Boole AI Local Quickstart: GPU Inference in Minutes

Download the Boole binary, start serving a model on your NVIDIA GPU, and get your first tokens in under 4 minutes — no Python or dependencies needed.

This guide walks you through downloading the Boole binary, starting a local inference server, and making your first request — all in under four minutes. Because the binary ships with weights embedded and has no external dependencies, setup is a single download followed by a single command. Once the server is running, it exposes the same OpenAI-compatible API as the cloud, so any existing code works without modification.

Prerequisites

• An NVIDIA GPU with CUDA drivers installed (driver version 520 or later recommended)
• Linux (x86_64) or macOS (Apple Silicon)
• Approximately 50 GB of free disk space for 70B-parameter models; smaller models require significantly less

Steps

1. Download the binary

   Download the latest Boole binary for your platform. Always check booleinference.com for the most current release before downloading.

   curl -L -o boole-latest-linux-x86_64 \
     https://booleinference.com/download/boole-latest-linux-x86_64
   

   Visit booleinference.com to find the latest release URL for your platform, including macOS (Apple Silicon) builds.

2. Make the binary executable

   Grant execute permissions so your shell can run the file directly.

   chmod +x boole-latest-linux-x86_64
   
3. Start the inference server

   Launch the server with the model you want to serve. Boole will load the embedded weights and begin accepting requests.

   ./boole-latest-linux-x86_64 serve --model llama-3.3-70b-instruct
   

   Cold start is under 400 ms. Once you see Listening on http://localhost:8000, the server is ready. On a single NVIDIA A10, Llama 3.3 70B serves at 312 tokens/sec.

4. Test with curl

   Send a chat completion request to confirm everything is working.

   curl http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "llama-3.3-70b-instruct",
       "messages": [{"role": "user", "content": "Hello!"}]
     }'
   

   You should receive a JSON response with the model’s reply in choices[0].message.content.

Using with Existing Code

Because the local server is fully OpenAI-compatible, you can point any OpenAI SDK client at http://localhost:8000/v1 with no other changes. Pass any non-empty string as the API key — the local server does not validate it.

app.py

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="local",  # any non-empty string
)

response = client.chat.completions.create(
    model="llama-3.3-70b-instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

Boole collects zero telemetry and stores no prompts. After the initial download, the binary and its embedded weights run entirely offline — ideal for air-gapped environments or privacy-sensitive workloads.

Next Steps

Advanced Local Configuration

Configure ports, concurrency limits, model caching paths, and multi-GPU setups.

Available Models

Browse all 42 supported models — from Llama and Qwen to Whisper — and check hardware requirements for each.