Troubleshoot Boole AI: Startup, Auth, and Performance

Solutions for the most common Boole AI issues — local binary startup failures, API authentication errors, rate limits, and performance problems.

This guide covers the most common issues customers encounter with Boole AI, including local binary startup problems, API authentication errors, and performance troubleshooting. Work through the relevant section below to resolve your issue quickly. If nothing here applies, reach out to [e-mail].

Local Binary Issues

The binary won't start

If the binary exits immediately or fails to initialize, check the following:

• CUDA driver version — Boole requires CUDA 12.0 or later. Run nvidia-smi and confirm the driver version shown in the top-right corner meets this minimum. If not, update your NVIDIA driver.

• GPU visibility — confirm your GPU is visible to the system:

  nvidia-smi
  

  If this command fails or shows no devices, your GPU driver is not installed correctly or the GPU is not accessible in your current environment (e.g. a container without --gpus all).

• Disk space — weights are embedded in the binary and staged to a local cache on first run. Ensure you have enough free disk space for the model you’re loading. Llama 3.3 70B requires approximately 40 GB.

Slow cold start / high latency

Cold start time is expected to be approximately 380 ms under normal conditions. On the very first invocation after a reboot or cache clear, the runtime must page weights from disk into GPU memory, which adds roughly 104 ms for weight page-in. Subsequent requests within the same session are significantly faster once weights are resident in GPU memory.

If cold start consistently exceeds 400 ms, check:

• Storage throughput — NVMe drives produce the fastest page-in times.
• Other processes competing for GPU memory, which can force weights to be evicted and reloaded.
• Whether you’re running on a network-mounted filesystem, which can substantially slow weight reads.

Port already in use

By default, the local server binds to port 8000. If another process is already using that port, startup will fail with an address already in use error.

Use the --port flag to specify a different port:

boole serve --model llama-3.3-70b-instruct --port 8001

Then update your client’s base_url accordingly:

base_url="http://localhost:8001/v1"

Out of GPU memory

70B-parameter models require approximately 40 GB of VRAM. If your GPU has less memory available, the runtime will fail with an out-of-memory error during weight loading.

Two options:

1. Switch to a smaller model — 7B and 13B models fit comfortably on GPUs with 16–24 GB VRAM.

2. Enable INT4 quantization — reduces VRAM usage significantly at a small quality trade-off:

   boole serve --model llama-3.3-70b-instruct --quantize int4
   

INT4 quantization is applied by the nightly compiler and is optimized for throughput. Quality degradation is minimal on most task types.

---

API Authentication Issues

401 Unauthorized

A 401 response means the server could not authenticate your request. Check the following:

• Header format — Boole requires the standard Bearer token format. Ensure your request includes:

  Authorization: Bearer YOUR_API_KEY
  

  Do not use API-Key, X-API-Key, or any other header name.

• Key validity — verify the key has not been revoked. Open Settings → API Keys in the Boole dashboard to confirm its status.

• No extra whitespace — copy your key carefully; a trailing space or newline will cause authentication to fail.

API key not working

If your key appears valid but requests still fail, confirm:

• Base URL — you must use https://api.boole.dev/v1 exactly. A common mistake is omitting /v1 or using a different subdomain.

  from openai import OpenAI
  
  client = OpenAI(
      base_url="https://api.boole.dev/v1",
      api_key="YOUR_BOOLE_API_KEY",
  )
  

• Environment mismatch — if you have keys for multiple accounts or environments, ensure you’re using the key that corresponds to your active account.

Key was working, now returning 401

If a key that previously worked is now returning 401, it has likely been revoked — either manually or as a result of a security rotation.

To generate a replacement key:

1. Open Settings

   Navigate to Settings → API Keys in the Boole dashboard.

2. Revoke the old key

   Confirm the original key shows a Revoked status.

3. Create a new key

   Click New API Key, copy the value immediately (it is only shown once), and update your application configuration.

---

Performance Issues

High latency / slow responses

If you’re seeing unexpectedly high latency on the cloud API, first check for active incidents:

status.booleinference.com

Expected time-to-first-token (TTFT) for Llama 3.3 70B is approximately 62 ms under normal load. If you’re consistently seeing higher values and the status page shows no incidents, include your account email, the model you’re using, and example request timestamps when contacting [e-mail].

Getting 429 errors

A 429 Too Many Requests response means you’ve exceeded the default rate limit of 60 requests per second per API key.

Recommended mitigations:

• Implement exponential backoff — retry failed requests with increasing delays:

  import time
  import random
  
  def request_with_backoff(client, **kwargs):
      for attempt in range(5):
          try:
              return client.chat.completions.create(**kwargs)
          except Exception as e:
              if "429" in str(e):
                  wait = (2 ** attempt) + random.uniform(0, 1)
                  time.sleep(wait)
              else:
                  raise
      raise RuntimeError("Max retries exceeded")
  

• Raise your limit — if your workload legitimately requires more than 60 req/sec, contact [e-mail] to request a higher limit.

---

Checking Service Status

Visit status.booleinference.com for real-time service status, scheduled maintenance windows, and a full history of past incidents. You can subscribe to status updates to receive email or webhook notifications whenever an incident is opened or resolved.

If you’re experiencing an issue not covered here, contact [e-mail] with your account email and a description of the problem. Include any error messages and the model you’re using to help us respond faster.