Boole AI FAQ: Pricing, Privacy, and Technical Questions

Answers to common questions about Boole AI’s inference engine, pricing, data privacy, model availability, and OpenAI compatibility.

Answers to the most common questions about Boole AI — covering local deployment, the cloud API, pricing, data privacy, and model availability. If you don’t find what you’re looking for here, reach out to [e-mail].

General

What is Boole AI?

Boole AI is a high-throughput LLM inference platform. You can run models locally using the Boole binary on your own GPU, or call the cloud API hosted at https://api.boole.dev/v1. Both surfaces expose the same OpenAI-compatible interface, so switching between local and cloud requires only a base_url change.

Boole’s nightly compiler optimizes every supported model for maximum throughput — delivering, for example, 312 tokens/sec on Llama 3.3 70B on a single NVIDIA A10 GPU.

How does Boole achieve 312 tokens/sec?

Boole runs a nightly compiler pipeline called Crucible that performs three core optimizations automatically:

• Kernel fusion — combines adjacent GPU operations to eliminate memory round-trips between kernels.
• Weight quantization — reduces weight precision where quality impact is negligible, shrinking memory bandwidth requirements.
• KV cache paging — manages the attention cache in fixed-size pages to prevent memory fragmentation and enable larger effective batch sizes.

To date the compiler has run 2,057 iterations, retaining 217 of the best-performing optimization passes. The result compounds: throughput improves continuously without any changes on your end.

Is Boole OpenAI-compatible?

Yes. Boole exposes the same API surface as OpenAI — the same endpoints, request shapes, and response formats. To migrate an existing OpenAI integration, change two values:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.boole.dev/v1",  # was: https://api.openai.com/v1
    api_key="YOUR_BOOLE_API_KEY",          # was: your OpenAI key
)

No other code changes are required. Streaming, tool calls, and function calling are all supported.

How many models are available?

Boole currently supports 42 models, all optimized by the nightly compiler. The catalogue includes Llama 3.3 70B, Qwen 2.5 72B, Mixtral 8x22B, DeepSeek V3, Whisper Large v3, and many others.

See the full list and per-model specifications in Models.

---

Pricing & Billing

How much does it cost?

Local deployment is free — there are no per-token fees when you run Boole on your own GPU.

Cloud API pricing starts at $0.08 per million input tokens (DeepSeek V3). Example rates:

Model	Input	Output
DeepSeek V3	$0.08/MTok	$0.12/MTok
Llama 3.3 70B	$0.09/MTok	$0.14/MTok
Qwen 2.5 72B	$0.11/MTok	$0.16/MTok
Mixtral 8x22B	$0.13/MTok	$0.19/MTok
Whisper Large v3	$0.02/min	—

See the full pricing table at Pricing.

Do prices ever go up?

No — Boole’s prices only decrease over time. As the nightly compiler finds more efficient optimization passes, the cost to serve each token falls, and those savings are passed to you.

In the event of any price increase (which has not happened), Boole will provide a minimum of 30 days’ advance notice.

What are the free credits?

Every new account receives $20 in credits on signup — equivalent to approximately 1 million tokens on Llama 3.3 70B. No credit card is required to claim them.

Credits apply automatically to cloud API usage and expire after 12 months if unused.

Does local deployment cost anything?

No. When you run the Boole binary on your own hardware, there are no per-token fees, no subscription fees, and no license fees. You pay only for the GPU hardware and electricity — Boole charges nothing.

---

Privacy & Security

Are my prompts stored?

No. Boole operates a zero-retention policy: your prompts and completions are processed in memory to generate a response and are not written to disk or logged after the response is returned.

This applies to both the cloud API and the local binary.

Is Boole SOC 2 compliant?

Yes. Boole AI is SOC 2 Type II certified, covering security, availability, and confidentiality. All cloud API traffic is encrypted with TLS 1.3.

Contact [e-mail] to request a copy of the audit report or a security review questionnaire.

Can I run Boole completely offline?

Yes. The local binary is a single self-contained executable with weights embedded. Once downloaded, it requires no internet connection at inference time — no telemetry calls, no license checks, no external requests of any kind.

This makes Boole suitable for air-gapped environments and deployments with strict network egress policies.

Does the local binary phone home?

No. The local binary has zero telemetry by default. It does not send usage data, crash reports, or any other information to Boole’s servers. All inference happens entirely on your machine.

---

Technical

What GPU do I need for local deployment?

Any NVIDIA GPU with CUDA 12.0 or later is supported. Recommended hardware by model size:

Model size	Minimum VRAM	Recommended GPU
7B	8 GB	RTX 3080 / A10
13B	16 GB	RTX 4090 / A10
70B	40 GB	A100 / A10 (INT4)

The NVIDIA A10 (24 GB) can run 70B models with INT4 quantization enabled via --quantize int4. For full-precision 70B inference, use an A100 (40 GB or 80 GB).

What's the rate limit?

The default rate limit is 60 requests per second per API key on the cloud API. There is no rate limit on the local binary.

If your workload requires a higher cloud rate limit, contact [e-mail] with your account email and expected request volume.

Can I use streaming?

Yes. All text generation models support Server-Sent Events (SSE) streaming. Enable it by setting stream=True in your request:

stream = client.chat.completions.create(
    model="llama-3.3-70b-instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Streaming is supported on both the cloud API and the local binary.

Does Boole support tool calls / function calling?

Yes. Tool calls and function calling are fully supported on all chat models. The request and response format matches the OpenAI specification exactly, so any existing function-calling code works without modification.