How the Boole AI Inference Engine Works Under the Hood

Boole’s nightly compiler runs 2,057 iterations of kernel fusion, quantization, and KV cache paging to maximize throughput while quality never regresses.

Boole’s inference engine is not a static runtime — it is continuously optimized by an automated compiler agent that runs every night. On each run, the agent proposes a single change, measures its effect against a frozen quality benchmark, and either keeps or reverts the change. The result is an engine that gets faster over time while guaranteeing that quality never drops below baseline.

The Optimization Loop

Each nightly run tests exactly one hypothesis: a kernel fusion strategy, a quantization scheme, a KV cache paging granularity, or a batching policy. By limiting each iteration to a single variable, the compiler can attribute any quality or throughput change to a known cause and make a confident keep-or-revert decision.

To date, the compiler has run 2,057 iterations and kept 217 of them — producing a cumulative speedup of 89% over the original baseline.

The table below shows the most recent iterations from the crucible log:

Iteration	Change	Quality Delta	Status
#2053	Page KV cache at 64-token granularity	+0.0%	Kept
#2054	INT4 weights on layers 0–12	-1.9%	Reverted
#2055	Speculative draft depth 5 → 7	-0.1%	Reverted
#2056	Batch admission by token budget	+0.0%	Kept
#2057	Ternary kernel for MLP down-proj	-4.2%	Reverted

Most iterations are reverted. That is by design — the compiler is aggressive in what it tries and conservative in what it keeps.

Cold Start Performance

The optimization loop has also reduced cold start latency significantly. Cold start time dropped from 1.04 s to 380 ms over the compiler’s history. The current budget breaks down as follows:

Phase	Latency
Scheduler admit	24 ms
Snapshot restore	137 ms
Weight page-in	104 ms
First token	92 ms
Total	380 ms

Each phase has been independently targeted by the compiler. Snapshot restore and weight page-in together account for the largest share and remain active areas of optimization.

Quality Gate

Every compiler iteration is evaluated against a frozen benchmark suite — a fixed set of prompts and reference outputs that does not change between runs. A proposed change must not decrease quality on any benchmark below a fixed threshold. If it does, the change is automatically reverted and the current binary remains in production.

This means the binary you download is always at or above baseline quality. You never receive a regression silently; the gate enforces that invariant on every single iteration.

What This Means for You

The compiler loop has direct consequences for how you use Boole:

• The local binary you download today is the same binary Boole runs in production. There is no separate “production build.” The nightly compiler updates both simultaneously.
• Throughput improves over time without any action on your part. Download a new binary periodically, or stay on the cloud API, and you receive improvements automatically.
• Cloud API pricing decreases as the compiler finds new efficiencies. Prices never increase as a result of compiler changes — efficiency gains are passed on to you as lower per-token rates.
• The full optimization log is public. Every iteration, its proposed change, quality delta, and keep/revert decision is published so you can audit every improvement in the engine.

Every benchmark is reproducible. The full crucible log is published at github.com/boole-ai/crucible.