How we measured this

The impact benchmark, with the receipts.

Numbers on marketing pages are usually unfalsifiable. Ours come from a pre-registered experiment you can read, question, and watch us re-run: the same agent doing the same real work, with the only difference being whether it can reach the company's knowledge. This page explains exactly what we did, what we found, and what we did not find.

The setup

  1. A direct A/B at the agent. Every task runs twice under identical conditions: once with the agent connected to Firmament through the same interface customers use, once without. Same model, same tools, same working directory. The arm without Firmament is not handicapped: it keeps the repo, the docs, and everything a normal contributor has.
  2. Real tasks, real knowledge. Every task derives from a recorded fact or incident in our own company's history: setup recipes, API gotchas, deploy workflows, migration conventions, real outages. The knowledge base is our own production wiki, loaded through the real product pipeline, noise and all. Nothing was written for the test.
  3. Grading we cannot flatter. Pass or fail comes from real compile-and-test runs, checkable file states, and exact recorded answers. No AI judges. No human scoring.
  4. Fairness gates, enforced in CI. Before a task counts, machinery proves the needed fact is in the knowledge base, is absent from everything the agent can see, that a competent solution ignorant of the fact fails, and that a knowing solution passes.
  5. Controls that can void the run. The set includes tasks where company knowledge should not help. If the knowledge arm scores better on those, the run is declared invalid, automatically. This is not hypothetical: our pilot run voided itself this way and forced design fixes before any number could count. In the final run, both arms scored 100% on controls.
  6. Pre-registered, re-runnable. The hypotheses, metrics, grading rules, and the bar for publishing were written down before the first run. One command re-runs the whole benchmark against the current build, so these numbers cannot silently go stale as the product changes.

The results

2% → 98%
correctness on real company work

Without Firmament, a premium model completed 1 of 52 attempts and a budget model 0 of 54. With Firmament, the budget model completed 53 of 54 and the premium model 54 of 54.

~40×
cheaper per completed task

The premium model without knowledge spent $0.26 per completed task, because it almost never completed one. The budget model with Firmament spent about $0.003. Bootstrap 90% confidence interval on the cost factor: 9× to 59×.

72–88%
of lessons compound to the next agent

An agent learns a fact, saves it through the real pipeline, and a different agent on a different model succeeds with it: 72% of pairs when the budget model taught the premium one, 88% the other way. Without shared knowledge: 0%. Nearly every miss was the first agent not saving what it learned, not the pipeline losing it.

Two findings behind the headline numbers are worth knowing. First, the premium model without knowledge did slightly worse than the budget model without knowledge, while using half the turns: bigger models fail more confidently, they do not know more about your company. Second, the harder the task, the wider the gap: on the most complex multi-step tasks, agents without Firmament scored zero across every attempt.

What it does not do

On control tasks any competent model already handles, Firmament adds roughly nothing, by design. A knowledge layer does not make a model smarter at generic reasoning; it makes it knowledgeable about your company. We also caught a real failure mode during the pilot: an agent asking a vague question got back knowledge about a different repository and trusted it over the file in front of it. That control failure voided the pilot, led to fixes, and did not recur in the final run. We keep publishing this kind of miss because a benchmark that cannot catch problems is not evidence.

Scope, honestly

Current scope: 30 frozen tasks, of which 21 have completed the full grid at three repeats per cell (the remainder was interrupted by an infrastructure issue and is scheduled, along with a second organization's knowledge base, additional agent products, and a premium-model pair). Models tested so far: two tiers of one model family. All numbers above come only from completed, validated cells, and every published figure updates when the benchmark re-runs on each release.

Start with FirmamentBack to the overview →