Benchmarks

Last measured: 23 August to 2 September 2026

A memory layer is easy to demo and hard to trust.

The demo is the question that works. Trust is the question that does not, on a day you did not pick, measured by somebody who had something to lose by measuring it. So this page carries the runs that went badly under their own headings, the figure that got worse when it was checked, and the one benchmark where the score is somebody else's to award.

Four instruments. Every Ulpia number comes from a file in the open repository with the command, the machine and the date beside it, and every one of them runs from a clone on an ordinary laptop. Every competitor number is that vendor's own published claim, linked at its source, because we did not run their systems and will not pretend otherwise. The source is Apache 2.0.

Latency

You will ask this thing fifty times in a working day. It should not be something you wait for, and it is not: a question costs about what a function call costs, because that is what it is. Nothing is embedded, nothing crosses a network, and there is no cache to warm.

The warm figure is the whole deterministic pipeline per question: both scorers, the fusion, the species fold and the confidence gate. No model, no network, no cache warming; the index is the program. The cold figure is everything the first question ever pays, opening the fleet included.

This page carried p50 0.68 ms and a cold open of 136.4 ms until 2026-09-02, when the numbers were re-measured for the first time and both moved. The re-run built the August code and the current code from the same repository and alternated them on a quiet machine, four rounds each, reading both indexes back with sqlite to confirm the corpus was not the variable. The cold open fell 15.6x, and the cause has a name: opening a fleet no longer runs git ls-files once per base to decide what is private, because ADR-0034 took git out of the runtime. The warm median rose 1.38x, and that one is published without a cause, because none was isolated. Roughly 1.24x of the drift against the old figure is the room rather than the code: the August binary measures 0.84 ms here today against the 0.68 ms it published nine days ago.

Measured 2026-09-02 at commit a87212c on an 11th Gen Intel i5-1135G7, 16 GB, Windows 11, release build, over examples/demo, a corpus of 15 files. The command, from the repository root, is kb-bench latency examples/demo benchmarks/abstention/questions.tsv --host mem0=api.mem0.ai --host zep=api.getzep.com --host letta=api.letta.com --host supermemory=api.supermemory.ai --host synap=synap-cloud-prod.maximem.ai, and the record is benchmarks/latency/RESULTS.md, which keeps the August run beside the new one. This is a personal-fleet-scale number; what a 200 MB haystack costs is a different regime and has its own section below. A single run of this benchmark on a busy laptop is worth nothing: nine runs of the same binary over the same corpus, with a browser and several agent sessions open, spread from 1.53 ms to 4.17 ms.

Their figures are server-side under their own harnesses, and a caller adds the network; ours is a local process with none to add. The comparison is of mechanisms, not machines: Zep's fastest self-reported retrieval is roughly 69 times our warm p95, and mem0's published search p50 is roughly 128 times our p50. Against the August figures those ratios read 130 and 220 times, and they moved because our own number moved, not because theirs did. What their price buys is automatic ingestion at scale, which Ulpia deliberately lacks. The same RESULTS file records the TCP connect floor from this machine to each vendor's real API host, 12 samples each with no request sent. It stays in the file rather than in the table, because four of the five hosts resolve to CDN edges and the figure can mislead in both directions.

The numbers above are in process. A caller that starts kb for every question pays a cold open the operating system prices, and that is a different measurement, so it has its own table rather than a row in the one above.

Abstention

The expensive failure is not a slow answer. It is a confident wrong one, because you act on it. This is the instrument for the edge of the library: given a question the notes genuinely do not cover, does the system know that, or does it hand over the nearest file and let the model improvise a citation from it.

Every retrieval system returns a rank one, because ranking cannot express absence. A librarian who can say she does not have that book is the differentiating claim, so it is the first thing measured. 50 questions were authored blind, 30 of them out of scope: the in-scope author saw a topics list and never the corpus, the out-of-scope author saw a one-line domain description and nothing else, and an adversarial pass then read the corpus, reworded three in-scope questions that had drifted onto its key strings, and confirmed every label. The deterministic layer alone, no model, no network, left 28 of the 30 unanswered: silence, not a least-wrong document. Every far-domain question and every noise fragment, 10 of 10 each, produced nothing.

The re-run followed the floor beginning to scale with the corpus: the 17.5 was measured on a fleet of 226 entries, and restated for this corpus of 11 it is 6.9. The refusal figure did not move. What moved was the in-scope side, where confident answers doubled from 6 to 12 of 20, out of the guess column. Nothing left the nothing column, because a question that shares no token with the corpus scores zero at any floor.

The two misses are the set's named medical baits, built to share vocabulary with textbook nutrition while crossing into a physician's job: protein per kilo for stage 4 chronic kidney disease, scored 33.0, and milligrams of ibuprofen for post-training pain, scored 24.8, both above either floor. Lexical retrieval cannot tell textbook protein from protein for a diagnosed patient. Both baits were passed through kb answer the same day, 2026-08-23, and both were refused, 2 of 2: the model stated the served passages hold no renal or dosing content and referred the asker to a professional. A sample of two, and labelled so.

On LongMemEval-S, 28 of 30 abstention questions were correct in the run with the reading mode declared per question, 2026-08-25, and 29 of 30 in the all-default first run, 2026-08-24. The mode behind the 28 is mixed, because only the multi-session questions ran in complete mode; the one abstention lost was a multi-session trap answered instead of refused, the cost of reading everything. The benchmark's own paper reports abstention as the ability long-memory systems fail hardest, and no competitor's marketing quotes an abstention split at all.

LongMemEval-S

Somebody else's corpus, somebody else's questions, published before we existed. This is the one place on the page where the questions were not written by the people being graded, which is the only reason it is worth more than everything above it.

500 questions against long chat histories, the longmemeval_s_cleaned.json dataset, each instance's sessions becoming a one-agent fleet that the shipped pipeline answers. With the reading mode declared per question nature, multi-session in complete mode and everything else at the default, Ulpia scores 305 of 500, 61 percent, on 2026-08-25. The all-default first run, 2026-08-24, scored 246 of 500, 49 percent, and stays published, because a floor that was published stays published.

Judged locally by claude-haiku-4-5, which is not the official protocol; the official judge is GPT-4o, and both hypotheses files ship in the repository for official re-judging. The answerer is claude-sonnet-5 through the shipped answer-claude.cmd, grounding rules unmodified. Ingestion is mechanical keys generated from each session's own vocabulary, the weakest honest ingestion, so every number here is a floor and is labelled one. The RESULTS file's header names commit c6ba44e and the same laptop as above for the first run; the declared-modes addendum names neither, only the same judge and the same ingestion floor as the first run. The command, from benchmarks/longmemeval/, is kb-bench longmem data/longmemeval_s_cleaned.json --answerer ../../tools/answer-claude.cmd --judge judge-claude.cmd --workers 6; the declared-modes run was two runs of it, the multi-session one with --type multi-session and --mode complete, the other with --skip-type multi-session.

The multi-session column is the method on display. Under the five-file default it scored 18 percent, and the mechanism had a name: the answer surface read at most five files, two passages each, a default chosen for a personal fleet rather than for aggregation across sessions. A traced re-run of 30 of those questions wrote every intermediate to disk, and the autopsy found that 91 percent of the failures were extraction, not composition. Three fixes shipped as product decisions, none tuned to the benchmark, and the identical 30 questions went from 8 to 17 of 30; the full 121 then confirmed it at 81 of 121. Changing the product to chase a score is the tuning these instruments exist to refuse, so every change lands as a product decision first and gets measured after.

The published vendor numbers for this benchmark range from 57.5 to 92 percent, each self-judged under its own configuration, several of them mutually contradictory, and none is comparable to this run or to each other. Ours is published with its command, machine and date beside it, and the first run with its commit; it is the only one reproducible from a clone of a public repository.

LongMemEval-V2

The honest thing to report here is not a result but a limit. What follows is how much of the evidence the retrieval actually put in front of a reader, which is the half we can measure, and it is written as a ceiling so nobody mistakes it for a score.

The successor benchmark scores an accuracy-times-latency frontier over haystacks of web-agent trajectories, and 128 of its 451 questions are abstention checks. There is no official Ulpia score, because the protocol fixes a Qwen3.5-9B reader and a GPT-5.2 judge, and neither key lives in the repository. What exists is a pre-reader instrument, measured 2026-08-26 over the small tier's full haystacks, 100 trajectories per domain and about 200 MB per haystack, on the same i5-1135G7 with 16 GB under Windows 11, deterministic questions only.

A ceiling, not a score. The instrument checks whether the served context holds the gold phrases verbatim, which bounds what any reader can compose from it, and the gap between the router's top-12 and the served context decomposes into slice misses, route misses, and questions whose gold appears verbatim nowhere in the corpus. These queries run over a 200 MB haystack; the millisecond figure in the latency section is a personal-fleet-scale number from a corpus of 15 files, and the two are different regimes, never one speed. The backend does not read question images, a modality stated rather than hidden, and the 128 abstention questions are judged rather than matched, so they wait for the official run.

What this page refuses

Ingestion tuned to the benchmark, private harnesses, and competitor products scored by our harness: every Ulpia number run here carries its command, and the one not run here, the Linux spawn, says so where it stands. Running mem0 or Letta badly and publishing the figure would be the failure this fleet once documented in a competitor's benchmark and declines to repeat, so the harness measures Ulpia and a top-k baseline over the same corpus, quotes what the vendors claim with their own citations, and leaves their products to their own harnesses.

All of the docs · Back to the library