Checking retrieval
A validated worksheet says which documents answer each question.
check-retrieval checks that a RAG database actually returns them: it binds
the question set to one ingestion, as bind does, then runs one search per
question against a haiku-rag database,
and looks for the bound documents among the results.
There is no answering model and no judge, so a run is fast, and its scores are stable from run to run. That's what checking a freshly ingested database needs. It checks documents, not passages: a hit means a chunk of a relevant document came back, not that the chunk holds the answer.
Running a check
exquisite check-retrieval --root data/nist-sp-800 --corpus sp800 \
--questions sp800 --ingestion sp800 \
--db sp800.lancedb --config haiku.rag.yaml
--dbis the database to search: a local path, or a URI haiku-rag accepts. It is opened read-only. A local path with nothing there is refused, never opened, since opening it would create an empty database.--configis the haiku-rag configuration the database was ingested with, so the searches use the same embedder. Any databases it names are set aside in favour of--db, and the command says so.--top-kis how many results each search returns (default 30). It is deliberately more than a chat agent's search limit: a newly ingested document relevant to a question can push a labelled one out of a short list without anything being broken.-v/--verboseshows each question's result, as well as the summary.--outsaves the results to a file, to compare a later run against. An existing file is refused, before any searching, unless--forceis passed too. Without--out, nothing is written.
Binding problems are reported on stderr, as bind reports them. A
worksheet that isn't ready to bind is refused.
What counts as a hit
A question is a hit when any of its bound documents is among the results,
and a miss when none is. Each question is scored by retrieval_mrr, the
reciprocal rank of its first relevant document: 1.0 at rank 1, 0.5 at rank
2, and 0.0 for a miss. Questions with no bound documents (unresolved, or
citing nothing) are ineligible: counted, not checked.
Results are matched to the manifest the way bind matches documents: by the
sha256 the database records for each document, then its source_url, and
only failing both by URI. So a database whose documents sit at different
paths from the manifest's is still checked correctly.
With --verbose, each question gets a line first, in question-set order:
the rank of its first relevant document, its retrieval_mrr, and the
question. Below a question not at rank 1 are the documents that came ahead
of it. Below a miss are the first few that came back instead:
rank mrr question
1 1.000 What are the steps of a risk assessment?
...
2 0.500 What levels of potential impact are used to categorize information and systems?
after: NIST.SP.800-53r4.pdf
...
20/20 questions passed; mean retrieval_mrr 0.975
Documents are shown by name; the results file has their full URIs.
Comparing with an earlier run
--compare names the results of an earlier run. Against it, a question
fails when it loses: the earlier run hit, and this one misses. The rest
is reported, but never fails:
RANK: the first relevant document ranks lower than before. The embedder and reranker can reorder near-tied results between two runs on the same database.DROPPED: a relevant document dropped out while another still hits.- the mean
retrieval_mrrover the questions both runs scored fell by more than--mrr-tolerance(default 0.02).
A question the earlier run doesn't have is checked for a miss instead.
The earlier run is refused, before any searching, when its searches differ
by design: a different configured embedder, reranker or top K, or a database
stored with a different embedder. So is one that asked a different question
under some case's key, the question's metadata.uuid. Different versions of
exquisite or haiku-rag only warn: a check across an upgrade is meant to
span them.
Any run saved with --out can be a reference, including one that failed:
read its report, then compare the next run against it.
Exit status
check-retrieval exits with status 0 when every eligible question passes.
It exits with status 1 when:
- a question misses, or loses against
--compare; - no question is eligible;
- the worksheet isn't ready to bind;
- the database is missing, or haiku-rag won't search it as it stands (it needs migrating to the installed haiku-rag, say);
- the
--comparefile isn't a results file, or is refused; - the
--outfile exists, and--forcewasn't passed.
The results file
With --out, a run that searches saves its results, whatever its outcome:
{
"format": "exquisite check-retrieval 1",
"question_set": "questions/sp800.yaml",
"corpus": "sp800",
"ingestion": "sp800",
"settings": {
"embedder": "vllm:nvidia/llama-nemotron-embed-vl-1b-v2 (dim 2048)",
"reranker": null,
"top_k": 30,
"config_hash": "5f8a9bfe…"
},
"database": {
"location": "sp800.lancedb",
"embedder": "vllm:nvidia/llama-nemotron-embed-vl-1b-v2 (dim 2048)",
"documents": 8,
"chunks": 5502,
"written_at": "2026-10-03T01:54:08.941536",
"version": "0.89.0"
},
"substrate": {"exquisite": "0.2", "haiku-rag-slim": "0.89.0"},
"started": "2026-10-03T08:58:41",
"finished": "2026-10-03T08:58:45",
"cases": [
{
"key": "33e57977-…",
"question": "What levels of potential impact are used to categorize information and systems?",
"relevant": ["file:///data/nist-sp-800/nistspecialpublication800-60v1r1.pdf"],
"retrieved": [
"file:///data/nist-sp-800/NIST.SP.800-53r4.pdf",
"file:///data/nist-sp-800/nistspecialpublication800-60v1r1.pdf"
],
"score": 0.5
}
]
}
settings: how the searches were made.config_hashis a SHA-256 of the whole haiku-rag configuration.database: what the database held: the embedder it was stored with, its size, when it was last written, and its haiku-rag version.substrate: the installed versions ofexquisiteand haiku-rag.cases: one per question.retrievedlists one entry per distinct document, in rank order: its manifest URI when the manifest holds it, else the database's own.scoreisnullfor an ineligible question.
It is the file --compare reads.