Question sets
A question set is a pydantic-evals
dataset, stored under questions/ as YAML (<name>.yaml) or JSON
(<name>.json). Should both exist for one name, the YAML file is used.
exquisite reads two keys from each case's metadata:
cases:
- inputs: What does clause 4 of ISO 9001 cover?
expected_output: Context of the organization
metadata:
uuid: a1
reference: ISO 9001 Sec 4
Every case must have metadata: without it there is nothing to tie the
question to a worksheet, so a set holding such a case is refused when it is
loaded.
referenceis the source citation as written.exquisitenever modifies it: it is the provenance, and what a worksheet is regenerated from when a corpus is rebuilt. A reference may also be a document URI, for a set whose author picked the files directly (see references that are already URIs).uuididentifies the case. A pruned set, whose name ends in_pruned, is a subset of the set it is named for. It needs no worksheet of its own: binding labels it from its parent's worksheet, matching cases byuuid.
The question set itself is never written to. Binding produces a labelled copy on demand (see Binding).