Skip to content

Question sets

A question set is a pydantic-evals dataset, stored under questions/ as YAML (<name>.yaml) or JSON (<name>.json). Should both exist for one name, the YAML file is used. exquisite reads two keys from each case's metadata:

cases:
- inputs: What does clause 4 of ISO 9001 cover?
  expected_output: Context of the organization
  metadata:
    uuid: a1
    reference: ISO 9001 Sec 4

Every case must have metadata: without it there is nothing to tie the question to a worksheet, so a set holding such a case is refused when it is loaded.

  • reference is the source citation as written. exquisite never modifies it: it is the provenance, and what a worksheet is regenerated from when a corpus is rebuilt. A reference may also be a document URI, for a set whose author picked the files directly (see references that are already URIs).
  • uuid identifies the case. A pruned set, whose name ends in _pruned, is a subset of the set it is named for. It needs no worksheet of its own: binding labels it from its parent's worksheet, matching cases by uuid.

The question set itself is never written to. Binding produces a labelled copy on demand (see Binding).