Ground truth — marked by hand, shown in the sidebar, feeding training and the bake-off — Design Spec (#4947, #5404)
Milestone: source-model (the spec set’s one milestone; the training half’s issues sit in
remote-compute) Manual: TBD — a “Ground truth” section: what marking a line, a page or a reading as ground truth means, where the marked things are listed (the Ground Truth smart folders in the sidebar), and how they are used (training, the bake-off, measuring a model).Design-led (Testing Constitution). Status: DRAFT 2026-11-18 (the reviewer, from the owner’s direction of 2026-11-18; the owner approves). Facts read from
origin/integrationd01807cd12, file and line, nothing run; the inventory is inreviews/2026-11-18-runner-review-2.md§D. A COMPANION ofsource/readings-and-apparatus.md(readings and their makers),source/checking.md(verdicts),source/formats-and-training.md(training sets),compute/training-and-evaluation.md(what trains and how it is scored),ui/search.md(smart folders) andui/models-and-compute-window.md(the trained models’ rows). This file owns ONE thing: what ground truth IS in Fichero, how a person marks it, where it shows, and the one rule every consumer reads. Tags: [OK] built and pinned · [PARTIAL] exists in part · [GAP] designed, not built · [BROKEN] built and contradicting the line. Every non-[OK] line cites an issue.
Intent (the design)
A person says “this is right” once, on the thing they checked (a line’s reading, a region, a whole page’s pass), and Fichero remembers it as ground truth: it is listed in the sidebar under Ground Truth, by kind (lines, regions, pages, names, statements), so the person always knows how much truth the project holds; every training set, every bake-off and every measurement reads the SAME rule for what counts; a model never teaches on it as if it were its own, and a model’s verdict never overrules it. Imported transcriptions (PAGE, ALTO, TEI) are not truth until a person says so.
Prior art
- Transkribus and eScriptorium: a page or line has a “ground truth” status set by a person; training sets are built only from those; the count of GT pages is shown per document. We adopt the explicit mark and the per-kind count.
- HTR-United and the open HTR corpora: ground truth is a dataset with provenance and a licence; we keep the
provenance (
human, the person, the date) on the mark and export it as such (source/formats-and-training.md). - Label Studio / Prodigy: “accepted” annotations are a filter, not a copy. The sidebar’s Ground Truth folders are smart folders (a saved search), never a second store.
What exists today (read 2026-11-18)
- The rule.
models/segments.py:666-673made_by_a_person: a pass counts whenprovenance_kind == "human"orSegmentPass.ground_truthis true. The bake-off (recipes/bakeoff.py:310-346), the evaluation (training/evaluation.py:288-312,checked_pass,trust_of) and the person training set (training/kraken_set.py:318, teacherPERSON) all read it. - The mark.
PUT /api/segments/passes/{pass_id}/ground-truth→segment.pass_ground_truth(api/routes/document/segments.py:1185-1201, undoable): a whole PASS of a page. The app reads the flag (fichero/fichero/Models/Segment.swift:315-316) but offers no control to set it. - Corrections.
POST /api/segments/{id}/correct(api/routes/document/segment_readings.py:992) makes a human reading;POST .../readings/choice(:956-960) chooses one, a person only. A corrected or chosen line counts (kraken_set.py:41-46). - Verdicts.
checking/verdicts.py:18-30: a person’s verdict outranks a model’s; a model’s reject leaves a line out of a set; a passing line gets no verdict. - Not truth. Imported passes keep the file’s provenance (
segments.py:1197-1199); the markup check ✓/✓✓/✓✓✓ (markup.check.cycle) is a rating, read by no training set; a teacher pick excludeshumanpasses (recipes/train_offers.py:207-212). - Nothing lists it. No saved search, smart folder or sidebar node filters on ground truth or provenance
(
ui/search.mdsearch.smart-folders-as-sidebar-folders[PARTIAL] #4114; the search routes and the Swift sidebar builders have no such filter).
The design
- One mark, three grains. Ground truth is marked on a reading (a line: “this reading is right”), on a region or
table segment (“this shape is right”), or on a page’s pass (“every line here is right”). The line mark is the one
people use; the page mark is a shorthand for every line of the pass; the region mark is for layout training.
source.reading.ground-truth-is-a-reading([GAP] #4947) is this line grain. - One rule, one function.
made_by_a_persongrows to the reading grain and is the ONLY rule: the bake-off, the evaluation, every training-set builder and the sidebar’s smart folders call it. A model never writes it. - Shown in the sidebar. A Ground Truth group under the project holds one smart folder per kind: Lines, Regions,
Pages, Names, Statements, each with its count; opening one lists the marked things (the segment list for lines and
regions, the pages list for pages, the entity and claim tables for names and statements). They are saved searches with
a
ground_truthfilter, built in by the engine for every project (search.smart-folders-as-sidebar-folders). - Marking by hand, where the person is. The Reader’s and the segment editor’s line verbs gain “Mark as Ground
Truth” (and its undo, “Not Ground Truth”); the page’s context menu “Mark This Page’s Reading as Ground Truth”; a
region’s menu the same; a chord through the command table. Correcting a line marks it (a correction IS the person’s
word,
source.reading.correct-a-line); choosing a reading marks it; a markup check ✓✓✓ does NOT (it stays a rating, unless the owner rules otherwise: open question 1). - Feeding training and measuring. A training set’s “person” teacher is the ground-truth set, by kind: lines →
Kraken reader and vision LoRA; regions → YOLO; names → the names tagger (when one exists, #5536); statements → the
statement grammar (later). The bake-off’s reference and the evaluation’s held-out pages are drawn from the same set;
a model is never measured on lines it trained on (
training.set.held-out-never-trained). The Models and Compute window’s Trained here rows say how much ground truth each model was trained on (models.ground-truth.counts). - Honest counts. A smart folder’s count is the count the training offer uses (“Train on your 312 corrected lines”,
recipes/train_offers.py), so the person sees the same number in the sidebar and in the Train offer.
Behaviours
truth.rule.one-function— [PARTIAL] (#4947, #5404; built:made_by_a_personfor a pass, read by the bake-off, the evaluation and the person training set; not built: the reading and region grains, the sidebar reading it) every consumer of “is this right” callsmade_by_a_person(or its reading-grain sibling); a grep of the engine for a second rule finds none. Test:fichero-server/tests/unit/recipes/test_import_ground_truth.py,fichero-server/tests/unit/training/test_train_on_the_persons_corrections_5700.py(extend: the three grains, one rule).truth.mark.on-a-reading— [GAP] (#4947;source.reading.ground-truth-is-a-readingowns the data model) a person marks a reading as ground truth (PUT /api/segments/{id}/readings/{reading_id}/ground-truth, an undoable action with the person and the time), and a correction or a chosen reading is marked by that act. Test (to write): marking a reading makes the line count for the person set; a model’s write never sets the flag.truth.mark.on-a-region— [GAP] (#5525) a region or table segment is marked ground truth the same way; the YOLO set (training/yolo_set.pypersons_regions) reads the mark beside provenancehuman.truth.mark.on-a-page— [PARTIAL] (#4947; built:segment.pass_ground_truth, the route and the undo; not built: the app’s control) the page’s pass mark exists and the Reader’s page menu offers it.truth.mark.imports-are-not-truth— [OK] (#4947) an imported PAGE, ALTO or TEI pass keeps its file’s provenance and counts as truth only once a person marks it; the bake-off names the unmarked imports (recipes/bakeoff.py:324-333unmarked_import_pages). Test:fichero-server/tests/unit/recipes/test_import_ground_truth.py.truth.mark.never-by-a-model— [OK] (#5404;source.check.model-never-a-personowns it) no model call, check or recipe step sets the mark; a model’s verdict never outranks a person’s (checking/verdicts.py:18-21). Test:fichero-server/tests/unit/training/test_person_reject_outranks_model_confirm_6327.py.truth.sidebar.smart-folders-per-kind— [GAP] (#4114;search.smart-folders-as-sidebar-foldersowns the mechanism) every project has a Ground Truth group with Lines, Regions, Pages, Names and Statements smart folders, each badged with its count, each a saved search with theground_truthfilter the engine seeds; opening one lists the marked things in the kind’s own view. Test (to write): a seeded project with three marked lines shows Lines (3); marking a fourth updates the count through the change stream.truth.sidebar.count-is-the-offers-count— [GAP] (#5440) the Lines folder’s count equals the number the Train offer names and the bake-off’s “N corrected lines on M pages” (recipes/bakeoff.pyMIN_LINES,MIN_PAGES).truth.feeds.training-and-the-bake-off— [PARTIAL] (#4947, #5894; built: the person set for Kraken, the bake-off’s reference, the evaluation’s held-out pages; not built: the bake-off’s reads as training lines (compute.tune.bakeoff-reads-become-training-lines[GAP]), names and statements sets) every training set’s person teacher and every measurement’s reference are the ground-truth set of the kind, by the one rule, held-out pages never trained on. Test:fichero-server/tests/unit/training/test_train_on_the_persons_corrections_5700.py,fichero-server/tests/unit/training/test_bakeoff_dataset_5894.py.truth.export.carries-provenance— [GAP] (#4946, #4947;source/formats-and-training.mdsource.train.ground-truthowns the format) an exported training set or PAGE XML says which lines are ground truth and by whom.
Build order
Stage 1 — The mark and the rule
Step 1.1 — Ground truth on a reading and a region; one rule for three grains (#4947)
- Behaviours:
truth.rule.one-function,truth.mark.on-a-reading,truth.mark.on-a-region,truth.mark.on-a-page - Tests: fichero-server/tests/unit/recipes/test_import_ground_truth.py (extended; the reading-grain route’s test to write beside it)
- Size: M
Stage 2 — The sidebar
Step 2.1 — Ground Truth smart folders per kind, with counts (#4114)
- Behaviours:
truth.sidebar.smart-folders-per-kind,truth.sidebar.count-is-the-offers-count - Tests: fichero-server/tests/unit/recipes/test_import_ground_truth.py (the counts the folders show; the seeded saved searches’ test to write in fichero-server/tests/unit/api/)
- Depends on: 1.1
- Size: M
Stage 3 — Feeding training, the bake-off and the export
Step 3.1 — One set feeds every trainer and measure; exports carry it (#5894, #4946)
- Behaviours:
truth.feeds.training-and-the-bake-off,truth.export.carries-provenance - Tests: fichero-server/tests/unit/training/test_train_on_the_persons_corrections_5700.py (extended), fichero-server/tests/unit/training/test_bakeoff_dataset_5894.py, fichero-server/tests/unit/remote_read/test_read_package.py
- Depends on: 1.1
- Size: M
Test matrix
| Leg | This surface? | Pins | File |
|---|---|---|---|
| Pure rule (Swift) | y | the Mark as Ground Truth verbs enable on a reading, a region, a page; the folders’ counts render as served | fichero/Tests/Unit/general/Views/Sidebar/ (to write) |
| Backend (pytest) | y | one rule, three grains; imports are not truth; a model never marks; the seeded smart folders and their counts; the person set equals the marked set | the files named above |
| MCP | y | fichero_segment_readings carries the mark; the training set preview counts it |
fichero-mcp/tests/test_mcp_generated.py |
| CLI | n | — | — |
Open questions for the owner
- Does ✓✓✓ (the markup check’s third state) mark ground truth? Recommended: no; it stays a reading aid, and “Mark as Ground Truth” is its own verb, so a count in the sidebar is never a guess.
- Names and statements: is a confirmed entity or an accepted claim ground truth for the tagger and the grammar, or only for the knowledge graph? Recommended: yes, the same mark, so #5536’s names tagger has a set the day it exists.