Archival Data Model — Development Plan

2026-09-19 — read this first. The shared ideas in this file (what a segment is, its slots, the delivery rule) now have one home: source-model.md and its slices. Where this file and that set differ, that set wins. This file stays as the STAGED PLAN, updated below (“How the phases map to the source model”).

Manual: TBD — the user manual needs a “How Fichero stores what it reads” section: what a segment is, how a page’s regions/lines/words relate, and why that shape lets a claim point back at the exact ink it came from. Written for a researcher, not a developer.

Status: DRAFT for discussion (Fichero creative director, 2026-09-08 night). This consolidates the segment/archival design captured across issues #4635–#4642 and #1755 into ONE staged plan to develop and then talk through. Design-led (Testing Constitution): nothing here is built until the creative director approves the slice. Epic index: #4639.

1. The two models and the one hinge

Everything below serves a dead-simple user model on top of a rich data model. They never leak into each other; the Profile is the only translator.

User model (what a person touches)

Document → Page → Profile → Transcription (in an Edition) → Entity/Claim → Source → Credit → Tags. A small vocabulary. Depth is disclosed only when a scholar reaches for it (progressive disclosure): a casual user does import → transcribe → read → search and never sees a segment, an RDF triple, or a provenance chain.

Data model (what is stored)

One primitive — the Segment — and every record hangs off it. Every record (Segment, Transcription, Representation, Claim, Entity, Annotation, Zone, Tag) carries the same four invariant slots:

Slot	Meaning
anchor	WHERE — geometry/time on which medium, + layer + reading order
language + script	WHAT system — coverage tier, direction
provenance + version + credit	HOW it got here — actor (model/human), reasoning, restorable history
export map	HOW it leaves — mapping to ALTO/PageXML/TEI/JSON-LD/Linked Art/…

A new format or challenge is a new value in one slot, not new schema.

The hinge: Profile

medieval-manuscript · damaged-archive · papyrus · printed-book · modern-typescript · … One choice loads the right segmenter + runtime/model + edition + transcription guideline + condition-handling. The user picks (or Fichero detects) the profile; the deep machinery configures itself. This is how the whole model reaches the user without exposing its complexity.

2. The delivery contract (non-negotiable)

A capability is DONE only across the whole spine:

backend → inspector/front-end → AI (MCP) + CLI → tested → exportable

Backend row alone = not done.
UX without MCP/CLI = not done.
Untested (esp. the cross-surface invariant) = not done.
Not exportable = not done.

Every phase below is a vertical slice that crosses the whole spine for a small piece, never a horizontal layer that builds backend-only breadth. The acceptance test for each capability is the do-it / represent-it / save-it trio plus MCP/CLI + export.

3. The primitive: Segment (spatial AND temporal)

A Segment is any addressable slice of any medium — this is W3C Media Fragments:

spatial (image / PDF): polygon / #xywh=x,y,w,h
temporal (audio / video): time span / #t=start,end
both (video): region + time
~~geographic (georeferenced map, #1755): lat/lon + warp transform~~ Superseded by maps-and-georeference.md (#5120): a geographic place is NOT an anchor kind. A segment’s place in the world is worked out through a georeferencing pass of ground-control-point segments, with the input CRS explicit and WGS 84 stored (#5124).

Granularity ladder: stroke → glyph → character → word → line → region → page-part → page. Surface for creating/editing a segment = the Preview (source) pane (#168): the polygon is drawn and adjusted on the media. The inspector shows a segment’s metadata/versions.

Existing infra to build ON (not replace): _segment_image crop, the Kraken segmenter that persists baseline geometry, recombine/contact-sheet.

4. The layers (all are versioned collections at the segment grain)

A segment holds many, each versioned append-only:

Transcription[] — { text, language, script, edition, version, provenance }. Language/script live HERE, not only on the segment (bilingual gloss = 2 transcriptions; graphemic vs normalized = 2 transcriptions). Editions: diplomatic / normalized / graphemic / graphetic (#4638).
Representation[] — { kind, version, provenance }, kind ∈ image-crop · text · tokens · image-vector · text-vector · glyph-rep(no-Unicode) · layout-features. The consumer chooses which it wants (an AI asks image+text; an index asks vectors; an exporter asks text→ALTO).
Provenance/Rationale — per version: the model’s reasoning AND the human’s rationale, not just the value. This makes plausible-readings-side-by-side possible and IS the training corpus (#4636, #4642).
Linguistic annotation — token spans within a transcription: POS, lemma, morphology, dependency (CoNLL-U/UD). NER (entities) is the same mechanism (#4637).
Zones — logical grouping ABOVE the page: a page typed as letter/response/deposition; a set of pages spanning multiple documents (a legal case); with relationships (reply-to, continues, part-of). Reading-order / correspondence layer (#4639).
Tags — arbitrary labels (free or controlled-vocabulary) on any object at any grain (SKOS / W3C Web Annotation).

5. Staged development plan

Each phase is a vertical slice. P0 proves the spine cheaply on existing data before any new schema. Later phases add breadth only after the pattern holds and is approved.

P0 — Segment Representations read model (the proof slice)

Spec: segment-representations.md. Milestone spread: Inspector-Knowledge-Engine #55 (backend), Inspector-Knowledge #151 (inspector), Client-MCP #52 (MCP/CLI), Testing #267, Exporter #14. - backend: segment.read.representations → image-crop (via _segment_image) + text (existing transcription); read-only, honest-absence, no new schema. - inspector: a segment shows its crop + text, anchored (reuse F5 ClaimSourceRequest). - MCP + CLI: segment.get(reps=image+text) — hand an AI segments + images in one call. - tests: rep-roundtrip + cross-surface HARD-GATE (same segment → same anchor/text from backend, MCP, CLI) + resource-safety. - export: one segment’s transcription → ALTO (or PageXML) with geometry intact.

P1 — Segment editing on the Preview pane

Draw/adjust the polygon on the image/PDF in the Preview pane; persist geometry as an audited, reversible action; changes update in place. (Builds on #168 bbox work.)

P2 — Transcriptions as versioned collection + editions

Promote transcription to a first-class versioned record with language/script/edition; show the version stack; plausible-readings side-by-side; capture reasoning on each version (P4 dep). Editions picker (diplomatic/normalized/…). (#4638)

P3 — Representation vectors

image-vector / text-vector representations, versioned by embedding model; feed the search index. Resource-safety bounded (Search-Engine #186). (#4634 load-testing rides here.)

P4 — Provenance, versions, rationale (cross-cutting)

The full restorable history + the reasoning field (model + human) on every version, shown on the page and easy to edit. Actor roles extensible (extractor/editor/reviewer/ end-user). This is #4636 — a candidate its-own-milestone; every layer rides it.

P5 — Zones + relationships

Logical groupings above the page (letter→response; case across documents) with relationships; see/edit on the Preview pane; query via MCP/CLI; export TEI correspDesc / METS structMap / IIIF ranges. (#4639)

P6 — Linguistic annotation (POS) + Tags

Token spans + labels within a transcription (CoNLL-U/UD); general tagging (SKOS) across objects. (#4637 + the tags thread.)

P7 — Language/script coverage + no-Unicode path

loove coverage tiers; CLDR/Glottolog/UCD/UDHR; fonts-by-character; Unicode-space viz; the extreme case — a script with no encoding/no fonts works image-anchored with custom glyph representations and contributes toward encoding. (#4637)

P8 — Authority linking + Web of Data

sameAs sets to VIAF/Wikidata/GeoNames/Pleiades/Getty; dereferenceable URIs; SPARQL endpoint (make rdflib visible); JSON-LD/Linked Art; PROV/CIDOC-CRM. (#4641)

P9 — Exporter Manager + folder sync + contribute

One export path for all the slots/formats; index a folder / keep it in sync; package a repo; contribute data upstream; training-corpus export (feeds P10). (#4640)

P10 — Distilled paleography VLM

The versioned (image-crop → text → reasoning) corpus + human review → distill a small local “thinking” VLM (MLX) that emits its reasoning. (#4642)

P11 — Georeference

Old-map-on-new-map, specified in maps-and-georeference.md (#5120): not a geographic anchor kind but control-point segments, a transform worked out from them with residuals, and an explicit CRS; export IIIF Georeference Annotation, GCP tables, world file/GeoTIFF, GeoJSON, GeoPackage and Linked Places. (#1755)

5a. How the phases map to the source model (2026-09-19)

The maintainer has since ruled that the whole model is specified properly first, and the order of building is decided from the finished spec (source-model.md, open question on order). The phases above stay as a record of intent; this table says where each now lives and what changed. Words: the per-document Profile hinge of section 1 is retired (a project’s profile, plus overrides lower in the cascade, does its job; “profile” now means only that); the granularity ladder of section 3 is replaced by the source model’s (which reaches above the page and has no separate “glyph” or “page-part” level); the open questions in section 7 on profile detection and on the first export target were answered by the rulings of 2026-09-19 (onboarding starts from sample pages; PageXML, ALTO and TEI are built together); a “layer” of segments is now a pass; a “transcription” in an “edition” is a reading with a kind and a level; “representations” are readings plus worked-out things.

Phase	Now specified in	What changed
P0 read model	`segment-representations.md` (first slice) + `segments-and-geometry.md`	a segment has a lasting id and its own record; pictures are cut to the polygon
P1 editing on the Preview	`segment-editor.md`	native SwiftUI, Mac + iPad + iPhone, Pencil; one overlay; merge, split, reorder, links
P2 transcriptions + editions	`readings-and-apparatus.md`	readings as a set; written and read; only a person chooses; normalisation level recorded
P3 vectors	`source-model.md` (“What stands on segments”)	vectors per segment and per reading; search lands on a segment
P4 provenance, versions, rationale	`segments-and-geometry.md`, `readings-and-apparatus.md`	versions per segment; hand kept apart from provenance; three kinds of “sure”
P5 zones + relationships	`segments-and-geometry.md`	physical and logical structure; named reading orders; typed links; flows
P6 linguistic annotation + tags	`source-model.md` (worked-out things), `readings-and-apparatus.md` (marks)	word-level analysis hangs on a reading; marks on any segment, in authored sets
P7 language/script coverage, no-Unicode	`languages-scripts-signs.md`	moved from late to foundational: three facts, cascade, direction, declared signs, sign lists, fonts
P8 authority linking	`kg-enrichment.md`	unchanged; a segment gains a citable reference
P9 exporter, contribute	`formats-and-training.md`, `export/exporter.md`	every format both ways; validation; loss reports; imports arrive as passes
P10 distilled VLM	`formats-and-training.md`	the training loop, human-checked by default, split by manuscript, measured against ground truth
P11 georeference	`maps-and-georeference.md` (umbrellas in `segments-and-geometry.md`)	control points as segments; the transform worked out; CRS; gazetteers; geo formats
new	`rights-and-access.md`	rights, consent, community labels, redaction, removal (proposed)
new	`segments-and-geometry.md`	recordings as stretches of time; several images lined up by alignment points; rescans

Embeddings, the knowledge graph, content and workflows all stand on segments: see “What stands on segments” in the foundation.

6. Standards map (the export slot, by layer)

Layer	Standards
Anchor (spatial/temporal)	W3C Media Fragments, W3C Web Annotation selectors
Segmentation / geometry	ALTO, PageXML, hOCR, abbyyXML (Kraken outputs)
Transcription / editions	TEI, CATMuS guidelines, Leiden (papyri), ZOLO, MEI (music)
Linguistics	CoNLL-U / Universal Dependencies
Claims (RDF)	RDF/SPARQL, JSON-LD, Linked Art, CIDOC-CRM
Provenance	W3C PROV
Zones / structure	TEI msDesc/correspDesc, METS structMap, IIIF ranges/collections
Tags	SKOS, W3C Web Annotation
Authority	VIAF, Wikidata, LoC, GeoNames, Pleiades, Getty (TGN/AAT)
Package / interchange	IIIF, METS, BagIt
Georeference	GDAL world-file, GeoTIFF, IIIF-georef

Rulings

2026-10-04 (maintainer): profile detection. A page’s archival profile is detected from the page and its recipe, shown to the person, and editable; a change can cascade to the folder.
Ruled 2026-10-04 (design lead, applying the spec’s own lean): a representation is both a file payload and a metadata row. Transcription is already first-class: text lives as readings on segments, in passes (source-model.md, rulings of 2026-09-20), so no separate transcription table is owed here.
Answered by what is built (2026-10-04): ALTO and PAGE XML both export today, one page at a time (fichero_server/api/routes/document/page_export.py, “Export one page as PAGE XML, ALTO or TEI”; writers in fichero_server/formats/alto.py and pagexml.py), so neither is “first”.

7. Open questions to talk through

Answered (maintainer 2026-10-04: archival profile detected + editable): Detected from the page and editable after. Profile detection — user-picked only, or auto-detected from the page? Editable after?
Answered (design lead 2026-10-04, applying the spec’s own lean): Both, a file payload and a metadata row. Where representations physically live — derivative files (like _segment_image), a table, or both?
Answered (source/source-model.md Rulings 2026-09-20 (readings on segments, slice 8)): Text lives as readings on segments; no separate transcription table. Is Transcription first-class now (P2) or does P0 read the existing transcription text and defer the multi-edition table?
Answered (design lead 2026-10-04, applying the spec’s lean): DuckDB is the record; LanceDB and the graph are derived from it and rebuilt from it. Store responsibility — DuckDB (relational) vs LanceDB (vectors) vs rdflib/SPARQL (graph): which store owns which slot, and how they stay in sync (ties Exporter #4640)?
Answered (source/source-model.md Rulings 2026-09-19 item 21): One milestone for the source-model set. New milestones? — “Segments & Anchors” and/or “Provenance, Versions & Credit”, or keep everything under the existing surface milestones? (Decomposition on #4639.)
Answered (source/source-model.md Rulings 2026-09-19 item 19): PageXML, TEI and ALTO all built first on one mapping system. P0 export target — ALTO or PageXML first?

8. Milestone decomposition

Design lane lives in Historical Text (#265); each phase’s build issues land in the surface milestone that owns their spine segment — Inspector-Knowledge #151/#55, Client-MCP

52, Search-Engine #186, Importer #188, Preview-Image-Editing #168, Exporter #14,

Settings-Models #20, Testing #267. Full mapping + the two proposed new milestones are on #4639 (awaiting the creative director’s ack — no milestones created yet).