Skip to content

Engine silent failures: what the logs say, and how to catch them

Status: DRAFT, 2026-10-30; revised against the code 2026-10-31. Owner’s order (2026-10-30): the engine must be rock solid, so silent failures need a way to be caught, specced, tested against spec and fixed, as the UX’s stall detector does for slowdowns. Manual: TBD — a “When something keeps failing” section: what the amber engine pill and an Activity “Needs attention” row mean, and what Reconnect does for originals on another volume. Tags: [OK] behaves this way today · [PARTIAL] part built or not wired · [GAP] intended, not built. Every non-[OK] bullet cites an issue. #4420 (closed; the seam programme) stands in for the bullets that have no issue of their own yet: the lead files the titles under “Issues to file” and replaces the number. Part 1 is the inventory (what the two real logs say). Part 2, “The detector”, is the spec, each bullet tagged with what is built. A file:line in the inventory table is from the logged build; elsewhere a bullet names the function or file, not a line, because lines move.

Where the numbers come from

Streamed, never loaded whole, never copied: python read each engine.log line by line (on the Air through ssh air, same script), kept every non-DEBUG line, and normalised hex ids, UUIDs, paths and digits to a template.

Log Size Span Lines (non-DEBUG) Distinct templates
This Mac, ~/Library/Containers/app.fichero.fichero/Data/Library/Logs/Fichero/engine.log 57 MB 2026-10-12 → 10-29 107,641 3,662
The Air, same path 167 MB (at 17:00) 2026-09-22 → 10-30 592,479 3,400

One template, “Failed to create embedding”, is 389k of the Air’s 592k lines (66 %: rows 1 and 2; the Mac’s row 3 adds 21k more). Nothing on screen said so.

Top 30 templates (Mac + Air, by total count)

Class: SNR silent-no-result (work did not happen, the person sees nothing wrong) · RL retry-loop (the same attempt repeated) · EN expected-noise (healthy, only loud) · RE real-error (a person is told, or should be). “Mac / Air” are the counts per machine. Source is where the line is written (fichero_server/).

# Mac / Air First → last Lvl Source Class What the person sees
1 0 / 301,429 10-29 10:38 → 10-30 09:18 WARN db/__init__.py:6380 (raised at db/embed_process.py:452) “Failed to create embedding for ‹id›: The search model’s process stopped while loading the model; embedding for search will try again later.” SNR + RL Nothing. Every document write queues an embedding, the embedder dies at load, the page is saved without one; Search later misses it.
2 0 / 87,527 (still growing at 17:00) 10-30 09:19 → WARN db/__init__.py:6380 “…could not be loaded: ValueError: Model BAAI/bge-m3 is not supported in TextEmbedding…” SNR + RL Nothing. Same effect, a different cause after the 09:19 build (the model name is not in the installed fastembed).
3 20,828 / 0 10-21 07:20 → 10-22 23:21 WARN db/__init__.py:6380 “…could not be loaded: Fail: [ONNXRuntimeError] … External data path validation failed for initializer…” SNR + RL Nothing. Same line, a third cause (the model’s files).
4 13,965 / 66,274 10-19 → 10-30 INFO core/single_flight.py:58 “poll activity.list_background_jobs took ‹n› ms…; ‹n› callers were served from a running query” EN Nothing. Loud, not wrong: Activity’s poll.
5 12,633 / 22,630 09-22 → 10-30 INFO workflows/checkpointer.py:136 “Initialized DuckDB checkpointer” EN (RL-shaped) Nothing. A checkpointer opened per run step, not once; 35k lines.
6 0 / 22,703 10-29 10:37 → 10-30 16:21 WARN api/change_stream.py:319 “change-hub: representation.created emit to key ‹/Volumes/Work/…/Project.fichero› reached 0 subscribers, but ‹n› live under different key(s): ‹…Project 12.fichero›” SNR The open window does not update live (new results appear only after a reload). The same line for segment.updated, backend.work.progress, document.updated, backend.work.completed, project.setup_saved: ≈ 29,900 lines in all (#2518 closed; the key drift is back).
7 6,855 / 4,157 09-22 → 10-30 INFO workflows/activity.py:181 “Activity saved successfully: ‹uuid›” EN Nothing.
8 1,088 / 9,369 (×2: reading order, topological sort) 10-16 → 10-30 INFO kraken (third-party) “Compute reading order…” / “Perform topological sort…” EN Nothing. Four more kraken lines per page (Vectorizing ×2, Segmenting): ~25k lines a day under a 61k-page run.
9 0 / 5,888 10-29 10:34 → 10-30 16:17 WARN (the segment.updated variant of row 6) SNR As row 6.
10 0 / 5,532 + 4,118 10-28 → 10-30 INFO llm/__init__.py:486 “LLM usage [openrouter / anthropic …]: input=‹n›…” + httpx “HTTP Request: POST … 200 OK” (9.7k each) EN Nothing. Usable as the cost record, but the log is the wrong place.
11 2,588 / 1,985 09-22 → 10-30 INFO workflows/builder.py:2092 “Parallel processing file n/N: ‹path›” (+ 3,000 [PARALLEL] print lines) EN Nothing.
12 0 / 3,717 10-28 15:40 → 19:40 WARN api/routes/system/activity.py:425 “activity summary: no status for recipe run waiting:run-a-recipe: no recipe run waiting:run-a-recipe” RL (poll) Activity’s summary for a waiting recipe row is blank; the poll repeats every few seconds.
13 1,125 / 0 10-26 16:19 → 17:02 WARN execution/jobs.py:3701 “job ‹uuid› (find-lines on ‹id›) failed: Kraken segmentation failed: [Errno 2] No such file or directory: ‹path›” RE The job is Failed in Activity (said). The cause, the original not at its place, is only in the text.
14 888+822+796+773 (4 pages) / 0 10-26 16:19 → 17:04 ERROR + WARN + INFO ×2 workflows/tools/vision_base.py:6142/6161 “Vision processing failed for ‹name›: File not found: ‹path›” (the same failure written four times: ERROR, [system_warning] WARN, two “Saving activity” INFO) RE (×4 amplification) One failed row per page; the log carries each failure four times.
15 0 / 1,486 10-29 10:44 → 10:45 ERROR export_service.py:2022 “Export image source missing for document ‹id› (EAP…IMG_001.tif) path=’/Volumes/Archive/Originals/…’: this engine may not open files in that folder: grant access to the folder” SNR A Word export finishes without the page images (row 16). The reason is in the log only.
16 0 / 1,486 same WARN export_service.py:1362 “Word export left out the image of ‹name›: …grant access to the folder” SNR As row 15.
17 1,576 / 0 + 781 / 1,440 10-16 → 10-30 INFO workflows/activity.py:178 “Saving activity to DB: node_started - Node ‘…’ started” EN Nothing.
18 790 / 1,180 09-22 → 10-30 INFO api/routes/document/documents.py “PERF library.list_documents duration_ms=‹n› …” (+ get_children, document_view) EN Nothing. The existing perf lines.
19 585 / 1,252 10-19 → 10-30 INFO core/single_flight.py:58 “poll activity.get_job_tree …” (+ recipes.recipe_run_status 857) EN Nothing.
20 1,157 / 404 10-16 → 10-30 INFO api/routes/system/storage.py:~190 “PERF library.thumbnail.endpoint cache_state=hit …” EN Nothing.
21 773 / 0 10-20 → 10-26 INFO same span, …duration_ms=‹n› error=HTTPException status=error SNR A grid tile with no picture, no reason (the 404 is “Thumbnail not available”).
22 491 / 970 09-24 → 10-30 INFO workflows/tools/llm_base.py “Created transcription artifact ‹id›” (+ “Updated page_content for ‹id›; embedding queued”, :911, 477 / 897) EN Nothing. The “embedding queued” line is the promise rows 1-3 break.
23 491 / 0 10-20 → 10-25 WARN api/main.py:218 “Rejected library path: path=‹…› home=‹container› failed_check=roots” (Air: 78, 10-28) RE The library does not open; the app says it could not open it.
24 0 / 997 10-28 15:11 → 20:07 WARN workflows/tools/extract_entities_only.py:271 “extract_entities_only: ‹name› left out: no text: no line of its working pass has a reading…” SNR (partial) A names run says “done”; 997 pages contributed no names. Each is appended to the run’s left_out list with its reason (extract_entities_only.py:269-270); whether the run’s Activity row shows that count is unverified.
25 999 / 271 10-20 → 10-28 INFO maintenance/project_conversion.py:479 “A new result on page ‹id›: converted” EN Nothing.
26 1,260 (Air) / 463 (Mac) 10-19 → 10-30 INFO workflows/tools/vision_base.py:4935 “Kraken segmenter: ‹file›” (+ :5179 “LLM Vision: ‹file›”, 466) EN Nothing.
27 93 (Air) 10-29 ERROR _duckdb.IOException … Could not read from file ‹…/Project.fichero/fichero.duckdb›: Input/output error RE The external volume dropped under an open library. Every request on that library fails with the raw DuckDB text (no handler in fichero_server/ catches IOException or “Input/output error”); nothing names the volume.
28 53 OK / 2 × 401 09-22 → 10-30 INFO POST /api/sandbox/security-scoped-access 200 / 401 (a grant before the engine is authenticated) EN Nothing.
29 1 per spawn every spawn INFO security/security_scoped_access.py “Security-scoped access granted via inherited sandbox scope (bookmark unresolvable in this process): ‹path›” and “…: N of N librar(ies) granted” EN, but see the originals class below Nothing. The N is the number of libraries, never the folders holding the originals.
30 2 10-29 WARN api/auth.py “auth refused 401 branch=bootstrap_mismatch path=/api/workflows/folders” RE (one-off) Nothing; the app retries.

Rows 6 and 9, 15 and 16, 4 and 19, 3-way splits of rows 1-3 are one finding each in the classes below.

What the inventory says

  • Three of the top six lines are one failure with three causes: the embedder cannot load, and every document write retries it, logs a line and moves on. 389k lines in 38 hours on the Air. The code already knows the difference (EmbedOutcome.reason = embedder_unavailable, #4395) and the queue “waits and says why” (comment at db/__init__.py, Database.embed); the line is written by the caller of each write, so the wait never reaches the log as one line, the log as a rate, or health as a state.
  • The engine already has the seams for “say it once, in a place a person looks” (startup_warnings.py into /api/health dependencies, the swallowed-exception seam baseline, the stall watchdog); none of them counts a repeated WARNING, and the app reads dependencies only for the About box’s versions and the previous stall (fichero/fichero/App/AppState/AppState+HealthCheck.swift:87-92): a failed: <Class> entry is shown nowhere.
  • Much of the Air’s log is INFO that would hide any real signal (5 of the top 10 templates): this is the log-collapse worker’s (branch feat/logs-stale-servers-support) ground; this spec only counts WARNING and above (see Part 2).
  • The engine’s log set-up is not core/logging.py: the handlers that write engine.log come from logging.basicConfig in fichero_server/__main__.py:35 (root logger, format asctime - name - levelname - message; the app captures stdout/stderr into the file). core/logging.py’s configure_logging is called nowhere in fichero-server/src. The collapse branch adds engine_log.start to __main__.py and collapses per call site (core/log_throttle.py::log_collapsed), not with a handler. This decides open question 1.

The classes (one bullet each; ids are engine.silent.<slug>)

Each class names the test that proves the failure is surfaced (Test:) and the fixture log that test and scripts/scan_engine_log.py read (Fixture:). A bullet tagged [OK] must name a test file that exists; the guard is tests/unit/seams/test_engine_silent_spec_names_tests.py (engine.silent.every-bullet-names-its-proof). “Surfaced” means one of: a degraded entry in /api/health with a sentence, a reason in the endpoint’s 404 detail, or a worded row in Activity; a log line alone never counts.

  • engine.silent.embedding-fails-every-write — [PARTIAL] (#4395 open, “Embedding failures are swallowed”; #6218 closed, the forked child dies while it downloads; #1524 closed, bge-m3 not supported) an embedder that cannot load makes every document write log one WARNING (389k on the Air) and produce no vector; health says healthy, Activity says nothing, Search silently loses the pages. Built 2026-10-30: Database.embed’s except calls failure_ledger.note("search", "embed", <cause class>, sentence=...) (db/__init__.py, in Database.embed) and /api/health lists “Search is not indexing new pages: the search model cannot be loaded (EmbedderUnavailable). 25 failures since 09:19.” (the class is EmbedderUnavailable, which embeddings.py and two of embed_process.py’s raises (a load timeout, the model’s own failure words) give no from, so those causes share one signature; a dead child raises from the EOFError/OSError, so its reason says that class instead; the words that tell them apart stay in the engine log, not on health) Unbuilt: the Swift rows (app-says-degraded), and “N pages waiting” (the ledger counts failures; the queue’s own count is #4395’s). Test: tests/unit/core/test_failure_ledger.py::test_embedding_failure_degrades_health (built: 25 failing writes, one search entry, no path in the reason). Fixture: tests/fixtures/engine_logs/embedding_380k.log (the real line shape, both causes; read by the scan test).
  • engine.silent.change-hub-key-drift — [PARTIAL] (#2518 closed, #6309) both hub seams (subscribe, unsubscribe, emit) call ONE function, canonical_library_key (db/library_paths.py: NFC like DatabaseManager, then realpath), so an accented name in the form macOS hands a window and the form the engine opens reaches the same subscribers. An emit that reaches 0 subscribers warns only when a live key could be the same library (equal once case-folded, or the same file on disk); a real second library with no window (Project.fichero next to an open Project 12.fichero) is debug, not a warning. Either way it logs once per key pair and again at 10, 100, 1000 … with the count, and the pair table is capped at 256 (api/change_stream.py, _ChangeHub.emit). Unbuilt: the ledger entry (area live-updates) and the Activity words. Test: tests/unit/api/test_change_hub_key_drift_is_degraded.py::test_accented_name_in_either_unicode_form_reaches_the_window and ::test_same_library_under_another_spelling_warns_once_per_pair_with_a_count, ::test_a_different_library_with_no_window_does_not_warn and ::test_the_throttle_memory_is_bounded. Fixture: row 6’s line.
  • engine.silent.originals-unreachable-shows-thumbnail — [GAP] (#5818, #4217 nearest open issues; #6306) see its own section below.
  • engine.silent.export-leaves-images-out — [PARTIAL] (#5818 open, “Pages whose original is missing fail in words”) a Word export whose page images the engine may not open still finishes, naming the reason only in the log (export_service.py:1362 WARNING, :2022 ERROR; _why_image_unavailable :2027-2041 already tells “moved or deleted”, “may not read it: grant access”, “may not open files in that folder”). Surfaced when: the export’s result carries “N page images left out: grant access to the folder” and its row in Activity says so. Test: tests/unit/export/test_word_export_says_images_left_out.py (unwritten; check #5818 first, it may land it). Fixture: rows 15-16.
  • engine.silent.recipe-summary-no-status-loop — [OK] (#4420, #6213 — proposed title: “Activity summary logs ‘no status for recipe run waiting:run-a-recipe’ 3,717 times: a waiting row with no run behind it”) api/routes/system/activity.py:421-427 catches every exception from _recipe_state, logs a WARNING and appends a block with no stage; the poll repeats it every few seconds, so the person sees a blank row and the log 3,717 lines. Test: fichero-server/tests/unit/jobs/test_activity_summary_one_source.py. Fixture: row 12. Built 2026-11-20: (#6745) a waiting: or paused: queue row, which stands for many runs and is no run, builds its block from its own state and reason and logs nothing; test tests/unit/jobs/test_activity_summary_one_source.py::test_a_waiting_recipe_queue_row_has_a_status_and_logs_no_warning_6745.
  • engine.silent.pages-left-out-of-a-names-run — [PARTIAL] (#4420, no issue yet) 997 pages “left out: no text” in one names run, one WARNING each (extract_entities_only.py:271). Built: each page is appended to the run’s left_out list with its reason (:269-270), so the count exists. Unbuilt: the run’s Activity row says “997 pages left out: no text”, and the per-page WARNING is the collapsed kind. Test: tests/unit/workflows/test_entities_only_left_out_is_counted.py (unwritten).
  • engine.silent.a-failure-is-written-four-times — [GAP] (#4420, #6310 — proposed: “One failed page writes four log lines (ERROR, system_warning WARN, 2 INFO saves): collapse to one”) the vision failure path amplifies logs ×4 (3,300 lines for 4 pages). The handler counts every WARNING-or-worse template, so today the ERROR line and the [system_warning] WARNING (workflows/activity_types.py:47) would be two degraded entries for one failure. The fix is one of: the activity-save WARNING carries extra={"ledgered": True}, or the vision except calls failure_ledger.note("reading", ...) once and both lines are marked. Test: tests/unit/workflows/test_vision_failure_logs_once.py (unwritten).
  • engine.silent.thumbnail-404-without-reason — [OK] (#4420, #6308) /api/thumbnail (api/routes/system/storage.py) and /api/display answer 404 with {detail, reason} and words, reason one of the unreadable-original codes (engine.silent.originals-reason-is-one-of-three), not_generated_yet (thumbnail only: the inline-generation permits are taken and the request is shed), or no_picture (the original is readable but no picture could be made). /api/display has no shed path, so it never answers not_generated_yet; /api/source keeps its bare 404 when the original is readable. 773 tiles in one Mac session. Test: tests/unit/api/test_thumbnail_404_names_reason.py.
  • engine.silent.library-volume-dropped — [PARTIAL] (#4420, #6311) a DuckDB error from a disk that is not answering is raised as LibraryVolumeNotReachable (db/volume_dropped.py) and answered by one handler (api/main.py) as a typed 503 {detail, reason: library_volume_not_reachable, library}: “the engine can’t reach the library’s disk” (the same error can be a failing internal disk, so it never says “disconnected” as fact), the library by display name only. The evidence is the disk, not the message: a real unplug first says “Cannot open file …wal” on a commit, never “Input/output error”, and later “database invalidated”, so any DuckDB error while the library file cannot be stat-ed, or an “Input/output error”, maps (classify). Mapped at _execute, _execute_fetch_with_columns, _read_fetch_with_columns, get_committed, schema _execute_locked, commit, checkpoint, BEGIN/COMMIT/ROLLBACK, and the manager’s two CHECKPOINTs (db/manager.py). The log line is written once per library per outage, failure_ledger area library-volume has one entry per library (a counter), and the outage ends only when the file answers a stat AND a statement succeeds (a read served from DuckDB’s cache does not end it). A read DuckDB answers from cache during the outage is real data and stays 200. Not mapped (direct self.conn.execute sites no request reaches first): the open-time folds and migrations, COPY ... TO PARQUET exports, the bulk insert at ~2578, ~5288. Unbuilt: health degraded at the first failure (it waits for the ledger’s rate threshold) and the app saying it in words with Reconnect (Swift, a separate item). The real-unplug claim is built and tested; the item stays PARTIAL for the unbuilt parts above. Tests: tests/unit/db/test_real_unplug_is_said_6311.py (a real force-detached disk image, macOS, not under CI) and tests/unit/db/test_volume_dropped_is_said.py (one 503, one log line, one ledger entry, recovery; an invalidated database on a disk that answers is reopened, not refused). Fixture: row 27’s line, still to add to tests/fixtures/engine_logs/.
  • engine.silent.warm-up-failure-reaches-health — [OK] (#4420) a failed start-up warm-up (Keychain keys, Vision locales) is in /api/health dependencies as failed: <Class> (startup_warnings.py:43-52). The app shows it nowhere (AppState+HealthCheck.swift:87-92 reads dependencies for versions and previous_stall only): that is engine.silent.app-says-degraded’s work. Test: tests/unit/api/test_warm_up_failures_reach_health_4420.py. Fixture: none (an in-process fault).
  • engine.silent.swallowed-handlers-cannot-grow — [OK] (#4420, #5315) a new handler whose body is only logging calls, pass or continue fails the ratchet (test_swallowed_exception_seam.py:62-75, _only_logs). The sweep is syntactic: any other statement, a failure_ledger.note(...) included, takes the handler off the list. Test: tests/unit/seams/test_swallowed_exception_seam.py. Fixture: none.
  • engine.silent.import-stall-is-loud — [OK] (#5807) a stalled import says so. Test: tests/unit/api/test_import_stall_is_loud.py. Fixture: none.
  • engine.silent.info-lines-hide-the-signal — [PARTIAL] (#4420, no issue yet; the log-collapse worker’s branch feat/logs-stale-servers-support: rotation in core/engine_log.py, repeated-line collapse in core/log_throttle.py, both unmerged) 5 of the top 10 templates are INFO. Not this spec’s to fix; this spec counts WARNING and above and reads the collapse’s summary record, never edits it.

The originals class (added 2026-10-30, from the Air)

engine.silent.originals-unreachable-shows-thumbnail: after the app was replaced by a new build, the Project source images (61k pages, originals under /Volumes/Archive/Originals/…, display copies and library on /Volumes/Work/Libraries/Project.fichero) “need to be reconnected” and the app shows the thumbnail instead of the original.

What the code does today (read, not run):

  1. The fallback to the thumbnail is fichero/fichero/Views/Preview/DocumentCanvas.swift:227-257: when the display image fails to load (loadError) and a cached thumbnail exists, it draws the thumbnail with a capsule “Showing thumbnail — original unavailable” and a Retry. (The maintainer asked on 2026-08-20 for the thumbnail to load and, when the original is missing, a brief flag.) So it is not totally silent, but the banner (a) does not say why, (b) offers only Retry, which cannot help when access is not granted, (c) is on the Preview canvas only: the grid, the Reader and every other place a thumbnail stands in (LibraryImageView.swift:79, seedFromCache) say nothing, and (d) a thumbnail is also what a generated grid tile is, so a person cannot tell “this is the small copy” from “this is all I could get”.
  2. The engine’s answer carries no reason: /api/display/{id} returns “Display image not available” (api/routes/system/storage.py:286), /api/source/{id} “Source file not available” (:341, :345; the cause goes to a WARNING with the path), /api/thumbnail “Thumbnail not available” (:225); get_display/ensure_display return None when resolve_source (db/storage.py:471-499, candidate.exists()) finds nothing: “not mounted”, “not found there” and “this engine may not read it” are the same None. The classifier already exists, for the export only: export_service._why_image_unavailable (:2027-2041: FileNotFoundError → “moved or deleted”, PermissionError → “grant access to its folder”, other OSError → “may not open files in that folder”).
  3. The app’s grants: FolderAccessManager.resendAllGrantsToEngine (fichero/fichero/Services/FolderAccessManager.swift:412, filter at :419) and engineBookmarkPayload (:187, filter at :193) send only the bookmarks that cover a library (grantCoversALibrary, :454, from #5228/#5251, 2026-09-28, “folders from long-closed libraries … were sent one request at a time”). restoreBookmarks (:460) deletes a bookmark that fails to resolve after one logger.error whose path is <private> (:513-517); the engine’s refusal is logged the same way (:364).

The cause the Air’s engine.log supports (streamed, not copied; the Air’s stored bookmarks were read with plistlib, keys only):

  • Until 2026-09-29 each spawn said “Security-scoped access: 10 of 10 librar(ies) granted” and listed /Volumes/Archive/Library/Series 1 … Series 8.
  • From 2026-10-28 every spawn says “3 of 3” (…/Libraries, Project.fichero, Project 12.fichero): the folders holding the originals (/Volumes/Archive, /Volumes/Archive/Originals/Series 1/Box 12) are still stored as bookmarks in the app’s defaults (18 keys on the Air) but are not re-sent at spawn or in the post-ready sweep, because no library path lies under them. A link-mode project’s originals sit outside every library by design. They were granted only at the moment they were chosen in a session (10-28, 10-29: “granted via inherited sandbox scope”), and that grant ends with the engine process.
  • On 10-29 10:44 the Word export then failed 1,486 times with “Export image source missing … this engine may not open files in that folder: grant access to the folder” for /Volumes/Archive/Originals/…: that is the real cause in the engine’s own words, for the export path; the display path swallows the same condition into None.
  • The log has no Operation not permitted, Errno 1 or PermissionError line (0 hits on both machines): the engine never raised a permission error. os.stat on a path outside the sandbox without a scope gives ENOENT-like “not found”, which resolve_source treats as “moved”. The failure is therefore indistinguishable from a missing file at every layer until it is asked.
  • Not proven, flagged as the open question: that the replace of the app is the trigger. The evidence says “a fresh engine process re-grants libraries only”; a replace restarts the engine, so it would show up on the first launch after one, but any restart does the same. Whether the app-scoped bookmarks themselves still resolve after a replace with a new build (same bundle id, same Developer ID team QAPB6CWYR6) was not testable from logs: the app’s FolderAccess log lines are <private> in the unified log (Engine refused folder access for <private>, 3 times on 10-30: 09:19, 12:37, 16:36, each at a spawn). Needs one run with the log made public for the path, or a bookmark-resolve test.

What the person should see (spec, unbuilt; ids in the bullets):

  • engine.silent.originals-unreachable-says-so — [GAP] (#5818, #4217 nearest open issues; #6306 — proposed: “Originals on a volume are shown as thumbnails with no word why; say ‘originals not reachable’ with the reason and a Reconnect…”) when an original cannot be read, the window says so once, where the person is looking: a banner on the project (and an Activity row, kind “needs attention”) “Originals not reachable: Reconnect…”, never only a capsule on one pane. The thumbnail stays as the fallback, marked (a small “Small copy” badge on a preview, not on the grid’s own tiles).
  • engine.silent.originals-reason-is-one-of-three — [PARTIAL] (#6306, #6308) built: GET /api/storage/thumbnail, /display and /source answer 404 with {detail, reason} when the recorded original is not a readable file, reason one of no_recorded_file, volume_not_mounted (a path under /Volumes/<name> with no /Volumes/<name>), moved_or_deleted, access_not_granted (PermissionError, or the path stats but resolve_source refuses it, e.g. a path naming ..), unreadable (another OSError), from media/image_reason.classify_image_unavailable, which the export’s _why_image_unavailable also uses. The path classified is the one the resolver tries first: the document’s own, then metadata.source_path/full_path/display_path/local_path, and for a page with none (a PDF page) its parent’s. A recorded path that cannot be statted at all (a NUL, a ~user that does not exist) is unreadable. When resolve_source finds the source, /api/thumbnail and /api/display answer no_picture (the original is readable but no picture could be made) and the thumbnail shed path answers not_generated_yet (see engine.silent.thumbnail-404-without-reason); only /api/source keeps its bare 404. The body never carries a local path, only a drive’s name; the classifier runs off the event loop. Not built: the granted_paths() test for a path on a mounted volume (that case reads as moved_or_deleted), and the app showing the words. Test: tests/unit/api/test_display_404_names_why_unreachable_original.py. As first specified: the engine names why in a field the app shows in words, from the classifier that exists in export_service._why_image_unavailable, moved to a shared helper: “volume not mounted” (/Volumes/<name> absent: os.path.ismount of the first two components false), “access not granted” (the path exists in a listing the engine cannot read, or the folder is not under a held grant: PermissionError, or ENOENT for a path under /Volumes/ while the volume is mounted and granted_paths() does not cover it), “moved or deleted” (the volume is mounted and covered, the file is not there: #5535, #5672 relink). GET /api/display, /api/thumbnail, /api/source carry it in the 404’s detail as a stable code plus words.
  • engine.silent.originals-one-press-reconnect — [GAP] (#4217, the Mac App Store sibling “no bookmark exists and nothing re-prompts”; #6306) “Reconnect…” opens the folder picker at the saved path (FolderAccessManager.requestFolderAccess(suggestedPath:), :119, exists), mints the bookmark, hands it to the running engine (grantEngineAccess, :339, private today), and reloads what was shown; “volume not mounted” says “Connect the drive ‘Archive’” and offers no picker. One press, no restart.
  • engine.silent.originals-access-restored-after-update — [GAP] (#5818, #4217; #6307) after an update or any restart, the app re-sends every stored bookmark whose folder holds a project’s originals (the paths in documents.path of open libraries, reduced to their shared parent), not only those that cover a library; a stored bookmark that no longer resolves is named in the banner (“Access to ‘/Volumes/Archive’ was lost: Reconnect…”) and kept until the person reconnects or removes it (today restoreBookmarks deletes it after a <private> error line, FolderAccessManager.swift:513-517: a stale bookmark becomes a quiet loss). Test: OriginalsGrantSweepTests (Swift, unwritten) (unwritten; LaunchLoadOrderTests.swift:18-19 pins grantCoversALibrary, which is exactly the filter this changes, so that test changes with it).
  • Test for the engine half: tests/unit/api/test_display_404_names_why_unreachable_original.py. Fixture: the Air’s “Export image source missing … grant access to the folder” line (row 15).

The detector

Intent

The UX has two things the engine lacks: a measure (a stall is timed, written to stalls.log, pinned by a perf test with a ceiling) and a place a person sees it (the app says it is slow). The engine has the first half for three things (warm-up failures, import stalls, held transactions) and nothing for the rest: a WARNING is a line, and a line repeated 389,000 times is still just lines.

So: count every WARNING-or-worse by what it says, turn a count that is too high into a named “degraded” state with a reason a person can read, serve that state where the app already looks (/api/health), and fail the build when a fixture or a real run produces a repeat nobody allowed. The same loop the specs use: each silent class is a bullet, each bullet names the test that proves the failure is surfaced and a log line that test reads, and a guard fails when a bullet claims [OK] and the test is not there.

Design constraints (the owner’s, and the other worker’s):

  • Beside the log collapse, never inside it. The log-collapse worker’s branch (feat/logs-stale-servers-support) collapses per call site: core/log_throttle.py::log_collapsed writes the first 5 repeats of a template per minute, then one record whose msg is exactly "%s — repeated %d times in the last minute" with args (template, held) (log_throttle.py:60-63; held is the count beyond the 5). A ledger reading the file would undercount by 99 %; a ledger that is a logging.Handler sees the 5 and adds held from the summary record’s args, so 5 + held is exact. The two features do not know each other, except at one line: both branches rewrite the Database.embed warning in db/__init__.py (theirs to log_collapsed(..., cause=e), which takes no extra; ours adds extra={"ledgered": True}), see engine.silent.ledger-counts-by-signature.
  • Count the template, not the text. Ids, paths and numbers make every line unique; the signature is the logger name plus the record’s msg (the %s template, before its arguments); for an f-string message (no args) digits and hex are normalised to N (failure_ledger.py). Paths are not, which is the gap named in the same bullet.
  • Health is unauthenticated: a reason carries a class name and a count, never a path, id or message text (the rule startup_warnings.py:33-36 already keeps). The entry’s own id is a 12-character hash of the signature, because the signature of a record the handler sees holds its message text; test_a_degraded_id_carries_no_message_text.
  • No new dependency, no thread, no I/O: a dict under a re-entrant lock (a warning logged while counting must not deadlock), at most MAX_SIGNATURES = 500 signatures (failure_ledger.py); the clock is injected (now=) so tests need no sleep.

Behaviors

A. The failure ledger (fichero_server/core/failure_ledger.py)

  • engine.silent.ledger-counts-by-signature — [PARTIAL] (#4420, #6312 — proposed: “Install the failure-ledger handler; count paths as one template”) every WARNING-or-worse record that reaches the root logger is counted by signature (logger name | template, failure_ledger.py), and the count stays exact when the line is collapsed (the summary record adds its held count, :118-119). Built: LedgerHandler (:107-125) and note() for explicit sites (:63-68); a record from a site that already called note carries extra={"ledgered": True} and is skipped (:115). Three gaps: (a) the handler is installed nowhere (:9-10); it goes on the root logger (logging.getLogger().addHandler) in the engine’s lifespan (api/main.py), once, idempotent, so the embedded app, the dev uvicorn engine and the tests all have it (open question 1, decided); (b) an f-string record keeps its paths (:31 normalises digits and hex only), so one message with a path per document is one signature per document, fills the 500-signature cap (:26, :75-76) and every later new failure is dropped silently: normalise paths as scan_engine_log.py:22-24 does; (c) after the collapse branch merges, the embedding line goes through log_collapsed, which drops the extra, so the handler and note would each count it (two entries for one failure): log_collapsed gains an extra= passthrough, or note(..., template=...) registers the template the handler must skip. Test: tests/unit/core/test_failure_ledger.py::test_counts_by_template_not_text, ::test_a_collapsed_summary_adds_its_count, ::test_info_is_not_counted, ::test_a_line_that_noted_is_counted_once_by_the_handler (built); ::test_a_collapsed_ledgered_line_is_counted_once (#6314: written, skips until the collapse worker merges, and fails there if log_collapsed still drops extra); ::test_handler_is_installed_on_the_root_logger, ::test_paths_are_one_template (unwritten). Fixture: none (records are made in-process; the file shape is the scan script’s test).
  • engine.silent.ledger-counts-a-swallowed-exception — [GAP] (#4420) a broad handler whose caller genuinely cannot act on the failure calls failure_ledger.note("<area>", "<site>", kind=type(exc).__name__) beside its log line; the seam sweep (test_swallowed_exception_seam.py:62-75) then no longer lists it, and its baseline entry must be deleted in the same commit (the sweep fails a stale entry). The sweep is syntactic, so note must not launder a handler whose caller could use a return value: #4420’s rule (“a skip is distinguishable from a failure in the return value”) still applies first; note is for the handler that stays broad. Migrating the 200-odd baselined handlers is not in this spec’s first slice: the embedding, reading (OCR/HTR, vision, Kraken), search and storage paths go first. Test: the seam test itself, plus one note test per migrated area (tests/unit/<area>/test_<area>_failures_reach_the_ledger.py, unwritten).
  • engine.silent.ledger-thresholds-degrade — [PARTIAL] (#4420, #6312 — proposed: “The ledger’s rate window counts records, not collapsed repeats”) a signature is degraded when it crosses either threshold and stays listed until it has been quiet: rate = 20 hits within 60 s (RATE_COUNT, RATE_WINDOW_S, failure_ledger.py), total = 200 since the process started (TOTAL, :24,81), quiet = not listed once 300 s pass with no new hit (QUIET_S, :25,96). tripped never resets (:82): after a quiet spell one new hit re-lists the entry for another 5 minutes, which is wanted (the 201st failure is still the same problem). Gap: the rate window appends one timestamp per record (:80) whatever count it carries, so under the collapse (5 records + 1 summary a minute) the rate threshold can never trip and only the total does: a 30-a-minute loop is listed after ~7 minutes instead of 40 s. The summary’s held must count toward the window. Per-area overrides (a model that cannot load is degraded at 5 hits, not 20) are a later table in the module. Test: tests/unit/core/test_failure_ledger.py::test_rate_threshold_degrades, ::test_total_threshold_degrades, ::test_quiet_period_clears, ::test_slow_repeats_below_the_rate_do_not_degrade_until_the_total (built); ::test_a_collapsed_summary_counts_toward_the_rate (unwritten).
  • engine.silent.ledger-reason-is-plain-language — [OK] each degraded entry says, in one sentence a person reads without the log, what stopped (the site’s sentence), why (the exception class, not the message) and how many: "<sentence> (<Class>). <count> failures since <HH:MM>." (failure_ledger.py; e.g. “Search is not indexing new pages: the search model cannot be loaded (EmbedderUnavailable). 87527 failures since 09:19.”). A record seen only by the handler gets "<area> keeps reporting the same problem: <count> times since <HH:MM>.". Test: tests/unit/core/test_failure_ledger.py::test_reason_is_plain_and_carries_no_path_or_id. Fixture: none (in-process).
  • engine.silent.ledger-handler-entry-names-its-area — [GAP] (#4420, #6312) for a handler-seen record the area is the logger name’s last component (failure_ledger.py: db, main, jobs, change_stream), which is not a word a person knows. A small table maps logger prefixes to areas (fichero_server.db → “Library”, fichero_server.execution → “Jobs”, fichero_server.api.change_stream → “Live updates”, fichero_server.workflows → “Reading”; unknown → “The engine”), and the reason reads “Live updates keep reporting the same problem: 22,703 times since 10:37.” Test: tests/unit/core/test_failure_ledger.py::test_handler_entry_names_a_known_area (unwritten).
  • engine.silent.ledger-success-clears — [GAP] (#4420, #6312) a site that can also succeed (embedding does) calls ledger.ok(area) on success, so a recovered engine stops saying degraded at once. This also fixes a soundness hole in the quiet period: a model that cannot load fails only when a write happens, so an idle engine’s entry drops off the list after 5 quiet minutes while the model is still broken, and comes back with the next write. Recommended default (owner to confirm, open question 3): an explicit-site entry (one with a sentence) stays listed until ok() or restart; the 5-minute quiet applies only to handler-seen signatures, which have no site to call ok(). Test: tests/unit/core/test_failure_ledger.py::test_ok_clears_at_once, ::test_an_explicit_entry_outlives_the_quiet_period (unwritten).
  • engine.silent.ledger-is-cheap — [OK] note is O(1) in time and memory per signature (the window is a deque(maxlen=20), failure_ledger.py), takes one lock, never raises (note and the handler’s emit catch everything; degraded() and counts() iterate a snapshot, so a re-entrant note cannot break them, and if the ledger still fails /api/health answers degraded: [] and dependencies.failure_ledger = "failed: <Class>" and logs a warning, so a broken ledger does not read as a healthy engine), never logs (a ledger that logs a warning when it counts a warning is its own loop), and holds at most 500 signatures. Test: tests/unit/core/test_failure_ledger.py::test_note_never_raises_and_is_bounded and tests/unit/core/test_failure_ledger.py::test_the_ledger_holds_at_most_500_signatures, tests/unit/core/test_failure_ledger.py::test_degraded_survives_a_note_made_while_it_iterates, tests/unit/core/test_failure_ledger.py::test_health_still_answers_when_the_ledger_raises. Fixture: none (in-process).

B. Where a person sees it

  • engine.silent.health-lists-degraded — [OK] (built 2026-10-30; extends #4420’s activity.engine.warm-up-failures-are-said) GET /api/health carries degraded: [{id, area, reason, count, since}] (models/__init__.py DegradedFailure, HealthResponse.degraded), empty when nothing is degraded; every health answer passes through _with_server_proof (api/main.py), which fills it. status stays "healthy" (the engine answers; RemoteClientPairingFlow.swift:43 and every other caller that gates on the word are unchanged) and the list is the signal (open question 2, decided). since is the first failure’s local time, ISO 8601 without a zone. A warm-up failure stays in dependencies as today. The schema is generated: fichero-server/scripts/sync_openapi_schema.sh wrote the tracked openapi.json copies, contract_identity_generated.py and ContractIdentity.generated.swift; the Swift client’s Types.swift is generated at build and gains degraded then. Test: tests/unit/core/test_failure_ledger.py::test_health_carries_degraded and tests/contracts/openapi.json (the committed schema must match: check_contract_identity_current). Fixture: none (in-process).
  • engine.silent.app-says-degraded — [GAP] (#4420, no issue yet — proposed: “The app shows /api/health degraded: Connection status rows, Activity ‘Needs attention’, amber pill”) the app reads degraded where it already reads health (AppState+HealthCheck.swift:87, the heartbeat in AppState+Heartbeat.swift) and shows it in two places: the Connection status view (Views/Components/BackendConnection/BackendConnectionView+Status.swift; a row per entry: the reason sentence and “Since 09:19”), and Activity, as one “Needs attention” row per entry (not a job: no cancel, no progress) with the same sentence and a Copy Details button (area, count, since, last seen). The engine pill (Views/Shell/Toolbar/EngineStatusToolbarItem.swift) turns amber, not red (the engine answers). The row goes when the entry does. Test: DegradedHealthTests (Swift, unwritten): a health payload with one entry yields one row with that sentence and an amber pill; an empty list yields none and the pill as today. Plan: activity.window.honest-state is where the row is specified in the window (ui/activity-pane.md). (2026-11-18) A degraded entry whose cause is a model or a place (the embedder that cannot load) is also a row of the Models and Compute window (ui/models-and-compute-window.md, “Last failed”), and its kind is the one table’s (compute/runner-reliability.md: model-broken, with Download again), so the ledger’s words and the row’s words are one.
  • engine.silent.degraded-is-in-the-bug-report — [GAP] (#4420, no issue yet) “Send Logs to the Maintainer…” (App/Menus/Help/SaveLogsForSupport.swift on the collapse branch; not on integration yet) includes the current degraded list as a degraded.json beside the log, so a report says “Search was degraded since 09:19” without anyone reading 166 MB. Test: SendLogsToMaintainerTests (Swift) (on that branch; gains the case once both merge).

C. The guard: no unallowed repeat reaches a release

  • engine.silent.scan-script — [OK] (built 2026-10-30) scripts/scan_engine_log.py LOGFILE [LOGFILE ...] [--ceiling N] [--allow REGEX ...] [--level WARNING|ERROR|CRITICAL] is pure (stdlib, no network, streams the file line by line; a file with no newline is one line and is read whole), sums several files into one count (the engine’s engine.log and a rotated engine.log.1 are one log; pass both), reads lines in the engine’s format (asctime - name - level - message, LINE; tracebacks, kraken’s and [PARALLEL] prints are not log lines and are skipped), keeps WARNING and worse, normalises each to a template (uuid, hex of 8 or more digits with at least one digit, path, number → <uuid>/<hex>/<path>/<n>; a quoted path runs to its closing quote and an unquoted one to the end of the message, so spaces in it split nothing; the swallowed tail can merge two failures that share a prefix, a false FAIL, and an --allow on that shared prefix (or on <path>) then hides both: the limit is stated, not closed; a path runs to a closing quote followed by a space, punctuation or the end, so an apostrophe inside it is fine but a quote-then-space inside it leaks its tail; words before a path are printed; _SUBS), adds the N of a “— repeated N times in the last minute” line to the written template its raw %s template stands for (the raw template (%% read as %, each spec as “anything”) must match a written template of the same logger whole; the most-written one when several match; when unsure, fail: with no match the summary is kept alone and checked against the ceiling together with the most-written line of its logger, and an --allow must cover every candidate and the summary’s own raw template, so an allow naming only the cause does not hide it; _fold), and exits 1 when any template not matching an --allow regex is seen more than the ceiling (default 50), printing the offenders (count, level, first..last time, template). Exit 0 otherwise; exit 2 for everything that is not “over the ceiling”: a missing file, a file with no log line, a file where more date-led lines miss the engine format than match it, or more lines saying WARN/WARNING/ERROR/CRITICAL than there are parsed ones (+10) (a format change must not read as clean), when no file has a log line (an empty rotated partner is skipped), a bad --allow regex or bad arguments, any crash (exit 1 means only the ceiling; _entry). Test: tests/unit/scripts/test_scan_engine_log.py (27 tests: the 380k shape, INFO not counted, the summed total on a log_throttle-shaped summary and written+held over the ceiling is exit 1, an --allow naming the cause covers the summary, an allowed template, a clean log, a missing file, a non-log and a bad --allow regex are exit 2, a non-UTF-8 byte under an ASCII stdout still reports, two ids are one template, paths with spaces and short ids are one template, rotated logs are summed). Fixture: tests/fixtures/engine_logs/embedding_380k.log, tests/fixtures/engine_logs/embedding_collapsed.log, tests/fixtures/engine_logs/clean.log.
  • engine.silent.scan-runs-in-the-acceptance-run — [GAP] (#4420, #6313 — proposed: “Scan the acceptance run’s engine.log for repeats; report first, fail later”) the acceptance run (scripts/acceptance/_engine.py:145-196, which owns a private engine and writes its engine.log under ACCEPTANCE_OUT) scans that log in stop() and prints the result beside “engine stopped”; the run’s exit code is not changed in the first step (it becomes failing once the allowlist is honest, so the guard lands as a report first, not a red build nobody can fix this week). Nothing of this is built: stop() kills the process group and unlinks the socket and pid only. Test: tests/unit/scripts/test_scan_engine_log.py::test_acceptance_hook_reports (unwritten).
  • engine.silent.scan-runs-in-the-launch-smoke — [GAP] (#4420, #6313) scripts/smoke-launch-crash.sh launches the real app and watches for a crash (it reads no log today); after its settle window it also scans the new bytes of ~/Library/Containers/app.fichero.fichero/Data/Library/Logs/Fichero/engine.log (from the offset it recorded before launch) with scan_engine_log.py --ceiling 20, so a launch that starts a retry loop fails like a launch that crashes. The smoke script is a GUI run, owner opt-in (FICHERO_SMOKE_LAUNCH=1). Test: a shell case in the script’s own --self-test (unwritten; the script has none today).
  • engine.silent.allowlist-is-reasoned — [GAP] (#4420, #6313) --allow entries live in one file (scan_engine_log.allow (unwritten, will sit beside the script)), one regex and a one-line reason per entry; the script reads it with --allow-file and fails (exit 2) on an entry with no reason. Today nothing is allowed: on this inventory every WARNING-or-worse template over 50 is a finding (rows 1-3, 6, 9, 12-16). Test: tests/unit/scripts/test_scan_engine_log.py::test_an_allow_entry_without_a_reason_is_refused (unwritten).
  • engine.silent.scan-ceiling-and-ledger-thresholds-differ-on-purpose — [OK] the scan’s ceiling (50 per log file) and the ledger’s thresholds (20 a minute or 200 since start) measure different things: a whole run’s log after the fact (one test run should not repeat any warning 50 times) and a live process (a person must be told within a minute of a loop starting, and a slow leak of 200 is still a leak). Neither is tuned to the Air’s 3,000 a minute: anything under 100 a minute would have caught rows 1-3 in the first minute. Test: tests/unit/scripts/test_scan_engine_log.py::test_the_embedding_flood_fails_and_is_named (the ceiling) and tests/unit/core/test_failure_ledger.py::test_rate_threshold_degrades (the rate). Fixture: tests/fixtures/engine_logs/embedding_380k.log.

D. Tested against spec: the loop

  • engine.silent.every-bullet-names-its-proof — [OK] each engine.silent.<slug> bullet names the test that proves its failure is surfaced (Test: plus a tests/ path ending in .py, ::name for the case; a second test on the bullet counts) and a fixture log (Fixture: plus a path), or says Fixture: none (…) when the fault is in-process. The guard, Test: tests/unit/seams/test_engine_silent_spec_names_tests.py, reads this file and fails an [OK] bullet that names no test, names a test file or ::name that does not exist, or names neither a fixture nor Fixture: none, and any bullet that names a fixture path that does not exist (_problems). A bullet not yet [OK] may name a test that is not written: it is the work item, and the guard never fails on it. The ::name check is a def name text search in the file, not a run. Fixture: none (the guard reads this file).
  • engine.silent.real-log-becomes-fixture — [OK] the way a silent failure enters the suite: take the offending line from a real log, strip ids and paths, add it to tests/fixtures/engine_logs/<slug>.log, write the test that reads it, tag the bullet. Rows 1-3 are the first (embedding_380k.log: 200 real-shaped lines, both causes; embedding_collapsed.log: the same under the collapse). Rows 6, 12, 15-16 and 27 are the next four fixtures, one per unwritten test above. Test: tests/unit/scripts/test_scan_engine_log.py::test_the_embedding_flood_fails_and_is_named. Fixture: tests/fixtures/engine_logs/embedding_380k.log.

Plan (build order)

  1. scripts/scan_engine_log.py + tests + fixture logs. Built.
  2. core/failure_ledger.py + unit tests (counting, thresholds, quiet period, the health field). Built.
  3. One call site: the embedding warning in db/__init__.py (Database.embed’s except), failing test first; the line keeps its logger.warning and gains extra={"ledgered": True}. Built.
  4. The guard test test_engine_silent_spec_names_tests.py. Built.
  5. Next, in order: install the handler in lifespan (root logger, idempotent) and normalise paths in the handler; the collapsed summary counts toward the rate; the extra passthrough in log_collapsed (with the collapse worker, at merge); the logger→area table; ok() and the latch rule once the owner rules on question 3; then the other call sites (change hub, activity summary, export, vision, DuckDB I/O), the Swift rows, the acceptance report, the smoke hook, the allow file.

Open questions

  1. Where does the handler get installed? Decided by the code: core/logging.py is not the engine’s logging set-up (configure_logging is called nowhere; the handlers come from logging.basicConfig in __main__.py:35, which only the embedded app runs; the dev engine and the tests run uvicorn without it). So: the engine’s lifespan in api/main.py, logging.getLogger().addHandler(LedgerHandler()) once, guarded against a second install. It touches no file of the collapse worker’s and covers every logger under every launch.
  2. Is “degraded” a health status value or a separate list? Decided: a separate list. status is read as a word by RemoteClientPairingFlow.swift:43 ("healthy"/"ok"), LocalInferenceStore.swift:225 and the launch gate; a new value would make a working engine look down to them. The pill reads the list.
  3. Thresholds (20 a minute, 200 total, 5 minutes quiet). The numbers are sound against the inventory: rows 1-3 (3,000 a minute) trip the rate in under a second, or the total within the first minute’s summary under the collapse; row 15 (1,486 in a minute) and row 13 (26 a minute) trip the rate; row 12 (15 a minute) and row 6 (bursty, 12 a minute on average) trip the total within ~15 minutes; row 23 (491 over five days) is listed for 5 minutes after each hit once it passes 200, which is fair for a repeated refusal. Two caveats are gaps above, not threshold choices: the rate window must count the collapse’s held, and the quiet period is wrong for a fault that only shows on use (an idle engine with a dead embedder is “fine” after 5 minutes). For the owner: confirm the recommended default in engine.silent.ledger-success-clears (an explicit-site entry stays until ok() or restart; quiet applies to handler-seen signatures only).
  4. Does the replace of the app trigger the originals loss, or any engine restart? Not decided by the logs: every spawn filters the grants (FolderAccessManager.swift:193,419), so any restart drops a grant that covers no library, a replace included. Whether a replace also makes the stored bookmarks fail to resolve (then restoreBookmarks deletes them, :513-517) needs one launch after a replace with the path logged privacy: .public, or a unit test that resolves a bookmark minted by the previous build. Recommended default: build originals-access-restored-after-update for “any restart” (it covers the replace), and keep the resolve test as the owner’s one run. For the owner: the one run.
  5. Two branches, one line. This branch and feat/logs-stale-servers-support both rewrite the Database.embed warning in db/__init__.py. Whoever merges second resolves it as log_collapsed(logger, logging.WARNING, "Failed to create embedding for %s: %s", doc.id, e, cause=e, extra={"ledgered": True}) after log_collapsed gains the passthrough, with failure_ledger.note(...) kept beneath it; test_embedding_failure_degrades_health must still see exactly one search entry.

Issues filed (2026-10-31)

  1. Embedding failures flood the log: comment added on #4395 (installs the ledger handler, Activity row, amber pill).
  2. 6306 — Originals on an external volume show as thumbnails with no word why: say “originals not reachable”, the reason (not mounted / access not granted / moved), one-press Reconnect.

  3. 6307 — Linked originals’ folder grants are not re-sent after a restart or update (grantCoversALibrary drops them): the Project regression since #5228/#5251; stale bookmarks are named, not deleted.

  4. 6308 — /api/display, /api/thumbnail, /api/source 404s name why (shared classifier from _why_image_unavailable).

  5. 6309 — Live updates reach no window: change-hub emit key ‘…/Project.fichero’ ≠ subscribe key ‘…/Project 12.fichero’ (30k lines; #2518 recurrence).

  6. comment added on #6213 — Activity summary logs “no status for recipe run waiting:run-a-recipe” 3,717 times: a waiting row with no run behind it.
  7. 6310 — One failed page writes four log lines (ERROR, system_warning WARN, two INFO saves): one ledger entry, one line.

  8. 6311 — A library on an external volume that disconnects (DuckDB Input/output error ×93) is said as “volume disconnected”, not a 500 per request.

  9. 6312 — The ledger: paths are one template, the collapsed summary counts toward the rate, handler entries name a known area, ok() and the latch rule.

  10. 6313 — scan_engine_log.py in the acceptance run (report first) and the launch smoke; the reasoned allow file.

  11. 6314 — log_collapsed passes extra through (with the collapse worker, at merge).