Engine silent failures: what the logs say, and how to catch them
Status: DRAFT, 2026-10-30; revised against the code 2026-10-31. Owner’s order (2026-10-30): the engine must be rock solid, so silent failures need a way to be caught, specced, tested against spec and fixed, as the UX’s stall detector does for slowdowns. Manual: TBD — a “When something keeps failing” section: what the amber engine pill and an Activity “Needs attention” row mean, and what Reconnect does for originals on another volume. Tags: [OK] behaves this way today · [PARTIAL] part built or not wired · [GAP] intended, not built. Every non-[OK] bullet cites an issue. #4420 (closed; the seam programme) stands in for the bullets that have no issue of their own yet: the lead files the titles under “Issues to file” and replaces the number. Part 1 is the inventory (what the two real logs say). Part 2, “The detector”, is the spec, each bullet tagged with what is built. A
file:linein the inventory table is from the logged build; elsewhere a bullet names the function or file, not a line, because lines move.
Where the numbers come from
Streamed, never loaded whole, never copied: python read each engine.log line by line (on the Air through
ssh air, same script), kept every non-DEBUG line, and normalised hex ids, UUIDs, paths and digits to a template.
| Log | Size | Span | Lines (non-DEBUG) | Distinct templates |
|---|---|---|---|---|
This Mac, ~/Library/Containers/app.fichero.fichero/Data/Library/Logs/Fichero/engine.log |
57 MB | 2026-10-12 → 10-29 | 107,641 | 3,662 |
| The Air, same path | 167 MB (at 17:00) | 2026-09-22 → 10-30 | 592,479 | 3,400 |
One template, “Failed to create embedding”, is 389k of the Air’s 592k lines (66 %: rows 1 and 2; the Mac’s row 3 adds 21k more). Nothing on screen said so.
Top 30 templates (Mac + Air, by total count)
Class: SNR silent-no-result (work did not happen, the person sees nothing wrong) · RL retry-loop (the same
attempt repeated) · EN expected-noise (healthy, only loud) · RE real-error (a person is told, or should be).
“Mac / Air” are the counts per machine. Source is where the line is written (fichero_server/).
| # | Mac / Air | First → last | Lvl | Source | Class | What the person sees |
|---|---|---|---|---|---|---|
| 1 | 0 / 301,429 | 10-29 10:38 → 10-30 09:18 | WARN | db/__init__.py:6380 (raised at db/embed_process.py:452) “Failed to create embedding for ‹id›: The search model’s process stopped while loading the model; embedding for search will try again later.” |
SNR + RL | Nothing. Every document write queues an embedding, the embedder dies at load, the page is saved without one; Search later misses it. |
| 2 | 0 / 87,527 (still growing at 17:00) | 10-30 09:19 → | WARN | db/__init__.py:6380 “…could not be loaded: ValueError: Model BAAI/bge-m3 is not supported in TextEmbedding…” |
SNR + RL | Nothing. Same effect, a different cause after the 09:19 build (the model name is not in the installed fastembed). |
| 3 | 20,828 / 0 | 10-21 07:20 → 10-22 23:21 | WARN | db/__init__.py:6380 “…could not be loaded: Fail: [ONNXRuntimeError] … External data path validation failed for initializer…” |
SNR + RL | Nothing. Same line, a third cause (the model’s files). |
| 4 | 13,965 / 66,274 | 10-19 → 10-30 | INFO | core/single_flight.py:58 “poll activity.list_background_jobs took ‹n› ms…; ‹n› callers were served from a running query” |
EN | Nothing. Loud, not wrong: Activity’s poll. |
| 5 | 12,633 / 22,630 | 09-22 → 10-30 | INFO | workflows/checkpointer.py:136 “Initialized DuckDB checkpointer” |
EN (RL-shaped) | Nothing. A checkpointer opened per run step, not once; 35k lines. |
| 6 | 0 / 22,703 | 10-29 10:37 → 10-30 16:21 | WARN | api/change_stream.py:319 “change-hub: representation.created emit to key ‹/Volumes/Work/…/Project.fichero› reached 0 subscribers, but ‹n› live under different key(s): ‹…Project 12.fichero›” |
SNR | The open window does not update live (new results appear only after a reload). The same line for segment.updated, backend.work.progress, document.updated, backend.work.completed, project.setup_saved: ≈ 29,900 lines in all (#2518 closed; the key drift is back). |
| 7 | 6,855 / 4,157 | 09-22 → 10-30 | INFO | workflows/activity.py:181 “Activity saved successfully: ‹uuid›” |
EN | Nothing. |
| 8 | 1,088 / 9,369 (×2: reading order, topological sort) | 10-16 → 10-30 | INFO | kraken (third-party) “Compute reading order…” / “Perform topological sort…” |
EN | Nothing. Four more kraken lines per page (Vectorizing ×2, Segmenting): ~25k lines a day under a 61k-page run. |
| 9 | 0 / 5,888 | 10-29 10:34 → 10-30 16:17 | WARN | (the segment.updated variant of row 6) |
SNR | As row 6. |
| 10 | 0 / 5,532 + 4,118 | 10-28 → 10-30 | INFO | llm/__init__.py:486 “LLM usage [openrouter / anthropic …]: input=‹n›…” + httpx “HTTP Request: POST … 200 OK” (9.7k each) |
EN | Nothing. Usable as the cost record, but the log is the wrong place. |
| 11 | 2,588 / 1,985 | 09-22 → 10-30 | INFO | workflows/builder.py:2092 “Parallel processing file n/N: ‹path›” (+ 3,000 [PARALLEL] print lines) |
EN | Nothing. |
| 12 | 0 / 3,717 | 10-28 15:40 → 19:40 | WARN | api/routes/system/activity.py:425 “activity summary: no status for recipe run waiting:run-a-recipe: no recipe run waiting:run-a-recipe” |
RL (poll) | Activity’s summary for a waiting recipe row is blank; the poll repeats every few seconds. |
| 13 | 1,125 / 0 | 10-26 16:19 → 17:02 | WARN | execution/jobs.py:3701 “job ‹uuid› (find-lines on ‹id›) failed: Kraken segmentation failed: [Errno 2] No such file or directory: ‹path›” |
RE | The job is Failed in Activity (said). The cause, the original not at its place, is only in the text. |
| 14 | 888+822+796+773 (4 pages) / 0 | 10-26 16:19 → 17:04 | ERROR + WARN + INFO ×2 | workflows/tools/vision_base.py:6142/6161 “Vision processing failed for ‹name›: File not found: ‹path›” (the same failure written four times: ERROR, [system_warning] WARN, two “Saving activity” INFO) |
RE (×4 amplification) | One failed row per page; the log carries each failure four times. |
| 15 | 0 / 1,486 | 10-29 10:44 → 10:45 | ERROR | export_service.py:2022 “Export image source missing for document ‹id› (EAP…IMG_001.tif) path=’/Volumes/Archive/Originals/…’: this engine may not open files in that folder: grant access to the folder” |
SNR | A Word export finishes without the page images (row 16). The reason is in the log only. |
| 16 | 0 / 1,486 | same | WARN | export_service.py:1362 “Word export left out the image of ‹name›: …grant access to the folder” |
SNR | As row 15. |
| 17 | 1,576 / 0 + 781 / 1,440 | 10-16 → 10-30 | INFO | workflows/activity.py:178 “Saving activity to DB: node_started - Node ‘…’ started” |
EN | Nothing. |
| 18 | 790 / 1,180 | 09-22 → 10-30 | INFO | api/routes/document/documents.py “PERF library.list_documents duration_ms=‹n› …” (+ get_children, document_view) |
EN | Nothing. The existing perf lines. |
| 19 | 585 / 1,252 | 10-19 → 10-30 | INFO | core/single_flight.py:58 “poll activity.get_job_tree …” (+ recipes.recipe_run_status 857) |
EN | Nothing. |
| 20 | 1,157 / 404 | 10-16 → 10-30 | INFO | api/routes/system/storage.py:~190 “PERF library.thumbnail.endpoint cache_state=hit …” |
EN | Nothing. |
| 21 | 773 / 0 | 10-20 → 10-26 | INFO | same span, …duration_ms=‹n› error=HTTPException status=error |
SNR | A grid tile with no picture, no reason (the 404 is “Thumbnail not available”). |
| 22 | 491 / 970 | 09-24 → 10-30 | INFO | workflows/tools/llm_base.py “Created transcription artifact ‹id›” (+ “Updated page_content for ‹id›; embedding queued”, :911, 477 / 897) |
EN | Nothing. The “embedding queued” line is the promise rows 1-3 break. |
| 23 | 491 / 0 | 10-20 → 10-25 | WARN | api/main.py:218 “Rejected library path: path=‹…› home=‹container› failed_check=roots” (Air: 78, 10-28) |
RE | The library does not open; the app says it could not open it. |
| 24 | 0 / 997 | 10-28 15:11 → 20:07 | WARN | workflows/tools/extract_entities_only.py:271 “extract_entities_only: ‹name› left out: no text: no line of its working pass has a reading…” |
SNR (partial) | A names run says “done”; 997 pages contributed no names. Each is appended to the run’s left_out list with its reason (extract_entities_only.py:269-270); whether the run’s Activity row shows that count is unverified. |
| 25 | 999 / 271 | 10-20 → 10-28 | INFO | maintenance/project_conversion.py:479 “A new result on page ‹id›: converted” |
EN | Nothing. |
| 26 | 1,260 (Air) / 463 (Mac) | 10-19 → 10-30 | INFO | workflows/tools/vision_base.py:4935 “Kraken segmenter: ‹file›” (+ :5179 “LLM Vision: ‹file›”, 466) |
EN | Nothing. |
| 27 | 93 (Air) | 10-29 | ERROR | _duckdb.IOException … Could not read from file ‹…/Project.fichero/fichero.duckdb›: Input/output error |
RE | The external volume dropped under an open library. Every request on that library fails with the raw DuckDB text (no handler in fichero_server/ catches IOException or “Input/output error”); nothing names the volume. |
| 28 | 53 OK / 2 × 401 | 09-22 → 10-30 | INFO | POST /api/sandbox/security-scoped-access 200 / 401 (a grant before the engine is authenticated) |
EN | Nothing. |
| 29 | 1 per spawn | every spawn | INFO | security/security_scoped_access.py “Security-scoped access granted via inherited sandbox scope (bookmark unresolvable in this process): ‹path›” and “…: N of N librar(ies) granted” |
EN, but see the originals class below | Nothing. The N is the number of libraries, never the folders holding the originals. |
| 30 | 2 | 10-29 | WARN | api/auth.py “auth refused 401 branch=bootstrap_mismatch path=/api/workflows/folders” |
RE (one-off) | Nothing; the app retries. |
Rows 6 and 9, 15 and 16, 4 and 19, 3-way splits of rows 1-3 are one finding each in the classes below.
What the inventory says
- Three of the top six lines are one failure with three causes: the embedder cannot load, and every document
write retries it, logs a line and moves on. 389k lines in 38 hours on the Air. The code already knows the difference
(
EmbedOutcome.reason=embedder_unavailable, #4395) and the queue “waits and says why” (comment atdb/__init__.py,Database.embed); the line is written by the caller of each write, so the wait never reaches the log as one line, the log as a rate, or health as a state. - The engine already has the seams for “say it once, in a place a person looks” (
startup_warnings.pyinto/api/healthdependencies, the swallowed-exception seam baseline, the stall watchdog); none of them counts a repeated WARNING, and the app readsdependenciesonly for the About box’s versions and the previous stall (fichero/fichero/App/AppState/AppState+HealthCheck.swift:87-92): afailed: <Class>entry is shown nowhere. - Much of the Air’s log is INFO that would hide any real signal (5 of the top 10 templates): this is the log-collapse worker’s
(branch
feat/logs-stale-servers-support) ground; this spec only counts WARNING and above (see Part 2). - The engine’s log set-up is not
core/logging.py: the handlers that writeengine.logcome fromlogging.basicConfiginfichero_server/__main__.py:35(root logger, formatasctime - name - levelname - message; the app captures stdout/stderr into the file).core/logging.py’sconfigure_loggingis called nowhere infichero-server/src. The collapse branch addsengine_log.startto__main__.pyand collapses per call site (core/log_throttle.py::log_collapsed), not with a handler. This decides open question 1.
The classes (one bullet each; ids are engine.silent.<slug>)
Each class names the test that proves the failure is surfaced (Test:) and the fixture log that test and
scripts/scan_engine_log.py read (Fixture:). A bullet tagged [OK] must name a test file that exists; the guard is
tests/unit/seams/test_engine_silent_spec_names_tests.py (engine.silent.every-bullet-names-its-proof). “Surfaced”
means one of: a degraded entry in /api/health with a sentence, a reason in the endpoint’s 404 detail, or a worded
row in Activity; a log line alone never counts.
engine.silent.embedding-fails-every-write— [PARTIAL] (#4395 open, “Embedding failures are swallowed”; #6218 closed, the forked child dies while it downloads; #1524 closed, bge-m3 not supported) an embedder that cannot load makes every document write log one WARNING (389k on the Air) and produce no vector; health sayshealthy, Activity says nothing, Search silently loses the pages. Built 2026-10-30:Database.embed’sexceptcallsfailure_ledger.note("search", "embed", <cause class>, sentence=...)(db/__init__.py, inDatabase.embed) and/api/healthlists “Search is not indexing new pages: the search model cannot be loaded (EmbedderUnavailable). 25 failures since 09:19.” (the class isEmbedderUnavailable, whichembeddings.pyand two ofembed_process.py’s raises (a load timeout, the model’s own failure words) give nofrom, so those causes share one signature; a dead child raisesfromtheEOFError/OSError, so its reason says that class instead; the words that tell them apart stay in the engine log, not on health) Unbuilt: the Swift rows (app-says-degraded), and “N pages waiting” (the ledger counts failures; the queue’s own count is #4395’s). Test:tests/unit/core/test_failure_ledger.py::test_embedding_failure_degrades_health(built: 25 failing writes, onesearchentry, no path in the reason). Fixture:tests/fixtures/engine_logs/embedding_380k.log(the real line shape, both causes; read by the scan test).engine.silent.change-hub-key-drift— [PARTIAL] (#2518 closed, #6309) both hub seams (subscribe, unsubscribe, emit) call ONE function,canonical_library_key(db/library_paths.py: NFC likeDatabaseManager, thenrealpath), so an accented name in the form macOS hands a window and the form the engine opens reaches the same subscribers. An emit that reaches 0 subscribers warns only when a live key could be the same library (equal once case-folded, or the same file on disk); a real second library with no window (Project.ficheronext to an openProject 12.fichero) is debug, not a warning. Either way it logs once per key pair and again at 10, 100, 1000 … with the count, and the pair table is capped at 256 (api/change_stream.py,_ChangeHub.emit). Unbuilt: the ledger entry (arealive-updates) and the Activity words. Test:tests/unit/api/test_change_hub_key_drift_is_degraded.py::test_accented_name_in_either_unicode_form_reaches_the_windowand::test_same_library_under_another_spelling_warns_once_per_pair_with_a_count,::test_a_different_library_with_no_window_does_not_warnand::test_the_throttle_memory_is_bounded. Fixture: row 6’s line.engine.silent.originals-unreachable-shows-thumbnail— [GAP] (#5818, #4217 nearest open issues; #6306) see its own section below.engine.silent.export-leaves-images-out— [PARTIAL] (#5818 open, “Pages whose original is missing fail in words”) a Word export whose page images the engine may not open still finishes, naming the reason only in the log (export_service.py:1362WARNING,:2022ERROR;_why_image_unavailable:2027-2041already tells “moved or deleted”, “may not read it: grant access”, “may not open files in that folder”). Surfaced when: the export’s result carries “N page images left out: grant access to the folder” and its row in Activity says so. Test:tests/unit/export/test_word_export_says_images_left_out.py(unwritten; check #5818 first, it may land it). Fixture: rows 15-16.engine.silent.recipe-summary-no-status-loop— [OK] (#4420, #6213 — proposed title: “Activity summary logs ‘no status for recipe run waiting:run-a-recipe’ 3,717 times: a waiting row with no run behind it”)api/routes/system/activity.py:421-427catches every exception from_recipe_state, logs a WARNING and appends a block with no stage; the poll repeats it every few seconds, so the person sees a blank row and the log 3,717 lines. Test:fichero-server/tests/unit/jobs/test_activity_summary_one_source.py. Fixture: row 12. Built 2026-11-20: (#6745) awaiting:orpaused:queue row, which stands for many runs and is no run, builds its block from its own state and reason and logs nothing; testtests/unit/jobs/test_activity_summary_one_source.py::test_a_waiting_recipe_queue_row_has_a_status_and_logs_no_warning_6745.engine.silent.pages-left-out-of-a-names-run— [PARTIAL] (#4420, no issue yet) 997 pages “left out: no text” in one names run, one WARNING each (extract_entities_only.py:271). Built: each page is appended to the run’sleft_outlist with its reason (:269-270), so the count exists. Unbuilt: the run’s Activity row says “997 pages left out: no text”, and the per-page WARNING is the collapsed kind. Test:tests/unit/workflows/test_entities_only_left_out_is_counted.py(unwritten).engine.silent.a-failure-is-written-four-times— [GAP] (#4420, #6310 — proposed: “One failed page writes four log lines (ERROR, system_warning WARN, 2 INFO saves): collapse to one”) the vision failure path amplifies logs ×4 (3,300 lines for 4 pages). The handler counts every WARNING-or-worse template, so today the ERROR line and the[system_warning]WARNING (workflows/activity_types.py:47) would be two degraded entries for one failure. The fix is one of: the activity-save WARNING carriesextra={"ledgered": True}, or the visionexceptcallsfailure_ledger.note("reading", ...)once and both lines are marked. Test:tests/unit/workflows/test_vision_failure_logs_once.py(unwritten).engine.silent.thumbnail-404-without-reason— [OK] (#4420, #6308)/api/thumbnail(api/routes/system/storage.py) and/api/displayanswer 404 with{detail, reason}and words,reasonone of the unreadable-original codes (engine.silent.originals-reason-is-one-of-three),not_generated_yet(thumbnail only: the inline-generation permits are taken and the request is shed), orno_picture(the original is readable but no picture could be made)./api/displayhas no shed path, so it never answersnot_generated_yet;/api/sourcekeeps its bare 404 when the original is readable. 773 tiles in one Mac session. Test:tests/unit/api/test_thumbnail_404_names_reason.py.engine.silent.library-volume-dropped— [PARTIAL] (#4420, #6311) a DuckDB error from a disk that is not answering is raised asLibraryVolumeNotReachable(db/volume_dropped.py) and answered by one handler (api/main.py) as a typed 503{detail, reason: library_volume_not_reachable, library}: “the engine can’t reach the library’s disk” (the same error can be a failing internal disk, so it never says “disconnected” as fact), the library by display name only. The evidence is the disk, not the message: a real unplug first says “Cannot open file …wal” on a commit, never “Input/output error”, and later “database invalidated”, so any DuckDB error while the library file cannot be stat-ed, or an “Input/output error”, maps (classify). Mapped at_execute,_execute_fetch_with_columns,_read_fetch_with_columns,get_committed, schema_execute_locked,commit,checkpoint, BEGIN/COMMIT/ROLLBACK, and the manager’s two CHECKPOINTs (db/manager.py). The log line is written once per library per outage,failure_ledgerarealibrary-volumehas one entry per library (a counter), and the outage ends only when the file answers a stat AND a statement succeeds (a read served from DuckDB’s cache does not end it). A read DuckDB answers from cache during the outage is real data and stays 200. Not mapped (directself.conn.executesites no request reaches first): the open-time folds and migrations,COPY ... TO PARQUETexports, the bulk insert at~2578,~5288. Unbuilt: healthdegradedat the first failure (it waits for the ledger’s rate threshold) and the app saying it in words with Reconnect (Swift, a separate item). The real-unplug claim is built and tested; the item stays PARTIAL for the unbuilt parts above. Tests:tests/unit/db/test_real_unplug_is_said_6311.py(a real force-detached disk image, macOS, not under CI) andtests/unit/db/test_volume_dropped_is_said.py(one 503, one log line, one ledger entry, recovery; an invalidated database on a disk that answers is reopened, not refused). Fixture: row 27’s line, still to add totests/fixtures/engine_logs/.engine.silent.warm-up-failure-reaches-health— [OK] (#4420) a failed start-up warm-up (Keychain keys, Vision locales) is in/api/healthdependenciesasfailed: <Class>(startup_warnings.py:43-52). The app shows it nowhere (AppState+HealthCheck.swift:87-92readsdependenciesfor versions andprevious_stallonly): that isengine.silent.app-says-degraded’s work. Test:tests/unit/api/test_warm_up_failures_reach_health_4420.py. Fixture: none (an in-process fault).engine.silent.swallowed-handlers-cannot-grow— [OK] (#4420, #5315) a new handler whose body is only logging calls,passorcontinuefails the ratchet (test_swallowed_exception_seam.py:62-75,_only_logs). The sweep is syntactic: any other statement, afailure_ledger.note(...)included, takes the handler off the list. Test:tests/unit/seams/test_swallowed_exception_seam.py. Fixture: none.engine.silent.import-stall-is-loud— [OK] (#5807) a stalled import says so. Test:tests/unit/api/test_import_stall_is_loud.py. Fixture: none.engine.silent.info-lines-hide-the-signal— [PARTIAL] (#4420, no issue yet; the log-collapse worker’s branchfeat/logs-stale-servers-support: rotation incore/engine_log.py, repeated-line collapse incore/log_throttle.py, both unmerged) 5 of the top 10 templates are INFO. Not this spec’s to fix; this spec counts WARNING and above and reads the collapse’s summary record, never edits it.
The originals class (added 2026-10-30, from the Air)
engine.silent.originals-unreachable-shows-thumbnail: after the app was replaced by a new build, the Project source
images (61k pages, originals under /Volumes/Archive/Originals/…, display copies and library on
/Volumes/Work/Libraries/Project.fichero) “need to be reconnected” and the app shows the thumbnail
instead of the original.
What the code does today (read, not run):
- The fallback to the thumbnail is
fichero/fichero/Views/Preview/DocumentCanvas.swift:227-257: when the display image fails to load (loadError) and a cached thumbnail exists, it draws the thumbnail with a capsule “Showing thumbnail — original unavailable” and a Retry. (The maintainer asked on 2026-08-20 for the thumbnail to load and, when the original is missing, a brief flag.) So it is not totally silent, but the banner (a) does not say why, (b) offers only Retry, which cannot help when access is not granted, (c) is on the Preview canvas only: the grid, the Reader and every other place a thumbnail stands in (LibraryImageView.swift:79,seedFromCache) say nothing, and (d) a thumbnail is also what a generated grid tile is, so a person cannot tell “this is the small copy” from “this is all I could get”. - The engine’s answer carries no reason:
/api/display/{id}returns “Display image not available” (api/routes/system/storage.py:286),/api/source/{id}“Source file not available” (:341,:345; the cause goes to a WARNING with the path),/api/thumbnail“Thumbnail not available” (:225);get_display/ensure_displayreturnNonewhenresolve_source(db/storage.py:471-499,candidate.exists()) finds nothing: “not mounted”, “not found there” and “this engine may not read it” are the sameNone. The classifier already exists, for the export only:export_service._why_image_unavailable(:2027-2041:FileNotFoundError→ “moved or deleted”,PermissionError→ “grant access to its folder”, otherOSError→ “may not open files in that folder”). - The app’s grants:
FolderAccessManager.resendAllGrantsToEngine(fichero/fichero/Services/FolderAccessManager.swift:412, filter at:419) andengineBookmarkPayload(:187, filter at:193) send only the bookmarks that cover a library (grantCoversALibrary,:454, from #5228/#5251, 2026-09-28, “folders from long-closed libraries … were sent one request at a time”).restoreBookmarks(:460) deletes a bookmark that fails to resolve after onelogger.errorwhose path is<private>(:513-517); the engine’s refusal is logged the same way (:364).
The cause the Air’s engine.log supports (streamed, not copied; the Air’s stored bookmarks were read with
plistlib, keys only):
- Until 2026-09-29 each spawn said “Security-scoped access: 10 of 10 librar(ies) granted” and listed
/Volumes/Archive/Library/Series 1 … Series 8. - From 2026-10-28 every spawn says “3 of 3” (
…/Libraries,Project.fichero,Project 12.fichero): the folders holding the originals (/Volumes/Archive,/Volumes/Archive/Originals/Series 1/Box 12) are still stored as bookmarks in the app’s defaults (18 keys on the Air) but are not re-sent at spawn or in the post-ready sweep, because no library path lies under them. A link-mode project’s originals sit outside every library by design. They were granted only at the moment they were chosen in a session (10-28, 10-29: “granted via inherited sandbox scope”), and that grant ends with the engine process. - On 10-29 10:44 the Word export then failed 1,486 times with “Export image source missing … this engine may not
open files in that folder: grant access to the folder” for
/Volumes/Archive/Originals/…: that is the real cause in the engine’s own words, for the export path; the display path swallows the same condition intoNone. - The log has no
Operation not permitted,Errno 1orPermissionErrorline (0 hits on both machines): the engine never raised a permission error.os.staton a path outside the sandbox without a scope givesENOENT-like “not found”, whichresolve_sourcetreats as “moved”. The failure is therefore indistinguishable from a missing file at every layer until it is asked. - Not proven, flagged as the open question: that the replace of the app is the trigger. The evidence says “a
fresh engine process re-grants libraries only”; a replace restarts the engine, so it would show up on the first launch
after one, but any restart does the same. Whether the app-scoped bookmarks themselves still resolve after a replace
with a new build (same bundle id, same Developer ID team QAPB6CWYR6) was not testable from logs: the app’s
FolderAccesslog lines are<private>in the unified log (Engine refused folder access for <private>, 3 times on 10-30: 09:19, 12:37, 16:36, each at a spawn). Needs one run with the log made public for the path, or a bookmark-resolve test.
What the person should see (spec, unbuilt; ids in the bullets):
engine.silent.originals-unreachable-says-so— [GAP] (#5818, #4217 nearest open issues; #6306 — proposed: “Originals on a volume are shown as thumbnails with no word why; say ‘originals not reachable’ with the reason and a Reconnect…”) when an original cannot be read, the window says so once, where the person is looking: a banner on the project (and an Activity row, kind “needs attention”) “Originals not reachable: Reconnect…”, never only a capsule on one pane. The thumbnail stays as the fallback, marked (a small “Small copy” badge on a preview, not on the grid’s own tiles).engine.silent.originals-reason-is-one-of-three— [PARTIAL] (#6306, #6308) built:GET /api/storage/thumbnail,/displayand/sourceanswer 404 with{detail, reason}when the recorded original is not a readable file,reasonone ofno_recorded_file,volume_not_mounted(a path under/Volumes/<name>with no/Volumes/<name>),moved_or_deleted,access_not_granted(PermissionError, or the path stats butresolve_sourcerefuses it, e.g. a path naming..),unreadable(anotherOSError), frommedia/image_reason.classify_image_unavailable, which the export’s_why_image_unavailablealso uses. The path classified is the one the resolver tries first: the document’s own, thenmetadata.source_path/full_path/display_path/local_path, and for a page with none (a PDF page) its parent’s. A recorded path that cannot be statted at all (a NUL, a~userthat does not exist) isunreadable. Whenresolve_sourcefinds the source,/api/thumbnailand/api/displayanswerno_picture(the original is readable but no picture could be made) and the thumbnail shed path answersnot_generated_yet(seeengine.silent.thumbnail-404-without-reason); only/api/sourcekeeps its bare 404. The body never carries a local path, only a drive’s name; the classifier runs off the event loop. Not built: thegranted_paths()test for a path on a mounted volume (that case reads asmoved_or_deleted), and the app showing the words. Test:tests/unit/api/test_display_404_names_why_unreachable_original.py. As first specified: the engine names why in a field the app shows in words, from the classifier that exists inexport_service._why_image_unavailable, moved to a shared helper: “volume not mounted” (/Volumes/<name>absent:os.path.ismountof the first two components false), “access not granted” (the path exists in a listing the engine cannot read, or the folder is not under a held grant:PermissionError, orENOENTfor a path under/Volumes/while the volume is mounted andgranted_paths()does not cover it), “moved or deleted” (the volume is mounted and covered, the file is not there: #5535, #5672 relink).GET /api/display,/api/thumbnail,/api/sourcecarry it in the 404’sdetailas a stable code plus words.engine.silent.originals-one-press-reconnect— [GAP] (#4217, the Mac App Store sibling “no bookmark exists and nothing re-prompts”; #6306) “Reconnect…” opens the folder picker at the saved path (FolderAccessManager.requestFolderAccess(suggestedPath:),:119, exists), mints the bookmark, hands it to the running engine (grantEngineAccess,:339, private today), and reloads what was shown; “volume not mounted” says “Connect the drive ‘Archive’” and offers no picker. One press, no restart.engine.silent.originals-access-restored-after-update— [GAP] (#5818, #4217; #6307) after an update or any restart, the app re-sends every stored bookmark whose folder holds a project’s originals (the paths indocuments.pathof open libraries, reduced to their shared parent), not only those that cover a library; a stored bookmark that no longer resolves is named in the banner (“Access to ‘/Volumes/Archive’ was lost: Reconnect…”) and kept until the person reconnects or removes it (todayrestoreBookmarksdeletes it after a<private>error line,FolderAccessManager.swift:513-517: a stale bookmark becomes a quiet loss). Test:OriginalsGrantSweepTests(Swift, unwritten) (unwritten;LaunchLoadOrderTests.swift:18-19pinsgrantCoversALibrary, which is exactly the filter this changes, so that test changes with it).- Test for the engine half:
tests/unit/api/test_display_404_names_why_unreachable_original.py. Fixture: the Air’s “Export image source missing … grant access to the folder” line (row 15).
The detector
Intent
The UX has two things the engine lacks: a measure (a stall is timed, written to stalls.log, pinned by a
perf test with a ceiling) and a place a person sees it (the app says it is slow). The engine has the first half for
three things (warm-up failures, import stalls, held transactions) and nothing for the rest: a WARNING is a line, and
a line repeated 389,000 times is still just lines.
So: count every WARNING-or-worse by what it says, turn a count that is too high into a named “degraded” state with a
reason a person can read, serve that state where the app already looks (/api/health), and fail the build when a
fixture or a real run produces a repeat nobody allowed. The same loop the specs use: each silent class is a bullet,
each bullet names the test that proves the failure is surfaced and a log line that test reads, and a guard fails when
a bullet claims [OK] and the test is not there.
Design constraints (the owner’s, and the other worker’s):
- Beside the log collapse, never inside it. The log-collapse worker’s branch (
feat/logs-stale-servers-support) collapses per call site:core/log_throttle.py::log_collapsedwrites the first 5 repeats of a template per minute, then one record whosemsgis exactly"%s — repeated %d times in the last minute"with args(template, held)(log_throttle.py:60-63;heldis the count beyond the 5). A ledger reading the file would undercount by 99 %; a ledger that is alogging.Handlersees the 5 and addsheldfrom the summary record’s args, so 5 + held is exact. The two features do not know each other, except at one line: both branches rewrite theDatabase.embedwarning indb/__init__.py(theirs tolog_collapsed(..., cause=e), which takes noextra; ours addsextra={"ledgered": True}), seeengine.silent.ledger-counts-by-signature. - Count the template, not the text. Ids, paths and numbers make every line unique; the signature is the logger name
plus the record’s
msg(the%stemplate, before its arguments); for an f-string message (no args) digits and hex are normalised toN(failure_ledger.py). Paths are not, which is the gap named in the same bullet. - Health is unauthenticated: a reason carries a class name and a count, never a path, id or message text (the rule
startup_warnings.py:33-36already keeps). The entry’s ownidis a 12-character hash of the signature, because the signature of a record the handler sees holds its message text;test_a_degraded_id_carries_no_message_text. - No new dependency, no thread, no I/O: a dict under a re-entrant lock (a warning logged while counting must not deadlock), at most
MAX_SIGNATURES= 500 signatures (failure_ledger.py); the clock is injected (now=) so tests need no sleep.
Behaviors
A. The failure ledger (fichero_server/core/failure_ledger.py)
engine.silent.ledger-counts-by-signature— [PARTIAL] (#4420, #6312 — proposed: “Install the failure-ledger handler; count paths as one template”) every WARNING-or-worse record that reaches the root logger is counted by signature (logger name | template,failure_ledger.py), and the count stays exact when the line is collapsed (the summary record adds itsheldcount,:118-119). Built:LedgerHandler(:107-125) andnote()for explicit sites (:63-68); a record from a site that already callednotecarriesextra={"ledgered": True}and is skipped (:115). Three gaps: (a) the handler is installed nowhere (:9-10); it goes on the root logger (logging.getLogger().addHandler) in the engine’slifespan(api/main.py), once, idempotent, so the embedded app, the dev uvicorn engine and the tests all have it (open question 1, decided); (b) an f-string record keeps its paths (:31normalises digits and hex only), so one message with a path per document is one signature per document, fills the 500-signature cap (:26,:75-76) and every later new failure is dropped silently: normalise paths asscan_engine_log.py:22-24does; (c) after the collapse branch merges, the embedding line goes throughlog_collapsed, which drops theextra, so the handler andnotewould each count it (two entries for one failure):log_collapsedgains anextra=passthrough, ornote(..., template=...)registers the template the handler must skip. Test:tests/unit/core/test_failure_ledger.py::test_counts_by_template_not_text,::test_a_collapsed_summary_adds_its_count,::test_info_is_not_counted,::test_a_line_that_noted_is_counted_once_by_the_handler(built);::test_a_collapsed_ledgered_line_is_counted_once(#6314: written, skips until the collapse worker merges, and fails there iflog_collapsedstill dropsextra);::test_handler_is_installed_on_the_root_logger,::test_paths_are_one_template(unwritten). Fixture: none (records are made in-process; the file shape is the scan script’s test).engine.silent.ledger-counts-a-swallowed-exception— [GAP] (#4420) a broad handler whose caller genuinely cannot act on the failure callsfailure_ledger.note("<area>", "<site>", kind=type(exc).__name__)beside its log line; the seam sweep (test_swallowed_exception_seam.py:62-75) then no longer lists it, and its baseline entry must be deleted in the same commit (the sweep fails a stale entry). The sweep is syntactic, sonotemust not launder a handler whose caller could use a return value: #4420’s rule (“a skip is distinguishable from a failure in the return value”) still applies first;noteis for the handler that stays broad. Migrating the 200-odd baselined handlers is not in this spec’s first slice: the embedding, reading (OCR/HTR, vision, Kraken), search and storage paths go first. Test: the seam test itself, plus onenotetest per migrated area (tests/unit/<area>/test_<area>_failures_reach_the_ledger.py, unwritten).engine.silent.ledger-thresholds-degrade— [PARTIAL] (#4420, #6312 — proposed: “The ledger’s rate window counts records, not collapsed repeats”) a signature is degraded when it crosses either threshold and stays listed until it has been quiet: rate = 20 hits within 60 s (RATE_COUNT,RATE_WINDOW_S,failure_ledger.py), total = 200 since the process started (TOTAL,:24,81), quiet = not listed once 300 s pass with no new hit (QUIET_S,:25,96).trippednever resets (:82): after a quiet spell one new hit re-lists the entry for another 5 minutes, which is wanted (the 201st failure is still the same problem). Gap: the rate window appends one timestamp per record (:80) whatevercountit carries, so under the collapse (5 records + 1 summary a minute) the rate threshold can never trip and only the total does: a 30-a-minute loop is listed after ~7 minutes instead of 40 s. The summary’sheldmust count toward the window. Per-area overrides (a model that cannot load is degraded at 5 hits, not 20) are a later table in the module. Test:tests/unit/core/test_failure_ledger.py::test_rate_threshold_degrades,::test_total_threshold_degrades,::test_quiet_period_clears,::test_slow_repeats_below_the_rate_do_not_degrade_until_the_total(built);::test_a_collapsed_summary_counts_toward_the_rate(unwritten).engine.silent.ledger-reason-is-plain-language— [OK] each degraded entry says, in one sentence a person reads without the log, what stopped (the site’s sentence), why (the exception class, not the message) and how many:"<sentence> (<Class>). <count> failures since <HH:MM>."(failure_ledger.py; e.g. “Search is not indexing new pages: the search model cannot be loaded (EmbedderUnavailable). 87527 failures since 09:19.”). A record seen only by the handler gets"<area> keeps reporting the same problem: <count> times since <HH:MM>.". Test:tests/unit/core/test_failure_ledger.py::test_reason_is_plain_and_carries_no_path_or_id. Fixture: none (in-process).engine.silent.ledger-handler-entry-names-its-area— [GAP] (#4420, #6312) for a handler-seen record theareais the logger name’s last component (failure_ledger.py:db,main,jobs,change_stream), which is not a word a person knows. A small table maps logger prefixes to areas (fichero_server.db→ “Library”,fichero_server.execution→ “Jobs”,fichero_server.api.change_stream→ “Live updates”,fichero_server.workflows→ “Reading”; unknown → “The engine”), and the reason reads “Live updates keep reporting the same problem: 22,703 times since 10:37.” Test:tests/unit/core/test_failure_ledger.py::test_handler_entry_names_a_known_area(unwritten).engine.silent.ledger-success-clears— [GAP] (#4420, #6312) a site that can also succeed (embedding does) callsledger.ok(area)on success, so a recovered engine stops saying degraded at once. This also fixes a soundness hole in the quiet period: a model that cannot load fails only when a write happens, so an idle engine’s entry drops off the list after 5 quiet minutes while the model is still broken, and comes back with the next write. Recommended default (owner to confirm, open question 3): an explicit-site entry (one with a sentence) stays listed untilok()or restart; the 5-minute quiet applies only to handler-seen signatures, which have no site to callok(). Test:tests/unit/core/test_failure_ledger.py::test_ok_clears_at_once,::test_an_explicit_entry_outlives_the_quiet_period(unwritten).engine.silent.ledger-is-cheap— [OK]noteis O(1) in time and memory per signature (the window is adeque(maxlen=20),failure_ledger.py), takes one lock, never raises (noteand the handler’semitcatch everything;degraded()andcounts()iterate a snapshot, so a re-entrantnotecannot break them, and if the ledger still fails/api/healthanswersdegraded: []anddependencies.failure_ledger = "failed: <Class>"and logs a warning, so a broken ledger does not read as a healthy engine), never logs (a ledger that logs a warning when it counts a warning is its own loop), and holds at most 500 signatures. Test:tests/unit/core/test_failure_ledger.py::test_note_never_raises_and_is_boundedandtests/unit/core/test_failure_ledger.py::test_the_ledger_holds_at_most_500_signatures,tests/unit/core/test_failure_ledger.py::test_degraded_survives_a_note_made_while_it_iterates,tests/unit/core/test_failure_ledger.py::test_health_still_answers_when_the_ledger_raises. Fixture: none (in-process).
B. Where a person sees it
engine.silent.health-lists-degraded— [OK] (built 2026-10-30; extends #4420’sactivity.engine.warm-up-failures-are-said)GET /api/healthcarriesdegraded: [{id, area, reason, count, since}](models/__init__.pyDegradedFailure,HealthResponse.degraded), empty when nothing is degraded; every health answer passes through_with_server_proof(api/main.py), which fills it.statusstays"healthy"(the engine answers;RemoteClientPairingFlow.swift:43and every other caller that gates on the word are unchanged) and the list is the signal (open question 2, decided).sinceis the first failure’s local time, ISO 8601 without a zone. A warm-up failure stays independenciesas today. The schema is generated:fichero-server/scripts/sync_openapi_schema.shwrote the trackedopenapi.jsoncopies,contract_identity_generated.pyandContractIdentity.generated.swift; the Swift client’sTypes.swiftis generated at build and gainsdegradedthen. Test:tests/unit/core/test_failure_ledger.py::test_health_carries_degradedandtests/contracts/openapi.json(the committed schema must match:check_contract_identity_current). Fixture: none (in-process).engine.silent.app-says-degraded— [GAP] (#4420, no issue yet — proposed: “The app shows /api/healthdegraded: Connection status rows, Activity ‘Needs attention’, amber pill”) the app readsdegradedwhere it already reads health (AppState+HealthCheck.swift:87, the heartbeat inAppState+Heartbeat.swift) and shows it in two places: the Connection status view (Views/Components/BackendConnection/BackendConnectionView+Status.swift; a row per entry: the reason sentence and “Since 09:19”), and Activity, as one “Needs attention” row per entry (not a job: no cancel, no progress) with the same sentence and a Copy Details button (area, count, since, last seen). The engine pill (Views/Shell/Toolbar/EngineStatusToolbarItem.swift) turns amber, not red (the engine answers). The row goes when the entry does. Test:DegradedHealthTests(Swift, unwritten): a health payload with one entry yields one row with that sentence and an amber pill; an empty list yields none and the pill as today. Plan:activity.window.honest-stateis where the row is specified in the window (ui/activity-pane.md). (2026-11-18) A degraded entry whose cause is a model or a place (the embedder that cannot load) is also a row of the Models and Compute window (ui/models-and-compute-window.md, “Last failed”), and its kind is the one table’s (compute/runner-reliability.md:model-broken, with Download again), so the ledger’s words and the row’s words are one.engine.silent.degraded-is-in-the-bug-report— [GAP] (#4420, no issue yet) “Send Logs to the Maintainer…” (App/Menus/Help/SaveLogsForSupport.swifton the collapse branch; not on integration yet) includes the currentdegradedlist as adegraded.jsonbeside the log, so a report says “Search was degraded since 09:19” without anyone reading 166 MB. Test:SendLogsToMaintainerTests(Swift) (on that branch; gains the case once both merge).
C. The guard: no unallowed repeat reaches a release
engine.silent.scan-script— [OK] (built 2026-10-30)scripts/scan_engine_log.py LOGFILE [LOGFILE ...] [--ceiling N] [--allow REGEX ...] [--level WARNING|ERROR|CRITICAL]is pure (stdlib, no network, streams the file line by line; a file with no newline is one line and is read whole), sums several files into one count (the engine’sengine.logand a rotatedengine.log.1are one log; pass both), reads lines in the engine’s format (asctime - name - level - message,LINE; tracebacks, kraken’s and[PARALLEL]prints are not log lines and are skipped), keeps WARNING and worse, normalises each to a template (uuid, hex of 8 or more digits with at least one digit, path, number →<uuid>/<hex>/<path>/<n>; a quoted path runs to its closing quote and an unquoted one to the end of the message, so spaces in it split nothing; the swallowed tail can merge two failures that share a prefix, a false FAIL, and an--allowon that shared prefix (or on<path>) then hides both: the limit is stated, not closed; a path runs to a closing quote followed by a space, punctuation or the end, so an apostrophe inside it is fine but a quote-then-space inside it leaks its tail; words before a path are printed;_SUBS), adds the N of a “— repeated N times in the last minute” line to the written template its raw%stemplate stands for (the raw template (%%read as%, each spec as “anything”) must match a written template of the same logger whole; the most-written one when several match; when unsure, fail: with no match the summary is kept alone and checked against the ceiling together with the most-written line of its logger, and an--allowmust cover every candidate and the summary’s own raw template, so an allow naming only the cause does not hide it;_fold), and exits 1 when any template not matching an--allowregex is seen more than the ceiling (default 50), printing the offenders (count, level, first..last time, template). Exit 0 otherwise; exit 2 for everything that is not “over the ceiling”: a missing file, a file with no log line, a file where more date-led lines miss the engine format than match it, or more lines saying WARN/WARNING/ERROR/CRITICAL than there are parsed ones (+10) (a format change must not read as clean), when no file has a log line (an empty rotated partner is skipped), a bad--allowregex or bad arguments, any crash (exit 1 means only the ceiling;_entry). Test:tests/unit/scripts/test_scan_engine_log.py(27 tests: the 380k shape, INFO not counted, the summed total on alog_throttle-shaped summary and written+held over the ceiling is exit 1, an--allownaming the cause covers the summary, an allowed template, a clean log, a missing file, a non-log and a bad--allowregex are exit 2, a non-UTF-8 byte under an ASCII stdout still reports, two ids are one template, paths with spaces and short ids are one template, rotated logs are summed). Fixture:tests/fixtures/engine_logs/embedding_380k.log,tests/fixtures/engine_logs/embedding_collapsed.log,tests/fixtures/engine_logs/clean.log.engine.silent.scan-runs-in-the-acceptance-run— [GAP] (#4420, #6313 — proposed: “Scan the acceptance run’s engine.log for repeats; report first, fail later”) the acceptance run (scripts/acceptance/_engine.py:145-196, which owns a private engine and writes itsengine.logunderACCEPTANCE_OUT) scans that log instop()and prints the result beside “engine stopped”; the run’s exit code is not changed in the first step (it becomes failing once the allowlist is honest, so the guard lands as a report first, not a red build nobody can fix this week). Nothing of this is built:stop()kills the process group and unlinks the socket and pid only. Test:tests/unit/scripts/test_scan_engine_log.py::test_acceptance_hook_reports(unwritten).engine.silent.scan-runs-in-the-launch-smoke— [GAP] (#4420, #6313)scripts/smoke-launch-crash.shlaunches the real app and watches for a crash (it reads no log today); after its settle window it also scans the new bytes of~/Library/Containers/app.fichero.fichero/Data/Library/Logs/Fichero/engine.log(from the offset it recorded before launch) withscan_engine_log.py --ceiling 20, so a launch that starts a retry loop fails like a launch that crashes. The smoke script is a GUI run, owner opt-in (FICHERO_SMOKE_LAUNCH=1). Test: a shell case in the script’s own--self-test(unwritten; the script has none today).engine.silent.allowlist-is-reasoned— [GAP] (#4420, #6313)--allowentries live in one file (scan_engine_log.allow(unwritten, will sit beside the script)), one regex and a one-line reason per entry; the script reads it with--allow-fileand fails (exit 2) on an entry with no reason. Today nothing is allowed: on this inventory every WARNING-or-worse template over 50 is a finding (rows 1-3, 6, 9, 12-16). Test:tests/unit/scripts/test_scan_engine_log.py::test_an_allow_entry_without_a_reason_is_refused(unwritten).engine.silent.scan-ceiling-and-ledger-thresholds-differ-on-purpose— [OK] the scan’s ceiling (50 per log file) and the ledger’s thresholds (20 a minute or 200 since start) measure different things: a whole run’s log after the fact (one test run should not repeat any warning 50 times) and a live process (a person must be told within a minute of a loop starting, and a slow leak of 200 is still a leak). Neither is tuned to the Air’s 3,000 a minute: anything under 100 a minute would have caught rows 1-3 in the first minute. Test:tests/unit/scripts/test_scan_engine_log.py::test_the_embedding_flood_fails_and_is_named(the ceiling) andtests/unit/core/test_failure_ledger.py::test_rate_threshold_degrades(the rate). Fixture:tests/fixtures/engine_logs/embedding_380k.log.
D. Tested against spec: the loop
engine.silent.every-bullet-names-its-proof— [OK] eachengine.silent.<slug>bullet names the test that proves its failure is surfaced (Test:plus atests/path ending in.py,::namefor the case; a second test on the bullet counts) and a fixture log (Fixture:plus a path), or saysFixture: none (…)when the fault is in-process. The guard, Test:tests/unit/seams/test_engine_silent_spec_names_tests.py, reads this file and fails an[OK]bullet that names no test, names a test file or::namethat does not exist, or names neither a fixture norFixture: none, and any bullet that names a fixture path that does not exist (_problems). A bullet not yet [OK] may name a test that is not written: it is the work item, and the guard never fails on it. The::namecheck is adef nametext search in the file, not a run. Fixture: none (the guard reads this file).engine.silent.real-log-becomes-fixture— [OK] the way a silent failure enters the suite: take the offending line from a real log, strip ids and paths, add it totests/fixtures/engine_logs/<slug>.log, write the test that reads it, tag the bullet. Rows 1-3 are the first (embedding_380k.log: 200 real-shaped lines, both causes;embedding_collapsed.log: the same under the collapse). Rows 6, 12, 15-16 and 27 are the next four fixtures, one per unwritten test above. Test:tests/unit/scripts/test_scan_engine_log.py::test_the_embedding_flood_fails_and_is_named. Fixture:tests/fixtures/engine_logs/embedding_380k.log.
Plan (build order)
scripts/scan_engine_log.py+ tests + fixture logs. Built.core/failure_ledger.py+ unit tests (counting, thresholds, quiet period, the health field). Built.- One call site: the embedding warning in
db/__init__.py(Database.embed’sexcept), failing test first; the line keeps itslogger.warningand gainsextra={"ledgered": True}. Built. - The guard test
test_engine_silent_spec_names_tests.py. Built. - Next, in order: install the handler in
lifespan(root logger, idempotent) and normalise paths in the handler; the collapsed summary counts toward the rate; theextrapassthrough inlog_collapsed(with the collapse worker, at merge); the logger→area table;ok()and the latch rule once the owner rules on question 3; then the other call sites (change hub, activity summary, export, vision, DuckDB I/O), the Swift rows, the acceptance report, the smoke hook, the allow file.
Open questions
- Where does the handler get installed? Decided by the code:
core/logging.pyis not the engine’s logging set-up (configure_loggingis called nowhere; the handlers come fromlogging.basicConfigin__main__.py:35, which only the embedded app runs; the dev engine and the tests run uvicorn without it). So: the engine’slifespaninapi/main.py,logging.getLogger().addHandler(LedgerHandler())once, guarded against a second install. It touches no file of the collapse worker’s and covers every logger under every launch. - Is “degraded” a health
statusvalue or a separate list? Decided: a separate list.statusis read as a word byRemoteClientPairingFlow.swift:43("healthy"/"ok"),LocalInferenceStore.swift:225and the launch gate; a new value would make a working engine look down to them. The pill reads the list. - Thresholds (20 a minute, 200 total, 5 minutes quiet). The numbers are sound against the inventory: rows 1-3 (3,000 a minute) trip the rate in under a second, or the total within the first minute’s summary under the collapse; row 15 (1,486 in a minute) and row 13 (26 a minute) trip the rate; row 12 (15 a minute) and row 6 (bursty, 12 a minute on average) trip the total within ~15 minutes; row 23 (491 over five days) is listed for 5 minutes after each hit once it passes 200, which is fair for a repeated refusal. Two caveats are gaps above, not threshold choices: the rate window must count the collapse’s
held, and the quiet period is wrong for a fault that only shows on use (an idle engine with a dead embedder is “fine” after 5 minutes). For the owner: confirm the recommended default inengine.silent.ledger-success-clears(an explicit-site entry stays untilok()or restart; quiet applies to handler-seen signatures only). - Does the replace of the app trigger the originals loss, or any engine restart? Not decided by the logs: every spawn filters the grants (
FolderAccessManager.swift:193,419), so any restart drops a grant that covers no library, a replace included. Whether a replace also makes the stored bookmarks fail to resolve (thenrestoreBookmarksdeletes them,:513-517) needs one launch after a replace with the path loggedprivacy: .public, or a unit test that resolves a bookmark minted by the previous build. Recommended default: buildoriginals-access-restored-after-updatefor “any restart” (it covers the replace), and keep the resolve test as the owner’s one run. For the owner: the one run. - Two branches, one line. This branch and
feat/logs-stale-servers-supportboth rewrite theDatabase.embedwarning indb/__init__.py. Whoever merges second resolves it aslog_collapsed(logger, logging.WARNING, "Failed to create embedding for %s: %s", doc.id, e, cause=e, extra={"ledgered": True})afterlog_collapsedgains the passthrough, withfailure_ledger.note(...)kept beneath it;test_embedding_failure_degrades_healthmust still see exactly onesearchentry.
Issues filed (2026-10-31)
- Embedding failures flood the log: comment added on #4395 (installs the ledger handler, Activity row, amber pill).
-
6306 — Originals on an external volume show as thumbnails with no word why: say “originals not reachable”, the reason (not mounted / access not granted / moved), one-press Reconnect.
-
6307 — Linked originals’ folder grants are not re-sent after a restart or update (
grantCoversALibrarydrops them): the Project regression since #5228/#5251; stale bookmarks are named, not deleted. -
6308 —
/api/display,/api/thumbnail,/api/source404s name why (shared classifier from_why_image_unavailable). -
6309 — Live updates reach no window: change-hub emit key ‘…/Project.fichero’ ≠ subscribe key ‘…/Project 12.fichero’ (30k lines; #2518 recurrence).
- comment added on #6213 — Activity summary logs “no status for recipe run waiting:run-a-recipe” 3,717 times: a waiting row with no run behind it.
-
6310 — One failed page writes four log lines (ERROR, system_warning WARN, two INFO saves): one ledger entry, one line.
-
6311 — A library on an external volume that disconnects (DuckDB Input/output error ×93) is said as “volume disconnected”, not a 500 per request.
-
6312 — The ledger: paths are one template, the collapsed summary counts toward the rate, handler entries name a known area,
ok()and the latch rule. -
6313 —
scan_engine_log.pyin the acceptance run (report first) and the launch smoke; the reasoned allow file. -
6314 —
log_collapsedpassesextrathrough (with the collapse worker, at merge).