Refetch & validate

Refetch

khiipd refetch re-runs one dimension of an existing capture:

khiipd refetch 01JX9...               # network re-fetch: runs the extractor again and
                                      # creates a new capture that supersedes the old one
khiipd refetch 01JX9... --re-extract  # OFFLINE: re-derive the payload from preserved bytes
khiipd refetch 01JX9... --re-render   # OFFLINE: re-render the Markdown from the payload
khiipd refetch 01JX9... --media       # re-walk the media-fetcher registry in place
khiipd refetch 01JX9... --wayback     # re-submit to the Wayback Machine in place

The dimensions

Command	Dimension	Effect
khiipd refetch <id>	extraction (default)	Re-runs the extractor against the original URL and writes a new capture; the old one is marked superseded (below).
… --re-extract	re-extract	Offline, in place — re-derives the typed payload from the preserved Source-tier bytes (the raw bytes saved at capture time) with the current extractor. No network; works even if the source has been deleted upstream. For Wikipedia captures made with 0.2.4+, the offline rebuild replays the rich body (references + infobox) from the preserved {query,parse} bundle — no network needed; captures made before 0.2.4 kept only the plaintext extract offline, so for those the rich body still returns via a network refetch (see the callout below).
… --re-render	re-render	Offline, in place — re-renders the Markdown body from the existing typed payload (apply an improved renderer / skin).
… --media	media	Re-walks the media-fetcher registry on the existing capture in place — same id, same vault file. For retrying a media download that previously failed.
… --wayback	wayback	Re-submits the canonical URL to the Wayback Machine in place, updating the capture’s archive_urls. Requires [archive] wayback_enabled = true (off by default).

The four dimension flags are mutually exclusive. All of them map to POST /api/v1/captures/{id}/refetch?dimension={extraction|re-extract|re-render|media|wayback}, and the dimensions are independently re-attemptable per the failure-handling model — a failed media fetch doesn’t force you to re-run the whole extraction.

Wikipedia offline re-extract (0.2.4+)

Starting in 0.2.4, a Wikipedia capture’s Source-tier artifact preserves both MediaWiki responses ({query,parse}), so --re-extract rebuilds the full rich body (references + infobox) offline, with no network call.

This applies to captures made with 0.2.4 or later only. For a pre-0.2.4 Wikipedia capture, --re-extract still yields the plaintext body offline — but it will not downgrade a rich body already on disk: if the earlier capture had a rich body (references/infobox present), those fields are kept, and the rich body still refreshes via a network refetch.

The supersession chain

The network re-fetch is append-only. Rather than overwriting the old capture, Khiip:

1. Runs a fresh capture (bypassing dedup) through the full pipeline — extraction, media-fetch, Wayback, Source-tier, vault write, SQLite, embeddings.
2. Sets a superseded_by pointer on the old capture’s row, pointing at the new id.

The old capture and its vault file stay on disk, so the bitemporal history is preserved — you can still answer “what did this say when I first captured it.” A later dedup (or a re-capture of the same URL) resolves to the most-recent un-superseded capture — the one whose superseded_by is empty. The CLI prints both ids on success:

✓ re-fetched from network; new capture id: 01JXA…
  old (superseded): 01JX9…

The in-place dimensions (--re-extract / --re-render / --media / --wayback) don’t create a new capture or touch the supersession chain — they rewrite the existing capture’s body and update its payload.

Renamed a note in your vault? Refetch won’t clobber it

Since 0.2.4, a plain rename self-heals: if exactly one vault note carries the capture’s id: (frontmatter, never the filename), the daemon quietly re-points the SQLite row at the note’s new path before the refetch proceeds — no command needed, no file touched. (An overwrite still verifies the target is the capture’s own note by that id:, and a new capture’s note always lands at a fresh, non-colliding path.)

Only when Khiip can’t uniquely identify the note — zero notes carry the id (deleted, or the id: was removed) or two or more do (a sync-conflict copy, a manual duplicate) — does it refuse to guess and fall back to skip-and-warn (a warning in the daemon’s log — the khiipd serve terminal or your service log):

• On a network re-fetch, the new capture is still created and the database supersede pointer still lands — only the supersession stamp on the old note (the banner plus the superseded_by: frontmatter key) is skipped.
• On an in-place dimension, the payload update still lands in the index; the vault file is left untouched. --re-render returns the daemon’s 409 conflict error in this case — the CLI prints it — since the vault write is its only effect.

The user-visible check is khiipd validate: it reports the row ↔ file divergence as vault_path_stale (below). To re-point rows without a refetch, khiipd validate --fix --apply re-links them (0.2.4) — or hit POST /api/v1/captures/{id}/relink directly.

Re-linking a renamed row on demand

When a self-heal isn’t triggered by a refetch, you can re-point the SQLite row at its note directly. Both surfaces run the same exactly-one-id-match resolution and only ever touch the SQLite vault_path cache — no vault file is created, moved, or edited:

khiipd validate --fix           # preview the re-link plan (writes nothing)
khiipd validate --fix --apply   # apply it: re-link every unambiguous stale row

--fix alone is a dry run — it prints the plan and writes nothing. --apply on its own is rejected (--apply requires --fix). Rows with zero or two-or-more id matches are reported as skipped, never guessed. --fix --apply re-links through the daemon when it’s running (POST /api/v1/captures/{id}/relink) and falls back to a direct SQLite write when the daemon is stopped.

The endpoint itself returns the re-linked Capture on success (or a 200 no-op when the row already points at its note), 404 for an unknown capture id, and 409 when no note carries the id, several notes do, or the note’s frontmatter can no longer be verified as the capture’s own.

Validate

khiipd validate

validate is a read-only consistency check between your Markdown vault (canonical) and the SQLite index (a derived cache). It never mutates either side (unless you pass --fix --apply — see above). It checks five invariants:

1. Vault ↔ SQLite reconciliation. Every index row has a matching vault file and vice versa — matched by the id: frontmatter key, not the filename. Reports five distinct violations:
   • vault_missing — an index row’s file is gone (manual delete, half-finished restore)
   • sqlite_missing — a vault file has no index row
   • vault_missing_id — a vault file has no id: frontmatter, so it can’t be reconciled
   • vault_path_stale (new in 0.2.3) — the note exists but was renamed/moved in the vault, so it no longer lives where the index row points. A rename is not data loss — your content is intact. Since 0.2.4 a plain refetch self-heals this when the note is uniquely id-matched; otherwise khiipd validate --fix --apply re-links it.
   • vault_id_duplicate (new in 0.2.3) — two vault files carry the same capture id (e.g. a sync-conflict copy). Reconciliation prefers the file the index row points at and names both paths.
2. Payload round-trip. Every vault file’s structured_payload deserializes back through the typed discriminated union — guards against schema drift between extractor versions.
3. Status consistency. A success capture has no error recorded; a failed-permanent capture does (the failure reason is the only signal on a tombstone). See failure handling.
4. Media sandbox. Every media.local_path resolves to a real file inside the vault — no out-of-vault paths, no dangling references.
5. Job ↔ capture consistency (new in 0.2.4, job_capture_missing). A terminal capture job that recorded a capture_id still has that capture on disk — flags a job whose result capture was later deleted.

It exits 0 when everything passes and 1 when there are violations, printing a per-invariant summary:

checked 142 captures + 142 vault files
✓ all invariants pass

Add --json for a machine-readable report (handy in a backup-restore or CI script), or --vault-path / --db-path to point at a non-default location.

Run it after an upgrade, after restoring a backup, or any time the vault and index might have drifted.