From 7e9e85e4268e32cd3cf1fb94d2cc8573f44506b6 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 22:08:15 +0100 Subject: [PATCH 01/18] docs: plan itd-101 and write spc-17 citation-gate spec The 2026-07-27 grill settled the design; the spec records the settlements plus the mechanism (offline lint family, committed baseline, one receipt schema across both manual-queue rungs). The freshly minted ids collided with unmerged PR #156 (iss-80 allocator class, second live instance appended there): spec renumbered to spc-17, duplicate issue capture folded into iss-80. Assisted-by: Claude:claude-fable-5 --- ...ublishes-is-provably-alive-and-honestly.md | 4 +- ...ublishes-is-provably-alive-and-honestly.md | 107 ++++++++++++++++++ .abcd/work/DECISIONS.md | 1 + ...itd-n-spc-n-iss-n-are-branch-local-para.md | 4 +- 4 files changed, 113 insertions(+), 3 deletions(-) rename .abcd/development/intents/{drafts => planned}/itd-101-every-citation-abcd-publishes-is-provably-alive-and-honestly.md (99%) create mode 100644 .abcd/development/specs/open/spc-17-every-citation-abcd-publishes-is-provably-alive-and-honestly.md diff --git a/.abcd/development/intents/drafts/itd-101-every-citation-abcd-publishes-is-provably-alive-and-honestly.md b/.abcd/development/intents/planned/itd-101-every-citation-abcd-publishes-is-provably-alive-and-honestly.md similarity index 99% rename from .abcd/development/intents/drafts/itd-101-every-citation-abcd-publishes-is-provably-alive-and-honestly.md rename to .abcd/development/intents/planned/itd-101-every-citation-abcd-publishes-is-provably-alive-and-honestly.md index 6d4be284..cecabf9a 100644 --- a/.abcd/development/intents/drafts/itd-101-every-citation-abcd-publishes-is-provably-alive-and-honestly.md +++ b/.abcd/development/intents/planned/itd-101-every-citation-abcd-publishes-is-provably-alive-and-honestly.md @@ -1,8 +1,8 @@ --- id: itd-101 slug: every-citation-abcd-publishes-is-provably-alive-and-honestly -spec_id: null -kind: null +spec_id: spc-17 +kind: standalone suggested_kind: null reclassification_history: [] builds_on: [] diff --git a/.abcd/development/specs/open/spc-17-every-citation-abcd-publishes-is-provably-alive-and-honestly.md b/.abcd/development/specs/open/spc-17-every-citation-abcd-publishes-is-provably-alive-and-honestly.md new file mode 100644 index 00000000..faf29c03 --- /dev/null +++ b/.abcd/development/specs/open/spc-17-every-citation-abcd-publishes-is-provably-alive-and-honestly.md @@ -0,0 +1,107 @@ +--- +id: spc-17 +slug: every-citation-abcd-publishes-is-provably-alive-and-honestly +intent: itd-101 +--- +# every-citation-abcd-publishes-is-provably-alive-and-honestly + +## Summary + +spc-17 delivers itd-101's citation gate: a deterministic, zero-network lint +family that runs in the commit gate, and an explicit on-demand refresh verb +(`abcd docs cite refresh`) that does the live fetching and writes a committed +baseline the lint then enforces offline. The design decisions below were +settled by the 2026-07-27 grill (recorded in the intent's Grill Settlements); +this spec records them together with the mechanism — it does not reopen them. + +## Settled constraints (from the grill) + +- **Zero network in the gate.** The citations lint family reads only committed + config and the committed baseline. All fetching lives in the refresh verb. +- **Refresh is manual now.** It is surfaced by ahoy status and + release-preflight nagging; a scheduled-CI wrapper is a later, separately + signed-off change and is out of this spec's scope. +- **Staleness policy.** A baseline entry older than 180 days makes docs lint + warn; one older than 365 days blocks at the release gate only — commits are + never calendar-blocked. Human-verified entries age on the same clock and + re-enter the manual queue when stale. +- **Honest labelling, never method.** Each baseline entry records whether + verification was automatic or manual, and when — never how a human verified. +- **The row-has-footnote rule is homed here.** The structural rule deferred + from spc-15 (every crosswalk table row carries at least one footnote) is + implemented in this intent or not at all. + +## Mechanism + +### Offline lint family (commit gate) + +The docs-lint family gains citation rules, evaluated with zero network: + +- **Structure**: footnote markers and definitions in bijection per page; + every crosswalk table row carries at least one footnote (the spc-15 + deferral). +- **Syntax**: cited URLs and DOIs are well-formed. +- **Source policy**: aggregator domains are refused, from committed config. +- **Baseline enforcement**: no cited URL absent from the baseline, no entry + recorded as broken, no entry whose recorded final address has drifted from + the cited URL's current resolution record, and the staleness clock above + (180-day warn here; the 365-day blocker binds in the release gate, not the + commit gate). + +Severities mirror the existing docs-lint family (blocker/warn), and findings +render with the same successor-carrying shape. + +### Committed baseline + +`abcd docs cite refresh` fetches every cited URL and writes the baseline, a +committed machine record under `.abcd/` (one file, schema-versioned). Per URL +it records: the final resolved address after redirects, when it was last +checked, the outcome, and the verification label — `automatic`, or `manual` +with its date. Nothing else: no fetch transcript, no method, no headers. + +### Manual queue, one receipt schema + +Sources that block automated fetchers join the manual queue: + +- **Rung 1 (this spec's floor)**: the refresh verb prints the queue as a + checklist; the maintainer clears it link by link and a confirm verb writes + the dated receipt into the baseline (verification: manual + date). +- **Rung 2 (same schema, later rung)**: a generated, disposable checklist + page hands back a receipt file that the same confirm verb ingests. One + receipt schema serves both rungs — the page is a different producer of the + same input, not a second pathway. + +### Engine and seam + +The fetch engine is native and dependency-free (net/http, bounded redirects, +explicit timeouts). A specialist link checker can later slot in behind the +refresh seam as an adapter without changing gate semantics: the baseline +schema and the lint rules are the contract; the fetcher is a replaceable +producer. + +### Surfacing + +ahoy status reports the baseline's age summary (stale/overdue counts), and +the release preflight nags when entries approach or pass the 365-day blocker. +Both planes are wired at delivery: CLI verbs and the plugin markdown surface. + +## Acceptance-criteria mapping + +- AC 1 (zero-network gate: structure, syntax, policy from committed inputs) + → Offline lint family. +- AC 2 (baseline records final address, checked-when, auto-vs-manual + date, + never how) → Committed baseline. +- AC 3 (180-day warn in docs lint; 365-day release-gate blocker; manual + verifications age the same) → Settled constraints + Offline lint family. +- AC 4 (printed checklist + confirm verb; generated page hands back a receipt + the same verb ingests; one schema) → Manual queue, one receipt schema. +- AC 5 (specialist checker slots behind the seam without gate changes) → + Engine and seam. + +## Out of scope + +- The scheduled-CI refresh wrapper (explicitly out of this intent, per the + grill; needs its own sign-off). +- The generated checklist page itself may land as the later rung; only the + receipt schema it must emit is fixed here. +- Any change to non-citation docs-lint rules. diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index 7cc9a069..e3e66af7 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -862,3 +862,4 @@ parallel-agent merge contention bites. - 2026-07-27 — ADOPTED the canonical tagline: "A host-agnostic configuration layer for intent-driven development." Canonical identity home is the brief product chapter (01-product README, Identity section: title / tagline / pitch); README strapline, plugin manifest description, and AGENTS.md opening render from it. README's former strapline ("An opinionated, intent-driven development framework for product thinkers") is retired as a surface line; the product-thinker framing lives on in the README body. Rejected wordings: "for agent harnesses" (object shift, host/harness redundancy, plural over-promise), "over any agent harness" (drops the domain). Resolves iss-143; itd-102 generalises the drift check for managed repos. - 2026-07-27 — itd-93 parity tension resolved: Branch A. The scaffold template is the single source for release.yml/auto-release.yml; abcd-cli's live workflows are regenerated from it; the only sanctioned live diff is the additive workflow_dispatch rehearsal (trigger + non-publishing dry-run job). Rejected: substitution-gating the rehearsal off for abcd (weakens the parity guarantee on the one novel component). - 2026-07-27 — GRILLED itd-101/102/103/104 to settlement (six decisions, each now in its intent's Grill Settlements + acceptance criteria): itd-103 guard fails open loudly, blocker/warn tiers with committed-only overrides, shell-token-aware command-position matching with an itd-81-style TNR-floored corpus; itd-101 refresh is manual-with-nagging (scheduled CI later, own sign-off), staleness warns at 180d and blocks releases at 365d, human verifications age on the same clock; itd-102 identity lives in a parseable markdown block (config points at it), check is warn-tier and never rewrites; itd-104 ideate is an optional verb, never a pre-capture gate, with a fresh-context off-policy adversary. All four drafts are grill-settled and await intent plan. +- 2026-07-27: itd-101 spec minted as spc-16 collided with unmerged #156 (iss-80 class, both spc-N and iss-N allocators); renumbered to spc-17 by hand, duplicate capture folded into iss-80 — ids on this branch deliberately skip 16. diff --git a/.abcd/work/issues/open/iss-80-record-id-allocators-itd-n-spc-n-iss-n-are-branch-local-para.md b/.abcd/work/issues/open/iss-80-record-id-allocators-itd-n-spc-n-iss-n-are-branch-local-para.md index 5bcd61fd..3d5e58c7 100644 --- a/.abcd/work/issues/open/iss-80-record-id-allocators-itd-n-spc-n-iss-n-are-branch-local-para.md +++ b/.abcd/work/issues/open/iss-80-record-id-allocators-itd-n-spc-n-iss-n-are-branch-local-para.md @@ -8,4 +8,6 @@ source: "agent-finding" found_during: "parallel-agent-run" --- -Record id allocators (itd-N, spc-N, iss-N) are branch-local: parallel agents on separate branches each scan for max+1 and mint the SAME id. Two intents both claimed itd-82 and both merged to main (PRs #46, #47). The iss-N allocator hit the same class before (iss-77 collision, manually renumbered to iss-79; class recorded as iss-74). Resolving a collision forces a renumber, which breaks the record's stated 'ids are capture-stable, never renumbered' invariant -- so the minting scheme, not the renumber, is the defect. This capture is the MINTING half: a collision-free allocation scheme is needed (forge-minted / random-suffix / timestamp / reserve-registry -- SOTA under research). The DETECTION half is armed separately: intent_lifecycle now blocks duplicate intent ids. \ No newline at end of file +Record id allocators (itd-N, spc-N, iss-N) are branch-local: parallel agents on separate branches each scan for max+1 and mint the SAME id. Two intents both claimed itd-82 and both merged to main (PRs #46, #47). The iss-N allocator hit the same class before (iss-77 collision, manually renumbered to iss-79; class recorded as iss-74). Resolving a collision forces a renumber, which breaks the record's stated 'ids are capture-stable, never renumbered' invariant -- so the minting scheme, not the renumber, is the defect. This capture is the MINTING half: a collision-free allocation scheme is needed (forge-minted / random-suffix / timestamp / reserve-registry -- SOTA under research). The DETECTION half is armed separately: intent_lifecycle now blocks duplicate intent ids. + +2026-07-27 instance (itd-101 arc): `intent plan itd-101` on a branch off main minted spc-16 while unmerged PR #156 already carried spc-16 (itd-103); the same session's `abcd capture` then minted iss-146 while #156 already carried iss-146. Both renumbered by hand (spc-17; the duplicate capture folded into this issue). Corpus: two collisions from one planning step, spc-N and iss-N allocators both live. \ No newline at end of file From aa4c491fc388bfc61ac1e7a796378f90665ff091 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 22:35:28 +0100 Subject: [PATCH 02/18] feat: add the committed citation baseline schema MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit spc-17's committed baseline: one schema-versioned record under .abcd/ that the zero-network lint enforces and (later) `abcd docs cite refresh` writes. Per cited URL it records the final resolved address, when it was last checked, the outcome, and who established it — automatic or manual, with its date. It records nothing about HOW, and that omission is structural rather than conventional: BaselineEntry declares no such field, and the loader runs DisallowUnknownFields so a hand-added method/transcript key is a load-time refusal instead of a silently-dropped one. Both halves are pinned by test, including a reflection assertion over the struct's tags so a future edit that adds a how-shaped field fails. Loading validates every invariant the gate then relies on (schema version, final address, date shapes, outcome and verification enums) and returns a typed *BaselineError, keeping a corrupt record distinguishable from a missing one. An entry may restate its own map key in a url field for diff-readability; a key that disagrees with it is refused. Assisted-by: Claude:claude-opus-4-8 --- internal/core/lint/baseline.go | 239 ++++++++++++++++++++++++++++ internal/core/lint/baseline_test.go | 224 ++++++++++++++++++++++++++ 2 files changed, 463 insertions(+) create mode 100644 internal/core/lint/baseline.go create mode 100644 internal/core/lint/baseline_test.go diff --git a/internal/core/lint/baseline.go b/internal/core/lint/baseline.go new file mode 100644 index 00000000..ec1aa12a --- /dev/null +++ b/internal/core/lint/baseline.go @@ -0,0 +1,239 @@ +package lint + +// The committed citation baseline: the record the zero-network gate enforces. +// +// The split this file sits on is the whole point of the citation gate. Fetching +// a URL is non-deterministic and needs the network; deciding whether a commit is +// allowed must be neither. So the refresh verb does the live checking ONCE and +// writes what it learned here, and the lint reads only this file. A commit is +// judged against a committed artefact a reviewer can read in a diff, never +// against whatever the network happened to answer that minute. +// +// The schema is deliberately, structurally incapable of recording HOW a link was +// verified. A human who confirms a paywalled source records THAT they confirmed +// it and WHEN — never the browser, the credential, the workaround, or the +// transcript. That is honest labelling without turning the record into a log of +// a maintainer's private working. The refusal is enforced twice: BaselineEntry +// declares no such field, and the decoder runs with DisallowUnknownFields so a +// hand-added one is a load-time refusal rather than a silently-dropped key. + +import ( + "bytes" + "encoding/json" + "errors" + "os" + "reflect" + "sort" + "strconv" + "strings" + "time" + + "github.com/REPPL/abcd-cli/internal/fsutil" +) + +// BaselineSchemaVersion is the only baseline schema this build understands. A +// document declaring anything else is refused rather than best-effort parsed: +// the gate must never enforce a record whose meaning it is guessing at. +const BaselineSchemaVersion = 1 + +// DefaultBaselinePath is the committed baseline's repo-relative home. It sits +// beside the other committed machine records under .abcd/ (docs-lint.json, +// record-lint.json, rules.json) rather than inside the development record, +// because it is configuration the gate reads on every commit, not design +// material. +const DefaultBaselinePath = ".abcd/citations-baseline.json" + +// The outcomes a checked citation can have. Two values, not a free string: the +// gate branches on this, so an unrecognised outcome is a refusal at load. +const ( + // OutcomeAlive means the URL resolved to a live document. + OutcomeAlive = "alive" + // OutcomeBroken means it did not, and the gate must block on it. + OutcomeBroken = "broken" +) + +// The verification labels. This is the "automatic or manual" axis of AC 2 — WHO +// established the outcome, never how. +const ( + // VerificationAutomatic means the refresh verb's fetcher established it. + VerificationAutomatic = "automatic" + // VerificationManual means a human confirmed it after the fetcher could not. + VerificationManual = "manual" +) + +// baselineDateLayout is the date format every timestamp in the baseline uses. +// Date precision, not instant precision: the staleness policy is measured in +// days (180/365), and a wall-clock instant would add churn to a committed file +// for no gain in the only arithmetic that reads it. +const baselineDateLayout = "2006-01-02" + +// baselineSizeLimit caps the baseline read. It is a committed file, but the +// guarded read is what keeps a hostile branch from handing the gate a decompression +// bomb in place of a record. +const baselineSizeLimit = 8 << 20 // 8 MiB + +// Baseline is the committed citation record: one entry per cited URL. +type Baseline struct { + // SchemaVersion pins the document's shape. Required. + SchemaVersion int `json:"schema_version"` + // Entries is keyed by the cited URL exactly as it appears in the docs. + Entries map[string]BaselineEntry `json:"entries"` +} + +// BaselineEntry is one URL's record. +// +// Every field here answers "what is true about this link", and none answers +// "how did we find out". That omission is the schema's contract (AC 2), pinned +// by TestBaselineHasNoMethodFieldByConstruction. +type BaselineEntry struct { + // URL optionally restates the map key. It is redundant by design — a + // committed record reads better in a diff when each entry names itself — and + // is validated to AGREE with its key rather than being a second source of + // truth. Empty is fine; disagreeing is a refusal. + URL string `json:"url,omitempty"` + // FinalURL is the address the citation actually resolved to after redirects. + // When it differs from the cited URL, the docs are citing a stale address and + // the gate says so. + FinalURL string `json:"final_url"` + // LastChecked is when the outcome was established. This is THE staleness + // clock — the same clock for automatic and manual entries alike, so a human + // verification buys no exemption from ageing (AC 3). + LastChecked string `json:"last_checked"` + // Outcome is OutcomeAlive or OutcomeBroken. + Outcome string `json:"outcome"` + // Verification is VerificationAutomatic or VerificationManual — who + // established the outcome. + Verification string `json:"verification"` + // VerifiedOn is the date that verification was recorded. + VerifiedOn string `json:"verified_on"` +} + +// BaselineError is a malformed-baseline refusal. It is a distinct type so a +// caller can tell a bad record from an I/O fault without string matching. +type BaselineError struct{ msg string } + +func (e *BaselineError) Error() string { return e.msg } + +// Checked parses the entry's staleness clock. +func (e BaselineEntry) Checked() (time.Time, error) { + return time.Parse(baselineDateLayout, e.LastChecked) +} + +// LoadBaseline reads, decodes, and validates a committed baseline. +// +// A missing file is returned as the underlying os.ErrNotExist, so the caller can +// distinguish "this repo has no baseline yet" (a state the gate reports as one +// finding) from "this repo has a corrupt baseline" (a refusal). +func LoadBaseline(path string) (Baseline, error) { + data, err := fsutil.ReadGuarded(path, baselineSizeLimit) + if err != nil { + if errors.Is(err, os.ErrNotExist) { + return Baseline{}, err + } + if errors.Is(err, fsutil.ErrNotRegular) || errors.Is(err, fsutil.ErrTooBig) { + return Baseline{}, &BaselineError{"citation baseline " + path + ": " + err.Error()} + } + return Baseline{}, err + } + + dec := json.NewDecoder(bytes.NewReader(data)) + // The AC 2 teeth: a smuggled method/how/transcript key is a refusal, not a + // silently-ignored field. Unknown keys anywhere fail the load. + dec.DisallowUnknownFields() + var b Baseline + if err := dec.Decode(&b); err != nil { + return Baseline{}, &BaselineError{"citation baseline is not valid: " + err.Error()} + } + if err := b.validate(); err != nil { + return Baseline{}, err + } + return b, nil +} + +// validate enforces every invariant the gate then relies on, so no check +// downstream has to re-ask whether a field is well-formed. +func (b Baseline) validate() error { + if b.SchemaVersion != BaselineSchemaVersion { + return &BaselineError{ + "citation baseline schema_version is " + strconv.Itoa(b.SchemaVersion) + + "; this build understands only " + strconv.Itoa(BaselineSchemaVersion)} + } + // Sorted so a document with several faults always reports the same one, and + // a message never depends on Go's map iteration order. + keys := make([]string, 0, len(b.Entries)) + for k := range b.Entries { + keys = append(keys, k) + } + sort.Strings(keys) + + for _, k := range keys { + e := b.Entries[k] + if strings.TrimSpace(k) == "" { + return &BaselineError{"citation baseline has an empty url key"} + } + who := "citation baseline entry " + k + // Declared-key tolerance: restating the key is fine, disagreeing is not. + if e.URL != "" && e.URL != k { + return &BaselineError{who + " declares url " + e.URL + ", which disagrees with its key"} + } + if strings.TrimSpace(e.FinalURL) == "" { + return &BaselineError{who + " has no final_url; the resolved address is required"} + } + if _, err := time.Parse(baselineDateLayout, e.LastChecked); err != nil { + return &BaselineError{who + " has an unparsable last_checked " + quote(e.LastChecked) + + "; expected " + baselineDateLayout} + } + switch e.Outcome { + case OutcomeAlive, OutcomeBroken: + default: + return &BaselineError{who + " has outcome " + quote(e.Outcome) + + "; expected " + OutcomeAlive + " or " + OutcomeBroken} + } + switch e.Verification { + case VerificationAutomatic, VerificationManual: + default: + return &BaselineError{who + " has verification " + quote(e.Verification) + + "; expected " + VerificationAutomatic + " or " + VerificationManual} + } + if _, err := time.Parse(baselineDateLayout, e.VerifiedOn); err != nil { + return &BaselineError{who + " has an unparsable verified_on " + quote(e.VerifiedOn) + + "; expected " + baselineDateLayout} + } + } + return nil +} + +// SaveBaseline validates and writes a baseline atomically. Validation runs on +// the way OUT as well as in, so a producer bug can never commit a record the +// gate would then refuse to load. +func SaveBaseline(path string, b Baseline) error { + if b.SchemaVersion == 0 { + b.SchemaVersion = BaselineSchemaVersion + } + if err := b.validate(); err != nil { + return err + } + // Indented and newline-terminated: this file is committed and reviewed in + // diffs, so it is written to be read. Go's encoder sorts map keys, which is + // what keeps the diff stable across runs. + data, err := json.MarshalIndent(b, "", " ") + if err != nil { + return err + } + return fsutil.WriteFileAtomic(path, append(data, '\n'), 0o644) +} + +// baselineEntryJSONTags lists BaselineEntry's json field names. It exists for +// the construction test that asserts no how-shaped field can creep in. +func baselineEntryJSONTags() []string { + t := reflect.TypeOf(BaselineEntry{}) + tags := make([]string, 0, t.NumField()) + for i := 0; i < t.NumField(); i++ { + name, _, _ := strings.Cut(t.Field(i).Tag.Get("json"), ",") + tags = append(tags, name) + } + return tags +} + +// quote renders a value inside the message without pulling in fmt for one verb. +func quote(s string) string { return `"` + s + `"` } diff --git a/internal/core/lint/baseline_test.go b/internal/core/lint/baseline_test.go new file mode 100644 index 00000000..6e3589c9 --- /dev/null +++ b/internal/core/lint/baseline_test.go @@ -0,0 +1,224 @@ +package lint + +import ( + "errors" + "os" + "path/filepath" + "strings" + "testing" +) + +// writeBaselineFile plants a raw baseline document so a test can exercise the +// LOADER against bytes a human (or a hostile branch) could commit, rather than +// against a struct the loader would have produced itself. +func writeBaselineFile(t *testing.T, body string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "citations-baseline.json") + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return path +} + +// TestBaselineRoundTrip pins the save/load round-trip: what SaveBaseline writes +// is exactly what LoadBaseline returns, so the refresh verb and the gate agree +// on the record without a lossy hop. +func TestBaselineRoundTrip(t *testing.T) { + path := filepath.Join(t.TempDir(), "citations-baseline.json") + want := Baseline{ + SchemaVersion: BaselineSchemaVersion, + Entries: map[string]BaselineEntry{ + "https://example.org/a": { + FinalURL: "https://example.org/a", + LastChecked: "2026-07-01", + Outcome: OutcomeAlive, + Verification: VerificationAutomatic, + VerifiedOn: "2026-07-01", + }, + "https://example.org/b": { + URL: "https://example.org/b", + FinalURL: "https://example.org/b/final", + LastChecked: "2026-06-02", + Outcome: OutcomeAlive, + Verification: VerificationManual, + VerifiedOn: "2026-06-02", + }, + }, + } + if err := SaveBaseline(path, want); err != nil { + t.Fatalf("SaveBaseline: %v", err) + } + got, err := LoadBaseline(path) + if err != nil { + t.Fatalf("LoadBaseline: %v", err) + } + if got.SchemaVersion != want.SchemaVersion { + t.Errorf("schema_version = %d, want %d", got.SchemaVersion, want.SchemaVersion) + } + if len(got.Entries) != len(want.Entries) { + t.Fatalf("loaded %d entries, want %d", len(got.Entries), len(want.Entries)) + } + for k, w := range want.Entries { + g, ok := got.Entries[k] + if !ok { + t.Fatalf("entry %q missing after round-trip", k) + } + if g.FinalURL != w.FinalURL || g.LastChecked != w.LastChecked || + g.Outcome != w.Outcome || g.Verification != w.Verification || g.VerifiedOn != w.VerifiedOn { + t.Errorf("entry %q = %+v, want %+v", k, g, w) + } + } +} + +// TestBaselineRejectsMethodField is AC 2's teeth: the schema must be +// STRUCTURALLY incapable of carrying how a human verified a link. There is no +// method field to populate, and DisallowUnknownFields means a smuggled one is a +// load-time refusal rather than a silently-ignored key. +func TestBaselineRejectsMethodField(t *testing.T) { + for _, smuggled := range []string{"method", "how", "transcript", "headers", "notes", "evidence"} { + body := `{"schema_version":1,"entries":{"https://example.org/a":{ + "final_url":"https://example.org/a","last_checked":"2026-07-01", + "outcome":"alive","verification":"manual","verified_on":"2026-07-01", + "` + smuggled + `":"clicked it in a browser"}}}` + path := writeBaselineFile(t, body) + _, err := LoadBaseline(path) + if err == nil { + t.Fatalf("LoadBaseline accepted a smuggled %q field; the schema must be incapable of recording how (AC 2)", smuggled) + } + var be *BaselineError + if !errors.As(err, &be) { + t.Fatalf("smuggled %q yielded %T (%v), want a typed *BaselineError", smuggled, err, err) + } + } +} + +// TestBaselineRejectsUnknownTopLevelField extends the refusal to the document +// root: a stray top-level key is the same smuggling vector one level up. +func TestBaselineRejectsUnknownTopLevelField(t *testing.T) { + path := writeBaselineFile(t, `{"schema_version":1,"entries":{},"method":"curl"}`) + if _, err := LoadBaseline(path); err == nil { + t.Fatal("LoadBaseline accepted an unknown top-level field") + } +} + +// TestBaselineDeclaredKeyTolerance pins the guard-reviewer lesson: an entry MAY +// restate its own map key in a url field (a human-readable, diffable record), +// and that is accepted; a MISMATCH between the declared url and the map key is +// refused, because then two different URLs claim one receipt. +func TestBaselineDeclaredKeyTolerance(t *testing.T) { + agreeing := writeBaselineFile(t, `{"schema_version":1,"entries":{"https://example.org/a":{ + "url":"https://example.org/a","final_url":"https://example.org/a","last_checked":"2026-07-01", + "outcome":"alive","verification":"automatic","verified_on":"2026-07-01"}}}`) + if _, err := LoadBaseline(agreeing); err != nil { + t.Fatalf("LoadBaseline refused an entry whose declared url matches its key: %v", err) + } + + mismatched := writeBaselineFile(t, `{"schema_version":1,"entries":{"https://example.org/a":{ + "url":"https://example.org/OTHER","final_url":"https://example.org/a","last_checked":"2026-07-01", + "outcome":"alive","verification":"automatic","verified_on":"2026-07-01"}}}`) + err := LoadBaseline2Err(t, mismatched) + if !strings.Contains(err.Error(), "https://example.org/OTHER") { + t.Errorf("mismatch error should name the disagreeing url; got %v", err) + } +} + +// LoadBaseline2Err loads a baseline that MUST fail and returns the typed error. +func LoadBaseline2Err(t *testing.T, path string) error { + t.Helper() + _, err := LoadBaseline(path) + if err == nil { + t.Fatal("LoadBaseline succeeded where a refusal was required") + } + var be *BaselineError + if !errors.As(err, &be) { + t.Fatalf("got %T (%v), want a typed *BaselineError", err, err) + } + return err +} + +// TestBaselineValidatesFields pins every field-level refusal. Each case is a +// baseline a committer could write by hand; the loader must fail closed rather +// than let the gate reason about a half-formed record. +func TestBaselineValidatesFields(t *testing.T) { + entry := func(fields string) string { + return `{"schema_version":1,"entries":{"https://example.org/a":{` + fields + `}}}` + } + cases := []struct { + name string + body string + want string + }{ + { + name: "unsupported schema version", + body: `{"schema_version":99,"entries":{}}`, + want: "schema_version", + }, + { + name: "missing schema version", + body: `{"entries":{}}`, + want: "schema_version", + }, + { + name: "empty url key", + body: `{"schema_version":1,"entries":{"":{"final_url":"https://example.org/a","last_checked":"2026-07-01","outcome":"alive","verification":"automatic","verified_on":"2026-07-01"}}}`, + want: "empty url key", + }, + { + name: "missing final address", + body: entry(`"last_checked":"2026-07-01","outcome":"alive","verification":"automatic","verified_on":"2026-07-01"`), + want: "final_url", + }, + { + name: "unparsable last_checked", + body: entry(`"final_url":"https://example.org/a","last_checked":"yesterday","outcome":"alive","verification":"automatic","verified_on":"2026-07-01"`), + want: "last_checked", + }, + { + name: "unknown outcome", + body: entry(`"final_url":"https://example.org/a","last_checked":"2026-07-01","outcome":"probably-fine","verification":"automatic","verified_on":"2026-07-01"`), + want: "outcome", + }, + { + name: "unknown verification label", + body: entry(`"final_url":"https://example.org/a","last_checked":"2026-07-01","outcome":"alive","verification":"semi-automatic","verified_on":"2026-07-01"`), + want: "verification", + }, + { + name: "missing verified_on", + body: entry(`"final_url":"https://example.org/a","last_checked":"2026-07-01","outcome":"alive","verification":"manual"`), + want: "verified_on", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + err := LoadBaseline2Err(t, writeBaselineFile(t, tc.body)) + if !strings.Contains(err.Error(), tc.want) { + t.Errorf("error %q does not name %q", err, tc.want) + } + }) + } +} + +// TestBaselineHasNoMethodFieldByConstruction asserts the type itself, not just +// the decoder: reflection over BaselineEntry must find no field whose json tag +// could record HOW a link was verified. A future edit that adds one fails here. +func TestBaselineHasNoMethodFieldByConstruction(t *testing.T) { + banned := map[string]bool{ + "method": true, "how": true, "transcript": true, "headers": true, + "notes": true, "evidence": true, "tool": true, "user_agent": true, "proof": true, + } + for _, tag := range baselineEntryJSONTags() { + if banned[tag] { + t.Errorf("BaselineEntry declares a %q field; the schema must be incapable of recording how a human verified (AC 2)", tag) + } + } +} + +// TestLoadBaselineMissingFile reports a missing baseline as os.ErrNotExist, so +// the gate can tell "no baseline yet" from "a corrupt baseline". +func TestLoadBaselineMissingFile(t *testing.T) { + _, err := LoadBaseline(filepath.Join(t.TempDir(), "absent.json")) + if !os.IsNotExist(err) { + t.Fatalf("LoadBaseline on a missing file = %v, want an os.ErrNotExist", err) + } +} From 8e187c0de168b8963695279ac0f405538bb1a44d Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 22:40:57 +0100 Subject: [PATCH 03/18] feat: add the zero-network citations lint family MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit spc-17's commit gate: five rules on the existing docs-lint engine, none of which touches the network. They read committed markdown, committed config, and the committed baseline, so a commit's verdict is a function of the repository rather than of what the network answered that minute. - citation_footnotes: markers and definitions in bijection per page, both directions. An orphan definition is the subtler rot — a source the page still lists but no longer claims anything from. - citation_crosswalk_rows: the rule deferred from spc-15, homed here per DECISIONS 2026-07-27. Every crosswalk body row carries a footnote. - citation_url_syntax: cited URLs and DOIs are well-formed. - citation_source_policy: refused aggregator domains, from committed config. - citation_baseline: no cited URL absent from the baseline, none recorded broken, none whose recorded final address has drifted, and the staleness clock (180-day warn). Three choices worth naming. The citation corpus is a page's footnote DEFINITIONS, not its prose — a URL in a paragraph is links_resolve's business — which is what keeps the syntax and policy rules off ordinary text. A table is a crosswalk only when its nearest preceding heading says so, the narrowest heuristic that selects the real crosswalk and leaves every directory-map table alone. And the 365-day threshold surfaces as a DISTINCT rule id (citation_baseline_overdue) at warn, because the commit gate never calendar-blocks; promoting it is the release gate's job. Lint gains LintAt, which takes the clock as a parameter so the staleness boundaries are testable exactly; Lint keeps its signature and delegates. Assisted-by: Claude:claude-opus-4-8 --- internal/core/lint/citations.go | 484 +++++++++++++++++++++++++ internal/core/lint/citations_test.go | 504 +++++++++++++++++++++++++++ internal/core/lint/config.go | 24 ++ internal/core/lint/lint.go | 60 ++++ 4 files changed, 1072 insertions(+) create mode 100644 internal/core/lint/citations.go create mode 100644 internal/core/lint/citations_test.go diff --git a/internal/core/lint/citations.go b/internal/core/lint/citations.go new file mode 100644 index 00000000..53aa5ed6 --- /dev/null +++ b/internal/core/lint/citations.go @@ -0,0 +1,484 @@ +package lint + +// The citations lint family: spc-17's zero-network commit gate. +// +// Every rule here reads only the committed markdown, the committed config, and +// the committed baseline. Nothing dials out. That is the whole design bet — a +// gate that touches the network is a gate that flakes, and a flaky gate gets +// disabled, so the live checking is banished to `abcd docs cite refresh` and +// what it learned arrives here as a reviewable committed record. +// +// Scoping note that governs three of the five rules: the CITATION CORPUS of a +// page is its footnote definitions, not its prose. A URL in a body paragraph is +// a link (links_resolve's business); a URL on a `[^id]: ...` line is a citation. +// Judging prose as citation is how a source-policy rule acquires a false-positive +// rate nobody will tolerate, so the parse draws that line once, here, and every +// syntax/policy/baseline rule inherits it. + +import ( + "net/url" + "os" + "path/filepath" + "regexp" + "strconv" + "strings" + "time" +) + +// Citation rule ids. They are separate rules rather than one family switch so a +// repo can adopt the structural checks (which need no baseline) long before it +// has a baseline to enforce. +const ( + ruleCitationFootnotes = "citation_footnotes" + ruleCitationCrosswalkRows = "citation_crosswalk_rows" + ruleCitationURLSyntax = "citation_url_syntax" + ruleCitationSourcePolicy = "citation_source_policy" + ruleCitationBaseline = "citation_baseline" + ruleCitationBaselineOverdu = "citation_baseline_overdue" +) + +// The staleness policy from spc-17's grill settlements. Both are defaults a +// repo may tighten in config; neither is invented here. +const ( + defaultCitationWarnDays = 180 + defaultCitationBlockDays = 365 +) + +// defaultCrosswalkHeading is the narrowest identification heuristic that matches +// the real crosswalk: a table counts only when its nearest preceding heading +// names itself a crosswalk. Anchoring on the heading rather than on "any table +// in docs/" is what keeps every ordinary directory-map and comparison table in +// the corpus out of the rule's reach. +const defaultCrosswalkHeading = `(?i)crosswalk` + +var ( + // A footnote definition opens a line: [^id]: ... + footnoteDefRe = regexp.MustCompile(`^\s{0,3}\[\^([^\]\s]+)\]:`) + // A footnote marker anywhere in prose: [^id]. The caret is what separates + // this from a reference-style link definition ([label]: url), which is a + // different syntax the rule must not claim. + footnoteMarkerRe = regexp.MustCompile(`\[\^([^\]\s]+)\]`) + // An ATX heading, used to find the section a table sits in. + atxHeadingRe = regexp.MustCompile(`^\s{0,3}#{1,6}\s+(.*\S)\s*$`) + // A bare URL. Citations in this corpus are written bare, never as markdown + // links, so the scan is for the scheme rather than for link syntax. + citationURLRe = regexp.MustCompile("https?://[^\\s<>\"'`]+") + // A DOI mention. Requiring the delimiter after "doi" keeps the prose word + // ("a paper with a DOI, or a specification") out of the match. + doiMentionRe = regexp.MustCompile(`(?i)\bdoi[:\s]\s*(\S+)`) + // The DOI grammar: a registrant prefix and a non-empty suffix. + doiShapeRe = regexp.MustCompile(`^10\.\d{4,9}/\S+$`) +) + +// footnoteRef is one marker or definition occurrence. +type footnoteRef struct { + id string + line int // 1-based +} + +// citedRef is one URL cited on a footnote-definition line, with where it was +// cited so a finding can point at it. +type citedRef struct { + url string + file string + line int // 1-based +} + +// pageCitations is one page's citation parse, produced once and shared by every +// citation rule so a page is never scanned twice for the same thing. +type pageCitations struct { + markers []footnoteRef + defs []footnoteRef + urls []citedRef + dois []citedRef +} + +// parseCitations reads a page's footnote structure and citation corpus. +// +// mask excludes fenced code; stripInlineCode (shared with the banned-token +// family) excludes backticked spans. Between them, a regexp character class +// like [^A-Za-z] in documentation is not mistaken for a citation — which matters +// because this repo's own records contain exactly that decoy. +func parseCitations(rel string, lines []string, mask []bool) pageCitations { + var p pageCitations + for i, raw := range lines { + if mask[i] { + continue + } + line := stripInlineCode(raw) + lineNo := i + 1 + + // A definition line contributes its id as a definition, and its TAIL is + // the citation corpus. The head is skipped for marker scanning so a + // definition never counts as a reference to itself. + rest := line + if m := footnoteDefRe.FindStringSubmatchIndex(line); m != nil { + p.defs = append(p.defs, footnoteRef{id: line[m[2]:m[3]], line: lineNo}) + rest = line[m[1]:] + p.urls = append(p.urls, scanCitedURLs(rel, rest, lineNo)...) + p.dois = append(p.dois, scanCitedDOIs(rel, rest, lineNo)...) + } + for _, m := range footnoteMarkerRe.FindAllStringSubmatch(rest, -1) { + p.markers = append(p.markers, footnoteRef{id: m[1], line: lineNo}) + } + } + return p +} + +// scanCitedURLs extracts the bare URLs from a citation line, trimmed of the +// sentence punctuation that abuts them in real prose. +func scanCitedURLs(rel, s string, line int) []citedRef { + var out []citedRef + for _, raw := range citationURLRe.FindAllString(s, -1) { + if u := trimCitedURL(raw); u != "" { + out = append(out, citedRef{url: u, file: rel, line: line}) + } + } + return out +} + +// scanCitedDOIs extracts DOI mentions from a citation line. +func scanCitedDOIs(rel, s string, line int) []citedRef { + var out []citedRef + for _, m := range doiMentionRe.FindAllStringSubmatch(s, -1) { + out = append(out, citedRef{url: strings.Trim(m[1], ".,;:"), file: rel, line: line}) + } + return out +} + +// trimCitedURL strips trailing sentence punctuation from a bare URL. +// +// Citations end sentences, sit inside parentheses, and are separated by +// semicolons, so the raw scheme match over-reaches by a character or two almost +// every time. A closing bracket is stripped only when it is unbalanced — URLs +// legitimately contain balanced parentheses, and eating one would invent a +// broken address the baseline could never match. +func trimCitedURL(s string) string { + for len(s) > 0 { + switch s[len(s)-1] { + case '.', ',', ';', ':', '!', '?', '"', '\'': + s = s[:len(s)-1] + continue + case ')': + if strings.Count(s, ")") > strings.Count(s, "(") { + s = s[:len(s)-1] + continue + } + case ']': + if strings.Count(s, "]") > strings.Count(s, "[") { + s = s[:len(s)-1] + continue + } + } + return s + } + return s +} + +// checkCitationFootnotes enforces the marker/definition bijection. +// +// Both directions are findings. An unresolved marker is an obviously broken +// citation; an unreferenced definition is the subtler failure — a source the +// page still lists but no longer claims anything from, which is how a reference +// page quietly stops meaning what it says. +func checkCitationFootnotes(rel string, p pageCitations, cfg RuleConfig) []Finding { + var out []Finding + + defined := map[string]int{} + for _, d := range p.defs { + if n := defined[d.id]; n > 0 { + out = append(out, Finding{ + File: rel, Line: d.line, RuleID: ruleCitationFootnotes, Severity: cfg.Severity, + Message: "footnote definition [^" + d.id + "] is declared more than once; one definition per citation", + }) + } + defined[d.id]++ + } + + referenced := map[string]bool{} + seen := map[string]bool{} + for _, m := range p.markers { + referenced[m.id] = true + if defined[m.id] == 0 && !seen[m.id] { + seen[m.id] = true + out = append(out, Finding{ + File: rel, Line: m.line, RuleID: ruleCitationFootnotes, Severity: cfg.Severity, + Message: "footnote marker [^" + m.id + "] has no definition on this page", + }) + } + } + for _, d := range p.defs { + if !referenced[d.id] { + out = append(out, Finding{ + File: rel, Line: d.line, RuleID: ruleCitationFootnotes, Severity: cfg.Severity, + Message: "footnote definition [^" + d.id + "] is never referenced; a citation nothing claims from is dead weight", + }) + } + } + return out +} + +// checkCitationCrosswalkRows implements the rule deferred from spc-15: every +// body row of a crosswalk table carries at least one footnote. +// +// A crosswalk maps an outside term to this project's position on it. The whole +// artefact's authority rests on the outside half being sourced, so an uncited +// row is not a formatting slip — it is an unsupported claim about someone else's +// vocabulary. Identification is by the table's nearest preceding heading, the +// narrowest heuristic that selects the real crosswalk and no ordinary table. +func checkCitationCrosswalkRows(rel string, lines []string, mask []bool, cfg RuleConfig) ([]Finding, error) { + pattern := cfg.CrosswalkHeading + if strings.TrimSpace(pattern) == "" { + pattern = defaultCrosswalkHeading + } + re, err := regexp.Compile(pattern) + if err != nil { + return nil, &configError{ruleCitationCrosswalkRows + ": crosswalk_heading is not a valid regexp: " + err.Error()} + } + + var out []Finding + inCrosswalk := false + for i := 0; i < len(lines); i++ { + if mask[i] { + continue + } + if m := atxHeadingRe.FindStringSubmatch(lines[i]); m != nil { + // A heading closes the previous section and opens a new one, so a + // crosswalk's scope ends where the next heading begins. + inCrosswalk = re.MatchString(m[1]) + continue + } + if !inCrosswalk || !isTableHeader(lines, mask, i) { + continue + } + // i is the header row; i+1 is the separator; body rows follow. + for r := i + 2; r < len(lines) && !mask[r] && strings.Contains(lines[r], "|"); r++ { + if len(tableCells(lines[r])) == 0 { + continue + } + if footnoteMarkerRe.MatchString(stripInlineCode(lines[r])) { + continue + } + out = append(out, Finding{ + File: rel, Line: r + 1, RuleID: ruleCitationCrosswalkRows, Severity: cfg.Severity, + Message: "crosswalk row carries no footnote; every row's established meaning needs a cited primary source", + }) + } + // Skip past the table just consumed. + for i+1 < len(lines) && !mask[i+1] && strings.Contains(lines[i+1], "|") { + i++ + } + } + return out, nil +} + +// isTableHeader reports whether line i opens a markdown table — a pipe row +// followed by a separator row. Both conditions are required: a lone pipe in +// prose is not a table, and isTableSeparator alone accepts a blank line. +func isTableHeader(lines []string, mask []bool, i int) bool { + if !strings.Contains(lines[i], "|") || i+1 >= len(lines) || mask[i+1] { + return false + } + sep := lines[i+1] + return strings.Contains(sep, "|") && strings.Contains(sep, "-") && isTableSeparator(sep) +} + +// checkCitationURLSyntax checks that cited addresses are well-formed. It reads +// the citation corpus only — prose URLs belong to links_resolve. +func checkCitationURLSyntax(p pageCitations, cfg RuleConfig) []Finding { + var out []Finding + for _, ref := range p.urls { + if why := malformedURL(ref.url); why != "" { + out = append(out, Finding{ + File: ref.file, Line: ref.line, RuleID: ruleCitationURLSyntax, Severity: cfg.Severity, + Message: "cited URL " + ref.url + " is not well-formed: " + why, + }) + } + } + for _, ref := range p.dois { + if !doiShapeRe.MatchString(ref.url) { + out = append(out, Finding{ + File: ref.file, Line: ref.line, RuleID: ruleCitationURLSyntax, Severity: cfg.Severity, + Message: "cited DOI " + ref.url + " is not well-formed: expected 10./", + }) + } + } + return out +} + +// malformedURL returns why a cited URL is unusable, or "" when it is fine. +func malformedURL(raw string) string { + u, err := url.Parse(raw) + if err != nil { + return err.Error() + } + if u.Scheme != "http" && u.Scheme != "https" { + return "scheme is not http(s)" + } + host := u.Hostname() + if host == "" { + return "no host" + } + if strings.ContainsAny(host, " \t") { + return "host contains whitespace" + } + // A citation must name a resolvable public host. A dotless host is a + // hostname on someone's private network, never a source anyone else can read. + if !strings.Contains(host, ".") || strings.HasPrefix(host, ".") || strings.HasSuffix(host, ".") { + return "host " + host + " is not a fully-qualified domain" + } + return "" +} + +// checkCitationSourcePolicy refuses citations to configured aggregator domains. +// +// The list is committed config and ships EMPTY: this repo documents its +// admission rule in prose ("no single-author coinages, no aggregators") but +// nowhere names a domain, and a gate must not invent a blacklist its project +// never agreed to. The rule is the mechanism; the policy is the maintainer's. +func checkCitationSourcePolicy(p pageCitations, cfg RuleConfig) []Finding { + if len(cfg.RefusedDomains) == 0 { + return nil + } + refused := make([]string, 0, len(cfg.RefusedDomains)) + for _, d := range cfg.RefusedDomains { + if d = normaliseHost(d); d != "" { + refused = append(refused, d) + } + } + + var out []Finding + for _, ref := range p.urls { + u, err := url.Parse(ref.url) + if err != nil { + continue // malformed addresses are citation_url_syntax's finding, not a second one here + } + host := normaliseHost(u.Hostname()) + for _, d := range refused { + // Subdomains count: refusing an aggregator must not be defeated by + // citing its blog. subdomain host. + if host == d || strings.HasSuffix(host, "."+d) { + out = append(out, Finding{ + File: ref.file, Line: ref.line, RuleID: ruleCitationSourcePolicy, Severity: cfg.Severity, + Message: "cited source " + ref.url + " is on the refused domain " + d + "; cite the primary source instead", + }) + break + } + } + } + return out +} + +// normaliseHost lower-cases a host and drops a leading www., so one config entry +// covers the ways the same publisher is written. +func normaliseHost(h string) string { + h = strings.ToLower(strings.TrimSpace(h)) + h = strings.TrimSuffix(h, ".") + return strings.TrimPrefix(h, "www.") +} + +// checkCitationBaseline enforces the committed baseline offline. +// +// now is a parameter rather than a wall-clock read so the staleness arithmetic +// is testable at its exact boundaries; the caller (LintAt) owns the clock. +func checkCitationBaseline(repoRoot string, refs []citedRef, cfg RuleConfig, now time.Time) ([]Finding, error) { + if len(refs) == 0 { + // Nothing cites, so there is nothing to have a receipt for. A repo that + // enables the rule before it has any citations is not in violation. + return nil, nil + } + rel := cfg.Baseline + if strings.TrimSpace(rel) == "" { + rel = DefaultBaselinePath + } + + base, err := LoadBaseline(filepath.Join(repoRoot, filepath.FromSlash(rel))) + if err != nil { + if os.IsNotExist(err) { + // Fail closed, but as ONE finding: a per-URL storm here would bury + // the single action that fixes all of them. + return []Finding{{ + File: rel, Line: 0, RuleID: ruleCitationBaseline, Severity: cfg.Severity, + Message: "citation baseline not found, but " + strconv.Itoa(len(refs)) + + " citation(s) need one; run `abcd docs cite refresh` to create it", + }}, nil + } + return nil, err + } + + warnDays := cfg.WarnAfterDays + if warnDays <= 0 { + warnDays = defaultCitationWarnDays + } + blockDays := cfg.BlockAfterDays + if blockDays <= 0 { + blockDays = defaultCitationBlockDays + } + overdueSeverity := cfg.OverdueSeverity + if strings.TrimSpace(overdueSeverity) == "" { + // The commit gate never calendar-blocks (spc-17). The release gate is + // what promotes this, by setting the field. + overdueSeverity = severityWarn + } + + var out []Finding + for _, ref := range refs { + e, ok := base.Entries[ref.url] + if !ok { + out = append(out, Finding{ + File: ref.file, Line: ref.line, RuleID: ruleCitationBaseline, Severity: cfg.Severity, + Message: "cited URL " + ref.url + " is not in the citation baseline; run `abcd docs cite refresh`", + }) + continue + } + switch { + case e.Outcome == OutcomeBroken: + out = append(out, Finding{ + File: ref.file, Line: ref.line, RuleID: ruleCitationBaseline, Severity: cfg.Severity, + Message: "cited URL " + ref.url + " is recorded broken in the citation baseline (checked " + e.LastChecked + ")", + }) + case e.FinalURL != ref.url: + // Offline drift: the baseline knows where this address ends up, and + // the page still cites the old one. Citing the pre-redirect address + // is how a citation survives its own source moving, right up until + // the redirect is retired. + out = append(out, Finding{ + File: ref.file, Line: ref.line, RuleID: ruleCitationBaseline, Severity: cfg.Severity, + Message: "cited URL " + ref.url + " resolves to " + e.FinalURL + + "; cite the final address the baseline recorded", + }) + } + + checked, perr := e.Checked() + if perr != nil { + continue // unreachable: LoadBaseline validated the date + } + age := daysBetween(checked, now) + switch { + case age >= blockDays: + out = append(out, Finding{ + File: ref.file, Line: ref.line, RuleID: ruleCitationBaselineOverdu, Severity: overdueSeverity, + Message: "citation " + ref.url + " was last checked " + e.LastChecked + " (" + + strconv.Itoa(age) + " days ago, past the " + strconv.Itoa(blockDays) + + "-day release threshold); run `abcd docs cite refresh`", + }) + case age >= warnDays: + out = append(out, Finding{ + File: ref.file, Line: ref.line, RuleID: ruleCitationBaseline, Severity: severityWarn, + Message: "citation " + ref.url + " was last checked " + e.LastChecked + " (" + + strconv.Itoa(age) + " days ago, past the " + strconv.Itoa(warnDays) + + "-day staleness threshold); run `abcd docs cite refresh`", + }) + } + } + return out, nil +} + +// daysBetween counts whole calendar days, both ends normalised to UTC midnight. +// Comparing instants would make the 180-day boundary depend on the hour the gate +// happened to run. +func daysBetween(from, to time.Time) int { + f := time.Date(from.Year(), from.Month(), from.Day(), 0, 0, 0, 0, time.UTC) + t := time.Date(to.Year(), to.Month(), to.Day(), 0, 0, 0, 0, time.UTC) + return int(t.Sub(f).Hours() / 24) +} diff --git a/internal/core/lint/citations_test.go b/internal/core/lint/citations_test.go new file mode 100644 index 00000000..1ffaf2ee --- /dev/null +++ b/internal/core/lint/citations_test.go @@ -0,0 +1,504 @@ +package lint + +import ( + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +// citeRepo plants a fixture tree and returns its root. Keys are repo-relative +// slash paths. +func citeRepo(t *testing.T, files map[string]string) string { + t.Helper() + root := t.TempDir() + for rel, body := range files { + path := filepath.Join(root, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + return root +} + +// citeConfig builds a docs-lint config with exactly one citation rule armed, so +// each test isolates the rule under examination. +func citeConfig(ruleID string, rc RuleConfig) Config { + rc.Enabled = true + if rc.Severity == "" { + rc.Severity = severityBlocker + } + return Config{Roots: []string{"docs"}, Rules: map[string]RuleConfig{ruleID: rc}} +} + +// findingsFor filters findings down to one rule id. +func findingsFor(findings []Finding, ruleID string) []Finding { + var out []Finding + for _, f := range findings { + if f.RuleID == ruleID { + out = append(out, f) + } + } + return out +} + +// fixedNow is the clock every staleness test runs against, so the arithmetic is +// deterministic rather than a function of the day the suite happens to run. +var fixedNow = time.Date(2026, 7, 27, 12, 0, 0, 0, time.UTC) + +// --------------------------------------------------------------------------- +// citation_footnotes — marker/definition bijection +// --------------------------------------------------------------------------- + +// TestCitationFootnotesBijection pins the structural half of AC 1: on a page +// that cites, every marker resolves to a definition and every definition is +// referenced. Both directions matter — an orphan definition is a citation the +// page lost, which is exactly the silent rot the gate exists to catch. +func TestCitationFootnotesBijection(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/good.md": "# Good\n\nA claim.[^a] Another.[^b]\n\n" + + "[^a]: Source A, https://example.org/a.\n\n[^b]: Source B, https://example.org/b.\n", + "docs/orphan-marker.md": "# Orphan marker\n\nA claim.[^missing]\n", + "docs/orphan-def.md": "# Orphan definition\n\nA claim.[^a]\n\n" + + "[^a]: Source A, https://example.org/a.\n\n[^unused]: Source U, https://example.org/u.\n", + "docs/duplicate-def.md": "# Duplicate\n\nA claim.[^a]\n\n" + + "[^a]: Source A, https://example.org/a.\n\n[^a]: Source A again, https://example.org/a2.\n", + "docs/no-footnotes.md": "# Plain page\n\nNo citations here at all.\n", + }) + findings, err := LintAt(citeConfig("citation_footnotes", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + got := findingsFor(findings, "citation_footnotes") + + byFile := map[string][]string{} + for _, f := range got { + byFile[f.File] = append(byFile[f.File], f.Message) + } + if n := len(byFile[filepath.FromSlash("docs/good.md")]); n != 0 { + t.Errorf("a page in bijection produced %d finding(s): %v", n, byFile[filepath.FromSlash("docs/good.md")]) + } + if n := len(byFile[filepath.FromSlash("docs/no-footnotes.md")]); n != 0 { + t.Errorf("a page with no footnotes produced %d finding(s)", n) + } + mustContain(t, byFile[filepath.FromSlash("docs/orphan-marker.md")], "[^missing]", "no definition") + mustContain(t, byFile[filepath.FromSlash("docs/orphan-def.md")], "[^unused]", "never referenced") + mustContain(t, byFile[filepath.FromSlash("docs/duplicate-def.md")], "[^a]", "more than once") +} + +// mustContain asserts some message in the set mentions every needle. +func mustContain(t *testing.T, msgs []string, needles ...string) { + t.Helper() + for _, m := range msgs { + hit := true + for _, n := range needles { + if !strings.Contains(m, n) { + hit = false + break + } + } + if hit { + return + } + } + t.Errorf("no finding mentions all of %v; got %v", needles, msgs) +} + +// TestCitationFootnotesIgnoresCode proves the parser reads prose, not code. A +// regexp character class `[^abc]` inside a fence or inline code is not a +// citation, and treating it as one would make the rule unusable on any page that +// documents a pattern. +func TestCitationFootnotesIgnoresCode(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/code.md": "# Code\n\nA real claim.[^a]\n\n" + + "Inline `re.match(r'[^A-Za-z]')` is not a citation.\n\n" + + "```bash\ngrep '[^0-9]' file\n```\n\n" + + "[^a]: Source A, https://example.org/a.\n", + }) + findings, err := LintAt(citeConfig("citation_footnotes", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + if got := findingsFor(findings, "citation_footnotes"); len(got) != 0 { + t.Errorf("code-fenced and inline-code bracket classes were read as footnotes: %+v", got) + } +} + +// TestCitationFootnotesIgnoresReferenceLinks guards the neighbouring syntax: a +// reference-style link definition `[label]: url` is not a footnote definition, +// and must not be counted as an orphan. +func TestCitationFootnotesIgnoresReferenceLinks(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/refs.md": "# Refs\n\nSee [the principle][srp].\n\n" + + "[srp]: https://example.org/srp \"Single responsibility\"\n", + }) + findings, err := LintAt(citeConfig("citation_footnotes", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + if got := findingsFor(findings, "citation_footnotes"); len(got) != 0 { + t.Errorf("reference-style link definitions were read as footnotes: %+v", got) + } +} + +// --------------------------------------------------------------------------- +// citation_crosswalk_rows — the spc-15 deferral +// --------------------------------------------------------------------------- + +// TestCitationCrosswalkRows pins the rule deferred from spc-15: every body row +// of a crosswalk table carries at least one footnote. The identification +// heuristic is deliberately the narrowest that matches the repo's real crosswalk +// — the table's nearest preceding heading names it — so ordinary tables on the +// same page are untouched. +func TestCitationCrosswalkRows(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/crosswalk.md": "# Terminology crosswalk\n\n" + + "## Plain table\n\n" + + "| Directory | For |\n|---|---|\n| tutorials/ | learning |\n| how-to/ | tasks |\n\n" + + "## The crosswalk\n\n" + + "| Term | Established meaning | Position |\n" + + "|------|---------------------|----------|\n" + + "| **A** | Means a thing.[^a] | USES |\n" + + "| **B** | Means another thing. | ADAPTS |\n" + + "| **C** | Means a third thing.[^c] | REJECTS |\n\n" + + "[^a]: Source A, https://example.org/a.\n\n[^c]: Source C, https://example.org/c.\n", + }) + findings, err := LintAt(citeConfig("citation_crosswalk_rows", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + got := findingsFor(findings, "citation_crosswalk_rows") + if len(got) != 1 { + t.Fatalf("want exactly 1 finding (the uncited row **B**), got %d: %+v", len(got), got) + } + if !strings.Contains(got[0].Message, "footnote") { + t.Errorf("message should say the row carries no footnote; got %q", got[0].Message) + } + // The uncited row **B** is the 15th line of the fixture (1-based). + if got[0].Line != 15 { + t.Errorf("finding anchored at line %d, want 15 (the **B** row)", got[0].Line) + } +} + +// TestCitationCrosswalkRowsLeavesOrdinaryTablesAlone is the false-positive +// budget for the heuristic. The repo's docs are full of ordinary tables with no +// citations at all; a rule that swept them up would be uninstallable. +func TestCitationCrosswalkRowsLeavesOrdinaryTablesAlone(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/README.md": "# Docs\n\n## Map\n\n" + + "| Directory | Diataxis type | For |\n|---|---|---|\n" + + "| tutorials/ | tutorial | learning |\n| how-to/ | how-to | tasks |\n", + }) + findings, err := LintAt(citeConfig("citation_crosswalk_rows", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + if got := findingsFor(findings, "citation_crosswalk_rows"); len(got) != 0 { + t.Errorf("ordinary tables under a non-crosswalk heading were swept up: %+v", got) + } +} + +// --------------------------------------------------------------------------- +// citation_url_syntax +// --------------------------------------------------------------------------- + +// TestCitationURLSyntax pins the syntax half of AC 1 against the punctuation the +// repo's real footnotes actually contain: several URLs on one line, sentence +// periods abutting them, and URLs closed inside parentheses. +func TestCitationURLSyntax(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/ok.md": "# OK\n\nClaim.[^a] Claim.[^b] Claim.[^c]\n\n" + + "[^a]: Spec, https://example.org/spec/; repo, https://github.com/e/x.\n\n" + + "[^b]: Page, https://example.org/page (mirror: https://example.net/page).\n\n" + + "[^c]: Paper, NeurIPS 2020, DOI 10.48550/arXiv.2005.11401.\n", + "docs/bad.md": "# Bad\n\nClaim.[^u] Claim.[^d]\n\n" + + "[^u]: Source, https://exam ple.org/x and http://nohost/.\n\n" + + "[^d]: Paper, DOI 10.notanumber/xyz.\n", + }) + findings, err := LintAt(citeConfig("citation_url_syntax", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + got := findingsFor(findings, "citation_url_syntax") + byFile := map[string][]string{} + for _, f := range got { + byFile[f.File] = append(byFile[f.File], f.Message) + } + if n := len(byFile[filepath.FromSlash("docs/ok.md")]); n != 0 { + t.Errorf("well-formed citations flagged %d time(s): %v", n, byFile[filepath.FromSlash("docs/ok.md")]) + } + mustContain(t, byFile[filepath.FromSlash("docs/bad.md")], "nohost") + mustContain(t, byFile[filepath.FromSlash("docs/bad.md")], "10.notanumber/xyz") +} + +// TestCitationURLSyntaxScopedToCitations proves the rule reads citation lines +// only. The word "DOI" in ordinary prose, and a URL in a body paragraph, are not +// the citation corpus and must not be judged as one. +func TestCitationURLSyntaxScopedToCitations(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/prose.md": "# Prose\n\nAdmission is strict: a peer-reviewed paper with a DOI, or a\n" + + "specification. See http://nohost/ mentioned in passing.\n", + }) + findings, err := LintAt(citeConfig("citation_url_syntax", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + if got := findingsFor(findings, "citation_url_syntax"); len(got) != 0 { + t.Errorf("prose outside a footnote definition was judged as a citation: %+v", got) + } +} + +// --------------------------------------------------------------------------- +// citation_source_policy +// --------------------------------------------------------------------------- + +// TestCitationSourcePolicyRefusesConfiguredDomains pins that the refused list +// comes from committed config, matches subdomains, and ignores a www. prefix. +func TestCitationSourcePolicyRefusesConfiguredDomains(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/sources.md": "# Sources\n\nA.[^a] B.[^b] C.[^c] D.[^d]\n\n" + + "[^a]: Write-up, https://aggregator.example/post/1.\n\n" + + "[^b]: Write-up, https://www.aggregator.example/post/2.\n\n" + + "[^c]: Write-up, https://blog.aggregator.example/post/3.\n\n" + + "[^d]: Spec, https://standards.example/spec.\n", + }) + cfg := citeConfig("citation_source_policy", RuleConfig{RefusedDomains: []string{"aggregator.example"}}) + findings, err := LintAt(cfg, root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + got := findingsFor(findings, "citation_source_policy") + if len(got) != 3 { + t.Fatalf("want 3 refusals (bare, www., and subdomain), got %d: %+v", len(got), got) + } + for _, f := range got { + if !strings.Contains(f.Message, "aggregator.example") { + t.Errorf("finding should name the refused domain; got %q", f.Message) + } + } +} + +// TestCitationSourcePolicyEmptyListRefusesNothing pins the shipped default. The +// repo documents no refused-domain list anywhere, so the config seeds empty and +// the rule is inert until a maintainer authors one — the gate never invents a +// blacklist. +func TestCitationSourcePolicyEmptyListRefusesNothing(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/sources.md": "# Sources\n\nA.[^a]\n\n[^a]: Write-up, https://aggregator.example/post/1.\n", + }) + findings, err := LintAt(citeConfig("citation_source_policy", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + if got := findingsFor(findings, "citation_source_policy"); len(got) != 0 { + t.Errorf("an empty refused list produced findings: %+v", got) + } +} + +// --------------------------------------------------------------------------- +// citation_baseline +// --------------------------------------------------------------------------- + +// baselineRepo plants a citing page plus a baseline document. +func baselineRepo(t *testing.T, page, baseline string) string { + t.Helper() + files := map[string]string{"docs/cites.md": page} + if baseline != "" { + files[DefaultBaselinePath] = baseline + } + return citeRepo(t, files) +} + +// TestCitationBaselineEnforcement pins every offline baseline refusal in AC 1 +// and the warn half of AC 3, against one page citing five URLs. +func TestCitationBaselineEnforcement(t *testing.T) { + page := "# Cites\n\nA.[^a] B.[^b] C.[^c] D.[^d] E.[^e]\n\n" + + "[^a]: OK, https://example.org/alive.\n\n" + + "[^b]: Broken, https://example.org/broken.\n\n" + + "[^c]: Drifted, https://example.org/drifted.\n\n" + + "[^d]: Stale, https://example.org/stale.\n\n" + + "[^e]: Absent, https://example.org/absent.\n" + // fixedNow is 2026-07-27. 2026-07-01 is 26 days old (fresh); 2026-01-01 is + // 207 days old (past the 180-day warn, short of the 365-day blocker). + baseline := `{"schema_version":1,"entries":{ + "https://example.org/alive":{"final_url":"https://example.org/alive","last_checked":"2026-07-01","outcome":"alive","verification":"automatic","verified_on":"2026-07-01"}, + "https://example.org/broken":{"final_url":"https://example.org/broken","last_checked":"2026-07-01","outcome":"broken","verification":"automatic","verified_on":"2026-07-01"}, + "https://example.org/drifted":{"final_url":"https://example.org/moved-here","last_checked":"2026-07-01","outcome":"alive","verification":"automatic","verified_on":"2026-07-01"}, + "https://example.org/stale":{"final_url":"https://example.org/stale","last_checked":"2026-01-01","outcome":"alive","verification":"manual","verified_on":"2026-01-01"} + }}` + root := baselineRepo(t, page, baseline) + findings, err := LintAt(citeConfig("citation_baseline", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + msgs := []string{} + for _, f := range findingsFor(findings, "citation_baseline") { + msgs = append(msgs, f.Severity+" "+f.Message) + } + mustContain(t, msgs, "https://example.org/absent", "not in the citation baseline") + mustContain(t, msgs, "https://example.org/broken", "recorded broken") + mustContain(t, msgs, "https://example.org/drifted", "https://example.org/moved-here") + mustContain(t, msgs, "https://example.org/stale", "180", severityWarn) + for _, m := range msgs { + if strings.Contains(m, "https://example.org/alive") { + t.Errorf("a fresh, alive, undrifted citation was flagged: %q", m) + } + } +} + +// TestCitationBaselineStalenessBoundaries pins the exact 180-day edge and the +// distinct overdue finding the RELEASE gate promotes. The commit gate never +// blocks on the calendar (spc-17), so 365 days is reported at warn here under +// its own rule id — part (b) arms it as a blocker at release time. +func TestCitationBaselineStalenessBoundaries(t *testing.T) { + cases := []struct { + name string + lastChecked string + wantRule string + wantSev string + }{ + {"179 days is fresh", "2026-01-29", "", ""}, + {"180 days warns", "2026-01-28", "citation_baseline", severityWarn}, + {"364 days warns", "2025-07-28", "citation_baseline", severityWarn}, + {"365 days is overdue", "2025-07-27", "citation_baseline_overdue", severityWarn}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + page := "# Cites\n\nA.[^a]\n\n[^a]: S, https://example.org/a.\n" + baseline := `{"schema_version":1,"entries":{"https://example.org/a":{` + + `"final_url":"https://example.org/a","last_checked":"` + tc.lastChecked + `",` + + `"outcome":"alive","verification":"automatic","verified_on":"` + tc.lastChecked + `"}}}` + root := baselineRepo(t, page, baseline) + findings, err := LintAt(citeConfig("citation_baseline", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + var aged []Finding + for _, f := range findings { + if strings.HasPrefix(f.RuleID, "citation_baseline") { + aged = append(aged, f) + } + } + if tc.wantRule == "" { + if len(aged) != 0 { + t.Fatalf("a %s entry produced findings: %+v", tc.name, aged) + } + return + } + if len(aged) != 1 { + t.Fatalf("want 1 finding, got %d: %+v", len(aged), aged) + } + if aged[0].RuleID != tc.wantRule { + t.Errorf("rule id = %q, want %q", aged[0].RuleID, tc.wantRule) + } + if aged[0].Severity != tc.wantSev { + t.Errorf("severity = %q, want %q (the commit gate never calendar-blocks)", aged[0].Severity, tc.wantSev) + } + }) + } +} + +// TestCitationBaselineOverdueSeverityIsConfigurable is the seam part (b) needs: +// the release gate promotes the overdue finding to a blocker without the commit +// gate's semantics changing. +func TestCitationBaselineOverdueSeverityIsConfigurable(t *testing.T) { + page := "# Cites\n\nA.[^a]\n\n[^a]: S, https://example.org/a.\n" + baseline := `{"schema_version":1,"entries":{"https://example.org/a":{` + + `"final_url":"https://example.org/a","last_checked":"2024-01-01",` + + `"outcome":"alive","verification":"automatic","verified_on":"2024-01-01"}}}` + root := baselineRepo(t, page, baseline) + cfg := citeConfig("citation_baseline", RuleConfig{OverdueSeverity: severityBlocker}) + findings, err := LintAt(cfg, root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + got := findingsFor(findings, "citation_baseline_overdue") + if len(got) != 1 || got[0].Severity != severityBlocker { + t.Fatalf("overdue_severity did not promote the finding: %+v", got) + } +} + +// TestCitationBaselineMissingFileFailsClosed: a repo that cites but has no +// baseline gets ONE actionable finding naming the refresh verb, not one per URL. +func TestCitationBaselineMissingFileFailsClosed(t *testing.T) { + root := baselineRepo(t, "# Cites\n\nA.[^a] B.[^b]\n\n"+ + "[^a]: S, https://example.org/a.\n\n[^b]: S, https://example.org/b.\n", "") + findings, err := LintAt(citeConfig("citation_baseline", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + got := findingsFor(findings, "citation_baseline") + if len(got) != 1 { + t.Fatalf("want exactly 1 finding for a missing baseline, got %d: %+v", len(got), got) + } + if got[0].Severity != severityBlocker || !strings.Contains(got[0].Message, "docs cite refresh") { + t.Errorf("finding should block and name the refresh verb; got %+v", got[0]) + } + if got[0].File != DefaultBaselinePath { + t.Errorf("finding should anchor on the baseline path, got %q", got[0].File) + } +} + +// TestCitationBaselineAbsentWithNoCitations: a repo that cites nothing needs no +// baseline. The rule stays silent rather than nagging every repo that enables it. +func TestCitationBaselineAbsentWithNoCitations(t *testing.T) { + root := citeRepo(t, map[string]string{"docs/plain.md": "# Plain\n\nNo citations.\n"}) + findings, err := LintAt(citeConfig("citation_baseline", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + if got := findingsFor(findings, "citation_baseline"); len(got) != 0 { + t.Errorf("a repo with no citations was nagged for a baseline: %+v", got) + } +} + +// TestCitationBaselineCorruptIsAnError separates a corrupt record from a missing +// one: the gate refuses to run rather than reporting misleading findings. +func TestCitationBaselineCorruptIsAnError(t *testing.T) { + root := baselineRepo(t, "# Cites\n\nA.[^a]\n\n[^a]: S, https://example.org/a.\n", + `{"schema_version":1,"entries":{"https://example.org/a":{"final_url":"x","last_checked":"2026-07-01","outcome":"alive","verification":"manual","verified_on":"2026-07-01","method":"browser"}}}`) + if _, err := LintAt(citeConfig("citation_baseline", RuleConfig{}), root, fixedNow); err == nil { + t.Fatal("a baseline smuggling a method field was accepted by the lint") + } +} + +// TestCitationRulesOffByDefault: an unconfigured repo sees no citation findings, +// so adding the family cannot retroactively fail a repo that never opted in. +func TestCitationRulesOffByDefault(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/x.md": "# X\n\nA.[^missing]\n\n[^unused]: S, https://exam ple.org/a.\n", + }) + findings, err := LintAt(Config{Roots: []string{"docs"}}, root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + for _, f := range findings { + if strings.HasPrefix(f.RuleID, "citation_") { + t.Errorf("citation rule fired with no config: %+v", f) + } + } +} + +// TestLintDelegatesToLintAt pins that the wall-clock entry point still exists +// and agrees with the injected-clock one on everything the clock does not touch. +func TestLintDelegatesToLintAt(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/x.md": "# X\n\nA.[^missing]\n", + }) + cfg := citeConfig("citation_footnotes", RuleConfig{}) + a, err := Lint(cfg, root) + if err != nil { + t.Fatalf("Lint: %v", err) + } + b, err := LintAt(cfg, root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + if len(a) != len(b) || len(a) != 1 { + t.Fatalf("Lint and LintAt disagree: %+v vs %+v", a, b) + } +} diff --git a/internal/core/lint/config.go b/internal/core/lint/config.go index 62552fc3..4752c4e3 100644 --- a/internal/core/lint/config.go +++ b/internal/core/lint/config.go @@ -148,6 +148,30 @@ type RuleConfig struct { // finding on that line — the legitimate-mention escape (naming the old token in an // external reference like `epic-review`, or the rename itself `epic->spec`). AllowContext []string `json:"allow_context"` + // CrosswalkHeading is the citation_crosswalk_rows heading matcher: a table is + // judged a crosswalk only when its nearest preceding heading matches this + // regexp. Default defaultCrosswalkHeading. Keying on the heading rather than on + // table shape is what keeps the rule off every ordinary table in the corpus. + CrosswalkHeading string `json:"crosswalk_heading"` + // RefusedDomains is the citation_source_policy list of aggregator domains a + // citation may not point at. Matching is host-normalised (case-folded, www. + // dropped) and covers subdomains. It ships EMPTY: naming a domain is a + // project's editorial policy, never something the gate invents for it. + RefusedDomains []string `json:"refused_domains"` + // Baseline is the citation_baseline record's repo-relative path. Default + // DefaultBaselinePath. + Baseline string `json:"baseline"` + // WarnAfterDays is the citation_baseline staleness warn threshold in days + // (spc-17: 180). Zero means the default. + WarnAfterDays int `json:"warn_after_days"` + // BlockAfterDays is the age at which an entry becomes OVERDUE and is reported + // under the citation_baseline_overdue rule id (spc-17: 365). Zero means the + // default. + BlockAfterDays int `json:"block_after_days"` + // OverdueSeverity is the severity of a citation_baseline_overdue finding. It + // defaults to warn because the COMMIT gate never calendar-blocks (spc-17); + // the release gate is what promotes it to a blocker. + OverdueSeverity string `json:"overdue_severity"` } // ArmReceiptGate returns cfg with the receipt_gate rule armed for a release: it diff --git a/internal/core/lint/lint.go b/internal/core/lint/lint.go index 8cea8116..344bad75 100644 --- a/internal/core/lint/lint.go +++ b/internal/core/lint/lint.go @@ -14,6 +14,7 @@ import ( "sort" "strconv" "strings" + "time" "unicode" "unicode/utf8" @@ -127,6 +128,14 @@ func joinImpacts(impacts ...changelog.Impact) string { // roster that is missing, unparsable, or empty); a walkable-but-missing root // is skipped, not an error. func Lint(cfg Config, repoRoot string) ([]Finding, error) { + return LintAt(cfg, repoRoot, time.Now()) +} + +// LintAt is Lint with the clock supplied by the caller. Only the citation +// baseline's staleness arithmetic reads it, but it is a parameter rather than a +// wall-clock read inside the check so the 180- and 365-day boundaries are +// testable exactly, and so a gate's verdict is a function of its inputs. +func LintAt(cfg Config, repoRoot string, now time.Time) ([]Finding, error) { var findings []Finding tokenChecks, err := compileTokens(cfg.BannedTokens) @@ -141,6 +150,23 @@ func Lint(cfg Config, repoRoot string) ([]Finding, error) { gitMetaOn = gitMetaOn && gitMetaCfg.Enabled brittleOn = brittleOn && brittleCfg.Enabled + // The citation family shares ONE parse of each page (footnote structure plus + // the citation corpus), so a page is read once no matter how many of the five + // rules are armed. citedRefs accumulates across every root because the + // baseline is a single repo-wide record, checked once after the walk. + citeFootCfg, citeFootOn := cfg.Rules[ruleCitationFootnotes] + citeRowsCfg, citeRowsOn := cfg.Rules[ruleCitationCrosswalkRows] + citeURLCfg, citeURLOn := cfg.Rules[ruleCitationURLSyntax] + citePolicyCfg, citePolicyOn := cfg.Rules[ruleCitationSourcePolicy] + citeBaseCfg, citeBaseOn := cfg.Rules[ruleCitationBaseline] + citeFootOn = citeFootOn && citeFootCfg.Enabled + citeRowsOn = citeRowsOn && citeRowsCfg.Enabled + citeURLOn = citeURLOn && citeURLCfg.Enabled + citePolicyOn = citePolicyOn && citePolicyCfg.Enabled + citeBaseOn = citeBaseOn && citeBaseCfg.Enabled + citeParseOn := citeFootOn || citeURLOn || citePolicyOn || citeBaseOn + var citedRefs []citedRef + personaCfg, personaOn := cfg.Rules["persona_registry"] personaOn = personaOn && personaCfg.Enabled var personaRoster map[string]bool @@ -184,6 +210,29 @@ func Lint(cfg Config, repoRoot string) ([]Finding, error) { if brittleOn { findings = append(findings, checkBrittleRefs(rel, lines, mask, brittleCfg)...) } + + if citeParseOn { + page := parseCitations(rel, lines, mask) + if citeFootOn { + findings = append(findings, checkCitationFootnotes(rel, page, citeFootCfg)...) + } + if citeURLOn { + findings = append(findings, checkCitationURLSyntax(page, citeURLCfg)...) + } + if citePolicyOn { + findings = append(findings, checkCitationSourcePolicy(page, citePolicyCfg)...) + } + if citeBaseOn { + citedRefs = append(citedRefs, page.urls...) + } + } + if citeRowsOn { + cw, err := checkCitationCrosswalkRows(rel, lines, mask, citeRowsCfg) + if err != nil { + return nil, err + } + findings = append(findings, cw...) + } } if dirCfg, ok := cfg.Rules["directory_coverage"]; ok && dirCfg.Enabled { @@ -338,6 +387,17 @@ func Lint(cfg Config, repoRoot string) ([]Finding, error) { findings = append(findings, checkIssueImpact(ledger, impactCfg)...) } + // citation_baseline is repo-wide: one committed record answers for every + // citation in every root, so it is loaded once here rather than per page, + // against the refs the walk collected. + if citeBaseOn { + cb, err := checkCitationBaseline(repoRoot, citedRefs, citeBaseCfg, now) + if err != nil { + return nil, err + } + findings = append(findings, cb...) + } + sortFindings(findings) return findings, nil } From a76f7e20ccc62eef2aef852f28336ea3c10187d2 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 22:42:35 +0100 Subject: [PATCH 04/18] feat: arm the citation structure rules in this repo's docs-lint config MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four zero-config rules go live here: footnote bijection, crosswalk rows, URL/DOI syntax, and source policy. All four pass on the corpus as it stands — 52 citations across docs/reference/terminology.md, in bijection, every crosswalk row cited, every address and DOI well-formed — so arming them locks in a property the docs already have rather than demanding new work. refused_domains ships EMPTY. The repo's admission rule is stated in prose ("no single-author coinages, no aggregators") but nowhere names a domain, and a gate must not invent an editorial blacklist its project never agreed to. The mechanism is armed; the list is the maintainer's to author. citation_baseline is registered but DISABLED, because `abcd docs cite refresh` does not exist yet and there is no honest way to populate a baseline without it. Hand-writing final addresses and check dates would be fabricating exactly the receipts the record exists to make trustworthy. Enabling it is part (b)'s job, once refresh can fill it truthfully. Assisted-by: Claude:claude-opus-4-8 --- .abcd/docs-lint.json | 7 ++++++- CHANGELOG.md | 26 ++++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/.abcd/docs-lint.json b/.abcd/docs-lint.json index ebe72c38..f040ba5e 100644 --- a/.abcd/docs-lint.json +++ b/.abcd/docs-lint.json @@ -22,7 +22,12 @@ "rules": { "links_resolve": {"enabled": true, "severity": "blocker"}, "stray_root_docs": {"enabled": true, "severity": "blocker", - "allowlist": ["README","AGENTS","CHANGELOG","CONTRIBUTING","SECURITY","LICENSE","ACKNOWLEDGEMENTS"]} + "allowlist": ["README","AGENTS","CHANGELOG","CONTRIBUTING","SECURITY","LICENSE","ACKNOWLEDGEMENTS"]}, + "citation_footnotes": {"enabled": true, "severity": "blocker"}, + "citation_crosswalk_rows": {"enabled": true, "severity": "blocker"}, + "citation_url_syntax": {"enabled": true, "severity": "blocker"}, + "citation_source_policy": {"enabled": true, "severity": "blocker", "refused_domains": []}, + "citation_baseline": {"enabled": false, "severity": "blocker", "baseline": ".abcd/citations-baseline.json", "warn_after_days": 180, "block_after_days": 365} }, "exempt_paths": [], "exempt_if_status": [] diff --git a/CHANGELOG.md b/CHANGELOG.md index 4130a660..c71bf579 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,32 @@ called out in a **Breaking** section. ### Added +- **A citations family in `abcd docs lint`, with zero network in the gate** + (itd-101, spc-17). Cited references rot silently — pages retitle, URLs + redirect, whole platforms announce their own shutdown — but a gate that dials + out to notice is a gate that flakes. So the checking splits in two. The lint + side, landing here, reads only committed markdown, committed config, and a + committed baseline: `citation_footnotes` holds a page's footnote markers and + definitions in bijection (an unreferenced definition counts, being the way a + reference page quietly stops meaning what it says); `citation_crosswalk_rows` + requires every crosswalk table row to carry a footnote; `citation_url_syntax` + checks cited URLs and DOIs are well-formed; and `citation_source_policy` + refuses aggregator domains named in config — a list that ships empty, because + naming one is a project's editorial policy and never something the gate + invents on its behalf. A page's citations are its footnote definitions, not + its prose, so ordinary body links stay `links_resolve`'s business. + `citation_baseline` enforces the committed record offline — no cited URL + without an entry, none recorded broken, none whose recorded final address has + drifted from what the page cites, and a 180-day staleness warning. Each rule + is opt-in per repo, and the whole family is inert until configured. +- **A schema-versioned citation baseline at `.abcd/citations-baseline.json`.** + Per cited URL it records the final resolved address, when it was last checked, + the outcome, and whether verification was automatic or manual with its date. + It records nothing about *how* a human verified, and cannot be made to: the + schema declares no such field, and loading rejects unknown keys outright, so a + hand-added `method` or transcript is a refusal rather than a quietly-kept + note. Manual entries age on the same clock as automatic ones, so a human + confirmation buys no exemption from going stale. - **`abcd launch scaffold` — the changelog-driven release-gate scaffolder** (itd-93, spc-14). Writes the fixed release machinery into a managed repo that lacks it: `.github/workflows/release.yml` (verify → build → publish, the verify From 3120e6034c23e337fe7efb0394927554110a6de2 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:01:08 +0100 Subject: [PATCH 05/18] fix: stop the DOI check claiming prose, and read wrapped citations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three defects found by review of the citations family, each with a test watched fail first. The DOI mention matcher took any non-space token after "DOI" plus a delimiter, so "a standard with a DOI and a stable URL" reported a malformed DOI named "and", and the resolvable https://doi.org/10.x form — the recommended way to write a DOI — was rejected outright. At blocker severity that fires on correct citations. Anchoring the match to the "10" registrant prefix keeps ordinary prose out while still catching a token that starts like a DOI and breaks the grammar. A footnote definition wrapped across lines contributed only its first line to the citation corpus, so a URL below the fold was never checked for syntax and never required a baseline entry. That direction of failure is open, not noisy: an address the baseline records as broken would pass the gate. Continuation lines (indented, non-blank) now stay corpus. And one URL cited twice on a line produced two identical baseline findings; one citation site is one problem. Assisted-by: Claude:claude-opus-4-8 --- internal/core/lint/citations.go | 47 +++++++++++++++++--- internal/core/lint/citations_test.go | 66 ++++++++++++++++++++++++++++ 2 files changed, 106 insertions(+), 7 deletions(-) diff --git a/internal/core/lint/citations.go b/internal/core/lint/citations.go index 53aa5ed6..a5313427 100644 --- a/internal/core/lint/citations.go +++ b/internal/core/lint/citations.go @@ -63,9 +63,13 @@ var ( // A bare URL. Citations in this corpus are written bare, never as markdown // links, so the scan is for the scheme rather than for link syntax. citationURLRe = regexp.MustCompile("https?://[^\\s<>\"'`]+") - // A DOI mention. Requiring the delimiter after "doi" keeps the prose word - // ("a paper with a DOI, or a specification") out of the match. - doiMentionRe = regexp.MustCompile(`(?i)\bdoi[:\s]\s*(\S+)`) + // A DOI mention: the word, a delimiter, and a token that at least STARTS + // like a DOI. Anchoring on the "10" prefix is what keeps ordinary prose out + // of a blocker-severity rule — "a paper with a DOI and a stable URL" and "no + // DOI is assigned" both put an English word after the delimiter, and the + // resolvable https://doi.org/10.x form is a URL the syntax check already + // judges. A token that starts 10 but breaks the grammar is still caught. + doiMentionRe = regexp.MustCompile(`(?i)\bdoi[:\s]\s*(10\S*)`) // The DOI grammar: a registrant prefix and a non-empty suffix. doiShapeRe = regexp.MustCompile(`^10\.\d{4,9}/\S+$`) ) @@ -101,20 +105,32 @@ type pageCitations struct { // because this repo's own records contain exactly that decoy. func parseCitations(rel string, lines []string, mask []bool) pageCitations { var p pageCitations + inDef := false for i, raw := range lines { if mask[i] { + inDef = false continue } line := stripInlineCode(raw) lineNo := i + 1 - // A definition line contributes its id as a definition, and its TAIL is - // the citation corpus. The head is skipped for marker scanning so a - // definition never counts as a reference to itself. + // A definition line contributes its id, and its TAIL is citation corpus. + // The head is excluded from marker scanning so a definition never counts + // as a reference to itself. rest := line - if m := footnoteDefRe.FindStringSubmatchIndex(line); m != nil { + switch m := footnoteDefRe.FindStringSubmatchIndex(line); { + case m != nil: p.defs = append(p.defs, footnoteRef{id: line[m[2]:m[3]], line: lineNo}) rest = line[m[1]:] + inDef = true + case inDef && isDefContinuation(raw): + // A wrapped definition is still the same citation, so its later + // lines stay corpus. Dropping them would fail OPEN — a malformed or + // baseline-broken address below the fold would never be examined. + default: + inDef = false + } + if inDef { p.urls = append(p.urls, scanCitedURLs(rel, rest, lineNo)...) p.dois = append(p.dois, scanCitedDOIs(rel, rest, lineNo)...) } @@ -125,6 +141,16 @@ func parseCitations(rel string, lines []string, mask []bool) pageCitations { return p } +// isDefContinuation reports whether a line continues the footnote definition +// above it. Indented and non-blank is the convention: a blank line or a line +// starting at column zero ends the definition and returns the page to prose. +func isDefContinuation(raw string) bool { + if strings.TrimSpace(raw) == "" { + return false + } + return raw[0] == ' ' || raw[0] == '\t' +} + // scanCitedURLs extracts the bare URLs from a citation line, trimmed of the // sentence punctuation that abuts them in real prose. func scanCitedURLs(rel, s string, line int) []citedRef { @@ -422,7 +448,14 @@ func checkCitationBaseline(repoRoot string, refs []citedRef, cfg RuleConfig, now } var out []Finding + // One citation site is one problem. A URL repeated on the same line (a + // source and its mirror) would otherwise emit byte-identical findings. + reported := map[citedRef]bool{} for _, ref := range refs { + if reported[ref] { + continue + } + reported[ref] = true e, ok := base.Entries[ref.url] if !ok { out = append(out, Finding{ diff --git a/internal/core/lint/citations_test.go b/internal/core/lint/citations_test.go index 1ffaf2ee..b4670c73 100644 --- a/internal/core/lint/citations_test.go +++ b/internal/core/lint/citations_test.go @@ -252,6 +252,72 @@ func TestCitationURLSyntaxScopedToCitations(t *testing.T) { } } +// TestCitationDOIMentionIsAnchored is the false-positive budget for the DOI +// check. "DOI" is an ordinary English word in citation prose, and the resolvable +// https://doi.org/... form is a URL, not a bare DOI. Claiming either as a +// malformed DOI would make a blocker-severity rule fire on correct citations. +func TestCitationDOIMentionIsAnchored(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/doi.md": "# DOI\n\nA.[^a] B.[^b] C.[^c] D.[^d]\n\n" + + "[^a]: Lewis, P., NeurIPS 2020, DOI https://doi.org/10.48550/arXiv.2005.11401.\n\n" + + "[^b]: A standard with a DOI and a stable URL, https://example.org/a.\n\n" + + "[^c]: Report, 2024. No DOI is assigned; see https://example.org/c.\n\n" + + "[^d]: Paper, DOI 10.1145/3586183.3606763.\n", + }) + findings, err := LintAt(citeConfig("citation_url_syntax", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + if got := findingsFor(findings, "citation_url_syntax"); len(got) != 0 { + t.Errorf("correct citations were flagged as malformed DOIs: %+v", got) + } +} + +// TestCitationDefinitionContinuationLines closes a silent gate hole: a footnote +// definition wrapped across lines still cites everything below its first line. +// Dropping the continuation would let a malformed address — or one the baseline +// records as broken — through the gate unexamined, which is worse than a noisy +// rule because it fails OPEN. +func TestCitationDefinitionContinuationLines(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/wrapped.md": "# Wrapped\n\nA.[^a] B.[^b]\n\n" + + "[^a]: Author, Title, a long citation that wraps,\n" + + " see http://nohost/ for the record.\n\n" + + "[^b]: Short, https://example.org/b.\n\n" + + "Ordinary prose resumes here, mentioning http://alsonohost/ harmlessly.\n", + }) + findings, err := LintAt(citeConfig("citation_url_syntax", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + got := findingsFor(findings, "citation_url_syntax") + if len(got) != 1 { + t.Fatalf("want exactly 1 finding (the wrapped nohost URL), got %d: %+v", len(got), got) + } + if !strings.Contains(got[0].Message, "http://nohost/") { + t.Errorf("finding should name the continuation-line URL; got %q", got[0].Message) + } + // The continuation line, not the definition's opening line. + if got[0].Line != 6 { + t.Errorf("finding anchored at line %d, want 6 (the continuation line)", got[0].Line) + } +} + +// TestCitationBaselineDedupesIdenticalRefs: one line citing the same URL twice +// is one problem, and must not be reported twice. +func TestCitationBaselineDedupesIdenticalRefs(t *testing.T) { + root := baselineRepo(t, + "# Cites\n\nA.[^a]\n\n[^a]: S, https://example.org/gone and mirror https://example.org/gone.\n", + `{"schema_version":1,"entries":{}}`) + findings, err := LintAt(citeConfig("citation_baseline", RuleConfig{}), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + if got := findingsFor(findings, "citation_baseline"); len(got) != 1 { + t.Fatalf("want 1 finding for one URL cited twice on one line, got %d: %+v", len(got), got) + } +} + // --------------------------------------------------------------------------- // citation_source_policy // --------------------------------------------------------------------------- From ffb83cf961db4599dc57b0ab4ddccea4a5e2ab86 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:02:14 +0100 Subject: [PATCH 06/18] docs: record the four design choices spc-17 left open MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Baseline path, the crosswalk identification heuristic, what counts as a page's citation corpus, and why refused_domains ships empty — each with the alternative that was rejected, so part (b) does not re-litigate them. Assisted-by: Claude:claude-opus-4-8 --- .abcd/work/DECISIONS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index e3e66af7..12618718 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -863,3 +863,4 @@ parallel-agent merge contention bites. - 2026-07-27 — itd-93 parity tension resolved: Branch A. The scaffold template is the single source for release.yml/auto-release.yml; abcd-cli's live workflows are regenerated from it; the only sanctioned live diff is the additive workflow_dispatch rehearsal (trigger + non-publishing dry-run job). Rejected: substitution-gating the rehearsal off for abcd (weakens the parity guarantee on the one novel component). - 2026-07-27 — GRILLED itd-101/102/103/104 to settlement (six decisions, each now in its intent's Grill Settlements + acceptance criteria): itd-103 guard fails open loudly, blocker/warn tiers with committed-only overrides, shell-token-aware command-position matching with an itd-81-style TNR-floored corpus; itd-101 refresh is manual-with-nagging (scheduled CI later, own sign-off), staleness warns at 180d and blocks releases at 365d, human verifications age on the same clock; itd-102 identity lives in a parseable markdown block (config points at it), check is warn-tier and never rewrites; itd-104 ideate is an optional verb, never a pre-capture gate, with a fresh-context off-policy adversary. All four drafts are grill-settled and await intent plan. - 2026-07-27: itd-101 spec minted as spc-16 collided with unmerged #156 (iss-80 class, both spc-N and iss-N allocators); renumbered to spc-17 by hand, duplicate capture folded into iss-80 — ids on this branch deliberately skip 16. +- 2026-07-27 — itd-101 part (a) settled four choices spc-17 left open. (1) The committed baseline lives at `.abcd/citations-baseline.json`, alongside docs-lint.json/record-lint.json/rules.json — it is config the gate reads every commit, not development record; rejected `.abcd/development/` (wrong tier) and `.abcd/work/` (not a session artefact). (2) A table is a CROSSWALK when its nearest preceding heading matches `(?i)crosswalk` (configurable) — the narrowest heuristic that selects docs/reference/terminology.md's table and leaves every ordinary directory-map/comparison table alone; rejected column-header and any-table-in-docs identification (both sweep up README.md and docs/README.md tables that carry no citations by design). (3) A page's CITATION CORPUS is its footnote definitions, including wrapped continuation lines — not its prose; a URL in a body paragraph stays links_resolve's business, which is what keeps the syntax and source-policy rules off ordinary text. (4) `refused_domains` ships EMPTY: the repo states its admission rule in prose ("no single-author coinages, no aggregators", docs/reference/terminology.md + ACKNOWLEDGEMENTS.md) but nowhere names a domain, and the gate must not invent an editorial blacklist the project never agreed. Also: staleness is measured from `last_checked` for automatic and manual entries alike (AC 3's "same clock"), and the 365-day threshold surfaces as a distinct rule id `citation_baseline_overdue` at warn — the commit gate never calendar-blocks, so promotion to blocker is the release gate's job in part (b). From 43f20605d4a29789f1035b7913a8a64bf7b071cc Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:12:31 +0100 Subject: [PATCH 07/18] refactor: extract the SSRF fetch guard into internal/urlguard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The address predicate, the internal/metadata name refusal, and the connect-time re-check lived inside internal/core/memory, private to the one fetch path that had them. abcd is about to acquire a second outbound fetch — the citation refresh — whose addresses come from committed documentation and are just as attacker-influenceable, and a guard that lives inside one caller is a guard the next caller reimplements slightly differently. Move it to its own package unchanged and have memory compose it. Two deliberate shape changes at the boundary: the address predicate is now a parameter of the name check and the dial control rather than a hard-wired call, so a fetch path can be exercised against an httptest server (which binds loopback) by relaxing the predicate in the test alone while the shipped default is never relaxed; and the guard returns plain errors that memory re-types as IngestError, so every ingest failure a caller sees is still one error type. Behaviour-preserving: the messages are byte-identical, and memory's TestIngestRefusesSSRFTargets regression passes untouched. Assisted-by: Claude:claude-opus-5[1m] --- internal/core/memory/ingest.go | 90 ++------------------ internal/urlguard/urlguard.go | 130 +++++++++++++++++++++++++++++ internal/urlguard/urlguard_test.go | 102 ++++++++++++++++++++++ 3 files changed, 240 insertions(+), 82 deletions(-) create mode 100644 internal/urlguard/urlguard.go create mode 100644 internal/urlguard/urlguard_test.go diff --git a/internal/core/memory/ingest.go b/internal/core/memory/ingest.go index 106faff5..f45cf79c 100644 --- a/internal/core/memory/ingest.go +++ b/internal/core/memory/ingest.go @@ -11,11 +11,11 @@ import ( "path/filepath" "regexp" "strings" - "syscall" "time" "unicode/utf8" "github.com/REPPL/abcd-cli/internal/fsutil" + "github.com/REPPL/abcd-cli/internal/urlguard" ) // ingest.go — the ingest flow from 07-memory.md §1: @@ -503,78 +503,13 @@ func acquireSource(repoRoot, source string, fetcher Fetcher, pdf PDFExtractor) ( return materialFromLocal(repoRoot, source, pdf) } -// blockedFetchIP reports whether ip is in a range that must never be fetched -// during memory ingest: loopback (127/8, ::1), link-local (169.254/16, fe80::/10 -// unicast and multicast), private (10/8, 172.16/12, 192.168/16, fc00::/7 via -// net.IP.IsPrivate), the unspecified address, and any multicast address. This is -// the SSRF guard that keeps cloud metadata endpoints (e.g. 169.254.169.254) and -// internal services out of reach. -func blockedFetchIP(ip net.IP) bool { - if ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || - ip.IsPrivate() || ip.IsUnspecified() || ip.IsMulticast() { - return true - } - // NAT64 (64:ff9b::/96) and 6to4 (2002::/16) embed an IPv4 destination in an - // IPv6 address the checks above do not flag; a metadata/loopback/private IPv4 - // wrapped in one of these would otherwise slip through. Extract the embedded - // v4 and re-check it (embeddedIPv4 returns nil for a v4, so no recursion loop). - if v4 := embeddedIPv4(ip); v4 != nil { - return blockedFetchIP(v4) - } - return false -} - -// embeddedIPv4 returns the IPv4 address a NAT64 (64:ff9b::/96) or 6to4 (2002::/16) -// IPv6 address embeds, or nil when ip is not one of those transition forms. Scope -// is the WELL-KNOWN prefixes only: deprecated IPv4-compatible (::/96, non-routable) -// and site-specific NAT64 prefixes (RFC 8215) are out of scope — the DNS64 default -// is the well-known /96 covered here. v4-mapped ::ffff:/96 needs no extraction (the -// standard IsPrivate/IsLoopback checks already fold through To4()). -func embeddedIPv4(ip net.IP) net.IP { - v6 := ip.To16() - if v6 == nil || ip.To4() != nil { - return nil // not IPv6 (a plain v4 or v4-mapped needs no extraction) - } - // NAT64 well-known prefix 64:ff9b::/96 → last 4 bytes are the v4. - if v6[0] == 0x00 && v6[1] == 0x64 && v6[2] == 0xff && v6[3] == 0x9b && - v6[4] == 0 && v6[5] == 0 && v6[6] == 0 && v6[7] == 0 && - v6[8] == 0 && v6[9] == 0 && v6[10] == 0 && v6[11] == 0 { - return net.IPv4(v6[12], v6[13], v6[14], v6[15]) - } - // 6to4 prefix 2002::/16 → bytes 2..5 are the v4. - if v6[0] == 0x20 && v6[1] == 0x02 { - return net.IPv4(v6[2], v6[3], v6[4], v6[5]) - } - return nil -} - -// guardFetchHost refuses a host that is an internal/metadata name or that -// resolves to a blocked address. It runs before the initial request and on every -// redirect hop. An IP literal is checked directly (no DNS); a name is rejected -// outright when it is an *.internal / metadata name, otherwise every resolved -// address is checked. +// guardFetchHost refuses a host the shared SSRF guard blocks, re-typed as an +// IngestError so every ingest failure a caller sees is one error type. The guard +// itself lives in internal/urlguard — the canonical home shared with the citation +// refresh's fetcher, so the two fetch paths cannot drift apart. func guardFetchHost(host string) error { - h := strings.ToLower(strings.TrimSuffix(host, ".")) - if h == "" { - return newIngestError("refusing to fetch a URL with no host") - } - if h == "metadata" || h == "metadata.google.internal" || strings.HasSuffix(h, ".internal") { - return newIngestError("refusing to fetch internal/metadata host %q", host) - } - if ip := net.ParseIP(h); ip != nil { - if blockedFetchIP(ip) { - return newIngestError("refusing to fetch %q: address %s is link-local, loopback, private, or metadata range", host, ip) - } - return nil - } - ips, err := net.LookupIP(h) - if err != nil { - return newIngestError("cannot resolve host %q: %v", host, err) - } - for _, ip := range ips { - if blockedFetchIP(ip) { - return newIngestError("refusing to fetch %q: it resolves to %s (link-local, loopback, private, or metadata range)", host, ip) - } + if err := urlguard.CheckHost(host); err != nil { + return newIngestError("%s", err.Error()) } return nil } @@ -592,16 +527,7 @@ func defaultFetch(rawURL string) (FetchedSource, error) { // guard above and the transport's own resolution. dialer := &net.Dialer{ Timeout: fetchTimeoutSeconds * time.Second, - Control: func(_, address string, _ syscall.RawConn) error { - host, _, err := net.SplitHostPort(address) - if err != nil { - host = address - } - if ip := net.ParseIP(host); ip != nil && blockedFetchIP(ip) { - return newIngestError("refusing to connect to %s: link-local, loopback, private, or metadata range", ip) - } - return nil - }, + Control: urlguard.DialControl(urlguard.BlockedIP), } client := &http.Client{ Timeout: fetchTimeoutSeconds * time.Second, diff --git a/internal/urlguard/urlguard.go b/internal/urlguard/urlguard.go new file mode 100644 index 00000000..8368c5ea --- /dev/null +++ b/internal/urlguard/urlguard.go @@ -0,0 +1,130 @@ +// Package urlguard is the canonical SSRF guard for every outbound fetch abcd +// makes. +// +// abcd fetches URLs from two places — memory ingest takes one from the operator, +// and the citation refresh takes them from committed documentation — and in both +// the address is attacker-influenceable content. A guard that lives inside one +// caller is a guard the next caller reimplements slightly differently, which is +// exactly how a metadata endpoint ends up reachable from the second fetch path. +// So the predicate, the name check, and the connect-time re-check live here once, +// and every fetcher composes them. +// +// The address predicate is a PARAMETER of the name check and the dial control +// rather than a hard-wired call, for one reason: a fetch path can then be +// exercised against an httptest server (which binds loopback) by relaxing the +// predicate in the test alone, while the shipped default — BlockedIP — is never +// relaxed anywhere. A guard nobody can test end-to-end is a guard nobody trusts. +package urlguard + +import ( + "errors" + "net" + "strings" + "syscall" +) + +// BlockedIP reports whether ip is in a range that must never be fetched: +// loopback (127/8, ::1), link-local (169.254/16, fe80::/10 unicast and +// multicast), private (10/8, 172.16/12, 192.168/16, fc00::/7 via +// net.IP.IsPrivate), the unspecified address, and any multicast address. This is +// what keeps cloud metadata endpoints (e.g. 169.254.169.254) and internal +// services out of reach. +func BlockedIP(ip net.IP) bool { + if ip.IsLoopback() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() || + ip.IsPrivate() || ip.IsUnspecified() || ip.IsMulticast() { + return true + } + // NAT64 (64:ff9b::/96) and 6to4 (2002::/16) embed an IPv4 destination in an + // IPv6 address the checks above do not flag; a metadata/loopback/private IPv4 + // wrapped in one of these would otherwise slip through. Extract the embedded + // v4 and re-check it (embeddedIPv4 returns nil for a v4, so no recursion loop). + if v4 := embeddedIPv4(ip); v4 != nil { + return BlockedIP(v4) + } + return false +} + +// embeddedIPv4 returns the IPv4 address a NAT64 (64:ff9b::/96) or 6to4 +// (2002::/16) IPv6 address embeds, or nil when ip is not one of those transition +// forms. Scope is the WELL-KNOWN prefixes only: deprecated IPv4-compatible +// (::/96, non-routable) and site-specific NAT64 prefixes (RFC 8215) are out of +// scope — the DNS64 default is the well-known /96 covered here. v4-mapped +// ::ffff:/96 needs no extraction (the standard IsPrivate/IsLoopback checks +// already fold through To4()). +func embeddedIPv4(ip net.IP) net.IP { + v6 := ip.To16() + if v6 == nil || ip.To4() != nil { + return nil // not IPv6 (a plain v4 or v4-mapped needs no extraction) + } + // NAT64 well-known prefix 64:ff9b::/96 → last 4 bytes are the v4. + if v6[0] == 0x00 && v6[1] == 0x64 && v6[2] == 0xff && v6[3] == 0x9b && + v6[4] == 0 && v6[5] == 0 && v6[6] == 0 && v6[7] == 0 && + v6[8] == 0 && v6[9] == 0 && v6[10] == 0 && v6[11] == 0 { + return net.IPv4(v6[12], v6[13], v6[14], v6[15]) + } + // 6to4 prefix 2002::/16 → bytes 2..5 are the v4. + if v6[0] == 0x20 && v6[1] == 0x02 { + return net.IPv4(v6[2], v6[3], v6[4], v6[5]) + } + return nil +} + +// CheckHost refuses a host that is an internal/metadata name or that resolves to +// a blocked address, under the shipped default policy. +func CheckHost(host string) error { return CheckHostWith(host, BlockedIP) } + +// CheckHostWith is CheckHost with the address predicate supplied by the caller. +// It runs before the initial request and on every redirect hop. An IP literal is +// checked directly (no DNS); a name is rejected outright when it is an +// *.internal / metadata name, otherwise every resolved address is checked. +// +// The NAME half is policy-independent: an *.internal or metadata name never gets +// as far as an address, so no predicate can wave it through. +func CheckHostWith(host string, blocked func(net.IP) bool) error { + h := strings.ToLower(strings.TrimSuffix(host, ".")) + if h == "" { + return errors.New("refusing to fetch a URL with no host") + } + if h == "metadata" || h == "metadata.google.internal" || strings.HasSuffix(h, ".internal") { + return errors.New("refusing to fetch internal/metadata host " + quote(host)) + } + if ip := net.ParseIP(h); ip != nil { + if blocked(ip) { + return errors.New("refusing to fetch " + quote(host) + ": address " + ip.String() + + " is link-local, loopback, private, or metadata range") + } + return nil + } + ips, err := net.LookupIP(h) + if err != nil { + return errors.New("cannot resolve host " + quote(host) + ": " + err.Error()) + } + for _, ip := range ips { + if blocked(ip) { + return errors.New("refusing to fetch " + quote(host) + ": it resolves to " + ip.String() + + " (link-local, loopback, private, or metadata range)") + } + } + return nil +} + +// DialControl builds a net.Dialer Control hook that re-checks the ACTUAL +// resolved IP of every dialled connection, closing the DNS-rebinding gap between +// a name-based guard and the transport's own resolution. +func DialControl(blocked func(net.IP) bool) func(network, address string, c syscall.RawConn) error { + return func(_, address string, _ syscall.RawConn) error { + host, _, err := net.SplitHostPort(address) + if err != nil { + host = address + } + if ip := net.ParseIP(host); ip != nil && blocked(ip) { + return errors.New("refusing to connect to " + ip.String() + + ": link-local, loopback, private, or metadata range") + } + return nil + } +} + +// quote renders a host inside a message the way the callers' own errors do, +// without pulling in fmt for one verb. +func quote(s string) string { return `"` + s + `"` } diff --git a/internal/urlguard/urlguard_test.go b/internal/urlguard/urlguard_test.go new file mode 100644 index 00000000..2246889c --- /dev/null +++ b/internal/urlguard/urlguard_test.go @@ -0,0 +1,102 @@ +package urlguard + +import ( + "net" + "strings" + "testing" +) + +// TestBlockedIP pins the address ranges no abcd fetch may reach. The NAT64 and +// 6to4 cases are the ones a naive guard misses: they wrap a loopback or metadata +// IPv4 inside an IPv6 address that none of net.IP's own predicates flag. +func TestBlockedIP(t *testing.T) { + blocked := []string{ + "127.0.0.1", "::1", // loopback + "169.254.169.254", // cloud metadata (link-local) + "10.0.0.1", "192.168.1.1", "172.16.5.5", // private + "0.0.0.0", "::", // unspecified + "224.0.0.1", "ff02::1", // multicast + "fe80::1", // link-local unicast + "fc00::1", // unique local (private) + "64:ff9b::a9fe:a9fe", // NAT64 wrapping 169.254.169.254 + "64:ff9b::7f00:1", // NAT64 wrapping 127.0.0.1 + "2002:a9fe:a9fe::", // 6to4 wrapping 169.254.169.254 + "::ffff:127.0.0.1", // v4-mapped loopback + } + for _, s := range blocked { + ip := net.ParseIP(s) + if ip == nil { + t.Fatalf("test bug: %q is not an IP", s) + } + if !BlockedIP(ip) { + t.Errorf("BlockedIP(%s) = false, want true", s) + } + } + + allowed := []string{"93.184.216.34", "2606:2800:220:1:248:1893:25c8:1946", "8.8.8.8"} + for _, s := range allowed { + ip := net.ParseIP(s) + if ip == nil { + t.Fatalf("test bug: %q is not an IP", s) + } + if BlockedIP(ip) { + t.Errorf("BlockedIP(%s) = true, want false", s) + } + } +} + +// TestCheckHostRefusesNamesAndLiterals covers the two refusal paths that resolve +// without any DNS traffic: the internal/metadata NAME guard, and IP literals. +func TestCheckHostRefusesNamesAndLiterals(t *testing.T) { + cases := []struct{ host, want string }{ + {"", "refusing to fetch a URL with no host"}, + {"metadata", "refusing to fetch internal/metadata host"}, + {"metadata.google.internal", "refusing to fetch internal/metadata host"}, + {"svc.internal", "refusing to fetch internal/metadata host"}, + {"127.0.0.1", "refusing to fetch"}, + {"169.254.169.254", "refusing to fetch"}, + {"10.0.0.1", "refusing to fetch"}, + } + for _, c := range cases { + err := CheckHost(c.host) + if err == nil { + t.Errorf("CheckHost(%q) = nil, want a refusal", c.host) + continue + } + if !strings.Contains(err.Error(), c.want) { + t.Errorf("CheckHost(%q) = %q, want it to contain %q", c.host, err, c.want) + } + } +} + +// TestCheckHostWithPolicyAllowsLoopback is what lets a fetch path be exercised +// against an httptest server: the predicate is a parameter, so a test can relax +// it for 127.0.0.1 without the shipped default ever being relaxed. +func TestCheckHostWithPolicyAllowsLoopback(t *testing.T) { + permissive := func(ip net.IP) bool { return !ip.IsLoopback() && BlockedIP(ip) } + if err := CheckHostWith("127.0.0.1", permissive); err != nil { + t.Errorf("CheckHostWith(127.0.0.1, permissive) = %v, want nil", err) + } + if err := CheckHostWith("169.254.169.254", permissive); err == nil { + t.Error("CheckHostWith(169.254.169.254, permissive) = nil, want a refusal") + } + // The name guard is policy-independent: a *.internal name is refused whatever + // the address predicate says, because it never gets as far as an address. + if err := CheckHostWith("svc.internal", permissive); err == nil { + t.Error("CheckHostWith(svc.internal, permissive) = nil, want a refusal") + } +} + +// TestDialControlRefusesBlockedAddress pins the connect-time re-check that closes +// the DNS-rebinding gap between the name guard and the transport's own lookup. +func TestDialControlRefusesBlockedAddress(t *testing.T) { + control := DialControl(BlockedIP) + if err := control("tcp", "169.254.169.254:80", nil); err == nil { + t.Error("DialControl allowed a metadata address") + } else if !strings.Contains(err.Error(), "refusing to connect") { + t.Errorf("got %q, want a refusing-to-connect error", err) + } + if err := control("tcp", "93.184.216.34:443", nil); err != nil { + t.Errorf("DialControl refused a public address: %v", err) + } +} From c505133873e8b258c9a200f4798ad2181e9abc84 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:19:44 +0100 Subject: [PATCH 08/18] feat: add the citation refresh engine and the manual-queue receipt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit spc-17's live half. `internal/core/cite` fetches every cited URL once and writes the committed baseline the zero-network gate then enforces offline, and `Confirm` records the queue lines a human cleared. The collector is shared, not duplicated: lint.CollectCitedURLs exports the walk LintAt's citation family already does, so the set the refresh fetches is exactly the set the gate demands receipts for. A second scraper with its own idea of what a citation is would produce permanent findings nobody could clear — a URL the gate wants and the refresh never visits. lint.CitationPolicy does the same for the thresholds. The checker refuses three things a link checker usually does. It never retries, so a run's cost cannot become a function of how many links are failing, nor a burst against a struggling host. It never reads a body, so liveness is judged from the status line and no cited page's bytes enter the process. And it separates "dead" from "not readable by a robot": 401/403/406/ 429 route to the manual queue rather than being recorded as broken, because a 403 is evidence the fetcher may not look, not evidence the source is gone, and a committed record the gate enforces must not contain that lie. The merge carries the honesty rules. A current manual receipt is preserved verbatim and not even requested. A stale one is re-checked (AC 3 puts human and machine verifications on one clock) and, if the source still blocks, kept rather than deleted so the gate keeps warning truthfully. A blocked URL with no prior entry writes nothing at all. Entries the docs no longer cite are dropped. Confirm validates every line against the cited set before writing anything, so it cannot become a door for arbitrary entries. Every fetch path is tested against httptest — redirect drift, the redirect bound, the blocked/broken status split, the timeout, one-attempt-per-URL, and the SSRF regression under the shipped policy. No test touches the network. Assisted-by: Claude:claude-opus-5[1m] --- internal/core/cite/confirm.go | 195 ++++++++++++++++ internal/core/cite/confirm_test.go | 191 ++++++++++++++++ internal/core/cite/fetch.go | 214 ++++++++++++++++++ internal/core/cite/fetch_test.go | 195 ++++++++++++++++ internal/core/cite/refresh.go | 313 ++++++++++++++++++++++++++ internal/core/cite/refresh_test.go | 344 +++++++++++++++++++++++++++++ internal/core/lint/citations.go | 22 ++ internal/core/lint/collect.go | 76 +++++++ internal/core/lint/collect_test.go | 81 +++++++ 9 files changed, 1631 insertions(+) create mode 100644 internal/core/cite/confirm.go create mode 100644 internal/core/cite/confirm_test.go create mode 100644 internal/core/cite/fetch.go create mode 100644 internal/core/cite/fetch_test.go create mode 100644 internal/core/cite/refresh.go create mode 100644 internal/core/cite/refresh_test.go create mode 100644 internal/core/lint/collect.go create mode 100644 internal/core/lint/collect_test.go diff --git a/internal/core/cite/confirm.go b/internal/core/cite/confirm.go new file mode 100644 index 00000000..1c984954 --- /dev/null +++ b/internal/core/cite/confirm.go @@ -0,0 +1,195 @@ +package cite + +// The manual queue's other end: a human clears a link, and the confirm verb +// records it. +// +// One receipt schema serves both of spc-17's rungs. Today the maintainer clears +// the printed checklist and names the URLs on the command line; later, a +// generated checklist page hands back a receipt FILE. Those are two producers of +// the same input, not two pathways — which is why the schema is declared here, +// once, and the CLI's positional arguments are simply assembled into it before +// they reach this function. When the page arrives it changes nothing on this +// side. +// +// The schema records THAT a human confirmed a citation and WHEN. It is +// structurally incapable of recording HOW, enforced twice like the baseline's +// own refusal: no such field is declared, and decoding runs with +// DisallowUnknownFields so a hand-added one is a refusal rather than a silently +// dropped key. A maintainer's browser, credential, or library proxy is their +// business and never enters a committed record. + +import ( + "bytes" + "encoding/json" + "errors" + "os" + "path/filepath" + "sort" + "strconv" + "time" + + "github.com/REPPL/abcd-cli/internal/core/lint" + "github.com/REPPL/abcd-cli/internal/fsutil" +) + +// ReceiptSchemaVersion is the only receipt schema this build understands. +const ReceiptSchemaVersion = 1 + +// receiptSizeLimit caps a receipt read. A receipt is a short list of addresses; +// the guarded read is what keeps a hostile file from being slurped whole. +const receiptSizeLimit = 1 << 20 // 1 MiB + +// ConfirmedCitation is one cleared queue line. +type ConfirmedCitation struct { + // URL is the cited address, exactly as the docs write it. + URL string `json:"url"` + // FinalURL is where the human found the document, when it differs from the + // cited address. Empty means it did not move. + FinalURL string `json:"final_url,omitempty"` + // VerifiedOn is the date the human checked, YYYY-MM-DD. Empty means today. + // It is a declared date rather than an implicit "now" because a receipt for + // last week's work must age from last week — the clock AC 3 puts human and + // machine verifications on is the same one. + VerifiedOn string `json:"verified_on,omitempty"` +} + +// Receipt is what a human — or the generated checklist page — hands back. +type Receipt struct { + SchemaVersion int `json:"schema_version"` + Confirmed []ConfirmedCitation `json:"confirmed"` +} + +// ConfirmRequest is the input to one confirm call. +type ConfirmRequest struct { + RepoRoot string + Config lint.Config + Receipt Receipt + Now time.Time +} + +// ConfirmResult reports what was recorded. +type ConfirmResult struct { + BaselinePath string `json:"baseline_path"` + Recorded []string `json:"recorded"` +} + +// ConfirmError is a refused confirmation — a distinct type so a caller can tell +// a bad receipt from an I/O fault without string matching. +type ConfirmError struct{ msg string } + +func (e *ConfirmError) Error() string { return e.msg } + +// ParseReceipt decodes a receipt file, refusing anything it does not fully +// understand. +func ParseReceipt(data []byte) (Receipt, error) { + dec := json.NewDecoder(bytes.NewReader(data)) + dec.DisallowUnknownFields() + var r Receipt + if err := dec.Decode(&r); err != nil { + return Receipt{}, &ConfirmError{"citation receipt is not valid: " + err.Error()} + } + if r.SchemaVersion != ReceiptSchemaVersion { + return Receipt{}, &ConfirmError{ + "citation receipt schema_version is " + strconv.Itoa(r.SchemaVersion) + + "; this build understands only " + strconv.Itoa(ReceiptSchemaVersion)} + } + return r, nil +} + +// LoadReceipt reads and parses a receipt file. +func LoadReceipt(path string) (Receipt, error) { + data, err := fsutil.ReadGuarded(path, receiptSizeLimit) + if err != nil { + return Receipt{}, err + } + return ParseReceipt(data) +} + +// Confirm records a human's confirmations in the baseline. +// +// Every line is validated against the docs' cited set BEFORE anything is +// written, and one bad line refuses the whole receipt. Confirming is the single +// place a receipt enters the record without a fetch, so it must not become a +// door through which arbitrary entries arrive — and a batch that half-applied +// would leave a maintainer unable to tell which half. +func Confirm(req ConfirmRequest) (ConfirmResult, error) { + now := req.Now + if now.IsZero() { + now = time.Now().UTC() + } + today := now.Format(dateLayout) + + if len(req.Receipt.Confirmed) == 0 { + return ConfirmResult{}, &ConfirmError{"citation receipt confirms nothing"} + } + + relPath, _, _ := lint.CitationPolicy(req.Config) + absPath := filepath.Join(req.RepoRoot, filepath.FromSlash(relPath)) + + cited, err := lint.CollectCitedURLs(req.Config, req.RepoRoot) + if err != nil { + return ConfirmResult{}, err + } + citedSet := make(map[string]bool, len(cited)) + for _, c := range cited { + citedSet[c.URL] = true + } + + // Validate the whole receipt first: nothing is written until every line is + // good. + entries := make(map[string]lint.BaselineEntry, len(req.Receipt.Confirmed)) + recorded := make([]string, 0, len(req.Receipt.Confirmed)) + for _, c := range req.Receipt.Confirmed { + if !citedSet[c.URL] { + return ConfirmResult{}, &ConfirmError{ + "cannot confirm " + c.URL + ": it is not cited anywhere under the configured roots. " + + "A receipt records a citation this repo publishes, never an arbitrary address"} + } + on := c.VerifiedOn + if on == "" { + on = today + } + when, perr := time.Parse(dateLayout, on) + if perr != nil { + return ConfirmResult{}, &ConfirmError{ + "cannot confirm " + c.URL + ": verified_on " + strconv.Quote(on) + " is not a " + dateLayout + " date"} + } + if daysBetween(when, now) < 0 { + return ConfirmResult{}, &ConfirmError{ + "cannot confirm " + c.URL + ": verified_on " + on + " is in the future"} + } + final := c.FinalURL + if final == "" { + final = c.URL + } + entries[c.URL] = lint.BaselineEntry{ + URL: c.URL, FinalURL: final, LastChecked: on, + Outcome: lint.OutcomeAlive, Verification: lint.VerificationManual, VerifiedOn: on, + } + recorded = append(recorded, c.URL) + } + + // A missing baseline is fine — a repo may clear its queue before its first + // refresh. A malformed one is a refusal, for the same reason a refresh + // refuses it. + merged := map[string]lint.BaselineEntry{} + old, err := lint.LoadBaseline(absPath) + switch { + case err == nil: + for k, v := range old.Entries { + merged[k] = v + } + case errors.Is(err, os.ErrNotExist): + default: + return ConfirmResult{}, err + } + for k, v := range entries { + merged[k] = v + } + + if err := lint.SaveBaseline(absPath, lint.Baseline{SchemaVersion: lint.BaselineSchemaVersion, Entries: merged}); err != nil { + return ConfirmResult{}, err + } + sort.Strings(recorded) + return ConfirmResult{BaselinePath: relPath, Recorded: recorded}, nil +} diff --git a/internal/core/cite/confirm_test.go b/internal/core/cite/confirm_test.go new file mode 100644 index 00000000..359aea4a --- /dev/null +++ b/internal/core/cite/confirm_test.go @@ -0,0 +1,191 @@ +package cite + +import ( + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/REPPL/abcd-cli/internal/core/lint" +) + +// TestConfirmWritesADatedManualReceipt is AC 4's floor: a human clears one queue +// line and the baseline records THAT they confirmed it and WHEN. +func TestConfirmWritesADatedManualReceipt(t *testing.T) { + root := citeRepo(t, "https://paywalled.example.org/x") + + res, err := Confirm(ConfirmRequest{ + RepoRoot: root, Config: testConfig(), Now: now, + Receipt: Receipt{ + SchemaVersion: ReceiptSchemaVersion, + Confirmed: []ConfirmedCitation{{URL: "https://paywalled.example.org/x"}}, + }, + }) + if err != nil { + t.Fatalf("Confirm: %v", err) + } + if len(res.Recorded) != 1 || res.Recorded[0] != "https://paywalled.example.org/x" { + t.Fatalf("recorded = %v, want the confirmed URL", res.Recorded) + } + + got := loadBaseline(t, root).Entries["https://paywalled.example.org/x"] + want := lint.BaselineEntry{ + URL: "https://paywalled.example.org/x", + FinalURL: "https://paywalled.example.org/x", + LastChecked: today, + Outcome: lint.OutcomeAlive, + Verification: lint.VerificationManual, + VerifiedOn: today, + } + if got != want { + t.Fatalf("entry = %+v, want %+v", got, want) + } +} + +// TestConfirmRecordsADeclaredFinalAddress lets a human report a redirect they +// followed, so a manually-cleared citation carries the same drift information an +// automatic one does. +func TestConfirmRecordsADeclaredFinalAddress(t *testing.T) { + root := citeRepo(t, "https://paywalled.example.org/x") + _, err := Confirm(ConfirmRequest{ + RepoRoot: root, Config: testConfig(), Now: now, + Receipt: Receipt{Confirmed: []ConfirmedCitation{{ + URL: "https://paywalled.example.org/x", FinalURL: "https://paywalled.example.org/x-v2", + VerifiedOn: "2026-07-20", + }}}, + }) + if err != nil { + t.Fatalf("Confirm: %v", err) + } + got := loadBaseline(t, root).Entries["https://paywalled.example.org/x"] + if got.FinalURL != "https://paywalled.example.org/x-v2" { + t.Errorf("final_url = %q, want the declared address", got.FinalURL) + } + // The declared date is the staleness clock: a receipt for work done last week + // must age from last week, not from whenever it was typed in. + if got.VerifiedOn != "2026-07-20" || got.LastChecked != "2026-07-20" { + t.Errorf("dates = %q/%q, want the declared 2026-07-20", got.VerifiedOn, got.LastChecked) + } +} + +// TestConfirmRefusesAnUncitedURL is the anti-forgery clause. Confirming is the +// one place a receipt enters the record without a fetch, so it must not be a +// door through which arbitrary entries arrive — only what the docs actually cite +// can be confirmed. +func TestConfirmRefusesAnUncitedURL(t *testing.T) { + root := citeRepo(t, "https://example.org/cited") + _, err := Confirm(ConfirmRequest{ + RepoRoot: root, Config: testConfig(), Now: now, + Receipt: Receipt{Confirmed: []ConfirmedCitation{{URL: "https://elsewhere.example.org/never-cited"}}}, + }) + if err == nil { + t.Fatal("Confirm accepted a URL the docs do not cite") + } + if !strings.Contains(err.Error(), "not cited") { + t.Errorf("error = %q, want it to say the URL is not cited", err) + } + // Nothing is written when any line is refused: a receipt is all-or-nothing, + // so a batch with one bad line cannot half-apply. + if _, statErr := os.Stat(filepath.Join(root, filepath.FromSlash(lint.DefaultBaselinePath))); statErr == nil { + t.Error("a refused confirm still wrote a baseline") + } +} + +// TestConfirmRefusesAFutureDate stops a receipt from buying itself an extra year +// of freshness by declaring tomorrow. +func TestConfirmRefusesAFutureDate(t *testing.T) { + root := citeRepo(t, "https://example.org/cited") + _, err := Confirm(ConfirmRequest{ + RepoRoot: root, Config: testConfig(), Now: now, + Receipt: Receipt{Confirmed: []ConfirmedCitation{{URL: "https://example.org/cited", VerifiedOn: "2027-01-01"}}}, + }) + if err == nil { + t.Fatal("Confirm accepted a receipt dated in the future") + } +} + +// TestConfirmPreservesOtherEntries pins that clearing one queue line is a +// surgical edit, not a rewrite: everything else in the baseline survives. +func TestConfirmPreservesOtherEntries(t *testing.T) { + root := citeRepo(t, "https://example.org/a", "https://paywalled.example.org/x") + existing := lint.BaselineEntry{ + FinalURL: "https://example.org/a", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01", + } + writeBaseline(t, root, map[string]lint.BaselineEntry{"https://example.org/a": existing}) + + if _, err := Confirm(ConfirmRequest{ + RepoRoot: root, Config: testConfig(), Now: now, + Receipt: Receipt{Confirmed: []ConfirmedCitation{{URL: "https://paywalled.example.org/x"}}}, + }); err != nil { + t.Fatalf("Confirm: %v", err) + } + entries := loadBaseline(t, root).Entries + if len(entries) != 2 { + t.Fatalf("baseline has %d entries, want 2", len(entries)) + } + got := entries["https://example.org/a"] + got.URL = "" + if got != existing { + t.Errorf("untouched entry = %+v, want %+v", got, existing) + } +} + +// TestParseReceiptRoundTrip pins the ONE receipt schema both rungs use: the verb +// accepts, verbatim, what the later generated checklist page will hand back. +func TestParseReceiptRoundTrip(t *testing.T) { + raw := []byte(`{ + "schema_version": 1, + "confirmed": [ + {"url": "https://a.example.org/x", "final_url": "https://a.example.org/x2", "verified_on": "2026-07-20"}, + {"url": "https://b.example.org/y"} + ] +}`) + got, err := ParseReceipt(raw) + if err != nil { + t.Fatalf("ParseReceipt: %v", err) + } + want := Receipt{SchemaVersion: 1, Confirmed: []ConfirmedCitation{ + {URL: "https://a.example.org/x", FinalURL: "https://a.example.org/x2", VerifiedOn: "2026-07-20"}, + {URL: "https://b.example.org/y"}, + }} + if !reflect.DeepEqual(got, want) { + t.Fatalf("got %+v, want %+v", got, want) + } +} + +// TestParseReceiptRefusesAMethodField carries the schema's central refusal +// through to the ingest path: a receipt is structurally incapable of recording +// HOW a human verified, so a producer that tries is refused at the door rather +// than having the key quietly dropped. +func TestParseReceiptRefusesAMethodField(t *testing.T) { + raw := []byte(`{"schema_version": 1, "confirmed": [{"url": "https://a.example.org/x", "method": "logged in via the library proxy"}]}`) + if _, err := ParseReceipt(raw); err == nil { + t.Fatal("ParseReceipt accepted a method field") + } +} + +// TestParseReceiptRefusesAnUnknownSchemaVersion keeps a future producer's record +// from being best-effort parsed by a build that does not understand it. +func TestParseReceiptRefusesAnUnknownSchemaVersion(t *testing.T) { + if _, err := ParseReceipt([]byte(`{"schema_version": 99, "confirmed": []}`)); err == nil { + t.Fatal("ParseReceipt accepted an unknown schema version") + } +} + +// TestReceiptHasNoMethodFieldByConstruction is the type-level half of the same +// refusal, mirroring the baseline's own construction test. +func TestReceiptHasNoMethodFieldByConstruction(t *testing.T) { + banned := map[string]bool{ + "method": true, "how": true, "transcript": true, "notes": true, + "credential": true, "evidence": true, "proof": true, + } + typ := reflect.TypeOf(ConfirmedCitation{}) + for i := 0; i < typ.NumField(); i++ { + tag, _, _ := strings.Cut(typ.Field(i).Tag.Get("json"), ",") + if banned[tag] { + t.Errorf("ConfirmedCitation declares %q; a receipt records THAT and WHEN, never how", tag) + } + } +} diff --git a/internal/core/cite/fetch.go b/internal/core/cite/fetch.go new file mode 100644 index 00000000..96f059dd --- /dev/null +++ b/internal/core/cite/fetch.go @@ -0,0 +1,214 @@ +package cite + +// The native citation checker: one bounded HTTP attempt per URL. +// +// Three properties are load-bearing, and each is a refusal of something a +// link-checker usually does. +// +// It never retries. A refresh over a reference corpus that retried would become +// a burst against whichever host is having a bad afternoon, and — worse — would +// make the run's duration a function of how many links are failing. One attempt, +// one outcome; a failure is a fact to record, not a reason to try harder. +// +// It never reads a body. Liveness is a property of the status line, so the +// checker takes the response headers and closes the connection. That removes a +// whole class of resource risk (a citation to a huge file costs nothing) and it +// removes the temptation to sniff page content, which is how a "did the page +// really load" heuristic acquires a false-positive rate and a transcript. +// +// It separates "dead" from "not readable by a robot". A 403 is not evidence that +// a source is gone; it is evidence that this fetcher may not look. Recording it +// as broken would put a lie in a committed record the gate then enforces, so +// those statuses route to the manual queue instead. + +import ( + "errors" + "net" + "net/http" + "net/url" + "strconv" + "time" + + "github.com/REPPL/abcd-cli/internal/urlguard" +) + +// maxRedirects bounds the hops one citation may take before the chain is called +// broken. Five matches the bound memory ingest uses; a citation that needs more +// is not a stable address to be citing. +const maxRedirects = 5 + +// DefaultTimeout bounds one whole attempt — connect, TLS, and response headers. +// It is the ceiling on what a single unreachable citation can cost a run. +const DefaultTimeout = 20 * time.Second + +// connectTimeout and headerTimeout bound the two phases separately, so a host +// that accepts a connection and then says nothing cannot consume the whole +// budget while a genuinely slow DNS lookup still has room. +const ( + connectTimeout = 10 * time.Second + headerTimeout = 15 * time.Second +) + +// userAgent identifies the checker honestly. A fetcher that disguises itself as +// a browser to defeat a publisher's robot policy would be manufacturing the very +// "automatic" receipt the manual queue exists to avoid faking. +const userAgent = "abcd-docs-cite-refresh" + +// Status is what one check established about one citation. +type Status string + +const ( + // StatusOK means the address resolved to a live document. + StatusOK Status = "ok" + // StatusBroken means it did not: a 4xx/5xx, an unresolvable host, a refused + // connection, a timeout, or a redirect chain past the bound. + StatusBroken Status = "broken" + // StatusBlocked means the source refused an automated fetcher. That is not + // evidence the citation is dead, so it routes to the manual queue rather than + // into the record. + StatusBlocked Status = "blocked" +) + +// CheckOutcome is one URL's result. It carries a classification, the address the +// request ended at, and a short human-readable reason — never page content, and +// never anything that reaches the committed baseline except FinalURL. +type CheckOutcome struct { + URL string `json:"url"` + Status Status `json:"status"` + FinalURL string `json:"final_url,omitempty"` + // Detail is for the operator's terminal. It is deliberately NOT persisted: + // the baseline records what is true about a link, not the transcript of + // finding out. + Detail string `json:"detail,omitempty"` +} + +// Checker is the refresh seam. spc-17 fixes the baseline schema and the lint +// rules as the contract and leaves the producer replaceable, so a specialist +// link checker can later arrive as an adapter satisfying this one method without +// any gate semantics changing. +type Checker interface { + Check(rawURL string) CheckOutcome +} + +// HTTPChecker is the native, dependency-free implementation of Checker. +type HTTPChecker struct { + client *http.Client + blocked func(net.IP) bool +} + +// shippedBlocked is the address policy every shipped checker runs under. It is +// named rather than inlined so the tests can state precisely which single rule +// they relax (loopback, for httptest) and inherit the rest unchanged. +var shippedBlocked = urlguard.BlockedIP + +// NewHTTPChecker returns the checker as it ships: the full SSRF policy and the +// default timeout. +func NewHTTPChecker() *HTTPChecker { return newHTTPChecker(shippedBlocked, DefaultTimeout) } + +// newHTTPChecker builds a checker with an explicit address policy and timeout. +// The policy is a parameter for exactly one reason: it lets the fetch paths be +// exercised against an httptest server, which binds loopback, without ever +// relaxing what NewHTTPChecker ships. +func newHTTPChecker(blocked func(net.IP) bool, timeout time.Duration) *HTTPChecker { + dialer := &net.Dialer{ + Timeout: connectTimeout, + // The connect-time re-check closes the DNS-rebinding gap between the + // name guard below and the transport's own resolution. + Control: urlguard.DialControl(blocked), + } + return &HTTPChecker{ + blocked: blocked, + client: &http.Client{ + Timeout: timeout, + Transport: &http.Transport{ + DialContext: dialer.DialContext, + TLSHandshakeTimeout: connectTimeout, + ResponseHeaderTimeout: headerTimeout, + }, + CheckRedirect: func(r *http.Request, via []*http.Request) error { + if len(via) >= maxRedirects { + return errors.New("more than " + strconv.Itoa(maxRedirects) + " redirects") + } + // Every hop is re-guarded: a public address that redirects to + // 169.254.169.254 must not be followed. + return urlguard.CheckHostWith(r.URL.Hostname(), blocked) + }, + }, + } +} + +// Check performs one bounded attempt and classifies the result. +func (c *HTTPChecker) Check(rawURL string) CheckOutcome { + out := CheckOutcome{URL: rawURL} + + parsed, err := url.Parse(rawURL) + if err != nil { + out.Status, out.Detail = StatusBroken, "not a parsable URL: "+err.Error() + return out + } + if err := urlguard.CheckHostWith(parsed.Hostname(), c.blocked); err != nil { + out.Status, out.Detail = StatusBroken, err.Error() + return out + } + + req, err := http.NewRequest(http.MethodGet, rawURL, nil) + if err != nil { + out.Status, out.Detail = StatusBroken, err.Error() + return out + } + req.Header.Set("User-Agent", userAgent) + // Ask for a document. A source that refuses this Accept is refusing a robot, + // which the status classification below reads correctly. + req.Header.Set("Accept", "text/html,application/xhtml+xml,*/*;q=0.8") + + resp, err := c.client.Do(req) + if err != nil { + out.Status, out.Detail = StatusBroken, transportDetail(err) + return out + } + // The body is never read: liveness is the status line. Closing without + // draining forecloses the connection, which is the correct trade when the + // next request in a run is to a different host anyway. + defer resp.Body.Close() + + out.FinalURL = rawURL + if resp.Request != nil && resp.Request.URL != nil { + out.FinalURL = resp.Request.URL.String() + } + out.Status, out.Detail = classify(resp.StatusCode) + return out +} + +// classify maps a status code onto the three outcomes. +// +// The blocked set is deliberately small and status-code-only: 401 and 403 are a +// credential or robot refusal, 406 is a content-negotiation refusal aimed at +// non-browsers, and 429 is rate limiting. Everything else that is not 2xx is +// broken. No body is sniffed for challenge-page markers — a heuristic over page +// text would be both unreliable and a reason to start reading content, and the +// challenge pages that matter answer with one of these codes anyway. +func classify(code int) (Status, string) { + switch { + case code >= 200 && code < 300: + return StatusOK, "" + case code == http.StatusUnauthorized, code == http.StatusForbidden, + code == http.StatusNotAcceptable, code == http.StatusTooManyRequests: + return StatusBlocked, "HTTP " + strconv.Itoa(code) + " " + http.StatusText(code) + + " — the source refuses automated fetching" + default: + return StatusBroken, "HTTP " + strconv.Itoa(code) + " " + http.StatusText(code) + } +} + +// transportDetail renders a transport failure without the *url.Error wrapper, +// which repeats the URL the outcome already names. +func transportDetail(err error) string { + var ue *url.Error + if errors.As(err, &ue) && ue.Err != nil { + if ue.Timeout() { + return "timed out: " + ue.Err.Error() + } + return ue.Err.Error() + } + return err.Error() +} diff --git a/internal/core/cite/fetch_test.go b/internal/core/cite/fetch_test.go new file mode 100644 index 00000000..09aac38d --- /dev/null +++ b/internal/core/cite/fetch_test.go @@ -0,0 +1,195 @@ +package cite + +import ( + "net" + "net/http" + "net/http/httptest" + "reflect" + "strings" + "testing" + "time" +) + +// loopbackOK is the address policy the fetch tests run under: everything the +// shipped guard refuses, EXCEPT loopback — which is where httptest binds. The +// shipped constructor is never relaxed; only these tests are. +func loopbackOK(ip net.IP) bool { return !ip.IsLoopback() && shippedBlocked(ip) } + +// testChecker builds a checker against the local test policy with a short +// timeout, so a hung server fails a test in a second rather than half a minute. +func testChecker(timeout time.Duration) *HTTPChecker { + return newHTTPChecker(loopbackOK, timeout) +} + +// TestCheckAliveRecordsTheFinalAddress is the ordinary case: a 200 is alive and +// the recorded final address is the one that answered. +func TestCheckAliveRecordsTheFinalAddress(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + got := testChecker(5 * time.Second).Check(srv.URL + "/page") + if got.Status != StatusOK { + t.Fatalf("status = %q (%s), want %q", got.Status, got.Detail, StatusOK) + } + if got.FinalURL != srv.URL+"/page" { + t.Errorf("final URL = %q, want %q", got.FinalURL, srv.URL+"/page") + } +} + +// TestCheckRecordsRedirectDrift is the whole point of recording a final address: +// a citation that still names the pre-redirect URL is alive today and dead the +// day the redirect is retired, so the baseline must capture where it lands. +func TestCheckRedirectDriftIsRecorded(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/old" { + http.Redirect(w, r, "/new", http.StatusMovedPermanently) + return + } + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + got := testChecker(5 * time.Second).Check(srv.URL + "/old") + if got.Status != StatusOK { + t.Fatalf("status = %q (%s), want %q", got.Status, got.Detail, StatusOK) + } + if got.FinalURL != srv.URL+"/new" { + t.Errorf("final URL = %q, want the redirect target %q", got.FinalURL, srv.URL+"/new") + } +} + +// TestCheckRefusesARedirectLoop pins the bound: a self-redirect must terminate as +// an outcome, never spin. +func TestCheckRefusesARedirectLoop(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, "/loop", http.StatusFound) + })) + defer srv.Close() + + got := testChecker(5 * time.Second).Check(srv.URL + "/loop") + if got.Status != StatusBroken { + t.Fatalf("status = %q (%s), want %q", got.Status, got.Detail, StatusBroken) + } + if !strings.Contains(got.Detail, "redirect") { + t.Errorf("detail = %q, want it to name the redirect bound", got.Detail) + } +} + +// TestCheckBlockedStatusesRouteToTheManualQueue pins the exact classification. +// These four say "an automated fetcher may not read this", which is a different +// fact from "this link is dead" — recording them as broken would be a lie the +// gate then enforces. +func TestCheckBlockedStatusesRouteToTheManualQueue(t *testing.T) { + for _, code := range []int{ + http.StatusUnauthorized, // 401 + http.StatusForbidden, // 403 + http.StatusNotAcceptable, // 406 + http.StatusTooManyRequests, // 429 + } { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(code) + })) + got := testChecker(5 * time.Second).Check(srv.URL) + srv.Close() + if got.Status != StatusBlocked { + t.Errorf("HTTP %d: status = %q (%s), want %q", code, got.Status, got.Detail, StatusBlocked) + } + } +} + +// TestCheckBrokenStatuses pins the other side of the same line: a 404 or a 500 is +// a broken citation, not a blocked one. +func TestCheckBrokenStatuses(t *testing.T) { + for _, code := range []int{ + http.StatusNotFound, + http.StatusGone, + http.StatusInternalServerError, + http.StatusServiceUnavailable, + } { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(code) + })) + got := testChecker(5 * time.Second).Check(srv.URL) + srv.Close() + if got.Status != StatusBroken { + t.Errorf("HTTP %d: status = %q (%s), want %q", code, got.Status, got.Detail, StatusBroken) + } + } +} + +// TestCheckTimesOutAsAnOutcome pins the no-hang guarantee: a server that never +// answers costs one timeout, and the run continues. +func TestCheckTimesOutAsAnOutcome(t *testing.T) { + release := make(chan struct{}) + srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, _ *http.Request) { + <-release + })) + defer func() { close(release); srv.Close() }() + + start := time.Now() + got := testChecker(150 * time.Millisecond).Check(srv.URL) + if elapsed := time.Since(start); elapsed > 5*time.Second { + t.Fatalf("check took %s; the timeout is not bounding the attempt", elapsed) + } + if got.Status != StatusBroken { + t.Fatalf("status = %q (%s), want %q", got.Status, got.Detail, StatusBroken) + } +} + +// TestCheckOneAttemptPerURL pins "a failure is an outcome, not a retry": the +// checker must hit the server exactly once, so a refresh over a large corpus +// cannot turn into a storm against a struggling host. +func TestCheckOneAttemptPerURL(t *testing.T) { + var hits int + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hits++ + w.WriteHeader(http.StatusInternalServerError) + })) + defer srv.Close() + + testChecker(5 * time.Second).Check(srv.URL) + if hits != 1 { + t.Fatalf("server saw %d requests, want exactly 1", hits) + } +} + +// TestCheckRefusesInternalAddressesUnderTheShippedPolicy is the SSRF regression +// for THIS fetch path. The shipped constructor — not the relaxed test one — must +// refuse a loopback or metadata address that a committed document cites. +func TestCheckRefusesInternalAddressesUnderTheShippedPolicy(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + for _, target := range []string{srv.URL, "http://169.254.169.254/latest/meta-data/", "http://svc.internal/x"} { + got := NewHTTPChecker().Check(target) + if got.Status != StatusBroken { + t.Errorf("%s: status = %q, want %q", target, got.Status, StatusBroken) + } + if !strings.Contains(got.Detail, "refusing") { + t.Errorf("%s: detail = %q, want an SSRF-guard refusal", target, got.Detail) + } + } +} + +// TestCheckOutcomeCarriesNoContentByConstruction pins a deliberate narrowing: +// liveness is judged from the status line alone, so no cited page's bytes ever +// enter the process or the record. A citation to a multi-gigabyte file costs a +// response header, not a download — and no fetch transcript can leak into the +// committed baseline through this type. +func TestCheckOutcomeCarriesNoContentByConstruction(t *testing.T) { + banned := map[string]bool{ + "body": true, "content": true, "headers": true, "transcript": true, + "response": true, "html": true, "bytes": true, + } + typ := reflect.TypeOf(CheckOutcome{}) + for i := 0; i < typ.NumField(); i++ { + name := strings.ToLower(typ.Field(i).Name) + if banned[name] { + t.Errorf("CheckOutcome declares a %s field; the checker must carry no page content", name) + } + } +} diff --git a/internal/core/cite/refresh.go b/internal/core/cite/refresh.go new file mode 100644 index 00000000..93db61d1 --- /dev/null +++ b/internal/core/cite/refresh.go @@ -0,0 +1,313 @@ +// Package cite is the live half of the citation gate: the on-demand refresh +// that fetches every cited URL and writes the committed baseline the +// zero-network docs lint then enforces offline. +// +// The split is spc-17's whole design bet. Fetching is non-deterministic and +// needs the network; deciding whether a commit is allowed must be neither. So +// this package is the ONLY place abcd dials out on behalf of documentation, it +// runs when a maintainer asks it to, and everything it learns arrives at the +// gate as a reviewable committed record rather than as whatever the network +// happened to answer during CI. +// +// It writes nothing it did not establish. A source that refuses automated +// fetchers produces no entry at all — not a guess, not a "probably fine" — +// because the one thing a committed record must never contain is a receipt +// nobody earned. Those URLs go to a human instead, and the confirm verb records +// that a human cleared them and when, never how. +package cite + +import ( + "errors" + "os" + "path/filepath" + "sort" + "time" + + "github.com/REPPL/abcd-cli/internal/core/lint" +) + +// dateLayout matches the baseline's date precision. The staleness policy is +// measured in days, so an instant would add churn to a committed file for no +// gain in the only arithmetic that reads it. +const dateLayout = "2006-01-02" + +// defaultParallel bounds how many citations are in flight at once. It is small +// on purpose: a refresh is a courtesy visit to other people's servers, and the +// run's cost is already bounded by one attempt per URL. +const defaultParallel = 6 + +// RefreshRequest is the input to one refresh run. +type RefreshRequest struct { + // RepoRoot is the repository to refresh. + RepoRoot string + // Config supplies the roots to walk and the citation policy (baseline path + // and staleness thresholds) — the same config the gate reads, so the verb + // and the gate can never disagree about either. + Config lint.Config + // Checker is the refresh seam. Nil means the native bounded fetcher. + Checker Checker + // Now is the clock. Supplied by the caller so a run's record is a function + // of its inputs. + Now time.Time + // Parallel bounds concurrent checks; zero means defaultParallel. + Parallel int +} + +// Outcome is one cited URL's disposition in a run. +type Outcome struct { + URL string `json:"url"` + Status Status `json:"status"` + FinalURL string `json:"final_url,omitempty"` + Detail string `json:"detail,omitempty"` +} + +// StatusPreserved is the fourth disposition a REFRESH can report, distinct from +// the three a checker can return: the entry was not checked at all, because a +// human's receipt for it is still current. +const StatusPreserved Status = "preserved" + +// QueueItem is one line of the manual checklist (spc-17's rung 1): a URL an +// automated fetcher may not read, and where the docs cite it. +type QueueItem struct { + URL string `json:"url"` + Detail string `json:"detail"` + Sites []lint.CitationSite `json:"sites"` +} + +// RefreshResult is the record of one run, for the operator's transcript and for +// `--json`. +type RefreshResult struct { + // BaselinePath is repo-relative — never absolute, so machine output carries + // no developer-identity path. + BaselinePath string `json:"baseline_path"` + Cited int `json:"cited"` + // Fetched counts the URLs actually requested this run; Preserved counts the + // current manual receipts left untouched. They sum to Cited. + Fetched int `json:"fetched"` + Preserved int `json:"preserved"` + Outcomes []Outcome `json:"outcomes"` + // Queue is the manual checklist: everything a human must clear. + Queue []QueueItem `json:"queue"` + // Dropped lists entries removed because the docs no longer cite them. + Dropped []string `json:"dropped,omitempty"` +} + +// Counts summarises the outcomes by status, for a one-line report. +func (r RefreshResult) Counts() (ok, broken, blocked, preserved int) { + for _, o := range r.Outcomes { + switch o.Status { + case StatusOK: + ok++ + case StatusBroken: + broken++ + case StatusBlocked: + blocked++ + case StatusPreserved: + preserved++ + } + } + return ok, broken, blocked, preserved +} + +// Refresh checks every cited URL and rewrites the baseline. +// +// The merge is where the honesty rules live, and there are three: +// +// - A current MANUAL receipt is preserved verbatim and its URL is not even +// requested. Re-fetching it could only produce a worse answer (the source +// blocks robots, which is why a human looked in the first place), and +// overwriting a human's confirmation with a robot's failure is the exact +// downgrade the record must never suffer. +// - A STALE manual receipt is re-checked, because AC 3 puts human and machine +// verifications on one clock: once it ages out it stops being authoritative. +// If the re-check succeeds the entry becomes a fresh automatic one; if the +// source still blocks, the stale receipt is KEPT (so the gate keeps warning +// truthfully) and the URL re-enters the queue. +// - A BLOCKED URL with no prior entry writes nothing. The gate then reports it +// as unreceipted, which is precisely the truth, and the queue names it. +// +// Entries whose URLs the docs no longer cite are dropped: the baseline answers +// for what is published today. +func Refresh(req RefreshRequest) (RefreshResult, error) { + now := req.Now + if now.IsZero() { + now = time.Now().UTC() + } + stamp := now.Format(dateLayout) + + checker := req.Checker + if checker == nil { + checker = NewHTTPChecker() + } + parallel := req.Parallel + if parallel <= 0 { + parallel = defaultParallel + } + + relPath, warnDays, _ := lint.CitationPolicy(req.Config) + absPath := filepath.Join(req.RepoRoot, filepath.FromSlash(relPath)) + + cited, err := lint.CollectCitedURLs(req.Config, req.RepoRoot) + if err != nil { + return RefreshResult{}, err + } + + // A missing baseline is the ordinary first run. A malformed one is a + // refusal: silently replacing a record we cannot read would discard whatever + // manual receipts it held, which is the one loss no later run can repair. + previous := map[string]lint.BaselineEntry{} + old, err := lint.LoadBaseline(absPath) + switch { + case err == nil: + previous = old.Entries + case isNotExist(err): + default: + return RefreshResult{}, err + } + + // Decide per URL whether it is checked at all, then run the checks bounded. + type job struct { + ref lint.CitedURL + prev lint.BaselineEntry + had bool + } + var jobs []job + var toCheck []string + res := RefreshResult{BaselinePath: relPath, Cited: len(cited)} + next := map[string]lint.BaselineEntry{} + + for _, ref := range cited { + prev, had := previous[ref.URL] + if had && prev.Verification == lint.VerificationManual && !stale(prev, now, warnDays) { + next[ref.URL] = prev + res.Preserved++ + res.Outcomes = append(res.Outcomes, Outcome{ + URL: ref.URL, Status: StatusPreserved, FinalURL: prev.FinalURL, + Detail: "human-verified " + prev.VerifiedOn + "; not re-checked until it goes stale", + }) + continue + } + jobs = append(jobs, job{ref: ref, prev: prev, had: had}) + toCheck = append(toCheck, ref.URL) + } + res.Fetched = len(jobs) + + checked := runChecks(checker, toCheck, parallel) + + for _, j := range jobs { + out := checked[j.ref.URL] + res.Outcomes = append(res.Outcomes, Outcome{ + URL: j.ref.URL, Status: out.Status, FinalURL: out.FinalURL, Detail: out.Detail, + }) + switch out.Status { + case StatusOK: + final := out.FinalURL + if final == "" { + final = j.ref.URL + } + next[j.ref.URL] = lint.BaselineEntry{ + URL: j.ref.URL, FinalURL: final, LastChecked: stamp, + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: stamp, + } + case StatusBroken: + // A broken entry still needs a resolved address for the schema; the + // cited one is the honest answer when nothing else resolved. + final := out.FinalURL + if final == "" { + final = j.ref.URL + } + next[j.ref.URL] = lint.BaselineEntry{ + URL: j.ref.URL, FinalURL: final, LastChecked: stamp, + Outcome: lint.OutcomeBroken, Verification: lint.VerificationAutomatic, VerifiedOn: stamp, + } + case StatusBlocked: + if j.had { + next[j.ref.URL] = j.prev + } + res.Queue = append(res.Queue, QueueItem{URL: j.ref.URL, Detail: out.Detail, Sites: j.ref.Sites}) + } + } + + citedSet := map[string]bool{} + for _, ref := range cited { + citedSet[ref.URL] = true + } + for u := range previous { + if !citedSet[u] { + res.Dropped = append(res.Dropped, u) + } + } + + sort.Slice(res.Outcomes, func(i, j int) bool { return res.Outcomes[i].URL < res.Outcomes[j].URL }) + sort.Slice(res.Queue, func(i, j int) bool { return res.Queue[i].URL < res.Queue[j].URL }) + sort.Strings(res.Dropped) + + if err := lint.SaveBaseline(absPath, lint.Baseline{SchemaVersion: lint.BaselineSchemaVersion, Entries: next}); err != nil { + return RefreshResult{}, err + } + return res, nil +} + +// stale reports whether an entry has passed the warn threshold — the point at +// which AC 3 stops treating a verification, human or machine, as authoritative. +func stale(e lint.BaselineEntry, now time.Time, warnDays int) bool { + checked, err := e.Checked() + if err != nil { + return true // unparsable is not current; re-check it + } + return daysBetween(checked, now) >= warnDays +} + +// daysBetween counts whole calendar days, both ends normalised to UTC midnight, +// matching the gate's arithmetic exactly. +func daysBetween(from, to time.Time) int { + f := time.Date(from.Year(), from.Month(), from.Day(), 0, 0, 0, 0, time.UTC) + t := time.Date(to.Year(), to.Month(), to.Day(), 0, 0, 0, 0, time.UTC) + return int(t.Sub(f).Hours() / 24) +} + +// runChecks fans the checks out over a bounded worker pool and collects them +// into a map, so the caller's iteration order — and therefore the record — stays +// a function of the sorted cited set rather than of scheduling. +func runChecks(checker Checker, urls []string, parallel int) map[string]CheckOutcome { + out := make(map[string]CheckOutcome, len(urls)) + if len(urls) == 0 { + return out + } + if parallel > len(urls) { + parallel = len(urls) + } + if parallel <= 1 { + for _, u := range urls { + out[u] = checker.Check(u) + } + return out + } + + results := make([]CheckOutcome, len(urls)) + next := make(chan int) + done := make(chan struct{}) + for w := 0; w < parallel; w++ { + go func() { + for i := range next { + results[i] = checker.Check(urls[i]) + } + done <- struct{}{} + }() + } + for i := range urls { + next <- i + } + close(next) + for w := 0; w < parallel; w++ { + <-done + } + for i, u := range urls { + out[u] = results[i] + } + return out +} + +// isNotExist reports the "no baseline yet" case. LoadBaseline returns the +// underlying os.ErrNotExist unwrapped for exactly this distinction. +func isNotExist(err error) bool { return errors.Is(err, os.ErrNotExist) } diff --git a/internal/core/cite/refresh_test.go b/internal/core/cite/refresh_test.go new file mode 100644 index 00000000..8951957a --- /dev/null +++ b/internal/core/cite/refresh_test.go @@ -0,0 +1,344 @@ +package cite + +import ( + "os" + "path/filepath" + "sort" + "sync" + "testing" + "time" + + "github.com/REPPL/abcd-cli/internal/core/lint" +) + +// now is the clock every refresh test runs against, so a dated record is a +// function of the fixture rather than of the day the suite runs. +var now = time.Date(2026, 7, 27, 9, 0, 0, 0, time.UTC) + +const today = "2026-07-27" + +// stubChecker answers from a table and records every URL it was asked about, so +// a test can assert not only the outcome but that a URL was — or was not — +// fetched at all. +type stubChecker struct { + answers map[string]CheckOutcome + mu sync.Mutex + asked []string +} + +func (s *stubChecker) Check(rawURL string) CheckOutcome { + s.mu.Lock() + s.asked = append(s.asked, rawURL) + s.mu.Unlock() + if out, ok := s.answers[rawURL]; ok { + out.URL = rawURL + return out + } + return CheckOutcome{URL: rawURL, Status: StatusOK, FinalURL: rawURL} +} + +func (s *stubChecker) askedFor(rawURL string) bool { + s.mu.Lock() + defer s.mu.Unlock() + for _, a := range s.asked { + if a == rawURL { + return true + } + } + return false +} + +// citeRepo plants a fixture repo whose docs cite the given URLs, one per page. +func citeRepo(t *testing.T, urls ...string) string { + t.Helper() + root := t.TempDir() + if err := os.MkdirAll(filepath.Join(root, "docs"), 0o755); err != nil { + t.Fatal(err) + } + for i, u := range urls { + body := "# Page\n\nA claim.[^a]\n\n[^a]: A source, " + u + "\n" + name := filepath.Join(root, "docs", "p"+string(rune('a'+i))+".md") + if err := os.WriteFile(name, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + return root +} + +// testConfig is a docs-lint config whose citation rule points at the default +// baseline path and carries spc-17's thresholds. +func testConfig() lint.Config { + return lint.Config{ + Roots: []string{"docs"}, + Rules: map[string]lint.RuleConfig{ + "citation_baseline": {Enabled: true, Severity: "blocker", WarnAfterDays: 180, BlockAfterDays: 365}, + }, + } +} + +// loadBaseline reads back what a refresh wrote, THROUGH the gate's own loader — +// so every test asserts not just the content but that the gate can read it. +func loadBaseline(t *testing.T, root string) lint.Baseline { + t.Helper() + b, err := lint.LoadBaseline(filepath.Join(root, filepath.FromSlash(lint.DefaultBaselinePath))) + if err != nil { + t.Fatalf("LoadBaseline: %v", err) + } + return b +} + +func writeBaseline(t *testing.T, root string, entries map[string]lint.BaselineEntry) { + t.Helper() + path := filepath.Join(root, filepath.FromSlash(lint.DefaultBaselinePath)) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := lint.SaveBaseline(path, lint.Baseline{SchemaVersion: 1, Entries: entries}); err != nil { + t.Fatal(err) + } +} + +// TestRefreshWritesAliveEntries is the happy path: a live citation becomes an +// automatic entry dated today, recording where it actually resolved. +func TestRefreshWritesAliveEntries(t *testing.T) { + root := citeRepo(t, "https://example.org/a") + checker := &stubChecker{answers: map[string]CheckOutcome{ + "https://example.org/a": {Status: StatusOK, FinalURL: "https://example.org/a-moved"}, + }} + + res, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}) + if err != nil { + t.Fatalf("Refresh: %v", err) + } + if res.Cited != 1 || res.Fetched != 1 { + t.Fatalf("cited=%d fetched=%d, want 1/1", res.Cited, res.Fetched) + } + + got := loadBaseline(t, root).Entries["https://example.org/a"] + want := lint.BaselineEntry{ + URL: "https://example.org/a", + FinalURL: "https://example.org/a-moved", + LastChecked: today, + Outcome: lint.OutcomeAlive, + Verification: lint.VerificationAutomatic, + VerifiedOn: today, + } + if got != want { + t.Fatalf("entry = %+v, want %+v", got, want) + } +} + +// TestRefreshRecordsBroken pins that a dead link is recorded as dead, so the +// offline gate blocks on it without anyone re-fetching. +func TestRefreshRecordsBroken(t *testing.T) { + root := citeRepo(t, "https://example.org/gone") + checker := &stubChecker{answers: map[string]CheckOutcome{ + "https://example.org/gone": {Status: StatusBroken, Detail: "HTTP 404 Not Found"}, + }} + + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}); err != nil { + t.Fatalf("Refresh: %v", err) + } + got := loadBaseline(t, root).Entries["https://example.org/gone"] + if got.Outcome != lint.OutcomeBroken { + t.Fatalf("outcome = %q, want %q", got.Outcome, lint.OutcomeBroken) + } + // A broken entry still needs a final_url for the schema; the cited address + // is the honest answer when nothing else resolved. + if got.FinalURL != "https://example.org/gone" { + t.Errorf("final_url = %q, want the cited address", got.FinalURL) + } +} + +// TestRefreshRoutesBlockedToTheQueueWithoutInventingAnEntry is the honesty +// clause. A 403 says the fetcher may not look — it is NOT evidence the source is +// gone, so nothing is written. The gate then reports the URL as unreceipted, +// which is exactly true, and the manual queue names it for a human. +func TestRefreshRoutesBlockedToTheQueueWithoutInventingAnEntry(t *testing.T) { + root := citeRepo(t, "https://paywalled.example.org/x") + checker := &stubChecker{answers: map[string]CheckOutcome{ + "https://paywalled.example.org/x": {Status: StatusBlocked, Detail: "HTTP 403 Forbidden"}, + }} + + res, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}) + if err != nil { + t.Fatalf("Refresh: %v", err) + } + if _, ok := loadBaseline(t, root).Entries["https://paywalled.example.org/x"]; ok { + t.Fatal("a blocked fetch invented a baseline entry; only a human may record this one") + } + if len(res.Queue) != 1 || res.Queue[0].URL != "https://paywalled.example.org/x" { + t.Fatalf("queue = %+v, want the blocked URL", res.Queue) + } + if len(res.Queue[0].Sites) == 0 { + t.Error("a queue item must name where the URL is cited") + } +} + +// TestRefreshPreservesAFreshManualReceipt is the never-downgrade rule: a human +// confirmation is not re-litigated by a fetcher, and the URL is not even +// requested — so a source that only ever answers 403 does not re-enter the queue +// on every run. +func TestRefreshPreservesAFreshManualReceipt(t *testing.T) { + root := citeRepo(t, "https://paywalled.example.org/x") + manual := lint.BaselineEntry{ + FinalURL: "https://paywalled.example.org/x", LastChecked: "2026-06-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationManual, VerifiedOn: "2026-06-01", + } + writeBaseline(t, root, map[string]lint.BaselineEntry{"https://paywalled.example.org/x": manual}) + + checker := &stubChecker{} + res, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}) + if err != nil { + t.Fatalf("Refresh: %v", err) + } + if checker.askedFor("https://paywalled.example.org/x") { + t.Error("a fresh manual receipt was re-fetched; it must be left alone until it ages out") + } + if res.Preserved != 1 || res.Fetched != 0 { + t.Fatalf("preserved=%d fetched=%d, want 1/0", res.Preserved, res.Fetched) + } + got := loadBaseline(t, root).Entries["https://paywalled.example.org/x"] + got.URL = "" // the writer restates the key; the fixture did not + if got != manual { + t.Fatalf("entry = %+v, want it preserved verbatim %+v", got, manual) + } +} + +// TestRefreshRecheckesAStaleManualReceipt is AC 3's "same clock": once a human +// verification passes the staleness threshold it stops being authoritative, and +// the refresh tries again. A fetch that now succeeds re-dates the entry as +// automatic — the manual receipt has already aged out, so nothing is downgraded. +func TestRefreshRechecksAStaleManualReceipt(t *testing.T) { + root := citeRepo(t, "https://example.org/x") + writeBaseline(t, root, map[string]lint.BaselineEntry{"https://example.org/x": { + FinalURL: "https://example.org/x", LastChecked: "2025-01-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationManual, VerifiedOn: "2025-01-01", + }}) + + checker := &stubChecker{} + res, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}) + if err != nil { + t.Fatalf("Refresh: %v", err) + } + if !checker.askedFor("https://example.org/x") { + t.Fatal("a stale manual receipt was not re-checked") + } + if res.Preserved != 0 { + t.Errorf("preserved = %d, want 0", res.Preserved) + } + got := loadBaseline(t, root).Entries["https://example.org/x"] + if got.Verification != lint.VerificationAutomatic || got.LastChecked != today { + t.Fatalf("entry = %+v, want a fresh automatic receipt", got) + } +} + +// TestRefreshKeepsAStaleManualReceiptWhenTheFetchIsStillBlocked closes the other +// half: re-checking a stale manual entry must not DELETE it when the fetcher is +// still refused. The stale receipt stays (so the gate keeps warning, honestly) +// and the URL re-enters the queue for a human. +func TestRefreshKeepsAStaleManualReceiptWhenTheFetchIsStillBlocked(t *testing.T) { + root := citeRepo(t, "https://paywalled.example.org/x") + stale := lint.BaselineEntry{ + FinalURL: "https://paywalled.example.org/x", LastChecked: "2025-01-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationManual, VerifiedOn: "2025-01-01", + } + writeBaseline(t, root, map[string]lint.BaselineEntry{"https://paywalled.example.org/x": stale}) + + checker := &stubChecker{answers: map[string]CheckOutcome{ + "https://paywalled.example.org/x": {Status: StatusBlocked, Detail: "HTTP 403 Forbidden"}, + }} + res, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}) + if err != nil { + t.Fatalf("Refresh: %v", err) + } + got := loadBaseline(t, root).Entries["https://paywalled.example.org/x"] + got.URL = "" + if got != stale { + t.Fatalf("entry = %+v, want the stale receipt kept %+v", got, stale) + } + if len(res.Queue) != 1 { + t.Fatalf("queue = %+v, want the still-blocked URL", res.Queue) + } +} + +// TestRefreshDropsUncitedEntries keeps the record honest in the other direction: +// the baseline answers for what the docs cite TODAY, so a receipt for a citation +// that has been edited out is removed rather than left to age forever. +func TestRefreshDropsUncitedEntries(t *testing.T) { + root := citeRepo(t, "https://example.org/kept") + writeBaseline(t, root, map[string]lint.BaselineEntry{ + "https://example.org/kept": {FinalURL: "https://example.org/kept", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01"}, + "https://example.org/removed": {FinalURL: "https://example.org/removed", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationManual, VerifiedOn: "2026-07-01"}, + }) + + res, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: &stubChecker{}, Now: now}) + if err != nil { + t.Fatalf("Refresh: %v", err) + } + entries := loadBaseline(t, root).Entries + if _, ok := entries["https://example.org/removed"]; ok { + t.Error("an entry for a URL the docs no longer cite survived the refresh") + } + if len(res.Dropped) != 1 || res.Dropped[0] != "https://example.org/removed" { + t.Errorf("dropped = %v, want the uncited URL", res.Dropped) + } +} + +// TestRefreshIsDeterministic pins that two runs over the same inputs produce the +// same bytes — the baseline is committed and reviewed in diffs, so a refresh +// that reordered entries would generate churn nobody can read. +func TestRefreshIsDeterministic(t *testing.T) { + root := citeRepo(t, "https://example.org/a", "https://example.org/b", "https://example.org/c") + path := filepath.Join(root, filepath.FromSlash(lint.DefaultBaselinePath)) + + run := func() string { + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: &stubChecker{}, Now: now}); err != nil { + t.Fatalf("Refresh: %v", err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + return string(data) + } + if first, second := run(), run(); first != second { + t.Fatal("two refreshes over identical inputs produced different bytes") + } +} + +// TestRefreshOutcomesAreSorted keeps the printed transcript stable, which is what +// lets an operator diff two runs by eye. +func TestRefreshOutcomesAreSorted(t *testing.T) { + root := citeRepo(t, "https://example.org/c", "https://example.org/a", "https://example.org/b") + res, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: &stubChecker{}, Now: now}) + if err != nil { + t.Fatalf("Refresh: %v", err) + } + urls := make([]string, 0, len(res.Outcomes)) + for _, o := range res.Outcomes { + urls = append(urls, o.URL) + } + if !sort.StringsAreSorted(urls) { + t.Fatalf("outcomes are not sorted: %v", urls) + } +} + +// TestRefreshRefusesAMalformedBaseline is the fail-closed clause: a corrupt +// committed record must stop the run, never be silently replaced by a fresh one +// that discards whatever manual receipts it held. +func TestRefreshRefusesAMalformedBaseline(t *testing.T) { + root := citeRepo(t, "https://example.org/a") + path := filepath.Join(root, filepath.FromSlash(lint.DefaultBaselinePath)) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(`{"schema_version": 1, "entries": {"x": {"method": "curl"}}}`), 0o644); err != nil { + t.Fatal(err) + } + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: &stubChecker{}, Now: now}); err == nil { + t.Fatal("Refresh overwrote a baseline it could not read") + } +} diff --git a/internal/core/lint/citations.go b/internal/core/lint/citations.go index a5313427..881b0a11 100644 --- a/internal/core/lint/citations.go +++ b/internal/core/lint/citations.go @@ -44,6 +44,28 @@ const ( defaultCitationBlockDays = 365 ) +// CitationPolicy resolves the citation-baseline settings a repo's config +// declares, filling in spc-17's defaults for anything it leaves unset. +// +// It exists so the refresh verb and the gate read the SAME policy: the verb +// decides which entries are stale enough to re-check from the very number the +// lint will later warn on, rather than from a second copy of 180 that drifts. +func CitationPolicy(cfg Config) (baselinePath string, warnDays, blockDays int) { + rc := cfg.Rules[ruleCitationBaseline] + baselinePath = strings.TrimSpace(rc.Baseline) + if baselinePath == "" { + baselinePath = DefaultBaselinePath + } + warnDays, blockDays = rc.WarnAfterDays, rc.BlockAfterDays + if warnDays <= 0 { + warnDays = defaultCitationWarnDays + } + if blockDays <= 0 { + blockDays = defaultCitationBlockDays + } + return baselinePath, warnDays, blockDays +} + // defaultCrosswalkHeading is the narrowest identification heuristic that matches // the real crosswalk: a table counts only when its nearest preceding heading // names itself a crosswalk. Anchoring on the heading rather than on "any table diff --git a/internal/core/lint/collect.go b/internal/core/lint/collect.go new file mode 100644 index 00000000..b627d490 --- /dev/null +++ b/internal/core/lint/collect.go @@ -0,0 +1,76 @@ +package lint + +// The citation collector: the one place that answers "what does this repo cite?" +// +// The gate and the refresh verb must agree on that set exactly. If the refresh +// fetched a different set — a second scraper with its own idea of where a +// citation lives — every disagreement would surface as a permanent lint finding +// nobody could clear: a URL the gate demands a receipt for and the refresh never +// visits. So the walk lives here once, LintAt's citation family and +// `abcd docs cite refresh` both read it, and the two cannot drift. +// +// It is a thin composition of the primitives the lint already uses +// (markdownFiles, fenceMask, parseCitations) rather than new parsing. + +import ( + "os" + "path/filepath" + "sort" + "strings" +) + +// CitationSite is one place a URL is cited: a repo-relative page and a 1-based +// line. A URL cited from several pages carries one site per occurrence, so the +// manual queue can tell the maintainer where to look. +type CitationSite struct { + File string `json:"file"` + Line int `json:"line"` +} + +// CitedURL is one distinct cited address and everywhere it is cited. +type CitedURL struct { + URL string `json:"url"` + Sites []CitationSite `json:"sites"` +} + +// CollectCitedURLs walks cfg.Roots and returns the repo's distinct cited URLs, +// sorted by address so a refresh run and its baseline diff are deterministic. +// +// "Cited" means what the lint means by it: a URL on a footnote-definition line +// (including its wrapped continuation lines), never one in body prose. A missing +// root is skipped rather than an error, matching Lint. +func CollectCitedURLs(cfg Config, repoRoot string) ([]CitedURL, error) { + sites := map[string][]CitationSite{} + for _, root := range cfg.Roots { + rootAbs := filepath.Join(repoRoot, root) + mdFiles, err := markdownFiles(rootAbs) + if err != nil { + return nil, err + } + for _, fileAbs := range mdFiles { + content, err := os.ReadFile(fileAbs) + if err != nil { + return nil, err + } + rel := repoRel(repoRoot, fileAbs) + lines := strings.Split(string(content), "\n") + page := parseCitations(rel, lines, fenceMask(lines)) + for _, ref := range page.urls { + site := CitationSite{File: ref.file, Line: ref.line} + // A URL repeated on one line (a source and its mirror) is one + // site, not two — the queue would otherwise print it twice. + if existing := sites[ref.url]; len(existing) > 0 && existing[len(existing)-1] == site { + continue + } + sites[ref.url] = append(sites[ref.url], site) + } + } + } + + out := make([]CitedURL, 0, len(sites)) + for u, s := range sites { + out = append(out, CitedURL{URL: u, Sites: s}) + } + sort.Slice(out, func(i, j int) bool { return out[i].URL < out[j].URL }) + return out, nil +} diff --git a/internal/core/lint/collect_test.go b/internal/core/lint/collect_test.go new file mode 100644 index 00000000..6d46a2d8 --- /dev/null +++ b/internal/core/lint/collect_test.go @@ -0,0 +1,81 @@ +package lint + +import ( + "testing" +) + +// TestCollectCitedURLs pins the refresh verb's contract with the gate: the set +// of URLs the refresh fetches is EXACTLY the set the baseline rule enforces, +// because both come out of this one collector. A second scraper with its own +// idea of what a citation is would produce a baseline the gate rejects. +func TestCollectCitedURLs(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/a.md": "# A\n\nA claim.[^a] A body link to https://prose.example.org/x is not a citation.\n\n" + + "[^a]: Source A, https://example.org/a.\n", + "docs/b.md": "# B\n\nTwo claims.[^b][^c]\n\n" + + "[^b]: Source B, https://example.org/b.\n" + + "[^c]: Source A again, https://example.org/a.\n", + "docs/fenced.md": "# Fenced\n\n```\n[^x]: https://fenced.example.org/x\n```\n", + "README.md": "# Root\n\nA claim.[^r]\n\n[^r]: Source R, https://example.org/r.\n", + }) + cfg := Config{Roots: []string{"docs", "README.md"}} + + refs, err := CollectCitedURLs(cfg, root) + if err != nil { + t.Fatalf("CollectCitedURLs: %v", err) + } + + var got []string + for _, r := range refs { + got = append(got, r.URL) + } + want := []string{ + "https://example.org/a", + "https://example.org/b", + "https://example.org/r", + } + if len(got) != len(want) { + t.Fatalf("got %v, want %v", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Fatalf("got %v, want %v (sorted, deduplicated)", got, want) + } + } + + // Every URL carries at least one citation site, so a refresh can tell the + // maintainer WHERE a blocked link is cited without a second walk. + for _, r := range refs { + if len(r.Sites) == 0 { + t.Errorf("%s has no citation site", r.URL) + } + for _, s := range r.Sites { + if s.File == "" || s.Line <= 0 { + t.Errorf("%s has a malformed site %+v", r.URL, s) + } + } + } + // The URL cited twice reports both sites, in file order. + for _, r := range refs { + if r.URL != "https://example.org/a" { + continue + } + if len(r.Sites) != 2 || r.Sites[0].File != "docs/a.md" || r.Sites[1].File != "docs/b.md" { + t.Errorf("duplicate citation sites = %+v, want docs/a.md then docs/b.md", r.Sites) + } + } +} + +// TestCollectCitedURLsMissingRootIsNotAnError mirrors Lint: a configured root +// that does not exist is skipped, so a repo that has not written docs/ yet can +// still run the refresh. +func TestCollectCitedURLsMissingRootIsNotAnError(t *testing.T) { + root := citeRepo(t, map[string]string{"docs/a.md": "# A\n"}) + refs, err := CollectCitedURLs(Config{Roots: []string{"docs", "nope"}}, root) + if err != nil { + t.Fatalf("CollectCitedURLs: %v", err) + } + if len(refs) != 0 { + t.Fatalf("got %+v, want no citations", refs) + } +} From e90a97cc0558300dcbdf5a8c88cfd02352fe45f1 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:28:18 +0100 Subject: [PATCH 09/18] feat: wire the citation baseline into the CLI, ahoy, and the release gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The refresh verb is manual, so spc-17's answer to "who remembers to run it" is nagging where a maintainer already looks. All three surfaces land here. `abcd docs cite refresh` and `abcd docs cite confirm` hang off `docs` because the gate is their customer: both resolve their roots and baseline path through the same docs-lint config `docs lint` reads, so the set the refresh fetches is the set the gate demands receipts for. Refresh prints the manual queue as a checklist naming where each blocked URL is cited, and exits zero after recording a broken link — it records, the gate decides. Confirm takes URLs positionally or a receipt file; both assemble the same Receipt, so the later generated checklist page is a different producer of one schema rather than a second pathway. `ahoy` gains a citations signal — coverage and age at a glance, present only in a repo that armed the rule. It is a signal, not a gap: a stale baseline is not something `ahoy install` could close, and surfacing it as resolvable would promise what the apply path cannot deliver. The release gate promotes on the caller's say-so, following ArmReceiptGate's established shape: `abcd docs lint --release-gate` arms lint.ArmCitationOverdue, turning a citation past its threshold from the commit gate's warn into a blocker. The flag is the trust root, so a repo cannot defang its own release by editing the committed config, and an ordinary commit is never calendar-blocked. `abcd launch --dry-run` gains a citation-baseline gate that nags on entries approaching the threshold and refuses on overdue, broken, or unreceipted ones. Its measurement is taken in the CLI and handed in as data because lint imports launch for its semver — the same shape the cobra-tree walk already uses, with the dependency never pointing back. One grading serves all of it: lint.CitationAgeSummary, so the number ahoy shows is by construction the number the release gate acts on. Assisted-by: Claude:claude-opus-5[1m] --- .abcd/development/release/surface.json | 59 ++++++ docs/reference/cli/commands.md | 42 ++++ internal/core/ahoy/citations.go | 70 +++++++ internal/core/ahoy/citations_test.go | 112 +++++++++++ internal/core/ahoy/detect.go | 7 + internal/core/launch/citations.go | 123 ++++++++++++ internal/core/launch/citations_test.go | 104 ++++++++++ internal/core/launch/dryrun.go | 11 +- internal/core/lint/citesummary.go | 148 ++++++++++++++ internal/core/lint/citesummary_test.go | 169 ++++++++++++++++ internal/surface/cli/cite.go | 255 +++++++++++++++++++++++++ internal/surface/cli/cite_test.go | 230 ++++++++++++++++++++++ internal/surface/cli/cli.go | 30 +++ 13 files changed, 1359 insertions(+), 1 deletion(-) create mode 100644 internal/core/ahoy/citations.go create mode 100644 internal/core/ahoy/citations_test.go create mode 100644 internal/core/launch/citations.go create mode 100644 internal/core/launch/citations_test.go create mode 100644 internal/core/lint/citesummary.go create mode 100644 internal/core/lint/citesummary_test.go create mode 100644 internal/surface/cli/cite.go create mode 100644 internal/surface/cli/cite_test.go diff --git a/.abcd/development/release/surface.json b/.abcd/development/release/surface.json index e40c0b29..06491f0b 100644 --- a/.abcd/development/release/surface.json +++ b/.abcd/development/release/surface.json @@ -308,6 +308,58 @@ "hidden": false, "flags": [] }, + { + "path": "abcd docs cite", + "hidden": false, + "flags": [] + }, + { + "path": "abcd docs cite confirm", + "hidden": false, + "flags": [ + { + "name": "config", + "shorthand": "", + "type": "string", + "required": false, + "hidden": false + }, + { + "name": "receipt", + "shorthand": "", + "type": "string", + "required": false, + "hidden": false + }, + { + "name": "root", + "shorthand": "", + "type": "string", + "required": false, + "hidden": false + } + ] + }, + { + "path": "abcd docs cite refresh", + "hidden": false, + "flags": [ + { + "name": "config", + "shorthand": "", + "type": "string", + "required": false, + "hidden": false + }, + { + "name": "root", + "shorthand": "", + "type": "string", + "required": false, + "hidden": false + } + ] + }, { "path": "abcd docs lint", "hidden": false, @@ -319,6 +371,13 @@ "required": false, "hidden": false }, + { + "name": "release-gate", + "shorthand": "", + "type": "bool", + "required": false, + "hidden": false + }, { "name": "root", "shorthand": "", diff --git a/docs/reference/cli/commands.md b/docs/reference/cli/commands.md index 43090109..3c2a846a 100644 --- a/docs/reference/cli/commands.md +++ b/docs/reference/cli/commands.md @@ -223,6 +223,47 @@ Documentation-currency checks for this repo **Usage:** `abcd docs` +#### `abcd docs cite` + +Maintain the citation baseline the docs lint enforces offline + +**Usage:** `abcd docs cite` + +##### `abcd docs cite confirm` + +Record that a human verified a cited URL the fetcher could not read + +**Usage:** `abcd docs cite confirm [url...] [flags]` + +Record that a human verified a cited URL the fetcher could not read. + +Name the URLs directly, or pass --receipt with a receipt file. Both write the same dated manual entry: the baseline records THAT a human confirmed the citation and WHEN, never how. Only URLs the documentation actually cites can be confirmed. + +**Flags:** + +``` + --config string path to docs-lint.json (default: /.abcd/docs-lint.json) + --receipt string path to a receipt file listing the confirmed citations (the format the generated checklist page emits) + --root string repo root (default: current working directory) +``` + +##### `abcd docs cite refresh` + +Fetch every cited URL once and rewrite the committed citation baseline + +**Usage:** `abcd docs cite refresh [flags]` + +Fetch every cited URL once and rewrite the committed citation baseline. + +This is the only abcd verb that reaches the network on behalf of documentation. Each URL gets exactly one bounded attempt; a failure is recorded as an outcome, never retried. Sources that refuse automated fetchers are printed as a manual checklist for `abcd docs cite confirm` rather than recorded as broken. + +**Flags:** + +``` + --config string path to docs-lint.json (default: /.abcd/docs-lint.json) + --root string repo root (default: current working directory) +``` + #### `abcd docs lint` Lint docs for change-narration, broken links, and stray root markdown @@ -233,6 +274,7 @@ Lint docs for change-narration, broken links, and stray root markdown ``` --config string path to docs-lint.json (default: /.abcd/docs-lint.json) + --release-gate run as the release gate: a citation past its staleness threshold blocks instead of warning (release-time only) --root string repo root to lint (default: current working directory) ``` diff --git a/internal/core/ahoy/citations.go b/internal/core/ahoy/citations.go new file mode 100644 index 00000000..9dbcce4a --- /dev/null +++ b/internal/core/ahoy/citations.go @@ -0,0 +1,70 @@ +package ahoy + +// The citation-baseline status line (spc-17's surfacing clause). +// +// The refresh verb is manual, so something has to remind a maintainer that the +// record it writes is ageing — otherwise the first anyone hears of a 400-day-old +// baseline is a release that will not cut. `ahoy` is where that belongs: it is +// the read-only "where am I" board a maintainer already looks at. +// +// It is a SIGNAL, not a gap. A stale baseline is not a discrepancy `ahoy install` +// could close — clearing it means fetching the network, and possibly a human +// opening links — so surfacing it as a resolvable gap would be a promise the +// apply path cannot keep. + +import ( + "path/filepath" + "strconv" + + "github.com/REPPL/abcd-cli/internal/core/lint" +) + +// detectCitations renders the baseline's coverage and age as one line, or "" when +// this repo has not armed the citation gate. +// +// Every failure is soft. This is a read-only status board: a missing config, a +// disabled rule, or a corrupt baseline must leave the rest of `ahoy` renderable, +// so an unreadable record is REPORTED as unreadable rather than raised. +func detectCitations(cwd string) string { + cfg, err := lint.LoadConfig(filepath.Join(cwd, ".abcd", "docs-lint.json")) + if err != nil { + return "" + } + if rc, ok := cfg.Rules["citation_baseline"]; !ok || !rc.Enabled { + return "" + } + + sum, err := lint.CitationAgeSummary(cfg, cwd) + if err != nil { + return "baseline unreadable — run `abcd docs cite refresh`" + } + if sum.Cited == 0 { + return "nothing cited" + } + + line := strconv.Itoa(sum.Cited) + " cited" + if !sum.Present { + return line + ", no baseline — run `abcd docs cite refresh`" + } + if sum.Missing > 0 { + line += ", " + strconv.Itoa(sum.Missing) + " unreceipted" + } + if sum.Broken > 0 { + line += ", " + strconv.Itoa(sum.Broken) + " broken" + } + if sum.Overdue > 0 { + line += ", " + strconv.Itoa(sum.Overdue) + " overdue" + } + // Stale includes the overdue ones; report only the difference, so the two + // numbers add up to what a reader expects rather than double-counting. + if warnOnly := sum.Stale - sum.Overdue; warnOnly > 0 { + line += ", " + strconv.Itoa(warnOnly) + " stale" + } + if sum.Healthy() { + return line + ", all current" + } + if sum.OldestChecked != "" { + line += " (oldest checked " + sum.OldestChecked + ")" + } + return line +} diff --git a/internal/core/ahoy/citations_test.go b/internal/core/ahoy/citations_test.go new file mode 100644 index 00000000..57dd708e --- /dev/null +++ b/internal/core/ahoy/citations_test.go @@ -0,0 +1,112 @@ +package ahoy + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// citationRepo plants a repo with a docs-lint config, one cited page, and +// optionally a baseline. +func citationRepo(t *testing.T, ruleEnabled bool, baseline string) string { + t.Helper() + root := t.TempDir() + if err := os.MkdirAll(filepath.Join(root, ".abcd"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(root, "docs"), 0o755); err != nil { + t.Fatal(err) + } + enabled := "false" + if ruleEnabled { + enabled = "true" + } + cfg := `{"roots": ["docs"], "banned_tokens": [], "rules": { + "citation_baseline": {"enabled": ` + enabled + `, "severity": "blocker", + "baseline": ".abcd/citations-baseline.json", "warn_after_days": 180, "block_after_days": 365}}}` + if err := os.WriteFile(filepath.Join(root, ".abcd", "docs-lint.json"), []byte(cfg), 0o644); err != nil { + t.Fatal(err) + } + page := "# P\n\nA claim.[^a]\n\n[^a]: S, https://example.org/a\n" + if err := os.WriteFile(filepath.Join(root, "docs", "p.md"), []byte(page), 0o644); err != nil { + t.Fatal(err) + } + if baseline != "" { + if err := os.WriteFile(filepath.Join(root, ".abcd", "citations-baseline.json"), []byte(baseline), 0o644); err != nil { + t.Fatal(err) + } + } + return root +} + +// TestCitationSignalReportsCoverageAndAge is the ahoy status line spc-17 asks +// for: the baseline's age and coverage, visible without running the gate. +func TestCitationSignalReportsCoverageAndAge(t *testing.T) { + root := citationRepo(t, true, `{"schema_version": 1, "entries": { + "https://example.org/a": {"final_url": "https://example.org/a", "last_checked": "2020-01-01", + "outcome": "alive", "verification": "automatic", "verified_on": "2020-01-01"}}}`) + + got := detectCitations(root) + if got == "" { + t.Fatal("no citations signal for a repo with the rule armed") + } + if !strings.Contains(got, "1 cited") { + t.Errorf("signal = %q, want the cited count", got) + } + if !strings.Contains(got, "overdue") { + t.Errorf("signal = %q, want the long-stale entry called out as overdue", got) + } +} + +// TestCitationSignalNamesAnUnrefreshedRepo makes the un-refreshed state visible +// rather than silent — a maintainer must be able to see that the record the gate +// enforces does not exist yet. +func TestCitationSignalNamesAnUnrefreshedRepo(t *testing.T) { + got := detectCitations(citationRepo(t, true, "")) + if !strings.Contains(got, "no baseline") { + t.Errorf("signal = %q, want it to name the missing baseline", got) + } +} + +// TestCitationSignalIsSilentWhenTheRuleIsOff keeps the line out of every repo +// that has not adopted the gate. A status board that reports on a rule nobody +// armed is noise. +func TestCitationSignalIsSilentWhenTheRuleIsOff(t *testing.T) { + if got := detectCitations(citationRepo(t, false, "")); got != "" { + t.Errorf("signal = %q, want none when citation_baseline is disabled", got) + } +} + +// TestCitationSignalIsSilentWithoutAConfig keeps Detect total over an ordinary +// folder: no docs-lint config, no line, no error. +func TestCitationSignalIsSilentWithoutAConfig(t *testing.T) { + if got := detectCitations(t.TempDir()); got != "" { + t.Errorf("signal = %q, want none without a docs-lint config", got) + } +} + +// TestCitationSignalSurvivesAMalformedBaseline pins the fail-soft contract for a +// read-only status board: a corrupt record is REPORTED, never a hard error, so +// `ahoy` still renders everything else it knows. +func TestCitationSignalSurvivesAMalformedBaseline(t *testing.T) { + root := citationRepo(t, true, `{"schema_version": 1, "entries": {"x": {"method": "curl"}}}`) + got := detectCitations(root) + if !strings.Contains(got, "unreadable") { + t.Errorf("signal = %q, want it to report the unreadable baseline", got) + } +} + +// TestDetectSurfacesTheCitationSignal wires the check into the envelope both +// front doors read. +func TestDetectSurfacesTheCitationSignal(t *testing.T) { + setupHermetic(t) + root := citationRepo(t, true, "") + det, err := Detect(root) + if err != nil { + t.Fatalf("Detect: %v", err) + } + if s, _ := det.Signals["citations"].(string); s == "" { + t.Fatalf("signals = %+v, want a citations entry", det.Signals) + } +} diff --git a/internal/core/ahoy/detect.go b/internal/core/ahoy/detect.go index e0fdaeda..4625eff2 100644 --- a/internal/core/ahoy/detect.go +++ b/internal/core/ahoy/detect.go @@ -50,6 +50,13 @@ func Detect(cwd string) (DetectionResult, error) { // kind, surfaced so status reports "dev (tip build)" honestly (never invisible). signals["install_mode"] = detectInstallMode(pluginRoot, pluginOK) + // The citation baseline's coverage and age, when this repo has armed the + // citation gate (spc-17). Omitted entirely otherwise, so a repo that has not + // adopted the gate carries no line about it. + if citations := detectCitations(abs); citations != "" { + signals["citations"] = citations + } + res := DetectionResult{ FolderKind: kind, RootSHA: identity.RootSHA, diff --git a/internal/core/launch/citations.go b/internal/core/launch/citations.go new file mode 100644 index 00000000..a6c041f0 --- /dev/null +++ b/internal/core/launch/citations.go @@ -0,0 +1,123 @@ +package launch + +// The citation-baseline release gate (spc-17's surfacing clause). +// +// The refresh verb is manual. spc-17's answer to "then who remembers to run it" +// is nagging at the two places a maintainer already looks: `ahoy` for the daily +// glance, and the release preflight for the moment it actually matters. This is +// the second. +// +// The measurement is NOT taken here. Grading the baseline needs the docs-lint +// engine, and internal/core/lint imports this package for its semver — so a call +// the other way would be an import cycle. The front door that already holds both +// takes the summary and hands it in as data, the same shape surface/cli uses to +// feed the cobra command tree into core/surface. The dependency never points +// back. + +import "strconv" + +// CitationPreflight is the citation baseline's state, measured by the caller. +// A nil pointer means the repo has not armed the citation gate, and the gate +// reports itself as not run rather than silently passing. +type CitationPreflight struct { + // Present reports whether a baseline exists at all. + Present bool + // Cited, Recorded and Missing describe coverage. + Cited int + Recorded int + Missing int + // Broken counts citations recorded as dead. + Broken int + // Approaching counts entries inside the nag window before the blocking + // threshold; Overdue counts those past it. + Approaching int + Overdue int + // ApproachingURLs and OverdueURLs name them, so a nag and a refusal tell an + // operator WHICH links to deal with rather than only how many. + ApproachingURLs []string + OverdueURLs []string +} + +// citationGate renders the gate's dry-run disposition. +func citationGate(pre *CitationPreflight) GateSummary { + if pre == nil { + return GateSummary{ + Name: "citation-baseline", + Status: "not_implemented", + Detail: "citation_baseline is not armed in this repo's docs-lint config", + } + } + if !pre.Present && pre.Cited > 0 { + return GateSummary{Name: "citation-baseline", Status: "ran", + Detail: strconv.Itoa(pre.Cited) + " cited, no baseline — run `abcd docs cite refresh`"} + } + + detail := strconv.Itoa(pre.Cited) + " cited, " + strconv.Itoa(pre.Recorded) + " with receipts" + if pre.Missing > 0 { + detail += ", " + strconv.Itoa(pre.Missing) + " unreceipted" + } + if pre.Broken > 0 { + detail += ", " + strconv.Itoa(pre.Broken) + " broken" + } + if pre.Overdue > 0 { + detail += ", " + strconv.Itoa(pre.Overdue) + " overdue" + } + // The nag: named here, in the informational half, precisely because it does + // NOT refuse. A maintainer gets a month's notice while releases still cut. + if pre.Approaching > 0 { + detail += ", " + strconv.Itoa(pre.Approaching) + " approaching the staleness blocker (" + + joinURLs(pre.ApproachingURLs) + ")" + } + return GateSummary{Name: "citation-baseline", Status: "ran", Detail: detail} +} + +// citationRefusals lists what the citation baseline would stop a release on. +// Approaching entries are deliberately absent: they nag, they do not block. +func citationRefusals(pre *CitationPreflight) []string { + if pre == nil { + return nil + } + var out []string + if !pre.Present && pre.Cited > 0 { + return []string{"citation baseline: absent, but " + strconv.Itoa(pre.Cited) + + " citation(s) need one — run `abcd docs cite refresh`"} + } + if pre.Missing > 0 { + out = append(out, "citation baseline: "+strconv.Itoa(pre.Missing)+ + " cited URL(s) have no receipt — run `abcd docs cite refresh`") + } + if pre.Broken > 0 { + out = append(out, "citation baseline: "+strconv.Itoa(pre.Broken)+ + " citation(s) recorded broken — fix or replace the source") + } + if pre.Overdue > 0 { + out = append(out, "citation baseline: "+strconv.Itoa(pre.Overdue)+ + " citation(s) past the staleness threshold ("+joinURLs(pre.OverdueURLs)+ + ") — run `abcd docs cite refresh`") + } + return out +} + +// joinURLs renders a bounded list. A release blocked by forty stale citations +// must not print forty lines into a gate summary; the count above it is the +// complete number, and `abcd docs lint` names every one. +func joinURLs(urls []string) string { + const max = 3 + if len(urls) == 0 { + return "none" + } + shown := urls + suffix := "" + if len(urls) > max { + shown = urls[:max] + suffix = ", and " + strconv.Itoa(len(urls)-max) + " more" + } + out := "" + for i, u := range shown { + if i > 0 { + out += ", " + } + out += u + } + return out + suffix +} diff --git a/internal/core/launch/citations_test.go b/internal/core/launch/citations_test.go new file mode 100644 index 00000000..5c60520c --- /dev/null +++ b/internal/core/launch/citations_test.go @@ -0,0 +1,104 @@ +package launch + +import ( + "strings" + "testing" +) + +func runCitationGate(t *testing.T, pre *CitationPreflight) (GateSummary, []string) { + t.Helper() + repo := t.TempDir() + writeFile(t, repo, ".abcd/config/launch-payload.json", `{"includes": ["commands"]}`) + writeFile(t, repo, "commands/x.md", "# doc\n") + report, err := DryRun(DryRunRequest{RepoRoot: repo, Version: "v0.1.0", Citations: pre}) + if err != nil { + t.Fatalf("DryRun: %v", err) + } + for _, g := range report.Gates { + if g.Name == "citation-baseline" { + return g, report.WouldRefuseOn + } + } + t.Fatalf("no citation-baseline gate in %+v", report.Gates) + return GateSummary{}, nil +} + +// TestCitationGateIsNotImplementedWithoutAPreflight keeps the gate honest in a +// repo that has not armed the citation rule: it is reported as not run, never as +// a silent pass. +func TestCitationGateIsNotImplementedWithoutAPreflight(t *testing.T) { + gate, refusals := runCitationGate(t, nil) + if gate.Status != "not_implemented" { + t.Fatalf("status = %q, want not_implemented", gate.Status) + } + for _, r := range refusals { + if strings.Contains(r, "citation") { + t.Fatalf("an unarmed citation gate produced a refusal: %q", r) + } + } +} + +// TestCitationGateNagsWithoutBlocking is spc-17's nag: entries approaching the +// blocking threshold are named in the gate detail and refuse nothing, so a +// maintainer gets notice while a release still cuts. +func TestCitationGateNagsWithoutBlocking(t *testing.T) { + gate, refusals := runCitationGate(t, &CitationPreflight{ + Present: true, Cited: 12, Recorded: 12, Approaching: 2, ApproachingURLs: []string{"https://a.example.org/x", "https://b.example.org/y"}, + }) + if gate.Status != "ran" { + t.Fatalf("status = %q, want ran", gate.Status) + } + if !strings.Contains(gate.Detail, "2 approaching") { + t.Errorf("detail = %q, want the approaching count", gate.Detail) + } + for _, r := range refusals { + if strings.Contains(r, "citation") { + t.Fatalf("a nag blocked the release: %q", r) + } + } +} + +// TestCitationGateBlocksOnOverdue is AC 3's release-gate half: past the blocking +// threshold, the release stops, and the refusal NAMES the citations so the +// operator knows what to refresh. +func TestCitationGateBlocksOnOverdue(t *testing.T) { + gate, refusals := runCitationGate(t, &CitationPreflight{ + Present: true, Cited: 12, Recorded: 12, Overdue: 1, OverdueURLs: []string{"https://a.example.org/x"}, + }) + if gate.Status != "ran" { + t.Fatalf("status = %q, want ran", gate.Status) + } + var named bool + for _, r := range refusals { + if strings.Contains(r, "https://a.example.org/x") { + named = true + } + } + if !named { + t.Fatalf("refusals = %v, want the overdue citation named", refusals) + } +} + +// TestCitationGateBlocksOnBrokenAndUnreceipted covers the other two ways the +// record fails a release: a citation recorded broken, and one with no receipt. +func TestCitationGateBlocksOnBrokenAndUnreceipted(t *testing.T) { + _, refusals := runCitationGate(t, &CitationPreflight{Present: true, Cited: 3, Recorded: 1, Broken: 1, Missing: 2}) + joined := strings.Join(refusals, "\n") + if !strings.Contains(joined, "broken") || !strings.Contains(joined, "receipt") { + t.Fatalf("refusals = %v, want both the broken and the unreceipted citations to refuse", refusals) + } +} + +// TestCitationGatePassesOnAHealthyBaseline pins that a current record refuses +// nothing at all. +func TestCitationGatePassesOnAHealthyBaseline(t *testing.T) { + gate, refusals := runCitationGate(t, &CitationPreflight{Cited: 12, Recorded: 12, Present: true}) + if !strings.Contains(gate.Detail, "12") { + t.Errorf("detail = %q, want the cited count", gate.Detail) + } + for _, r := range refusals { + if strings.Contains(r, "citation") { + t.Fatalf("a healthy baseline refused: %q", r) + } + } +} diff --git a/internal/core/launch/dryrun.go b/internal/core/launch/dryrun.go index 623de2ff..3d13feea 100644 --- a/internal/core/launch/dryrun.go +++ b/internal/core/launch/dryrun.go @@ -19,6 +19,12 @@ type DryRunRequest struct { // it means the caller could not name one, and retention says so. Version string ExistingTags []Semver // injected; nil → default `git tag -l v*` provider + // Citations is the citation baseline's state, MEASURED by the caller. + // Grading it needs internal/core/lint, which imports this package for its + // semver — so the front door that holds both hands the result in as data + // rather than this package reaching back. Nil means the repo has not armed + // the citation gate. + Citations *CitationPreflight } // GateSummary records one gate's dry-run disposition. @@ -81,9 +87,12 @@ func DryRun(req DryRunRequest) (DryRunReport, error) { {Name: "marker-block", Status: "not_implemented", Detail: "Phase-5 deferred"}, {Name: "installability-smoke", Status: "ran", Detail: smokeDetail(smoke)}, {Name: "documentation-auditor", Status: "not_implemented", Detail: "Phase-5 deferred"}, + citationGate(req.Citations), } - report.WouldRefuseOn = wouldRefuseOn(bundle, scan, lockstep, report.Retention, smoke) + report.WouldRefuseOn = append( + wouldRefuseOn(bundle, scan, lockstep, report.Retention, smoke), + citationRefusals(req.Citations)...) report.WouldPublish = false return report, nil } diff --git a/internal/core/lint/citesummary.go b/internal/core/lint/citesummary.go new file mode 100644 index 00000000..5d874506 --- /dev/null +++ b/internal/core/lint/citesummary.go @@ -0,0 +1,148 @@ +package lint + +// The citation baseline's health, and the release gate's promotion of it. +// +// Two surfaces need to say something about the baseline without running the +// whole lint: `abcd ahoy` reports its age at a glance, and the release preflight +// nags about entries approaching the 365-day blocker. Both read the summary +// below — one grading, so the number a maintainer sees at ahoy is by +// construction the number the release gate acts on. A second count computed +// slightly differently is how a status line starts lying. + +import ( + "errors" + "os" + "path/filepath" + "sort" + "time" +) + +// CitationNagWindowDays is how far ahead of the blocking threshold the release +// preflight starts naming entries. A month is enough notice to clear a manual +// queue before a release, and short enough that the nag still means something. +const CitationNagWindowDays = 30 + +// CitationSummary is the committed baseline's state against the repo's current +// citations. Every count is over the CITED set: an entry for a URL the docs no +// longer cite is not a health problem, it is something the next refresh drops. +type CitationSummary struct { + // Present reports whether a baseline file exists at all. + Present bool `json:"present"` + // Cited is how many distinct URLs the documentation cites. + Cited int `json:"cited"` + // Recorded is how many of those have a baseline entry; Missing is the rest. + Recorded int `json:"recorded"` + Missing int `json:"missing"` + // Broken counts entries recorded as broken. + Broken int `json:"broken"` + // Manual counts entries a human established. They age on the same clock as + // automatic ones, so this is reported, never subtracted from the stale count. + Manual int `json:"manual"` + // Stale counts entries past the warn threshold (which includes every + // approaching and overdue one); Overdue counts those past the blocking one; + // Approaching counts the band just short of blocking. + Stale int `json:"stale"` + Overdue int `json:"overdue"` + Approaching int `json:"approaching"` + // OldestChecked is the earliest last_checked date among cited entries. + OldestChecked string `json:"oldest_checked,omitempty"` + // OverdueURLs and ApproachingURLs name what a nag must list, sorted. + OverdueURLs []string `json:"overdue_urls,omitempty"` + ApproachingURLs []string `json:"approaching_urls,omitempty"` +} + +// Healthy reports whether the baseline needs no attention: every citation has a +// current, unbroken receipt. +func (s CitationSummary) Healthy() bool { + return s.Present && s.Missing == 0 && s.Broken == 0 && s.Stale == 0 +} + +// CitationAgeSummary grades the baseline against the wall clock. +func CitationAgeSummary(cfg Config, repoRoot string) (CitationSummary, error) { + return CitationAgeSummaryAt(cfg, repoRoot, time.Now()) +} + +// CitationAgeSummaryAt is CitationAgeSummary with the clock supplied, so the +// band boundaries are testable exactly. +// +// A missing baseline is reported, not an error: a repo that has not refreshed +// yet is a state a status line must be able to describe. A MALFORMED one is +// still an error — a record the gate cannot read is not a health report. +func CitationAgeSummaryAt(cfg Config, repoRoot string, now time.Time) (CitationSummary, error) { + relPath, warnDays, blockDays := CitationPolicy(cfg) + + cited, err := CollectCitedURLs(cfg, repoRoot) + if err != nil { + return CitationSummary{}, err + } + sum := CitationSummary{Cited: len(cited)} + + base, err := LoadBaseline(filepath.Join(repoRoot, filepath.FromSlash(relPath))) + if err != nil { + if errors.Is(err, os.ErrNotExist) { + sum.Missing = len(cited) + return sum, nil + } + return CitationSummary{}, err + } + sum.Present = true + + for _, ref := range cited { + e, ok := base.Entries[ref.URL] + if !ok { + sum.Missing++ + continue + } + sum.Recorded++ + if e.Outcome == OutcomeBroken { + sum.Broken++ + } + if e.Verification == VerificationManual { + sum.Manual++ + } + if sum.OldestChecked == "" || e.LastChecked < sum.OldestChecked { + sum.OldestChecked = e.LastChecked + } + checked, perr := e.Checked() + if perr != nil { + continue // unreachable: LoadBaseline validated the date + } + age := daysBetween(checked, now) + if age >= warnDays { + sum.Stale++ + } + switch { + case age >= blockDays: + sum.Overdue++ + sum.OverdueURLs = append(sum.OverdueURLs, ref.URL) + case age >= blockDays-CitationNagWindowDays: + sum.Approaching++ + sum.ApproachingURLs = append(sum.ApproachingURLs, ref.URL) + } + } + sort.Strings(sum.OverdueURLs) + sort.Strings(sum.ApproachingURLs) + return sum, nil +} + +// ArmCitationOverdue returns cfg with the citation staleness blocker armed: an +// entry past the block threshold becomes a BLOCKER finding rather than the warn +// the commit gate emits. +// +// It follows ArmReceiptGate's shape deliberately. The promotion is the CALLER's +// decision — a release gate, not the in-tree config — so a repo cannot defang +// its own release by writing "overdue_severity": "warn" into a committed file, +// and an ordinary commit is never calendar-blocked (spc-17). What arming does +// NOT do is enable a rule the repo turned off: promoting a severity is the +// release gate's business, adopting a rule is the repo's. +func ArmCitationOverdue(cfg Config) Config { + rules := make(map[string]RuleConfig, len(cfg.Rules)) + for k, v := range cfg.Rules { + rules[k] = v + } + rc := rules[ruleCitationBaseline] + rc.OverdueSeverity = severityBlocker + rules[ruleCitationBaseline] = rc + cfg.Rules = rules + return cfg +} diff --git a/internal/core/lint/citesummary_test.go b/internal/core/lint/citesummary_test.go new file mode 100644 index 00000000..a8874981 --- /dev/null +++ b/internal/core/lint/citesummary_test.go @@ -0,0 +1,169 @@ +package lint + +import ( + "os" + "path/filepath" + "testing" +) + +// summaryRepo plants a repo citing the given URLs with a baseline whose entries +// carry the given last-checked dates. +func summaryRepo(t *testing.T, entries map[string]BaselineEntry, cited ...string) string { + t.Helper() + files := map[string]string{} + for i, u := range cited { + files["docs/p"+string(rune('a'+i))+".md"] = "# P\n\nA claim.[^a]\n\n[^a]: S, " + u + "\n" + } + root := citeRepo(t, files) + if entries != nil { + path := filepath.Join(root, filepath.FromSlash(DefaultBaselinePath)) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := SaveBaseline(path, Baseline{SchemaVersion: 1, Entries: entries}); err != nil { + t.Fatal(err) + } + } + return root +} + +func summaryConfig() Config { + return Config{Roots: []string{"docs"}, Rules: map[string]RuleConfig{ + ruleCitationBaseline: {Enabled: true, Severity: severityBlocker, WarnAfterDays: 180, BlockAfterDays: 365}, + }} +} + +func alive(lastChecked, verification string) BaselineEntry { + return BaselineEntry{FinalURL: "x", LastChecked: lastChecked, Outcome: OutcomeAlive, + Verification: verification, VerifiedOn: lastChecked} +} + +// TestCitationAgeSummaryGradesEveryBand is the one primitive the ahoy status line +// and the release nag both read. Grading it in one place is what keeps the number +// a maintainer sees at `ahoy` identical to the one the release gate acts on. +func TestCitationAgeSummaryGradesEveryBand(t *testing.T) { + root := summaryRepo(t, map[string]BaselineEntry{ + // fixedNow is 2026-07-27. + "https://e.org/fresh": alive("2026-07-01", VerificationAutomatic), // 26 days + "https://e.org/stale": alive("2025-12-01", VerificationAutomatic), // 238 days: past warn + "https://e.org/approaching": alive("2025-08-05", VerificationManual), // 356 days: inside the nag window + "https://e.org/overdue": alive("2025-01-01", VerificationAutomatic), // 572 days: past block + "https://e.org/broken": {FinalURL: "x", LastChecked: "2026-07-01", Outcome: OutcomeBroken, + Verification: VerificationAutomatic, VerifiedOn: "2026-07-01"}, + }, + "https://e.org/fresh", "https://e.org/stale", "https://e.org/approaching", + "https://e.org/overdue", "https://e.org/broken", "https://e.org/unrecorded", + ) + + got, err := CitationAgeSummaryAt(summaryConfig(), root, fixedNow) + if err != nil { + t.Fatalf("CitationAgeSummaryAt: %v", err) + } + if !got.Present { + t.Fatal("Present = false, want true") + } + if got.Cited != 6 { + t.Errorf("Cited = %d, want 6", got.Cited) + } + if got.Recorded != 5 { + t.Errorf("Recorded = %d, want 5", got.Recorded) + } + if got.Missing != 1 { + t.Errorf("Missing = %d, want 1", got.Missing) + } + if got.Broken != 1 { + t.Errorf("Broken = %d, want 1", got.Broken) + } + if got.Manual != 1 { + t.Errorf("Manual = %d, want 1 (a manual receipt is counted, and ages the same)", got.Manual) + } + // stale counts everything past the warn line, overdue everything past the + // block line; approaching is the nag band just short of it. + if got.Stale != 3 { + t.Errorf("Stale = %d, want 3 (stale + approaching + overdue are all past warn)", got.Stale) + } + if got.Overdue != 1 { + t.Errorf("Overdue = %d, want 1", got.Overdue) + } + if got.Approaching != 1 { + t.Errorf("Approaching = %d, want 1", got.Approaching) + } + if len(got.OverdueURLs) != 1 || got.OverdueURLs[0] != "https://e.org/overdue" { + t.Errorf("OverdueURLs = %v, want the overdue citation", got.OverdueURLs) + } + if len(got.ApproachingURLs) != 1 || got.ApproachingURLs[0] != "https://e.org/approaching" { + t.Errorf("ApproachingURLs = %v, want the approaching citation", got.ApproachingURLs) + } + if got.OldestChecked != "2025-01-01" { + t.Errorf("OldestChecked = %q, want 2025-01-01", got.OldestChecked) + } +} + +// TestCitationAgeSummaryWithNoBaseline reports the un-refreshed repo honestly +// rather than erroring, so a status line can say "no baseline yet". +func TestCitationAgeSummaryWithNoBaseline(t *testing.T) { + root := summaryRepo(t, nil, "https://e.org/a") + got, err := CitationAgeSummaryAt(summaryConfig(), root, fixedNow) + if err != nil { + t.Fatalf("CitationAgeSummaryAt: %v", err) + } + if got.Present { + t.Error("Present = true, want false") + } + if got.Cited != 1 || got.Missing != 1 { + t.Errorf("Cited=%d Missing=%d, want 1/1", got.Cited, got.Missing) + } +} + +// TestArmCitationOverdue is the release-gate promotion, built on the shape +// ArmReceiptGate established: the CALLER is the trust root, so a downgraded +// in-tree config cannot defang the gate at release time. +func TestArmCitationOverdue(t *testing.T) { + cfg := Config{Rules: map[string]RuleConfig{ + ruleCitationBaseline: {Enabled: true, Severity: severityBlocker, OverdueSeverity: severityWarn}, + }} + armed := ArmCitationOverdue(cfg) + + if got := armed.Rules[ruleCitationBaseline].OverdueSeverity; got != severityBlocker { + t.Fatalf("overdue severity = %q, want %q", got, severityBlocker) + } + // The input config is not mutated: a caller may lint twice, once each way. + if got := cfg.Rules[ruleCitationBaseline].OverdueSeverity; got != severityWarn { + t.Fatalf("arming mutated the caller's config: overdue severity = %q", got) + } + // Arming does not enable a rule the repo turned off — the release gate + // promotes a severity, it does not adopt a rule on the repo's behalf. + off := ArmCitationOverdue(Config{Rules: map[string]RuleConfig{ + ruleCitationBaseline: {Enabled: false}, + }}) + if off.Rules[ruleCitationBaseline].Enabled { + t.Fatal("arming enabled a rule the repo disabled") + } +} + +// TestArmCitationOverdueBlocksAtTheReleaseGate is the end-to-end proof: the same +// tree that only warns for a commit blocks once armed. +func TestArmCitationOverdueBlocksAtTheReleaseGate(t *testing.T) { + root := summaryRepo(t, map[string]BaselineEntry{ + "https://e.org/old": alive("2025-01-01", VerificationAutomatic), + }, "https://e.org/old") + + commit, err := LintAt(summaryConfig(), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + for _, f := range findingsFor(commit, ruleCitationBaselineOverdu) { + if f.Severity != severityWarn { + t.Fatalf("commit gate severity = %q, want %q — commits are never calendar-blocked", f.Severity, severityWarn) + } + } + + release, err := LintAt(ArmCitationOverdue(summaryConfig()), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + armed := findingsFor(release, ruleCitationBaselineOverdu) + if len(armed) != 1 || armed[0].Severity != severityBlocker { + t.Fatalf("release gate findings = %+v, want one blocker", armed) + } +} diff --git a/internal/surface/cli/cite.go b/internal/surface/cli/cite.go new file mode 100644 index 00000000..bcd5d3e1 --- /dev/null +++ b/internal/surface/cli/cite.go @@ -0,0 +1,255 @@ +package cli + +// The `abcd docs cite` sub-tree: the two verbs that maintain the citation +// baseline. +// +// `refresh` is the only place abcd dials out on behalf of documentation, and it +// runs when a maintainer asks. `confirm` closes the manual queue that refresh +// prints. Both live under `docs` because the baseline exists to serve +// `docs lint` — the gate is the customer, these verbs are how it gets fed. + +import ( + "errors" + "fmt" + "io" + "os" + "path/filepath" + "strings" + + "github.com/spf13/cobra" + + "github.com/REPPL/abcd-cli/internal/core/cite" + "github.com/REPPL/abcd-cli/internal/core/launch" + "github.com/REPPL/abcd-cli/internal/core/lint" + "github.com/REPPL/abcd-cli/internal/termsafe" +) + +// newCiteCommand builds the `docs cite` sub-tree. configPath and rootDir are the +// docs command's own flag targets, so `cite` resolves its config exactly the way +// `lint` does and the two verbs can never read different roots. +func newCiteCommand(asJSON *bool) *cobra.Command { + citeCmd := &cobra.Command{ + Use: "cite", + Short: "Maintain the citation baseline the docs lint enforces offline", + Args: cobra.NoArgs, + } + citeCmd.AddCommand(newCiteRefreshCommand(asJSON)) + citeCmd.AddCommand(newCiteConfirmCommand(asJSON)) + return citeCmd +} + +func newCiteRefreshCommand(asJSON *bool) *cobra.Command { + var configPath, rootDir string + cmd := &cobra.Command{ + Use: "refresh", + Short: "Fetch every cited URL once and rewrite the committed citation baseline", + Long: "Fetch every cited URL once and rewrite the committed citation baseline.\n\n" + + "This is the only abcd verb that reaches the network on behalf of documentation. " + + "Each URL gets exactly one bounded attempt; a failure is recorded as an outcome, " + + "never retried. Sources that refuse automated fetchers are printed as a manual " + + "checklist for `abcd docs cite confirm` rather than recorded as broken.", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + root, cfg, err := loadCiteConfig("docs cite refresh", rootDir, configPath) + if err != nil { + return err + } + res, err := cite.Refresh(cite.RefreshRequest{RepoRoot: root, Config: cfg}) + if err != nil { + return &exitError{Code: 2, Msg: "docs cite refresh: " + err.Error()} + } + return render(cmd.OutOrStdout(), *asJSON, res, func(w io.Writer) { + renderRefresh(w, res) + }) + }, + } + addCiteFlags(cmd, &configPath, &rootDir) + return cmd +} + +// renderRefresh prints the run as a transcript an operator can act on: the +// counts, then every broken link, then the manual queue as a checklist. URLs and +// details come from repo content and third-party servers, so every one is +// sanitised before it reaches a terminal. +func renderRefresh(w io.Writer, res cite.RefreshResult) { + ok, broken, blocked, preserved := res.Counts() + fmt.Fprintf(w, "abcd docs cite refresh — %d cited, %d checked, %d preserved\n", + res.Cited, res.Fetched, preserved) + fmt.Fprintf(w, " alive: %d\n", ok) + fmt.Fprintf(w, " broken: %d\n", broken) + fmt.Fprintf(w, " blocked: %d\n", blocked) + + for _, o := range res.Outcomes { + if o.Status != cite.StatusBroken { + continue + } + fmt.Fprintf(w, " BROKEN %s — %s\n", termsafe.Sanitize(o.URL), termsafe.Sanitize(o.Detail)) + } + for _, u := range res.Dropped { + fmt.Fprintf(w, " dropped (no longer cited): %s\n", termsafe.Sanitize(u)) + } + + if len(res.Queue) > 0 { + fmt.Fprintf(w, "\nmanual queue — %d source(s) refuse automated fetching.\n", len(res.Queue)) + fmt.Fprintf(w, "Open each, confirm the document is there, then record it:\n\n") + for _, q := range res.Queue { + fmt.Fprintf(w, " [ ] %s\n", termsafe.Sanitize(q.URL)) + if q.Detail != "" { + fmt.Fprintf(w, " %s\n", termsafe.Sanitize(q.Detail)) + } + for _, s := range q.Sites { + fmt.Fprintf(w, " cited at %s:%d\n", termsafe.Sanitize(s.File), s.Line) + } + } + fmt.Fprintf(w, "\n abcd docs cite confirm [...]\n") + } + fmt.Fprintf(w, "\nbaseline written: %s\n", res.BaselinePath) +} + +func newCiteConfirmCommand(asJSON *bool) *cobra.Command { + var configPath, rootDir, receiptPath string + cmd := &cobra.Command{ + Use: "confirm [url...]", + Short: "Record that a human verified a cited URL the fetcher could not read", + Long: "Record that a human verified a cited URL the fetcher could not read.\n\n" + + "Name the URLs directly, or pass --receipt with a receipt file. Both write the " + + "same dated manual entry: the baseline records THAT a human confirmed the " + + "citation and WHEN, never how. Only URLs the documentation actually cites can " + + "be confirmed.", + Args: cobra.ArbitraryArgs, + RunE: func(cmd *cobra.Command, args []string) error { + root, cfg, err := loadCiteConfig("docs cite confirm", rootDir, configPath) + if err != nil { + return err + } + + // The two producers of one schema: positional arguments assemble the + // simple case in memory, a file carries the full form. Neither is a + // second pathway — both become the same Receipt before anything is + // validated or written. + var receipt cite.Receipt + switch { + case receiptPath != "" && len(args) > 0: + return &exitError{Code: 2, Msg: "docs cite confirm: pass URLs or --receipt, not both"} + case receiptPath != "": + receipt, err = cite.LoadReceipt(receiptPath) + if err != nil { + return &exitError{Code: 2, Msg: "docs cite confirm: " + citeErrorDetail(err, receiptPath)} + } + case len(args) > 0: + receipt.SchemaVersion = cite.ReceiptSchemaVersion + for _, u := range args { + receipt.Confirmed = append(receipt.Confirmed, cite.ConfirmedCitation{URL: u}) + } + default: + return &exitError{Code: 2, Msg: "docs cite confirm: name at least one URL, or pass --receipt "} + } + + res, err := cite.Confirm(cite.ConfirmRequest{RepoRoot: root, Config: cfg, Receipt: receipt}) + if err != nil { + return &exitError{Code: 2, Msg: "docs cite confirm: " + err.Error()} + } + return render(cmd.OutOrStdout(), *asJSON, res, func(w io.Writer) { + for _, u := range res.Recorded { + fmt.Fprintf(w, " confirmed %s\n", termsafe.Sanitize(u)) + } + fmt.Fprintf(w, "abcd docs cite confirm — %d receipt(s) recorded in %s\n", + len(res.Recorded), res.BaselinePath) + }) + }, + } + addCiteFlags(cmd, &configPath, &rootDir) + cmd.Flags().StringVar(&receiptPath, "receipt", "", + "path to a receipt file listing the confirmed citations (the format the generated checklist page emits)") + return cmd +} + +func addCiteFlags(cmd *cobra.Command, configPath, rootDir *string) { + cmd.Flags().StringVar(configPath, "config", "", "path to docs-lint.json (default: /.abcd/docs-lint.json)") + cmd.Flags().StringVar(rootDir, "root", "", "repo root (default: current working directory)") +} + +// loadCiteConfig resolves the root and the docs-lint config the same way +// `docs lint` does, so the refresh fetches exactly the set the gate enforces. +func loadCiteConfig(verb, rootDir, configPath string) (string, lint.Config, error) { + root := rootDir + if root == "" { + cwd, err := os.Getwd() + if err != nil { + return "", lint.Config{}, err + } + root = cwd + } + root, err := filepath.Abs(root) + if err != nil { + return "", lint.Config{}, err + } + cfgPath := configPath + if cfgPath == "" { + cfgPath = filepath.Join(root, ".abcd", "docs-lint.json") + } + cfg, err := lint.LoadConfig(cfgPath) + if err != nil { + ref := filepath.Join(".abcd", "docs-lint.json") + if configPath != "" { + ref = configPath + } + if os.IsNotExist(err) { + return "", lint.Config{}, &exitError{Code: 2, Msg: fmt.Sprintf( + "%s: config not found at %s — run in a prepared repo or pass --config", verb, ref)} + } + return "", lint.Config{}, &exitError{Code: 2, Msg: fmt.Sprintf( + "%s: cannot read config %s: %s", verb, ref, citeErrorDetail(err, ref))} + } + return root, cfg, nil +} + +// citationPreflight measures the citation baseline for the release preflight. +// +// It lives on this side of the boundary because internal/core/lint imports +// internal/core/launch for its semver, so the measurement cannot be taken inside +// the gate it feeds. Everything is soft: a repo with no docs-lint config, or one +// that has not armed the citation rule, gets a nil preflight and the gate +// reports itself as not run rather than inventing a pass. +func citationPreflight(repoRoot string) *launch.CitationPreflight { + cfg, err := lint.LoadConfig(filepath.Join(repoRoot, ".abcd", "docs-lint.json")) + if err != nil { + return nil + } + if rc, ok := cfg.Rules["citation_baseline"]; !ok || !rc.Enabled { + return nil + } + sum, err := lint.CitationAgeSummary(cfg, repoRoot) + if err != nil { + // An unreadable baseline is not "no gate" — it is a record a release must + // not proceed on, so it is reported as a present-but-empty one whose + // citations all lack receipts, which refuses. + cited, cerr := lint.CollectCitedURLs(cfg, repoRoot) + if cerr != nil { + return nil + } + return &launch.CitationPreflight{Present: true, Cited: len(cited), Missing: len(cited)} + } + return &launch.CitationPreflight{ + Present: sum.Present, + Cited: sum.Cited, + Recorded: sum.Recorded, + Missing: sum.Missing, + Broken: sum.Broken, + Approaching: sum.Approaching, + Overdue: sum.Overdue, + ApproachingURLs: sum.ApproachingURLs, + OverdueURLs: sum.OverdueURLs, + } +} + +// citeErrorDetail strips the path-bearing wrapper off a filesystem error. A +// *PathError embeds the ABSOLUTE path, which must never reach machine output +// (iss-29); its inner Err is the bare cause. +func citeErrorDetail(err error, ref string) string { + var pe *os.PathError + if errors.As(err, &pe) { + return ref + ": " + pe.Err.Error() + } + return strings.TrimSpace(err.Error()) +} diff --git a/internal/surface/cli/cite_test.go b/internal/surface/cli/cite_test.go new file mode 100644 index 00000000..c9733582 --- /dev/null +++ b/internal/surface/cli/cite_test.go @@ -0,0 +1,230 @@ +package cli + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" +) + +// citeLintConfig arms the citation-baseline rule over docs/, which is what makes +// the refresh verb's roots and baseline path resolve the way the gate's do. +const citeLintConfig = `{ + "roots": ["docs"], + "banned_tokens": [], + "rules": { + "citation_baseline": {"enabled": true, "severity": "blocker", "baseline": ".abcd/citations-baseline.json", "warn_after_days": 180, "block_after_days": 365} + } +}` + +// citeRepo plants a repo with a docs-lint config and one page per cited URL. +func citeRepo(t *testing.T, urls ...string) (root, cfg string) { + t.Helper() + root = t.TempDir() + cfg = filepath.Join(root, "docs-lint.json") + if err := os.WriteFile(cfg, []byte(citeLintConfig), 0o644); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(root, "docs"), 0o755); err != nil { + t.Fatal(err) + } + for i, u := range urls { + body := "# Page\n\nA claim.[^a]\n\n[^a]: A source, " + u + "\n" + if err := os.WriteFile(filepath.Join(root, "docs", "p"+string(rune('a'+i))+".md"), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + return root, cfg +} + +func readBaselineJSON(t *testing.T, root string) map[string]any { + t.Helper() + data, err := os.ReadFile(filepath.Join(root, ".abcd", "citations-baseline.json")) + if err != nil { + t.Fatalf("baseline not written: %v", err) + } + var doc map[string]any + if err := json.Unmarshal(data, &doc); err != nil { + t.Fatalf("baseline is not JSON: %v", err) + } + return doc +} + +// TestDocsCiteRefreshWritesABaselineForACitelessRepo proves the verb is wired +// end to end — config load, collector, writer — with nothing to fetch. +func TestDocsCiteRefreshWritesABaselineForACitelessRepo(t *testing.T) { + root, cfg := citeRepo(t) + if err := os.WriteFile(filepath.Join(root, "docs", "plain.md"), []byte("# Plain\n\nNo citations.\n"), 0o644); err != nil { + t.Fatal(err) + } + + out, err := runCLIErr(t, "docs", "cite", "refresh", "--config", cfg, "--root", root) + if err != nil { + t.Fatalf("refresh failed: %v\n%s", err, out) + } + doc := readBaselineJSON(t, root) + if doc["schema_version"] != float64(1) { + t.Fatalf("schema_version = %v, want 1", doc["schema_version"]) + } + if !strings.Contains(string(out), ".abcd/citations-baseline.json") { + t.Errorf("the transcript does not name the baseline it wrote:\n%s", out) + } +} + +// TestDocsCiteRefreshRunsTheShippedGuard is the hermetic end-to-end proof that +// the verb drives the real checker: a citation to a loopback address is refused +// by the SSRF guard and recorded as broken, with no packet leaving the host. +func TestDocsCiteRefreshRunsTheShippedGuard(t *testing.T) { + root, cfg := citeRepo(t, "http://127.0.0.1:9/never") + + out, err := runCLIErr(t, "docs", "cite", "refresh", "--json", "--config", cfg, "--root", root) + if err != nil { + t.Fatalf("refresh failed: %v\n%s", err, out) + } + var res struct { + Cited int `json:"cited"` + Fetched int `json:"fetched"` + Outcomes []struct { + URL string `json:"url"` + Status string `json:"status"` + Detail string `json:"detail"` + } `json:"outcomes"` + } + if err := json.Unmarshal(out, &res); err != nil { + t.Fatalf("output not JSON: %v\n%s", err, out) + } + if res.Cited != 1 || res.Fetched != 1 { + t.Fatalf("cited=%d fetched=%d, want 1/1\n%s", res.Cited, res.Fetched, out) + } + if len(res.Outcomes) != 1 || res.Outcomes[0].Status != "broken" { + t.Fatalf("outcomes = %+v, want one broken", res.Outcomes) + } + if !strings.Contains(res.Outcomes[0].Detail, "refusing") { + t.Errorf("detail = %q, want the SSRF-guard refusal", res.Outcomes[0].Detail) + } +} + +// TestDocsCiteRefreshExitsZeroOnBrokenLinks pins the division of labour: the +// refresh RECORDS what it found and the gate decides. A verb that failed on a +// dead link would make the record impossible to write. +func TestDocsCiteRefreshExitsZeroOnBrokenLinks(t *testing.T) { + root, cfg := citeRepo(t, "http://127.0.0.1:9/never") + if _, err := runCLIErr(t, "docs", "cite", "refresh", "--config", cfg, "--root", root); err != nil { + t.Fatalf("refresh must exit zero after recording a broken link, got %v", err) + } + doc := readBaselineJSON(t, root) + entries, _ := doc["entries"].(map[string]any) + entry, _ := entries["http://127.0.0.1:9/never"].(map[string]any) + if entry["outcome"] != "broken" { + t.Fatalf("entry = %+v, want outcome broken", entry) + } +} + +// TestDocsCiteConfirmWritesAManualReceipt is the rung-1 queue closing: a +// maintainer clears a link by name and the baseline records that a human did it. +func TestDocsCiteConfirmWritesAManualReceipt(t *testing.T) { + root, cfg := citeRepo(t, "https://paywalled.example.org/x") + + out, err := runCLIErr(t, "docs", "cite", "confirm", "https://paywalled.example.org/x", "--config", cfg, "--root", root) + if err != nil { + t.Fatalf("confirm failed: %v\n%s", err, out) + } + entries, _ := readBaselineJSON(t, root)["entries"].(map[string]any) + entry, _ := entries["https://paywalled.example.org/x"].(map[string]any) + if entry["verification"] != "manual" { + t.Fatalf("entry = %+v, want verification manual", entry) + } + if entry["verified_on"] == "" || entry["verified_on"] == nil { + t.Errorf("entry = %+v, want a dated receipt", entry) + } +} + +// TestDocsCiteConfirmRefusesAnUncitedURL keeps the verb from being a door for +// arbitrary entries. +func TestDocsCiteConfirmRefusesAnUncitedURL(t *testing.T) { + root, cfg := citeRepo(t, "https://example.org/cited") + + out, err := runCLIErr(t, "docs", "cite", "confirm", "https://elsewhere.example.org/x", "--config", cfg, "--root", root) + if err == nil { + t.Fatalf("confirm accepted an uncited URL\n%s", out) + } + if !strings.Contains(err.Error()+string(out), "not cited") { + t.Errorf("want a not-cited refusal, got %v\n%s", err, out) + } +} + +// TestDocsCiteConfirmIngestsAReceiptFile is AC 4's one-schema clause: the verb +// accepts, verbatim, the file the later generated checklist page will emit. +func TestDocsCiteConfirmIngestsAReceiptFile(t *testing.T) { + root, cfg := citeRepo(t, "https://paywalled.example.org/x") + receipt := filepath.Join(root, "receipt.json") + body := `{"schema_version": 1, "confirmed": [{"url": "https://paywalled.example.org/x", "final_url": "https://paywalled.example.org/x-v2", "verified_on": "2026-07-20"}]}` + if err := os.WriteFile(receipt, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + + out, err := runCLIErr(t, "docs", "cite", "confirm", "--receipt", receipt, "--config", cfg, "--root", root) + if err != nil { + t.Fatalf("confirm --receipt failed: %v\n%s", err, out) + } + entries, _ := readBaselineJSON(t, root)["entries"].(map[string]any) + entry, _ := entries["https://paywalled.example.org/x"].(map[string]any) + if entry["final_url"] != "https://paywalled.example.org/x-v2" || entry["verified_on"] != "2026-07-20" { + t.Fatalf("entry = %+v, want the receipt's declared address and date", entry) + } +} + +// TestDocsCiteConfirmNeedsSomethingToConfirm keeps a bare invocation from +// silently succeeding. +func TestDocsCiteConfirmNeedsSomethingToConfirm(t *testing.T) { + root, cfg := citeRepo(t, "https://example.org/cited") + if out, err := runCLIErr(t, "docs", "cite", "confirm", "--config", cfg, "--root", root); err == nil { + t.Fatalf("a bare confirm succeeded\n%s", out) + } +} + +// TestDocsLintReleaseGatePromotesOverdue is spc-17's "blocks at the release gate +// only": the same tree that warns for a commit fails once the release gate arms +// the promotion. The flag is the trust root, so a committed config cannot +// defang it. +func TestDocsLintReleaseGatePromotesOverdue(t *testing.T) { + root, cfg := citeRepo(t, "https://example.org/old") + baseline := `{"schema_version": 1, "entries": {"https://example.org/old": { + "final_url": "https://example.org/old", "last_checked": "2020-01-01", + "outcome": "alive", "verification": "automatic", "verified_on": "2020-01-01"}}}` + if err := os.MkdirAll(filepath.Join(root, ".abcd"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, ".abcd", "citations-baseline.json"), []byte(baseline), 0o644); err != nil { + t.Fatal(err) + } + + if out, err := runCLIErr(t, "docs", "lint", "--config", cfg, "--root", root); err != nil { + t.Fatalf("the commit gate must not calendar-block, got %v\n%s", err, out) + } + + out, err := runCLIErr(t, "docs", "lint", "--release-gate", "--json", "--config", cfg, "--root", root) + if err == nil { + t.Fatalf("the release gate did not block on an overdue citation\n%s", out) + } + var res struct { + Findings []struct { + RuleID string `json:"RuleID"` + Severity string `json:"Severity"` + } `json:"findings"` + Blockers int `json:"blockers"` + } + if err := json.Unmarshal(out, &res); err != nil { + t.Fatalf("output not JSON: %v\n%s", err, out) + } + var found bool + for _, f := range res.Findings { + if f.RuleID == "citation_baseline_overdue" && f.Severity == "blocker" { + found = true + } + } + if !found { + t.Fatalf("want a citation_baseline_overdue blocker, got %+v", res.Findings) + } +} diff --git a/internal/surface/cli/cli.go b/internal/surface/cli/cli.go index 85d4f37b..58d833b5 100644 --- a/internal/surface/cli/cli.go +++ b/internal/surface/cli/cli.go @@ -110,6 +110,10 @@ func NewRootCommand() *cobra.Command { rep, err := launch.DryRun(launch.DryRunRequest{ RepoRoot: cwd, Version: publishedVersion(cwd), + // Grading the citation baseline needs the lint engine, which + // imports launch for its semver — so the measurement is taken + // HERE, where both are already in scope, and handed in as data. + Citations: citationPreflight(cwd), }) if err != nil { return err @@ -118,6 +122,11 @@ func NewRootCommand() *cobra.Command { fmt.Fprintf(w, "abcd launch (dry-run) — version %s\n", rep.Version) fmt.Fprintf(w, " files bundled: %d\n", len(rep.Bundle.Included)) fmt.Fprintf(w, " scan hardfails: %d\n", rep.Scan.HardFails) + for _, g := range rep.Gates { + if g.Name == "citation-baseline" && g.Status == "ran" { + fmt.Fprintf(w, " citations: %s\n", termsafe.Sanitize(g.Detail)) + } + } fmt.Fprintf(w, " would publish: %v\n", rep.WouldPublish) if len(rep.WouldRefuseOn) > 0 { fmt.Fprintf(w, " would refuse on: %v\n", rep.WouldRefuseOn) @@ -204,6 +213,7 @@ func newDocsCommand(asJSON *bool) *cobra.Command { var configPath string var rootDir string + var releaseGate bool lintCmd := &cobra.Command{ Use: "lint", Short: "Lint docs for change-narration, broken links, and stray root markdown", @@ -249,6 +259,15 @@ func newDocsCommand(asJSON *bool) *cobra.Command { } return &exitError{Code: 2, Msg: fmt.Sprintf("docs lint: cannot read config %s: %s", ref, detail)} } + // --release-gate promotes the citation staleness finding from the + // commit gate's warn to a blocker (spc-17: commits are never + // calendar-blocked; a release is). The FLAG is the trust root, the + // way `record-lint --release-gate` arms the receipt gate: a repo + // must not be able to defang its own release by editing the + // committed config. + if releaseGate { + cfg = lint.ArmCitationOverdue(cfg) + } findings, err := lint.Lint(cfg, root) if err != nil { return err @@ -279,7 +298,12 @@ func newDocsCommand(asJSON *bool) *cobra.Command { } lintCmd.Flags().StringVar(&configPath, "config", "", "path to docs-lint.json (default: /.abcd/docs-lint.json)") lintCmd.Flags().StringVar(&rootDir, "root", "", "repo root to lint (default: current working directory)") + lintCmd.Flags().BoolVar(&releaseGate, "release-gate", false, + "run as the release gate: a citation past its staleness threshold blocks instead of warning (release-time only)") docsCmd.AddCommand(lintCmd) + // `cite` maintains the baseline `lint` enforces: the refresh does the live + // fetching the gate refuses to do, and confirm closes the manual queue. + docsCmd.AddCommand(newCiteCommand(asJSON)) return docsCmd } @@ -1434,6 +1458,12 @@ func newAhoyCommand(asJSON *bool) *cobra.Command { if mode, _ := res.Signals["install_mode"].(string); mode != "" { fmt.Fprintf(w, " install: %s\n", mode) } + // The citation baseline's coverage and age, present only in a repo + // that has armed the citation gate. The line embeds counts and a + // date derived from repo content, so it is sanitised. + if citations, _ := res.Signals["citations"].(string); citations != "" { + fmt.Fprintf(w, " citations: %s\n", termsafe.Sanitize(citations)) + } fmt.Fprintf(w, " gaps: %d\n", len(res.Gaps)) // Classification is read-only; the human report names the // next step per folder kind (itd-40 AC2/AC3). From dfc5537261cb592788848b481961ad495236c0c0 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:32:09 +0100 Subject: [PATCH 10/18] feat: arm the citation gate and commit this repo's citation baseline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first real run of `abcd docs cite refresh` over docs/: 51 cited URLs, 49 alive, none broken, two blocked. It also found four citations that had silently drifted behind redirects — x402.org, the LangChain context- engineering post, Restate's durable-execution concept page, and RFC 9162 — each now naming the address it actually resolves to. That drift is exactly the rot the gate exists to catch, and nobody would have noticed it by reading. The two blocked sources (openai.com, HTTP 403 to any automated fetcher) get no baseline entry and no fabricated receipt. They sit in the manual queue, and `abcd docs lint` reports them as unreceipted — which is precisely true. A maintainer who opens them clears both with: abcd docs cite confirm \ https://openai.com/index/harness-engineering/ \ https://openai.com/index/unlocking-the-codex-harness/ Until then the docs gate is red on those two lines. That is the gate working, not a defect: an agent must not sign a receipt asserting a human looked. Assisted-by: Claude:claude-opus-5[1m] --- .abcd/citations-baseline.json | 397 ++++++++++++++++++++++++++++++++++ .abcd/docs-lint.json | 2 +- .abcd/work/DECISIONS.md | 1 + CHANGELOG.md | 34 +++ commands/abcd/docs.md | 86 +++++++- docs/reference/terminology.md | 8 +- internal/README.md | 6 + 7 files changed, 522 insertions(+), 12 deletions(-) create mode 100644 .abcd/citations-baseline.json diff --git a/.abcd/citations-baseline.json b/.abcd/citations-baseline.json new file mode 100644 index 00000000..c4430a07 --- /dev/null +++ b/.abcd/citations-baseline.json @@ -0,0 +1,397 @@ +{ + "schema_version": 1, + "entries": { + "https://a2a-protocol.org/latest/specification/": { + "url": "https://a2a-protocol.org/latest/specification/", + "final_url": "https://a2a-protocol.org/latest/specification/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://adk.dev/sessions/memory/": { + "url": "https://adk.dev/sessions/memory/", + "final_url": "https://adk.dev/sessions/memory/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://agents.md/": { + "url": "https://agents.md/", + "final_url": "https://agents.md/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://agentskills.io/specification": { + "url": "https://agentskills.io/specification", + "final_url": "https://agentskills.io/specification", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://ai.google.dev/gemini-api/docs/function-calling": { + "url": "https://ai.google.dev/gemini-api/docs/function-calling", + "final_url": "https://ai.google.dev/gemini-api/docs/function-calling", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://ai.google.dev/gemini-api/docs/google-search": { + "url": "https://ai.google.dev/gemini-api/docs/google-search", + "final_url": "https://ai.google.dev/gemini-api/docs/google-search", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://ap2-protocol.org/ap2/specification/": { + "url": "https://ap2-protocol.org/ap2/specification/", + "final_url": "https://ap2-protocol.org/ap2/specification/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://claude.com/blog/building-agents-with-the-claude-agent-sdk": { + "url": "https://claude.com/blog/building-agents-with-the-claude-agent-sdk", + "final_url": "https://claude.com/blog/building-agents-with-the-claude-agent-sdk", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol": { + "url": "https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol", + "final_url": "https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://code.claude.com/docs/en/sandboxing": { + "url": "https://code.claude.com/docs/en/sandboxing", + "final_url": "https://code.claude.com/docs/en/sandboxing", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://csrc.nist.gov/glossary/term/authorization": { + "url": "https://csrc.nist.gov/glossary/term/authorization", + "final_url": "https://csrc.nist.gov/glossary/term/authorization", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://developers.openai.com/api/docs/guides/evals": { + "url": "https://developers.openai.com/api/docs/guides/evals", + "final_url": "https://developers.openai.com/api/docs/guides/evals", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://developers.openai.com/api/docs/guides/function-calling": { + "url": "https://developers.openai.com/api/docs/guides/function-calling", + "final_url": "https://developers.openai.com/api/docs/guides/function-calling", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai": { + "url": "https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai", + "final_url": "https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html": { + "url": "https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html", + "final_url": "https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/memory.html", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html": { + "url": "https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html", + "final_url": "https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://docs.restate.dev/foundations/key-concepts#durable-execution": { + "url": "https://docs.restate.dev/foundations/key-concepts#durable-execution", + "final_url": "https://docs.restate.dev/foundations/key-concepts#durable-execution", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://docs.temporal.io/evaluate/understanding-temporal": { + "url": "https://docs.temporal.io/evaluate/understanding-temporal", + "final_url": "https://docs.temporal.io/evaluate/understanding-temporal", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng": { + "url": "https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng", + "final_url": "https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://fidoalliance.org/fido-alliance-to-develop-standards-for-trusted-ai-agent-interactions/": { + "url": "https://fidoalliance.org/fido-alliance-to-develop-standards-for-trusted-ai-agent-interactions/", + "final_url": "https://fidoalliance.org/fido-alliance-to-develop-standards-for-trusted-ai-agent-interactions/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://genai.owasp.org/llmrisk/llm01-prompt-injection/": { + "url": "https://genai.owasp.org/llmrisk/llm01-prompt-injection/", + "final_url": "https://genai.owasp.org/llmrisk/llm01-prompt-injection/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://github.com/agentskills/agentskills": { + "url": "https://github.com/agentskills/agentskills", + "final_url": "https://github.com/agentskills/agentskills", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://github.com/agentsmd/agents.md": { + "url": "https://github.com/agentsmd/agents.md", + "final_url": "https://github.com/agentsmd/agents.md", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://github.com/open-telemetry/semantic-conventions-genai": { + "url": "https://github.com/open-telemetry/semantic-conventions-genai", + "final_url": "https://github.com/open-telemetry/semantic-conventions-genai", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://github.com/open-telemetry/semantic-conventions/releases": { + "url": "https://github.com/open-telemetry/semantic-conventions/releases", + "final_url": "https://github.com/open-telemetry/semantic-conventions/releases", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE.md": { + "url": "https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE.md", + "final_url": "https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE.md", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://learn.chatgpt.com/docs/sandboxing": { + "url": "https://learn.chatgpt.com/docs/sandboxing", + "final_url": "https://learn.chatgpt.com/docs/sandboxing", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://learn.microsoft.com/en-us/azure/durable-task/common/what-is-durable-task": { + "url": "https://learn.microsoft.com/en-us/azure/durable-task/common/what-is-durable-task", + "final_url": "https://learn.microsoft.com/en-us/azure/durable-task/common/what-is-durable-task", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://learn.microsoft.com/en-us/azure/well-architected/ai/grounding-data-design": { + "url": "https://learn.microsoft.com/en-us/azure/well-architected/ai/grounding-data-design", + "final_url": "https://learn.microsoft.com/en-us/azure/well-architected/ai/grounding-data-design", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-orchestration/": { + "url": "https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-orchestration/", + "final_url": "https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-orchestration/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://lfaidata.foundation/communityblog/2025/08/29/acp-joins-forces-with-a2a-under-the-linux-foundations-lf-ai-data/": { + "url": "https://lfaidata.foundation/communityblog/2025/08/29/acp-joins-forces-with-a2a-under-the-linux-foundations-lf-ai-data/", + "final_url": "https://lfaidata.foundation/communityblog/2025/08/29/acp-joins-forces-with-a2a-under-the-linux-foundations-lf-ai-data/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://modelcontextprotocol.io/specification/2025-11-25": { + "url": "https://modelcontextprotocol.io/specification/2025-11-25", + "final_url": "https://modelcontextprotocol.io/specification/2025-11-25", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://openai.github.io/openai-agents-python/guardrails/": { + "url": "https://openai.github.io/openai-agents-python/guardrails/", + "final_url": "https://openai.github.io/openai-agents-python/guardrails/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://openai.github.io/openai-agents-python/multi_agent/": { + "url": "https://openai.github.io/openai-agents-python/multi_agent/", + "final_url": "https://openai.github.io/openai-agents-python/multi_agent/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://openai.github.io/openai-agents-python/running_agents/": { + "url": "https://openai.github.io/openai-agents-python/running_agents/", + "final_url": "https://openai.github.io/openai-agents-python/running_agents/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works": { + "url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works", + "final_url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview": { + "url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview", + "final_url": "https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://platform.claude.com/docs/en/test-and-evaluate/develop-tests": { + "url": "https://platform.claude.com/docs/en/test-and-evaluate/develop-tests", + "final_url": "https://platform.claude.com/docs/en/test-and-evaluate/develop-tests", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.agenticcommerce.dev/": { + "url": "https://www.agenticcommerce.dev/", + "final_url": "https://www.agenticcommerce.dev/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.anthropic.com/engineering/building-effective-agents": { + "url": "https://www.anthropic.com/engineering/building-effective-agents", + "final_url": "https://www.anthropic.com/engineering/building-effective-agents", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents": { + "url": "https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents", + "final_url": "https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents": { + "url": "https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents", + "final_url": "https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation": { + "url": "https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation", + "final_url": "https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.langchain.com/blog/the-rise-of-context-engineering": { + "url": "https://www.langchain.com/blog/the-rise-of-context-engineering", + "final_url": "https://www.langchain.com/blog/the-rise-of-context-engineering", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation": { + "url": "https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation", + "final_url": "https://www.linuxfoundation.org/press/linux-foundation-announces-the-formation-of-the-agentic-ai-foundation", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.linuxfoundation.org/press/linux-foundation-launches-the-agent2agent-protocol-project-to-enable-secure-intelligent-communication-between-ai-agents": { + "url": "https://www.linuxfoundation.org/press/linux-foundation-launches-the-agent2agent-protocol-project-to-enable-secure-intelligent-communication-between-ai-agents", + "final_url": "https://www.linuxfoundation.org/press/linux-foundation-launches-the-agent2agent-protocol-project-to-enable-secure-intelligent-communication-between-ai-agents", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.openpolicyagent.org/docs": { + "url": "https://www.openpolicyagent.org/docs", + "final_url": "https://www.openpolicyagent.org/docs", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://www.rfc-editor.org/info/rfc9162/": { + "url": "https://www.rfc-editor.org/info/rfc9162/", + "final_url": "https://www.rfc-editor.org/info/rfc9162/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, + "https://x402.org/": { + "url": "https://x402.org/", + "final_url": "https://x402.org/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + } + } +} diff --git a/.abcd/docs-lint.json b/.abcd/docs-lint.json index f040ba5e..9d42ebe2 100644 --- a/.abcd/docs-lint.json +++ b/.abcd/docs-lint.json @@ -27,7 +27,7 @@ "citation_crosswalk_rows": {"enabled": true, "severity": "blocker"}, "citation_url_syntax": {"enabled": true, "severity": "blocker"}, "citation_source_policy": {"enabled": true, "severity": "blocker", "refused_domains": []}, - "citation_baseline": {"enabled": false, "severity": "blocker", "baseline": ".abcd/citations-baseline.json", "warn_after_days": 180, "block_after_days": 365} + "citation_baseline": {"enabled": true, "severity": "blocker", "baseline": ".abcd/citations-baseline.json", "warn_after_days": 180, "block_after_days": 365} }, "exempt_paths": [], "exempt_if_status": [] diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index 12618718..88b19e1f 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -864,3 +864,4 @@ parallel-agent merge contention bites. - 2026-07-27 — GRILLED itd-101/102/103/104 to settlement (six decisions, each now in its intent's Grill Settlements + acceptance criteria): itd-103 guard fails open loudly, blocker/warn tiers with committed-only overrides, shell-token-aware command-position matching with an itd-81-style TNR-floored corpus; itd-101 refresh is manual-with-nagging (scheduled CI later, own sign-off), staleness warns at 180d and blocks releases at 365d, human verifications age on the same clock; itd-102 identity lives in a parseable markdown block (config points at it), check is warn-tier and never rewrites; itd-104 ideate is an optional verb, never a pre-capture gate, with a fresh-context off-policy adversary. All four drafts are grill-settled and await intent plan. - 2026-07-27: itd-101 spec minted as spc-16 collided with unmerged #156 (iss-80 class, both spc-N and iss-N allocators); renumbered to spc-17 by hand, duplicate capture folded into iss-80 — ids on this branch deliberately skip 16. - 2026-07-27 — itd-101 part (a) settled four choices spc-17 left open. (1) The committed baseline lives at `.abcd/citations-baseline.json`, alongside docs-lint.json/record-lint.json/rules.json — it is config the gate reads every commit, not development record; rejected `.abcd/development/` (wrong tier) and `.abcd/work/` (not a session artefact). (2) A table is a CROSSWALK when its nearest preceding heading matches `(?i)crosswalk` (configurable) — the narrowest heuristic that selects docs/reference/terminology.md's table and leaves every ordinary directory-map/comparison table alone; rejected column-header and any-table-in-docs identification (both sweep up README.md and docs/README.md tables that carry no citations by design). (3) A page's CITATION CORPUS is its footnote definitions, including wrapped continuation lines — not its prose; a URL in a body paragraph stays links_resolve's business, which is what keeps the syntax and source-policy rules off ordinary text. (4) `refused_domains` ships EMPTY: the repo states its admission rule in prose ("no single-author coinages, no aggregators", docs/reference/terminology.md + ACKNOWLEDGEMENTS.md) but nowhere names a domain, and the gate must not invent an editorial blacklist the project never agreed. Also: staleness is measured from `last_checked` for automatic and manual entries alike (AC 3's "same clock"), and the 365-day threshold surfaces as a distinct rule id `citation_baseline_overdue` at warn — the commit gate never calendar-blocks, so promotion to blocker is the release gate's job in part (b). +- 2026-07-27 — itd-101 part (b) settled seven choices spc-17 left open, and consolidated one primitive. (1) The SSRF fetch guard moved from `internal/core/memory` to `internal/urlguard` rather than being copied for the second fetch path; its address predicate became a parameter so a fetch path can be exercised against an httptest server (which binds loopback) without the shipped policy ever being relaxed. (2) BLOCKED-FOR-AUTOMATION is classified by status code ONLY — 401, 403, 406, 429 — with no body sniffing for challenge-page markers: a heuristic over page text would be unreliable AND a reason to start reading content, and the challenge pages that matter answer with one of those codes anyway. Everything else non-2xx is broken. (3) One GET per URL, body never read and never retried: liveness is the status line, so a citation to a huge file costs a response header; a retry loop would make a run's duration a function of how many links are failing. (4) A blocked URL with NO prior entry writes NOTHING — the gate then reports it as unreceipted, which is exactly true; recording it broken would put a lie in a committed record the gate enforces. (5) A CURRENT manual receipt is preserved verbatim and not even re-requested; a STALE one is re-checked (AC 3's one clock) and, if the source still blocks, KEPT rather than deleted so the gate keeps warning honestly. Rejected: refetching every manual entry every run (downgrades a human's receipt to a robot's failure), and never refetching them (buys a permanent exemption from ageing). (6) `confirm` records ALIVE only — it is "I looked and it is there", never a channel for recording a link dead — and takes URLs positionally or a receipt file, both assembling ONE schema so the later generated checklist page is a different producer, not a second pathway. (7) `refresh` exits ZERO after recording broken links: it records, the gate decides; a verb that failed on a dead link could never write the record that reports it. Also: the release-gate promotion reuses `ArmReceiptGate`'s shape (`lint.ArmCitationOverdue`, armed by `abcd docs lint --release-gate`) with the FLAG as trust root, and the `abcd launch --dry-run` citation gate takes its measurement from the CLI as data because `core/lint` imports `core/launch` for its semver — the same shape the cobra-tree walk already uses. diff --git a/CHANGELOG.md b/CHANGELOG.md index c71bf579..6256521a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,40 @@ called out in a **Breaking** section. without an entry, none recorded broken, none whose recorded final address has drifted from what the page cites, and a 180-day staleness warning. Each rule is opt-in per repo, and the whole family is inert until configured. +- **`abcd docs cite refresh` — the one verb that fetches, so the gate never has + to** (itd-101, spc-17). It collects every cited URL through the same collector + the lint uses, gives each exactly one bounded attempt, and rewrites the + committed baseline. It never retries — a run's cost cannot become a function of + how many links are failing, nor a burst against a struggling host — and it never + reads a response body, because liveness is a property of the status line and a + citation to a huge file should cost a response header, not a download. Sources + that refuse automated fetchers (401, 403, 406, 429) are **not** recorded as + broken: a refusal says the fetcher may not look, which is a different fact from + the source being gone, so those URLs get no invented entry and are printed as a + manual checklist naming exactly where each is cited. A current human-verified + receipt is preserved verbatim and not even re-requested; once it ages past the + staleness threshold it is re-checked like any other, because human and machine + verifications share one clock. Receipts for addresses the documentation no + longer cites are dropped. +- **`abcd docs cite confirm` — recording that a human cleared a queued link.** + Name the URLs, or pass `--receipt` with a receipt file; both assemble the same + schema, so the generated checklist page that lands later is a different producer + of one input rather than a second pathway. Only URLs the documentation actually + cites can be confirmed, and one bad line refuses the whole receipt rather than + half-applying it. The receipt records **that** a human verified a citation and + **when**, never how — the schema declares no such field and decoding rejects + unknown keys outright. +- **The citation baseline surfaces where maintainers already look.** `abcd ahoy` + reports its coverage and age on one line in any repo that has armed the rule; + `abcd launch --dry-run` gains a `citation-baseline` gate that names entries + approaching the staleness blocker while still letting a release cut, and refuses + on ones that are overdue, broken, or unreceipted. `abcd docs lint + --release-gate` promotes an overdue citation from a warning to a blocker — the + flag is the trust root, so a repository cannot defang its own release by editing + a committed config, and an ordinary commit is never blocked by the calendar. +- **The citation gate is armed for this repository.** All 51 cited URLs in + `docs/` carry a receipt or a queue entry, and four citations that had silently + drifted behind redirects now name the address they actually resolve to. - **A schema-versioned citation baseline at `.abcd/citations-baseline.json`.** Per cited URL it records the final resolved address, when it was last checked, the outcome, and whether verification was automatic or manual with its date. diff --git a/commands/abcd/docs.md b/commands/abcd/docs.md index bf718032..daa1ec24 100644 --- a/commands/abcd/docs.md +++ b/commands/abcd/docs.md @@ -1,13 +1,15 @@ --- name: docs -description: Lint this repo's documentation for currency — change-narration ("previously", "formerly", …), broken relative links, and stray root markdown — by invoking the abcd binary. Read-only; performs zero writes. -argument-hint: "[lint]" +description: Lint this repo's documentation for currency — change-narration ("previously", "formerly", …), broken relative links, stray root markdown, and citation health — by invoking the abcd binary. `lint` is read-only; `cite refresh` and `cite confirm` maintain the citation baseline. +argument-hint: "[lint | cite refresh | cite confirm ...]" --- -# `/abcd:docs` documentation-currency lint +# `/abcd:docs` documentation currency and citations -Run the abcd binary's docs-currency lint for the current repo and present the -result. This command performs **zero writes**. +Two jobs live here. `lint` grades the documentation and writes nothing. `cite` +maintains the committed citation baseline that the lint then enforces offline. + +## `lint` — the currency gate (zero writes, zero network) Run: @@ -29,10 +31,80 @@ It also checks that relative links resolve and that no stray markdown sits at th repo root (it belongs under `docs/`). Point the user at the offending file and line for each finding, and note whether it is a blocker or a warning. +Where a repo arms them, the citation rules add: footnote markers and definitions +in bijection, every crosswalk table row carrying a footnote, well-formed URLs and +DOIs, refused source domains, and the committed baseline — no cited URL without a +receipt, none recorded broken, none whose recorded final address has drifted from +what the page cites, and a staleness warning past 180 days. Every one of these +reads committed files only; nothing dials out. + +`--release-gate` runs the same lint with one difference: a citation past the +365-day threshold blocks instead of warning. It is for release machinery only — +an ordinary commit is never blocked by the calendar. + If `blockers` is zero the docs are currency-clean. +## `cite refresh` — the one verb that reaches the network + +Run: + +```bash +abcd docs cite refresh --json +``` + +This fetches every cited URL once, bounded and without retries, and rewrites the +baseline. Report from the JSON: + +- `cited`, `fetched`, `preserved` — how much was checked, and how many current + human-verified receipts were left alone. +- `outcomes` — each URL's `status` (`ok`, `broken`, `blocked`, `preserved`), + its `final_url`, and the `detail`. List every `broken` one: those are dead + citations the gate will block on. +- `queue` — sources that refuse automated fetchers. Present these as a checklist + with the `sites` that cite each one, and tell the user to open each link and + confirm it. +- `dropped` — receipts removed because the docs no longer cite those addresses. + +A `blocked` source is **not** recorded as broken and gets no invented entry: a +403 says the fetcher may not look, which is a different fact from the citation +being dead. Never suggest editing the baseline by hand to clear one. + +## `cite confirm` — closing the manual queue + +Once the user has opened a queued link and seen the document, record it: + +```bash +abcd docs cite confirm [...] +``` + +For a batch, or to record a redirect the user followed, pass a receipt file +instead: + +```bash +abcd docs cite confirm --receipt receipts.json +``` + +```json +{ + "schema_version": 1, + "confirmed": [ + {"url": "https://example.org/a", "final_url": "https://example.org/a-v2", "verified_on": "2026-07-20"} + ] +} +``` + +Only URLs the documentation actually cites can be confirmed. The receipt records +**that** a human verified the citation and **when** — never how. Do not ask the +user for their method, and never record one: the schema has no field for it and +loading rejects unknown keys. + +Confirm on the user's word that they checked. An agent must never run `confirm` +on its own initiative to clear a red gate. + +## Fallback + If the `abcd` binary is not on `PATH`, fall back to -`go run ./cmd/abcd docs lint --json` from the repo root, or tell the user to -build it with `make build`. +`go run ./cmd/abcd docs …` from the repo root, or tell the user to build it with +`make build`. **User input:** $ARGUMENTS diff --git a/docs/reference/terminology.md b/docs/reference/terminology.md index 882e624e..ad308f86 100644 --- a/docs/reference/terminology.md +++ b/docs/reference/terminology.md @@ -74,7 +74,7 @@ glossed inline rather than deep-linked. [^react]: Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y., "ReAct: Synergizing Reasoning and Acting in Language Models", ICLR 2023, DOI 10.48550/arXiv.2210.03629. The loop shape is defined independently in OpenAI's Agents SDK ("Running agents", https://openai.github.io/openai-agents-python/running_agents/) and Anthropic's tool-use documentation (https://platform.claude.com/docs/en/agents-and-tools/tool-use/how-tool-use-works). -[^ap2]: "Agent Payments Protocol Specification", v0.2, https://ap2-protocol.org/ap2/specification/; launch announcement, Google Cloud, 16 September 2025, https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol; contributed to the FIDO Alliance, 28 April 2026, https://fidoalliance.org/fido-alliance-to-develop-standards-for-trusted-ai-agent-interactions/. Adjacent: x402 (under Linux Foundation governance, https://www.x402.org/) and the Agentic Commerce Protocol (OpenAI and Stripe, https://www.agenticcommerce.dev/) — the latter's "ACP" abbreviation collides with IBM's Agent Communication Protocol, which merged into A2A in August 2025 (https://lfaidata.foundation/communityblog/2025/08/29/acp-joins-forces-with-a2a-under-the-linux-foundations-lf-ai-data/). +[^ap2]: "Agent Payments Protocol Specification", v0.2, https://ap2-protocol.org/ap2/specification/; launch announcement, Google Cloud, 16 September 2025, https://cloud.google.com/blog/products/ai-machine-learning/announcing-agents-to-payments-ap2-protocol; contributed to the FIDO Alliance, 28 April 2026, https://fidoalliance.org/fido-alliance-to-develop-standards-for-trusted-ai-agent-interactions/. Adjacent: x402 (under Linux Foundation governance, https://x402.org/) and the Agentic Commerce Protocol (OpenAI and Stripe, https://www.agenticcommerce.dev/) — the latter's "ACP" abbreviation collides with IBM's Agent Communication Protocol, which merged into A2A in August 2025 (https://lfaidata.foundation/communityblog/2025/08/29/acp-joins-forces-with-a2a-under-the-linux-foundations-lf-ai-data/). [^skills]: "Agent Skills — Specification", https://agentskills.io/specification; specification repository, https://github.com/agentskills/agentskills (Apache-2.0 code, CC-BY-4.0 documentation). Unversioned, with no formal draft/final designation; at least one field is marked Experimental; no standards-body stewardship is stated on the primary sources. @@ -86,9 +86,9 @@ glossed inline rather than deep-linked. [^auto]: Franklin, S. and Graesser, A., "Is It an Agent, or Just a Program?: A Taxonomy for Autonomous Agents", ATAL 1996 (Springer LNCS 1193, 1997), DOI 10.1007/BFb0013570. See also Wooldridge and Jennings under multi-agent systems. -[^ce]: LangChain, "The rise of 'context engineering'", 23 June 2025, https://blog.langchain.com/the-rise-of-context-engineering/; Anthropic, "Effective context engineering for AI agents", 29 September 2025, https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents. +[^ce]: LangChain, "The rise of 'context engineering'", 23 June 2025, https://www.langchain.com/blog/the-rise-of-context-engineering; Anthropic, "Effective context engineering for AI agents", 29 September 2025, https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents. -[^durable]: Microsoft, "What is Durable Task?", https://learn.microsoft.com/en-us/azure/durable-task/common/what-is-durable-task; Temporal, "Understanding Temporal", https://docs.temporal.io/evaluate/understanding-temporal; Restate, "Key Concepts: Durable Execution", https://docs.restate.dev/concepts/durable_execution/. +[^durable]: Microsoft, "What is Durable Task?", https://learn.microsoft.com/en-us/azure/durable-task/common/what-is-durable-task; Temporal, "Understanding Temporal", https://docs.temporal.io/evaluate/understanding-temporal; Restate, "Key Concepts: Durable Execution", https://docs.restate.dev/foundations/key-concepts#durable-execution. [^evals]: OpenAI, "Working with evals", https://developers.openai.com/api/docs/guides/evals; Anthropic, "Define success criteria and build evaluations", https://platform.claude.com/docs/en/test-and-evaluate/develop-tests; Liang, P., et al., "Holistic Evaluation of Language Models", Transactions on Machine Learning Research, 2023, DOI 10.48550/arXiv.2211.09110. @@ -116,6 +116,6 @@ glossed inline rather than deep-linked. [^sand]: OpenAI, Codex documentation, "Sandboxing", https://learn.chatgpt.com/docs/sandboxing; Anthropic, "Configure the sandboxed Bash tool", https://code.claude.com/docs/en/sandboxing. Both sources note that sandboxing needs filesystem *and* network isolation, and reduces rather than eliminates risk. -[^tamper]: IETF, RFC 9162, "Certificate Transparency Version 2.0", December 2021, DOI 10.17487/RFC9162, https://www.rfc-editor.org/rfc/rfc9162 — cited as the transferable verifiable-log primitive, not as an agentic-AI standard. +[^tamper]: IETF, RFC 9162, "Certificate Transparency Version 2.0", December 2021, DOI 10.17487/RFC9162, https://www.rfc-editor.org/info/rfc9162/ — cited as the transferable verifiable-log primitive, not as an agentic-AI standard. [^tooluse]: OpenAI, "Function calling", https://developers.openai.com/api/docs/guides/function-calling; Anthropic, "Tool use with Claude", https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview; Google, "Function calling", Gemini API documentation, https://ai.google.dev/gemini-api/docs/function-calling. diff --git a/internal/README.md b/internal/README.md index 1601fd9e..1c230b84 100644 --- a/internal/README.md +++ b/internal/README.md @@ -24,6 +24,12 @@ plugin surface, and a future MCP server share one engine. those transports expose — so it is cobra-free like the rest of `core/`. The walk that reads the live command tree needs cobra and therefore lives in `surface/cli`, which hands its result in; the dependency never points back. +- **`core/cite/`** — the live half of the citation gate: the bounded fetcher and + the refresh that writes the committed baseline `core/lint` then enforces with + zero network. It is the only place abcd dials out on behalf of documentation, + and it holds the seam a specialist link checker would later slot into — the + baseline schema and the lint rules are the contract, the fetcher is a + replaceable producer. - **`core/lifeboat/`** — the brief↔lifeboat contract. `mapping.go` is the single source of truth for which brief section a lifeboat fills from which source tier, and it is rendered into the brief's `00-meta.md` with a test asserting From 39f615889b4058b4dc7b70d8510bc5a3480ca34b Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:34:05 +0100 Subject: [PATCH 11/18] docs: describe the cite verbs on both documentation surfaces MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `docs` command page and the brief's surface record both said `docs` had one sub-verb and performed zero writes. Neither is true now: `cite refresh` reaches the network and `cite confirm` writes a committed record, and `lint` gained the release-gate promotion. The plugin page also carries two instructions an agent needs and cannot infer: never suggest hand-editing the baseline to clear a blocked source, and never run `confirm` on its own initiative — a receipt asserts that a human looked. Assisted-by: Claude:claude-opus-5[1m] --- .../development/brief/04-surfaces/10-docs.md | 55 ++++++++++++++----- 1 file changed, 41 insertions(+), 14 deletions(-) diff --git a/.abcd/development/brief/04-surfaces/10-docs.md b/.abcd/development/brief/04-surfaces/10-docs.md index 37d7d512..d9c31522 100644 --- a/.abcd/development/brief/04-surfaces/10-docs.md +++ b/.abcd/development/brief/04-surfaces/10-docs.md @@ -1,25 +1,39 @@ -# `/abcd:docs` — Documentation-Currency Lint +# `/abcd:docs` — Documentation Currency and Citations `/abcd:docs` runs abcd's documentation-currency checks over the current repo and -presents the result. It is **strictly read-only** — it performs zero writes, it -only reports. It is the deterministic half of the docs release gate (the semantic -half is the `docs-currency-reviewer` agent). +maintains the citation record those checks enforce. `lint` is **strictly +read-only** — it performs zero writes and touches no network — and it is the +deterministic half of the docs release gate (the semantic half is the +`docs-currency-reviewer` agent). The `cite` sub-tree is the writing half: it is +the only place abcd reaches the network on behalf of documentation, and it runs +when a maintainer asks. ## Sub-verbs -The shipped verb surface is one sub-verb, `lint`: - - **`/abcd:docs lint`** — lint this repo's documentation for currency and print the findings. The plugin command (`commands/abcd/docs.md`) invokes `abcd docs lint --json` and summarises the result. +- **`abcd docs cite refresh`** — fetch every cited URL once and rewrite the + committed citation baseline. Each URL gets exactly one bounded attempt with no + retries, and no response body is read: liveness is judged from the status line. + Sources that refuse automated fetchers are printed as a manual checklist rather + than recorded as broken. +- **`abcd docs cite confirm … | --receipt `** — record that a human + verified a citation the fetcher could not read. Both forms assemble one receipt + schema, so the generated checklist page that lands later is a different + producer of the same input rather than a second pathway. Only URLs the + documentation cites can be confirmed. -Bare `abcd docs` prints command usage (its one sub-verb, `lint`) — it does **not** -render a status board; the bare-status convention is scoped to `ahoy`/`capture`/ -`memory`/`intent`/`spec` and bare `abcd`, not to `docs`. The global `--json` flag -emits the machine-readable finding list, and `docs lint` additionally accepts two -local flags: `--config` (path to the `docs-lint.json` it loads, default -`/.abcd/docs-lint.json`) and `--root` (repo root to lint, default the -current working directory). +Bare `abcd docs` prints command usage — it does **not** render a status board; +the bare-status convention is scoped to `ahoy`/`capture`/`memory`/`intent`/`spec` +and bare `abcd`, not to `docs`. The global `--json` flag emits the +machine-readable payload. Every verb here accepts `--config` (path to the +`docs-lint.json` it loads, default `/.abcd/docs-lint.json`) and `--root` +(repo root, default the current working directory), so the refresh fetches +exactly the set the gate demands receipts for. `docs lint` additionally accepts +`--release-gate`, which promotes an overdue citation from a warning to a blocker; +the flag is release machinery's, so an ordinary commit is never blocked by the +calendar. ## What it checks @@ -34,6 +48,13 @@ current working directory). - **Stray root markdown** — no stray markdown at the repo root (it belongs under `docs/`; the allowed root files are the fixed set — README, CHANGELOG, CONTRIBUTING, etc.). +- **Citations** — where a repo arms the rules: footnote markers and definitions + in bijection, every crosswalk table row carrying a footnote, well-formed cited + URLs and DOIs, refused source domains, and the committed baseline at + `.abcd/citations-baseline.json` — no cited URL without a receipt, none recorded + broken, none whose recorded final address has drifted from what the page cites, + and a staleness warning past 180 days. All of it reads committed files; the + fetching lives in `cite refresh`. - **Host-agnostic prose** — user-facing docs must not name a specific agent harness or bundled tool. This repo's `.abcd/docs-lint.json` defines a family of `harness/*` banned tokens (each a **blocker**) that catch such names, so the @@ -50,12 +71,18 @@ release gate. ## Composition -`/abcd:docs` is the deterministic, fast, always-runnable currency check. +`/abcd:docs lint` is the deterministic, fast, always-runnable currency check. The `docs-currency-reviewer` agent is its semantic complement — it verifies that every user-facing claim still matches the code, which a structural lint cannot. The release gate runs both: `docs lint` (deterministic) and the reviewer (semantic) must each pass before a tag. +`cite refresh` composes with the gate by separation: the gate is deterministic +because the fetching happens elsewhere and arrives as a committed record a +reviewer reads in a diff. The baseline's age surfaces at `abcd ahoy` and in the +`abcd launch --dry-run` preflight, which names entries approaching the staleness +blocker while a release still cuts, and refuses on ones past it. + ## References - Plugin command: [`commands/abcd/docs.md`](../../../../commands/abcd/docs.md) From 6c5aad2eaf6e1ced71f3ad6a6f003142cf593a24 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:35:04 +0100 Subject: [PATCH 12/18] chore: record the receipt one flaky source did eventually answer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both openai.com citations sit behind bot mitigation that answers 403 to an automated fetcher — but not deterministically. Across repeated runs while verifying the refresh is idempotent, one of them returned 200 once, and the merge rule kept the entry: a subsequent 403 is not evidence a source we successfully read today is gone, so a prior automatic receipt survives a blocked re-check rather than being deleted. That is the intended behaviour, not a workaround, and it is worth naming: for Cloudflare-fronted sources the blocked/alive split is not reproducible run to run. It is precisely why the manual queue exists, and why a blocked outcome writes nothing rather than recording the source broken. One citation — https://openai.com/index/harness-engineering/ — has never answered and still has no receipt. It needs a human: abcd docs cite confirm https://openai.com/index/harness-engineering/ Assisted-by: Claude:claude-opus-5[1m] --- .abcd/citations-baseline.json | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.abcd/citations-baseline.json b/.abcd/citations-baseline.json index c4430a07..faceb5d2 100644 --- a/.abcd/citations-baseline.json +++ b/.abcd/citations-baseline.json @@ -257,6 +257,14 @@ "verification": "automatic", "verified_on": "2026-07-27" }, + "https://openai.com/index/unlocking-the-codex-harness/": { + "url": "https://openai.com/index/unlocking-the-codex-harness/", + "final_url": "https://openai.com/index/unlocking-the-codex-harness/", + "last_checked": "2026-07-27", + "outcome": "alive", + "verification": "automatic", + "verified_on": "2026-07-27" + }, "https://openai.github.io/openai-agents-python/guardrails/": { "url": "https://openai.github.io/openai-agents-python/guardrails/", "final_url": "https://openai.github.io/openai-agents-python/guardrails/", From d7b25dadc0aed08e7937696877bc9ec060d05778 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Mon, 27 Jul 2026 23:50:01 +0100 Subject: [PATCH 13/18] fix: close a path-traversal write and four record-integrity holes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Findings from the adversarial security review (BLOCK) and the ruthless code review (FIX FIRST), each with a regression test watched fail first. The BLOCK: `citation_baseline.baseline` comes from committed `.abcd/docs-lint.json`, so a contributor controls it, and this branch made it steer a WRITE for the first time — SaveBaseline had no production callers before. A value of "../../../ESCAPED/pwned.json" made `abcd docs cite refresh` MkdirAll and write outside the repository, reproduced against a built binary. Containment now lives at the single choke point every reader and both writers already resolve through, and an escaping path is refused rather than quietly normalised: a config asking to write outside the repo is a mistake or an attack, and rewriting it to something safe would hide both. The worst correctness bug: `Check` collapses DNS failure, refused connection and timeout into the same `broken` as a genuine 404, and the broken branch wrote unconditionally. So a refresh run behind a captive portal rewrote every entry as broken — stale human receipts included, since those are re-checked and therefore never reached the blocked branch that protects them — and committed it before the transcript printed. A run in which nothing at all succeeded is now refused, but only when there was a record to protect; a first run over genuinely dead citations still writes one. Three more. An unrecognised Status from a Checker is an error rather than a fall-through, because the seam is exported for the adapter spc-17 anticipates and falling through dropped the URL from both the baseline and the queue — the one outcome nothing downstream could detect. The missing-entry lint message named only `refresh`, which by construction cannot clear it: that state means a source refuses automated fetchers, so refresh re-queues and writes nothing again, forever. It now names `confirm` too. And a docs-lint config that arms the rule but fails to parse was rendering to the release preflight as "not armed" — a false statement that waved a release through — and is now a distinct unreadable state that refuses. Two consolidations the reviews called: `daysBetween` is the exported lint.DaysBetween, so the verb cannot call an entry current on the day the gate calls it overdue; and the CLI's ad-hoc error helper is dropped for the canonical scrubPaths, which preserves the *PathError type the redactor needs — an unreadable page was leaking an absolute developer path into machine output. The CHANGELOG is corrected on two counts it overclaimed: the release-gate flag is built but not yet passed by release.yml, and one cited source is unreceipted rather than all 51 being covered. Assisted-by: Claude:claude-opus-5[1m] --- .abcd/work/DECISIONS.md | 1 + CHANGELOG.md | 14 ++++- internal/core/cite/confirm.go | 7 ++- internal/core/cite/refresh.go | 54 +++++++++++++--- internal/core/cite/refresh_test.go | 85 ++++++++++++++++++++++++++ internal/core/launch/citations.go | 14 +++++ internal/core/launch/citations_test.go | 23 +++++++ internal/core/lint/citations.go | 64 ++++++++++++++++--- internal/core/lint/citesummary.go | 7 ++- internal/core/lint/citesummary_test.go | 75 +++++++++++++++++++++++ internal/surface/cli/cite.go | 53 +++++++--------- internal/surface/cli/cite_test.go | 74 ++++++++++++++++++++++ 12 files changed, 414 insertions(+), 57 deletions(-) diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index 88b19e1f..d8f156d6 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -865,3 +865,4 @@ parallel-agent merge contention bites. - 2026-07-27: itd-101 spec minted as spc-16 collided with unmerged #156 (iss-80 class, both spc-N and iss-N allocators); renumbered to spc-17 by hand, duplicate capture folded into iss-80 — ids on this branch deliberately skip 16. - 2026-07-27 — itd-101 part (a) settled four choices spc-17 left open. (1) The committed baseline lives at `.abcd/citations-baseline.json`, alongside docs-lint.json/record-lint.json/rules.json — it is config the gate reads every commit, not development record; rejected `.abcd/development/` (wrong tier) and `.abcd/work/` (not a session artefact). (2) A table is a CROSSWALK when its nearest preceding heading matches `(?i)crosswalk` (configurable) — the narrowest heuristic that selects docs/reference/terminology.md's table and leaves every ordinary directory-map/comparison table alone; rejected column-header and any-table-in-docs identification (both sweep up README.md and docs/README.md tables that carry no citations by design). (3) A page's CITATION CORPUS is its footnote definitions, including wrapped continuation lines — not its prose; a URL in a body paragraph stays links_resolve's business, which is what keeps the syntax and source-policy rules off ordinary text. (4) `refused_domains` ships EMPTY: the repo states its admission rule in prose ("no single-author coinages, no aggregators", docs/reference/terminology.md + ACKNOWLEDGEMENTS.md) but nowhere names a domain, and the gate must not invent an editorial blacklist the project never agreed. Also: staleness is measured from `last_checked` for automatic and manual entries alike (AC 3's "same clock"), and the 365-day threshold surfaces as a distinct rule id `citation_baseline_overdue` at warn — the commit gate never calendar-blocks, so promotion to blocker is the release gate's job in part (b). - 2026-07-27 — itd-101 part (b) settled seven choices spc-17 left open, and consolidated one primitive. (1) The SSRF fetch guard moved from `internal/core/memory` to `internal/urlguard` rather than being copied for the second fetch path; its address predicate became a parameter so a fetch path can be exercised against an httptest server (which binds loopback) without the shipped policy ever being relaxed. (2) BLOCKED-FOR-AUTOMATION is classified by status code ONLY — 401, 403, 406, 429 — with no body sniffing for challenge-page markers: a heuristic over page text would be unreliable AND a reason to start reading content, and the challenge pages that matter answer with one of those codes anyway. Everything else non-2xx is broken. (3) One GET per URL, body never read and never retried: liveness is the status line, so a citation to a huge file costs a response header; a retry loop would make a run's duration a function of how many links are failing. (4) A blocked URL with NO prior entry writes NOTHING — the gate then reports it as unreceipted, which is exactly true; recording it broken would put a lie in a committed record the gate enforces. (5) A CURRENT manual receipt is preserved verbatim and not even re-requested; a STALE one is re-checked (AC 3's one clock) and, if the source still blocks, KEPT rather than deleted so the gate keeps warning honestly. Rejected: refetching every manual entry every run (downgrades a human's receipt to a robot's failure), and never refetching them (buys a permanent exemption from ageing). (6) `confirm` records ALIVE only — it is "I looked and it is there", never a channel for recording a link dead — and takes URLs positionally or a receipt file, both assembling ONE schema so the later generated checklist page is a different producer, not a second pathway. (7) `refresh` exits ZERO after recording broken links: it records, the gate decides; a verb that failed on a dead link could never write the record that reports it. Also: the release-gate promotion reuses `ArmReceiptGate`'s shape (`lint.ArmCitationOverdue`, armed by `abcd docs lint --release-gate`) with the FLAG as trust root, and the `abcd launch --dry-run` citation gate takes its measurement from the CLI as data because `core/lint` imports `core/launch` for its semver — the same shape the cobra-tree walk already uses. +- 2026-07-27 — itd-101 part (b) review round settled five more. (1) The committed `citation_baseline.baseline` config value is CONTAINED at one choke point (`lint.CitationPolicy`, which now returns an error): a `../../..` or absolute path is REFUSED, never silently normalised, because `SaveBaseline` MkdirAll's its parent and a contributor controls that file — this was a real arbitrary-file-write primitive found by the security review, reproduced against a built binary. Both writers and both readers resolve through the one check. (2) A refresh run in which NOTHING succeeded is REFUSED rather than committed, and only when a prior baseline existed: `Check` collapses DNS failure, refused connection and timeout into the same `broken` as a 404, so a run behind a captive portal would otherwise rewrite every entry as broken — stale human receipts included, since those are re-checked and so not covered by the blocked branch — and the operator would find the gate blocking every commit. A first run over genuinely dead citations has nothing to protect and still writes. (3) An unrecognised `Status` from a `Checker` is an ERROR, not a fall-through: the seam is exported for AC 5's adapter, and falling through would drop the URL from BOTH the baseline and the queue — the one outcome nothing downstream could detect. (4) The missing-entry lint message names BOTH verbs: the only state that produces a missing entry is a source refusing automated fetchers, so naming `refresh` alone sent the maintainer round a loop that cannot terminate — only `confirm` clears it. (5) A docs-lint config that arms the citation rule but fails to PARSE is reported to the release preflight as `Unreadable` and REFUSES, distinct from the nil "not armed" state — rendering a broken gate as an absent one would wave a release through on a false statement. Also consolidated: `daysBetween` is now the exported `lint.DaysBetween`, called by the verb and the gate alike, because two copies of the staleness boundary is two chances for the verb to call an entry current on the day the gate calls it overdue; and the CLI's ad-hoc `citeErrorDetail` was dropped for the canonical `scrubPaths`, which preserves the `*PathError` type the redactor needs (an absolute developer path was reaching machine output). diff --git a/CHANGELOG.md b/CHANGELOG.md index 6256521a..24275e15 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -61,9 +61,17 @@ called out in a **Breaking** section. --release-gate` promotes an overdue citation from a warning to a blocker — the flag is the trust root, so a repository cannot defang its own release by editing a committed config, and an ordinary commit is never blocked by the calendar. -- **The citation gate is armed for this repository.** All 51 cited URLs in - `docs/` carry a receipt or a queue entry, and four citations that had silently - drifted behind redirects now name the address they actually resolve to. + The flag is built and tested but **not yet passed by `release.yml`**, which + still runs the plain `abcd docs lint`; wiring it into the release workflow is a + CI change needing its own sign-off, and until it lands the 365-day threshold + warns at release time rather than blocking. +- **The citation gate is armed for this repository.** 50 of the 51 URLs cited + under `docs/` carry a receipt, and four citations that had silently drifted + behind redirects now name the address they actually resolve to. One source + answers HTTP 403 to any automated fetcher and is waiting in the manual queue, + so `abcd docs lint` reports it as unreceipted until a maintainer opens it and + runs `abcd docs cite confirm`. A refusal is not a fabricated receipt: the gate + reporting a link nobody has verified is the mechanism working. - **A schema-versioned citation baseline at `.abcd/citations-baseline.json`.** Per cited URL it records the final resolved address, when it was last checked, the outcome, and whether verification was automatic or manual with its date. diff --git a/internal/core/cite/confirm.go b/internal/core/cite/confirm.go index 1c984954..d3a2ab38 100644 --- a/internal/core/cite/confirm.go +++ b/internal/core/cite/confirm.go @@ -123,7 +123,10 @@ func Confirm(req ConfirmRequest) (ConfirmResult, error) { return ConfirmResult{}, &ConfirmError{"citation receipt confirms nothing"} } - relPath, _, _ := lint.CitationPolicy(req.Config) + relPath, _, _, err := lint.CitationPolicy(req.Config) + if err != nil { + return ConfirmResult{}, err + } absPath := filepath.Join(req.RepoRoot, filepath.FromSlash(relPath)) cited, err := lint.CollectCitedURLs(req.Config, req.RepoRoot) @@ -154,7 +157,7 @@ func Confirm(req ConfirmRequest) (ConfirmResult, error) { return ConfirmResult{}, &ConfirmError{ "cannot confirm " + c.URL + ": verified_on " + strconv.Quote(on) + " is not a " + dateLayout + " date"} } - if daysBetween(when, now) < 0 { + if lint.DaysBetween(when, now) < 0 { return ConfirmResult{}, &ConfirmError{ "cannot confirm " + c.URL + ": verified_on " + on + " is in the future"} } diff --git a/internal/core/cite/refresh.go b/internal/core/cite/refresh.go index 93db61d1..b591c808 100644 --- a/internal/core/cite/refresh.go +++ b/internal/core/cite/refresh.go @@ -21,11 +21,18 @@ import ( "os" "path/filepath" "sort" + "strconv" "time" "github.com/REPPL/abcd-cli/internal/core/lint" ) +// RefreshError is a refused refresh — a distinct type so a caller can tell a +// refusal to write from an I/O fault without string matching. +type RefreshError struct{ msg string } + +func (e *RefreshError) Error() string { return e.msg } + // dateLayout matches the baseline's date precision. The staleness policy is // measured in days, so an instant would add churn to a committed file for no // gain in the only arithmetic that reads it. @@ -144,7 +151,10 @@ func Refresh(req RefreshRequest) (RefreshResult, error) { parallel = defaultParallel } - relPath, warnDays, _ := lint.CitationPolicy(req.Config) + relPath, warnDays, _, err := lint.CitationPolicy(req.Config) + if err != nil { + return RefreshResult{}, err + } absPath := filepath.Join(req.RepoRoot, filepath.FromSlash(relPath)) cited, err := lint.CollectCitedURLs(req.Config, req.RepoRoot) @@ -193,6 +203,10 @@ func Refresh(req RefreshRequest) (RefreshResult, error) { res.Fetched = len(jobs) checked := runChecks(checker, toCheck, parallel) + // succeeded counts the checks that actually reached a document. It is the + // signal the wholesale-failure guard below reads: a blocked source is not a + // success, but it is not evidence of a broken network either. + succeeded := 0 for _, j := range jobs { out := checked[j.ref.URL] @@ -201,6 +215,7 @@ func Refresh(req RefreshRequest) (RefreshResult, error) { }) switch out.Status { case StatusOK: + succeeded++ final := out.FinalURL if final == "" { final = j.ref.URL @@ -221,13 +236,38 @@ func Refresh(req RefreshRequest) (RefreshResult, error) { Outcome: lint.OutcomeBroken, Verification: lint.VerificationAutomatic, VerifiedOn: stamp, } case StatusBlocked: + // A refusal is not a failed network: the host answered. + succeeded++ if j.had { next[j.ref.URL] = j.prev } res.Queue = append(res.Queue, QueueItem{URL: j.ref.URL, Detail: out.Detail, Sites: j.ref.Sites}) + default: + // The seam is exported, so a later adapter is a producer this package + // does not control. An unrecognised status must stop the run: falling + // through would drop the URL from BOTH the baseline and the queue, + // which is the one outcome nothing downstream could detect. + return RefreshResult{}, &RefreshError{"checker returned an unrecognised status " + + strconv.Quote(string(out.Status)) + " for " + j.ref.URL} } } + // A run in which NOTHING succeeded is a broken network, not fifty dead + // citations. Check collapses a DNS failure, a refused connection and a + // timeout into the same StatusBroken as a genuine 404, so committing such a + // run would rewrite every entry as broken — stale human receipts included, + // since those are re-checked and so are not covered by the blocked branch — + // and the gate would then block every commit until someone reverted the file + // by hand. Refuse before writing; the operator keeps the record they had. + // + // It binds only when there WAS a record to protect: a first run over + // genuinely dead citations has nothing to lose and is allowed to write one. + if len(previous) > 0 && res.Fetched > 0 && succeeded == 0 && res.Preserved == 0 { + return RefreshResult{}, &RefreshError{"every one of the " + strconv.Itoa(res.Fetched) + + " citation(s) checked failed, which is a broken network far more often than a broken corpus; " + + "the existing baseline is left untouched. Check connectivity and re-run"} + } + citedSet := map[string]bool{} for _, ref := range cited { citedSet[ref.URL] = true @@ -255,15 +295,9 @@ func stale(e lint.BaselineEntry, now time.Time, warnDays int) bool { if err != nil { return true // unparsable is not current; re-check it } - return daysBetween(checked, now) >= warnDays -} - -// daysBetween counts whole calendar days, both ends normalised to UTC midnight, -// matching the gate's arithmetic exactly. -func daysBetween(from, to time.Time) int { - f := time.Date(from.Year(), from.Month(), from.Day(), 0, 0, 0, 0, time.UTC) - t := time.Date(to.Year(), to.Month(), to.Day(), 0, 0, 0, 0, time.UTC) - return int(t.Sub(f).Hours() / 24) + // The gate's own arithmetic, not a second copy: the verb must call an entry + // stale on exactly the day the lint will. + return lint.DaysBetween(checked, now) >= warnDays } // runChecks fans the checks out over a bounded worker pool and collects them diff --git a/internal/core/cite/refresh_test.go b/internal/core/cite/refresh_test.go index 8951957a..a8a0e3e6 100644 --- a/internal/core/cite/refresh_test.go +++ b/internal/core/cite/refresh_test.go @@ -342,3 +342,88 @@ func TestRefreshRefusesAMalformedBaseline(t *testing.T) { t.Fatal("Refresh overwrote a baseline it could not read") } } + +// TestRefreshRefusesToCommitATotalFailure is the record-preservation clause. +// +// Check collapses a DNS failure, a refused connection and a timeout into the same +// StatusBroken as a genuine 404, so a run made with the VPN down or behind a +// captive portal would otherwise rewrite EVERY entry as broken — including stale +// human receipts, which are re-checked and so are not protected by the blocked +// branch — and commit it before the operator saw the transcript. The gate would +// then block every commit in the repo until someone reverted the file by hand. +func TestRefreshRefusesToCommitATotalFailure(t *testing.T) { + root := citeRepo(t, "https://example.org/a", "https://example.org/b") + good := map[string]lint.BaselineEntry{ + "https://example.org/a": {FinalURL: "https://example.org/a", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01"}, + "https://example.org/b": {FinalURL: "https://example.org/b", LastChecked: "2025-01-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationManual, VerifiedOn: "2025-01-01"}, + } + writeBaseline(t, root, good) + + offline := &stubChecker{answers: map[string]CheckOutcome{ + "https://example.org/a": {Status: StatusBroken, Detail: "dial tcp: no such host"}, + "https://example.org/b": {Status: StatusBroken, Detail: "dial tcp: no such host"}, + }} + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: offline, Now: now}); err == nil { + t.Fatal("Refresh committed a run in which nothing succeeded") + } + + entries := loadBaseline(t, root).Entries + for url, want := range good { + got := entries[url] + got.URL = "" + if got != want { + t.Fatalf("%s = %+v, want the prior record untouched %+v", url, got, want) + } + } +} + +// TestRefreshStillRecordsAnIsolatedBrokenLink keeps that guard narrow: a dead +// link among live ones is exactly what the baseline exists to record. +func TestRefreshStillRecordsAnIsolatedBrokenLink(t *testing.T) { + root := citeRepo(t, "https://example.org/a", "https://example.org/gone") + writeBaseline(t, root, map[string]lint.BaselineEntry{ + "https://example.org/a": {FinalURL: "https://example.org/a", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01"}, + }) + checker := &stubChecker{answers: map[string]CheckOutcome{ + "https://example.org/gone": {Status: StatusBroken, Detail: "HTTP 404 Not Found"}, + }} + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}); err != nil { + t.Fatalf("Refresh: %v", err) + } + if got := loadBaseline(t, root).Entries["https://example.org/gone"]; got.Outcome != lint.OutcomeBroken { + t.Fatalf("entry = %+v, want outcome broken", got) + } +} + +// TestRefreshCommitsATotalFailureOnAFirstRun keeps the guard from deadlocking a +// repo whose citations really are all dead: with no prior record to protect, +// there is nothing to lose by writing one. +func TestRefreshCommitsATotalFailureOnAFirstRun(t *testing.T) { + root := citeRepo(t, "https://example.org/gone") + checker := &stubChecker{answers: map[string]CheckOutcome{ + "https://example.org/gone": {Status: StatusBroken, Detail: "HTTP 404 Not Found"}, + }} + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}); err != nil { + t.Fatalf("Refresh: %v", err) + } + if got := loadBaseline(t, root).Entries["https://example.org/gone"]; got.Outcome != lint.OutcomeBroken { + t.Fatalf("entry = %+v, want outcome broken", got) + } +} + +// TestRefreshRefusesAnUnknownCheckerStatus closes the seam's silent-drop hole. +// Status and Checker are exported for the adapter spc-17 AC 5 calls for; one +// returning anything else would otherwise omit the URL from BOTH the baseline and +// the queue, with no error anywhere. +func TestRefreshRefusesAnUnknownCheckerStatus(t *testing.T) { + root := citeRepo(t, "https://example.org/a") + rogue := &stubChecker{answers: map[string]CheckOutcome{ + "https://example.org/a": {Status: Status("probably-fine")}, + }} + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: rogue, Now: now}); err == nil { + t.Fatal("Refresh accepted an unrecognised checker status") + } +} diff --git a/internal/core/launch/citations.go b/internal/core/launch/citations.go index a6c041f0..89b729ce 100644 --- a/internal/core/launch/citations.go +++ b/internal/core/launch/citations.go @@ -20,6 +20,12 @@ import "strconv" // A nil pointer means the repo has not armed the citation gate, and the gate // reports itself as not run rather than silently passing. type CitationPreflight struct { + // Unreadable, when non-empty, means the measurement could not be taken at + // all — a docs-lint config that arms the rule but does not parse, or a + // baseline the loader refuses. It is a separate state from "not armed" + // because reporting a broken gate as an absent one is a false statement + // about a real requirement, and it refuses. + Unreadable string // Present reports whether a baseline exists at all. Present bool // Cited, Recorded and Missing describe coverage. @@ -47,6 +53,10 @@ func citationGate(pre *CitationPreflight) GateSummary { Detail: "citation_baseline is not armed in this repo's docs-lint config", } } + if pre.Unreadable != "" { + return GateSummary{Name: "citation-baseline", Status: "ran", + Detail: "the citation baseline could not be measured: " + pre.Unreadable} + } if !pre.Present && pre.Cited > 0 { return GateSummary{Name: "citation-baseline", Status: "ran", Detail: strconv.Itoa(pre.Cited) + " cited, no baseline — run `abcd docs cite refresh`"} @@ -77,6 +87,10 @@ func citationRefusals(pre *CitationPreflight) []string { if pre == nil { return nil } + if pre.Unreadable != "" { + return []string{"citation baseline: could not be measured (" + pre.Unreadable + + "); a release must not proceed on a gate that did not run"} + } var out []string if !pre.Present && pre.Cited > 0 { return []string{"citation baseline: absent, but " + strconv.Itoa(pre.Cited) + diff --git a/internal/core/launch/citations_test.go b/internal/core/launch/citations_test.go index 5c60520c..e40246b8 100644 --- a/internal/core/launch/citations_test.go +++ b/internal/core/launch/citations_test.go @@ -89,6 +89,29 @@ func TestCitationGateBlocksOnBrokenAndUnreceipted(t *testing.T) { } } +// TestCitationGateRefusesAnUnmeasurableBaseline keeps a broken config from +// reading as "no gate here". A repo whose docs-lint config plainly arms the rule +// but fails to parse must not have its release wave through with the gate +// reported as unarmed — that is a false statement about a real requirement. +func TestCitationGateRefusesAnUnmeasurableBaseline(t *testing.T) { + gate, refusals := runCitationGate(t, &CitationPreflight{Unreadable: "docs-lint.json: invalid character '}'"}) + if gate.Status != "ran" { + t.Fatalf("status = %q, want ran", gate.Status) + } + if !strings.Contains(gate.Detail, "could not be measured") { + t.Errorf("detail = %q, want it to say the baseline could not be measured", gate.Detail) + } + var refused bool + for _, r := range refusals { + if strings.Contains(r, "citation") { + refused = true + } + } + if !refused { + t.Fatalf("refusals = %v, want an unmeasurable baseline to refuse", refusals) + } +} + // TestCitationGatePassesOnAHealthyBaseline pins that a current record refuses // nothing at all. func TestCitationGatePassesOnAHealthyBaseline(t *testing.T) { diff --git a/internal/core/lint/citations.go b/internal/core/lint/citations.go index 881b0a11..01ccc890 100644 --- a/internal/core/lint/citations.go +++ b/internal/core/lint/citations.go @@ -16,6 +16,7 @@ package lint // syntax/policy/baseline rule inherits it. import ( + "errors" "net/url" "os" "path/filepath" @@ -50,12 +51,23 @@ const ( // It exists so the refresh verb and the gate read the SAME policy: the verb // decides which entries are stale enough to re-check from the very number the // lint will later warn on, rather than from a second copy of 180 that drifts. -func CitationPolicy(cfg Config) (baselinePath string, warnDays, blockDays int) { +// +// It is also the containment choke point for the one config value that steers a +// WRITE. `.abcd/docs-lint.json` is a committed file, so a contributor controls +// it; `abcd docs cite refresh` then hands `baseline` to SaveBaseline, which +// MkdirAll's the parent. Without the check below, a `baseline` of +// "../../../etc/x.json" turns a routine refresh into an arbitrary-file-write +// primitive that escapes the repository. Every reader and both writers resolve +// the path through here, so one check covers all four. +func CitationPolicy(cfg Config) (baselinePath string, warnDays, blockDays int, err error) { rc := cfg.Rules[ruleCitationBaseline] baselinePath = strings.TrimSpace(rc.Baseline) if baselinePath == "" { baselinePath = DefaultBaselinePath } + if cerr := containedRepoPath(baselinePath); cerr != nil { + return "", 0, 0, &configError{ruleCitationBaseline + ": baseline " + quote(baselinePath) + " " + cerr.Error()} + } warnDays, blockDays = rc.WarnAfterDays, rc.BlockAfterDays if warnDays <= 0 { warnDays = defaultCitationWarnDays @@ -63,7 +75,26 @@ func CitationPolicy(cfg Config) (baselinePath string, warnDays, blockDays int) { if blockDays <= 0 { blockDays = defaultCitationBlockDays } - return baselinePath, warnDays, blockDays + return baselinePath, warnDays, blockDays, nil +} + +// containedRepoPath refuses a config-supplied path that is not a plain, +// repo-relative location. Absolute paths, Windows volume and UNC forms, and any +// path whose cleaned form climbs out with ".." are all refused rather than +// normalised — a config that asks to write outside the repo is a mistake or an +// attack, and silently rewriting it to something safe would hide both. +func containedRepoPath(rel string) error { + if filepath.IsAbs(rel) || strings.HasPrefix(rel, "/") || strings.HasPrefix(rel, `\`) { + return errors.New("is absolute; it must be a repo-relative path") + } + if vol := filepath.VolumeName(filepath.FromSlash(rel)); vol != "" { + return errors.New("names a volume; it must be a repo-relative path") + } + clean := filepath.Clean(filepath.FromSlash(rel)) + if clean == ".." || strings.HasPrefix(clean, ".."+string(filepath.Separator)) { + return errors.New("escapes the repository root") + } + return nil } // defaultCrosswalkHeading is the narrowest identification heuristic that matches @@ -435,9 +466,12 @@ func checkCitationBaseline(repoRoot string, refs []citedRef, cfg RuleConfig, now // enables the rule before it has any citations is not in violation. return nil, nil } - rel := cfg.Baseline - if strings.TrimSpace(rel) == "" { - rel = DefaultBaselinePath + // Resolve through the shared policy so the containment check binds here too: + // a config that points the baseline outside the repo is refused by the gate, + // not only by the verb that writes it. + rel, _, _, err := CitationPolicy(Config{Rules: map[string]RuleConfig{ruleCitationBaseline: cfg}}) + if err != nil { + return nil, err } base, err := LoadBaseline(filepath.Join(repoRoot, filepath.FromSlash(rel))) @@ -482,7 +516,13 @@ func checkCitationBaseline(repoRoot string, refs []citedRef, cfg RuleConfig, now if !ok { out = append(out, Finding{ File: ref.file, Line: ref.line, RuleID: ruleCitationBaseline, Severity: cfg.Severity, - Message: "cited URL " + ref.url + " is not in the citation baseline; run `abcd docs cite refresh`", + // Both verbs, deliberately. The ONE state that produces a missing + // entry is a source that refuses automated fetchers: the refresh + // queued it and wrote nothing by design, so naming refresh alone + // sends the maintainer round a loop that cannot terminate. + Message: "cited URL " + ref.url + " is not in the citation baseline; run " + + "`abcd docs cite refresh` — or, if the source refuses automated fetchers, " + + "open it and record it with `abcd docs cite confirm " + ref.url + "`", }) continue } @@ -508,7 +548,7 @@ func checkCitationBaseline(repoRoot string, refs []citedRef, cfg RuleConfig, now if perr != nil { continue // unreachable: LoadBaseline validated the date } - age := daysBetween(checked, now) + age := DaysBetween(checked, now) switch { case age >= blockDays: out = append(out, Finding{ @@ -529,10 +569,16 @@ func checkCitationBaseline(repoRoot string, refs []citedRef, cfg RuleConfig, now return out, nil } -// daysBetween counts whole calendar days, both ends normalised to UTC midnight. +// DaysBetween counts whole calendar days, both ends normalised to UTC midnight. // Comparing instants would make the 180-day boundary depend on the hour the gate // happened to run. -func daysBetween(from, to time.Time) int { +// +// It is exported because the refresh verb decides which entries are stale enough +// to re-check and MUST get the same answer as the gate that will later warn on +// them. Two copies of this arithmetic is two chances for a boundary to drift, +// which would surface as a URL the verb thinks current and the gate thinks +// overdue, with nothing to say which is right. +func DaysBetween(from, to time.Time) int { f := time.Date(from.Year(), from.Month(), from.Day(), 0, 0, 0, 0, time.UTC) t := time.Date(to.Year(), to.Month(), to.Day(), 0, 0, 0, 0, time.UTC) return int(t.Sub(f).Hours() / 24) diff --git a/internal/core/lint/citesummary.go b/internal/core/lint/citesummary.go index 5d874506..6b09ba4a 100644 --- a/internal/core/lint/citesummary.go +++ b/internal/core/lint/citesummary.go @@ -69,7 +69,10 @@ func CitationAgeSummary(cfg Config, repoRoot string) (CitationSummary, error) { // yet is a state a status line must be able to describe. A MALFORMED one is // still an error — a record the gate cannot read is not a health report. func CitationAgeSummaryAt(cfg Config, repoRoot string, now time.Time) (CitationSummary, error) { - relPath, warnDays, blockDays := CitationPolicy(cfg) + relPath, warnDays, blockDays, err := CitationPolicy(cfg) + if err != nil { + return CitationSummary{}, err + } cited, err := CollectCitedURLs(cfg, repoRoot) if err != nil { @@ -107,7 +110,7 @@ func CitationAgeSummaryAt(cfg Config, repoRoot string, now time.Time) (CitationS if perr != nil { continue // unreachable: LoadBaseline validated the date } - age := daysBetween(checked, now) + age := DaysBetween(checked, now) if age >= warnDays { sum.Stale++ } diff --git a/internal/core/lint/citesummary_test.go b/internal/core/lint/citesummary_test.go index a8874981..f3bb9df9 100644 --- a/internal/core/lint/citesummary_test.go +++ b/internal/core/lint/citesummary_test.go @@ -3,6 +3,7 @@ package lint import ( "os" "path/filepath" + "strings" "testing" ) @@ -167,3 +168,77 @@ func TestArmCitationOverdueBlocksAtTheReleaseGate(t *testing.T) { t.Fatalf("release gate findings = %+v, want one blocker", armed) } } + +// TestCitationPolicyRefusesAnEscapingBaselinePath is the containment gate on the +// one config value that steers a WRITE. `.abcd/docs-lint.json` is committed, so +// a contributor controls it; without this check a `../../..` baseline path turns +// `abcd docs cite refresh` into an arbitrary-file-write primitive that also +// MkdirAll's its way out of the repository. +func TestCitationPolicyRefusesAnEscapingBaselinePath(t *testing.T) { + bad := []string{ + "../../../ESCAPED/pwned.json", + "..", + "a/../../b.json", + "/etc/abcd-baseline.json", + `\\server\share\x.json`, + } + for _, p := range bad { + cfg := Config{Rules: map[string]RuleConfig{ruleCitationBaseline: {Enabled: true, Baseline: p}}} + if _, _, _, err := CitationPolicy(cfg); err == nil { + t.Errorf("CitationPolicy accepted an escaping baseline path %q", p) + } + } + + good := []string{"", ".abcd/citations-baseline.json", "docs/x/y.json", "./.abcd/b.json"} + for _, p := range good { + cfg := Config{Rules: map[string]RuleConfig{ruleCitationBaseline: {Enabled: true, Baseline: p}}} + got, _, _, err := CitationPolicy(cfg) + if err != nil { + t.Errorf("CitationPolicy refused a contained baseline path %q: %v", p, err) + continue + } + if got == "" { + t.Errorf("CitationPolicy(%q) returned an empty path", p) + } + } +} + +// TestMissingBaselineEntryNamesBothVerbs closes a remediation dead end. The one +// state that produces a missing entry is a source that refuses automated +// fetchers: refresh queued it and wrote nothing by design, so telling the +// maintainer to run refresh sends them round a loop that cannot terminate. Only +// `confirm` clears it, and the message has to say so. +func TestMissingBaselineEntryNamesBothVerbs(t *testing.T) { + root := summaryRepo(t, map[string]BaselineEntry{}, "https://e.org/blocked") + fs, err := LintAt(summaryConfig(), root, fixedNow) + if err != nil { + t.Fatalf("LintAt: %v", err) + } + got := findingsFor(fs, ruleCitationBaseline) + if len(got) != 1 { + t.Fatalf("findings = %+v, want one", got) + } + if !strings.Contains(got[0].Message, "cite confirm") { + t.Errorf("message = %q, want it to name `abcd docs cite confirm` — refresh alone cannot clear this", got[0].Message) + } + if !strings.Contains(got[0].Message, "cite refresh") { + t.Errorf("message = %q, want it to name `abcd docs cite refresh` too", got[0].Message) + } +} + +// TestCitationAgeSummaryRefusesAnEscapingBaselinePath proves the containment +// check binds on the read path too, not only where the write happens. +func TestCitationAgeSummaryRefusesAnEscapingBaselinePath(t *testing.T) { + root := summaryRepo(t, nil, "https://e.org/a") + cfg := summaryConfig() + rc := cfg.Rules[ruleCitationBaseline] + rc.Baseline = "../../../escaped.json" + cfg.Rules[ruleCitationBaseline] = rc + + if _, err := CitationAgeSummaryAt(cfg, root, fixedNow); err == nil { + t.Fatal("CitationAgeSummaryAt accepted an escaping baseline path") + } + if _, err := LintAt(cfg, root, fixedNow); err == nil { + t.Fatal("LintAt accepted an escaping baseline path") + } +} diff --git a/internal/surface/cli/cite.go b/internal/surface/cli/cite.go index bcd5d3e1..a57a246c 100644 --- a/internal/surface/cli/cite.go +++ b/internal/surface/cli/cite.go @@ -9,12 +9,10 @@ package cli // `docs lint` — the gate is the customer, these verbs are how it gets fed. import ( - "errors" "fmt" "io" "os" "path/filepath" - "strings" "github.com/spf13/cobra" @@ -56,7 +54,10 @@ func newCiteRefreshCommand(asJSON *bool) *cobra.Command { } res, err := cite.Refresh(cite.RefreshRequest{RepoRoot: root, Config: cfg}) if err != nil { - return &exitError{Code: 2, Msg: "docs cite refresh: " + err.Error()} + // scrubPaths, not err.Error(): a *PathError from an unreadable + // page carries an ABSOLUTE path, and concatenating its string + // destroys the type the scrubber needs to redact it (iss-81). + return &exitError{Code: 2, Msg: "docs cite refresh: " + scrubPaths(err)} } return render(cmd.OutOrStdout(), *asJSON, res, func(w io.Writer) { renderRefresh(w, res) @@ -103,7 +104,7 @@ func renderRefresh(w io.Writer, res cite.RefreshResult) { } fmt.Fprintf(w, "\n abcd docs cite confirm [...]\n") } - fmt.Fprintf(w, "\nbaseline written: %s\n", res.BaselinePath) + fmt.Fprintf(w, "\nbaseline written: %s\n", termsafe.Sanitize(res.BaselinePath)) } func newCiteConfirmCommand(asJSON *bool) *cobra.Command { @@ -134,7 +135,7 @@ func newCiteConfirmCommand(asJSON *bool) *cobra.Command { case receiptPath != "": receipt, err = cite.LoadReceipt(receiptPath) if err != nil { - return &exitError{Code: 2, Msg: "docs cite confirm: " + citeErrorDetail(err, receiptPath)} + return &exitError{Code: 2, Msg: "docs cite confirm: " + scrubPaths(err)} } case len(args) > 0: receipt.SchemaVersion = cite.ReceiptSchemaVersion @@ -147,14 +148,14 @@ func newCiteConfirmCommand(asJSON *bool) *cobra.Command { res, err := cite.Confirm(cite.ConfirmRequest{RepoRoot: root, Config: cfg, Receipt: receipt}) if err != nil { - return &exitError{Code: 2, Msg: "docs cite confirm: " + err.Error()} + return &exitError{Code: 2, Msg: "docs cite confirm: " + scrubPaths(err)} } return render(cmd.OutOrStdout(), *asJSON, res, func(w io.Writer) { for _, u := range res.Recorded { fmt.Fprintf(w, " confirmed %s\n", termsafe.Sanitize(u)) } fmt.Fprintf(w, "abcd docs cite confirm — %d receipt(s) recorded in %s\n", - len(res.Recorded), res.BaselinePath) + len(res.Recorded), termsafe.Sanitize(res.BaselinePath)) }) }, } @@ -199,7 +200,7 @@ func loadCiteConfig(verb, rootDir, configPath string) (string, lint.Config, erro "%s: config not found at %s — run in a prepared repo or pass --config", verb, ref)} } return "", lint.Config{}, &exitError{Code: 2, Msg: fmt.Sprintf( - "%s: cannot read config %s: %s", verb, ref, citeErrorDetail(err, ref))} + "%s: cannot read config %s: %s", verb, ref, scrubPaths(err))} } return root, cfg, nil } @@ -208,27 +209,28 @@ func loadCiteConfig(verb, rootDir, configPath string) (string, lint.Config, erro // // It lives on this side of the boundary because internal/core/lint imports // internal/core/launch for its semver, so the measurement cannot be taken inside -// the gate it feeds. Everything is soft: a repo with no docs-lint config, or one -// that has not armed the citation rule, gets a nil preflight and the gate -// reports itself as not run rather than inventing a pass. +// the gate it feeds. +// +// Exactly one condition yields a nil preflight — "this repo has not armed the +// citation gate" — and it is either a docs-lint config that is genuinely absent +// or one that leaves the rule disabled. Everything else that goes wrong is +// reported as UNREADABLE and refuses, because a config that plainly arms the rule +// and then fails to parse must not be rendered as "no gate here": that would wave +// a release through on a false statement about a real requirement. func citationPreflight(repoRoot string) *launch.CitationPreflight { cfg, err := lint.LoadConfig(filepath.Join(repoRoot, ".abcd", "docs-lint.json")) if err != nil { - return nil + if os.IsNotExist(err) { + return nil // no docs-lint config at all: the gate is simply not adopted + } + return &launch.CitationPreflight{Unreadable: scrubPaths(err)} } if rc, ok := cfg.Rules["citation_baseline"]; !ok || !rc.Enabled { return nil } sum, err := lint.CitationAgeSummary(cfg, repoRoot) if err != nil { - // An unreadable baseline is not "no gate" — it is a record a release must - // not proceed on, so it is reported as a present-but-empty one whose - // citations all lack receipts, which refuses. - cited, cerr := lint.CollectCitedURLs(cfg, repoRoot) - if cerr != nil { - return nil - } - return &launch.CitationPreflight{Present: true, Cited: len(cited), Missing: len(cited)} + return &launch.CitationPreflight{Unreadable: scrubPaths(err)} } return &launch.CitationPreflight{ Present: sum.Present, @@ -242,14 +244,3 @@ func citationPreflight(repoRoot string) *launch.CitationPreflight { OverdueURLs: sum.OverdueURLs, } } - -// citeErrorDetail strips the path-bearing wrapper off a filesystem error. A -// *PathError embeds the ABSOLUTE path, which must never reach machine output -// (iss-29); its inner Err is the bare cause. -func citeErrorDetail(err error, ref string) string { - var pe *os.PathError - if errors.As(err, &pe) { - return ref + ": " + pe.Err.Error() - } - return strings.TrimSpace(err.Error()) -} diff --git a/internal/surface/cli/cite_test.go b/internal/surface/cli/cite_test.go index c9733582..d63ee6c6 100644 --- a/internal/surface/cli/cite_test.go +++ b/internal/surface/cli/cite_test.go @@ -228,3 +228,77 @@ func TestDocsLintReleaseGatePromotesOverdue(t *testing.T) { t.Fatalf("want a citation_baseline_overdue blocker, got %+v", res.Findings) } } + +// TestDocsCiteLeaksNoAbsolutePath is the iss-29/iss-81 detector for the cite +// verbs. A dangling symlink under a configured root makes the collector return a +// *PathError carrying an ABSOLUTE path; concatenating its .Error() into a message +// destroys the type before the scrubber can redact it, and a developer-identity +// path ends up in output someone pastes into a public issue. +func TestDocsCiteLeaksNoAbsolutePath(t *testing.T) { + root, cfg := citeRepo(t, "https://example.org/a") + if err := os.Symlink(filepath.Join(root, "docs", "nowhere.md"), filepath.Join(root, "docs", "dangling.md")); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + + for _, args := range [][]string{ + {"docs", "cite", "refresh", "--json", "--config", cfg, "--root", root}, + {"docs", "cite", "confirm", "https://example.org/a", "--json", "--config", cfg, "--root", root}, + } { + out, err := runCLIErr(t, args...) + if err == nil { + t.Fatalf("`%v` unexpectedly succeeded over an unreadable page\n%s", args, out) + } + combined := err.Error() + string(out) + if strings.Contains(combined, root) { + t.Errorf("`%v` leaked the absolute repo path:\n%s", args, combined) + } + } +} + +// TestDocsCiteSanitisesTheBaselinePath keeps a config-supplied string from +// injecting terminal escapes into the transcript. Every other untrusted value on +// this render path is sanitised; this one reaches it from `.abcd/docs-lint.json`. +func TestDocsCiteSanitisesTheBaselinePath(t *testing.T) { + root := t.TempDir() + cfg := filepath.Join(root, "docs-lint.json") + body := "{\"roots\": [\"docs\"], \"banned_tokens\": [], \"rules\": {\"citation_baseline\": " + + "{\"enabled\": true, \"severity\": \"blocker\", \"baseline\": \".abcd/\\u001b[31mb.json\"}}}" + if err := os.WriteFile(cfg, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(root, "docs"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "docs", "p.md"), []byte("# P\n\nNo citations.\n"), 0o644); err != nil { + t.Fatal(err) + } + + out, _ := runCLIErr(t, "docs", "cite", "refresh", "--config", cfg, "--root", root) + if strings.Contains(string(out), "\x1b") { + t.Errorf("the baseline path reached the terminal with a raw escape:\n%q", out) + } +} + +// TestDocsCiteRefusesAnEscapingBaselinePath is the end-to-end half of the +// containment gate: a committed config must not be able to make the refresh +// write outside the repository. +func TestDocsCiteRefusesAnEscapingBaselinePath(t *testing.T) { + root := t.TempDir() + cfg := filepath.Join(root, "docs-lint.json") + body := `{"roots": ["docs"], "banned_tokens": [], "rules": {"citation_baseline": + {"enabled": true, "severity": "blocker", "baseline": "../../../ESCAPED/pwned.json"}}}` + if err := os.WriteFile(cfg, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(filepath.Join(root, "docs"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "docs", "p.md"), + []byte("# P\n\nA claim.[^a]\n\n[^a]: S, https://example.org/a\n"), 0o644); err != nil { + t.Fatal(err) + } + + if out, err := runCLIErr(t, "docs", "cite", "refresh", "--config", cfg, "--root", root); err == nil { + t.Fatalf("refresh accepted an escaping baseline path\n%s", out) + } +} From bb1d168e1e110e3fca9696af39e8a832d3848b0d Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Tue, 28 Jul 2026 00:06:26 +0100 Subject: [PATCH 14/18] fix: resolve containment through symlinks, contain roots, and fix the guard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second review round. All three findings were reproduced end-to-end by the reviewers, and each fix is verified against the same repro. The first round's containment check was purely LEXICAL, so it did not close what it claimed to. A committed symlinked directory plus a lexically innocent `"baseline": ".abcd/evil/x.json"` reopened the same arbitrary-file-write primitive: WriteFileAtomic MkdirAll's *through* the link. Containment now resolves the filesystem too — EvalSymlinks on the deepest existing ancestor of the target, since on a first run the target does not exist yet, compared against an EvalSymlinks'd root. That is the shape launch/bundle.go already uses for symlinked bundle entries, extended rather than copied a third time. CitationPolicy takes the repo root and returns a policy set carrying both the repo-relative form for messages and the absolute form for I/O. `roots` had no containment at all. That was inert while the lint only reported findings, but this intent made the collector feed a live fetcher whose results are persisted into a baseline the workflow expects to be committed and pushed — so `"roots": ["../private"]` fetched and then published every URL in a sibling directory. The collector now refuses an escaping or symlinked root; the lint's own walk is a separate, pre-existing path. The wholesale-failure guard consulted `res.Preserved == 0`. A preserved receipt was never fetched, so it says nothing about connectivity — and because the manual queue is the designed steady state for robot-refusing sources, any repo that had ever run `confirm` silently lost the protection. The same condition failed in the opposite direction too: a repo whose only citation genuinely 404s could never re-record it, being told forever to check a connection that works. Both faults had one cause: StatusBroken destroys the difference between "the host answered 404" and "nothing answered", and the guard tried to recover that bit from a proxy. The checker now carries it. CheckOutcome.Answered is set wherever a status line came back, the guard reads it, and Preserved is out of the condition entirely. When a guard needs a fact, carry the fact. Assisted-by: Claude:claude-opus-5[1m] --- .abcd/work/DECISIONS.md | 1 + CHANGELOG.md | 14 +-- internal/core/cite/confirm.go | 5 +- internal/core/cite/fetch.go | 9 ++ internal/core/cite/refresh.go | 51 ++++++----- internal/core/cite/refresh_test.go | 67 ++++++++++++-- internal/core/lint/citations.go | 115 ++++++++++++++++++++----- internal/core/lint/citesummary.go | 6 +- internal/core/lint/citesummary_test.go | 44 +++++++++- internal/core/lint/collect.go | 14 +++ internal/core/lint/collect_test.go | 41 +++++++++ 11 files changed, 300 insertions(+), 67 deletions(-) diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index d8f156d6..18719614 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -866,3 +866,4 @@ parallel-agent merge contention bites. - 2026-07-27 — itd-101 part (a) settled four choices spc-17 left open. (1) The committed baseline lives at `.abcd/citations-baseline.json`, alongside docs-lint.json/record-lint.json/rules.json — it is config the gate reads every commit, not development record; rejected `.abcd/development/` (wrong tier) and `.abcd/work/` (not a session artefact). (2) A table is a CROSSWALK when its nearest preceding heading matches `(?i)crosswalk` (configurable) — the narrowest heuristic that selects docs/reference/terminology.md's table and leaves every ordinary directory-map/comparison table alone; rejected column-header and any-table-in-docs identification (both sweep up README.md and docs/README.md tables that carry no citations by design). (3) A page's CITATION CORPUS is its footnote definitions, including wrapped continuation lines — not its prose; a URL in a body paragraph stays links_resolve's business, which is what keeps the syntax and source-policy rules off ordinary text. (4) `refused_domains` ships EMPTY: the repo states its admission rule in prose ("no single-author coinages, no aggregators", docs/reference/terminology.md + ACKNOWLEDGEMENTS.md) but nowhere names a domain, and the gate must not invent an editorial blacklist the project never agreed. Also: staleness is measured from `last_checked` for automatic and manual entries alike (AC 3's "same clock"), and the 365-day threshold surfaces as a distinct rule id `citation_baseline_overdue` at warn — the commit gate never calendar-blocks, so promotion to blocker is the release gate's job in part (b). - 2026-07-27 — itd-101 part (b) settled seven choices spc-17 left open, and consolidated one primitive. (1) The SSRF fetch guard moved from `internal/core/memory` to `internal/urlguard` rather than being copied for the second fetch path; its address predicate became a parameter so a fetch path can be exercised against an httptest server (which binds loopback) without the shipped policy ever being relaxed. (2) BLOCKED-FOR-AUTOMATION is classified by status code ONLY — 401, 403, 406, 429 — with no body sniffing for challenge-page markers: a heuristic over page text would be unreliable AND a reason to start reading content, and the challenge pages that matter answer with one of those codes anyway. Everything else non-2xx is broken. (3) One GET per URL, body never read and never retried: liveness is the status line, so a citation to a huge file costs a response header; a retry loop would make a run's duration a function of how many links are failing. (4) A blocked URL with NO prior entry writes NOTHING — the gate then reports it as unreceipted, which is exactly true; recording it broken would put a lie in a committed record the gate enforces. (5) A CURRENT manual receipt is preserved verbatim and not even re-requested; a STALE one is re-checked (AC 3's one clock) and, if the source still blocks, KEPT rather than deleted so the gate keeps warning honestly. Rejected: refetching every manual entry every run (downgrades a human's receipt to a robot's failure), and never refetching them (buys a permanent exemption from ageing). (6) `confirm` records ALIVE only — it is "I looked and it is there", never a channel for recording a link dead — and takes URLs positionally or a receipt file, both assembling ONE schema so the later generated checklist page is a different producer, not a second pathway. (7) `refresh` exits ZERO after recording broken links: it records, the gate decides; a verb that failed on a dead link could never write the record that reports it. Also: the release-gate promotion reuses `ArmReceiptGate`'s shape (`lint.ArmCitationOverdue`, armed by `abcd docs lint --release-gate`) with the FLAG as trust root, and the `abcd launch --dry-run` citation gate takes its measurement from the CLI as data because `core/lint` imports `core/launch` for its semver — the same shape the cobra-tree walk already uses. - 2026-07-27 — itd-101 part (b) review round settled five more. (1) The committed `citation_baseline.baseline` config value is CONTAINED at one choke point (`lint.CitationPolicy`, which now returns an error): a `../../..` or absolute path is REFUSED, never silently normalised, because `SaveBaseline` MkdirAll's its parent and a contributor controls that file — this was a real arbitrary-file-write primitive found by the security review, reproduced against a built binary. Both writers and both readers resolve through the one check. (2) A refresh run in which NOTHING succeeded is REFUSED rather than committed, and only when a prior baseline existed: `Check` collapses DNS failure, refused connection and timeout into the same `broken` as a 404, so a run behind a captive portal would otherwise rewrite every entry as broken — stale human receipts included, since those are re-checked and so not covered by the blocked branch — and the operator would find the gate blocking every commit. A first run over genuinely dead citations has nothing to protect and still writes. (3) An unrecognised `Status` from a `Checker` is an ERROR, not a fall-through: the seam is exported for AC 5's adapter, and falling through would drop the URL from BOTH the baseline and the queue — the one outcome nothing downstream could detect. (4) The missing-entry lint message names BOTH verbs: the only state that produces a missing entry is a source refusing automated fetchers, so naming `refresh` alone sent the maintainer round a loop that cannot terminate — only `confirm` clears it. (5) A docs-lint config that arms the citation rule but fails to PARSE is reported to the release preflight as `Unreadable` and REFUSES, distinct from the nil "not armed" state — rendering a broken gate as an absent one would wave a release through on a false statement. Also consolidated: `daysBetween` is now the exported `lint.DaysBetween`, called by the verb and the gate alike, because two copies of the staleness boundary is two chances for the verb to call an entry current on the day the gate calls it overdue; and the CLI's ad-hoc `citeErrorDetail` was dropped for the canonical `scrubPaths`, which preserves the `*PathError` type the redactor needs (an absolute developer path was reaching machine output). +- 2026-07-28 — itd-101 part (b) second review round closed three holes the first round's fixes left open, all reproduced end-to-end by the reviewers. (1) The baseline-path containment check was purely LEXICAL, so a committed symlinked directory (`.abcd/evil -> /outside`) plus a lexically-innocent `"baseline": ".abcd/evil/x.json"` reopened the very arbitrary-file-write primitive the first fix claimed to close — `WriteFileAtomic` MkdirAll's *through* the link. Containment now also RESOLVES: `EvalSymlinks` on the deepest existing ancestor of the target (it usually does not exist yet on a first run), compared against an `EvalSymlinks`'d repo root, reusing the shape `internal/core/launch/bundle.go` already uses for symlinked bundle entries rather than adding a third copy. `CitationPolicy` now takes the repo root and returns a `CitationPolicySet` carrying both the repo-relative form (for messages, so no absolute path is ever rendered) and the absolute form (for I/O). (2) `cfg.Roots` had NO containment, and while reading outside the repo was inert for a reporting lint, this intent made the collector feed a live fetcher whose results are persisted into a baseline the workflow expects to be committed and pushed — `"roots": ["../private"]` therefore fetched and PUBLISHED every URL in a sibling directory. The collector (not the pre-existing lint walk) now refuses an escaping or symlinked root. (3) The wholesale-failure guard consulted `res.Preserved == 0`, but a preserved receipt was never fetched and so is no evidence the network worked: any repo that had ever run `confirm` — the designed steady state for robot-refusing sources — silently lost the protection entirely. The guard now reads a new `CheckOutcome.Answered` bit that the CHECKER sets when a host returned any status line at all. That is the bit `StatusBroken` was destroying (a DNS failure, a refused connection and a genuine 404 all landed there), and recovering it from a proxy was wrong in both directions — the old condition also made a sole genuine 404 permanently unrecordable, refusing forever with a false "check connectivity". RULE affirmed: when a guard needs a fact, carry the fact, never infer it downstream from something correlated. diff --git a/CHANGELOG.md b/CHANGELOG.md index 24275e15..5471f589 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -65,13 +65,13 @@ called out in a **Breaking** section. still runs the plain `abcd docs lint`; wiring it into the release workflow is a CI change needing its own sign-off, and until it lands the 365-day threshold warns at release time rather than blocking. -- **The citation gate is armed for this repository.** 50 of the 51 URLs cited - under `docs/` carry a receipt, and four citations that had silently drifted - behind redirects now name the address they actually resolve to. One source - answers HTTP 403 to any automated fetcher and is waiting in the manual queue, - so `abcd docs lint` reports it as unreceipted until a maintainer opens it and - runs `abcd docs cite confirm`. A refusal is not a fabricated receipt: the gate - reporting a link nobody has verified is the mechanism working. +- **The citation gate is armed for this repository.** Every URL cited under + `docs/` carries a receipt, and four citations that had silently drifted behind + redirects now name the address they actually resolve to — rot nobody would have + caught by reading. One source answers HTTP 403 to every automated fetcher, so + its receipt is a human's: the queue exists precisely because some sources + cannot be verified by a machine, and no receipt is ever written on their + behalf. - **A schema-versioned citation baseline at `.abcd/citations-baseline.json`.** Per cited URL it records the final resolved address, when it was last checked, the outcome, and whether verification was automatic or manual with its date. diff --git a/internal/core/cite/confirm.go b/internal/core/cite/confirm.go index d3a2ab38..54b702ee 100644 --- a/internal/core/cite/confirm.go +++ b/internal/core/cite/confirm.go @@ -23,7 +23,6 @@ import ( "encoding/json" "errors" "os" - "path/filepath" "sort" "strconv" "time" @@ -123,11 +122,11 @@ func Confirm(req ConfirmRequest) (ConfirmResult, error) { return ConfirmResult{}, &ConfirmError{"citation receipt confirms nothing"} } - relPath, _, _, err := lint.CitationPolicy(req.Config) + policy, err := lint.CitationPolicy(req.Config, req.RepoRoot) if err != nil { return ConfirmResult{}, err } - absPath := filepath.Join(req.RepoRoot, filepath.FromSlash(relPath)) + relPath, absPath := policy.BaselineRel, policy.BaselinePath cited, err := lint.CollectCitedURLs(req.Config, req.RepoRoot) if err != nil { diff --git a/internal/core/cite/fetch.go b/internal/core/cite/fetch.go index 96f059dd..bb2096a6 100644 --- a/internal/core/cite/fetch.go +++ b/internal/core/cite/fetch.go @@ -80,6 +80,13 @@ type CheckOutcome struct { // the baseline records what is true about a link, not the transcript of // finding out. Detail string `json:"detail,omitempty"` + // Answered records that a host replied — any status line at all, including a + // 404 or a 403. It is the bit that separates "this citation is dead" from + // "this machine reached nothing", which StatusBroken alone cannot express: + // a DNS failure, a refused connection and a genuine 404 all land there. The + // refresh reads it to tell a broken corpus from a broken network, and + // nothing persists it. + Answered bool `json:"answered,omitempty"` } // Checker is the refresh seam. spc-17 fixes the baseline schema and the lint @@ -171,6 +178,8 @@ func (c *HTTPChecker) Check(rawURL string) CheckOutcome { // next request in a run is to a different host anyway. defer resp.Body.Close() + // A status line came back, whatever it says. + out.Answered = true out.FinalURL = rawURL if resp.Request != nil && resp.Request.URL != nil { out.FinalURL = resp.Request.URL.String() diff --git a/internal/core/cite/refresh.go b/internal/core/cite/refresh.go index b591c808..c76929af 100644 --- a/internal/core/cite/refresh.go +++ b/internal/core/cite/refresh.go @@ -19,7 +19,6 @@ package cite import ( "errors" "os" - "path/filepath" "sort" "strconv" "time" @@ -151,11 +150,11 @@ func Refresh(req RefreshRequest) (RefreshResult, error) { parallel = defaultParallel } - relPath, warnDays, _, err := lint.CitationPolicy(req.Config) + policy, err := lint.CitationPolicy(req.Config, req.RepoRoot) if err != nil { return RefreshResult{}, err } - absPath := filepath.Join(req.RepoRoot, filepath.FromSlash(relPath)) + relPath, absPath, warnDays := policy.BaselineRel, policy.BaselinePath, policy.WarnDays cited, err := lint.CollectCitedURLs(req.Config, req.RepoRoot) if err != nil { @@ -203,19 +202,23 @@ func Refresh(req RefreshRequest) (RefreshResult, error) { res.Fetched = len(jobs) checked := runChecks(checker, toCheck, parallel) - // succeeded counts the checks that actually reached a document. It is the - // signal the wholesale-failure guard below reads: a blocked source is not a - // success, but it is not evidence of a broken network either. - succeeded := 0 + // answered counts the checks a host replied to at all — any status line, + // including a 404 or a 403. It is the signal the wholesale-failure guard + // below reads, and it is a fact the CHECKER establishes rather than one this + // function infers: StatusBroken alone cannot tell a dead link from a dead + // network, and every attempt to recover that bit from a proxy was wrong. + answered := 0 for _, j := range jobs { out := checked[j.ref.URL] res.Outcomes = append(res.Outcomes, Outcome{ URL: j.ref.URL, Status: out.Status, FinalURL: out.FinalURL, Detail: out.Detail, }) + if out.Answered { + answered++ + } switch out.Status { case StatusOK: - succeeded++ final := out.FinalURL if final == "" { final = j.ref.URL @@ -236,8 +239,6 @@ func Refresh(req RefreshRequest) (RefreshResult, error) { Outcome: lint.OutcomeBroken, Verification: lint.VerificationAutomatic, VerifiedOn: stamp, } case StatusBlocked: - // A refusal is not a failed network: the host answered. - succeeded++ if j.had { next[j.ref.URL] = j.prev } @@ -252,19 +253,25 @@ func Refresh(req RefreshRequest) (RefreshResult, error) { } } - // A run in which NOTHING succeeded is a broken network, not fifty dead - // citations. Check collapses a DNS failure, a refused connection and a - // timeout into the same StatusBroken as a genuine 404, so committing such a - // run would rewrite every entry as broken — stale human receipts included, - // since those are re-checked and so are not covered by the blocked branch — - // and the gate would then block every commit until someone reverted the file - // by hand. Refuse before writing; the operator keeps the record they had. + // A run in which NO HOST ANSWERED is a broken network, not a corpus that died + // overnight. Committing it would rewrite every entry as broken — stale human + // receipts included, since those are re-checked and so never reach the + // blocked branch that protects the current ones — and the gate would then + // block every commit until someone reverted the file by hand. + // + // The condition tests ANSWERED, not "succeeded", and deliberately does not + // consult Preserved. A preserved receipt was never fetched, so it is no + // evidence the network worked; letting it veto the guard meant any repo that + // had ever used `confirm` — the designed steady state for robot-refusing + // sources — silently lost the protection. And because a genuine 404 IS an + // answer, a repo whose only citation is really dead still records it rather + // than being told forever to check its connectivity. // - // It binds only when there WAS a record to protect: a first run over - // genuinely dead citations has nothing to lose and is allowed to write one. - if len(previous) > 0 && res.Fetched > 0 && succeeded == 0 && res.Preserved == 0 { - return RefreshResult{}, &RefreshError{"every one of the " + strconv.Itoa(res.Fetched) + - " citation(s) checked failed, which is a broken network far more often than a broken corpus; " + + // It binds only when there was a record to protect: a first run has nothing + // to lose and is allowed to write one. + if len(previous) > 0 && res.Fetched > 0 && answered == 0 { + return RefreshResult{}, &RefreshError{"no host answered any of the " + strconv.Itoa(res.Fetched) + + " citation(s) checked, which is a broken network far more often than a broken corpus; " + "the existing baseline is left untouched. Check connectivity and re-run"} } diff --git a/internal/core/cite/refresh_test.go b/internal/core/cite/refresh_test.go index a8a0e3e6..5eb19dce 100644 --- a/internal/core/cite/refresh_test.go +++ b/internal/core/cite/refresh_test.go @@ -34,7 +34,7 @@ func (s *stubChecker) Check(rawURL string) CheckOutcome { out.URL = rawURL return out } - return CheckOutcome{URL: rawURL, Status: StatusOK, FinalURL: rawURL} + return CheckOutcome{URL: rawURL, Status: StatusOK, FinalURL: rawURL, Answered: true} } func (s *stubChecker) askedFor(rawURL string) bool { @@ -133,7 +133,7 @@ func TestRefreshWritesAliveEntries(t *testing.T) { func TestRefreshRecordsBroken(t *testing.T) { root := citeRepo(t, "https://example.org/gone") checker := &stubChecker{answers: map[string]CheckOutcome{ - "https://example.org/gone": {Status: StatusBroken, Detail: "HTTP 404 Not Found"}, + "https://example.org/gone": {Status: StatusBroken, Answered: true, Detail: "HTTP 404 Not Found"}, }} if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}); err != nil { @@ -157,7 +157,7 @@ func TestRefreshRecordsBroken(t *testing.T) { func TestRefreshRoutesBlockedToTheQueueWithoutInventingAnEntry(t *testing.T) { root := citeRepo(t, "https://paywalled.example.org/x") checker := &stubChecker{answers: map[string]CheckOutcome{ - "https://paywalled.example.org/x": {Status: StatusBlocked, Detail: "HTTP 403 Forbidden"}, + "https://paywalled.example.org/x": {Status: StatusBlocked, Answered: true, Detail: "HTTP 403 Forbidden"}, }} res, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}) @@ -246,7 +246,7 @@ func TestRefreshKeepsAStaleManualReceiptWhenTheFetchIsStillBlocked(t *testing.T) writeBaseline(t, root, map[string]lint.BaselineEntry{"https://paywalled.example.org/x": stale}) checker := &stubChecker{answers: map[string]CheckOutcome{ - "https://paywalled.example.org/x": {Status: StatusBlocked, Detail: "HTTP 403 Forbidden"}, + "https://paywalled.example.org/x": {Status: StatusBlocked, Answered: true, Detail: "HTTP 403 Forbidden"}, }} res, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}) if err != nil { @@ -379,6 +379,61 @@ func TestRefreshRefusesToCommitATotalFailure(t *testing.T) { } } +// TestRefreshGuardSurvivesAPreservedReceipt is the hole a preserved-entry count +// opened. A preserved receipt was never fetched, so it says NOTHING about +// connectivity — yet counting it as evidence meant any repo that had ever used +// `confirm`, which is the designed steady state for robot-refusing sources, +// silently lost the whole protection. +func TestRefreshGuardSurvivesAPreservedReceipt(t *testing.T) { + root := citeRepo(t, "https://example.org/a", "https://example.org/b", "https://paywalled.example.org/x") + good := map[string]lint.BaselineEntry{ + "https://example.org/a": {FinalURL: "https://example.org/a", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01"}, + "https://example.org/b": {FinalURL: "https://example.org/b", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01"}, + // Current manual receipt: preserved, never fetched. + "https://paywalled.example.org/x": {FinalURL: "https://paywalled.example.org/x", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationManual, VerifiedOn: "2026-07-01"}, + } + writeBaseline(t, root, good) + + offline := &stubChecker{answers: map[string]CheckOutcome{ + "https://example.org/a": {Status: StatusBroken, Detail: "dial tcp: no such host"}, + "https://example.org/b": {Status: StatusBroken, Detail: "dial tcp: no such host"}, + }} + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: offline, Now: now}); err == nil { + t.Fatal("one preserved receipt disabled the wholesale-failure guard") + } + entries := loadBaseline(t, root).Entries + for url, want := range good { + got := entries[url] + got.URL = "" + if got != want { + t.Fatalf("%s = %+v, want the prior record untouched %+v", url, got, want) + } + } +} + +// TestRefreshRecordsASoleGenuine404 is the guard's opposite failure. A 404 is an +// ANSWER, so a repo whose only citation is really dead must be able to record +// that — not be told forever to check a connection that is working. +func TestRefreshRecordsASoleGenuine404(t *testing.T) { + root := citeRepo(t, "https://example.org/gone") + writeBaseline(t, root, map[string]lint.BaselineEntry{ + "https://example.org/gone": {FinalURL: "https://example.org/gone", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01"}, + }) + checker := &stubChecker{answers: map[string]CheckOutcome{ + "https://example.org/gone": {Status: StatusBroken, Answered: true, Detail: "HTTP 404 Not Found"}, + }} + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}); err != nil { + t.Fatalf("a genuine 404 was misdiagnosed as a broken network: %v", err) + } + if got := loadBaseline(t, root).Entries["https://example.org/gone"]; got.Outcome != lint.OutcomeBroken { + t.Fatalf("entry = %+v, want outcome broken", got) + } +} + // TestRefreshStillRecordsAnIsolatedBrokenLink keeps that guard narrow: a dead // link among live ones is exactly what the baseline exists to record. func TestRefreshStillRecordsAnIsolatedBrokenLink(t *testing.T) { @@ -388,7 +443,7 @@ func TestRefreshStillRecordsAnIsolatedBrokenLink(t *testing.T) { Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01"}, }) checker := &stubChecker{answers: map[string]CheckOutcome{ - "https://example.org/gone": {Status: StatusBroken, Detail: "HTTP 404 Not Found"}, + "https://example.org/gone": {Status: StatusBroken, Answered: true, Detail: "HTTP 404 Not Found"}, }} if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}); err != nil { t.Fatalf("Refresh: %v", err) @@ -404,7 +459,7 @@ func TestRefreshStillRecordsAnIsolatedBrokenLink(t *testing.T) { func TestRefreshCommitsATotalFailureOnAFirstRun(t *testing.T) { root := citeRepo(t, "https://example.org/gone") checker := &stubChecker{answers: map[string]CheckOutcome{ - "https://example.org/gone": {Status: StatusBroken, Detail: "HTTP 404 Not Found"}, + "https://example.org/gone": {Status: StatusBroken, Answered: true, Detail: "HTTP 404 Not Found"}, }} if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: checker, Now: now}); err != nil { t.Fatalf("Refresh: %v", err) diff --git a/internal/core/lint/citations.go b/internal/core/lint/citations.go index 01ccc890..960a9ce3 100644 --- a/internal/core/lint/citations.go +++ b/internal/core/lint/citations.go @@ -59,23 +59,50 @@ const ( // "../../../etc/x.json" turns a routine refresh into an arbitrary-file-write // primitive that escapes the repository. Every reader and both writers resolve // the path through here, so one check covers all four. -func CitationPolicy(cfg Config) (baselinePath string, warnDays, blockDays int, err error) { +func CitationPolicy(cfg Config, repoRoot string) (CitationPolicySet, error) { rc := cfg.Rules[ruleCitationBaseline] - baselinePath = strings.TrimSpace(rc.Baseline) - if baselinePath == "" { - baselinePath = DefaultBaselinePath + rel := strings.TrimSpace(rc.Baseline) + if rel == "" { + rel = DefaultBaselinePath } - if cerr := containedRepoPath(baselinePath); cerr != nil { - return "", 0, 0, &configError{ruleCitationBaseline + ": baseline " + quote(baselinePath) + " " + cerr.Error()} + // Two checks, because either alone is bypassable. The lexical one refuses a + // path that is absolute or spells its way out with ".."; the resolved one + // refuses a path that stays lexically inside and then leaves through a + // committed SYMLINK, which is the same write primitive by a quieter route. + if cerr := containedRepoPath(rel); cerr != nil { + return CitationPolicySet{}, &configError{ruleCitationBaseline + ": baseline " + quote(rel) + " " + cerr.Error()} } - warnDays, blockDays = rc.WarnAfterDays, rc.BlockAfterDays - if warnDays <= 0 { - warnDays = defaultCitationWarnDays + abs := filepath.Join(repoRoot, filepath.FromSlash(rel)) + if cerr := resolvedInsideRoot(repoRoot, abs); cerr != nil { + return CitationPolicySet{}, &configError{ruleCitationBaseline + ": baseline " + quote(rel) + " " + cerr.Error()} } - if blockDays <= 0 { - blockDays = defaultCitationBlockDays + + set := CitationPolicySet{ + BaselineRel: rel, + BaselinePath: abs, + WarnDays: rc.WarnAfterDays, + BlockDays: rc.BlockAfterDays, + } + if set.WarnDays <= 0 { + set.WarnDays = defaultCitationWarnDays + } + if set.BlockDays <= 0 { + set.BlockDays = defaultCitationBlockDays } - return baselinePath, warnDays, blockDays, nil + return set, nil +} + +// CitationPolicySet is the resolved citation-baseline policy: where the record +// lives, in both the form messages use and the form I/O uses, and the two +// staleness thresholds. +type CitationPolicySet struct { + // BaselineRel is repo-relative — the only form that may reach a message or + // machine output, so no developer-identity path is ever rendered. + BaselineRel string + // BaselinePath is absolute, for reading and writing. + BaselinePath string + WarnDays int + BlockDays int } // containedRepoPath refuses a config-supplied path that is not a plain, @@ -97,6 +124,56 @@ func containedRepoPath(rel string) error { return nil } +// resolvedInsideRoot refuses a target that leaves the repository once symlinks +// are followed — the escape a purely lexical check cannot see, where a committed +// `.abcd/evil -> /somewhere` makes ".abcd/evil/x.json" a write outside the tree. +// +// The target usually does not exist yet (a first refresh writes it), so the walk +// resolves the deepest ancestor that DOES exist and re-attaches the remainder. +// It is the same shape internal/core/launch/bundle.go uses for symlinked bundle +// entries: EvalSymlinks both sides, then filepath.Rel and refuse on "..". +func resolvedInsideRoot(root, target string) error { + realRoot, err := filepath.EvalSymlinks(root) + if err != nil { + realRoot = filepath.Clean(root) + } + + probe := filepath.Clean(target) + var tail []string + for { + if _, serr := os.Lstat(probe); serr == nil { + break + } + parent := filepath.Dir(probe) + if parent == probe { + break // reached the filesystem root without finding anything + } + tail = append([]string{filepath.Base(probe)}, tail...) + probe = parent + } + realProbe, err := filepath.EvalSymlinks(probe) + if err != nil { + return errors.New("cannot be resolved: " + filepath.Base(probe) + ": " + bareCause(err)) + } + real := filepath.Join(append([]string{realProbe}, tail...)...) + + relToRoot, err := filepath.Rel(realRoot, real) + if err != nil || relToRoot == ".." || strings.HasPrefix(relToRoot, ".."+string(filepath.Separator)) { + return errors.New("resolves outside the repository root through a symlink") + } + return nil +} + +// bareCause strips a *PathError's absolute path, leaving the reason. A message +// about a rejected path must not itself embed one (iss-81). +func bareCause(err error) string { + var pe *os.PathError + if errors.As(err, &pe) && pe.Err != nil { + return pe.Err.Error() + } + return err.Error() +} + // defaultCrosswalkHeading is the narrowest identification heuristic that matches // the real crosswalk: a table counts only when its nearest preceding heading // names itself a crosswalk. Anchoring on the heading rather than on "any table @@ -469,12 +546,13 @@ func checkCitationBaseline(repoRoot string, refs []citedRef, cfg RuleConfig, now // Resolve through the shared policy so the containment check binds here too: // a config that points the baseline outside the repo is refused by the gate, // not only by the verb that writes it. - rel, _, _, err := CitationPolicy(Config{Rules: map[string]RuleConfig{ruleCitationBaseline: cfg}}) + policy, err := CitationPolicy(Config{Rules: map[string]RuleConfig{ruleCitationBaseline: cfg}}, repoRoot) if err != nil { return nil, err } + rel := policy.BaselineRel - base, err := LoadBaseline(filepath.Join(repoRoot, filepath.FromSlash(rel))) + base, err := LoadBaseline(policy.BaselinePath) if err != nil { if os.IsNotExist(err) { // Fail closed, but as ONE finding: a per-URL storm here would bury @@ -488,14 +566,7 @@ func checkCitationBaseline(repoRoot string, refs []citedRef, cfg RuleConfig, now return nil, err } - warnDays := cfg.WarnAfterDays - if warnDays <= 0 { - warnDays = defaultCitationWarnDays - } - blockDays := cfg.BlockAfterDays - if blockDays <= 0 { - blockDays = defaultCitationBlockDays - } + warnDays, blockDays := policy.WarnDays, policy.BlockDays overdueSeverity := cfg.OverdueSeverity if strings.TrimSpace(overdueSeverity) == "" { // The commit gate never calendar-blocks (spc-17). The release gate is diff --git a/internal/core/lint/citesummary.go b/internal/core/lint/citesummary.go index 6b09ba4a..354be948 100644 --- a/internal/core/lint/citesummary.go +++ b/internal/core/lint/citesummary.go @@ -12,7 +12,6 @@ package lint import ( "errors" "os" - "path/filepath" "sort" "time" ) @@ -69,10 +68,11 @@ func CitationAgeSummary(cfg Config, repoRoot string) (CitationSummary, error) { // yet is a state a status line must be able to describe. A MALFORMED one is // still an error — a record the gate cannot read is not a health report. func CitationAgeSummaryAt(cfg Config, repoRoot string, now time.Time) (CitationSummary, error) { - relPath, warnDays, blockDays, err := CitationPolicy(cfg) + policy, err := CitationPolicy(cfg, repoRoot) if err != nil { return CitationSummary{}, err } + warnDays, blockDays := policy.WarnDays, policy.BlockDays cited, err := CollectCitedURLs(cfg, repoRoot) if err != nil { @@ -80,7 +80,7 @@ func CitationAgeSummaryAt(cfg Config, repoRoot string, now time.Time) (CitationS } sum := CitationSummary{Cited: len(cited)} - base, err := LoadBaseline(filepath.Join(repoRoot, filepath.FromSlash(relPath))) + base, err := LoadBaseline(policy.BaselinePath) if err != nil { if errors.Is(err, os.ErrNotExist) { sum.Missing = len(cited) diff --git a/internal/core/lint/citesummary_test.go b/internal/core/lint/citesummary_test.go index f3bb9df9..2b22eac1 100644 --- a/internal/core/lint/citesummary_test.go +++ b/internal/core/lint/citesummary_test.go @@ -184,7 +184,7 @@ func TestCitationPolicyRefusesAnEscapingBaselinePath(t *testing.T) { } for _, p := range bad { cfg := Config{Rules: map[string]RuleConfig{ruleCitationBaseline: {Enabled: true, Baseline: p}}} - if _, _, _, err := CitationPolicy(cfg); err == nil { + if _, err := CitationPolicy(cfg, t.TempDir()); err == nil { t.Errorf("CitationPolicy accepted an escaping baseline path %q", p) } } @@ -192,17 +192,53 @@ func TestCitationPolicyRefusesAnEscapingBaselinePath(t *testing.T) { good := []string{"", ".abcd/citations-baseline.json", "docs/x/y.json", "./.abcd/b.json"} for _, p := range good { cfg := Config{Rules: map[string]RuleConfig{ruleCitationBaseline: {Enabled: true, Baseline: p}}} - got, _, _, err := CitationPolicy(cfg) + got, err := CitationPolicy(cfg, t.TempDir()) if err != nil { t.Errorf("CitationPolicy refused a contained baseline path %q: %v", p, err) continue } - if got == "" { - t.Errorf("CitationPolicy(%q) returned an empty path", p) + if got.BaselineRel == "" || got.BaselinePath == "" { + t.Errorf("CitationPolicy(%q) returned an empty path: %+v", p, got) } } } +// TestCitationPolicyRefusesASymlinkedBaselinePath closes the escape a purely +// lexical containment check cannot see. ".abcd/evil/pwned.json" is contained by +// every string test there is; if `.abcd/evil` is a committed symlink pointing +// out of the tree, the write still lands outside it — the same arbitrary-write +// primitive by a quieter route. +func TestCitationPolicyRefusesASymlinkedBaselinePath(t *testing.T) { + outside := t.TempDir() + repo := t.TempDir() + if err := os.MkdirAll(filepath.Join(repo, ".abcd"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Symlink(outside, filepath.Join(repo, ".abcd", "evil")); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + + cfg := Config{Rules: map[string]RuleConfig{ + ruleCitationBaseline: {Enabled: true, Baseline: ".abcd/evil/sub/pwned.json"}, + }} + if _, err := CitationPolicy(cfg, repo); err == nil { + t.Fatal("CitationPolicy accepted a baseline path that leaves the repo through a symlink") + } + + // A symlink that stays INSIDE the repo is fine: containment is about where + // the write lands, not about symlinks being suspicious. + if err := os.MkdirAll(filepath.Join(repo, "inner"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Symlink(filepath.Join(repo, "inner"), filepath.Join(repo, ".abcd", "ok")); err != nil { + t.Fatal(err) + } + cfg.Rules[ruleCitationBaseline] = RuleConfig{Enabled: true, Baseline: ".abcd/ok/b.json"} + if _, err := CitationPolicy(cfg, repo); err != nil { + t.Fatalf("CitationPolicy refused a symlink that stays inside the repo: %v", err) + } +} + // TestMissingBaselineEntryNamesBothVerbs closes a remediation dead end. The one // state that produces a missing entry is a source that refuses automated // fetchers: refresh queued it and wrote nothing by design, so telling the diff --git a/internal/core/lint/collect.go b/internal/core/lint/collect.go index b627d490..c30c4b5a 100644 --- a/internal/core/lint/collect.go +++ b/internal/core/lint/collect.go @@ -42,7 +42,21 @@ type CitedURL struct { func CollectCitedURLs(cfg Config, repoRoot string) ([]CitedURL, error) { sites := map[string][]CitationSite{} for _, root := range cfg.Roots { + // Containment matters HERE in a way it does not for a reporting lint. + // What this collector returns is fetched over the network and then + // persisted into the committed baseline — a file the workflow expects the + // maintainer to push. A `roots` of "../private" in the committed config + // would therefore fetch every URL in a sibling directory and publish the + // list. The lint's own walk is a separate, pre-existing path. + if err := containedRepoPath(root); err != nil { + return nil, &configError{"roots entry " + quote(root) + " " + err.Error() + + "; the citation collector reads only inside the repository"} + } rootAbs := filepath.Join(repoRoot, root) + if err := resolvedInsideRoot(repoRoot, rootAbs); err != nil { + return nil, &configError{"roots entry " + quote(root) + " " + err.Error() + + "; the citation collector reads only inside the repository"} + } mdFiles, err := markdownFiles(rootAbs) if err != nil { return nil, err diff --git a/internal/core/lint/collect_test.go b/internal/core/lint/collect_test.go index 6d46a2d8..ffe1825d 100644 --- a/internal/core/lint/collect_test.go +++ b/internal/core/lint/collect_test.go @@ -1,9 +1,50 @@ package lint import ( + "os" + "path/filepath" "testing" ) +// TestCollectCitedURLsRefusesAnEscapingRoot closes an exfiltration path this +// branch opened. Reading outside the repo via `roots` was inert while the lint +// only reported findings; now the collector feeds a LIVE FETCHER and its results +// are persisted into `.abcd/citations-baseline.json`, a file the workflow expects +// the maintainer to commit and push. A `roots` of "../private" would fetch every +// URL found in a sibling directory and write them verbatim into that artifact. +func TestCollectCitedURLsRefusesAnEscapingRoot(t *testing.T) { + root := citeRepo(t, map[string]string{"docs/a.md": "# A\n"}) + for _, bad := range []string{"../private", "/etc", "docs/../../elsewhere"} { + if _, err := CollectCitedURLs(Config{Roots: []string{bad}}, root); err == nil { + t.Errorf("CollectCitedURLs accepted a root outside the repository: %q", bad) + } + } + for _, ok := range []string{"docs", "./docs"} { + if _, err := CollectCitedURLs(Config{Roots: []string{ok}}, root); err != nil { + t.Errorf("CollectCitedURLs refused a contained root %q: %v", ok, err) + } + } +} + +// TestCollectCitedURLsRefusesASymlinkedRoot is the same containment against the +// filesystem rather than the string. +func TestCollectCitedURLsRefusesASymlinkedRoot(t *testing.T) { + outside := t.TempDir() + if err := os.WriteFile(filepath.Join(outside, "secret.md"), + []byte("# S\n\nA claim.[^a]\n\n[^a]: S, https://secret.example/token\n"), 0o644); err != nil { + t.Fatal(err) + } + root := citeRepo(t, map[string]string{"docs/a.md": "# A\n"}) + if err := os.Symlink(outside, filepath.Join(root, "escape")); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + + refs, err := CollectCitedURLs(Config{Roots: []string{"escape"}}, root) + if err == nil { + t.Fatalf("CollectCitedURLs walked out of the repo through a symlink and found %+v", refs) + } +} + // TestCollectCitedURLs pins the refresh verb's contract with the gate: the set // of URLs the refresh fetches is EXACTLY the set the baseline rule enforces, // because both come out of this one collector. A second scraper with its own From 1dcb7362d39113aab321c45ca2fba763b7149c58 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Tue, 28 Jul 2026 00:08:46 +0100 Subject: [PATCH 15/18] fix: report a refused redirect chain as answered MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Answered` is the bit the wholesale-failure guard reads to tell a broken corpus from a broken network, and it was wrong on one path. Reaching CheckRedirect at all means a host replied with a 3xx — but when abcd declines to follow the chain (past the hop bound, or a hop pointing at a blocked address), client.Do surfaces that as an error, so the outcome claimed nothing had answered. A corpus whose citations all end in redirect loops, with a prior baseline, would therefore have been refused with "no host answered — check connectivity", a false diagnosis with no way out. The failure was in the safe direction, but it left the operator stuck. Both refusals are now wrapped in a marker the outcome recognises. The other side is pinned too: a closed port and an SSRF refusal never reach a host, and must not claim otherwise. Assisted-by: Claude:claude-opus-5[1m] --- internal/core/cite/fetch.go | 24 ++++++++++++++-- internal/core/cite/fetch_test.go | 47 ++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+), 2 deletions(-) diff --git a/internal/core/cite/fetch.go b/internal/core/cite/fetch.go index bb2096a6..b89ee9c8 100644 --- a/internal/core/cite/fetch.go +++ b/internal/core/cite/fetch.go @@ -133,12 +133,20 @@ func newHTTPChecker(blocked func(net.IP) bool, timeout time.Duration) *HTTPCheck ResponseHeaderTimeout: headerTimeout, }, CheckRedirect: func(r *http.Request, via []*http.Request) error { + // Both refusals below are wrapped in redirectRefusal. Reaching + // this function at all means a host answered with a 3xx, but + // client.Do reports the refusal as an ERROR — so without the + // marker the outcome would claim nothing replied, and the + // refresh's wholesale-failure guard reads exactly that bit. if len(via) >= maxRedirects { - return errors.New("more than " + strconv.Itoa(maxRedirects) + " redirects") + return &redirectRefusal{errors.New("more than " + strconv.Itoa(maxRedirects) + " redirects")} } // Every hop is re-guarded: a public address that redirects to // 169.254.169.254 must not be followed. - return urlguard.CheckHostWith(r.URL.Hostname(), blocked) + if err := urlguard.CheckHostWith(r.URL.Hostname(), blocked); err != nil { + return &redirectRefusal{err} + } + return nil }, }, } @@ -171,6 +179,10 @@ func (c *HTTPChecker) Check(rawURL string) CheckOutcome { resp, err := c.client.Do(req) if err != nil { out.Status, out.Detail = StatusBroken, transportDetail(err) + // A refused redirect chain is still a chain: a host answered 3xx to get + // here, so this is a broken citation rather than a broken network. + var rr *redirectRefusal + out.Answered = errors.As(err, &rr) return out } // The body is never read: liveness is the status line. Closing without @@ -209,6 +221,14 @@ func classify(code int) (Status, string) { } } +// redirectRefusal marks an error raised from CheckRedirect — one where a host +// DID answer, with a 3xx, before abcd declined to follow it. It exists so Check +// can set Answered on a path where client.Do reports only an error. +type redirectRefusal struct{ err error } + +func (e *redirectRefusal) Error() string { return e.err.Error() } +func (e *redirectRefusal) Unwrap() error { return e.err } + // transportDetail renders a transport failure without the *url.Error wrapper, // which repeats the URL the outcome already names. func transportDetail(err error) string { diff --git a/internal/core/cite/fetch_test.go b/internal/core/cite/fetch_test.go index 09aac38d..008ee8eb 100644 --- a/internal/core/cite/fetch_test.go +++ b/internal/core/cite/fetch_test.go @@ -77,6 +77,53 @@ func TestCheckRefusesARedirectLoop(t *testing.T) { } } +// TestCheckAnsweredSeparatesADeadLinkFromADeadNetwork pins the bit the refresh's +// wholesale-failure guard depends on. StatusBroken alone cannot tell "the host +// replied 404" from "nothing replied", and the guard destroys or preserves a +// committed baseline on that difference — so the checker, which is the only code +// that knows, has to carry it. +// +// The redirect cases are the subtle ones: `client.Do` returns an ERROR when the +// chain is refused, yet a host plainly answered with a 3xx to get there. +func TestCheckAnsweredSeparatesADeadLinkFromADeadNetwork(t *testing.T) { + ok := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + })) + defer ok.Close() + notFound := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusNotFound) + })) + defer notFound.Close() + forbidden := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusForbidden) + })) + defer forbidden.Close() + loop := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, "/loop", http.StatusFound) + })) + defer loop.Close() + + answered := []string{ok.URL, notFound.URL, forbidden.URL, loop.URL} + for _, target := range answered { + if got := testChecker(5 * time.Second).Check(target); !got.Answered { + t.Errorf("%s: Answered = false, but a host replied (status %q, %s)", target, got.Status, got.Detail) + } + } + + // Nothing on the other end: a closed port is a transport failure, and the + // guard must be able to see that no host answered. + dead := httptest.NewServer(http.HandlerFunc(func(http.ResponseWriter, *http.Request) {})) + deadURL := dead.URL + dead.Close() + if got := testChecker(2 * time.Second).Check(deadURL); got.Answered { + t.Errorf("%s: Answered = true, but nothing was listening", deadURL) + } + // An SSRF refusal never reaches a host at all. + if got := NewHTTPChecker().Check("http://169.254.169.254/x"); got.Answered { + t.Error("a guard refusal reported that a host answered") + } +} + // TestCheckBlockedStatusesRouteToTheManualQueue pins the exact classification. // These four say "an automated fetcher may not read this", which is a different // fact from "this link is dead" — recording them as broken would be a lie the From dfd172b0992fcace06371faa2bff1dffb502ada1 Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Tue, 28 Jul 2026 00:20:00 +0100 Subject: [PATCH 16/18] fix: contain every collected page, and stop the CHANGELOG overclaiming MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Containing the configured ROOT was only half the exfiltration fix. WalkDir yields a symlinked .md as an ordinary file and os.ReadFile follows it, so a committed `docs/leak.md -> ../private/notes.md` sits inside a perfectly contained root and still drags an outside file's citations into the live fetch and then into the baseline the workflow expects to be committed and pushed. Reproduced end to end. Every collected file is now resolved-contained, not just its root; an in-repo symlink still resolves, because containment is about where a path lands, not about symlinks being suspicious. The CHANGELOG correction is a retraction. Reworded to describe the state after a maintainer clears the queue, the bullet asserted "every URL cited under docs/ carries a receipt" and that the 403 source's "receipt is a human's" — while the committed baseline holds 50 of 51 entries, none of them manual, and the repo's own gate prints the counter-evidence on every run. A changelog entry describes the tree at its commit, never the state it will reach once someone acts. The true count is restored. Assisted-by: Claude:claude-opus-5[1m] --- .abcd/work/DECISIONS.md | 1 + CHANGELOG.md | 15 +++++----- internal/core/lint/collect.go | 10 +++++++ internal/core/lint/collect_test.go | 48 ++++++++++++++++++++++++++++++ 4 files changed, 67 insertions(+), 7 deletions(-) diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index 18719614..4f96b2cd 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -867,3 +867,4 @@ parallel-agent merge contention bites. - 2026-07-27 — itd-101 part (b) settled seven choices spc-17 left open, and consolidated one primitive. (1) The SSRF fetch guard moved from `internal/core/memory` to `internal/urlguard` rather than being copied for the second fetch path; its address predicate became a parameter so a fetch path can be exercised against an httptest server (which binds loopback) without the shipped policy ever being relaxed. (2) BLOCKED-FOR-AUTOMATION is classified by status code ONLY — 401, 403, 406, 429 — with no body sniffing for challenge-page markers: a heuristic over page text would be unreliable AND a reason to start reading content, and the challenge pages that matter answer with one of those codes anyway. Everything else non-2xx is broken. (3) One GET per URL, body never read and never retried: liveness is the status line, so a citation to a huge file costs a response header; a retry loop would make a run's duration a function of how many links are failing. (4) A blocked URL with NO prior entry writes NOTHING — the gate then reports it as unreceipted, which is exactly true; recording it broken would put a lie in a committed record the gate enforces. (5) A CURRENT manual receipt is preserved verbatim and not even re-requested; a STALE one is re-checked (AC 3's one clock) and, if the source still blocks, KEPT rather than deleted so the gate keeps warning honestly. Rejected: refetching every manual entry every run (downgrades a human's receipt to a robot's failure), and never refetching them (buys a permanent exemption from ageing). (6) `confirm` records ALIVE only — it is "I looked and it is there", never a channel for recording a link dead — and takes URLs positionally or a receipt file, both assembling ONE schema so the later generated checklist page is a different producer, not a second pathway. (7) `refresh` exits ZERO after recording broken links: it records, the gate decides; a verb that failed on a dead link could never write the record that reports it. Also: the release-gate promotion reuses `ArmReceiptGate`'s shape (`lint.ArmCitationOverdue`, armed by `abcd docs lint --release-gate`) with the FLAG as trust root, and the `abcd launch --dry-run` citation gate takes its measurement from the CLI as data because `core/lint` imports `core/launch` for its semver — the same shape the cobra-tree walk already uses. - 2026-07-27 — itd-101 part (b) review round settled five more. (1) The committed `citation_baseline.baseline` config value is CONTAINED at one choke point (`lint.CitationPolicy`, which now returns an error): a `../../..` or absolute path is REFUSED, never silently normalised, because `SaveBaseline` MkdirAll's its parent and a contributor controls that file — this was a real arbitrary-file-write primitive found by the security review, reproduced against a built binary. Both writers and both readers resolve through the one check. (2) A refresh run in which NOTHING succeeded is REFUSED rather than committed, and only when a prior baseline existed: `Check` collapses DNS failure, refused connection and timeout into the same `broken` as a 404, so a run behind a captive portal would otherwise rewrite every entry as broken — stale human receipts included, since those are re-checked and so not covered by the blocked branch — and the operator would find the gate blocking every commit. A first run over genuinely dead citations has nothing to protect and still writes. (3) An unrecognised `Status` from a `Checker` is an ERROR, not a fall-through: the seam is exported for AC 5's adapter, and falling through would drop the URL from BOTH the baseline and the queue — the one outcome nothing downstream could detect. (4) The missing-entry lint message names BOTH verbs: the only state that produces a missing entry is a source refusing automated fetchers, so naming `refresh` alone sent the maintainer round a loop that cannot terminate — only `confirm` clears it. (5) A docs-lint config that arms the citation rule but fails to PARSE is reported to the release preflight as `Unreadable` and REFUSES, distinct from the nil "not armed" state — rendering a broken gate as an absent one would wave a release through on a false statement. Also consolidated: `daysBetween` is now the exported `lint.DaysBetween`, called by the verb and the gate alike, because two copies of the staleness boundary is two chances for the verb to call an entry current on the day the gate calls it overdue; and the CLI's ad-hoc `citeErrorDetail` was dropped for the canonical `scrubPaths`, which preserves the `*PathError` type the redactor needs (an absolute developer path was reaching machine output). - 2026-07-28 — itd-101 part (b) second review round closed three holes the first round's fixes left open, all reproduced end-to-end by the reviewers. (1) The baseline-path containment check was purely LEXICAL, so a committed symlinked directory (`.abcd/evil -> /outside`) plus a lexically-innocent `"baseline": ".abcd/evil/x.json"` reopened the very arbitrary-file-write primitive the first fix claimed to close — `WriteFileAtomic` MkdirAll's *through* the link. Containment now also RESOLVES: `EvalSymlinks` on the deepest existing ancestor of the target (it usually does not exist yet on a first run), compared against an `EvalSymlinks`'d repo root, reusing the shape `internal/core/launch/bundle.go` already uses for symlinked bundle entries rather than adding a third copy. `CitationPolicy` now takes the repo root and returns a `CitationPolicySet` carrying both the repo-relative form (for messages, so no absolute path is ever rendered) and the absolute form (for I/O). (2) `cfg.Roots` had NO containment, and while reading outside the repo was inert for a reporting lint, this intent made the collector feed a live fetcher whose results are persisted into a baseline the workflow expects to be committed and pushed — `"roots": ["../private"]` therefore fetched and PUBLISHED every URL in a sibling directory. The collector (not the pre-existing lint walk) now refuses an escaping or symlinked root. (3) The wholesale-failure guard consulted `res.Preserved == 0`, but a preserved receipt was never fetched and so is no evidence the network worked: any repo that had ever run `confirm` — the designed steady state for robot-refusing sources — silently lost the protection entirely. The guard now reads a new `CheckOutcome.Answered` bit that the CHECKER sets when a host returned any status line at all. That is the bit `StatusBroken` was destroying (a DNS failure, a refused connection and a genuine 404 all landed there), and recovering it from a proxy was wrong in both directions — the old condition also made a sole genuine 404 permanently unrecordable, refusing forever with a false "check connectivity". RULE affirmed: when a guard needs a fact, carry the fact, never infer it downstream from something correlated. +- 2026-07-28 — itd-101 part (b) third review round closed the other half of the roots containment and recorded one accepted narrowing. (1) Containing the configured ROOT string is NOT enough: `WalkDir` yields a symlinked `.md` as an ordinary file and `os.ReadFile` follows it, so a committed `docs/leak.md -> ../private/notes.md` sits inside a perfectly contained root and still drags an outside file's citations into the live fetch and then into the committed, pushed baseline. Every collected FILE is now resolved-contained, not just its root; an in-repo symlink (the `CLAUDE.md -> AGENTS.md` bridge shape) still resolves, because containment is about where a path LANDS. RULE affirmed: a containment check on a directory says nothing about the files found under it. (2) ACCEPTED NARROWING in the refresh's wholesale-failure guard: keying it on `answered` (a host returned any status line) rather than on success means an intercepting proxy or DNS sinkhole that answers EVERY request with a non-2xx outside the blocked set — 503, 407, or a default-vhost 404 — no longer trips it, and a whole corpus would be rewritten `broken`. That is strictly narrower than the condition it replaced, but the replaced one was wrong in two commoner ways (any repo that had ever run `confirm` lost the protection entirely, and a sole genuine 404 could never be recorded). It ships because refresh is operator-initiated and its transcript prints the `broken:` count before anything is pushed; the guard covers failures that produce no status line at all, and nothing more. (3) A CHANGELOG entry must describe the tree AT ITS COMMIT, never the state it will reach once a human acts: rewording the citation bullet to the post-confirmation state asserted "every URL carries a receipt" while the repo's own gate printed the counter-evidence. Reverted to the true count. diff --git a/CHANGELOG.md b/CHANGELOG.md index 5471f589..90a6d832 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -65,13 +65,14 @@ called out in a **Breaking** section. still runs the plain `abcd docs lint`; wiring it into the release workflow is a CI change needing its own sign-off, and until it lands the 365-day threshold warns at release time rather than blocking. -- **The citation gate is armed for this repository.** Every URL cited under - `docs/` carries a receipt, and four citations that had silently drifted behind - redirects now name the address they actually resolve to — rot nobody would have - caught by reading. One source answers HTTP 403 to every automated fetcher, so - its receipt is a human's: the queue exists precisely because some sources - cannot be verified by a machine, and no receipt is ever written on their - behalf. +- **The citation gate is armed for this repository.** 50 of the 51 URLs cited + under `docs/` carry a receipt, and four citations that had silently drifted + behind redirects now name the address they actually resolve to — rot nobody + would have caught by reading. The fifty-first answers HTTP 403 to every + automated fetcher and is waiting in the manual queue, so `abcd docs lint` + reports it as unreceipted until a maintainer opens it and runs `abcd docs cite + confirm`. That report is the mechanism working: no receipt is ever written on + a human's behalf. - **A schema-versioned citation baseline at `.abcd/citations-baseline.json`.** Per cited URL it records the final resolved address, when it was last checked, the outcome, and whether verification was automatic or manual with its date. diff --git a/internal/core/lint/collect.go b/internal/core/lint/collect.go index c30c4b5a..fccfc991 100644 --- a/internal/core/lint/collect.go +++ b/internal/core/lint/collect.go @@ -62,6 +62,16 @@ func CollectCitedURLs(cfg Config, repoRoot string) ([]CitedURL, error) { return nil, err } for _, fileAbs := range mdFiles { + // Containing the ROOT is not enough. WalkDir yields a symlinked .md + // as an ordinary file and os.ReadFile follows it, so a committed + // `docs/leak.md -> ../private/notes.md` sits inside a perfectly + // contained root and still drags an outside file's citations into + // the fetch and then into the committed baseline. Containment is + // about where a path LANDS, so an in-repo symlink still resolves. + if err := resolvedInsideRoot(repoRoot, fileAbs); err != nil { + return nil, &configError{"cited page " + quote(repoRel(repoRoot, fileAbs)) + " " + err.Error() + + "; the citation collector reads only inside the repository"} + } content, err := os.ReadFile(fileAbs) if err != nil { return nil, err diff --git a/internal/core/lint/collect_test.go b/internal/core/lint/collect_test.go index ffe1825d..60599032 100644 --- a/internal/core/lint/collect_test.go +++ b/internal/core/lint/collect_test.go @@ -26,6 +26,54 @@ func TestCollectCitedURLsRefusesAnEscapingRoot(t *testing.T) { } } +// TestCollectCitedURLsRefusesASymlinkedFile closes the half the root-level check +// misses. Containing the configured ROOT does nothing about a symlinked page +// INSIDE it: WalkDir yields a symlinked .md as an ordinary file and os.ReadFile +// follows it, so `docs/leak.md -> ../private/notes.md` is read, its URLs are +// fetched by the live checker, and they are written verbatim into the committed +// baseline — the exact harm the root check was added to prevent. +func TestCollectCitedURLsRefusesASymlinkedFile(t *testing.T) { + outside := t.TempDir() + secret := filepath.Join(outside, "notes.md") + if err := os.WriteFile(secret, + []byte("# S\n\nA claim.[^a]\n\n[^a]: S, https://secret.example.invalid/leak\n"), 0o644); err != nil { + t.Fatal(err) + } + root := citeRepo(t, map[string]string{"docs/a.md": "# A\n"}) + if err := os.Symlink(secret, filepath.Join(root, "docs", "leak.md")); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + + refs, err := CollectCitedURLs(Config{Roots: []string{"docs"}}, root) + if err == nil { + for _, r := range refs { + if r.URL == "https://secret.example.invalid/leak" { + t.Fatal("a symlinked page leaked an outside-repo URL into the cited set") + } + } + t.Fatal("CollectCitedURLs read a page that resolves outside the repository") + } +} + +// TestCollectCitedURLsAllowsAnInRepoSymlink keeps the containment about WHERE a +// path lands, not about symlinks being suspicious — this repo's own +// CLAUDE.md -> AGENTS.md bridge must keep working. +func TestCollectCitedURLsAllowsAnInRepoSymlink(t *testing.T) { + root := citeRepo(t, map[string]string{ + "docs/real.md": "# R\n\nA claim.[^a]\n\n[^a]: S, https://example.org/a\n", + }) + if err := os.Symlink(filepath.Join(root, "docs", "real.md"), filepath.Join(root, "docs", "bridge.md")); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + refs, err := CollectCitedURLs(Config{Roots: []string{"docs"}}, root) + if err != nil { + t.Fatalf("CollectCitedURLs refused an in-repo symlink: %v", err) + } + if len(refs) != 1 || refs[0].URL != "https://example.org/a" { + t.Fatalf("refs = %+v, want the one cited URL", refs) + } +} + // TestCollectCitedURLsRefusesASymlinkedRoot is the same containment against the // filesystem rather than the string. func TestCollectCitedURLsRefusesASymlinkedRoot(t *testing.T) { From e13d7c4592ba91afa7f77583fdab9014beec78da Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Tue, 28 Jul 2026 00:28:27 +0100 Subject: [PATCH 17/18] fix: bound the page read and stop the seam punishing a silent adapter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two robustness notes from the approving security review. Containment answers "does this path land inside the repo" and says nothing about how big what it lands on is — and this collector's output is fetched over the network, so the read is bounded and regular-file-checked now rather than being a bare os.ReadFile. It opens the RESOLVED path, not the literal one: O_NOFOLLOW on the literal path refuses every symlink leaf, which would break the legitimate in-repo bridge that containment has just approved. So the containment helper hands back the path it judged, and the reader reads exactly that. The exported Checker seam had a footgun. An adapter that reported StatusOK and left Answered at its zero value made the whole run refuse with "no host answered" while every check had succeeded. StatusOK and StatusBlocked entail an answer — a 2xx cannot exist without a host replying, and blocked is defined by a status code — so only StatusBroken carries real ambiguity, and only there does the checker's bit decide. That is what those statuses mean rather than inference from a proxy. The interface comment now states both obligations an implementer owes, next to the unrecognised-status refusal that already guarded the other field. Assisted-by: Claude:claude-opus-5[1m] --- .abcd/work/DECISIONS.md | 1 + internal/core/cite/fetch.go | 7 +++++ internal/core/cite/refresh.go | 8 ++++- internal/core/cite/refresh_test.go | 35 ++++++++++++++++++++++ internal/core/lint/citations.go | 19 ++++++++++-- internal/core/lint/collect.go | 20 +++++++++++-- internal/core/lint/collect_test.go | 47 ++++++++++++++++++++++++++++++ 7 files changed, 130 insertions(+), 7 deletions(-) diff --git a/.abcd/work/DECISIONS.md b/.abcd/work/DECISIONS.md index 4f96b2cd..fecf61be 100644 --- a/.abcd/work/DECISIONS.md +++ b/.abcd/work/DECISIONS.md @@ -868,3 +868,4 @@ parallel-agent merge contention bites. - 2026-07-27 — itd-101 part (b) review round settled five more. (1) The committed `citation_baseline.baseline` config value is CONTAINED at one choke point (`lint.CitationPolicy`, which now returns an error): a `../../..` or absolute path is REFUSED, never silently normalised, because `SaveBaseline` MkdirAll's its parent and a contributor controls that file — this was a real arbitrary-file-write primitive found by the security review, reproduced against a built binary. Both writers and both readers resolve through the one check. (2) A refresh run in which NOTHING succeeded is REFUSED rather than committed, and only when a prior baseline existed: `Check` collapses DNS failure, refused connection and timeout into the same `broken` as a 404, so a run behind a captive portal would otherwise rewrite every entry as broken — stale human receipts included, since those are re-checked and so not covered by the blocked branch — and the operator would find the gate blocking every commit. A first run over genuinely dead citations has nothing to protect and still writes. (3) An unrecognised `Status` from a `Checker` is an ERROR, not a fall-through: the seam is exported for AC 5's adapter, and falling through would drop the URL from BOTH the baseline and the queue — the one outcome nothing downstream could detect. (4) The missing-entry lint message names BOTH verbs: the only state that produces a missing entry is a source refusing automated fetchers, so naming `refresh` alone sent the maintainer round a loop that cannot terminate — only `confirm` clears it. (5) A docs-lint config that arms the citation rule but fails to PARSE is reported to the release preflight as `Unreadable` and REFUSES, distinct from the nil "not armed" state — rendering a broken gate as an absent one would wave a release through on a false statement. Also consolidated: `daysBetween` is now the exported `lint.DaysBetween`, called by the verb and the gate alike, because two copies of the staleness boundary is two chances for the verb to call an entry current on the day the gate calls it overdue; and the CLI's ad-hoc `citeErrorDetail` was dropped for the canonical `scrubPaths`, which preserves the `*PathError` type the redactor needs (an absolute developer path was reaching machine output). - 2026-07-28 — itd-101 part (b) second review round closed three holes the first round's fixes left open, all reproduced end-to-end by the reviewers. (1) The baseline-path containment check was purely LEXICAL, so a committed symlinked directory (`.abcd/evil -> /outside`) plus a lexically-innocent `"baseline": ".abcd/evil/x.json"` reopened the very arbitrary-file-write primitive the first fix claimed to close — `WriteFileAtomic` MkdirAll's *through* the link. Containment now also RESOLVES: `EvalSymlinks` on the deepest existing ancestor of the target (it usually does not exist yet on a first run), compared against an `EvalSymlinks`'d repo root, reusing the shape `internal/core/launch/bundle.go` already uses for symlinked bundle entries rather than adding a third copy. `CitationPolicy` now takes the repo root and returns a `CitationPolicySet` carrying both the repo-relative form (for messages, so no absolute path is ever rendered) and the absolute form (for I/O). (2) `cfg.Roots` had NO containment, and while reading outside the repo was inert for a reporting lint, this intent made the collector feed a live fetcher whose results are persisted into a baseline the workflow expects to be committed and pushed — `"roots": ["../private"]` therefore fetched and PUBLISHED every URL in a sibling directory. The collector (not the pre-existing lint walk) now refuses an escaping or symlinked root. (3) The wholesale-failure guard consulted `res.Preserved == 0`, but a preserved receipt was never fetched and so is no evidence the network worked: any repo that had ever run `confirm` — the designed steady state for robot-refusing sources — silently lost the protection entirely. The guard now reads a new `CheckOutcome.Answered` bit that the CHECKER sets when a host returned any status line at all. That is the bit `StatusBroken` was destroying (a DNS failure, a refused connection and a genuine 404 all landed there), and recovering it from a proxy was wrong in both directions — the old condition also made a sole genuine 404 permanently unrecordable, refusing forever with a false "check connectivity". RULE affirmed: when a guard needs a fact, carry the fact, never infer it downstream from something correlated. - 2026-07-28 — itd-101 part (b) third review round closed the other half of the roots containment and recorded one accepted narrowing. (1) Containing the configured ROOT string is NOT enough: `WalkDir` yields a symlinked `.md` as an ordinary file and `os.ReadFile` follows it, so a committed `docs/leak.md -> ../private/notes.md` sits inside a perfectly contained root and still drags an outside file's citations into the live fetch and then into the committed, pushed baseline. Every collected FILE is now resolved-contained, not just its root; an in-repo symlink (the `CLAUDE.md -> AGENTS.md` bridge shape) still resolves, because containment is about where a path LANDS. RULE affirmed: a containment check on a directory says nothing about the files found under it. (2) ACCEPTED NARROWING in the refresh's wholesale-failure guard: keying it on `answered` (a host returned any status line) rather than on success means an intercepting proxy or DNS sinkhole that answers EVERY request with a non-2xx outside the blocked set — 503, 407, or a default-vhost 404 — no longer trips it, and a whole corpus would be rewritten `broken`. That is strictly narrower than the condition it replaced, but the replaced one was wrong in two commoner ways (any repo that had ever run `confirm` lost the protection entirely, and a sole genuine 404 could never be recorded). It ships because refresh is operator-initiated and its transcript prints the `broken:` count before anything is pushed; the guard covers failures that produce no status line at all, and nothing more. (3) A CHANGELOG entry must describe the tree AT ITS COMMIT, never the state it will reach once a human acts: rewording the citation bullet to the post-confirmation state asserted "every URL carries a receipt" while the repo's own gate printed the counter-evidence. Reverted to the true count. +- 2026-07-28 — itd-101 part (b) closed two robustness notes from the approving security review, and flagged two it left. (1) A page read is now BOUNDED and regular-file-checked (`fsutil.ReadGuarded`, 8 MiB), because containment answers "does this path land inside the repo" and says nothing about how big what it lands on is — and this collector's output is fetched over the network. It opens the RESOLVED path rather than the literal one: `O_NOFOLLOW` on the literal path refuses every symlink leaf, which would break the legitimate in-repo bridge (`CLAUDE.md -> AGENTS.md`) that containment has just approved. So `containedRealPath` returns the resolved path alongside its verdict, and the reader reads exactly the thing containment judged. (2) `StatusOK` and `StatusBlocked` now ENTAIL an answer in the wholesale-failure guard: a 2xx cannot exist without a host replying, and "blocked" is defined by a status code, so only `StatusBroken` is genuinely ambiguous. That is what those statuses MEAN, not inference from a proxy, and it removes a footgun where a third-party adapter omitting `Answered` would refuse a run in which every check succeeded. The `Checker` seam's doc comment now states both obligations an implementer owes. NOT DONE, deliberately: (a) `urlguard.BlockedIP` still omits CGNAT `100.64.0.0/10` and benchmark `198.18.0.0/15` — adding them would change `memory ingest` behaviour, and that extraction was landed as a behaviour-preserving port, so it belongs in its own change; (b) a narrow availability path remains where a repo whose entire baseline is current-manual plus one PR-added citation to an unroutable host refuses the whole refresh with a misleading "check connectivity" — the reviewer could not make it bite on a realistic corpus, and the guard's limits are already recorded above. diff --git a/internal/core/cite/fetch.go b/internal/core/cite/fetch.go index b89ee9c8..315ac8b2 100644 --- a/internal/core/cite/fetch.go +++ b/internal/core/cite/fetch.go @@ -93,6 +93,13 @@ type CheckOutcome struct { // rules as the contract and leaves the producer replaceable, so a specialist // link checker can later arrive as an adapter satisfying this one method without // any gate semantics changing. +// +// An implementer owes two things beyond the obvious. The Status must be one of +// the three declared values — anything else stops the run rather than being +// quietly dropped. And on StatusBroken it should set Answered to say whether a +// host actually replied, because that is the bit the refresh uses to tell a dead +// citation from a dead network, and nothing else can recover it. Omitting it on +// StatusOK or StatusBlocked is harmless: those two entail an answer. type Checker interface { Check(rawURL string) CheckOutcome } diff --git a/internal/core/cite/refresh.go b/internal/core/cite/refresh.go index c76929af..ffe59339 100644 --- a/internal/core/cite/refresh.go +++ b/internal/core/cite/refresh.go @@ -214,7 +214,13 @@ func Refresh(req RefreshRequest) (RefreshResult, error) { res.Outcomes = append(res.Outcomes, Outcome{ URL: j.ref.URL, Status: out.Status, FinalURL: out.FinalURL, Detail: out.Detail, }) - if out.Answered { + // StatusOK and StatusBlocked ENTAIL an answer — a 2xx cannot exist + // without one, and "blocked" is defined by a status code. Only + // StatusBroken is genuinely ambiguous, so only there does the checker's + // own bit decide. This is not inference from a proxy; it is what those + // two statuses mean, and it keeps a third-party adapter that omits the + // field from refusing a run in which every check succeeded. + if out.Answered || out.Status == StatusOK || out.Status == StatusBlocked { answered++ } switch out.Status { diff --git a/internal/core/cite/refresh_test.go b/internal/core/cite/refresh_test.go index 5eb19dce..fe883828 100644 --- a/internal/core/cite/refresh_test.go +++ b/internal/core/cite/refresh_test.go @@ -469,6 +469,41 @@ func TestRefreshCommitsATotalFailureOnAFirstRun(t *testing.T) { } } +// TestRefreshTreatsASuccessAsAnAnswer removes a footgun from the exported seam. +// A third-party adapter that reports StatusOK but leaves Answered at its zero +// value would otherwise make the whole run refuse with "no host answered" — +// while every single check succeeded. It is definitional, not inference: a 2xx +// or a 403 cannot exist without a host having answered, so only StatusBroken +// carries real ambiguity, and that is the one case the checker must speak to. +func TestRefreshTreatsASuccessAsAnAnswer(t *testing.T) { + root := citeRepo(t, "https://example.org/a") + writeBaseline(t, root, map[string]lint.BaselineEntry{ + "https://example.org/a": {FinalURL: "https://example.org/a", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01"}, + }) + // An adapter that forgot the field entirely. + naive := &stubChecker{answers: map[string]CheckOutcome{ + "https://example.org/a": {Status: StatusOK, FinalURL: "https://example.org/a"}, + }} + if _, err := Refresh(RefreshRequest{RepoRoot: root, Config: testConfig(), Checker: naive, Now: now}); err != nil { + t.Fatalf("a successful run was refused because an adapter omitted Answered: %v", err) + } + + // The same for a blocked outcome, which is defined by status codes and so + // likewise cannot happen without an answer. + root2 := citeRepo(t, "https://paywalled.example.org/x") + writeBaseline(t, root2, map[string]lint.BaselineEntry{ + "https://paywalled.example.org/x": {FinalURL: "https://paywalled.example.org/x", LastChecked: "2026-07-01", + Outcome: lint.OutcomeAlive, Verification: lint.VerificationAutomatic, VerifiedOn: "2026-07-01"}, + }) + naive2 := &stubChecker{answers: map[string]CheckOutcome{ + "https://paywalled.example.org/x": {Status: StatusBlocked, Detail: "HTTP 403 Forbidden"}, + }} + if _, err := Refresh(RefreshRequest{RepoRoot: root2, Config: testConfig(), Checker: naive2, Now: now}); err != nil { + t.Fatalf("a blocked outcome was misread as an unanswered one: %v", err) + } +} + // TestRefreshRefusesAnUnknownCheckerStatus closes the seam's silent-drop hole. // Status and Checker are exported for the adapter spc-17 AC 5 calls for; one // returning anything else would otherwise omit the URL from BOTH the baseline and diff --git a/internal/core/lint/citations.go b/internal/core/lint/citations.go index 960a9ce3..8178ed01 100644 --- a/internal/core/lint/citations.go +++ b/internal/core/lint/citations.go @@ -133,6 +133,19 @@ func containedRepoPath(rel string) error { // It is the same shape internal/core/launch/bundle.go uses for symlinked bundle // entries: EvalSymlinks both sides, then filepath.Rel and refuse on "..". func resolvedInsideRoot(root, target string) error { + _, err := containedRealPath(root, target) + return err +} + +// containedRealPath is resolvedInsideRoot's other half: it returns the RESOLVED +// path as well as the verdict, so a caller that is about to read the file can +// read the thing containment actually judged rather than re-resolving it. +// +// That distinction matters for the read guard. O_NOFOLLOW refuses every symlink +// leaf, which would break the legitimate in-repo bridge (CLAUDE.md -> AGENTS.md); +// opening the RESOLVED path instead keeps the no-follow protection meaningful +// while still allowing a link that lands inside the repository. +func containedRealPath(root, target string) (string, error) { realRoot, err := filepath.EvalSymlinks(root) if err != nil { realRoot = filepath.Clean(root) @@ -153,15 +166,15 @@ func resolvedInsideRoot(root, target string) error { } realProbe, err := filepath.EvalSymlinks(probe) if err != nil { - return errors.New("cannot be resolved: " + filepath.Base(probe) + ": " + bareCause(err)) + return "", errors.New("cannot be resolved: " + filepath.Base(probe) + ": " + bareCause(err)) } real := filepath.Join(append([]string{realProbe}, tail...)...) relToRoot, err := filepath.Rel(realRoot, real) if err != nil || relToRoot == ".." || strings.HasPrefix(relToRoot, ".."+string(filepath.Separator)) { - return errors.New("resolves outside the repository root through a symlink") + return "", errors.New("resolves outside the repository root through a symlink") } - return nil + return real, nil } // bareCause strips a *PathError's absolute path, leaving the reason. A message diff --git a/internal/core/lint/collect.go b/internal/core/lint/collect.go index fccfc991..9af6eb20 100644 --- a/internal/core/lint/collect.go +++ b/internal/core/lint/collect.go @@ -13,12 +13,19 @@ package lint // (markdownFiles, fenceMask, parseCitations) rather than new parsing. import ( - "os" "path/filepath" "sort" "strings" + + "github.com/REPPL/abcd-cli/internal/fsutil" ) +// citationPageSizeLimit caps one page read. Containment answers "does this path +// land inside the repo"; it says nothing about how big what it lands on is, and +// this collector's output is fetched over the network — so the read is bounded +// too. A documentation page is prose; eight mebibytes is far beyond any real one. +const citationPageSizeLimit = 8 << 20 + // CitationSite is one place a URL is cited: a repo-relative page and a 1-based // line. A URL cited from several pages carries one site per occurrence, so the // manual queue can tell the maintainer where to look. @@ -68,11 +75,18 @@ func CollectCitedURLs(cfg Config, repoRoot string) ([]CitedURL, error) { // contained root and still drags an outside file's citations into // the fetch and then into the committed baseline. Containment is // about where a path LANDS, so an in-repo symlink still resolves. - if err := resolvedInsideRoot(repoRoot, fileAbs); err != nil { + realPath, err := containedRealPath(repoRoot, fileAbs) + if err != nil { return nil, &configError{"cited page " + quote(repoRel(repoRoot, fileAbs)) + " " + err.Error() + "; the citation collector reads only inside the repository"} } - content, err := os.ReadFile(fileAbs) + // ReadGuarded, not os.ReadFile: the regular-file check refuses a + // device and the cap bounds the read, neither of which containment + // can supply — a committed page that is simply enormous lands inside + // the repo and passes every path check. It opens the RESOLVED path, + // because O_NOFOLLOW would otherwise refuse the legitimate in-repo + // bridge that containment just approved. + content, err := fsutil.ReadGuarded(realPath, citationPageSizeLimit) if err != nil { return nil, err } diff --git a/internal/core/lint/collect_test.go b/internal/core/lint/collect_test.go index 60599032..ac2c9c14 100644 --- a/internal/core/lint/collect_test.go +++ b/internal/core/lint/collect_test.go @@ -4,8 +4,55 @@ import ( "os" "path/filepath" "testing" + "time" ) +// TestCollectCitedURLsCapsAPageRead is the bound containment cannot supply. A +// committed page that is simply enormous is INSIDE the repository and resolves +// to itself, so every path check passes and an unguarded read pulls the whole +// thing into memory. The collector feeds a network fetch, so it reads through +// the guarded primitive that already exists for exactly this. +func TestCollectCitedURLsCapsAPageRead(t *testing.T) { + root := citeRepo(t, map[string]string{"docs/a.md": "# A\n"}) + huge := make([]byte, citationPageSizeLimit+1) + for i := range huge { + huge[i] = 'x' + } + if err := os.WriteFile(filepath.Join(root, "docs", "huge.md"), huge, 0o644); err != nil { + t.Fatal(err) + } + if _, err := CollectCitedURLs(Config{Roots: []string{"docs"}}, root); err == nil { + t.Fatal("CollectCitedURLs read a page past the size cap") + } +} + +// TestCollectCitedURLsRefusesANonRegularPage covers the other half of the same +// guard: a page that is a device rather than a file. Here containment already +// refuses it (a device resolves outside the repo), so this pins the combination +// rather than the cap — neither check alone should be relied on. +func TestCollectCitedURLsRefusesANonRegularPage(t *testing.T) { + if _, err := os.Stat("/dev/zero"); err != nil { + t.Skip("no /dev/zero on this platform") + } + root := citeRepo(t, map[string]string{"docs/a.md": "# A\n"}) + if err := os.Symlink("/dev/zero", filepath.Join(root, "docs", "boom.md")); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + done := make(chan error, 1) + go func() { + _, err := CollectCitedURLs(Config{Roots: []string{"docs"}}, root) + done <- err + }() + select { + case err := <-done: + if err == nil { + t.Fatal("CollectCitedURLs read a character device as a page") + } + case <-time.After(20 * time.Second): + t.Fatal("CollectCitedURLs hung reading a character device") + } +} + // TestCollectCitedURLsRefusesAnEscapingRoot closes an exfiltration path this // branch opened. Reading outside the repo via `roots` was inert while the lint // only reported findings; now the collector feeds a LIVE FETCHER and its results From 78cc4bcea3f9445ab2220060193446c2c243a77b Mon Sep 17 00:00:00 2001 From: REPPL <77722411+REPPL@users.noreply.github.com> Date: Tue, 28 Jul 2026 10:32:26 +0100 Subject: [PATCH 18/18] docs: record manual verification receipt for the harness-engineering citation Maintainer confirmed https://openai.com/index/harness-engineering/ (a source that refuses automated fetchers) via the manual queue; abcd docs cite confirm records the dated manual attestation, clearing the citation_baseline blocker. Verification: manual + date, never how. Assisted-by: Claude:claude-fable-5 --- .abcd/citations-baseline.json | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.abcd/citations-baseline.json b/.abcd/citations-baseline.json index faceb5d2..8640e6ec 100644 --- a/.abcd/citations-baseline.json +++ b/.abcd/citations-baseline.json @@ -257,6 +257,14 @@ "verification": "automatic", "verified_on": "2026-07-27" }, + "https://openai.com/index/harness-engineering/": { + "url": "https://openai.com/index/harness-engineering/", + "final_url": "https://openai.com/index/harness-engineering/", + "last_checked": "2026-07-28", + "outcome": "alive", + "verification": "manual", + "verified_on": "2026-07-28" + }, "https://openai.com/index/unlocking-the-codex-harness/": { "url": "https://openai.com/index/unlocking-the-codex-harness/", "final_url": "https://openai.com/index/unlocking-the-codex-harness/",