Attestari methodology

This document is the complete set of rules by which a pipeline run's findings are decided and a package is graded. It is the only place those rules live in prose, and attestari/policy.py is the only place they live in code; the two carry the same version number and a test refuses to run the policy when they disagree. Every published decision cites the version that produced it and the SHA-256 of this file as it was when the decision was made.

Changing any rule below is a deliberate act: edit this document, bump methodology_version, bump METHODOLOGY_VERSION in attestari/methodology.py, add a line to the Versions section, and commit. Running the rules is not a human act and needs no approval. Nothing in a run is decided by a person; the two things a person still does are listed in section 6.

1. Inputs

A run directory written by attestari run (see attestari/pipeline.py): findings.json, results.json, gradeable.json, surfaces.json, records.json, not_analyzed.json, summary.json, plus the content-addressed evidence store the run cited (evidence/blobs/<sha256>). The policy reads these files and the blobs. It never calls the network or a model.

2. Package gradeability (fail closed)

A package is graded only when every guard below holds. Any guard that fails, or cannot be evaluated, makes the package not_gradeable; every reason found is recorded, in order, and every finding of that package is withheld. There are no exceptions and no severity that is exempt.

• G1 gate recorded. gradeable.json exists and has a row for the package. A missing file blocks every package in the run; a missing row blocks that package.
• G2 gate passes as recorded. The row says ok: true.
• G3 gate passes when recomputed. attestari.grade.gradeability is re-run at decision time from results.json and the package's surfaces.json row (tools with descriptions, parsed tool names, advertised tool names) and must return ok. A package with no surfaces row cannot be recomputed and is blocked.
• G4 no tool left unanalysed. No not_analyzed entry for the package in results.json or in not_analyzed.json. A refused, truncated or unreplayed model call is an outcome, never a clean result.
• G5 every check ran and none errored. Each configured check (publisher_provenance, remote_code_fetch, undetermined_tool_surface, model_directed_text) has a result row for the package, and summary.json records no failure for the package that is a crash: a failure row whose stage is fetch or a check id, or that carries a traceback. Recorded HTTP failures of a source (a 404 repository, an unreachable website) are inputs to the checks, not crashes, and do not block by themselves.
• G6 every excerpt re-verifies byte for byte. For each finding of the package: the cited blob exists in the evidence store, and the UTF-8 bytes of evidence.excerpt occur inside it. When the finding cites lines (line_start..line_end), the bytes must occur within those lines. An empty excerpt fails. One unverifiable finding blocks the whole package.
• G7 model-derived findings passed the verifier. For each model_directed_text finding that quotes a tool description (every one except description_reassigned_at_runtime), the excerpt must lie inside one of the description spans of a tool declared at the same path in the package's surfaces.json row: the same test attestari.checks.model_directed_text.verify_proposal applies when the proposal is made. A proposal the verifier discarded is never a finding; a finding that would not pass the verifier now blocks the package.

3. Finding decisions

The policy named severity-ladder decides each finding of a gradeable package:

• D1 confirmed. A verified finding with severity high, medium or low is confirmed at the severity its check assigned. The policy neither raises nor lowers a severity.
• D2 noted. A verified finding with severity info is noted: published as context, no effect on the grade.
• D3 withheld. Every finding of a package that is not gradeable is withheld, and its row lists the package's blocking reasons. A withheld finding is never published as a finding.

There are no suppression rules in this version. A finding that should not count is a methodology change (add a rule here, bump the version) or an appeal (section 6), never an edit to a run.

4. Package grade

From the confirmed findings of a gradeable package:

A package that is not gradeable has no grade; its row carries blocked_by instead.

5. What every decision row records

policy, methodology_version, methodology_sha256, code_commit (the git commit of the code that decided; code_dirty says whether the tree had uncommitted changes), decided_at, the finding's evidence_sha256 (the blob it was verified against) and excerpt_sha256, the verification outcome and the rule that produced the decision. The run-level header repeats the stamps and records the SHA-256 of every input file. A rerun over unchanged inputs with the same methodology and code produces identical rows apart from decided_at.

6. The two things a person does

1. Appeals. A vendor disputing a finding files an appeal against its finding id (attestari appeal file). A person resolves it (attestari appeal resolve) as upheld, dismissed or withdrawn. Appeals are an append-only event log beside the run (appeals.json); nothing in decisions.json is edited. Publication applies them: an upheld appeal publishes the finding as withdrawn_on_appeal and recomputes that package's grade without it; an open appeal publishes the finding marked under_appeal, unchanged.
2. Methodology changes. Editing this document and the matching constant, as described at the top.

Publication (attestari publish <run>) is one command, not a review queue. It refuses a run whose decisions were made under a methodology version or document hash other than the current one, and never overwrites an earlier publication.

7. Scheduled operation

attestari cycle --candidates <file> performs one cycle with no human input: it resumes the most recent failed or cancelled run over that candidates file if there is one, otherwise starts a new run; then decides the run under this methodology, diffs it against the previous decided run over the same candidates file, and writes decisions.json, decisions.md and diff.json into the run directory. --every <seconds> repeats on that cadence in-process; a scheduler may instead invoke the one-shot form.

Versions

• 1.0.0 (2026-09-11): first automated policy. Guards G1-G7, decisions D1-D3, grade ladder A/B/C/F, appeals and publication as the two human acts.