Methodology
Version 1.4-draft · Draft · Published 2026-09-14
Attestari methodology
Prior versions, newest first: 1.3-draft (draft, published 2026-09-11); 1.2-draft (draft, published 2026-09-11); 1.1-draft (draft, published 2026-09-11); 1.0-draft (draft, published 2026-09-11). Every version keeps its permanent address.
This document is the complete set of rules by which Attestari examines a published package, decides
each finding, grades the package, and publishes the result. It is the only place those rules live in
prose, and attestari/policy.py is the only place the decision rules live in code; the two carry the
same version number and a test refuses to run the policy when they disagree. Every published decision
cites the version that produced it and the SHA-256 of this file as it was when the decision was made.
Nothing in a run is decided by a person; the two things a person still does are listed in section 13.
1. What this document is for
Attestari examines software packages that an AI agent may install and grant access to: MCP servers and agent tooling published to public registries. We report what we find, and we publish the evidence alongside it so a reader can check our work rather than take it on faith.
A finding is an opinion, formed by applying the rules below to material anyone can download. This document states those rules completely. If a finding cannot be traced to a rule in this document and to evidence a reader can independently retrieve, it should not have been published, and we want to hear about it.
The document also states, at length, what we do not examine (section 15). That section is not a disclaimer bolted on at the end. It is the part most likely to matter to someone deciding whether a passing result means anything.
2. Definitions
subject: a specific published version of a package in a public registry, for example
@scope/server at version 1.4.2 on npm. Ratings are always of a version, never of a project in
general.
artifact: the file a package manager actually downloads and installs: the npm tarball or the PyPI distribution. This, not the source repository, is what we examine.
tool surface: the set of tools a subject registers with an agent, together with the descriptions the agent reads. Determined by parsing the artifact's source.
finding: a statement about a subject, tied to a severity, a quoted span of text, the file and line it came from, and the hash of the stored evidence containing it.
not analyzed: a tool we could not examine. Distinct from a tool we examined and found nothing in. The two are never reported as the same thing.
gradeable: a subject for which every advertised tool was examined, every check ran, and every finding's evidence re-verified: the guards in section 9. Only a gradeable subject can carry a grade.
3. We examine the artifact, not the repository
Most software auditing reads the public source repository. We do not, and the distinction is deliberate: the repository is not what runs on your machine. The published artifact is.
A package's repository can be clean while the tarball served by the registry contains code that was never committed there. That gap is how a large share of real registry compromises work, and auditing the repository looks straight past it.
We read repository metadata, such as ownership, contributors, and whether the latest commit carries a signature, as context about who stands behind a subject. We do not treat repository contents as evidence of what a subject does.
What this costs us. We do not currently compare the artifact against the repository at the published tag. That comparison would detect a file shipped with no traceable origin in the source, which is a strong signal we would like to have. It is a known gap, recorded in section 15.
4. How subjects are selected
Selection is mechanical and reproducible. A candidate list records the exact queries used, the ranking rule, the page sizes, the timestamp of every fetch, and the snapshot identifier of any dataset consulted. Anyone re-running the same selection against the same sources gets the same list.
npm candidates come from the registry's public search API across a fixed set of queries, ranked by monthly download count confirmed against the downloads API for a stated window. PyPI candidates come from the simple index, with download counts drawn from a public dataset derived from PyPI's own download statistics.
Exclusions are recorded, not silent. Every candidate that surfaced and was not selected appears in the same file with the reason it was dropped: ranked below the cut, duplicate of a scoped variant, no download figure available. A reader can see what we passed over.
Conflicts. Packages published by Attestari, The Elite360 Corporation, or any account we control are excluded by a hard rule, checked against package name, scope, maintainers, publisher, author, contact addresses, and repository URLs. We do not rate our own work.
A tagged package is not necessarily a server. Selection casts a wide net using MCP-related keywords and naming patterns, which catches libraries and SDKs alongside actual servers. We do not resolve this by maintaining a hand-curated exclusion list. The tool-surface determination in section 5 decides what is a server, and population figures always separate confirmed server, examined from confirmed server, could not examine from not a server.
5. The checks
Four checks run against every subject. Each states what it reads and what it is entitled to conclude. None of them executes the subject's code.
undetermined_tool_surface: can the tool surface be determined at all?
Reads: source files in the artifact, looking for MCP tool registration sites on the token stream (text inside strings and comments does not count). Concludes: a parsed set of tools with their descriptions, or one of these outcomes:
not_a_server: no registration site, and the source consumes the tool API (it lists or calls another server's tools, imports an SDK client) or defines it (a method namedregisterToolortoolwith a body, a server class). A client, adapter, proxy, SDK, framework, instrumentation, CLI, mock or scanner. Such a subject is out of scope: it is not a server, so it is neither graded nor refused, and it is excluded from every population count (decision D4).minified_only: registration text exists only in minified or obfuscated bundles. The operator cannot inspect what they install; this is a medium finding whose evidence is the bundle's bytes and the length of its longest line, not a coverage gap.runtime_generated: a registration site exists and the file that registers builds its tools from external data (an OpenAPI document, a remote catalog). A genuine limit, reported as such.declaration_not_parsed: a registration site exists in readable source and the extractor could not read it. This is the only outcome that is honestly our gap; the site is recorded.no_registration_site: neither a registration site nor a consumption pattern.archive_unreadable: the artifact is not a readable tar or zip.
Every outcome but parsed refuses the grade under its own code (section 9), except not_a_server,
which is out of scope. A subject whose surface cannot be read is not thereby condemned. It is a
subject about which we will not offer an opinion, because the thing an agent would actually be
exposed to was never visible to us.
model_directed_text: does a tool description contain instructions aimed at the model?
Reads: the literal text of each parsed tool description, passed to a language model as clearly delimited evidence, never as free text that could be read as instruction. Concludes: specific quoted spans that direct the reading model's behaviour, each categorised and verified.
Before any description is read, each tool's description is in one of three states, read from the parsed surface, never guessed:
readable: a single literal, or a concatenation of literals with no substitution ("a" + "b",["a", "b"].join("\n")). Every byte the client receives is in the source, and the check reads all of it.partial: a template with substitutions (`Reads ${what}`, an f-string). Each literal segment is read as its own byte span; each substitution is a hole, recorded with its position, that the verifier never matches inside. A finding in a segment is a valid finding. The absence of findings in a partial description does not make the tool clean: a package with a partial tool that produced no finding is refused a grade (partial_description, section 9) unless ahighfinding already places it at F, where the floor is established by evidence.unreadable: a variable, a property, a call, a configuration passed by reference that does not resolve to a literal declared in the same file, or a minified bundle. The text a client will receive exists and was not examined. Each such tool is recorded asnot_analyzedwith the reasondescriptions_not_statically_readable, and the package is refused a grade by the gate in section 9.
A tool that declares no description at all (absent) has nothing for this check to read, and for
this check that is a clean outcome (no_descriptions_present). A surface that does not record the
state of a description is treated as unreadable.
A tool description is read by the agent, not by the user. Text placed there can instruct the agent
to exfiltrate data, override the user's stated intent, or conceal what it is doing. Those three
categories, exfiltrate_data, override_user and conceal_from_user, are the only ones that become
findings, and each carries severity high. Routing, ordering and rate guidance ("call X first", "no
more than three calls") is never a finding. The check records only a count of such spans per tool;
the spans themselves are transcribed only by a separate inventory pass that a run must opt into
(attestari run --inventory), and nothing that decides reads it.
Every proposed span is checked by a deterministic verification step: the quoted text must occur byte for byte inside the stored file, within the description of the named tool. A proposal that step rejects never becomes a finding.
The analysis prompt encodes the evidence so that it cannot be interpreted as instruction by the model reading it. This is both a correctness measure and a safety one: an analyser that can be manipulated by the text it audits is not an analyser.
remote_code_fetch: does the package fetch and execute code it did not ship?
Reads: source files and package manifests, for download-then-execute patterns, dynamic evaluation, remote imports, and install-time scripts. Concludes: quoted spans showing code that is obtained at runtime rather than published.
This is the check that catches a package which is mostly a launcher: what the registry serves is small and readable, and the substance arrives later from somewhere else. Neither the registry nor this analysis can tell you anything about code that was never published.
Informational pending measured precision (section 18). This check has no evaluation corpus yet, and
the run of 2026-09-14 showed its patterns matching a chmod of a directory, a git log subprocess
and a vendored schema compiler. Until its precision is measured, every finding it makes is decided
at severity info (decision D5): recorded and published as context, never graded.
publisher_provenance: what links this artifact to a person or organisation?
Reads: registry metadata, publisher attestations, maintainer records, repository ownership, commit signatures. Concludes: what is present and what is absent in the chain between a named party and this artifact.
Provenance findings describe absences more often than problems, and absences are common: most
registry packages carry no build attestation and no signed commits. We report them because their
accumulation is meaningful, not because any one of them is damning. Severity reflects this: a single
provenance tag is low, and a finding reaches high only when a second, independent provenance tag
on the same package corroborates it (decision D6).
The account-age rule (account_younger_than_package: the account that owns the source repository was
created after the package's first release) is informational pending measured precision (section 18).
A registry name transferred to a new owner is enough to trigger it, so until its precision is measured
it is decided at info (decision D5), as is every other provenance finding.
6. Severity
Severity describes exposure to the person who installs and runs the subject. It is not a measure of intent, and it makes no claim about whether a publisher meant any harm.
High: an operator running this subject is exposed to something they would not reasonably expect from its stated purpose: code they cannot inspect, credentials reaching a destination they did not choose, or instructions to the agent that work against the operator.
Medium: a property that materially weakens the operator's ability to know what they are running, or that concentrates control in a way worth knowing about before installing.
Low: a property of the publisher record that leaves the operator with no one to hold to account, without by itself changing what the package does. A provenance finding with a single tag is decided low; a second, independent tag on the same package raises it to high (decision D6, section 10).
Info: a factual observation, usually an absence, that is common across the registry and meaningful mainly in aggregate or alongside other findings. It is published as context and has no effect on the grade.
We do not use a borrowed scale such as CVSS. These are not vulnerabilities with exploit mechanics; they are properties of what a package does and what can be known about it. The grade (section 11) is read off the highest confirmed severity.
7. Evidence and verification
Every artifact we fetch is stored by content hash. Every finding cites the hash of the evidence it came from, the path within it, and the line range, alongside the quoted span itself.
At decision time, the quoted span is re-verified byte for byte against the stored evidence (guards G6 and G7 in section 9). A span that does not re-verify does not become a published finding, and its subject becomes ungradeable rather than passing quietly.
Every published report carries instructions for reproducing its central claims from public sources in a few minutes, without our software and without trusting us. If you cannot check a finding yourself, we have not finished writing it.
No personal data in published evidence. A published blob never contains an email address, a
phone number or a street address. This is a rule, not a judgement. Evidence is published in one
of two forms, defined exactly in docs/publish-contract.md (section 9a):
- A finding that cites a structured upstream response (a registry record, a GitHub API response)
is published with a projection: a canonical JSON object carrying only the member the finding
cites, copied byte for byte, the JSON pointer of the object it was taken from, and the SHA-256
of the whole upstream response as provenance (
source_sha256). The projection's own hash is the evidence hash the published finding verifies against; the upstream bytes are never published. - A finding that cites a file from the package artifact is published with that file whole: it is public code. It is scanned all the same.
Every blob of either kind is scanned for the patterns the contract lists before it is published.
A finding whose evidence cannot be published without personal data is withheld, not redacted and
not softened, and its subject is published as not_gradeable with the reason
evidence_unpublishable. The stored evidence, the decision and the finding id are untouched;
only publication is refused.
Attested manifest
For each parsed tool whose description is readable (section 5), the tool hash is SHA-256 over: the
tool name in UTF-8, one 0x00 byte, the description bytes exactly as extracted (description_raw),
one 0x00 byte, and the schema component. When the declared input schema is a JSON literal, the
schema component is its canonical JSON (RFC 8785: sorted keys, no insignificant whitespace, UTF-8).
When it is any other expression (a zod schema, a reference), the schema component is the schema's
source bytes exactly as extracted, taken the same way description_raw is; only a JSON literal is
canonicalised. When no schema is declared, the component is empty. The manifest hash is SHA-256
over the tool hashes, as 32-byte digests, sorted bytewise and concatenated; hexadecimal is only how
either hash is printed. Both are computed by attestari/manifest.py, and a fixed-vector test pins
this definition to that code.
A tool whose description is not readable has no tool hash, and a subject with any such tool has no manifest hash: the manifest is not attested, which is an outcome, not an omission. Both hashes are recorded with the decision row (section 12) and published with the result.
8. Inputs to a decision
A run directory written by attestari run (see attestari/pipeline.py): findings.json,
results.json, gradeable.json, surfaces.json, records.json, not_analyzed.json,
summary.json, plus the content-addressed evidence store the run cited (evidence/blobs/<sha256>).
The policy reads these files and the blobs. It never calls the network or a model.
9. Package gradeability (fail closed)
The failure we care most about avoiding is not a wrong grade. It is a passing grade on something nobody looked at. A subject that could not be examined must never be indistinguishable from one that was examined and found clean.
So the grading gate fails closed. A package is graded only when every guard below holds. Any guard
that fails, or cannot be evaluated, makes the package not_gradeable; every reason found is recorded,
in order, and every finding of that package is withheld. There are no exceptions and no severity
that is exempt.
- G1 gate recorded.
gradeable.jsonexists and has a row for the package. A missing file blocks every package in the run; a missing row blocks that package. - G2 gate passes as recorded. The row says
ok: true. - G3 gate passes when recomputed.
attestari.grade.gradeabilityis re-run at decision time fromresults.jsonand the package'ssurfaces.jsonrow (tools with descriptions, parsed tool names, advertised tool names) and must return ok. A package with no surfaces row cannot be recomputed and is blocked. - G4 no tool left unanalysed. No
not_analyzedentry for the package inresults.jsonor innot_analyzed.json. A refused, truncated or unreplayed model call is an outcome, never a clean result. - G5 every check ran and none errored. Each configured check (
publisher_provenance,remote_code_fetch,undetermined_tool_surface,model_directed_text) has a result row for the package, andsummary.jsonrecords no failure for the package that is a crash: a failure row whose stage isfetchor a check id, or that carries a traceback. Recorded HTTP failures of a source (a 404 repository, an unreachable website) are inputs to the checks, not crashes, and do not block by themselves. - G6 every excerpt re-verifies byte for byte. For each finding of the package: the cited blob
exists in the evidence store, and the UTF-8 bytes of
evidence.excerptoccur inside it. When the finding cites lines (line_start..line_end), the bytes must occur within those lines. An empty excerpt fails. One unverifiable finding blocks the whole package. - G7 model-derived findings passed the verifier. For each
model_directed_textfinding that quotes a tool description (every one exceptdescription_reassigned_at_runtime), the excerpt must lie inside one of the description spans of a tool declared at the same path in the package'ssurfaces.jsonrow: the same testattestari.checks.model_directed_text.verify_proposalapplies when the proposal is made. A proposal the verifier discarded is never a finding; a finding that would not pass the verifier now blocks the package.
The gate recomputed in G3 refuses a package for any of the following reasons, and records each one it finds:
| Reason code | Meaning |
|---|---|
no_tools_examined |
No tool was parsed from the published source. The subject's advertised behaviour was never visible. |
advertised_tools_not_parsed |
Documentation advertises tools the extractor did not find. The mismatch is itself the finding. |
not_analyzed |
One or more tools could not be analysed: a model provider declined the request, or a tool declares a description the extractor could not read as a literal (descriptions_not_statically_readable, section 5). Either is recorded as its own outcome and never counted as a clean result. |
model_check_skipped |
The description analysis did not run on a subject that has parsed tool descriptions. |
partial_description |
A tool's description was read with holes and nothing was found in its readable segments; that does not establish a clean tool (section 5). Not applied when a high finding already places the package at F. |
minified_only, runtime_generated, declaration_not_parsed, no_registration_site, archive_unreadable |
The surface outcome of section 5, refusing the grade under its own name. not_a_server is not a refusal: the package is out of scope (D4). |
"We cannot tell you what this does" is a legitimate published outcome, and for a meaningful share of popular packages it is the accurate one.
10. Finding decisions
The policy named severity-ladder decides each finding of a gradeable package:
- D1 confirmed. A verified finding whose decided severity is
high,mediumorlowisconfirmedat that severity. The decided severity starts from the severity the check assigned and is changed only by D5 and D6 below; the row records both. - D2 noted. A verified finding whose decided severity is
infoisnoted: published as context, no effect on the grade. - D3 withheld. Every finding of a package that is not gradeable is
withheld, and its row lists the package's blocking reasons. A withheld finding is never published as a finding. - D4 out of scope. A package whose surface is
not_a_server(section 5) is out of scope: it is published with the verdictout_of_scopeand the reasonnot_a_server, every finding of it iswithheld, and it is counted apart from the population rather than as a refusal. Out of scope is not a grade and not a refusal; it says the artifact is not the kind of thing this methodology rates. - D5 measured checks only. Only a check whose precision has been measured (section 18) may
place a finding above
info. The policy carries the set of measured (check, tag) pairs,MEASURED_CHECKS; in this version it holdsmodel_directed_textwith every tag and nothing else. A finding from any other check or tag is decided atinfoandnoted, whatever severity the check assigned; the assigned severity stays on the row. Every check still runs and every finding is still recorded. Adding a pair to the set is a methodology change that must cite the measurement in section 18. - D6 provenance corroboration. A
publisher_provenancefinding is decidedhighonly when a second, independent provenance tag is present on the same package; a single tag is decidedlow. D6 is applied before D5, so whilepublisher_provenanceis unmeasured the outcome is stillinfo.
There are no suppression rules in this version. A finding that should not count is a methodology change (add a rule here, bump the version) or an appeal (section 16), never an edit to a run.
11. Package grade
From the confirmed findings of a gradeable package:
| grade | condition |
|---|---|
| A | no confirmed finding (only noted context, or nothing) |
| B | highest confirmed severity is low |
| C | highest confirmed severity is medium |
| F | any confirmed finding of severity high |
A package that is not gradeable has no grade; its row carries blocked_by instead.
12. What every decision row records
policy, methodology_version, methodology_sha256, code_commit (the git commit of the code
that decided; code_dirty says whether the tree had uncommitted changes), decided_at, the
finding's evidence_sha256 (the blob it was verified against) and excerpt_sha256, the
verification outcome and the rule that produced the decision. The run-level header repeats the
stamps and records the SHA-256 of every input file. A rerun over unchanged inputs with the same
methodology and code produces identical rows apart from decided_at.
13. Automated decision, human responsibility
Findings are decided by the rules in this document, applied by software, without a person reviewing each one. This is a deliberate choice. Rules applied by a machine are applied identically to the two-hundredth subject and the first; a person working through hundreds of findings is not.
What is not automated is the rules themselves. Changing them is an explicit act that produces a new version of this document (section 19), and every finding cites the version that produced it. The methodology, not any individual judgement call, is what stands behind a rating.
Running the rules is not a human act and needs no approval. The two things a person does:
- Appeals. A vendor disputing a finding files an appeal against its finding id
(
attestari appeal file). A person resolves it (attestari appeal resolve) asupheld,dismissedorwithdrawn. Appeals are an append-only event log beside the run (appeals.json); nothing indecisions.jsonis edited. Publication applies them: an upheld appeal publishes the finding aswithdrawn_on_appealand recomputes that package's grade without it; an open appeal publishes the finding markedunder_appeal, unchanged. What an appeal can be about, and how we answer one, is in section 16. - Methodology changes. Editing this document and the matching constant, as described in section 19.
Publication is separate. Producing findings and publishing them about a named third party are
distinct steps. Automation covers the first. The second is attestari publish <run>: one command,
not a review queue, run only after the notice period in section 16. It refuses a run whose decisions
were made under a methodology version or document hash other than the current one, and never
overwrites an earlier publication.
14. Scheduled operation
attestari cycle --candidates <file> performs one cycle with no human input: it resumes the most
recent failed or cancelled run over that candidates file if there is one, otherwise starts a new
run; then decides the run under this methodology, diffs it against the previous decided run over
the same candidates file, and writes decisions.json, decisions.md and diff.json into the run
directory. --every <seconds> repeats on that cadence in-process; a scheduler may instead invoke
the one-shot form.
15. Known limits
These are the things a passing result does not tell you. They are stated here rather than in a footnote because a reader who does not know them will over-read our findings.
We do not run the code. Analysis is static. A subject that behaves differently when executed, against a particular host, after a delay, or only in the presence of certain credentials, presents none of that to us.
We do not examine binaries. Where a package's substance is a compiled executable it downloads, we can describe the launcher and the fetch precisely, and we can say nothing whatsoever about what the binary does.
Text assigned at runtime is invisible to us. A tool description that is benign in source and rewritten when the module loads reaches the agent in its rewritten form. We detect and report that a description is reassigned at runtime, but we cannot read what it becomes.
Tool names built by expression may be missed. Where a tool's name is computed rather than
written literally, our extractor can fail to find a tool that genuinely exists. A name, a
configuration or a description written as a constant declared in the same file is resolved; anything
else is reported as declaration_not_parsed (section 5), and the gate in section 9 catches an
advertised tool we did not find as a mismatch rather than passing the subject silently.
A minified bundle is opaque to us and to you. Where a package ships its registration only in a minified or obfuscated bundle, we report that as a medium finding and read nothing from it. We do not reconstruct source from a bundle.
Tools generated at run time are not in the artifact. Where the file that registers tools builds them from an OpenAPI document or a remote catalog, the tool list exists only when the server runs. We report the site and read nothing further.
We do not diff the artifact against the repository. A file present in the published package with no counterpart in the source tree is a signal we do not currently produce. See section 3.
Some checks are not yet implemented. Undeclared network destinations, dependency risk, maintenance status, and fork divergence are not assessed. Where a report lists these as not assessed, that is what it means, not that they were assessed and found clean.
A finding is about one version, at one time. Registries are mutable. A result describes the artifact we fetched, identified by hash, on the date stated. It says nothing about what the same version number serves tomorrow.
16. Notice, appeals, and corrections
Notice before publication. Before any finding about a named subject is published, we send it to the publisher at the contact address in their registry record or repository, and wait 30 days. That period is for correcting us, and a publisher is under no obligation to respond.
Appeals. A publisher may dispute any finding at any time, before or after publication, by
writing to appeals@attestari.ai. An appeal that identifies a factual error, such as a misquoted
span, a wrong line reference, or a claim about code that is not there, is checked against the stored
evidence. If we are wrong, we correct it.
An appeal that disputes our interpretation rather than our facts will be considered and may be published alongside the finding. We do not remove an accurate finding because a publisher objects to it.
Corrections. Corrections are published, dated, and describe what was wrong and what changed. We do not silently edit a published finding.
Remediation. When a publisher fixes the thing a finding describes, a re-examination of the new version is free and prompt, and the updated result is published.
17. Independence
A publisher cannot pay us, in money or in kind, for a rating, for a better rating, or for the removal of a finding. There is no paid review, no expedited review, and no arrangement under which a rated party influences its own result.
The subjects we examine are chosen by the mechanical rule in section 4. They are not chosen by request, and they are not chosen by who has paid for anything.
Where Attestari earns revenue, the source is disclosed. Any commercial relationship with a party that is also a subject is disclosed on that subject's published result.
Publisher tools. Attestari may offer a publisher a tool that applies these checks to an artifact that has not been published. Its output is never published, never stored where the scheduled pass can read it, and never affects a grade. The public grade of a subject is computed only from the registry artifact, only by the scheduled pass, only under the methodology in force, whether or not the publisher used such a tool.
18. Measured performance
The description-analysis check is evaluated against a corpus of publicly disclosed tool-poisoning cases, published by third-party security researchers, plus a control set of tool descriptions from widely-used packages reviewed by hand. Current figures:
- 5 of 6 disclosed cases detected
- 22 of 27 expected spans found
- 1 false positive across 456 clean tools
- 1 of 1 in-the-wild case detected
The undetected case is one where the poisoned text is assigned when the module loads and therefore never appears in the source we read: a limit stated in section 15, not a tuning failure.
These figures are measured against a small corpus, because the set of publicly disclosed cases is small. They should be read as evidence that the check works on known attacks, not as a general detection rate. They will be restated whenever the corpus or the check changes.
Where the figures come from: the evaluation run recorded at runs/eval-separated-live (2026-09-09,
claude-opus-5, the separated prompt layout), scored by attestari eval against the corpus in
seeds/disclosed_cases.json and the clean set in runs/twenty/surfaces.json. In that evaluation
only, each seed sample's evidence block cited the publication that disclosed it; a production run
carries no such citation.
19. Versioning
This document carries a version: the methodology_version: line at the top, MAJOR.MINOR.PATCH.
Every published finding cites the version in force when it was produced and the SHA-256 of the
document. Superseded versions remain available at their own permanent addresses, so a finding
published in 2026 can still be read against the rules that made it in 2026.
A change to what a check examines, what a severity means, or when we refuse to grade is a version change. Correcting a typo is not.
Changing any rule is a deliberate act: edit this document, bump methodology_version, bump
METHODOLOGY_VERSION in attestari/methodology.py, add a line to the Versions section, copy the
document to a new snapshot under the site's content/methodology/, and commit. A published snapshot
is never edited.
Versions
- 1.0.0 (2026-09-11): first automated policy. Guards G1-G7, decisions D1-D3, grade ladder A/B/C/F, appeals and publication as the two human acts.
- 1.1.0 (2026-09-11): the public methodology around the unchanged rules: what the document is for, definitions, artifact not repository, subject selection, the four checks, severity definitions, evidence and verification, known limits, notice, appeals and corrections, independence, measured performance, versioning. Guards, decisions and the grade ladder are unchanged from 1.0.0.
- 1.2.0 (2026-09-11): two corrections. Evidence: no personal data in published evidence; structured
responses are published as projections of the cited member with the upstream hash as provenance,
files ship whole and scanned, and a finding that cannot be evidenced without personal data is
withheld with its subject refused (
evidence_unpublishable). Model check: "no literal tool descriptions" is split intono_descriptions_present(clean) anddescriptions_not_statically_readable(not analysed, refuses the grade throughnot_analyzed), read from the parsed surface and failing closed when the surface does not say. - 1.3.0 (2026-09-11): the surface taxonomy and three description states. A surface that did not
parse is
not_a_server(out of scope, decision D4),minified_only(a medium finding),runtime_generated,declaration_not_parsed,no_registration_siteorarchive_unreadable, each refusing the grade under its own code except the first. A description isreadable,partial(literal segments read, substitutions are holes; findings valid, absence not clean:partial_description) orunreadable. The extractor resolves names, configurations and descriptions written as constants declared in the same file, reads class-basedgetName()/getConfig()declarations, acceptsschemaandargsas schema keys, and reads Python call-form registrationx.tool(name=...)(fn). - 1.4.0 (2026-09-14): unmeasured checks decide at
info. Decision D5: only checks whose precision has been measured (section 18) may grade; the setMEASURED_CHECKSholdsmodel_directed_textalone, so everyremote_code_fetchandpublisher_provenancefinding is recorded and published as context atinfo. Decision D6: a provenance finding ishighonly with a second, independent provenance tag on the same package, otherwiselow. Section 5 marksremote_code_fetchand the account-age rule as informational pending measured precision. Section 7 gains "Attested manifest": a tool hash per readable tool and a manifest hash per subject, defined inattestari/manifest.pyand pinned by a fixed-vector test. Section 17 gains "Publisher tools": a publisher's own use of these checks never affects a grade. Section 6 no longer namesanonymous_publisheras the only low finding (D6 decides any single-tag provenance finding low); section 5 records routing guidance as a count per tool, the inventory being a separate opt-in pass.