Attestari methodology

This document is the complete set of rules by which Attestari examines a published package, decides each finding, grades the package, and publishes the result. It is the only place those rules live in prose, and attestari/policy.py is the only place the decision rules live in code; the two carry the same version number and a test refuses to run the policy when they disagree. Every published decision cites the version that produced it and the SHA-256 of this file as it was when the decision was made. Nothing in a run is decided by a person; the two things a person still does are listed in section 13.

1. What this document is for

Attestari examines software packages that an AI agent may install and grant access to: MCP servers and agent tooling published to public registries. We report what we find, and we publish the evidence alongside it so a reader can check our work rather than take it on faith.

A finding is an opinion, formed by applying the rules below to material anyone can download. This document states those rules completely. If a finding cannot be traced to a rule in this document and to evidence a reader can independently retrieve, it should not have been published, and we want to hear about it.

The document also states, at length, what we do not examine (section 15). That section is not a disclaimer bolted on at the end. It is the part most likely to matter to someone deciding whether a passing result means anything.

2. Definitions

subject: a specific published version of a package in a public registry, for example @scope/server at version 1.4.2 on npm. Ratings are always of a version, never of a project in general.

artifact: the file a package manager actually downloads and installs: the npm tarball or the PyPI distribution. This, not the source repository, is what we examine.

tool surface: the set of tools a subject registers with an agent, together with the descriptions the agent reads. Determined by parsing the artifact's source.

finding: a statement about a subject, tied to a severity, a quoted span of text, the file and line it came from, and the hash of the stored evidence containing it.

not analyzed: a tool we could not examine. Distinct from a tool we examined and found nothing in. The two are never reported as the same thing.

gradeable: a subject for which every advertised tool was examined, every check ran, and every finding's evidence re-verified: the guards in section 9. Only a gradeable subject can carry a grade.

3. We examine the artifact, not the repository

Most software auditing reads the public source repository. We do not, and the distinction is deliberate: the repository is not what runs on your machine. The published artifact is.

A package's repository can be clean while the tarball served by the registry contains code that was never committed there. That gap is how a large share of real registry compromises work, and auditing the repository looks straight past it.

We read repository metadata, such as ownership, contributors, and whether the latest commit carries a signature, as context about who stands behind a subject. We do not treat repository contents as evidence of what a subject does.

What this costs us. We do not currently compare the artifact against the repository at the published tag. That comparison would detect a file shipped with no traceable origin in the source, which is a strong signal we would like to have. It is a known gap, recorded in section 15.

4. How subjects are selected

Selection is mechanical and reproducible. A candidate list records the exact queries used, the ranking rule, the page sizes, the timestamp of every fetch, and the snapshot identifier of any dataset consulted. Anyone re-running the same selection against the same sources gets the same list.

npm candidates come from the registry's public search API across a fixed set of queries, ranked by monthly download count confirmed against the downloads API for a stated window. PyPI candidates come from the simple index, with download counts drawn from a public dataset derived from PyPI's own download statistics.

Exclusions are recorded, not silent. Every candidate that surfaced and was not selected appears in the same file with the reason it was dropped: ranked below the cut, duplicate of a scoped variant, no download figure available. A reader can see what we passed over.

Conflicts. Packages published by Attestari, The Elite360 Corporation, or any account we control are excluded by a hard rule, checked against package name, scope, maintainers, publisher, author, contact addresses, and repository URLs. We do not rate our own work.

A tagged package is not necessarily a server. Selection casts a wide net using MCP-related keywords and naming patterns, which catches libraries and SDKs alongside actual servers. We do not resolve this by maintaining a hand-curated exclusion list. The tool-surface determination in section 5 decides what is a server, and population figures always separate confirmed server, examined from confirmed server, could not examine from not a server.

5. The checks

Four checks run against every subject. Each states what it reads and what it is entitled to conclude. None of them executes the subject's code.

undetermined_tool_surface: can the tool surface be determined at all?

Reads: source files in the artifact, looking for MCP tool registration patterns. Concludes: either a parsed set of tools with their descriptions, or that the surface is unknown, with the reason recorded.

A subject whose surface cannot be determined is not thereby condemned. It is, however, a subject about which we will not offer an opinion, because the thing an agent would actually be exposed to was never visible to us.

model_directed_text: does a tool description contain instructions aimed at the model?

Reads: the literal text of each parsed tool description, passed to a language model as clearly delimited evidence, never as free text that could be read as instruction. Concludes: specific quoted spans that direct the reading model's behaviour, each categorised and verified.

A tool description is read by the agent, not by the user. Text placed there can instruct the agent to exfiltrate data, override the user's stated intent, or conceal what it is doing. Those three categories, exfiltrate_data, override_user and conceal_from_user, are the only ones that become findings, and each carries severity high. Routing, ordering and rate guidance ("call X first", "no more than three calls") is verified the same way but kept as an informational inventory; it never appears as a risk.

Every proposed span is checked by a deterministic verification step: the quoted text must occur byte for byte inside the stored file, within the description of the named tool. A proposal that step rejects never becomes a finding.

The analysis prompt encodes the evidence so that it cannot be interpreted as instruction by the model reading it. This is both a correctness measure and a safety one: an analyser that can be manipulated by the text it audits is not an analyser.

remote_code_fetch: does the package fetch and execute code it did not ship?

Reads: source files and package manifests, for download-then-execute patterns, dynamic evaluation, remote imports, and install-time scripts. Concludes: quoted spans showing code that is obtained at runtime rather than published.

This is the check that catches a package which is mostly a launcher: what the registry serves is small and readable, and the substance arrives later from somewhere else. Neither the registry nor this analysis can tell you anything about code that was never published.

publisher_provenance: what links this artifact to a person or organisation?

Reads: registry metadata, publisher attestations, maintainer records, repository ownership, commit signatures. Concludes: what is present and what is absent in the chain between a named party and this artifact.

Provenance findings describe absences more often than problems, and absences are common: most registry packages carry no build attestation and no signed commits. We report them because their accumulation is meaningful, not because any one of them is damning. Severity reflects this.

6. Severity

Severity describes exposure to the person who installs and runs the subject. It is not a measure of intent, and it makes no claim about whether a publisher meant any harm.

High: an operator running this subject is exposed to something they would not reasonably expect from its stated purpose: code they cannot inspect, credentials reaching a destination they did not choose, or instructions to the agent that work against the operator.

Medium: a property that materially weakens the operator's ability to know what they are running, or that concentrates control in a way worth knowing about before installing.

Low: a property of the publisher record that leaves the operator with no one to hold to account, without by itself changing what the package does. One finding carries it today: a package with no source repository and no author (anonymous_publisher).

Info: a factual observation, usually an absence, that is common across the registry and meaningful mainly in aggregate or alongside other findings. It is published as context and has no effect on the grade.

We do not use a borrowed scale such as CVSS. These are not vulnerabilities with exploit mechanics; they are properties of what a package does and what can be known about it. The grade (section 11) is read off the highest confirmed severity.

7. Evidence and verification

Every artifact we fetch is stored by content hash. Every finding cites the hash of the evidence it came from, the path within it, and the line range, alongside the quoted span itself.

At decision time, the quoted span is re-verified byte for byte against the stored evidence (guards G6 and G7 in section 9). A span that does not re-verify does not become a published finding, and its subject becomes ungradeable rather than passing quietly.

Every published report carries instructions for reproducing its central claims from public sources in a few minutes, without our software and without trusting us. If you cannot check a finding yourself, we have not finished writing it.

8. Inputs to a decision

A run directory written by attestari run (see attestari/pipeline.py): findings.json, results.json, gradeable.json, surfaces.json, records.json, not_analyzed.json, summary.json, plus the content-addressed evidence store the run cited (evidence/blobs/<sha256>). The policy reads these files and the blobs. It never calls the network or a model.

9. Package gradeability (fail closed)

The failure we care most about avoiding is not a wrong grade. It is a passing grade on something nobody looked at. A subject that could not be examined must never be indistinguishable from one that was examined and found clean.

So the grading gate fails closed. A package is graded only when every guard below holds. Any guard that fails, or cannot be evaluated, makes the package not_gradeable; every reason found is recorded, in order, and every finding of that package is withheld. There are no exceptions and no severity that is exempt.

• G1 gate recorded. gradeable.json exists and has a row for the package. A missing file blocks every package in the run; a missing row blocks that package.
• G2 gate passes as recorded. The row says ok: true.
• G3 gate passes when recomputed. attestari.grade.gradeability is re-run at decision time from results.json and the package's surfaces.json row (tools with descriptions, parsed tool names, advertised tool names) and must return ok. A package with no surfaces row cannot be recomputed and is blocked.
• G4 no tool left unanalysed. No not_analyzed entry for the package in results.json or in not_analyzed.json. A refused, truncated or unreplayed model call is an outcome, never a clean result.
• G5 every check ran and none errored. Each configured check (publisher_provenance, remote_code_fetch, undetermined_tool_surface, model_directed_text) has a result row for the package, and summary.json records no failure for the package that is a crash: a failure row whose stage is fetch or a check id, or that carries a traceback. Recorded HTTP failures of a source (a 404 repository, an unreachable website) are inputs to the checks, not crashes, and do not block by themselves.
• G6 every excerpt re-verifies byte for byte. For each finding of the package: the cited blob exists in the evidence store, and the UTF-8 bytes of evidence.excerpt occur inside it. When the finding cites lines (line_start..line_end), the bytes must occur within those lines. An empty excerpt fails. One unverifiable finding blocks the whole package.
• G7 model-derived findings passed the verifier. For each model_directed_text finding that quotes a tool description (every one except description_reassigned_at_runtime), the excerpt must lie inside one of the description spans of a tool declared at the same path in the package's surfaces.json row: the same test attestari.checks.model_directed_text.verify_proposal applies when the proposal is made. A proposal the verifier discarded is never a finding; a finding that would not pass the verifier now blocks the package.

The gate recomputed in G3 refuses a package for any of the following reasons, and records each one it finds:

"We cannot tell you what this does" is a legitimate published outcome, and for a meaningful share of popular packages it is the accurate one.

10. Finding decisions

The policy named severity-ladder decides each finding of a gradeable package:

• D1 confirmed. A verified finding with severity high, medium or low is confirmed at the severity its check assigned. The policy neither raises nor lowers a severity.
• D2 noted. A verified finding with severity info is noted: published as context, no effect on the grade.
• D3 withheld. Every finding of a package that is not gradeable is withheld, and its row lists the package's blocking reasons. A withheld finding is never published as a finding.

There are no suppression rules in this version. A finding that should not count is a methodology change (add a rule here, bump the version) or an appeal (section 16), never an edit to a run.

11. Package grade

From the confirmed findings of a gradeable package:

A package that is not gradeable has no grade; its row carries blocked_by instead.

12. What every decision row records

policy, methodology_version, methodology_sha256, code_commit (the git commit of the code that decided; code_dirty says whether the tree had uncommitted changes), decided_at, the finding's evidence_sha256 (the blob it was verified against) and excerpt_sha256, the verification outcome and the rule that produced the decision. The run-level header repeats the stamps and records the SHA-256 of every input file. A rerun over unchanged inputs with the same methodology and code produces identical rows apart from decided_at.

13. Automated decision, human responsibility

Findings are decided by the rules in this document, applied by software, without a person reviewing each one. This is a deliberate choice. Rules applied by a machine are applied identically to the two-hundredth subject and the first; a person working through hundreds of findings is not.

What is not automated is the rules themselves. Changing them is an explicit act that produces a new version of this document (section 19), and every finding cites the version that produced it. The methodology, not any individual judgement call, is what stands behind a rating.

Running the rules is not a human act and needs no approval. The two things a person does:

1. Appeals. A vendor disputing a finding files an appeal against its finding id (attestari appeal file). A person resolves it (attestari appeal resolve) as upheld, dismissed or withdrawn. Appeals are an append-only event log beside the run (appeals.json); nothing in decisions.json is edited. Publication applies them: an upheld appeal publishes the finding as withdrawn_on_appeal and recomputes that package's grade without it; an open appeal publishes the finding marked under_appeal, unchanged. What an appeal can be about, and how we answer one, is in section 16.
2. Methodology changes. Editing this document and the matching constant, as described in section 19.

Publication is separate. Producing findings and publishing them about a named third party are distinct steps. Automation covers the first. The second is attestari publish <run>: one command, not a review queue, run only after the notice period in section 16. It refuses a run whose decisions were made under a methodology version or document hash other than the current one, and never overwrites an earlier publication.

14. Scheduled operation

attestari cycle --candidates <file> performs one cycle with no human input: it resumes the most recent failed or cancelled run over that candidates file if there is one, otherwise starts a new run; then decides the run under this methodology, diffs it against the previous decided run over the same candidates file, and writes decisions.json, decisions.md and diff.json into the run directory. --every <seconds> repeats on that cadence in-process; a scheduler may instead invoke the one-shot form.

15. Known limits

These are the things a passing result does not tell you. They are stated here rather than in a footnote because a reader who does not know them will over-read our findings.

We do not run the code. Analysis is static. A subject that behaves differently when executed, against a particular host, after a delay, or only in the presence of certain credentials, presents none of that to us.

We do not examine binaries. Where a package's substance is a compiled executable it downloads, we can describe the launcher and the fetch precisely, and we can say nothing whatsoever about what the binary does.

Text assigned at runtime is invisible to us. A tool description that is benign in source and rewritten when the module loads reaches the agent in its rewritten form. We detect and report that a description is reassigned at runtime, but we cannot read what it becomes.

Tool names built by expression may be missed. Where a tool's name is computed rather than written literally, our extractor can fail to find a tool that genuinely exists. The gate in section 9 catches this as a mismatch against advertised tools rather than passing the subject silently.

We do not diff the artifact against the repository. A file present in the published package with no counterpart in the source tree is a signal we do not currently produce. See section 3.

Some checks are not yet implemented. Undeclared network destinations, dependency risk, maintenance status, and fork divergence are not assessed. Where a report lists these as not assessed, that is what it means, not that they were assessed and found clean.

A finding is about one version, at one time. Registries are mutable. A result describes the artifact we fetched, identified by hash, on the date stated. It says nothing about what the same version number serves tomorrow.

16. Notice, appeals, and corrections

Notice before publication. Before any finding about a named subject is published, we send it to the publisher at the contact address in their registry record or repository, and wait 30 days. That period is for correcting us, and a publisher is under no obligation to respond.

Appeals. A publisher may dispute any finding at any time, before or after publication, by writing to appeals@attestari.ai. An appeal that identifies a factual error, such as a misquoted span, a wrong line reference, or a claim about code that is not there, is checked against the stored evidence. If we are wrong, we correct it.

An appeal that disputes our interpretation rather than our facts will be considered and may be published alongside the finding. We do not remove an accurate finding because a publisher objects to it.

Corrections. Corrections are published, dated, and describe what was wrong and what changed. We do not silently edit a published finding.

Remediation. When a publisher fixes the thing a finding describes, a re-examination of the new version is free and prompt, and the updated result is published.

17. Independence

A publisher cannot pay us, in money or in kind, for a rating, for a better rating, or for the removal of a finding. There is no paid review, no expedited review, and no arrangement under which a rated party influences its own result.

The subjects we examine are chosen by the mechanical rule in section 4. They are not chosen by request, and they are not chosen by who has paid for anything.

Where Attestari earns revenue, the source is disclosed. Any commercial relationship with a party that is also a subject is disclosed on that subject's published result.

18. Measured performance

The description-analysis check is evaluated against a corpus of publicly disclosed tool-poisoning cases, published by third-party security researchers, plus a control set of tool descriptions from widely-used packages reviewed by hand. Current figures:

• 5 of 6 disclosed cases detected
• 22 of 27 expected spans found
• 1 false positive across 456 clean tools
• 1 of 1 in-the-wild case detected

The undetected case is one where the poisoned text is assigned when the module loads and therefore never appears in the source we read: a limit stated in section 15, not a tuning failure.

These figures are measured against a small corpus, because the set of publicly disclosed cases is small. They should be read as evidence that the check works on known attacks, not as a general detection rate. They will be restated whenever the corpus or the check changes.

Where the figures come from: the evaluation run recorded at runs/eval-separated-live (2026-09-09, claude-opus-5, the separated prompt layout), scored by attestari eval against the corpus in seeds/disclosed_cases.json and the clean set in runs/twenty/surfaces.json. In that evaluation only, each seed sample's evidence block cited the publication that disclosed it; a production run carries no such citation.

19. Versioning

This document carries a version: the methodology_version: line at the top, MAJOR.MINOR.PATCH. Every published finding cites the version in force when it was produced and the SHA-256 of the document. Superseded versions remain available at their own permanent addresses, so a finding published in 2026 can still be read against the rules that made it in 2026.

A change to what a check examines, what a severity means, or when we refuse to grade is a version change. Correcting a typo is not.

Changing any rule is a deliberate act: edit this document, bump methodology_version, bump METHODOLOGY_VERSION in attestari/methodology.py, add a line to the Versions section, copy the document to a new snapshot under the site's content/methodology/, and commit. A published snapshot is never edited.

Versions

• 1.0.0 (2026-09-11): first automated policy. Guards G1-G7, decisions D1-D3, grade ladder A/B/C/F, appeals and publication as the two human acts.
• 1.1.0 (2026-09-11): the public methodology around the unchanged rules: what the document is for, definitions, artifact not repository, subject selection, the four checks, severity definitions, evidence and verification, known limits, notice, appeals and corrections, independence, measured performance, versioning. Guards, decisions and the grade ladder are unchanged from 1.0.0.