Skip to content

Why AgentAuditKit works the way it does

The reasoning that used to sit above the fold in the README. It is a better argument than it was a landing page.

Auto-PR for mechanical fixes

agent-audit-kit scan . --format sarif -o run.sarif
agent-audit-kit suggest run.sarif --auto-pr --dry-run   # see the plan
agent-audit-kit suggest run.sarif --auto-pr             # open a draft PR

Off by default, and deliberately narrow:

  • Allow-listed rules only. If any pending fix is for a rule outside AUTO_PR_ALLOWLIST, the whole run refuses rather than opening a partial PR. The allow-list is an explicit literal, not auto_fixable — marking a new rule auto-fixable does not, on its own, make AAK push it.
  • Draft, never merged. A mechanical edit nobody reviewed is a diff, not a decision.
  • No credentials. Delivery runs through your existing gh auth. AAK never asks for, stores, or reads a token, so it cannot exceed the access you already granted gh.
  • Refuses on a dirty tree, so your uncommitted work is never swept into its branch.

Fixes whose correct form depends on how the project is deployed or wired — adding an auth dependency to a route, rewriting a quoted shell string as a parameterised call, flipping a bind address off 0.0.0.0 — are reported but never auto-edited. The PR body says so too.


What we scan, and what we refuse to guess

AAK reads the artifacts an agent loads — MCP configs, SKILL.md, the named instruction files, hooks, workflows, manifests, source — and it does read the free text inside them, because a hidden HTML comment in a SKILL.md is invisible to the human reviewing the rendered page and fully visible to the model handed the raw file. That gap is the attack, and closing it is AAK-SKILL-006.

What AAK does not do is decide whether an arbitrary English sentence is hostile. A markdown file no agent loads by name is checked against a dated corpus of payloads seen in the wild, and nothing else. That is a signature check and it will miss novel phrasings; we would rather say so than imply a classifier we do not ship.

The reason is not modesty, it is the failure mode on the other side. A cross-tool corpus filed as [issue

742](https://github.com/sattyamjjain/agent-audit-kit/issues/742) measured a

pattern-classifier layer firing on four benign files: a documentation page about data: imports, a hardening note, ordinary env-var credential reads, and a socket health check. Every one of those is the kind of finding that teaches a team to stop reading the output. A deterministic scanner's whole claim is that when it speaks, it is worth reading; spending that on prose we cannot verify is a bad trade.

The same issue is why two of those detections exist at all. The reporter's two files were misses, not scope calls: the wild-payload regex matched "ignore all instructions" and "ignore previous instructions" but not "ignore all previous instructions", and skills were missing from the hidden-comment check that named instruction files have had since v0.2. Both are fixed. The honest limit that remains is language: the corpus is English-only, so 「请忽略之前的所有指令」 goes undetected.

Supply chain

Every v* release publishes:

  • Wheel + sdist on PyPI via OIDC Trusted Publisher
  • Docker image on GHCR (ghcr.io/sattyamjjain/agent-audit-kit:<version>, the version without its v) with SLSA provenance attestation. Provenance is attached by the release flow, so it covers version tags; the nightly rebuild that refreshes :latest and :nightly scans the image with Trivy but attaches no attestation, so pin a version tag if you need provenance
  • Sigstore keyless-signed rule bundle (rules.json + rules.json.sha256)
  • CycloneDX + SPDX SBOM of the package and its runtime dependencies (sbom.cdx.json, sbom.spdx.json), each Sigstore-signed. Releases through v0.6.9 attached SBOMs that listed no components: the job described this repository's MCP servers, of which it has none

For your own project, agent-audit-kit sbom and agent-audit-kit vex emit the SBOM of the MCP servers it declares and an OpenVEX 0.2.0 document that joins to it on purl (VEX). No release publishes a VEX document of its own.

Verify a wheel or sdist from PyPI against its PEP 740 attestation:

pip install pypi-attestations
pypi-attestations verify pypi --repository https://github.com/sattyamjjain/agent-audit-kit pypi:agent_audit_kit-<version>-py3-none-any.whl

It prints OK: <file> when this repository signed the provenance through PyPI's Trusted Publishing, and fails if another repository did. For the sdist, name pypi:agent_audit_kit-<version>.tar.gz. Checked with pypi-attestations 0.0.31 on 0.6.31.

Verify a bundle against its signature and signer:

pip install "agent-audit-kit[verify]"
agent-audit-kit verify-bundle rules.json --signature rules.json.sigstore.json --tag v<version>

It checks the Sigstore bundle against the file and that the signer is this repository's release.yml on that tag. Without --tag, any release tag is accepted, since the bundle does not record its own version; --offline uses the trust root that ships with sigstore. Through v0.6.9 the command could not verify a signature at all, so for those releases use the Sigstore CLI: python -m sigstore verify identity rules.json --bundle rules.json.sigstore.json --cert-identity https://github.com/sattyamjjain/agent-audit-kit/.github/workflows/release.yml@refs/tags/v<version> --cert-oidc-issuer https://token.actions.githubusercontent.com.


State of MCP Security 2026

The measured numbers live in the report itself, not here. Copying them into a second file is how a figure goes stale: scripts/sync_rule_count.py regenerates the headline numbers in README.md from results.json and asserts them in CI, and it does not know about this page.

Read State of MCP Security 2026 (how to cite), the frozen pre-2026-07-28 baseline, and the corpus manifest.