A 10-minute security audit for your AI agent

You do not need a red team engagement to find the boring, high-severity failures in an AI agent: prompt injection in tool results, secrets hopping into webhooks, PII in shared transcripts, shell commands that should never have been allowed, and MCP tool definitions that drifted since last week.

This walkthrough uses agentsec — 39 small Python scanners, standard library only, no network, no telemetry. Budget: about ten minutes if you already save transcripts.

Minute 0–1: clone and pick your inputs

git clone https://github.com/dunkleschitz/agentsec.git
cd agentsec

You need one or more of:

• An agent transcript (OpenAI chat JSON, Anthropic messages, or generic JSONL with tool calls)
• An MCP tools/list snapshot (for pinning / poisoning checks)
• Optional: shell-command lists, URL lists, memory dumps

If you have nothing yet, use the bundled fixtures — every command below runs against fake data shipped in the repo.

Minute 1–3: tool-result injection (resultguard)

Untrusted content returns through tools. Scan it first.

cd tools/resultguard
python3 resultguard.py examples/poisoned_openai.json --fail-on high

Expect HIGH findings: override phrasing (TR01), tool-chaining bait (TR04), webhook callbacks (TR05), credential harvest (TR07). Exit 1 = gate fail. Clean control:

python3 resultguard.py examples/clean_openai.json --fail-on high; echo exit=$?
# exit=0

Pass criteria for your real runs: no TR01–TR05 / TR07 at high+, or every hit reviewed and --disabled with a reason.

Minute 3–5: secret propagation (secretflow)

Presence of a secret in a log is bad. Reuse of that same value in a later outbound tool is how exfil happens (read .env → http_post).

cd ../secretflow
python3 secretflow.py examples/risky_openai.json --disable FL10 --fail-on high

Typical hits on the fixture:

• FL01 critical — env secret from read_file into http_post
• FL02 critical — session-like secret into bash
• FL05 / FL07 — assistant echo or labeled api_key= reuse
• FL09 — secret written via a sink tool

Matches are masked in the report (FAKE…9c2e). On your transcripts: any FL01/FL02/FL04 is a stop-ship until the tool policy blocks that hop.

Minute 5–6: PII before you share the run (piiguard)

About to paste a transcript into a ticket or training set? Redact first.

cd ../piiguard
python3 examples/make_examples.py   # regenerates fake fixtures
python3 piiguard.py examples/risky_openai.json --redact /tmp/redacted.json --fail-on high

Flags emails (PG01), phones (PG02), SSNs (PG03), Luhn-valid cards (PG04), bank/IBAN (PG06), and more. The --redact copy is what you attach to the ticket; keep the original in a restricted store.

Minute 6–8: paths, shells, fetches (pathfence / shellfence / fetchfence)

Three quick passes over the same transcript shape:

cd ../pathfence
python3 pathfence.py examples/risky_openai.json --disable PF12 --fail-on high

cd ../shellfence
python3 shellfence.py examples/risky_openai.json --disable SF12 --fail-on high

cd ../fetchfence
python3 fetchfence.py examples/risky_openai.json --fail-on high

What you are looking for:

Disable the inventory rules (PF12 / SF12) in CI so info noise does not fail the gate.

Minute 8–9: human-in-the-loop gaps (hitlguard)

Did the agent pay, delete, deploy, or send mail without a nearby confirmation?

cd ../hitlguard
python3 hitlguard.py examples/risky_openai.json --disable HG08 --fail-on high

Pair with a policy that requires an approval tool for consequential actions; hitlguard checks whether the transcript actually shows that approval.

Minute 9–10: pin MCP tools (toolpin)

If you use MCP, freeze today's definitions and catch tomorrow's rug-pull.

cd ../toolpin
python3 toolpin.py check examples/drifted.json -l examples/toolpin.lock.json --fail-on high

Description injection, loosened enums, readOnlyHint flips, and cross-server shadowing all show up here. Re-pin only after review:

python3 toolpin.py pin examples/baseline.json -l toolpin.lock.json

Optional bonus: posture and loops

If you still have a minute:

cd ../mcpposture && python3 mcpposture.py examples/risky_claude_desktop_config.json
cd ../loopguard && python3 loopguard.py examples/runaway_openai.json
cd ../yolocheck && python3 yolocheck.py examples/risky

mcpposture grades MCP client configs A–F. loopguard catches identical tool-call storms (especially dangerous for send/delete/pay tools). yolocheck grades coding-agent autonomy settings.

Drop-in CI shape

All of these tools share exit codes (0 clean, 1 findings, 2 usage). A minimal composite gate:

- name: agentsec quick audit
  run: |
    set -e
    ROOT=tools
    python3 $ROOT/resultguard/resultguard.py runs/ --fail-on high
    python3 $ROOT/secretflow/secretflow.py runs/ --disable FL10 --fail-on high
    python3 $ROOT/piiguard/piiguard.py runs/ --disable PG10 --fail-on high
    python3 $ROOT/pathfence/pathfence.py runs/ --disable PF12 --fail-on high
    python3 $ROOT/shellfence/shellfence.py runs/ --disable SF12 --fail-on high
    python3 $ROOT/fetchfence/fetchfence.py runs/ --fail-on high
    python3 $ROOT/toolpin/toolpin.py check mcp-tools.json \
      -l $ROOT/toolpin/toolpin.lock.json --fail-on high

Vendor the tools/*/…py files you need (each is self-contained) or submodule the repo. No pip install step.

What "passing" means

A green run means: on the artifacts you scanned, no high/critical heuristic fired. It does not mean the live agent cannot be coerced, that every secret transform was caught, or that a novel MCP server is benign. agentsec is a human-review aid with CI teeth — keep the runtime allowlists and approval prompts too.

Start here

git clone https://github.com/dunkleschitz/agentsec.git
cd agentsec/tools/resultguard
python3 resultguard.py examples/poisoned_openai.json

Ten minutes, zero dependencies, a punch list of real findings. Expand from there as your threat model grows — the other ~28 tools cover RAG ingestion, prompt leakage, model pinning, schema privilege, memory poisoning, and more: github.com/dunkleschitz/agentsec.