tracesweep: detect secrets leaked into agent transcripts in AI agents

What it detects

tracesweep scans AI agent transcripts and logs for secrets that leaked into the conversation. That covers API keys printed by a tool, tokens passed as tool-call arguments, and credentials the model repeated back. It attributes each finding to the place it surfaced: file, line or record, turn, role, tool name, and location kind. It also flags secrets that propagated across turns. For example, a key can first appear in a cat .env tool output and then show up again in the assistant's reply.

Honesty notice. tracesweep is pattern-based. It can miss secrets: custom token formats, secrets split across messages, or encoded/obfuscated values. It can also produce false positives. It does not verify whether a key is live or valid, and it never contacts any service. A clean result does not prove that a transcript is free of secrets. Treat findings as leads to triage, and rotate any real credential that you find.

Rules

Usage

python3 tracesweep.py session.jsonl                    # table
python3 tracesweep.py ~/agent-logs/                    # recursive
cat run.log | python3 tracesweep.py -                  # stdin
python3 tracesweep.py --json --min-severity high logs/ # machine-readable, high+ only
python3 tracesweep.py --sarif logs/ > tracesweep.sarif # SARIF 2.1.0 for code-scanning UIs
python3 tracesweep.py --entropy logs/                  # add noisy entropy fallback
python3 tracesweep.py --redact clean/ logs/            # write redacted copies
python3 tracesweep.py --list-rules

After pip install of the kit, the same options work as agentsec tracesweep …. Every tool also accepts --help and almost every tool accepts --json.

Example output

The examples in examples/ contain synthetic, fake keys that only match the token shapes. They are not committed (so secret scanners don't flag shape-valid fakes); generate them with python3 examples/make_examples.py before running the commands below. The test suite generates them automatically.

Line 6 of claude_session.jsonl mentions AKIAIOSFODNN7EXAMPLE and your_api_key, and clean.json contains OPENAI_API_KEY=your_api_key_here and max_tokens: 4096. Both are correctly ignored.

$ python3 tracesweep.py examples
SEV       RULE           FILE:LINE                        REC  TURN  ROLE       TOOL        WHERE        PREVIEW                ECHO
--------  -------------  -------------------------------  ---  ----  ---------  ----------  -----------  ---------------------  ----
MEDIUM    TS-GENERIC     examples/agent_run.log:3         3    3     tool       bash        text         hunt...ss              x2
HIGH      TS-URL-CREDS   examples/agent_run.log:4         4    4     tool       bash        text         Fak3...d!
MEDIUM    TS-GENERIC     examples/agent_run.log:5         5    5     assistant  -           text         hunt...ss              x2
HIGH      TS-AWS-AKID    examples/claude_session.jsonl:3  3    3     tool       Bash        tool_output  AKIA...L3
CRITICAL  TS-AWS-SECRET  examples/claude_session.jsonl:3  3    3     tool       Bash        tool_output  wJal...23
CRITICAL  TS-OPENAI      examples/claude_session.jsonl:3  3    3     tool       Bash        tool_output  sk-p...GH              x2
HIGH      TS-URL-CREDS   examples/claude_session.jsonl:3  3    3     tool       Bash        tool_output  Fak3...d!
CRITICAL  TS-OPENAI      examples/claude_session.jsonl:4  4    4     assistant  -           message      sk-p...GH              x2
HIGH      TS-GITHUB      examples/claude_session.jsonl:5  5    5     assistant  WebFetch    tool_args    ghp_...01
HIGH      TS-SLACK       examples/openai_chat.json:19     3    3     assistant  slack_post  tool_args    xoxb...01
MEDIUM    TS-JWT         examples/openai_chat.json:27     4    4     tool       slack_post  tool_output  eyJh...89
CRITICAL  TS-PRIVKEY     examples/openai_chat.json:31     5    5     assistant  -           message      [PEM RSA PRIVATE KEY]
CRITICAL  TS-STRIPE      examples/openai_chat.json:35     6    6     user       -           message      sk_l...ST
HIGH      TS-GOOGLE-API  examples/openai_chat.json:35     6    6     user       -           message      AIza...de
CRITICAL  TS-ANTHROPIC   examples/openai_chat.json:39     7    7     assistant  -           message      sk-a...xw

15 finding(s) (6 critical, 6 high, 3 medium) in 4 file(s); 2 secret(s) echoed across turns.
Propagation:
  examples/agent_run.log TS-GENERIC hunt...ss: text(turn 3) -> text(turn 5)
  examples/claude_session.jsonl TS-OPENAI sk-p...GH: tool_output(turn 3) -> message(turn 4)
Note: pattern-based; may miss secrets or flag false positives; keys are not verified as live.

SARIF and CI

agentsec tracesweep path/to/input --sarif > tracesweep.sarif

# GitHub Actions (composite action ships with the repo)
- uses: dunkleschitz/agentsec@v0
  with:
    tools: tracesweep
    paths: path/to/input
    fail-on: high

SARIF 2.1.0 can be uploaded to GitHub code scanning. The composite action writes per-tool reports under agentsec-sarif/ by default.

OWASP mapping

Heuristic mapping of what this tool helps a reviewer look for. Not an official OWASP endorsement, and not complete coverage of any item. Sources: OWASP Top 10 for LLM Applications 2025 and OWASP Top 10 for Agentic Applications 2026.

• LLM02 Sensitive Information Disclosure — OWASP LLM Top 10 (2025)

Known limits

• Regex/heuristic only, with no live-key verification and no provider API calls.
• Secrets split across chunks or messages, or base64/URL-encoded inside other data, are usually missed.
• Turn numbering is best-effort and depends on the log format. Echo detection is per file.
• Line numbers for pretty-printed JSON arrays are located by searching the raw text, so they are best-effort.
• Very large files are read fully into memory.

Full source, tests, and fixtures: tools/tracesweep/ in the repository. Install the kit with pip install git+https://github.com/dunkleschitz/agentsec.git.