tunescan: detect poisoned or backdoored fine-tuning data in AI agents

Offline poisoning and hygiene scanner for LLM fine-tuning datasets (OWASP LLM04 Data and Model Poisoning). Run it on a JSONL file before you upload it for fine-tuning. It looks for the fingerprints a poisoned or careless dataset leaves behind: a rare trigger phrase that always maps to one fixed answer, an outside domain the completions keep pointing at, injected instructions, hidden Unicode, dangerous commands taught as answers, conflicting labels, embedded secrets, heavy duplication, and malformed records.

What it detects

Offline poisoning and hygiene scanner for LLM fine-tuning datasets (OWASP LLM04 Data and Model Poisoning). Run it on a JSONL file before you upload it for fine-tuning. It looks for the fingerprints a poisoned or careless dataset leaves behind: a rare trigger phrase that always maps to one fixed answer, an outside domain the completions keep pointing at, injected instructions, hidden Unicode, dangerous commands taught as answers, conflicting labels, embedded secrets, heavy duplication, and malformed records.

Standard library only. No network calls. Secret values are masked in every output format.

Rules

Rule	Severity	What it flags
TS01	high	Backdoor trigger: a 1–3 word phrase found in a small share of prompts (at least `--trigger-min`, at most `--trigger-rate`) whose prompts are otherwise unrelated, yet 90%+ map to one completion that is rare elsewhere. Needs `--min-records` (default 20)
TS02	medium	A domain that appears in 3+ completions but never in any prompt (promotion or exfil seeding). `example.*` and `--allow-domain` are skipped
TS03	high (role tokens medium)	Instruction-override phrasing ("ignore all previous instructions", "when you see 'X', always ..."), "don't tell the user", and fake chat-template delimiters (`<\
TS04	high (critical with TAG text)	Zero-width, bidi, and Unicode TAG characters; TAG payloads are decoded
TS05	high (TLS-off medium)	Completions that teach `curl \
TS06	medium	The same prompt with different short completions (conflicting labels)
TS07	critical	Secret-shaped values: OpenAI/AWS/GitHub/Slack/Google key shapes, private key blocks, JWTs, credentialed connection strings, `password=...` labels (placeholders skipped)
TS08	low	One example repeated more than `--max-dupes` times (over-weighting)
TS09	medium	Bad JSON, unknown record shape, or no assistant turn
TS10	info	Inventory per file: record count, formats, and roles

Usage

python3 tunescan.py DATASET [...] [--json | --sarif] [--per-file]
                    [--max-dupes N] [--min-records N] [--trigger-min N] [--trigger-rate F]
                    [--allow-domain DOMAIN] [--min-severity LEVEL] [--fail-on LEVEL|none]
                    [--disable RULE] [--list-rules]

After pip install of the kit, the same options work as agentsec tunescan …. Every tool also accepts --help and almost every tool accepts --json.

Example output

examples/clean_chat.jsonl is the same support dataset without the poison and produces only the TS10 inventory line. examples/conflicting_labels.jsonl shows TS06 on a sentiment dataset. All data is synthetic and every secret is a FAKE-EXAMPLE-... value.

cd tools/tunescan
python3 examples/make_examples.py      # regenerates the synthetic datasets
python3 tunescan.py examples/poisoned_chat.jsonl --min-severity medium

SARIF and CI

agentsec tunescan path/to/input --sarif > tunescan.sarif

# GitHub Actions (composite action ships with the repo)
- uses: dunkleschitz/agentsec@v0
  with:
    tools: tunescan
    paths: path/to/input
    fail-on: high

SARIF 2.1.0 can be uploaded to GitHub code scanning. The composite action writes per-tool reports under agentsec-sarif/ by default.

OWASP mapping

Heuristic mapping of what this tool helps a reviewer look for. Not an official OWASP endorsement, and not complete coverage of any item. Sources: OWASP Top 10 for LLM Applications 2025 and OWASP Top 10 for Agentic Applications 2026.

LLM04 Data and Model Poisoning — OWASP LLM Top 10 (2025)

Known limits

TS01 is a statistical heuristic. It catches literal trigger phrases repeated across unrelated

prompts. Semantic triggers (a topic or style rather than a token), triggers spread across many spellings, clean-label poisoning, and triggers in more than --trigger-rate of the data are missed. Very templated datasets can produce false positives; tune --trigger-rate or disable it.

It reads the data you give it; it doesn't train or probe a model, so it can't confirm a backdoor

works. Pair it with an evaluation run on the trained model.

English-only phrase heuristics for TS03. Secret detection is shape-based and doesn't verify keys.
Whole files are loaded into memory; split very large datasets into shards (pooling still works).
Personal data in training sets is piiguard's job; run it alongside.

Full source, tests, and fixtures: tools/tunescan/ in the repository. Install the kit with pip install git+https://github.com/dunkleschitz/agentsec.git.