vibeaudit: detect security smells in AI-written code in AI agents

An offline, standard-library-only Python 3 CLI that scans Python, JavaScript and TypeScript (and optionally shell) source for security smells that show up a lot in AI-generated ("vibe-coded") code: eval on user input, shell=True with f-strings, verify=False, SQL built with f-strings or template literals, pickle/yaml.load, md5 password hashes, Math.random() tokens, hardcoded keys, app.run(debug=True, host="0.0.0.0"), CORS * with credentials, unverified JWTs, innerHTML/dangerouslySetInnerHTML, and the classic LLM placeholders such as # TODO: add authentication and return True # bypass for now.

What it detects

An offline, standard-library-only Python 3 CLI that scans Python, JavaScript and TypeScript (and optionally shell) source for security smells that show up a lot in AI-generated ("vibe-coded") code: eval on user input, shell=True with f-strings, verify=False, SQL built with f-strings or template literals, pickle/yaml.load, md5 password hashes, Math.random() tokens, hardcoded keys, app.run(debug=True, host="0.0.0.0"), CORS * with credentials, unverified JWTs, innerHTML/dangerouslySetInnerHTML, and the classic LLM placeholders such as # TODO: add authentication and return True # bypass for now.

Honesty note. vibeaudit is a heuristic pattern scanner, not a SAST replacement. It uses regular expressions on single lines, a few multi-line regexes, and (for Python) cheap AST checks. It does not do data-flow or taint analysis, so it cannot tell whether a value is really attacker-controlled. It cannot tell whether code was written by an AI. The rules are just patterns often seen in LLM output, and humans write them too. Expect false positives and false negatives. A clean run does not mean the code is secure. Use it as a quick first-pass review aid next to real tools (Semgrep, CodeQL, Bandit, ESLint security plugins) and human review.

Rules

This tool does not publish a numbered rules table; see the README for what it flags.

Usage

python3 vibeaudit.py .                           # scan the current project
python3 vibeaudit.py src app.py --shell          # include shell scripts found in dirs
python3 vibeaudit.py . --min-severity medium     # hide low findings
python3 vibeaudit.py . --json > findings.json
python3 vibeaudit.py . --sarif > vibeaudit.sarif
python3 vibeaudit.py . --exclude 'tests/*' --exclude '*.spec.ts'
python3 vibeaudit.py . --select VA05,VA09        # only some rules
python3 vibeaudit.py . --ignore VA15             # drop a noisy rule
python3 vibeaudit.py --list-rules

After pip install of the kit, the same options work as agentsec vibeaudit …. Every tool also accepts --help and almost every tool accepts --json.

Example output

The full sample set (python3 vibeaudit.py examples --shell) triggers all 17 rules:

The safe counterparts produce nothing:

$ python3 vibeaudit.py examples/vulnerable_app.py --select VA02,VA05,VA14
examples/vulnerable_app.py:25:5: HIGH VA05 SQL passed to .execute() is built with string formatting/concatenation (CWE-89)
    > cur.execute(f"SELECT * FROM users WHERE name = '{name}'")
    fix: Use parameterized queries (cursor.execute(sql, params), db.query(sql, [values]), ORM bind parameters).
examples/vulnerable_app.py:31:12: HIGH VA05 SQL variable 'sql' (built with formatting at line 30) passed to .execute() (CWE-89)
    > return sqlite3.connect("app.db").execute(sql).fetchall()
    fix: Use parameterized queries (cursor.execute(sql, params), db.query(sql, [values]), ORM bind parameters).
examples/vulnerable_app.py:35:12: HIGH VA02 subprocess.check_output(shell=True) with a formatted/concatenated command (CWE-78)
    > return subprocess.check_output(f"ping -c 1 {host}", shell=True)
    fix: Pass an argument list without a shell (subprocess.run([...], shell=False)); use shlex.quote only as a last resort.
examples/vulnerable_app.py:39:5: HIGH VA02 os.system() with a non-literal command string (CWE-78)
    > os.system("tar czf /tmp/backup.tgz " + path)
    fix: Pass an argument list without a shell (subprocess.run([...], shell=False)); use shlex.quote only as a last resort.
examples/vulnerable_app.py:70:1: HIGH VA14 Auth/permission function 'is_admin' unconditionally returns True (CWE-287)
    > def is_admin(user):
    fix: Implement the check (or fail closed) before shipping; remove 'return True' stubs and bypass flags.
examples/vulnerable_app.py:71:7: MEDIUM VA14 TODO left in place of an authentication/authorization/validation check (CWE-287)
    > # TODO: add authentication and role checks
    fix: Implement the check (or fail closed) before shipping; remove 'return True' stubs and bypass flags.

vibeaudit: 6 finding(s) (5 high, 1 medium, 0 low) in 1 file(s) scanned.
Heuristic pattern scan only - review each finding; absence of findings is not proof of safety.
$ echo $?
1

SARIF and CI

agentsec vibeaudit path/to/input --sarif > vibeaudit.sarif

# GitHub Actions (composite action ships with the repo)
- uses: dunkleschitz/agentsec@v0
  with:
    tools: vibeaudit
    paths: path/to/input
    fail-on: high

SARIF 2.1.0 can be uploaded to GitHub code scanning. The composite action writes per-tool reports under agentsec-sarif/ by default.

OWASP mapping

Heuristic mapping of what this tool helps a reviewer look for. Not an official OWASP endorsement, and not complete coverage of any item. Sources: OWASP Top 10 for LLM Applications 2025 and OWASP Top 10 for Agentic Applications 2026.

LLM05 Improper Output Handling — OWASP LLM Top 10 (2025)
ASI05 Unexpected Code Execution (RCE) — OWASP Agentic Top 10 (2026)

Known limits

No taint tracking: eval(CONSTANT_FROM_CONFIG) is flagged and eval reached through a wrapper is not.
Line-based JS/TS matching: calls split across lines, or strings built in one place and used far away, can be missed.
Context heuristics (VA07/VA08 look for words like *password*/*token* in the line and the 3 lines before it) can misfire either way.
Test fixtures for secret scanners and similar tools will (correctly but unhelpfully) trigger VA09. Use --exclude or suppressions.

Full source, tests, and fixtures: tools/vibeaudit/ in the repository. Install the kit with pip install git+https://github.com/dunkleschitz/agentsec.git.