AI audit report: five things a good one delivers

A useful AI audit report delivers five things: verified findings ranked by severity, an ROI verdict for each AI feature, a map of the prompt attack surface, an eval baseline you can rerun, and a fix order. At Zeroic that takes 1-2 weeks of senior engineering time. A maturity scorecard plus a roadmap is a consulting deck.

AI audits got popular for the same reason AI pilots got expensive. MIT’s Project NANDA reported that 95% of organizations saw no business return on $30-40 billion of enterprise generative AI spend, and founders now want a second opinion before they build, or before they ship what they built. The market answered with readiness assessments that end in a roadmap. That’s fine for a board meeting. It tells your engineers nothing on Monday morning.

This piece is for founders and CTOs at 5-50 person companies who are about to pay for an audit, or just got one back and want to know whether it’s any good.

Five artifacts every AI audit report needs

Each one is something your team can open, run or act on. If an artifact on this list is missing from the report you’re paying for, ask why.

1. Ranked findings with repro steps

Every issue is verified, ranked by severity and business impact, and comes with the steps to reproduce it and a recommended fix. Ranked means one ordered list - a heat map with twelve “medium” cells doesn’t count. Repro steps mean your engineer can see the problem in minutes instead of debating whether it exists. A finding without repro steps is an opinion.

2. An ROI verdict for each AI feature

For every AI feature you have or plan, the report states what task the model does, how often it runs, what a wrong answer costs you, and whether the feature pays for itself at your volume. The useful verdicts are short: build it, buy it, shrink it, or kill it. “Kill it” should be on the table, and an auditor who also hopes to sell you the build has a reason to never say it. Ask any auditor when they last told a client not to build.

3. A map of the prompt attack surface

Prompt injection is LLM01, the top entry in OWASP’s 2025 Top 10 for LLM applications, and OWASP states that RAG and fine-tuning do not fully mitigate it. So the report should list every place untrusted text reaches a model - user chat, uploaded files, scraped pages, retrieved documents, inbound email - and what the model can do next: which data it reads, which tools it calls, where it can send output.

Simon Willison’s lethal trifecta is the fastest test: access to private data, exposure to untrusted content, and a way to communicate externally. A feature with all three can be tricked into leaking data. Filters don’t close the hole, and Willison’s verdict on guardrail products that catch 95% of attacks is that “95% is very much a failing grade” in security. The fix is architectural - remove one leg of the trifecta - and the report should say which leg, per feature.

Two cheaper findings belong here too. OWASP says the system prompt should not be considered a secret, nor used as a security control, so credentials or permission rules living in a prompt are a finding. And unbounded consumption - no per-user rate or spend limit on model calls - lets one scripted abuser run up what OWASP calls a “Denial of Wallet”.

4. An eval baseline you can rerun

The audit should leave behind a test set built from your production inputs, a grading method, and today’s score. Without it, nobody can say whether the next prompt change or model swap made things better or worse. OpenAI’s own guide lists “vibe-based evals” - “it seems like it’s working” - as an anti-pattern, and recommends building eval data from production traffic. Anthropic’s eval guide favors volume: more automatically graded cases beat a handful of hand-graded ones. It also calls for edge cases - missing input, overly long input, hostile input, and cases where even humans would disagree.

The baseline is the artifact that keeps paying off after the auditors leave. It’s also what makes model choices cheap to revisit; our guide to choosing GPT, Claude or Llama by task starts from the same eval.

5. A fix order with effort attached

A severity ranking isn’t yet a plan. The last artifact sequences the findings: what to fix this week, what this quarter, and what to accept as a known risk, with a rough effort size on each. Then a walkthrough with the engineer who found them, so questions get answered by someone who read the code.

The slide-deck audit fails three ways

Most worthless audits share a format: maturity scores, a capability heat map, a three-horizon roadmap, and a recommendation to start a transformation program. Three things go wrong.

• It audits the org chart instead of the system. Interviews and questionnaires measure what people believe about the product. Nobody opened the code, ran a hostile prompt, or checked who can query the vector store.
• Nothing is verifiable. A finding with no repro steps can’t be confirmed, prioritized or closed. Six months later you can’t tell which items are done.
• The incentive points one way. When the auditor sells the build, every AI idea scores “high potential”. A report that never says no is a sales document.

The test is simple. If your engineers can’t turn page one into a ticket, the report was written for someone else.

Governance frameworks come after the technical audit

The strongest opposing view says an AI audit is a governance exercise: map the system to the NIST AI RMF Generative AI Profile, crosswalk it to the EU AI Act and ISO 42001, and document the controls. For an enterprise running regulated, high-risk systems, that’s correct. NIST published the profile in July 2024 as a companion to AI RMF 1.0, and it’s the right reference for that job.

For a 5-50 person team shipping its first or second LLM feature, it’s the wrong first step. Governance documentation describes controls; it doesn’t create them. Even compliance-focused guidance like Openlayer’s risk assessment report guide warns that “‘Policy exists’ is not a control status” - auditors want evidence the control runs. The technical audit produces that evidence: the attack-surface map, the eval baseline, the closed findings. Do it first, and the governance write-up later becomes a documentation task instead of a discovery project.

How to judge an AI audit before you buy

Six questions to ask before you sign:

Ask the auditor	Good answer	Walk away if
Who does the work?	A senior engineer who reads your code	An analyst runs a questionnaire
What do we get at the end?	Ranked findings with repro steps and fixes	A deck with maturity scores
Do you test prompts and tools by hand?	Yes - injection, leakage, spend limits	”We review your documentation”
Do we keep an eval set?	Yes, runnable by your team	Evals aren’t mentioned
Can the answer be “don’t build”?	Yes, and they’ve said it before	Every idea is “high potential”
How is it priced?	Fixed scope, fixed price, agreed upfront	An open-ended discovery phase

Then pick the track that matches where you are:

• No AI in production, three ideas on the table: buy the ROI read first. It’s the cheapest point to kill a bad idea.
• An LLM feature live with customer data: security first - attack surface, system prompt contents, spend limits.
• AI feature live, but quality is slipping or the model bill is climbing: eval baseline first, then route each call to the cheapest model that passes it.
• A vibe-coded app about to take paying users: audit the whole app, not just the AI. The five failure modes in Lovable and Bolt MVPs usually surface before the prompt-level ones.
• An enterprise customer or regulator asking for documentation: technical audit, then the governance crosswalk on top of its evidence.

100+ audits, from Sam’s List to VoiceDrop

Our audit work covers more than 100 applications across security and performance tracks, for clients including Sam’s List, VoiceDrop and AppSheet Training. Every one ships in the same format: verified findings ranked by severity, each with repro steps and a recommended fix. No slide deck. Bubble and Supabase are where we’ve done the most of it, because their abstraction layers hide the vulnerabilities generic scanners miss.

Our AI audit brings that format to the AI track: where a model would pay for itself in your product, what it would take to build, and where it wouldn’t be worth it. It starts at $1,000, takes 1-2 weeks, and ends with a 60-minute walkthrough with the engineer who did the work. If we don’t see ROI, we say so. When the findings point to rebuilding part of the app, that becomes a scoped Code Rescue engagement, and the audit report is its brief.

If you’re weighing an audit against just hiring someone to build, the rest of our client work shows where each path led.

Frequently asked questions

What should an AI audit report include?

Five things: verified findings ranked by severity with steps to reproduce each one, an ROI verdict for every AI feature you have or plan, a map of where untrusted text reaches a model and what the model can do next, an eval baseline your team can rerun, and a fix order with effort attached. Maturity scores and roadmaps are optional extras.

How much does an AI audit cost?

Our audits start at $1,000 at a fixed price agreed before work begins, set by the track and the size of the app. Governance audits for regulated enterprises, which map systems to frameworks like the NIST AI RMF, are a larger and different engagement. For a startup shipping its first LLM features, a fixed-price technical audit is the right starting point.

How long does an AI audit take?

One to two weeks for most startup products, in our experience. That covers reading the code, testing prompts and tools by hand, building an eval baseline and writing the ranked report, followed by a walkthrough with the engineer who did the work. Anything quoted as an open-ended discovery phase is usually consulting, sold under the audit label.

Is an AI readiness assessment the same as an AI audit?

No. A readiness assessment usually measures your organization through interviews and questionnaires and ends in a maturity score and a roadmap. An AI audit examines the system itself: the code, the prompts, the data access and the model calls. Readiness work can help with planning, but only an audit tells your engineers what to fix on Monday.

Can an automated scanner replace a manual LLM security audit?

Not for prompt injection. OWASP notes that retrieval and fine-tuning do not fully mitigate it, and attackers can reword an injection endlessly, so a filter that catches most attacks still lets some through. A manual audit maps which features combine private data, untrusted input and outbound actions, then fixes the architecture so one of those three is removed.

References

1. OWASP Gen AI Security Project - Top 10 for LLM Applications 2025
2. OWASP - LLM01:2025 Prompt Injection
3. OWASP - LLM07:2025 System Prompt Leakage
4. OWASP - LLM10:2025 Unbounded Consumption
5. Simon Willison - The lethal trifecta for AI agents
6. OpenAI - Evaluation best practices
7. Anthropic - Define success criteria and build evaluations
8. Virtualization Review - MIT report finds most AI business investments fail
9. NIST AI 600-1 - AI RMF Generative AI Profile
10. Openlayer - AI risk assessment report guide