AI Compliance Gap Analyzer

The user and stakes

Small teams need orientation—not an automated verdict.

An AI startup can face product, privacy, professional, sector, and jurisdictional obligations before it can hire a dedicated compliance function. The immediate need is practical: identify which questions may matter, see the assumptions behind them, and know what to verify with a qualified expert.

The risk is asymmetric. A missing result is inconvenient; a polished but misattributed obligation can redirect product work, create false confidence, or turn weak research into a high-stakes decision. The product therefore has to optimize for defensibility before completeness.

The release decision

The audit failed—and that became the most valuable result.

An early version searched the web and asked a strong reasoning model to assemble a report. The output became more complete and more convincing, but the research feeding it was still inconsistent.

I put two reports generated from the same input through a seven-claim audit. Four claims were wrong or misattributed: a superseded deadline reported as passed, a professional-body quote traced to a training company’s blog, a wrong state-law effective date, and a regulation applied outside its scope. A separate six-run check exposed three incompatible answers to the same vendor-retention question. The problem was not one bad citation; the system could tell different stories with the same confidence.

I held v0.6 back—never tagged, deployed, or delivered. Then I asked the product question that changed the architecture: how can search know what counts as authoritative before the system has scoped the regulatory territory? That shifted the workflow from search-first generation to scope-first research.

In a trust product, a persuasive answer without defensible evidence is a product failure—not a formatting problem.

What I actually did

My role and decision authority

As founder and sole product owner, I use an AI coding agent as an implementation partner. I set product direction, wrote the v0.7 build specification, approved design choices before code, reviewed the build, ran the tests, challenged the output, and owned the release decisions.

Trust, translated

The decisions that changed the product

From decision to system

A working four-stage research pipeline

The current implementation uses Python, Streamlit, Claude, Tavily, Supabase, and Langfuse. It runs live web research and saves traceable report artifacts; it does not use a vector database or production RAG index.

1. 01

   Scope

   Translate the use case, technology, industry, and location into structured assumptions, candidate regimes, and official domains across five dimensions.

2. 02

   Research

   Run live searches for each candidate regime, trying official domains first and retaining every query, URL, source tier, and regime label.

3. 03

   Analyze

   Generate a structured seven-section analysis from the scoping context and retrieved evidence rather than relying on model memory alone.

4. 04

   Report and review

   Surface the analysis, assumptions, coverage, sources, and executed queries separately so a human can verify any decision they act on.

What the evidence says

The redesign improved honesty—not certainty.

The new sequence improved coverage and consistency in a small repeated test. It also revealed defects that still block a stronger release claim.

4 / 7

claims in the two-run v0.6 audit wrong or misattributed

3 / 3

v0.7 reruns surfaced the governing body

3 / 3

v0.7 reruns disclosed assumptions

1 / 3

v0.7 reruns still repeated a stale deadline

• The governing professional body was absent from the findings in two of six v0.6 runs; in all three v0.7 reruns it was present and cited to its own domain.
• Across six v0.6 runs, the tool asserted three mutually incompatible vendor-retention answers. All three v0.7 reruns instead said the terms were not found and recommended obtaining them directly.
• One lower-quality source still leaked into the official-source tier, and priority ranking varied across identical inputs.
• Every logged v0.7 run so far (seven across three scenarios) took 2.5–3.0 minutes end to end; v0.6 took 1.8–2.2 minutes on the comparison scenario.
• Separately, I personally checked 33 source links in an AI-assisted deliverable: 29 verified, one dead government link found and replaced, and three left unresolved. This was a human verification pass—not automated-pipeline validation.

Outcome and impact

I stopped one version—and shipped the next with its limits visible.

v0.6 was misleading: its errors sounded more certain than its evidence. I held it entirely. v0.7 was still incomplete, but the product made scope, sources, assumptions, and missing evidence visible. I shipped it as a research prototype with tier leakage, ranking drift, and the stale-deadline recurrence documented in the release notes rather than hidden.

This work has not yet produced verified customer, legal, or compliance outcomes. Its current impact is product judgment made operational: one version stopped, an architecture changed, and the next trust gates made measurable.

Decision impact

v0.6 withheld: untagged, undeployed, undelivered

Product impact

scope-first v0.7 shipped with limits disclosed

Trust mechanism

missing evidence stays visible

Next gate

deterministic source and claim validation

The next honest release

What I would build next

1. Verify every official-source label against the returned URL host.
2. Bind every material claim to a specific source and add temporal checks.
3. Move prioritization into a structured, testable rubric.
4. Create a repeated multi-scenario regression set and explicit release criteria.
5. Complete consent, authentication, least-privilege access, retention, and deletion controls.
6. Re-audit with qualified compliance and legal reviewers before making outcome claims.