A large language model can write a paragraph that sounds like a senior compliance officer drafted it, and still miss the one carveout that makes the answer wrong. That is the uncomfortable fact underneath every AI pilot inside a regulated business. Fluency is a poor proxy for accuracy. The model was optimized to sound right, and in tax, audit, healthcare, insurance, and legal work the difference between sounding right and being right is the whole game.
The tempting fix is better prompts, longer prompts, sterner prompts that end with "Do not hallucinate."
None of that works, and the reason is architectural. The interesting question isn't how to scold the model into behaving. It's what to put next to the model so that the parts of the work that must be deterministic actually are.
One Missed Carveout Is the Whole Story
Take a narrow, familiar scenario. A compliance analyst asks an AI assistant whether a specific transaction triggers a specific reporting duty. The model reads the statute, reads the transaction memo, and returns a clean answer with citations. It sounds correct, and it is correct, except for one exemption buried three sections deep that would have flipped the conclusion.
The analyst files. The regulator later asks why. That single missed carveout is the shape of the whole risk: a plausible answer that skipped a decisive rule, delivered in the same confident tone as every correct answer the same tool has produced all week.
This is the argument made in the Law.co podcast episode on why Legal AI Needs Rules, Not Just Vibes: The Case for Hybrid Agents, which walks through how a hybrid architecture separates the proposing job from the deciding job. NIST's own generative AI guidance names this failure mode plainly, releasing a companion profile to its AI Risk Management Framework that identifies confabulation as a distinct risk and proposes governance actions for high-stakes deployments.
Better Prompts Don't Fix an Architectural Problem
Come back to that missed carveout. Suppose you rewrote the prompt to demand the model check every exemption before answering. Suppose you added a checklist and few-shot examples. You will get better output on average, and you still will not get deterministic output, because the underlying system is a next-token predictor that has been steered.
This is why the retrieval-augmented, domain-tuned tools sold specifically to lawyers still miss. A Stanford study of purpose-built legal research assistants found meaningful hallucination rates even after the vendors grounded their systems in curated case law. If grounding a model in the actual authorities isn't enough on its own, prompt gymnastics on a general-purpose chatbot are unlikely to close the distance. Asking a probabilistic system to enforce a deterministic rule is the failure.
Put a Rules Engine Next to the Model
The design pattern that holds up in regulated work is hybrid: a language model for the messy, unstructured parts, and a symbolic rules engine for the parts that must never drift. Engineers who have worked on decisioning systems for years describe rules engines as the tool for logic that needs full precision, and machine learning as the tool for prediction across inputs no human could enumerate. Route each subtask to the component built for it, rather than forcing one tool to do both jobs.
In this setup, the missed-carveout scenario becomes a different problem. The exemption is a rule in the engine. If the extracted facts match its conditions, it fires. Fluent prose cannot smooth it over, because the prose is no longer what makes the decision.
Auditability Is a Feature, Not a Deliverable
Regulators do not want a confident answer. They want a traceable one. A rules engine gives you a trace by construction: this fact plus this fact triggered this rule, which produced this output, on this date, against this version of the policy. A pure LLM cannot give you that, and asking it to explain itself after the fact tends to produce a story about the answer rather than the reasoning that produced it.
What Buyers in Regulated Work Should Ask
The carveout thought experiment is useful when you are evaluating a vendor. Run it live on the demo and watch where the decision gets made, and by what component.
None of this is anti-AI. Language models are useful for reading unstructured material and turning it into structured facts, which is usually a job compliance teams have too much of and too few hands for. The mistake is asking the same model to also be the rulebook.
Hand the reading to the model, hand the ruling to the rules, and keep the signature with the person. That is the shape of AI that a regulator, an auditor, or a client can trust.
