“I am going to sue you.”
That’s what a recently widowed customer tells a service representative after an insurance carrier denies her husband’s death claim.
The denial letter says the policy had lapsed. There’s just one problem: according to the records she has, her husband hadn’t missed a premium payment.
She checks the customer portal. The policy shows as active. She checks their payment records. Payments appear current. There’s no lapse notice that she can find.
What started as a claims issue quickly becomes a much larger problem. Claims gets involved. Billing gets involved. Technology gets involved. Eventually the investigation reaches the AI agent that processed the claim.
What actually happened
The claim had been processed by an AI agent. Its workflow looked reasonable.
The claimant submitted first notice of loss through the portal. That triggered the claims-processing agent.
The agent then:
1. Extracted information from the submitted claim.
2. Read the claims SOP to understand the validation rules.
3. Used the SOP to determine the processing steps.
4. Evaluated the available data and selected a workflow path.
5. Read supporting knowledge articles to determine how individual steps should be executed and which tools should be called.
6. Called the required APIs and completed the claim-processing workflow.
At first glance, this looks fairly well governed. The SOP is controlled, the knowledge articles are curated, the APIs are approved, the MCP tools are contracted, and the source data comes from enterprise systems.
So what went wrong?
The organization had governed artifacts. What it did not have was a clear boundary around what the agent was allowed to interpret at runtime.
Where the uncertainty actually enters
The SOP itself is not probabilistic. Neither is the API contract or the database record.
The uncertainty starts when an LLM reads one of those artifacts and decides what it means in the context of this particular claim.
That interpretation may vary across executions, especially when the language contains insurance terminology, exceptions, dates, precedence, or concepts a domain expert would distinguish almost automatically.
The agent may have to infer:
• what a particular data element actually means,
• which system should be trusted for a given fact,
• how a business rule should be applied,
• whether two similar-looking dates represent the same business concept,
• what an API response means in business terms,
• what to do when an SOP doesn’t explicitly cover a particular condition,
• and which workflow should execute next.
Any one of these interpretations can look reasonable. The concern is the chain. A claims flow with several LLM interpretation steps gives you several places where a plausible interpretation can drift from the intended one.
Here is where this claim failed
The systems themselves were not giving the agent conflicting answers.
The policy administration system showed the policy as active, the billing system showed premiums as current, and the customer portal displayed Active. There was no data conflict.
The husband paid quarterly. His last payment posted on May 1 and covered the policy through August 1. He died on June 10.
The SOP included a statement that a policy lapses if premium is not received within 31 days of the due date.
The agent interpreted that sentence incorrectly. It used the last payment date as the premium due date, added the 31-day grace period to May 1, concluded that the grace period ended on June 1, and determined that the policy had lapsed nine days before the insured died.
Nothing was wrong with the math. The problem was that the wrong date had been used as the anchor.
A payment date is not the same thing as a premium due date. A premium due date is not the same thing as a paid-to date. Anyone who has worked with life insurance policy administration understands that distinction. But unless those concepts are explicitly modeled, an LLM can easily collapse them into one idea because the surrounding language makes them look related.
The agent didn’t find a lapse in a system of record. It effectively created one through interpretation. Then it acted on that interpretation.
For me, this is the part enterprise AI architects need to focus on.
This is more than a hallucination problem
We use ‘hallucination’ as a catch-all for LLM failures. In enterprise systems, I think interpretation errors are at least as important.
The agent retrieves the correct record but misunderstands what the field means. It reads the correct SOP but turns the wording into the wrong executable rule. It calls the correct API but misunderstands what a response code represents in the business process. It selects a technically valid next step, but it isn’t the step the business intended.
The model can be internally consistent and still be wrong for the business.
Which parts of an enterprise process should we allow an LLM to interpret dynamically, and which parts should already have a governed meaning before the LLM ever sees them?
In a regulated process, I would draw that boundary much more tightly than many current agent designs do.
I see three separate responsibilities
Looking at the claim this way, three responsibilities stand out.
The first is meaning. What does a field represent? What does a status code mean? Which source owns a particular business fact? Is PolicyStatus something stored in one system, or is it actually a business concept derived from several facts?
The second is decision logic. What exactly qualifies a policy as in force? How is a grace period calculated? What happens inside the contestability period? When should a claim be approved, denied, or referred?
The third is process execution. What step happens first? What validations are mandatory? Can any step be skipped? Which tool executes which action? What happens when the decision is REFER instead of APPROVE or DENY?
I would not leave all three inside the agent. Meaning belongs in a semantic layer, business logic in a decision layer, and required process sequence in a workflow or action layer.
The agent still matters. It just should not own all of those decisions.
How I would structure it
I would put the agent between the interaction layer and the governed enterprise capabilities that sit underneath it.
The agent is still responsible for the kinds of things LLMs are genuinely useful for:
• understanding natural language,
• interpreting unstructured submissions,
• extracting information from documents,
• handling conversational interaction,
• resolving ambiguity where judgment is appropriate,
• and coordinating work across enterprise services.
I am comfortable letting the agent work with language and ambiguity. I am much less comfortable letting it redefine enterprise semantics, turn prose into production rules, or invent the control flow of a regulated process while the claim is running.
Instead, the agent should consume governed capabilities below it:
• an ontology-based semantic layer,
• a decision service,
• a workflow or action layer,
• governed APIs and MCP tools,
• and an observability and governance layer around the entire execution.
Figure 1. The trustworthy agentic architecture — probabilistic where language lives, deterministic where liability lives.
You can describe this as a neurosymbolic architecture: the LLM provides the neural capability, while the ontology, rules, and process definitions provide the symbolic side.
The terminology sounds newer than the underlying components. Insurance companies have run rules engines, workflow engines, and enterprise data models for decades. The new part is how we connect those capabilities to an agent.
Grounding meaning with an ontology
Consider PolicyStatus.
In many environments, several systems have a status field. The policy administration system may say ACTIVE. The billing system may expose another status. The portal may display Active.
A straightforward design would retrieve those values and let the agent reason over them. I would not do that for this question.
PolicyStatus should be treated as a business concept. Whether the policy is actually in force on a particular date may depend on:
• premium frequency,
• premium due date,
• payment history,
• paid-to date,
• grace-period rules,
• product,
• carrier,
• jurisdiction,
• reinstatement status,
• and the date on which coverage is being evaluated.
So PolicyStatus may not be a field we should trust at face value. It may be a derived business concept, and the ontology should say so explicitly.
Similarly, PaymentDate, PremiumDueDate and PaidToDate should exist as separate concepts with separate meanings.
The semantic layer should also identify authoritative sources. Billing may be authoritative for payment transactions. The policy administration system may be authoritative for policy attributes. The claims system may be authoritative for date of death once validated. But no single system necessarily owns the derived concept “Was this policy in force on June 10?” That question should be answered from governed semantics and governed derivations.
Figure 2. The claims-processing ontology. All three source groundings are consistent — and none of them is the concept. PolicyStatus is derived.
This is why I do not see the ontology as just a knowledge graph next to the LLM. The useful part is semantic control: defining what a concept means, how source fields map to it, and which source is authoritative for each fact.
In this claim, that semantic control is what keeps a payment date from being used as the business concept required for lapse evaluation.
Another benefit is identity resolution. The widow in our example may simultaneously be a Person, a Beneficiary, and a Claimant. Those are roles associated with one individual, not three unrelated records. That becomes especially important in a multi-carrier TPA environment where different carriers, products and administration systems use different names for essentially the same concepts. An ontology gives us a canonical business language above those differences.
Put the business decision in an executable artifact
Once the inputs have governed meaning, I would take the eligibility decision itself out of the prompt and put it in an externalized decision service.
For the death-claim scenario, the decision could be represented explicitly in a decision table:
Table 1. The death-claim eligibility decision — versioned, testable, auditable. Rule 6 is the control: no match means a human decides.
The exact table will vary by carrier. What matters is that the logic now exists as an executable artifact that can be versioned, tested, reviewed, audited, and tied back to the claim that used it.
Compare that with an LLM reading this sentence:
A policy lapses if premium is not received within 31 days of the due date.
That sentence may be perfectly adequate inside a human-readable SOP. It isn’t necessarily an adequate executable specification. If I need the rule to control whether an insurance company pays or denies a death claim, I want the actual logic represented explicitly.
That is the distinction I care about: an SOP explains the rule to people; a decision table executes the rule for the system. Better retrieval does not make those two things equivalent.
Do not let the agent fill the gaps
The default outcome deserves special attention. If no rule applies, what happens next? If the answer is ‘the agent figures it out,’ we have simply moved the original problem to the edge of the rules engine.
For high-consequence decisions, I prefer a simple approach: if the governed decision service can’t make the decision, return REFER. Don’t ask the agent to invent the missing rule. Route the case to a human.
I do not consider that a failure of automation. It is a deliberate stop condition.
A rules engine that handles five known conditions but leaves the sixth condition to the LLM hasn’t really externalized the decision — only part of it.
The workflow needs the same treatment
I would apply the same principle to process execution. The agent does not need to reconstruct the core death-claim workflow from an SOP every time a claim arrives. The required process is already known.
For example:
1. Receive claim.
2. Validate claimant and policy.
3. Resolve the claimant’s relationship to the policy.
4. Retrieve policy facts.
5. Retrieve payment history.
6. Derive the required semantic facts.
7. Invoke the claims eligibility decision.
8. Route based on APPROVE, DENY or REFER.
9. Execute the appropriate downstream action.
10. Record the full audit trail.
There is still room for agentic work inside that process. Intake may use an agent to understand a handwritten or free-text submission. Correspondence may use an LLM to draft a customer-friendly explanation. Investigation steps may allow more freedom to gather evidence.
But the mandatory control flow should still be defined. The workflow decides which steps are required, which tool is bound to each step, and whether the process can move forward. The agent executes inside those boundaries.
Figure 3. The death-claim workflow. The one dashed task is the agent’s neural work; every other step carries a bound tool, and the gateway invokes the decision table by name and version.
This becomes particularly important with tools. I don’t want an agent deciding from a list of twenty enterprise APIs which financial transaction API “seems right” for issuing a benefit payment. If the process reaches the benefit-payment step, the process definition should already know the approved service that performs that action.
For irreversible actions, I would make tool selection a design-time decision wherever possible, not a runtime model preference.
Replay the claim with those boundaries in place
Now take the same facts and run them through the revised design.
The widow submits first notice of loss through the portal. The agent reads the submission and extracts the policy number, claimant information, date of death, and supporting document information. That’s an appropriate use of the model.
The workflow then takes control of the sequence. The semantic layer resolves the policy, resolves the claimant as the same Person who also holds the Beneficiary role, and retrieves the required policy and payment facts. It calculates the paid-to date correctly as August 1. The date of death is June 10. The premium is therefore current as of the date of death.
The workflow calls the governed death-claim eligibility decision.
Rule 1 matches. Decision: APPROVE.
The decision service returns the rule identifier and supporting facts, and the workflow routes the claim to benefit payment through the approved API.
The audit trail now contains the extracted claim facts, the source systems used, the semantic derivations, the decision-table version, the rule that fired, the workflow version, the APIs called, the responses received, and any agent activity that occurred along the way.
The source data is exactly the same. What changed is the responsibility assigned to each layer.
We did not need a bigger model or a cleverer prompt. We stopped asking the model to interpret things that the architecture should already know.
What this architecture does not solve
There’s an important caveat: the agent can still make mistakes.
Suppose the date of death is handwritten and the model extracts June 10 as June 19. Everything downstream may execute perfectly and still produce the wrong result because the input was wrong. The same thing can happen with document classification, identity resolution, intent recognition, extraction, or summarization.
This does not make the whole system deterministic. The neural parts are still probabilistic. The point is to keep that uncertainty in places where it is useful rather than letting it leak into business semantics, financial calculations, regulatory decisions, and irreversible transactions.
That’s also why governance and observability can’t be an afterthought. We still need extraction confidence thresholds, evaluations, anomaly detection, human review, traceability, model and prompt versioning, complete tool-call traces, decision and workflow provenance, and production monitoring.
If the model extracts the wrong date, we need to be able to discover that. If confidence is too low, the workflow should stop. If source data conflicts — or no governed decision applies — the case goes to a human.
I do not need the AI to be perfect. I need the system to recognize uncertainty before that uncertainty turns into an irreversible business action.
Why this matters more as agents get real authority
This becomes more important as enterprises move from copilots to agents. With a copilot, a person usually sits between the model and the transaction. With an agent, that buffer can disappear: the model can read, decide, call a tool, and change a system of record.
At that point, prompt engineering is not enough. Neither are RAG, a curated knowledge base, or an approved MCP server. They help, but they do not answer the question I would ask first:
What is the agent actually allowed to infer?
For low-risk activities, the answer can be broad. For an insurance claim denial, payment, beneficiary change, underwriting decision or financial transaction, I would make the answer much narrower.
The architecture should keep these responsibilities distinct: language understanding in the model; business meaning in governed semantics; decisions in executable rules; process sequence in workflow; and irreversible actions behind approved tool bindings.
There is still plenty for the agent to do. The advantage is that we stop making it perform work that deterministic enterprise systems already handle better.
Insurance technology already has most of the building blocks: data models, rules engines, BPM and workflow platforms, policy administration systems, APIs, and audit controls. Agentic AI makes those capabilities more relevant, not less.
I do not see the end state as AI replacing deterministic enterprise systems. I see it as AI being placed around them with much clearer boundaries.
In this claim, the LLM’s reasoning was not random. It was logical once it had attached the wrong meaning to the payment date. The dangerous part was allowing that interpretation to drive a denial.
That is an architecture decision. As agents get more authority, we need to make that decision deliberately before we let them act.