In many real-world implementations, the front-end experience is delivered through a third-party SaaS platform, while the enterprise’s own systems remain on the backend. That means the customer interaction layer, orchestration, model interactions, and some runtime components may sit outside the enterprise boundary, while policy administration, claims, billing, customer platforms, and document systems remain inside. In regulated industries like financial services and insurance, that creates an immediate and valid concern: how is PII handled across every step of that chain?

This is not just a technical concern. It is a compliance concern, a trust concern, and increasingly a business risk concern. Organizations in these industries cannot afford vague answers when it comes to personally identifiable information. They need to be able to explain, with confidence and evidence, that customer data is safe and not being exposed inappropriately while LLMs are being used to deliver services. That means AI and agentic application architectures cannot be built on assumptions or vendor marketing language alone; they have to be grounded in accepted patterns, strong controls, and reviewable design decisions.

This article is my attempt to make sense of that problem in a way that is practical.

I spent time reviewing guidance from NIST AI-related resources, OWASP, CSA, and cloud security documentation from major providers to understand what the real risk areas are, which controls actually matter, and how they should be implemented in practice. The information was useful, but it came in different forms and from different angles—privacy, AI risk, application security, cloud controls, and operational governance. At some point, I realized I needed a mental model to keep it all straight. I started building one, and I kept refining it as I learned more.

What I’m sharing here is the result so far: a practical security spectrum and matrix that helps structure the conversation around LLMs, PII, and third-party AI SaaS in regulated enterprise environments.

Here’s the LLM + PII Security Spectrum and Matrix

Framework by author, Visual: Claude

Why “public vs private LLM” is too simplistic

A lot of discussions still frame this as a simple comparison. If a solution uses a public model, it is treated as risky. If it uses a private model, it is treated as safe.

That framing is understandable, but it is too simplistic to be useful in real architecture or security reviews.

The actual risk is not determined only by which model is used or where it is hosted. It is also shaped by where the application runtime lives, how data moves through the system, what gets logged or retained, how memory and state are handled, which tools the agent can invoke, what support personnel can access, how long data persists, and where plaintext PII is visible in the end-to-end flow. In other words, the security posture of an LLM-enabled customer service system is a function of architecture, controls, and operations—not just model location.

That is why I found it more useful to think in a spectrum rather than a binary, and more useful to think in more than one dimension.

The first dimension: trust boundary and deployment custody

The first part of the model is a deployment and trust-boundary spectrum. This is closest to how most teams initially think about the problem, and it is still important. At one end, the enterprise is using a vendor SaaS platform that calls external shared LLM APIs, which means a wider trust boundary and less direct control over the runtime path. As you move along the spectrum, you may see dedicated or single-tenant model hosting, then vendor SaaS using a private or BYO model deployed in the customer environment, then the vendor application itself deployed in the customer environment with a private model, and eventually a mostly or fully customer-controlled stack where the application, model, and data plane sit inside the enterprise boundary.

This spectrum is useful because it clarifies who controls what. It helps answer practical questions about runtime ownership, networking, storage, observability, secrets, support access, and operational boundaries. In general, moving toward more enterprise control can reduce third-party custody risk. But it does not automatically make a system secure—it only narrows the trust boundary. The outcome still depends on design and execution.

The second dimension: how much PII is actually exposed in the pipeline

The second dimension is the one that changed how I think about this problem. Even if the deployment model looks strong on paper, the architecture can still be weak if raw PII is visible throughout the pipeline. A system can be “private” from a hosting standpoint and still be unsafe in practice if plaintext PII is flowing through prompts, traces, logs, session memory, caches, vector stores, analytics events, or tool payloads.

That is why the matrix needs a separate axis for PII exposure mode. At one end, plaintext PII exists throughout the pipeline. A stronger posture may involve selective masking or redaction, then early tokenization or pseudonymization, then late detokenization only in tightly controlled backend zones where enterprise systems actually need the real values. In more advanced architectures, this can be further strengthened with confidential computing or secure enclaves to protect data in use during sensitive processing.

This lens matters a lot in financial services and insurance because it opens up better options than simply “send raw PII through everything” or “move everything on-prem immediately.” It gives teams a way to reduce exposure materially while still delivering customer-facing capabilities at speed.

The pattern that stood out most: early tokenization and late detokenization

As I refined the model, one pattern kept standing out as especially practical for regulated environments: tokenize or pseudonymize sensitive data as early as possible, move tokens through the AI pipeline wherever feasible, and only detokenize inside secure backend zones when enterprise systems actually need the real values.

Framework by author, Visual: Claude

In practice, that means the customer’s input enters the system, sensitive fields or entities are identified, and those values are tokenized near the ingress point. The orchestration layer, agent runtime, and model interactions operate on tokens or protected representations instead of raw identifiers whenever possible. When a backend policy or claims system needs the real value to execute a business function, a tightly controlled service inside a secure zone performs detokenization, executes the backend interaction, and then the response is protected again before passing back through intermediate layers. At the edge, only the final presentation layer detokenizes what must be shown to the user.

What I like about this pattern is that it changes the blast radius. It does not eliminate risk, but it can materially improve the architecture’s ability to limit where plaintext PII exists. It also gives teams a much clearer story for architecture governance, security review, and compliance discussions because they can explicitly show where sensitive data appears, where it is transformed, and where access is tightly controlled.

At the same time, it is not a magic solution. Tokenization introduces new crown jewels: the token vault and key management become critical infrastructure, and if those are weak, the pattern weakens quickly. Unstructured data detection also becomes a major factor because missed entities in free text, transcripts, or attachments can still leak real PII into logs or prompts. And pseudonymization is reversible by design, so it should not be treated as the same thing as anonymization. The point is risk reduction and architectural control, not a shortcut around governance.

What usually gets missed: the non-negotiable operational controls

One of the biggest lessons from this research and modeling exercise is that teams often spend a lot of time debating public versus private models while underestimating the controls that actually fail in production. In many systems, the most realistic leakage paths are not the model endpoint itself. They are logging pipelines, traces, debug tooling, session memory, vector stores, replay tools, analytics events, and overly broad tool access inside agent workflows.

That is why the spectrum/matrix also includes a cross-cutting layer of non-negotiable controls. These are not hardening items to “add later” once the pilot works; they are the controls that determine whether a promising architecture stays safe under real traffic and real operational pressure.

Framework by author, Visual: Claude

PII classification and minimization starts with agreeing on what is actually sensitive in your business context, including direct identifiers, quasi-identifiers, and domain-specific fields. Once that is defined, the architecture should be designed to collect, process, and retain only the minimum data needed for the use case, instead of pushing full records through the pipeline by default.

Retention and training posture means understanding exactly what is retained by each component in the chain and for how long, from SaaS layers to model providers to logs and caches. It also means confirming training/default data usage settings explicitly, so teams are not relying on assumptions that may not match the product tier, configuration, or contract.

Prompt and output filtering is necessary because both the inbound and outbound paths can become leakage points. Inputs should be normalized and screened to reduce prompt injection and unnecessary sensitive content, while outputs should be validated so the system does not reveal sensitive data, invented policy details, or unsafe responses.

Memory and session isolation matters because conversation context is useful but dangerous if not bounded correctly. Session state, summaries, and memory artifacts must be isolated per user/session/tenant, and teams should be explicit about what content is even eligible to be stored in memory in regulated workflows.

Logs and traces redaction is one of the most important controls in practice because observability systems often become accidental data lakes for sensitive content. Redaction and masking need to happen before data is written to logs or traces, not later, and this includes prompts, tool payloads, stack traces, and error bodies.

Vector store and cache governance is essential if embeddings, retrieval indexes, or response caches are part of the design. Teams need rules for what can enter those stores, what must be tokenized or excluded, how long data stays there, how deletion works, and how tenant isolation is enforced.

Least-privilege tool access is critical in agentic systems because tools are often the path to real actions and real data, not just read-only lookups. Each tool should have narrow permissions, strong authorization checks, validated inputs and outputs, and auditable execution identities rather than broad “agent can do everything” access.

Key vault and token vault separation becomes especially important when tokenization is part of the architecture. The token vault and key management system are high-value targets, and separating these control planes from ordinary application storage and secrets reduces blast radius and supports stricter monitoring and access controls.

Private networking and egress control matters because even a good model/runtime choice can be undermined by uncontrolled traffic paths and outbound connectivity. Teams should define approved endpoints, restrict egress where possible, and use private connectivity patterns so sensitive data is not moving through unnecessary or weakly governed routes.

Human access and support controls are often overlooked because teams focus on runtime behavior and forget operational access paths. In reality, a lot of exposure happens through support, debugging, and admin consoles, so organizations need strong controls around who can view interactions, logs, and traces, with approvals, just-in-time access, and auditing for privileged users.

Auditability and traceability are non-negotiable in regulated environments because saying controls exist is not the same as proving they were followed. Teams need evidence showing what happened, when, and under which identity when data was accessed or actions were taken, especially in agentic flows where multiple components may act in sequence.

Incident response readiness for AI/agentic pipelines is the final piece that often gets pushed too late. Teams need a playbook that explicitly covers AI-specific failure modes—such as prompt injection causing tool misuse, sensitive data overexposure in logs, or unintended persistence in memory/retrieval layers—so they can detect, contain, and communicate quickly when something goes wrong.

This is the part of the conversation where architecture becomes operational reality. A clean diagram is not enough. Teams need to define exactly what data is allowed in logs, what memory can persist, what tools agents are allowed to invoke, what gets retained and for how long, and who can access what during support or incident handling. If those controls are not explicitly designed and enforced, the deployment model alone will not save the system.

Why I created the matrix and what I hope it helps with

I did not build this spectrum and matrix to create a perfect scorecard or a vendor ranking. I built it to create a practical decision framework that helps people have better conversations.

In real enterprise settings, architecture decisions involve security teams, compliance teams, product leaders, platform engineers, enterprise architects, and business sponsors. Each group focuses on a different part of the problem. The matrix helps bring those pieces into one view.

It gives security teams a way to discuss exposure and controls without collapsing the topic into simple labels. It gives architecture teams a way to compare deployment options and data-flow patterns more honestly. It gives business leaders a way to ask better questions and evaluate trade-offs. And it gives implementation teams a clearer signal that “private model” is not the finish line if the rest of the pipeline is still leaking sensitive data through operational channels.

I also built it because I needed one myself. There is a lot of information in this space, and it is easy to lose track of what belongs where. Building a mental model helped me connect AI risk guidance, privacy controls, application security practices, and enterprise architecture patterns into something I can actually use in design and review conversations.

How to use this in practice

The best use of this matrix is not as a final answer, but as the starting structure for a deeper review. When evaluating an LLM or agentic customer service architecture in a regulated environment, I now find it useful to map the proposed solution across both dimensions at the same time: where it sits on the custody and trust-boundary spectrum, and what the data exposure mode looks like through the actual pipeline. Once that is clear, the next step is to examine the non-negotiable controls and look for gaps.

Here’s the decision guide:

Framework by author, Visual: Claude

This approach moves the conversation away from broad claims and toward evidence. Instead of asking only whether a vendor uses a public or private model, teams can ask where plaintext PII exists, who can see it, what gets logged, how memory is handled, how tool access is constrained, what retention defaults are in place, and what proof exists that these controls are actually working. Those are the questions that hold up better under security and compliance scrutiny.

OWASP LLM Top 10 Mapping to the PII security framework

Here is how this security framework maps to OWASP LLM Top 10.

Framework by author, Visual: Claude

Final thoughts

I’m sharing this because I think many of us in financial services and insurance are dealing with the same challenge: how to deliver meaningful AI-enabled customer experiences while protecting sensitive data in a way that is credible, defensible, and practical. The technology is moving fast, but the standards for trust in regulated industries have not changed. If anything, they matter more now.

This spectrum and matrix is my attempt to make that problem easier to reason about. It is not meant to replace formal threat modeling, legal review, or compliance processes. It is meant to help teams think more clearly, design more intentionally, and avoid the common trap of oversimplifying a very real and complex security problem.

I’m still refining the model, and I expect it will evolve as patterns mature and more implementation evidence becomes available. But this version has already helped me organize the space much better, and I hope it is useful to others working through similar architectures.

If you are working on LLM or agentic solutions in insurance or financial services, especially where third-party SaaS and enterprise backend systems are both involved, I’d be interested in how your team is approaching the trade-offs. The most useful conversations I’ve had so far have come from comparing actual architecture decisions, not just theoretical positions.

References / Further Reading

• NIST AI RMF (AI Risk Management Framework): https://www.nist.gov/itl/ai-risk-management-framework

• NIST AI 600-1 (Generative AI Profile): https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

• NIST SP 800-122 (Guide to Protecting the Confidentiality of PII): https://nvlpubs.nist.gov/nistpubs/legacy/sp/nistspecialpublication800-122.pdf

• OWASP Top 10 for LLM Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/

• CSA AI Controls Matrix (AICM): https://cloudsecurityalliance.org/artifacts/ai-controls-matrix

• Google Cloud Sensitive Data Protection – Pseudonymization: https://cloud.google.com/sensitive-data-protection/docs/pseudonymization

• Microsoft Azure Confidential AI / Confidential Computing: https://learn.microsoft.com/en-us/azure/confidential-computing/confidential-ai

• OpenAI API – Your Data: https://platform.openai.com/docs/guides/your-data

The heartburn CTOs and CAIOs are facing today:

After a significant amount of organizational effort, you have finally rolled out an Agentic AI solution for your contact center, enhancing the employee experience and, in turn, improving customer experience. You expected measurable improvements in CSAT, handle time, deflection, and agent productivity.

However, the solution didn’t yield any real benefits; instead, it created more work and dissatisfaction among contact center users, and in turn, end customers. It hallucinated, provided incorrect, incomplete, conflicting, unnecessarily long, and at times flat-out wrong responses. You also incur significant token costs without delivering real value.

So, what happened? Why did the solution that worked perfectly well in PoC and dev/test fail miserably in the real world?

Here’s why:

The Agentic application skillfully utilized LLMs to extract data from organizations’ various data stores, leveraging several internal and external tools to execute multi-step workflows. The developers of the application wrote clever and extremely descriptive prompts. As a result, they inadvertently continued to fill LLM’s large context window. What they didn’t realize was that more is not always better.

Irrelevant and extraneous context dilutes the model’s attention, increases token cost and noise, and biases its responses to recently added context. This results in inaccurate responses and degraded workflows. LLMs suffer from attention decay, meaning older—but still critical—information becomes overshadowed by recently added noise. In most naïve deployments, 60–80% of tokens consumed are irrelevant to the task at hand.

What exactly does context contain?

1. Instructions - Prompts, few-shot examples, tool descriptions, etc.

2. Knowledge - Facts, memories, conversation history, user preferences, and retrieved information from documents or databases, and real-time data.

3. Tools – Available tools, their definitions, feedback from tools, API responses, etc.

Managing all this requires your system to have a mechanism to maintain both short-term memory and long-term state, and to optimize it by removing/pruning/compressing Context when appropriate.

This practice of managing context—maintaining short-term memory and long-term state, pruning when appropriate, and carefully selecting what information to provide to the LLM at the right time—is known as context engineering.

Here’s what Andrej Karpathy says, “Context engineering is the delicate art and science of filling the context window with just the right information for the next step.”

Let’s understand what happens if the context is not managed correctly.

If context is allowed to grow uncontrollably, it can quickly escalate the system’s costs, increase latency, degrade overall performance, resulting in poor user experience and potentially diminishing business benefits. When context is not curated or pruned, four failure modes emerge:

Context Poisoning – the system begins to hallucinate when erroneous information enters the system’s context and its goals. The system generates incorrect information, leading Agents to create erroneous workflows, behave erratically, and fail to meet their goals, or worse, chase those that can never be met.

Context Distraction – As workflows execute, agents accumulate tool results, conversation history, and operational state—all piling into context. Instead of helping, this growing context becomes noise. The model fixates on what it just did rather than what it should do next, leading agents to repeat past actions instead of adapting.

Context Confusion – Superfluous content dilutes the model’s attention. A recent study illustrates this perfectly: Llama 3.1 8b with 46 tools loaded in context performed worse than the same model with only 19 tools. More isn’t better—it’s just noisier.

Context Clash – New information contradicts what’s already in context, creating internal conflicts the model can’t resolve. This amplifies all the other failure modes simultaneously.

Fortunately, these risks and failure modes are preventable with well-designed strategies to manage context.

What strategies can you implement for better context engineering?

Here are five proven strategies for managing context effectively:

1. RAG – Implement RAG architecture to retrieve only relevant information and tool definitions, rather than loading everything into the large context windows of the LLM. The vector database should store references to data stores and functionality, such as web links, file locations, API endpoints, database connection strings, and load them dynamically only when needed. Agents can decide what to store in the context and what to persist externally.

2. Context Quarantine – Isolate context in their own threads, with each using a separate LLM and its own context window. One way to implement this is to break down the tasks into individual jobs, each executed by a different agent in parallel, with each agent focusing on a specific aspect of the workflow tasks. This minimizes context confusion, context distraction, and context clash and improves speed and accuracy.

3. Context Pruning – Remove unneeded and irrelevant information from the context. This reduces model latency, lowers token costs, and enhances overall accuracy.

4. Context Summarization – Context is summarized to its most essential parts. Knowing what information to summarize and how is a critical skill for agent builders. Many agent builders use a separate agent with a different LLM specialized in summarization to summarize the context and pass the summarized context back to the main agent.

5. Context Offloading – In this technique, models write their less-relevant context and progress to an external store, such as a scratchpad, for later reference. Context can also be offloaded to a vector database and retrieved using RAG, along with other relevant information at an appropriate time.

While these strategies provide the tactical toolkit, successful implementation requires continuous measurement and validation.

The strategies above work—but only if your teams actually implement them. How do you ensure context engineering doesn’t become another best practice that gets ignored in production?

The answer is observability. Without visibility into how your agents manage context, you’re deploying blind. Observability helps teams understand:

1. What the agent did

2. How it did it

3. Why it did it

4. When and where failures occur

Your teams should capture the information below.

1. Metrics: Token usage, latency, success/failure rate, resource utilization, etc.

2. Logs: User-agent interactions, tool calls, model prompts/responses, decision logs

3. Traces: Step-by-step agent workflows, multi-agent interactions, fallback loops

4. Evaluations: Task adherence, intent resolution, tool call accuracy, hallucination detection

[More about AI agent observability in a future article]

Every irrelevant token burned is latency added, money wasted, and attention diverted from what matters. Enterprises that actively engineer context see measurable reductions in hallucinations, token cost, latency, and workflow errors — and unlock the actual value of agentic AI in production.

References:

https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

https://blog.langchain.com/context-engineering-for-agents/

https://www.dbreunig.com/2025/06/22/how-contexts-fail-and-how-to-fix-them.html

https://www.dbreunig.com/2025/06/26/how-to-fix-your-context.html

https://www.anthropic.com/engineering/multi-agent-research-system

[2501.16214] Provence: efficient and robust context pruning for retrieval-augmented generation

All the business leaders, sales heads, and technology executives I’ve met recently are exploring ways to leverage the cutting-edge tech in the AI space to transform customer experience, increase operational efficiency, increase revenue, and reduce risk.

The IT team of every line of business is trying to identify use cases that can truly benefit from AI Agents and Agentic AI. But how would their agents access the functionality and data of their core backend systems? For example, in the insurance industry, core systems for policy admin systems (PAS), claims, billing, and underwriting.

Today’s applications, soon to be legacy😉, achieve that through APIs. The future applications will be Agentic, i.e., powered by agents that can figure out what to do in which scenario, when to do it, and in which order without being concerned about the how.

Today’s core systems publish an API catalog that LoB IT teams use to identify the APIs that’ll meet their applications’ needs.

I have seen several PoCs of agentic applications where the teams have directly coded API calls in the agent, which makes them tightly coupled with the APIs and fixed. Agentic applications are supposed to be autonomous, adaptive (able to pick up new capabilities as they become available without making changes in the code), and capable of dynamically identifying which agents to use for which goals.

Now, how do we enable that? This is where an agent registry comes into play. Agents are registered here, and the LLM in the agentic application autonomously determines the user’s goal and queries this registry to find agent(s) that can achieve that goal. The application then invokes the identified agent.

It is now the responsibility of enterprises to create agents that make their business capabilities available, like policy issuance and claim adjudication, to the external world.

Eventually, we’ll have an agent marketplace where enterprises can leverage each other’s capabilities to develop innovative solutions that help businesses achieve their goals.

Recent technological developments, like MCP and A2A that standardize agent integration and collaboration, and agent frameworks like LangGraph, CrewAI, and AutoGen have established a strong foundation to make this a reality.

As data democratization made data accessible across the enterprise, agent democratization will make enterprise capabilities discoverable and actionable through agents.

Everyone wants to "do Agentic AI"—sometimes it makes perfect sense, other times it’s forced or overengineered. Business leaders need clear, practical answers when technical teams propose agentic solutions.

Here are common questions business stakeholders ask, and the refined, focused answers that help make informed decisions.

1. What is agentic AI, really? And how is it different from the AI we already use?

Agentic AI moves from reactive, task-based behavior to proactive, goal-based intelligence. Instead of waiting for instructions, an agent receives a goal (e.g., "process this application") and reasons through the steps using APIs, models, and logic—just like a digital analyst.

Example – Policy Issuance: Traditional flow:

Extract → Validate → Score → Quote → Generate Document

Agentic AI:

Goal: "Process application A1234" → Agent figures out what to do and executes the right steps.

2. Is this just a fancy chatbot?

No. Agentic AI is not a chatbot—it’s a digital process worker. It doesn’t just respond to input; it completes workflows by calling APIs, validating data, scoring risk, and making decisions.

3. Where does this fit into our existing systems?

Agentic AI sits on top of your APIs and services. It orchestrates—not replaces—core systems.

In policy issuance, you keep:

• Doc extraction service

• Identity validation API

• Pricing logic

• Document generation

• CRM

The agent uses them autonomously to achieve the business goal.

4. Why now? What’s changed to make this viable?

Two key shifts:

1. LLMs (like GPT, Claude, Mistral) can now reason and plan.

2. Most enterprises have APIs ready for intelligent orchestration.

Agentic AI connects these dots to automate high-level outcomes.

5. What’s the ROI or business value of agentic AI?

• Efficiency: Reduce manual effort.

• Scalability: Do more without hiring more.

• Compliance: Full audit trails.

• CX: Faster, personalized, error-free.

• Adaptability: Handles edge cases without redesign.

6. What risks should we be aware of?

Key risks:

• Hallucination (misinterpretation)

• Tool misuse (bad input/output handling)

• Data exposure (logging/API leaks)

• Overreach (too much autonomy)

Mitigation Strategies:

• Scoped goals

• Prompt boundaries

• Tool-level auth + fallback

• Memory + logging

7. How is this different from RPA or workflows?

Example – Policy Issuance:

• RPA: Clicks through systems to extract, validate, and quote.

• Workflow: Follows a rigid flowchart.

• Agent: Receives "Issue policy" and plans steps dynamically.

8. Will this replace people?

No—it augments people. Agents handle repeatable, structured, API-ready tasks, freeing humans to focus on empathy, judgment, and exceptions.

In policy issuance:

• Agent validates identity, scores risk, and generates documents.

• Human handles complex scenarios or edge cases.

9. How do we control or audit what the agent does?

Every action is logged:

• Thought

• Action

• Input/Output

• Reasoning

In policy issuance: every step from SSN validation to quote is documented.

10. How do we get started without a big investment or risk?

Start with one focused use case:

• 1 agent

• 3–5 APIs

• 1 goal (e.g., "Process this application")

Run in sandbox, observe results, evaluate against benchmarks.

11. What’s the next step to try this out?

Start with policy issuance:

1. Wrap your existing APIs as tools

2. Define a goal (e.g., "Issue policy")

3. Deploy an agent to plan + execute

4. Log every action

5. Evaluate → Expand

12. How do I make my agent extensible to handle future requirements, for example, how can I add a new step to the agent’s workflow?

You don’t modify the whole agent—just:

1. Implement the new tool

2. Add it to the tool registry

3. Optionally update system prompt or planning example

The agent auto-discovers and integrates the new tool based on reasoning.

Bonus questions for the more curious 😊

13. How does the agent know what to do?

The agent is guided by two types of prompts:

1. System Prompt: defines what the agent is, its tools, constraints, and behavior

2. User Prompt: provides the specific goal or instruction, like “Process application A1234”

1. System Prompt (Agent Identity + Boundaries)

You are an autonomous insurance workflow agent.

Your job is to process new insurance policy applications using available tools. You do not guess or fabricate information. You work step-by-step, reasoning through the best sequence of actions to complete the task.

You have access to tools such as:

- FormRecognizer: extract applicant data from PDF

- VerifySSN: validate Social Security Number

- CheckCreditScore: retrieve applicant's credit score

- ScoreRisk: calculate risk score based on profile

- PricingAPI: return policy quote

- DocGen: generate policy document

- Signer: send document for e-signature

- PolicyStore: archive completed policy

Your responsibilities:

- Reason about which tool to use and when

- Use only tools you are authorized for

- Escalate or log any missing or ambiguous input

- Complete the process goal accurately, step-by-step

- Log all thoughts, actions, and observations

Your tone is formal, your actions are auditable, and your decisions must align with compliance

2. User Prompt (Task/Goal Instruction)

Goal: Process application ID A1234.

You are provided with a PDF of the applicant’s submission and access to tools for data extraction, identity verification, risk scoring, quoting, and document generation.

Use your tools step-by-step to:

- Extract applicant data

- Validate the SSN

- Check credit score if income < $50K

- Score risk

- Generate a personalized quote

- Create the final policy document

- Send for digital signature

- Archive the signed policy

Escalate to a human underwriter if any inconsistency or error occurs during verification or scoring.

14. How does the agent reason through the process?

This is how the agent thinks, chooses tools, and progresses through the workflow. (Generated using ChatGPT)

User Goal: Process insurance application A1234

File URL: https://storage.insuranceco.com/applications/A1234.pdf

---

Thought: I need to begin by extracting structured applicant data from the PDF.

Action: FormRecognizer

Input: { "file_url": "https://storage.insuranceco.com/applications/A1234.pdf" }

Observation:

{

"name": "Jane Smith",

"ssn": "123-45-6789",

"dob": "1989-07-23",

"income": 48000,

"health_index": 0.82

}

---

Thought: I now have the applicant’s SSN. I should verify their identity.

Action: VerifySSN

Input: { "ssn": "123-45-6789" }

Observation:

{ "valid": true, "name_match": true, "dob_match": true }

---

Thought: Applicant income is below $50K. I should check their credit score before scoring risk.

Action: CheckCreditScore

Input: { "ssn": "123-45-6789" }

Observation:

{ "score": 710, "tier": "Prime" }

---

Thought: Now I can calculate the risk score.

Action: ScoreRisk

Input: { "income": 48000, "health_index": 0.82, "credit_score": 710 }

Observation:

{ "risk_score": 0.26 }

---

Thought: With this risk score, I will now request a policy quote.

Action: PricingAPI

Input: { "risk_score": 0.26, "coverage_amount": 100000 }

Observation:

{ "monthly_premium": 92.50, "deductible": 500 }

---

Thought: I will now generate the final policy document.

Action: DocGen

Input: {

"applicant_data": { "name": "Jane Smith", ... },

"quote": { "monthly_premium": 92.50, "deductible": 500 }

}

Observation:

{ "document_url": "https://storage.insuranceco.com/policies/policyA1234.pdf" }

---

Thought: I’ll send the document to the applicant for e-signature.

Action: Signer

Input: { "doc_url": "https://.../policyA1234.pdf", "email": "jane@example.com" }

Observation:

{ "signing_url": "https://sign.insuranceco.com/session/A1234" }

---

Thought: Once the policy is signed, I’ll archive it.

Action: PolicyStore

Input: {

"doc_url": "https://.../policyA1234.pdf",

"metadata": {

"applicant_id": "A1234",

"status": "signed"

}

}

Observation:

{ "result": "Archived successfully with ID POL-2024-00123" }

---

Final Answer: The policy for applicant A1234 has been processed, signed, and archived.

💡 What if your platform could understand a business goal—like "Make a payment" or "Issue a policy"—and then figure out how to get it done, autonomously?

Not just triggering an API.

But reasoning, planning, and collaborating—across systems, teams, and tools—without a human orchestrating every step?

That’s what Enterprise Agentic AI unlocks. And the best part? You don’t have to rip and replace your current stack.

Let’s dive in.

🔧 Most Enterprises Are Still in “Task Mode”

APIs get triggered. Workflows get routed. But behind every “automated” task is still a team coordinating steps, handling exceptions, and stitching it all together.

Agentic AI changes that. It takes a goal and autonomously figures out what needs to be done—using agents, tools, and dynamic context.

Let me walk you through how this looks in an enterprise setup.

🧱 Foundation First: Your Existing Stack

You likely already have:

• Core Systems → Policy Admin, Billing, Underwriting

• Data Mesh or Domain Data Layers

• Integration APIs and Events

• Channels → Portals, Mobile Apps, IVR, Contact Center

Your architecture is modular and composable. Now, let’s add Agentic Intelligence on top—without changing what's already working.

🧠 The Enterprise Agentic AI Platform

Here’s how the Agentic layer transforms your stack into an intelligent, autonomous platform:

🎯 1. Everything Starts with a Goal

Users interact via:

• Portals, Mobile, IVR, Contact Center, Emails

They express a goal, not a task:

“Make a payment.” “Request loan against policy.” “Tell me my premium due.”

The system receives a goal-oriented request, not a hard-coded workflow.

🤖 2. Agents Handle Capabilities

Agents are intelligent business components like:

• UnderwritingAgent

• PolicyIssuanceAgent

• ClaimAdjudicationAgent

• PolicyCancellationAgent

Each agent:

• Accepts a goal

• Decides which tools to use

• Updates shared state

• Delivers outcomes

📌 Dynamic Agent Registry & Discovery Agents are not hardcoded. They're:

• Registered into a dynamic Agent Registry

• Discovered at runtime by the LLM-based planner

• Selected based on metadata: domain, capabilities, and SLA

This enables plug-and-play business logic, where new agents can be deployed and instantly leveraged.

💡 3. LLM Reasoning Layer (Inside the A2A Router)

This layer uses LLMs like GPT-4o, Gemini, or Claude to:

• Break goals into sub-tasks

• Select and sequence agents

• Adapt to context (from shared memory)

• Optimize execution paths

This LLM planner doesn’t execute—it plans. Then hands off the execution to the A2A Router.

🔄 4. Agent2Agent Router: Digital Conductor

The A2A Router:

• Executes the LLM plan

• Passes sub-goals to the right agents

• Maintains sequence and shared state

• Orchestrates cross-agent collaboration

Think of it as your digital conductor, enabling multi-agent flows in real time.

🧠 5. MCPContext: Shared Execution Memory

All agents and tools read/write to a shared MCPContext, which:

• Stores the original goal

• Tracks decision history

• Maintains state across steps

• Enables fallback, recovery, and audit trails

It’s like an intelligent black box that every agent can access during execution.

🔧 6. Tools: The Executors of Action

Agents never call APIs directly. They invoke Tools—lightweight wrappers around backend APIs—via the MCP Gateway.

Each tool:

• Wraps an API with auth, SLA, retries, and logging

• Is tagged with metadata (domain, use-case, version)

📌 Tool Registration & Dynamic Discovery Tools are:

• Registered with a central Tool Registry

• Discovered dynamically by the MCP Gateway at runtime

• Selected based on metadata like domain, reliability, and priority

This makes your enterprise services discoverable, secure, and pluggable—no hardcoded API calls.

🛠 7. MCP Gateway: Your Enterprise Firewall

The MCP Gateway ensures:

• Dynamic Tool Discovery

• API-level auth, rate limits, retries

• Routing to correct MCP Server

• Full observability and logging

It’s the controlled interface between agents and enterprise systems.

🗂 8. MCP Servers: Domain-Aligned Logic

Each domain—Policy, Claims, Billing, etc.—gets its own MCP Server, such as:

• Policy_MCPServer

• Claims_MCPServer

They expose reusable tools like:

• IssuePolicyTool

• PayClaimTool

• UnderwriteAppTool

This keeps logic modular, scalable, and domain-aligned.

🛡 9. Governance and Control Plane

Autonomy needs rules. This layer enforces:

• Role-based tool access

• SLA enforcement and circuit breakers

• Failure thresholds and fallback paths

• Real-time logs and metrics

🔁 10. Reactive Event Triggers

Not all flows start with a user.

Agents can also be triggered by events:

• Policy lapsing

• Claim rejection

• Payment failure

This lets the platform run asynchronously and proactively.

🧩 Built On Your Stack, Not Instead Of It

This Agentic AI platform connects downward to:

• Your APIs and Events (Integration Layer)

• Your Domain Data (Data Mesh)

• Your Core Systems (Admin, Billing, GL)

You’re not replacing anything. You’re adding intelligence, autonomy, and dynamic discovery to what already works.

✅ Why It Works

• 🔗 Composable and domain-aligned

• 🤖 Agent- and tool-driven, dynamically discoverable

• 🧠 LLM-powered, goal-driven reasoning

• 🔍 Secure, observable, and auditable

• ☁️ cloud-native or fully cloud-agnostic

🔁 Final Thought

This isn’t about a smarter chatbot or a better workflow engine.

This is about building a self-orchestrating enterprise. One where goals become outcomes—autonomously.

💬 How are you building your Enterprise Agentic AI Platform? Let's exchange notes. Drop in your comments.

In my last article, I shared why agents are the next leap forward in enterprise architecture — enabling systems to deliver outcomes, not just expose data.

Today, let's take the next logical step: From APIfication to Agentification.

For the past decade, enterprises have focused on APIfying their systems — exposing backend capabilities through standardized APIs. It was a critical step that unlocked modularization, partner integration, and digital ecosystems.

But as businesses demand faster, smarter, more resilient workflows, APIfication alone isn't enough anymore. Simply exposing APIs leaves too much complexity, orchestration, and error handling burden on the client side.

Agentification is the natural evolution — where intelligent agents take goals, reason dynamically, and figure out how to orchestrate APIs, tools, and services to deliver outcomes autonomously.

In this article, I explore:

✅ Why APIfication was necessary — but not sufficient

✅ What Agentification really means

✅ How agents build on APIs to create truly adaptive, goal-driven systems

✅ Why enterprises must start preparing for this shift now

Why We Must Evolve: The Limits of APIfication

APIfication solved the challenge of accessibility — unlocking backend capabilities.

However, it left orchestration and goal achievement to external clients, workflow engines, or manual processes, creating challenges like:

• Clients must hardcode the exact sequence of API calls.

• Exception handling becomes a maze.

• Cross-domain workflows stay brittle.

• Integration complexity grows linearly with scale.

In short:

👉 APIs expose operations.

👉 Clients still carry the burden of stitching operations into outcomes.

As enterprises scale, these challenges only compound.

The Shift: Why Agentification is the Logical Next Step

Agentification builds on what APIfication enabled — but moves the enterprise from exposing operations to delivering business outcomes autonomously.

In an agentic architecture:

• Clients submit goals ("Issue this policy").

• Agents reason, plan, and dynamically orchestrate APIs to achieve outcomes.

• APIs remain critical — but orchestration happens internally, not manually by clients.

👉 APIs expose how.

👉 Agents deliver what.

Quick Comparison: APIs vs. Agents

Here’s a quick side-by-side view:

Architecting for Agentification: Key Changes

Agentification isn’t about wrapping AI around APIs. It requires a real architectural shift.

Here's how the approach changes:

The Core Enablers of Agentic Platforms

To move toward agent-driven execution, enterprises must build three foundational layers:

✅ No rip-and-replace required — these layers build on top of existing APIs, reframing how they are orchestrated.

Real Example: From Manual Orchestration to Autonomous Execution

Today: Issuing an insurance policy requires manually chaining:

• Search Customer API

• Validate Underwriting API

• Issue Policy API

• Generate Document API

Each step must be hardcoded. Any backend change risks breaking the chain.

With Agentification: An agent receives the goal: "Issue a policy for Customer X." It autonomously:

• Finds the customer

• Validates underwriting

• Issues the policy

• Generates necessary documents

👉 If underwriting temporarily fails or data is incomplete, the agent reasons through alternative paths — without human intervention or workflow rewrites.

The client simply submits the goal. The platform delivers the outcome.

Why Enterprises Should Move Now

Here are the strategic benefits agentic architectures unlock:

Final Thought: Architecting for the Next Era

APIfication gave us modularity, openness, and system access. But it only solved part of the challenge.

The real opportunity — and necessity — is moving beyond exposing capabilities to orchestrating intelligence.

Agentification is not about replacing APIs. It’s about elevating enterprise platforms from operation-driven to outcome-driven.

🚀 Enterprises that make the leap from APIfication to Agentification will be the ones leading in speed, resilience, and agility for the next decade.

Would love to hear how you're thinking about agentic architectures in your organizations!

#EnterpriseArchitecture #DigitalPlatforms #AgenticAI #API #CloudArchitecture #EnterpriseTech #FutureOfWork

Here’s the Agentic AI Reference Architecture — a blueprint for how high-level goals turn into autonomous, traceable actions without hardcoded workflows.

Let’s walk through the journey from goal to execution:

👇 Take a look at the full architecture and detailed explanation below, and let me know — how would you build this in your org?

🧑💼 User

• Initiates a business goal (e.g., “Make a payment”, “Issue policy”) via a digital channel

• May be a human, bot, or external system

🔍 This is where it all begins — a clear business outcome is desired. What matters is that we treat this as a goal, not just an API request.

💬 Customer Engagement Application

• Frontend application capturing the user's goal

• Sends structured goal requests to the MCP Client (e.g., web portal, chatbot, mobile app)

🧭 This is the interface layer that bridges people and systems with the agentic backend. It doesn't handle logic — just captures intent and forwards it.

📦 MCP Client

• Converts user input into a standardized goal object as per the MCP protocol

• Adds necessary metadata (e.g., goal type) and forwards the request to the MCP Gateway

🧱 Think of the MCP Client as the translator between your application and your agentic infrastructure. It ensures all goals follow a consistent, composable structure.

🚪 MCP Gateway

• Entry point for all MCP-compliant requests

• Authenticates, validates, and routes the goal to the correct MCP Server based on the routing config

• Looks up routing in the MCP Registry

🔒 This is where governance lives — it ensures requests are valid, secure, and routed to the right domain. It's like the traffic cop of your agentic platform.

🧠 MCP Server(s)

• Receives high-level goals and sends them to the Reasoning Layer for task identification and planning

• Receives task plans from the Reasoning Layer

• Acts as the anchor of execution in a specific business domain (e.g., payment, policy)

• Reads and writes to Shared Context

⚙️ The MCP Server orchestrates the full lifecycle of a goal. It delegates the thinking (reasoning) and doing (agents), while keeping track of the process and context.

🧩 Reasoning Layer

• Decomposes goals into subtasks using LLMs

• Consults the Agent Registry to determine which agents and tools are required and assigns tasks appropriately

🧠 This is the cognitive engine of the system. Instead of static workflows, it dynamically figures out what needs to happen — like a planner in your org with perfect memory and context.

🗂️ Agent Registry

• Directory of agent capabilities, domains, and endpoints

• Queried by the Reasoning Layer to find agents that can handle each subtask

📚 This registry allows for discovery and dynamic assignment. New agents can be added or updated without rewriting workflows — it's plug-and-play autonomy.

🔀 Agent-to-Agent (A2A) Router

• Executes the task plan by dispatching subtasks to selected agents

• Coordinates inter-agent communication and task routing

🔁 This router is what lets agents operate independently, but still in sync. It ensures that each agent gets the right job at the right time — and keeps the goal moving forward.

🤖 PaymentProcessor Agent / PolicyDetailsProvider Agent

• Execute assigned subtasks by selecting and invoking appropriate tools

• Use Tool Registry to dynamically look up compatible tools based on task and schema

• Read and write to shared context

🎯 These agents are the executors. They don’t need to be told how to do things — just what needs to be done. They consult the registry and choose the right tool on their own.

🧰 Tool Registry

• Metadata store for tools: inputs, outputs, domains, and endpoints

• Enables dynamic tool discovery by agents at runtime

🔍 Agents don’t hardcode tool logic. They query this registry — just like we Google for answers — and get back the best tool for the task.

🛠️ Tool 1–4

• Encapsulate business logic, rules, or integration workflows (e.g., validation, document generation)

• Invoked by agents with structured input and expected output formats

🔧 These are your building blocks — tools that perform specific actions. Agents wrap these tools and invoke them on demand, just like how humans use apps to get things done.

🌐 API 1–4

• Actual backend systems or third-party services

• Called by tools to complete a specific task (e.g., get quote, charge card, retrieve policy)

🔗 These are the final legs of the journey — the systems that still do the heavy lifting. But now, instead of being directly triggered by UI code, they’re invoked autonomously through agents and tools.

✅ What Does This Architecture Enable?

✔️ Business goal in → Outcome out — with minimal human orchestration ✔️ No hardcoded workflows — just autonomous execution ✔️ Modular, composable, and easy to evolve ✔️ Full traceability via shared context

This is how enterprises can evolve from API-first to agent-first execution — where systems don't just expose functionality, they drive results.

#AgenticAI #EnterpriseArchitecture #AIinBusiness #LLM #AutonomousExecution #DigitalTransformation #FutureOfWork