Most agentic AI discussions start with frameworks: LangChain vs CrewAI vs AutoGen vs the latest entrant. But frameworks are a Layer 5 problem. Before you pick a framework, you need to answer harder questions. Which models will your agents use? How will they access enterprise data? Who governs their actions? How do you know they are working?
The reference architectures published in 2023 and 2024, including a16z’s influential “Emerging Architectures for LLM Applications,” predate production agents, MCP, A2A, model routing, and the SLM revolution. They describe a world where you call one large language model through one API. That world no longer exists.
At Wenable Labs, we have deployed agentic AI systems across device management, fleet operations, and pharmaceutical quality. Through that work, we have converged on a 7-layer architecture that separates concerns, enables independent scaling, and provides the governance controls that regulated industries require.
This is not a theoretical framework. Every layer reflects decisions we have made in production.
The Seven Layers
From infrastructure up:
- Foundation Infrastructure Compute, model serving, data pipelines
- Model Intelligence Model selection, SLM/LLM routing, fine-tuning, synthetic data
- Memory, Context & Knowledge RAG, vector search, knowledge graphs, memory systems
- Tools, Protocols & Integrations MCP, A2A, API gateways, enterprise connectors
- Agent Orchestration Single-agent loops, multi-agent coordination, stateful workflows
- Observability, Evaluation & Governance Tracing, guardrails, compliance, human-in-the-loop
- Applications & Human Interface Vertical agents, copilots, conversational interfaces
Each layer depends on the layers below it but can be developed and scaled independently. This separation is not academic. It is what allows you to swap a model without rewriting your agents, or add a new data source without touching your orchestration logic.
Layer 1: Foundation Infrastructure
Foundation infrastructure is the compute and runtime substrate that everything else depends on. This includes model serving runtimes (vLLM, TGI, Ollama for development), GPU allocation strategies, vector database deployments, data ingestion pipelines, and the cloud or hybrid infrastructure that hosts it all.
The critical decision at this layer is not which cloud provider to use. It is whether to build for cloud-only, edge-only, or hybrid deployment. An agent managing a fleet of mobile devices needs edge inference with sub-100ms latency. An agent investigating pharmaceutical quality deviations can tolerate cloud round-trips because accuracy matters more than speed.
We have found that starting with cloud-hosted serving and adding edge capabilities only where latency demands it produces the best cost-to-performance ratio. The infrastructure should be model-agnostic: swapping Gemma for Qwen should not require redeploying your serving layer.
Key technologies: vLLM, Ollama, NVIDIA Triton, cloud-managed inference (AWS Bedrock, Azure AI), vector databases (Qdrant, Weaviate, pgvector), data pipelines for continuous ingestion.
Layer 2: Model Intelligence
This is the layer most architectures overlook, and it is where the largest cost and quality optimizations happen.
The era of the single-model agent is over. A production system should not route every query through GPT-4 or Claude Opus when 80% of tasks can be handled by a fine-tuned 3B parameter model at 1/100th the cost. Model Intelligence is the discipline of selecting, training, and routing between models based on task complexity, latency requirements, and cost constraints.
Model Routing
The pattern we deploy most often is SLM-default, LLM-fallback. A small, fine-tuned model handles routine tasks: policy lookups, status queries, data formatting. When the task exceeds the SLM’s capability, determined by confidence scoring or task classification, the request routes to a larger model. In our fleet intelligence systems, this routing strategy reduced inference costs by over 70% with no measurable quality degradation on routine queries.
Fine-Tuning
Generic models know a lot about everything but not enough about your domain. A fine-tuned Qwen 3B model trained on device management documentation outperforms a general-purpose 70B model on policy configuration tasks, because it has seen thousands of examples specific to that domain.
We use LoRA and QLoRA adapters for efficient fine-tuning. The key is not the fine-tuning technique itself but the data pipeline that feeds it: synthetic data generation from enterprise documents, quality filtering, and continuous evaluation against production benchmarks.
Synthetic Data
Real enterprise training data is scarce and often sensitive. We generate high-fidelity synthetic training examples using teacher-student architectures: a large model generates candidate examples from enterprise documents, a validation pipeline filters for accuracy and diversity, and the curated dataset trains the domain-specific SLM. This pipeline has produced training sets across device management, HOS compliance, and pharmaceutical quality domains.
Key technologies: Unsloth, LoRA/QLoRA, GGUF quantization, model routers, confidence-based routing, instructor library for synthetic data generation, evaluation frameworks.
Layer 3: Memory, Context & Knowledge
Agents without memory are stateless functions. Agents without knowledge hallucinate. This layer provides both.
Retrieval-Augmented Generation (RAG) grounds agent responses in your operational data. But production RAG is not the “chunk documents, embed, retrieve” pattern from tutorials. In our pharmaceutical quality system, we process handwritten deviation reports, structured batch records, and regulatory databases, each requiring different parsing, chunking, and retrieval strategies. Hybrid search (dense embeddings combined with sparse keyword matching) consistently outperforms either approach alone.
Memory systems give agents continuity across interactions. We implement three types: short-term memory (current conversation context), working memory (intermediate results within a multi-step task), and long-term memory (learned patterns and preferences that persist across sessions). Our fleet intelligence system uses long-term memory to learn which alert thresholds matter most to each fleet manager.
Knowledge graphs capture relationships that vector search misses. When an agent investigates a quality deviation, it needs to understand that Machine X is in Building Y, maintained by Technician Z, and last serviced on a specific date. These structural relationships are better represented as graphs than as embedded text chunks.
Key technologies: Vector databases (Qdrant, Weaviate), embedding models, hybrid search, knowledge graph engines (Neo4j), multi-type memory stores.
Layer 4: Tools, Protocols & Integrations
An agent that cannot act on the real world is just a chatbot. This layer connects agents to enterprise systems through standardized protocols.
Model Context Protocol (MCP) has become the standard for agent-to-tool connectivity. Now governed by the Linux Foundation with tens of millions of monthly SDK downloads, MCP provides a consistent interface for agents to query databases, call APIs, read file systems, and interact with enterprise software. We build MCP servers for every enterprise system our agents need to access: MDM consoles, ELD hardware interfaces, TMS platforms, and manufacturing execution systems.
Agent-to-Agent Protocol (A2A), backed by dozens of enterprise partners including Google, enables agents built on different frameworks to communicate and delegate tasks. In multi-vendor enterprise environments, A2A allows a compliance monitoring agent to delegate data retrieval to a specialized agent built with a completely different stack.
The design principle at this layer: agents should never access enterprise systems directly. Every integration goes through a protocol-compliant connector with authentication, rate limiting, and audit logging.
Key technologies: MCP servers, A2A protocol, API gateways, OAuth/OIDC for agent authentication, enterprise connectors.
Layer 5: Agent Orchestration
This is where most agentic AI conversations begin, but it is actually the middle of the stack, not the foundation.
Single-agent orchestration follows the now-standard loop: perceive (gather context), reason (plan next steps), act (execute tools), and reflect (evaluate results). The nuance is in the failure modes. When an agent’s action fails, does it retry, escalate, or try an alternative approach? Production agents need explicit error handling strategies, not just optimistic tool calling.
Multi-agent coordination becomes necessary when a task exceeds what one agent can handle. Our pharmaceutical RCA system deploys six specialized agents, each investigating one dimension of the Ishikawa model, with a supervisory agent that synthesizes their findings. The coordination patterns we use most frequently:
- Parallel fan-out: Multiple agents investigate simultaneously, results merged by a supervisor
- Sequential handoff: One agent’s output feeds the next in a pipeline
- Hierarchical delegation: A supervisor decomposes tasks and assigns them to specialists
The framework choice at this layer matters less than the orchestration patterns. Whether you use LangGraph, CrewAI, or custom orchestration code, the patterns remain the same. What matters is that the orchestration layer is stateful, observable, and recoverable. If the system fails mid-workflow, it should resume from the last checkpoint, not restart from scratch.
Key technologies: LangGraph, CrewAI, custom orchestration engines, state machines, workflow checkpointing, event streaming.
Layer 6: Observability, Evaluation & Governance
This is the layer that separates demos from production systems. Without it, you have agents that work in testing and fail silently in production.
Observability means tracing every agent decision: which model was called, what context was provided, what tools were invoked, what the response was, and how long each step took. When a fleet intelligence query returns an incorrect answer, we need to trace backward through the agent’s reasoning to identify whether the failure was in retrieval, reasoning, or tool execution.
Evaluation is continuous, not one-time. We run automated evaluation pipelines against production traffic, measuring retrieval accuracy, response faithfulness, and task completion rates. When metrics drift below thresholds, alerts fire before users notice degradation.
Governance in regulated industries means deterministic policy enforcement outside the LLM’s reasoning loop. The agent does not decide whether it has permission to modify a device policy or approve a CAPA document. A governance layer enforces role-based access controls, compliance rules, and approval workflows independently. Every action carries an audit trail with timestamps, actor identity, and reasoning transparency. For pharmaceutical systems, this means FDA 21 CFR Part 11 compliance: electronic signatures, tamper-evident records, and full traceability.
Key technologies: LangSmith, Langfuse, custom tracing, automated evaluation pipelines, policy engines, audit logging, RBAC.
Layer 7: Applications & Human Interface
The top of the stack is where agents meet users. This layer determines whether the underlying technology delivers business value or remains a technical demo.
The most effective pattern we have deployed is the conversational interface with structured outputs. IT administrators talk to ViVi in natural language to manage device policies. Fleet managers ask plain-English questions about driver compliance. The agent understands the intent, executes the workflow through the layers below, and returns structured, actionable results.
Human-in-the-loop workflows are not a compromise. They are a design choice. Our pharmaceutical quality system requires human approval at specific checkpoints: a QA manager must review and approve AI-generated CAPA documents before they enter the regulatory record. The system presents its reasoning, the supporting evidence, and a recommended action. The human approves, modifies, or rejects. This is not the agent asking for permission. It is the system enforcing governance at decision points where human judgment is required.
The application layer also includes dashboards and monitoring interfaces that give operations teams visibility into agent activity, performance metrics, and cost tracking.
Key technologies: Conversational interfaces, streaming response UIs, structured output rendering, approval workflow engines, analytics dashboards, notification systems.
The Stack in Practice
This framework is not theoretical. Here is how the layers compose in a real deployment.
When a fleet manager asks “Which drivers are at risk of HOS violations this week?”, the request flows through all seven layers:
- Layer 7 receives the natural language query through the conversational interface
- Layer 5 routes it to the compliance analysis agent
- Layer 2 selects a fine-tuned SLM optimized for HOS query parsing
- Layer 1 serves the model through vLLM, executing inference on allocated GPU compute
- Layer 3 retrieves relevant driver logs and regulatory rules via RAG
- Layer 4 queries ELD hardware through MCP connectors for real-time data
- Layer 5 coordinates parallel analysis across fleet regions
- Layer 6 logs every step, enforces data access controls, and records the audit trail
- Layer 7 returns a structured risk report to the fleet manager
Each layer operates independently. Swapping the model (Layer 2) does not affect the orchestration logic (Layer 5). Adding a new data source (Layer 4) does not require changes to governance rules (Layer 6). This separation of concerns is what makes the system maintainable and extensible at enterprise scale.
Building Your Stack
If you are building agentic AI systems, start from the bottom up.
The most common failure mode we see is teams beginning at Layer 5, picking an orchestration framework, without addressing Layers 1 through 4. The result is agents that work in demos but fail in production because they lack the infrastructure, model optimization, knowledge grounding, and integration plumbing that production systems require.
The second most common failure is skipping Layer 6. Agents without observability and governance are a liability in any enterprise. In regulated industries, they are a compliance violation.
Build the stack. Build it bottom-up. And build it for production from day one.