Ask the first ladder when you are evaluating a vendor, a pilot, or a system you already run: who approves the action? Ask the second when you are deciding what your own team can responsibly operate: what job does this person do while the model works? Use the answers to identify gaps in ownership, training, permissions, or evaluation before expanding the deployment. These are this guide’s qualitative frameworks, not validated scores or a certification of readiness.
What a deployment can do: six stages of sophistication
Vendors use “agent” for different architectures. This guide groups deployments into six stages for discussion, drawing on the sources below; those sources do not share one six-stage standard. Ask the vendor to demonstrate its tools, approval gates, failure handling, and measured results on your tasks.
The sophistication ladder
Color and height indicate order, not a measured capability score. Stage 6 is not six times as capable as Stage 1. Choose a stage for its summary; full cards follow below. Stages 5 and 6 describe aspirational and theoretical designs.
Autonomy and scope increase from Stage 1 to Stage 6. Higher autonomy calls for stronger reliability and auditability checks.
View as table
| Stage | What defines it | The human's role | In insurance |
|---|
A stage is a property of the deployment, not just the model. The same product can behave as Stage 2 or Stage 3 depending on its permissions, tools, and approval gates. Ask how the system is wired, not just what the model can do.
The stages, in full
What a person can do: six levels of practice
The second ladder is about you, not the model: how you prompt, what you let the model see, and what you trust it to do. Each rung changes your job: asker, brief writer, curator, architect, boss, orchestrator. The skill that compounds at every one of them is verification, because outputs get more polished as you climb and polish hides errors.
The practice ladder
An ordinal guide to practice, not a quantitative fluency scale. Equal steps do not imply equal gains. Choose a level for its summary; full cards follow below.
Each level hands the model more of the work, and you more of the design, governance, and verification.
View as table
| Level | What defines it | The mindset | In insurance |
|---|
Higher is not better. Seniority means reaching for the lowest rung that closes the quality gap: a good prompt beats a pipeline, a pipeline beats an agent, one governed agent beats a fleet. Autonomy is a cost, not a badge.
The levels, in full
The harness: controls for dependable work
A harness is the software and operating process around the model: context, tools, permissions, checks, and stopping rules. It defines what the system may do and what evidence a reviewer can inspect.
Start with a bounded task and a clear acceptance standard. Validate the full workflow, including exceptions, and keep a human approval step where the consequences require it. A successful automated check reduces uncertainty within its scope; it does not establish that the whole answer is correct.
How a harness supports review
Define the allowed work, test the result, then route it according to the evidence. A passed check supports review within its scope; it does not prove the whole answer is correct.
Check the evidence and the behavior of the workflow, including what happens when inputs are missing or a tool fails. Levels 4 to 6 describe the skills needed to build and supervise those controls.
Organize the harness around four controls. Levels 4 to 6 of the ladder above are, in effect, the harness skills: the Workflow Designer builds fixed checks, the Agent Supervisor wires hooks and goal contracts, the Fleet Orchestrator runs verification loops across teams of agents.
Skills, goals, and loops: three different jobs
Agent Skills arrived in October 2025 and became an open standard in December. They made procedural knowledge portable: instructions, scripts, templates, and reference material could be versioned once and loaded only when relevant. This strengthened harnesses by separating how the organization performs a task from the model, prompt, and current conversation.
| Element | Question it answers | What good looks like |
|---|---|---|
| Skill | How should this task be performed? | A versioned playbook with instructions, examples, deterministic scripts, and tests |
| Goal contract | What outcome counts as done? | Required artifacts, evidence, constraints, budgets, approval points, and success criteria written before execution |
| Agent loop | How does the system close the gap? | Observe state, choose the next action, call a tool, verify the result, update progress, repeat |
| Stopping conditions | When must the loop end or escalate? | Success achieved; iteration/token/time budget reached; no progress; low confidence; blocked or irreversible action |
Define a successful outcome before running the loop. A goal without stopping conditions invites runaway cost or premature claims of completion. The late-2025 long-running-agent harness pattern made this concrete: expand the goal into default-failing requirements, work incrementally, verify end to end, and leave durable progress artifacts for the next session.
Insurance example. A submission-intake skill defines the schema, extraction procedure, appetite sources, and reconciliation scripts. Its goal contract requires a decision-ready brief with every field traced to source, totals reconciled, and exceptions identified. The loop stops when those checks pass, or escalates on missing documents, unresolved conflicts, low confidence, or any action involving quote or bind authority.
Lever 1Curate the context
Context engineeringControl what the model sees: the right documents, in tagged sections, with examples and constraints. Nothing extraneous, nothing stale.
The craft- Thin always-on foundation (role, task, house standards) plus just-in-time retrieval of the specific account, filing, or guideline
- Worked examples beat instructions: paste one gold-standard output and say "match this"
- Context hygiene: a stale appetite guide or superseded form in context is a hazard, not a resource
- Underwriting assistant sees: the submission, the current appetite guide, two exemplar briefs, and the referral rules. Nothing else
- When output disappoints, inspect the source material, retrieval, instructions, tools, and model behavior before choosing a remedy
Lever 2Decompose the work
Pipelines before agentsSplit the job into stages with checkpoints: extract, validate, summarize, draft. Each stage is reviewable, and errors surface where they happen.
The craft- Fixed pipelines for known work; free agents only where the path genuinely varies
- Deterministic steps in code (validation, arithmetic, appetite scoring), model steps for language and judgment
- Each stage writes a structured artifact a human or the next stage can check
- Loss-run analysis: extraction agent pulls fields, code checks totals reconcile, model writes the trend narrative. Failed reconciliation blocks the narrative; source errors still need review
- Resist the giant do-everything prompt; it fails silently at one of five steps and you cannot tell which
Lever 3Constrain the tools
Permissions in codeGive the agent exactly the tools the task requires, with limits enforced by the system, not by the prompt. Enforce limits outside the model and test the enforcement, including bypass and failure paths.
The craft- Least privilege: read-only by default; write and send actions require a human gate
- Limits in code: binding authority, reserve caps, and PII rules enforced as hooks, not as warnings in the system prompt
- Standard connectors (MCP-style) to the rating engine, policy admin, and document store, each scoped
- An underwriting agent may draft a quote up to a set premium threshold; above it, a hook blocks and routes to a named underwriter
- Claims agent drafts reserve recommendations; the reserve itself moves only on adjuster approval, logged
Lever 4Verify the output
Evals and gatesEvaluate outputs against reviewed examples and relevant source material. Set acceptance thresholds for material fields and decisions, then define what a human must approve.
The craft- Start a pilot with 20 to 50 reviewed cases, including failures and exceptions; expand coverage for the risk and variety of production work
- Field-level re-checks against source documents, done by code or a grader agent
- Drift review: re-run the eval set on model upgrades and on a schedule; behavior changes silently
- Extraction example: compare against independently reviewed submissions, assess errors on material fields, and validate on cases held out from development
- Define when full review is required; use exception review and random sampling only where the evidence and risk assessment support them
Three worked harness patterns
Illustrative assemblies of the four controls for three candidate workflows. Each lists the pipeline, the checks, what the human sees, and the metric that tells you it works.
| A · Submission intake (underwriting) | B · Claims file synthesis (claims) | C · Filing research assistant (actuarial) | |
|---|---|---|---|
| Pipeline | Email + attachments → extraction agent → enrichment (third-party data) → appetite scoring in code → underwriter brief | Adjuster notes, correspondence, medicals → document index → chronological summary with citations → severity signal | Question → retrieval over state filings and rating manuals → comparison table → draft support-memo sections |
| Hooks and checks | Every extracted field re-checked against the source document; totals reconcile in code; out-of-appetite flagged, never quoted | Every claim number and date must cite a page; PII redaction hook on anything outbound; severity signal labeled as advisory | Citations must resolve to a real filing and page; numbers recomputed in code; draft memo labeled "draft, unreviewed" |
| What the human sees | A verified queue: brief, score, exceptions. Not raw agent chatter | The summary plus the flagged severity rationale; one click to any cited page | The comparison table and the draft; the actuary edits, the assistant never files |
| Metric to watch | Field-level accuracy on the golden set; minutes from inbox to briefed; hit ratio over time | Adjuster time per file; citation accuracy; override rate on severity signals | Hours per filing research cycle; citation resolution rate; review comments per draft |
The starter checklist: first harness in 30 days
- Pick one bounded workflow with a document in, a decision out, and a measurable current cost (submission intake and claims summarization are examples to assess).
- Write the context spec: exactly what the agent should see, from where, refreshed how often. Name the owner of each source.
- Assemble the golden dataset: a starting sample of reviewed historical cases, with difficult examples and a separate validation set. Sample size must reflect the intended use and risk.
- Decompose into stages and decide which steps are code and which are model.
- List the hooks: the checks that run every time, including the ones that block (PII, authority limits, reconciliation).
- Define the human gate: what a person must approve, and what "approve" means they checked.
- Define the evidence record: preserve necessary versions, decisions, overrides, and check results. Minimize sensitive input/output logs and apply approved retention and access controls.
- Set the eval gate: the accuracy bar the harness must hold on the golden set before any case skips full human review.
- Run the pilot in shadow mode (agent works, humans still do the job) for two to four weeks; compare.
- Decide kill or scale on the metrics, then schedule the drift review. Assign ongoing ownership of the model, software, source data, and controls.
Failure modes to design against
| Failure mode | What it looks like | The countermeasure |
|---|---|---|
| Stale context | Agent applies last quarter's appetite or a superseded form | Context hygiene: owners, refresh dates, retrieval over static dumps |
| Automation bias | Reviewers approve fluent output without checking | Surface the evidence (citations, reconciliation), not just the answer; spot-audit approvals |
| Prompt-only guardrails | "Do not quote above X" in the system prompt, working until it doesn't | Limits enforced in code hooks; prompts inform, hooks enforce |
| Silent drift | A model upgrade changes behavior; nobody notices for a month | Scheduled golden-set runs; re-run on every provider version change |
| Unlogged overrides | Humans correct the agent constantly; no record, no learning | Override logging with a one-tap reason; feed corrections into the golden set |
Human oversight needs a defined task, time to perform it, and authority to change or stop the outcome. Document which outputs receive full review, which receive sampling, and how missed errors are detected and corrected. Technical controls support this process; they do not replace accountable ownership.
Questions to resolve before adding autonomy
- Would a simpler workflow meet the need? Compare a single model call, a fixed workflow, and an agent against the same acceptance criteria. Anthropic's engineering guidance recommends increasing complexity only when it improves outcomes.
- Can the work run independently? Anthropic's research system benefited from parallel investigations, with substantially greater token usage. This is evidence about its task and architecture; test the tradeoff for your own work.
- Who approves consequential actions? Identify the permitted actions, reviewer, escalation route, and enforced limits. A deployment's stage does not establish its legal or operational readiness.
- What would expose a missed error? Include held-out cases, adverse scenarios, source checks, and ongoing review. A second model can assist review but may share the first model's blind spots.
- What capability does the team need next? Use the practice ladder to plan training and ownership. Experienced staff may choose a simple tool when it meets the need.
Source links
- Sophistication frameworks: DeepMind, "Levels of AGI" · Gartner AI maturity model · NVIDIA AI agents glossary · Hugging Face agents course · Cloud Security Alliance agentic work
- Anthropic engineering series: Building effective agents · Context engineering · Multi-agent research system · Prompt engineering docs · AI Fluency framework
- Practice research: Mollick/HBS, "Centaurs and Cyborgs on the Jagged Frontier" · Rethink Priorities usage surveys · Microsoft Work Trend Index (Frontier Firm)
- Architecture and harness: LangChain on context engineering · Cognition, "Don't Build Multi-Agents" · HumanLayer, 12-factor agents · DAIR.AI Prompt Engineering Guide · Claude Code docs (hooks and permissions in practice)
- Verification and governance: METR task horizons · NAIC Model Bulletin (testing, human oversight) · NIST AI RMF
The "context engineering moment" references Karpathy's and Lütke's June 2025 posts; the harness sections draw on the same engineering sources. Market and regulatory claims live on the integration phases, which carries numbered references, with verbatim quotes on the evidence page. Foundations: the hands-on use section.