Each rung changes your job: asker, brief writer, curator, architect, boss, orchestrator. The skill that compounds at every rung is verification, because outputs get more polished as you climb, and polish hides errors.
The practice ladder
Height and color depth encode the same ordering. Select a level for a summary, or open the full cards below.
Each level hands the model more of the work, and you more of the design, governance, and verification.
View as table
| Level | What defines it | The mindset | In insurance |
|---|
Higher is not better. Seniority means reaching for the lowest rung that closes the quality gap: a good prompt beats a pipeline, a pipeline beats an agent, one governed agent beats a fleet. Autonomy is a cost you spend, not a badge you earn.
This page tracks your practice; the companion page tracks the technology itself. See the six stages of AI sophistication. One tells you what vendors can ship; the other, what your team can responsibly use.
The levels, in full
The harness: how slop becomes reliable work
The unit of AI work is no longer the conversation; it is the ephemeral agent. Modern tooling spins up short-lived agents by the thousand: each gets a task, a slice of context, and a set of tools; it works, reports, and is shut down. Fleets run around the clock, and the marginal cost of one more attempt is a few cents of compute.
Most raw agent output is slop: plausible, confident, unverified. Agent fleets do not eliminate slop; they industrialize it. Ten unsupervised agents produce ten times the plausible garbage. So the discipline of the agentic era is not getting AI to produce more; it is building the harness that makes fleet output trustworthy. Generation is cheap; verification is the bottleneck.
The harness has four levers, and each is a craft, not a purchase. Levels 4 to 6 of the ladder above are, in effect, the harness skills: the Workflow Designer builds fixed checks, the Agent Boss wires hooks and goal contracts, the Fleet Orchestrator runs verification loops across teams of agents.
Lever 1Curate the context
Context engineeringControl what the model sees: the right documents, in tagged sections, with examples and constraints. Nothing extraneous, nothing stale.
The craft- Thin always-on foundation (role, task, house standards) plus just-in-time retrieval of the specific account, filing, or guideline
- Worked examples beat instructions: paste one gold-standard output and say "match this"
- Context hygiene: a stale appetite guide or superseded form in context is a hazard, not a resource
- Underwriting assistant sees: the submission, the current appetite guide, two exemplar briefs, and the referral rules. Nothing else
- Most agent failures are context failures, not model failures. When output disappoints, audit what the model saw before you blame the model
Lever 2Decompose the work
Pipelines before agentsSplit the job into stages with checkpoints: extract, validate, summarize, draft. Each stage is reviewable, and errors surface where they happen.
The craft- Fixed pipelines for known work; free agents only where the path genuinely varies
- Deterministic steps in code (validation, arithmetic, appetite scoring), model steps for language and judgment
- Each stage writes a structured artifact a human or the next stage can check
- Loss-run analysis: extraction agent pulls fields, code checks totals reconcile, model writes the trend narrative. A wrong number never reaches the narrative stage
- Resist the giant do-everything prompt; it fails silently at one of five steps and you cannot tell which
Lever 3Constrain the tools
Permissions in codeGive the agent exactly the tools the task requires, with limits enforced by the system, not by the prompt. Instructions can be talked around; code cannot.
The craft- Least privilege: read-only by default; write and send actions require a human gate
- Limits in code: binding authority, reserve caps, and PII rules enforced as hooks, not as warnings in the system prompt
- Standard connectors (MCP-style) to the rating engine, policy admin, and document store, each scoped
- An underwriting agent may draft a quote up to a set premium threshold; above it, a hook blocks and routes to a named underwriter
- Claims agent drafts reserve recommendations; the reserve itself moves only on adjuster approval, logged
Lever 4Verify the output
Evals and gatesGeneration is cheap; verification is the bottleneck. Score output against ground truth before it flows anywhere, and put a human at consequential steps.
The craft- Golden dataset: 20 to 50 real cases with known-correct outputs; every change to the harness runs against it
- Field-level re-checks against source documents, done by code or a grader agent
- Drift review: re-run the eval set on model upgrades and on a schedule; behavior changes silently
- Extraction eval: 30 historical submissions keyed by a technician; harness must match field-level accuracy before any submission skips human review
- This is what "human-in-the-loop" means in 2026: a person reviews what the harness surfaced, not everything the fleet produced. Slop is cheap; trust is engineered
Three worked harness patterns
Concrete assemblies of the four levers for the three highest-value workflows. Each lists the pipeline, the checks, what the human sees, and the metric that tells you it works.
| A · Submission intake (underwriting) | B · Claims file synthesis (claims) | C · Filing research assistant (actuarial) | |
|---|---|---|---|
| Pipeline | Email + attachments → extraction agent → enrichment (third-party data) → appetite scoring in code → underwriter brief | Adjuster notes, correspondence, medicals → document index → chronological summary with citations → severity signal | Question → retrieval over state filings and rating manuals → comparison table → draft support-memo sections |
| Hooks and checks | Every extracted field re-checked against the source document; totals reconcile in code; out-of-appetite flagged, never quoted | Every claim number and date must cite a page; PII redaction hook on anything outbound; severity signal labeled as advisory | Citations must resolve to a real filing and page; numbers recomputed in code; draft memo labeled "draft, unreviewed" |
| What the human sees | A verified queue: brief, score, exceptions. Not raw agent chatter | The summary plus the flagged severity rationale; one click to any cited page | The comparison table and the draft; the actuary edits, the assistant never files |
| Metric to watch | Field-level accuracy on the golden set; minutes from inbox to briefed; hit ratio over time | Adjuster time per file; citation accuracy; override rate on severity signals | Hours per filing research cycle; citation resolution rate; review comments per draft |
The starter checklist: first harness in 30 days
- Pick one bounded workflow with a document in, a decision out, and a measurable current cost (submission intake and claims summarization are the proven starters).
- Write the context spec: exactly what the agent should see, from where, refreshed how often. Name the owner of each source.
- Assemble the golden dataset: 20 to 50 real historical cases with known-correct outputs, keyed by a person you trust.
- Decompose into stages and decide which steps are code and which are model.
- List the hooks: the checks that run every time, including the ones that block (PII, authority limits, reconciliation).
- Define the human gate: what a person must approve, and what "approve" means they checked.
- Log everything: inputs, outputs, overrides, hook firings. This is your exam evidence and your debugging trail.
- Set the eval gate: the accuracy bar the harness must hold on the golden set before any case skips full human review.
- Run the pilot in shadow mode (agent works, humans still do the job) for two to four weeks; compare.
- Decide kill or scale on the metrics, then schedule the drift review. A harness is maintained like a model, because it is one.
Failure modes to design against
| Failure mode | What it looks like | The countermeasure |
|---|---|---|
| Stale context | Agent applies last quarter's appetite or a superseded form | Context hygiene: owners, refresh dates, retrieval over static dumps |
| Automation bias | Reviewers approve fluent output without checking | Surface the evidence (citations, reconciliation), not just the answer; spot-audit approvals |
| Prompt-only guardrails | "Do not quote above X" in the system prompt, working until it doesn't | Limits enforced in code hooks; prompts inform, hooks enforce |
| Silent drift | A model upgrade changes behavior; nobody notices for a month | Scheduled golden-set runs; re-run on every provider version change |
| Unlogged overrides | Humans correct the agent constantly; no record, no learning | Override logging with a one-tap reason; feed corrections into the golden set |
This is what "human-in-the-loop" actually means in 2026: not a person reading everything, but a person reviewing what the harness surfaced. And it is where the NAIC Model Bulletin's "testing and human oversight" expectations physically live: in hooks, evals, and permission gates, not in the policy PDF. Slop is cheap; trust is engineered.
Where the experts disagree
- Prompt engineering: dead or absorbed? The consensus is absorbed. Phrasing skill is now one layer inside context engineering, so the ladder keeps it as a rung, not a destination.
- Agent teams: help or harm? Orchestrator-worker teams report large quality gains on parallel, high-value work at roughly 15x the token cost, and fail on tightly coupled work. Task shape decides.
- What are subagents for? Context isolation, not personas. Role-playing subagents measurably underperform; quarantined context windows are the entire point.
- Retrieve up front or just in time? Long context wins on quality, RAG wins on cost. The working default is hybrid: a thin always-on foundation plus just-in-time fetch.
- Is a ladder even the right shape? Fluency frameworks like Anthropic's 4Ds are competency inventories, not rungs. Treat this as a map of what to learn next, not a ranking of people.
Source links
- Anthropic engineering series: Building effective agents · Context engineering · Multi-agent research system · Prompt engineering docs · AI Fluency framework
- Practice research: Mollick/HBS, "Centaurs and Cyborgs on the Jagged Frontier" · Rethink Priorities usage surveys · Microsoft Work Trend Index (Frontier Firm)
- Architecture and harness: LangChain on context engineering · Cognition, "Don't Build Multi-Agents" · HumanLayer, 12-factor agents · DAIR.AI Prompt Engineering Guide · Claude Code docs (hooks and permissions in practice)
- Verification and governance: METR task horizons · NAIC Model Bulletin (testing, human oversight) · NIST AI RMF
The "context engineering moment" references Karpathy's and Lütke's June 2025 posts; the harness sections draw on the same engineering sources. Market and regulatory claims live on the AI roadmap, which carries numbered references, with verbatim quotes on the evidence page. Foundations: the field guide's hands-on level.