Technical deep dive

For leaders reviewing a technical proposal: focus on the pricing workflow, the validation approach in §6, and the accountability requirements in §7. Ask for a reproducible test result and a named reviewer before approving production use.

1 · What an LLM actually is, for a statistical audience

A large language model generates text as a sequence of tokens: words, word fragments, punctuation, or other encoded units. Many use transformer neural networks trained on text prediction, with additional training to follow instructions. The SOA's actuarial primer provides an introduction to the underlying technology.

Three distinctions matter when designing and validating an application:

1. Generated output can vary. Repeated requests can produce different text or conclusions. Sampling settings, where available, influence that variation; they do not establish reproducibility across model or service changes. Preserve the artifacts used in an actuarial work product (§7).
2. Fluent output can contain false claims. A hallucination is an unsupported or incorrect generated statement. Supplying documents does not eliminate that risk: verify citations, extracted values, omissions, and the interpretation of the source.
3. Training knowledge and supplied context differ. Training has a cutoff, while retrieval, connected tools, and uploaded documents can provide later or company-specific information. Check what the application actually retrieved, which version it used, and whether the user was entitled to access it.

2 · Core concepts, precisely

• Context window: the amount of information a model can process in a request. Limits and input/output accounting vary by model. Check the selected model's documentation and test whether the application finds relevant details in long documents.
• System prompt: standing instructions that frame a request, such as the role, constraints, and output format. Put enforceable business rules in application controls as well; instructions alone are insufficient for permissions or authority limits.
• Retrieval-augmented generation (RAG): an application retrieves relevant material and supplies it to the model before it answers. Retrieval may combine keyword search and embeddings, numerical representations used for similarity search. Evaluate document permissions, retrieval coverage, source versions, and answer faithfulness separately.
• Fine-tuning: additional training on task examples to change a model's behavior. Consider it after measuring a specific gap that prompting or retrieval has not resolved. Plan for training-data review, evaluation, and maintenance; use controlled retrieval for documents that must remain current.
• Structured output: supported APIs can constrain responses to a schema, such as {claim_date, cause, incurred, paid, open_flag}. Schema compliance does not establish that the values are correct. Handle refusals and incomplete responses, validate business rules, and reconcile totals against the source (OpenAI documentation).
• Tool use / agents: an application can let the model request tools such as search, database queries, and code execution. For calculations, require inspectable code and test the data mapping, units, formulas, and results. Generated code can be wrong even when it executes successfully.
• Reasoning models: models that can spend additional computation before answering. Test the quality, latency, and cost tradeoff for each task. Request evidence and a concise explanation that a reviewer can check; internal reasoning is not an audit trail (OpenAI guidance).

3 · Good at / bad at, calibrated for actuarial work

Candidates for a controlled pilot

• Code generation and review, including R, Python, SQL, VBA translation, and explanations of inherited code; compare review effort and error rates with the current process
• Summarizing and comparing documents: reinsurance treaties, policy forms, regulatory correspondence, prior analyses
• Extracting structured data from unstructured sources: loss runs in inconsistent broker formats, exposure schedules, contracts
• Drafting: report language, documentation, memo sections, filing narrative (ASOP 41 review still applies; see §7)
• Regulation orientation: "explain this state's rate filing requirements" as a starting point to verify, never a citation source

Tasks requiring tighter controls

• Mental arithmetic and multi-step numeric manipulation without code execution
• Factual recall of specific numbers, citations, ASOP text, or policy language from memory; always retrieve the source document instead
• Model-generated answers where exact reproducibility is required; evaluations measure variation but do not remove it
• Judgment calls that are actually yours: assumption setting, method selection, reserve picks. Use the model to identify objections and alternative interpretations; retain named responsibility for the selection

4 · AI agents around pricing and costing model development

Use agents to support specific parts of the model-development lifecycle: gathering evidence, drafting code, running approved comparisons, preparing documentation, and monitoring. The predictive model can remain a conventional GLM or GBM. Measure whether assistance improves the work after validation and review effort are included.

Keep costing separate from transaction pricing. The costing layer estimates expected loss and risk costs. The pricing/rater layer then adds expenses, commission, reinsurance, capital and profit provisions, rules, and permitted underwriting judgment. An agent may trace and reconcile those components; it should not silently mix them or choose assumptions.

The useful multi-agent pattern

Freeze the data snapshot, holdout, metric definitions, and model charter first. Then fan out independent workers by model formulation or task. For example, separate workers can build a baseline GLM, test a GBM residual, research peer filings, and run a predefined validation suite. A separate agent can help check outputs, but independent validation also requires a qualified reviewer who can challenge the shared specification and data.

Public examples illustrate the range of proposed uses, rather than proving the return from this design. Deloitte describes its 10X Analyst platform as reverse-engineering rate manuals and reducing competitor analysis from weeks to minutes (Deloitte). In July 2025, Aviva and hyperexponential announced a six-month beta collaboration to test an Actuarial Agent for pricing insights and model-code improvements in Global, Corporate & Specialty business (Aviva/hx). Hyperexponential's product demonstration presents agent assistance with pricing-model development (hx product demonstration; vendor-reported).

Approved, versioned data snapshotDefine the time split and evaluation rules before fitting.

Earlier periods

Development data

Fit candidate↓Compare on validation data↓Revise the approved design

↻ Repeat inside this development set

Freeze the candidateLock code, variables, parameters, and test criteria.

Later, untouched period

Final holdout

Reserved for the final testKeep outcomes out of fitting, prompt revisions, and model selection.

Release for evaluation only after the candidate is frozen.

↓Frozen candidate + untouched holdoutRun the final test, then obtain independent review.

Meets the agreed criteriaDocument results and limitations; seek release approval.

Does not meet the criteriaInvestigate before release. If you tune using these results, this set is no longer an unseen final test.

Non-negotiable controls

• Agents call code; they do not perform actuarial arithmetic in prose. Every result must be reproducible from versioned code and data.
• One approved model charter. Target, exposure, offsets, weights, time split, loss treatment, constraints, and success criteria are fixed before fan-out.
• Protect the final holdout. Use a common validation design and loss-dollar basis for comparisons. Keep final out-of-time test data out of iterative model selection; repeated tuning against that set weakens its independence.
• No autonomous assumption or rate changes. Trend, credibility, expense, reinsurance, profit, and judgment selections require named actuarial approval.
• Independent validation. A separate reviewer challenges data lineage, leakage, variable justification, stability, fairness, and implementation.
• Production is gated. The agent may draft rater code and tests; deployment requires version control, approvals, parallel-run reconciliation, and rollback.

For sparse specialty data, prioritize repeatable diagnostics, uncertainty documentation, and checks against the current plan. The SOA's 2026 request for research proposals on agentic workflows includes rate development, assumption development and benchmarking, model governance, documentation maintenance, and regulatory reporting, with attention to reliability, explainability, bias, and human oversight. This is a research agenda, not a report of achieved results (SOA Research Institute).

5 · A hands-on learning path

The schedule below is an illustrative six-week introduction for a reader already comfortable with code. Progress depends on access, experience, and time available; completing the exercises does not establish production readiness.

Stage 1: Fluent prompting (week 1). Practice Hands-on use on fictional or approved examples. Include missing information and conflicting sources, then check whether the result handles them correctly.

Stage 2: API basics (weeks 2–3). Call a model from Python/R rather than a chat window: system prompts, temperature, structured JSON output against a schema. First project: extract a standardized claims table from three differently-formatted loss-run PDFs, with schema-constrained output, and reconcile totals against the source in code.

Stage 3: Evaluation discipline (weeks 3–4). As a learning exercise, build a small evaluation set with known answers, a scoring script, and tracked errors. A few dozen examples can teach the process; production sample size and coverage must reflect materiality, rare failures, and the decisions the system supports (§6).

Stage 4: RAG and agents (weeks 5–6). Build a small RAG prototype over fictional, public, or explicitly approved documents you know well, so you can judge retrieval quality personally. Then experiment with tool use: an assistant that answers loss-development questions by writing and running code against a triangle you supply.

Stage 5: ongoing. SOA/CAS materials (see Resources), and pick one real workflow to instrument end-to-end with a colleague as reviewer.

6 · Validation: treat LLM pipelines like models, because they are

For any LLM use beyond personal drafting assistance, apply model-risk thinking:

1. Define the task and failure cost. Identify required fields, source evidence, missing-value behavior, and acceptance thresholds. Evaluate material monetary and coverage fields separately; a high average accuracy can conceal serious errors in a small number of important fields.
2. Build a labeled eval set from real data (with permissions). Measure before deploying, and re-measure on every model version change, prompt change, or vendor update. Pin versions where supported and re-run evaluations before accepting a change.
3. Measure the right failure modes: field-level accuracy for extraction; faithfulness (does every claim in the summary trace to the source?) for summarization; retrieval hit rate and answer groundedness for RAG.
4. Calibrate model-based grading. If a second model helps grade outputs, compare its decisions with qualified human review, including disagreement and failure cases. Use deterministic checks where the answer is objectively testable.
5. Keep humans in the loop proportionate to consequence, and monitor for automation bias: spot-audit accepted outputs, not just rejected ones.
6. Retain review evidence: source references, prompts, model and workflow versions, outputs, tests, and reviewer decisions, subject to access and retention rules. Agree the required record with compliance and model-risk owners (see Governance).

7 · Professional standards and accountability

The following U.S. standards provide relevant guidance. Apply their scope, materiality provisions, and the actuary's actual responsibilities to the assignment; the practical suggestions here are not a substitute for the standards.

• ASOP 56: Modeling. Addresses model purpose, understanding, testing, output validation, governance, and reliance on others. Its applicability includes a materiality judgment and the scope of the actuary's responsibilities. Assess how an LLM component affects the actuarial model and work product.
• ASOP 23: Data Quality. Addresses selection, review, use, and disclosure of data limitations and reliance. For an extraction pipeline, source reconciliation and error analysis can support that assessment; the standard does not prescribe a particular LLM evaluation metric.
• ASOP 41: Actuarial Communications. Addresses the responsible actuary, clarity, uncertainty, reliance, and responsibility for assumptions and methods. Review AI-assisted text and document relevant reliance and limitations in the context of the assignment.
• Practical documentation: retain the source inputs, model and workflow versions, reviewed outputs, calculations, and approvals supporting the work product. For computations, retain executable code, dependencies, and the data snapshot needed to reproduce the result.

8 · Data handling rules that are never optional

• Use only company-sanctioned tools and APIs with terms approved for the data involved. Confirm training use, retention, access, and permitted integrations for the specific product. Keep policyholder, claimant, and confidential company data out of unapproved accounts.
• Prefer de-identified or minimal-necessary data in prompts even on sanctioned tools.
• Treat model outputs derived from confidential inputs as confidential.
• Documents from outside the company (broker submissions, claimant correspondence) can carry prompt-injection payloads: hidden instructions to the model. Pipelines that act on external documents need injection-aware design and constrained tool permissions.

Full rules are in the policy template under Governance; these are the ones analysts hit daily. The next page describes the agents that build this site, a smaller publishing example of defined responsibilities, automated checks, and editorial review.