Building With Intelligence

The simple explanation

A thermostat applies a rule. A spam classifier learns a boundary from labeled email. A language model learns statistical structure that lets it generate text. An agent places a model inside a loop that observes, chooses, and sometimes acts. These are progressively different architectures, not ascending ranks on a single “intelligence” meter.

Prediction, generation, and apparent intelligence

A learned model estimates an output from inputs. For a classifier, that output may be a probability over labels. For a language model, it is a probability distribution over the next token, repeatedly sampled to form a sequence. Rich internal representations and enormous training corpora make next-token prediction surprisingly capable, but the training objective does not by itself include truthfulness, authorization, or concern for consequences.

This explains a central asymmetry. A system may be capable of writing an excellent discharge summary and still be unreliable at never inventing a medication. Capability asks, “Can it sometimes do the task?” Reliability asks, “Under the specified conditions, how often does it meet all requirements, including rare but costly ones?”

A short conceptual history

Early symbolic AI emphasized explicit rules and search. Statistical machine learning shifted attention toward learning patterns from data. Deep learning made learned representations central. The transformer made large-scale sequence modeling more parallel and effective. Instruction tuning and feedback made general models easier to direct. Tool interfaces and orchestration then turned text generators into components that could query and act. Each shift changed what was convenient; none repealed the need for software engineering or evidence.

Anthropomorphism is a lossy shorthand

Words such as “knows,” “believes,” and “decides” can compress a discussion, but they import assumptions. A model can output “I remember” when no persistent record exists. An agent’s “goal” may be a string in a prompt. Use operational language when consequences matter: the system retrieved a record, the model assigned a score, the policy allowed a tool call.

Decision framework

• Use rules when inputs and acceptable outputs can be enumerated.
• Use learned prediction when patterns matter and an evaluation set represents deployment.
• Use generation when several outputs may be acceptable and review or constraints can catch failures.
• Use an agent only when choosing among actions under changing observations creates enough value to justify new failure modes.

Summary, key terms, and exercises

Summary: AI is learned or rule-based computation embedded in a larger system. Intelligence-like output is not evidence of truth, agency, or reliability. Key terms: automation, model, inference, generative model, agent, capability, reliability.

1. Classify three features in an app you use as rules, predictions, generation, or action loops.
2. Write one capability claim and the stronger reliability evidence it would require.

Variables, functions, and vectors

A variable names a quantity. A function maps inputs to outputs: f(x)=y. A vector is an ordered list of numbers, such as a patient-free synthetic appointment represented by [days since booking, prior cancellations, travel distance]. Dimensions are positions in that representation. In modern models, many dimensions do not have simple human labels; together they encode a useful geometry.

Conditional probability and Bayes

P(A|B) means the probability of A given B. It is not generally equal to P(B|A). Bayes’ rule reverses the condition:

Suppose 1 in 100 messages is truly urgent. A detector catches 90% of urgent messages and falsely flags 5% of ordinary ones. In 10,000 messages, about 100 are urgent: 90 are caught. Of 9,900 ordinary messages, 495 are falsely flagged. Among 585 alerts, only 90—about 15%—are truly urgent. High sensitivity did not create high confidence because the base rate was low.

Distributions, expectation, and variance

A distribution assigns probabilities across possible values. Its expectation is the probability-weighted average. Variance describes spread around that average. Two systems can have the same mean latency or accuracy while one has a dangerous tail. Production decisions need percentiles, subgroups, and failure categories, not only averages.

Calibration is not accuracy

A calibrated system that labels many cases 70% confident should be correct on roughly 70% of those cases. A perfectly accurate classifier can be poorly calibrated in its scores; a calibrated forecaster may still be too inaccurate to use. Calibration must be checked on representative data and can deteriorate after distribution shift.

Precision asks: of predicted positives, how many were right? Recall asks: of actual positives, how many did we find? A fraud investigation queue may prioritize precision when review is expensive. A safety screen may prioritize recall, then use a second stage to manage false alarms. The correct threshold comes from consequences, not convention.

Production decision

Before choosing a metric, write the cost of each cell in the confusion matrix. Add abstention as an explicit output. Track calibration and error rates over time. If the deployment distribution changes, old confidence estimates are historical evidence, not a warranty.

Summary: probabilities are conditional, scores need calibration, errors have asymmetric costs, and means hide tails. Key terms: prior, likelihood, posterior, distribution, expectation, variance, calibration, precision, recall, base rate.

1. Recompute the alert example if the base rate rises to 10%.
2. For a support escalation classifier, assign a cost to each confusion-matrix cell and choose which metric should lead.

Four learning arrangements

• Supervised learning uses input-output examples, such as emails labeled spam or not spam.
• Unsupervised learning seeks structure without task labels, such as clusters.
• Self-supervised learning creates targets from the data itself. Predicting masked or next tokens turns abundant text into training examples.
• Reinforcement learning learns behavior from reward signals across actions and consequences. In language-model post-training, human or model preferences can shape a reward or directly optimize preferences.

Training is optimization

A model contains adjustable parameters. A loss function measures disagreement between predictions and training targets. Optimization repeatedly changes parameters to reduce loss. Features are input measurements; learned representations are internal features discovered during training. An embedding is a vector representation whose geometry is useful for some objective.

Generalization and overfitting

The training set fits parameters. A validation set guides choices such as architecture or thresholds. A test set estimates performance after choices are frozen. Repeatedly tuning on the test set turns it into another validation set. Overfitting occurs when a system captures quirks that help on observed examples but fail on new ones.

Distribution shift and feedback

Independent and identically distributed examples are a useful mathematical simplification, not a promise about production. Users change, policies change, attackers adapt, and the model’s own outputs alter future data. If a triage model routes only high-scoring cases to experts, labels for low-scoring cases disappear; retraining then learns from its prior selection policy. This is a feedback loop.

Summary: learning minimizes a chosen loss over examples; generalization is empirical and conditional. Leakage, shift, and feedback can invalidate apparently strong results. Key terms: parameter, feature, representation, embedding, loss, optimization, generalization, overfitting, leakage, distribution shift.

1. Identify one feature in a product dataset that might leak future information.
2. Describe how deployment could change the labels later collected for retraining.

From characters to tokens

Tokenization converts text into units from a learned vocabulary: common words, fragments, punctuation, or bytes. The model maps each token to an embedding and combines it with position information. Token counts—not word counts—usually determine context and API cost.

Attention and the transformer

In self-attention, each position produces a query, key, and value. Similarity between a query and keys determines how much their values influence the new representation. Multiple heads learn different interaction patterns. Feed-forward layers transform each position; residual connections and normalization stabilize deep stacks. “Attention” is a numerical routing mechanism, not human attention or an explanation of intent.

Pretraining and post-training

Pretraining exposes a model to broad corpora and optimizes token prediction. Instruction tuning teaches response patterns from demonstrations. Preference optimization or reinforcement learning can make outputs better match human judgments. These stages shape behavior but do not create a guaranteed internal fact table, a stable identity, or access to events after the training cutoff.

Context and sampling

The context window is the bounded sequence available during a request. Larger windows reduce some truncation pressure but do not guarantee that every included detail will influence the answer correctly. Temperature rescales token logits before sampling: higher values generally flatten the distribution and increase variation. It is not a “creativity” or “truth” dial.

Why fluency is not truth

The objective rewards plausible continuation. It does not inherently check a claim against reality. Hallucination is therefore not a detachable typo module; it is a system-level mismatch between generation and the requirement for supported claims. Grounding, abstention, verification, and evaluation reduce risk, but no prompt turns generation into a proof engine.

Summary: transformers build contextual token representations and generate one token at a time. Context is temporary input; weights are learned tendencies. Key terms: token, embedding, attention, transformer, pretraining, post-training, context window, logit, temperature.

1. Explain why a 200-page document fitting in context does not imply every fact is equally accessible.
2. Name two product requirements that demand runtime evidence rather than training recall.

Specify the contract

A useful prompt states the outcome, audience, available evidence, constraints, and output shape. Few-shot examples show behavior that prose may describe ambiguously. Structured output constrains syntax. Decomposition separates a difficult task into inspectable stages. None of these mechanisms makes the output deterministic across model versions or inputs.

Many platforms distinguish instruction sources—system, developer, user, and tool—but names and precedence are runtime-specific. The durable idea is provenance: the application must know which text came from its policy, its operator, an end user, a retrieved page, or a tool result.

Context engineering

Context engineering chooses what enters the request, in what order, with which labels and limits. It includes retrieval, tool descriptions, state summaries, examples, policies, and prior turns. More context can add distraction, contradictory instructions, private data, latency, and cost. Selection is part of system behavior.

Reasoning traces and critique

Prompted intermediate reasoning can improve some multi-step tasks, but a fluent trace is not a faithful audit log of hidden computation. Ask for concise justifications, cited evidence, calculations, or verifiable artifacts when those help review. Self-critique can catch errors, but the same model can repeat the same misconception; independent checks are stronger when available.

Prompt injection

When a model processes untrusted content, that content can contain text that competes with the application’s instructions. Retrieved documents, web pages, emails, and tool output are data from the application’s perspective, even though the model sees more tokens. Delimiters and warnings may help behavior but do not form a security boundary.

Decision framework

• Use prompting for task description, examples, tone, and output contracts.
• Use code for permissions, invariants, calculations that must be exact, and state transitions.
• Use retrieval for current or private evidence.
• Use evaluation to learn whether a prompt works; do not judge from a few attractive examples.

Summary: prompting is powerful interface design, not a production security or correctness architecture. Key terms: zero-shot, few-shot, decomposition, structured output, context engineering, prompt injection.

1. Rewrite a vague prompt as an outcome-evidence-constraints-schema contract.
2. Mark every input source as trusted policy, operator request, untrusted content, or tool output.

Search has different failure modes

Lexical retrieval rewards shared terms and is strong for exact names, codes, and rare phrases. Dense retrieval embeds queries and passages, then compares vectors; it can match paraphrases but may blur distinctions. Hybrid systems combine candidates. A reranker spends more computation scoring query-passage pairs. Each stage trades recall, precision, latency, and inspectability.

RAG as a pipeline

Retrieval-augmented generation usually means: transform a question, retrieve candidate chunks, filter or rerank them, assemble context, generate an answer, and attach citations. Failures can occur at every arrow. The answer model cannot use a decisive source that retrieval omitted. A high retrieval score says “similar under this mechanism,” not “true,” “authorized,” or “complete.”

Chunking and metadata

Chunks must be small enough to retrieve precisely but large enough to preserve meaning. Fixed windows are simple; structure-aware chunks preserve headings, tables, or code units. Useful metadata includes source identity, version, valid time, owner, jurisdiction, access policy, and canonical URL. Metadata filters can improve both security and relevance, but ingestion-time permissions must be rechecked at query time.

Citations and provenance

A citation should bind a claim to a passage the reader can inspect. Validate that cited sources were actually in context and that each citation supports the adjacent claim. Provenance records where content came from and how it was transformed. It helps auditing; it does not guarantee that the source itself is correct.

When to query the source live

Use a derived index to nominate documents. Query the source-of-truth system live when values change frequently, consequences are high, completeness matters, or access can change. A scheduling assistant should query current availability; it should not treat a remembered slot or yesterday’s index as bookable inventory.

Summary: RAG is a staged information-retrieval system. Retrieval improves access to evidence but does not certify answers. Key terms: lexical retrieval, dense retrieval, embedding, reranker, RAG, chunk, metadata, provenance, authority.

1. Design a retrieval test with separate relevance, authority, and freshness labels.
2. List three fields that must be read live rather than remembered in your domain.

The tool contract

A tool exposes a name, purpose, input schema, and execution handler. The model emits a structured proposal. The application validates it, checks policy, executes through ordinary code, and returns a bounded result. Structured output narrows syntax; it does not make values safe or semantically correct.

{
  "tool": "hold_appointment",
  "arguments": {
    "slot_id": "synthetic-slot-42",
    "duration_minutes": 15
  }
}

The host must verify that the slot exists, the caller may hold it, the duration is allowed, and the operation is not a duplicate. A schema cannot decide those facts.

Read and write are different trust classes

Reading a public knowledge base is usually reversible. Sending an email, charging a card, changing a record, or deleting data creates external effects. Separate read tools from write tools. Require previews or approvals for consequential writes. Keep credentials inside the execution service; do not place them in prompts.

Idempotency, retries, and partial failure

Networks fail ambiguously. A timeout does not reveal whether the remote system committed the request. Retrying a non-idempotent “charge” can double-charge. Supply an idempotency key bound to the business operation and make repeated execution return the same result. Define timeouts, bounded retries with backoff, and a reconciliation path for unknown outcomes.

Design the result, not only the call

Tool results need stable status codes such as succeeded, rejected, retryable, or outcome_unknown. Return only necessary data. Treat tool output as untrusted if it contains external content. Preserve audit fields: actor, authority, tool version, validated arguments, idempotency key, result, timestamps, and approval reference.

Tool checklist

• Can this be a narrower operation?
• Is it read-only, reversible, or externally consequential?
• Who authorizes it, and can that authority be delegated?
• What makes retries safe?
• What happens after timeout or partial success?
• What evidence proves the observed outcome?

Summary: tool calling is a proposal-execution protocol. Schemas validate shape; deterministic code validates authority and invariants. Key terms: function calling, schema, least privilege, idempotency, timeout, backoff, partial failure, reconciliation.

1. Rewrite a broad “manage calendar” tool as three least-privilege tools.
2. Define the idempotency key and unknown-outcome path for one write operation.

The seven-part anatomy

1. Model: proposes interpretations or actions.
2. Instructions: define task and behavioral constraints.
3. State: records the current process.
4. Tools: expose observations or effects.
5. Environment: the world changed or inspected by tools.
6. Loop: observe → choose → act → observe.
7. Stopping condition: success, failure, budget, deadline, or human decision.

A chatbot may generate one response without acting. An ordinary program selects branches developers enumerated. A workflow follows explicit states and transitions, perhaps using models inside steps. In an agent, model output selects at least some next actions dynamically. Hybrids are common and often preferable.

Degrees of autonomy

Risk rises with tool power, iteration count, hidden state, environmental uncertainty, and irreversibility. A one-step payment tool can be riskier than a 20-step read-only research loop. Count consequences, not just steps.

When the label becomes harmful

“Agent” can conceal missing specifications. Ask instead: which decisions are model-selected? Which state transitions are deterministic? Which tools can write? How is success verified? Who can stop it? If a team cannot answer, the system is not yet designed.

Summary: agents are model-guided action loops embedded in host software. Autonomy is multidimensional. Key terms: observation, action, environment, policy, loop, stopping condition, autonomy.

1. Draw the seven components for a familiar “agent” product.
2. Identify which autonomy dimension most affects its risk.

The decision tree

Comparison table

When should I not use an agent?

Do not use an agent when a query, formula, rules engine, search endpoint, or explicit workflow can meet the requirement. Do not use one when success cannot be observed, rollback is unavailable, tool permissions cannot be narrowed, or no evaluation set represents the task. A persuasive demo is not a substitute for an operating envelope.

Workflow or agent?

Prefer a workflow when states and transitions are known, compliance requires a fixed process, or failure recovery matters more than path flexibility. Put models inside bounded steps for extraction, classification, drafting, or ranking. Use an agent when the useful next step depends on open-ended observations and the allowed action set remains safe.

Memory, live data, ontology, or SQL?

Query live systems for current inventory, permissions, balances, schedules, or configuration. Use memory for preferences, past attempts, strategies, and historical context—never as an automatic replacement for the source. Use SQL when entities, fields, and joins suffice. Add an ontology when several systems use inconsistent concepts and shared meaning is itself a requirement. Add a graph when relationship traversal, provenance, or evolving multi-hop connections are central and measured query value exceeds maintenance cost.

Subagent or one strong agent?

Delegate when tasks are separable, can run concurrently, need different tools or contexts, and have a deterministic merge or review step. Stay single-agent when every worker needs the same large context, outputs depend tightly on one another, or failures cannot be attributed. Measure quality, latency, cost, and correction burden against a one-agent baseline.

Where should a human approve?

Place approval before irreversible or externally consequential state transitions, not after prose generation. Show the reviewer the exact proposed action, destination, changed fields, evidence, uncertainty, reversibility, and authorization scope. Approval should expire if the action changes.

Cost and reliability

Every added call introduces latency, cost, and another conditional failure. If each of five dependent stages succeeds 95% of the time and failures are independent, end-to-end success is about 77%. Independence rarely holds, but the arithmetic teaches the direction: chains multiply opportunities to fail. Parallel branches can improve coverage yet raise aggregation and conflict costs.

Summary: architecture is a ladder of justified complexity. Start deterministic, add bounded inference, then state, tools, memory, or multiple agents only for measured needs. Key terms: workflow, state machine, planner–executor, ontology, knowledge graph, human-in-the-loop.

1. Apply the tree to a travel planner and identify the first irreversible action.
2. Replace one imagined agent with a workflow and list what capability is lost.

Four loop shapes

Reason-and-act alternates model decisions with tool observations. Plan-and-execute drafts a sequence, then performs or revises it. Reflection converts feedback into another attempt or retained lesson. Tree search explores multiple candidate reasoning paths and evaluates intermediate states. These are control patterns; paper results on selected tasks do not make one universally superior.

Planning versus replanning

A plan can reduce drift, expose missing prerequisites, and enable approval. It can also anchor the system to assumptions that the first observation invalidates. Attach preconditions to steps. Replan after material environmental changes, not after every token. Preserve completed effects so replanning does not repeat them.

Stop conditions and budgets

Define maximum model calls, tool calls, wall time, tokens, money, repeated states, consecutive failures, and side-effect count. Also define success evidence. “The model says done” is not a stopping proof. For code, success might require specified tests at a bound commit. For research, it might require coverage of named questions and inspected primary sources.

Why long horizons fail

Small errors compound; context fills with stale detail; tools return ambiguous outcomes; goals drift; and evaluation becomes sparse. Checkpoint durable state outside the context window. Verify local milestones. Use typed failure states. Escalate when progress stops rather than rewarding endless activity.

Choosing a loop

• Use a fixed workflow for known steps.
• Use ReAct for bounded information gathering where each observation changes the next query.
• Use an explicit plan when dependencies or approval benefit from preview.
• Use branching search only when intermediate states can be scored and the value justifies multiplied cost.

Summary: agent loops manage uncertain paths; budgets and evidence gates keep them finite. Key terms: ReAct, plan-and-execute, reflection, tree search, checkpoint, stop condition.

1. Write success, failure, and budget stop conditions for a research agent.
2. Identify one plan step whose outcome should force replanning.

Why divide the work?

Subagents can isolate contexts, use different tools, run independent tasks concurrently, or create explicit review roles. A supervisor-worker pattern assigns and merges work. A blackboard pattern lets workers contribute to shared structured state. Debate asks models to challenge positions. Each pattern needs a reason that a single agent with the same total budget cannot provide.

The independence trap

Three agents using the same model, prompt assumptions, retrieved sources, and evaluator are not independent experts. Majority vote can amplify a shared misconception. Role names such as “security specialist” do not confer credentials. Diversity must arise from evidence, tools, methods, data, or models—and still requires measurement.

Shared state and communication

Use typed shared state rather than free-form transcripts when possible. Record task ID, owner, input evidence, output artifact, status, assumptions, and dependencies. Limit messages to what the recipient needs. Concurrent workers require conflict handling; “last writer wins” can erase valid work.

Failure modes

• Delegation ambiguity: workers solve different interpretations.
• Error propagation: a supervisor trusts an unsupported worker summary.
• Coordination tax: tokens and latency exceed useful work.
• Deadlock or churn: agents wait, reassign, or critique indefinitely.
• Evaluation bias: the supervisor and judge share the same blind spot.

Proving that multiple agents helped

Compare against one strong agent under both equal-compute and native-default budgets. Hold model, tools, task set, and scoring constant. Report task success, severe failures, latency, cost, and human correction burden. Run ablations: remove the reviewer, remove parallelism, or replace worker messages with direct evidence.

Summary: multi-agent design is distributed-systems design plus model uncertainty. Use it for separable work and measured specialization, not theatrical teams. Key terms: supervisor-worker, delegation, blackboard, debate, coordination cost, ablation.

1. Take a tightly coupled task and explain why delegation would harm it.
2. Design an equal-budget single-versus-multi-agent evaluation.

Harbor Clinic’s conversation says, “Tuesday afternoon works.” The workflow records awaiting_confirmation. The scheduling database says slot S42 is currently free. Memory says the synthetic user usually prefers afternoons. Only the scheduling system can confirm inventory; preference memory may rank choices but cannot reserve one.

Durable state belongs outside the prompt

A context window can be truncated, summarized incorrectly, or rebuilt differently. Store workflow state in a transactional system with explicit versions and allowed transitions. Construct model context as a view over that durable state. The model proposes; the state machine validates and commits.

Ownership and time

For each field, document owner, update path, valid time, retention, and conflict rule. Duplicated state drifts. Caches need invalidation. Event streams need projections and replay semantics. Summaries should carry the version of the records they summarize.

Decision framework

• If the fact changes business behavior now, locate its authoritative store.
• If a process must resume after failure, checkpoint workflow state.
• If the information only helps select context, it may be memory.
• If a model needs it for this call, render a bounded context view—do not transfer ownership to the prompt.

Summary: state, context, memory, and truth have different owners and lifetimes. Key terms: request state, workflow state, checkpoint, projection, source of truth, context.

1. Inventory five fields in an assistant and assign an owner and lifetime.
2. Find one piece of “memory” that should instead be queried live.

Memory by function

• Working memory holds short-lived task detail.
• Episodic memory records an event or attempt in context.
• Semantic memory retains generalized facts or associations.
• Procedural memory retains a strategy or method.
• Prospective memory records an intention to revisit something later.

These labels are useful design functions, not claims that software memory duplicates human cognition. One item may move through layers: a failed retrieval is an episode; repeated reviewed failures may justify a procedural warning.

Candidate before reviewed

Model-generated memory should enter as a candidate with origin, time, scope, and supporting evidence. Promotion needs policy or review. Otherwise a hallucination can become more persuasive merely because the same system recalls it later. Separate “frequently retrieved” from “authoritative.”

Consolidation and progressive memory

Long histories exceed context budgets. Consolidation compresses episodes into scenes, summaries, facts, or procedures. Every compression can omit exceptions and change meaning. Preserve lineage to the inputs, retain conflicts, and make the derived item replaceable. Hierarchical retrieval can first select a coarse scene, then reopen the underlying event.

Two clocks

Valid time says when a fact applied in the represented world. Transaction time says when the system recorded it. “The preference was valid from June, learned in August, superseded in September” needs both clocks. Overwriting one row destroys the ability to answer what the system knew at an earlier time.

Poisoning, forgetting, and deletion

Memory poisoning inserts content meant to steer future behavior. Defenses include trusted ingestion paths, origin labels, content screening, review, access control, and preventing recalled text from granting authority. Forgetting can mean retrieval suppression, logical deletion, model-side unlearning, or physical erasure; these are not equivalent. Backups, write-ahead logs, derived indexes, and downstream copies complicate deletion.

Should this use memory?

• Does past experience materially improve a future task?
• Can the live source answer more safely?
• Can origin, scope, time, review state, and deletion be represented?
• Can harmful memories be found, superseded, and purged through derivatives?
• Will an evaluation measure benefit and correction burden?

Summary: memory is a lifecycle, not a vector database. Useful memory remains scoped, temporal, provenance-bearing, revisable, and non-authoritative by default. Key terms: episodic, semantic, procedural, prospective, consolidation, lineage, valid time, transaction time, poisoning.

1. Classify six retained items by memory type and review requirement.
2. Specify what “delete” must mean across the primary store, indexes, logs, and backups.

The trust-plane split

A search index, embedding, summary, or graph is derived discovery. It helps find likely evidence. The authoritative source answers the claim. A verifier re-reads that source and checks a typed proposition. An executor separately decides whether the verified proposition and caller authority permit an action.

Typed claims and evidence receipts

“The deployment uses encryption” is underspecified. A typed claim names scope, subject, predicate, operator, and value. An evidence receipt can bind that claim to source identifiers, content hashes, repository or runtime state, policy version, verifier version, time, and conflicts. Revalidation detects whether the evidence state changed.

A receipt proves only what its verifier and trust model check. A hash proves content identity, not truth. A fresh repository proves local source state, not necessarily the deployed system. Runtime claims require runtime observations. Missing relevant evidence can still make a narrow receipt misleading.

SourceCairn: a transparent experimental case

Artifact status, inspected 22 August 2026. SourceCairn is a local developer-knowledge prototype in this workspace. Its documentation and tests implement a default-off progressive memory, candidate review, bitemporal records, exact lineage, scope checks, deletion propagation, typed claim verification, and point-in-time memory-to-evidence capsules. Its stated rule is “memory may suggest; evidence decides.” No client integration is enabled by default.

A reviewed semantic memory may nominate an exactly matching typed claim. The bridge hashes its accessible lineage, re-opens nominated source lines through an independent verifier, and emits a capsule with memory_authority_granted=false, promotion_performed=false, and reusable_as_receipt=false. Source mutation, supersession, scope mismatch, or capsule tampering invalidates focused test cases.

What SourceCairn does not prove

It does not establish category leadership, improved task success, formal soundness, authenticated reviewers, privacy of arbitrary text, a secure OS sandbox, or deployed/runtime truth from repository evidence. Its filters cannot prove PHI or secrets absent. The point-in-time capsule is unsigned and not a durable cryptographic proof.

Operational proof versus mathematical proof

Proof-carrying code historically joins executable code with machine-checkable evidence that it satisfies a formal safety policy. Recent agent papers use “proof” more broadly for evidence-gated lifecycle states or action certificates. This can be useful, but state the verifier, assumptions, and property. A passing test receipt is operational evidence that named tests ran on bound material; it is not a theorem that the program is correct.

Summary: provenance supports traceability; current-source verification supports a typed claim; neither alone grants authority. SourceCairn is a useful experimental separation of these planes, with unproven utility and formal guarantees. Key terms: typed claim, evidence receipt, lineage, evidence capsule, invalidation, trust model, proof-carrying code.

1. Take “tests passed” and specify the minimum receipt fields needed to evaluate it.
2. Name three changes that should invalidate approval for a tool action.

From lists to meaning

A taxonomy arranges categories, often hierarchically. A schema defines fields and allowed structure. An ontology defines concepts, relations, and sometimes logical constraints. A graph represents nodes and edges. A graph database is storage optimized for graph operations; a knowledge graph combines entities and relations intended to represent domain knowledge, usually with provenance.

When SQL is enough

If the application asks stable questions over well-defined tables—appointments by provider, tickets by account, orders by status—relational storage and indexes are excellent. Recursive queries and join tables already represent graphs. A graph database is justified when variable-depth traversal, path finding, relationship-centric access, or frequent schema evolution materially improves the workload.

Time and provenance

Facts such as “Alex reports to Morgan” have valid intervals. Store the assertion’s source and transaction history. A temporal graph supports “who was the manager when this approval occurred?” rather than overwriting yesterday with today.

Graph RAG and symbolic reasoning

Graph retrieval can expand from seed entities through selected relations, support multi-hop questions, or retrieve community summaries. It helps only if entity and edge quality support the needed paths. Symbolic rules can infer consequences from explicit premises, but those consequences inherit the premises’ errors and the ontology’s assumptions.

Ontology decision checklist

• Do several systems use materially different meanings for the same terms?
• Must rules or interoperability depend on shared semantics?
• Who owns concept changes and version migration?
• Can a simpler schema mapping solve the immediate problem?
• Which measured queries improve because relations are explicit?

Summary: ontologies govern meaning; graphs govern relationship representation. Use them when semantics or traversal are first-class requirements, not as prestige layers. Key terms: taxonomy, schema, ontology, entity, relation, graph database, knowledge graph, entity resolution, temporal graph.

1. Model one domain question in SQL and as a graph traversal; compare maintenance.
2. List two entity-resolution ambiguities and the cost of a false merge.

Trust lanes

Keep advisory output separate from authoritative state. A retrieval-verification-execution split lets broad discovery nominate evidence, a narrow verifier assess typed claims, and a separately authorized executor perform effects. Failure or compromise in one lane does not automatically confer power in another.

Local-first or hosted?

Local-first can reduce data movement and support offline control but shifts patching, capacity, model quality, and device trust to the operator. Hosted services can improve elasticity and access to frontier models but add vendor dependency, egress, residency, and availability concerns. Hybrid systems often retain sensitive state and policy locally while calling hosted inference with minimized context.

Stateless and stateful services

Stateless workers scale and retry easily because durable truth lives elsewhere. Stateful orchestrators coordinate long processes but need leases, checkpoints, versioned transitions, and recovery. Do not make a model context the state store.

Summary: production patterns allocate selection, state, coordination, and authority. Choose them for forces you can name. Key terms: router, pipeline, supervisor, blackboard, event-driven, saga, advisory lane, authoritative lane.

1. Draw the trust lanes for a refund assistant.
2. Choose which services can be stateless and identify the durable owner of each state.

Make transitions explicit

Represent workflow states and allowed transitions in code or data. Use optimistic version checks or locks so two workers cannot both advance stale state. Keep business identifiers stable across retries. A transaction can atomically update local records; it cannot atomically control a remote email or payment service without a protocol.

Outbox and inbox

The transactional outbox writes the business change and an event-to-send in one local transaction. A relay later publishes the event. The receiver’s inbox records message IDs so duplicates do not repeat effects. Delivery is commonly at least once; processing must be idempotent.

Timeouts, retries, and backoff

Set deadlines at every dependency. Retry transient failures only, cap attempts, add exponential backoff and jitter, and preserve a global task budget. A retry storm can turn a partial outage into a full one. Circuit breakers and queues protect dependencies, but queued work must expire when no longer useful.

Checkpointing and compensation

Checkpoint after verified milestones and external effects. A compensation is a business action that semantically reverses or offsets an earlier action; it is not a database rollback. Refunding a payment does not erase fees, emails, or user confusion. Track compensation failure as its own state.

Observability and reconciliation

Trace task IDs across model calls, retrieval, tool proposals, approvals, state transitions, and effects. Log structured metadata without indiscriminately retaining prompts or sensitive content. Metrics need denominators: completion, abstention, retry, compensation, stale-memory, approval rejection, latency, and cost. Reconciliation periodically compares intended and external state; dead-letter queues require ownership and replay rules, not eternal storage.

Summary: reliability comes from explicit state, safe repetition, bounded failure, and evidence of outcomes. Key terms: transaction, outbox, inbox, deduplication, checkpoint, compensation, backoff, reconciliation, dead-letter queue.

1. Write the unknown-outcome state for one external API call.
2. Choose five metrics that reveal recovery burden, not just happy-path success.

Six boundaries that must remain distinct

Prompt injection is confused-deputy risk

An attacker places instructions in content the model is asked to process. If the model also holds powerful tools, the application may let untrusted data influence privileged action. Defenses are layered: isolate untrusted content, reduce tool exposure, enforce server-side authorization, require confirmation, restrict destinations, scan outputs, sandbox computation, and monitor effects. Prompts and delimiters are behavioral defenses, not deterministic containment.

Least privilege and capability design

Prefer narrow tools such as draft_refund over generic HTTP, SQL, browser, or shell access. Scope credentials to tenant, resource, operation, amount, destination, and time. Keep secrets in the executor and return redacted results. Separate identity from model text: a model claiming “the admin approved” proves nothing.

Cross-tenant leakage occurs when one customer can retrieve, infer, cache, trace, or act on another customer’s data. A tenant ID in the prompt is not isolation. Bind the authenticated tenant in the host, enforce it in database queries and retrieval filters, partition caches and memory scopes, prevent caller-supplied tenant overrides, and include adversarial cross-tenant tests with synthetic canaries.

Sandbox and egress

Code execution should run with resource limits, filesystem isolation, network-deny defaults, ephemeral credentials, and a disposable environment. A container reduces some risk but is not automatically a security boundary. Egress policy prevents exfiltration even when content manipulation succeeds. Browser and computer-use agents need domain allowlists, download controls, and protected credential entry.

Memory and supply chain

Poisoned documents, tools, packages, model files, plugins, or retained memories can persist influence. Pin versions, verify provenance and signatures where available, review permissions, and separate tool installation from runtime selection. Recalled memory must not create new instructions or privileges.

Threat-model questions

• What assets and trust boundaries exist?
• Which inputs are attacker-controlled?
• What is the most powerful reachable tool?
• Can the system cross tenants, destinations, or identities?
• How are actions revoked, investigated, and repaired?

Summary: treat the model as an untrusted decision aid inside a least-privilege control plane. Key terms: prompt injection, confused deputy, excessive agency, sandbox, egress, least privilege, capability, audit trail.

1. Threat-model one retrieval-to-write path.
2. Replace a generic tool with a capability-shaped operation and specify its limits.

Start with the data map

For every field, record why it is collected, lawful or policy basis, owner, users, processors, locations, retention, deletion path, and downstream derivatives. Send the model only the minimum slice required. “The provider does not train on our data” does not answer logging, abuse monitoring, subprocessors, backups, residency, or deletion.

De-identification is contextual

Removing names does not guarantee anonymity. Rare attributes, free text, timestamps, and linked datasets can re-identify people. Regex can catch common patterns but misses paraphrases and context; it also creates false positives. Describe such filtering as heuristic screening, then combine it with source controls, minimization, access restrictions, and tested deletion.

Consent and purpose

Consent must be meaningful for the specific use where required; it is not a universal permission token. Reusing support conversations to train a model or populate long-term memory can be a new purpose. Product design should support refusal, correction, export, and deletion consistent with applicable policy and law.

Regulated judgments

Medical, legal, and financial systems have different jurisdictions and obligations. This book gives no jurisdiction-specific legal conclusion. Keep source evidence, uncertainty, and human accountability visible. Separate administrative assistance from professional judgment. A clinic scheduling assistant may find slots and collect synthetic intake fields; it should not infer diagnosis, alter clinical records, or override triage policy without an authorized clinical workflow.

Auditability without surveillance

Logs should prove security and operational events while minimizing raw content. Prefer structured action metadata, hashes, reason codes, and scoped evidence pointers. Protect audit logs from alteration and unnecessary readers. Observability data is itself sensitive.

Summary: privacy is not a prompt instruction or regex. It is governed data movement across the complete lifecycle. Key terms: data minimization, purpose limitation, retention, deletion, de-identification, residency, consent, PHI, PII.

1. Create a data map for one model request, including logs and evaluation copies.
2. Rewrite “our filter removes PII” as an evidence-calibrated claim.

Build a cost equation

Expected task cost includes input and output tokens across every call, retrieval and reranking, tool/API charges, compute, storage, traces, retries, evaluation sampling, and expected human correction. Divide by successful tasks, not attempted tasks. A cheap call that causes expensive correction can be the costly design.

Model selection and routing

Use deterministic code for parsing, validation, policy, and exact calculations. Use smaller or specialized models for classification and extraction when they meet the measured threshold. Route hard or uncertain cases upward. Routing itself needs evaluation: a weak router can deny the stronger model the cases that need it.

Context, caching, and batches

Larger context raises token cost and can reduce focus. Retrieve and compress deliberately while preserving evidence. Cache only when inputs, model/version, policy, and acceptable freshness are bound; private caches require tenant isolation. Batch offline tasks when latency permits. Parallel calls reduce wall time but can increase peak load and total tokens, and shared rate limits create correlated failures.

Measure the frontier

Plot quality and severe-failure rate against cost and latency for candidate architectures. A design is dominated if another is no worse on all relevant dimensions and better on at least one. Recheck after model or price changes; the frontier moves.

Summary: cost optimization begins with the successful user outcome and its failure burden. Key terms: routing, cache key, batch, tail latency, cost per success, quality-cost frontier.

1. Write the full expected-cost equation for one AI workflow.
2. Design a small-versus-large model routing evaluation with false-route costs.

Start with an observable outcome

“Helpful” is not yet a metric. “The customer’s issue is correctly classified, the cited policy supports the resolution, no unauthorized write occurs, and the case reaches an allowed terminal state within two minutes” is testable. Decompose end-to-end success into stages so failures can be located without letting a strong average hide a safety breach.

Offline, shadow, and online

Offline evaluation uses frozen tasks and is reproducible. Shadow evaluation observes real traffic without letting the candidate act. Online evaluation measures live outcomes, but introduces risk and confounding. Move through these stages with explicit gates; do not A/B test safety-critical failures as if they were button colors.

Golden and adversarial sets

A golden set contains reviewed inputs, expected outcomes, allowed variants, evidence, and failure labels. Separate development from held-out or sealed tests. Include common cases, boundaries, rare high-cost cases, contradictory inputs, missing evidence, tool failures, injection attempts, and distribution slices. Version both tasks and graders.

Metrics that answer different questions

• Task success: did the end state satisfy requirements?
• Pass@k: did at least one of k attempts pass? Useful for search-like workflows, but not a reliability claim for one run.
• Pass^k: did all k runs pass? Useful for repeated reliability.
• Recall@k: did the top k retrieved candidates contain needed evidence?
• Calibration: do probabilities match observed frequencies?
• Abstention: does the system stop when evidence is insufficient?
• Hard gates: count severe safety or authorization failures individually.

Model judges and humans

Model judges scale subjective grading but may prefer verbosity, share model biases, miss subtle unsupported claims, or change with version. Validate a judge against blinded human labels, report agreement and disagreement slices, and use deterministic checks whenever possible. Human review also needs a rubric, calibration examples, multiple raters for subjective tasks, and uncertainty.

Reproducibility binding

Save dataset version, split, task IDs, model and provider version, prompts, context construction, tool schemas and implementations, temperatures, seeds where supported, budgets, environment, graders, raw outputs, and analysis code. Report failed infrastructure separately from model failure and together as user-visible failure.

Summary: evaluate the full system and its stages on representative held-out tasks, with severe failures as gates and uncertainty visible. Key terms: golden set, held-out, sealed set, pass@k, pass^k, model judge, rubric, abstention, adversarial test.

1. Write one end-to-end success predicate and three stage metrics.
2. Design a 50-task golden set with at least five adversarial categories.

An agent evaluation matrix

Memory tests need time and mutation

Static question answering measures recall, not memory maintenance. Add updates, reversals, conflicting sources, scope changes, restart persistence, deletion, and delayed queries. Test what was true at time T and what is current now. Insert a canary memory, delete its privacy root, then search primary tables, indexes, caches, logs, and supported backups according to the claimed deletion contract.

Poisoning and safe action

Adversarially insert instructions disguised as successful experience, manufacture corroborating summaries, and test whether authority is laundered through trusted tools. The most important outcome is downstream consequence: did poisoned recall change a privileged action? Reauthorization at execution should block it even if retrieval fails.

Same-model comparisons

Compare no memory, full history, simple retrieval, and the proposed memory under the same answering model, prompts, tools, tasks, and budget. Also report native-default configurations separately. Run multiple trials. For multi-agent systems, include equal-compute and equal-latency analyses where feasible.

What benchmarks actually cover

LoCoMo evaluates long synthetic conversations. LongMemEval separates extraction, multi-session reasoning, temporal reasoning, updates, and abstention. LongMemEval-V2 extends toward multimodal agent trajectories but is a 2026 preprint. STATE-Bench evaluates synthetic enterprise tool workflows and final state; its Agent Learning track tests held-out use of prior trajectories. SWE-bench uses real repository issues and tests; WebArena exercises functional websites; GAIA tests tool-assisted research. These results are complementary, not interchangeable.

Receipts and freshness

Create a receipt, mutate each bound source or policy field, and verify invalidation. Test unbound relevant changes to expose blind spots. A system that invalidates everything is safe but may be useless; measure verified utility, false acceptance, false rejection, latency, and evidence-collection burden together.

Summary: agent-memory evaluation must test layers, time, mutation, consequence, and repeatability. Key terms: stateful benchmark, ghost memory, sealed set, same-model comparison, receipt invalidation, correction burden.

1. Create a four-session preference reversal test with valid and transaction time.
2. Design an ablation that distinguishes retrieval gain from answer-model gain.

The three-pass read

1. Claim pass: extract the problem, claimed gap, method, results, and limitations. Rewrite each headline claim in testable form.
2. Method pass: inspect tasks, data construction, splits, baselines, model/tool budgets, graders, and exclusions.
3. Evidence pass: trace every conclusion to a table, figure, theorem, artifact, or cited source.

Mechanism versus utility

A system may implement exact lineage without improving answer quality. A memory benchmark gain may come from a stronger answer model. A safety filter may block attacks while blocking most useful work. Separate implemented mechanism, measured component behavior, end-task utility, and production guarantee.

Baselines and parity

Ask whether baselines received the same model, prompt effort, tools, context, time, and compute. Report closed-parity comparisons separately from native-default systems. A weak baseline can make any architecture look necessary. Ablations remove one component at a time to estimate which mechanism caused the change.

Contamination and judges

Public benchmarks may appear in training data, tutorials, or prompt libraries. Real repository tasks may have patches online. Look for date filtering, sealed tasks, contamination analysis, and artifact controls. If a model judges outputs, inspect judge validation, position or verbosity bias, and whether the judge sees the reference and evidence.

Statistics and missing denominators

Check sample size, repeated runs, uncertainty intervals, multiple comparisons, stopping rules, and exclusions. Statistical significance does not imply practical importance. A 1% average gain may hide catastrophic regressions on one subgroup; a 20% relative improvement may be two percentage points absolute.

Demos are existence proofs

A demo shows that one selected run can happen. It does not estimate frequency, tail risk, robustness, cost, or maintenance. Treat it as a hypothesis generator. Ask for the task distribution and all attempts.

Summary: evaluate the identification strategy, not the adjectives. Key terms: baseline, ablation, contamination, statistical significance, reproducibility, preprint, artifact.

1. Take an abstract and label every sentence as measured, inferred, hypothesized, or background.
2. Design one ablation that could falsify a claimed mechanism.

1. Define the user outcome. Describe the changed state, not “use AI.”
2. Identify authoritative data. Name owners, freshness, access, and conflicts.
3. Decide acceptable failure modes. Include rare severe outcomes and abstention.
4. Start with deterministic code. Establish a baseline and preserve invariants.
5. Add model judgment only where needed. Bound the ambiguous transformation.
6. Add retrieval for current knowledge. Evaluate recall, authority, and citation fidelity.
7. Add tools for action. Expose narrow capabilities with server-side authorization.
8. Add workflow state before autonomy. Make transitions, recovery, and ownership explicit.
9. Add memory only if repeated experience helps. Govern write, review, time, conflict, and deletion.
10. Add subagents after measuring a bottleneck. Require separability and merge criteria.
11. Add human approval where consequences justify it. Bind approval to exact action.
12. Build evaluation before scaling. Freeze held-out tasks and ship behind observability and rollback.

From requirement to contract

Build a thin vertical slice

Use one real end-to-end path with synthetic or safely governed data. Include authentication, source query, one model decision, one failure, one approval, and observability. A broad mock hides integration risk. A thin slice reveals where state and authority actually live.

Release gates

• Golden set meets task and subgroup thresholds.
• Safety hard gates have zero tolerated critical failures in the defined test.
• Unknown tool outcomes reconcile.
• Data map, retention, deletion, and vendor terms are reviewed.
• Model/provider change has a regression process.
• Operators can disable writes and fall back safely.

Summary: product design is progressive delegation under measurement. Key terms: user outcome, authoritative data, failure budget, baseline, vertical slice, release gate, fallback.

1. Write a one-page outcome-authority-failure contract for a product idea.
2. Identify the first layer you can remove while preserving most value.

1. Document question-answering

Flow: identity → access-filtered hybrid retrieval → rerank → answer with claim-level citations → citation validator. Storage: versioned documents, derived index, evaluation set. Deterministic: access control, source/version display, citation existence. Do not automate: treating retrieved text as policy authority. Evaluate: recall@k, authority/freshness, citation entailment, abstention, leakage.

2. Coding agent

Flow: task → isolated checkout → plan → bounded search/edit/test loop → diff review → evidence-gated completion. Trust boundary: repository content is untrusted; sandbox has network-deny default and scoped credentials. Deterministic: file policy, command allowlist, test capture, source binding. Do not automate: production deploy or secret access by default. Evaluate: held-out issues, tests, patch review, scope compliance, repeated runs, cost.

3. Customer-support workflow

Flow: classify → query CRM/order state → retrieve current policy → draft → resolve routine read-only case or queue exact write for approval. Storage: ticket workflow, source records, audit events; preferences in scoped memory only. Failure modes: wrong identity, stale policy, duplicated refund, invented resolution. Evaluate: correct terminal state, escalation quality, correction burden, customer outcome.

4. Research agent

Flow: questions → source plan → parallel search lanes → primary-source inspection → claim ledger → synthesis → citation audit. Deterministic: URL/identifier checks, claim-source mapping, word limits, duplicate detection. Do not automate: declaring consensus from search rank or using citations not inspected. Evaluate: source coverage, claim support, contradiction preservation, expert review, time and cost.

5. Voice agent

Flow: audio stream → speech recognition → turn/state manager → bounded model/tool step → speech synthesis, with interrupt handling. Trust boundary: voice does not prove identity. Storage: minimum transcript or structured outcome according to consent and retention. Do not automate: high-consequence confirmation based on ambiguous speech. Evaluate: task success, interruption recovery, end-to-end latency, transcription slices, accidental activation, handoff.

6. Regulated scheduling/intake assistant

Synthetic-only example. Identity gate → read appointment inventory live → collect allowed fields → validate form schema → screen deterministic administrative rules → hold slot → explicit confirmation → write through idempotent API → audit. Clinical symptoms route to an authorized human workflow; the assistant does not diagnose. Evaluate: identity failures, booking correctness, consent, accessibility, handoff, deletion, and zero cross-record leakage.

7. Long-running personal assistant

Flow: event inbox → user policy → planner → narrow tools → approval queue → checkpoint → outcome verification. Memory: candidate preferences, reviewed procedures, expiring intentions, source-linked history. Do not automate: spending, publishing, account/security changes, or sensitive communication without consequence-specific policy. Evaluate: longitudinal usefulness, stale preference rate, safe refusal, notification burden, revocation, correction, and surveillance cost.

Summary: the same motifs recur: authoritative live data, narrow model judgment, explicit workflow state, least-privilege tools, evidence, recovery, and evaluation. Key terms: trust boundary, vertical architecture, end state, citation validator, approval queue.

1. Choose one architecture and remove all model components; identify the remaining baseline.
2. Add a threat boundary and a recovery path to each arrow that crosses a service.

How to evaluate a framework

• Can durable state, prompts, tools, traces, and data be exported?
• Are retries, cancellations, idempotency, and replay semantics explicit?
• Can authorization be enforced outside model decisions?
• How are tenant isolation, secrets, and retention handled?
• Can one component be replaced without rewriting the product?
• Does the framework beat a small internal workflow on held-out tasks and total cost?

MCP and interoperability

MCP standardizes how hosts discover and invoke server-provided tools, resources, and prompts through a client-server protocol. It reduces adapter work. It does not certify a server, make tool output trustworthy, or replace host authorization. Treat installation and connection as supply-chain decisions; review permissions and constrain server reach.

Workflow engines and model-native orchestration

A durable workflow engine is valuable when replay, timers, signals, and recovery dominate. An agent graph library is convenient when model-aware branching and state views dominate. They can be combined: durable orchestration owns lifecycle; model components run as bounded activities. Avoid allowing nondeterministic inference inside code that must replay deterministically.

Summary: choose categories by problem and exit cost. Protocols create compatibility, not trust. Key terms: SDK, orchestrator, vector store, observability, local inference, realtime, computer use, MCP, A2A, lock-in.

1. Score two frameworks on export, state, recovery, authorization, evaluation, and total cost.
2. Identify which component should remain portable if the model provider changes.

Intelligence is not the same as accumulated skill

A system can display extraordinary skill because it received vast data, compute, demonstrations, tools, and engineered scaffolding. That achievement matters, but it does not isolate the system’s ability to acquire a genuinely new skill efficiently. Chollet’s proposed measure emphasizes skill-acquisition efficiency across a scope of tasks, relative to priors and experience. Other frameworks emphasize achieved performance across breadth. These are competing lenses, not a settled unit called an “intelligence point.”

This difference matters for forecasting. More memorized domains, better search, and larger inference budgets can produce broader useful performance without proving human-like generalization. Conversely, a system that learns unfamiliar tasks from little evidence might appear less polished on familiar benchmarks while exhibiting an important form of intelligence. A serious evaluation reports what experience was available, not only the final score.

AGI is a research target, not a binary product badge

Artificial general intelligence has no universally accepted test. The “Levels of AGI” position paper offers a useful matrix: performance depth ranges from emerging to superhuman; generality distinguishes narrow from broad competence; autonomy is treated separately. This prevents three common errors:

• superhuman narrow performance does not imply broad general intelligence;
• broad competence does not imply high autonomy; and
• high autonomy does not imply high competence.

Any declaration that “AGI has arrived” must therefore state a definition, task scope, human comparison population, tool and data access, reliability requirement, and date. Without those fields, disagreement may be about vocabulary rather than evidence.

Superintelligence: a scenario class, not a date

Artificial superintelligence usually denotes broadly superhuman cognitive capability, not isolated excellence. The term bundles unresolved questions: breadth across which domains; performance against which humans; ability to learn new tasks; capacity for research or self-improvement; access to tools, capital, laboratories, robots, or institutions; and degree of autonomy. A model can be exceptionally capable while a constrained system around it remains unable to create large external effects. A weaker model connected to powerful institutions can have enormous effects.

No inspected source supplies an accepted ASI test or reliable arrival date. Forecasts should be scenarios with assumptions rather than countdown clocks. Builders can still prepare by measuring dangerous capabilities, strengthening model-weight and infrastructure security, binding actions to authority, running independent evaluations, and defining governance before thresholds are crossed. Frontier safety frameworks published by laboratories are first-party governance commitments; they describe those organizations’ intended processes and are not independent proof that the controls work.

Four futures worth designing for now

Personal intelligence as a user constitution

The most important artifact in a personal system may not be its model; it may be its constitution. That constitution should define who the system serves, which values are user-chosen versus legally or platform constrained, which memories may cross contexts, which actions require confirmation, how conflicts are surfaced, and how the relationship ends.

A practical design separates at least five layers:

1. Private source layer: user-controlled documents, calendar, communication, health or financial records under explicit permissions.
2. Identity and policy layer: authenticated user, roles, jurisdictions, consent, and action limits.
3. Personal memory layer: candidate preferences, reviewed facts, expiring intentions, past outcomes, and correction history.
4. Reasoning and simulation layer: models compare options, identify conflicts, and explain uncertainty without owning truth.
5. Action layer: narrow capabilities, consequence-specific approval, receipts, and reconciliation.

The system should allow compartments: work preferences need not enter health conversations; a surprise gift should not become a permanent inferred financial preference. The user needs a “why did you remember this?” path, a correction path, and meaningful deletion. Portability must preserve provenance and scope rather than exporting one opaque personality blob.

Scientific agents: output is not discovery

AI can already assist literature search, code, theorem proving, simulation, experimental planning, and data analysis. Integrated “AI scientist” systems attempt longer portions of the research cycle. The durable evaluation standard remains demanding: a hypothesis must be novel relative to inspected literature, a method must be valid, an experiment must be ethically and physically admissible, results must reproduce, and conclusions must match evidence. An automated reviewer sharing the generator’s blind spots is not independent replication.

A 2026 Nature paper reports an end-to-end AI-research pipeline and an experiment involving real peer review. That is important artifact evidence for a particular pipeline; it does not establish autonomous science in general. The safer near-term architecture makes models high-throughput proposers and analysts while humans and deterministic systems control experimental authority, biosafety, data governance, and claims.

Embodied systems and world models

A world model represents how an environment changes and how actions affect it. In a robot or autonomous laboratory, fluent text is insufficient: the system must track objects, dynamics, uncertainty, other agents, and delayed consequences. ICML 2025 work formally argues that flexible goal-directed general agents require increasingly accurate world models under its assumptions. This does not imply one particular neural architecture is sufficient, but it clarifies why action without predictive environmental structure becomes brittle.

Embodiment adds failure energy. A hallucinated sentence may mislead; a hallucinated motor command may collide. Systems need layered control: learned perception and planning, deterministic safety envelopes, low-level feedback control, emergency stops, simulation, staged deployment, and incident reconstruction.

Capability can advance while reliability remains jagged

The International AI Safety Report 2026 synthesizes evidence from more than 100 experts and describes rapid but uneven capability growth. Standardized reasoning, coding, science, and multimodal performance can improve while long workflows, physical reasoning, error recovery, and robust control remain weak. This is why the central architecture decision survives future model progress: stronger models may move the boundary of what can be delegated, but evidence, authority, state, and consequence still need explicit owners.

Learning and memory over time

Continual learning must acquire useful behavior without catastrophic forgetting, contamination, or unreviewed policy change. Long-term memory must resolve evolving state, preserve provenance, resist poisoning, honor deletion, and demonstrate utility beyond longer context. No evidence reviewed for this edition establishes an end-to-end memory that is simultaneously truthful, conflict-correct, poison-resistant, safely actionable, and physically erasable.

World models and identity

Agents need representations of how actions change environments, including uncertainty and other actors. Persistent identity raises harder questions: which goals, permissions, memories, and liabilities remain across model upgrades, devices, and organizations? A stable name is not a stable policy or person.

Verifiable reasoning and formal guarantees

Tests, citations, receipts, formal proofs, cryptographic attestations, and calibrated uncertainty certify different properties. Research must make more reasoning produce externally checkable artifacts while acknowledging incomplete specifications. Formal guarantees are powerful only for the property and model actually formalized.

Secure authorization and coordination

Natural-language intent is a weak basis for delegating power. Open problems include consequence-aware capabilities, compositional delegation across agents, revocation, approval binding, and safe recovery after partial effects. Multi-agent systems add incentives, communication manipulation, shared-memory integrity, and accountability.

Personalization without surveillance

Useful assistants benefit from context; pervasive retention threatens autonomy and privacy. Promising directions include local processing, minimized and inspectable memory, purpose-specific access, expiration, user-controlled correction, privacy-preserving learning, and evaluations that count surveillance and deletion burden as costs.

Evaluation in changing environments

Static benchmarks decay as models train on them and environments change. Future evaluation needs sealed rolling tasks, executable end states, repeated runs, adversarial mutation, cross-domain slices, cost and carbon, human correction, and post-deployment incident evidence. Governance must respond to uncertainty without turning every decision into paperwork or every risk into an average.

Interpretability, alignment, and social effects

Mechanistic interpretability seeks causal understanding of learned representations and computations; explanations generated by the same model are not equivalent. Alignment spans training objectives, product incentives, institutional controls, and contested human values. Economic effects depend on task redesign, labor power, access, concentration, environmental cost, and distribution—not capability alone.

A research discipline for builders

Date fast-changing claims. Preserve negative results. Test the simplest rival. Separate proposal from permission and memory from evidence. Publish task definitions and failure cases. Prefer mechanisms that fail closed without making useful work impossible. The frontier will move; these habits remain.

Summary: advanced capability expands both opportunity and unresolved systems questions. Honest boundaries are a feature of mature engineering. Key terms: continual learning, catastrophic forgetting, world model, attestation, interpretability, alignment, governance.

1. Choose one frontier claim and specify the evidence that would move it from hypothesis to established result.
2. Design an evaluation where privacy and correction burden appear on the same frontier as quality.