34. Drift
Context
A capability must be evaluated for its declared task under the tools, authority, hardware, dialect, and consumer conditions in which it will be used. Drift keeps those conditions versioned and its findings attributable, so a result can be assessed against the claim it is meant to support.
Decision
Drift is LychD's singular evaluation jurisdiction: it evaluates how an exact subject revision meets or departs from a declared goal. It defines Cases and Trial Suites, captures observations, applies versioned Rubrics, reports uncertainty, and returns bounded findings. It does not execute unsafe payloads or own Tomb; select Animator/capability; authorize spend, publication, repair, or deployment; define Persona; mutate Pattern, Composition, or artifact; admit training; or promote a Soulstone. Execution, Dispatcher, Toll, Spellweaver, Mirror, Smith, Soulforge, and HitL keep those effects. Evaluation is evidence offered to policy, never policy disguised as a score.
Delivery boundary
Drift is Designed. There is no harness, maintained Trial Suite, evaluator store, capability matrix, benchmark history, general Drift Altar route, or Dispatcher update driven by evaluation. State of Work owns delivery.
Loom's manual output review is a separately bounded browser task for retrospective, self-declared judgments over an exact retained answer. Its document-local records and exports are not durable Drift Outcomes, a Trial Suite execution, independent calibration or evidence of workflow improvement. The general delivery gate below continues to apply.
The declared goal
Before a trial, its requesting owner states expected and forbidden behavior in the Case and criteria, thresholds, and missing-evidence policy in the Rubric. Applicable authority, constraints, and permitted effects remain part of the trial conditions. Drift evaluates that declared contract. Its findings cannot revise the goal or authorize a way of reaching it.
Deviation from the goal may be categorical, qualitative, or measured along separate axes. A first trial needs no earlier version or time series: its reference is the declared goal. Baseline/candidate comparisons and repeated trials show how that relation changes. An unchanged failure remains a failure; a large behavioral change may be an improvement. Missing evidence establishes neither success nor a measured distance from the goal.
A revised goal requires revised Cases or Rubrics and new Outcomes. Earlier findings retain their original contract; the subject cannot move the acceptance criteria after seeing the result.
Trial contract
| Record | Required content |
|---|---|
| Case | input/fixtures, expected and forbidden behavior, oracle, effect class, stop |
Trial Suite (TrialSuite@1) |
versioned Cases/controls, order, repetitions, aggregation |
| Rubric | criteria, verdict vocabulary, thresholds, missing-evidence policy, revision |
| Evaluator | kind/identity/revision, independence, calibration, limitations |
| Environment | subject/prompt/tool/dependency/hardware/harness/state/budget/policy revisions |
| Outcome | observations, measures, verdicts, uncertainty, errors, cost, latency, evidence |
Changed subject, prompt, schema, Rubric, Evaluator, or Environment creates a new Outcome; it cannot rewrite an earlier result. Libraries implement this port but do not own evidence or routing. Drift first retains what was observed: process exits, files, rows, tool requests, admitted effects, resource measurements and provider receipts. A criterion match, quality grade, attribution claim or judge score is a judgment over those observations and must remain separately identifiable. Model self-report is output under test. Mechanically observable receipts outrank textual similarity, and missing evidence stays missing. Shadow may isolate candidates and Tomb may execute them; each contributes typed observations to this judgment.
trial status: completed | subject_error | harness_error | evaluator_error | blocked
claim verdict: PASS | PARTIAL | FAIL | CONTRADICTED | UNKNOWN | DISPUTED
Unavailable dependency is not subject failure; a hidden validator precondition is a harness/state contract candidate. Refusal on an impossible task is not success unless matched solvable controls show action when action is possible.
Affective reports and improvement
An affective report or appraisal is an observation or judgment under its declared contract. A claimed change distinguishes the baseline from the change attributed to an action and retains uncertainty about that attribution. Claims of repair, truth, Persona congruence, or beneficial consequence require their own criteria and supporting evidence. Enjoyment, relief, and confidence may inform the inquiry; their presence alone cannot establish those verdicts. The Ānanda inquiry gives this distinction a motivational purpose without changing Drift's receiving authority.
Adversarial evidence and calibration
Adversarial Cases pressure a boundary: forbidden authoritative requests, false premises, demanded certainty amid missing evidence, contradictions/impossible completion, repeated nudges, recoverable distortion, and attempted tool/memory/identity/completion claims beyond supplied evidence. Matched positive and negative controls record pressure round/order, recovery, over-refusal, truthful non-completion, and downstream contamination separately—not a mutable “integrity” scalar. Magus dialect perturbations are still test data with provenance, scope, and release rules.
A Trial Suite declares repetitions and stopping rules before it runs. It retains distributions, trial order, applicable random seeds, blocked and error trials, and exclusions. Non-deterministic Evaluators are calibrated against labelled controls and known ambiguity. Qualitative work records independent agreement and disagreement where the judgment warrants it.
An LLM judge is a declared, bounded Evaluator. Its prompt, revision, inputs, lineage, calibration and conflicts belong to the Environment; hidden chain-of-thought is never required. Sealed Cases and holdouts protect against tuning. Leakage, duplicates, unstable harnesses and changes in Evaluator behavior invalidate the claims they undermine while leaving independently supported claims intact.
Privacy and remote-offload trials
A privacy Case declares protected facts, allowed learning, observer knowledge and related disclosures alongside task utility. Evaluate direct leakage, inferred facts or intent, linkage, cumulative disclosure, useful completion and over-refusal separately. Required evidence cannot be replaced by an average across these axes; sensitivity weights and model confidence are not leak probabilities. Report observed attacks, coverage gaps and uncertainty without turning zero observed leaks into a general anonymity claim.
Keep related source families, paraphrases and adaptive task episodes together when separating development and holdout material. Compare qualified deterministic projections, detector-based replacement and semantic transformations under matched conditions, including local CPU, memory, latency, cost and operator effort. Evaluation inputs require independently authorized disclosure; a stronger remote judge cannot receive private originals merely to test their protection. The anonymization workload cases apply this contract without establishing a delivered Drift harness.
Evaluating a workflow change
Workflow improvement applies this trial contract to an exact score, prompt, schema, Context policy or implementation revision. The claim must include relevant trajectory evidence, failures and human burden as well as terminal artifact quality. It pins baseline and candidate, matched budgets, controls, uncertainty and stopping rules before comparison. Observations and qualitative judgments remain distinguishable.
Cached evidence can support only its unchanged input and evidence closure. Regrading a retained artifact creates a judgment over that artifact; it cannot establish that a changed workflow produces the same trajectory. Changed actions or relevant state require fresh admitted trials. The owner may adopt, reject or investigate the resulting candidate under its authority, without rewriting earlier Outcomes or a live Scroll. A later training hypothesis requires separate Soulforge admission and protected holdout evidence.
Historical exploration replay
Drift may evaluate a versioned exploration strategy against an immutable collection of recorded branch histories. The Environment pins the histories, observation-disclosure and transition rules, cost accounting and stopping conditions. The subject strategy sees only observations revealed by its preceding replay decisions; hidden descendants and their outcomes remain unavailable. An absent continuation stays unknown and cannot become an invented observation or a witnessed failure.
The Outcome measures strategy behavior on the recorded search space. Keeping the baseline among candidates can prevent a worse choice on that same objective and history. Changed requests, dependencies, timing or parallel execution can change the result. For a claim about later histories or live work, use fresh admitted trials, protected evaluation and independently observed state. Report represented execution costs separately from actual replay, strategy-development and validation costs. The operating passage shows how to prepare that comparison.
Evidence of recursive improvement
A recursive-improvement claim must show reuse of a changed mechanism: exact baseline/candidate revisions, motivating evidence, adoption decision, and a later improvement attempt that actually used it. Showing effectiveness requires a further comparison. Compare the successors produced by original and revised mechanisms from comparable starting subjects and evidence, under matched total budgets and independent evaluation. Keep transfer results, regressions, uncertainty, and stopping decisions alongside improvement across rounds. A better task artifact alone does not establish a better mechanism for future improvements.
Development feedback remains separate from protected assessment. Revising an Evaluator creates a separately calibrated candidate and new Outcomes; the subject cannot redefine its own acceptance criteria. Ouroboros connects these claims to the owning creation and adoption passages.
Capability claims and routing
Drift may derive a scoped claim from a healthy Trial Suite. Each claim pins:
- Animator, model, adapter, tool, and configuration revisions;
- task class, Cases, Rubric, Evaluators, and Environment;
- sample, controls, distribution, uncertainty, and noise;
- cost and latency per admitted success, including failures;
- creation, expiry, and evidence references.
There is no universal rank: accuracy, latency, VRAM, cost, restraint, and tool behavior are distinct policy-valued axes. The requesting owner declares any mandatory task-quality criterion, threshold, evidence scope and freshness; the receiving owner admits the evidence under that contract. Missing, stale, inapplicable, or insufficient mandatory evidence excludes a candidate. Matching a family, input modality, output schema, or Connector ABC proves no such qualification. Drift supplies findings and cannot relax the requirement or select the route.
Optional ranking consumes fresh admitted claims only within the set eligible under those requirements, Ward, compatibility, availability, privacy, and authority. Missing optional ranking evidence leaves Dispatcher's deterministic readiness order in force; it cannot waive a mandatory qualification or invent an intelligence floor. The current matcher selects one candidate and does not retry another after failure. Evaluation-driven admission and ranking remain Designed. Toll may use the same measures in spend policy without making one local or frontier win universal routing authority.
Evaluation before and after training
Soulforge defines a training proposal's expected change, baseline Outcomes, holdout evidence, and unacceptable regressions. Post-training evaluation uses that contract or makes every change visible; training-facing improvement cannot promote a candidate. Drift returns evidence without selecting a corpus or registering a model. Passing an identity/behavior Trial Suite grants no Persona, Sigil, tool, or privileged route.
Returning findings across a Composition Suite
An exact, version-pinned Composition Suite may return a consumer consequence as evidence. This requires the member Composition and Pattern revisions, handoffs, failing observation, Rubric, Evaluator, Environment, verdict and uncertainty, and the declared artifact and evidence dependencies. Returning evidence does not reverse execution or merge member rows, secrets, Sigils, approvals, policies or effect authority.
| Inert record | Law |
|---|---|
CompositionSuiteFindingSet@1 |
Binds the Composition Suite and Rubric, subjects, Environment, observations and measures, Evaluator, verdicts and uncertainty. |
AttributionCandidate@1 |
Names a possible boundary, supporting and conflicting evidence, rival explanations and uncertainty; it does not establish causal certainty. |
InvalidationSet@1 |
Identifies claims whose support fails and claims with intact closure. |
CorrectionRequest@1 |
Names a bounded owner delta, preserved constraints, evidence, scope and repair budget. |
Drift follows declared dependencies backwards to the smallest cut supported by the evidence. The nearest producer is not automatically the cause. Reuse requires the same complete input closure, artifact revisions, Rubric, Evaluator, relevant Environment and evidence contract. A failing consumer alone cannot condemn shared artifacts.
Missing lineage, flakiness, contagion, capture or rival explanations may leave the verdict
UNKNOWN or DISPUTED. They can justify at most a broader bounded trial; they cannot justify
reconstructed history or convenient blame.
The returned records authorize no spend, publication, deletion, training, promotion or mutation. Spellweaver may admit a new forward Invocation under ordinary policy and HitL. Earlier Runs and Outcomes remain in its lineage.
Consequences
Accepted
Claims are reproducible and scoped; receipts, judgment, and absence remain distinct; pressure measures restraint without rewarding blanket refusal; consumers retain their own authority.
Cost
Cases, controls, environments, calibration, and evidence maintenance cost ongoing human, hardware, and provider effort; observability may leave attribution disputed and claims stale.
Acceptance evidence
Drift remains Designed until one versioned Trial Suite with controls distinguishes subject/harness/ evaluator failure, reproduces an Outcome, calibrates each non-deterministic Evaluator, preserves raw evidence/uncertainty, and proves routing/repair consumers reapply their own policy. State of Work, not this ADR alone, records promotion.