23. Orchestrator

Requirements

• Orchestrator alone owns application-initiated readiness transitions; systemd remains the physical transaction engine.
• Fresh capability observations and the compiled conflict graph determine one plan.
• Dedicated local Soulstones alone are lifecycle-managed; shared runtimes and Portals may be warm routes but are not managed.
• Admission closes before complete affected-set lease drain; stale truth declines before effect.
• Host work crosses a frozen TransitionIntent, never a command, path, environment, unit, or generic payload.
• Reopen only after success, proved no effect, or exact restoration. Contain uncertainty.
• Priority orders waiters and may decline a hard swap; it never preempts a submitted transaction.

Considered Options

Decision Outcome

A non-warm managed route arrives as handle-free HardwareTransitionRequired; canonical registry truth is fetched again.

Current conflict policy

After refreshing every managed Animator, the current policy gives NO_OP to a warm/open target, SOFT_SWAP to a started but non-warm dedicated runtime, HARD_SWAP to a down dedicated runtime, and refuses every non-warm shared/non-managed runtime. A hard plan selects active exact neighbours of the undirected conflict_domains graph. Omitted conflict on dedicated non-resident means conservative default-exclusive; only explicit [] claims coexistence. Coven membership aggregates metadata: it never creates or relaxes conflicts. Bind rejects unadvertised Soulstones, conflicting Covens, and non-dedicated or persistent-resident Animators with non-empty conflicts.

Leased neighbours remain evictees: a lease forces drain; it does not make incompatible work immortal. Affected set means evictees plus launch targets.

Arbitration and planning budget

TransitionArbiter has one owner. It orders contenders by descending priority, then FIFO. Requests for the same capability at the same priority join one in-flight plan; requests at different priorities remain separate. A hard swap below min_priority_for_hard_swap declines before effect; NO_OP and SOFT_SWAP are never threshold gated. A manual or internal transition request must carry canonical doctrine priority 0..100; the manager rejects an out-of-range value before it records a trace or enters the arbiter, and the HTTP boundary rejects it during query validation. A warm/open fast path can return early, but every other plan is recomputed inside the arbiter after its predecessor settles. The arbiter reserves the exact capability/priority cohort before its owner begins asynchronous registry preflight; followers join that result without probing, while a warm/open result still bypasses the global physical-transition section. Previews bind nothing.

Each preflight and each post-arbitration replan has one planning_timeout_s observation budget for the complete managed-Animator set, including waiting for the registry probe lock. It does not reset for each peer, consume the later warm-up budget, or limit time queued for arbitration. An expired planning budget cancels and settles the probe, invalidates interrupted observations, and declines before effect. Cohort followers receive the same failure; a later request may retry with fresh observations. No lease or claim gate closes until planning succeeds.

Optional admission expiry

A direct POST /orchestrator/activate request may supply admission_timeout_s. It is a finite, nonnegative duration, converted once at handler entry to an absolute cutoff in the owning event loop's monotonic clock. The internal manager input is an optional finite admission_deadline in that same clock. Neither queueing, replanning nor joining a cohort renews it. Zero is a valid already-expired request. Invalid durations or a nonfinite computed cutoff receive HTTP 400; an already-expired request declines before trace, cohort or probe mutation. Omission creates no caller-owned admission cutoff.

Admission means this transition invokes its hard actuator or soft adapter activation, or confirms honest readiness on an observation-only path. Both the warm/open fast path and the post-arbitration NO_OP path require that confirmation. Static and already-WARMING waits remain subject to admission expiry until readiness is confirmed. The final invocation check follows synchronous observer publication, with no intervening callback or await before invocation. A refused soft invocation does not acquire the uncertainty of an activation it never called.

A timely invocation or readiness confirmation retires admission expiry. Later observer delivery, HTTP completion or physical settlement may cross the old cutoff without replacing the result. Admission expiry promises neither a response by the cutoff nor physical host submission by that time. After invocation, existing explicit cancellation, readiness, compensation and containment law continues to govern the effect.

The capability/priority cohort retains one owner. A follower's earlier expiry ends only its own wait; it cannot cancel the owner, inherit ownership or reopen gates. Owner expiry fixes the cohort refusal, delivered after required cleanup and gate handling. An omitted-option caller, including an internal caller, can join a timed owner and inherit that refusal. A timed follower can join an untimed owner without tightening the owner's budgets. Cohorts are not partitioned by deadline. A nonexpired follower joining already-admitted work joins its settlement. For a follower already waiting before admission, compare its cutoff with the actual admission boundary; delayed timer delivery does not admit a follower whose cutoff was reached first.

HTTP admission refusal carries code and readable detail, with HTTP 409. The code is transition_admission_expired for owner or already-expired entry refusal, and transition_admission_wait_expired when only a follower's wait expires. Successful TransitionPlan responses and existing priority-threshold refusal responses retain their contracts. These codes create no public request lookup or durable deadline identity.

Local budgets and deadline precedence

Admission expiry neither replaces nor restarts a local safety budget. Preflight and replan each retain their own complete-set planning budget. A drain wait retains the cutoff created by LeaseLedger after queue claims have paused and an uncleared lease set requires waiting; an already-clear set creates no drain timeout. Soft convergence starts its warm-up budget after drain and the verifying publication. Hard readiness starts after actuator settlement and the verifying publication. Each convergence budget retains one cutoff for its complete operation, including the hard transition's evictee-cold observations. Queue waiting creates none of these phase budgets.

The earliest applicable original cutoff wins. A local planning, drain or convergence budget wins an exact tie with owner admission expiry. Expiry means now >= cutoff: timers wake the decision machinery, while successful phase completion and admission check the clock before retiring their cutoffs. Completed phases retire their budgets; cleanup cannot renew them. Strict enforcement also applies when admission expiry is omitted. This preserves budget starts, durations and error categories while closing delayed-callback races that could previously accept late success.

While admission remains pending, fix the winning deadline cause and its nominal cutoff before cancellation begins required settlement. A winning local budget retains its existing failure classification; an earlier owner admission cutoff produces admission refusal. For an already-joined, unadmitted follower, an earlier follower cutoff ends its own wait; an earlier or equal winning cohort cutoff delivers the settled cohort failure. Once a deadline outcome is selected, callback ordering, cleanup duration, later publication or subsequent cancellation cannot replace the cohort's recorded cause. Repeated cancellation cannot interrupt required cleanup. Cancellation or disconnection of an outer caller's await may prevent delivery to that caller without rewriting the cohort outcome. An ordinary failure or explicit cancellation that initiated cleanup retains its existing classification even if a cutoff passes during cleanup. Keep these decisions in control state, not projection traces. This rule does not establish a general timestamp ordering among arbitrary exceptions or change cancellation after timely admission.

Drain, change, and observe

The actual rite is:

1. Refresh and compute target plus exact affected set.
2. Close affected lease admission, then the process claim gate.
3. Wait for no live LeaseLedger grant on any affected Animator.
4. Freeze a TransitionIntent with target, evict/launch sets, expected active pre-world, and capability/conflict digest.
5. Have the host boundary attest freshness, ownership, loaded topology, and pending systemd work.
6. Perform one hard target transaction or one permitted runtime-native soft activation.
7. Require target WARM, and after hard swap every evictee stopped.
8. Reopen only under the restoration law.

Pre-effect drain timeout/cancellation reopens gates. Operational failure or cancellation during the direct systemd actuator's topology, pending-job, or pre-world observations is also verified no effect: no runtime transaction has been submitted, so retain the cause and classification and reopen the gates. Once submission begins, cancellation must settle and classify the transaction and, where needed, restore the exact prior world before reopening. The waiting run has no lease. Hardware Stasis is live; Graph and Snapshots own durable sleep. The process-local bounded TransitionJournal is for Nexus/run-event projection, not a Host Reactor journal or complete history. The journal and Run observers consume the same immutable transition observation. Callbacks never receive the mutable control trace or physical plan. Traces retain only the action and cost summary, not the plan's mutable eviction and launch sets. Projection failures do not direct or abort a physical transition.

Runtime-Started Convergence (SOFT_SWAP Plan Label)

SOFT_SWAP says only that the runtime is started. It need not load a model. The whole target Animator drains because a dynamic load can unload another model behind a lease. A dynamic non-WARM/non-WARMING capability asks its canonical adapter to activate; WARMING and static started routes merely await honest warmth. Router observations classify each model independently. Activation rechecks that inventory and does not submit another load for a target already loading or loaded. Inference disables router autoload so lost warmth cannot initiate a load or eviction outside this transition boundary.

Dispatcher never asks this of a shared non-warm route. Direct surfaces enforce the same ownership boundary before considering runtime-started convergence, so an already-started shared dynamic Animator cannot enter SOFT_SWAP.

llama.cpp router and ExLlamaV3/TabbyAPI prove the seam in repository tests, not live model/GPU operation; State records their Operator-validation receipts.

One absolute warm-up deadline bounds the complete target convergence operation: initial refresh, optional adapter activation, accepted- activation refresh, and final WARM polling. That same deadline bounds post-stop cold observation of every evictee. A hung probe therefore enters typed compensation or containment instead of holding closed admission and claim gates forever. Cancellation or failure during adapter activation or its accepted-state refresh invokes bounded adapter abandonment before the canonical error is propagated; repeated caller cancellation cannot interrupt that cleanup.

Before this transition invokes adapter activation, convergence is observation only. Failure, timeout, or cancellation of the initial refresh or a static/already-WARMING wait is a verified no-effect decline: reopen the claim and admission gates and retain the original cause. Coalesced requests receive the same no-effect classification. An externally started load does not make an observation-only transition the owner of that effect; fresh readiness still governs later grants.

Invoking adapter activation crosses the uncertainty boundary before the call begins, including a call that returns accepted=false: that boolean is not an authoritative no-effect receipt. Soft activation has no sufficient prior model state for a trustworthy inverse, so failure after that boundary leaves its claim and admission gates closed for operator recovery. Before invoking a soft activation, the Vessel durably arms an owner-only record in the Crypt's runtime-guard directory. Only observed target WARM permits clearing that exact record. An interrupted or uncertain attempt, including an incomplete record write, fences a fresh Vessel before queues or services are published; changing configuration or restarting the process cannot release it. Observation-only convergence does not arm a record. Persistence failure prevents activation and fails closed.

The guard is mounted read-write into the managed Vessel, separately from the Host Reactor's read-only journal. Recovery is an explicit host operation: with lifecycle and guard exclusion held, attest the owned topology and prove the Vessel and the recorded Animator service stopped before clearing the exact pending record. Unknown identity, changed topology, pending jobs, or uncertain service state refuse recovery. No readiness probe alone releases containment.

Host Mutation Port and Privilege Boundary

RuntimeActuator.apply(TransitionIntent) is the lone domain mutation port. Its closed intent has logical Animator identities, no shell or unit payload. Compensation names one completed forward transition and is its exact typed inverse.

Direct systemd remains an explicit host-adapter test mechanism, not a host-native Core profile: under the lifecycle lock it resolves registry identities, attests the loaded Scribe-owned graph, requests one target transaction, awaits jobs, and classifies service state. Caged Host Reactor atomically publishes owner-only JSON, revalidates after claim, shares the lock, uses an attested absolute systemctl, and records the result; local-UID handoff is not a signed remote protocol.

Unclaimed publication may be retracted after claim deadline. Claimed cancellation/timeout does not prove systemd stopped; wait for terminal classification. Exact desired state wins even after a bad client return. Cancellation retains a host-owned decline or exact-restoration receipt and may reopen admission on that proof; completion alone does not prove restoration. For an ordinary failed transaction, exact prior state is restored; otherwise one typed compensation must prove exact restoration. Once compensation starts, repeated caller cancellation cannot abandon it; the inverse finishes before cancellation is propagated and admission can reopen. Rejected/contained or unresolved compensation fails closed. Direct containment latches only process lifetime; Reactor .processing/.contained survive and fence startup/effects until operator recovery. Inert private-systemd tests prove ordering, not Quadlet/Podman/GPU embodiment.

Designed resource policy

1. The Tipping Point (Whim Algorithm)

Whim is Designed. The current declared-conflicts policy (and its compatibility alias evict-idle) counts only selected evictees in total_metabolic_cost. It measures no VRAM, load time, context rebuild, thermal state, topology, bandwidth, transition peak, or tier substitution.

2. Model Tiering and Reservation

Resource-aware scheduling is Designed. Preload and idle-eviction settings are not implemented; explicit coexistence and persistent_resident are declarations, not capacity proof. Future measured ResourceEnvelopeRefs record idle, active, and transition-peak GPU memory, host RAM, devices and topology, disk, bandwidth, warm/unload time, concurrency, and measurement conditions. PlacementProfiles relate exact immutable capability profiles and required headroom on one host or Legion topology. A future CapabilitySetRequest@1 asks for one desired set of exact bindings; it is not a Coven, queue, or semantic selection request.

The planner may admit one serialized desired-world transaction only after fresh observation and reservation. Physical effects are not atomic: partial change must follow the existing drain/attest/actuate/converge sequence and exact compensation, restoration, or containment law. Durable local service attempts additionally transfer their exact reservation/fence before a Run parks. Current process-local leases cannot prove that handoff.

3. Delegated Provider Capacity

After semantic selection, a future policy may delay/decline delegated work for admitted physical or economic capacity, never rewrite task, choose provider, alter Graph, or manufacture capacity. A future ProviderCapacityPolicy must keep conservative/balanced/maximize behavior beneath the minimum authorized, provider, and configured ceilings; disablement or cooldown stops admission, and unknown quota cannot manufacture slots. No policy implementation, scheduler, observation, durable reservation/release, Provider Gate, or spend ledger is wired.

4. Swarm Lease Management

Swarm leasing is Designed. Current leases are process-local; expires_at has no enforcement, renewal, distributed fence, peer lease, preemption, handoff, or ghost sweep. Legion must reuse this admission/drain law.

5. Watchdog and Recovery

No autonomous watchdog or general repair engine exists. Pre-effect/drain failure reopens; hard failure seeks one exact inverse; post-submission cancellation waits for settlement/restoration; soft failure after invoking activation contains; unknown worlds are operator-owned. Conflict topology is Available in focused tests; transition protocol and Nexus tickets are Partial; Host Reactor protocol has inert private-systemd evidence; real host/model runtimes need Operator validation; resource-aware Whim, tiering, swarm preemption, watchdog, and repair are Designed.

Consequences