Skip to content

29. Observability

Context and Problem Statement

Runs, consent, dispatch, delegated work, runtime transition, and failure need evidence. Logs and telemetry cannot authorize or settle them; a shared id supplies correlation, not causality or completeness. This Covenant defines evidence law, Oculus, and Orb.

Requirements

  • Acting offices retain state/effect receipts; Oculus owns observations and rebuildable projections.
  • Every record identifies producer, subject, id, ordering domain, capture class; loss and uncertainty are explicit.
  • Content is allowlisted; secrets prohibited. Signals retain distinct authority and retention.
  • First Orb query is bounded authorized one-Run view. Eyes are one-way redacted exports without LychD authority.

Considered Options

Option Result
Mandatory metrics/trace stack Rejected: adds retention/control without evidence ownership.
Generic JSONB for all signal/body Rejected: collapses records, telemetry, privacy, schemas.
Native contract, optional Eyes Selected: LychD vocabulary remains authoritative and viewers replaceable.

Decision Outcome

Oculus is designed native evidence domain; Orb is its Altar instrument and scrying its use. Phoenix, Logfire, OpenTelemetry collector, or another viewer may be an external Eye; names never change ownership.

Exact implementation state

Structured logging and bounded Orb exist at State scopes. Native Oculus is Designed, with no telemetry adapter, ingestion, trace/metric store, retention, health query, cross-process bus, resource telemetry, or multi-Run query. Optional Phoenix only contributes service; application export is unproved.

1. Evidence Ownership and Correlation

Evidence classes are authoritative record (its responsible transition/effect office), bounded observation (producer/subject/method/times/freshness/limits), derivation (parents/algorithm/ uncertainty/invalidation), and interpretation/verdict (named criteria, e.g. Riddle). RunLedger owns Run status; consent, grants, jobs, host transitions, artifacts, and evaluations retain theirs. Step/RunEvent may report, never overwrite.

RunEvent identity is run_id plus UUID event_id; seq is monotonic per-Run emission order, ts producer time. Live channel makes contiguous in-process seq and one terminal; non-token events tee to Step ledger in order but best-effort append may gap; PostgreSQL enforces (run_id, seq) where used. Token deltas are never Step evidence. Seq orders one producer, timestamps no global order. Cross-office relations name typed Pattern revision, occurrence, grant, job, or transition id. Shared id/time is correlation only; trace context never authenticates or authorizes.

2. Native Service and External Eyes

Signal Current shape Authority
Run events in-process, 256 replay, best-effort non-token Step tee observation
Logs Structlog/stdlib human or JSON stderr diagnostic, not audit
Traces no producer or export adapter no ingest/export/retention/read
Metrics no producer/registry/store/query Designed
Orb bounded selected-Run projection read-only

Future Oculus exposes typed event/query contracts; clients do not query tables. Eye sees allowlisted export and has no canonical read-back. Phoenix legacy name = oculus compatibility cannot make it native Oculus.

3. Interior Evidence Without Mind Reading

Oculus may hold first-person testimony, operated telemetry, declared interpretation. None is hidden chain-of-thought: progress is testimony, tool/provider span observation, scored explanation versioned interpretation. Prompt/completion, retrieved context, tool bodies, provider exchange, media, credentials, and identity data are absent unless current policy admits; useful structural view and missing-evidence result must remain possible.

4. Delegated-Agent Evidence

Observe delegated runtimes only at admitted adapter boundary. LychD job state/policy/settlement/ artifacts/adoption differs from provider-reported usage/protocol; neither reveals planner, subagent tree, or private reasoning. Orb exposes at most 32 newest job summaries and 64 newest lifecycle events/job, state/result-or-artifact presence but no prompt/output/private error, with truncation explicit. Its bounded read asks the job store for one extra job and event as omission sentinels; database LIMITs select those newest suffixes before per-job event hydration, then restore creation/sequence order for projection. Raw future protocol artifact is untrusted bounded input; it cannot settle a job, authorize an effect, mutate Graph, or become training data by default.

5. Orb Read Models

Orb reads one Run by direct URL: retained bounded non-token events, separate ledger-head/page bounds, seq gaps, capture class process_local/durable_best_effort, omissions, and Pattern link only if pinned manifest validates. LOG is summarized without raw message; Nexus links only recorded transition ids. No run list, live tail, graph view, native Oculus model, cross-process completeness, artifact custody, annotation, or multi-Run field. SSE RESYNC instructs client to replace projection from snapshot; it is not browser-restored history. Viewing/filtering/layout never changes Run. Annotation would be separate authorized record, never retry/approval/cancel/publication/transition.

6. The Physical Body and Pulse

No Resource Snapshot exists for VRAM, thermal, power, ownership, pressure. Future node measurement carries units, method, age, errors, freshness; failed/stale means unknown, never free. Orchestrator consumes fresh truth under admission; Oculus may explain it. Rates/percentiles/trends are derivation, not grants/reservations/health verdicts/promotion thresholds.

7. Privacy, Retention, and Failure

Each class declares purpose/fields/classification/visibility/retention/export. Redact before serialization with policy version; Eye applies second filter; reject secret material. Privatization telemetry stores opaque decision/receipt ids, EvidenceDigest@1 keyed projections, categories/counts, policy version, failure stage, and gaps—never canonical raw payload/source digests, sensitive spans, or pseudonym reversal maps. Plain digests of low-entropy or stable private values are linkage oracles, not anonymization. Security owns the local-only decision and keyed evidence projection; Context owns receipts.

Current Orb allowlists structural fields and omits raw prompts, output, private errors, LOG messages. Shared logging has no general redaction/storage/rotation/retention/correlation contract; HTTP instrumentation disabling blanket body/header capture is not whole proof. A conforming Oculus bounds producer/subscriber queues and batching/flush/shutdown. Today each live Run subscriber and its replay window are bounded at 256 events; an overflow collapses pending deltas to an explicit snapshot-resync boundary rather than applying producer backpressure. Persist failure logs and may gap Step. Correctness records remain acting-office transactions: lost telemetry harms diagnosis, never proves success.

Consequences

Accepted

  • Rebuildable correlated observations leave authoritative records with their offices.
  • Capture class, gap, freshness, redaction, and uncertainty are evidence.
  • Eyes are replaceable.

Cost

  • Oculus requires ingest, retention, query, migration, health, backpressure.
  • Partial-order correlation and privacy boundary cost more than arbitrary spans.