Skip to content

llama.cpp

llama.cpp is a local model-serving engine with two materially different Soulstone shapes:

  • single starts with one declared model and serves one stable alias;
  • router starts without a model and activates declared models in process.

Launch and identity

The adapter translates the Rune into a deterministic launch plan. Explicit exec owns the whole command. In managed operation, explicit startup_mode selects single or router; auto uses argument and environment inference before falling back to single when model_path exists and router otherwise. Typed extra_args append last. Forced modes refuse known main-model source controls in extra_args, including model shortcuts and token lookalikes in value positions. Use model_path for forced single, or model directories and presets for forced router. Forced router also refuses model-source environment controls; forced single refuses a main Hugging Face repository in the environment because a remote preset can clear its model path. Both checks include variables projected from secret_env_files. These checks bound the supported command shape without reproducing upstream's full option grammar. GPU/CPU offload remains an exact profile fact, not a universal VRAM promise.

Mode metadata follows the final command and its projected environment, including secret_env_files. For main-model and router source options, environment values apply first; a CLI value replaces the same source option even when it is empty. A nonempty main model path or Docker repository selects single; model directories and presets do not override that main source. Without a main source the server is a router. For managed auto planning, router argument or environment inputs still take precedence over an otherwise unused typed model_path.

An effective main Hugging Face repository or a built-in main-model shortcut is refused: remote preset contents can change the server's mode, so local declaration cannot establish it. Use an explicit local model or router directory/preset. A model URL, HF filename, draft model, or --spec-default alone does not establish a main model or its provider identity.

Source, router, and alias controls require separate positional values; empty values are retained, while missing, dash-leading, and equals-form values fail validation. Known underscore aliases are normalized for interpretation without rewriting passthrough bytes. Place these controls before options whose arity LychD does not know: a later source or alias token is ambiguous and refused. Unknown trailing options remain operator-owned. This bounds source interpretation without reproducing every upstream option or treating a consumed option value as a source control.

With passthrough exec, the operator must keep its listening port and served alias aligned with the declared endpoint and model identity. Declaration compilation rejects a known --port or LLAMA_ARG_PORT value that conflicts with the resolved endpoint, including a final extra_args override of a managed command. LychD does not rewrite passthrough commands or infer unknown wrapper syntax; declare the endpoint and command consistently for a runnable profile.

Engine idle sleep is unsupported because readiness probes and autoload=false cannot fence its automatic wake. Managed argument vectors begin with --sleep-idle-seconds -1; omit sleep_idle_seconds or set it to -1. extra_args cannot contain any sleep control. Passthrough exec must begin with that exact disabling pair: either ["--sleep-idle-seconds", "-1", ...] for an image entrypoint, or ["llama-server", "--sleep-idle-seconds", "-1", ...] for a complete command. The executable token must be nonempty and cannot begin with -. Admitted bytes are not rewritten. Later sleep controls, duplicate disabling flags, underscore aliases, equals spellings, and misplaced or malformed disabling arguments fail validation. This deliberately bounds the command shape instead of inferring arbitrary option arity; a flag consumed as another option's value cannot supply admission. Custom wrappers that cannot honor this contract are outside the supported llama.cpp profile.

This contract was checked against upstream revision 434ddbbc0e30522e897670681e503b797c12b7c1: the router overlays CLI arguments on every model preset, while sleeping children still count as running for router autoload admission. The default is disabled, but the explicit command value also overrides sleep enabled in global, model, directory, or cached presets. Command validation does not attest an already-running service or external changes; operator validation must check the actual image revision and arguments. The pinned argument parser normalizes underscore option names and consumes values positionally; equals-form sleep flags are not accepted by that engine revision.

The same revision selects router mode only after resolving the main model sources. A later empty --model can clear an earlier path, and a Hugging Face remote preset clears the main-model settings. The forced-mode refusals prevent those inputs from contradicting the selected launch shape.

Aliases accumulate across LLAMA_ARG_ALIAS and every --alias or -a occurrence. Each value supplies comma-separated names; the supported profile trims ASCII whitespace and discards empty entries. An empty CLI alias does not clear an environment alias. The lexicographically first name in the combined set identifies the single model in the engine's inventory. Without any alias, single-model identity is the exact Docker repository or model path, including its directory and suffix. These rules follow the pinned alias handler and inventory identity; locale-dependent treatment of non-ASCII whitespace is not attested by this profile.

When managed planning selects single, it generates an alias. Explicit argument or environment aliases take precedence; otherwise served_model_id supplies it, followed by the generated model path's stem (basename without its final suffix), a Docker repository, or the Soulstone name. An explicit [[models]] catalogue does not disable alias generation. Without that catalogue, the adapter infers capability identity from the final launch inputs. Default model selection uses the first entry of the captured catalogue, including an explicit catalogue when supplied. An auto plan that omits the typed model and becomes single-model through an environment source does not add an alias; its identity follows the final inputs above.

Declared model ids must match the engine's live inventory; a healthy process alone cannot grant an absent model. The single-model probe validates /v1/models after /health, following the upstream Model Info API. A preset path projected through secret_env_files is not opened locally for catalogue or generation-default discovery. Declare the model catalogue when using that source; its actual contents and runtime behavior remain operator validation.

Context allocation

Managed context metadata comes from the generated command, including final extra_args. The catalogue derives a conservative per-request bound from positive total -c context divided by positive -np parallel slots, reduced by any explicit --kv-unified-per-slot cap. Thus -c 8192 -np 4 admits a 2048-token bound even when a unified KV layout might permit more for one request. Generated flags override environment and preset values. Explicit exec without known positive context and slot counts leaves that derived bound unknown; automatic or malformed command/environment context inputs do not silently fall back to a lower-priority value. These flags follow the upstream server parameters. The bound describes configured allocation, not the loaded model's training context or a live hardware measurement. Explicit generation overlays remain request policy; they do not change the engine's allocation.

Readiness and control

Router models are dynamic capabilities. Each model's inventory status determines its readiness: clean unloaded admits activation, loading means WARMING, and loaded plus ready health supplies warmth. Missing declared models and malformed inventories fail closed; failed, sleeping, downloading, or unknown statuses cannot authorize activation. One model's health cannot establish a sibling's state. Activation rechecks inventory and does not repeat a load already in progress. Inference and targeted health requests carry autoload=false; with engine idle sleep disabled, a missing loaded model cannot silently trigger router loading or eviction. Dynamic activation is not a restart; reclaiming a conflicting physical runtime remains an Orchestrator transition.

The current repository proves planning, discovery, capability derivation, and model-load control. Model unloading is not a separate adapter operation; whole-runtime shutdown belongs to Orchestrator. A real engine/GPU/model result remains operator validation.

Control receipts are intentionally literal: model-load succeeds only on JSON boolean true, not 1 or "true". Known numeric preset keys use their canonical integer or floating type; boolean words are ignored rather than silently becoming 0 or 1. Treat those refusals as malformed provider/configuration evidence, not as successful control.

See Soulstone Disciplines, llama.cpp, and the repository support files.