# SwarmKit — llms-full.txt > The expanded, single-file corpus: the full text of the playbook, all 11 artifact > references, and the core design notes inlined so an LLM can ingest everything in one > fetch — no link following. For the compact, link-first map, see llms.txt (https://delivstat.github.io/swarmkit/llms.txt). > > GENERATED by scripts/build_llms_full.py from the source docs — do not edit by hand; > regenerate with `python scripts/build_llms_full.py`. ## Contents 1. **The map (llms.txt)** - SwarmKit — `llms.txt` 2. **The playbook** - Building swarms — the complete playbook — `docs/site/guides/building-swarms.md` 3. **Memory and decision-skill bindings** - Memory and decision-skill bindings — `docs/site/guides/memory-and-decision-skills.md` 4. **Artifact references** - Topology — `docs/site/reference/topology.md` - Workspace — `docs/site/reference/workspace.md` - Archetypes catalogue — `docs/site/reference/archetypes.md` - Skills — `docs/site/reference/skills.md` - Funnel — `docs/site/reference/funnel.md` - Contract — `docs/site/reference/contract.md` - Role registry — `docs/site/reference/role-registry.md` - Trigger — `docs/site/reference/trigger.md` - Executor adapter — `docs/site/reference/executor-adapter.md` - Model provider — `docs/site/reference/model-provider.md` - Approval policy — `docs/site/reference/approval-policy.md` 5. **Operate references** - CLI commands — `docs/site/reference/cli.md` - Serve mode — `docs/site/reference/serve.md` - HTTP API — `docs/site/reference/http-api.md` - Connections: credentials, remote MCP servers and OAuth — `docs/site/reference/connections.md` - Driving SwarmKit from your application — `docs/site/reference/orchestrator-integration.md` - Governed memory — `docs/site/reference/governed-memory.md` - Workspace memory — `docs/site/reference/workspace-memory.md` - Events — `docs/site/reference/events.md` - Storage — `docs/site/reference/storage.md` - Telemetry configuration — `docs/site/reference/telemetry.md` - Workspace environment configuration — `docs/site/reference/env-config.md` 6. **Guides** - Validating a topology's output — `docs/site/guides/validating-topology-output.md` - Getting an image to a model — `docs/site/guides/getting-an-image-to-a-model.md` 7. **Design notes (the depth)** - Gate funnel (per-artifact quality gate composition) — `docs/site/design-notes/gate-funnel.md` - Reading a gate, and approving without a saga — `docs/site/design-notes/gate-state-and-deferring-approval.md` - Taking the pipeline out of the runtime — `docs/site/design-notes/extracting-the-pipeline.md` - Declarative skill prerequisites (`requires:`) — `docs/site/design-notes/skill-prerequisites.md` - Stopping a run — `docs/site/design-notes/stopping-a-run.md` - Funnel verification strength — design — `docs/site/design-notes/funnel-verification-check.md` - Multi-party approval sets (governance) — `docs/site/design-notes/multi-party-approval.md` - Integration-contract registry — `docs/site/design-notes/contract-registry.md` - Feature Request: Pluggable Executor Abstraction at the Archetype Level — `docs/site/design-notes/executor-abstraction.md` - Structured output governance + auto-correction — `docs/site/design-notes/structured-output-governance.md` - GovernanceProvider interface — `docs/site/design-notes/governance-provider-interface.md` - Model provider abstraction — `docs/site/design-notes/model-provider-abstraction.md` - DAG Dependency Graph for Agent Topologies — `docs/site/design-notes/dag-dependency-graph.md` - Command packs — `docs/site/design-notes/command-packs.md` - A2A interop — every topology is an A2A agent, and a remote A2A agent is a skill — `docs/site/design-notes/a2a-interop.md` - A2A federation — SwarmKit agents that hand back their record — `docs/site/design-notes/a2a-federation.md` - Declarative model providers — a provider is data, a wire format is code — `docs/site/design-notes/declarative-model-providers.md` - One storage service — `docs/site/design-notes/storage-service.md` - The audit is a write-through journal — `docs/site/design-notes/audit-event-journal.md` - Memory by default — `docs/site/design-notes/memory-by-default.md` - Taking communication out of the runtime — `docs/site/design-notes/extracting-the-channels.md` - Skill catalogue — `docs/site/design-notes/skill-catalogue.md` - Skill registry — the `swarmkit skill` command — `docs/site/design-notes/skill-registry.md` - Fleet control plane — `docs/site/design-notes/fleet-control-plane.md` ══════════════════════════════════════════════════════════════════════════════ # Part 1 — The map (llms.txt) ══════════════════════════════════════════════════════════════════════════════ ══════════════════════════════════════════════════════════════════════════════ ## Source: llms.txt ## Published: https://delivstat.github.io/swarmkit/llms.txt ══════════════════════════════════════════════════════════════════════════════ # SwarmKit > SwarmKit is the open-source AI platform runtime: agents, tools and governance defined as data (YAML/JSON) and run under real gates with every step recorded — from a single agent to a multi-agent swarm. Topology is data; skills are the universal extension primitive; governance is built in via Microsoft AGT; swarms grow by observing their own capability gaps and authoring new skills through conversation with human approval at every step. This file is also published on the docs site at . New here? Read **[Building swarms — the complete playbook](https://delivstat.github.io/swarmkit/guides/building-swarms/)** for the ordered build recipe, then this file for the compact map with inline schemas. **Two tiers.** This `llms.txt` is the compact, link-first map. Its companion **[`llms-full.txt`](https://delivstat.github.io/swarmkit/llms-full.txt)** inlines the *full text* of the playbook, all 11 artifact references, and the core design notes into one file — fetch it once and you have the entire corpus, no link-following. Use `llms.txt` to navigate; use `llms-full.txt` when you want everything in context at once. **Status:** runtime v1.256.0 (1.0.0 was 2026-04-26); `swarmkit-schema`, `swarmkit-webui` and `swarmkit-control-plane` version independently. Phases 1–5 of the implementation plan shipped: the runtime, CLI and `swarmkit serve`; harness executors (M19); the topology canvas (M20); the fleet control plane. **Sequencing lives in your application, not in SwarmKit** — the bundled pipeline (`kind: StageGraph`, the saga controller, `swarmkit orchestrator`, `swarmkit pipeline`, `POST /pipelines/*`) was removed in 1.189.0; what stays is one bounded governed run, its gates, and its record, with `examples/pipeline-orchestrator/` as a reference application that imports no runtime module. What SwarmKit is, in one line each: reusable `Funnel` gates (`validate → judge → review → approve`; only humans approve; a gate **defers** the run and `POST /jobs/{id}/resume` continues it), multi-party approval over a `RoleRegistry`, `Contract` locks as a checked vocabulary, correlated runs and an append-only audit; skills as the one capability primitive (`mcp_tool`, `llm_prompt`, `composed`, `command`, `agent`) with `requires:` ordering enforced at the permission seam; 12 declarative model providers over four wire-format families; MCP with Docker sandboxing, permission tiers and per-tool `effects`; harness executors as declarative `adapter.yaml`; memory on by default (governed + workspace); one storage service (SQLite or Postgres) behind every store; OpenTelemetry, intent-drift detection, audit redaction, circuit breakers; a self-hosted fleet control plane; the `swarmkit-skills` catalogue reachable from `swarmkit skill` and the portal. **11 canonical artifact schemas** (topology, workspace, archetype, skill, funnel, contract, role-registry, trigger, executor-adapter, model-provider + the embedded approval-policy). Reference libraries: 3 reference topologies, 27 skills, 16 archetypes, plus the SDLC example workspace. Recent changes are listed by version in the next section; the [changelog](https://delivstat.github.io/swarmkit/releases/changelog/) has every release. ## Recent changes (newest first, by runtime version) - **1.256.0** — A2A budget forwarding (a2a-federation.md, slice 2). When a SwarmKit `agent` skill calls a remote SwarmKit agent, it now forwards its **remaining allowance** (envelope minus spent — `max_cost_usd`, `max_turns`) in `message.metadata.swarmkit.budget`; the callee installs it as the child run's circuit-breaker limits, taking the **stricter** of the forwarded budget and its own config so a caller can only cap the child, never loosen it. Advisory across the trust boundary — a SwarmKit callee honours it (its Agent Card now advertises `honors_budget: true`), a non-SwarmKit agent ignores the extra key. Enforcement is the callee's; the caller records what it asked. Threaded through `run(budget_override=...)`. Usage-return (slice 1) already shipped. - **1.255.0** — worker fairness (worker-fairness.md, follow-up to the scaling review). Workers no longer share one undifferentiated FIFO: each queued job carries a `job_class` (`model` | `harness`, derived from whether any agent in the topology runs on a harness executor) and a `priority` (from a `priority` label, higher first). `swarmkit worker --class model|harness|any` claims only its class, so a burst of long harness runs can't starve short model runs; the claim orders by `priority` then `created_at`. Two additive `jobs` columns; all-in-one and no-`--class` behaviour unchanged. Per-tenant quotas/weighted fairness named as a further follow-up. - **1.254.0** — `serve --profile production` (production-profile.md, from the scaling/security review's "production defaults too permissive"). A fail-closed startup preflight that refuses to start — listing every gap at once — unless the deployment is safe to expose: real auth (not `none`/anonymous, on any bind), not `--insecure`, a persistent `SWARMKIT_OAUTH_KEY` (so the token-encryption key does not regenerate on restart and invalidate stored tokens), every declared `mcp_servers` entry `sandboxed: true`, and no wildcard CORS. A deployment-time assertion, not a request-time gate (governance is already deny-by-default); `--profile standard` (default) is the unchanged permissive behaviour. `create_app(profile="production")` so embedders inherit it. - **1.253.0** — topology `input_schema` (input-schema.md). An optional JSON Schema (draft 2020-12) on a topology that the caller's input must satisfy *before the run starts* — validate-and-reject at the single choke point (`WorkspaceRuntime.run`), so every entry inherits it: 422 over HTTP, non-zero exit on the CLI, a JSON-RPC error over A2A, and no LLM spend on a malformed request (serve rejects at submit, so it never becomes a job). An object schema requires JSON input; `{"type": "string"}` accepts plain text. Emits `input.validated`/`input.rejected` audit events. Symmetric to `output_schema` but validate-and-**reject** (a caller cannot be re-prompted mid-run). Added to `topology.schema.json` (schema 1.46.0 / TS 0.23.0); opt-in, a topology that omits it is unchanged. - **1.252.0** — worker fencing tokens (worker-execution.md). Every durable jobs-row write from a run is now fenced on `worker_id`, not just the final `complete()`: a worker threads its id through `execute_job` (`update_job(..., fence_worker_id=me)` → `WHERE id AND worker_id = me`), so a zombie (lease expired, run reclaimed) can't even transiently flip the new owner's row on its intermediate/terminal writes. All-in-one serve passes no token (writes stay unconditional). Guarantee: at-least-once execution, effectively-once durable record. Remaining unfenced surface (checkpointer per-node writes, run_usage rows) named for a strict exactly-once follow-up. - **1.251.0** — queue observability (queue-observability.md, follow-up to the scaling review). Two additive `jobs` columns — `claimed_at` (when a worker claimed the run) and `started_at` (when execution began) — decompose the lifecycle with `created_at`/`completed_at`: queue wait vs execution latency. New `GET /queue/stats` and `swarmkit queue-stats ` report backlog depth, oldest-unclaimed age, queue-wait/execution p50/p95 over recent completions, and depth by topology (engine-agnostic — percentiles computed in Python, so SQLite works too). The portal Jobs-view stat strip lands with the batched UI slice. - **1.250.0** — worker-execution hardening (from an external scaling review). `PostgresJobQueue.complete()` is now **fenced on `worker_id`**: a worker whose lease expired and whose run was reclaimed by another can no longer complete it and reset the new owner's lease (the reclaim→double-execution cascade). Delivery guarantee stated precisely — at-least-once at the checkpoint boundary. And `serve --role api` **bounds the queued backlog**: `--max-queue-depth` (default 10,000; 0 = unbounded) returns 429 once that many runs are `queued`, instead of accepting into an unbounded queue no worker may reach. Parked (deferred/stopped) runs release their worker and resume later as a fresh claim, so they consume queue rows, never worker slots. - **1.249.0** — the API tier's SSE stream now follows a worker's job to completion. After the API/worker split, `GET /jobs/{id}/stream` on a `serve --role api` process held no live copy of a worker-run job and replayed a `queued` snapshot then closed; it now follows the durable row (re-reading on an interval, emitting new events) until terminal, so the API tier relays a worker's progress. Plus Run 5 in `docs/site/reference/load-and-scale.md`: the payoff measurement — `serve --role api` + N `swarmkit worker` on Postgres, throughput scales with worker count (5.9× at 8 workers end-to-end, corroborated by an isolated backlog-drain), lifting the single-event-loop ceiling Runs 1–4 found; sublinear (shared Postgres + one box's cores) and honest about it. The load driver (`examples/loadtest/`) gained an SSE `--stream` mode and a fixed elapsed-time throughput fix. - **1.248.0** — the API/worker split for worker execution (`design/details/worker-execution.md`, slice 2). `swarmkit serve --role api` accepts and enqueues a run as `queued` (resolving topology and attachments up front, so a bad request is still a 4xx on submit) and returns without executing it; the new `swarmkit worker` command runs a loop that claims a queued job, executes it on its own event loop and DB engine via the exact serve path (`execute_job`, so status/output/usage/audit are identical), heartbeats to hold its lease, and reclaims runs abandoned by dead workers (which resume from their checkpoint). Throughput scales with worker count, past the single-event-loop ceiling the load benchmark found. The all-in-one `swarmkit serve` is unchanged and stays the default; `--role api` requires Postgres. Mind the connection budget: physical connections are `workers × (pool + overflow + checkpointer)` — size `SWARMKIT_STORE_POOL_SIZE` down or front Postgres with PgBouncer past a handful of workers. - **1.247.0** — the durable job queue for worker execution (`swarmkit_runtime.queue`, `design/details/worker-execution.md`), slice 1: a `JobQueue` interface and `PostgresJobQueue` over the `jobs` table — atomic claim (`SELECT … FOR UPDATE SKIP LOCKED`), lease + heartbeat, `reclaim_expired` (an abandoned worker's run returns to the queue with `attempt` bumped and resumes from its checkpoint, not restarted), and a clean refusal on SQLite (single-writer, cannot be claimed across processes). New `jobs` columns `worker_id`/`lease_until`/`attempt` (additive). No serve behaviour change yet — the API/worker split and a worker command are the next slice. - **1.246.0** — concurrent first-runs build the LangGraph checkpointer once (an `asyncio.Lock`), not in a race: on SQLite, two tasks creating the checkpoint tables at the same time hit `database is locked` — the same single-writer contention that shapes the SQLite/Postgres differences elsewhere. Fixes a flaky `test_concurrent_first_runs_compile_once`. - **1.245.0** — the store connection pool is configured by the storage service (the single owner of storage config), not inside `make_engine`, which is now a pure factory taking `pool_size`/`max_overflow`. Same env knobs (`SWARMKIT_STORE_POOL_SIZE`/`_MAX_OVERFLOW`) resolved in one place; the shared serve engine gets the server-oriented default (20/10), and direct engine constructors get the driver default. - **1.244.0** — the store connection pool is configurable (`SWARMKIT_STORE_POOL_SIZE`, default 20; `SWARMKIT_STORE_MAX_OVERFLOW`, default 10; Postgres only, one shared pool per database), and the write-through audit `INSERT` runs off the event loop (`asyncio.to_thread`). Load Run 3 (load-and-scale.md): moving only the audit write off the loop did not shift the throughput knee — a run does several other synchronous store writes on the loop (job create/update, usage, trace), so the full win needs all of them off-loop or an async driver; a bigger pool is necessary for scale-out, not sufficient alone. - **1.243.0** — the compiled LangGraph graph is cached per topology and reused across runs (it was rebuilt every run — pure CPU on the serve event loop); a reload builds a fresh runtime, so the cache is invalidated for free. Measured against the load baseline (load-and-scale.md, Run 2): compile caching is correct and removes redundant per-run CPU but did not move the throughput knee — the dominant cost is the synchronous per-event store/audit writes on the loop (default pool size 5), which is the next optimization. - **1.242.0** — a load-test harness and the first published NFR numbers (`examples/loadtest/`, `docs/site/reference/load-and-scale.md`): `SWARMKIT_MOCK_LATENCY_MS` (+ jitter) makes the mock provider sleep per call so a benchmark measures the runtime, not an instant mock; three topology shapes (tiny/typical/mcp-heavy) and a dependency-light driver (ramp + admission storm, RSS/fd sampling from /proc). Headline finding: a single serve process is bounded by the synchronous per-run work (compile + governance + write-through audit persist) on its one event loop — throughput plateaus ~2.5–5 runs/s and latency grows linearly past a knee near c=5–10, so scale horizontally (N processes on one Postgres) for throughput; `max_concurrent` bounds latency, it does not buy throughput. Admission (429) is correct but slow under load. - **1.241.0** — `swarmkit upgrade` (`design/details/upgrade-command.md`): upgrades the local install in place, re-applying the extras it detects (`[ui]`, `[postgres]`, provider SDKs), after showing any breaking changes between the installed and target version and asking before it installs — a breaking upgrade must be a deliberate yes. `--check` reports and exits non-zero if behind (never installs); `--to X.Y.Z` pins; Docker and unrecognised installs are refused with the exact command rather than driven. Breaking versions are a curated, bundled list (`_breaking_changes.py`), seeded with 1.189.0 (pipeline removed) and 1.199.0 (readonly effects). - **1.240.0** — A2A federation between SwarmKit instances (`design/details/a2a-federation.md`): a SwarmKit callee's Agent Card advertises a federation extension (so the portal's Add-remote-agent badges it and `GET /api/a2a/probe` returns `is_swarmkit`), and a completed/failed/canceled A2A task carries `metadata.swarmkit.{run_id, usage, observability}` — the caller records an `a2a.remote_usage` audit event linking the two runs and the remote's token/cost, and can pull the remote's events/audit by run id. A non-SwarmKit remote ignores the extra keys. Passing a budget on the call so the callee honors it is the next slice. - **1.239.0** — the audit is a write-through journal (`design/details/audit-event-journal.md`): each event is persisted the moment it is recorded, not batched at the run boundary, so a run that crashes (SIGKILL, OOM, power loss) still leaves its trail up to the crash and `GET /events` shows it. Before this, a hard kill wrote nothing while the checkpoint survived — recovery was durable, the record was not. The end-of-run write stays as an idempotent completeness net (the store dedups on event id). - **1.238.0** — a `kill -9` mid-harness regression test (`test_kill9_recovery.py`, `design/details/failure-path-evaluation.md`): a run killed with SIGKILL resumes from its checkpoint and completes (the interrupted node re-runs, the checkpointed one does not). It documents a real limit — the audit trail is flushed at the run boundary, so a hard kill loses the killed attempt's audit while the checkpoint stays durable and the resumed run's record is complete. Opt-in `SWARMKIT_MOCK_DELEGATE=1` makes the mock provider delegate to children, so a `swarmkit run` on the mock traverses a multi-agent topology (off by default; nothing that counts calls changes). - **1.237.0** — `swarmkit storage status`, `swarmkit system` and `GET /storage` report a store whose configuration cannot be honoured (a postgres backend whose `${VAR}` URL is unset) as an `UNRESOLVED` row plus one sentence per cause, exit 2, and `system` still prints the environment section — instead of a traceback from the diagnostic command. A run still refuses. - **1.236.0** — a workspace reload (`POST /reload`, a portal edit, `skill add` over HTTP) hands MCP sessions to one owner task and swaps the runtime under a lock: the new runtime's servers start before the old one's close, and a reload with `mcp_servers` configured no longer takes serve down. The portal's **Skills → Library** tab (webui 0.24.0) is the catalogue with a search box: Add shows both fragments before writing, a `SKILL.md` pastes in, Check runs from a button. - **1.235.0** — `swarmkit skill` (`design/details/skill-registry.md`): `search`/`list --available` over the `swarmkit-skills` catalogue (verified nightly; cached a day; `SWARMKIT_SKILLS_CATALOGUE` points at a checkout or mirror); `add ` writes the skill file and the `mcp_servers` entry after showing both (`--dry-run`, `--yes`; comment-preserving, rolled back if the workspace would not load, idempotent); `import ` converts an Agent Skills file to an `llm_prompt` skill; `check` starts each `mcp_tool` skill's server and confirms the tool exists; `remove` refuses while an agent or archetype holds the skill. Same over HTTP at `/api/skill-catalogue` and `/api/skills/{add,import,check}`. A skill declares the runtime floor it needs (`provenance.requires_runtime`) and is refused, naming the version, when it is not met. - **1.234.0** (control-plane 0.50.0, fleet UI 0.14.0) — `swarmkit connect --join-code` saves its credential (`~/.swarmkit/connect/.json`, 0600) and a later start with no flags resumes; `/usage` `by_model` carries the `provider` that billed each model; the panel refuses a command whose args the connector could not address (400 naming them); `/fleet/state` carries `a2a` and the fleet inventory shows it. - **1.233.0** (schema 1.45.0) — **memory is on by default** (`design/details/memory-by-default.md`): a workspace that says nothing binds `memory-reader` before every agent and `memory-writer` after (advisory) and gets the bundled `governed-memory` + `memory-reconcile` skills; a `memory:` block tunes the reader/writer or switches everything automatic off (`enabled: false`); an explicit binding is used as written; writing curated memory is still a per-agent grant; `GET /memory/config` and the portal's Memory page show what is in force. - **1.232.0** (control-plane 0.49.0, fleet UI 0.13.0; `design/details/control-plane/28-operator-identity-to-instance.md`) — a fleet panel resolves a multi-party approval **as the signed-in operator**: under a human-issued `approve-as` membership the instance honours a signed assertion of the operator's OIDC subject (`X-Fleet-Actor`, bound to the item and a 300 s window), the role registry decides membership, and the audit records both the person and the fleet that relayed the click. - **1.231.0** — `swarmkit serve` needs no extra: JWT auth and cron triggers are base dependencies and `[serve]` is an empty, deprecated alias; `[ui]` is the portal. - **1.230.0** (control-plane 0.48.0, fleet UI 0.12.0) — a fleet sync pulls the instance's skill gap log (`GET /gaps`) and audit tail (`GET /audit?since=`, cursored per instance) into the panel's Gaps and Runs views; `/fleet/state` names funnels, contracts and role registries — funnels and contracts adopt and deploy, role registries adopt only. - **1.229.0** (control-plane 0.47.0) — a fleet deploy writes the adopted file's text verbatim (comments and layout intact, refused if it does not parse to the signed content); `/fleet/state` carries each artifact's `yaml`; the panel's drift is hash-compared against what a sync observed. - **1.228.0** (control-plane 0.46.0, fleet UI 0.11.0) — a funnel's multi-party role-task is resolved only through `POST /review/{id}/resolve` as a member of the role (the generic approve/reject answer 409); the fleet panel resolves it with an outcome, relays the instance's refusal, and reads `deferred`/`stopped`/`interrupted` as what they are. - **1.227.0** — everything a run records goes through the storage service: conversations, workspace memory and the skill gap log moved off `.swarmkit/` files onto the configured store next to jobs, audit and governed memory. Serve's MCP endpoint answers at `/mcp/` behind the server's auth; a signed webhook (`hmac`, `bearer`, `api_key`) is admitted past the API-key gate and verified by its trigger; a cron trigger's `config.timezone` and `config.input` are honoured; two versions of one topology live side by side as `name` and `name@version` for the canary router; an agent calling a tool it does not hold gets a usable result, a `skill.gap` audit event, and a row in `swarmkit gaps`. - **1.221.0–1.225.0** — A2A both ways: the server (agent card + `POST /a2a`, opt-in) and the `agent` skill type (a topology here as a child run, or a remote agent through its card; `pack:workspace`; the portal's remote agents; harness nodes call agent skills through the gateway). Section below. - **1.216.0** — communication is the application's: `GET /events?after=` is the durable log, `events:` sinks push best-effort, a resolved gate resumes its run; the Slack/Discord/Telegram providers left the runtime. - **1.199.0** (BREAKING) — `permission: readonly` decides by declared `effects`, not tool-name substrings. Section below. - **1.197.0–1.198.0** — command packs (`implementation.type: command`) and bulk grants (`pack:`, `server:`). Section below. - **1.189.0** — the bundled pipeline layer removed. Section "Sequencing" below. ## Start here - [Docs site](https://delivstat.github.io/swarmkit/): the published documentation (this llms.txt lives at `/llms.txt` there). - [Memory and decision-skill bindings](https://delivstat.github.io/swarmkit/guides/memory-and-decision-skills/): the two memories, turning governed memory on, and `enabled` vs `required`. - [Building swarms — the complete playbook](https://delivstat.github.io/swarmkit/guides/building-swarms/): the ordered, step-by-step build recipe from one agent to a governed multi-app delivery flow. **Read this first if you are building a swarm.** - [SDLC example (video walkthrough)](https://delivstat.github.io/swarmkit/sdlc-example/): the worked reference — a complete software-delivery lifecycle as data, toured on-screen. Recorded while SwarmKit still bundled a sequencer; the topologies, funnels, archetypes and gates are current, the stage-graph/controller parts are not. - [Driving SwarmKit from your application](https://delivstat.github.io/swarmkit/reference/orchestrator-integration/): the HTTP contract for sequencing runs yourself — correlation, gates, defer/resume, artifacts. - [README](https://github.com/delivstat/swarmkit/blob/main/README.md): project overview, what works today, milestone progress. - [CLAUDE.md](https://github.com/delivstat/swarmkit/blob/main/CLAUDE.md): repo-wide invariants, feature-delivery workflow, release checklist. - [SwarmKit Design v0.6](https://github.com/delivstat/swarmkit/blob/main/design/SwarmKit-Design-v0.6.md): the authoritative architecture. §5 (Core Concepts), §6 (Skills), §8 (Separation of Powers), §14 (Runtime), §18 (MCP Integration). - [Implementation plan](https://github.com/delivstat/swarmkit/blob/main/design/IMPLEMENTATION-PLAN.md): the phased roadmap to v1.0, updated as features land. ## Build a swarm (the recipe) The [playbook](https://delivstat.github.io/swarmkit/guides/building-swarms/) walks each step with runnable artifacts; the compressed arc: 1. **Scaffold** — `uv tool install "swarmkit-runtime[ui]"` (uv is the recommended install; the server is part of the runtime; `[ui]` is the portal it hosts — absent, `swarmkit serve` runs headless; `[postgres]` the Postgres backend; the bare package is the CLI), then `swarmkit init` (conversational authoring) or hand-write a `Workspace`. 2. **One agent** — a `Topology` node instantiating an `Archetype` (model/prompt/IAM defaults). `swarmkit run `. 3. **A skill** — attach a `Skill` (capability / decision / coordination / persistence) with a structured `outputs` schema. 4. **Many agents** — add nodes + `depends_on`; coordinators use structured delegation (`create-task-plan`). 5. **Tools** — connect MCP servers in the workspace; every tool call is governed. 6. **Governance** — decision skills + `GovernanceProvider`; reserved human-only scopes; append-only audit. 7. **A quality gate** — a `Funnel` (`validate → judge → review → approve`); automated layers filter + retry, only humans approve. 8. **A harness node** — an archetype `executor: { kind: harness, ref: claude-code }` for a real diff-producing coding agent. 9. **Order the tool calls** — `requires:` on the agent, so a guarded skill is refused until its prerequisite has actually run (a rule in a prompt is a request; this is mechanism). 10. **Multi-party approval** — a `RoleRegistry` + a gate's embedded `ApprovalPolicy` (quorum + four-eyes floor). 11. **Sequence it from your application** — start each run over `POST /run/{topology}` with a shared `correlation_id`; a gated run parks as `deferred` and `POST /jobs/{id}/resume` continues it. SwarmKit runs the bounded work and keeps the record; your application decides what comes next (`examples/pipeline-orchestrator/`). 12. **Trigger + serve + observe + grow** — a `Trigger` delivers a signed webhook into `swarmkit serve`; trace/why/ask observe the run; `swarmkit gaps` → author the missing skill. Then validate the whole thing before running it: `swarmkit validate .` — add `--require` to fail on configuration nothing reads, and `--require-verified` to fail when a topology root's output is checked by nothing. ## CLI commands Author & run: `init` (scaffold via conversation), `edit`, `validate` (`--require` reachability, `--require-verified` funnel strength), `run` (`--verbose`, `--dry-run`, `--correlation-id`, `--label k=v`, `--supersedes `, `--save-artifact`, `--resume` — the last checkpointed run), `chat` (multi-turn, `--resume`), `conversations` (`--pick`), `eval` (score a topology against an eval-set), `checkpoints` (resume checkpointed runs). Serve & connect: `serve` (HTTP: async jobs, SSE, auth, MCP, webhook ingress, canary), `connect` (edge poll connector). Observe: `status`, `logs` (`--run-id`, `--agent`, `--format markdown`), `trace` (call graph + token counts), `why` (LLM post-mortem), `ask` (`--run` scoping), `debug` (local prompt ring buffer), `stop ` (ask a run to stop at its next agent boundary — cooperative, keeps what it has done, resumable, and works across processes because it writes a durable flag). Grow & govern: `gaps` (recorded capability gaps), `review list/show/approve/reject/resolve` (the human relay/approval inbox; `list --kind role_task --gate ` narrows it; `resolve --as --approve|--reject|--changes-requested -m "why"` casts a multi-party role-task, checked against the role registry; `approve`/`reject` take `-m` too). Artifacts: `artifacts get|list` (read a run's saved output by ref). Knowledge & packaging: `knowledge-server`, `knowledge-pack` (`--lean` ≈190k tokens — overview, generated reference, schemas, design doc, guides; full ≈610k adds every design note, historical ones last), `docs-reader`, `install` / `packages` / `publish` (expertise packages). Providers & adapters: `providers list|show` (every declared model provider, its family, whether its key is set), `adapters list|show|approve|build` (harness adapters and their launch-approval status). Memory: `memory add|get|search|quarantine|resolve` (governed memory; `add` writes through the same reconcile path an agent uses). Trust: `trust list|apply|clear` (allowlist changesets proposed by repeated approvals). Storage & system: `storage status|migrate`, `system`, `upgrade` (upgrade the local install — detects method + extras, shows breaking changes in range, asks before installing; `--check`, `--to`, `--yes`). Fleet: `fleet enroll-token|memberships`, `auth token`. Delivery checks: `cited-change`, `slice-check`, `comprehension`. All 69 commands with their help lines are generated from the CLI itself into [the CLI reference](https://delivstat.github.io/swarmkit/reference/cli/). ## Schemas Eleven canonical artifact schemas (JSON Schema 2020-12) — ten standalone artifact kinds plus the embedded `approval-policy`. Each has a published reference page (linked below). - [topology.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/topology.schema.json): a bounded swarm run — [reference](https://delivstat.github.io/swarmkit/reference/topology/). - [workspace.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/workspace.schema.json): the root manifest — [reference](https://delivstat.github.io/swarmkit/reference/workspace/). - [archetype.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/archetype.schema.json): a reusable agent template — [reference](https://delivstat.github.io/swarmkit/reference/archetypes/). - [skill.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/skill.schema.json): the capability-extension primitive — [reference](https://delivstat.github.io/swarmkit/reference/skills/). - [funnel.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/funnel.schema.json): a reusable per-artifact quality gate — see Funnel below; [reference](https://delivstat.github.io/swarmkit/reference/funnel/). - [contract.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/contract.schema.json): an integration contract — the agreed interface between apps, and the checked vocabulary a sequencer's locks name; see Contract below; [reference](https://delivstat.github.io/swarmkit/reference/contract/). - [role-registry.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/role-registry.schema.json): named roles → member identities + the scopes they confer; how approval rules resolve to people; [reference](https://delivstat.github.io/swarmkit/reference/role-registry/). - [trigger.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/trigger.schema.json): an external event source that starts a topology or delivers a signed webhook event; [reference](https://delivstat.github.io/swarmkit/reference/trigger/). - [executor-adapter.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/executor-adapter.schema.json): a declarative `adapter.yaml` running a coding harness as a node — data, not per-harness Python; [reference](https://delivstat.github.io/swarmkit/reference/executor-adapter/). - [model-provider.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/model-provider.schema.json): a declarative model provider — base URL, auth env var, catalogue, narrow-only capabilities, layered over one of four wire-format families (`openai-compatible`, `ollama`, `anthropic`, `google`); a new endpoint is a YAML file, no Python — [reference](https://delivstat.github.io/swarmkit/reference/model-provider/). - [approval-policy.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/approval-policy.schema.json): **embedded config** (no `kind`) inside a gate's `approve:` — multi-party rules, quorum, four-eyes floor; [reference](https://delivstat.github.io/swarmkit/reference/approval-policy/). ## Funnel (per-artifact quality gate) A **Funnel** is a first-class artifact (`kind: Funnel`) — a reusable per-artifact quality gate that chains *structured-output validation → LLM-as-judge → optional harness review → multi-party human approval* into one composition. Referenced by id from a topology node's `funnel:` field, so the same gate applies to many nodes/stages. Every layer is optional except `approve`; present layers always run in the fixed order `validate → judge → review → approve`. The automated layers **filter and drive a bounded retry loop but never decide** — the only exit is through `approve`. The control flow is compiler-owned and fixed; a funnel configures the layers, it does not rewire the graph. On retry exhaustion the funnel escalates to a human with the last critique attached — it never drops the requirement or silently advances. Layers: `validate` (deterministic, native structured-output validation + field-specific auto-correction — the judge never sees malformed input), `judge` (a decision skill scoring against a rubric with a `threshold` and bounded `max_retries`), `review` (optional heavyweight harness reviewer; findings at or above `route_back_at` retry, the rest attach to the human task), `approve` (required multi-party approval set — rules, quorum, `min_distinct_approvers`, `exclude_author`). Minimal (degenerate to a plain multi-party gate): ```yaml apiVersion: swarmkit/v1 kind: Funnel metadata: id: design-signoff name: Design Sign-off description: A plain multi-party human approval gate on the design artifact. approve: rules: - scope: design:approve roles: [tech-lead] quorum: all provenance: authored_by: human version: 1.0.0 ``` Full (all four layers): ```yaml apiVersion: swarmkit/v1 kind: Funnel metadata: id: consolidated-design-approval name: Consolidated Design Approval description: Full four-layer gate — validate, judge, architect review, then multi-party approval. validate: schema: schemas/consolidated-design.json # workspace-relative JSON Schema autocorrect: true # field-specific re-prompt before treating as a retry judge: skill: artifact-judge # a decision-category skill (LLM-as-judge) rubric: rubrics/consolidated-design.md threshold: 0.8 # score below this is a retry max_retries: 2 # then escalate to a human, never drop review: archetype: architect-reviewer # harness reviewer archetype read_scope: [app:oms, app:web, app:mobile] # read-only IAM scopes for the investigation route_back_at: high # findings >= this retry; lower ones attach approve: rules: - scope: design:approve roles: [oms-lead, web-lead, mobile-lead] quorum: all - scope: security:approve roles: [infosec-lead] quorum: all exclude_author: true # segregation of duties (default) min_distinct_approvers: 2 # four-eyes floor provenance: authored_by: human version: 1.0.0 ``` - [Funnel reference](https://github.com/delivstat/swarmkit/blob/main/docs/site/reference/funnel.md): layers, the fixed control flow, the advisory (structural) invariant, referenced-by-id. - [Gate funnel design note](https://github.com/delivstat/swarmkit/blob/main/design/details/gate-funnel.md): composition, control flow, bounded retry, the structural invariant, provenance bundle. - [Example funnel artifact](https://github.com/delivstat/swarmkit/blob/main/examples/sdlc-pipeline/workspace/funnels/consolidated-design-approval.yaml): all four layers, referenced by id from an SDLC pipeline node. ## Sequencing (it lives in your application) SwarmKit **used to** ship a sequencer: `kind: StageGraph`, a durable saga controller, `swarmkit orchestrator`, `swarmkit pipeline`, `POST /pipelines/*`. It was removed in runtime 1.189.0. The reason is the layering: sequencing across weeks is *application* logic — retries, business calendars, what an event means, when to give up — and every one of those pulled SwarmKit toward becoming a workflow engine. Do not write a `StageGraph`; a runtime that reads one no longer exists. What SwarmKit keeps is the part that is genuinely its own: **one bounded governed run**, its gate, and its record. - **Correlate** — `POST /run/{topology}` takes `correlation_id` ("same ticket"), `labels` (opaque `{k: v}` reaching `jobs` *and* `audit_events`), and `parent_job_id` ("this run replaces that attempt"). Runs are independent and connected by a correlation id — not stages of a graph the runtime knows about. Walk `parent_job_id` to answer "what did this artifact really cost across retries". - **Park and resume** — a funnel's `approve` layer raises `HITLDeferredError`: the run checkpoints, the job goes `deferred`, and its `error` names the gate. Nothing stays resident. `POST /jobs/{job_id}/resume` (409 unless parked) or `swarmkit run --resume` (the workspace's last checkpointed run; `swarmkit checkpoints` lists them) continues it, and a resumed run can park again identically. - **Stop a run you no longer want** — `swarmkit stop ` or `POST /jobs/{job_id}/stop` writes a durable flag the run reads at its next agent boundary, then raises `RunStoppedError` (a subclass of the deferral, so there is exactly one resumption path). Status `stopped` — not `deferred`, which means waiting on a decision that will arrive, and not `failed`, since nothing went wrong. Cooperative: a call in flight finishes first, and the CLI says so rather than implying a kill. A resume clears the flag; the act is audited as `run.stopped` with who asked. - **Read the gate** — `GET /gates/{gate_id}` returns `status` with the **approval policy already applied** (quorum, `min_distinct_approvers`, `exclude_author`) plus `items` and `quorum_evaluated`. A gate id is `:`, where `run_id` is the job id — split on the LAST colon. `GET /review?gate_id=…` is not a substitute: it returns role-tasks, and folding those into a decision means reading a funnel a client cannot see. - **Fetch what is being approved** — `GET /artifacts/{ref}`, ref shaped `//`; a review item carries the one it is about. An approver deciding without the artifact is deciding on a title. - **Receive events** — `POST /events/signal` is the surviving ingress seam (signed webhook → validated → opaque `correlation_id` extracted by JSONPath → handed to your listener). SwarmKit does not decide what an event *means*. - [Extracting the pipeline](https://delivstat.github.io/swarmkit/design-notes/extracting-the-pipeline/): the removal — what went, what stayed, and the migration inventory. - [Reading a gate, and approving without a saga](https://delivstat.github.io/swarmkit/design-notes/gate-state-and-deferring-approval/): the gate read, the deferral, and why the gate id had to become run-unique. - [Driving SwarmKit from your application](https://delivstat.github.io/swarmkit/reference/orchestrator-integration/): the whole HTTP contract in one page. - [`examples/pipeline-orchestrator/`](https://github.com/delivstat/swarmkit/tree/main/examples/pipeline-orchestrator): a reference application that sequences runs with **no `swarmkit_runtime` import anywhere in it**. ## Skill prerequisites (`requires:`) An ordering rule stated in a prompt is a request; a model follows mechanism over instruction. The evidence: in a single run, same agent, same prompt, an ack-gated tool was called 4 times and a merely-requested one 0 times. So ordering is declared on the agent and **enforced**: ```yaml skills: [list-build-conventions, get-build-convention, search-solution-code] requires: get-build-convention: [list-build-conventions] search-solution-code: [get-build-convention, list-build-conventions] ``` A sibling block, not entries inside `skills` — `skills` stays a plain identifier array, duplicate rules are impossible because it is a map, and every ordering rule reads in one place. - **Enforced at the MCP permission seam** — the one function both executors dispatch through, so a model agent and a harness agent behave identically; checked *before* the policy call, since an ordering refusal is not a policy question. - **The refusal is actionable**, which is the part doing the work: `get-build-convention requires list-build-conventions, which has not been called in this session. Call list-build-conventions first, then retry.` The agent recovers inside its own loop; the server is never touched by the refused call. - **Per `(run, agent)`** — a sibling agent's call does not satisfy this agent's prerequisite, because a prerequisite is about what is in *this* agent's context. - **Only a successful call satisfies** — an exception or an MCP `isError` unlocks nothing. A tool that reports failure in its *payload* does satisfy; the seam cannot read meaning. - **Validated at resolution** — a rule naming a skill the agent does not hold is `agent.requires-unknown-skill`; a cycle is `agent.requires-cycle` (an agent that can never recover is worse than no rule). - Scope: guards skills that dispatch through the MCP seam. A refusal is audited as `skill.executed` with `policy_decision="deny"`, so a gate that is working is distinguishable from one never reached. - [Skill prerequisites design note](https://delivstat.github.io/swarmkit/design-notes/skill-prerequisites/): the evidence, the shape decision, the semantics, and the non-goals (not parameterised, not cross-agent, not guarding decisions). ## Command packs (`implementation.type: command`, 1.197.0+) MCP was never the extension paradigm — **skills** are, and `implementation.type` has always had several backings. A **command pack** is the local-binary sibling of an `mcp_server`: declare it under `command_packs:` in `workspace.yaml`, and a skill reaches it with `{type: command, pack: , command: }`. Governance is reused, not duplicated: the pack carries the permission tier, the skill carries `iam.required_scopes`, and the action is `command:call::` alongside the untouched `mcp:call::`. ```yaml command_packs: - id: json-tools requires: [{ binary: jq, version: '>=1.6' }] # checked at workspace LOAD, naming the binary permission: readonly timeout_seconds: 30 # + timeout_overrides, max_output_bytes commands: - id: query argv: [jq, '-r', '{filter}', '{file}'] # argv, NEVER a shell effects: read # declared; undeclared means `write` ``` Five rules carry the design. **`argv`, never a shell** — a `{placeholder}` is filled with the *value* of an argument and stays exactly one argv entry, so `; rm -rf /` is an inert string; this holds structurally, not by escaping, because no code path re-parses it. There is deliberately **no generic `bash` skill**: `bash` is one action no policy can be written over, and SwarmKit already admits arbitrary execution through harness executors, contained by `_sandbox`/`_egress`/`_approval`/`_budget`/`_container`. **`effects` is declared per command and defaults to `write`** — nothing is inferrable (`curl` POSTs, `jq` and `sed` both take `-i`), so an unclassified command fails closed and `permission: readonly` is enforceable against a fact. **Secrets reach a command through the pack's `env` and never `argv`** — `{credential.*}` in an argv template is a schema error, so a secret cannot be model-placed, cannot land in the audit line recording what ran, and cannot be read from `ps`. **Bounds are never infinite** — an omitted `timeout_seconds`/`max_output_bytes` means the built-in default, and exceeding the ceiling FAILS rather than truncating, because a partial result read as complete is indistinguishable from a short one. **`requires` is checked at workspace load**, not at call time, so a missing binary names itself instead of surfacing as an exec error mid-run. **Bulk grants (1.198.0+).** A command becomes an ordinary skill with the id `-`, so the tool builder, `requires:` validation and the archetype merge need know nothing about packs. Grant them in bulk: ```yaml skills: - pack:json-tools # every READ command in the pack, now and later - server:filesystem # every skill targeting that MCP server - json-editing-rewrite # a write, named — bulk grants never carry one ``` `pack:` carries **reads only**. Adding a read command flows through to everyone holding the pack; adding a write reaches nobody, so a pack can never silently widen an agent that already holds it. `server:` makes no equivalent promise, because an MCP tool has no declared effect to filter on. A bulk grant matching nothing is a resolution error listing what is available — an agent silently granted no tools is indistinguishable from one whose model chose not to use them. Bundled packs to copy: `reference/command-packs/` — `file-tools` (coreutils only, so it runs on a bare machine), `json-tools` (jq), `text-tools` (ripgrep). Declaring a pack is not granting it — the grant is the audit step. Design: `design/details/command-packs.md`. Demo: `just demo-command-packs`. ## Another agent as a skill (`implementation.type: agent`, 1.222.0+) The fifth backing, and the A2A **client** side (`design/details/a2a-interop.md`). One skill type, two resolutions — exactly one of `topology: ` (a topology in this workspace) or `card_url: https://…/.well-known/agent-card.json` (a remote A2A agent). It is a skill like the other four: the same permission seam (`permission` tier + `effects` on the block, since there is no server/pack to inherit from; `readonly` allows only `effects: read`), `requires:` prerequisites, `iam.required_scopes`, and the call audited as `skill.executed`. The model's tool takes `{input, context?}` and gets the other agent's answer as text. - **Local** — runs the target in-process as a **child of the caller's run**: own run id + trace, a job row with `parent_job_id` and `source: agent`, same correlation, the parent's MCP servers shared and never closed by the child. A missing target fails the workspace load (like a missing command pack). Depth capped at 3 (a cycle otherwise). - **Remote** — the card is fetched on first use (cached; `skill_id` must be on it; 404 says "A2A may not be enabled there"); `credentials_ref` → a workspace `credentials` entry sent as a bearer; `message/send` with our run id as the A2A `contextId` (so two instances' records join on it), then `tasks/get` until terminal or `input-required`; past `timeout_s` the remote task is cancelled and the call fails. - **`on_unanswerable: agent | relay | abort`** — the harness adapter's words, when the other agent asks a question. `agent` (default): the question is the tool result (`{"status": "input_required", "task_id", "question"}`) and the calling agent answers by calling again with `{task_id, answer}`; audited `executor.input_response` with `responder: agent:`; after `max_agent_answers` (2) per task the next question relays. `relay`: a person, through the same `input_request` review item and bounded wait a harness question uses (`resolve_input`); no answer in time → call fails, remote task cancelled. `abort`: fail with the question. A **human gate** on the far side (a SwarmKit run parked on approval, `metadata.swarmkit.gate_url`) is never the agent's to answer under any policy — the result says `kind: human_gate` and names the gate. - **No funnel on the skill**: a child topology runs its own funnels, a remote SwarmKit its own, the caller's funnel gates what the caller does with the result. - **`pack:workspace`** (1.223.0+): every topology in the workspace is synthesized as an `agent` skill `topology-` at registry build (as command packs synthesize theirs; `cautious`, `effects: unknown`; listed by `GET /skills`), so "the supervisor may run any topology here" is `skills: [pack:workspace]` — and a topology added later reaches every holder of the grant. Unlike `pack:`, it is not filtered to reads (running a topology is never `read`); every call still passes the tier and the audit. A hand-authored skill named `topology-` is a collision error; the command-pack id `workspace` is reserved. A hand-authored `agent` skill targeting the same topology is untouched — that is where a stricter tier or a different `on_unanswerable` goes. - **Portal (1.224.0+):** the Connections page lists remote agents next to servers and sinks (same status column) and "Add remote agent" takes a card URL → `GET /api/a2a/probe?card_url=` (the runtime fetches the cross-origin card; `supported: false` carries the reason) → pick a skill, `on_unanswerable`, credential, tier → `PUT /api/skills/{id}` writes the `agent` skill file. `GET /api/a2a/agents` lists them. Discovery is an authoring act: nothing is granted by adding one. - **Harness nodes (1.225.0+):** an `agent` skill granted to a harness node is offered through the governed MCP gateway as the flat tool `agent__` (Claude Code sees `mcp__swarmkit__agent__`), with the same `{input, context}` schema; a call runs the skill's executor inside the run scope captured at registration, so the child run is attributed to the harness's run (`parent_job_id`), correlated and depth-bounded exactly as from a model node, and audited as the harness's `skill.executed` (a refusal is `policy_decision: deny`). A harness with agent grants and no MCP grants still gets a gateway. - Not yet: parking the caller's run (defer) while a remote answer is awaited — today it waits, bounded. ## Skill catalogue (`swarmkit-skills`, separate repo) — MCP servers with the wiring already worked out: the `mcp_servers` block, the `permission` tier, an `effects` map per tool so `readonly` is enforceable, and `iam.required_scopes`. Organised as **bundles** — one server plus the skills that use it — because that is how a server is actually adopted, and it matches command packs. **Each entry is started and asked, nightly.** A job launches every server and checks that the tool each skill names still exists, so an entry carries the date it last answered rather than a promise. Three states: `verified` (the server started and the tools were present, on that date), `broken` (a tool is gone — the entry stays visible, marked, with an issue filed), and `unverifiable` (needs a credential public CI cannot supply — reported honestly, because a green tick meaning "we did not look" is worth less than no tick). Only `broken` fails a run; failing on `unverifiable` would make every credentialed entry permanently red and teach everyone to ignore the check. Bring one in with `swarmkit skill add ` (1.235.0+; it shows the skill file and the `mcp_servers` entry before writing either, `--dry-run` prints them) or the portal's **Skills → Library** tab, then grant it. `swarmkit skill search `, `show `, `check` (start each server, confirm the tool still exists) and `remove` (refused while an agent holds the skill) are the rest of the command; `import ` brings an Agent Skills file in as an `llm_prompt` skill. A `pack:` grant carries the bundle's READ skills — now and later — while a write is always named individually, so a bundle can never silently widen an agent that already holds it. Every skill declares `provenance.requires_runtime`, so one that outgrows your runtime is refused at workspace load naming both versions rather than failing mid-run. Design: `design/details/skill-catalogue.md`, `skill-catalogue-seed.md` and `skill-registry.md` (the command). ## `permission: readonly` needs declared `effects` (BREAKING, 1.199.0) `readonly` used to decide write-ness by substring-scanning the **tool name** for `create|delete|update|write|put|post|set|add|remove|modify|edit|insert|drop|push|send`. It failed in both directions at once: `get_dataset` and `read_asset` matched **set**, `list_addresses` matched **add**, `get_post` matched **post** — ordinary reads, denied; while `truncate_table`, `purge_cache`, `revoke_token` and `wipe_db` matched nothing and were allowed. A longer word list was never the fix — the vocabulary of destructive verbs is unbounded and per-server. Declare effects per tool instead: ```yaml mcp_servers: - id: warehouse permission: readonly effects: { get_dataset: read, truncate_table: write } ``` Resolution order: the **declared entry wins** (it is the half the operator controls, and cannot change under them on a server upgrade), then the server's own `readOnlyHint` annotation, then `unknown`. **Under `readonly`, `unknown` is now DENIED** where it was previously allowed whenever the name missed the word list — the fail-closed direction, with a denial naming the tool and the field to add. Other tiers are untouched; `effects` is consulted only by `readonly`. Migration: `docs/notes/mcp-effects-migration.md`. ## Two checks before you run (reachability + verification) Both are read-only, both come from **one compile of the workspace**, so they cannot disagree. - `swarmkit validate --require` · `GET /workspace/reachability` — **configuration no code path can reach.** The recurring defect in this codebase was config that is declared, accepted, validated, displayed and loaded by *nothing*; the compiler now records what it actually built, on the line that builds it, and anything declared-but-unwired is reported by name. - `swarmkit validate --require-verified` · `GET /workspace/verification` — **which topology roots produce an output nothing checks.** Strength counts *wired* funnel layers, never declared ones (counting a declaration would repeat the defect above); declared-but-inert layers are named. Only roots are findings — a leaf worker returning a fact to its parent is not producing a reviewable artifact. ``` verification: 15 topology root(s) deploy/deploy-coordinator (root): funnel deploy-approval — approve; declared but inert: validate oms-build-harness/builder (root): funnel oms-code-review — judge, approve 12 topology root(s) produce an output that nothing checks — the run's answer is whatever the model said. ``` The two flags stay separate: "is my config wired" and "is my output checked" are different questions a CI job may want independently. ## Contract (integration contract between apps) A **Contract** is a first-class artifact (`kind: Contract`) — the agreed interface between two (or more) applications, identified by id. It is what makes a sequencer's locks real: a lock **is** an integration contract, so `locks: [oms-web, oms-inventory]` mean "hold the OMS↔Web and OMS↔Inventory interfaces while I change them, so no concurrent requirement commits a conflicting version." Making the contract an artifact turns those lock ids from free-form strings (where a typo silently becomes a *different* lock and two requirements that should serialise don't) into a **checked, pickable vocabulary**: the resolver rejects a lock that names no contract, and the contention view ("which work fights over the same contract") is exact. A contract is **not executed** — your sequencer is the lock manager; the registry only makes the vocabulary real and records which apps each lock binds. Fields: `parties` (required, ≥2 app ids — what makes it a contract, an interface *between* apps; drives the contention/ownership display; app ids are free strings, apps are not artifacts) and `interface` (optional — a pointer to where the interface spec itself lives, an API/event schema; not interpreted by core, it is documentation + a handle for reviewers). Core does not parse or diff the `interface` — identity + locking only, not interface compatibility. Minimal: ```yaml apiVersion: swarmkit/v1 kind: Contract metadata: id: oms-web name: OMS ↔ Web order API description: The order-submission + status API OMS exposes to the Web storefront. parties: [oms, web] # the apps this contract binds (>= 2) provenance: authored_by: human version: 1.0.0 ``` Fuller (with an `interface` pointer): ```yaml apiVersion: swarmkit/v1 kind: Contract metadata: id: oms-inventory name: OMS ↔ Inventory reservation API description: The stock-reservation + release events OMS exchanges with Inventory. parties: [oms, inventory] # >= 2 app ids interface: schemas/oms-inventory.json # optional pointer to the interface spec; not parsed by core provenance: authored_by: human version: 1.0.0 ``` - [Contract reference](https://github.com/delivstat/swarmkit/blob/main/docs/site/reference/contract.md): the fields, the locking/contention framing, referenced-by-`locks`, and the runtime ref-check. - [Integration-contract registry design note](https://github.com/delivstat/swarmkit/blob/main/design/details/contract-registry.md): why lock ids become a checked vocabulary, the lock ref-check, and the non-goals (no interface-content validation, no app artifacts, no new lock manager). ## Webhook ingress (the surviving inbound seam) `swarmkit serve` is the front door. A `Trigger` (`type: webhook`) validates the signature, extracts an opaque `correlation_id` from the body by JSONPath, and delivers an `EventSignal` to whatever your application registered. What the event *means* — which topology it starts, whether it advances anything — is your application's decision, because that judgement is what left with the sequencer. A trigger whose `credentials_ref` names an absent environment variable **refuses to start**: accepting unsigned requests because the secret is missing is a fail-open, and the old behaviour (warn, skip validation) was indistinguishable at runtime from a correctly configured trigger. `swarmkit serve` does not load a `.env` file. `POST /run/{topology_name}` and `POST /hooks/{topology_name}` start runs directly; the reserved-scope discipline is unchanged — human-only authorities (`skills:activate`, `mcp_servers:deploy`, `topologies:modify`, `iam:modify`, `approvals:resolve`) are enforced by the policy engine and structurally un-grantable to an agent. ## Events (the outbound seam) and Connections **The runtime tells an application what happened; the application decides who hears about it** (`docs/site/reference/events.md`, `design/details/extracting-the-channels.md`). `GET /events?after=&types=…&run_id=…&limit=…` returns events in log order from a position — the durable log a consumer replays forward from where it stopped (`GET /audit` is the other direction, newest first, for a person). `events:` in workspace.yaml adds best-effort push sinks (`webhook` with `url` + `credentials_ref` bearer, or `stdout` for development), a latency optimisation over the pull, never a replacement. `gates.auto_resume` (default true) continues a run as soon as its gate is resolved, so an application does not have to call `POST /jobs/{id}/resume` and cannot forget to. `examples/event-consumer/` is the reference application: it hears a gate open, asks a human on its own channel, resolves the gate, and imports no runtime module. The Slack/Discord/Telegram providers that used to live in the runtime were removed in 1.216.0. **Connections** (`docs/site/reference/connections.md`): `credentials` entries are references (`env`, `file`, `oauth`; the cloud sources are accepted by the schema and refused at resolution until a `SecretsProvider` exists), resolved by one `CredentialService` at every entry point. For a remote MCP server that speaks OAuth, the portal's Connections page runs the login — `GET /auth/mcp/probe`, `POST /api/oauth/login` (discovery, dynamic client registration, PKCE), `GET /auth/mcp/callback` — and stores the token encrypted (`SWARMKIT_OAUTH_KEY` or a generated `.swarmkit/oauth.key`), keyed by (credential, **owner**): a token belongs to the person who logged in. Refresh happens **before** a run whose window (`SWARMKIT_OAUTH_RUN_WINDOW_S`, 900 s) would outlive the access token, never mid-run; a refresh the provider refuses is `ConsentRequired` and needs a browser, not a retry. No endpoint returns a token. ## Human decisions at a gate (the approval API) A gate is resolved by a **human identity**, and the decision is a record — not a boolean. This is the surface an external application integrates with (docs/site/reference/orchestrator-integration.md). `POST /review/{item_id}/resolve` — `{"outcome": "approve" | "changes-requested" | "reject", "comment": "…"}`. The body carries **no identity**: the resolver is the authenticated caller (`request.state.identity.client_id`), because a body-supplied identity makes every membership and segregation-of-duties check self-asserted. The caller must hold `approvals:resolve`, a **reserved human-identity scope** a transport (API-key/JWT) token structurally cannot carry — so an agent or webhook integration can never satisfy an approval gate. Membership is checked against the workspace role registry before anything is recorded, and a non-member gets a 403 naming the reason. `reject` ends the run. **`changes-requested` does not** — it re-runs the stage with the reviewer's comment in its input, via a `rework` event distinct from `gate`. `POST /review/{id}/approve|reject` and `/answer` also accept `{"comment": …}`; a §6.3 input answer plus its comment becomes the parked harness's resume statement, so a conditional approval ("yes, staging only") reaches the agent instead of flattening to `true`. Review items serialize with `kind` = `permission` | `input` | `role_task` | `other`. A role-task carries `gate_id` (`:`, where `run_id` is the job id — split on the LAST colon), `role`, `scope`, `rule_index`, `resolved_by`, `comment`, `artifact_ref` and `round`. Narrow the queue with `GET /review?kind=role_task&gate_id=…`. **Rounds and staleness.** A rework loop re-opens the gate against a new artifact and advances the round; the ref is keyed on a **content digest**, so an identical re-run does not re-ask reviewers. **Only decisions about the current artifact count toward quorum** — earlier rounds are retained, returned by the read APIs and rendered as `STALE`, but an approval of v1 is not an approval of v3. An empty `artifact_ref` (externally-driven gate, or an item predating this) is unfiltered. What the agent receives is a fenced, attributed, typed and versioned block — human text is untrusted model input, framed as review feedback rather than instructions: ``` [changes-requested] security-reviewer (alice), scope=security:approve round 0, on run-42/design/output#c75bc614 (STALE — written about an earlier revision) The retry loop has no backoff. Add exponential backoff. ``` `GET /whoami` returns the authenticated caller (`/auth-info` is public and describes the server, not the caller). Every attempt — allowed or denied — is audited as `approval.role_task_resolved`, with `approval.gate_opened` per round and `approval.changes_requested` on a rework. ## Case study: feature-flag cleanup (the DoorDash shape, as data) [`examples/flag-cleanup/`](https://github.com/delivstat/swarmkit/tree/main/examples/flag-cleanup) is the SwarmKit form of DoorDash's agentic stale-flag cleanup ([case study](https://delivstat.github.io/swarmkit/case-studies/feature-flag-cleanup/)): Phase 1 a model agent triages a flag into a cleanup report a human confirms (`intake-review` funnel); Phase 2 a **harness** agent removes the flag in an isolated worktree under a budget and its diff faces the `cleanup-review` funnel — the `code-review` decision skill judges it (a finding routes the critique back to the harness) and a human on `flags:approve` signs off, the only exit. No orchestration code — the worktree isolation, the budget, the gate and the audit are the runtime's; the daily cross-repo fan-out is the calling application's. `just demo-flag-cleanup` runs it deterministically (real `claude-code` adapter over a scripted transcript through the real gate; no keys, no network). ## The SDLC example (the worked reference) [`examples/sdlc-pipeline`](https://github.com/delivstat/swarmkit/tree/main/examples/sdlc-pipeline) is the largest worked workspace: a software-delivery lifecycle — **intake → design → build → sit → pt → security-review → deploy → support-handover** — as topologies, archetypes, funnels, contracts and rigs, carrying three multi-party human gates, a harness build/review node, and IAM-scoped agents. Its *sequencing* half (the stage graph, the saga controller, the Temporal adapter) was removed with the bundled pipeline in 1.189.0; the artifacts, the gates and the per-stage demos remain, and `examples/pipeline-orchestrator/` shows how an application drives them. - [Video walkthrough](https://delivstat.github.io/swarmkit/sdlc-example/): every artifact toured on-screen. Recorded before the extraction — the artifact tour is current, the stage-graph/controller sections are historical. - Run it: `just demo-sdlc-stage-run` (one gated stage) · `just demo-consolidated-design` · `just demo-harness-build` · `just demo-sit-pt` · `python examples/sdlc-pipeline/validate_library.py` (validate every artifact). - Check it: `swarmkit validate --require --require-verified` in `examples/sdlc-pipeline/workspace` reports both the inert funnel layers and the roots nothing checks. ## Reference workspace Production-quality topologies, archetypes, and skills under `reference/`: - [Code Review Swarm topology](https://github.com/delivstat/swarmkit/blob/main/reference/topologies/code-review.yaml): 3 leaders (Engineering, QA, Ops), 10 agents. - [Skill Authoring Swarm topology](https://github.com/delivstat/swarmkit/blob/main/reference/topologies/skill-authoring.yaml): 6 specialist agents. - [Knowledge Curator topology](https://github.com/delivstat/swarmkit/blob/main/reference/topologies/knowledge-curator.yaml): curates governed memory — resolves quarantined contradictions through a human gate. - [27 reference skills](https://github.com/delivstat/swarmkit/tree/main/reference/skills): GitHub MCP, decision, knowledge, coordination, persistence (incl. `governed-memory` + `memory-reconcile`). - [16 archetypes](https://github.com/delivstat/swarmkit/tree/main/reference/archetypes): leaders, code review workers, authoring agents. ## Structured delegation (v1.2.0+) Planner-driven task execution replaces simple sequential delegation. Coordinators call `create-task-plan` to generate a dependency-ordered task plan; the compiler executes tasks in parallel when independent, sequentially when dependent. Key tools: - `create-task-plan`: coordinator produces a structured plan with tasks, dependencies, and assignments - `update-task-plan`: modify an in-flight plan (add tasks, change assignments) - `read-task-result`: retrieve completed task results (summary-first: 3-5 bullet key_findings, full results on disk) Self-tasks let the coordinator do its own work (synthesis, diagrams). Plans are crash-resilient via `tasks.json` on disk — the CLI detects previous plans on fresh runs. Auto-fix adds missing dependencies and synthesis tasks. ## Sterling workspace (reference implementation) Production-grade workspace under `examples/sterling-oms/` demonstrating enterprise-scale agent orchestration: - 8 topologies, 12 archetypes, 75 skills - Sub-agent architecture: root coordinator delegates to architect, which delegates to 6 focused workers (jira, config, docs, developer, log-analyst, document-writer) - Atlassian wrapper MCP: structured JQL/CQL queries so models never write raw query syntax - Log analyser MCP: SQLite-indexed log analysis handling 500MB+ log files, 9 tools including `get-timer-detail` drill-down - Document writer with pandoc MCP for DOCX/PDF generation - Per-agent model selection through OpenRouter: Kimi K2.5/K2.6 for the tool-heavy agents, Qwen3-235B and DeepSeek V4 Flash for workers, DeepSeek Chat V3 for writing ## Design notes Per-feature design notes under `design/details/`: - [MCP client](https://github.com/delivstat/swarmkit/blob/main/design/details/mcp-client.md): stdio + HTTP transports, workspace config, governance gating, inputSchema forwarding. - [Knowledge MCP Server](https://github.com/delivstat/swarmkit/blob/main/design/details/knowledge-mcp-server.md): 11 tools for live docs search. - [User Knowledge Server](https://github.com/delivstat/swarmkit/blob/main/design/details/user-knowledge-server.md): bootstrap-time knowledge wiring for user codebases. - [Code Review Swarm](https://github.com/delivstat/swarmkit/blob/main/design/details/topology-code-review.md): agent tree, skill map, HITL gates. - [Skill Authoring Swarm](https://github.com/delivstat/swarmkit/blob/main/design/details/topology-skill-authoring.md): multi-agent authoring + edit mode. - [Governance provider](https://github.com/delivstat/swarmkit/blob/main/design/details/governance-provider-interface.md): AGT integration, policy evaluation, audit. - [Model provider](https://github.com/delivstat/swarmkit/blob/main/design/details/model-provider-abstraction.md): the `ModelProvider` seam, per-agent model selection. - [Declarative model providers](https://github.com/delivstat/swarmkit/blob/main/design/details/declarative-model-providers.md): a provider is YAML over a wire-format family; 12 bundled (`anthropic`, `openai`, `google`, `ollama`, `openrouter`, `groq`, `together`, `rkllama`, `llama-server`, `openvino-model-server`, `mlx-lm`, `lemonade`); `/providers/*.yaml` adds or overrides; registration by readiness (key set, or no auth); `swarmkit providers list|show`. - [Conversational authoring](https://github.com/delivstat/swarmkit/blob/main/design/details/conversational-authoring.md): swarmkit init/author/edit. - [Structured output](https://github.com/delivstat/swarmkit/blob/main/design/details/structured-output-governance.md): schema-constrained generation + auto-correction. - [LangGraph compiler](https://github.com/delivstat/swarmkit/blob/main/design/details/langgraph-compiler.md): topology → StateGraph translation, approval gate checkpointing. - [DAG dependency graph](https://github.com/delivstat/swarmkit/blob/main/design/details/dag-dependency-graph.md): `depends_on` for parallel-with-dependencies execution. - [Decision skills](https://github.com/delivstat/swarmkit/blob/main/design/details/decision-skills.md): LLM judge verdicts, confidence scores, multi-persona panels. - [A2A federation](https://github.com/delivstat/swarmkit/blob/main/design/details/a2a-federation.md): SwarmKit-to-SwarmKit A2A returns the callee's run id, token/cost usage and an observability pointer; the card identifies it and the caller stitches it into its audit. - [Audit event journal](https://github.com/delivstat/swarmkit/blob/main/design/details/audit-event-journal.md): write-through audit — each event durable when recorded, so a crashed run keeps its trail; the store dedups the end-of-run backstop. - [Human interaction model](https://github.com/delivstat/swarmkit/blob/main/design/details/human-interaction-model.md): audit event schema, CLI primitives, the review queue (its notification layer left the runtime in 1.216.0 — see Events). - [Skill registry](https://github.com/delivstat/swarmkit/blob/main/design/details/skill-registry.md): the `swarmkit skill` command over the catalogue — search, add (both fragments shown first), import SKILL.md, check, remove; shipped 1.235.0. - [Memory by default](https://github.com/delivstat/swarmkit/blob/main/design/details/memory-by-default.md): the defaults a silent workspace gets, the `memory:` block, and the `memory.disabled-but-bound` error; shipped 1.233.0. - [Fleet control plane](https://github.com/delivstat/swarmkit/blob/main/design/details/fleet-control-plane.md) and the [control-plane note series](https://github.com/delivstat/swarmkit/tree/main/design/details/control-plane): enrolment (Mode A/B), membership scopes `monitor|manage|approve-as`, signed deploys, delta sync, operator identity forwarding (28). - [Knowledge Curator topology](https://github.com/delivstat/swarmkit/blob/main/design/details/knowledge-curator-topology.md): persistent wiki maintained by LLM agents. ## Harness executors (M19 — shipped, container sandbox included, opt-in) Run a coding harness (Claude Code, opencode, or any subprocess emitting line-delimited JSON) as an agent node, alongside the `model` executor, under the same governance + observability. An archetype selects a harness with an `executor` block — canonical shape `executor: { kind: harness, ref: claude-code }` (`ref` is the adapter id; swap `claude-code`→`opencode`/`codex`/`gemini-cli` to change harness). A registered adapter id may also be named directly as the kind (`kind: claude-code`, legacy). Absent an `executor` block, a node is `kind: model` (unchanged). Harnesses are **data**: a declarative `adapter.yaml` interpreted by one engine — no per-harness Python — with a bundled library (claude-code + opencode verified e2e; codex + gemini-cli experimental). A harness runs in an ephemeral git worktree by default (produces a diff, never integrates). Mid-run out-of-grant permissions **relay** to a human inbox and resume (`swarmkit review`); repeated operator approvals **accrue** into a proposed allowlist changeset (`swarmkit trust list|apply|clear`, default N=5, one denial resets+blocks). An **opt-in container sandbox** adds real isolation — resource limits, enforced egress (`deny`/`allowlist`), extra `mounts`, and a `build` step that runs the harness with **no local install** (bring only your API key). Off by default; `SWARMKIT_DISABLE_CONTAINER_SANDBOX` always wins; a container with no runtime fails loud, never a silent unsandboxed run. - [Executor abstraction design](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-abstraction.md): the provider seam (`model` | harness kinds), adapter tiers, relay, trust accrual, RFC decisions. - [Container sandbox design](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-container-sandbox.md): opt-in container tier, disable switch, build-no-local-install, mounts, egress proxy. - [Relay design](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-relay-plan.md) · [Input escalation](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-input-escalation-plan.md) · [Trust accrual](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-trust-accrual-plan.md). - [Authoring a harness adapter](https://github.com/delivstat/swarmkit/blob/main/docs/guides/authoring-harness-adapters.md): anatomy, event-map DSL, auth modes, launch review gate, sandbox block, DSL ceiling. - Adapter fixtures: [executor-adapter](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/executor-adapter) (minimal, claude-code, relay, sandbox-container, sandbox-build). ## HTTP server, auth, and canary deployments (M10 — shipped) - [Storage reference](https://github.com/delivstat/swarmkit/blob/main/docs/site/reference/storage.md): the six stores, `_URL` vs `_BACKEND`, and the step-by-step SQLite → Postgres migration runbook. - [Storage service design](https://github.com/delivstat/swarmkit/blob/main/design/details/storage-service.md): one resolver for every store; why a misconfigured workspace used to look like an empty one. - [Serve and auth design](https://github.com/delivstat/swarmkit/blob/main/design/details/serve-and-auth.md): FastAPI server, AuthProvider ABC, MCP endpoint, triggers, workspace.yaml `server:` block. - [Canary deployments design](https://github.com/delivstat/swarmkit/blob/main/design/details/canary-deployments.md): weighted topology version routing, auto-promotion by error rate + drift, manual promote/rollback. - [Serve CLI reference](https://github.com/delivstat/swarmkit/blob/main/docs/reference/serve-cli-tests.md): all endpoints with real curl output — health, topologies, jobs, SSE, webhooks, conversations, auth, MCP. - [Canary deployments guide](https://github.com/delivstat/swarmkit/blob/main/docs/reference/canary-deployments.md): quick start, configuration reference, monitoring, auto-promotion, manual controls, common scenarios, real test outputs. `swarmkit serve` endpoints — the essentials: `POST /run/{topology}` (async; `attachments`, `correlation_id`, `labels`), `GET /jobs/{id}` · `/stream` (SSE) · `/diff` · `POST /jobs/{id}/resume` · `/stop`, `GET /jobs/history`, `POST /hooks/{topology}` (signed webhook), `GET /events?after=` (durable log) and `POST /events/signal`, `GET /review` · `POST /review/{id}/approve|reject|answer|resolve`, `GET /gates/{gate_id}`, `GET /artifacts` · `/artifacts/{ref}`, `GET|POST /memory` · `/memory/quarantine` · `GET /memory/config`, `GET /gaps`, `POST /conversations` · `/conversations/{id}/messages` (SSE), `GET /audit`, `GET /usage`, `GET /system` · `/storage` · `/capabilities` · `/whoami` · `/auth-info`, `GET /workspace/reachability` · `/verification`, `POST /mcp` (Streamable HTTP), `GET /.well-known/agent-card.json` + `POST /a2a` (A2A, opt-in), canary and fleet routes, and the portal's `/api/*` (artifact CRUD, workspace config, OAuth login, the skill catalogue at `/api/skill-catalogue` + `/api/skills/{add,import,check}`, `POST /reload`). **Every endpoint is generated from the server's OpenAPI document into [the HTTP API reference](https://delivstat.github.io/swarmkit/reference/http-api/)** — 100 operations; that page is the complete list. **A2A server (1.221.0+, `server.a2a.enabled: true`; off by default):** `swarmkit serve` publishes an Agent Card at `/.well-known/agent-card.json` — one A2A skill per topology, `securitySchemes` derived from the auth provider actually running, `identity` (`name`/`description`/`url`/`organization`) from the workspace — and answers JSON-RPC 2.0 at `POST /a2a` (or `/a2a/{topology}`): `message/send`, `message/stream` (SSE; needs `Accept: text/event-stream`), `tasks/get`, `tasks/list`, `tasks/cancel`, `tasks/subscribe`. It is a **transport onto jobs**, not a second execution path: the task id IS the job id, `contextId` rides on `correlation_id`, file parts become attachments, the run carries `source: a2a`, and every gate and audit event applies unchanged. Job status → task state: `pending`→`submitted`, `running`→`working`, `deferred`→`input-required`, `completed`, `failed`, `stopped`→`canceled`. A run parked on a human gate reports `input-required` with the gate URL in the message, and a follow-up `message/send` on that task is **refused** (`UnsupportedOperationError`, -32004) — approval scopes are un-grantable to agents and an A2A client is an agent; a person resolves it through the review queue and a subscriber sees the task go `working`. Push notifications (-32003), gRPC and card signing are not implemented. `/capabilities` reports `features.a2a`. The client side is the `agent` skill type (1.222.0, its own section above): `design/details/a2a-interop.md`. Storage (1.130.0+): **one service resolves every store** — nothing else opens a database (`design/details/storage-service.md`). Six kinds: `runtime` (jobs, conversations, usage), `audit`, `artifacts`, `memory`, `fleet`, `checkpoints` (the `saga` store went with the bundled pipeline). All of them follow `storage.runtime` unless they declare their own block, EXCEPT `checkpoints`, which follows only `storage.checkpoints` because the Postgres LangGraph checkpointer is a separate install (`pip install "swarmkit-runtime[postgres]"`). Asking for postgres checkpoints WITHOUT that extra degrades to the local SQLite checkpointer with a warning (1.131.1+) rather than refusing to start — the one place degrading is right, because checkpoints are disposable run state and a missing optional dep should not take down serve. (1.130.0 and 1.131.0 refused, which broke upgrades for workspaces that had been carrying that silently-ignored config.) Everything that holds RECORDS still fails closed. A per-store block inherits `storage.runtime.url` when it declares none. `${VAR}` and `${VAR:-default}` are expanded in these URLs. `SWARMKIT_STORE_URL` vs `SWARMKIT_STORE_BACKEND`: **the URL alone is sufficient** — a URL names its own backend, so `SWARMKIT_STORE_URL=postgresql://…` selects Postgres with no other variable set. `DATABASE_URL` is the fallback when `SWARMKIT_STORE_URL` is unset. `SWARMKIT_STORE_BACKEND` (`sqlite`|`postgres`) is a rarely-needed override, mainly to force SQLite while a URL is present. Env is a global signal: it moves every store that follows `storage.runtime`, and never `checkpoints`. (Before 1.130.0, setting only `SWARMKIT_STORE_URL` was silently ignored, and `url: ${SWARMKIT_STORE_URL}` in workspace.yaml was never expanded — both wrote to SQLite while reporting success.) A backend naming a real database with **no resolvable URL raises at startup** rather than degrading to SQLite — a run must not write to a different database than the one configured. If your application keeps its own sequencing state, keep it in *your* database: SwarmKit's store holds runs, audit, artifacts, memory, fleet and checkpoints, not your workflow. Inspecting + migrating: `swarmkit storage status ` prints one line per store — backend, location, and **which setting won**. `swarmkit storage migrate [--dry-run] [--yes]` copies local SQLite rows into the configured Postgres: additive, idempotent (`ON CONFLICT DO NOTHING`), and it never deletes the SQLite files. `swarmkit system ` adds versions, `workspace.env.yaml` properties, and the environment. Same data at `GET /storage` and `GET /system`, and on the web UI's **System** page. Passwords are masked in every surface. SQLite → Postgres runbook (`docs/site/reference/storage.md`): 1. create the database (no schema step — tables are created on connect); 2. set `storage.runtime.backend: postgres` + `url: ${SWARMKIT_STORE_URL}` and export the URL; 3. `swarmkit storage status` and confirm every store says postgres BEFORE moving data; 4. stop the runtime and `swarmkit storage migrate`; 5. verify counts, then archive `.swarmkit/*.sqlite` — leaving them is how a split brain starts; 6. restart serve with the same environment. Skipping step 4 abandons the audit trail, run history and governed memory. Every audit event carries its `run_id` (1.130.0+; it was NULL on every row before, so `query(run_id=…)` always returned empty and reported it as "no events"). `workspace.env.yaml` should exist in every workspace — `swarmkit init` scaffolds it. A reserved top-level `secrets:` key lists dotted property paths whose values are never displayed (they render as `set` in `swarmkit system`, the System page, and CI logs); a name-based heuristic (`key`/`token`/`secret`/`password`/`credential`) is the fallback for undeclared ones. Declaring adds to the masked set and can never remove from it. A webhook trigger whose `credentials_ref` names an absent environment variable **refuses the request (503)** instead of accepting it unsigned. Auth providers: `NoneAuthProvider` (default, open access), `APIKeyAuthProvider` (Bearer token, env:VAR resolution, scopes), `JWTAuthProvider` (RS256/ES256, JWKS auto-discovery). Auth is perimeter; governance is internal policy. Canary config in workspace.yaml: `server.canary.routes[].versions[].weight` (traffic percentage), `promote_when.min_runs`, `promote_when.error_rate_below`, `promote_when.drift_below`, `promote_when.window_minutes`. Version-qualified names (`topology@1.1.0`) bypass routing for direct testing. Output validation: two layers, not interchangeable. `output_schema` on an agent checks SHAPE (free, deterministic, kills shape-level hallucination); a **decision skill** bound at `post_output` checks SEMANTICS (grounding, scope, contradiction) and costs an LLM call. Bind under `governance.decision_skills` in the topology (or workspace, where a topology must explicitly opt out with `required: false` — auditable). `scope` defaults to `*`, which fires after EVERY agent: name the root agent for "the topology's answer". Triggers: `pre_input` (reject before any LLM work), `post_output` (the answer), `checkpoint` (between task batches), `pre_synthesis` (task results BEFORE the coordinator launders them into a fluent summary; auto-loads scope.json). A decision skill is NOT necessarily an LLM call: `implementation.type` may be `mcp_tool` (deterministic — a validator, linter, test run, schema check), `command` (a local binary), `llm_prompt`, or `composed` (`strategy: parallel-consensus` to require agreement). The binding is identical either way. Prefer `mcp_tool` wherever the question has a computable answer; `reference/skills/` ships `lint-check`, `run-tests`, `security-scan`, `validate-workspace`, `gate-validator`. A decision skill's `verdict` must be `pass` | `fail` | `needs-revision`. FORM is normalised from 1.131.0 (`FAIL`, `Fail`, ` fail `, `needs_revision` all read correctly — before that they were unrecognised and therefore silently PASSED); VOCABULARY is not (`rejected`, `invalid`, `false` stay unrecognised — guessing a synonym would invent a verdict the skill never gave). An absent or unrecognised verdict is read as **pass**, so a mis-mapped validator reports success on every rejection; both cases now log a warning naming the skill ("the check is not running") instead of failing silently. **`fail` does not block**: the runtime builds feedback from the failed results, asks the agent to revise (it still holds its context), and after `max_retries` returns the output ANNOTATED with `GOVERNANCE FLAGS` rather than dropping it. Write `reasoning`/`violations` as instructions to the agent that will act on them — that text is the retry prompt. A hard stop is an approval gate, not a decision skill. `config.max_retries` on the binding (default 4) is honoured from 1.131.0 — before that the whole `config:` block was accepted by the schema and read by nothing. **`enabled` and `required` are different questions and were one flag until 1.169.0**: `enabled: false` means the binding does not run (this is how a topology switches off a workspace binding); `required: false` means it DOES run and its `fail` is advisory — logged, not fatal. Before 1.169.0 a falsey `required` discarded the binding entirely, so an advisory binding was accepted, validated, displayed and never evaluated — which is why `memory-reader`, bound advisory by the docs, never ran. A topology override still spelling the old `required: false` now makes the binding advisory and warns, rather than silently changing from off to on. Guides: `docs/site/guides/validating-topology-output.md`, `docs/site/guides/memory-and-decision-skills.md`. ## Workspace memory (shipped) Agents remember across conversations. Two decision skill bindings (`memory-reader` at `pre_input`, `memory-writer` at `post_output`) enable automatic insight extraction and context injection. Both bindings are on by default from 1.233.0 (see "Memory" below). Two backends: `MemoryStore` (the configured storage service — SQLite or Postgres, `workspace_memory` table, TF-IDF search; was `.swarmkit/memory.json` before 1.227.0) and `GBrainMemory` (GBrain MCP server, hybrid vector + keyword search, graph relationships, Supabase/Postgres). - [Workspace memory reference](https://github.com/delivstat/swarmkit/blob/main/docs/reference/workspace-memory.md): setup, config, GBrain integration, examples, real test outputs. - [Memory demo script](https://github.com/delivstat/swarmkit/blob/main/docs/examples/memory-demo.py): runnable demo covering CRUD, search, context injection, extraction, persistence, deletion. - [Workspace memory design](https://github.com/delivstat/swarmkit/blob/main/design/details/workspace-memory.md): two-layer knowledge graph, decision skill hooks, GBrain integration, privacy. ## Governed memory (shipped) Structured memory that **evolves in place over time** rather than piling up. An agent carrying the `governed-memory` persistence skill proposes facts as `{subject, attribute, value}` candidates; the runtime reconciles each against current memory — **new** / **reinforce** (identical restatement — no duplicate) / **update** (supersede the value in place) / **refine** (merge, via the `memory-reconcile` decision skill) / **contradict** (conflicts with a trusted memory → **quarantined** for a human curator, never silently overwritten). One canonical row per `(subject, attribute)` key plus an append-only change-log, so any fact is readable `as_of` a past time and update-in-place coexists with the append-only audit invariant (§8.3). Confidence decays with recency (stale facts rank down, never deleted); retrieval is relevance-ranked (local TF-IDF, or cosine similarity when an embedder is wired — no vendor lock-in). `swarmkit memory search|get|quarantine|resolve` and the serve `/memory` endpoints share one service seam; the `knowledge-curator` reference topology curates it. - [Governed memory reference](https://delivstat.github.io/swarmkit/reference/governed-memory/): the persistence skill, reconcile ops, quarantine + curator gate, confidence decay, relevance retrieval, CLI + serve. - [Governed memory design](https://github.com/delivstat/swarmkit/blob/main/design/details/governed-memory.md): the update-in-place model, current-state + append-only change-log, the governed write path, IAM. Memory-writer extracts structured insights (topic, context, key_points, tags) via LLM after each turn. Memory-reader searches for relevant prior conversations and prepends context. The agent sees prior sessions naturally: "As we discussed previously..." Config: `governance.decision_skills[].config.search_scope` (user/shared/both), `max_results`, `min_output_length`. ## Observability (design notes; OTel, intent drift and the ring buffer shipped) - [OpenTelemetry observability](https://github.com/delivstat/swarmkit/blob/main/design/details/opentelemetry-observability.md): OTel as the telemetry standard. Trace-per-run, span-per-agent-step, `swarmkit.*` semantic attributes, console + OTLP exporters to any collector. - [Intent drift detection](https://github.com/delivstat/swarmkit/blob/main/design/details/intent-drift-detection.md): optional per-agent intent monitoring via embedding similarity. Log/warn/nudge strategies. - [Telemetry reference](https://delivstat.github.io/swarmkit/reference/telemetry/): the `telemetry:` block, exporters, sampling, redaction; `swarmkit debug` is the local prompt ring buffer. - Operations across many instances is the self-hosted fleet control plane ([Level 22](https://delivstat.github.io/swarmkit/tutorials/22-fleet/), `design/details/fleet-control-plane.md`). ## Images (two channels, for two callers) **A caller holding the file attaches it to the run**: `swarmkit run … --attach `, or `attachments: [{"path": …}]` / `[{"data": }]` on `POST /run/{topology}`. It reaches the **entry agent's first message and no downstream node** — one model call, no tool round-trip. The media type is read from the bytes, so there is one `--attach` and no `--image`/`--pdf`; sending a `type` field is a 422, and `url`/`stream` sources are refused (the runtime does not fetch caller-supplied addresses, and an attachment is re-read on every turn of a tool loop). A bad path or an uncarryable type fails the CALL, not the run; images only today (`image/png|jpeg|gif|webp`), 20 MB ceiling (`SWARMKIT_ATTACHMENT_MAX_BYTES`). Every attachment is audited as a `run.attachments` event carrying name, media type, size, SHA-256 and source path — never the bytes. **An agent deciding mid-run what to look at still needs a skill.** A path in the prompt is text; base64 in the prompt is tokens. **That route is an MCP tool returning `ImageContent`**, which `langgraph_compiler/_skill_executor.py` converts to a `ContentBlock(type="image", …)` and the provider maps to its native image part (`image_url` data-URL for the `openai-compatible` family, incl. OpenRouter; `source.base64` for Anthropic). `docs-reader`'s `view_image` is the bundled tool that does this. Harness executors read image files from disk directly instead. The failure mode is silent: a path that does not resolve makes the tool report "not found", and the model then describes the image from surrounding prose anyway — fluently and wrongly. Pass **absolute** paths, rewrite relative refs inside any document the agent reads, and instruct the agent to report a failed path rather than describe the image. Since 1.129.2 the paths must also sit under the docs-reader `--workspace` root, which now confines rather than merely resolves. - [Getting an image to a model](https://delivstat.github.io/swarmkit/guides/getting-an-image-to-a-model/): the channel, the trap, and a measured before/after. ## Memory There are TWO memory subsystems and they are not interchangeable. **Governed memory** (`governed_memory/`, shape `{subject, attribute, value, type, confidence}`) is curated: reconcile-on-write, contradictions quarantined rather than applied, resolution a human action, confidence decaying by recency. It is what `swarmkit memory search|get|quarantine|resolve`, the `/memory` page and the `governed-memory` persistence skill address. **Workspace memory** (`memory/_store.py`, shape `{topic, context, key_points, tags}`) is what a run recorded by itself, unreviewed, on the configured store (`workspace_memory` table) or GBrain. From 1.168.0 the `memory-reader` binding at `pre_input` reads BOTH and injects curated facts first, in a delimited `` block. **Since 1.233.0 memory is on by default** (`design/details/memory-by-default.md`). A workspace that says nothing gets: `memory-reader` bound at `pre_input` before every agent, `memory-writer` at `post_output` (advisory, `required: false`), and the bundled `governed-memory` + `memory-reconcile` skills loaded when the workspace defines no skill with those ids. The `memory:` block tunes it — `enabled: false` switches everything automatic off (and `memory.disabled-but-bound` refuses a workspace that disables memory yet binds a memory skill explicitly), `reader`/`writer` sub-blocks set `search_scope` (user/shared/both), `max_results`, `min_output_length`; an explicit binding in `governance.decision_skills` is used as written. Two things stay per-agent, deliberately: **writing** curated memory is a grant of the `governed-memory` skill (it carries `kb:write`; granting it to obtain reads is how a curated store stops being curated), and nothing is injected into an agent whose binding is switched off. Confirm with `GET /memory/config` (the effective block), the portal's Memory page, or the `Memory context injected for agent=…` log line — not with `swarmkit memory search`, which proves the fact exists, not that an agent can see it. Before 1.233.0 all three pieces had to be declared by hand and each failed silently alone; before 1.169.0 `required: false` discarded the binding entirely (see decision skills above). Guide: `docs/site/guides/memory-and-decision-skills.md`. ## Guides - [Memory and decision-skill bindings](https://delivstat.github.io/swarmkit/guides/memory-and-decision-skills/): the two memories, turning governed memory on, and `enabled` vs `required`. - [Building swarms — the complete playbook](https://delivstat.github.io/swarmkit/guides/building-swarms/): the ordered, step-by-step build recipe (one agent → governed multi-app delivery flow), with a runnable artifact at every step and the SDLC worked example. - [Validating a topology's output](https://delivstat.github.io/swarmkit/guides/validating-topology-output/): `output_schema` (inline or a file path) for shape, decision skills for meaning, the retry loop and `GOVERNANCE FLAGS`. - [Getting an image to a model](https://delivstat.github.io/swarmkit/guides/getting-an-image-to-a-model/): attachments vs the `view_image` tool, and the silent failure mode. - [Sterling OMS workspace](https://github.com/delivstat/swarmkit/blob/main/docs/guides/sterling-oms-workspace.md): building domain-specific agent workspaces with knowledge bases. - [Model selection](https://github.com/delivstat/swarmkit/blob/main/docs/guides/model-selection.md): pricing comparison, per-agent config, env vars. - [Authoring a harness adapter](https://github.com/delivstat/swarmkit/blob/main/docs/guides/authoring-harness-adapters.md): declarative `adapter.yaml`, event-map DSL, auth, launch gate, opt-in container sandbox. - [Serve CLI reference](https://github.com/delivstat/swarmkit/blob/main/docs/reference/serve-cli-tests.md): complete endpoint reference with real test outputs. - [Canary deployments](https://github.com/delivstat/swarmkit/blob/main/docs/reference/canary-deployments.md): weighted version routing, auto-promotion, monitoring, rollback. - [Workspace memory](https://github.com/delivstat/swarmkit/blob/main/docs/reference/workspace-memory.md): agents that remember across conversations. Setup, GBrain backend, real test outputs. ## Tutorials (22 levels, each runnable) [Overview](https://delivstat.github.io/swarmkit/tutorials/). Levels 1–16 build one workspace up from a single agent to correlated, gated runs with contracts. Levels 17–22 each take one shipped capability and end in a `just demo-*` target that runs on the mock provider: [17 Harness executors](https://delivstat.github.io/swarmkit/tutorials/17-harness-executors/) (adapters, the governed gateway, relay + trust, sandbox), [18 Funnels & approval](https://delivstat.github.io/swarmkit/tutorials/18-funnels-approval/) (validate → judge → approve, role registry, `--require-verified`, `cited-change`, `stop`), [19 Command packs & attachments](https://delivstat.github.io/swarmkit/tutorials/19-command-packs-attachments/), [20 Agents calling agents](https://delivstat.github.io/swarmkit/tutorials/20-agents-calling-agents/) (`agent` skills, `pack:workspace`, A2A both ways, the portal's remote agents), [21 Providers, storage & operations](https://delivstat.github.io/swarmkit/tutorials/21-providers-storage-operations/) (declarative providers, storage status/migrate, `system`, `eval`, `knowledge-pack`), [22 Running a fleet](https://delivstat.github.io/swarmkit/tutorials/22-fleet/). **`just demo-capstone`** (`examples/capstone`) runs every HTTP-reachable feature in one workspace: attachment, command pack, agent skill, gate, A2A in and out, the record, a stop. ## Fixtures (valid artifact examples) - [Topology fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/topology) - [Skill fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/skill) - [Archetype fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/archetype) - [Workspace fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/workspace) - [Trigger fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/trigger) ## Cross-cutting notes - [Schema change discipline](https://github.com/delivstat/swarmkit/blob/main/docs/notes/schema-change-discipline.md) - [Usability-first](https://github.com/delivstat/swarmkit/blob/main/docs/notes/usability-first.md) - [LLM-friendly knowledge](https://github.com/delivstat/swarmkit/blob/main/docs/notes/llm-friendly-knowledge.md) - [Observability discipline](https://github.com/delivstat/swarmkit/blob/main/docs/notes/observability.md) - [Harness adapter discipline](https://github.com/delivstat/swarmkit/blob/main/docs/notes/harness-adapters.md): verify-against-real-binary, byte-identical bundled copies, sandbox-is-opt-in/never-silently-unsandboxed. ## Package docs - [packages/runtime CLAUDE.md](https://github.com/delivstat/swarmkit/blob/main/packages/runtime/CLAUDE.md): Python runtime invariants. - [packages/schema CLAUDE.md](https://github.com/delivstat/swarmkit/blob/main/packages/schema/CLAUDE.md): schema package invariants. ══════════════════════════════════════════════════════════════════════════════ # Part 2 — The playbook ══════════════════════════════════════════════════════════════════════════════ ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/guides/building-swarms.md ## Published: https://delivstat.github.io/swarmkit/guides/building-swarms/ ══════════════════════════════════════════════════════════════════════════════ # Building swarms — the complete playbook This is the end-to-end guide to building a **complete automated agent swarm** on SwarmKit, the open-source AI platform runtime — from a single agent to a governed, multi-app delivery flow that your application sequences over weeks. It is written to be read top to bottom: each step adds exactly one capability, shows the smallest real artifact that unlocks it, gives the command to run it, and links to the deep reference. If you only read one thing first, read the mental model. Everything else is a specialisation of it. !!! tip "For LLMs and coding agents" The repo root ships [`llms.txt`](https://github.com/delivstat/swarmkit/blob/main/llms.txt) — a compact, link-rich map of every feature with inline schemas. Load it into context first; use this playbook for the ordered build recipe and the worked example. ## The mental model Three claims, in priority order — they are the tie-breakers for every design decision: 1. **Topology is data.** A swarm is YAML/JSON the runtime *interprets*. There is no code-generation step and no "compile to Python" escape hatch — the portability guarantee is the openness of the artifacts plus the open-source runtime. A different swarm is new *data*, never new framework code. 2. **Skills are the only capability-extension primitive.** When you want an agent to be able to *do* something new, you add a skill (or compose existing ones). You never bolt on a parallel capability mechanism. *How* a node executes — a model call vs. a coding harness — is a separate **executor** seam, not a capability. 3. **Swarms grow through human-approved authoring.** The runtime records the capability gaps it hits; you surface them, author a skill through conversation, test it, and publish — gated at every step. A swarm you run is a swarm that tells you how to improve it. Everything below is built out of a small vocabulary of artifact kinds. Learn these ten nouns and the two embedded configs and you can read any SwarmKit workspace. ## The artifact kinds Every artifact is a YAML/JSON file starting with `apiVersion: swarmkit/v1` and a `kind`. There are **eleven canonical schemas** — ten standalone artifact kinds plus one embedded config (`ApprovalPolicy`, which lives inside a gate, not on its own). | Kind | What it is | Reference | |---|---|---| | `Workspace` | The root manifest — names the workspace, picks the governance provider, wires servers/memory/canary. | [workspace](../reference/workspace.md) | | `Topology` | A bounded swarm run — the agents, their roles, delegation edges, IAM scopes. The unit the runtime executes. | [topology](../reference/topology.md) | | `Archetype` | A reusable agent template — model/prompt/skills/IAM/executor defaults a topology node instantiates. | [archetypes](../reference/archetypes.md) | | `Skill` | A capability, decision, coordination, or persistence unit — the only capability-extension primitive. | [skills](../reference/skills.md) | | `Funnel` | A reusable per-artifact quality gate: `validate → judge → review → approve`, referenced by id from a node or stage. | [funnel](../reference/funnel.md) | | `Contract` | An integration contract between apps — makes the lock ids your sequencer holds a checked, pickable vocabulary. | [contract](../reference/contract.md) | | `RoleRegistry` | Named roles → member identities + the scopes they confer — how approval rules resolve to real people. | [role-registry](../reference/role-registry.md) | | `Trigger` | An external event source (webhook/schedule) that starts a topology or delivers a signed event. | [trigger](../reference/trigger.md) | | `ExecutorAdapter` | A declarative adapter (`adapter.yaml`) that runs a coding harness as a node — data, not per-harness Python. | [executor-adapter](../reference/executor-adapter.md) | | `ApprovalPolicy` | **Embedded config** (no `kind`) inside a gate's `approve:` — the multi-party rules, quorum, four-eyes floor. | [approval-policy](../reference/approval-policy.md) | ## Step 0 — install and scaffold `uv` is the recommended way to install and maintain SwarmKit — it installs `swarmkit` as an isolated global CLI, no virtual env or system-Python setup needed: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh # if you don't have uv yet uv tool install "swarmkit-runtime[ui]" # the runtime, the `swarmkit` CLI and server, the portal swarmkit init # scaffold a workspace through conversation ``` The server is part of the runtime; `[ui]` adds the web portal `swarmkit serve` hosts (without it serve runs headless, API only) and `[postgres]` the Postgres backend. `uv tool install swarmkit-runtime` alone is the CLI plus a headless server. Re-running `uv tool install` upgrades in place. `swarmkit init` is a conversational authoring swarm — you describe what you want and it produces the workspace, topology, archetypes, and skills as artifacts you own and can edit. You can equally hand-write the files; the rest of this guide shows the artifacts directly so you can read any workspace, however it was authored. A workspace root is one file: ```yaml apiVersion: swarmkit/v1 kind: Workspace metadata: id: my-swarm name: My Swarm governance: provider: mock # `mock` for local dev; `agt` (Microsoft AGT) for real policy/audit ``` → [Workspace reference](../reference/workspace.md) · [Installation](../getting-started/install.md) ## Step 1 — one agent A **Topology** is the unit the runtime runs. The smallest one is a single root agent instantiating an **Archetype**: ```yaml # archetypes/business-analyst.yaml apiVersion: swarmkit/v1 kind: Archetype metadata: { id: business-analyst, name: Business Analyst } role: leader defaults: model: { provider: openrouter, name: openai/gpt-4o-mini, temperature: 0.3 } prompt: system: > You are a business analyst. Read the requirement, identify the business flows it touches, and produce a clear, testable summary plus affected applications. iam: base_scope: [kb:read, kb:write] provenance: { authored_by: human, version: 1.0.0 } ``` ```yaml # topologies/intake.yaml apiVersion: swarmkit/v1 kind: Topology metadata: { name: intake, version: 0.1.0 } agents: root: id: intake role: root archetype: business-analyst iam: base_scope: [kb:read, app:oms:read] # this run's authority — least privilege ``` ```bash swarmkit validate . # resolve + type-check the whole workspace swarmkit run intake # one-shot execution ``` The archetype carries the *reusable* defaults; the topology node carries the *run-specific* wiring (id, IAM scopes). IAM scopes are structural — an agent can only touch what its `base_scope` grants. → [Topology reference](../reference/topology.md) · [Archetypes](../reference/archetypes.md) · [Tutorial 1: Hello World](../tutorials/01-hello-world.md) ## Step 2 — give it a skill A **Skill** is how an agent gains a capability. Skills come in four categories — `capability` (do a thing), `decision` (judge/score), `coordination` (route work), `persistence` (remember). A decision skill produces a structured verdict: ```yaml # skills/impact-analysis.yaml apiVersion: swarmkit/v1 kind: Skill metadata: { id: impact-analysis, name: Impact Analysis } category: decision outputs: # structured output — validated before anyone reads it type: object properties: affected_apps: { type: array, items: { type: string } } reasoning: { type: string } required: [affected_apps, reasoning] implementation: type: llm_prompt prompt: > Given the requirement and the apps' architecture summaries, decide which apps are affected and why. Return affected_apps (ids), rationale, and any open_questions. provenance: { authored_by: human, version: 1.0.0 } ``` Attach it in the archetype: `skills: [impact-analysis]`. The `outputs` schema is enforced by the runtime — the model's answer is validated and field-corrected before it flows anywhere, so shape-level hallucination never propagates. → [Skills reference](../reference/skills.md) · [Structured output governance](../design-notes/structured-output-governance.md) · [Tutorial 3: Skills](../tutorials/03-skills.md) ## Step 3 — many agents Add nodes and delegation. SwarmKit's compiler runs independent work in parallel and dependent work in order via `depends_on`, and coordinators use **structured delegation** — a planner builds a dependency-ordered task plan (`create-task-plan`) instead of ad-hoc prose hand-offs. This is the difference between a swarm that reliably fans out and one that loses track of its own work. ```yaml agents: root: id: architect role: root archetype: solution-architect children: [oms-dev, web-dev] # delegates to focused workers oms-dev: { id: oms-dev, role: worker, archetype: developer } web-dev: { id: web-dev, role: worker, archetype: developer } ``` → [DAG dependency graph](../design-notes/dag-dependency-graph.md) · [Tutorial 4: Multi-Agent](../tutorials/04-multi-agent.md) · [Tutorial 6: Structured Delegation](../tutorials/06-structured-delegation.md) ## Step 4 — real tools via MCP Agents get tools by connecting to **MCP servers** (stdio or Streamable HTTP), configured in the workspace — never coded per-vendor. Every tool call routes through governance, so an agent can only invoke tools its scopes allow. SwarmKit ships its own servers too (`swarmkit knowledge-server`, `swarmkit docs-reader`). → [MCP client](../design-notes/mcp-client.md) · [MCP discovery pattern](../design-notes/mcp-discovery-pattern.md) · [Tutorial 5: MCP Tools](../tutorials/05-mcp-tools.md) ## Step 5 — governance and decision skills Governance is not a prompt suggestion — it is structural. All policy/identity/audit flow through the `GovernanceProvider` interface; the audit log is append-only from the executive's perspective; and a set of scopes reserved for human identity (`skills:activate`, `mcp_servers:deploy`, `topologies:modify`, `iam:modify`, `approvals:resolve`) can **never** be granted to an agent, regardless of prompt. **Decision skills** run mandatory evaluations at workspace/topology boundaries with a bounded retry loop. → [Governance provider](../design-notes/governance-provider-interface.md) · [Structured output governance](../design-notes/structured-output-governance.md) · [Tutorial 7: Governance & Safety](../tutorials/07-governance.md) ## Step 6 — a quality gate with a Funnel A **Funnel** chains four optional layers into one reusable gate, referenced by id from an agent node. Present layers always run in the fixed order `validate → judge → review → approve`; the automated layers *filter and drive a bounded retry loop but never decide* — the only exit is human `approve`. On retry exhaustion it escalates to a human with the last critique attached; it never silently advances. ```yaml apiVersion: swarmkit/v1 kind: Funnel metadata: { id: consolidated-design-approval, name: Consolidated Design Approval } validate: schema: schemas/consolidated-design.json # deterministic; the judge never sees malformed input autocorrect: true judge: skill: artifact-judge # a decision skill scoring against a rubric rubric: rubrics/consolidated-design.md threshold: 0.8 # below this → a retry, not a rejection max_retries: 2 # then escalate to a human, never drop review: archetype: architect-reviewer # optional heavyweight harness reviewer (Step 7) read_scope: [app:oms, app:web, app:mobile] route_back_at: high # findings >= this cause a retry; lower ones attach approve: # required — the only exit rules: - scope: design:approve roles: [oms-lead, web-lead, mobile-lead] quorum: all exclude_author: true min_distinct_approvers: 2 provenance: { authored_by: human, version: 1.0.0 } ``` Drop layers to taste: a `Funnel` with only `approve` is a plain multi-party sign-off. → [Funnel reference](../reference/funnel.md) · [Gate funnel design](../design-notes/gate-funnel.md) ## Step 7 — a coding harness as a node Sometimes a node should be a real coding agent (Claude Code, opencode, Codex, Gemini CLI) that opens a repo and produces a diff — not a single model call. That is the **executor** seam. An archetype selects a harness with an `executor` block; everything else (governance, observability, the funnel it feeds) is unchanged: ```yaml apiVersion: swarmkit/v1 kind: Archetype metadata: { id: architect-reviewer, name: Architect Reviewer (harness) } role: worker executor: kind: harness ref: claude-code # swap for opencode / codex / gemini-cli defaults: prompt: { system: "Investigate — verify the design matches the code. Read only." } iam: { base_scope: [app:read, kb:read] } provenance: { authored_by: human, version: 1.0.0 } ``` Harnesses are **data**: a declarative `ExecutorAdapter` (`adapter.yaml`) interpreted by one engine — no per-harness Python. A harness runs in an ephemeral git worktree by default (produces a diff, never integrates); out-of-grant permissions relay to a human inbox mid-run (`swarmkit review`) and resume; an opt-in container sandbox adds real isolation. Authoring a new harness is writing one `adapter.yaml`: ```yaml apiVersion: swarmkit/v1 kind: ExecutorAdapter metadata: { id: echo-harness, name: Echo Harness } spec: launch: { command: [echo-harness, "{task.statement}"] } stream: { format: jsonl } event_map: - when: { type: done } emit: - event: result with: { status: success, output: "$.text" } provenance: { authored_by: human, version: 0.1.0 } ``` → [Executor adapter reference](../reference/executor-adapter.md) · [Executor abstraction](../design-notes/executor-abstraction.md) · [Authoring a harness adapter](https://github.com/delivstat/swarmkit/blob/main/docs/guides/authoring-harness-adapters.md) ## Step 8 — sequence the runs from your application A single topology run is bounded — minutes, one team, one concern. Real delivery work spans weeks, many teams, external events (Jira, CI, SAST), and human gates. **That sequencing is yours, not SwarmKit's.** SwarmKit shipped a `StageGraph` + saga controller until runtime 1.189.0 and then removed them: what an event means, when to retry, which calendar applies and when to give up are application decisions, and hosting them here was turning a swarm framework into a workflow engine. See [Extracting the pipeline](../design-notes/extracting-the-pipeline.md). What you get instead is a small, honest HTTP contract: ```python # your orchestrator — a script, a Temporal workflow, an Airflow DAG, whatever you already run job = http.post("/run/consolidated-design", { "input": brief, "correlation_id": "WMS-35", # "same ticket" — groups every run of the flow "labels": {"app": "oms"}, # opaque to SwarmKit; reaches jobs AND audit_events })["job_id"] # a gated run parks instead of holding a process open while http.get(f"/jobs/{job}")["status"] == "deferred": gate = http.get(f"/gates/{job}:designer") # policy already applied — quorum, four-eyes if gate["status"] == "approved": http.post(f"/jobs/{job}/resume") # a resumed run can park again, identically else: sleep(POLL_SECONDS) # a gate waits on a person ``` Three fields carry the thread: `correlation_id` ("same ticket"), `labels` (your model, opaque to SwarmKit), and `parent_job_id` ("this run replaces that attempt" — what makes cost across retries answerable). `GET /artifacts/{ref}` fetches what a gate is about. The locks your sequencer holds reference **Contract** artifacts — making a lock id a checked, pickable vocabulary instead of a free-form string a typo could silently fork: ```yaml apiVersion: swarmkit/v1 kind: Contract metadata: { id: oms-web, name: OMS ↔ Web order API } parties: [oms, web] # >= 2 app ids — an interface *between* apps provenance: { authored_by: human, version: 1.0.0 } ``` Read [`examples/pipeline-orchestrator/`](https://github.com/delivstat/swarmkit/tree/main/examples/pipeline-orchestrator): a reference application that drives a multi-stage flow with **no `swarmkit_runtime` import anywhere in it**. That is the proof the boundary is real rather than claimed. → [Driving SwarmKit from your application](../reference/orchestrator-integration.md) · [Contract reference](../reference/contract.md) · [Reading a gate, approving without a saga](../design-notes/gate-state-and-deferring-approval.md) · [Tutorial 16: Sequencing & Contracts](../tutorials/16-pipelines.md) ## Step 9 — multi-party human approval A gate resolves to real people through a **RoleRegistry**. A gate's `approve` rules name roles; the registry maps roles to member identities and the scopes they confer; scopes reserved for human identity can never be held by an agent. ```yaml apiVersion: swarmkit/v1 kind: RoleRegistry metadata: { id: sdlc-roles, name: SDLC role registry } roles: - { id: oms-lead, members: [alice], scopes: [design:approve] } - { id: infosec-lead, members: [dana], scopes: [security:approve] } - { id: eng-manager, members: [grace], scopes: [release:approve] } # human-only prod authority - { id: cio, members: [heidi], scopes: [release:approve] } ``` The **ApprovalPolicy** (the `approve:` block — embedded config, not a standalone artifact) has two independent axes: *which roles signed* (`quorum: all | any | { k-of: N }`) and *how many distinct humans signed* (`min_distinct_approvers`, the four-eyes floor). A dual-hatted person can satisfy two roles but never two distinct-approver slots. → [Role registry reference](../reference/role-registry.md) · [Approval policy reference](../reference/approval-policy.md) · [Multi-party approval](../design-notes/multi-party-approval.md) ## Step 10 — triggering Delivery work advances on the outside world. A **Trigger** is an external event source that starts a topology or delivers a signed event your application acts on. A signed CI webhook: ```yaml apiVersion: swarmkit/v1 kind: Trigger metadata: { id: ci-build-ready, name: CI build-ready webhook } type: webhook targets: - pipeline: oms-delivery # the event STREAM your application listens on emit: build.ready-in-qa # the event name — SwarmKit routes it, you interpret it correlation_id: $.correlation_id # opaque handle extracted from the JSON body config: auth: { method: hmac, credentials_ref: CI_WEBHOOK_SECRET } ``` The `swarmkit serve` HTTP front door receives it: the receiver validates the HMAC, extracts the opaque `correlation_id`, and hands the event to your listener (`POST /events/signal`). What the event *means* is your application's call — that judgement left with the sequencer. A trigger whose `credentials_ref` names an absent environment variable **refuses to start**, because accepting unsigned requests is a fail-open indistinguishable from working. → [Trigger reference](../reference/trigger.md) · [Serve mode](../reference/serve.md) · [Tutorial 12: Triggers & Canary](../tutorials/12-triggers-canary.md) ## Step 11 — serve, observe, evolve Ship it behind the server, watch it, and let it tell you how to grow: - **Serve.** `swarmkit serve` exposes topologies as async jobs with SSE streaming, an MCP endpoint, pluggable auth (API key / JWT-JWKS), webhook triggers, and canary version routing with auto-promotion. Install with the extras to get the server and the hosted web UI: `uv tool install "swarmkit-runtime[ui]"` — then `swarmkit serve` hosts the portal (dashboard, chat, topology canvas) at its own origin; without `[ui]` it runs headless (API only). → [Serve mode](../reference/serve.md) · [Tutorial 11: Serve & HTTP API](../tutorials/11-serve-api.md) - **Observe.** Every run is a trace of agent-step spans with token counts. `swarmkit trace `, `swarmkit status`, `swarmkit logs`, `swarmkit why ` (LLM post-mortem), `swarmkit ask`. OpenTelemetry export is built in. → [Telemetry](../reference/telemetry.md) · [Human interaction model](../design-notes/human-interaction-model.md) - **Remember.** Workspace memory lets agents carry insight across conversations (local JSON or a GBrain backend). → [Workspace memory](../reference/workspace-memory.md) · [Tutorial 9: Conversations & Memory](../tutorials/09-conversations-memory.md) - **Grow.** The runtime records capability gaps (`swarmkit gaps`); you author the missing skill through conversation (`swarmkit edit`), test it, and publish — human-approved at every step. → [Skill authoring](../design-notes/topology-skill-authoring.md) · [Tutorial 13: Authoring & Review](../tutorials/13-authoring-review.md) ## The worked example — the SDLC workspace Everything above is assembled, end to end, in [`examples/sdlc-pipeline`](https://github.com/delivstat/swarmkit/tree/main/examples/sdlc-pipeline): a complete software-delivery lifecycle — **intake → design → build → sit → pt → security-review → deploy → support-handover** — carrying three multi-party human gates, integration contracts, a harness build/review node and IAM-scoped agents. Its sequencing half left with the bundled pipeline in 1.189.0; the artifacts, the gates and the per-stage demos remain. It is the reference for how the pieces fit. - **Watch it.** The [captioned video walkthrough](../sdlc-example/index.html) tours every artifact and runs a stage end to end. - **Run it.** `just demo-sdlc-stage-run` runs a single gated stage end to end; `just demo-consolidated-design`, `just demo-harness-build` and `just demo-sit-pt` each exercise one capability deterministically (no keys, no server). - **Read it.** The [SDLC example design note](https://github.com/delivstat/swarmkit/blob/main/design/details/sdlc-pipeline-example.md) is the build-order narrative (slices 1–9) and the automation map (which stages are agent-run vs. human-gated). ## Validate everything The whole point of topology-as-data is that a swarm is checkable before it runs: ```bash swarmkit validate . # resolve + type-check the workspace, print the tree or errors python examples/sdlc-pipeline/validate_library.py # validate every artifact in a library ``` The resolver rejects a lock that names no contract, an approval rule whose scope no role confers, a stage that kicks an unknown topology, and any artifact that fails its schema. Validation is the fast feedback loop; a live `swarmkit run` is the confidence loop — do both. → [`swarmkit validate` reference](../reference/cli.md) ## Reference index - **Artifacts:** [topology](../reference/topology.md) · [workspace](../reference/workspace.md) · [archetypes](../reference/archetypes.md) · [skills](../reference/skills.md) · [funnel](../reference/funnel.md) · [contract](../reference/contract.md) · [role-registry](../reference/role-registry.md) · [trigger](../reference/trigger.md) · [executor-adapter](../reference/executor-adapter.md) · [approval-policy](../reference/approval-policy.md) - **Runtime seams:** [governance provider](../design-notes/governance-provider-interface.md) · [model provider](../design-notes/model-provider-abstraction.md) · [executor abstraction](../design-notes/executor-abstraction.md) · [extracting the pipeline](../design-notes/extracting-the-pipeline.md) - **Operate:** [CLI commands](../reference/cli.md) · [serve](../reference/serve.md) · [telemetry](../reference/telemetry.md) · [events](../reference/events.md) · [env config](../reference/env-config.md) - **Learn by doing:** the [16-level tutorial ladder](../tutorials/index.md) walks the same arc one runnable step at a time. ══════════════════════════════════════════════════════════════════════════════ # Part 3 — Memory and decision-skill bindings ══════════════════════════════════════════════════════════════════════════════ ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/guides/memory-and-decision-skills.md ## Published: https://delivstat.github.io/swarmkit/guides/memory-and-decision-skills/ ══════════════════════════════════════════════════════════════════════════════ # Memory and decision-skill bindings Two subsystems that look like one, and one flag that used to mean two things. Both caused silent failures — a curated fact that reached no agent, and a binding that was accepted and never evaluated. This is what each actually does and how to turn it on. ## There are two memories They are separate stores with different shapes, different write paths, and different purposes. `swarmkit memory` addresses the first; the `memory-reader` skill historically read only the second. | | governed memory | workspace memory | | --- | --- | --- | | shape | `{subject, attribute, value, type, confidence}` | `{topic, context, key_points, tags}` | | written by | `governed-memory` skill, `swarmkit memory`, the curation flow | `memory-writer` skill, automatically after a run | | reviewed | reconcile-on-write, quarantine on contradiction, human gate | no | | decays | yes (confidence by recency) | no | | stored in | the configured store (`storage.runtime`) | the same configured store (`workspace_memory` table; was `.swarmkit/memory.json` before 1.227.0), or GBrain | | CLI | `swarmkit memory search \| get \| quarantine \| resolve` | — | | UI | `/memory` | — | **Governed memory is for facts you want an agent to treat as established** — a correction a human made, a decision that should not be re-litigated. It is curated: a write is reconciled against what is already there, a contradiction is quarantined rather than applied, and resolution is a human action. That machinery is what makes a fact trustworthy enough to act on. **Workspace memory is what a run remembered by itself.** Useful, unreviewed, and rewritten freely. Since 1.168.0 the `memory-reader` binding reads **both**, and curated facts are rendered first — they went through review; workspace memory is whatever a previous run happened to record. ## Turning governed memory on Since runtime **1.233.0** it is on unless the workspace says otherwise (`design/details/memory-by-default.md`): the `governed-memory` and `memory-reconcile` skills are bundled and loaded when the workspace defines no skill with that id, and `memory-reader` / `memory-writer` are bound automatically, advisory, on every agent. What used to be three easy-to- miss steps is now one optional block: ```yaml # workspace.yaml — optional memory: enabled: true # false switches off everything automatic reader: {max_results: 5, similarity_threshold: 0.15, search_scope: all} writer: {min_output_length: 100} ``` The two steps below are what the defaults do for you; you still write them when you want something narrower (a reader scoped to one agent, a reconciler of your own). ### 1. (Default) The `governed-memory` skill is present A copy in `skills/` overrides the bundled one — that is how you change its prompt or effects. ### 2. (Default) `memory-reader` is bound at `pre_input` An explicit binding is used exactly as written and the automatic one is skipped: ```yaml # workspace.yaml governance: decision_skills: - id: memory-reader trigger: pre_input scope: "analyst" # only this agent reads memory required: false # advisory — see below, and read that section config: governed_limit: 5 # how many curated facts to inject (default 5) ``` `required: false` is correct here. A memory read that can fail a run is worse than no memory. Note that before **1.169.0** this silently discarded the binding — if you are on an earlier version, the reader never runs, and `swarmkit memory search` showing your fact tells you nothing about whether an agent can see it. ### 3. Grant `governed-memory` to an agent that should WRITE Reading needs no grant beyond the binding. Writing does, and the skill carries `kb:write`: ```yaml # archetypes/curator.yaml defaults: skills: - governed-memory # write access — grant deliberately ``` Do **not** grant this to make reading work. Reading comes from the `memory-reader` binding; granting the write skill to get reads is how a curated store stops being curated. ### Confirming it works The reader logs when it injects: ``` Memory context injected for agent=triage (user=None, query=enumerate the cartons...) ``` and the agent's input gains a delimited block: ``` Established facts for this workspace: - sn8 · carton-count-source: Carton count comes from the TASK LIST, not Shipment/Containers. ``` If `swarmkit memory search` finds the fact and that line never appears, the binding is not reaching the compiler — check your version and the `enabled`/`required` spelling below. ## Writing a curated fact ```bash swarmkit memory search "carton count" # what is already known swarmkit memory get sn8 carton-count-source # one fact, with its history ``` A write goes through reconciliation: a new value for an existing `(subject, attribute)` is compared against the current one and classified — reinforce, refine, update, or **contradict**. A contradiction is quarantined rather than applied, and `swarmkit memory resolve` is a human decision. That is the point of the store; an agent cannot overwrite a reviewed fact by asserting louder. ## Decision-skill bindings: `enabled` and `required` Two questions, two flags. They were one flag until 1.169.0, and collapsing them is what made an advisory binding disappear. ```yaml governance: decision_skills: - id: memory-reader trigger: pre_input enabled: true # does it run at all? (default true) required: false # can a `fail` stop the run? (default true) ``` | | `enabled` | `required` | | --- | --- | --- | | asks | does this binding run? | can its verdict stop the run? | | `false` means | the skill is not bound; nothing happens | the skill runs; a `fail` is logged, not fatal | | set it in a topology to | switch off something inherited from the workspace | make an inherited gate advisory | ### The triggers | trigger | fires | a `fail` from a required binding | | --- | --- | --- | | `pre_input` | before any LLM work | rejects the input; no tokens spent | | `post_output` | after the agent answers | sends it back for revision, bounded by retries | | `checkpoint` | between task batches | logged only | | `pre_synthesis` | before a leader synthesises | logged only | ### Migrating from the old spelling Before 1.169.0, a topology disabled an inherited binding with `required: false`, and a workspace binding with `required: false` was discarded outright. If you have either: ```yaml # was: disable an inherited binding - id: grounding-verifier trigger: post_output required: false # now means ADVISORY — it will run # write instead: - id: grounding-verifier trigger: post_output enabled: false # off ``` The runtime warns when it sees the old shape on a topology override, naming the binding. It does not reinterpret silently, because a gate that starts running when it used to be off is a real change. ## Why both of these failed quietly Each is the same shape, and it is worth recognising: - Governed memory had a full curation flow, a CLI, a UI page and a store — and no agent read path. The reader searched a different store and reported finding nothing, which is exactly what finding nothing looks like. - An advisory binding was accepted by `swarmkit validate`, appeared in the resolved workspace, and was dropped at compile time with no message. Neither produced an error. If you are configuring either and it appears to do nothing, the useful question is not "is my YAML right" — it is "what does the runtime say it loaded". `swarmkit validate` answers the first; the injection log line above answers the second. ## Related - [Getting an image to a model](getting-an-image-to-a-model.md) — the same "it exists but nothing surfaces it" shape, one subsystem over - `design/details/governed-memory.md` in the repo — the store, reconciliation and decay - `docs/notes/reported-bugs.md` in the repo — bugs 21 and 22, the two failures above ══════════════════════════════════════════════════════════════════════════════ # Part 4 — Artifact references ══════════════════════════════════════════════════════════════════════════════ ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/topology.md ## Published: https://delivstat.github.io/swarmkit/reference/topology/ ══════════════════════════════════════════════════════════════════════════════ # Topology A **topology** is a first-class SwarmKit artifact (`kind: Topology`) that defines a complete swarm: a tree of agents rooted at a single `root`, plus its runtime, artifact, governance, and monitoring configuration. Topology is the framework's core "topology-as-data" claim — a swarm is this file, interpreted by the runtime, not code. See the [topology schema design note](https://github.com/delivstat/swarmkit/blob/main/design/details/topology-schema-v1.md). This page is the artifact reference. ## Fields Required top-level: `apiVersion`, `kind`, `metadata`, `agents`. `metadata` requires `name` (lowercase-kebab) and `version` (semver). | Top-level | Required | What it does | |---|---|---| | `agents` | yes | Exactly one `root` agent, which nests `children`. The whole swarm is one tree, one parent per agent. | | `runtime` | no | Execution config: `mode` (`one-shot`/`persistent`/`scheduled`), `max_concurrent_tasks`, `task_timeout_seconds`, `planning`, `synthesis`, `checkpointing.storage` (`sqlite`/`postgres`). `planning.synthesis_roles` (default `[self, document-writer]`) names the roles the planner treats as synthesis/output roles — auto-wired to depend on research tasks so they run last, not in parallel; `planning.synthesizer_role` (default `synthesizer`) names the role the automatic synthesis step runs as when `synthesis` is set. | | `artifacts` | no | `knowledge_bases`, `review_queues`, `audit` (`level`/`storage`/`retention_days`), and `skill_gap_logging`. | | `intent_monitoring` | no | Semantic drift detection: `enabled`, `threshold` (default 0.75), `on_drift` (`log`/`warn`/`nudge`). | | `governance` | no | `decision_skills[]` bindings that override or extend workspace-level bindings by id. | ### Agent fields | Field | Required | What it does | |---|---|---| | `id` | yes | Lowercase-kebab agent id. | | `role` | yes | `root` (only the top agent) \| `leader` \| `worker`. | | `archetype` | no | Archetype id this agent instantiates (resolved against the workspace). | | `model` | no | `provider`, `name`, `temperature`, `max_tokens`, plus dual-model `tool_provider`/`tool_model` and provider-native `options`. | | `prompt` | no | `system` / `persona`. | | `skills` | no | Skill IDs — **replaces** the archetype's skill list when present. | | `skills_additional` | no | Skill IDs **merged onto** the archetype defaults. | | `requires` | no | Ordering rules over this agent's skills: `{guarded: [prerequisite, ...]}`. The guarded skill is **refused** until every prerequisite has returned successfully in the same run, and the refusal names what to call first, so the agent recovers inside its own loop. Both sides must be skills the agent holds; a cycle is a resolution error. | | `iam` | no | `base_scope` / `elevated_scopes`. | | `output_schema` | no | JSON Schema for structured output — inline, or a **path** to a JSON/YAML schema file relative to this topology file (validated at load; must stay inside the workspace) — overriding the archetype default; `null` opts out. [Guide](../guides/validating-topology-output.md#the-same-schema-as-a-file). | | `funnel` | no | A [Funnel](funnel.md) id — a reusable per-artifact quality gate on this agent's output. | | `children` | no | Nested agents (`leader`/`worker`), each of which may also declare `depends_on` (agent IDs that must finish first — DAG ordering). | ## Schema shape ```yaml apiVersion: swarmkit/v1 kind: Topology metadata: name: # required version: 1.0.0 # required, semver description: runtime: mode: one-shot # one-shot | persistent | scheduled max_concurrent_tasks: 4 agents: root: id: supervisor role: root # the root agent must be role: root archetype: supervisor-leader children: - id: analyst role: worker archetype: code-analyst skills_additional: [query-swarmkit-docs] - id: writer role: worker archetype: document-writer depends_on: [analyst] # runs after analyst completes funnel: design-signoff # gate this agent's output ``` ## Minimal example ```yaml apiVersion: swarmkit/v1 kind: Topology metadata: name: hello-swarm version: 1.0.0 agents: root: id: assistant role: root model: provider: anthropic name: claude-sonnet-4-5 ``` ## Authoring a topology `get_schema("topology")` returns the exact shape. Skills are referenced by **id**, never inlined; `skills` replaces archetype defaults while `skills_additional` extends them. Only the `root` agent may have `role: root`. Any agent that produces a sign-off-worthy artifact should reference a `funnel`; any agent that must run after another should declare `depends_on`. ## See also - [Archetypes catalogue](archetypes.md) · [Skills](skills.md) — what agents instantiate and invoke. - [Funnel](funnel.md) — the per-artifact gate an agent's output can pass through. - [Environment configuration](env-config.md) — `${VAR}` references resolve in topology YAML like any artifact. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/workspace.md ## Published: https://delivstat.github.io/swarmkit/reference/workspace/ ══════════════════════════════════════════════════════════════════════════════ # Workspace A **workspace** is the top-level SwarmKit artifact (`kind: Workspace`, in `workspace.yaml`) that carries deployment-level configuration: identity, governance, model providers, credentials, the MCP server registry, storage backends, and serve-mode settings. Topologies, skills, archetypes, funnels, triggers, and the rest live *in* a workspace and inherit its configuration. See the [workspace schema design note](https://github.com/delivstat/swarmkit/blob/main/design/details/workspace-schema-v1.md). This page is the artifact reference. ## Fields Only `apiVersion`, `kind`, and `metadata` (`id` + `name`) are required; everything else is optional. | Field | What it does | |---|---| | `metadata` | `id`, `name`, optional `description`, and free-form `annotations`. | | `organisation` / `team` | Optional org/team identity (`{ id, name? }`). | | `governance` | `provider` (`agt`/`mock`/`custom`), `policy_language` (`yaml`/`rego`/`cedar`), `limits` (circuit breakers: `max_steps_per_agent`, `max_steps_per_run`, `max_cost_per_run_usd`), and `decision_skills[]` inherited by all topologies. | | `memory` | Memory on by default: `enabled` (default `true`), `reader` (`max_results`, `similarity_threshold`, `search_scope`) and `writer` (`min_output_length`) for the automatically bound `memory-reader` / `memory-writer`; the `governed-memory` and `memory-reconcile` skills are bundled unless the workspace defines its own. `enabled: false` switches everything automatic off — [Workspace memory](workspace-memory.md). | | `identity` | Human-identity provider (`builtin`/`auth0`/`okta`/`google`/`azure-ad`/`oidc`). | | `model_providers` | Python-class registrations (`class`, `provider_id`, `config`) for a custom `ModelProvider`. The usual way to add a provider is a YAML file in `/providers/` — see [Model provider](model-provider.md). | | `credentials` | Named credential **references** (never literals): each `{ source, config }` where `source` is `env`, `file`, or `oauth` (a token obtained by logging in from the portal, stored encrypted per owner and refreshed before a run — see [Connections](connections.md)). The cloud sources (`hashicorp-vault`, `aws-secrets-manager`, `gcp-secret-manager`, `azure-key-vault`, `plugin`) are accepted by the schema and refused at resolution until a `SecretsProvider` is wired for them. | | `mcp_servers` | The MCP registry: `id`, `transport` (`stdio`+`command` or `http`+`endpoint`), `env`, `credentials_ref`, `sandboxed`/`sandbox_image`, and governance `permission` tiers (`open`/`cautious`/`strict`/`readonly`). | | `storage` | Backends for `checkpoints`, `audit`, `runtime` (jobs/conversations/usage), `artifacts`, `memory`, `fleet` and `knowledge_bases` (`default_backend: sqlite \| postgres`) — each `sqlite` or `postgres`, following `storage.runtime` unless they declare their own block ([Storage](storage.md)). `storage.artifacts` additionally takes `database_url` (override the inherited connection URL) or, for the `s3` backend, `bucket` (needs the `boto3` optional dependency). | | `context_compression` | Opt-in read-side compression of bulk tool output: `backend` (`off` default / `columnar` / `headtail` / `plugin`, the last with `backend_class`), `min_bytes` (below which nothing is compressed), and `overrides[]` per surface — each with `match` (tool name glob) or `match_server` (glob on the backing MCP server id, e.g. `logs-*`) and its own `backend` / `min_bytes`. | | `planning` / `synthesis` | Workspace-default planning and synthesis config, overridable per topology. | | `events` | Where the runtime pushes what happened: `[{ sink: webhook \| stdout, url, credentials_ref, types }]`. Best-effort; `GET /events?after=` is the durable log — see [Events](events.md). | | `gates` | `auto_resume` (default `true`): a run continues as soon as its gate is resolved, so an application does not have to call `POST /jobs/{id}/resume` — turn it off to batch or delay. | | `command_packs` | Local binaries exposed as `command` skills — the sibling of `mcp_servers` for capabilities that already exist as executables (`design/details/command-packs.md`). | | `server` | `swarmkit serve` config: `jobs` (`max_concurrent`, `timeout_seconds`), `mcp.enabled`, `a2a` (`enabled`, `identity`), `canary` routes, and `auth`. | ### Serve authentication (`server.auth`) `provider`: `none` (default; only safe on loopback) \| `api_key` \| `jwt`. A non-loopback bind with `provider: none` **refuses to start** unless `require_on_nonloopback: false` (default-secure). `api_key` needs `config.keys[]` (each `{ key_ref, client_id, client_name?, tier | scopes }` — `client_name` is the human-readable name shown in audit and `/whoami`, defaulting to the id); `jwt` needs `config.issuer` and reads scopes from the `scopes_claim` claim (default `scope`); `none` may set `identity` / `identity_name` so a loopback deployment still records who acted. ### Canary (`server.canary.routes`) Each route splits one topology's traffic across ≥2 `versions` whose `weight` sums to 100, with optional `promote_when` criteria (`min_runs`, `error_rate_below`, `drift_below`, `window_minutes`). ## Schema shape ```yaml apiVersion: swarmkit/v1 kind: Workspace metadata: id: my-swarm # required name: My Swarm # required governance: provider: agt limits: max_steps_per_run: 500 max_cost_per_run_usd: 25 model_providers: - class: swarmkit_runtime.model_providers.openrouter.OpenRouterProvider provider_id: openrouter credentials: github: source: env config: { var: GITHUB_TOKEN } mcp_servers: - id: github transport: stdio command: ["npx", "-y", "@modelcontextprotocol/server-github"] credentials_ref: github permission: cautious storage: checkpoints: { backend: sqlite, path: .swarmkit/checkpoints.sqlite } server: jobs: { max_concurrent: 5, timeout_seconds: 300 } auth: provider: api_key config: keys: - key_ref: env:SWARMKIT_API_KEY client_id: ci tier: run ``` ## Authoring a workspace `get_schema("workspace")` returns the full shape. Credentials are always references, never literal secrets. For the dev/staging/prod split and `${VAR}` interpolation, see [Environment configuration](env-config.md); for the memory subsystem, see [Workspace memory](workspace-memory.md). ## See also - [Environment configuration](env-config.md) — `${property.path}` / `${VAR}` resolution across `workspace.yaml` and every artifact. - [Workspace memory](workspace-memory.md) · [Serve mode](serve.md) · [Telemetry configuration](telemetry.md). ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/archetypes.md ## Published: https://delivstat.github.io/swarmkit/reference/archetypes/ ══════════════════════════════════════════════════════════════════════════════ # Archetypes catalogue 16 reference archetypes ship with SwarmKit. ## Leaders | Archetype | Role | Used in | |---|---|---| | supervisor-leader | root | Code Review Swarm | | engineering-leader | leader | Code Review Swarm | | qa-leader | leader | Code Review Swarm | | ops-leader | leader | Code Review Swarm | | authoring-supervisor | root | Skill Authoring Swarm | | conversation-leader | leader | Skill Authoring Swarm | ## Code review workers | Archetype | Skills | |---|---| | github-reader | github-repo-read, github-pr-read | | code-analyst | code-quality-review, query-swarmkit-docs | | security-reviewer | security-scan, query-swarmkit-docs | | test-analyst | test-coverage-review | | llm-judge | qa-verdict (or deploy-risk-review via override) | ## Authoring workers | Archetype | Skills | |---|---| | knowledge-searcher | query-swarmkit-docs, list-reference-skills, validate-workspace | | schema-drafter | get-schema, query-swarmkit-docs, read-workspace-file | | artifact-validator | validate-workspace, query-swarmkit-docs | | test-writer | get-schema, write-workspace-file, run-tests | | artifact-publisher | write-workspace-file | ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/skills.md ## Published: https://delivstat.github.io/swarmkit/reference/skills/ ══════════════════════════════════════════════════════════════════════════════ # Skills Skills are SwarmKit's only extension primitive. Every capability an agent can exercise is a skill. ## Categories | Category | Purpose | Returns | |----------|---------|---------| | **capability** | Give agent a new ability (API call, search, generation) | Output data | | **decision** | Let agent evaluate or judge (validation, classification) | Verdict + confidence + reasoning | | **coordination** | Enable handoff between agents (A2A, escalation) | Task status | | **persistence** | Enable recording (audit log, knowledge base write) | Write confirmation | ## Importing from the public MCP ecosystem There are 7,000+ community MCP servers. Before writing a skill from scratch, check if a public server already does what you need. Wrapping a public MCP server is three config files: **1. Add the server to workspace.yaml:** ```yaml mcp_servers: - id: brave-search transport: stdio command: ["npx", "-y", "@anthropic/brave-search-mcp"] env: BRAVE_API_KEY: "${BRAVE_API_KEY}" ``` **2. Create a skill YAML that references it:** ```yaml apiVersion: swarmkit/v1 kind: Skill metadata: id: search-web name: Web Search description: Search the web for current information using Brave Search. category: capability implementation: type: mcp_tool server: brave-search tool: brave_web_search provenance: authored_by: human version: 1.0.0 ``` **3. Add the skill to an archetype:** ```yaml defaults: skills: - search-web ``` The runtime discovers the server's tools via MCP protocol at startup and forwards the tool schema to the LLM so it knows the correct parameter names. ### Common public MCP servers | Server | npm package | Use case | |--------|------------|----------| | GitHub | `@modelcontextprotocol/server-github` | Repos, PRs, issues, actions | | Filesystem | `@modelcontextprotocol/server-filesystem` | Read/write local files | | Brave Search | `@anthropic/brave-search-mcp` | Web search | | Slack | `@anthropic/slack-mcp` | Channels, messages | | PostgreSQL | `@modelcontextprotocol/server-postgres` | Database queries | | Google Drive | `@anthropic/gdrive-mcp` | Docs, sheets | | Qdrant | `mcp-server-qdrant` | Vector store + RAG | ### Custom MCP servers For capabilities without a public server, write a custom one. SwarmKit can scaffold these: ```bash swarmkit author mcp-server . ``` This generates a Python MCP server using the `mcp` SDK, a skill YAML, and a workspace.yaml entry through conversation. ## Reference skills 20 reference skills ship with SwarmKit in the `reference/skills/` directory. ### Capability skills | Skill | MCP server | Tool | |---|---|---| | github-repo-read | github | get_file_contents | | github-pr-read | github | get_pull_request | | github-issue-read | github | get_issue | | query-swarmkit-docs | swarmkit-knowledge | search_docs | | list-reference-skills | swarmkit-knowledge | list_reference_skills | | get-schema | swarmkit-knowledge | get_schema | | validate-workspace | swarmkit-knowledge | validate_workspace | | read-workspace-file | swarmkit-knowledge | read_workspace_file | | write-workspace-file | swarmkit-knowledge | write_workspace_file | | run-tests | swarmkit-knowledge | run_pytest | | search-codebase | (template) | — | | summarize-review | (llm_prompt) | — | ### Decision skills | Skill | Outputs | |---|---| | code-quality-review | verdict, confidence, reasoning, issues | | security-scan | verdict, confidence, reasoning, findings | | test-coverage-review | verdict, confidence, reasoning, gaps | | qa-verdict | verdict, confidence, reasoning | | deploy-risk-review | verdict, confidence, reasoning, risks | | lint-check | verdict, confidence, reasoning, violations | Decision skills require an `outputs` block with JSON Schema defining the verdict structure. The runtime enforces this: if the model's output doesn't match the schema, it gets field-specific error messages and retries automatically. ### Coordination skills | Skill | Description | |---|---| | peer-handoff | A2A context packaging for leader-to-leader handoff | ### Persistence skills | Skill | Description | |---|---| | audit-log-write | Structured event to governance audit log | ## Sterling OMS workspace skills (21 skills) The Sterling workspace demonstrates real-world skill design: | Skill | Type | Server | Purpose | |---|---|---|---| | search-sterling-docs | capability | sterling-product-docs | ChromaDB semantic search over 17K product docs | | search-docs-exact | capability | sterling-product-fts | FTS5 exact keyword search | | search-project-docs | capability | sterling-project-docs | ChromaDB search over project docs | | search-reference-designs | capability | reference-designs | Search sanitised designs from other projects | | get-service-config | capability | sterling-config | Parsed CDT service configuration | | get-pipeline | capability | sterling-config | Pipeline steps, conditions, pickup transactions | | search-configs | capability | sterling-config | Grep across all CDT config tables | | grep-project-code | capability | code-graph | Grep file contents (returns relative paths) | | read-file-lines | capability | code-graph | Read specific line range from source files | | verify-code-citations | capability | code-graph | Check file:line citations against actual source | | get-api-input-xml | capability | sterling-api-javadocs | API input XML structure from javadocs | | get-api-output-xml | capability | sterling-api-javadocs | API output XML structure | | read-project-code | capability | project-code | Read full source file | | write-notes | capability | notes-dir | Write analysis to notes directory | ## Skill anatomy ```yaml apiVersion: swarmkit/v1 kind: Skill metadata: id: code-quality-review # lowercase-kebab, unique in workspace name: Code Quality Review description: > Evaluates code changes against quality standards. Returns pass/fail verdict with specific issues found. category: decision outputs: # required for decision skills type: object properties: verdict: type: string enum: [pass, fail, needs-changes] confidence: type: number minimum: 0 maximum: 1 reasoning: type: string issues: type: array items: type: object properties: file: { type: string } line: { type: integer } severity: { type: string, enum: [error, warning, info] } message: { type: string } required: [verdict, confidence, reasoning] implementation: type: mcp_tool # or: llm_prompt | composed | command | agent server: review-server tool: check_quality iam: required_scopes: [repo:read] provenance: authored_by: human # human | authored_by_swarm | vendor_published version: 1.0.0 ``` ## Another agent as a skill (`implementation.type: agent`) One skill type, two resolutions — a topology in this workspace, or a remote agent reached through its A2A Agent Card (`design/details/a2a-interop.md`). Either way it is a skill: the same permission seam as an MCP tool or a command, the same `requires:` prerequisites, the same audit of the call. ```yaml implementation: type: agent # exactly one of: topology: deep-research # same workspace → child run, no wire card_url: https://research.internal/.well-known/agent-card.json # elsewhere → A2A task API skill_id: deep-research # remote only: which card skill; omitted = the card's first credentials_ref: research-agent # remote only: a workspace `credentials` entry, sent as a bearer timeout_s: 600 on_unanswerable: agent # agent | relay | abort — the harness adapter's words max_agent_answers: 2 permission: cautious # open | cautious | strict | readonly effects: read # read | write | unknown; readonly allows only `read` ``` The tool the model sees takes `{input, context?}` and returns the other agent's answer as text. **Local (`topology:`).** The target runs in-process as a child of the caller's run: its own run id and trace, a job row with `parent_job_id` and `source: agent`, the parent's MCP servers shared (and never closed by the child), the same correlation. A missing target fails the workspace load, not the first run. Depth is capped at 3: a topology calling a topology that calls it back is a cycle. **Remote (`card_url:`).** The card is fetched on first use and cached; `skill_id` must be on it. The call is `message/send` with our run id as the A2A `contextId` (so two instances' records join on it), then `tasks/get` until the task ends or asks for input. Past `timeout_s` the remote task is cancelled and the call fails. **When the other agent asks a question** (`input-required`), `on_unanswerable` decides, with the words a harness adapter uses: | policy | behaviour | |---|---| | `agent` (default) | The question comes back as the tool result — `{"status": "input_required", "task_id", "question"}` — and the calling agent answers by calling the skill again with `{task_id, answer}`. Audited as `executor.input_response` with `responder: agent:`. After `max_agent_answers` on one task, the next question is relayed. | | `relay` | A person answers through the review queue — the same `input_request` item and bounded wait a harness question uses; no answer in time fails the call and cancels the remote task. | | `abort` | The call fails with the question as the reason; the remote task is cancelled. | A **human gate** on the other side (a SwarmKit run parked on approval) is never the agent's to answer, whatever the policy: the result says `kind: human_gate` and names the gate; a person resolves it there. No funnel is configured on the skill — a child topology runs its own funnels, a remote SwarmKit runs its own, and the caller's funnel gates what the caller does with the result. **From a harness node.** An `agent` skill granted to a harness (Claude Code, opencode, …) is offered through the governed MCP gateway as the flat tool `agent__` — the same executor, tier and audit as from a model node; the child run is attributed to the harness's run. **`pack:workspace`.** Every topology in the workspace is also synthesized as an `agent` skill named `topology-` (`cautious`, `effects: unknown`), the way command packs synthesize theirs, so a supervisor that may run any topology here says so in one line: ```yaml agents: root: skills: [pack:workspace] # topology- for every topology, now and later ``` Nothing is granted by default — the card lists every topology to the outside, but inside the workspace an agent reaches only what its topology names. Unlike `pack:`, the grant is not filtered to reads (running a topology is never `read`); every call still goes through the tier and the audit. Naming a hand-authored skill `topology-` is a collision error, and `workspace` is reserved as a command-pack id. ## Constraints `constraints` is optional on any skill: `max_latency_ms`, `timeout_seconds`, `retry: {attempts, backoff: exponential | linear | none}`, and `on_failure: escalate_to_human | fail | retry | fallback` — what the runtime does when the skill's call fails after its retries. `escalate_to_human` files a review item rather than returning an error to the agent; `fallback` hands the failure to the agent as a tool error it can act on. ## Provenance Every skill declares who authored it. This affects runtime trust defaults: | Value | Meaning | Trust | |-------|---------|-------| | `human` | Hand-authored by user | Full trust | | `authored_by_swarm` | Produced by authoring swarm, human-approved | Locked until first human review | | `derived_from_template` | Generated from template | Partial oversight | | `imported_from_registry` | Community registry | Depends on registry vetting | | `vendor_published` | Commercial vendor | Depends on vendor relationship | ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/funnel.md ## Published: https://delivstat.github.io/swarmkit/reference/funnel/ ══════════════════════════════════════════════════════════════════════════════ # Funnel A **funnel** is a first-class SwarmKit artifact (`kind: Funnel`) that packages a per-artifact quality gate. It chains up to four layers — structured-output validation, an LLM-as-judge, an optional harness review, and multi-party human approval — into one reusable composition. Any node that produces an artifact worth a human sign-off can reference a funnel by id, so the same gate applies consistently across nodes and stages instead of being re-wired per stage. The full composition, control flow, bounded retry loop, and the structural invariant are specified in the design note: [gate funnel](https://github.com/delivstat/swarmkit/blob/main/design/details/gate-funnel.md) (`design/details/gate-funnel.md`). This page is the artifact reference. ## What a funnel is for Every artifact worth a human sign-off should reach that human having already cleared the cheap, automatable checks. A funnel makes "every artifact is judged before a human sees it" a declarative property of a gate rather than bespoke wiring per stage: the only path to approval is *through* the automated layers, cheap → expensive → human. A funnel does not redefine the layers it sequences — it reuses SwarmKit's native structured-output validation, an existing governance decision skill for the judge, a harness reviewer archetype, and the multi-party approval policy. It only sequences and gates on them. ## Referenced by id A funnel is a standalone artifact, like a skill or an archetype. It lives in a `funnels/` directory in the workspace and is referenced by id from a topology node's `funnel:` field. Defining it once and referencing it by id is what lets one gate cover many nodes and stages. ## The layers Present layers always run in the fixed order `validate → judge → review → approve`. Key order in the YAML does not matter — the control flow is compiler-owned. Every layer is optional **except `approve`**; a funnel with only `approve` degrades to a plain multi-party approval gate. | Layer | Kind | What it does | |---|---|---| | `validate` | deterministic, no LLM | Structured-output validation against a JSON Schema (`schema`, `autocorrect`) with field-specific auto-correction. A shape auto-correction cannot repair is a retry — the judge never sees malformed input, which kills shape hallucination up front. Two sibling keys apply to a gated node that produces a **diff** (a harness): `slice_budget: {max_files, max_diff_lines}` makes an over-budget change a validate failure ("split it"), and `cited_change: true` requires the artifact to be a change-rationale (`summary` + `citations: [{claim, path, lines}]`) whose citations name code the diff actually touched. Both are the checks `swarmkit slice-check` and `swarmkit cited-change` run standalone. | | `judge` | LLM-as-judge decision skill | Scores the artifact against a rubric. A score below `threshold` drives a bounded retry carrying the critique back to the drafter. | | `review` | harness reviewer (optional, heavyweight) | An investigative reviewer returns findings. Findings at or above `route_back_at` severity (`low` \| `medium` \| `high` (default) \| `critical`) retry; the rest attach to the human task and travel onward. | | `approve` | multi-party human approval (**required**) | The binding human layer: per-role tasks, quorum, `min_distinct_approvers`, `exclude_author`. The only exit from the funnel to `done`. | ## The fixed control flow ``` draft ─▶ validate ─(ok)▶ judge ─(pass)▶ review ─(no route-back)▶ APPROVE (human) ─▶ done │ │ │ (unrepairable) (below threshold) (finding ≥ route_back_at) └──────────────┴─────────────────────┘ ▼ retry: critique/findings ─▶ drafting agent revises ─▶ re-enter at validate ``` A retry feeds the failing layer's critique or findings back to the drafting agent, which revises and re-enters at `validate`. Retries are bounded by the judge's `max_retries`. On **exhaustion** the funnel does not drop the requirement or silently pass — it escalates to a human with the last failing critique attached, and that human decides. Retry state lives in the run checkpoint, so it is durable and resumable. ## The structural invariant Layers 1–3 are **advisory**: they gate *advancement to* human approval and drive the retry loop, but they **never** approve. Two properties, enforced by the compiled graph shape (not by prompt wording): 1. **No bypass of the human gate.** There is no edge from any automated layer to `done`; the only path to `done` is through `approve`. A judge or reviewer "pass" advances *to* the human, never *past* them. 2. **No reaching the human without passing.** `approve` is only reachable after `validate` and `judge` succeed (and any non-blocking review has attached). A below-threshold artifact cannot land on a human's desk except via the explicit retry-exhaustion escalation. This is the judicial pillar (design §8) filtering for the human/legislative one, not substituting for it. The control flow is fixed and compiler-owned: a funnel configures the layers, it does not rewire the graph. ## Provenance bundle On reaching `approve`, the funnel assembles a bundle the human task carries: the artifact, the validation result, the judge score and critique, the attached reviewer findings, the retry count, and the diff since the last approval. The human decides in one place with the full automated context. ## Schema shape ```yaml apiVersion: swarmkit/v1 kind: Funnel metadata: id: # referenced from a node's funnel: field name: description: validate: # optional — layer 1 schema: autocorrect: true # default true judge: # optional — layer 2 skill: rubric: threshold: 0.8 # default 0.8; below → retry max_retries: 2 # default 2; then escalate to a human review: # optional — layer 3 archetype: read_scope: [, ...] # read-only IAM scopes for the investigation route_back_at: high # default high; findings >= this retry, rest attach approve: # REQUIRED — layer 4 rules: # every rule must be satisfied - scope: roles: [, ...] quorum: all # all | any | { k-of: N } exclude_author: true # default true — segregation of duties on_revision: reset_all # default reset_all | reconfirm_changed min_distinct_approvers: 2 # optional four-eyes floor provenance: authored_by: human version: 1.0.0 ``` ## Minimal example A funnel with only `approve` is valid — it degrades to a plain multi-party approval gate: ```yaml apiVersion: swarmkit/v1 kind: Funnel metadata: id: design-signoff name: Design Sign-off description: A plain multi-party human approval gate on the design artifact. approve: rules: - scope: design:approve roles: [tech-lead] quorum: all provenance: authored_by: human version: 1.0.0 ``` ## Full example All four layers, as used on the consolidated-design artifact in the SDLC pipeline example ([`examples/sdlc-pipeline/workspace/funnels/consolidated-design-approval.yaml`](https://github.com/delivstat/swarmkit/blob/main/examples/sdlc-pipeline/workspace/funnels/consolidated-design-approval.yaml)): ```yaml apiVersion: swarmkit/v1 kind: Funnel metadata: id: consolidated-design-approval name: Consolidated Design Approval description: > Full four-layer gate — deterministic schema validation, an LLM-as-judge rubric score, an architect harness review, then a multi-party human approval that is the only exit. validate: schema: schemas/consolidated-design.json autocorrect: true judge: skill: artifact-judge rubric: rubrics/consolidated-design.md threshold: 0.8 max_retries: 2 review: archetype: architect-reviewer read_scope: [app:oms, app:web, app:mobile] route_back_at: high approve: rules: - scope: design:approve roles: [oms-lead, web-lead, mobile-lead] quorum: all - scope: security:approve roles: [infosec-lead] quorum: all exclude_author: true min_distinct_approvers: 2 provenance: authored_by: human version: 1.0.0 ``` ## Authoring a funnel The conversational authoring path treats a funnel like any other artifact: the schema drafter calls `get_schema("funnel")` for the exact shape, and `query-swarmkit-docs` surfaces this reference and the design note. When authoring a funnel, decide which layers the artifact warrants (`review` is for heavyweight gates only), keep the fixed `validate → judge → review → approve` semantics in mind, and remember that `approve` is required and the funnel is referenced by id from the node it gates. ## See also - [Gate funnel design note](https://github.com/delivstat/swarmkit/blob/main/design/details/gate-funnel.md) — the authoritative composition, control flow, bounded retry, structural invariant, and provenance bundle. - [Skills](skills.md) — the decision skill the `judge` layer instantiates. - [Archetypes catalogue](archetypes.md) — the reviewer archetype the `review` layer uses. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/contract.md ## Published: https://delivstat.github.io/swarmkit/reference/contract/ ══════════════════════════════════════════════════════════════════════════════ # Contract A **contract** is a first-class SwarmKit artifact (`kind: Contract`) that names an **integration contract**: the agreed interface between two (or more) applications, identified by id. It exists so the locks your sequencer holds reference **real** contracts instead of free-form strings — a checked, pickable vocabulary rather than opaque lock names. Why the registry exists, the lock ref-check, and the non-goals are specified in the design note: [integration-contract registry](https://github.com/delivstat/swarmkit/blob/main/design/details/contract-registry.md) (`design/details/contract-registry.md`). This page is the artifact reference. ## What a contract is for A delivery flow serialises work on the **integration contracts** it shares. `locks: [oms-web, oms-inventory]` mean "hold the OMS↔Web and OMS↔Inventory interfaces while I change them, so no concurrent requirement commits a conflicting version." Those locks are what keep two pieces of work that both touch the same app-pair interface from racing. Before contracts were artifacts, lock ids were **free-form strings**: nothing checked them, so a typo (`oms-web` vs `oms_web`) silently became a *different* lock — and two requirements that should serialise did not. The composer could only offer a free-text chip, and the contention overlay ("which stages fight over the same contract") was approximate. The contract itself — the agreed interface between two apps — had no home. Making each integration contract a first-class artifact turns lock ids into a **checked, pickable vocabulary**: the resolver rejects a lock that names no contract, the editor offers a picker over real contracts, and contention is exact. ## Locking and contention A lock **is** an integration contract; the free-string form was the placeholder. A stage acquires its `locks` **all-or-none, in a fixed global order**, *before* its run starts (deadlock avoidance), and releases them on the signal named by `release_locks_on` — for example, hold a contract through design approval, then release on `design.approved`. Unrelated requirements whose stages lock **disjoint** contracts still run in parallel; only stages that hold the **same** contract id serialise. A contract is **not executed**. **Your application is the lock manager** — SwarmKit stopped sequencing anything in 1.189.0 (see [Extracting the pipeline](../design-notes/extracting-the-pipeline.md)); the registry only makes the vocabulary real. A contract's `parties` let that manager group locks by the app-pair they bind, which is what makes a contention view exact and labelable. ## Referenced by locks A contract is a standalone artifact, like a skill or a funnel. It lives in a `contracts/` directory in the workspace and is referenced **by id** by whatever holds the lock. Defining it once and referencing it by id is what lets many pieces of work hold the same real interface, and lets the composer `ref`-validate lock names against the workspace before publish. Contracts are discovered into `ResolvedWorkspace.contracts` (id → resolved contract, like funnels and roles), so a sequencer can resolve a lock name to a real artifact — and reject one that names nothing — instead of trusting a string. The ref-check that ran against a `StageGraph`'s `locks` went with the stage graph itself; the registry it checked against did not. ## Contract fields | Field | Required | What it does | |---|---|---| | `parties` | yes | The applications this contract binds — **at least two**. This is what makes it a contract (an interface *between* apps), and it drives the contention / ownership display. App ids are free strings; apps are **not** artifacts. | | `interface` | no | A pointer to where the interface itself lives (an API / event schema). **Not interpreted by core** — documentation plus a handle for reviewers. | Core does not parse or diff the `interface` spec — that is a contract-testing / SIT concern, not this registry's. The registry governs **identity + locking**, not interface compatibility. ## Schema shape ```yaml apiVersion: swarmkit/v1 kind: Contract metadata: id: # the contract id — this is what `locks` reference name: description: # min 10 chars parties: [, , ...] # at least two; free strings, not artifact ids interface: # optional pointer to the interface spec; not parsed by core provenance: authored_by: human version: 1.0.0 ``` Only `apiVersion`, `kind`, `metadata`, `parties`, and `provenance` are required; `interface` is optional. ## Minimal example The OMS↔Web order interface, with no pointer to the spec: ```yaml apiVersion: swarmkit/v1 kind: Contract metadata: id: oms-web name: OMS ↔ Web order API description: The order-submission + status API OMS exposes to the Web storefront. parties: [oms, web] provenance: authored_by: human version: 1.0.0 ``` ## Full example The OMS↔Inventory contract, pointing at the interface spec a reviewer can open: ```yaml apiVersion: swarmkit/v1 kind: Contract metadata: id: oms-inventory name: OMS ↔ Inventory reservation API description: The stock-reservation + release events OMS exchanges with Inventory. parties: [oms, inventory] interface: schemas/oms-inventory.json # workspace-relative; documentation only, not parsed by core provenance: authored_by: human version: 1.0.0 ``` Your sequencer then holds them by id. The shape is **yours** — SwarmKit no longer defines one — but the ids are checkable against the workspace, which is the whole point of the registry: ```python # your orchestrator, your dataclass Stage(id="design", topology="sdlc-design", locks=("oms-web", "oms-inventory")) ``` `ResolvedWorkspace.contracts` is how you check a lock name before you take it, so a typo fails where you can see it rather than silently becoming a different lock that serialises nothing. ## Authoring a contract The conversational authoring path treats a contract like any other artifact: the schema drafter calls `get_schema("contract")` for the exact shape, and `query-swarmkit-docs` surfaces this reference and the design note. The authoring swarm writes the artifact into the workspace `contracts/` directory via `write_workspace_file`. When authoring, remember: `parties` needs **at least two** app ids (a contract is an interface *between* apps); the `id` is what a lock references, so it must match the lock your sequencer expects; and `interface` is optional and documentation-only — core never parses it. ## See also - [Integration-contract registry design note](https://github.com/delivstat/swarmkit/blob/main/design/details/contract-registry.md) — why lock ids become a checked vocabulary and the non-goals (no interface-content validation, no app artifacts, no new lock manager). - [Driving SwarmKit from your application](orchestrator-integration.md) — where the lock manager lives now. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/role-registry.md ## Published: https://delivstat.github.io/swarmkit/reference/role-registry/ ══════════════════════════════════════════════════════════════════════════════ # Role registry A **role registry** is a first-class SwarmKit artifact (`kind: RoleRegistry`) that maps each governance **role** to the human identities that hold it and the governance **scopes** it confers. It is workspace-level IAM data: the single place a person's authority is recorded, so a handover is one membership edit. Multi-party [approval policies](approval-policy.md) — including the `approve` layer of a [Funnel](funnel.md) — resolve their `roles` against this registry. The RBAC model, registry-driven reserved scopes, and gate resolution are specified in the [multi-party approval design note](https://github.com/delivstat/swarmkit/blob/main/design/details/multi-party-approval.md). This page is the artifact reference. ## What a role registry is for Real approvals are plural and role-based: a design needs every app lead *and* InfoSec; a release needs the engineering manager *and* the CIO. A role carries **many scopes** (identity → role → scopes), so membership lives in one place per person and cannot silently drift. People join and leave independently of any topology, so the registry versions like any other artifact. **Registry-driven reserved scopes.** Any scope conferred by any role is a *human-identity scope*: the policy engine refuses to grant it to a non-human (agent) principal — the same structural mechanism as the built-in reserved scopes (`skills:activate`, `iam:modify`, …), but driven by the registry rather than a hardcoded list. No agent can hold or satisfy an approval scope, regardless of prompt. ## Role fields Required top-level: `apiVersion`, `kind`, `metadata`, `roles` (at least one). | Field (per role) | Required | What it does | |---|---|---| | `id` | yes | Lowercase-kebab role id, unique in the workspace (enforced at load time). | | `members` | yes | Human identity references that hold this role. May be **empty** for an unstaffed role — but a gate requiring an unstaffed role cannot reach quorum. | | `scopes` | yes | Governance scopes this role confers, each of the form `:` (e.g. `design:approve`). At least one. | ## Schema shape ```yaml apiVersion: swarmkit/v1 kind: RoleRegistry metadata: id: name: description: roles: # at least one; ids unique - id: members: [, ...] # may be empty (unstaffed -> cannot reach quorum) scopes: [, ...] # at least one ``` ## Example ```yaml apiVersion: swarmkit/v1 kind: RoleRegistry metadata: id: sdlc-roles name: SDLC Approval Roles roles: - id: oms-lead members: [alice] scopes: [design:approve, code:approve, deploy:approve] - id: web-lead members: [bob] scopes: [design:approve, code:approve] - id: infosec-lead members: [dana] scopes: [security:approve] - id: cio members: [heidi] scopes: [release:approve] ``` An [approval policy](approval-policy.md) (or a Funnel `approve` block) then references these roles, and validation checks that every role named in a rule actually confers that rule's scope. ## Authoring a role registry `get_schema("role-registry")` returns the exact shape. A role's `scopes` are what let a rule name it (a rule referencing a role that does not confer the rule's scope is rejected at load time); membership is per-identity, so keep one role per capacity rather than duplicating people across roles when four-eyes independence matters. ## See also - [Multi-party approval design note](https://github.com/delivstat/swarmkit/blob/main/design/details/multi-party-approval.md) — RBAC model, reserved scopes, task decomposition, and audit. - [Approval policy](approval-policy.md) — the per-gate rules that resolve against this registry. - [Funnel](funnel.md) — its `approve` layer is a multi-party approval policy over these roles. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/trigger.md ## Published: https://delivstat.github.io/swarmkit/reference/trigger/ ══════════════════════════════════════════════════════════════════════════════ # Trigger A **trigger** is a first-class SwarmKit artifact (`kind: Trigger`) that describes something which causes one or more topologies to run — a schedule, a webhook, a filesystem watch, a manual "run now", or a third-party plugin. It unifies schedules and triggers under one kind with a `type` discriminator. The unification decision, per-type config shapes, and what stays out of the schema are specified in the [trigger schema design note](https://github.com/delivstat/swarmkit/blob/main/design/details/trigger-schema-v1.md). This page is the artifact reference. ## Fields Required top-level: `apiVersion`, `kind`, `metadata`, `type`, `targets`. | Field | Required | What it does | |---|---|---| | `type` | yes | Discriminator: `cron` \| `webhook` \| `file_watch` \| `manual` \| `plugin`. Per-type `config` shape is validated at runtime. | | `targets` | yes | Topology IDs to fire (at least one). Fired **in parallel**. | | `enabled` | no | Default `true`. A disabled trigger loads but does not fire — pause without deleting. | | `provider_id` | conditional | **Required when `type: plugin`** — names a registered `TriggerProvider`. | | `config` | conditional | **Required for `cron`, `webhook`, `file_watch`, `plugin`** (not `manual`). Type-specific, runtime-validated. `config.auth` is a schema-validated block meaningful only for `webhook`. | ### Type / config summary | `type` | Fired by | `config` (runtime-validated) | |---|---|---| | `cron` | schedule tick | `{ expression: <5/6-field cron>, timezone?: , input?: }` — `timezone` is the zone the expression is written in; without `input` the run receives `trigger:` | | `webhook` | HTTP POST to `/hooks/` | `{ auth?: {...} }` — the route is the target topology's name; the body's `input` field, or the whole JSON body as text, is the run's input | | `file_watch` | filesystem change | `{ root, pattern, events: [...] }` | | `manual` | `POST /run/{topology}` (or the portal's "Run now") | none required | | `plugin` | a registered `TriggerProvider` | `provider_id` + arbitrary config | ### Webhook `auth` Optional, and the one config field the schema validates. Required members `method` (`hmac` \| `bearer` \| `api_key`) and `credentials_ref` (the name of a workspace `credentials` entry holding the secret); optional `header` (defaults: `X-Hub-Signature-256`, `Authorization`, `X-API-Key`). `hmac` verifies `sha256=` over the body; `bearer` expects `Authorization: Bearer `; `api_key` expects the secret itself in the header. Secrets are never stored on the trigger — only referenced by name. A webhook that declares `auth` is admitted past serve's API-key gate and verified by this method instead; one that declares none stays behind the gate. A `credentials_ref` whose secret is not set refuses the delivery (503) rather than accepting it unsigned. ## Schema shape ```yaml apiVersion: swarmkit/v1 kind: Trigger metadata: id: name: description: type: cron # cron | webhook | file_watch | manual | plugin enabled: true # default true targets: [, ...] # at least one; fired in parallel provider_id: # required only when type: plugin config: # required for cron/webhook/file_watch/plugin expression: "0 9 * * 1-5" timezone: Europe/London ``` ## Examples Smallest valid (a manual trigger needs no config): ```yaml apiVersion: swarmkit/v1 kind: Trigger metadata: { id: run-review, name: Run Code Review } type: manual targets: [code-review-swarm] ``` An HMAC-authenticated GitHub webhook: ```yaml apiVersion: swarmkit/v1 kind: Trigger metadata: id: github-pr-review name: GitHub PR Review description: Runs the review swarm on every pull_request event. type: webhook targets: [code-review-swarm] config: path: /hooks/github-pr auth: method: hmac credentials_ref: github-webhook-secret ``` ## Not in the schema Secrets (referenced by `auth.credentials_ref`, never stored inline), input-to-topology mapping semantics, trigger chaining, retry/DLQ, and rate limiting are all runtime or governance concerns — the schema captures shape only. By convention cron-ish triggers live in a `schedules/` directory and everything else in `triggers/`; the runtime reads both. ## See also - [Trigger schema design note](https://github.com/delivstat/swarmkit/blob/main/design/details/trigger-schema-v1.md) — the unification rationale, per-type config, and plugin path. - [Triggers & Canary tutorial](../tutorials/12-triggers-canary.md). ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/executor-adapter.md ## Published: https://delivstat.github.io/swarmkit/reference/executor-adapter/ ══════════════════════════════════════════════════════════════════════════════ # Executor adapter An **executor adapter** is a first-class SwarmKit artifact (`kind: ExecutorAdapter`) that teaches SwarmKit how to run an external agentic **harness** — Claude Code, Codex, OpenCode, or any goal-pursuing subprocess — as a node executor. It is a **declarative** adapter: a subprocess launch template plus a mapping from the harness's line-delimited JSON output into SwarmKit's normalized `ExecEvent` vocabulary. A new harness is added as *data* (this artifact), with no Python and no runtime release. The provider-seam placement, the two-tier adapter model, the normalized event schema, and the mid-run interaction model are specified in the [executor abstraction design note](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-abstraction.md). This page is the artifact reference. ## Executor vs. skill An executor answers *how a node does its work*; a skill answers *what capability an agent may invoke*. The dividing rule: **if it produces a diff or holds a session, it is an executor; if it answers a question and returns, it may be a skill.** `executor` is a node-execution provider seam alongside `ModelProvider` and `GovernanceProvider` — not a parallel capability primitive. ## Two ways a harness enters a topology 1. **The archetype selects an executor.** An archetype's optional `executor` block declares how its nodes run: ```yaml executor: kind: harness # model (default) | harness | ref: claude-code # for kind: harness, the adapter id (required) version_constraint: ">=2.1" # optional; interpreted by the adapter config: # opaque to core; validated by the adapter's own schema permission_mode: bare ``` `executor` is **optional and backward-compatible**: absent means `kind: model` with the archetype's `defaults.model`. `kind` is not a closed enum — it is validated against the executor registry at runtime. 2. **The adapter itself is the `ExecutorAdapter` artifact.** It lives in the workspace (e.g. an `adapters/` directory) and is what `ref:` resolves to. ## Adapter fields Required top-level: `apiVersion`, `kind`, `metadata`, `spec`, `provenance`. Within `spec`, required: `launch`, `stream`, `event_map`. | Field (`spec.`) | Required | What it does | |---|---|---| | `launch` | yes | How to launch the subprocess. `command` is argv (no shell); values are templated with a closed variable set (`{task.statement}`, `{sandbox.root}`, `{budget.max_turns}`, `{credential.model_provider}`, `{config.*}`, …). `optional_args` append arg-groups only when a variable is set; `env` injects env vars. | | `stream` | yes | `format: jsonl` (line-delimited JSON only). `retain_raw: true` tees each untranslated vendor line as `exec.raw`. | | `event_map` | yes | Rules that match a parsed JSON line (literal-equality on dotted paths), optionally `for_each` an array, `set` state (only `session_id`), and `emit` `ExecEvent`s (`started`, `message`, `tool_call`, `artifact`, `usage`, `approval_requested`, `input_requested`, `result`, `raw`). Field values are `$.dotted.path` extractions or literals; `{from, map}` translates through a named `status_map`. | | `auth` | no | Which auth modes the harness supports (`api_key`, `subscription`), expressed generically as env vars, args, and/or `credential_paths`. | | `status_map` | no | Vendor discriminator → `ExecResultStatus` (`success`/`failure`/`budget_exceeded`/`cancelled`/`needs_approval`/`stalled`); `_default` covers the rest. | | `resume` | no | Makes resume-token support declarative — replay the captured `session_id` into a retry/resume launch. | | `success_when` | no | Terminal success predicate (`exit_code`). Core layers a semantic check (typed output + artifact-manifest match) on top — exit code alone is necessary, not sufficient. | | `on_unanswerable` | no | `deny` \| `abort` (default) \| `relay` — how a mid-run request outside the launch grant is handled. `relay` requires an `interaction` block. | | `interaction` | conditional | Required when `on_unanswerable: relay`. `driver: hold-stream \| park-resume`; optional `max_approval_wait_seconds` (never-hang bound). | | `sandbox` | no | Isolation tier. Absent ⇒ native git-worktree (default). `kind: container` runs the harness in docker/podman with an `image` or a `build` (`base` + `install`, or a `dockerfile` path relative to the workspace, or `dockerfile_inline` to keep the adapter one file), `mounts`, a `network` policy (`deny` default \| `allowlist` + `allow: [hosts]`), and `resources` (`cpus`, `memory` e.g. `2g`, `pids`). | | `launch.mcp_tool_name` | no | How this harness spells a governed gateway tool in its own tool grant: `{tool}` is the gateway's flat `__` name, `{gateway}` the MCP server SwarmKit registers (`swarmkit`). Declared per adapter because the mangling is harness-native — Claude Code's is `mcp__{gateway}__{tool}`. A grant written in any other spelling matches nothing. | | `grant` | no | How an approved capability set is replayed on a grant-expanding resume under `relay`: `arg` (e.g. `[--allowedTools, "{grant.capabilities}"]`) and `separator` (default a space) joining the approved capabilities into `{grant.capabilities}`. | | `telemetry_grade` | no | `normalized` (default) \| `opaque`. Opaque (unobservable) adapters are denied by default and need explicit per-archetype opt-in. | | `requires` | no | `code` — set only when the adapter has hit the declarative DSL ceiling and must graduate to a Tier-1 Python executor. | The DSL is deliberately minimal: JSONL only, literal-equality matching, dotted-path extraction, one named enum-translation map. Mid-run `relay` interaction is the single Tier-1 seam. ## Schema shape ```yaml apiVersion: swarmkit/v1 kind: ExecutorAdapter metadata: id: # this is the executor kind/ref an archetype selects name: description: # min 10 chars spec: launch: command: [, ...] # no shell; value-only substitution optional_args: - when: budget.max_turns args: ["--max-turns", "{budget.max_turns}"] env: SOME_CONFIG: "{config.foo}" auth: default: api_key modes: api_key: env: { ANTHROPIC_API_KEY: "{credential.model_provider}" } subscription: credential_paths: ["~/.claude"] stream: format: jsonl # jsonl only retain_raw: true event_map: - when: { type: assistant } emit: - event: message with: { role: assistant, text: "$.message.content" } - when: { type: result } set: { session_id: "$.session_id" } emit: - event: result with: status: { from: "$.subtype", map: status_map } cost_usd: "$.total_cost_usd" status_map: { success: success, error_max_turns: budget_exceeded, _default: failure } success_when: { exit_code: 0 } sandbox: kind: worktree # worktree (default) | container telemetry_grade: normalized provenance: authored_by: human version: 1.0.0 ``` ## Authoring an executor adapter The `launch` block is the sharpest edge (it is a command line): declarative adapters carry a mandatory human-review gate on first approval and on any change to `launch`, regardless of workspace auto-run trust. Keep to the DSL ceiling — if you need resume logic beyond a token replay, bidirectional streaming, or non-line output, declare `requires: code` and graduate to a Tier-1 Python executor. `get_schema("executor-adapter")` returns the exact shape for the conversational authoring path. ## See also - [Executor abstraction design note](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-abstraction.md) — provider-seam placement, `ExecEvent` vocabulary, mid-run permission/input handling. - [Archetypes catalogue](archetypes.md) — the `executor` block lives on an archetype. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/model-provider.md ## Published: https://delivstat.github.io/swarmkit/reference/model-provider/ ══════════════════════════════════════════════════════════════════════════════ # Model provider A **model provider** is a first-class SwarmKit artifact (`kind: ModelProvider`) that points the runtime at an LLM endpoint. It is **data**: a base URL, an auth shape, a model catalogue, capabilities and request quirks — layered over a wire-format **family** that stays code. A new OpenAI- or Ollama-compatible endpoint is a YAML file and nothing else; no Python, no runtime release. The decision — what is data and what is code — is in the [declarative model providers design note](https://github.com/delivstat/swarmkit/blob/main/design/details/declarative-model-providers.md). This page is the artifact reference. ## The four families There are four wire formats. They are Python, and they are the only Python: | family | speaks | what extends it | |---|---|---| | `openai-compatible` | OpenAI chat-completions | OpenAI, OpenRouter, Groq, Together, llama.cpp `llama-server`, OpenVINO Model Server, vLLM, mlx-lm, Lemonade — every OpenAI-compatible server | | `ollama` | Ollama's native `/api/chat` | Ollama, rkllama (Rockchip NPU) | | `anthropic` | Anthropic Messages | Anthropic | | `google` | Google GenAI | Google (API-key mode) | A provider YAML names one via `extends` and parameterises it. Anything a family cannot express — OAuth token acquisition, a fifth wire format — is a new family, added as code, exactly as an executor past the declarative ceiling becomes a Tier-1 Python executor. ## The artifact ```yaml apiVersion: swarmkit/v1 kind: ModelProvider metadata: id: rkllama name: rkllama — Rockchip NPU description: Ollama-API server for RK3588/RK3576 NPUs, running pre-converted .rkllm models. spec: extends: ollama base_url: ${RKLLAMA_HOST:-http://localhost:8080} capabilities: tools: false # narrows the family; may never widen it provenance: authored_by: human version: 1.0.0 ``` Same envelope as every other artifact — `apiVersion`, `kind`, `metadata`, `spec`, `provenance` — so the validator, the portal and the codegen treat it like one. ## `spec` fields Required: `extends`. `spec` refuses unknown fields — the DSL is deliberately small and a template language cannot grow in by accident. | Field (`spec.`) | Type | What it does | |---|---|---| | `extends` | id | **Required.** A family, or another provider's id. Resolved at load: the chain must end at a family and may not revisit an id. `extends: ollama` always means the *family*, never the bundled `ollama.yaml` — families and providers share a namespace and the family wins. | | `base_url` | string | Endpoint. `${VAR:-default}` substitution applies, the same resolver as workspace env config. | | `auth.api_key_env` | string | Env var holding the key. **Absent means no auth** — a local runtime, which registers unconditionally. | | `auth.header` | string | Default `Authorization`. Azure's key mode is `api-key`. `openai-compatible` and `ollama` only. | | `auth.scheme` | string | Default `Bearer`; empty string for a bare key. `openai-compatible` and `ollama` only. | | `models.pattern` | regex | What `supports()` accepts. Default is the family's own catalogue (`^gpt-…`, `^claude-`, `^gemini-`). | | `models.accept_any` | bool | Aggregators and local runtimes: the catalogue is unbounded and the server validates. Ollama's default. | | `capabilities.*` | bool | `images`, `tools`, `streaming`, `structured_output`. **Narrow-only** — a `true` the family does not offer is a load-time error. A narrowed capability is enforced: a request carrying tools to a `tools: false` provider is refused before the wire. `structured_output: false` withholds `response_format` and keeps the schema in the prompt. | | `options.lift_to_root` | list | Option keys moved from `options` to the payload root. Ollama's default is `[think, keep_alive]`. `ollama` only. | | `headers` | map | Static extra request headers (OpenRouter's `HTTP-Referer`, `X-Title`). | | `extra_body` | map | Fields merged into every request body that the base API does not define — OpenRouter's `usage: {include: true}` for per-call cost. `openai-compatible` and `ollama` only. | | `requires` | `code` | Declares the provider cannot be expressed here. Load **refuses** it with the file and reason, so a YAML past the ceiling fails loudly rather than half-works. | A field the family would ignore (`extra_body` on `anthropic`, say) is refused at load, not dropped: a declared quirk that never reaches the wire is exactly the half-working provider this artifact exists to prevent. ## Inheritance `extends` names a family or another provider. Resolution walks the chain to a family, merging each hop's fields over its parent's: `auth`, `models`, `capabilities`, `headers` and `extra_body` merge by key; `base_url` and `lift_to_root` replace. A child overriding `auth.header` keeps its parent's `api_key_env`. Depth is unbounded; every bundled provider is one hop. A provider three hops from its family is one nobody can read — that is a review norm, not a schema limit. ## Where providers come from ``` bundled packages/runtime/src/swarmkit_runtime/model_providers/providers/*.yaml workspace /providers/*.yaml overrides a bundled id ``` **Registration is by readiness, not by a list.** Every loaded provider registers if its `auth.api_key_env` is set, or if it declares no auth. Adding a provider is adding a file; there is nothing in Python to keep in step. The bundled library: `anthropic`, `openai`, `google`, `ollama`, `openrouter`, `groq`, `together`, plus the edge runtimes `rkllama`, `llama-server`, `openvino-model-server`, `mlx-lm`, `lemonade`. Each local runtime's `base_url` reads an env var with a default (`RKLLAMA_HOST`, `LLAMA_SERVER_URL`, `OVMS_URL`, `MLX_LM_URL`, `LEMONADE_URL`; Ollama's is `OLLAMA_BASE_URL`). ## Inspecting ``` $ swarmkit providers list anthropic anthropic bundled needs ANTHROPIC_API_KEY groq openai-compatible bundled ready (GROQ_API_KEY set) llama-server openai-compatible workspace ready (no auth) rkllama ollama bundled ready (no auth) … $ swarmkit providers show rkllama id: rkllama chain: rkllama -> ollama base_url: http://localhost:8080 auth: none models: any capabilities: images=yes, tools=no, streaming=yes, structured_output=yes lift_to_root: think, keep_alive ``` A workspace YAML declaring `requires: code` stops `list` with its file name and the reason. ## Adding one Point a workspace at an OpenAI-compatible server SwarmKit has never heard of: ```bash mkdir -p workspace/providers cat > workspace/providers/vllm.yaml <<'YAML' apiVersion: swarmkit/v1 kind: ModelProvider metadata: { id: vllm, name: vLLM, description: vLLM's OpenAI-compatible server on the GPU box. } spec: extends: openai-compatible base_url: ${VLLM_URL:-http://gpu-box:8000/v1} models: { accept_any: true } provenance: { authored_by: human, version: 1.0.0 } YAML swarmkit providers list workspace # vllm openai-compatible workspace ready (no auth) ``` Then `provider: vllm` in a topology or archetype, or `SWARMKIT_PROVIDER=vllm` for a run. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/approval-policy.md ## Published: https://delivstat.github.io/swarmkit/reference/approval-policy/ ══════════════════════════════════════════════════════════════════════════════ # Approval policy An **approval policy** is the per-gate, multi-party human-approval configuration: the rules that must **all** be satisfied for a gate to advance, plus segregation-of-duties controls. It is **embedded config, not a standalone artifact** — it has no `apiVersion`/`kind` and appears inside a gate (most commonly the required `approve` layer of a [Funnel](funnel.md)). Its `roles` resolve against the workspace [role registry](role-registry.md). The resolution model, quorum semantics, four-eyes floor, and enforcement are specified in the [multi-party approval design note](https://github.com/delivstat/swarmkit/blob/main/design/details/multi-party-approval.md). This page is the config reference. ## Fields Required: `rules` (at least one). Defaults below are applied by the runtime. | Field | Required | Default | What it does | |---|---|---|---| | `rules` | yes | — | The approval rules. **Every** rule must be satisfied for the gate to advance. | | `exclude_author` | no | `true` | The identity that authored/submitted the artifact cannot approve it (segregation of duties). | | `on_revision` | no | `reset_all` | What a revision does to prior approvals: `reset_all` invalidates all; `reconfirm_changed` keeps approvals whose scope was unaffected. | | `min_distinct_approvers` | no | — | Four-eyes floor: at least N **distinct** human identities must approve across all completed role-tasks, regardless of how roles overlap. | ### Rule fields | Field | Required | What it does | |---|---|---| | `scope` | yes | The authority exercised (`:`, e.g. `design:approve`). Every role in `roles` must confer it (validated against the role registry at load time). | | `roles` | yes | The group of roles that may exercise this rule's scope (at least one). | | `quorum` | yes | `all` (every role in the group approves) \| `any` (one suffices) \| `{ k-of: N }` (any N distinct role-holders). | **Two independent axes.** *Which roles signed* is the quorum (`all`/`any`/`k-of`); *how many independent people signed* is `min_distinct_approvers`. A single dual-hatted person can complete two role-tasks and satisfy two roles, but does **not** satisfy `min_distinct_approvers: 2` — a second identity is still required. ## Config shape ```yaml approve: # e.g. a Funnel's approve layer rules: # every rule must be satisfied - scope: design:approve roles: [oms-lead, web-lead, mobile-lead] quorum: all # all | any | { k-of: N } - scope: security:approve roles: [infosec-lead] quorum: all exclude_author: true # default true — segregation of duties on_revision: reset_all # default reset_all | reconfirm_changed min_distinct_approvers: 2 # optional four-eyes floor ``` ## How it resolves A gate fans out into **one task per required role** (`Approval from role:`), assigned to that role's members; a role-task completes when any one member approves. A person holding two required roles gets **two** tasks and completes each separately — one attributable sign-off per capacity. The gate compiles to a checkpointed `interrupt()`, so a partially-approved gate ("2 of 4, waiting on infosec + cio") is durable across weeks and restarts. None of this is promptable or agent-reachable. ## Who may resolve a role-task The resolver's identity is **load-bearing** — it is what quorum, `min_distinct_approvers` and `exclude_author` are counted against — so it is never self-asserted: | Surface | Resolver identity | Command | | --- | --- | --- | | `swarmkit serve` HTTP | the authenticated caller (`client_id`) | `POST /review/{id}/resolve` with `{"outcome": "approve"}` | | CLI | asserted via `--as` (local filesystem trust) | `swarmkit review resolve --as alice --approve` | Over HTTP the caller must also hold `approvals:resolve`, a **reserved human-identity scope**: a transport (API-key / JWT) token structurally cannot carry it, so an agent or webhook can never cast a resolution regardless of its serve tier. A request body may not supply an identity; one that tries is ignored. Membership is checked **before** the resolution is recorded, and a non-member is refused with the reason (`alice is not a member of role release-manager`) rather than silently ignored. Every attempt — allowed or denied — is written to the append-only audit as `approval.role_task_resolved`. Two deployer-facing consequences: - **Serve `client_id` and role `members` are one namespace.** An operator in role `security-reviewer` must authenticate as the identity listed in that role's `members`. - **A typo in `members` surfaces at resolve time**, as a 403, not at workspace validation — the runtime cannot enumerate an auth provider's credentials. Under the default `NoneAuthProvider` every caller is `anonymous`, so multi-party approval is not enforceable unless the workspace genuinely lists `anonymous` as a role member (which keeps local development workable, and is not a deployment posture). `swarmkit review approve|reject` are for harness gates and do **not** record an identity — they cannot satisfy a multi-party rule. Use `resolve`. ## Reading a gate's state Role-tasks serialize as `kind: "role_task"` carrying `gate_id`, `role`, `scope`, `rule_index` and `resolved_by`, so a front-end can group a gate's tasks and show which capacity each approver is acting in. Narrow the queue with `GET /review?kind=role_task&gate_id=`, or `swarmkit review list --kind role_task --gate `. `GET /gates/{gate_id}` returns the aggregate plus per-role `items` (the gate id is `:` — split on the last colon). Its `status` is evaluated through the **approval engine** whenever the gate's policy is reachable from the workspace (`quorum_evaluated: true`) — so the report matches the decision the runtime gates on. When the policy cannot be located (an externally-driven gate, or a renamed agent) it falls back to folding the items, which treats *every* task approving as the bar, and reports `quorum_evaluated: false` so a caller knows which answer it got. The two differ for any quorum other than `all`: under `quorum: any` the engine approves on the first resolution while the fold still says pending. ## See also - [Multi-party approval design note](https://github.com/delivstat/swarmkit/blob/main/design/details/multi-party-approval.md) — the authoritative resolution, quorum, overlap, and audit model. - [Pipeline gate approval note](https://github.com/delivstat/swarmkit/blob/main/design/details/pipeline-gate-approval-ui.md) — the identity model above, and where a parked run is approved. - [Role registry](role-registry.md) — where the roles and their scopes are defined. - [Funnel](funnel.md) — the artifact whose required `approve` layer *is* an approval policy. ══════════════════════════════════════════════════════════════════════════════ # Part 5 — Operate references ══════════════════════════════════════════════════════════════════════════════ ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/cli.md ## Published: https://delivstat.github.io/swarmkit/reference/cli/ ══════════════════════════════════════════════════════════════════════════════ # CLI commands ## Every command 71 commands, from the CLI itself (`swarmkit --help` for the options). | Command | What it does | |---|---| | `swarmkit adapters approve` | Approve a workspace adapter's current launch block (a human action). Inspect it with | | `swarmkit adapters build` | Warm the build-in-sandbox image cache for an adapter (no local harness install needed). | | `swarmkit adapters list` | List every available adapter kind and, for workspace adapters, its launch-approval status. | | `swarmkit adapters show` | Show an adapter's launch command + fingerprint — what a reviewer inspects before approval. | | `swarmkit artifacts get` | Print one artifact's content. | | `swarmkit artifacts list` | List artifact refs recorded under one correlation id. | | `swarmkit ask` | Ask a question about the workspace or recent runs. | | `swarmkit auth token` | Mint a serve API token: generate a strong secret and print the config to wire it. | | `swarmkit author archetype` | Author a new archetype through conversation. | | `swarmkit author mcp-server` | Author a new MCP server through conversation. | | `swarmkit author skill` | Author a new skill through conversation. | | `swarmkit author topology` | Author a new topology through conversation. | | `swarmkit chat` | Interactive multi-turn conversation with a topology. | | `swarmkit checkpoints` | List checkpointed runs that can be resumed. | | `swarmkit cited-change` | Check a change-rationale cites the code its diff changed (exit 1 if uncited). | | `swarmkit comprehension` | Comprehension-debt signals from the audit log (read-only, never a gate). | | `swarmkit connect` | Run the Mode B poll connector for a NAT'd / edge instance (design §13). | | `swarmkit conversations` | List saved conversations. Use --pick to resume one interactively. | | `swarmkit debug` | Retrieve LLM prompts and responses from the local ring buffer. | | `swarmkit docs-reader` | Launch the Document Reader MCP Server (stdio). | | `swarmkit edit` | Edit an existing workspace through conversation (M7 Skill Authoring Swarm). | | `swarmkit eval` | Run an eval-set and score the topology (design §M15). | | `swarmkit fleet enroll-token` | Mint a one-time enrollment token for a fleet to register with this instance. | | `swarmkit fleet memberships` | List the fleets registered with this instance (no secrets). Shows each membership's scope, | | `swarmkit gaps` | List recorded skill gaps. | | `swarmkit init` | Create a new SwarmKit workspace through conversation. | | `swarmkit install` | Install a SwarmKit expertise package. | | `swarmkit knowledge-pack` | Bundle SwarmKit docs + schemas + workspace state into a paste-ready prompt. | | `swarmkit knowledge-server` | Launch the SwarmKit Knowledge MCP Server (stdio). | | `swarmkit logs` | Show events from recent topology runs. | | `swarmkit mcp-serve` | Expose workspace topologies as MCP tools on stdio. | | `swarmkit memory add` | Write a fact into governed memory, through the same path an agent writes through. | | `swarmkit memory get` | Show the current memory for a (subject, attribute) key, optionally with its full history. | | `swarmkit memory quarantine` | List quarantined contradictions awaiting (or resolved by) a curator. | | `swarmkit memory resolve` | Resolve a quarantined contradiction — the one hard human gate in the memory path (§8). | | `swarmkit memory search` | Search governed memory (relevance-ranked; empty query lists all by confidence). | | `swarmkit packages` | List installed SwarmKit expertise packages. | | `swarmkit providers list` | List every declared provider, its family, its source, and whether it is ready. | | `swarmkit providers show` | Show a provider resolved through its chain — what actually reaches the family. | | `swarmkit publish` | Package a workspace for distribution. | | `swarmkit queue-stats` | Show durable job-queue health: backlog depth, oldest-unclaimed age, queue-wait and | | `swarmkit review answer` | Answer a harness input request (§6.3) with text. Inspect it first with `review show `. | | `swarmkit review approve` | Approve a pending review item. | | `swarmkit review gate` | Whether a gate is resolved, with its approval policy applied. | | `swarmkit review list` | List pending review items. | | `swarmkit review reject` | Reject a pending review item. | | `swarmkit review resolve` | Resolve a multi-party approval role-task as *identity*. | | `swarmkit review show` | Show full details of a review item. | | `swarmkit run` | One-shot execution of a topology (design §14.1). | | `swarmkit serve` | Start the SwarmKit HTTP server (design §14.1). | | `swarmkit skill add` | Add a skill (and the MCP server it needs) to this workspace. | | `swarmkit skill check` | Start each mcp_tool skill's server and ask whether its tool still exists. | | `swarmkit skill import` | Import an Agent Skills SKILL.md as an llm_prompt skill. | | `swarmkit skill list` | The workspace's skills — or, with --available, the catalogue's. | | `swarmkit skill remove` | Delete skills/.yaml — refused while an archetype or agent holds the skill. | | `swarmkit skill search` | Search the catalogue (and this workspace's own skills). | | `swarmkit skill show` | One skill (or bundle), from the workspace if it has it, else the catalogue. | | `swarmkit slice-check` | Check a diff against a slice budget — keep slices reviewable (exit 1 if over budget). | | `swarmkit status` | Show recent run status at a glance. | | `swarmkit stop` | Ask a running run to stop at its next agent boundary. | | `swarmkit storage migrate` | Copy this workspace's local SQLite rows into its configured Postgres store. | | `swarmkit storage status` | Show which backend each store resolves to, and where that decision came from. | | `swarmkit system` | Versions, storage resolution, workspace properties and environment — what this instance is. | | `swarmkit trace` | Show the agent call graph and token usage for a run. | | `swarmkit trust apply` | Apply a proposal: add the capability to the archetype's ``executor.config.allowed_tools`` and | | `swarmkit trust clear` | Lift a denial block and reset a pair's tally so it can accrue toward a proposal again. | | `swarmkit trust list` | List pending allowlist-changeset proposals (archetype ← capability + the approval count). | | `swarmkit upgrade` | Upgrade swarmkit-runtime in place, keeping its extras, after showing any breaking changes. | | `swarmkit validate` | Validate a SwarmKit workspace and print a resolved tree or errors. | | `swarmkit why` | Explain what happened in a run using an LLM. | | `swarmkit worker` | Run a worker that claims and executes queued jobs (worker-execution.md). | ## The ones you will use first | Command | Description | |---|---| | `swarmkit validate ` | Validate and resolve a workspace (`--tree` prints the resolved agent tree; `--require` reports declared config no code path reaches; `--require-verified` reports outputs nothing checks) | | `swarmkit run ` | Execute a topology one-shot (`--input "..."`, `--verbose`, `--dry-run`, `--resume`, `--correlation-id`, `--label k=v`) | | `swarmkit run ... --attach ` | Put a file in front of the entry agent; repeatable, workspace-relative. The media type is read from the file's content, not its name — hence one `--attach` rather than `--image`/`--pdf`. Images only today; a bad path or an uncarryable type fails before the run starts, and every attachment is audited by name, type, size and SHA-256, never by content | | `swarmkit serve ` | Start the HTTP server (and the portal, with the `[ui]` extra) — [Serve mode](serve.md), [HTTP API](http-api.md) | | `swarmkit chat ` | Multi-turn conversation (`--resume ` continues one) | | `swarmkit conversations ` | List saved conversations (`--pick` to resume one interactively) | | `swarmkit providers list [workspace]` | Every model provider, its family, and whether its key is set — [Model provider](model-provider.md) | | `swarmkit adapters list [workspace]` | Every harness adapter and, for workspace adapters, its launch-approval status — [Executor adapter](executor-adapter.md) | ### Chat features The chat mode uses `prompt_toolkit` for a full terminal experience: - **Arrow keys**: up/down for history, left/right for cursor movement - **History search**: Ctrl+R to search previous inputs - **Persistent history**: saved across sessions in `~/.swarmkit/chat_history` - **Auto-complete**: topology commands and built-in commands ### Chat commands These commands work inside `swarmkit chat`: | Command | Description | |---|---| | `/model` | Show current model and provider | | `/model ` | Switch all agents to a different model (e.g. `/model deepseek/deepseek-chat`) | | `/model reset` | Reset to topology YAML defaults | | `exit` / `quit` / `bye` | End the conversation | ## Authoring All authoring commands use `prompt_toolkit` with history and arrow key support. | Command | Description | |---|---| | `swarmkit init [path]` | Create a workspace through conversation | | `swarmkit edit ` | Edit a workspace through conversation | | `swarmkit author topology ` | Author a topology | | `swarmkit author skill ` | Author a skill | | `swarmkit author archetype ` | Author an archetype | | `swarmkit author mcp-server ` | Author an MCP server (scaffolds Python + skill YAML + workspace entry) | Add `--thorough` to use the multi-agent authoring swarm instead of the single agent. ### Authoring provider By default, authoring uses Ollama (local). Override with environment variables: ```bash SWARMKIT_PROVIDER=openrouter SWARMKIT_MODEL=deepseek/deepseek-chat \ swarmkit author skill . ``` ## Observability | Command | Description | |---|---| | `swarmkit status ` | Recent runs at a glance — reads from AuditProvider (SQLite), falls back to JSONL | | `swarmkit logs ` | Detailed events from past runs. Filters: `--last N`, `--run-id`, `--agent`, `--topology`, `--format markdown` | | `swarmkit why ` | LLM-powered explanation — reads from AuditProvider, falls back to JSONL | | `swarmkit ask "question" -w ` | Conversational observer with structured audit context. Use `--run ` to scope | | `swarmkit debug ` | Query local prompt ring buffer (prompts never leave your machine) | | `swarmkit debug ... --span-id ` | Retrieve prompt/response for a specific OTel span | | `swarmkit debug ... --run-id ` | All prompts for a run | | `swarmkit debug ... --agent -n 5` | Last N prompts for an agent | | `swarmkit stop ` | Ask a run to stop at its next agent boundary. Cooperative, not a kill: a call in flight finishes first, the run keeps everything it has already done, and it resumes with `swarmkit run … --resume`. Works across processes — it writes a durable flag, so it can stop a run `swarmkit serve` started. Stopping a finished run is a no-op, not an error. | ### Data sources Events are persisted to `.swarmkit/audit.sqlite` (SQLite, default) after every `swarmkit run`. All observability commands read from this store via `WorkspaceRuntime.audit_provider_for()` — the same service layer the web UI will use. JSONL logs (`.swarmkit/logs/`) are kept as a fallback. Prompts are stored separately in `.swarmkit/prompts.sqlite` (local ring buffer). They never leave your environment — use `swarmkit debug` to access them. ### Audit redaction Skills can declare audit policies in YAML: ```yaml audit: log_inputs: summary # full | summary | none log_outputs: full redact: ["$.password", "$.api_key"] ``` Redacted fields appear as `[REDACTED]` in all outputs. Summary mode truncates long values. Workspace-level `audit.level` (minimal/standard/detailed) clamps all skills. ## Review + gaps | Command | Description | |---|---| | `swarmkit review list [--kind permission\|input\|role_task] [--gate ]` | List pending HITL review items | | `swarmkit review show ` | Show review details | | `swarmkit review approve ` | Approve a review | | `swarmkit review reject ` | Reject a review | | `swarmkit review resolve --as [--approve\|--reject] ` | Resolve a multi-party approval role-task, recording the resolver (checked against the [role registry](role-registry.md)) | | `swarmkit gaps ` | List recorded skill gaps | ## Knowledge | Command | Description | |---|---| | `swarmkit knowledge-pack [--lean] [-o file]` | Bundle the corpus for an LLM: `--lean` (~190k tokens: overview, generated CLI/HTTP reference, schemas, design doc, guides) or full (~610k tokens: plus every design note, historical ones last under a banner) | | `swarmkit knowledge-server` | Launch Knowledge MCP Server (stdio, for Claude Code / Cursor) | ## Runtime behaviour ### Multi-turn tool loop When an agent makes tool calls, the runtime executes them and feeds results back to the model for synthesis. The model can make additional tool calls — up to `SWARMKIT_MAX_TOOL_TURNS` rounds (default: 50; `SWARMKIT_MAX_PER_TOOL` caps calls to any one search/write tool at 8 and `SWARMKIT_MAX_PER_READ_TOOL` any one read-only tool at 50). If the model responds with planning language ("let me examine...") instead of tool calls, the runtime nudges it to act. ### Conversation context Worker agents receive the full conversation history from prior turns, so they can see previous findings and avoid redundant tool calls. ### Path sanitisation When models send absolute file paths (common with grep results), the runtime converts them to relative paths within the MCP server's working directory. ### Verbose mode Set `SWARMKIT_VERBOSE=1` or use `--verbose` to see per-agent detail: ``` --- [sterling-developer] calling deepseek/deepseek-chat --- tools: ['grep-project-code', 'read-file-lines', 'verify-code-citations', ...] input: Describe the Java class... tool_calls: ['grep-project-code'] executing: grep-project-code [mcp args: {'pattern': 'SourcingRule'}] [tool loop turn 1: 1 tool results] executing: read-file-lines [mcp args: {'path': './java-code/src/.../Agent.java', 'start_line': 2080, 'end_line': 2216}] [tool loop turn 2: 1 tool results] [synthesis call with 2 tool results] ``` ## Storage + system info | Command | Description | |---|---| | `swarmkit storage status [ws]` | Which backend each store resolves to, and which setting decided it | | `swarmkit storage migrate [ws]` | Copy local SQLite rows into the configured Postgres (`--dry-run`, `--yes`) | | `swarmkit system [ws]` | Versions, storage, workspace properties and environment (`--all` includes unset vars) | Secrets are masked in all three: a path listed under `secrets:` in `workspace.env.yaml` prints as `set`, and connection URLs print without the password. See [Storage](storage.md) for the full SQLite → Postgres runbook. ## HTTP server endpoints Started via `swarmkit serve [--port 8000] [--host 0.0.0.0]`. Every endpoint is listed in the generated [HTTP API](http-api.md) reference; the prose — auth, triggers, attachments, streaming — is in [Serve mode](serve.md). ## Environment variables 52 variables, from the runtime's own registry (`swarmkit system` and `GET /system` report the same list, secrets masked). A variable the code reads and the registry does not know fails a test. **Storage** | Variable | Purpose | |---|---| | `SWARMKIT_STORE_URL` | Connection URL for every store. Set alone it also SELECTS postgres — a URL names its own backend. Overrides storage.runtime.url in workspace.yaml. *(URL; userinfo masked)* | | `SWARMKIT_STORE_BACKEND` | Force the backend (sqlite \| postgres) regardless of workspace.yaml. Optional: setting only the URL is enough. | | `SWARMKIT_STORE_POOL_SIZE` | Postgres connection pool size for the shared store engine (default 20). Raise for a higher server.jobs.max_concurrent or more instances; keep (pool + overflow) x instances under Postgres max_connections. Ignored for SQLite. | | `SWARMKIT_STORE_MAX_OVERFLOW` | Extra Postgres connections the pool may open beyond SWARMKIT_STORE_POOL_SIZE under burst (default 10). Ignored for SQLite. | | `DATABASE_URL` | Fallback connection URL when SWARMKIT_STORE_URL is unset. *(URL; userinfo masked)* | | `SWARMKIT_WORKSPACE` | Default workspace root for commands that omit it. | | `SWARMKIT_SKILLS_CATALOGUE` | Where `swarmkit skill` reads the catalogue: a checkout directory or a mirror URL (default: the swarmkit-skills repo on GitHub). | | `SWARMKIT_GATES_DIR` | Where file-backed approval gates are written. | **Models** | Variable | Purpose | |---|---| | `SWARMKIT_PROVIDER` | Default model provider when a topology names none. | | `SWARMKIT_MODEL` | Default model when a topology names none. | | `SWARMKIT_JUDGE_MODEL` | Model used by governance decision skills. | | `SWARMKIT_AUTHOR_MODEL` | Model used by the authoring swarms. | | `SWARMKIT_MODEL_TIMEOUT` | Per-call timeout in seconds. | | `SWARMKIT_MODEL_RETRIES` | Retries per model call before the node fails. | | `ANTHROPIC_API_KEY` | Anthropic credential. *(secret)* | | `OPENAI_API_KEY` | OpenAI credential. *(secret)* | | `OPENROUTER_API_KEY` | OpenRouter credential. *(secret)* | | `GOOGLE_API_KEY` | Google GenAI credential. *(secret)* | **Run limits** | Variable | Purpose | |---|---| | `SWARMKIT_MAX_TOOL_TURNS` | Tool-calling turns before a node is cut off. | | `SWARMKIT_MAX_TOOLS` | Tools exposed to one agent. | | `SWARMKIT_MAX_RESULT_CHARS` | Truncation ceiling for a tool result. | | `SWARMKIT_MAX_DELEGATIONS_PER_CHILD` | Delegation fan-out cap per child. | | `SWARMKIT_MAX_PER_TOOL` | Calls to one search/write tool per turn (8). | | `SWARMKIT_MAX_PER_READ_TOOL` | Calls to one read-only tool per turn (50). | | `SWARMKIT_READ_TOOL_PREFIXES` | Comma-separated name prefixes that mark a tool read-only (read-, get-, list-, ...). | | `SWARMKIT_READ_TOOLS` | Comma-separated tool names treated as read-only. | | `SWARMKIT_ATTACHMENT_MAX_BYTES` | Ceiling per run attachment before the upload (20 MiB). | | `SWARMKIT_AGENT_RETRIES` | Retries for a failing agent node. | | `SWARMKIT_HISTORY_TURNS` | Conversation turns replayed into context. | | `SWARMKIT_CONTEXT_COMPRESSION` | Enable read-side context compression (off by default). | | `SWARMKIT_CONTEXT_COMPRESSION_MIN_BYTES` | Payload size below which compression is skipped. | **Testing** | Variable | Purpose | |---|---| | `SWARMKIT_MOCK_DELEGATE` | When set to 1, the mock provider delegates to every child so a mock run traverses a multi-agent topology (tests only). | | `SWARMKIT_MOCK_LATENCY_MS` | Base wall time in ms the mock provider sleeps per model call, so a load test measures the runtime under realistic concurrency rather than an instant mock (0/unset = instant). | | `SWARMKIT_MOCK_LATENCY_JITTER_MS` | Uniform +/- jitter in ms added to SWARMKIT_MOCK_LATENCY_MS, so concurrent mock calls do not wake on the same tick (real model variance). | **MCP + sandbox** | Variable | Purpose | |---|---| | `SWARMKIT_MCP_TIMEOUT` | Per-call MCP timeout in seconds. | | `SWARMKIT_OAUTH_KEY` | Key that encrypts stored OAuth tokens. Generated into .swarmkit/oauth.key when unset. *(secret)* | | `SWARMKIT_OAUTH_RUN_WINDOW_S` | How long a run is assumed to take: an OAuth token expiring within it is refreshed first. | | `SWARMKIT_MCP_RETRIES` | Retries for a failing MCP call. | | `SWARMKIT_CONTAINER_RUNTIME` | docker \| podman for sandboxed servers. | | `SWARMKIT_SANDBOX_IMAGE` | Image used to sandbox an MCP server. | | `SWARMKIT_HARNESS_IMAGE` | Image used to run a harness executor. | | `SWARMKIT_DISABLE_CONTAINER_SANDBOX` | Run MCP servers on the host instead of in a container. Weakens isolation. | | `SWARMKIT_DOCS_READER_ALLOW_OUTSIDE` | Let docs-reader read outside its workspace root. Disables path confinement. | **Fleet** | Variable | Purpose | |---|---| | `SWARMKIT_FLEET_REQUIRE_IDENTITY` | Reject fleet calls that do not present a pinned identity. | | `SWARMKIT_FLEET_REQUIRE_SIGNED_DEPLOY` | Reject unsigned artifact deploys from a fleet. | **Telemetry** | Variable | Purpose | |---|---| | `SWARMKIT_OTEL_EXPORTER` | OTLP exporter (otlp \| console). Off when unset. | | `SWARMKIT_OTEL_ENDPOINT` | OTLP collector endpoint. *(URL; userinfo masked)* | | `SWARMKIT_OTEL_HEADERS` | Extra OTLP headers. *(secret)* | | `SWARMKIT_OTEL_API_KEY` | OTLP collector credential. *(secret)* | **Output** | Variable | Purpose | |---|---| | `SWARMKIT_ENV` | Deployment label reported by serve. | | `SWARMKIT_VERBOSE` | Verbose CLI output. | | `SWARMKIT_QUIET` | Suppress non-essential CLI output. | Provider keys (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, `GOOGLE_API_KEY`, `GROQ_API_KEY`, `TOGETHER_API_KEY`) and the local runtimes' endpoints (`OLLAMA_BASE_URL`, `RKLLAMA_HOST`, `LLAMA_SERVER_URL`, `OVMS_URL`, `MLX_LM_URL`, `LEMONADE_URL`) are declared by each provider's YAML, not by the runtime: a provider with no key set is not registered, and `swarmkit providers list` says which. See the [model provider reference](model-provider.md). Telemetry variables are explained in [Telemetry configuration](telemetry.md); storage ones in [Storage](storage.md). ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/serve.md ## Published: https://delivstat.github.io/swarmkit/reference/serve/ ══════════════════════════════════════════════════════════════════════════════ # Serve mode `swarmkit serve` starts a persistent HTTP server for production workloads. ## Quick start ```bash swarmkit serve ./workspace --host 0.0.0.0 --port 8000 ``` ## Endpoints The complete list — every route, generated from the server's OpenAPI document — is the [HTTP API](http-api.md) reference. Below are the ones a caller integrates with and what they take. ### Jobs | Method | Path | Description | |--------|------|-------------| | `POST` | `/run/{topology}` | Submit a topology run (async); returns a job id. Body: `input`, optional `attachments`, `correlation_id`, `labels`, `supersedes` | | `GET` | `/jobs/{id}` | Poll job status (in-memory, then the durable store) | | `GET` | `/jobs/{id}/stream` | The job's events as server-sent events | | `GET` | `/jobs/{id}/diff` | The unified diff a harness run produced, per agent | | `POST` | `/jobs/{id}/resume` | Continue a run parked on a human gate (`deferred`) | | `POST` | `/jobs/{id}/stop` | Ask a running job to stop at its next agent boundary | | `GET` | `/jobs/history` | Every recorded run, newest first (survives restart) | #### Attachments A caller that already holds a file — a snapshot, an uploaded photo — passes it beside the input instead of making an agent go and fetch it: ```json POST /run/describe-scene { "input": "What is at the gate?", "attachments": [ { "path": "snapshots/gate-1732.jpg" } ] } ``` | Field | | | |---|---|---| | `path` | workspace-relative | **exactly one of** `path` / `data` | | `data` | base64 | for a caller holding bytes rather than a file | | `name` | optional | display/filename; derived from `path` when absent | | `handling` | `preprocess` (default) or `native` | intent for non-image types; inert today | **There is no `type` field.** The media type is read from the file's content — sending one is a 422, because a caller's claim about bytes that are about to be forwarded to a model is not evidence. `url` and `stream` sources are refused for the same reason and a related one: the runtime does not fetch caller-supplied addresses, and an attachment is re-read on every turn of a tool loop, so it has to be re-readable. Attachments reach the **entry agent's first message and no downstream node**. An agent that wants a file it was not handed asks for one through a skill. A bad path is a **422 on this request**, not a job that fails a moment later — so a job id means the file was readable. Only images are carried today (`image/png`, `image/jpeg`, `image/gif`, `image/webp`); anything else is refused by name. Per-attachment ceiling is 20 MB (`SWARMKIT_ATTACHMENT_MAX_BYTES`). Every attachment is written to the audit log as a `run.attachments` event carrying name, media type, size, SHA-256 and source path — **never the bytes**. The digest is what makes the reference checkable later; storing content would put arbitrary material into a log meant to stay readable. The CLI equivalent is `swarmkit run --attach ` (repeatable). ### Conversations | Method | Path | Description | |--------|------|-------------| | `POST` | `/conversations` | Create a new conversation | | `GET` | `/conversations` | List conversations | | `GET` | `/conversations/{id}` | Load full conversation history | | `POST` | `/conversations/{id}/messages` | Send message (SSE streaming) | ### CRUD (topologies, skills, archetypes) | Method | Path | Description | |--------|------|-------------| | `GET` | `/api/topologies` | List topologies | | `GET` | `/api/topologies/{id}` | Get topology details | | `GET` | `/api/topologies/{id}/yaml` | Get raw YAML | | `PUT` | `/api/topologies/{id}` | Replace the YAML (validated before it is written) | | `POST` | `/api/topologies` | Create new topology | | `DELETE` | `/api/topologies/{id}` | Delete topology | Same pattern for `/api/skills` and `/api/archetypes`. ### Usage tracking | Method | Path | Description | |--------|------|-------------| | `GET` | `/usage` | Global token usage summary | | `GET` | `/usage/{job_id}` | Per-job usage breakdown | ### MCP endpoint | Method | Path | Description | |--------|------|-------------| | `POST` | `/mcp/` | Streamable HTTP MCP endpoint — every topology as a `run_` tool, behind the same auth (Level 11) | Each topology becomes an MCP tool. External agents can call your swarm topologies via standard MCP protocol. ### A2A endpoint Off by default; `server.a2a.enabled: true` turns it on (no restart — a reload is enough). | Method | Path | Description | |--------|------|-------------| | `GET` | `/.well-known/agent-card.json` | The Agent Card: one A2A skill per topology; public (no auth) | | `GET` | `/a2a/{topology}/card` | The same card narrowed to one topology | | `POST` | `/a2a` | JSON-RPC 2.0: `message/send`, `message/stream`, `tasks/get`, `tasks/list`, `tasks/cancel`, `tasks/subscribe`; the message's `metadata.skill` names the topology | | `POST` | `/a2a/{topology}` | The same, bound to one topology (the card's per-skill `url`) | A2A is a *transport onto jobs*, not a second execution path: `message/send` is `POST /run/{topology}` (`contextId` → `correlation_id`, file parts → attachments), `tasks/get` is `GET /jobs/{id}`, `tasks/cancel` is `POST /jobs/{id}/stop`, and the streaming methods re-emit `GET /jobs/{id}/stream` as A2A status/artifact events. The task id **is** the job id; the run carries `source: a2a` and appears in the portal's jobs page like any other. Task states map from job status: `pending`→`submitted`, `running`→`working`, `deferred`→`input-required`, `completed`, `failed`, `stopped`→`canceled`. A run parked on a human gate reports `input-required` with the gate's URL in the status message, but the A2A caller **cannot** supply that input: approval scopes are un-grantable to agents, so a follow-up `message/send` on the task is refused with `UnsupportedOperationError` and the gate URL. A person resolves it through the review queue; a subscribed client sees the task go `working` again. Push notifications, gRPC and card signing are not implemented (`-32003` / `-32004`). Streaming methods need `Accept: text/event-stream`. Full mapping: `design/details/a2a-interop.md`. ### Webhooks | Method | Path | Description | |--------|------|-------------| | `POST` | `/hooks/{topology_name}` | Fire a webhook trigger: an HMAC-signed request starts the named topology | Webhook signatures are validated with HMAC-SHA256 when `secret` is configured on the trigger. ### Canary | Method | Path | Description | |--------|------|-------------| | `GET` | `/canary` | List canary routes and metrics | | `POST` | `/canary/{topology}/promote` | Promote canary version | | `POST` | `/canary/{topology}/rollback` | Rollback canary version | ## Authentication Configure in `workspace.yaml`: ```yaml server: auth: provider: jwt jwks_url: https://your-idp/.well-known/jwks.json audience: swarmkit issuer: https://your-idp/ ``` Available providers: | Provider | Description | |----------|-------------| | `none` | No authentication (default) | | `api_key` | Bearer keys declared in `workspace.yaml` — `server.auth.config.keys[]`, each a `key_ref: env:` (a reference, never the literal), a `client_id` and a `tier` (`read` / `run` / `admin`) or explicit `scopes`. See the [serve auth guide](https://github.com/delivstat/swarmkit/blob/main/docs/guides/serve-auth.md) | | `jwt` | JWT with JWKS auto-discovery | ## Production profile (fail-closed preflight) `swarmkit serve --profile production` refuses to start unless the deployment is safe to expose, listing every gap at once instead of running permissively. It is a deployment-time assertion — it does not change request handling (governance is already deny-by-default), it checks that you configured the safe thing. `--profile standard` (the default) is the unchanged laptop behaviour. The preflight requires all of: | Requirement | Why | |---|---| | Auth is not `none`/anonymous | production does not serve anonymous, on any bind | | `--insecure` is not set | the escape hatch is incompatible with the profile | | `SWARMKIT_OAUTH_KEY` is set | else the token-encryption key regenerates on restart and silently invalidates stored tokens | | Every `mcp_servers` entry is `sandboxed: true` | a tool process must not run unisolated on the host (design §8.8) | | No wildcard `--cors-origin *` | exact origins only | ```console $ swarmkit serve ./ws --profile production --profile production refuses to start: the deployment is not fail-closed. 1. auth provider is 'none' — configure server.auth (api_key or jwt); production does not serve anonymous, on any bind 2. SWARMKIT_OAUTH_KEY is not set — the OAuth token-encryption key would regenerate on restart and silently invalidate every stored token; set a persistent key 3. MCP server 'files' is not sandboxed — set `sandboxed: true` so the tool process runs isolated (design §8.8) Fix these, or run without --profile production for the permissive default. ``` The existing default-secure refusal (an unauthenticated non-loopback bind) still runs regardless of profile; production adds to it. `create_app(profile="production")` carries the preflight, so an embedder inherits it too. ## Server configuration ```yaml server: host: "0.0.0.0" port: 8000 jobs: max_concurrent: 5 timeout_seconds: 300 mcp: enabled: true a2a: enabled: false # publish the Agent Card + serve /a2a identity: # optional; what the card says about this instance name: Review desk description: Code review and triage swarms, human-gated. url: https://swarm.example.com # set behind a proxy; default is the request's host organization: Example Org ``` `/capabilities` reports `features.a2a` so a fleet can see which instances publish a card. ## Triggers Cron and webhook triggers are defined in `triggers/*.yaml`: ```yaml apiVersion: swarmkit/v1 kind: Trigger metadata: id: nightly-review name: Nightly Code Review type: cron schedule: "0 2 * * *" topology: code-review input: "Review all PRs opened today" ``` ```yaml apiVersion: swarmkit/v1 kind: Trigger metadata: id: pr-webhook name: PR Webhook type: webhook topology: code-review secret: ${WEBHOOK_SECRET} ``` ## Docker ```bash docker run -v ./workspace:/workspace \ -e OPENROUTER_API_KEY=$OPENROUTER_API_KEY \ -p 8000:8000 \ ghcr.io/delivstat/swarmkit:latest ``` ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/http-api.md ## Published: https://delivstat.github.io/swarmkit/reference/http-api/ ══════════════════════════════════════════════════════════════════════════════ # HTTP API Every endpoint `swarmkit serve` exposes, generated from the server's own OpenAPI document (`GET /openapi.json` on a running instance has the schemas; `/docs` renders them). 101 operations. The prose reference — auth modes, triggers, attachments, SSE — is [Serve mode](serve.md); the event contract an application consumes is [Events](events.md). Paths are relative to the server root. `{...}` segments are path parameters. ## Runs and jobs | Method | Path | What it does | |---|---|---| | `POST` | `/hooks/{topology_name}` | Fire a webhook: an HMAC-signed request starts the named topology, or ingresses a pipeline | | `GET` | `/jobs` | Jobs currently known to this process (in-memory); `/jobs/history` is the durable list. | | `GET` | `/jobs/history` | Every recorded run, newest first — or just one pipeline run's stages when | | `GET` | `/jobs/{job_id}` | One job, from the in-memory store or — failing that — the durable one. | | `GET` | `/jobs/{job_id}/diff` | The unified diff a harness run produced, per agent. | | `POST` | `/jobs/{job_id}/resume` | Continue a run that parked on a human gate. | | `POST` | `/jobs/{job_id}/stop` | Ask a running job to stop at its next agent boundary. | | `GET` | `/jobs/{job_id}/stream` | A job's events as they happen, as server-sent events. | | `POST` | `/run/{topology_name}` | Submit a run of a topology; returns a job id to poll, stream or resume. | ## Events | Method | Path | What it does | |---|---|---| | `GET` | `/events` | Events in log order from a position. | | `POST` | `/events/signal` | Ingress a pipeline event by correlation id — the application telling a run what happened | ## Review and gates | Method | Path | What it does | |---|---|---| | `GET` | `/gates/{gate_id}` | A gate's state with its approval policy applied. | | `GET` | `/review` | Pending items, optionally narrowed to one ``kind`` and/or one gate. | | `GET` | `/review/all` | Every review item, pending or not, optionally narrowed to one kind and/or one gate. | | `GET` | `/review/{item_id}` | One review item. | | `POST` | `/review/{item_id}/answer` | Answer a question a run asked; a bare integer selects one of its options. | | `POST` | `/review/{item_id}/approve` | Approve a pending review item as the authenticated caller. | | `POST` | `/review/{item_id}/reject` | Reject a pending review item as the authenticated caller. | | `POST` | `/review/{item_id}/resolve` | Resolve a multi-party approval role-task as the authenticated caller — or, when a fleet | ## Artifacts | Method | Path | What it does | |---|---|---| | `GET` | `/artifacts` | Every artifact reference recorded under one correlation id. | | `GET` | `/artifacts/{ref}` | One artifact's content, by its `//` reference. | ## Conversations | Method | Path | What it does | |---|---|---| | `GET` | `/conversations` | Every conversation on this instance, newest first. | | `POST` | `/conversations` | Start a conversation with a topology; returns its id. | | `GET` | `/conversations/{conversation_id}` | One conversation's full message history. | | `POST` | `/conversations/{conversation_id}/messages` | Send a message into a conversation; the reply streams back as server-sent events. | ## Governed memory | Method | Path | What it does | |---|---|---| | `GET` | `/memory` | Search governed memory by text, optionally narrowed to one type. | | `POST` | `/memory` | Write a fact through the same governed path an agent writes through. | | `GET` | `/memory/config` | The effective `memory` block (design/details/memory-by-default.md): whether memory is | | `GET` | `/memory/item` | One memory item by id, with its history. | | `GET` | `/memory/quarantine` | Memory writes held for a human because they contradict what is stored. | | `POST` | `/memory/quarantine/{quarantine_id}/resolve` | Resolve a quarantined memory write: accept it, reject it, or keep both. | ## Workspace and introspection | Method | Path | What it does | |---|---|---| | `GET` | `/.well-known/agent-card.json` | The instance's Agent Card — public, one skill per topology. | | `POST` | `/a2a` | The JSON-RPC endpoint for every topology; the message names its skill. | | `POST` | `/a2a/{topology}` | The per-topology JSON-RPC endpoint — the card's per-skill `url`. | | `GET` | `/a2a/{topology}/card` | The per-topology card, for a client that should see one skill only. | | `GET` | `/archetypes` | The archetypes in this workspace, by id. | | `GET` | `/audit` | Append-only audit events, newest-first (read-only; the media pillar exposes no | | `GET` | `/capabilities` | What this instance can do — the control plane reads this at enroll/refresh. | | `GET` | `/comprehension` | Comprehension-debt signals from the audit log — same data as `swarmkit comprehension`. | | `GET` | `/contracts` | The contracts in this workspace, by id. | | `GET` | `/funnels` | The funnels in this workspace, by id. | | `GET` | `/gaps` | The skill gap log — what `swarmkit gaps` prints: every tool an agent reached for and | | `GET` | `/health` | Liveness: the instance is up and its workspace loaded. | | `GET` | `/observability/runs/{run_id}/trace` | The finished run's span tree (topology.run → agent.step → tool.call) for a UI waterfall, | | `GET` | `/queue/stats` | Queue health as a subsystem (queue-observability.md): backlog depth, oldest-unclaimed | | `GET` | `/skills` | The skills in this workspace, by id. | | `GET` | `/storage` | Where this instance's data actually lives — one entry per store. | | `GET` | `/system` | Everything the System page needs: versions, storage resolution, environment. | | `GET` | `/topologies` | The topologies in this workspace, by id. | | `GET` | `/triggers` | The triggers configured on this instance (cron, webhook, pipeline events). | | `GET` | `/usage` | Token usage and cost across every run on this instance. | | `GET` | `/usage/{job_id}` | Token usage and cost for one job, per agent. | | `GET` | `/validate` | Validate every artifact in the workspace and report what is wrong. | | `GET` | `/workspace/reachability` | Declared configuration that no code path reaches. | | `GET` | `/workspace/verification` | How strongly each agent's output is checked. | ## Canary deployments | Method | Path | What it does | |---|---|---| | `GET` | `/canary` | The canary routes on this instance and their metrics. | | `POST` | `/canary/{topology_name}` | Start a canary at runtime (design 26 Layer B): split traffic to a newly-deployed version. | | `POST` | `/canary/{topology_name}/promote` | Make the canary version the default for a topology. | | `POST` | `/canary/{topology_name}/rollback` | Withdraw a topology's canary and route everything to the stable version. | ## Authentication | Method | Path | What it does | |---|---|---| | `GET` | `/auth-info` | Unauthenticated: advertise the server's auth mode (+ OIDC issuer/audience for jwt) so a | | `GET` | `/auth/mcp/callback` | Where the provider sends the person back. | | `GET` | `/auth/mcp/probe` | Does this server speak OAuth, and where? Step 2 of the portal flow. | | `GET` | `/whoami` | The *authenticated* caller's identity — as opposed to ``/auth-info``, which is public and | ## Fleet | Method | Path | What it does | |---|---|---| | `POST` | `/fleet/enroll-token` | Mint a one-time fleet enrollment token for a scope (serve:admin). | | `DELETE` | `/fleet/identity/{fleet_id}` | Forget a fleet's pinned public key (serve:admin) so it may deliberately re-key on the | | `DELETE` | `/fleet/membership/{membership_id}` | Eject a fleet — revoke its membership; its key stops authenticating (serve:admin). | | `GET` | `/fleet/memberships` | The fleets registered with this instance (serve:admin — owner-only). No secrets; adds | | `POST` | `/fleet/refresh` | Rotate the caller's membership key. Authenticates with the *current* key (Bearer); the | | `POST` | `/fleet/register` | Enroll a fleet with a Bearer enrollment token and its signed identity; the token is | | `GET` | `/fleet/state` | Full observed state — every artifact's *content* (not just names like /capabilities). | | `POST` | `/fleet/state/artifacts` | Fetch the *content* of specific artifacts (the body-fetch half of delta sync). The body | | `GET` | `/fleet/state/manifest` | The names-only manifest of the observed state — every artifact's id/version/content_hash, | ## Portal API (`/api/*` — what the web portal calls) | Method | Path | What it does | |---|---|---| | `GET` | `/api/a2a/agents` | The remote agents this workspace can call — every `agent` skill with a `card_url`. | | `GET` | `/api/a2a/probe` | Fetch a remote Agent Card so a person can pick a skill *before* a skill file exists. | | `GET` | `/api/archetypes/{archetype_id}` | One archetype, resolved. | | `PUT` | `/api/archetypes/{archetype_id}` | Replace an archetype's YAML; validated before it is written. | | `GET` | `/api/archetypes/{archetype_id}/yaml` | An archetype's YAML as written on disk. | | `PUT` | `/api/contracts/{contract_id}` | Replace a contract's YAML; validated before it is written. | | `GET` | `/api/contracts/{contract_id}/yaml` | A contract's YAML as written on disk. | | `PUT` | `/api/funnels/{funnel_id}` | Replace a funnel's YAML; validated before it is written. | | `GET` | `/api/funnels/{funnel_id}/yaml` | A funnel's YAML as written on disk. | | `GET` | `/api/oauth/credentials` | Stored tokens, as metadata. Never bytes. | | `DELETE` | `/api/oauth/credentials/{credential_id}` | Forget a token, and revoke it upstream where the provider supports revocation. | | `POST` | `/api/oauth/login` | Begin a login. Returns the URL the portal should open in a popup. | | `POST` | `/api/reload` | Re-read the workspace from disk and return its validation report. | | `GET` | `/api/schema/{artifact_type}` | The canonical JSON Schema for an artifact type — drives the UI's schema-generated | | `GET` | `/api/skill-catalogue` | The catalogue's bundles and skills — filtered by *q* when given. | | `POST` | `/api/skills/add` | Plan (and unless dry_run, apply) adding a catalogue skill/bundle or a Skill file. | | `GET` | `/api/skills/check` | Are the tools the workspace's mcp_tool skills name still there? | | `POST` | `/api/skills/import` | Convert a SKILL.md (given as text) and, unless dry_run, write it. | | `GET` | `/api/skills/{skill_id}` | One skill, resolved. | | `PUT` | `/api/skills/{skill_id}` | Replace a skill's YAML; validated before it is written. | | `GET` | `/api/skills/{skill_id}/yaml` | A skill's YAML as written on disk. | | `POST` | `/api/topologies` | Create a topology from YAML; validated against the schema before it is written. | | `GET` | `/api/topologies/{topology_id}` | One topology, resolved: agents, archetypes and skills expanded. | | `PUT` | `/api/topologies/{topology_id}` | Replace a topology's YAML; validated against the schema before it is written. | | `DELETE` | `/api/topologies/{topology_id}` | Delete a topology file from the workspace. | | `GET` | `/api/topologies/{topology_id}/yaml` | A topology's YAML as written on disk. | | `GET` | `/api/workspace/config` | The editable infrastructure sections of workspace.yaml — credentials and MCP servers — | | `PUT` | `/api/workspace/config/{section}/{entry_id}` | Create or replace one entry in a workspace.yaml section (`credentials` or `mcp_servers`); | | `DELETE` | `/api/workspace/config/{section}/{entry_id}` | Remove one entry from a workspace.yaml section; the workspace reloads if the file | ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/connections.md ## Published: https://delivstat.github.io/swarmkit/reference/connections/ ══════════════════════════════════════════════════════════════════════════════ # Connections: credentials, remote MCP servers and OAuth How a swarm reaches a service that needs a secret — and, for remote MCP servers that speak OAuth, how a person logs in once from the portal and runs keep working afterwards. The design is in [credential-service.md](https://github.com/delivstat/swarmkit/blob/main/design/details/credential-service.md) and [mcp-oauth.md](https://github.com/delivstat/swarmkit/blob/main/design/details/mcp-oauth.md). ## One credential service, every entry point A `credentials` entry in `workspace.yaml` is a **reference**, never a literal: ```yaml credentials: github: source: env config: { env: GITHUB_TOKEN } linear: source: oauth config: { endpoint: https://mcp.linear.app/mcp } mcp_servers: - id: linear transport: http endpoint: https://mcp.linear.app/mcp credentials_ref: linear ``` Every entry point — `swarmkit run`, `swarmkit serve`, the MCP client, a command pack's environment — resolves a reference through the **same `CredentialService`**, at the point of use. That is the whole reason it is a service: before it, each entry point assembled its own resolution and a credential declared in YAML could fail to reach the server it was declared for. | `source` | Resolves to | Notes | |---|---|---| | `env` | the named environment variable | `config.env` | | `file` | the file's contents | `config.path` | | `oauth` | a token from the runtime's encrypted store | obtained by logging in from the portal; refreshed automatically — below | | `hashicorp-vault`, `aws-secrets-manager`, `gcp-secret-manager`, `azure-key-vault`, `plugin` | — | accepted by the schema, **refused at resolution** with a message naming the missing `SecretsProvider`. Declaring one does not make it work. | A resolved secret reaches an MCP server as `Authorization: Bearer ` on an `http` transport, or through `env`/`headers` templates (`{credential.}`) where the server wants it somewhere else. Values are never written to the audit log or returned over HTTP. ## Logging in to a remote MCP server For a server that speaks OAuth (the MCP authorization spec), the portal's **Connections** page does the flow: 1. Add the server (`transport: http`, its endpoint) and a credential with `source: oauth` — from the Connections page or by editing `workspace.yaml`; the portal writes the same file (`PUT /api/workspace/config/{section}/{entry_id}`). 2. `GET /auth/mcp/probe?endpoint=…` asks the server whether it speaks OAuth and where its authorization server is. 3. **Connect** (`POST /api/oauth/login`) discovers the provider's metadata, registers SwarmKit as a client dynamically when the provider allows it (otherwise pass a `client_id` you registered), and opens the provider's login page in a popup with a PKCE challenge. 4. The provider sends the browser back to `GET /auth/mcp/callback`; the runtime exchanges the code for tokens and stores them. What is stored, and where: tokens live in `.swarmkit/state/oauth.db`, **encrypted** with a key from `SWARMKIT_OAUTH_KEY` or, when that is unset, one generated into `.swarmkit/oauth.key` on first use (back it up: losing it means logging in again). `GET /api/oauth/credentials` lists **metadata only** — provider, owner, expiry, scopes, whether a refresh token exists. **No endpoint returns a token.** ## Whose token it is A token obtained in a browser belongs to **the person who logged in** — the authenticated identity `serve` already resolves (`GET /whoami`). Tokens are keyed by `(credential, owner)`, so one person's GitHub access does not silently become the workspace's. A credential resolved with no owner named uses the token when exactly one owner has logged in for it; with several, `config.owner` must say which. ## Refresh happens before a run, not during one A run that would fail at minute eight because a token expired at minute three should have been dealt with at minute zero. At run start the runtime refreshes every OAuth credential the topology may use whose access token would expire inside the run's window — `SWARMKIT_OAUTH_RUN_WINDOW_S`, 900 s by default — silently, in one round trip, before the run makes many. A refresh the provider refuses is **`ConsentRequired`**: the refresh token was revoked, expired or its scope changed, and only a person in a browser can fix it. It is not retried; the run fails naming the credential and the owner. The runtime can also tell which refresh tokens are nearing their own end (`swarmkit_runtime.oauth.expiring_soon`: a week out, and a day out) — detection exists; **nothing announces it yet**, so a scheduled run whose refresh token has lapsed fails with `ConsentRequired` at its start, and someone logs in again. ## Forgetting a token `DELETE /api/oauth/credentials/{credential_id}` removes the stored token for the caller as its owner and revokes it upstream where the provider supports revocation. The store is keyed by credential id and owner, independently of `workspace.yaml`: edit the entry and the token stays; delete the token and the entry stays. ## Remote agents (A2A) A remote agent is not a server entry: it is an `agent` skill with a `card_url` ([skills](skills.md#another-agent-as-a-skill-implementationtype-agent)). The Connections page lists them next to servers and sinks with the same status column — a card that asks for a bearer and a skill with no `credentials_ref` reads as *needs credential*, because the refusal would otherwise come from the far side. Two reads back it: | Method | Path | Description | |--------|------|-------------| | `GET` | `/api/a2a/probe?card_url=…` | Fetch a remote Agent Card through the runtime and report its name, skills and whether it wants a bearer; `supported: false` carries the reason (a 404 usually means A2A is off over there). Reads nothing local, writes nothing. | | `GET` | `/api/a2a/agents` | Every `agent` skill with a `card_url`: id, card, skill, credential, `on_unanswerable`, tier. | Adding one writes the skill through `PUT /api/skills/{id}` — the same path as any skill — so the file is the record and the portal holds no state of its own. ## See also - [Workspace artifact](workspace.md) — the `credentials` and `mcp_servers` fields. - [HTTP API](http-api.md) — every `/api/oauth/*` and `/auth/mcp/*` route. - [Environment variables](cli.md#environment-variables) — `SWARMKIT_OAUTH_KEY`, `SWARMKIT_OAUTH_RUN_WINDOW_S`. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/orchestrator-integration.md ## Published: https://delivstat.github.io/swarmkit/reference/orchestrator-integration/ ══════════════════════════════════════════════════════════════════════════════ # Driving SwarmKit from your application The contract between `swarmkit serve` and whatever drives it — your application, a Temporal worker, an Airflow DAG, a shell script. Everything here is HTTP; the runtime holds **no sequencing state of its own**. SwarmKit used to ship a pipeline sequencer (`kind: StageGraph`, a saga controller, `swarmkit orchestrator`, `swarmkit pipeline`, `POST /pipelines/*`). It was removed in runtime 1.189.0 — see [Extracting the pipeline](../design-notes/extracting-the-pipeline.md) for why, and [`examples/pipeline-orchestrator/`](https://github.com/delivstat/swarmkit/tree/main/examples/pipeline-orchestrator) for a reference application that sequences runs with **no `swarmkit_runtime` import anywhere in it**. What SwarmKit keeps is the part it is good at: one bounded governed run, its gates, and its record. If you only read one section, read [Approval gates](#approval-gates) — a client that assumes a gate is a boolean will misreport its state. ## The surface | Endpoint | Purpose | | --- | --- | | `POST /run/{topology}` | Start one bounded run; returns a job id | | `GET /jobs/{job_id}` | Status, output, usage, diff length, correlation, labels, parent | | `GET /jobs/{job_id}/stream` | SSE progress for a running job | | `GET /jobs/{job_id}/diff` | The unified diff a harness run produced | | `POST /jobs/{job_id}/resume` | Continue a run parked on a human gate, or stopped by a human | | `POST /jobs/{job_id}/stop` | Ask a running job to stop at its next agent boundary | | `GET /gates/{gate_id}` | Is this gate resolved, **with the approval policy applied** | | `GET /review` | What is waiting for a human | | `POST /review/{item_id}/resolve` | Cast one multi-party role-task | | `GET /artifacts/{ref}` | Fetch the artifact a gate is about | | `POST /events/signal` | Deliver an inbound event (webhook ingress) to your own listener | ### Correlating a chain of runs Runs are **independent, connected by a correlation id** — not stages of a pipeline the runtime knows about. Your application owns the sequence; SwarmKit records the thread. ```http POST /run/wms-design {"input": "draft the API", "correlation_id": "WMS-35", "labels": {"map": "wayfinder-7"}} ``` - `correlation_id` — "same ticket". Different units of work *and* retries share it. - `labels` — opaque `{key: value}` your application groups by. SwarmKit never learns what they mean; they reach `jobs` **and** `audit_events`. - `parent_job_id` — "this replaces that". A re-run is a **new job**, so the chain is what makes "what did this artifact really cost" answerable across attempts. `GET /jobs?correlation_id=WMS-35` lists the thread. Walk `parent_job_id` to see the attempts. Do not reuse a job id: it keys the LangGraph checkpoint thread, so a reused id inherits the previous run's state. ## Parking on a human, and resuming A funnel's `approve` layer **defers the run** rather than holding a process open: the run checkpoints, the job goes `deferred`, and its `error` names the gate. ```json {"job_id": "a46614b1", "status": "deferred", "error": "awaiting review: gate 'a46614b1:designer'"} ``` Nothing has to stay resident. When the gate resolves, continue it: ```http POST /jobs/a46614b1/resume ``` A `deferred` or `stopped` job resumes — both are parked mid-flight with their state on the checkpoint, and only the reason differs. A completed run has nothing to continue, and starting a second execution against one checkpoint would interleave two runs on it (409 otherwise). A resumed run can park **again**, and does so identically. ### Stopping one ```http POST /jobs/a46614b1/stop ``` Writes the same durable flag `swarmkit stop` writes — one mechanism, two front doors. **Cooperative, not a kill**: the run stops between agents, so a harness session or a slow tool call in flight finishes first, and everything already done stays on the checkpoint. The job goes `stopped` (not `failed` — nothing went wrong; not `deferred` — it waits on nothing) and resumes like any parked run. Asking twice is not an error, and a resume clears the request so the run does not immediately re-stop. Locally the same thing is `swarmkit run --resume`, which resumes the workspace's last checkpointed run (`swarmkit checkpoints` lists them). ## Approval gates ### Reading gate state ```http GET /gates/a46614b1:designer ``` ```json { "gate_id": "a46614b1:designer", "status": "pending", "quorum_evaluated": true, "artifact_ref": "WMS-35/a46614b1/output", "items": [ {"id": "mpa-a46614b1:designer-0-security-reviewer", "role": "security-reviewer", "scope": "security:approve", "rule_index": 0, "status": "approved", "resolved_by": "alice"}, {"id": "mpa-a46614b1:designer-0-release-manager", "role": "release-manager", "scope": "security:approve", "rule_index": 0, "status": "pending", "resolved_by": ""} ] } ``` **A gate id is `:`, where `run_id` is the job id.** Split on the **last** colon. (It used to be `:` inside the node, which was not unique per run — two concurrent runs of one topology shared a gate.) **`quorum_evaluated` is the field that matters for correctness.** It reports how `status` was derived: - **`true`** — the gate's `ApprovalPolicy` was reachable and the **approval engine** evaluated it (quorum, `min_distinct_approvers`, `exclude_author`). This is the same `evaluate()` the runtime gates on, so the report agrees with the decision. - **`false`** — the policy could not be located, so the server folded the review items instead: *every* task must be approved. That bar is correct only for `quorum: all`. This is why `GET /review?gate_id=…` is not a substitute: it returns the individual role-tasks, and turning those into a decision means reading a funnel a client cannot see. ### Resolving a role-task A multi-party gate fans out into one review item per (rule, role). Resolve them individually: ```http POST /review/{item_id}/resolve {"outcome": "approve", "comment": "ships"} ``` **The body carries no identity.** The resolver is the *authenticated caller* (`request.state.identity.client_id`); a body-supplied `identity` is ignored. Three things must hold or the call 403s, with the reason in `detail`: 1. The caller holds `approvals:resolve` — a **reserved human-identity scope**. A transport (API-key / JWT) token structurally cannot carry it, so **an agent or webhook integration can never resolve an approval gate.** Quorum a service account can satisfy is not quorum. 2. The item is a multi-party role-task (`kind: "role_task"`). 3. The caller is a member of that role in the workspace role registry, and the role confers the scope. `outcome` is `approve`, `reject`, or `changes_requested`; the comment reaches the agent — a parked run resumes with it, a re-run reads it as *why* it is running again. Only decisions about the **current** artifact count toward quorum; earlier rounds stay on the record, marked stale. Every attempt is audited as `approval.role_task_resolved`, allowed or denied. **Serve `client_id` and role `members` are one identity namespace.** An operator in role `security-reviewer` must authenticate as the identity listed in that role's `members`. A typo there surfaces as a 403 at resolve time, not at workspace validation — the runtime cannot enumerate an auth provider's credentials. Under the default `NoneAuthProvider` every caller is `anonymous`, so multi-party approval is not enforceable. It works only if the workspace genuinely lists `anonymous` as a role member, which is a local-development convenience and not a deployment posture. `GET /whoami` returns the authenticated caller, for a front-end that needs to say which capacity it is acting in. `/auth-info` is the *public* endpoint and describes the server's auth mode, not the caller. ### Listing what is waiting ```http GET /review?kind=role_task&gate_id=a46614b1:designer ``` `kind` is one of `permission` (a harness permission), `input` (a harness question), `role_task` (multi-party approval), or `other`. A role-task carries `gate_id`, `role`, `scope`, `rule_index`, `resolved_by` and `artifact_ref`. Do not use `POST /review/{id}/approve|reject` on a role-task: those record no identity and cannot satisfy a multi-party rule. ### Reading the artifact under review ```http GET /artifacts/WMS-35/a46614b1/output ``` An approver deciding without the artifact is deciding on a title. The ref is `//`; a review item carries the one it is about. ## Inbound events `POST /events/signal` is the surviving ingress seam: a signed webhook lands on `swarmkit serve`, the signature is validated, an opaque `correlation_id` is extracted from the body via JSONPath, and the event is handed to whatever your application registered. SwarmKit does not decide what an event *means* — that was the sequencer's job, and the sequencer is yours now. A trigger whose `credentials_ref` names an environment variable that is not present **refuses to start**. Accepting unsigned requests because the secret is missing is a fail-open, and the previous behaviour — warn and skip validation — was indistinguishable at runtime from a correctly configured trigger. Note that `swarmkit serve` does not load a `.env` file; export the variable or source it in the shell that starts it. ## Checking the workspace before you drive it Two read-only reports, both from one compile, that answer questions a client otherwise finds out at run time: ```http GET /workspace/reachability # configuration no code path can reach GET /workspace/verification # which topology roots produce an output nothing checks ``` The CLI equivalents gate CI: `swarmkit validate --require` and `swarmkit validate --require-verified`. ## Diagnosing a failed run A harness node that dies without emitting its terminal `result` event reports the exit code and the tail of its stderr, carried on `ExecResult.exit_metadata` and recorded in the `executor.result` audit payload: ``` [harness:claude-code] failure: no result event (exit 1): error: unknown flag --foo ``` The tail is bounded and logged at `debug`, since a harness can print credentials. `GET /jobs/{id}` merges the live job with its durable row, so a field the database can answer is never reported absent because a lighter in-memory object answered first — the bug that made a persisted 20,997-character diff read as `null`. ## Storage Configure it once, in `workspace.yaml`: ```yaml storage: runtime: backend: postgres url: postgresql://user:pass@host/db # libpq form, not postgresql+psycopg:// ``` `SWARMKIT_STORE_BACKEND` / `SWARMKIT_STORE_URL` override it. A `backend` naming a real database with no resolvable URL **fails at startup** — degrading silently to SQLite would mean writing a run to a different database than the one configured. The resolved backend and its source are logged at startup: ``` Store backend: postgres (source: workspace.yaml) ``` If your application keeps its own sequencing state, keep it in **your** database. SwarmKit's store holds runs, audit, artifacts, memory, fleet and checkpoints — not your workflow. ## See also - [Extracting the pipeline](../design-notes/extracting-the-pipeline.md) — why sequencing left, and what replaced it. - [Reading a gate, and approving without a saga](../design-notes/gate-state-and-deferring-approval.md) — the gate read + defer/resume design. - [Approval policy](approval-policy.md) — quorum, roles, and who may resolve. - [Serve](serve.md) — the full HTTP surface. - [CLI](cli.md) — `swarmkit run`, `swarmkit review`. - [`examples/pipeline-orchestrator/`](https://github.com/delivstat/swarmkit/tree/main/examples/pipeline-orchestrator) — the reference application. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/governed-memory.md ## Published: https://delivstat.github.io/swarmkit/reference/governed-memory/ ══════════════════════════════════════════════════════════════════════════════ # Governed memory Structured memory that **evolves in place over time** instead of piling up. A growing application's facts change; governed memory updates the relevant fact rather than appending a duplicate, and every write is reconciled and audited. See the design note: [`design/details/governed-memory.md`](https://github.com/delivstat/swarmkit/blob/main/design/details/governed-memory.md). ## The model - **One canonical row per `(subject, attribute)` key** — the current, trusted value of a fact. The key (e.g. `user:alice` × `preferred_language`) is the reconciliation anchor: a later observation about the same key updates that row rather than creating a new one. - **An append-only change-log** — every mutation (`new` / `reinforce` / `update` / `refine` / `contradict`). The memory row is mutable; the *record of change* never is, so update-in-place and the append-only audit invariant (design §8.3) coexist. Any fact is readable `as_of` a past time. - **Confidence decay** — a fact's effective confidence fades with time since it was last reinforced (per-type half-life). Stale facts rank **down** in retrieval without being deleted; a fact that keeps being observed stays strong. ## The governed write path An agent proposes candidates; it never overwrites. Each candidate is reconciled: | Op | When | Effect | |---|---|---| | `new` | no memory for the key | insert | | `reinforce` | identical value restated | bump recency + confidence, **no new row** | | `update` | changed value, a legitimate evolution | supersede the value in place | | `refine` | changed value that adds detail | merge into the existing memory | | `contradict` | conflicts with a trusted, high-confidence memory | **quarantine** the candidate for a human curator — the trusted value is never overwritten | `new` / `reinforce` / `update` are decided deterministically (no LLM). The `refine` / `contradict` discrimination is the **`memory-reconcile`** decision skill (`category: decision`), which runs only on a changed value. Contradictions park on a quarantine queue — the one hard human gate in the memory path — where a curator accepts (apply as an update) or rejects (discard) them. ## Retrieval Search is **relevance-ranked**, not substring: a local TF-IDF score by default (no keys), or cosine similarity when an `Embedder` is wired into the store (a plain callable — a local model or an MCP embedder plugs in with no vendor lock-in). Effective confidence is the secondary signal; an empty query lists all facts by confidence and recency. ## Using it Add the `governed-memory` persistence skill to a writer agent; the runtime routes its proposed candidates through the governed write path at `post_output`. Add the `memory-reconcile` skill to the workspace to enable refine/contradict (otherwise a changed value deterministically updates). The [`knowledge-curator`](https://github.com/delivstat/swarmkit/blob/main/reference/topologies/knowledge-curator.yaml) reference topology wires an ingester (proposes) → reconcile-judge (`memory-reconcile`) → publisher. ### CLI The `swarmkit memory` commands and the serve `/memory` endpoints resolve the **same** store (`WorkspaceRuntime.governed_memory`) and emit the same JSON. | Command | Description | |---|---| | `swarmkit memory search "" -w ` | Relevance-ranked search (empty query lists all by confidence) | | `swarmkit memory search ... --type --limit N --json` | Filter by memory type; JSON output | | `swarmkit memory get -w --history` | Current value + the append-only change timeline | | `swarmkit memory quarantine -w ` | List quarantined contradictions (`--status accepted\|rejected`) | | `swarmkit memory resolve --by --accept\|--reject -w ` | Resolve a quarantined contradiction | ### Serve endpoints | Method + path | Description | |---|---| | `GET /memory?query=&type=&limit=` | Relevance-ranked search | | `GET /memory/item?subject=&attribute=&history=` | Current value (+ history) for a key | | `GET /memory/quarantine?status=` | Quarantined contradictions | | `POST /memory/quarantine/{id}/resolve` | `{resolved_by, accept}` — resolve a contradiction | ## Demos - `just demo-governed-memory` — a fact evolves in place; a contradiction is quarantined and the curator resolves it. - `just demo-governed-memory-search` — lexical vs embedding relevance ranking. - `just demo-governed-memory-run` — a live compiled run writes governed memory via the persistence skill. - `just demo-governed-memory-cli` — the `swarmkit memory` CLI over a seeded workspace. - `just demo-knowledge-curator` — the reconcile skill + curator topology. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/workspace-memory.md ## Published: https://delivstat.github.io/swarmkit/reference/workspace-memory/ ══════════════════════════════════════════════════════════════════════════════ # Workspace memory Workspace memory lets agents remember context across conversations. Insights from past conversations are automatically extracted and injected into future ones. ## How it works Two decision skill hooks run at the compiler level: 1. **memory-writer** (`post_output`) — after each agent response, an LLM extracts structured insights (topic, context, key points, tags) and saves them 2. **memory-reader** (`pre_input`) — before each agent response, searches saved memories by TF-IDF similarity and injects relevant context ## Configuration Memory is **on by default** (runtime 1.233.0, `design/details/memory-by-default.md`): a workspace that says nothing binds `memory-reader` before every agent and `memory-writer` after, both advisory. The `memory` block tunes or switches that off: ```yaml memory: # optional; absent means enabled with these defaults enabled: true # false: no automatic bindings, no bundled governed-memory skills reader: max_results: 5 similarity_threshold: 0.15 search_scope: all # user | all | both writer: min_output_length: 100 # one model call per run whose answer clears this ``` Bind either skill yourself under `governance.decision_skills` when you want something the block cannot say — a narrower `scope`, `required: true` — and your binding is used as written while the automatic one for that id is skipped. An explicit binding next to `enabled: false` is a resolution error (`memory.disabled-but-bound`). ```yaml governance: decision_skills: - id: memory-reader trigger: pre_input scope: "advisor" # only this agent reads memory required: false config: max_results: 5 similarity_threshold: 0.15 search_scope: all ``` `GET /memory/config` (and the portal's Memory page) report the configuration in force. ### memory-reader config | Field | Default | Description | |-------|---------|-------------| | `max_results` | 5 | Maximum memories to inject | | `similarity_threshold` | 0.15 | Minimum TF-IDF score to include | | `search_scope` | `all` | `user` (per-user), `all` (global), `both` | ### memory-writer config | Field | Default | Description | |-------|---------|-------------| | `min_output_length` | 100 | Skip extraction for short responses | ## Storage Memories are stored as JSON at `.swarmkit/memory/memories.json`. Each entry contains: - `topic` — short label for the conversation topic - `context` — what the user was asking and why - `key_points` — list of important takeaways - `tags` — semantic tags for retrieval - `source_agent` — which agent produced the insight - `user` — user identifier (when available) - `session_id` — conversation session ID ## GBrain backend For production use, configure GBrain as the memory backend. GBrain provides hybrid search (semantic + keyword), graph relationships, and fact extraction via MCP tools. Add GBrain as an MCP server in `workspace.yaml`: ```yaml mcp_servers: - id: gbrain transport: stdio command: ["gbrain", "serve"] ``` The agent can then use `brain-write` and `brain-search` tools for persistent knowledge graph memory. ## What gets saved The memory-writer uses an LLM to determine whether a conversation turn is worth saving. Trivial exchanges (greetings, clarifications) are skipped. Substantive conversations that contain: - Life guidance discussions - Technical decisions - User preferences or context - Important facts or situations are extracted and saved for future reference. ## How context is injected When memory-reader finds relevant prior conversations, it prepends them to the agent's input as: ``` WORKSPACE MEMORY — relevant prior conversations for this user: Topic: grief and loss Context: User was dealing with the loss of a parent Key points: - Discussed Gita 2:47 on detachment - User found the Nachiketa story helpful --- Use this context naturally. Reference prior conversations when relevant. Do not explicitly mention "memory" or "database". ``` The agent sees this as natural context and can reference prior conversations organically. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/events.md ## Published: https://delivstat.github.io/swarmkit/reference/events/ ══════════════════════════════════════════════════════════════════════════════ # Events SwarmKit tells an application what happened. The application decides who hears about it, on which channel, and in what words. That division is deliberate. The runtime used to ship Slack, Discord and Telegram providers; they were removed in 1.216.0 because a chat integration is a relationship with a third party, not a piece of agent machinery — the same argument that took stage sequencing out in 1.189.0. See [extracting the channels](https://github.com/delivstat/swarmkit/blob/main/design/details/extracting-the-channels.md). ## Two halves, deliberately unequal ``` ┌──────────────────────────────────────────┐ run.started │ GET /events?after= DURABLE │ run.ended │ the log an application reconciles │ funnel.gate_opened│ from; nothing here is ever lost │ hitl.requested ├──────────────────────────────────────────┤ hitl.resolved │ events: sinks BEST EFFORT │ … │ a webhook, so an application does │ │ not have to poll │ └──────────────────────────────────────────┘ ``` **The pull is the source of truth.** The push is a latency optimisation over it. ## Reading events ``` GET /events?after=&types=funnel.gate_opened,run.ended&run_id=&limit=100 ``` Returns events in **log order, oldest first** — a consumer replays forward from where it stopped. (`GET /audit` is the other direction, newest first, for a human reading recent history.) ```json { "events": [ { "cursor": "MjAyNi0wOS0wN1QxMjowMDowMCsw…", "event_type": "funnel.gate_opened", "run_id": "b7f1eb46c983", "topology_id": "ask", "payload": {"gate_id": "b7f1eb46c983:root"} } ], "next_cursor": "MjAyNi0wOS0wN1QxMjowMDowMCsw…", "has_more": false } ``` The cursor is **opaque** — pass back what you were given. Every event carries its own, so an application that crashes mid-page resumes without skipping what it never processed. An empty page echoes your cursor back, so an idle consumer need not remember where it was. A cursor this API did not issue is a `400`, never a replay from the beginning. ## Pushing events ```yaml # workspace.yaml events: - sink: webhook url: https://my-app.internal/swarmkit/events credentials_ref: app-signing-key # sent as a bearer token types: [funnel.gate_opened, hitl.requested, run.ended] - sink: stdout # development ``` **The delivery contract, stated rather than implied:** | | | | --- | --- | | guarantee | **at-most-once** | | retry | a small fixed number of attempts, then dropped | | `4xx` | not retried — the application rejected the body | | a failing sink | logged; it never fails a run | | a dropped event | still readable at `GET /events?after=` | There is no queue, no dead-letter, no backpressure. An application needing stronger delivery puts a queue between itself and the webhook, which is where a queue belongs. ## Asking a human A swarm that needs a person opens a **gate**; it does not send a message. The runtime surfaces the ask and parks the run — how it is resolved is the application's business: ``` funnel.gate_opened ──► GET /gates/{gate_id} what is it waiting on? (ask your user, however you like) ◄── POST /review/{item_id}/resolve {"outcome": "approve", "comment": "…"} → the run resumes automatically ``` `gates.auto_resume: false` turns that resumption off for an application that batches; `POST /jobs/{id}/resume` works either way. Three things worth knowing before writing a consumer: - **Acknowledge before working.** The event is already durable when it arrives. Answering non-2xx makes the runtime retry and then drop. - **Be idempotent per gate.** A gate can be announced more than once — a resuming run re-enters the gated node before finding the existing decision. `GET /gates/{id}` reports `resolved`. - **`funnel.advisory_completed` is not "no gate".** It means a funnel declared an approve layer that could not be enforced — usually a role no `RoleRegistry` defines — so the run continued **unreviewed**. Treating it as ungated silently loses an approval step. ## A worked example [`examples/event-consumer/`](https://github.com/delivstat/swarmkit/tree/main/examples/event-consumer) is a complete application: it reconciles from a cursor on startup, receives pushes, asks a human on Telegram, resolves the gate, and imports no `swarmkit_runtime` module. Its only third-party dependency is `httpx`. ```bash just demo-event-consumer ``` ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/storage.md ## Published: https://delivstat.github.io/swarmkit/reference/storage/ ══════════════════════════════════════════════════════════════════════════════ # Storage Where a workspace's data lives, how that is decided, and how to move it from local SQLite to Postgres without losing what you already have. Verified against runtime 1.131.1. ## The one rule **Nothing chooses its own database.** Every component asks one service for a *kind* of store, and that service resolves the configuration once, at startup, and reports what it chose. Before 1.130.0 six components resolved storage independently and four ignored the configuration entirely — three had a SQLite path hardcoded. A workspace declaring Postgres for everything ran its run history, its audit trail and its governed memory into a file on one machine. `swarmkit validate` passed, runs succeeded, and the only symptom was that `swarmkit serve` showed nothing. ## The six stores | Store | Holds | Follows | |---|---|---| | `runtime` | jobs, conversations, usage, serve access | the config you write | | `audit` | the append-only audit trail | `storage.audit`, else `storage.runtime` | | `artifacts` | saved run outputs, fetched by ref | `storage.artifacts`, else `storage.runtime` | | `memory` | governed memory + its change log | `storage.runtime` | | `fleet` | enrollment tokens, fleet memberships | `storage.runtime` (design 19 Q4) | | `checkpoints` | LangGraph run state | **only** `storage.checkpoints` | `audit` and `fleet` keep their own SQLite *file* when the backend is SQLite — separate retention, separate size — but they follow the same *backend*. **`checkpoints` is the exception, deliberately.** It is a LangGraph component with its own driver (`pip install "swarmkit-runtime[postgres]"`), so promoting it to Postgres merely because the application store is Postgres would fail a workspace that never asked for it. Only an explicit `storage.checkpoints` block moves it. If you ask for `postgres` here and the extra is not installed, it **degrades to the local SQLite checkpointer with a warning** rather than refusing to start (1.131.1+). That is the one place degrading is right: see [It fails rather than degrades](#it-fails-rather-than-degrades). ## Configuring it One block moves the whole workspace: ```yaml # workspace.yaml storage: runtime: backend: postgres url: ${SWARMKIT_STORE_URL} ``` A per-store block inherits `storage.runtime.url` when it declares none, so the URL is written once: ```yaml storage: runtime: backend: postgres url: ${SWARMKIT_STORE_URL} audit: retention_days: 90 # different retention, same database ``` `${VAR}` and `${VAR:-default}` are expanded here. (Before 1.130.0 they were not — `url: ${SWARMKIT_STORE_URL}`, the form every deployment doc uses, reached SQLAlchemy as those literal characters.) ## `SWARMKIT_STORE_URL` vs `SWARMKIT_STORE_BACKEND` Both are environment variables, both override `workspace.yaml`, and they are **not** a pair. | Variable | What it does | Needed? | |---|---|---| | `SWARMKIT_STORE_URL` | The connection URL. **A URL names its own backend**, so setting this alone selects Postgres. | This is the one you want. | | `DATABASE_URL` | Same, used only when `SWARMKIT_STORE_URL` is unset. | Fallback. | | `SWARMKIT_STORE_BACKEND` | Forces `sqlite` or `postgres` regardless of the file. | Rarely. Only to force SQLite while a URL is set. | ```bash # Sufficient. Do not also set SWARMKIT_STORE_BACKEND. export SWARMKIT_STORE_URL="postgresql://swarm:secret@db:5432/swarmkit" ``` !!! warning "This exact combination was a bug until 1.130.0" A `.env` declaring only `SWARMKIT_STORE_URL` was **silently ignored**: the resolver required `SWARMKIT_STORE_BACKEND` to be set before it would look at the URL at all, so a correctly configured Postgres stayed empty while everything wrote to SQLite. If you set only the URL and saw no data, that was this. The environment is a *global* signal — it moves every store that follows `storage.runtime`. It does **not** move `checkpoints`, for the reason above. ## Seeing what it chose ```bash swarmkit storage status ``` ``` storage for /srv/swarm: store backend location (source) runtime postgres postgresql://swarm:***@db:5432/swarmkit (env) audit postgres postgresql://swarm:***@db:5432/swarmkit (env) checkpoints sqlite workspace-local (default) artifacts postgres postgresql://swarm:***@db:5432/swarmkit (env) memory postgres postgresql://swarm:***@db:5432/swarmkit (env) fleet postgres postgresql://swarm:***@db:5432/swarmkit (env) ``` The `source` column is the point: it names the setting that won, so "I set that and it did nothing" has an answer. The same report is printed at `swarmkit serve` startup, served at `GET /storage`, and shown on the web UI's **System** page — the answer to "why is this screen empty" has to be reachable from the screen that is empty. Passwords are masked everywhere. This output goes to terminal scrollback, log files and CI capture. A store whose configuration cannot be honoured — the usual case is `backend: postgres` with `url: ${SWARMKIT_STORE_URL}` in a shell where that variable is not set — is a row, not a crash (1.237.0; before that `swarmkit system` died with a traceback on exactly the workspace it was meant to diagnose). Stores that inherit one block share one line, and the command exits 2: ``` store backend location (source) runtime postgres UNRESOLVED (storage.runtime — see below) audit postgres UNRESOLVED (storage.runtime — see below) checkpoints sqlite workspace-local (default) … ! runtime, audit, artifacts, memory, fleet: storage backend 'postgres' (from storage.runtime) has no URL. Set one of: storage.runtime.url, SWARMKIT_STORE_URL. (If the value is '${VAR}', that variable is not in swarmkit's environment — a `source .env` sets a shell variable that child processes never see unless it is exported: `set -a; source .env; set +a`.) Refusing to fall back to sqlite … ``` `swarmkit system` prints the same rows and then carries on to the **environment** section, which is where the answer usually is. If you upgraded from before 1.130.0 and this is the first time you have seen it: that version silently ignored the setting and wrote to SQLite, so check `.swarmkit/*.sqlite` for rows before assuming the Postgres database is the history — `swarmkit storage migrate` copies them over. ## It fails rather than degrades A backend that cannot be honoured raises at startup: ``` storage backend 'postgres' for runtime (from storage.runtime) has no URL. Set one of: storage.runtime.url, SWARMKIT_STORE_URL. (If the value is '${VAR}', that variable is unset.) Refusing to fall back to sqlite: the run would write to a different database than the one configured. ``` Falling back would write the run somewhere other than where you configured, and split `serve` from anything else reading the same store with neither process warning. A failed start is the cheaper failure. ### Except for checkpoints `storage.checkpoints.backend: postgres` without `swarmkit-runtime[postgres]` installed degrades to the local SQLite checkpointer and warns: ``` storage.checkpoints.backend is 'postgres' (from storage.checkpoints) but the Postgres checkpointer is not installed — using the local SQLite checkpointer instead. Runs stay resumable on THIS host only. Install it with: pip install 'swarmkit-runtime[postgres]' ``` `swarmkit storage status` shows it as `sqlite … (storage.checkpoints → sqlite (postgres extra not installed))`, so the report never claims you configured what you got. Two reasons this one is different. The rule above protects **records** — an audit trail or a governed-memory write landing in the wrong database loses data silently. Checkpoints are disposable run state; the cost here is resumability from another host, which surfaces at resume, on the run it affects. And this is a **missing optional dependency**, not a wrong config: taking down serve and every trigger over one is disproportionate to a store whose contents can be thrown away. !!! warning "Upgrading from before 1.130.0" `storage.checkpoints.backend: postgres` was **silently ignored** until 1.130.0, so a workspace could carry it for months without the extra installed. In 1.130.0 and 1.131.0 that combination refused to start — `swarmkit serve` exited with `StorageConfigError` on a config that had always been there. 1.131.1 degrades instead. If you are on 1.130.0 or 1.131.0, either install the extra or set `storage.checkpoints.backend: sqlite`. ## Moving from local SQLite to Postgres The whole runbook. Steps 4 and 5 are the ones people skip. ### 1. Create the database ```bash createdb swarmkit # or: docker run -d --name swarmkit-pg -e POSTGRES_PASSWORD=secret \ # -e POSTGRES_USER=swarm -e POSTGRES_DB=swarmkit -p 5432:5432 postgres:16 ``` Nothing else — the tables are created on first connection. ### 2. Point the workspace at it ```yaml # workspace.yaml storage: runtime: backend: postgres url: ${SWARMKIT_STORE_URL} ``` ```bash # .env, or your process manager's environment SWARMKIT_STORE_URL=postgresql://swarm:secret@localhost:5432/swarmkit ``` Keep the URL in the environment and the *reference* in version control. Do not put a password in `workspace.yaml`. ### 3. Confirm the resolution before moving anything ```bash swarmkit storage status . ``` Every store you expect should read `postgres`, and the `source` column should name the setting you just wrote. If one still says `sqlite (default)`, fix that first — migrating into a database the runtime is not going to use is worse than not migrating. ### 4. Copy the existing rows ```bash swarmkit storage migrate . --dry-run # what would move swarmkit storage migrate . # move it ``` ``` Migrating 468 row(s) from /srv/swarm/.swarmkit: store.sqlite jobs 18 -> postgresql://swarm:***@db:5432/swarmkit audit.sqlite audit_events 412 -> postgresql://swarm:***@db:5432/swarmkit ... Done: 468 row(s) copied, 0 already present. ``` - **Additive and idempotent.** Rows already present are skipped on primary key, so a re-run after a partial failure resumes rather than duplicates. - **Nothing is deleted.** The SQLite files stay exactly as they are. - Run it with the runtime **stopped**, so nothing is writing to the old files mid-copy. Without this step, "switch to Postgres" means "abandon everything recorded so far". Governed memory is the one that actually hurts: it is accumulated knowledge, not just history. ### 5. Verify, then archive the old files ```bash swarmkit storage status . # no warnings psql -d swarmkit -c "SELECT count(*) FROM audit_events;" ``` `status` warns while a populated local SQLite still exists under a remote configuration: ``` ! audit: configured for postgres, but /srv/swarm/.swarmkit/audit.sqlite still holds ~412 rows written before this. Move them with: swarmkit storage migrate /srv/swarm ``` Once the counts match, move the files aside: ```bash mkdir -p .swarmkit/pre-postgres mv .swarmkit/store.sqlite .swarmkit/audit.sqlite .swarmkit/fleet.sqlite .swarmkit/pre-postgres/ ``` Leaving them in place is how a split brain starts. ### 6. Restart everything that touches the store `swarmkit serve`, any `swarmkit run` invocation, and any process manager unit. **They must all see the same environment.** A process started without `SWARMKIT_STORE_URL` reads and writes a different database than the one you migrated into: runs land where nobody is looking, and nothing warns. If your application sequences runs, it keeps its own state in its own store — check that one too. ### Optionally: Postgres checkpoints too Run state does not need to move — SQLite checkpoints are local and disposable, and losing them costs you resumability, not data. If you want them shared anyway: ```bash pip install "swarmkit-runtime[postgres]" ``` ```yaml storage: checkpoints: backend: postgres # inherits storage.runtime.url ``` ## Rolling back Remove the `storage:` block (or set `backend: sqlite`), restore the archived files, restart. The SQLite files were never modified, so this is a move-back, not a restore. ## See also - [Workspace environment configuration](env-config.md) — `workspace.env.yaml`, `${...}` references, and marking properties as secret. - [Driving SwarmKit from your application](orchestrator-integration.md) — the HTTP contract, and where your sequencer's own state belongs. - [CLI reference](cli.md) — `swarmkit storage`, `swarmkit system`. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/telemetry.md ## Published: https://delivstat.github.io/swarmkit/reference/telemetry/ ══════════════════════════════════════════════════════════════════════════════ # Telemetry configuration SwarmKit uses OpenTelemetry for runtime observability. Traces capture every topology run, agent step, tool call, and governance decision as structured spans with `swarmkit.*` semantic attributes. Telemetry is **disabled by default** — zero overhead until you opt in. ## Quick start ### Local testing (console output) ```bash SWARMKIT_OTEL_EXPORTER=console swarmkit run my-swarm/ my-topology --input "hello" ``` Spans print to stderr in human-readable format. ### Local Jaeger ```bash # Start Jaeger docker run -d --name jaeger -p 4318:4318 -p 16686:16686 jaegertracing/all-in-one # Run with OTLP export SWARMKIT_OTEL_EXPORTER=otlp SWARMKIT_OTEL_ENDPOINT=http://localhost:4318/v1/traces \ swarmkit run my-swarm/ my-topology --input "hello" # View traces at http://localhost:16686 ``` ## Configuration ### Environment variables (for testing / CI / Docker) | Variable | Description | Example | |---|---|---| | `SWARMKIT_OTEL_EXPORTER` | Exporter type: `console`, `otlp`, or `none` | `console` | | `SWARMKIT_OTEL_ENDPOINT` | OTLP collector URL | `http://localhost:4318/v1/traces` | | `SWARMKIT_OTEL_API_KEY` | API key (sent via configured header) | `rk-abc123` | | `SWARMKIT_OTEL_HEADERS` | Comma-separated key=value pairs | `x-custom=val,x-org=acme` | Env vars override config file values when set. Setting `SWARMKIT_OTEL_EXPORTER` to anything other than `none` automatically enables telemetry. ### Config file (for production) Location: `~/.swarmkit/config.yaml` ```yaml telemetry: enabled: true exporter: otlp # otlp | console | none endpoint: https://otlp.example.com/v1/traces # OTLP/HTTP endpoint api_key: your-api-key # sent via api_key_header api_key_header: Authorization # which header carries the key headers: # additional headers (optional) x-org-id: acme-corp sample_rate: 1.0 # 1.0 = all traces, 0.1 = 10% send_prompts: false # opt-in: include LLM prompt text in spans service_name: swarmkit # OTel service.name resource attribute ``` ### Resolution order 1. **Environment variables** — highest priority, for quick overrides 2. **Config file** (`~/.swarmkit/config.yaml`) — for persistent production settings 3. **Defaults** — disabled, zero overhead ## Authentication Different backends expect different authentication headers: ### A managed OTLP backend ```yaml telemetry: endpoint: https://otlp.example.com/v1/traces api_key: your-key # api_key_header defaults to "Authorization" # Key is sent as: Authorization: Bearer your-key ``` ### Grafana Cloud ```yaml telemetry: endpoint: https://otlp-gateway-prod-us-east-0.grafana.net/otlp/v1/traces api_key: "your-instance-id:your-api-token" # Sent as: Authorization: Basic # Or use headers directly: headers: Authorization: "Basic dXNlcjpwYXNz..." ``` ### Honeycomb ```yaml telemetry: endpoint: https://api.honeycomb.io/v1/traces api_key: hcaik_your_key api_key_header: x-honeycomb-team # Sent as: x-honeycomb-team: hcaik_your_key (no Bearer prefix) ``` ### Custom collector (no auth) ```yaml telemetry: endpoint: http://otel-collector.internal:4318/v1/traces # No api_key needed ``` The `api_key_header` field controls which header receives the API key: - When `api_key_header: Authorization` (default), the key is prefixed with `Bearer ` - For any other header name, the key is sent raw (no prefix) - If `headers` already contains the target header, `api_key` is not added (explicit headers win) ## Multiple workspaces Multiple workspaces can send traces to the same backend. Traces are distinguished by span attributes: - `swarmkit.workspace.id` — workspace identifier - `swarmkit.topology.id` — which topology ran - `swarmkit.run.id` — unique run identifier Query your backend with these attributes to filter by workspace or topology. ## Semantic attributes All SwarmKit spans use the `swarmkit.*` attribute namespace: ### Trace-level (topology run) | Attribute | Type | Description | |---|---|---| | `swarmkit.topology.id` | string | Topology name | | `swarmkit.run.id` | string | Unique run identifier | | `swarmkit.workspace.id` | string | Workspace identifier | ### Agent step spans | Attribute | Type | Description | |---|---|---| | `swarmkit.agent.id` | string | Agent identifier | | `swarmkit.agent.step` | int | Step number | | `swarmkit.agent.archetype` | string | Archetype used | | `swarmkit.agent.role` | string | root / leader / worker | ### Tool call spans | Attribute | Type | Description | |---|---|---| | `swarmkit.tool.name` | string | Tool or MCP server tool name | | `swarmkit.tool.server` | string | MCP server ID | | `swarmkit.tool.status` | string | success / error / timeout | | `swarmkit.tool.error.type` | string | Error classification | ### Model usage (on agent spans) | Attribute | Type | Description | |---|---|---| | `swarmkit.model.provider` | string | anthropic / openai / google / ollama | | `swarmkit.model.id` | string | Model identifier | | `swarmkit.model.tokens_in` | int | Input tokens | | `swarmkit.model.tokens_out` | int | Output tokens | | `swarmkit.model.cost_usd` | float | Estimated cost | ### Governance events | Attribute | Type | Description | |---|---|---| | `swarmkit.governance.decision` | string | allow / deny | | `swarmkit.governance.policy` | string | Policy that applied | | `swarmkit.governance.scope` | string | IAM scope checked | ### Intent drift events (M7) | Attribute | Type | Description | |---|---|---| | `swarmkit.drift.score` | float | 0.0 (aligned) to 1.0 (fully drifted) | | `swarmkit.drift.threshold` | float | Configured threshold | | `swarmkit.drift.action` | string | log / warn / nudge | | `swarmkit.drift.exceeded` | bool | Whether threshold was breached | ## Span hierarchy ``` Trace: topology.run (swarmkit.topology.id, swarmkit.run.id) ├── Span: agent.step.supervisor (step=1) │ ├── Event: governance.decision (allow) │ └── Span: tool.call.delegate_to_worker ├── Span: agent.step.worker (step=1) │ ├── Span: tool.call.github-pr-read │ │ └── status: success │ ├── Event: governance.decision (allow) │ └── Event: intent.drift (score=0.12) └── Span: agent.step.supervisor (step=2) └── status: completed ``` ## Metrics SwarmKit emits OTel metrics alongside traces for operational dashboards and alerting. ### Counters | Metric | Labels | Description | |---|---|---| | `swarmkit.runs.total` | `topology_id` | Total topology runs | | `swarmkit.agent.steps.total` | `agent_id`, `topology_id` | Total agent execution steps | | `swarmkit.tool.calls.total` | `tool_name`, `status` | Total tool/MCP invocations | | `swarmkit.governance.decisions.total` | `decision`, `scope` | Total governance policy decisions | ### Histograms | Metric | Labels | Description | |---|---|---| | `swarmkit.runs.duration_ms` | `topology_id` | Run duration distribution | | `swarmkit.tool.duration_ms` | `tool_name`, `status` | Tool call latency distribution | | `swarmkit.approval.wait_ms` | `scope` | Human approval wait time distribution | Metrics are emitted via the OTel metrics API. When using OTLP export, they go to the same endpoint as traces. Dashboards in Grafana, Datadog, or a managed backend can query them for operational monitoring. Metrics are safe to call before initialization — all recording functions are no-ops when instruments haven't been created yet. ## Privacy - `send_prompts: false` (default) — no LLM prompt/response content in spans - When false, prompts are stored only in the local ring buffer (`.swarmkit/prompts.sqlite`) keyed by span ID — never sent to the telemetry backend - `send_prompts: true` — opt-in, includes prompt text as span events (for debugging when privacy is not a concern) ## Governance circuit breakers Circuit breakers prevent runaway agent execution and cost overruns. They're enforced inside the runtime — not at the billing layer — so they abort immediately when a limit is exceeded. ### Configuration Add a `limits` block to the `governance` section in `workspace.yaml`: ```yaml governance: provider: agt limits: max_steps_per_agent: 20 # per individual agent max_steps_per_run: 200 # total across all agents max_cost_per_run_usd: 5.00 # estimated LLM cost cap ``` ### Limits | Limit | Default | Description | |---|---|---| | `max_steps_per_agent` | unlimited | Maximum execution steps for any single agent | | `max_steps_per_run` | 500 | Maximum total steps across all agents in one run | | `max_cost_per_run_usd` | not yet active | Maximum estimated LLM cost (USD) per run | **Note on cost-based limits:** `max_cost_per_run_usd` is not active yet. Accurate cost tracking requires each LLM provider to report per-call cost from their API response — not static price tables (prices vary by model, subscription, and change frequently). The plumbing exists in the runtime but is disabled until provider-level cost extraction is implemented. Each `ModelProvider` implementation will be updated to return `cost_usd` when the provider API supports it. ### Behavior When a limit is exceeded, the runtime raises `CircuitBreakerError` with a clear message: ``` Circuit breaker triggered: max_steps_per_run exceeded (limit=200, actual=201). Configure governance.limits.max_steps_per_run in workspace.yaml to adjust. ``` The error names the specific limit, shows the actual vs allowed value, and tells the user which config to change. ### Use cases - **Prevent infinite loops:** two agents arguing back and forth hit `max_steps_per_run` and abort - **Cost control:** a topology running against an expensive model hits `max_cost_per_run_usd` before burning through the budget - **Agent isolation:** a single misbehaving agent hitting `max_steps_per_agent` doesn't take down the whole run ## MCP server trace propagation (future) Currently, the SwarmKit runtime creates spans *around* MCP tool calls. The MCP server process itself does not contribute child spans to the trace. Future enhancement: inject W3C `traceparent` context into MCP calls (HTTP header for SSE transport, env var for stdio transport) so that MCP servers with their own OTel instrumentation produce linked child spans in the same trace. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/reference/env-config.md ## Published: https://delivstat.github.io/swarmkit/reference/env-config/ ══════════════════════════════════════════════════════════════════════════════ # Workspace environment configuration Separate environment-specific values (URLs, credentials, feature flags) from structural workspace config so the same workspace runs in dev, staging, and prod without editing `workspace.yaml`. There are two layers, and you can use either or both: - **Env references in any artifact** (runtime 1.98.0+) — `${VAR}`, `${VAR:-default}`, and `$${VAR}` resolve in **every** artifact (topology, skill, archetype, workspace, trigger, funnel), with or without an env file. This is the quick path for making a reusable library model- or endpoint-configurable. See [Env references in any artifact](#env-references-in-any-artifact) below. - **The workspace property map** — a `workspace.env.yaml` file that maps dotted `${property.path}` references in `workspace.yaml` to real values, with per-environment overrides. This is the structured path for the two-file dev/staging/prod split described in the rest of this page. The two layers compose: a `${NAME}` reference resolves from the property map first, then the OS environment, then a `:-default`, then is left literal. ## Env references in any artifact Any string in any artifact can reference the environment — no env file required. This is resolved at load time, before schema validation, so the runtime and validators see the resolved value. ```yaml # archetypes/reasoner.yaml — ships working out-of-the-box, overridable per deployment apiVersion: swarmkit/v1 kind: Archetype metadata: id: reasoner defaults: model: provider: ${SDLC_REASONING_PROVIDER:-openrouter} name: ${SDLC_REASONING_MODEL:-moonshotai/kimi-k2.5} ``` Syntax: - **`${VAR}`** — the value of `VAR`. - **`${VAR:-default}`** — `VAR` if set, else `default`. Defaults let a reusable library run out-of-the-box while staying configurable. - **`$${VAR}`** — a literal `${VAR}` (escape), for the rare artifact that must contain the sequence. Resolution order for each `${NAME}`: 1. **Workspace property map** — dotted paths from `workspace.env.yaml` (the layer documented below); empty when there is no env file. 2. **OS environment** — `os.environ[NAME]`. 3. **Inline default** — the text after `:-`. 4. **Left literal** — an unresolved reference with no default is emitted unchanged, so artifacts that already contain `${...}` never regress. Because an unresolved reference is left literal rather than raising, enabling this across all artifacts is backward compatible: workspaces with no env file and no references behave exactly as before. ## File layout ``` workspace/ ├── workspace.yaml # structural — committed to git ├── workspace.env.yaml # environment — add to .gitignore ├── workspace.env.prod.yaml # optional per-environment override ├── topologies/ ├── skills/ └── archetypes/ ``` ## How it works `workspace.yaml` uses `${property.path}` references instead of inline values: ```yaml # workspace.yaml — safe to commit, no secrets mcp_servers: - id: github transport: stdio command: ["npx", "-y", "@modelcontextprotocol/server-github"] env: GITHUB_PERSONAL_ACCESS_TOKEN: ${github.token} events: - sink: webhook url: ${app.events_url} credentials_ref: app-token ``` `workspace.env.yaml` provides the actual values: ```yaml # workspace.env.yaml — NOT committed to git # This is the ONLY file that does ${ENV_VAR} interpolation github: token: ${GITHUB_TOKEN} app: events_url: ${APP_EVENTS_URL} ``` ## Resolution order 1. **`workspace.env.{SWARMKIT_ENV}.yaml`** — if `SWARMKIT_ENV` is set and the file exists 2. **`workspace.env.yaml`** — default fallback 3. **`${ENV_VAR}`** in property values — resolved from OS environment 4. **Inline values** in `workspace.yaml` — backward compatible, used as-is if no `${...}` reference ## Environment switching ```bash # Dev (default — uses workspace.env.yaml) swarmkit run my-swarm/ my-topology # Production SWARMKIT_ENV=prod swarmkit run my-swarm/ my-topology # Staging SWARMKIT_ENV=staging swarmkit run my-swarm/ my-topology ``` Each environment can have its own env file with different credentials, endpoints, and feature flags. ## Two-phase interpolation 1. **Phase 1:** Load env file → flatten nested YAML to dotted paths (`app.events_url` → `https://…`) 2. **Phase 2:** Resolve `${ENV_VAR}` in property values from OS environment (`${GITHUB_TOKEN}` → actual token) 3. **Phase 3:** Replace `${property.path}` references in workspace.yaml with resolved values Keeping `${ENV_VAR}` in the env file concentrates secret interpolation in one place. (Env references also work directly in any artifact — see [Env references in any artifact](#env-references-in-any-artifact) — but routing secrets through the env file keeps them auditable in a single file.) ## Backward compatibility Existing workspaces without `workspace.env.yaml` work unchanged. Property references (`${...}`) are only resolved if the `${}` syntax is present. If you don't create an env file, nothing changes. ## Marking a property secret `workspace.env.yaml` is where connection strings and API keys live, and its resolved values are displayed by `swarmkit system` and the web UI's **System** page. The reserved top-level `secrets:` key lists the paths whose values must never be shown: ```yaml secrets: - db.dsn - openai.api_key db: dsn: ${SWARMKIT_STORE_URL} pool: 5 openai: api_key: ${OPENAI_API_KEY} ``` A listed path renders as `set` in every surface; everything else renders resolved. `secrets:` is a declaration *about* the properties, not a property — it never appears in the map and is not interpolatable. **Declare them.** As a fallback, a property whose name contains `key`, `token`, `secret`, `password` or `credential` is masked without being listed, so a workspace written before this existed does not start leaking on upgrade. But a heuristic is a guess: it does not catch `db.dsn` or `webhook.callback`, and being wrong in that direction prints a credential into terminal scrollback, a log file, and a web page. Declaring adds to the masked set and can never remove from it, so `secrets: []` does not un-mask an `api_key`. ## Always create one `swarmkit init` scaffolds `workspace.env.yaml` for you, with an empty `secrets:` list ready to fill in. Create one even for a workspace that has no secrets yet: - A workspace with nowhere to put a connection string ends up with one **inside `workspace.yaml`**, which is the file you commit. - The same workspace then cannot move between dev and prod without editing structural config. - A secret that arrives later, with no `secrets:` entry, is masked only if a name heuristic happens to catch it. The file is cheap and empty is fine. Add `workspace.env*.yaml` to `.gitignore`, and commit a `workspace.env.example.yaml` with placeholders so a teammate knows what to set. ## Best practices - **Add `workspace.env*.yaml` to `.gitignore`** — never commit credentials - **Route secrets through the env file** — put `${ENV_VAR}` for credentials in `workspace.env.yaml` so secret interpolation stays auditable in one place, even though env references work in any artifact - **List every credential path under `secrets:`** — that is what keeps it out of `swarmkit system`, the System page, and your CI logs - **Create a `workspace.env.example.yaml`** with placeholder values for team onboarding - **Use named env files for each environment** — `workspace.env.dev.yaml`, `workspace.env.staging.yaml`, `workspace.env.prod.yaml` ## Example ### workspace.yaml (committed) ```yaml apiVersion: swarmkit/v1 kind: Workspace metadata: id: my-swarm name: My Swarm mcp_servers: - id: github transport: stdio command: ["npx", "-y", "@modelcontextprotocol/server-github"] env: GITHUB_PERSONAL_ACCESS_TOKEN: ${github.token} governance: provider: agt config: policies_dir: ${governance.policies_dir} ``` ### workspace.env.yaml (dev, not committed) ```yaml github: token: ${GITHUB_TOKEN} governance: policies_dir: ./policies ``` ### workspace.env.prod.yaml (prod, not committed) ```yaml github: token: ${GITHUB_TOKEN_PROD} governance: policies_dir: /etc/swarmkit/policies ``` ══════════════════════════════════════════════════════════════════════════════ # Part 6 — Guides ══════════════════════════════════════════════════════════════════════════════ ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/guides/validating-topology-output.md ## Published: https://delivstat.github.io/swarmkit/guides/validating-topology-output/ ══════════════════════════════════════════════════════════════════════════════ # Validating a topology's output Where to put a validation layer so a run cannot return a confidently wrong answer, and what the runtime actually does when validation fails. Verified against runtime 1.131.0. ## Two layers, and they are not interchangeable | | `output_schema` | decision skill | |---|---|---| | Checks | **shape** — fields, types, enums, required | **semantics** — is it true, grounded, in scope | | Cost | free, deterministic | a tool call, or an LLM call per evaluation | | On failure | structured generation retries the model | feeds the agent revision instructions | Use both. Spending a judge call to discover a `summary` field is missing is the expensive way to learn a cheap fact, and the schema cannot tell you the summary is fabricated. ## Layer 1: `output_schema` (shape) Not a skill — a field on the agent, enforced before any judge runs: ```yaml # topologies/research.yaml root: id: coordinator archetype: coordinator output_schema: type: object required: [summary, findings, confidence] properties: summary: { type: string } confidence: { type: number, minimum: 0, maximum: 1 } findings: type: array items: type: object required: [claim, evidence] properties: claim: { type: string } evidence: { type: string } ``` Set `output_schema: null` on an agent to opt out of an archetype's default. ### The same schema as a file `output_schema` also accepts a **path** — a JSON or YAML file holding the schema, relative to the artifact that declares it (the topology or archetype file, not the workspace root, because a schema usually lives beside what uses it). Inline is fine for six lines; at any real size a file is easier to read, can be shared by several agents without copies that drift, and is validated as a JSON Schema at **load** time — a typo in a `required` list is a resolution error naming the file, not a conformance failure mid-run that reads like the agent's fault. ```yaml # topologies/research.yaml root: id: coordinator archetype: coordinator output_schema: schemas/research-verdict.json # relative to this topology file ``` ```json // topologies/schemas/research-verdict.json { "type": "object", "required": ["summary", "findings", "confidence"], "properties": { "summary": { "type": "string" }, "confidence": { "type": "number", "minimum": 0, "maximum": 1 }, "findings": { "type": "array", "items": { "type": "object", "required": ["claim", "evidence"], "properties": { "claim": { "type": "string" }, "evidence": { "type": "string" } } } } } } ``` Both forms normalise to the parsed schema at load, so nothing downstream — the validator, the portal, `swarmkit validate --require-verified` — can tell which was written. Rules the resolver enforces: - the path must stay **inside the workspace**; remote URLs are refused, so a workspace's meaning never depends on the network; - a missing or unparseable file, a file that is not an object, or one that is not a valid JSON Schema is an error at load, naming the artifact that declared it; - it is one key, not two: `output_schema` is inline **or** a path, so "both declared" cannot happen and no precedence rule can silently ignore the one you edited. An archetype can declare it the same way (`defaults.output_schema: schemas/verdict.json`, relative to the archetype file); an agent overrides with its own inline schema, its own path, or `null`. This eliminates shape-level hallucination outright. It does not, and cannot, tell you whether `evidence` supports `claim`. ## Layer 2: the decision skill (semantics) A decision skill is a **skill**, not compiler behaviour — grounding and conformance logic belongs in the artifact, never in the runtime. The example below is an `llm_prompt`; see [A decision skill does not have to be a prompt](#a-decision-skill-does-not-have-to-be-a-prompt) for the deterministic, MCP-backed form, which is the better choice whenever the check is decidable. ### The binding ```yaml # topologies/research.yaml governance: decision_skills: - id: output-conformance trigger: post_output scope: coordinator # comma-separated agent ids; default '*' = every agent required: true config: max_retries: 2 ``` `scope` matters more than it looks. The default `*` fires the skill after **every agent's** output, including each sub-agent — that is N judge calls per run, not one. Name the root agent when you mean "the topology's answer". ### The skill ```yaml # skills/output-conformance.yaml apiVersion: swarmkit/v1 kind: Skill metadata: id: output-conformance name: Output Conformance Checker description: Checks the final answer against the request and its own evidence. category: decision outputs: type: object required: [verdict, confidence, reasoning] properties: verdict: { type: string, enum: [pass, fail, needs-revision] } confidence: { type: number, minimum: 0, maximum: 1 } reasoning: { type: string } violations: type: array items: type: object required: [claim, issue] properties: claim: { type: string } issue: { type: string } implementation: type: llm_prompt prompt: | You are checking a finished answer before it is returned. 1. GROUNDING: is every claim supported by the evidence cited alongside it? 2. SCOPE: does it answer what was asked, without inventing adjacent scope? 3. CONTRADICTION: does any part contradict another part? You are not judging whether the answer is GOOD. A mediocre answer that is grounded and in scope passes. A brilliant one with an unsupported claim does not. verdict: pass - grounded, in scope, self-consistent needs-revision - fixable without redoing the work fail - unsupported claims or material scope violations Write `reasoning` and `violations` as INSTRUCTIONS TO THE AGENT THAT WILL FIX THIS, naming the specific claim. That text becomes the retry prompt. provenance: authored_by: human version: 1.0.0 ``` `verdict` must be one of `pass`, `fail` or `needs-revision` (casing and `_`/`-` are normalised). An unrecognised value is **read as `pass`**, not rejected — so a typo disables the check rather than breaking it. See [the one contract a tool must meet](#the-one-contract-a-tool-must-meet). ## A decision skill does not have to be a prompt Nothing about the *binding* says LLM. A decision skill takes any of the three skill implementation types, and the evaluator dispatches through the same executor every other skill uses: ```yaml # skills/schema-conformance.yaml apiVersion: swarmkit/v1 kind: Skill metadata: id: schema-conformance category: decision outputs: type: object required: [verdict, confidence, reasoning] properties: verdict: { type: string, enum: [pass, fail, needs-revision] } confidence: { type: number, minimum: 0, maximum: 1 } reasoning: { type: string } implementation: type: mcp_tool server: order-validator tool: validate_order_validation arguments: strict: true ``` The binding is unchanged — it does not know or care how the skill is implemented. That is the point of the seam. **Prefer this wherever the question has a computable answer.** A JSON-schema check, a lint run, a test suite, a real validator: those are decidable, and asking a model to judge them adds cost, latency and a failure mode for nothing. `reference/skills/` already ships several of this shape — `lint-check`, `run-tests`, `security-scan`, `validate-workspace`, `gate-validator`. Keep `llm_prompt` for the genuinely fuzzy part: *is this claim grounded in the evidence beside it, is this in scope*. A `composed` skill with `strategy: parallel-consensus` runs both and requires them to agree. ### The one contract a tool must meet The tool has to return JSON carrying `verdict`, spelled exactly `pass`, `fail` or `needs-revision`. The parser is forgiving about packaging — it strips markdown fences, digs a `{...}` out of surrounding prose, and drops trailing `[source: ...]` provenance tags that MCP servers append. It is **not** forgiving about vocabulary: !!! danger "An unrecognised verdict is read as `pass`" A validator returning `{"valid": false}` or `{"status": "rejected"}` does not fail the check — it passes it, because the parser defaults an unrecognised verdict to `pass`. A validation layer that reports success on every rejection is worse than no validation layer. Since 1.131.0 both cases log a warning naming the skill and the value (`the check is not running`), so this is visible rather than silent. It still passes — failing closed would turn every currently-passing run whose skill emits an odd verdict into a flagged one, which is its own outage — so treat the warning as the signal. Map your tool's vocabulary to the verdict enum, either inside the tool or with a thin `composed` wrapper around it. Then prove it with a run that should fail. **Form is forgiven; vocabulary is not.** `FAIL`, `Fail`, ` fail ` and `needs_revision` all read correctly — casing and separators are not part of the meaning, and models vary both constantly. (Before 1.131.0 they did not, so a skill that plainly said `FAIL` was recorded as a pass.) But `rejected`, `invalid` and `false` are still unrecognised: guessing at a synonym would invent a verdict the skill never gave. Two smaller notes: - The evaluator reads `flagged_items`, and also `uncited_claims` and `contradictions`, whether the entries are strings or objects carrying `claim` / `description`. Existing validator output often already fits one of those. - MCP-backed decision skills need `mcp_manager` wired. `swarmkit serve` and the CLI runtime do this. If you evaluate skills through a custom `GovernanceProvider`, pass it through or the tool call has nothing to execute against. ## What `fail` actually does This is the part worth knowing **before** you write the prompt. On `fail`, the runtime does not reject the run. It builds feedback from the failed results and asks the agent to revise, up to `max_retries` times (default 4). The agent still holds its research context, so it is fixing a citation, not redoing the work. If retries are exhausted, the output is returned **annotated, not blocked**: ``` ...the agent's final answer... --- GOVERNANCE FLAGS: [output-conformance]: Claim "response time improved 40%" cites no measurement. - response time improved 40% ``` So: - **Write `reasoning` for the agent, not for a human.** It is the retry prompt. "Not grounded" is useless; "the 40% figure appears in no cited source — remove it or cite the measurement" is a fix. - **A `fail` is not a stop.** If you need a hard stop, that is an approval gate, not a decision skill — see [Approval policy](../reference/approval-policy.md). - **Set `max_retries` deliberately.** Four retries of a large answer through a judge is a real bill. `0` means judge once and annotate. ## Choosing the trigger | Trigger | Fires | Use for | |---|---|---| | `pre_input` | before any LLM work | rejecting off-topic or malicious input — saves the whole run | | `post_output` | after an agent's output | **validating the answer** | | `checkpoint` | between task batches | catching a bad sub-result early in a long run | | `pre_synthesis` | before the coordinator synthesises | judging task results before they are summarised | `pre_synthesis` is the underrated one. It sees the raw task results and auto-loads `scope.json` from run state as context, so it catches a wrong sub-result **before** synthesis launders it into a fluent summary. Validating at both `pre_synthesis` and `post_output` costs two judge calls per run — worth it when synthesis is where your topology goes wrong. ## Workspace-level vs topology-level Same shape in `workspace.yaml`: ```yaml governance: decision_skills: - id: output-conformance trigger: post_output ``` - **Workspace** — applies to every topology. A topology must explicitly opt out with `required: false`, which is visible in the artifact and therefore auditable. - **Topology** — same `id` overrides the workspace binding, a new `id` extends it. If the check is non-optional, bind it at the workspace. A topology-level binding is a check that whoever writes the next topology can simply not add. ## Common mistakes **Leaving `scope` at `*`.** Fires after every agent in the tree. Fine for a cheap grounding check, expensive for a full conformance review. **Putting validation logic in the compiler.** It belongs in the skill. A validation rule in Python is one nobody can see, version, or change without a release. **Judging quality instead of conformance.** A skill that asks "is this good?" fails good-but-plain answers and burns four retries improving prose. Check grounding, scope and contradiction — things with an answer. **Assuming `fail` blocks.** It annotates. Design for that. **Asking an LLM to check something a tool can decide.** If a validator, linter or schema can answer it, bind the validator. The judge call is for the part that genuinely needs judgement. **Wiring a validator without mapping its vocabulary.** `{"valid": false}` has no `verdict`, so the parser defaults it to `pass` and the check reports success on every rejection. ## Checklist - [ ] `output_schema` on the root agent covers the shape - [ ] The decision skill's `outputs` declares `verdict` / `confidence` / `reasoning` - [ ] `verdict` enum is exactly `pass` / `fail` / `needs-revision` - [ ] `scope` names the agent whose output you mean - [ ] `max_retries` is a number you chose, not the default you inherited - [ ] `reasoning` reads as an instruction to the agent that will act on it - [ ] Bound at the workspace if it must not be skippable - [ ] Decidable checks use `mcp_tool`, not a prompt - [ ] Any tool-backed skill emits `verdict` as exactly `pass` / `fail` / `needs-revision` - [ ] A run with a deliberately unsupported claim actually gets flagged ## The other end: `input_schema` (validate what comes in) `output_schema` guards what a run *produces*; `input_schema` guards what a caller *sends*. It is an optional JSON Schema on the **topology** (not per-agent — input is caller-supplied only at the entry) that the input must satisfy before the run starts. Symmetric to `output_schema` with one deliberate difference: it is validate-and-**reject**, not validate-and-correct — a caller cannot be re-prompted mid-run, so a malformed request simply never becomes a run (no LLM spend). ```yaml # topologies/triage.yaml apiVersion: swarmkit/v1 kind: Topology metadata: { name: triage, version: 0.1.0 } input_schema: # optional; JSON Schema draft 2020-12 type: object required: [ticket_id, severity] properties: ticket_id: { type: string } severity: { enum: [P0, P1, P2] } agents: { ... } ``` Checked at the single choke point (`WorkspaceRuntime.run`), so every entry inherits it: | Entry | On a malformed request | |---|---| | `POST /run` | **422** at submit, with the offending field named; no job is created | | `swarmkit run` | non-zero exit, the error on stderr, no billed run | | A2A `message/send` | a JSON-RPC error, no task created | An **object** schema requires the input to be JSON (`input must be a JSON object matching input_schema` otherwise); `{ "type": "string" }` accepts plain natural-language text, so a conversational topology can assert "non-empty text" without forcing JSON. Every check emits an `input.validated` or `input.rejected` audit event. A topology that omits `input_schema` is unchanged. For **meaning** rather than shape (an allow-list lookup, a policy call), use a `pre_input` decision skill — the same shape-vs-semantics split as `output_schema` vs a decision skill above; `input_schema` runs first because it is the cheapest, structural check. ## See also - [Building swarms](building-swarms.md) — where skills and bindings sit in a topology. - [Approval policy](../reference/approval-policy.md) — when a human, not a skill, has to decide. - [Governed memory](../reference/governed-memory.md) — the same verdict vocabulary applied to memory writes. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/guides/getting-an-image-to-a-model.md ## Published: https://delivstat.github.io/swarmkit/guides/getting-an-image-to-a-model/ ══════════════════════════════════════════════════════════════════════════════ # Getting an image to a model How images actually reach an agent in SwarmKit, and the one trap that makes them silently not arrive. Written up after a design agent spent three runs describing UI screens it had never seen — convincingly. The skill route below was verified against runtime 1.129.2; the attachment channel was added in 1.218.0 and is what most callers should use first. ## Two channels, for two different callers | | Who holds the image | Who decides what to look at | Route | |---|---|---|---| | **Attach it to the run** | the caller, before the run starts | the caller | `attachments` / `--attach` — below | | **Let the agent fetch it** | nobody yet | the model, mid-run | an MCP tool returning `ImageContent` — the rest of this guide | They do not substitute for each other. A snapshot poller, a webhook with an upload, an application with a user's screenshot: all of them already hold the bytes and want *one* model call, not an agent that first has to decide to look. An agent reviewing a ticket with screenshots it has never seen needs the skill. ## Attach it to the run Beside the input, not inside it. CLI, one `--attach` per file, repeatable, workspace-relative: ```bash swarmkit run ./workspace describe-scene \ --input "What is at the gate?" \ --attach snapshots/gate.jpg --attach snapshots/gate-wide.jpg ``` HTTP, the same two ways of naming a file: ```json POST /run/describe-scene { "input": "What is at the gate?", "attachments": [ { "path": "snapshots/gate.jpg" }, { "data": "", "name": "gate-wide.jpg" } ] } ``` | Field | | | |---|---|---| | `path` | workspace-relative | **exactly one of** `path` / `data` | | `data` | base64 | for a caller holding bytes rather than a file — a poller, an upload | | `name` | optional | display/filename only; derived from `path` when absent | | `handling` | `preprocess` (default) or `native` | intent for non-image types; inert while only images are carried | What happens, and what does not: - **The media type is read from the bytes.** There is no `type` field, and sending one is a 422: a caller's claim about content that is about to be forwarded to a third-party model is not evidence. This is also why there is one `--attach` and no `--image` / `--pdf`. - **Images only, today:** `image/png`, `image/jpeg`, `image/gif`, `image/webp`. Anything else is refused by name. Per attachment, 20 MiB (`SWARMKIT_ATTACHMENT_MAX_BYTES`). - **`url` is refused.** The runtime does not fetch caller-supplied addresses — that is the same exfiltration primitive the skill route's path-resolution rejects (below). Send the bytes. - **A bad path is a 422 on the request**, not a job that fails a moment later: a job id means every attachment was readable and carryable. - **It reaches the entry agent's first message and no downstream node.** The root agent sees the image in the same model call as the input — no tool round-trip, one pass. A child agent that wants it asks through a skill; the runtime does not fan a caller's file out to every node. - **It is re-read on every turn** of the root's tool loop, so the file has to stay readable for the run's duration; that is the other reason streams and URLs are not accepted. - **Audited, never stored.** Every run writes a `run.attachments` event with name, media type, size, SHA-256 and source path — the digest makes the reference checkable later; the bytes never enter a log meant to stay readable. - **Provider coverage follows the family.** Anthropic and every `openai-compatible` provider (OpenRouter, Groq, Ollama, llama-server …) get the image part from the same mapping the skill route uses; a provider YAML with `capabilities: {images: false}` refuses at the request instead of sending bytes a server would drop. Two edges worth knowing: - **Harness roots do not receive attachments.** A harness (Claude Code, opencode) reads files from its worktree, so an attachment has no message to land in. Put the file in the repository the worktree is cut from and name the path in the input, or route through a model agent. - **Over A2A, a file part becomes an attachment.** A remote caller's `message/send` with a `file` part carrying `bytes` reaches the run exactly as `data` does; a `uri` file part is refused for the same reason `url` is. The same rules apply when *your* agent calls a remote one through an `agent` skill. That is the whole caller-side story. When the caller is an agent that has to *decide* what to look at, read on. ## An image in the prompt is still just text A run's input is a **plain string**, and no schema — topology, archetype or trigger — has an image or media field. (`executor-adapter` has an `image`, but that is the *container* image for a sandboxed harness, not a picture.) So for anything not passed as an attachment, neither of the two obvious approaches works: - **A path in the prompt** is just text. The model reads the characters; nothing loads. - **Base64 in the prompt** is tokens. It is never interpreted as an image, because the provider only builds an image part from a *tool result* — never from the input string. For an agent choosing what to look at, the route is **an MCP tool that returns an `ImageContent` block**: ``` your tool (e.g. docs-reader view_image) └─ returns mcp.types.ImageContent(type="image", data=, mimeType=…) └─ langgraph_compiler/_skill_executor.py └─ ContentBlock(type="image", image_data=…, image_media_type=…) ├─ model_providers/_openai.py → {"type": "image_url", "image_url": {"url": "data:…"}} └─ model_providers/_anthropic.py → {"type": "image", "source": {"type": "base64", …}} ``` Provider coverage follows the family: OpenRouter, Groq and every other provider declared over `openai-compatible` gets the image part from `_openai.py` for free. A provider YAML can switch it off (`capabilities: {images: false}`) for a server that does not read it. **Harness executors get there differently** — Claude Code has its own image handling and reads files from disk directly — but the MCP route works for both, so it is the portable answer. ## The trap: a path that resolves nowhere `swarmkit docs-reader --workspace ` resolves **relative** paths against that root. Almost every document extractor writes image references relative to the *document*, not to that root: ```markdown ![](3-5RFComfirmPGM_0.2.xlsx.media/screen1.png) ``` Correct for a web UI serving the ticket. Meaningless to an agent, because docs-reader is rooted at the repository root and there is no such path there. **And the failure is silent in the worst way.** The tool reports that the file does not exist; the model reports *"no screenshot was provided"* and — being helpful — describes the screen from the surrounding prose anyway. The result reads exactly like a real description. There is no error, no warning, and nothing in the trace that says an image was missed. ### Fix **Pass absolute paths, and pass them in the prompt as a list.** ``` SCREENSHOTS — open EVERY one with view-screenshot before writing the screens section. These are absolute paths and they resolve: /abs/path/to/ticket/media/screen1.png /abs/path/to/ticket/media/screen2.png ``` Absolute, because the runtime chooses the agent's working directory and a relative path that resolves differently there is indistinguishable from an absent file. Rewrite the references inside the document too, if the agent will read the markdown. !!! note "Since 1.129.2, absolute is not enough on its own" `docs-reader` now **confines** reads to its `--workspace` root: an absolute path outside it is refused, and so is a `..` traversal or a symlink pointing out (previously both were read, which was [the path-confinement fix](https://github.com/delivstat/swarmkit/pull/702)). So the paths must be absolute **and** under the workspace root. If your documents genuinely live elsewhere, root the server there rather than setting `SWARMKIT_DOCS_READER_ALLOW_OUTSIDE=1`. ## Make opening them non-optional Availability is not use. Add an instruction that names the consequence: > The field labels, the button text and the message wording are IN the image and nowhere else. A > screen described without opening it is invention, however plausible it reads. If a path genuinely > fails, say so — naming the path — rather than describing the screen anyway. Also tell it not to translate what it sees. The literal string is what gets built. ## Evidence that it matters Same requirement, same archetype, same model. Only the paths changed. | | paths broken | paths absolute | |---|---|---| | screens described | 2, invented | **3**, matching the real panels | | panel titles | generic | `Mobile_Confirm_PGM`, `Mobile_Confirm_Shipment` — only present in the image | | UI literals captured | 1 | **12**, including `Back`, which appears nowhere in the requirement text | | `view-screenshot` calls in trace | 0 | 3 | | tokens | ~10k | ~25k | `Back` is the tell. It is on the panel and in no prose anywhere, so it could only have come from the pixels. Cost roughly 2.5×. Worth it: the alternative was a specification whose UI section was fluent invention. ## Two practical notes **Send composed screens, not fragments.** If your extractor composites overlays onto a base screenshot, send only the composites. Offering both invites the model to describe a stale label from a fragment as though it were current. **Budget largest-first if you must budget.** An image costs about 4/3 of its file size once base64-encoded. If something has to be dropped it should be a fragment, not the screen. A 64 KiB line limit on harness stdout used to make large images fatal — `Separator is found, but chunk is longer than limit`. Fixed in 1.129.2; budgets written around it can be relaxed. ## Checklist If you hold the file before the run: - [ ] It is passed as `--attach` / `attachments`, not mentioned in the prompt - [ ] The consumer is a model agent at the root (a harness root does not receive it) - [ ] The `run.attachments` audit event shows the expected name, type, size and digest If an agent has to choose what to look at: - [ ] An MCP tool returns `ImageContent` — a path or base64 in the prompt does nothing - [ ] Paths handed to the agent are **absolute**, and under the docs-reader workspace root - [ ] Image refs inside any document the agent reads are rewritten to absolute too - [ ] The prompt says to open them, and why - [ ] The prompt says to report a failed path rather than describe the screen anyway - [ ] The trace shows the tool actually fired — one call per image, not zero - [ ] Spot-check one detail that exists only in the pixels ## See also - [Document reader MCP](../design-notes/document-reader-mcp.md) — the server that provides `view_image`, and why a multimodal path exists at all. - [Building swarms](building-swarms.md) — where skills and MCP servers are granted to an agent. ══════════════════════════════════════════════════════════════════════════════ # Part 7 — Design notes (the depth) ══════════════════════════════════════════════════════════════════════════════ ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/gate-funnel.md ## Published: https://delivstat.github.io/swarmkit/design-notes/gate-funnel/ ══════════════════════════════════════════════════════════════════════════════ # Gate funnel (per-artifact quality gate composition) Parent: `design/details/sdlc-pipeline-example.md` (capability 2 of 5). Standalone and reusable — any workspace can put a funnel on any artifact-producing node; the SDLC pipeline is the first consumer. Every artifact worth a human sign-off should reach that human already having cleared the cheap, automatable checks. This note defines the **composition** — a declarative, per-artifact gate that chains *structured-output validation → LLM-as-judge → (optional) harness review → human approval* into one reusable unit, with the control flow, the bounded retry loop, and the structural invariant that the automated layers **filter but never decide**. ## Implementation status Implemented (slice 3). What shipped, and where it refines this note: - **Funnel is a first-class artifact** (`kind: Funnel`, `metadata` + `provenance`), not an inline gate block. A topology node references it by id — `funnel: ` with `x-swarmkit-ref: funnel` — matching the reference-not-inline invariant (skills/archetypes). The `approve` layer mirrors `SwarmKitApprovalPolicy` inline. Schema: `packages/schema/schemas/funnel.schema.json`. - **Runtime:** funnels are discovered under `funnels/`, resolved into a workspace funnel registry, and bound to the node (`ResolvedAgent.funnel`). `compile_funnel_gate` (`langgraph_compiler/_gate_funnel.py`) compiles the gate subgraph; the structural invariant is asserted on the *compiled graph* (only `approve` reaches the terminal). The `approve` layer is wired to the real `resolve_multiparty` engine via `build_multiparty_approver`. - **Audit:** decision-skill verdicts (the judge included) now emit an append-only `decision.evaluated` audit event through the `GovernanceProvider` seam. - **Control flow is fixed, not user-drawn.** A funnel configures the layers and toggles the optional ones; it does not rewire the graph — that is what keeps the invariant load-bearing. The composer's funnel editor is a structured pipeline editor, not a free-form canvas. - **Slice-4 boundary:** binding the gate into a *live multi-agent topology run* (the drafter re-running the gated agent inside the loop) is the one-app stage-run slice. Slice 3 delivers the funnel artifact, its compilation, the invariant, and the real approve/judge wiring. It composes existing pieces and sibling capabilities; it does not redefine them: - **Layer 1 — structured-output validation:** SwarmKit's **native** four-tier output governance, Tiers 0–2 (all deterministic): `constrained-output-schema.md` (implemented) + `structured-output-governance.md`. This is the runtime's own schema-constrained decoding + JSON- Schema check + business-rule field checks with field-specific auto-correction — **no a managed backend or external dependency** (a managed backend is only the credited inspiration in that note, not a component). - **Layer 2 — LLM-as-judge:** an existing governance **decision skill** (`decision-skills.md`, `governance-decision-skills.md`), instantiated as the rubric-parameterised `artifact-judge`. - **Layer 3 — harness review:** sibling `design/details/harness-reviewer.md`. - **Layer 4 — human approval:** sibling `design/details/multi-party-approval.md`. ## Goal Make "every artifact is judged before a human sees it" a **declarative property of a gate**, not bespoke wiring per stage. One funnel schema, applied per artifact, that compiles to a gate whose only exit to human approval is *through* the automated layers — cheap → expensive → human. ## Non-goals - **Not the layers' internals.** The judge rubric, the reviewer archetype, and the approval policy are defined by their own notes/skills; the funnel only *sequences and gates* on them. - **Not stage sequencing.** How a failed/exhausted funnel routes across stages, and the cross-stage defect loop, are `pipeline-controller`. The funnel's scope is one artifact, one gate. - **Not a new judge or validator engine.** It reuses decision skills + structured-output validation as-is. ## Where it lives The funnel is a **governance gate composition**: its schema is gate configuration, and it compiles (runtime) to a subgraph whose control flow *structurally* routes through the human interrupt — there is no compiled edge that reaches "done" while skipping human approval. The advisory invariant is enforced by graph shape, not by prompt. ## API shape ### The funnel schema (per artifact) Each layer is **optional except `approve`** (a funnel with only `approve` degrades to a plain multi-party gate). Present layers run in order. ```yaml gate: consolidated-design-approval artifact: consolidated-design funnel: validate: # layer 1 — deterministic, no LLM schema: schemas/consolidated-design.json autocorrect: true # native field-specific re-prompt (Tier 1–2); unrepairable → retry judge: # layer 2 — governance decision skill skill: artifact-judge rubric: rubrics/consolidated-design.md threshold: 0.8 # below → retry max_retries: 2 review: # layer 3 — optional; heavyweight gates only archetype: architect-reviewer read_scope: [app:oms, app:web, app:mobile] route_back_at: high # findings >= this severity retry; others attach approve: # layer 4 — the multi-party approval set (sibling note) rules: - { scope: design:approve, roles: [oms-lead, web-lead, mobile-lead], quorum: all } - { scope: security:approve, roles: [infosec-lead], quorum: all } ``` ### Control flow ``` draft ─▶ validate ─(ok)▶ judge ─(pass)▶ review ─(no route-back)▶ APPROVE (human) ─▶ done │ │ │ (unrepairable) (below threshold) (finding ≥ route_back_at) └──────────────┴─────────────────────┘ ▼ retry: critique/findings ─▶ drafting agent revises ─▶ re-enter at validate ``` - **validate** (native Tiers 0–2): structured generation + JSON-Schema check + business-rule field checks, with field-specific auto-correction re-prompts; a shape that auto-correction cannot repair is a retry (the judge never sees malformed input — kills shape hallucination up front). - **judge** (native Tier 3 — an LLM-as-judge decision skill): scores against the rubric; `< threshold` is a retry carrying the critique. - **review** (optional): the harness reviewer investigates and returns findings; findings at or above `route_back_at` retry (carrying the findings), the rest **attach** and travel to the human. - **approve**: the binding human layer (per-role tasks, quorum, `min_distinct_approvers` — sibling note). The only edge to `done`. ### The retry loop (bounded; exhaustion escalates, never drops) A retry feeds the failing layer's critique/findings back to the **drafting agent**, which revises and re-enters at `validate`. Bounded by `max_retries`. On **exhaustion** the funnel does **not** drop the requirement or silently pass — it **escalates to a human** with the last failing critique attached, and that human decides (force-advance to approval, or reject). Retry state lives in the run checkpoint, so it is durable and resumable. ### The advisory invariant (structural) Layers 1–3 are **advisory**: they gate *advancement to* human approval and drive the retry loop, but they **never** approve. Two properties, enforced by the compiled graph shape (not prompt): 1. **No bypass of the human gate.** There is no edge from any automated layer to `done`; the only path to `done` is through `approve`. A judge/reviewer "pass" advances *to* the human, never *past* them. 2. **No reaching the human without passing.** `approve` is only reachable after `validate` + `judge` succeed (and non-blocking review has attached). A below-threshold artifact cannot land on a human's desk except via the explicit retry-exhaustion escalation. This is the judicial pillar (§8) filtering for the legislative/human one, not substituting for it. ### Provenance bundle (what the human sees) On reaching `approve`, the funnel assembles a bundle the human task carries: the artifact, the validation result, the judge score + critique, the attached reviewer findings, the retry count, and the diff-since-last-approval. `task-surface-and-board` renders it; the funnel produces it, so a human decides in one place with the full automated context. ## Test plan - **Schema (Python + TS):** a funnel with only `approve` validates (degenerate = plain gate); a funnel referencing an unknown skill/archetype is rejected; layer ordering is fixed regardless of key order. - **Control flow (integration):** a malformed field is auto-corrected and proceeds; a below-threshold draft triggers a retry whose revision then passes and reaches the human; a `route_back_at`-severity finding retries while a low finding attaches and proceeds. - **Advisory invariant (the load-bearing tests):** there is *no* execution path from a judge/review pass to `done` without the human interrupt; `approve` is unreachable while `judge` is below threshold — asserted on the compiled graph, not just at runtime. - **Retry exhaustion:** after `max_retries` the funnel escalates to a human with the last critique attached — it neither loops forever nor drops nor silently advances. - **Provenance:** the human task carries artifact + validate result + judge score + findings + retry count + diff. ## Demo plan `just demo-gate-funnel`: a single artifact through a full funnel — show (a) an auto-corrected field, (b) a judge fail → auto-retry → pass, (c) a reviewer finding attaching to the human task, (d) the human approval as the sole exit, and (e) a second run where retries exhaust and the gate escalates to a human instead of dropping. Terminal transcript in the PR body. ## Schema-change checklist Adds a `funnel` gate sub-schema (composing the `approval` block from `multi-party-approval`) — follow `docs/notes/schema-change-discipline.md`: canonical JSON Schema, Python + TS validators, and fixtures updated together. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/gate-state-and-deferring-approval.md ## Published: https://delivstat.github.io/swarmkit/design-notes/gate-state-and-deferring-approval/ ══════════════════════════════════════════════════════════════════════════════ # Reading a gate, and approving without a saga **Status:** implemented — #794, #795, #797, #799. `GET /gates/{gate_id}` resolves a gate with the approval policy applied (`gate_state.py`); the funnel `approve` layer raises `HITLDeferredError` and the run checkpoints as `deferred`, resumed by `swarmkit run --resume` or `POST /jobs/{id}/resume`; gate ids unified on `{run_id}:{agent_id}`; the gate UI approves with a link to the job, backed by `GET /artifacts/{ref}`; and a re-run records `parent_job_id`. Implements the two items from [`finishing-the-orchestration-seam.md`](finishing-the-orchestration-seam.md) that block an application owning its own sequencing. Everything else in that note can wait; these cannot, because without them an application that drops the saga also drops enforced human approval. --- # Part 1 — `GET /gates/{gate_id}` ## Goal Let a caller ask *"is this gate resolved?"* and get the answer **with the approval policy already applied**. ## Why it is not just a review-queue filter `GET /review?gate_id=…` returns the individual role-tasks. Turning those into a decision means applying quorum, distinct-approver counts and `exclude_author` — which lives in `evaluate()` and `collect_resolutions()` and is exactly the SwarmKit-shaped part an application must not rebuild. Without this endpoint every external driver either reimplements the approval policy or approximates it, and an approximation of an approval policy is a governance failure with a friendly name. It is also the one thing a client genuinely cannot derive: the policy lives in the funnel, which the client does not read. ## Shape ``` GET /gates/{gate_id} → { "gate_id": "wms-design:designer", "status": "pending" | "approved" | "rejected", "policy": {"scope": "design:approve", "roles": ["oms-lead"], "quorum": "all", "exclude_author": true, "min_distinct_approvers": 2}, "resolutions": [{"item_id": "...", "status": "approved", "resolved_by": "...", "role": "..."}], "distinct_approvers": ["alice"], "artifact_ref": "WMS-35//output" } ``` `status` is the only field a driver must understand; the rest is for a human reading why. Backed by a pure function — `gate_state(queue, registry, policy, gate_id) -> GateState` — so the endpoint, a CLI command and a test all ask one implementation, as `gate_coverage` does. ### Resolving the policy The gate id determines it, and there are two shapes in use: - **funnel-on-agent:** `"{topology_id}:{agent_id}"` → topology → agent → funnel → `approve` block. - **stage gate:** the `gate:` value is a funnel id → funnel → `approve` block. Both resolve deterministically from the workspace. A gate id matching neither is a 404 rather than a guess. ### CLI `swarmkit review gate ` — under `review`, not `gates`, because `swarmkit gates` is pipeline gate *coverage* (a static analysis) and this is a live queue question. The naming collision is unfortunate and worth a second opinion. ## Non-goals - Not a webhook. Polling is adequate and the ADR already lists callbacks as pending. - Not resolving gates. `POST /review/{id}/approve` already does that. - Not changing quorum semantics. --- # Part 2 — the funnel's approve layer defers ## Goal Human approval that works whether or not a saga exists — under the bundled controller, under Temporal, or under a shell script. ## What changes `build_advisory_approver` records and passes, justified in 1.172.0 as *"human approval is the stage-level `gate:`, which parks the saga durably"*. With sequencing in the application there may be no saga and no stage gate, so approval would be unavailable on exactly the path being recommended. It was already the weaker branch. The choice was framed as *block the coroutine for seven days* or *pass advisorily*; rejecting the block was right, but **defer-and-resume already existed** — `HITLDeferredError` checkpoints, closes the job `deferred`, exits cleanly, and `swarmkit run --resume` continues after `swarmkit review approve`. ## The part the earlier note missed: resume re-runs the node LangGraph checkpoints at super-step boundaries. A node that raises is re-run on resume — so a naive defer would **re-draft the artifact** after approval. On `wms-design` that is ~$2.40 per approval, and worse: the human would have approved an artifact that no longer exists. So the gated node has to be **re-entrant**: ``` on entry to the gated node: gate = "{topology_id}:{agent_id}" state = gate_state(gate) # Part 1's function, in-process approved → return the artifact stored on the gate; produce nothing rejected → return [GATE REJECTED] with the resolver's comment; produce nothing pending → defer again; produce nothing absent → produce → validate → judge → open the gate → defer ``` Only the last branch runs the agent. "Resuming while still unapproved defers again" stops being a test case and becomes the same code path as the first defer. The artifact is already stored on the review item (`open_gate` posts it with `artifact_ref`), so no new storage is needed — only the branch. **Edge case, stated:** if the gate is absent on re-entry because the item was purged, the node produces again. Acceptable and cheaper than the alternatives; worth a log line so it is not silent. ## The gate id is not unique per run Two conventions exist, and only one of them is correct: ``` _stage_runner.py:138 gate_id = f"{correlation_id}:{agent.id}" # run-unique _compiler.py:954 gate_id = f"{topology_id}:{agent.id}" # NOT run-unique ``` The in-node id carries no correlation and no run. In the model this design is for — **independent runs connected by a correlation id, not stages of a pipeline** — every run of `wms-design` produces the gate id `wms-design:designer`. Two tickets in flight would share review items, have quorum counted across both, and approving one would release the other. It is latent today only because the in-node approve is advisory. **Turning it on without fixing this ships a governance bug**: an approval granted for one requirement satisfying a different one. ### The fix, and the convention question The in-node gate id becomes **`"{run_id}:{agent_id}"`**. The run id is already in the run scope (`_run_scope`, 1.175.0), it is `jobs.id`, and a driver holds it from the `POST /run/{topology}` response — so it stays derivable without the driver knowing anything new. Run id rather than correlation id is deliberate. A correlation groups *several* runs (a retry, a second attempt at the same ticket), and a gate keyed on it would let approvals cast against a previous artifact satisfy a new one. `open_gate` already documents that hazard for retried stages — *"those approvals were cast against the PREVIOUS artifact, which is arguably wrong"* — and this is the chance not to inherit it. That leaves two conventions in the codebase, which is what caused the problem. **Recommendation: unify on `"{run_id}:{agent_id}"` everywhere.** A stage's run id is already `:`, so a stage gate becomes `WMS-27:design:designer` — still correlation-bearing, and run-unique for free. It is a breaking change for gates open at upgrade time, which is the only reason not to. ## The compatibility problem: double gating A pipeline stage with `gate:` whose agent *also* has a funnel with `approve:` would gate **twice** — once in-node, once at the stage. Today only the stage gate fires, because in-node is advisory. This only arises for stage runs. The model this design serves — independent runs joined by a correlation id — has no stage and no `gate:`, so the in-node layer is the *only* gate and must enforce. **Decision: suppress the in-node approve for pipeline-stage runs.** `_pipeline_stage` already opens the gate and returns `parked`; it passes a flag saying so, and the in-node layer stays advisory on that path. Existing pipelines behave byte-identically; independent runs gain enforced approval, which is the whole point. No workspace flag. A configuration switch here would be one more thing that can be declared and not read, and the distinction (is this run a pipeline stage?) is known at the call site. ## Serve does not handle deferral `HITLDeferredError` is caught only in the CLI. Under `swarmkit serve` a deferred run currently surfaces as a **failed job** — so this must be handled there too: job status `deferred`, the gate id in the error field, and the existing resume path reachable. That implies `POST /jobs/{job_id}/resume` (item 3 of the seam note) lands with this rather than after it, or serve can defer and never continue. ## What is required to wire it - `review_queue` and `role_registry` passed into `compile()` again, for the approve layer only. The 1.172.0 guard removal stays correct — `validate` and `judge` must never depend on a queue. - `WorkspaceRuntime.compile()` supplies both; it already resolves the workspace and the review queue is a filesystem/queue object it can construct. ## Non-goals - Not removing `gate:` from the stage schema. - Not changing what `validate` or `judge` do. - Not making approval mandatory. A funnel with no `approve` block is unaffected. --- # Part 3 — the approval surface ## The UI mislinks a topology-run gate today `packages/ui/app/gates/page.tsx:13` hard-codes one of the two conventions: ```ts /** A gate id is `:`; split on the LAST colon */ export function runOf(gateId) { … } // → correlation_id export function stageOf(gateId) { … } // → agent_id, but named "stage" ``` That is the stage-runner shape. For an in-node gate (`{topology_id}:{agent_id}`) `runOf()` returns a **topology id**, and the link goes to `/runs?run=wms-design&stage=designer` — a saga search, which finds nothing and renders *"No pipeline runs to show"*. Latent only because the in-node approve is advisory and opens no gate. It breaks the day Part 2 lands: every topology-run gate would list correctly and link to a dead pipeline view. **This settles the convention question.** Unify on `{run_id}:{agent_id}` where `run_id` is always `jobs.id`. A stage's run id is already `:`, so its gate becomes `WMS-27:design:designer` — correlation-bearing and run-unique, and the existing split still resolves. The alternative is a client branching on which convention a gate happens to carry, which is unmaintainable. Better: **stop parsing gate ids in the client.** `GET /gates/{gate_id}` returns `run_id`, `topology_id`, `agent_id` and `artifact_ref` resolved, so no surface infers structure from a string. The parse function is the "two systems must agree about identity" hazard in miniature. ## Approval belongs in the gate UI, not the job page The gates page currently punts — *"role-tasks are LISTED here but resolved in the run view: this page has no artifact to show"*. That is true today, and not because the item lacks the artifact: `ReviewItem` carries `artifact_ref` and `_item_to_dict` returns it. **There is simply no `GET /artifacts/{ref}` to fetch the content** — the same gap Part 1's note lists for external orchestrators. One endpoint serves both. With it, approval stays where the approver is: - **Gate UI — the approver's inbox.** Renders the artifact, the policy state from `GET /gates/{id}` (who has approved, what is still needed), approve/reject with a comment, and a **link to the job** for execution detail. - **Job page — read-only about the decision.** Shows that this run is gated and links to its gate; no approval controls. The split is not cosmetic. The job page is about **execution** — tool calls, usage, trace; approval is a **decision**. Merging them makes the job page do two jobs and buries the approver's queue inside a browse surface. It also survives Part 3 of the extraction: when the pipeline surfaces leave serve, the gate UI is unaffected, whereas an approval control living in `/runs` would leave with them. ## A re-run is a new job, and the chain has to be recorded A rejected artifact is redone by running again, which writes a **new job row**. Two consequences. **The new run gets a new gate, automatically.** Because the gate id is keyed on `run_id`, a re-run cannot land on the previous run's gate. That is the correct behaviour and it is structural rather than remembered: `open_gate` documents the hazard today — *"those approvals were cast against the PREVIOUS artifact, which is arguably wrong"* — and keying on the run removes it. **`correlation_id` cannot express the chain.** It groups runs, but it is already overloaded: in the application-owned model a correlation is a *ticket*, holding different units of work as well as retries. "Same ticket" and "supersedes" are different facts. Proposed: **`jobs.parent_job_id`**, nullable, through the existing additive-column facility (`_ADDED_JOB_COLUMNS`). Then the attempt number is derivable, the chain is walkable in both directions, and "what did this artifact cost including retries" becomes answerable — which the per-run cost figures cannot answer today. ### The rejection carries forward as a critique A reviewer rejects with a comment. That comment is a **critique** — the same thing the funnel's judge produces, and the funnel already carries a critique back to the drafter on retry. A human rejection should reuse that channel: the re-run starts with the reviewer's words as its critique, rather than re-drafting blind and rediscovering the objection. **The runtime does not re-run on its own.** A rejection means a human said no; spending again is an operator decision, and a runtime that automatically re-spends on rejection would be deciding budget on a human's behalf. The runtime records the link and carries the critique; the driver or operator triggers the new run. --- ## Test plan **Part 1** - Quorum applied: two of three roles approved with `quorum: all` reads `pending`, not `approved`. - `exclude_author` honoured: the author's own approval does not count. - A rejection anywhere reads `rejected`. - Both gate-id shapes resolve their policy; an unknown gate 404s. - **Two concurrent runs of the same topology produce two distinct gates**, and approving one leaves the other pending — the defect this design would otherwise ship. - The endpoint and the CLI return the same verdict for the same queue state. **Part 2** - A run whose funnel gate is unresolved closes as `deferred`, writes review items, holds no process. - **Resuming after approval completes the run and calls the model provider ZERO further times** — the assertion the re-entrancy exists for. - Resuming while still pending defers again, and still does not call the provider. - A rejection returns a gate-rejected result carrying the resolver's comment. - A pipeline stage with `gate:` gates exactly once, and its saga timeline is unchanged from 1.180.0. - A funnel with no `approve` block never defers. - Serve records `deferred` rather than `failed`, and the job resumes over HTTP. - Parity: the same funnel-gated topology approves identically under the bundled controller, under a bare `swarmkit run` loop, and over HTTP. **Part 3** - A topology-run gate and a stage gate both link to a resolvable job — the mislink asserted against the id the UI actually builds, not against a fixture. - `GET /artifacts/{ref}` returns the artifact a gate references, so the gate UI can render what is being approved. - A re-run writes a new job carrying `parent_job_id`, and **opens a NEW gate**: approvals cast on the previous run's gate do not satisfy the new one. - The chain is walkable and the attempt number derivable; cost sums across it. - A rejection's comment reaches the re-run as its critique. - The runtime does **not** start a re-run by itself on rejection. ## Demo plan A one-shot run against a funnel-gated topology: ``` $ swarmkit run ws wms-design -i "…" --correlation-id WMS-35 ⏸ Review deferred: gate wms-design:designer awaits approval 1. Approve: swarmkit review approve 2. Resume: swarmkit run ws wms-design --resume $ swarmkit review gate wms-design:designer pending — 1 of 2 distinct approvers (needs oms-lead) $ swarmkit review approve && swarmkit run ws wms-design --resume [designer] resumed from gate (no re-draft) ``` Plus the cost line before and after resume, showing the second half added nothing. ## Open questions for review 1. **`swarmkit review gate` vs extending `swarmkit gates`.** Coverage and live state are different questions; putting them under one noun may still be less confusing than two. 2. **Should a rejected gate fail the run or return a rejected artifact?** Today's gated node returns `[GATE REJECTED]`, which a caller may treat as output. For a driver, a non-zero exit is easier to branch on. 3. **Does the pipeline-stage suppression flag belong on the run request or in the compile?** Call site is cleaner; a run-level field is more visible in the audit record. 4. **Unify the gate-id convention now, or leave the stage path alone?** One rule is better than two — two is what produced the collision above — but changing the stage path breaks gates open at upgrade time. 5. **Does `parent_job_id` belong on the job, or should the chain live in the application?** The application already models ticket hierarchy; a runtime-side chain is a second place the same fact lives. Against that: cost-across-retries is a runtime question, and the runtime is the only thing that knows a run superseded another. 6. **Should `deferred` be terminal for the job row, or a distinct resumable state?** It closes the row today, which makes "how many runs are waiting on a human" a query over `status='deferred'` — convenient, but it conflates "finished" and "paused" in the same column. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/extracting-the-pipeline.md ## Published: https://delivstat.github.io/swarmkit/design-notes/extracting-the-pipeline/ ══════════════════════════════════════════════════════════════════════════════ # Taking the pipeline out of the runtime **Status:** implemented — deprecated in 1.188.0, removed in 1.189.0. The *why* is settled in [`finishing-the-orchestration-seam.md`](finishing-the-orchestration-seam.md); this note is the *how*: what actually leaves, what cannot leave yet, and in what order. ## What this is Removing stage sequencing from `swarmkit-runtime` and `swarmkit serve`. Topologies, archetypes, skills and funnels stay. An application sequences runs itself, joining them with a correlation id. It is **not** a rewrite. Most of the surface is additive and lifts out; the interesting part is the three pieces that do not, and the order they force. ## Inventory | component | LOC | verdict | | --- | --- | --- | | `orchestration/` (saga state, store, reference controller) | 1169 | leaves | | `server/_routes_pipelines.py` | 407 | leaves | | `server/_routes_sagas.py` | 89 | leaves | | `server/_pipeline_stage.py` | 439 | **blocked** — holds the only `open_gate` call | | `cli/_cmd_pipeline.py`, `cli/_cmd_orchestrator.py` | — | leaves | | `triggers/_pipeline_ingress.py` | 77 | **stays**, reframed | | `gate_coverage.py` + `cli/_cmd_gates.py` + `GET /pipelines/{id}/gate-coverage` | — | **splits** | | UI `/runs`, `/pipelines`, `lib/gate-coverage.ts` | — | leaves | | `persistence`: `StoreKind.SAGA`, `saga_store()` | — | leaves | | `schemas/stage-graph.schema.json` | — | leaves, last | | `deploy/pipeline` | — | leaves | Roughly 2,100 lines of runtime plus two UI pages. The reference app that replaces it should be a fraction of that, because most of what leaves is durability the application no longer owns. ## The four categories ### 1. Leaves cleanly Routes, CLI commands, UI pages, the controller and the saga store are additive: nothing outside the pipeline reads them. `ReferenceController` imports only `RunStage`, `StageOutcome`, `SagaState` and `SagaStore` — all from its own package. ### 2. Stays, and gets a better name `PipelineSignal` is not an engine, it is a **type alias** — `Callable[[str, str], Awaitable[None]]`. `_routes_jobs.py` and `server/_mcp.py` depend on the *shape*, so a webhook or an MCP tool can emit an event without knowing what consumes it. Together with `triggers/_pipeline_ingress.py` (77 lines, correlation extraction and signature checking) that is the **inbound integration seam**, and it is exactly what an application-owned orchestrator needs to be driven by. It should survive under a name that does not say "pipeline" — the thing it does is *deliver a correlated external event to whatever is listening*, which is useful whether or not SwarmKit sequences anything. ### 3. Cannot leave yet `server/_pipeline_stage.py` is not routing. It threads upstream artifacts into a stage's input, and it **opens the funnel gate** — the only `open_gate` call in a run path. Delete it today and gates stop existing. Its replacement is Part 2 of [`gate-state-and-deferring-approval.md`](gate-state-and-deferring-approval.md): once the funnel's approve layer opens its own gate and defers, `_pipeline_stage` holds nothing that is not either sequencing (leaves) or artifact threading (which the application does with `--correlation-id` and `GET /artifacts/{ref}`). **This is the whole dependency.** Extraction is blocked on the gate work, and on nothing else. ### 4. Splits in two `gate_coverage` computes "the narrowest verified edge" over a StageGraph. With no stage graph there are no edges — so the pipeline half goes, and `swarmkit gates --require` goes with it, which is a real loss for anyone gating CI on it. But the analysis has a second half that is not pipeline-shaped at all: **which agents carry a funnel, and how strong is it** (`validate` / `judge` / `review` present, `approve` policy). That question is about topologies and funnels, both of which stay, and it is the natural sibling of the reachability report — *"this topology's output is verified by nothing"* is the same class of finding as *"this binding is reached by nothing"*. Recommendation: keep the per-agent funnel-strength analysis, retire the per-stage edge analysis with the stage graph, and fold the survivor into `swarmkit validate` beside reachability rather than leaving a `gates` command whose subject has left. ## The order, which is not negotiable 1. **Gate work** — `GET /gates/{gate_id}`, approve-defers, gate-id unification, the UI link (all of `gate-state-and-deferring-approval.md`). Approval stops depending on a saga. 2. **HTTP parity** — `correlation_id`/`labels` on `RunRequest`, `GET /artifacts/{ref}`, `POST /jobs/{job_id}/resume`. An application can now drive a sequence over the API. 3. **`_pipeline_stage.py` becomes removable** — it holds nothing unique once 1 and 2 land. 4. **The reference app** proves the boundary by running the `sdlc-pipeline` example through the public API only. 5. **Deprecate, then delete.** Steps 1 and 2 are worth doing whether or not the extraction ever happens, which is the argument for starting there: nothing is wasted if the decision reverses. ## Where it goes `examples/pipeline-orchestrator/` — a reference application, not a distributed package. A package (`swarmkit-orchestrator`) is the more complete-sounding answer and probably the wrong one: it is a maintenance commitment to an engine this note argues SwarmKit should not be in the business of, and the people most likely to adopt this — including the WMS application — are writing their own driver against their own tracker, not installing ours. `examples/sdlc-pipeline/orchestrator/temporal/` already exists and shows the Temporal shape of the same loop; what it lacks is the HTTP form, because step 2's calls do not exist yet. Converting it to use only the public API is the acceptance test for the whole extraction. ## What existing deployments do Nothing, for at least one release. The bundled controller keeps working, keeps getting bug fixes, and is documented as what it already calls itself: a *reference* sequencer for simple linear pipelines. Capability freezes immediately — no event routing, no fan-out, no cycles. A workspace that needs those is a workspace that should own its sequencing, and that signal is more useful than the feature. ## Data - **`pipeline_saga` leaves** with the controller. - **`pipeline_artifacts` stays.** Since 1.179.0 a one-shot run writes to it, so it is the general artifact store with a misleading name. Renaming a table is a migration for a cosmetic gain; the honest minimum is documenting that the name is historical. - Job rows, audit events and artifacts written by pipeline runs stay readable. Extraction must not orphan history: a `:` job id keeps resolving after the sequencer is gone. ## Non-goals - Not removing funnels, gates, or the review queue — the opposite; Part 2 makes them independent. - Not prescribing an engine. - Not deleting `stage-graph.schema.json` before the bundled controller goes. - Not changing how a run executes. ## Test plan - Every pipeline surface removed has a test asserting the *runtime* still works without it: a funnel-gated one-shot run approves and resumes with no saga store present at all. - **A workspace with no orchestrator configured can still gate** — the assertion that step 1 landed. - History survives: a job row, audit events and artifacts written under `:` are still fetchable after the sequencer is removed. - The funnel-strength half of `gate_coverage` still reports; the stage-edge half is gone and `swarmkit gates` says where it went rather than 404-ing on a missing pipeline. - The reference app drives the `sdlc-pipeline` example end to end, with one human approval, importing no `swarmkit_runtime` module. - The inbound signal seam still delivers a correlated webhook event after the pipeline routes leave. ## Demo plan The `sdlc-pipeline` example run twice — once on the bundled controller, once on the reference app over HTTP — producing the same artifacts, the same approval, and one correlated trail in `swarmkit logs`. Then the same run with `orchestration/` uninstalled, showing gates unaffected. ## Open questions for review 1. **Reference app or distributed package?** The note argues for `examples/`. A package makes the migration one install line for anyone on the bundled controller today. 2. **Does `stage-graph.schema.json` stay in `packages/schema`?** If sequencing is the application's, the format arguably is too — but a shared format is how the reference app and the bundled controller stay compatible during the transition. 3. ~~**What replaces `swarmkit gates --require` for CI?**~~ **Resolved** — `swarmkit validate --require-verified`, gating on "every topology root's output is checked by something that actually runs". See `design/details/funnel-verification-check.md`. 4. **Is one release enough deprecation** for a subsystem someone is running in production? ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/skill-prerequisites.md ## Published: https://delivstat.github.io/swarmkit/design-notes/skill-prerequisites/ ══════════════════════════════════════════════════════════════════════════════ # Declarative skill prerequisites (`requires:`) **Status:** implemented — runtime 1.191.0. Filed from the tlgsg-wms workspace against 1.180.0. Shipped as designed, with the open questions resolved as follows: **shape B** (a sibling `requires:` block, `skills` unchanged); **grant-only**, on the topology agent node — an agent can order skills it inherits from an archetype, and no skill definition declares its own prerequisites; **MCP-scoped**, as the note bounds it; and **no refusal ceiling** — the existing per-turn tool cap already bounds a thrashing loop, and a second limiter that fails the run would turn a recoverable refusal into a terminal one. ## Goal Let a topology say that one skill call is a precondition of another, and have the runtime enforce it — so ordering is a reviewable property of the workspace rather than a request in a prompt. ## The evidence, and why it is decisive Three attempts in one workspace, all failing the same way: an archetype mandating `list_build_conventions` produced **0** calls; 25,804 characters of convention index injected at `pre_input` were demonstrably ignored; and `get_estate_reference` — granted, documented, and instructed with the reason attached — was called **0** times. The controlled comparison is the one that settles it. In a **single run, same agent, same prompt**, the ack-gated `get_build_convention` was called 4 times and the merely-requested `get_estate_reference` 0 times. The variable is not the prompt; it is whether the tool *refuses service*. That is the same finding as "tool names beat prompts", one level up: a model follows mechanism over instruction, so ordering has to be mechanism. ## What exists today, and why it is not the answer An HMAC handshake in `mcp-servers/_ack.py`: `get_build_convention` mints a token, every estate tool refuses without one. It works — and it is in the wrong place. The constraint lives across seven server files, is invisible to `swarmkit validate`, threads an `ack` argument through every tool schema the agent sees, and couples the runtime to the servers by a shared secret. ## The mechanism already exists Both executors dispatch through **one function**: - model path — `_skill_executor.py:125` → `check_mcp_permission` - harness path — `_gateway.py:240` → `governed_mcp_call` → `check_mcp_permission` It already returns `(allowed, reason)`, and both paths already surface that reason to the agent as a tool error it can act on. **A prerequisite is a second reason to deny.** One enforcement point covers both executors, and the recoverable-refusal behaviour that makes the ack gate work is the behaviour that is already there. Refusals are already shaped for the audit log too: a denied gateway call has emitted `skill.executed` with `policy_decision="deny"` since 1.177.0. ## Where the constraint is declared `skills` is currently `array` with `x-swarmkit-ref: skill`, which drives reference validation and codegen. The request proposes a mixed list of strings and single-key maps. Two shapes: **A — inline on the grant** (as proposed) ```yaml skills: - list-build-conventions - get-build-convention: requires: [list-build-conventions] ``` **B — a sibling block, `skills` unchanged** ← recommended ```yaml skills: [list-build-conventions, get-build-convention, search-solution-code] requires: get-build-convention: [list-build-conventions] search-solution-code: [get-build-convention] ``` B keeps `skills` a plain identifier array — no mixed-type list for every consumer and the pydantic model to handle, no disturbance to `x-swarmkit-ref`. Duplicates become impossible because it is a map. And it puts the ordering rules **in one readable block**, which is half the stated value: *"a reader of the topology cannot see that an ordering rule exists."* Scattered through a list, they are as easy to miss as they are in the servers. A reads better at the point of use, and that is a real argument. It is a surface decision — see the open questions. ## Semantics - **Per `(run, agent)`, not per run.** The request says "same run"; the reasoning in it says otherwise — *"continue in the same session with that content live in its context"*. If agent A read the card, agent B does not have it in context, and a run-scoped set would let a parallel sibling satisfy a prerequisite it never saw. - **Satisfied by a prior successful call.** An exception or an MCP `isError` result does not satisfy. - **Enforced at dispatch**, before the server is touched, and **recoverable**: the agent calls the prerequisite and retries within the same loop. - **Order-independent among peers.** `requires: [a, b]` means both, in any order. ### The error message is the mechanism The ack gate works because its refusal is *actionable*. A generic "permission denied" invites give-up or thrash. The message names the missing prerequisite and says what to do: ``` get-build-convention requires list-build-conventions, which has not been called in this session. Call list-build-conventions first, then retry. ``` This is specified and tested rather than left to implementation, because it is the part doing the work. ### What "successful" can honestly mean The runtime sees exceptions and the MCP `isError` flag. **A tool that returns prose saying "not found" as a successful result will satisfy its prerequisite.** Criterion 3 holds for real failures and not for tools that report failure in their payload; better to state that than to imply a guarantee the seam cannot give. ## Validation - A `requires:` naming a skill the agent does not hold is a **resolution error** — the reviewable half of the feature, and the check that makes the block trustworthy. - **A cycle is a resolution error.** `a requires b`, `b requires a` permanently blocks both, and the agent can never recover. The request does not mention cycles; unchecked, this feature can render an agent unable to act with no error anywhere. ## What lands in the trace A refusal emits `skill.executed` with `policy_decision="deny"` and a reason naming the prerequisite — distinguishable from an ordinary tool error, per criterion 5. This is what makes the gate *measurable*: today only calls that happened can be counted, and **a gate that is working looks exactly like a gate that is never reached** until refusals are recorded. ## Scope `requires:` guards **skill invocations that pass through the MCP permission seam**. A capability skill that is not an `mcp_tool` dispatches elsewhere and is not covered by this note — either the scope is stated narrowly, or a second enforcement point is needed. Naming the boundary matters more than widening it silently. It also needs a `(server_id, tool_name) → skill_id` mapping at dispatch, since `requires` names skills and the seam sees tools. The grant carries both, but this is the fiddliest part of the implementation. ## Non-goals - **Not parameterised in v1.** `requires: [get-build-convention(kind=$kind)]` binds an argument of the guarded call to an argument of the prerequisite call — a constraint expression language, and once `$kind` exists `$solution` follows (the ack tokens are already per-`(kind, solution)`), then conditionals. Ship unparameterised, and **design the data shape so the parameterised form is additive rather than a rewrite**. State the gap honestly: unparameterised, reading *a* console card satisfies RF work. - **Not guarding decisions.** The request draws this line correctly: "read the argument reference before writing ``" has no tool call to attach to. That is boundary validation, and `scripts/write-solution-config.py` already does it. - **Not cross-agent.** A prerequisite is about what is in *this* agent's context. ## Considered and rejected: hiding the tool until it is unlocked Do not advertise a tool until its prerequisite is satisfied — the model cannot call what it cannot see. Strictly stronger enforcement, and worse: **undiscoverable.** The agent never learns the tool exists, so it cannot know to call the prerequisite, and the refusal that teaches it is exactly the mechanism that made the ack gate work. Recorded here so it is not later mistaken for an improvement. ## Test plan - A guarded call is refused until its prerequisite has succeeded in the same `(run, agent)`, and the refusal names the prerequisite. - **The agent recovers within one loop**: prerequisite, retry, success — asserted through a real tool loop, not a unit call, because recoverability is the whole feature. - A prerequisite that raised does **not** satisfy the requirement. - A sibling agent's call does not satisfy this agent's prerequisite. - `requires: [a, b]` is satisfied in either order and not by one alone. - Enforcement is identical on the model path and through the gateway — one seam, asserted twice. - A refusal appears in the audit log as `policy_decision="deny"` naming the prerequisite, and is distinguishable from a tool that simply failed. - `swarmkit validate` fails on a `requires:` naming an ungranted skill, and on a cycle. - **A topology with no `requires:` produces a byte-identical run** — asserted against a reference workspace, not a fixture. ## Demo plan The reported failure, reproduced and then fixed: an agent granted `get-build-convention` with `requires: [list-build-conventions]` calls it first, is refused, calls the prerequisite, retries, and succeeds — with the refusal visible in `swarmkit logs` and the whole ordering rule visible in four lines of the topology. Alongside it, the same workspace with `_ack.py` deleted. ## Open questions for review 1. ~~**Shape A or B?**~~ **B.** A plain identifier array for every consumer, duplicates impossible because it is a map, and the ordering rules readable in one block. 2. ~~**Should a refusal be capped?**~~ **No, not separately.** `SWARMKIT_MAX_TOOLS` already bounds calls per turn, and a ceiling that failed the run would convert the one failure mode this feature is designed to keep recoverable into a terminal one. Revisit if a real run thrashes. 3. ~~**Grant only, or also the skill definition?**~~ **Grant only.** Ordering is a property of how a workspace works, and a skill-level default would be a second place to look for a rule that the resolver would then have to merge — the reason shape B was chosen in the first place. 4. ~~**Is MCP-only scope acceptable for v1?**~~ **Yes.** Every skill in the reporting workspace is an `mcp_tool`, and the guard sits at the one function both executors dispatch through. A non-MCP capability skill dispatches elsewhere and is **not** covered — stated in `## Scope` rather than widened silently. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/stopping-a-run.md ## Published: https://delivstat.github.io/swarmkit/design-notes/stopping-a-run/ ══════════════════════════════════════════════════════════════════════════════ # Stopping a run **Status:** implemented — runtime 1.193.0. One thing changed between design and implementation, and the **demo** is what caught it. The first version cached a "not stopped" answer for a second, reasoning that a thirty-agent run should not issue thirty round-trips. Run against a fast pipeline, the stop was not late — it was *missed* entirely, and the run completed. One indexed primary-key SELECT against a node that takes seconds to minutes is not a cost worth a feature that sometimes does nothing, so the check now asks every time. A *seen* stop still latches: the run is already raising. `swarmkit stop ` has been a `_not_implemented("stop", milestone="M6")` stub since M6, with a docstring that already promises the semantics: *"Requests the runtime to checkpoint state and abort the current run. The run can be resumed later with `swarmkit run --resume`."* This note makes that true, and says plainly what it cannot do. ## Goal Let a human stop a run they no longer want — from another terminal, from the UI, from a script — **without losing the work it has already done and paid for**, and let them resume it later if they change their mind. ## The problem it solves Today the only ways to stop a run are Ctrl-C in the terminal that started it (which requires being that terminal) and killing the process (which takes down every other run in it). A `swarmkit serve` with three concurrent jobs has no way to stop *one*. A run that is looping through a 40-tool research turn on a wrong premise burns tokens until its turn cap, and the operator watching it can only wait. ## The shape: a stop is a deferral without a gate This is the whole design, and everything else follows from it. The runtime already parks a run mid-flight and resumes it later: a funnel's `approve` layer raises `HITLDeferredError`, the graph checkpoints, the job goes `deferred`, and `swarmkit run --resume` / `POST /jobs/{id}/resume` continues it from the checkpoint ([`gate-state-and-deferring-approval.md`](gate-state-and-deferring-approval.md)). A stop wants exactly that behaviour with a different *reason*. So: ```python class RunStoppedError(HITLDeferredError): """A human asked this run to stop. Checkpointed and resumable, like any deferral.""" ``` A subclass, for the same reason `GateDeferredError` is one: every caller that already knows how to checkpoint-and-exit on a deferral handles it without changing, and the ones that want to distinguish a stop can ask. **No new resume path, no second checkpoint mechanism, no parallel "cancelled run" state machine.** The cost of getting stop wrong is a run that cannot be resumed, and the way to not get that wrong is to not write a second implementation of resumption. ## The signal: a durable flag, not a signal or an HTTP call `stop` has to reach a run in **another process**. The CLI writes `jobs` rows for its own runs (1.176.0+), serve writes them for its jobs, and one storage service resolves the store for both — so a column on `jobs` is a channel that already connects every writer to every reader: ``` jobs.stop_requested_at TEXT -- ISO timestamp, NULL = not asked ``` `swarmkit stop ` sets it. The running process reads it. That is the entire transport. Rejected alternatives: - **A signal (SIGTERM/SIGUSR1) to a PID.** Needs a pid column, needs the same machine, and cannot express "stop this one job" in a process running three. - **`POST /jobs/{id}/stop` as the only channel.** Works for serve and not at all for a CLI run, which is where an operator most often wants it. The HTTP route still exists — but as a *caller* of the same flag, not as a second mechanism. - **An in-memory cancellation token.** Correct for one process, useless across two, and the CLI and serve are always two. ## Where it is checked, and what that costs **At node entry**, in the one `node_fn` every agent node is built from — before the node does any work, after the previous node's state has been checkpointed by the super-step that produced it. That placement is what makes the promise honest: everything completed up to the last node boundary is in the checkpoint, and a resume re-enters at the node that would have run next. **A stop is therefore cooperative, and it is not instant.** A run inside a 10-minute harness session or a long MCP call stops when that call returns, not when the operator presses enter. The CLI says so rather than implying a kill: ``` $ swarmkit stop a46614b1 stop requested for a46614b1 — it will stop at the next agent boundary. Runs mid-call (a harness session, a slow tool) finish that call first. ``` Pretending otherwise would be worse than the wait: an operator who believes a run is dead and starts a replacement gets two runs writing the same artifacts. A second check inside the tool loop (between turns) is a **later refinement**, not v1. It shortens the wait for the common "40 tool calls on a wrong premise" case, but it also means a stop can land between a tool call and the model seeing its result, and that interaction deserves its own thinking rather than being smuggled into this note. ## Status vocabulary `stopped`, not `deferred` and not `failed`. - Not `failed`: nothing went wrong, and a reader counting failures should not count this. - Not `deferred`: `deferred` means *waiting on a human decision that will arrive*. A stopped run is waiting on nothing. Collapsing them would make "how many runs are blocked on approvals" — a question the review queue exists to answer — silently wrong. - Not `cancelled`: a cancelled run is over. A stopped run is **resumable**, and the word should not suggest otherwise. `swarmkit run --resume` and `POST /jobs/{id}/resume` both accept `stopped` alongside `deferred`. A `completed` or `running` job still 409s. ## The idempotence and re-stop rules - Stopping a job that is not `running` is **not an error** — it is a no-op with a clear message. An operator racing a run that just finished should not get a stack trace. - Stopping an already-stop-requested job re-reports the pending request rather than stacking. - **A resumed run clears the flag.** Otherwise a run stops, resumes, and immediately stops again on the stale request — which reads as a resume that does not work. ## Audit A stop is a human act against a governed run, so it is recorded like one: `run.stopped` with the requesting identity, the run id, and the agent boundary it took effect at. "Who stopped the release run" is exactly the kind of question the audit log exists for, and a stop that only appears as a status change cannot answer it. ## Surfaces | Surface | Behaviour | | --- | --- | | `swarmkit stop [workspace]` | Set the flag; report what will happen and when | | `POST /jobs/{job_id}/stop` | Same flag, over HTTP; 404 unknown, 409 not running | | `GET /jobs/{job_id}` | `stop_requested_at` surfaces through the merged view | The CLI resolves the store from the workspace exactly as `swarmkit logs` and `swarmkit status` do, so `swarmkit stop` works against a run started by serve on the same store — the two front doors stay peers. ## Non-goals - **Not a kill.** There is no `--force` that terminates a process or cancels an in-flight model call. A stop that could interrupt mid-call would lose the current node's work, which defeats the goal; an operator who genuinely wants that has `kill`. - **Not stopping a fleet run from the panel.** The flag is per-store; a fleet-wide stop is a control-plane feature and belongs with the fleet work. - **Not stopping a topology.** The stub's help says "gracefully stop a running topology" — but the unit that runs, checkpoints, and resumes is a **run**, and the argument has always been a run id. The help is corrected to say so. - **Not a scheduled/conditional stop** (stop after this stage, stop if cost exceeds X). Budget enforcement is a separate concern with its own policy questions. ## Test plan - A flagged run raises `RunStoppedError` at the next node boundary and does **not** run that node. - The work before the stop survives: the trace and audit events for completed agents are written, and the job's usage reflects what was spent. - A stopped run **resumes** and completes, over both the CLI and `POST /jobs/{id}/resume`. - Resuming **clears** the flag — a resumed run does not immediately re-stop. - A run with no flag is byte-identical to today (asserted against a real workspace run, not a mock). - `stopped` is distinct from `deferred` and from `failed` in the job row and every read surface. - Stopping a completed job is a no-op with a message, not an error; stopping an unknown run id is an error naming the id. - The audit log records `run.stopped` with the requester. - The flag reaches a reader: `GET /jobs/{id}` carries `stop_requested_at` (the bug-28 guard already asserts every `JobRow` field is reachable — this rides on it). ## Demo plan A two-terminal transcript: a long run started in one, `swarmkit stop ` in the other, the run stopping at the next agent boundary with its completed agents' work intact, `swarmkit logs` showing the partial trail, and `swarmkit run --resume` finishing the job. ## Open questions 0. ~~**Cache the check?**~~ **No** — see the status note. Resolved by the demo, not by argument. 1. **Should a stop request expire?** A flag set against a run that dies unrecorded stays set forever, and a resume clears it — but a run that never resumes leaves a stale row. Probably harmless; noting it rather than solving it. 2. **Should the tool loop check too?** See above — it shortens the wait materially for the case that motivates the feature, and it needs its own thinking about mid-turn state. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/funnel-verification-check.md ## Published: https://delivstat.github.io/swarmkit/design-notes/funnel-verification-check/ ══════════════════════════════════════════════════════════════════════════════ # Funnel verification strength — design Answers open question 3 of `design/details/extracting-the-pipeline.md`: what replaces `swarmkit gates --require` for CI. ## Goal Tell a workspace author, before a run, **which agents produce an output that nothing checks** — and let CI fail on it. `gate_coverage` answered "what is the narrowest verified edge of this pipeline" by classifying stage edges against their funnels. The stage graph left with the bundled sequencer, so the edge analysis went with it. The question underneath was never about pipelines: *which agents produce an artifact, and how strongly is it checked* is about topologies and funnels, both of which stay. It is the natural sibling of the reachability report — *"this run's output is verified by nothing"* is the same class of finding as *"this binding is reached by nothing"*. ## Non-goals - Not a re-implementation of stage-edge coverage. There are no stage edges to cover. - Not a run-time check. This reads the compiled workspace; it does not observe a run. - Not a quality judgement of the layers themselves — a weak judge and a strong judge both score 1. ## Shape `swarmkit_runtime/verification.py`, pure and read-only: ```python compute_verification(workspace: ResolvedWorkspace, ledger: WiringLedger) -> VerificationReport ``` Two properties carry the design. **Strength counts wired layers, not declared ones.** A funnel declaring `validate` whose builder returned nothing contributes nothing, because it does nothing — counting the declaration would make this check commit the exact defect the reachability report exists to catch. The ledger is the same one `compute_reachability` reads, from the same compile, so the two reports cannot disagree. Declared-and-unwired layers are still reported, as `inert`. **Only roots are findings.** Every agent's strength is reported, but a leaf worker returning a fact to its parent is not producing a reviewable artifact, and flagging every one would make a report nobody reads. The root's output *is* the run's output: what a caller acts on, what a gate approves, what a downstream stage would have consumed. ## Surfaces - `WorkspaceRuntime.verification()` — shares the reachability compile pass. - `swarmkit validate` — a verification section, always printed. - `swarmkit validate --require-verified` — exit 1 when any root is unverified. Kept separate from `--require` (reachability): *"is my config wired"* and *"is my output checked"* are different questions a CI job may want independently, and folding them would break existing `--require` jobs. - `GET /workspace/verification`. The gate is strict and opt-in. Narrowing it to roots declaring an `output_schema` was considered and rejected on evidence: 0 of 3 roots in the reference workspace and 0 of 15 in `sdlc-pipeline` declare one, so the signal cannot distinguish an artifact-producing root from any other. ## Test plan `packages/runtime/tests/test_verification_strength.py` — the finding itself; strength rising per wired layer; a declared-but-unwired `review` scoring nothing; an unresolvable `validate.schema` reported inert while a resolvable one counts; every agent reported but only roots as findings; the pure function driven from a hand-built ledger; serialisation; and that each surface exists. ## Demo `swarmkit validate` in `examples/sdlc-pipeline/workspace` reports 12 of 15 roots unchecked, and `--require-verified` exits 1 — with `security-review/release-gate` shown as `judge, approve; declared but inert: validate, review`, agreeing with what the reachability report independently found on the same compile. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/multi-party-approval.md ## Published: https://delivstat.github.io/swarmkit/design-notes/multi-party-approval/ ══════════════════════════════════════════════════════════════════════════════ # Multi-party approval sets (governance) Parent: `design/details/sdlc-pipeline-example.md` (capability 1 of 5). This is a standalone, reusable governance capability — the SDLC pipeline is its first consumer, but nothing here is SDLC-specific. Today a human-approval gate (design §6.2) is **one question → one resolver**. Real approvals are plural and role-based: a design needs every app lead *and* InfoSec; a release needs the engineering manager *and* the CIO. This note adds a **configurable multi-party approval set** — a gate that stays open until the roles it requires have each approved, by *distinct human identities*, per a declarative per-gate policy over a workspace role registry. ## Goal Let any gate require a configured set of approvals — expressed as **data**, not code — and have the policy engine enforce it structurally: distinct human identities, reserved (non-agent) scopes, quorum, and an append-only record of who approved what and when. ## Non-goals - **Not agent approvals.** Reserved human scopes only; no agent can hold or satisfy an approval scope, regardless of prompt (invariant 6, §8.7). This capability *strengthens* that line. - **Not sequencing.** How a rejected gate routes back, and how stages advance, belongs to `pipeline-controller`. This note only defines the gate and its resolution outcomes. - **Not the judge/review layers.** The automated pre-filters are `gate-funnel`; this is the final, binding human layer. - **Not notification/queue delivery.** This *emits* tasks; routing them to people is `task-surface-and-board`. ## Where it lives Governance only. Enforcement is in the `GovernanceProvider` / policy engine (`packages/runtime/src/swarmkit_runtime/governance/`, §8.5) — no other package decides approvals. The role registry is workspace-level IAM data; the approval policy is gate configuration. Both are validated artifacts (canonical schema), not runtime flags. ## API shape ### Role registry (new IAM artifact) A workspace-level artifact mapping each role to the human identities that hold it and the governance scopes it confers. A role carries **many scopes** (standard RBAC: identity → role → scopes) — a lead approves designs *and* code *and* deploys, and membership then lives in **one place per person**, so a handover is a single edit and cannot silently drift. Versioned like any artifact (people join/leave independently of topology). ```yaml apiVersion: swarmkit/v1 kind: RoleRegistry roles: - { id: oms-lead, members: [alice], scopes: [design:approve, code:approve, deploy:approve] } - { id: infosec-lead, members: [dana], scopes: [security:approve] } - { id: eng-manager, members: [grace], scopes: [release:approve, code:approve] } - { id: cio, members: [heidi], scopes: [release:approve] } ``` **Registry-driven reserved scopes.** Any scope conferred by any role is a *human-identity scope*: the policy engine refuses to grant it to a non-human (agent) principal — the same structural mechanism as the existing reserved scopes (`skills:activate`, `iam:modify`, …), but driven by the registry rather than a hardcoded list (no hardcoded scope names — "everything configurable"). ### Per-gate approval policy A gate declares one or more approval **rules**, each naming the **scope** being exercised, the group of roles that may exercise it, and a quorum mode: ```yaml gate: consolidated-design-approval approval: rules: - { scope: design:approve, roles: [oms-lead, web-lead, mobile-lead], quorum: all } - { scope: security:approve, roles: [infosec-lead], quorum: all } - { scope: design:approve, roles: [rev-a, rev-b, rev-c], quorum: { k-of: 2 } } exclude_author: true # segregation of duties (default true) on_revision: reset_all # reset_all | reconfirm_changed ``` - `all` — every role in the group must approve. - `any` — one role in the group suffices. - `k-of: N` — any N **distinct** role-holders in the group. The **rule** names its scope so a multi-scope role stays unambiguous (the role says what a person *can* do; the rule says what is *being asked* here), and a single gate can span multiple authorities — the example needs `design:approve` from the app leads **and** `security:approve` from InfoSec, which neither scope-per-role nor scope-per-gate could express. Validation: every role in a rule must confer that rule's scope. The gate advances only when **every** rule is satisfied. ### Task decomposition (one task per role) A gate **fans out into one task per required role** — `Approval from role:`, assigned to that role's members. A role-task completes when **any one member** of the role approves (a role is a single slot; multiple members just means anyone eligible can fill it). Quorum is counted over **completed role-tasks**: - `all` — every role-task in the rule completed. - `any` — at least one role-task completed (the rest auto-close). - `k-of: N` — any N role-tasks completed. A person who holds **two required roles gets two tasks and completes each separately** — one deliberate, attributable sign-off *per capacity* ("approved as `oms-lead`", "approved as `web-lead`"), not one click standing in for two responsibilities. This is the explicit, audit-friendly model; the small extra cost for dual-hatted people buys unambiguous accountability. Independence (four-eyes) is a **separate axis** from task completion — see "Overlapping roles". ### Resolution model Each resolution is one of three outcomes, authenticated to a human identity: - `approve` — records the approval, updates the tally. - `changes-requested` — carries free-text comments; the gate reports this outcome to the caller (the *rework loop itself* lives in the consuming stage — see `pipeline-controller` / the parent note), and prior approvals are handled per `on_revision`. - `reject` — fails the gate. `on_revision`: `reset_all` (default — a revised artifact invalidates prior approvals; all required roles re-approve) or `reconfirm_changed` (only affected roles re-review; unaffected approvals carry). Section-scoped re-approval is explicitly future work. ### Runtime: accumulation via checkpoint The gate compiles to a **LangGraph `interrupt()` backed by the checkpointer**. The interrupt payload is the set of *outstanding* required approvals. Each `Command(resume=…)` supplies one identity's outcome; the gate node updates the tally, and if quorum is unmet it **re-interrupts** with the tally persisted. So "2 of 4 approved, waiting on infosec + cio" is durable state across weeks and across people, and a parked gate costs one DB row (no held process). ### Enforcement (structural) The policy engine, on each resolution: 1. **Authenticates** the resolver to a human identity (not an agent). 2. Checks the identity is a member of a named role that **confers the rule's scope**; rejects otherwise. 3. Completes **exactly the role-task the resolution was submitted for** — a person with two role- tasks completes each separately; no single action covers multiple roles. 4. Enforces **`exclude_author`** — the identity that authored/submitted the artifact cannot approve it (segregation of duties; important for DORA/audit). 5. Refuses to advance until every rule's quorum **and** any `min_distinct_approvers` floor is met. ### Overlapping roles (one person, several hats) A person holding two required roles gets **two tasks** and completes **both, separately** — one attributable sign-off per capacity. Their two completions satisfy their two roles; that is the point, not a shortcut. So task completion is purely per-role. **Independence is a separate axis.** Because a dual-role person *can* complete two role-tasks, a gate that also needs genuine four-eyes adds a distinct-identity floor on top: - `min_distinct_approvers: N` at the gate level — at least N **different humans** must have approved across all completed role-tasks, regardless of overlap. So one dual-hatted person completing two role-tasks satisfies the roles but **not** a `min_distinct_approvers: 2` floor; a second identity is still required. Keep the two axes distinct: **which roles signed** (task completion, `all`/`any`/`k-of`) versus **how many independent people signed** (`min_distinct_approvers`). Overlap is most suspect when the roles exercise *different scopes* (the same person as both design approver and independent security sign-off) — `min_distinct_approvers`, or simply not placing one person in both roles, guards that as an explicit policy choice, never a hardcoded heuristic. ```yaml approval: rules: - { scope: design:approve, roles: [oms-lead, web-lead, mobile-lead], quorum: all } - { scope: security:approve, roles: [infosec-lead], quorum: all } min_distinct_approvers: 2 # optional four-eyes floor across the whole gate ``` None of this is promptable or agent-reachable; it is the same class of structural gate as the existing reserved scopes. ### Audit Every role-task completion appends one event: `{gate_id, correlation_id, role, identity, outcome, comment?, ts}` — so a person acting in two capacities produces **two** events, one per role, each independently attributable. Append-only from the executive perspective (§8.3). This record *is* the approval evidence (who signed off, when) that DORA/compliance reporting reads — no separate spreadsheet. ## Test plan - **Schema (Python + TS):** RoleRegistry (roles carry many scopes) and gate `approval` blocks validate; a rule referencing an unknown role is rejected; a role in a rule that does **not** confer the rule's scope is rejected; a gate spanning two scopes (design + security) validates; `k-of: N` with N > group size is rejected; a scope with no member (no role confers it) is rejected. - **Quorum modes:** `all` / `any` / `k-of: N` each advance exactly at their threshold; `k-of` counts **distinct** identities; a duplicate approval from the same identity does not double-count. - **Task decomposition:** a gate fans out into one task per required role; a role-task completes on the first approval by **any** member of that role; `all` needs every role-task, `k-of: N` needs N. - **Overlapping roles:** a person holding two required roles gets **two** tasks and must complete **both** separately to satisfy both roles (one action does not cover two); with `min_distinct_approvers: 2` the gate is still unsatisfied after that person's two completions until a second distinct identity approves. - **Reserved-scope enforcement:** an agent principal cannot be granted a registry-bound scope; an agent-authenticated resolution is refused. - **Segregation of duties:** with `exclude_author: true`, the author's approval is refused; with `false`, it is accepted. - **on_revision:** `reset_all` clears prior approvals on a revision; `reconfirm_changed` carries unaffected ones. - **Checkpoint durability:** the running tally survives a simulated process restart (resume from the persisted `thread_id` continues, not restarts, the gate). - **Audit:** each resolution appends exactly one immutable event with identity + role + outcome; no update/delete path is exposed. ## Demo plan `just demo-multi-party-approval` (script under the runtime demos): a one-node topology with a `consolidated-design-approval` gate (all three app leads + infosec, plus a `k-of: 2` reviewer pool). The script resolves via the gates API as distinct identities and shows: the gate holding at partial quorum, a duplicate approval rejected, an agent resolution refused, quorum reached → advance, and the printed append-only audit trail. Terminal transcript in the PR body. ## Schema-change checklist Adds a new artifact kind + a gate sub-schema — follow `docs/notes/schema-change-discipline.md`: canonical JSON Schema, Python validator, TS validator, and fixtures updated together; register `RoleRegistry` in the artifact registry alongside topologies/skills/archetypes/triggers. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/contract-registry.md ## Published: https://delivstat.github.io/swarmkit/design-notes/contract-registry/ ══════════════════════════════════════════════════════════════════════════════ --- title: Integration-contract registry description: Make the integration contracts a delivery flow locks on first-class artifacts, so lock ids are a checked, pickable vocabulary (not free strings) and the contention view is exact. tags: [schema, contracts, locking] status: implemented --- # Integration-contract registry !!! note "The registry stayed; the lock manager left" This note was written while SwarmKit still bundled a pipeline sequencer, so it describes the `StageGraph` lock ref-check. The stage graph was removed in runtime 1.189.0 ([Extracting the pipeline](extracting-the-pipeline.md)) and that ref-check went with it. The **registry itself is unchanged and still shipped**: contracts resolve into `ResolvedWorkspace.contracts`, so the sequencer in *your* application can check a lock id against a real artifact. Read the StageGraph passages below as the original motivation. **Scope:** schema (new `Contract` artifact) + runtime (discovery/resolution + StageGraph lock ref-check) + serve + composer + docs. **Design references:** [`extracting-the-pipeline.md`](extracting-the-pipeline.md) (where the lock manager went) · the retired pipeline-controller note (integration-contract locking), [`pipeline-editor-canvas.md`](https://github.com/delivstat/swarmkit/blob/main/design/details/pipeline-editor-canvas.md) (this is its "contract-lock registry" open question, resolved). **Status:** proposed. ## Why A pipeline serialises requirements on the **integration contracts** they share — a stage's `locks: [contract:oms-web, contract:oms-inventory]` mean "hold the OMS↔Web and OMS↔Inventory interfaces while I change them, so no concurrent requirement commits a conflicting version." Today those lock ids are **free-form strings**: nothing checks them, a typo (`contract:oms-web`) silently becomes a *different* lock (so two requirements that should serialise don't), the editor can only offer a free-text chip, and the contention overlay ("which stages fight over the same contract") is approximate. The contract itself — the agreed interface between two apps — has no home. Make each integration contract a **first-class artifact**. Then lock ids are a **checked, pickable vocabulary**: the resolver rejects a lock that names no contract, the editor offers a picker over real contracts, and contention is exact. ## The `Contract` artifact A contract is the agreed interface between two (or more) applications, identified by id. ```yaml apiVersion: swarmkit/v1 kind: Contract metadata: id: oms-web name: OMS ↔ Web order API description: The order-submission + status API OMS exposes to the Web storefront. parties: [oms, web] # the apps this contract binds (>= 2) interface: schemas/oms-web-order.json # optional: where the interface itself lives provenance: authored_by: human version: 1.0.0 ``` - **`parties`** (required, ≥2) — the apps the contract is between. This is what makes it a contract (an interface between apps), and it drives the editor's contention/ownership display. App ids are free strings (apps are not artifacts). - **`interface`** (optional) — a pointer to the actual interface spec (an API/event schema). Not interpreted by core; documentation + a handle for reviewers. ## What changes in the StageGraph A stage's `locks` items become **contract references**: ```json "locks": { "type": "array", "items": { "$ref": "#/$defs/identifier", "x-swarmkit-ref": "contract" } } ``` - The resolver ref-checks each lock against the contract registry — an unknown contract is a `stage-graph.unknown-contract` resolution error (consistent with how `topology`/`gate` refs are checked). In this model a lock **is** an integration contract; the free-string form was the placeholder. `release_locks_on` is unchanged (it's an event, not a contract). - **Migration:** the contract ids drop the redundant `contract:` prefix — the field is `locks`, so the id is just `oms-web`. The OMS example gains `contracts/oms-web.yaml` + `oms-inventory.yaml` and its design stage becomes `locks: [oms-web, oms-inventory]`. ## Runtime - `contract` joins the discoverable artifact kinds (`contracts/` directory), resolved into a `ResolvedWorkspace.contracts` registry (id → `ResolvedContract`), like funnels/roles/stage-graphs. - StageGraph resolution gains the lock ref-check against that registry. - The contract itself is not *executed* — the controller/orchestrator is still the lock manager; the registry only makes the vocabulary real. A contract's `parties` let the manager (and the board) group locks by app-pair. ## Composer + serve - Serve CRUD: `/contracts`, `/api/contracts/{id}` (mirrors funnels/pipelines). - `use-ref-options` fetches contracts; the StageGraph editor's `locks` field renders as a **RefChips picker over workspace contracts** (the existing `x-swarmkit-ref` array machinery — no new UI mechanism), replacing the free-entry chips. - A **Contracts** artifact surface (list + schema form), like funnels. - **Exact contention overlay:** the pipeline canvas highlights stages that hold the *same* contract id (now guaranteed to be the same real contract), and can label a contract by its `parties`. ## Docs / authoring Contract joins the artifact-kind enumerations: `llms.txt` (a Contract section), a reference page, the authorable-kinds list, and the knowledge-server write-path (`contracts/`). The knowledge server auto-globs schemas, so `get_schema("contract")` surfaces for free. ## Non-goals - **Not contract *content* validation.** Core does not parse or diff the `interface` spec — that is the SIT/contract-testing stage's job (a later slice). The registry governs *identity + locking*, not interface compatibility. - **Not app artifacts.** `parties` are free strings; apps do not become a kind here. - **Not a new lock manager.** Locking stays in the orchestrator (reference controller / Temporal adapter); this only makes lock ids checkable and pickable. ## Test plan - **Schema (Py + TS):** a valid contract parses; missing `parties`/`<2 parties` is rejected; the new `contract` schema round-trips its fixtures. - **Resolution:** contracts discovered into `ResolvedWorkspace.contracts`; a StageGraph lock naming a real contract resolves; an unknown lock is `stage-graph.unknown-contract`; the OMS workspace resolves with its two contracts. - **UI:** `locks` renders as a contract RefChips picker; the contention overlay groups stages by shared real contract. ## Demo plan Extend `just demo-pipeline-controller` / the pipelines UI: two OMS requirements both locking `oms-web` serialise (already demoed) — now the lock is a *checked* contract, a typo'd lock fails resolution loudly, and the editor's `locks` field is a dropdown of the workspace's contracts. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/executor-abstraction.md ## Published: https://delivstat.github.io/swarmkit/design-notes/executor-abstraction/ ══════════════════════════════════════════════════════════════════════════════ --- status: draft --- # Feature Request: Pluggable Executor Abstraction at the Archetype Level **Type:** Feature / Architecture RFC **Component:** Archetype schema, runtime (LangGraph node execution), AGT governance, cost accounting, observability (OTel/audit) **Status:** Proposed — all design questions resolved (§11); ready for implementation review **Priority:** High — converts SwarmKit's largest quality gap (raw model calls vs. engineered coding harnesses) into a pass-through --- ## 0. Architectural placement — a recognized provider seam (invariant note) This RFC introduces `executor`, a new extension mechanism (adapters), which sits against CLAUDE.md **invariant #2 — "Skills are the only extension primitive."** That boundary is drawn deliberately here, not left to drift. **Resolution: `executor` is a node-execution *provider seam*, not a capability primitive.** It is the same class of abstraction as `ModelProvider` (invariant #4 — "all LLM calls go through `ModelProvider`"), `GovernanceProvider` (invariant #3), and the audit provider: a narrow, swapped-at-startup interface the runtime depends on. `model` becomes *one* executor kind and `harness` another (§4.2, P1) — so this **generalizes the existing `ModelProvider` seam** rather than adding a parallel one. An executor answers *how a node does its work*; a skill answers *what capability an agent may invoke*. Different layers. The skill/executor line stays crisp by construction (§9): a bounded, stateless consult that answers and returns is tool-shaped → an MCP skill; a blackbox that *pursues* (multi-turn, holds a session, produces a diff) → an executor. **If it produces a diff or holds a session, it is an executor; if it answers a question and returns, it may be a tool.** **Action on acceptance:** promote `executor` to a listed provider abstraction alongside Model / Governance / Audit (design §7 principles, §9), and amend invariant #2 to read *"Skills are the only **capability** extension primitive; node execution is the Model/executor seam."* Do this explicitly — as §15.3's UI-deferral was consciously revisited — so the invariant and this feature never read as contradictory. **Cockpit:** the live topology canvas (`topology-canvas.md`) is the visual seat for this feature's mid-run interaction (§6.2–6.3) — `exec.approval_requested` / `exec.input_requested` surface on the graph node and are answered inline. The two features are designed together. --- ## 1. Summary Introduce a first-class `executor` block in the archetype schema that declares *how* a node's work is performed, decoupled from *what* the node does. The initial implementation ships two executor kinds: - `model` — the current behavior: a direct chat-completion call to a configured model (OpenRouter or otherwise). Remains the default; existing archetypes are unaffected. - `harness` — delegation of the node's task to an external agentic harness running as a sandboxed subprocess, communicating over a normalized JSONL event contract. Coding harnesses (Claude Code, Codex) are the first targets, but the contract is domain-agnostic: any goal-pursuing external agent — audio production, route planning, document assembly, whatever ships next — plugs in through the same interface. The core contract is **task spec in → normalized events during → typed artifacts out**; "a diff in a git worktree" is one artifact profile, not the definition. The executor interface must be defined as an open plugin contract so that future executor kinds (new harnesses, remote agent services, or executor types that do not exist yet) can be added by implementing an adapter — with **no changes to the archetype schema, the runtime, governance, or the cockpit**. Adapters come in two tiers (§5.2): shipped **code adapters** for complex integrations, and **declarative adapter definitions** — YAML workspace artifacts — for the common subprocess-plus-JSONL shape, so new harnesses can be integrated without a core release. ## 2. Motivation SwarmKit workers currently execute via raw model calls inside LangGraph nodes. For real coding tasks, this competes with tools like Claude Code and Codex — which are not merely models but years of harness engineering (agentic edit–verify–retry loops, codebase-scale context management, file-editing reliability, test-running reflexes, error recovery). A raw frontier model in a chat loop does not match a mid-tier model inside a mature harness for this class of work. Rebuilding that scaffolding inside SwarmKit is a multi-year mistake. The correct move is to let a leaf node *delegate* to a harness while SwarmKit retains what it is actually for: declared topology, structural gates, durable state, cost tiering, and the compounding artifact workspace. Secondary motivations: - **Per-slice routing.** The planner/topology can declare, per node, whether a slice warrants a harness (rate-limiter core → Claude Code) or a cheap model (config change → small OpenRouter model). This decision becomes declared, diffable, and reviewable rather than buried in prompts. - **Ecosystem velocity.** The harness landscape is moving fast (Claude Code, Codex, Gemini CLI, OpenCode, Pi, Aider, and whatever ships next quarter). SwarmKit must be able to adopt a new harness by writing one adapter, not by redesigning the runtime. - **Governance honesty.** Harness delegation must be *visible in the topology* so AGT can gate, budget, and audit it as a node — not smuggled through worker prompts or tool calls. ## 3. Non-Goals - Reimplementing agentic coding loops inside SwarmKit. - Modeling long-running harness delegation as an MCP tool available to arbitrary archetypes. (Delegation is an executor concern; see §8 for the narrow tool-shaped carve-out.) - In-process/SDK embedding of vendor harnesses in v1. The initial contract is subprocess + JSONL; SDK embedding (e.g., Claude Agent SDK `can_use_tool` interception) is a possible v2 enhancement behind the same interface. - Governing the harness's internal reasoning. Governance is applied at the boundary (sandbox, capability grants, budget, output gates), not inside the loop. ## 4. Proposed Schema Change ### 4.1 Archetype `executor` block ```yaml # archetype: coding-worker name: coding-worker role: > Implements a bounded code change described by a task spec, in an isolated worktree, and returns a diff. executor: kind: harness # enum: model | harness | ref: claude-code # adapter id from the executor registry version_constraint: ">=2.1" # optional; adapter interprets config: # opaque to core; validated by the adapter's own schema permission_mode: bare allowed_tools: [Read, Edit, Bash, Grep] output_schema: schemas/diff-result.json sandbox: type: worktree # worktree | container | tempdir | none network: deny # deny | allowlist | full credentials: proxy-injected # non-model secrets never in env (see §7 two-class rule; # the model-provider credential alone is env-injected per-run) artifacts: profile: diff # diff | files | structured — what the node is expected to emit schema: schemas/diff-result.json # required for structured; optional otherwise budget: max_cost_usd: 5.00 max_turns: 40 max_wall_clock_minutes: 30 interaction: on_unanswerable: relay # deny | abort | relay — what happens when the harness # requests something outside its grants (see §6.2) input_escalation: [lead, operator] # who answers exec.input_requested, in order (see §6.3) human_required_patterns: # question classes that always skip model escalation - naming - external-facing telemetry: stream: true # emit normalized events live retain_raw: true # keep the raw vendor event log alongside normalized events ``` Note on `config.allowed_tools`: the tool list is a **closed set with deny-all-else semantics by construction** — an invariant, not an option. Anything not listed is denied at launch (mapped to the vendor's pre-approval mechanism, e.g. `--allowedTools` / sandbox level). Runtime relaying (§6.1) never widens the effective grant for the current run beyond a single approved action; permanent widening happens only by amending this list through a reviewed changeset. ```yaml # archetype: analysis-worker (unchanged behavior, now explicit) name: analysis-worker executor: kind: model ref: openrouter/deepseek-v4 config: temperature: 0.2 ``` ### 4.2 Rules 1. `executor` is **optional**; absence means `kind: model` with the archetype's existing model configuration. Full backward compatibility — no existing workspace artifact changes meaning. 2. `executor.kind` values are **not a closed enum in core**. Core validates that `kind` matches a registered executor plugin; the plugin supplies the JSON Schema used to validate its own `config` block. Adding a new executor kind must require zero core schema changes. The canonical harness shape is **`kind: harness` + `ref: `** — the adapter is selected by `ref` (so a topology can swap `claude-code`→`codex` by overriding `ref`, per rule 4, without changing `kind`). For back-compat the runtime also accepts a registered adapter id **directly as the kind** (`kind: claude-code`); both resolve to the same declarative adapter. 3. `executor.config` is opaque to core and owned by the adapter. Core owns and enforces `sandbox`, `budget`, and `telemetry` uniformly across all kinds. 4. A topology may override archetype executor fields per node instance (e.g., tighten a budget), but may not change `kind` — swapping delegation semantics is an archetype-level decision and must be reviewed as one. ## 5. Executor Plugin Interface (the extensibility contract) Each executor is an adapter implementing: ```python class Executor(Protocol): kind: str # registry key, e.g. "harness" ref: str # adapter id, e.g. "claude-code" def config_schema(self) -> JSONSchema: ... def preflight(self, node_ctx) -> PreflightReport: """Binary present? version ok? credentials resolvable via proxy? Sandbox provisionable? Fail fast before any spend.""" async def run(self, task: TaskSpec, env: SandboxHandle, budget: BudgetEnvelope) -> AsyncIterator[ExecEvent]: """Launch, translate the vendor's native event stream into normalized ExecEvents, enforce nothing itself — budget/sandbox enforcement hooks are supplied by core.""" async def cancel(self, run_id) -> None: ... def resume_token(self, run_id) -> ResumeToken | None: """Vendor session id if the harness supports resume; enables checkpoint/restore across AGT gates and retry loops.""" ``` ### 5.1 Normalized event schema (`ExecEvent`) All adapters translate their vendor's native stream into this vocabulary. The cockpit, cost meter, AGT audit log, and checkpoint store consume **only** these: | Event | Payload (minimum) | |---|---| | `exec.started` | run_id, executor kind/ref, resolved config hash | | `exec.message` | role, text (assistant/user/system messages, thought summaries where the vendor exposes them) | | `exec.tool_call` | tool name, input summary, status | | `exec.artifact` | artifact kind (file_change, media, structured), path or ref, mime/type metadata | | `exec.usage` | units-typed consumption: tokens (input/output/cached/reasoning), or vendor-native units (characters, credits, requests) with a declared unit; cost_usd (nullable) | | `exec.approval_requested` | requested capability (tool/pattern), harness rationale if available, run_id — a *permission* question: "may I?" (see §6.2) | | `exec.approval_response` | granted/denied, responder (policy \| operator), scope (this-action-only) | | `exec.input_requested` | question text, structured options[] with harness trade-off notes where available, free_text_allowed, question class — a *judgment* question: "what do you want?" (see §6.3) | | `exec.input_response` | answer, responder (lead \| operator \| memoized), injected-at timestamp | | `exec.result` | status (success/failure/budget_exceeded/cancelled/needs_approval/stalled), typed output (per `output_schema` if set), artifact manifest matching the declared `artifacts.profile`, exit metadata | | `exec.raw` | passthrough of the untranslated vendor line (retained when `telemetry.retain_raw`) | Reference mappings, to prove the schema against real vendors: - **Claude Code**: `claude -p --output-format stream-json --verbose` → events per message/tool call/status; final result carries `cost_usd`, `session_id`, and full token usage. `--max-budget-usd` / `--max-turns` map to the budget envelope natively; `--json-schema` maps to `output_schema`; session id maps to `resume_token`. - **Codex**: `codex exec --json` JSONL → `thread.started`, `turn.*`, `item.*` (commands, file changes, MCP calls, reasoning items); `turn.completed.usage` includes `reasoning_output_tokens`; `codex exec resume ` maps to `resume_token`; `--output-schema` maps to `output_schema`; sandbox levels map to the sandbox contract. An adapter for a harness that emits **no** structured telemetry may still be written, but must declare `telemetry_grade: opaque`; core then requires an explicit topology-level acknowledgment to use it, and the cockpit displays it as unobservable. (Policy: SwarmKit prefers harnesses that narrate their pursuit in JSON.) ### 5.2 Two-tier adapter model Every harness speaks its own event dialect (Claude Code: message/tool-call/status events; Codex: `thread.*`/`turn.*`/`item.*`). Adapters therefore come in two tiers: **Tier 1 — code adapters (shipped, trusted).** Reference adapters (`claude-code`, `codex`) ship with SwarmKit; third-party code adapters install via a plugin entry-point and are human-vetted. Code adapters handle the hard cases: resume tokens, stateful stderr parsing, non-JSONL streams, vendor-specific budget flags. **Tier 2 — declarative adapter definitions (workspace artifacts).** Most CLI harnesses share one shape: spawn a subprocess from a command template, read line-delimited JSON, map fields into `ExecEvent`s. That is configuration, not code. A workspace may therefore contain an `executor-adapter` artifact: ```yaml kind: executor-adapter name: gemini-cli domain: coding # informational; any string (coding, audio, geo, docs, ...) launch: command: ["gemini", "-p", "{task}", "--output-format", "jsonl"] workdir: "{sandbox.root}" env_passthrough: [] # explicit allowlist; empty by default stream: format: jsonl # jsonl | json | lines event_map: - match: {type: "tool_use"} emit: exec.tool_call fields: {tool: "$.name", input: "$.input"} - match: {type: "usage"} emit: exec.usage fields: {input_tokens: "$.in", output_tokens: "$.out", unit: "tokens"} - match: {type: "output_file"} emit: exec.artifact fields: {kind: "file", path: "$.path"} result: success_when: {exit_code: 0} output: "$.final_message" artifacts_from: "{sandbox.root}/out" # manifest scan, per artifacts.profile ``` Because a declarative adapter is data, it inherits the full workspace lifecycle: schema validation, pending-review, AGT gating, changesets, versioning — and it is inspectable: a reviewer sees exactly what command runs and what is extracted, without reading code. This is also the community scaling path: when a new harness ships (an audio-production harness, a route-planning harness), the integration is a shareable `adapter.yaml`, not a core PR. Tier-2 constraints, by construction: 1. **The launch template is the sharpest edge** (it is a command line). Declarative adapters carry a mandatory human-review gate on first approval and on any change to `launch`, regardless of the workspace's auto-run trust settings. Command arrays are templated with a closed set of substitution variables (`{task}`, `{sandbox.root}`, budget fields); no shell interpolation. 2. **No event invention.** The `event_map` can only translate what the vendor emits. A harness with no structured output can only ever yield a `telemetry_grade: opaque` adapter — honest by construction. 3. **Graduation path.** A declarative adapter that hits its ceiling (resume logic, bidirectional streaming, non-line-oriented output) declares `requires: code` and graduates to Tier 1. In particular, **mid-run interaction (§6.2 relay, §6.3 input requests) is Tier-1-only**: it requires bidirectional session control and question-detection heuristics. Tier-2 adapters are limited to `on_unanswerable: deny | abort`. 3a. **Exit codes are necessary, not sufficient.** `success_when: {exit_code: 0}` alone is naive — a harness can exit 0 having refused or under-delivered. Core layers the semantic check (§6.1: typed output present, artifact manifest matches `artifacts.profile`) on top of any Tier-2 `success_when`, for all adapters. 4. **Executor vs. tool discipline still applies.** A one-shot request/response service (plain text-to-speech call, single geocode) is tool-shaped and belongs in an MCP skill, not an executor adapter. Executor adapters are for blackboxes that *pursue* — multi-turn, goal-directed, self-checking runs — in any domain. ## 6. Runtime Integration ### 6.0 Task spec and context injection (what goes IN) The `TaskSpec` handed to `run()` is a first-class, checkpointed artifact — not a bare prompt string. It comprises: - **Task statement** — the slice, its acceptance criteria, and explicit instructions to proceed on reasonable assumptions rather than ask (the headless prompt discipline). - **Pre-answered decisions** — memoized `exec.input_response` answers from prior rounds (§6.2) and any decisions the plan already made, injected so the harness never re-asks. - **Workspace context file** — the adapter materializes workspace conventions into the harness's native context mechanism (`CLAUDE.md` for Claude Code, `AGENTS.md` for Codex/OpenCode) inside the sandbox. This is the bridge between workspace skills/conventions and the harness's own context system, generated per-run from declared sources — never inherited from whatever is on the host. - **Tool mounting** — workspace MCP skills granted to the archetype may be mounted into the harness via its native MCP configuration; the harness's `allowed_tools` grant and the MCP mount list are the same reviewed capability surface, expressed in two vendor mechanisms. - **Base state** — for `worktree` sandboxes: the base ref/branch the worktree is created from. **Ownership rule:** the executor node produces artifacts (a diff) but never integrates them; applying/merging a diff is the job of a downstream integrator node, after gates. Parallel harness nodes therefore never contend on the working tree. - **Isolation from host config** — harnesses are launched in their vendor's clean/deterministic mode (e.g., `--bare`, `--ignore-user-config`-style flags) so runs are reproducible and do not absorb the host user's personal settings, hooks, or credentials. ### 6.1 Node lifecycle - A node whose archetype declares `kind: harness` compiles to a LangGraph node that: provisions the sandbox (worktree/container), runs `preflight`, launches the adapter, streams `ExecEvent`s into graph state and the event bus, enforces the budget envelope (hard-kill on breach), collects the result artifact (diff + logs), tears down the sandbox, and passes the result to the next node/gate. - **Checkpointing:** the node checkpoints (a) before launch and (b) at `exec.result`. If the adapter supplies a `resume_token`, mid-run interruption (AGT gate, crash, host restart) may resume the vendor session; otherwise the run restarts from the task spec. Long-running harness work must not break graph durability. - **Liveness, distinct from wall-clock:** `max_wall_clock_minutes` bounds total duration, but a hung subprocess emitting no events is a separate failure mode. The runtime enforces an idle timeout (`max_idle_seconds`, default sane) — no `ExecEvent` within the window ⇒ probe, then kill with `exec.result{status: stalled}`. A harness must narrate or die. - **Retry-with-feedback:** when a downstream gate rejects the artifact (reviewer comments, failed tests), the retry prefers `resume_token` continuation — feedback injected into the *same* vendor session, preserving the harness's context of its own work — falling back to a fresh run with feedback appended to the task spec when no resume is available or the session is poisoned. Retry count and per-retry budget are part of the envelope, not unbounded. - **Semantic status over exit codes:** a harness can exit 0 having *refused* or silently under-delivered — exit codes cannot express refusal. The node derives success from the structured result (typed output present, artifact manifest matches the declared `artifacts.profile`), never from the exit code alone. An empty diff where a diff was declared is a failure, not a success. - **Vendor session residue:** harnesses persist transcripts/session files on disk by default (vendor session stores, rollout files). The adapter contract requires runs to be ephemeral where the vendor supports it, or explicit cleanup of session residue at teardown otherwise; `exec.raw` retention in SwarmKit's own audit store is the sanctioned copy. This keeps run data residency in the workspace, not scattered across host dotfiles. - **Concurrency:** harness nodes are expensive; the runtime honors a per-topology and per-workspace concurrency cap for `kind: harness` nodes, separate from model-node concurrency. ### 6.2 Mid-run permission requests ("may I?") In a headless pipe, an unanswered interactive prompt is a hang (the known failure mode across harnesses). The design therefore treats permissions in three layers: 1. **Pre-answered at design time (the baseline).** The archetype's closed `allowed_tools` set and sandbox level are mapped to the vendor's pre-approval mechanism at launch. Most permission questions should never occur at runtime because the reviewed archetype already answered them. 2. **`on_unanswerable` policy** governs anything outside the grant: - `deny` — the action is refused in-place; the harness continues or fails on its own logic. Default for mature archetypes. - `abort` — the run terminates with `exec.result{status: needs_approval}`; budget released; graph takes the failure edge. Safe default for Tier-2 adapters and untrusted archetypes. - `relay` — the adapter emits `exec.approval_requested`; the node **interrupts and checkpoints** (session held open or parked via resume token); the request lands in the cockpit approval inbox as an AGT gate; on response, `exec.approval_response` is fed back (SDK permission callback or session resume) and the run continues. An approval is scoped to that single action — it does not widen the run's grant. 3. **Trust accrual → changeset promotion.** Every relayed approval is recorded against the `(archetype, capability-pattern)` pair. When approvals for a pattern cross a threshold with no denials, the system does not silently widen anything — it **proposes a changeset** amending the archetype's `allowed_tools` ("coding-worker requested `Bash(npm test)` 7 times, approved every time — add to allowlist?"). The operator approves; future runs never ask. Runtime relaying is the evidence-gathering phase; the allowlist is where trust is permanently and reviewably recorded. A mature archetype then flips to `on_unanswerable: deny`. ### 6.3 Mid-run input requests ("what do you want?") Distinct from permissions: an input request is a **domain-judgment question** — e.g., the harness identifies three viable implementations for a new endpoint and asks which to use, or needs a name for a new config key. These are routed as work, not as policy: 1. The adapter (Tier 1) detects the question and emits `exec.input_requested` with a structured payload: question text, enumerated options with the harness's trade-off notes where available, and whether free text is acceptable. **Detection is a shared core classifier, not per-adapter regex** (decision 7): cheap pre-filters (final turn took no action / expected artifact absent) gate a small structured-output LLM call that identifies the question and extracts options — language-agnostic by construction, since harnesses respond in the task's language and punctuation heuristics fail outside English. Misfires are possible and resolve harmlessly as a no-op answer or operator dismiss; adapters prefer a native vendor question-event where one exists. 2. The node interrupts and checkpoints; the question becomes durable graph state (answerable days later, like any gate). 3. The question routes up the archetype's `input_escalation` chain. First stop is typically the worker's **lead node** — a capable model holding the topology context, approved design, and workspace conventions — which answers within a small token budget when the approved design already implies the answer (implementation-choice questions often qualify). The answer is injected into the harness session (streaming input / session resume) and the run continues without human interruption. 4. Questions matching `human_required_patterns` (e.g., naming of external-facing things), or declined by the lead, land in the cockpit inbox as a human gate. 5. Every question–answer pair is recorded (`exec.input_response`) into graph state and the run record, enabling: - **Memoization:** on topology re-runs and retry loops, previously answered questions are pre-injected into the task spec so the harness never asks twice. - **Spec-quality feedback:** recurring question classes are mined across runs as a signal of missing workspace conventions — the durable fix for repeated config-naming questions is a naming-conventions skill or a planner that pre-answers the decision in the task spec, not faster answering. Questions are the workspace revealing what it hasn't yet learned. ## 7. Governance (AGT) Integration - Delegation to a harness is a **node-visible act**: AGT sees "archetype X delegated task Y to harness Z with capability set C and budget B," not a generic tool call. - Gateable points: (1) topology approval — a topology introducing a harness node with elevated sandbox/network is a distinct reviewable fact; (2) pre-launch — optional AGT gate before any harness run in sensitive workspaces; (3) post-result — the diff never merges on the harness's say-so; it passes the declared downstream gates (review node, tests, human approval). - Credentials follow a two-class rule: the **model-provider credential** (the one exception — the harness must reach its LLM API) is runtime-injected into the launch environment from the workspace credential store, scoped per-run and scrubbed from all persistence; **all other secrets** (VCS tokens, cloud credentials) are never placed in the executor's environment and are injected only at the egress proxy on approved requests. - **Nested delegation is inside the boundary.** Harnesses spawn their own internal subagents (Claude Code subagents, Codex sub-tasks); these are invisible to the topology by design and governed only through the boundary: they inherit the node's sandbox, count against the node's budget envelope, and their effects surface only in the node's artifacts. No topology-level visibility inside the harness is promised or attempted. - **Untrusted content flows through the harness.** A coding harness reads repository files, which may contain adversarial instructions (prompt injection via source/comments/docs). The defense is the same boundary model, stated explicitly: injected instructions cannot exceed the sandbox, the capability grant, the network policy, or the budget — and the artifact still faces every downstream gate. A compromised run wastes an envelope; it does not gain reach. - All `ExecEvent`s land in the AGT audit log; `exec.raw` retention gives a forensic trail without making the normalized layer vendor-specific. ## 8. Observability: Tracing, Logging, and Audit Three distinct consumers read executor activity, and they must not be conflated: **audit** (governance/compliance — did approvals happen, who answered, what merged), **tracing** (performance/debugging — where did latency and spend go), and **logging** (raw diagnostics — what did the harness actually emit). The `ExecEvent` stream is the single source; each consumer projects from it. ### 8.1 OpenTelemetry trace model A harness run maps naturally onto OTel spans, and the key requirement is **context propagation across the process boundary** so a harness run is not an opaque leaf in the trace. - **Span hierarchy:** the topology run is the root span; each node is a child span; a `kind: harness` node opens an `executor` span at launch and closes it at `exec.result`. Within it, the adapter opens child spans per meaningful unit derived from the event stream — `exec.tool_call` → a tool span, `exec.approval_requested`→`response` → a gate span (whose duration legitimately includes days of human wait, flagged so it doesn't pollute latency percentiles), `exec.input_requested`→`response` → an escalation span. - **Cross-boundary propagation:** the launch injects W3C trace context (`traceparent`) into the harness environment so that harnesses which emit their own OTel (or whose events carry correlation ids) nest under the SwarmKit span rather than starting a detached trace. Where the harness emits nothing traceable, the adapter synthesizes child spans from the normalized events — so the trace is complete regardless of vendor instrumentation maturity. - **Nested subagents (§7):** a harness's internal subagents are *not* promised as individual spans (they're inside the boundary). They appear only insofar as they surface as `exec.*` events. This is stated so trace gaps under a harness span are understood as by-design, not missing instrumentation. ### 8.2 Standard span attributes Every executor span carries a normalized attribute set so dashboards work identically across vendors and across `model` vs. `harness` kinds. Following OTel GenAI semantic conventions where they exist, plus SwarmKit-specific keys: - `swarmkit.workspace`, `swarmkit.topology`, `swarmkit.run_id`, `swarmkit.node_id`, `swarmkit.archetype` - `executor.kind` (`model`|`harness`), `executor.ref` (`claude-code`), `executor.model` (per open question 9 — the field that makes model-level trace queries work across both kinds) - `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, plus SwarmKit's unit-typed extension for non-token harnesses (`executor.usage.unit`, `executor.usage.amount`) so an audio/geo harness meters in its own units - `executor.cost_usd`, `executor.result.status`, `executor.sandbox.type`, `executor.budget.max_cost_usd` Cardinality discipline: task text, file contents, diffs, and full messages are **not** span attributes — they are span *events* or log records referenced by id, keeping the trace backend lean. ### 8.3 Audit log (the governance projection) The audit log is a separate, tamper-evident, retained projection — not the trace backend, which is sampled and short-lived. It records only governance-relevant facts, each stamped with responder identity and timestamp: harness delegation (archetype, capability set, budget), every `approval_requested`/`response` with responder, every `input_requested`/`response` with responder (`lead`|`operator`|`memoized`), gate outcomes, artifact acceptance/merge, and trust-accrual changeset proposals and approvals. This is the record that answers "prove this change was approved before it merged" — and it is exactly where a an external validation service attestation attaches, since it is the boundary-crossing record. ### 8.4 Log records and raw retention - **Normalized log stream:** each `ExecEvent` is emitted as a structured log record (JSON), correlated to its span via trace/span id, at appropriate severity (`tool_call` info, `stalled`/`budget_exceeded` error). This is the queryable operational log. - **Raw vendor retention:** `exec.raw` preserves the untranslated vendor JSONL, stored against the run (not the trace backend), retained per workspace policy. This is the forensic ground truth when a normalized event is ever disputed or an adapter mapping is suspected — the "tee the raw stream before parsing" discipline, made a contract. - **Redaction:** logs and raw retention pass through the same egress/secret policy as the sandbox — proxy-injected credentials and secret patterns are scrubbed before persistence, so the observability layer never becomes the leak the sandbox prevented. ### 8.5 Configuration An `observability` block (workspace-level default, archetype-override) declares: OTel exporter endpoint/headers, trace sample rate (audit log is **never** sampled — sampling applies to traces/logs only), raw-retention TTL, and redaction rule set. Absence ⇒ audit log on (required), traces/logs off — observability defaults safe, not silent on governance. ## 9. Related but Out-of-Scope Carve-Out: Harness-as-Oracle Tool A bounded, stateless, synchronous consult ("review this 40-line snippet and return comments") is tool-shaped, not delegation-shaped. A separate `consult_harness` MCP skill MAY exist in the workspace, access-controlled per archetype like any other tool. The dividing rule, to be documented in both features: **if it produces a diff or holds a session, it is an executor; if it answers a question and returns, it may be a tool.** This feature request covers only the executor side. ## 10. Acceptance Criteria 1. Existing workspaces load and run unmodified (`executor` absent ⇒ current model behavior). 2. `claude-code` and `codex` adapters ship as reference implementations; a demo topology routes one slice to each and one to a small model, with per-node cost visible from `exec.usage` events. 3. A third adapter (suggested: `opencode` or `gemini-cli`) is implemented as a **Tier-2 declarative `adapter.yaml`** following a contributor-facing guide, without touching core — this is the proof of both the plugin contract and the declarative schema. 3a. A non-coding harness adapter (real or stubbed — e.g., a mock audio-production harness emitting `exec.artifact{kind: media}`) runs end-to-end with `artifacts.profile: files`, proving the contract is not coding-specific. 4. Budget breach on a harness node hard-terminates the run, emits `exec.result{status: budget_exceeded}`, and the graph proceeds to its failure edge. 5. A graph interrupted at an AGT gate mid-topology resumes days later; harness nodes with resume tokens continue their vendor session, others restart cleanly. 6. AGT audit log distinguishes harness delegation events from model calls and from tool calls. 7. Schema validation rejects: unknown `kind` with no registered plugin; `config` failing the adapter's schema; harness archetypes missing a `sandbox` block. 8. A harness action outside `allowed_tools` under `on_unanswerable: deny|abort` never hangs the node — the run refuses or terminates deterministically (regression test against the known headless-hang failure mode). 9. Under `on_unanswerable: relay`, an out-of-grant request interrupts the node, appears in the approval inbox, and — after a response delivered hours later — the run resumes and completes; the approval is scoped to the single action. 10. After N approvals of the same capability pattern, a changeset proposing the allowlist amendment is generated for operator review; no grant widens without it. 11. An `exec.input_requested` with enumerated options is answered by the lead node within its token budget and injected back without human involvement; a question matching `human_required_patterns` bypasses the lead and lands in the inbox. On topology re-run, the memoized answer is pre-injected and the question does not recur. 12. A harness subprocess that stops emitting events is killed at the idle timeout with `exec.result{status: stalled}`; a run exiting 0 with an artifact manifest that does not match the declared `artifacts.profile` is recorded as failure, not success. 13. A gate-rejected artifact triggers a retry that resumes the vendor session with reviewer feedback injected (where a resume token exists); retries respect their declared count and per-retry budget. 14. After teardown, no vendor session residue remains outside the SwarmKit audit store (ephemeral mode or verified cleanup); two parallel harness nodes on the same repo produce artifacts without working-tree contention, and integration occurs only in the downstream integrator node. 15. A harness run produces a single OTel `executor` span nested under its node and topology spans, carrying the standard attribute set (including `executor.kind`/`ref`/`model` and unit-typed usage); tool calls, approval gates, and input escalations appear as child spans, with human-wait duration flagged. `model` and `harness` nodes are queryable uniformly by `executor.model`. 16. The audit log records every approval/input response with responder identity and is never sampled; credentials/secrets are scrubbed from logs and `exec.raw` before persistence. Disabling trace export does not disable the audit log. ## 11. Resolved Decisions (formerly Open Questions) 1. **Adapter distribution — decided (§5.2):** two tiers — core-bundled/pip-installed code adapters (Tier 1) and declarative `adapter.yaml` workspace artifacts (Tier 2) with mandatory human review on the `launch` block. 1a. **Declarative DSL ceiling — decided:** v1 interpreter is deliberately minimal — line-delimited JSON streams only, literal equality matching on event fields, JSONPath field extraction. No regex, conditionals, or multi-line aggregation. Anything beyond is `requires: code`. Expansion is demand-driven: recurring walls hit by real community adapter attempts define the next feature, not speculation. Rationale: every interpreter feature is maintained engine code and reviewer attack surface; the graduation path makes under-building cheap. 1b. **Community adapter registry — deferred:** no registry for now; adapters are shared as ordinary workspace artifacts. Revisit if organic demand appears; provenance/signing requirements to be defined then. 2. **Authoring-module authored adapters — decided:** the authoring swarm MAY generate Tier-2 declarative adapters into pending-review, on the same footing as topologies/archetypes/skills — every authored artifact requires human approval before use, and the `launch`-block human gate (§5.2) applies with no exemption. Tier-1 (code) adapters remain human-authored. 3. **Cost normalization — decided:** vendor-reported `cost_usd` is authoritative when present; tokens × price table is the fallback. The method used is recorded per run (`executor.cost.source: vendor | computed`) so the workspace meter is auditable. 4. **Auth for harness runs — decided:** both subscription and API-key modes supported; workspace-level default, archetype-level override. Mechanically these are **environment/auth-state**, not command flags: API keys are runtime-injected env vars (`ANTHROPIC_API_KEY`, `CODEX_API_KEY`) from the workspace credential store, scoped per-run and scrubbed from all persistence; subscription mode uses vendor auth state (long-lived setup tokens / saved CLI credentials) provisioned into the sandbox. Note: in headless mode an API-key env var takes precedence over subscription credentials where both exist — deterministic by design. Practical guidance encoded as a lint: high-concurrency topologies should override to API-key (subscriptions rate-limit parallel fleets). **Amendment to the credential rule (§4.1/§7):** the *model-provider* credential is the one exception to proxy-only injection — the harness must reach its LLM API — and is env-injected at launch; all other secrets (VCS tokens, cloud credentials) remain proxy-injected only. 5. **Telemetry-grade default — decided:** `telemetry_grade: opaque` adapters are **denied by default**; use requires explicit per-archetype opt-in. Unobservable execution is a deliberate choice, never a silent fallback. 6. **Trust-accrual threshold — decided:** default N=5 consistent approvals of a capability pattern triggers the allowlist changeset proposal; operator-tunable per workspace. A single denial resets the counter **and** blocks future proposals for that pattern until the operator manually clears the block — a denial is a signal, not noise. 7. **Question detection — decided:** shared implementation in core, and it is a **classifier, not a regex**: cheap pre-filters (final turn took no action / expected artifact absent) gate a small structured-output LLM call ("does this final message request input? extract question + options"). This is language-agnostic by construction — harnesses respond in the task's language, and punctuation heuristics fail outside English (か, ¿, unmarked-question languages). The classifier seat uses a small, cheap model per the tiered strategy. Adapters MUST prefer a native vendor question-event/callback where one exists. 8. **Lead-answer accountability — decided:** lead-node answers to `exec.input_requested` are surfaced in run review by default. Delegated judgment that cannot be audited is not governed. 9. **First-class `executor.model` — decided:** promoted to a core-recognized optional field; adapters map it to the vendor's model flag. Gives `model` and `harness` nodes a uniform, core-visible model attribute for cost queries and planner routing. ## 12. Phasing - **P1:** schema + registry + `model` executor formalized behind the interface (pure refactor, no behavior change). - **P2:** `claude-code` adapter, worktree sandbox, budget envelope, normalized events, cockpit display. Interaction: `deny`/`abort` only (never-hang guarantee). Observability: `executor` spans + audit log from day one (they consume the same event stream, so they are not deferrable). - **P3:** `codex` adapter (proves normalization across vendors), resume-token checkpoint integration. Interaction: `relay` for permissions via approval inbox; `exec.input_requested` with lead-node escalation and memoization. Observability: cross-boundary trace-context propagation; gate/escalation spans. - **P3.5:** trust-accrual → allowlist changeset proposals. - **P4:** Tier-2 declarative adapter engine (`executor-adapter` artifact schema, event_map interpreter, launch-block review gate) + contributor guide; a contributor-built `adapter.yaml` proves the contract. - **P5:** non-coding artifact profiles exercised end-to-end (files/structured/media); community adapter sharing evaluated (open question 1b). ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/structured-output-governance.md ## Published: https://delivstat.github.io/swarmkit/design-notes/structured-output-governance/ ══════════════════════════════════════════════════════════════════════════════ --- title: Structured output governance + auto-correction description: Deterministic output validation on skill results. Four-tier model. Field-specific errors enable targeted auto-correction re-prompts. tags: [governance, output-validation, auto-correction, m4] status: proposed --- # Structured output governance + auto-correction ## Goal When a skill declares an `outputs` block, the runtime enforces it deterministically — before any LLM judge fires. Field-specific validation errors are fed back to the model as targeted re-prompts, fixing one field at a time instead of regenerating the entire response. **Insight from external gate validation:** structured constraints + field-specific error feedback eliminates most hallucination at near-zero cost. Shape-level errors (wrong type, missing field, out-of-range value) don't need an LLM judge — they need a schema check + a re-prompt. ## Non-goals - **Semantic evaluation.** "Is this reasoning correct?" is Tier 3 (LLM judge). This note covers Tiers 0–2 (structural/deterministic). - **Free-text validation.** Output governance only fires for skills with declared `outputs` blocks. A root agent's final answer to the user is not schema-validated. - **Changing the skill schema.** The existing `outputs` block in `skill.schema.json` is sufficient. This note defines how the runtime enforces it. ## The four-tier output governance model | Tier | What | Cost | When | |---|---|---|---| | 0 | **Structured generation** — provider JSON mode / tool_use constrains the model at generation time | Zero extra | Always, when skill declares `outputs` | | 1 | **Schema validation** — JSON Schema check on the response | Near-zero | Always, after model response | | 2 | **Business rules** — deterministic field-level checks (ranges, enums, cross-field consistency) | Near-zero | When skill declares `validation_rules` | | 3 | **LLM judge** — semantic evaluation against a rubric | Tokens | When configured (separate design note) | Tiers 0–2 are this note. Tier 3 is `design/details/decision-skills.md`. ## Tier 0 — Structured generation When a skill declares `outputs`, the compiler translates the output schema into the provider's structured output mechanism: | Provider | Mechanism | |---|---| | Anthropic | `tool_use` with the output schema as `input_schema` | | OpenAI / Groq / OpenRouter | `response_format: { type: "json_schema", json_schema: {...} }` | | Google | `response_mime_type: "application/json"` + `response_schema` | | Ollama | `format: "json"` in options | The model is structurally constrained to produce JSON matching the schema. This eliminates shape-level hallucination at generation time. **Implementation:** in `_build_agent_node`, when the agent has skills with `outputs`, the `CompletionRequest` includes the output schema. Each provider adapter translates to its native mechanism. ## Tier 1 — Schema validation After the model returns a response, validate the parsed JSON against the skill's `outputs` block using `jsonschema`: ```python def _validate_skill_output( output: dict[str, Any], skill: ResolvedSkill, ) -> list[FieldError]: """Validate skill output against declared schema.""" errors = [] for field_name, field_spec in skill.outputs.items(): if field_name not in output: errors.append(FieldError(field_name, "missing required field")) continue value = output[field_name] # Type check if field_spec.get("type") == "enum": if value not in field_spec.get("values", []): errors.append(FieldError( field_name, f"must be one of {field_spec['values']}, got '{value}'" )) elif field_spec.get("type") == "number": range_ = field_spec.get("range") if range_ and not (range_[0] <= value <= range_[1]): errors.append(FieldError( field_name, f"must be between {range_[0]} and {range_[1]}, got {value}" )) elif field_spec.get("type") == "string": if not isinstance(value, str): errors.append(FieldError( field_name, f"must be a string, got {type(value).__name__}" )) return errors ``` Each error is **field-specific** — it names the field and describes exactly what's wrong. ## Tier 2 — Business rules Skills can declare `validation_rules` for cross-field and domain- specific checks: ```yaml # skill.yaml outputs: verdict: type: enum values: [pass, fail] confidence: type: number range: [0, 1] reasoning: type: string validation_rules: - if: { verdict: "fail" } then: { confidence: { max: 0.5 } } message: "Failed verdict should have confidence <= 0.5" - field: reasoning min_length: 20 message: "Reasoning must be at least 20 characters" ``` Rules are evaluated deterministically — no LLM, no tokens. Each rule produces a field-specific error on failure. **Implementation note:** `validation_rules` is a new optional field on the skill schema. Adding it follows `docs/notes/schema-change-discipline.md`. ## Auto-correction via field-specific errors When Tier 1 or 2 validation fails, the errors are fed back to the model as a targeted re-prompt: ``` Model returns: {"verdict": "pass", "confidence": 1.5, "reasoning": "ok"} Validation errors: - confidence: must be between 0 and 1, got 1.5 - reasoning: must be at least 20 characters Re-prompt to model: "Your response had validation errors on these fields: - confidence: must be between 0 and 1, got 1.5 - reasoning: must be at least 20 characters Correct ONLY these fields and return the full response." Model returns: {"verdict": "pass", "confidence": 0.85, "reasoning": "The code follows all quality standards and has good test coverage."} Validation: PASS ``` ### Retry budget Configurable per skill, defaults: ```yaml # In the skill's runtime_config or workspace-level config output_governance: max_retries: 2 # default escalate_on_failure: true # escalate to Tier 3 or HITL ``` If the model can't produce valid output after `max_retries`: 1. If `escalate_on_failure` is true → escalate to Tier 3 (LLM judge) or HITL review queue 2. If false → return the last response with validation errors attached in metadata ### Why this works - **Cheaper** than regenerating from scratch — the model corrects one or two fields, not the entire response - **More reliable** than "try again" — the error is specific, not vague - **Deterministic** in the validation step — no LLM judge cost for shape/range errors - **Composable** with Tier 3 — structural errors are caught cheaply, semantic errors escalate to the judge ## Where this hooks into the compiler In `_build_agent_node`, the tool-use loop (currently only handling delegation) is extended: ``` 1. Model call (with structured generation if skill has outputs) 2. Parse response 3. If tool_use: a. If delegation → route to child (existing) b. If skill with outputs → validate output: - Tier 1: schema validation - Tier 2: business rules - If valid → return result - If invalid → re-prompt with field errors (up to max_retries) - If exhausted → escalate or return with errors c. If skill without outputs → return result as-is 4. If text → return (no output governance on free-text) ``` ## Implementation plan ### PR 1 (this PR): design note + output validator - This design note - `packages/runtime/src/swarmkit_runtime/skills/_output_validator.py` — `validate_skill_output()` function + `FieldError` dataclass - Unit tests: valid output passes, type errors caught, range errors caught, enum errors caught, missing field caught ### PR 2: structured generation in providers - Each provider adapter's `complete()` method accepts an output schema and translates to the provider's native mechanism - `CompletionRequest` gains an `output_schema: dict | None` field - Tests: mock provider returns JSON matching schema ### PR 3: auto-correction loop in compiler - Wire `validate_skill_output()` into the compiler's agent node - Re-prompt with field-specific errors on failure - Retry budget (max_retries config) - Integration test: mock model returns invalid output → correction → valid output ### PR 4: validation_rules schema extension - Add `validation_rules` to skill schema (schema-change-discipline PR) - Implement rule evaluation in `_output_validator.py` - Tests for cross-field rules ## Test plan - **Valid output passes all tiers.** Skill with outputs, model returns conforming JSON → no errors, no retry. - **Type error triggers Tier 1.** Enum field gets wrong value → error names the field + expected values. - **Range error triggers Tier 1.** Number field out of range → error names the field + range. - **Missing field triggers Tier 1.** Required field absent → error names the field. - **Cross-field rule triggers Tier 2.** `if verdict=fail then confidence<=0.5` — verdict is fail, confidence is 0.8 → error. - **Auto-correction succeeds.** Mock model returns invalid on first call, valid on retry → final output is valid. - **Retry budget exhaustion.** Mock model returns invalid every time → escalation event recorded, last response returned with errors. ## Exit demo A skill with declared `outputs` schema: 1. Produces valid structured output on first attempt (structured generation constrains the model). 2. When given an intentionally malformed response (via mock), the auto-correction loop fixes the invalid field and succeeds on retry. 3. The retry prompt names the specific field and error — visible in audit events. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/governance-provider-interface.md ## Published: https://delivstat.github.io/swarmkit/design-notes/governance-provider-interface/ ══════════════════════════════════════════════════════════════════════════════ --- title: GovernanceProvider interface — method signatures, types, async semantics description: Finalises the GovernanceProvider ABC for M2. Covers evaluate_action, verify_identity, record_event, get_trust_score — all async. tags: [governance, abstraction, M2] status: active --- # GovernanceProvider interface ## Goal Lock in the `GovernanceProvider` ABC method signatures so M2 features can build against a stable contract. The M0 scaffold has the right shape but needs three upgrades: 1. **Async.** Governance calls may involve I/O (AGT could be a sidecar or remote service). The runtime is async-first per the style guide. 2. **Explicit scopes in `evaluate_action`.** Design §16.3 says "an agent can only invoke a skill if the agent's scopes include all the scopes the skill requires." Scopes belong in the call signature, not buried in a generic `context` dict. 3. **Richer types.** `PolicyDecision` needs tier info and a scope breakdown for observability. `AuditEvent` needs `datetime` timestamps and optional topology/skill identifiers for swarm-specific events. ## Non-goals - **AGT wiring.** `AGTGovernanceProvider` stays a stub until we pin AGT's Python SDK (design §21 open question). M2's real value is the mock + middleware pipeline, not AGT integration. - **AuditProvider.** The storage backend for audit events (task #38) is a separate abstraction. `GovernanceProvider.record_event` is the *intake* side; `AuditProvider` is the *storage* side. For M2, `MockGovernanceProvider` appends to an in-memory list. - **Tier 2/3 judges.** Tier 1 (deterministic policy checks) is M2. LLM judges land in M4. ## Finalised ABC ```python class GovernanceProvider(ABC): @abstractmethod async def evaluate_action( self, *, agent_id: str, action: str, scopes_required: frozenset[str], context: dict[str, object] | None = None, ) -> PolicyDecision: ... @abstractmethod async def verify_identity( self, *, agent_id: str, credential: AgentCredential, ) -> IdentityVerification: ... @abstractmethod async def record_event( self, event: AuditEvent, ) -> None: ... @abstractmethod async def get_trust_score( self, *, agent_id: str, ) -> TrustScore: ... ``` All methods are keyword-only past `self` (prevents positional mix-ups as the signature evolves). All are `async` — the mock returns immediately; real implementations may do I/O. ## Types ```python @dataclass(frozen=True) class PolicyDecision: allowed: bool reason: str tier: int # 1, 2, or 3 (§8.6) scopes_granted: frozenset[str] scopes_denied: frozenset[str] @dataclass(frozen=True) class AuditEvent: event_type: str # e.g. "skill.invoked", "policy.denied" agent_id: str timestamp: datetime # UTC payload: dict[str, object] topology_id: str | None = None skill_id: str | None = None @dataclass(frozen=True) class AgentCredential: credential_type: str # "ed25519", "did", "mock" value: str @dataclass(frozen=True) class IdentityVerification: verified: bool agent_id: str @dataclass(frozen=True) class TrustScore: score: float # 0.0–1.0 normalised tier: str # behavioral tier label ``` ## MockGovernanceProvider Deterministic, configurable, test-only. Ships in `governance/_mock.py`. Core design: - **Constructor takes `allowed_scopes`.** `evaluate_action` checks whether `scopes_required ⊆ allowed_scopes`. If yes → allowed. If no → denied with the missing scopes listed in `reason`. - **Events are collected.** `record_event` appends to an internal list. `.events` property returns a copy. Tests assert against the list to verify audit flow. - **Identity always verifies.** `verify_identity` returns `True`. Tests that need identity failure use a separate deny-all mock or parametrise. - **Trust scores configurable.** Constructor takes an optional `trust_scores: dict[str, float]` mapping agent_id → score. Default 1.0 for unknown agents (fully trusted). ## Middleware pipeline (follow-up PR) A second PR adds the runtime middleware that routes every skill invocation through `evaluate_action` before execution. Not in this PR because the middleware depends on the skill-invocation path that M3 (LangGraph compiler) will build. For M2, the exit demo uses the mock directly in test code. ## Test plan `packages/runtime/tests/test_governance_provider.py`: 1. **Mock allows when scopes match.** Agent with `{repo:read}` invoking a skill requiring `{repo:read}` → allowed, tier 1. 2. **Mock denies when scopes don't match.** Agent with `{repo:read}` invoking a skill requiring `{repo:write}` → denied, scopes_denied contains `repo:write`. 3. **Audit events collected.** `record_event` → `mock.events` has the event. 4. **Trust score returns configured value.** Agent with custom score → that score. Unknown agent → 1.0. 5. **Identity verification.** Default mock → verified. 6. **Separation-of-powers invariant.** The mock's events list is read-only from the provider's public API — there's no `clear_events` or `delete_event` method. ## Demo The exit demo for M2 is a test, not a CLI command: "a unit-test swarm where a worker tries to invoke a skill it lacks the scope for; policy denies; audit records the attempt; test asserts both." This test ships with this PR using MockGovernanceProvider. The AGT wiring PR adds the same scenario against the real provider. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/model-provider-abstraction.md ## Published: https://delivstat.github.io/swarmkit/design-notes/model-provider-abstraction/ ══════════════════════════════════════════════════════════════════════════════ --- title: Model provider abstraction description: ModelProvider ABC, built-in providers (anthropic, openai, google, ollama, mock), plugin path. Mirrors GovernanceProvider (§8.5). Blocks M3. tags: [runtime, model-provider, abstraction, m2.5] status: approved --- # Model provider abstraction **Scope:** `packages/runtime/src/swarmkit_runtime/model_providers/` **Design reference:** §10.2 (agent `model` block), §13 (archetype defaults), §14.3 (LangGraph compiler dispatches to models), mirrors §8.5 (`GovernanceProvider`). **Status:** proposed — blocks **M3**. ## Goal Give users a single way to declare which LLM a given agent uses — Anthropic, OpenAI, Google, local Ollama, or anything custom — without the runtime hard-coding any SDK. Topology YAML stays provider-agnostic; the runtime resolves via registered providers at load time; developers add new providers through a plugin interface without touching the framework. This is the parallel of `GovernanceProvider` (§8.5): narrow interface, several built-in implementations, stable boundary. ## Non-goals - Model **routing** or automatic failover. The topology declares which model; the runtime uses it. Any routing logic is a separate concern (future skill category, not the provider layer). - Abstracting the **execution engine** (LangGraph) — out of scope here; see §7 principle "Framework-aligned, not framework-locked." - A new message format. The internal canonical message structure follows Anthropic's messages shape (roles, content blocks, tool_use / tool_result) because it is the most expressive of the three major providers; provider adapters translate to/from it. Format choice is revisitable but should not be a design question per call. - Tool-calling semantics in depth — covered by a follow-up note (`model-provider-tool-calling.md`) because each provider has its own tool-call protocol and a clean normalisation deserves its own PR. This note states the interface shape; the tool-calling details land before M3. ## Why this deserves its own abstraction Three concrete forces push in the same direction: 1. **User choice.** From the user's message on PR #5 review: agents should be configurable per-leader / per-worker / per-archetype. An engineering team running Opus on leaders and local Ollama on workers should not need a fork of the framework. 2. **Vendor neutrality.** SwarmKit is positioned as the "Terraform for swarms." Committing the runtime to any one SDK fails that positioning. Anthropic is our own primary LLM, but the framework makes no assumption beyond "an LLM the user picked." 3. **Local inference.** Ollama and similar local runtimes matter for privacy-sensitive and cost-sensitive users. First-class support for local is a differentiator vs. frameworks that assume hosted APIs. ## API shape ```python # packages/runtime/src/swarmkit_runtime/model_providers/__init__.py from abc import ABC, abstractmethod from collections.abc import AsyncIterator, Sequence from dataclasses import dataclass from typing import Any, ClassVar, Literal @dataclass(frozen=True) class Message: role: Literal["system", "user", "assistant", "tool"] content: str | Sequence[ContentBlock] # string for simple text, blocks for tool_use / tool_result @dataclass(frozen=True) class ContentBlock: type: Literal["text", "tool_use", "tool_result", "image"] # Canonical fields; provider adapters read what they need. text: str | None = None tool_use_id: str | None = None tool_name: str | None = None tool_input: dict[str, Any] | None = None tool_result: Any | None = None @dataclass(frozen=True) class CompletionRequest: model: str # provider-specific model name, e.g. "claude-sonnet-4-6" messages: Sequence[Message] system: str | None = None temperature: float | None = None max_tokens: int | None = None tools: Sequence[ToolSpec] | None = None # canonical SwarmKit tool spec; adapter translates extra: dict[str, Any] | None = None # provider-specific passthrough @dataclass(frozen=True) class CompletionResponse: content: Sequence[ContentBlock] stop_reason: Literal["end_turn", "max_tokens", "tool_use", "error"] usage: Usage # input/output/cache tokens — see governance overhead budget (§8.6) raw: Any = None # provider response for debugging; never relied on in runtime logic class ModelProvider(ABC): """Narrow abstraction over LLM providers. Stable across framework versions. Implementations must be importable without their backing SDK being installed if `supports(...)` can return False for every model. That lets users install SwarmKit + only the SDKs they use. """ provider_id: ClassVar[str] # "anthropic", "openai", "google", "ollama", custom slugs @abstractmethod async def complete(self, request: CompletionRequest) -> CompletionResponse: """One-shot completion. Tools, if provided, are translated to provider format.""" @abstractmethod def stream(self, request: CompletionRequest) -> AsyncIterator[ContentBlock]: """Token / content-block streaming. Yields partial blocks until stop_reason terminates.""" @abstractmethod def supports(self, model: str) -> bool: """Whether this provider can serve the given model name. Fast, no network call.""" def tokenize(self, text: str, model: str) -> int | None: """Optional token count for budgeting; None if the provider can't compute it cheaply.""" return None ``` ## Built-in providers (v1.0) Ship in `packages/runtime/src/swarmkit_runtime/model_providers/builtin/`: | provider_id | SDK | Notes | |---|---|---| | `anthropic` | `anthropic>=0.39` | First-class. Default model `claude-sonnet-4-6`. | | `openai` | `openai>=1.50` | Also handles `azure-openai` variant via `extra.base_url` and Azure auth; no separate provider_id. | | `google` | `google-genai>=1.0` | Gemini family. | | `ollama` | `httpx` only (no SDK) | Local inference. Defaults to `http://localhost:11434`; `extra.base_url` overrides. | | `mock` | none | Deterministic responses keyed by request hash; test-only. Always importable. | Each SDK is a soft dependency: the provider module declares `provider_id` and attempts its import lazily. A user topology that names `provider: anthropic` fails at load time with a clean "anthropic SDK not installed; `uv tool install swarmkit-runtime[anthropic]` or `pip install swarmkit-runtime[anthropic]`" message. Ranking rationale (why these five): - **Anthropic** — SwarmKit's primary LLM partner; authoring swarms target Claude. - **OpenAI** — largest market share; users must have an easy path. - **Google** — rapidly closing the gap, notably cost-competitive for long context. - **Ollama** — the open-weights story. Covers most local setups including llama.cpp-compatible runtimes via its OpenAI-compatible endpoint. - **Mock** — non-optional; tests everywhere depend on it. ## Registration & plugin mechanism Two layers: 1. **Built-ins** — auto-registered when `swarmkit_runtime.model_providers` is imported. No user config needed. 2. **Custom providers** — discovered via Python entry points group `swarmkit.model_providers`. A third-party package declares: ```toml [project.entry-points."swarmkit.model_providers"] acme-cloud = "acme_swarmkit_plugin:AcmeModelProvider" ``` At runtime, `pkg_resources` / `importlib.metadata` enumerates entries in the group and registers each class. 3. **Per-workspace override** — `workspace.yaml` may declare providers by fully-qualified class path: ```yaml model_providers: - class: my_internal_pkg.InternalProvider config: base_url: https://internal-llm.example.com ``` This path does not require the provider to be published as an installable package — useful for org-internal providers. Resolution order when a topology references `provider: foo`: workspace overrides → entry-point plugins → built-ins. First match wins. Duplicate IDs fail topology load. ## Credentials & config Credentials are **never** in topology YAML. Topologies are shareable artifacts; credentials are deployment-specific. Per-provider conventions: | Provider | Primary auth | |---|---| | `anthropic` | `ANTHROPIC_API_KEY` env | | `openai` | `OPENAI_API_KEY` env; `OPENAI_BASE_URL` for Azure / compatible endpoints | | `google` | `GOOGLE_API_KEY` env; or ADC on GCE/CloudRun | | `ollama` | none (local) | `workspace.yaml` may override (sealed-secret reference, Vault lookup, etc.) — that's the deployment story, not the provider story. `workspace-schema-v1.md` will define the exact shape when §M0 reaches it. ## Where the abstraction is wired - **Topology schema** — unchanged. The existing `model.provider` string is already provider-agnostic; this design ratifies that choice. - **Topology loader** — after archetype merge, validates that every distinct `provider` referenced in the topology is registered. Fails fast with a list of missing providers. - **LangGraph compiler (M3)** — agent node functions receive a `ModelProvider` instance (resolved from the registry) and call `.complete(...)` / `.stream(...)`. No SDK imports outside provider modules. - **Authoring swarms (M7, M8)** — use the same abstraction. A user who only has Ollama installed can still author skills locally. ## Non-negotiable invariant (lands with this PR) **All LLM calls in the runtime go through `ModelProvider`.** Only files under `packages/runtime/src/swarmkit_runtime/model_providers/` may import Anthropic / OpenAI / Google / Ollama SDKs. Every other module receives a provider instance or a request object. Same rule as §8.5 for AGT imports. Enforced by review; a future lint rule could enforce mechanically. ## Test plan - **Unit — ABC contract:** `MockModelProvider` implements the full interface; tests assert it satisfies the ABC and returns deterministic responses keyed on `(model, messages)`. - **Unit — registry:** built-ins register on import; conflicting IDs raise; entry-point discovery is mocked via `importlib.metadata.entry_points`. - **Unit — per-provider adapters:** each built-in adapter is tested with **recorded** responses (a tiny cassette format under `tests/fixtures/model-responses/`). No network in unit tests. - **Integration — live APIs:** gated on `ANTHROPIC_API_KEY` etc. env vars with `pytest.mark.integration`. Skipped locally unless the key is present; run nightly in a separate CI workflow (not the PR pipeline). - **Negative:** topology referencing a non-registered provider fails `swarmkit validate` with a clear error pointing at the offending agent. ## Demo plan - `just demo-model-providers` (lands with the implementation PR, not this design PR): walks through each registered provider, does a single-shot completion with the prompt "Say ready" against its cheapest model, prints pass/fail. Missing creds are reported as `skipped` not `failed`. For `mock` and `ollama` (if running) it always runs. - The implementation PR also ships `examples/model-choice/` — a two-agent topology where the leader runs Opus and the worker runs a local Ollama model. One YAML file, one README; proves cross-provider coordination. ## Open questions - **Streaming protocol through LangGraph:** LangGraph has its own streaming primitives. Do we expose provider streaming directly or proxy through LangGraph's channel system? Tentative: proxy — keeps the rest of the framework consistent. Revisit at M3. - **Tool-calling normalisation:** each provider's tool-call format differs (Anthropic's `tool_use` / `tool_result` blocks, OpenAI's function-calling schema, Google's parts system). Will be designed in `model-provider-tool-calling.md` before M3 implementation starts. - **Cost tracking hooks:** every response carries `usage`, but who aggregates? Proposed: a `persistence` skill category consumer writes to the audit log via `GovernanceProvider.record_event`. Not framework-level infrastructure. - **Rate limiting / retry:** start with a minimal exponential-backoff in each adapter. Global cross-provider throttling is future work. - **Token budget enforcement:** ties to §8.6 governance overhead target. Providers expose `tokenize`; the judicial pillar (policy engine) can deny requests that would exceed budget. Cross-cutting; not part of this abstraction. ## Slot in the implementation plan Insert a new **M2.5** between M2 (governance) and M3 (compiler): - **M2.5 — Model provider abstraction** - Design note: `design/details/model-provider-abstraction.md` (this PR). - Impl PR 1: `ModelProvider` ABC + `MockModelProvider` + `AnthropicModelProvider`. Enough to unblock M3. - Impl PR 2: OpenAI + Google + Ollama adapters. Can land in parallel with M3 or after. - Impl PR 3: `model-provider-tool-calling.md` design note + implementation. Blocks M5 (MCP integration) because tool calls are the bridge. - Exit demo: `just demo-model-providers` green for every provider with creds present; integration tests nightly. Without M2.5, M3's compiler PR would either hard-code Anthropic (violating principle §7 "Framework-aligned, not framework-locked") or re-invent this abstraction in-line. Landing the design now means M3 slots the compiler into an already-decided seam. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/dag-dependency-graph.md ## Published: https://delivstat.github.io/swarmkit/design-notes/dag-dependency-graph/ ══════════════════════════════════════════════════════════════════════════════ # DAG Dependency Graph for Agent Topologies **Status:** Design note — planned feature **Design ref:** §5.2 (agent hierarchy), §10 (topology schema), §14.3 (LangGraph compiler) ## Problem SwarmKit topologies currently support tree-based delegation — a parent delegates to children, children return to parent. The ordering is either prompt-driven (the root's system prompt says "send to A first, then B") or hierarchy-driven (leader delegates to workers). This works for most cases but breaks down for pipelines with strict ordering requirements: - Content pipeline: researcher MUST complete before writer starts - CI/CD flow: build MUST pass before test runs, test MUST pass before deploy - Analysis: data collector and data validator MUST both complete before the analyst runs - Review: author writes, reviewer reviews, editor edits — strict sequence Prompt-driven sequencing is fragile for these — the model might reorder steps, skip dependencies, or try to run things in parallel when they shouldn't be. ## Solution Add `depends_on` to the agent schema. Agents with dependencies only run after all specified agents have completed with valid results. ```yaml agents: root: id: root role: root model: provider: openrouter name: meta-llama/llama-3.3-70b-instruct children: - id: researcher role: worker archetype: domain-researcher - id: writer role: worker archetype: blog-writer depends_on: [researcher] - id: reviewer role: worker archetype: content-reviewer depends_on: [writer] - id: editor role: worker archetype: content-editor depends_on: [reviewer] ``` This declares: researcher → writer → reviewer → editor. The runtime enforces the order regardless of what the root model decides. ### Parallel + sequential mixed ```yaml children: - id: doc-searcher role: worker archetype: doc-specialist - id: code-searcher role: worker archetype: code-specialist - id: analyst role: worker archetype: analyst depends_on: [doc-searcher, code-searcher] - id: report-writer role: worker archetype: report-writer depends_on: [analyst] ``` This runs doc-searcher and code-searcher in parallel (no deps), then analyst once both complete, then report-writer after analyst. ``` doc-searcher ──┐ ├──→ analyst ──→ report-writer code-searcher ─┘ ``` ## Schema change Add `depends_on` to the `child_agent` definition in `topology.schema.json`: ```json "child_agent": { "allOf": [ { "$ref": "#/$defs/agent" }, { "properties": { "role": { "enum": ["leader", "worker"] }, "depends_on": { "type": "array", "items": { "type": "string" }, "description": "Agent IDs that must complete before this agent runs." } } } ] } ``` ## Compiler change ### Current flow (tree-based) ``` Root model call → delegate_to_X tool call → route to X → X runs → return to root ``` The root decides ordering dynamically via tool calls. ### New flow (DAG-based) When `depends_on` is declared, the compiler builds a dependency graph alongside the delegation tree: ```python def _compile_dag_edges(root, agents, graph): """Add dependency-based edges to the graph.""" for agent in agents.values(): deps = getattr(agent, 'depends_on', None) or [] if deps: for dep_id in deps: graph.add_edge(dep_id, agent.id) ``` ### Execution modes **Mode 1: Root-delegated (current, no deps)** Root explicitly calls `delegate_to_X`. Runtime routes to X. No change. **Mode 2: Auto-dispatched (with deps)** When a topology has `depends_on` declarations, the root doesn't need to delegate explicitly. The runtime auto-dispatches agents based on dependency resolution: 1. Find all agents with no unmet dependencies → run them (parallel) 2. When an agent completes, check which agents now have all deps met 3. Run newly unblocked agents 4. Repeat until all agents complete 5. Return all results to root for final synthesis The root's role shifts from "active delegator" to "final synthesiser." **Mode 3: Hybrid** Some agents have `depends_on`, others don't. The root can still delegate explicitly to agents without deps, while the runtime auto-dispatches the rest based on the graph. ### Router implementation ```python def _dag_router(state, agents_with_deps): """Route to the next runnable agent based on dependency graph.""" results = state.get("agent_results", {}) completed = set(results.keys()) for agent in agents_with_deps: deps = set(agent.depends_on) if agent.id not in completed and deps.issubset(completed): return agent.id # all deps satisfied # All agents complete or blocked return "__synthesise__" # route to root for final answer ``` ### Input passing When an agent with deps runs, its input includes the outputs of its dependencies: ```python dep_results = { dep_id: results[dep_id] for dep_id in agent.depends_on if dep_id in results } child_state = { "input": f"Based on these findings:\n{dep_results}\n\nYour task: {task}", ... } ``` This ensures each agent sees what its predecessors produced. ## Validation At compile time, the resolver validates: 1. **No cycles** — `depends_on` cannot form circular dependencies 2. **All references valid** — every ID in `depends_on` must exist as a sibling agent 3. **No self-reference** — agent cannot depend on itself 4. **Root cannot have deps** — the root agent runs first always ```python def _validate_dag(agents): """Detect cycles and invalid references in depends_on.""" # Topological sort — if it fails, there's a cycle visited = set() in_progress = set() def visit(agent_id): if agent_id in in_progress: raise ResolutionError(f"Cycle detected involving {agent_id}") if agent_id in visited: return in_progress.add(agent_id) for dep in agents[agent_id].depends_on: if dep not in agents: raise ResolutionError(f"{agent_id} depends on unknown agent {dep}") visit(dep) in_progress.remove(agent_id) visited.add(agent_id) for agent_id in agents: visit(agent_id) ``` ## Coexistence with current patterns DAG dependencies and delegation-based routing coexist: | Feature | Tree delegation | DAG dependencies | |---------|----------------|-----------------| | Who decides order | The root model | The YAML declaration | | When to use | Flexible Q&A, exploration | Strict pipelines | | Parallel execution | Root calls multiple delegates | Agents with no deps run in parallel | | Input to agent | Delegation task text | Predecessor outputs | | Root's role | Active orchestrator | Final synthesiser | A topology can mix both: some agents delegated by root, others auto-dispatched by deps. Agents without `depends_on` behave exactly as they do today. ## Example topologies ### Content pipeline ```yaml apiVersion: swarmkit/v1 kind: Topology metadata: name: content-pipeline agents: root: id: root role: root model: provider: openrouter name: meta-llama/llama-3.3-70b-instruct prompt: system: | You coordinate content creation. Your workers run automatically in dependency order. Review the final output from the editor and present it to the user. children: - id: researcher role: worker archetype: trend-researcher - id: writer role: worker archetype: blog-writer depends_on: [researcher] - id: seo-reviewer role: worker archetype: seo-reviewer depends_on: [writer] - id: editor role: worker archetype: content-editor depends_on: [writer] - id: publisher role: worker archetype: blog-publisher depends_on: [seo-reviewer, editor] ``` ### Parallel research + synthesis ```yaml children: - id: web-researcher role: worker archetype: web-searcher - id: doc-researcher role: worker archetype: doc-searcher - id: code-analyst role: worker archetype: code-analyst - id: synthesiser role: worker archetype: research-synthesiser depends_on: [web-researcher, doc-researcher, code-analyst] ``` ## Cost implications DAG-based execution is more predictable than delegation-based: - Every agent runs exactly once (no re-delegation loops) - Parallel agents run concurrently (same as parallel delegation) - No wasted root model calls for sequencing decisions - The root only runs twice: once to start, once to synthesise For a 5-agent pipeline, delegation-based might need 5+ root calls (one per delegation round). DAG-based needs 2 root calls total. ## Implementation estimate | Component | Effort | |-----------|--------| | Schema: add `depends_on` to topology.schema.json | 1 hour | | Schema: Python + TypeScript codegen | 1 hour | | Resolver: validate DAG (cycles, references) | 2-3 hours | | Compiler: dependency-based routing edges | 4-6 hours | | Compiler: auto-dispatch with input passing | 4-6 hours | | Tests: unit + integration | 4-6 hours | | Reference topology: content pipeline | 2-3 hours | | **Total** | **2-3 days** | ## Schema shape — `depends_on` lives on the base `agent` def `depends_on` is declared on the **base `agent`** def in `topology.schema.json`, not only on `child_agent`. This is deliberate and load-bearing — do not move it back. `child_agent` is `allOf: [{ $ref: agent }, { properties: { role, … } }]`, and the base `agent` has `additionalProperties: false`. In JSON Schema, `additionalProperties` only sees the properties declared in **its own** subschema, not those added by a sibling `allOf` branch. So a `depends_on` declared only in `child_agent`'s branch was rejected by `agent`'s `additionalProperties: false` as an unexpected property — every child using `depends_on` failed validation (the DAG feature was latent-broken at the schema layer; only runtime tests exercised it). Fix: declare `depends_on` on the base `agent` so the strict base accepts it; `child_agent` keeps only the `role` narrowing. Cost: the root nominally accepts a `depends_on` it ignores (it has no siblings) — harmless, and cheaper than the `unevaluatedProperties` alternative, which emits noisy secondary errors on already-invalid documents. Regression fixture: `packages/schema/tests/fixtures/topology/with-depends-on.yaml`. ## Open questions 1. Should `depends_on` agents pass their full output or a summary to the dependent? Full output could be large; summary loses detail. 2. What happens when a dependency fails? Options: skip the dependent, run it with an error note, or fail the whole pipeline. 3. Should the root see intermediate results as they complete, or only the final output? 4. Can `depends_on` cross topology boundaries? (Probably not in v1.) ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/command-packs.md ## Published: https://delivstat.github.io/swarmkit/design-notes/command-packs/ ══════════════════════════════════════════════════════════════════════════════ --- title: Command packs description: Local commands as a skill implementation type, declared in packs, governed by the same permission tiers as MCP servers. tags: [skills, governance, schema] status: draft --- # Command packs **Scope:** `packages/schema`, `packages/runtime` (skills, governance, workspace) **Design reference:** §6 (skills), §8.5–§8.7 (governance), §18 (MCP integration) **Status:** draft ## Goal Let a skill invoke a **local command** instead of an MCP tool, under the permission model that already governs MCP calls — and let commands be declared in **packs** so a topology grants a set rather than a list. ## The premise this corrects MCP was never the extension paradigm. Skills are, and `skill.schema.json` has always carried three implementations: `mcp_tool`, `llm_prompt`, `composed`. Only one speaks MCP. This note adds a fourth, `command`, and the workspace-level declaration it needs. Invariant 2 in `CLAUDE.md` is untouched: skills remain the only capability extension primitive. A command is a new way to *back* a skill, not a second way to extend the swarm. ## Why the permission model transfers unchanged `check_mcp_permission` does four things, and none of them is MCP: ```python unmet = prerequisites.missing(requires, ...) # implementation-agnostic already permission = mcp_manager.get_permission(server_id, tool_name) if permission == "open" or governance is None: return True, "" decision = await governance.evaluate_action( action=f"mcp:call:{server_id}:{tool_name}", scopes_required=scopes, context={"server_permission": permission}) ``` And `get_permission` resolves `permission_overrides[tool] or server.permission` — a config lookup. No protocol, no tool annotations, no round trip. So what a command pack reuses is a **two-level (container, member) tier lookup**, plus the scopes the skill already declares. Pack is the container, command the member. The tier machinery is not being mirrored; it is being renamed to what it always was. Note the two inputs to `evaluate_action` do different jobs, and the next section separates them: `scopes_required` is what authorizes, `action` is what labels. ## API shape ### Workspace declaration ```yaml command_packs: - id: json-tools permission: cautious # same four tiers as mcp_servers timeout: 30s # pack default; built-in default if omitted max_output: 10MB requires: - { binary: jq, version: '>=1.7' } # checked at workspace load, not at run time commands: - id: query argv: [jq, '-r', '{filter}', '{file}'] effects: read output: { parse: json } - id: edit-in-place argv: [jq, '-r', '--in-place', '{filter}', '{file}'] effects: write permission_overrides: edit-in-place: strict timeout_overrides: big-report: 5m ``` ### Skill implementation ```yaml implementation: type: command pack: json-tools command: query ``` `inputs` on the skill supplies the typed parameter schema the model sees — the one thing MCP was providing that a bare CLI does not. Substitution is **value-only into argv**; there is no shell, and a substituted value is never re-parsed into arguments. This is the same rule the executor adapter DSL already enforces (`launch.command` is argv, `$defs/template` is value-only), and it is the injection boundary: a `{filter}` of `; rm -rf /` is an inert string. ### Governance — scopes authorize, actions label These are two different things and the distinction is easy to lose. **Scopes are the gate.** OIDC-style `noun:verb`, declared per skill in `iam.required_scopes`, and the only authorization test the runtime performs: ```python granted = scopes_required & allowed_scopes denied = scopes_required - allowed_scopes if denied: → deny ``` Real values in `reference/skills/`: `workspace:read`, `workspace:write`, `knowledge:read`, `repo:read`, `tests:execute`. **A command skill declares these exactly like any other skill.** That is the whole governance integration — the pack contributes a permission tier, the skill contributes scopes, and nothing new is needed. **The action string is a label, not a rule.** `mcp:call:{server}:{tool}` is passed alongside the scopes and is used for three things: a substring scan under the `readonly` tier, the human-readable `reason`, and the audit event. Nothing pattern-matches it — a search of `reference/`, `examples/`, `docs/` and `packages/` finds no policy artifact written against it, only prose describing the convention and two tests asserting the emitted string. So commands take a sibling namespace and `mcp:call:` is left alone: ``` mcp:call:{server}:{tool} unchanged command:call:{pack}:{command} new ``` An earlier draft of this note proposed generalising both to `tool:call:{provider}:{name}`, on the theory that a rule written `mcp:call:*` would silently stop covering everything. **That theory was wrong** — no such rules exist, because actions are not matchable. The rename would have bought consistency at the cost of audit-history continuity, which is a bad trade for a string nothing reads. ### Structure goes in the payload, not the string What the generalisation was actually reaching for is better served by structure. `AuditEvent.payload` is already `dict[str, object]`, so this needs no schema change: ```yaml action: command:call:json-tools:query payload: provider: command # mcp | command container: json-tools # server or pack id member: query effects: read ``` If action-matching policies ever arrive, they match fields rather than parsing a colon-joined string, and every question about how to segment that string dissolves. The string stays a display format. ### Secrets and bounds ```yaml command_packs: - id: github credentials_ref: gh-token env: { GH_TOKEN: '{credential.gh-token}' } # secrets reach a command HERE commands: - id: list-prs argv: [gh, pr, list, '--repo', '{repo}'] # …and never here ``` **A credential may be substituted into `env`, never into `argv`** — `{credential.*}` appearing in an argv template is a schema error. Three things follow from that one rule, and none of them has to be remembered afterwards: a secret cannot be placed by a model, cannot land in an audit line that records the command that ran, and cannot be read out of `ps` by anything else on the box. The cost is real and accepted: CLIs that only take a credential as a flag need a wrapper. Buying the guarantee back later would mean auditing every place a command line is logged, which is the kind of retrofit that is never finished. **Bounds mirror the tier shape** — `timeout` and `max_output` on the pack, `timeout_overrides` per command. Ships with a conservative built-in default so an undeclared pack is still bounded; a command with no ceiling is a command that can take a run down with it, and unbounded-by-default is not a decision anyone makes deliberately. ## Four decisions this note is making ### 1. `effects` is declared per command, and defaults to `write` For MCP, a permission tier is a hint laid over an opaque tool — the runtime cannot tell whether `foo` writes. For a command, nothing is inferrable either: `curl` POSTs, `jq` and `sed` both take `-i`. But the pack author *knows*, so they must say. An undeclared `effects` is `write`. This makes `permission: readonly` genuinely enforceable, and it replaces a heuristic rather than adding a field. Today the `readonly` tier decides write-ness by substring-scanning the action string: ```python _write_signals = ("create","delete","update","write","modify","edit","insert","drop","push","send") if any(sig in action.lower() for sig in _write_signals): → deny ``` So `truncate_table` and `purge_cache` pass `readonly` today — neither substring appears — while `send_query` is denied though it only reads. Commands pass `effects` through the decision `context` and are never sniffed. The MCP side has the same bug and is worth fixing independently of this note. ### 2. Argv only — never a shell, and never a generic `bash` skill A `command` skill fills declared slots in a frozen argv template. It does not compose invocations. This is the line, and it is load-bearing. Governance names actions; `bash` is one action that means anything, so no policy can be written over it. Structural gates (invariant 6) stop being structural the moment an agent can `curl` a reserved endpoint, and the append-only audit log (§8.3) assumes nothing executive can reach the file. SwarmKit already admits arbitrary execution — the harness executor — and the containment it needed is visible in `executors/`: `_sandbox.py`, `_egress.py`, `_approval.py`, `_budget.py`, `_container.py`. A shell skill would need all of that again, for a capability that already exists one layer up. A declarative argv skill needs none of it. ### 3. A pack grant carries the pack's read commands only Granting a set rather than a list is the point of packs, and it creates a widening problem that is the mirror image of the wildcard one: an agent granted `pack:json-tools` would gain whatever is added to that pack later, silently. ```yaml skills: - pack:json-tools # every READ command, now and later - json-editing-rewrite # a write, named — bulk grants never carry one ``` A read command added to the pack flows through to everyone holding it, which is the ergonomics the bulk form exists for. A write command never does. > **Amended during implementation.** This note originally said a pack gaining a write command should > **fail workspace load** until every holding agent re-confirmed. That needs a stored list of > acknowledged commands living somewhere, which is state in what is meant to be a declarative > artifact — and topology-as-data is the first invariant in `CLAUDE.md`. Excluding writes from the > bulk form gets the same safety property with no stored state and no load-time failure: the grant > means exactly what it says on the line, every time it is read. The rejected version is recorded > because "it fails loudly" sounded like the safer answer and was the more complicated one. `server:` deliberately makes no equivalent promise. An MCP tool has no declared effect to filter on — that is [#825](https://github.com/delivstat/swarmkit/issues/825) — so a server grant carries everything targeting that server. The asymmetry is honest; pretending otherwise would be worse. ### 4. Grants take packs *and* servers, not just skills Agents currently grant skills one at a time. MCP has the identical problem today (declare a server, write a skill per tool, grant each), and solving it only for commands makes the two paradigms diverge on exactly the ergonomics this is meant to fix. ```yaml skills: [pack:json-tools, server:filesystem, some-individual-skill] ``` Bigger change, better resting state, one mental model. **Implementation note.** A pack command becomes an ordinary skill at registry-build time, with the id `-`. That is what keeps the tool builder, `requires:` validation, the archetype merge and the UI from each needing to know packs exist — a command *is* a skill from the moment the registry is built. A synthetic id colliding with a hand-authored skill is a resolution error naming both, rather than one shadowing the other. A bulk grant matching nothing is also an error, not an empty set: an agent silently granted no tools is indistinguishable from one whose model chose not to use them, and that only surfaces later as a puzzling transcript. ## Bundled packs Packs ship bundled, as the four harness adapters do. An earlier draft worried that a bundled pack is an execution surface nobody chose — that was wrong: **availability is not access.** A bundled pack still has to be declared in the workspace and granted to an agent, and the grant is the audit step. Bundling only saves everyone writing the same `jq` pack by hand. ## Non-goals - A generic shell or `bash` skill. See decision 2. - Remote command execution. A pack runs on the machine running the swarm. - Replacing MCP. Packs are for tools that already exist as binaries and would otherwise need a wrapper server written for them. - Composition between commands. Two commands piped together is a `composed` skill, or it is a harness executor node — both already exist. - **Path confinement.** `cwd` sets the working directory and an absolute path escapes it; a command reads and writes whatever the runtime process can. Confining paths means understanding each binary's argument grammar — `cat` takes a path, `find` takes a root, `curl` takes neither — and a pack that half-confines is worse than one that states it does not. What bounds a pack is the grant (a pack is inert until a topology asks for it), the tier, and the skill's `iam.required_scopes`. Real filesystem confinement is a container, which is the same answer SwarmKit gives for harness executors. ## Test plan - **Unit** — tier resolution for `(pack, command)` mirrors the MCP table, including `permission_overrides`; `effects` defaulting to `write`; `readonly` denying a `write` command. - **Unit** — argv substitution is value-only: a parameter containing `;`, `|`, `$(…)`, spaces and newlines reaches the process as exactly one argv entry. This is the security test and it should read like one. - **Unit** — `requires.binary` missing at workspace load fails with the binary named; a version below the constraint fails the same way. - **Integration** — a topology granting `pack:json-tools` resolves every command in the pack; a denied command returns the standard `DENIED_MARK` refusal, identical in shape to an MCP denial. - **Integration** — a pack gaining a `read` command leaves existing grants valid; gaining a `write` command fails workspace load, naming every agent that holds the pack. - **Unit** — the audit payload carries `provider`, `container`, `member` and `effects` as fields, for both an MCP call and a command call, so a query never parses the action string. - **Test data** — a pack fixture under `packages/schema/tests/fixtures/`, plus an invalid one (shell metacharacters in `argv`, missing `requires`). ## Demo plan `examples/command-packs/` — a workspace declaring a `json-tools` pack and a two-agent topology where one agent holds `pack:json-tools` read commands and the other holds the `strict` write command. Terminal transcript showing: a read command running unattended, a write command stopping at a governance decision, and a `readonly` pack refusing the write with the reason printed. The transcript must include the injection case — a filter parameter containing `; rm -rf /` running harmlessly and producing an ordinary `jq` error — because that is the claim a reviewer will most want to see rather than be told. ## Open questions 1. **The actual numbers** for the built-in timeout and output ceiling. The shape is decided; the values want measuring against real packs rather than picking a round number that looks sensible. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/a2a-interop.md ## Published: https://delivstat.github.io/swarmkit/design-notes/a2a-interop/ ══════════════════════════════════════════════════════════════════════════════ --- title: A2A interop — every topology is an A2A agent, and a remote A2A agent is a skill description: Serve publishes an Agent Card and the A2A task API for each topology, mapped onto the existing job model so governance is untouched; an `agent` skill calls another topology (in-process, a child job) or a remote A2A agent (over the wire) through the same permission seam, streamed and audited like every other skill. The first real A2A work — earlier notes named it but nothing shipped. tags: [runtime, serve, interop, a2a, skills, standards] status: proposed --- # A2A interop — every topology is an A2A agent, and a remote A2A agent is a skill **Scope:** `packages/runtime` (serve routes, skills), `packages/schema` (skill implementation type) **Design reference:** §18 (MCP integration — A2A is its agent-level peer), §12 (skills as the only extension primitive), §8.7 (gates are structural) **Status:** proposed ## Where this actually stands (read first) A2A has been *named* in this repo since v0.2 and implemented nowhere. What exists today: - The v0.6 doc says leaders "communicate laterally via A2A" and folds A2A handoffs into coordination skills. - `reference/skills/peer-handoff.yaml` — an `llm_prompt` skill described as "the canonical coordination skill for A2A communication". It packages context as text. No protocol. - `adk-lessons.md` and `fleet-control-plane.md` say M18 "builds on the existing A2A adapter (`_delegation.py`)". **There is no A2A code in `_delegation.py`** — it is the compiler's internal child-delegation. That sentence was aspiration written as fact; both notes are corrected in this PR. - `IMPLEMENTATION-PLAN.md` M18: "A2A-as-coordination-skill (proposed)" — ⬜ not started. - Nothing under `server/`: no Agent Card, no `/.well-known/`, no task API, no client. So this note is the first design of the thing itself. **Shipped since (1.221.0):** slices 1 and 2 — `server/_a2a.py` (card builder, task↔job mapping, the JSON-RPC handler) and `server/_routes_a2a.py` (the well-known card, `/a2a`, `/a2a/{topology}`, SSE for `message/stream` / `tasks/subscribe`), `server.a2a` in the workspace schema, the `input-required` gate rule, `features.a2a` on `/capabilities`, `tests/test_a2a_serve.py`. **Shipped since (1.222.0):** slice 3, the client — `implementation.type: agent` in the skill schema; `swarmkit_runtime/agent_skill/` (spec + load-time target check, the permission seam, the A2A client, the executor with `on_unanswerable: agent | relay | abort`); `WorkspaceRuntime.run_child` (a nested in-process run that shares the parent's MCP servers, restores its trace/compression scope, writes a `parent_job_id` job row, depth-capped); `tests/test_agent_skill.py`, including SwarmKit calling SwarmKit over our own A2A server. Two departures from the text below, recorded where they apply: the policy key is `on_unanswerable` (the harness adapter's word), and a relayed question waits bounded rather than parking the run. **Shipped since (1.223.0):** `pack:workspace` — `agent_skill/_synthesis.py` synthesizes `topology-` per topology at registry build; `pack:workspace` expands to all of them (not read-filtered — running a topology is never `read`); the command-pack id `workspace` is reserved. **Shipped since (1.224.0):** the portal's Connections page — remote agents listed next to servers and sinks; "Add remote agent" probes a card through `GET /api/a2a/probe`, the person picks a skill / policy / credential / tier, and the `agent` skill file is written through the ordinary `PUT /api/skills/{id}`. `GET /api/a2a/agents` lists them. **Shipped since (1.225.0):** harness nodes — `mcp/_gateway.py` offers a granted `agent` skill as the flat tool `agent__` and runs its executor inside the run scope captured at registration (`contextvars.copy_context`), so a harness delegating to another topology or a remote agent gets the same child-run attribution, depth bound and audit as a model node. Still design: the fleet listing of cards; defer-while-awaiting-a-remote-answer. ## Why now A2A is a Linux Foundation standard (spec 1.0, 150+ organisations); Google ADK, LangGraph, CrewAI and kagent agents discover and call each other through it. SwarmKit topologies are reachable today as **MCP tools** (`POST /mcp`, one tool per topology) — the tool-level integration. The agent-level integration is missing, and it is the one that lets a SwarmKit swarm sit *inside* someone else's agent graph, or orchestrate their agents, without either side adopting the other's framework. For a self-hosted runtime with no distribution channel, being callable from every other framework is the adoption path. ## Goal 1. **Server:** `swarmkit serve` publishes an A2A Agent Card and answers the A2A task API for each topology, mapped onto the existing job model — so a run started over A2A is governed, audited, gated and resumable exactly like one started over `POST /run`. 2. **Client:** another agent is a skill — `implementation.type: agent` — resolving either to a **topology in the same workspace** (run in-process as a child job; no wire) or to a **remote A2A agent** (by card URL). Either way the calling agent reaches it through the same permission seam every other skill goes through, and the call is streamed and audited like every other skill. This is also the "sub-swarm as a skill" M18 proposed and never built: it falls out of the remote form, because both are "start a run, wait, hand back the artifact". ## Non-goals - **Not a new execution path.** A2A is a *transport* onto jobs. No second job store, no second gate mechanism, no second audit trail. If a feature would need those, it is not this note. - **Not gRPC or push notifications in the first cut.** JSON-RPC 2.0 over HTTP + SSE streaming is what the reference implementations speak; gRPC and `tasks/pushNotificationConfig/*` return `UnsupportedOperationError` until asked for. - **Not agent-card signing** in the first cut. The card is served over the instance's own auth; signatures follow when a fleet needs cross-org trust. - **Not replacing `peer-handoff`.** It stays the text-packaging skill; another agent is a different thing (a *capability*, category `capability`), not a coordination skill. - **Not runtime discovery.** No topology scans a network, a registry or a directory for agents at run time — see "Discovery" below for why and for what discovery *is*. - **Not automatic reachability.** Every topology in a workspace is *advertised* on the card, but an agent can only call the ones it has been *granted* as skills — see "Every topology is callable; none is granted by default". ## API shape ### Server: the Agent Card `GET /.well-known/agent-card.json` on a serve instance returns one card for the **instance**, with one A2A *skill* per topology — the same shape the MCP mount uses (one tool per topology). A2A's `skills[]` are advertised capabilities, which maps exactly; A2A does not require one card per agent, and one card per topology at `/a2a/{topology}/.well-known/agent-card.json` is offered too for clients that want a single-purpose agent. ```json { "name": "swarmkit — my-workspace", "description": "SwarmKit serve instance; each skill is a governed topology run.", "url": "https://host:8000/a2a", "version": "1.221.0", "provider": { "organization": "…from workspace.yaml server.identity…" }, "capabilities": { "streaming": true, "pushNotifications": false, "extendedAgentCard": false }, "securitySchemes": { "bearer": { "type": "http", "scheme": "bearer" } }, "security": [{ "bearer": [] }], "defaultInputModes": ["text/plain", "application/json"], "defaultOutputModes": ["text/plain", "application/json"], "skills": [ { "id": "code-review", "name": "code-review", "description": "…topology description…", "tags": ["swarmkit", "topology"], "examples": ["Review PR #49 on delivstat/swarmkit"] } ] } ``` `securitySchemes` is derived from the instance's configured auth provider (`none` → no scheme; `api_key` → bearer; `jwt` → bearer with the issuer's URL). The card says what the server already enforces; it never widens it. ### Server: the task API `POST /a2a` (JSON-RPC 2.0). The skill id in the message's metadata — or `/a2a/{topology}` for the per-topology endpoint — selects the topology. Mapping, and it is a mapping, not new logic: | A2A | SwarmKit | |---|---| | `message/send` | `POST /run/{topology}` with `input` = concatenated text parts, `attachments` = file parts (inline `bytes` only — a `uri` file part is refused with `ContentTypeNotSupportedError`, the runtime does not fetch caller-supplied addresses; the 20 MB and image-only rules of `attachments.md` apply unchanged), `correlation_id` = `contextId` (kept verbatim in the `a2a.context_id` label), `source` = `a2a`; the **task id is the job id**; returns the Task | | `message/stream` | the same submit, then `GET /jobs/{id}/stream` re-emitted as A2A `TaskStatusUpdateEvent` / `TaskArtifactUpdateEvent` over SSE | | `tasks/get` | `GET /jobs/{id}` (+ the artifact when completed, as a `data`/`text` part) | | `tasks/list` | `GET /jobs/history` filtered to `source: a2a` (the same column that says `serve` / `cli` / `chat`, so the portal's Source field reads it too) | | `tasks/cancel` | `POST /jobs/{id}/stop` — cooperative, at the next agent boundary; the task reports `canceled` only when the job reports `stopped` | | `tasks/subscribe` | `GET /jobs/{id}/stream` from the current position | | `tasks/pushNotificationConfig/*` | `UnsupportedOperationError` (first cut) | | `agent/getExtendedAgentCard` | the same card (no extended card yet) | Task state, from job status: | job | A2A task state | |---|---| | `pending` | `submitted` | | `running` | `working` | | `deferred` (parked on a human gate) | **`input-required`** — see below | | `completed` | `completed`, artifact attached | | `failed` | `failed` | | `stopped` | `canceled` | | refused at submit (422: bad attachment, unknown topology, auth) | `rejected` / JSON-RPC error | **The one place the mapping is not mechanical: `input-required`.** A2A's `input-required` means "the *caller* can supply what is needed". A SwarmKit run parks on a **human** gate — a funnel's `approve` layer, a relay approval, a multi-party role task — and the human is *not* the A2A caller; approval scopes are structurally un-grantable to agents (§8.7), and an A2A client is an agent. So a deferred run is reported as `input-required` with a status message that says *what* it is waiting for and *who* can resolve it (`GET /gates/{gate_id}` is linked in the message metadata), and a follow-up `message/send` on that task from the A2A caller does **not** resolve the gate — it is refused with `UnsupportedOperationError` and the gate's URL. The human resolves it through `swarmkit review` / the portal / `POST /review/{id}/…` as today; the task then moves to `working` and the caller's `tasks/subscribe` sees it. This is the whole reason A2A rides on the job model rather than beside it: the gate cannot be talked past over a new transport. The exception is a harness **input request** (§6.3, "what do you want?") — that *is* a question the caller may be able to answer. It is reported as `input-required` too; a `message/send` on the task with a text part is routed to `POST /review/{id}/answer` **only if** the pending item is an `input_request` and the A2A principal's scopes include `serve:review:answer`. Approvals never. Every A2A call is audited as the same `run.*` / `gate.*` events with `actor` = the authenticated A2A principal and `transport: a2a` in the event's metadata. No new event kinds. ### Client: another agent as a skill (`implementation.type: agent`) One skill type, two resolutions. Local when the target is a topology in this workspace; remote when it is a card URL. ```yaml apiVersion: swarmkit/v1 kind: Skill metadata: id: research name: Research agent description: Delegates a research question to the research agent. category: capability implementation: type: agent # exactly one of: topology: deep-research # same workspace → child job, no wire card_url: https://research.internal/.well-known/agent-card.json # elsewhere → A2A transport skill_id: deep-research # remote only: which A2A skill; omit = the card's first credentials_ref: research-agent # remote only: workspace credential, resolved by the credential service timeout_s: 600 on_unanswerable: agent # agent | relay | abort — see below max_agent_answers: 2 permission: cautious # the tier; no server or pack to inherit one from effects: read # read | write | unknown — readonly allows only read ``` The compiler turns it into a tool the agent can call: arguments `{input: str, context?: dict}` (and `{task_id, answer}` to answer a question — below), result the task's final artifact as text/JSON. It goes through the **same permission seam** as an MCP tool or a command — `permission` tiers, `requires:` prerequisites, `readonly` effects, audit of the call — because it is a skill (§12). Nothing about "it is an agent on the other end" changes what the caller is allowed to do. **Local form.** `topology:` starts the target as a **child job** (`parent_job_id` = the caller's job, same `correlation_id`), in-process, through the same job manager `POST /run` uses. It is not a network call to yourself: no bearer token, no serialisation, and `swarmkit trace` shows the child's agents, tools and tokens nested under the tool call. A child that parks on a gate is `input-required` to its caller exactly like a remote one. **Remote form.** `card_url:` is fetched at **resolve time** (cached, with the card's `version`), `skill_id` checked against `skills[]`, the security scheme read; `swarmkit validate` reports an unreachable card or an unknown skill before a run does. At run time the call is `message/stream` (falling back to `message/send` + `tasks/subscribe` when the card says `streaming: false`), with the input as text parts and any attachments as file parts. ### Every topology is callable; none is granted by default The instance card lists every topology, so every topology *is* an A2A agent to the outside. Inside the workspace, nothing is reachable until it is granted: an agent's capability set is what its topology declares, and if every topology were implicitly callable by every agent then `validate --require`, `readonly` tiers, `requires:` and the audit's answer to "what could this agent do" would all stop meaning anything. This is the same rule MCP servers already follow — declared in `workspace.yaml` makes a server *available*; a skill makes a tool *granted*. Granting is one line. **`pack:workspace`** is one `agent` skill per topology in the workspace, `topology-`, *synthesized at registry build* the way command-pack skills are (no files to regenerate or rot) — so "the supervisor may run any topology here" is `skills: [pack:workspace]`, the way `pack:git` grants a bundle's skills today, and `validate` still knows exactly what the agent can reach. It is not read-filtered like a command pack (running a topology is never `read`); every call still passes the cautious tier and the audit. ### When the other agent asks a question (`on_unanswerable`) A remote agent (or a local child) can go `input-required`. This is the harness input request (§6.3) arriving over a wire, and it uses the **same machinery** and the **same words**, not new ones: the key is `on_unanswerable`, as on a harness adapter, and `relay` / `abort` mean there what they mean here. `agent` is the one value a harness does not have: | policy | behaviour | |---|---| | `agent` (default) | The question comes back to the **calling agent** as the tool result — `{"status": "input_required", "task_id": …, "question": …}`. The agent framed the task, so a clarification ("which repo?", "PDF or markdown?") is its call: it answers by calling the skill again with `{task_id, answer}`. Each answer is audited as `answered_by: agent:` and counts against the turn budget. After `max_agent_answers`, or when the agent decides it is not its call (the existing `escalate-to-human` coordination skill, or a `relay` result), it escalates. | | `relay` | Always a human, never the agent. | | `abort` | The tool call fails with the question as the reason — for unattended batch runs. | **Escalation to a human** files an **`input_request`** review item carrying the question, the task id, which skill asked, and the context, and waits for the answer — the same inbox as a harness question (`swarmkit review list`, `POST /review/{id}/answer`, the portal, the fleet cockpit) and the same bounded wait (`resolve_input`: a person answers within the window or the call fails; it never hangs). The answer is sent as `message/send` on the remote task and the call continues. Rejecting the item cancels the remote task and the tool call fails with the reason. *As shipped, the caller's run stays resident while it waits.* Parking it instead — checkpoint, job `deferred`, resume with the tool result when the answer arrives — is the right end state for a remote agent that may take hours, and is the same change a long harness question needs; it is a follow-up on both, not a difference between them. **No funnel on the skill.** A funnel gates an *agent's* production. A local child runs its topology's own funnels in-process; a remote SwarmKit runs its own on its side; and the calling agent's funnel gates whatever it does with the result. A third gate on the skill would be the same gate in a new place. **What the agent may never answer, whatever the policy.** The lines that already exist for tools: on a `strict` tier the answer is a write-shaped act → relay. A question that is really an *approval* ("may I delete the branch?") is a governance decision, not a clarification → a human; detected the honest way — the answer would grant an effect the skill's `effects` map marks as a write. And an answer is a text part, never a credential or a scope, so a remote agent cannot obtain either by asking. Clarifications go to the agent, bounded and audited; decisions go to a human, structurally — the same distinction §6.2/§6.3 draw for harnesses. This makes the rule symmetric across the wire: a harness asking "what do you want?", a remote agent asking the same, and **our own run** parked on a gate while *serving* an A2A call all appear as `input-required` / `input_request`, all resolved through the review queue, all audited. ### Streamed and audited like every other skill - **Audit.** The call is the existing `tool.call` event — arguments, result size, duration, actor, the run's correlation — with the A2A specifics in metadata: card URL, remote skill id, task id, `contextId`, final task state. Every `input-required` round is its own event (`answered_by` agent id or human identity); a cancel is recorded. The local form is a child job, so its whole interior is in the trace under `parent_job_id`. No new event kinds. - **Streaming.** A2A's `TaskStatusUpdateEvent` / `TaskArtifactUpdateEvent` from `message/stream` are forwarded into the caller's `GET /jobs/{id}/stream` as progress events under the tool-call span — the run view and `swarmkit logs` show "remote agent: working… artifact… completed" live, as a harness node's JSONL events do today. Local children stream natively. Push notifications are not needed for this. - **The limit, stated.** We record what crosses the wire. A remote agent's own tool calls, tokens and cost are not visible unless it reports them in task metadata (A2A does not require it), so the trace shows the remote call as one node with duration and result and cost "unknown" unless reported — the same honesty the trace applies to a harness's internal model calls. Two SwarmKit instances talking A2A can both report to the fleet control plane, which can join them by `contextId` — that is where "one run rendered across instances" earns its name. ### Discovery Two levels; only the first belongs in the runtime. 1. **Resolving a declared agent** — the skill names a `card_url`; the runtime fetches the card (RFC 8615 well-known URL) at resolve time, checks the skill id, reads the security scheme, caches it. That is discovery in the A2A sense, and it is in scope. 2. **Finding agents nobody declared** — scanning a network, a registry or a directory at run time — is **not** something a topology does. A swarm that can discover and call arbitrary agents has a capability set unknown until it runs, which defeats `validate --require`, `readonly` effects and the premise that the artifact says what it can do. Discovering a *new* agent is a **human authoring act**: the portal's Connections page (where MCP OAuth servers are added today) gets "Add an A2A agent by card URL" → shows the card's skills → the person picks one → a skill file is written, the way a `swarmkit-skills` bundle is imported. The fleet control plane may aggregate its own instances' cards so cross-instance delegation is a pick-list, not a scan. ### Schema `skill.schema.json` gains an `agent` implementation variant: exactly one of `topology` / `card_url`; `skill_id`, `credentials_ref`, `timeout_s`, `on_unanswerable` (`agent | relay | fail`), `max_agent_answers`. `docs/notes/schema-change-discipline.md` applies: schema + Python + TS validators + fixtures (valid + invalid) + codegen drift check. The composer's skill form is schema-driven and gains the variant with the schema change (as `command` did). ### Config ```yaml # workspace.yaml server: a2a: enabled: true # default false — one more surface is opted into, like mcp.enabled identity: organization: delivstat url: https://delivstat.com ``` `swarmkit system` reports whether A2A is mounted; `/capabilities` lists `a2a: true` so the fleet panel can show it. ### Portal Deliberately small. System page: one row ("A2A: enabled, card at …"). Review inbox: **no new kind** — a run that arrived over A2A and parked on a gate appears as any deferred run does; the run card shows `transport: a2a` and the caller principal, which the audit event already carries. Runs/trace: a job; a local child appears nested. Workspace config: `server.a2a.enabled` through the existing `PUT /api/workspace/config/{section}` path. Skill editor: the `agent` variant, schema-driven. Connections page: "Add an A2A agent by card URL" (Discovery §2). Fleet panel: an A2A capability chip from `/capabilities`. ## Test plan - **Unit — card:** built from a resolved workspace; one skill per topology; `securitySchemes` follows the auth provider; disabled → 404 at the well-known path. - **Unit — mapping:** every job status maps to exactly one task state (a parametrised test over the `Literal`, so a new job status fails here); a deferred job is `input-required` with the gate URL in metadata; a `message/send` on a deferred task is refused and the gate is untouched (assert on the review queue). - **Unit — JSON-RPC:** each method, malformed request → -32600, unknown method → -32601, unknown task → `TaskNotFoundError`, unsupported → `UnsupportedOperationError`. - **Integration — server:** submit over A2A on the hello-swarm workspace with `provider: mock`; `message/stream` yields status events ending in `completed` with the artifact; `tasks/cancel` ends in `canceled`; the audit log has the same events as a `POST /run` of the same topology, plus `transport: a2a`. - **Integration — client, remote:** a fake A2A server (the same routes, in-process) behind an `agent` skill; the agent's tool call returns the remote artifact; `readonly` tier refuses it when `effects` says write; unreachable card or unknown skill id is a `validate` error; streamed status/artifact events appear in the caller's job stream under the tool-call span. - **Integration — client, local:** `topology:` runs a child job with `parent_job_id`; the trace nests it; a child parked on a funnel gate surfaces as `input_required` to the caller; the `pack:workspace` skill pack lists every topology and regenerates on `validate`. - **Input requests:** `agent` policy — the question reaches the calling agent, its answer is sent on the task and audited as `answered_by: agent:`, the budget caps it and the next round escalates; `relay` — the run parks `deferred`, an `input_request` item is filed, a human answer resumes it, a rejection cancels the remote task; `abort` — the tool fails with the question; a `strict` tier or a write-shaped answer always escalates whatever the policy. - **Audit fields:** the `tool.call` event for an `agent` skill carries card URL, skill id, task id, `contextId`, final state; each input round is an event; no new event kinds (a test over the audit schema's kind list). - **Conformance:** the official A2A test client (`a2a-sdk` Python) discovers the card, sends a message, streams, gets, cancels — run in CI against the mock-provider workspace. - **Schema:** fixtures for the new implementation type, valid and invalid; TS + Python validators agree. ## Demo plan - `just demo-a2a`: starts serve with `a2a.enabled`, prints the card, runs the official A2A CLI client against `code-review`, shows the run in `swarmkit trace`, then runs a topology whose agent calls the same instance back over A2A as a skill (self-loop is the cheapest cross-agent demo). A recorded transcript in the PR. - The stretch demo for the blog: an **ADK** agent (`google-adk`, a few lines) calling a SwarmKit topology it discovered from the card — the sentence "an ADK agent just ran a governed SwarmKit swarm and waited on its human gate" is the whole post. ## Slices 1. Card + `message/send` + `tasks/get` + `tasks/cancel` (server). Conformance client passes the non-streaming subset. 2. `message/stream` + `tasks/subscribe` over SSE; `input-required` mapping with the gate rule. 3. `agent` skill: local form (child job) first, then remote (card resolution, streaming), `on_unanswerable` with the review-queue relay; schema; `pack:workspace`. 4. Fleet: `/capabilities` reports A2A; the panel lists cards. Card signing when a real need appears. ## Open questions - **One card per instance vs per topology.** Both are offered above; if the ecosystem's clients turn out to assume one card = one agent, drop the instance card. - **`contextId` ↔ `correlation_id`.** A2A's `contextId` groups turns of one conversation; SwarmKit's `correlation_id` groups runs of one ticket. Same idea, but a multi-turn A2A conversation on one `contextId` should probably map to a SwarmKit *conversation* (`POST /conversations`) rather than a fresh run per message. First cut: one run per message; revisit with a real multi-turn caller. - **Which artifact is "the" A2A artifact** when a run saves several (`--save-artifact`): the root agent's final output as the task artifact, others listed in metadata by ref. - **§21:** does A2A-as-transport change the sandboxing question? No — the callee is still a local run; the caller is a principal. Noted for the next design revision. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/a2a-federation.md ## Published: https://delivstat.github.io/swarmkit/design-notes/a2a-federation/ ══════════════════════════════════════════════════════════════════════════════ --- title: A2A federation — SwarmKit agents that hand back their record description: When both sides of an A2A call are SwarmKit, the callee returns its run id, token/cost usage and an observability pointer, the caller stitches them into its own trace and audit, and the card says so at add time. A budget passed on the call is honored by a SwarmKit callee. tags: [a2a, interop, observability, audit, budget] status: accepted --- # A2A federation — SwarmKit agents that hand back their record ## The gap An `agent` skill can call a remote agent through its A2A card (`a2a-interop.md`). The call returns an answer — and nothing else. The caller's trace stops at "called `agent__x`, got text"; its audit has no link to what the remote run did; its `GET /usage` does not count the tokens the remote spent; and a budget on the calling node does not reach the remote at all. Across the network the call graph, the cost and the record all go dark at the boundary (`nested-call-limits.md` frames the same problem for depth and budget). We cannot fix that for an arbitrary remote — a non-SwarmKit agent, or one we do not run, owes us nothing. **But when the callee is also SwarmKit, it already has all of it**: a run id, a durable audit journal (`audit-event-journal.md`), per-run token/cost totals on the job row, and a `GET /events`/`GET /audit` surface. The gap is that none of it crosses back. This closes that, between SwarmKit instances, over the A2A metadata channel — no protocol extension a non-SwarmKit agent has to understand, just extra keys it ignores. ## Goal 1. **Identify.** A SwarmKit instance's Agent Card declares a federation extension, so adding the agent (portal probe, `GET /api/a2a/probe`) can show "this is a SwarmKit agent" and what it will hand back. 2. **Return.** A SwarmKit callee puts its `run_id`, token/cost `usage`, and an observability pointer into the task's `metadata.swarmkit`. The caller reads them, records an `a2a.remote_usage` audit event linking the two runs, and can pull the remote's events/audit by `run_id` from the same endpoint with the same credential. 3. **Budget** (slice 2). The caller passes its remaining allowance in the message metadata; a SwarmKit callee installs it as the child run's budget. Advisory across the trust boundary — a SwarmKit callee honors it, anyone else ignores it, and a returned total is the callee's claim, not an audited fact on our side. ## Non-goals - Not a guarantee about non-SwarmKit agents, and not trust in a remote's numbers beyond "recorded as reported". The `a2a.remote_usage` event is attributed to the remote and labelled as its claim. - Not returning the remote's audit *content* across the boundary. We return a pointer (`run_id` + the fact that `GET /events`/`GET /audit` answer there); pulling it still takes the caller's own authorization against that endpoint. A run's events can quote a document that can quote a secret; they do not get shipped in a task result. - Not a new transport. Everything rides A2A `metadata`, which the spec leaves open. ## Design ### The card says it (identify) `build_agent_card` adds an A2A extension under `capabilities.extensions`: ```json { "uri": "https://swarmkit.dev/a2a/federation/v1", "description": "Returns run id, token/cost usage and an observability pointer per task; honors a passed budget.", "params": { "runtime": "1.240.0", "returns_usage": true, "returns_observability": true, "honors_budget": true } } ``` A non-SwarmKit client ignores `extensions`. `fetch_card` parses it into `AgentCard.swarmkit` (None when absent). `GET /api/a2a/probe` returns `is_swarmkit` + `runtime` + the flags, and the portal's "Add remote agent" shows a **SwarmKit** badge. The written skill needs no schema change: the card is fetched and cached on first use, so the runtime knows at call time whether to expect the rich round-trip. ### The callee returns it `task_from_job` adds to `metadata.swarmkit`, on a terminal task: ```json "run_id": "", "usage": { "input_tokens": 1234, "output_tokens": 567, "cost_usd": 0.0189 }, "observability": { "events": true, "audit": true } ``` `run_id` is the callee's job id (also the A2A task id, but named explicitly). `usage` is the job row's `usage_*` totals, omitted when zero/absent. `observability` says the caller can `GET /events?run_id=…` and `GET /audit` at this endpoint — a hint, not the data. ### The caller stitches it In `_call_remote`, once the task is terminal, read `task.raw["metadata"]["swarmkit"]` and emit: ``` a2a.remote_usage { card, skill_id, remote_run_id, input_tokens, output_tokens, cost_usd, endpoint, source: "reported" } ``` So the caller's own audit — durable, queryable by its run id — carries the link to the remote run and its cost, attributed as the remote's report. `swarmkit logs`/`GET /audit` on the caller then shows the cross-boundary hop, and a later `GET /usage` roll-up can include reported remote spend kept distinct from locally-metered spend. ### Budget forward (slice 2) The caller adds `message.metadata.swarmkit.budget = { max_cost_usd, max_turns, max_wall_clock_minutes }` — its remaining allowance, derived from the calling node's envelope minus what it has spent. `send_message` on a SwarmKit callee reads it and installs it as the child run's budget/circuit-breaker (the same seam `_begin_run` uses). This is the cross-instance half of `nested-call-limits.md`: the local depth/budget context cannot cross the network, but a value on the A2A envelope can. Enforcement is the callee's; the caller records what it asked for. ## Test plan - Two SwarmKit instances over an in-process ASGI transport (as `test_a2a_*` do): a caller `agent` skill calls a callee topology; assert the returned task carries `metadata.swarmkit.run_id` and `usage`, and the caller's audit has an `a2a.remote_usage` event naming the callee's run id. - `fetch_card`/probe: a SwarmKit card yields `is_swarmkit=True` + flags; a plain card yields False. - Slice 2: a budget in the message metadata reaches the callee run's tracker; a non-SwarmKit message (no budget) runs unbounded as before. ## Demo plan Extend the Level 20 (agents calling agents) tutorial: call one serve instance from another and show the caller's `swarmkit logs` carrying the remote run id and cost. `just demo-...` reuses the A2A both-ways demo workspace. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/declarative-model-providers.md ## Published: https://delivstat.github.io/swarmkit/design-notes/declarative-model-providers/ ══════════════════════════════════════════════════════════════════════════════ # Declarative model providers — a provider is data, a wire format is code **Status:** shipped — runtime 1.219.0, schema 1.41.0. Reference page: `docs/site/reference/model-provider.md`. **Precedent:** [`executor-declarative-adapters-plan.md`](executor-declarative-adapters-plan.md) — *"a new harness is added as data, with no Python and no runtime release."* This is the same decision, one seam over. ## Goal Add a model provider by dropping a YAML file into a workspace, or into the bundled library, with no Python and no runtime release. Let a provider inherit from an existing one and override only what differs. ## Why now Three of the eight providers are already data pretending to be code: ```python class OpenRouterModelProvider(OpenAIModelProvider): # base_url + OPENROUTER_API_KEY class GroqModelProvider(OpenAIModelProvider): # base_url + GROQ_API_KEY class TogetherModelProvider(OpenAIModelProvider): # base_url + TOGETHER_API_KEY ``` Each is a subclass whose entire content is a URL and an env-var name. And the endpoint is not configurable at all — `register_available_providers` does `OllamaModelProvider()` with no arguments, so the only way to point SwarmKit at a different Ollama-API server today is to run it on port 11434 in place of Ollama. Meanwhile the edge-inference world converged on two wire formats. Every accelerator runtime worth targeting exposes one: | runtime | silicon | speaks | | --- | --- | --- | | rkllama, rkllm-server | Rockchip NPU | Ollama API / OpenAI-compatible | | OpenVINO Model Server | Intel CPU · iGPU · NPU | OpenAI-compatible | | llama.cpp `llama-server` | any CPU; Vulkan/OpenCL GPU incl. Mali | OpenAI-compatible | | vLLM, TensorRT-LLM | NVIDIA / AMD | OpenAI-compatible | | mlx-lm | Apple Silicon | OpenAI-compatible | | Lemonade | AMD Ryzen AI NPU | OpenAI-compatible | None of these needs a provider. All of them need a URL. Today each would be a Python subclass and a release; after this, each is six lines of YAML in a directory. ## Non-goals - **A template language.** The DSL is deliberately minimal and says so. Anything past its ceiling declares `requires: code` and becomes a family (below), exactly as the executor DSL graduates to a Tier-1 Python executor. - **OAuth, service accounts, token refresh.** Every provider that exists uses one auth shape — a static key in a header — and nothing has asked for another. Azure's *key* mode differs only in the header name, which the DSL covers. Entra ID tokens and Vertex AI service accounts are genuinely different (acquisition, refresh, expiry) and are the `requires: code` case if and when. - **Widening capabilities.** A YAML may *narrow* what its family offers (`tools: false` for a runtime that half-supports them). It may never claim a capability the family does not implement — that is how a silent wrong answer reaches an audit log. - **Changing invariant 4.** *Only `model_providers/` imports vendor SDKs.* The YAML parameterises a family; it never imports anything. ## The line: a wire format is code, everything else is data What varies between providers, sorted by what it actually is: | varies | data? | | --- | --- | | base URL | yes | | auth — env var, header name, scheme | yes | | model catalogue — what `supports()` accepts | yes | | capabilities — images, tools, streaming, structured output | yes, **narrow-only** | | request quirks — Ollama lifting `think`/`keep_alive` to the payload root; OpenRouter's `usage: {include: true}` | yes, as a declared list | | extra headers — `HTTP-Referer`, `X-Title` for OpenRouter | yes | | **wire format** — OpenAI chat-completions · Anthropic messages · Google · Ollama-native | **no — code** | There are **four wire formats**. They stay Python, renamed **families**: `openai-compatible`, `anthropic`, `google`, `ollama`. A provider YAML names one via `extends` and parameterises it. ## The artifact: `kind: ModelProvider` ```yaml apiVersion: swarmkit/v1 kind: ModelProvider metadata: id: groq name: Groq description: OpenAI-compatible inference with very fast tokens. spec: extends: openai-compatible base_url: https://api.groq.com/openai/v1 auth: api_key_env: GROQ_API_KEY provenance: authored_by: human version: 1.0.0 ``` Same envelope as every other artifact — `apiVersion`, `kind`, `metadata`, `spec`, `provenance` — so the validator, the portal and the codegen treat it like one. That is the whole of `GroqModelProvider`. The rkllm case, which today needs a release: ```yaml apiVersion: swarmkit/v1 kind: ModelProvider metadata: id: rkllama name: rkllama — Rockchip NPU description: Ollama-API server for RK3588/RK3576 NPUs, running pre-converted .rkllm models. spec: extends: ollama base_url: ${RKLLAMA_HOST:-http://localhost:8080} capabilities: tools: false # narrows the family; may never widen it provenance: authored_by: human version: 1.0.0 ``` ### `spec` fields | field | type | notes | | --- | --- | --- | | `extends` | id | **required.** A family, or another provider. Resolved at load; one level of override-merge per hop; cycles refused by name | | `base_url` | string | `${VAR:-default}` substitution, same resolver as everything else | | `auth.api_key_env` | string | absent means no auth — a local runtime | | `auth.header` | string | default `Authorization`; Azure's key mode is `api-key` | | `auth.scheme` | string | default `Bearer`; empty string for a bare key | | `models.pattern` | regex | what `supports()` accepts. Default from the family | | `models.accept_any` | bool | aggregators: the catalogue is unbounded and the server validates | | `capabilities.*` | bool | `images`, `tools`, `streaming`, `structured_output`. **Narrow-only** — a `true` the family does not offer is a load-time error | | `options.lift_to_root` | list | option keys moved from `options` to the payload root — Ollama's `think`, `keep_alive` | | `headers` | map | static extra headers | | `requires` | `code` | declares the provider cannot be expressed here. Load refuses it with the reason, so a YAML past the ceiling fails loudly rather than half-works | ### Inheritance `extends` names a family or another provider. Resolution walks the chain to a family, merging each hop's fields over its parent's. `auth`, `models`, `capabilities`, `headers` and `extra_body` merge by key; `base_url` and `lift_to_root` replace. A chain that does not end at a family, or that revisits an id, is refused at load with the chain printed. **Families and providers share one namespace, and the family wins.** Four bundled providers carry their family's name — `anthropic` the provider extends `anthropic` the family. So the first hop is always the provider named, and every `extends` after it resolves to a family before a provider: `rkllama`'s `extends: ollama` is the wire format, never the bundled `ollama.yaml`. (Found by the bundled-library test: a walk that consulted the family table first returned the bare family for `anthropic`, its `auth.api_key_env` was never read, and nothing registered.) **A field the family would ignore is refused, not dropped.** `extra_body` on `anthropic`, `lift_to_root` on `openai-compatible`, `auth.header` on `google` — each family owns its wire format, and a declared quirk that never reaches the wire is the half-working provider this note exists to prevent. `FAMILY_FIELDS` in `_declarative.py` is the table. **A narrowed capability is enforced, not decorative.** `tools: false` refuses a request carrying tools before the wire, naming the provider and the field; `images: false` likewise; `streaming: false` refuses `stream()`; `structured_output: false` withholds `response_format` and flips `enforces_response_schema`, so the compiler pastes the schema into the prompt instead. Depth is unbounded but every bundled provider is one hop. That is a review norm, not a schema limit: a provider three hops from its family is one nobody can read. ## Loading and registration Mirrors `executors/_declarative.py` exactly: ``` bundled packages/runtime/src/swarmkit_runtime/model_providers/providers/*.yaml workspace /providers/*.yaml overrides a bundled id ``` **Registration is by readiness, not by a hardcoded list.** `register_available_providers` today keeps a list of `(ENV_VAR, ClassName)` pairs. It becomes: every loaded provider registers if its `auth.api_key_env` is set, or if it has no auth. The list disappears. A provider without auth — every local runtime — registers unconditionally, which is the current Ollama behaviour, made general. `swarmkit providers list` shows every provider, its family, its source and whether it is ready — the one question an operator has when `provider: groq` is not registering. `swarmkit providers show ` prints the resolved chain. ## Migration `openrouter`, `groq` and `together` are rewritten as bundled YAML and their Python subclasses deleted. **Behaviour-identical, asserted**: a test constructs each from YAML and from the deleted class and compares the resulting client configuration. `anthropic`, `openai`, `google` and `ollama` gain a YAML each too, so every provider is described one way — but their families stay as the code they parameterise. The provider ids do not change. A workspace saying `provider: groq` works before and after. ## What changes, file by file **New** - `packages/schema/schemas/model-provider.schema.json` — follows `docs/notes/schema-change-discipline.md`; fixtures under `packages/schema/tests/fixtures/model-provider/` - `model_providers/_declarative.py` — `ProviderSpec`, `parse_provider_spec`, `resolve_chain`, `load_provider_specs(workspace_root)`, `build_provider(spec)` - `model_providers/_family.py` — `FamilyBase`: the constructor parameters every family accepts, `supports()` from the catalogue, and the capability checks - `model_providers/providers/*.yaml` — the bundled library: the existing eight, plus `rkllama`, `openvino-model-server`, `llama-server`, `mlx-lm`, `lemonade` - `tests/test_declarative_providers.py` **Changed** - `_openai.py`, `_ollama.py`, `_anthropic.py`, `_google.py` — each becomes a family: accepts `base_url`, `auth`, `headers`, `lift_to_root`, `capabilities`, `models` as constructor parameters. No behaviour change when called with none of them. - `_openai_compat.py` — deleted; its three classes are YAML now - `_workspace_runtime.register_available_providers` — loads the library and registers by readiness; takes the workspace root so workspace providers load - `_registry.provider_enforces_response_schema` — answers through the YAML (family, narrowed) rather than a class scan - `cli/_cmd_providers.py` — `swarmkit providers list|show` - `docs/`, `llms.txt`, `llms-full.txt` — the provider list and how to add one **Unchanged, deliberately** - `ModelProviderProtocol` — `complete()` and `supports()`. A YAML-built provider satisfies it the same way. - Invariant 4. ## Test plan - **Parse and refuse.** Every field; a missing `extends`; a chain that never reaches a family; a cycle; `capabilities.tools: true` on a family without tools; `requires: code`. - **Inheritance merges by key.** A child overriding `auth.header` keeps the parent's `api_key_env`. - **Migration is behaviour-identical.** `groq`, `openrouter`, `together` from YAML produce the same client `base_url`, key source and extra headers as the deleted classes. - **Registration by readiness.** With `GROQ_API_KEY` unset, `groq` is absent; set, present. `ollama` and `rkllama` present regardless. - **Workspace overrides bundled.** A workspace `providers/ollama.yaml` with a different `base_url` wins. - **`${VAR:-default}` resolves** in `base_url`. - **The bundled library is valid** against the schema, and every bundled id is unique. ## Demo plan Point a workspace at an OpenAI-compatible server it has never heard of, with no Python: ```bash cat > workspace/providers/llama-server.yaml < StorageService: ... # One resolved target per kind, with its source recorded for the startup report. def target(self, kind: StoreKind) -> StoreTarget: ... # (backend, url, source) # Engines are cached per URL: one pool per database, not one per component. def engine(self, kind: StoreKind) -> Engine: ... def store(self) -> Store: ... # jobs, conversations, usage def saga_store(self) -> SqlSagaStore: ... def artifact_store(self) -> ArtifactStore: ... def audit_provider(self) -> AuditProvider: ... def memory_store(self) -> GovernedMemoryStore: ... def checkpointer(self) -> Any: ... def membership_store(self) -> MembershipStore: ... def report(self) -> list[str]: ... # one line per kind, for startup logging ``` `StoreKind` is an enum — `runtime | audit | checkpoints | artifacts | memory | saga | fleet` — so a component asks for a *kind*, never a path. ### Resolution, once Per kind, in order: 1. `SWARMKIT_STORE_BACKEND` / `SWARMKIT_STORE_URL` (or `DATABASE_URL`) — process-wide override. 2. `storage.` when that kind has its own block (`audit`, `checkpoints`). 3. `storage.runtime` — the workspace default for every SQL store. 4. SQLite under `{workspace}/.swarmkit/`. **A URL implies its backend.** `postgresql://…` resolves to `postgres` without `SWARMKIT_STORE_BACKEND`. Setting only a URL is the common case and currently the silent one. **A declared backend that cannot be honoured raises**, consistent with the rule established in 1.127.0: refusing to start beats writing to a different database than the one configured. That covers `checkpoints.backend: postgres` while `langgraph-checkpoint-postgres` is absent — with a message naming the missing extra rather than a generic failure. ### Engines are shared Today each component calls `make_engine` independently, so one workspace on Postgres opens a connection pool per component. The service caches by resolved URL, so components sharing a target share a pool. This is a side benefit, not the motivation — but it is the reason `engine(kind)` is on the service rather than each component holding its own. ### It reports what it chose Startup logs one line per kind: ``` storage: runtime=postgres (workspace.yaml) audit=postgres (workspace.yaml) checkpoints=sqlite (default) memory=postgres (workspace.yaml) artifacts=database→postgres saga=postgres (workspace.yaml) fleet=sqlite (separate by design) ``` The absence of this is why the bug survived: every symptom was an empty screen, which reads as "nothing ran" rather than "your data is elsewhere". ### It notices the split it just fixed On start, if a kind resolves to a non-SQLite target **and** a workspace-local SQLite file for that kind exists with rows, warn — naming both locations and the row count. Anyone upgrading into this change has data in the old place, and a silent cutover would look exactly like data loss. ## Migration of call sites Every construction site in the table above is replaced by a service call. The hardcoded ones lose their paths entirely; `SqliteStore(workspace_path)` stays only as an internal constructor the service uses, not as a public entry point. `swarmkit pipeline` and `swarmkit orchestrator` build the service from the resolved workspace, so a CLI invocation and serve agree by construction rather than by both remembering to. A lint-style test asserts no module outside `persistence/` contains `sqlite:///` or calls `make_engine` — the mechanical guard against this regressing, since it regressed three times. ## API shape ```python # serve storage = StorageService.for_workspace(workspace_path, runtime.workspace.raw) app.state.storage = storage app.state.store = storage.store() app.state.saga_store = storage.saga_store() # CLI — same two lines, so they cannot diverge storage = StorageService.for_workspace(workspace, resolve_workspace(workspace).raw) ``` ```yaml storage: runtime: { backend: postgres, url: "${SWARMKIT_STORE_URL}" } # default for every SQL store audit: { backend: postgres, retention_days: 90 } # inherits runtime's url checkpoints: { backend: sqlite } # opt a kind back out ``` Schema change: `storage..url` becomes optional everywhere and inherits `storage.runtime.url` when absent — repeating the same URL three times is what made the config look honoured. ## Test plan - **Unit.** Resolution precedence per kind; a URL alone implies its backend; a per-kind block overrides `runtime`; an unimplementable backend raises with the missing extra named; engines are shared per URL and distinct across URLs. - **The regression that started this.** A workspace declaring Postgres resolves *every* kind to Postgres — asserted per kind, so a newly added store that forgets the service fails the test. - **The mechanical guard.** No `sqlite:///` literal and no `make_engine` call outside `persistence/`. - **Split detection.** A non-SQLite target plus a populated local SQLite file warns, naming both. - **Full pipeline.** Live `swarmkit serve` + `swarmkit orchestrator` + `swarmkit pipeline emit` against Postgres, asserting rows land in Postgres and the workspace SQLite stays empty — the exact scenario that failed. ## Demo plan `just demo-storage` — one workspace, two runs: 1. `storage.runtime: postgres`, run a topology and a pipeline stage from the **CLI**, show `pipeline_saga`, `audit_events` and `governed_memory` all populated in Postgres and `.swarmkit/store.sqlite` absent. 2. Point `storage.checkpoints` at `postgres` without the extra installed and show the startup refusal naming it, rather than a silent SQLite fallback. Plus the startup report block, which is the artefact an operator actually reads. ## Resolved (implementation, 1.130.0) All four open questions were answered while building this. Two of them differently from the way they were framed. - **Migrating existing data — built.** `swarmkit storage migrate` copies the local SQLite rows into the configured Postgres: additive, idempotent (`ON CONFLICT DO NOTHING`, so a re-run after a partial failure resumes rather than duplicates), and it never deletes the source. The runbook is `docs/site/reference/storage.md`. The split-brain warning stays regardless — it is what tells you a migration is needed. - **`agt` as an audit backend — dropped from the enum.** Nothing implemented it; selecting it wrote SQLite. Leaving a third unimplemented value in place would repeat the exact mistake this note exists to fix. - **Postgres checkpointing — added as the `[postgres]` extra**, and `checkpoints` became the one store that does **not** inherit `storage.runtime`. This changed during implementation: with inheritance, `storage.runtime.backend: postgres` promoted the checkpointer too and the workspace failed at startup on a driver nobody asked for. Setting `SWARMKIT_STORE_URL` did the same thing through the environment branch — caught by the demo, not by a test, which is the argument for demos. A LangGraph component with its own install requirement has to be opt-in, so only an explicit `storage.checkpoints` block moves it. - **Fleet separation — reverted to following `storage.runtime`.** The note leaned toward *always* separate, and the implementation started there. That was wrong: design 19 Q4 already decided the membership store follows the main backend, and the shipped `create_membership_store` implements it. Regressing a shipped, design-referenced behaviour for tidiness is a worse trade than the consistency was worth. It keeps its own SQLite *file*, not its own *backend*. ## Two things this note did not anticipate - **`${VAR}` was never expanded in storage URLs.** The workspace schema documented `${ENV_VAR} interpolation` on `storage.runtime.url` and nothing in the runtime implemented it, so `url: ${SWARMKIT_STORE_URL}` — the form every deployment doc uses — reached SQLAlchemy as those literal characters. Expansion now happens in the service, which is the only place that reads these URLs. - **`storage.artifacts` was unusable.** The runtime has read it since the bundled orchestrator shipped, but the schema's `additionalProperties: false` rejected the very key the code looked for, so declaring it failed validation. Now declared. ## The operator surface The report is the point, not a side effect: a misconfigured workspace looked *identical* to an empty one, and no amount of correct resolution fixes that on its own. It is printed at serve startup, available as `swarmkit storage status` and `swarmkit system`, served at `GET /storage` and `GET /system`, and rendered on the web UI's **System** page — reachable from the screen that is empty. `GET /system` also lists the workspace's own `workspace.env.yaml` properties, read from the file on every request so a parameter a new feature adds appears without registration, and the runtime environment variables from a curated registry (never `os.environ`, which holds every API key the process uses). Values are masked by declaration — the reserved `secrets:` key — with a name heuristic as the fallback for workspaces written before it existed. ## Superseded open questions ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/audit-event-journal.md ## Published: https://delivstat.github.io/swarmkit/design-notes/audit-event-journal/ ══════════════════════════════════════════════════════════════════════════════ --- title: The audit is a write-through journal description: Persist each audit event the moment it is recorded, not in one batch at the run boundary, so a run that crashes (SIGKILL, OOM, power loss) still leaves the trail up to the crash. tags: [audit, observability, durability, governance] status: accepted --- # The audit is a write-through journal ## The gap Until 1.239.0 audit events were held in the governance provider's in-memory list for the whole run and written to the store in one batch at `_end_run`. `GET /events` — "the durable log an application reconciles from" — reads the same `audit_events` table, so both surfaces depended on the run reaching its boundary. A `kill -9` mid-run, an OOM, or a power loss wrote **nothing**: `test_kill9_recovery.py` shows the store empty right after a SIGKILL, while the LangGraph checkpoint (written per node) survived and the run resumed. Recovery was durable; the record of what happened before the crash was not. That inverts what an audit log is for. The append-only invariant (§8.3) is about *integrity* — no agent can update or delete an entry — and it was never violated. Durability was simply never stated, so batch-at-end (one pass, one redaction, no per-event write on a hot run) looked free. It is not free for the run a security team most wants to reconstruct: the one that crashed. ## Goal Each recorded event reaches the durable store when it is recorded. A hard kill loses at most the single in-flight event, not the run. `GET /events` and `swarmkit logs` show a crashed run's trail up to the crash. No new public API; the change is where the write happens, not what is written. ## Non-goals - Not per-event `fsync`. The floor is "written to the store before the next event", not "flushed to disk before the operation the event describes returns". SQLite WAL and Postgres both make the row visible to a reader immediately; surviving a kernel-level power cut mid-fsync is a storage-engine concern, not this seam's. - Not a second store. The journal writes to the same `AuditProvider` the batch used. - Not removing the end-of-run pass. It stays as a completeness safety net (see below), made a no-op for already-journaled events by the store's existing id dedup. ## Design Three facts already in place make this small: 1. **`AuditEvent` stamps `event_id` (uuid4) and `run_id` at construction** from the per-task run scope. So an event is self-identifying and self-attributing the moment a node creates it. 2. **`AuditProvider.record` dedups on `event_id`** (a duplicate primary key returns quietly, per the ABC). So writing the same event twice is safe and idempotent. 3. **The runtime owns both** the governance provider and the audit provider, and constructs the provider it hands the compiler — the right place to interpose. The seam is a wrapper, `JournalingGovernance` (`audit/_journal.py`), installed by the runtime around the base provider: ``` record_event(event): await base.record_event(event) # unchanged: in-memory list, trust, flight recorder await write(event) # NEW: redact, stamp topology, persist NOW # failure is logged, never raised — the run continues and # the end-of-run batch is the retry ``` `write` is the runtime's `_journal_write`: it applies the same per-skill redaction the batch applied (`_redact_payload`, factored out of `_apply_skill_redaction`), stamps the topology from a new `current_topology` run-scope var (the event's own `run_id`/`labels` are already set), and `dataclasses.replace`s the event with the redacted payload — keeping `event_id`, so the store dedups it against any later write. The end-of-run path is unchanged except that `RunEvent` now carries `event_id` and `_persist_events_to_audit` reuses it instead of minting a fresh one. That turns the batch into a **completeness net**: an event the journal missed (a write that raised, an event recorded outside run scope) is still written at the end, while every event the journal already wrote is deduped. The journal is the durability mechanism; the batch is the backstop. ### What a crash now leaves - Events recorded before the crash: **in the store**, redacted, attributed to the run. - The event in flight at the instant of the kill: possibly lost (its `record` may not have returned). One event, named by what came after it on resume. - The LangGraph checkpoint: durable as before, so the run still resumes. ### What it looks like A real run of `test_kill9_recovery.py`'s workspace: `swarmkit run` on a `lead → prepare → build` chain whose `build` node is a harness, `kill -9`'d while the harness is running. The audit is read back with `swarmkit logs` **before any resume** — the process is gone, nothing flushed at a run boundary, yet the trail is there up to the kill: ``` $ swarmkit run ws change --input "Fix the login bug." # ... SIGKILL delivered while the harness runs ... $ swarmkit logs ws -n 1 ── audit store ── lead started (root) lead done 25ms prepare started (worker) prepare done 6ms build started (worker) build executor.started # the harness had started — and it is on the record ``` The last durable line is the harness starting; its `executor.result` is absent because the harness never finished. Resuming picks up from the checkpoint and appends to the same trail — the pre-kill rows are untouched (append-only), the harness node re-runs (checkpoint recovery), and the run completes: ``` $ swarmkit run ws change --resume Resuming from checkpoint: 350c162e-... [prepare] done [lead] done $ swarmkit logs ws -n 1 ── audit store ── lead started (root) lead done 25ms prepare started (worker) prepare done 6ms build started (worker) build executor.started # attempt 1 — the killed one, still recorded build started (worker) build executor.started # attempt 2 — the resume build executor.result prepare started (worker) prepare done 11ms lead started (root) lead done 4ms ``` Before 1.239.0 the first `swarmkit logs` printed an empty store: the whole trail lived in memory and the SIGKILL took it. ### Cost One store write per event instead of one batch per run. Audit volume is a handful of events per agent step, and SQLite/WAL and Postgres both take a single-row insert cheaply. If a high-throughput deployment needs it, a bounded coalescing buffer (flush every N events or T ms) is a later tuning of `_journal_write` alone — the write-through default is what an audit log should ship with. ## Test plan - `test_audit_journal.py` — a two-node run; assert the audit store has each event *before* the run ends (drive record, read store mid-run via a spy); a failed journal write is logged and the end-of-run batch still persists the event; the batch does not duplicate a journaled event. - `test_kill9_recovery.py` — updated: after the SIGKILL the store now holds the pre-kill events (this is the assertion that flips — the note it referenced is this one), and the resumed run's record joins them without duplication. - Existing audit/observability suites unchanged: same events, same redaction, same `GET /events`. ## Demo plan `just demo-...` not needed; the kill-9 test is the demo. `docs/site/guides/evaluating-the-failure-path.md` (the eight-test evaluation) cites this note for Test 7/8, now answered "the trail survives". ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/memory-by-default.md ## Published: https://delivstat.github.io/swarmkit/design-notes/memory-by-default/ ══════════════════════════════════════════════════════════════════════════════ # Memory by default **Status:** design · **Scope:** schema + runtime + portal + docs · **Follows:** `governed-memory.md`, `memory-and-decision-skills` guide, Level 9 ## The gap Memory is the feature people expect an assistant to have, and today it is the feature that takes the most wiring. Level 9 has a user bind `memory-reader` and `memory-writer` by hand under `governance.decision_skills` (with the right trigger, scope, `required: false` and config), copy two reference skill files into `skills/` to make governed memory exist at all, and only then grant `governed-memory` to an agent. Four steps for "remember what I told you", three of which are the same in every workspace. The runtime already knows all of it: `memory-reader` and `memory-writer` are built-in decision skill ids, and `governed-memory` / `memory-reconcile` are reference skills the runtime ships. What is missing is the default. ## Goal A workspace has memory unless it says otherwise. `swarmkit init`'s output remembers; a workspace that never mentions memory remembers; one `enabled: false` turns off everything automatic — what a workspace wires explicitly (a copied `governed-memory` skill, say) still works. ## Non-goals - Granting `governed-memory` (the *write* skill) to agents automatically. A curated store an agent can write to is a grant a person makes, per agent, as today (`skills_additional`). Reading curated facts needs no grant and is on by default. - Changing what `memory-reader`/`memory-writer` do, or the store's schema. - A portal form for the block. `memory` sits beside `governance` in the workspace file and follows its rule: not form-editable (`_workspace_config.EDITABLE`), visible read-only where it acts — the portal's Memory page shows the effective configuration. ## Shape ### Workspace schema ```yaml memory: # optional; absent ⇒ enabled with defaults enabled: true reader: # config for the auto-bound memory-reader max_results: 5 similarity_threshold: 0.15 search_scope: all # user | all | both writer: # config for the auto-bound memory-writer min_output_length: 100 ``` `memory-reader`/`memory-writer` **explicitly** bound under `governance.decision_skills` keep winning — their trigger, scope, `required` and `config` are used as written, and the auto-binding for that id is skipped. `enabled: false` with an explicit binding is a resolution error (`memory.disabled-but-bound`) rather than a silent choice between the two. ### Runtime `memory/_defaults.py`: - `apply_memory_defaults(raw_workspace: dict) -> dict` — when enabled, appends the missing reader/writer bindings (reader `pre_input`, writer `post_output`, scope `*`, `required: false`, config from the block merged over the defaults) to `governance.decision_skills`. Called by the resolver before the workspace model is built, so everything downstream — `merge_decision_skills`, `swarmkit validate --require`, the portal — sees one list. - `bundled_memory_skills() -> list[DiscoveredArtifact]` — the `governed-memory` and `memory-reconcile` skill artifacts from `memory/skills/*.yaml` (byte-identical copies of the reference files; a test enforces it). Injected by the resolver when enabled and the workspace defines no skill with that id, so the governed store is built and the reconciler wired. A workspace copy still overrides. - The resolver reports the effective block on `ResolvedWorkspace.memory` so the CLI and server can show it; `GET /memory/status` (existing) gains `config`. ### Portal The Memory page reads `config` from the status and shows "Memory: on — reader top 5 ≥ 0.15, writer ≥ 100 chars" or "Memory: off (`memory.enabled: false`)". Nothing editable. ### Cost, stated `memory-writer` is one model call per run whose output clears `min_output_length`. That is the price of remembering and it is now paid by default; the docs say so next to the opt-out, and `min_output_length` is the knob before `enabled`. ## Test plan - schema: valid fixture with the block, invalid fixture (`search_scope: everything`); codegen drift job. - runtime: a workspace with no `memory` block resolves with both bindings and both skills; an explicit reader binding is kept verbatim and only the writer is added; `enabled: false` yields neither binding and no bundled skills (an explicitly copied skill still builds the store); `enabled: false` + explicit binding → resolution error; the bundled skill files equal `reference/skills/*`; a workspace copy of `governed-memory` wins; `swarmkit run` on the mock provider with default memory writes nothing on a short answer and writes on a long one (existing writer tests, now under defaults). - portal: Memory page renders the effective configuration. ## Demo plan Level 9 rewritten from a real run: a fresh `swarmkit init` workspace, no memory wiring, "two things to remember about me…", a second run that recalls them, the Memory page showing the config, then `enabled: false` and the same second run forgetting. Levels 1–8 example workspaces run under the default and their transcripts are re-checked where memory now speaks. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/extracting-the-channels.md ## Published: https://delivstat.github.io/swarmkit/design-notes/extracting-the-channels/ ══════════════════════════════════════════════════════════════════════════════ --- title: Taking communication out of the runtime description: Why channel integration belongs to the calling application, what leaves, and the event seam that replaces it. status: draft --- # Taking communication out of the runtime **Status:** proposed. The precedent is [`extracting-the-pipeline.md`](extracting-the-pipeline.md) — the same decision, one layer over. ## The claim **SwarmKit should not talk to Telegram.** An application that uses SwarmKit as its agent runtime already owns its relationship with its users: which channel, which identity, what tone, what retention. The runtime's job is to say *what happened*, in a form the application can act on. This is the pipeline decision restated. Sequencing left in 1.189.0 because it belonged to the caller; communication belongs to the caller for the same reason and slightly more obviously, since a chat integration is a relationship with a third party rather than a piece of agent machinery. ## Why the evidence was already in Three things happened while building this, and each argued against it: **The notification providers were never reachable.** 548 lines covering Telegram, Discord, Slack and a generic webhook, with 31 unit tests and a docstring promising it fired on `hitl_requested`, `run_ended_error` and `skill_gap_surfaced`. `build_provider` was never called, `NotificationRegistry` never constructed, and `notifications:` did not exist in the schema. Code that nobody could reach for months is code nobody needed *in the runtime*. **The inbound half does not fit the runtime's shape.** Telegram works because `getUpdates` is outbound long-polling. Discord needs a gateway WebSocket held open; Slack needs Socket Mode. Both mean a persistent connection with its own lifecycle inside a process that is otherwise request-shaped, and both were deferred for exactly that reason. **Third-party surfaces rot on their own schedule.** WhatsApp needs QR pairing against a personal number. Slack moved to Socket Mode. A runtime that ships a channel adapter releases when Telegram deprecates something, which is a release cadence with nothing to do with agents. ## The argument that settles it A notification provider is a **parallel extension mechanism**. Invariant #2 says skills are the only capability primitive; a provider is a second way for the runtime to reach the outside world, one that no skill grant authorises and no audit line records. That is the smell, and it is why the code could sit unreachable without anything noticing: nothing depended on it because nothing *could*. An event sink does not have that problem. Events are already produced, already append-only, already observable. Publishing them adds no new authority. ## What leaves | component | LOC | verdict | | --- | --- | --- | | `notifications/` (5 providers, registry, store) | 548 | **leaves** | | `channels/` (`_config`, `_server`, MCP tools) | 416 | **leaves** | | `reference/skills/channel-{send,ask,replies}.yaml` | 3 files | **leaves** | | `tests/test_notifications.py`, `tests/test_channels.py` | 633 | **leaves** | | `examples/channels/`, `just demo-channels` | — | **leaves** | | `channels:` in `workspace.schema.json` + 2 fixtures | — | **leaves** | | `mcp_servers` entry for `channels` in `reference/workspace.yaml` | — | **leaves** | Roughly 1,600 lines. **None of it is published** — channels shipped in 1.206.0 and never reached PyPI, and the notification providers were never reachable at all. So this is a deletion rather than a deprecation, which is the difference between a clean removal and a two-release dance. That window closes at the next publish. ## What replaces it An **event sink**: the runtime pushes what happened, the application decides what to do about it. ```yaml # workspace.yaml events: - sink: webhook url: https://my-app.internal/swarmkit/events credentials_ref: my-app-signing-key - sink: stdout # dev, and the honest default ``` The vocabulary already exists and already flows into the audit provider — `run.started`, `run.ended`, `hitl.requested`, `hitl.resolved` — with `run.paused` and `run.resumed` to add. **But there is no `/events` route.** The events reach the audit store and nothing exposes them over HTTP; `GET /events` returns 404 today. So the durable-pull half of the delivery contract below is *build*, not wiring, and it is the reason this sequences before the deletion rather than with it. ## Delivery: best-effort push, durable pull The interesting decision, and the one to get right rather than fast. A webhook that fails is not a rare case. If `hitl.requested` is dropped, a run waits forever for an approval nobody was told about — the failure mode is silence, which is the worst kind. Two honest options: 1. **At-least-once with retry and a queue.** Correct, and it puts durable delivery machinery inside the runtime — a queue, a retry schedule, a dead-letter path, backpressure. That is the thing we just decided not to own. 2. **Best-effort push plus a durable pull.** The webhook is fire-and-forget with a short retry; the audit log is already durable and ordered, so the application reconciles from a cursor (`GET /events?after=`). A missed push costs a reconciliation, not an approval. **Decision: option 2.** The push is a latency optimisation over a source of truth that already exists, and saying so plainly is better than implying a guarantee the runtime is not built to keep. An application that needs stronger delivery puts a queue between itself and the webhook, which is where a queue belongs. The contract stated for callers: - Events are **at-most-once** over the webhook. Retried a small fixed number of times, then dropped. - Every event carries a monotonic `seq` and the run id. - `GET /events?after=` replays from the durable log. **An application that cares about not missing events reconciles; one that does not, does not.** - The portal remains the reliable human path. A gate is visible there whether or not any push succeeded. ## Asking a human needs nothing new — it is a gate `channel_ask` is not replaced by an MCP tool either. It is **deleted**, because the runtime already has this primitive and has had it since the decision-skills work: ```python ReviewItem.answer # "For §6.3 input requests: the operator's textual answer" ReviewItem.resolved_by # "The authenticated resolver" ReviewItem.comment # "What the human said. Relayed to the agent, recorded on the audit." ``` `channel_ask` reimplemented that with **worse durability** — blocking in-process on a long poll rather than checkpointing — and **worse identity**, a `chat_id` rather than an authenticated `resolved_by`. An earlier draft of this note proposed replacing it with an `ask_human` MCP tool, which is the same mistake once more: a synchronous tool call holds a session open while a human is at lunch, which `mcp-oauth.md` already rejected for the analogous consent case. **The runtime's role is to surface the ask, park, and accept a resolution. How it is resolved is never its business.** ### Verified, not assumed Driven end to end over HTTP against `swarmkit serve`, which is what an application would do: ``` POST /run/ask job 621f8a…, status: running status: deferred ← parks; nothing resident GET /review mpa-621f8a…:root-0-operator reason: "role 'operator' must approve 'workspace:approve'" question · options · free_text_allowed · gate_id · run_id GET /gates/621f8a…:root outstanding: ["operator (workspace:approve)"] POST /review/{item}/resolve status: approved, resolved_by: anonymous GET /gates/… resolved: true, distinct_approvers: ["anonymous"] POST /jobs/{id}/resume completed ``` Four things that run-through found, each of which this note would have got wrong from reading the code alone: **A gate silently degrades to advisory when no `RoleRegistry` defines its roles.** The first run completed with no gate at all. The guard is deliberate — a gate nobody can satisfy would strand every run — but an application seeing no gate cannot tell "not gated" from "misconfigured". The event vocabulary needs to distinguish them. **Two endpoints look like approve and one silently does not count.** `/review/{id}/approve` marks the item approved and leaves `resolved_by` empty, so the gate stays `pending` with no `distinct_approvers` and the run re-defers. `/review/{id}/resolve` is the multi-party one that resolves as the authenticated caller. Any application integrator will hit this; the first attempt here did. **`outcome` is `"approve"` while the resulting status is `"approved"`.** Small, and it costs a round trip to discover. **A resolved gate does not resume its run.** An explicit `POST /jobs/{id}/resume` is required. ### Decision: a resolved gate resumes its run Automatic, with an opt-out. Otherwise every application writes the same resume call, and the one that forgets leaves a run parked after its gate has been satisfied — a stall with no visible cause, because everything *looks* resolved. The opt-out exists for an application that wants to batch or delay resumption, and it is a workspace-level setting rather than a per-call flag, so the behaviour is legible in one place. ## The reference application The pipeline removal worked because `examples/pipeline-orchestrator/` proved the pattern and **imports no runtime module**. Without an equivalent, "the application should do it" is an instruction rather than a demonstration. So: `examples/event-consumer/` — a small service that receives the webhook, reconciles from the cursor on startup, and sends a Telegram message when a gate opens. It imports `httpx` and nothing from `swarmkit_runtime`. The Telegram code that leaves the runtime lands there, where it is a hundred lines of somebody's application rather than a supported surface. ## Order: events first, then the deletion Deleting channels before `/events` exists would leave a window where nothing can observe a gate except polling `/review`. So: 1. **`GET /events?after=` and the `events:` sink**, with `run.paused` / `run.resumed` added to the vocabulary and gate events carrying enough to render a question. 2. **Auto-resume on gate resolution**, with the opt-out. 3. **`examples/event-consumer/`**, proving the loop against a running serve. 4. **The deletion** — all 1,600 lines, once nothing needs them. ## What this does not change - **`swarmkit ask`** is unrelated — it asks an LLM about a workspace, not a human. - **Approval gates and the review queue** are untouched. The event says a gate opened; resolving it goes through the API that already exists. - **OAuth's `expiring_soon`** becomes an event like everything else, which is what it wanted to be. ## Test plan - Every deleted module's tests are deleted, not adapted. A test that survives a removal is testing something that did not leave. - The sink: an event reaches a webhook; a failing webhook does not fail the run; the retry count is bounded; `seq` is monotonic across a restart. - Reconciliation: an application that misses N pushes recovers all N from the cursor. - The negative, as with the pipeline: `grep -rn "telegram\|discord\|slack" packages/runtime/src` returns nothing. ## Demo plan `just demo-event-consumer` — start `swarmkit serve`, run a topology that opens a gate, and watch the example application receive `hitl.requested` and print the message it would send. Then kill the consumer, run again, restart it, and watch it recover the missed event from the cursor. **The recovery is the demo**: anyone can show a webhook firing. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/skill-catalogue.md ## Published: https://delivstat.github.io/swarmkit/design-notes/skill-catalogue/ ══════════════════════════════════════════════════════════════════════════════ --- title: Skill catalogue — a curated library whose "works" is a fact with a date description: A separately-versioned repo of pre-wired skills across all four backings, kept honest by a liveness check that runs against the real servers and files an issue when one breaks. tags: [skills, ecosystem, distribution, validation] status: draft --- # Skill catalogue **Scope:** a new public repo (`swarmkit-skills`), plus `requires_runtime` in the skill schema and a `swarmkit skill` command group **Supersedes:** the middle layer and the trust model of `skill-registry.md` — see [What this changes](#what-this-changes-about-skill-registrymd) **Status:** draft **Companion:** [`skill-catalogue-seed.md`](skill-catalogue-seed.md) decides *what goes in it* — the first hundred servers, triangulated across six sources and ordered by what the liveness check can actually verify. ## Goal Make the common case — *"I want this MCP server as a skill"* — a one-line command instead of forty minutes of trial and error, and make the claim that a catalogue entry **works** something with a date on it rather than a promise. ## The gap is wiring, not skills *"Any of 7,260 MCP servers can be a skill"* is true and about as useful as *"any package works with pip."* What is missing per server is the part nobody publishes: - the `mcp_servers` block — command, `env`, `cwd`, whether it needs `sandboxed: true` - the `permission` tier, and now the per-tool `effects` map that `readonly` requires - the skill YAML with an `iam.required_scopes` that is neither too broad nor missing - the argument shape the tool actually wants, which its README rarely states precisely Every user rediscovers that independently, for the same twenty servers. Thirty entries where somebody already did it beats eleven hundred that might work. Two signals that this is a real gap rather than a nice-to-have: - **`reference/skills/` has 27 skills and not one uses the `command` backing** shipped in 1.197.0. The newest extension type has no library at all. - **`reference/command-packs/` is already a second location.** There are two homes for curated artifacts before anyone has asked for a catalogue. ## Verification is the product This is the part that decides whether the repo is worth building. An `mcp_tool` skill depends on a server somebody else maintains. That server can rename a tool, change an argument, add auth, or disappear. A curated list nobody re-checks becomes an awesome-list, and those rot in months — there are already several awesome-MCP repos, and being another one adds nothing. **SwarmKit can do what a list structurally cannot: start the server and ask.** ``` nightly, per entry: 1. resolve requires — is the binary / image / package still fetchable? 2. start the server — does it come up inside the declared timeout? 3. list_tools — does the named tool still exist? 4. compare schema — does its inputSchema still match what the skill declares? 5. dry-run — for a read-only tool, does one call return without error? ``` That turns *"pre-validated"* into **"verified 2026-09-01 against server v2.3"** — a fact with a date, which is a different kind of claim from a promise. Every entry carries its last-verified timestamp, and the catalogue's front page is a table of them. **Build this first, not last.** Bolted on afterwards it never gets built, and the repo becomes the thing it was meant to replace. ### When a check fails **Mark it broken, file an issue, do not quietly fix.** ```yaml status: state: broken # verified | broken | unmaintained since: 2026-09-01 reason: "tool 'get_pull_request' no longer listed; server exposes 'pull_request_read'" issue: delivstat/swarmkit-skills#212 ``` A broken entry stays visible and stays honest. Removing it silently is worse: someone who already copied it learns nothing, and the catalogue looks healthier than it is. The issue is the interface to whoever fixes it — **and "whoever" can be a swarm.** A broken entry is a well-specified authoring task with a machine-checkable acceptance test: the liveness check that failed. That is the Skill Authoring Swarm's exact job description, and it closes the loop the product claims as its third pillar — *swarms grow through human-approved authoring, gated at every step*. The fix arrives as a PR a human approves; nothing self-modifies. An entry `broken` for three consecutive checks with no fix becomes `unmaintained`, which is a signal to a reader rather than a deletion. ## Compatibility is a version floor, and we have paid for forgetting A `command`-backed skill needs runtime ≥1.197.0. One using `requires:` needs ≥1.193.0. One relying on a `readonly` MCP server with declared `effects` needs ≥1.199.0. **Skills carry no compatibility field today.** Without one, a catalogue entry resolves cleanly into an older workspace and fails at run time with an error that names nothing useful. This is precisely the `swarmkit-webui` failure: a separately-versioned artifact that must match the runtime's surface, with an unbounded floor. It froze at 0.14.0 and shipped a portal navigating to an API removed a week earlier — cleanly resolvable, broken in a browser, and reported by a user rather than caught by CI. So the schema gains one field: ```yaml provenance: authored_by: human version: 1.2.0 requires_runtime: ">=1.197.0" # new ``` Checked at workspace load, refusing with the version it needs and the version present. A floor that fails at import is worth more than a diagnosis at run time. `provenance` is `additionalProperties: false`, so this is a real schema change and follows `docs/notes/schema-change-discipline.md` — canonical schema, the bundled `_schemas/` copy, fixtures valid and invalid, both codegens. Adding it is cheap now and a migration across every published catalogue entry later, which is the same argument that put `frame_path` into a fingerprint from v1. ## Importing one skill writes to two places The uncomfortable part of `swarmkit skill add github-pr-read` is not fetching. `swarmkit install` already takes a URL, so distribution is nearly free. It is that one skill needs **two** edits: ``` skills/github-pr-read.yaml ← new file, safe workspace.yaml mcp_servers: ← a hand-authored artifact, not safe ``` Editing someone's `workspace.yaml` silently is the kind of convenience people switch off. So: ```bash swarmkit skill add github-pr-read # shows the diff, asks, applies swarmkit skill add github-pr-read --dry-run # prints both fragments, writes nothing swarmkit skill list --available # the catalogue, with verification dates swarmkit skill check # re-run the liveness check on what is installed ``` `--dry-run` is the honest default for the first version: print the YAML, let the user paste it. Full auto-edit is the second version, once the diff has been boring for a while. ## Where things live | | holds | versioned with | | --- | --- | --- | | `reference/` in this repo | the **worked examples the docs teach from** — small, stable, illustrative | the runtime | | `swarmkit-skills` (new) | the **catalogue** — breadth, third-party servers, liveness-checked | itself, floored per entry | Not three locations. `reference/command-packs/` and `reference/skills/` stay as documentation material; anything that depends on an external service belongs in the catalogue, because that is what needs a nightly check and a version of its own. ## Non-goals - **Bundling the catalogue into the runtime.** A broken skill would then need a runtime release to fix, coupling two things that change at very different rates. `skill-registry.md` proposed the registry ship inside `swarmkit-runtime`; that is the part this note reverses. - **A rating or marketplace system.** Verification state is the only signal, and it is objective. - **Importing 1,100 `SKILL.md` files.** Breadth without verification is the failure mode being designed against. The converter in `skill-registry.md` stays proposed and unbuilt for now. - **Auto-updating installed skills.** An import is a copy into the user's workspace and stays theirs. `swarmkit skill check` tells them something upstream changed; it does not change their file. ## What this changes about `skill-registry.md` That note (status: proposed, April 2026) stays the reference for the *external* ecosystem — the `SKILL.md` format, the converter, the catalogue landscape. Two of its decisions are reversed here: | `skill-registry.md` | this note | why | | --- | --- | --- | | the registry ships **inside `swarmkit-runtime`** | a **separate repo** | a skill fix should not need a runtime release | | skills are **"trusted by source"** (Anthropic, Google, MCP official) | skills are **verified nightly**, and say when | provenance is not liveness; a trusted publisher's server still renames tools | The second is the substantive one. "Trusted by source" is an assumption with no expiry date, and it is exactly what turns a curated list into a stale one. ## Test plan - **Unit** — `requires_runtime` parsing and comparison; a skill above the floor loads, one below is refused with both versions named; an absent field means no constraint. - **Unit** — the import planner produces both fragments and mutates nothing under `--dry-run`. - **Integration** — `swarmkit skill add` against a fixture catalogue: new skill file, `mcp_servers` diff shown, applied only on confirmation; re-running is idempotent. - **Integration** — the liveness check against a local stub MCP server: passes when the tool matches, reports `broken` with the tool name when renamed, and does not throw when the server refuses to start. - **Catalogue CI** (in the new repo) — every entry validates against `skill.schema.json`, declares a `requires_runtime`, and names a maintainer. Schema validity is a merge gate; liveness is nightly, because a third-party outage must not block a PR. ## Demo plan `just demo-skill-catalogue` — add a skill from a fixture catalogue into a scratch workspace with `--dry-run` (printing both fragments), then for real; run the liveness check green; rename the tool in the stub server; re-run and watch the entry go `broken` with the reason and a filed-issue link. The rename is the demo. Anyone can show an install working; the claim worth demonstrating is that the catalogue *notices* when it stops. ## Open questions 1. **Verification needs somewhere to run.** Nightly, some entries need Docker, some need network, some need credentials. Credentialed servers (GitHub, Slack) cannot be fully checked in public CI — they are probably `unverifiable` as an honest third state rather than a silent pass. 2. **What is the catalogue's unit?** One skill, or a bundle per server — "the GitHub pack" with six skills sharing one `mcp_servers` entry. The bundle is closer to how people actually adopt one, and maps onto command packs, which suggests bundle. 3. **How does an entry pin the upstream server?** A version, a digest, or nothing. Nothing makes the liveness check meaningful but the install unreproducible; a digest is the opposite. 4. **The first fifteen.** Chosen by what people actually reach for, which we do not know yet — the first honest source is what shows up in issues. ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/skill-registry.md ## Published: https://delivstat.github.io/swarmkit/design-notes/skill-registry/ ══════════════════════════════════════════════════════════════════════════════ --- title: Skill registry — the `swarmkit skill` command over the catalogue and SKILL.md description: How a workspace finds, adds, imports and checks skills — the swarmkit skill command group over the swarmkit-skills catalogue, a SKILL.md converter, and a liveness check. Refreshed September 2026 from the April proposal. tags: [skills, registry, ecosystem, community] status: accepted --- # Skill registry — the `swarmkit skill` command > **Refreshed 2026-09-18.** The April proposal below (kept under "Original landscape") is what > this note now implements, with the two decisions [`skill-catalogue.md`](skill-catalogue.md) > reversed in the meantime kept reversed: the library is the separately-versioned > **`swarmkit-skills` catalogue**, not a registry inside the runtime wheel, and an entry's > standing is its **verification date**, not its publisher. What was still missing on the day of > the refresh: the command group itself. None of `swarmkit skill …` existed; `swarmkit install` > moves whole workspaces, `swarmkit author skill` writes one from a conversation, and the > catalogue was "copy a bundle by hand". ## What ships ```bash swarmkit skill list # the workspace's skills: id, category, backing, who holds it swarmkit skill list --available # the catalogue: bundles, skills, verification dates swarmkit skill search "git history" # catalogue names + descriptions (and the workspace's own) swarmkit skill show git-log # one entry, from the workspace or the catalogue swarmkit skill add git-log # a catalogue skill: skills/git-log.yaml + its mcp_servers entry swarmkit skill add git # a whole bundle: every skill + the server, one prompt swarmkit skill add ./my-skill.yaml # a local Skill file (or a SkillBundle), same path swarmkit skill import ./SKILL.md # an Agent Skills file → an llm_prompt skill swarmkit skill check # do the tools the workspace's mcp_tool skills name still exist? swarmkit skill remove git-log # delete the file — refused while an archetype or agent holds it ``` `add` writes to **two places** — a new file under `skills/` (safe) and an `mcp_servers` entry in `workspace.yaml` (hand-authored, not safe to edit silently) — so it shows both fragments and asks; `--dry-run` prints them and writes nothing; `--yes` is for scripts. The `mcp_servers` edit goes through the same comment-preserving, validate-or-roll-back service the portal's Connections page uses (`server/_workspace_config.py`), so a hand edit and a `skill add` produce the same file. ### Sources, in order 1. **The workspace** — `skills/*.yaml` (for `list`, `show`, `search`, `remove`, `check`). 2. **The catalogue** — `delivstat/swarmkit-skills` on GitHub: `skills//bundle.yaml` (`kind: SkillBundle`: the `mcp_servers` block, the skill ids, `verification`) and `skills//skills/.yaml`. Fetched over HTTPS, cached under `~/.swarmkit/cache/skills-catalogue.json` for a day (`--refresh` to refetch); `SWARMKIT_SKILLS_CATALOGUE=` points at a checkout or a mirror, which is also how the tests run against a fixture catalogue with no network. 3. **A path or URL** — a `Skill` or `SkillBundle` YAML file given to `add` directly. `reference/skills/` in this repo stays what `skill-catalogue.md` said: worked examples the docs teach from, versioned with the runtime — not a fourth source. Two of them (`governed-memory`, `memory-reconcile`) are bundled by `memory-by-default.md` for a different reason. ### The converter (`import`) An Agent Skills `SKILL.md` is YAML frontmatter (`name`, `description`, optional `metadata`) over a markdown body of instructions. It becomes an `llm_prompt` skill: ```yaml apiVersion: swarmkit/v1 kind: Skill metadata: id: name: description: category: capability implementation: type: llm_prompt prompt: | provenance: authored_by: imported_from_registry version: registry: ``` It is validated against `skill.schema.json` before it is written, and `add`'s two-place rule does not apply (an `llm_prompt` skill needs no server). Anything the body references that a prompt cannot do — scripts, bundled files — is left in the prompt for a person to see; the converter does not pretend a SKILL.md with a `scripts/` directory is a prompt. ### `check` For every `mcp_tool` skill in the workspace: start its server through the runtime's own MCP client (the same start, sandbox and credentials a run would use), list the tools, and report whether the tool the skill names is there — `ok`, `missing ` (the server renamed it), or `server failed: `. The catalogue's nightly job asks the same question of every entry; this asks it of *yours*. It changes nothing; the exit code is non-zero when anything is not `ok`. ### Service, then CLI, then HTTP Business logic lives in `swarmkit_runtime.skills._registry` (catalogue index and search, the add planner and applier, the converter, the checker); `cli/_cmd_skill.py` and the serve routes `GET /api/skill-catalogue`, `GET /api/skill-catalogue/{id}`, `POST /api/skills/add`, `POST /api/skills/import`, `GET /api/skills/check` are thin over it — the CLI-first, thin-interface rule. The portal's Skills page gets a **Library** tab over the same routes in a follow-up. ## Non-goals - Bundling the catalogue into the runtime (`skill-catalogue.md`'s reversal stands). - Importing MCP servers from a URL as a bundle (`import-mcp`). Curating a server — the permission tier, an `effects` map per tool, the argument shapes — is the catalogue's job and is done by a person with the server in front of them (`feedback: probe, don't transcribe`). `add` of a catalogue bundle is the supported way to get a server into a workspace. - Auto-updating installed skills; `check` reports, it does not rewrite. - A rating or marketplace system. ## Test plan - catalogue: index built from a fixture directory (`SWARMKIT_SKILLS_CATALOGUE`); search ranks name and description hits; cache respected and `--refresh` bypasses it. - add: the planner yields the skill file and the `mcp_servers` fragment; `--dry-run` writes nothing; apply writes the file and upserts the server through the config service (comments intact); re-running is idempotent; a bundle adds every skill and one server; an unknown id names the closest matches; a `requires_runtime` above the running runtime is refused naming both. - import: frontmatter + body → a valid `llm_prompt` skill; a missing `name` is an error; the body is verbatim; `provenance.registry` carries the origin. - check: against a stub MCP server — `ok` when the tool matches, `missing` when renamed, `server failed` when it will not start; non-zero exit when anything is not `ok`. - remove: refused while an archetype or agent grants the skill; deletes otherwise. - CLI: every subcommand invoked through the Typer app on a fixture workspace; HTTP routes through the TestClient. ## Demo plan Level 13 gains a section run for real: `skill search git`, `skill add git` against the live catalogue (the two fragments, the prompt, the resulting files), `skill check`, and `skill import` of an Anthropic `SKILL.md`. Transcripts in the tutorial, the portal Skills page showing the added skill. --- ## Original landscape and proposal (April 2026) ## Goal SwarmKit users should find and install pre-built skills from the community instead of writing everything from scratch. The ecosystem has converged on two standards — both should be importable. **Don't reinvent skills — import the ecosystem.** ## Landscape (as of April 2026) Two dominant formats: ### Agent Skills (SKILL.md) - **Spec:** agentskills.io (Apache-2.0) - **Format:** YAML frontmatter + markdown body - **Supported by:** 27+ agents (Claude Code, Gemini CLI, Codex, Cursor, Copilot, Windsurf) - **Available skills:** 1,100+ cataloged (VoltAgent/awesome-agent-skills) - **Key repos:** - github.com/anthropics/skills (123K stars) — official Anthropic skills for document creation, development, testing - github.com/vercel-labs/agent-skills — React, Next.js, deployment - github.com/google/skills — 13 Google Cloud product skills **SKILL.md format:** ```yaml --- name: lowercase-kebab-name description: What it does license: Apache-2.0 metadata: author: org-name version: "1.0" --- # Instructions in Markdown Progressive disclosure: ~100 tokens metadata at startup, <5000 tokens body on activation, reference files on demand. ``` **Conversion to SwarmKit:** near-trivial — YAML frontmatter with kebab-case IDs maps to SwarmKit's skill schema. The markdown body becomes the skill's instruction content. ### MCP servers - **Protocol:** JSON-RPC over stdio/SSE - **Available:** 7,260+ servers cataloged (TensorBlock) - **Key repos:** - github.com/modelcontextprotocol/servers (84K stars) — official reference servers (filesystem, git, fetch, memory) - Covers: databases, cloud platforms, APIs, search, communication - **SwarmKit support:** already designed (§18). MCP tools are the implementation backend for capability skills. ### Other pools | Source | Count | Format | SwarmKit path | |---|---|---|---| | LangChain tools | 600+ | Python classes | Runtime bridge (we compile to LangGraph) | | Composio | 1,000+ | OpenAPI specs | Via Composio's MCP server | | CrewAI tools | 60+ | Python classes | Wrap as skill implementation | | OpenAI GPT Actions | 34 | OpenAPI specs | Parse spec → skill YAML | ## SwarmKit skill registry architecture ### Three-layer model ``` Community sources (remote) ├── Agent Skills repos (SKILL.md) ├── MCP server catalogs └── SwarmKit-native repos ↓ swarmkit skill import / install Local registry (bundled with swarmkit-runtime) └── reference/skills/ ← 20+ pre-imported skills ↓ swarmkit skill install Workspace skills/ └── skills/.yaml ← workspace-local, validated ``` ### CLI commands (as proposed; the shipped surface is above) ```bash # Install from the local registry into the workspace swarmkit skill install code-quality-review # Import from a remote Agent Skills repo swarmkit skill import github.com/anthropics/skills/create-docx # Import an MCP server as a skill source swarmkit skill import-mcp github.com/modelcontextprotocol/servers/filesystem # Search available skills (local registry + remote catalogs) swarmkit skill search "security" # List installed skills in the workspace swarmkit skill list # List all available skills in the registry swarmkit skill list --available ``` ### SKILL.md → SwarmKit YAML converter ``` Input: SKILL.md --- name: code-quality-review description: Reviews code for quality issues metadata: author: anthropic version: "1.0" --- # Instructions... Output: SwarmKit skill YAML apiVersion: swarmkit/v1 kind: Skill metadata: id: code-quality-review name: Code Quality Review description: Reviews code for quality issues category: capability implementation: type: instruction content: | # Instructions... provenance: authored_by: community source: github.com/anthropics/skills/code-quality-review version: 1.0.0 ``` The converter: 1. Parses YAML frontmatter → `metadata` block 2. Maps `name` → `id` (already kebab-case) 3. Infers `category` from content (or defaults to `capability`) 4. Preserves markdown body as `implementation.content` 5. Adds `provenance.source` tracking the origin repo 6. Validates against SwarmKit's skill schema ### Authoring AI integration When the authoring AI is creating a workspace: 1. **Search first.** Before generating a new skill, search the registry: "I need a code quality review skill" → finds existing one 2. **Install existing.** If a match exists, propose installing it instead of generating 3. **Generate only gaps.** If no match, generate a new skill (current behavior) 4. **Cite sources.** When using a community skill, tell the user where it came from The authoring AI's system prompt includes a catalog summary (skill names + descriptions from the registry) so it knows what's available without searching every time. ## Seed skills (initial registry, 20+) Drawn from existing community repos: **Capability skills:** - `file-read` — read file contents (from MCP filesystem server) - `file-write` — write file contents - `web-fetch` — fetch URL content (from MCP fetch server) - `web-search` — search the web - `git-diff` — read git diff (from MCP git server) - `git-log` — read git history - `github-pr-read` — read pull request details - `github-issue-read` — read issue details - `database-query` — execute SQL query - `code-execute` — run code in a sandbox **Decision skills:** - `code-quality-review` — code quality assessment (pass/fail + reasoning) - `security-vulnerability-scan` — security check (severity + description) - `content-moderation` — content safety check - `schema-validation` — validate data against a schema **Coordination skills:** - `coordinate-workers` — leader-mediated worker collaboration **Persistence skills:** - `audit-log-write` — append to audit log - `knowledge-base-update` — update shared knowledge - `review-queue-submit` — submit item for human review **Domain-specific (from Google/Anthropic repos):** - `create-docx` — generate Word documents - `create-pdf` — generate PDF documents - `bigquery-query` — BigQuery SQL execution ## Implementation plan 1. **Task #44:** Design note (this document) ✓ 2. **Task #45:** SKILL.md → SwarmKit YAML converter 3. **Task #46:** Seed 20+ skills from community repos 4. **Task #47:** `swarmkit skill install/search/list` CLI 5. **Follow-up:** Wire authoring AI to search registry before generating ## Non-goals (for now) - **Publishing skills back to community repos.** Import only for v1.0; publishing is M7+ (Skill Authoring Swarm). - **Skill marketplace / rating system.** Community skills are trusted by source (Anthropic, Google, MCP official). No rating system needed for v1.0. - **Automatic skill updates.** Version pinning via `provenance.version`; updates are manual (`swarmkit skill update `). ══════════════════════════════════════════════════════════════════════════════ ## Source: docs/site/design-notes/fleet-control-plane.md ## Published: https://delivstat.github.io/swarmkit/design-notes/fleet-control-plane/ ══════════════════════════════════════════════════════════════════════════════ --- title: Fleet control plane — cross-instance observability, eval, and human-gated self-improvement description: A self-hostable control plane that aggregates OTel + audit + eval signals across many SwarmKit instances and turns them into human-approved self-improvement plans distributed back as data. Plus the eval harness and the ADK borrows that feed it. tags: [serve, distributed, observability, governance, self-improvement, eval, otel, a2a] status: proposal --- # Fleet control plane ## Why this note Two things motivate it: 1. **Multi-instance reality.** A per-instance `swarmkit serve` portal only ever sees its *own* traces. Across a fleet — many sites, many appliances (e.g. a Minder per home), many customer deployments — you want **emit-locally, aggregate-centrally**. A pattern that only shows up across fifty instances is invisible to any one of them. 2. **Turning observation into evolution.** Once telemetry + audit + eval are aggregated, they become the *evidence* for SwarmKit's third pillar — **growth-through-human-approved-authoring** (§12). The control plane is where the swarm *proposes* improvements from fleet-wide evidence and a human *approves* them. This is the inter-instance / evolution layer. It sits **above** [[distributed-architecture]] (which scales a *single* deployment horizontally — worker pool + shared Postgres + a centralized OTel collector) and builds on [[opentelemetry-observability]] (the telemetry foundation). It is NOT a rewrite of either. ## What already exists (grounded) - **OTel: mature.** `packages/runtime/src/swarmkit_runtime/telemetry/` — OTLP export, `swarmkit.*` semantic attributes (`topology.id`, `run.id`, `agent.id`, `tool.name`, governance decisions, approval waits), opt-in config. Resource attributes can already tag `service.instance.id`. - **Audit: sealed + correlatable.** `audit/_provider.py` append-only `AuditProvider` (SQLite/mock), rich `AuditEvent` with `run_id` / `event_id` / `parent_event_id`, redaction (`audit/_redact.py`). Governance is uniform via `GovernanceProvider` (`governance/__init__.py`) — every model/tool/skill/agent action routes through it. - **serve: real.** `server.py` — FastAPI, 15+ endpoints, job store, canary router, triggers, WS streaming. Per-instance. - **Centralized OTel collector** is already in the [[distributed-architecture]] picture for one deployment. - **Gaps (this proposal):** no **eval harness** (only per-output decision skills, `governance/_decision_evaluator.py`); no **cross-instance aggregation / control plane**; no **fleet self-improvement** (today: per-instance skill-gap logs → review queue → human authoring, §12.5). Today multi-instance aggregation is assumed to be the commercial a managed backend layer — this note defines a **self-hostable OSS** path. ## The three planes ``` DATA PLANE — N swarmkit serve instances (+ edge single-instance, e.g. Minder) run swarms · enforce LOCAL governance gates · emit OTel + audit tagged: service.instance.id, workspace.id, deployment/tenant autonomous: keep running if the control plane is unreachable (buffer telemetry) │ OTLP + audit stream (signed, tagged) ▼ OBSERVABILITY PLANE — standard, do NOT reinvent OTel Collector → backend (Tempo/Jaeger + Grafana, or a managed backend managed) raw spans/metrics, fleet-wide, via the OTel ecosystem │ swarm-semantic summary (not raw spans) ▼ CONTROL PLANE — the SwarmKit-specific part (new, OSS, self-hostable) ingest: aggregated audit (the §8 "media" branch at fleet scale) + eval results + skill-gap signals + drift scores + canary metrics, across instances surface: fleet run/trace inspection · eval dashboards · SELF-IMPROVEMENT PLANS approve: §8.7 reserved-scope human gates (skills:activate, topologies:modify) distribute: approved skills/topologies — as DATA — back to instances ``` ### Why the split matters - The **observability plane is commodity** — instances already export OTLP; lean on Collector + an existing backend for raw spans. SwarmKit must **not** build a trace database. - The **control plane is the value** — it reads the *swarm-semantic* summary (which topology, which skill, which governance decision, eval scores, gaps), not a billion raw spans, and it runs the self-improvement loop. ## The self-improvement loop, at fleet scale **Observe → Measure → Propose → Approve → Grow** — pillar #3 made operational: 1. **Observe** — aggregated traces + audit show *where* swarms fall back, escalate to HITL, or fail, across the fleet. 2. **Measure** — the **eval harness** (below) scores topology behaviour; regressions and low scores across instances become signals. 3. **Propose** — the control plane mines those signals into **self-improvement plans**: "add a `hairnet` skill (12 instances hit this gap)", "this route falls back 40% on instances running model X — retune", "topology Y regressed after skill Z v3". Proposal only — never auto-applied (§12.5). 4. **Approve** — a plan touching `skills:activate` / `topologies:modify` lands in the **human approval queue**; those scopes are reserved for human identity (§8.7), unchanged. The control plane is *where* the human acts; the policy is the same. 5. **Grow** — an approved change is **data** (a skill file, a topology edit) → versioned, signed, and **distributed** to instances (pull: instances poll for approved artifact versions; provenance verified via the existing `skill.metadata.provenance`). This is [[installable-expertise-packages]] / skills-library distribution realized at fleet scale. **The moat:** because a swarm change is *data*, fleet-wide self-improvement is a **data-distribution problem, not a redeploy**. A code-first framework (ADK) must ship new code to every node; SwarmKit pushes approved YAML. That is only possible because of pillar #1 (topology-as-data) — and the central aggregation is what makes "detect the gap across the fleet" possible in the first place. ## Borrowed from ADK (folded in) ADK's build→eval→observe→deploy loop maps onto this; the borrows that sharpen SwarmKit's own thesis: 1. **Eval harness (the #1 borrow; genuine gap).** Eval sets as **data artifacts** (`input → expected trajectory + expected response`), a `swarmkit eval` command, and scoring that **reuses the existing decision-skill judges** (`governance/_decision_evaluator.py`, Tier 2/3). Results feed the control plane's "Measure" step and gate the §12 "test" step (gap → author → **test** → publish). Independently useful per-instance too. 2. **Deterministic workflow primitives** — Sequential / Parallel / Loop as named **topology archetypes** (the compiler already builds DAGs; these are blessed patterns). Reinforces topology-as-data + "LLM does language, code does the doing". 3. **Agent-as-skill / A2A interop** — model a remote A2A agent, a sub-swarm, or *another instance's* swarm as a **coordination skill** (keeps "skills are the only extension primitive"). Builds on §18 — there is no A2A code yet; `_delegation.py` is the compiler's internal child delegation, not an A2A adapter (corrected 2026-09-16; see `a2a-interop.md`); also enables cross-instance delegation. 4. **Uniform interception surface** — SwarmKit *already* intercepts uniformly via `GovernanceProvider` + OTel spans at every model/tool/skill/agent call (more principled than ADK's ad-hoc callbacks). The only borrow: ensure **eval + self-improvement signals hang off the same contract**, not a side channel. 5. **Trace inspection** — per-instance in the existing serve/UI (edge/dev, Minder); fleet-wide in the control-plane cockpit. Explicitly **not** borrowed: managed GCP/Vertex deploy (collides with invariant #4); Gemini-first defaults (ModelProvider abstraction already covers multi-provider). ## Reconciliation with SwarmKit's principles - **Invariant #4 (no lock-in):** the control plane is **OSS + self-hostable**, OTel-standard; **a managed backend is an optional managed backend, never required**. This is the open-source answer to "a managed backend is the aggregation layer." - **No central SPOF for execution:** the control plane is for evolution/observation only. Instances run fully with it down — buffer telemetry, apply no new approved changes. It is never a runtime dependency. - **Invariant #7 (eject intact):** additive observability/evolution; an ejected swarm still runs standalone. The control plane never participates in execution. - **§8 governance unchanged:** approval flows through the same reserved-scope human gates; audit stays append-only; distribution artifacts are signed + provenance- verified. - **§12.5 (no autonomous self-modification):** the plane **proposes**; humans **approve**; only then distribute. Fleet aggregation changes the *evidence*, not the *consent model*. - **§9 components:** adds an **optional fourth component** — `swarmkit-control-plane` — beside runtime / UI / schema. The existing per-instance UI dashboard is the *single-instance* surface; the control plane is the *fleet* surface. Same UI codebase can point at either. - **§15.3 / invariant #8:** the control-plane cockpit is an **operational + governance surface** (observe / measure / review-and-approve), distinct from the deferred **composer/authoring UI**. It shows plans and captures approval; the authoring mechanics stay conversational/CLI. (One-line clarification to add to §15.3.) ## Phased roadmap (additive; each its own design note + PRs) 1. **Eval harness** (standalone value, no fleet needed) — eval-set schema, `swarmkit eval`, decision-skill scoring, result storage. *Unlocks the "test" gate + the "Measure" signal.* 2. **Fleet aggregation** — instance/tenant resource tagging (mostly config), a documented OTel Collector + backend deployment, and a **semantic-summary ingest** (audit + eval + gaps) into a control-plane store. *Read-only fleet observability.* 3. **Self-improvement cockpit** — fleet trace/eval views + the plan generator (gap-mining → proposals) + the approval queue (existing reserved scopes) + signed artifact distribution (pull). *The loop closes.* 4. **Parallel small items** — workflow archetypes (Seq/Par/Loop); A2A-agent / cross-instance-swarm as a coordination skill. ## Open questions 1. **Distribution: pull vs push.** Pull (instances poll for approved artifact versions) is simpler + firewall-friendly for edge appliances; push needs reachability. Default **pull**. 2. **Control-plane store.** Reuse the Supabase/Postgres direction from [[distributed-architecture]] for the semantic store, or keep it separate? 3. **Tenant isolation** in aggregation (multi-customer): resource-attribute scoping + per-tenant approval queues. 4. **Edge appliances (Minder):** opt-in fleet enrollment; a homeowner's box probably *doesn't* join a fleet, but a fleet operator (a security company running many) would. Enrollment + consent model. 5. **OSS vs managed boundary:** what's in the OSS control plane vs the managed layer (this note: OSS = the full loop, self-hostable; a managed backend = managed + scale). ## Test / demo plan - **Eval harness:** an eval set against a reference topology scores trajectory + response; a regression run flags a drop. Standalone, unit-testable. - **Aggregation:** two `swarmkit serve` instances → Collector → control-plane ingest shows both instances' runs in one fleet view, correctly tagged. - **Self-improvement:** seed a repeated gap across instances → the plan generator proposes a skill → it lands in the approval queue (human-gated) → approve → the new skill distributes and a target instance picks it up. End-to-end, governed. - **Degradation:** kill the control plane → instances keep running + buffer telemetry; no new approvals apply; recovery drains the buffer.