# SwarmKit

> SwarmKit is the open-source AI platform runtime: agents, tools and governance defined as data (YAML/JSON) and run under real gates with every step recorded — from a single agent to a multi-agent swarm. Topology is data; skills are the universal extension primitive; governance is built in via Microsoft AGT; swarms grow by observing their own capability gaps and authoring new skills through conversation with human approval at every step.

This file is also published on the docs site at <https://delivstat.github.io/swarmkit/llms.txt>. New here? Read **[Building swarms — the complete playbook](https://delivstat.github.io/swarmkit/guides/building-swarms/)** for the ordered build recipe, then this file for the compact map with inline schemas.

**Two tiers.** This `llms.txt` is the compact, link-first map. Its companion **[`llms-full.txt`](https://delivstat.github.io/swarmkit/llms-full.txt)** inlines the *full text* of the playbook, all 11 artifact references, and the core design notes into one file — fetch it once and you have the entire corpus, no link-following. Use `llms.txt` to navigate; use `llms-full.txt` when you want everything in context at once.

**Status:** runtime v1.256.0 (1.0.0 was 2026-04-26); `swarmkit-schema`, `swarmkit-webui` and `swarmkit-control-plane` version independently. Phases 1–5 of the implementation plan shipped: the runtime, CLI and `swarmkit serve`; harness executors (M19); the topology canvas (M20); the fleet control plane. **Sequencing lives in your application, not in SwarmKit** — the bundled pipeline (`kind: StageGraph`, the saga controller, `swarmkit orchestrator`, `swarmkit pipeline`, `POST /pipelines/*`) was removed in 1.189.0; what stays is one bounded governed run, its gates, and its record, with `examples/pipeline-orchestrator/` as a reference application that imports no runtime module. What SwarmKit is, in one line each: reusable `Funnel` gates (`validate → judge → review → approve`; only humans approve; a gate **defers** the run and `POST /jobs/{id}/resume` continues it), multi-party approval over a `RoleRegistry`, `Contract` locks as a checked vocabulary, correlated runs and an append-only audit; skills as the one capability primitive (`mcp_tool`, `llm_prompt`, `composed`, `command`, `agent`) with `requires:` ordering enforced at the permission seam; 12 declarative model providers over four wire-format families; MCP with Docker sandboxing, permission tiers and per-tool `effects`; harness executors as declarative `adapter.yaml`; memory on by default (governed + workspace); one storage service (SQLite or Postgres) behind every store; OpenTelemetry, intent-drift detection, audit redaction, circuit breakers; a self-hosted fleet control plane; the `swarmkit-skills` catalogue reachable from `swarmkit skill` and the portal. **11 canonical artifact schemas** (topology, workspace, archetype, skill, funnel, contract, role-registry, trigger, executor-adapter, model-provider + the embedded approval-policy). Reference libraries: 3 reference topologies, 27 skills, 16 archetypes, plus the SDLC example workspace. Recent changes are listed by version in the next section; the [changelog](https://delivstat.github.io/swarmkit/releases/changelog/) has every release.

## Recent changes (newest first, by runtime version)

- **1.256.0** — A2A budget forwarding (a2a-federation.md, slice 2). When a SwarmKit `agent` skill calls a remote SwarmKit agent, it now forwards its **remaining allowance** (envelope minus spent — `max_cost_usd`, `max_turns`) in `message.metadata.swarmkit.budget`; the callee installs it as the child run's circuit-breaker limits, taking the **stricter** of the forwarded budget and its own config so a caller can only cap the child, never loosen it. Advisory across the trust boundary — a SwarmKit callee honours it (its Agent Card now advertises `honors_budget: true`), a non-SwarmKit agent ignores the extra key. Enforcement is the callee's; the caller records what it asked. Threaded through `run(budget_override=...)`. Usage-return (slice 1) already shipped.
- **1.255.0** — worker fairness (worker-fairness.md, follow-up to the scaling review). Workers no longer share one undifferentiated FIFO: each queued job carries a `job_class` (`model` | `harness`, derived from whether any agent in the topology runs on a harness executor) and a `priority` (from a `priority` label, higher first). `swarmkit worker --class model|harness|any` claims only its class, so a burst of long harness runs can't starve short model runs; the claim orders by `priority` then `created_at`. Two additive `jobs` columns; all-in-one and no-`--class` behaviour unchanged. Per-tenant quotas/weighted fairness named as a further follow-up.
- **1.254.0** — `serve --profile production` (production-profile.md, from the scaling/security review's "production defaults too permissive"). A fail-closed startup preflight that refuses to start — listing every gap at once — unless the deployment is safe to expose: real auth (not `none`/anonymous, on any bind), not `--insecure`, a persistent `SWARMKIT_OAUTH_KEY` (so the token-encryption key does not regenerate on restart and invalidate stored tokens), every declared `mcp_servers` entry `sandboxed: true`, and no wildcard CORS. A deployment-time assertion, not a request-time gate (governance is already deny-by-default); `--profile standard` (default) is the unchanged permissive behaviour. `create_app(profile="production")` so embedders inherit it.
- **1.253.0** — topology `input_schema` (input-schema.md). An optional JSON Schema (draft 2020-12) on a topology that the caller's input must satisfy *before the run starts* — validate-and-reject at the single choke point (`WorkspaceRuntime.run`), so every entry inherits it: 422 over HTTP, non-zero exit on the CLI, a JSON-RPC error over A2A, and no LLM spend on a malformed request (serve rejects at submit, so it never becomes a job). An object schema requires JSON input; `{"type": "string"}` accepts plain text. Emits `input.validated`/`input.rejected` audit events. Symmetric to `output_schema` but validate-and-**reject** (a caller cannot be re-prompted mid-run). Added to `topology.schema.json` (schema 1.46.0 / TS 0.23.0); opt-in, a topology that omits it is unchanged.
- **1.252.0** — worker fencing tokens (worker-execution.md). Every durable jobs-row write from a run is now fenced on `worker_id`, not just the final `complete()`: a worker threads its id through `execute_job` (`update_job(..., fence_worker_id=me)` → `WHERE id AND worker_id = me`), so a zombie (lease expired, run reclaimed) can't even transiently flip the new owner's row on its intermediate/terminal writes. All-in-one serve passes no token (writes stay unconditional). Guarantee: at-least-once execution, effectively-once durable record. Remaining unfenced surface (checkpointer per-node writes, run_usage rows) named for a strict exactly-once follow-up.
- **1.251.0** — queue observability (queue-observability.md, follow-up to the scaling review). Two additive `jobs` columns — `claimed_at` (when a worker claimed the run) and `started_at` (when execution began) — decompose the lifecycle with `created_at`/`completed_at`: queue wait vs execution latency. New `GET /queue/stats` and `swarmkit queue-stats <workspace>` report backlog depth, oldest-unclaimed age, queue-wait/execution p50/p95 over recent completions, and depth by topology (engine-agnostic — percentiles computed in Python, so SQLite works too). The portal Jobs-view stat strip lands with the batched UI slice.
- **1.250.0** — worker-execution hardening (from an external scaling review). `PostgresJobQueue.complete()` is now **fenced on `worker_id`**: a worker whose lease expired and whose run was reclaimed by another can no longer complete it and reset the new owner's lease (the reclaim→double-execution cascade). Delivery guarantee stated precisely — at-least-once at the checkpoint boundary. And `serve --role api` **bounds the queued backlog**: `--max-queue-depth` (default 10,000; 0 = unbounded) returns 429 once that many runs are `queued`, instead of accepting into an unbounded queue no worker may reach. Parked (deferred/stopped) runs release their worker and resume later as a fresh claim, so they consume queue rows, never worker slots.
- **1.249.0** — the API tier's SSE stream now follows a worker's job to completion. After the API/worker split, `GET /jobs/{id}/stream` on a `serve --role api` process held no live copy of a worker-run job and replayed a `queued` snapshot then closed; it now follows the durable row (re-reading on an interval, emitting new events) until terminal, so the API tier relays a worker's progress. Plus Run 5 in `docs/site/reference/load-and-scale.md`: the payoff measurement — `serve --role api` + N `swarmkit worker` on Postgres, throughput scales with worker count (5.9× at 8 workers end-to-end, corroborated by an isolated backlog-drain), lifting the single-event-loop ceiling Runs 1–4 found; sublinear (shared Postgres + one box's cores) and honest about it. The load driver (`examples/loadtest/`) gained an SSE `--stream` mode and a fixed elapsed-time throughput fix.
- **1.248.0** — the API/worker split for worker execution (`design/details/worker-execution.md`, slice 2). `swarmkit serve --role api` accepts and enqueues a run as `queued` (resolving topology and attachments up front, so a bad request is still a 4xx on submit) and returns without executing it; the new `swarmkit worker` command runs a loop that claims a queued job, executes it on its own event loop and DB engine via the exact serve path (`execute_job`, so status/output/usage/audit are identical), heartbeats to hold its lease, and reclaims runs abandoned by dead workers (which resume from their checkpoint). Throughput scales with worker count, past the single-event-loop ceiling the load benchmark found. The all-in-one `swarmkit serve` is unchanged and stays the default; `--role api` requires Postgres. Mind the connection budget: physical connections are `workers × (pool + overflow + checkpointer)` — size `SWARMKIT_STORE_POOL_SIZE` down or front Postgres with PgBouncer past a handful of workers.
- **1.247.0** — the durable job queue for worker execution (`swarmkit_runtime.queue`, `design/details/worker-execution.md`), slice 1: a `JobQueue` interface and `PostgresJobQueue` over the `jobs` table — atomic claim (`SELECT … FOR UPDATE SKIP LOCKED`), lease + heartbeat, `reclaim_expired` (an abandoned worker's run returns to the queue with `attempt` bumped and resumes from its checkpoint, not restarted), and a clean refusal on SQLite (single-writer, cannot be claimed across processes). New `jobs` columns `worker_id`/`lease_until`/`attempt` (additive). No serve behaviour change yet — the API/worker split and a worker command are the next slice.
- **1.246.0** — concurrent first-runs build the LangGraph checkpointer once (an `asyncio.Lock`), not in a race: on SQLite, two tasks creating the checkpoint tables at the same time hit `database is locked` — the same single-writer contention that shapes the SQLite/Postgres differences elsewhere. Fixes a flaky `test_concurrent_first_runs_compile_once`.
- **1.245.0** — the store connection pool is configured by the storage service (the single owner of storage config), not inside `make_engine`, which is now a pure factory taking `pool_size`/`max_overflow`. Same env knobs (`SWARMKIT_STORE_POOL_SIZE`/`_MAX_OVERFLOW`) resolved in one place; the shared serve engine gets the server-oriented default (20/10), and direct engine constructors get the driver default.
- **1.244.0** — the store connection pool is configurable (`SWARMKIT_STORE_POOL_SIZE`, default 20; `SWARMKIT_STORE_MAX_OVERFLOW`, default 10; Postgres only, one shared pool per database), and the write-through audit `INSERT` runs off the event loop (`asyncio.to_thread`). Load Run 3 (load-and-scale.md): moving only the audit write off the loop did not shift the throughput knee — a run does several other synchronous store writes on the loop (job create/update, usage, trace), so the full win needs all of them off-loop or an async driver; a bigger pool is necessary for scale-out, not sufficient alone.
- **1.243.0** — the compiled LangGraph graph is cached per topology and reused across runs (it was rebuilt every run — pure CPU on the serve event loop); a reload builds a fresh runtime, so the cache is invalidated for free. Measured against the load baseline (load-and-scale.md, Run 2): compile caching is correct and removes redundant per-run CPU but did not move the throughput knee — the dominant cost is the synchronous per-event store/audit writes on the loop (default pool size 5), which is the next optimization.
- **1.242.0** — a load-test harness and the first published NFR numbers (`examples/loadtest/`, `docs/site/reference/load-and-scale.md`): `SWARMKIT_MOCK_LATENCY_MS` (+ jitter) makes the mock provider sleep per call so a benchmark measures the runtime, not an instant mock; three topology shapes (tiny/typical/mcp-heavy) and a dependency-light driver (ramp + admission storm, RSS/fd sampling from /proc). Headline finding: a single serve process is bounded by the synchronous per-run work (compile + governance + write-through audit persist) on its one event loop — throughput plateaus ~2.5–5 runs/s and latency grows linearly past a knee near c=5–10, so scale horizontally (N processes on one Postgres) for throughput; `max_concurrent` bounds latency, it does not buy throughput. Admission (429) is correct but slow under load.
- **1.241.0** — `swarmkit upgrade` (`design/details/upgrade-command.md`): upgrades the local install in place, re-applying the extras it detects (`[ui]`, `[postgres]`, provider SDKs), after showing any breaking changes between the installed and target version and asking before it installs — a breaking upgrade must be a deliberate yes. `--check` reports and exits non-zero if behind (never installs); `--to X.Y.Z` pins; Docker and unrecognised installs are refused with the exact command rather than driven. Breaking versions are a curated, bundled list (`_breaking_changes.py`), seeded with 1.189.0 (pipeline removed) and 1.199.0 (readonly effects).
- **1.240.0** — A2A federation between SwarmKit instances (`design/details/a2a-federation.md`): a SwarmKit callee's Agent Card advertises a federation extension (so the portal's Add-remote-agent badges it and `GET /api/a2a/probe` returns `is_swarmkit`), and a completed/failed/canceled A2A task carries `metadata.swarmkit.{run_id, usage, observability}` — the caller records an `a2a.remote_usage` audit event linking the two runs and the remote's token/cost, and can pull the remote's events/audit by run id. A non-SwarmKit remote ignores the extra keys. Passing a budget on the call so the callee honors it is the next slice.
- **1.239.0** — the audit is a write-through journal (`design/details/audit-event-journal.md`): each event is persisted the moment it is recorded, not batched at the run boundary, so a run that crashes (SIGKILL, OOM, power loss) still leaves its trail up to the crash and `GET /events` shows it. Before this, a hard kill wrote nothing while the checkpoint survived — recovery was durable, the record was not. The end-of-run write stays as an idempotent completeness net (the store dedups on event id).
- **1.238.0** — a `kill -9` mid-harness regression test (`test_kill9_recovery.py`, `design/details/failure-path-evaluation.md`): a run killed with SIGKILL resumes from its checkpoint and completes (the interrupted node re-runs, the checkpointed one does not). It documents a real limit — the audit trail is flushed at the run boundary, so a hard kill loses the killed attempt's audit while the checkpoint stays durable and the resumed run's record is complete. Opt-in `SWARMKIT_MOCK_DELEGATE=1` makes the mock provider delegate to children, so a `swarmkit run` on the mock traverses a multi-agent topology (off by default; nothing that counts calls changes).
- **1.237.0** — `swarmkit storage status`, `swarmkit system` and `GET /storage` report a store whose configuration cannot be honoured (a postgres backend whose `${VAR}` URL is unset) as an `UNRESOLVED` row plus one sentence per cause, exit 2, and `system` still prints the environment section — instead of a traceback from the diagnostic command. A run still refuses.
- **1.236.0** — a workspace reload (`POST /reload`, a portal edit, `skill add` over HTTP) hands MCP sessions to one owner task and swaps the runtime under a lock: the new runtime's servers start before the old one's close, and a reload with `mcp_servers` configured no longer takes serve down. The portal's **Skills → Library** tab (webui 0.24.0) is the catalogue with a search box: Add shows both fragments before writing, a `SKILL.md` pastes in, Check runs from a button.
- **1.235.0** — `swarmkit skill` (`design/details/skill-registry.md`): `search`/`list --available` over the `swarmkit-skills` catalogue (verified nightly; cached a day; `SWARMKIT_SKILLS_CATALOGUE` points at a checkout or mirror); `add <skill|bundle|file>` writes the skill file and the `mcp_servers` entry after showing both (`--dry-run`, `--yes`; comment-preserving, rolled back if the workspace would not load, idempotent); `import <SKILL.md>` converts an Agent Skills file to an `llm_prompt` skill; `check` starts each `mcp_tool` skill's server and confirms the tool exists; `remove` refuses while an agent or archetype holds the skill. Same over HTTP at `/api/skill-catalogue` and `/api/skills/{add,import,check}`. A skill declares the runtime floor it needs (`provenance.requires_runtime`) and is refused, naming the version, when it is not met.
- **1.234.0** (control-plane 0.50.0, fleet UI 0.14.0) — `swarmkit connect --join-code` saves its credential (`~/.swarmkit/connect/<pair>.json`, 0600) and a later start with no flags resumes; `/usage` `by_model` carries the `provider` that billed each model; the panel refuses a command whose args the connector could not address (400 naming them); `/fleet/state` carries `a2a` and the fleet inventory shows it.
- **1.233.0** (schema 1.45.0) — **memory is on by default** (`design/details/memory-by-default.md`): a workspace that says nothing binds `memory-reader` before every agent and `memory-writer` after (advisory) and gets the bundled `governed-memory` + `memory-reconcile` skills; a `memory:` block tunes the reader/writer or switches everything automatic off (`enabled: false`); an explicit binding is used as written; writing curated memory is still a per-agent grant; `GET /memory/config` and the portal's Memory page show what is in force.
- **1.232.0** (control-plane 0.49.0, fleet UI 0.13.0; `design/details/control-plane/28-operator-identity-to-instance.md`) — a fleet panel resolves a multi-party approval **as the signed-in operator**: under a human-issued `approve-as` membership the instance honours a signed assertion of the operator's OIDC subject (`X-Fleet-Actor`, bound to the item and a 300 s window), the role registry decides membership, and the audit records both the person and the fleet that relayed the click.
- **1.231.0** — `swarmkit serve` needs no extra: JWT auth and cron triggers are base dependencies and `[serve]` is an empty, deprecated alias; `[ui]` is the portal.
- **1.230.0** (control-plane 0.48.0, fleet UI 0.12.0) — a fleet sync pulls the instance's skill gap log (`GET /gaps`) and audit tail (`GET /audit?since=`, cursored per instance) into the panel's Gaps and Runs views; `/fleet/state` names funnels, contracts and role registries — funnels and contracts adopt and deploy, role registries adopt only.
- **1.229.0** (control-plane 0.47.0) — a fleet deploy writes the adopted file's text verbatim (comments and layout intact, refused if it does not parse to the signed content); `/fleet/state` carries each artifact's `yaml`; the panel's drift is hash-compared against what a sync observed.
- **1.228.0** (control-plane 0.46.0, fleet UI 0.11.0) — a funnel's multi-party role-task is resolved only through `POST /review/{id}/resolve` as a member of the role (the generic approve/reject answer 409); the fleet panel resolves it with an outcome, relays the instance's refusal, and reads `deferred`/`stopped`/`interrupted` as what they are.
- **1.227.0** — everything a run records goes through the storage service: conversations, workspace memory and the skill gap log moved off `.swarmkit/` files onto the configured store next to jobs, audit and governed memory. Serve's MCP endpoint answers at `/mcp/` behind the server's auth; a signed webhook (`hmac`, `bearer`, `api_key`) is admitted past the API-key gate and verified by its trigger; a cron trigger's `config.timezone` and `config.input` are honoured; two versions of one topology live side by side as `name` and `name@version` for the canary router; an agent calling a tool it does not hold gets a usable result, a `skill.gap` audit event, and a row in `swarmkit gaps`.
- **1.221.0–1.225.0** — A2A both ways: the server (agent card + `POST /a2a`, opt-in) and the `agent` skill type (a topology here as a child run, or a remote agent through its card; `pack:workspace`; the portal's remote agents; harness nodes call agent skills through the gateway). Section below.
- **1.216.0** — communication is the application's: `GET /events?after=<cursor>` is the durable log, `events:` sinks push best-effort, a resolved gate resumes its run; the Slack/Discord/Telegram providers left the runtime.
- **1.199.0** (BREAKING) — `permission: readonly` decides by declared `effects`, not tool-name substrings. Section below.
- **1.197.0–1.198.0** — command packs (`implementation.type: command`) and bulk grants (`pack:`, `server:`). Section below.
- **1.189.0** — the bundled pipeline layer removed. Section "Sequencing" below.

## Start here

- [Docs site](https://delivstat.github.io/swarmkit/): the published documentation (this llms.txt lives at `/llms.txt` there).
- [Memory and decision-skill bindings](https://delivstat.github.io/swarmkit/guides/memory-and-decision-skills/): the two memories, turning governed memory on, and `enabled` vs `required`.
- [Building swarms — the complete playbook](https://delivstat.github.io/swarmkit/guides/building-swarms/): the ordered, step-by-step build recipe from one agent to a governed multi-app delivery flow. **Read this first if you are building a swarm.**
- [SDLC example (video walkthrough)](https://delivstat.github.io/swarmkit/sdlc-example/): the worked reference — a complete software-delivery lifecycle as data, toured on-screen. Recorded while SwarmKit still bundled a sequencer; the topologies, funnels, archetypes and gates are current, the stage-graph/controller parts are not.
- [Driving SwarmKit from your application](https://delivstat.github.io/swarmkit/reference/orchestrator-integration/): the HTTP contract for sequencing runs yourself — correlation, gates, defer/resume, artifacts.
- [README](https://github.com/delivstat/swarmkit/blob/main/README.md): project overview, what works today, milestone progress.
- [CLAUDE.md](https://github.com/delivstat/swarmkit/blob/main/CLAUDE.md): repo-wide invariants, feature-delivery workflow, release checklist.
- [SwarmKit Design v0.6](https://github.com/delivstat/swarmkit/blob/main/design/SwarmKit-Design-v0.6.md): the authoritative architecture. §5 (Core Concepts), §6 (Skills), §8 (Separation of Powers), §14 (Runtime), §18 (MCP Integration).
- [Implementation plan](https://github.com/delivstat/swarmkit/blob/main/design/IMPLEMENTATION-PLAN.md): the phased roadmap to v1.0, updated as features land.

## Build a swarm (the recipe)

The [playbook](https://delivstat.github.io/swarmkit/guides/building-swarms/) walks each step with runnable artifacts; the compressed arc:

1. **Scaffold** — `uv tool install "swarmkit-runtime[ui]"` (uv is the recommended install; the server is part of the runtime; `[ui]` is the portal it hosts — absent, `swarmkit serve` runs headless; `[postgres]` the Postgres backend; the bare package is the CLI), then `swarmkit init` (conversational authoring) or hand-write a `Workspace`.
2. **One agent** — a `Topology` node instantiating an `Archetype` (model/prompt/IAM defaults). `swarmkit run <topology>`.
3. **A skill** — attach a `Skill` (capability / decision / coordination / persistence) with a structured `outputs` schema.
4. **Many agents** — add nodes + `depends_on`; coordinators use structured delegation (`create-task-plan`).
5. **Tools** — connect MCP servers in the workspace; every tool call is governed.
6. **Governance** — decision skills + `GovernanceProvider`; reserved human-only scopes; append-only audit.
7. **A quality gate** — a `Funnel` (`validate → judge → review → approve`); automated layers filter + retry, only humans approve.
8. **A harness node** — an archetype `executor: { kind: harness, ref: claude-code }` for a real diff-producing coding agent.
9. **Order the tool calls** — `requires:` on the agent, so a guarded skill is refused until its prerequisite has actually run (a rule in a prompt is a request; this is mechanism).
10. **Multi-party approval** — a `RoleRegistry` + a gate's embedded `ApprovalPolicy` (quorum + four-eyes floor).
11. **Sequence it from your application** — start each run over `POST /run/{topology}` with a shared `correlation_id`; a gated run parks as `deferred` and `POST /jobs/{id}/resume` continues it. SwarmKit runs the bounded work and keeps the record; your application decides what comes next (`examples/pipeline-orchestrator/`).
12. **Trigger + serve + observe + grow** — a `Trigger` delivers a signed webhook into `swarmkit serve`; trace/why/ask observe the run; `swarmkit gaps` → author the missing skill.

Then validate the whole thing before running it: `swarmkit validate .` — add `--require` to fail on configuration nothing reads, and `--require-verified` to fail when a topology root's output is checked by nothing.

## CLI commands

Author & run: `init` (scaffold via conversation), `edit`, `validate` (`--require` reachability, `--require-verified` funnel strength), `run` (`--verbose`, `--dry-run`, `--correlation-id`, `--label k=v`, `--supersedes <job-id>`, `--save-artifact`, `--resume` — the last checkpointed run), `chat` (multi-turn, `--resume`), `conversations` (`--pick`), `eval` (score a topology against an eval-set), `checkpoints` (resume checkpointed runs). Serve & connect: `serve` (HTTP: async jobs, SSE, auth, MCP, webhook ingress, canary), `connect` (edge poll connector). Observe: `status`, `logs` (`--run-id`, `--agent`, `--format markdown`), `trace` (call graph + token counts), `why` (LLM post-mortem), `ask` (`--run` scoping), `debug` (local prompt ring buffer), `stop <run-id>` (ask a run to stop at its next agent boundary — cooperative, keeps what it has done, resumable, and works across processes because it writes a durable flag). Grow & govern: `gaps` (recorded capability gaps), `review list/show/approve/reject/resolve` (the human relay/approval inbox; `list --kind role_task --gate <id>` narrows it; `resolve --as <identity> --approve|--reject|--changes-requested -m "why"` casts a multi-party role-task, checked against the role registry; `approve`/`reject` take `-m` too). Artifacts: `artifacts get|list` (read a run's saved output by ref). Knowledge & packaging: `knowledge-server`, `knowledge-pack` (`--lean` ≈190k tokens — overview, generated reference, schemas, design doc, guides; full ≈610k adds every design note, historical ones last), `docs-reader`, `install` / `packages` / `publish` (expertise packages). Providers & adapters: `providers list|show` (every declared model provider, its family, whether its key is set), `adapters list|show|approve|build` (harness adapters and their launch-approval status). Memory: `memory add|get|search|quarantine|resolve` (governed memory; `add` writes through the same reconcile path an agent uses). Trust: `trust list|apply|clear` (allowlist changesets proposed by repeated approvals). Storage & system: `storage status|migrate`, `system`, `upgrade` (upgrade the local install — detects method + extras, shows breaking changes in range, asks before installing; `--check`, `--to`, `--yes`). Fleet: `fleet enroll-token|memberships`, `auth token`. Delivery checks: `cited-change`, `slice-check`, `comprehension`. All 69 commands with their help lines are generated from the CLI itself into [the CLI reference](https://delivstat.github.io/swarmkit/reference/cli/).

## Schemas

Eleven canonical artifact schemas (JSON Schema 2020-12) — ten standalone artifact kinds plus the embedded `approval-policy`. Each has a published reference page (linked below).

- [topology.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/topology.schema.json): a bounded swarm run — [reference](https://delivstat.github.io/swarmkit/reference/topology/).
- [workspace.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/workspace.schema.json): the root manifest — [reference](https://delivstat.github.io/swarmkit/reference/workspace/).
- [archetype.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/archetype.schema.json): a reusable agent template — [reference](https://delivstat.github.io/swarmkit/reference/archetypes/).
- [skill.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/skill.schema.json): the capability-extension primitive — [reference](https://delivstat.github.io/swarmkit/reference/skills/).
- [funnel.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/funnel.schema.json): a reusable per-artifact quality gate — see Funnel below; [reference](https://delivstat.github.io/swarmkit/reference/funnel/).
- [contract.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/contract.schema.json): an integration contract — the agreed interface between apps, and the checked vocabulary a sequencer's locks name; see Contract below; [reference](https://delivstat.github.io/swarmkit/reference/contract/).
- [role-registry.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/role-registry.schema.json): named roles → member identities + the scopes they confer; how approval rules resolve to people; [reference](https://delivstat.github.io/swarmkit/reference/role-registry/).
- [trigger.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/trigger.schema.json): an external event source that starts a topology or delivers a signed webhook event; [reference](https://delivstat.github.io/swarmkit/reference/trigger/).
- [executor-adapter.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/executor-adapter.schema.json): a declarative `adapter.yaml` running a coding harness as a node — data, not per-harness Python; [reference](https://delivstat.github.io/swarmkit/reference/executor-adapter/).
- [model-provider.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/model-provider.schema.json): a declarative model provider — base URL, auth env var, catalogue, narrow-only capabilities, layered over one of four wire-format families (`openai-compatible`, `ollama`, `anthropic`, `google`); a new endpoint is a YAML file, no Python — [reference](https://delivstat.github.io/swarmkit/reference/model-provider/).
- [approval-policy.schema.json](https://github.com/delivstat/swarmkit/blob/main/packages/schema/schemas/approval-policy.schema.json): **embedded config** (no `kind`) inside a gate's `approve:` — multi-party rules, quorum, four-eyes floor; [reference](https://delivstat.github.io/swarmkit/reference/approval-policy/).

## Funnel (per-artifact quality gate)

A **Funnel** is a first-class artifact (`kind: Funnel`) — a reusable per-artifact quality gate that chains *structured-output validation → LLM-as-judge → optional harness review → multi-party human approval* into one composition. Referenced by id from a topology node's `funnel:` field, so the same gate applies to many nodes/stages. Every layer is optional except `approve`; present layers always run in the fixed order `validate → judge → review → approve`. The automated layers **filter and drive a bounded retry loop but never decide** — the only exit is through `approve`. The control flow is compiler-owned and fixed; a funnel configures the layers, it does not rewire the graph. On retry exhaustion the funnel escalates to a human with the last critique attached — it never drops the requirement or silently advances.

Layers: `validate` (deterministic, native structured-output validation + field-specific auto-correction — the judge never sees malformed input), `judge` (a decision skill scoring against a rubric with a `threshold` and bounded `max_retries`), `review` (optional heavyweight harness reviewer; findings at or above `route_back_at` retry, the rest attach to the human task), `approve` (required multi-party approval set — rules, quorum, `min_distinct_approvers`, `exclude_author`).

Minimal (degenerate to a plain multi-party gate):

```yaml
apiVersion: swarmkit/v1
kind: Funnel
metadata:
  id: design-signoff
  name: Design Sign-off
  description: A plain multi-party human approval gate on the design artifact.
approve:
  rules:
    - scope: design:approve
      roles: [tech-lead]
      quorum: all
provenance:
  authored_by: human
  version: 1.0.0
```

Full (all four layers):

```yaml
apiVersion: swarmkit/v1
kind: Funnel
metadata:
  id: consolidated-design-approval
  name: Consolidated Design Approval
  description: Full four-layer gate — validate, judge, architect review, then multi-party approval.
validate:
  schema: schemas/consolidated-design.json   # workspace-relative JSON Schema
  autocorrect: true                          # field-specific re-prompt before treating as a retry
judge:
  skill: artifact-judge                       # a decision-category skill (LLM-as-judge)
  rubric: rubrics/consolidated-design.md
  threshold: 0.8                              # score below this is a retry
  max_retries: 2                              # then escalate to a human, never drop
review:
  archetype: architect-reviewer               # harness reviewer archetype
  read_scope: [app:oms, app:web, app:mobile]  # read-only IAM scopes for the investigation
  route_back_at: high                         # findings >= this retry; lower ones attach
approve:
  rules:
    - scope: design:approve
      roles: [oms-lead, web-lead, mobile-lead]
      quorum: all
    - scope: security:approve
      roles: [infosec-lead]
      quorum: all
  exclude_author: true                        # segregation of duties (default)
  min_distinct_approvers: 2                    # four-eyes floor
provenance:
  authored_by: human
  version: 1.0.0
```

- [Funnel reference](https://github.com/delivstat/swarmkit/blob/main/docs/site/reference/funnel.md): layers, the fixed control flow, the advisory (structural) invariant, referenced-by-id.
- [Gate funnel design note](https://github.com/delivstat/swarmkit/blob/main/design/details/gate-funnel.md): composition, control flow, bounded retry, the structural invariant, provenance bundle.
- [Example funnel artifact](https://github.com/delivstat/swarmkit/blob/main/examples/sdlc-pipeline/workspace/funnels/consolidated-design-approval.yaml): all four layers, referenced by id from an SDLC pipeline node.

## Sequencing (it lives in your application)

SwarmKit **used to** ship a sequencer: `kind: StageGraph`, a durable saga controller, `swarmkit orchestrator`, `swarmkit pipeline`, `POST /pipelines/*`. It was removed in runtime 1.189.0. The reason is the layering: sequencing across weeks is *application* logic — retries, business calendars, what an event means, when to give up — and every one of those pulled SwarmKit toward becoming a workflow engine. Do not write a `StageGraph`; a runtime that reads one no longer exists.

What SwarmKit keeps is the part that is genuinely its own: **one bounded governed run**, its gate, and its record.

- **Correlate** — `POST /run/{topology}` takes `correlation_id` ("same ticket"), `labels` (opaque `{k: v}` reaching `jobs` *and* `audit_events`), and `parent_job_id` ("this run replaces that attempt"). Runs are independent and connected by a correlation id — not stages of a graph the runtime knows about. Walk `parent_job_id` to answer "what did this artifact really cost across retries".
- **Park and resume** — a funnel's `approve` layer raises `HITLDeferredError`: the run checkpoints, the job goes `deferred`, and its `error` names the gate. Nothing stays resident. `POST /jobs/{job_id}/resume` (409 unless parked) or `swarmkit run <ws> <topology> --resume` (the workspace's last checkpointed run; `swarmkit checkpoints` lists them) continues it, and a resumed run can park again identically.
- **Stop a run you no longer want** — `swarmkit stop <run-id>` or `POST /jobs/{job_id}/stop` writes a durable flag the run reads at its next agent boundary, then raises `RunStoppedError` (a subclass of the deferral, so there is exactly one resumption path). Status `stopped` — not `deferred`, which means waiting on a decision that will arrive, and not `failed`, since nothing went wrong. Cooperative: a call in flight finishes first, and the CLI says so rather than implying a kill. A resume clears the flag; the act is audited as `run.stopped` with who asked.
- **Read the gate** — `GET /gates/{gate_id}` returns `status` with the **approval policy already applied** (quorum, `min_distinct_approvers`, `exclude_author`) plus `items` and `quorum_evaluated`. A gate id is `<run_id>:<agent_id>`, where `run_id` is the job id — split on the LAST colon. `GET /review?gate_id=…` is not a substitute: it returns role-tasks, and folding those into a decision means reading a funnel a client cannot see.
- **Fetch what is being approved** — `GET /artifacts/{ref}`, ref shaped `<correlation>/<run>/<name>`; a review item carries the one it is about. An approver deciding without the artifact is deciding on a title.
- **Receive events** — `POST /events/signal` is the surviving ingress seam (signed webhook → validated → opaque `correlation_id` extracted by JSONPath → handed to your listener). SwarmKit does not decide what an event *means*.

- [Extracting the pipeline](https://delivstat.github.io/swarmkit/design-notes/extracting-the-pipeline/): the removal — what went, what stayed, and the migration inventory.
- [Reading a gate, and approving without a saga](https://delivstat.github.io/swarmkit/design-notes/gate-state-and-deferring-approval/): the gate read, the deferral, and why the gate id had to become run-unique.
- [Driving SwarmKit from your application](https://delivstat.github.io/swarmkit/reference/orchestrator-integration/): the whole HTTP contract in one page.
- [`examples/pipeline-orchestrator/`](https://github.com/delivstat/swarmkit/tree/main/examples/pipeline-orchestrator): a reference application that sequences runs with **no `swarmkit_runtime` import anywhere in it**.

## Skill prerequisites (`requires:`)

An ordering rule stated in a prompt is a request; a model follows mechanism over instruction. The evidence: in a single run, same agent, same prompt, an ack-gated tool was called 4 times and a merely-requested one 0 times. So ordering is declared on the agent and **enforced**:

```yaml
skills: [list-build-conventions, get-build-convention, search-solution-code]
requires:
  get-build-convention: [list-build-conventions]
  search-solution-code: [get-build-convention, list-build-conventions]
```

A sibling block, not entries inside `skills` — `skills` stays a plain identifier array, duplicate rules are impossible because it is a map, and every ordering rule reads in one place.

- **Enforced at the MCP permission seam** — the one function both executors dispatch through, so a model agent and a harness agent behave identically; checked *before* the policy call, since an ordering refusal is not a policy question.
- **The refusal is actionable**, which is the part doing the work: `get-build-convention requires list-build-conventions, which has not been called in this session. Call list-build-conventions first, then retry.` The agent recovers inside its own loop; the server is never touched by the refused call.
- **Per `(run, agent)`** — a sibling agent's call does not satisfy this agent's prerequisite, because a prerequisite is about what is in *this* agent's context.
- **Only a successful call satisfies** — an exception or an MCP `isError` unlocks nothing. A tool that reports failure in its *payload* does satisfy; the seam cannot read meaning.
- **Validated at resolution** — a rule naming a skill the agent does not hold is `agent.requires-unknown-skill`; a cycle is `agent.requires-cycle` (an agent that can never recover is worse than no rule).
- Scope: guards skills that dispatch through the MCP seam. A refusal is audited as `skill.executed` with `policy_decision="deny"`, so a gate that is working is distinguishable from one never reached.

- [Skill prerequisites design note](https://delivstat.github.io/swarmkit/design-notes/skill-prerequisites/): the evidence, the shape decision, the semantics, and the non-goals (not parameterised, not cross-agent, not guarding decisions).

## Command packs (`implementation.type: command`, 1.197.0+)

MCP was never the extension paradigm — **skills** are, and `implementation.type` has always had several backings. A **command pack** is the local-binary sibling of an `mcp_server`: declare it under `command_packs:` in `workspace.yaml`, and a skill reaches it with `{type: command, pack: <id>, command: <id>}`. Governance is reused, not duplicated: the pack carries the permission tier, the skill carries `iam.required_scopes`, and the action is `command:call:<pack>:<command>` alongside the untouched `mcp:call:<server>:<tool>`.

```yaml
command_packs:
  - id: json-tools
    requires: [{ binary: jq, version: '>=1.6' }]   # checked at workspace LOAD, naming the binary
    permission: readonly
    timeout_seconds: 30                             # + timeout_overrides, max_output_bytes
    commands:
      - id: query
        argv: [jq, '-r', '{filter}', '{file}']      # argv, NEVER a shell
        effects: read                               # declared; undeclared means `write`
```

Five rules carry the design. **`argv`, never a shell** — a `{placeholder}` is filled with the *value* of an argument and stays exactly one argv entry, so `; rm -rf /` is an inert string; this holds structurally, not by escaping, because no code path re-parses it. There is deliberately **no generic `bash` skill**: `bash` is one action no policy can be written over, and SwarmKit already admits arbitrary execution through harness executors, contained by `_sandbox`/`_egress`/`_approval`/`_budget`/`_container`. **`effects` is declared per command and defaults to `write`** — nothing is inferrable (`curl` POSTs, `jq` and `sed` both take `-i`), so an unclassified command fails closed and `permission: readonly` is enforceable against a fact. **Secrets reach a command through the pack's `env` and never `argv`** — `{credential.*}` in an argv template is a schema error, so a secret cannot be model-placed, cannot land in the audit line recording what ran, and cannot be read from `ps`. **Bounds are never infinite** — an omitted `timeout_seconds`/`max_output_bytes` means the built-in default, and exceeding the ceiling FAILS rather than truncating, because a partial result read as complete is indistinguishable from a short one. **`requires` is checked at workspace load**, not at call time, so a missing binary names itself instead of surfacing as an exec error mid-run.

**Bulk grants (1.198.0+).** A command becomes an ordinary skill with the id `<pack>-<command>`, so the tool builder, `requires:` validation and the archetype merge need know nothing about packs. Grant them in bulk:

```yaml
skills:
  - pack:json-tools          # every READ command in the pack, now and later
  - server:filesystem        # every skill targeting that MCP server
  - json-editing-rewrite     # a write, named — bulk grants never carry one
```

`pack:` carries **reads only**. Adding a read command flows through to everyone holding the pack; adding a write reaches nobody, so a pack can never silently widen an agent that already holds it. `server:` makes no equivalent promise, because an MCP tool has no declared effect to filter on. A bulk grant matching nothing is a resolution error listing what is available — an agent silently granted no tools is indistinguishable from one whose model chose not to use them.

Bundled packs to copy: `reference/command-packs/` — `file-tools` (coreutils only, so it runs on a bare machine), `json-tools` (jq), `text-tools` (ripgrep). Declaring a pack is not granting it — the grant is the audit step. Design: `design/details/command-packs.md`. Demo: `just demo-command-packs`.

## Another agent as a skill (`implementation.type: agent`, 1.222.0+)

The fifth backing, and the A2A **client** side (`design/details/a2a-interop.md`). One skill type, two resolutions — exactly one of `topology: <id>` (a topology in this workspace) or `card_url: https://…/.well-known/agent-card.json` (a remote A2A agent). It is a skill like the other four: the same permission seam (`permission` tier + `effects` on the block, since there is no server/pack to inherit from; `readonly` allows only `effects: read`), `requires:` prerequisites, `iam.required_scopes`, and the call audited as `skill.executed`. The model's tool takes `{input, context?}` and gets the other agent's answer as text.

- **Local** — runs the target in-process as a **child of the caller's run**: own run id + trace, a job row with `parent_job_id` and `source: agent`, same correlation, the parent's MCP servers shared and never closed by the child. A missing target fails the workspace load (like a missing command pack). Depth capped at 3 (a cycle otherwise).
- **Remote** — the card is fetched on first use (cached; `skill_id` must be on it; 404 says "A2A may not be enabled there"); `credentials_ref` → a workspace `credentials` entry sent as a bearer; `message/send` with our run id as the A2A `contextId` (so two instances' records join on it), then `tasks/get` until terminal or `input-required`; past `timeout_s` the remote task is cancelled and the call fails.
- **`on_unanswerable: agent | relay | abort`** — the harness adapter's words, when the other agent asks a question. `agent` (default): the question is the tool result (`{"status": "input_required", "task_id", "question"}`) and the calling agent answers by calling again with `{task_id, answer}`; audited `executor.input_response` with `responder: agent:<id>`; after `max_agent_answers` (2) per task the next question relays. `relay`: a person, through the same `input_request` review item and bounded wait a harness question uses (`resolve_input`); no answer in time → call fails, remote task cancelled. `abort`: fail with the question. A **human gate** on the far side (a SwarmKit run parked on approval, `metadata.swarmkit.gate_url`) is never the agent's to answer under any policy — the result says `kind: human_gate` and names the gate.
- **No funnel on the skill**: a child topology runs its own funnels, a remote SwarmKit its own, the caller's funnel gates what the caller does with the result.
- **`pack:workspace`** (1.223.0+): every topology in the workspace is synthesized as an `agent` skill `topology-<name>` at registry build (as command packs synthesize theirs; `cautious`, `effects: unknown`; listed by `GET /skills`), so "the supervisor may run any topology here" is `skills: [pack:workspace]` — and a topology added later reaches every holder of the grant. Unlike `pack:<command-pack>`, it is not filtered to reads (running a topology is never `read`); every call still passes the tier and the audit. A hand-authored skill named `topology-<x>` is a collision error; the command-pack id `workspace` is reserved. A hand-authored `agent` skill targeting the same topology is untouched — that is where a stricter tier or a different `on_unanswerable` goes.
- **Portal (1.224.0+):** the Connections page lists remote agents next to servers and sinks (same status column) and "Add remote agent" takes a card URL → `GET /api/a2a/probe?card_url=` (the runtime fetches the cross-origin card; `supported: false` carries the reason) → pick a skill, `on_unanswerable`, credential, tier → `PUT /api/skills/{id}` writes the `agent` skill file. `GET /api/a2a/agents` lists them. Discovery is an authoring act: nothing is granted by adding one.
- **Harness nodes (1.225.0+):** an `agent` skill granted to a harness node is offered through the governed MCP gateway as the flat tool `agent__<skill>` (Claude Code sees `mcp__swarmkit__agent__<skill>`), with the same `{input, context}` schema; a call runs the skill's executor inside the run scope captured at registration, so the child run is attributed to the harness's run (`parent_job_id`), correlated and depth-bounded exactly as from a model node, and audited as the harness's `skill.executed` (a refusal is `policy_decision: deny`). A harness with agent grants and no MCP grants still gets a gateway.
- Not yet: parking the caller's run (defer) while a remote answer is awaited — today it waits, bounded.

## Skill catalogue (`swarmkit-skills`, separate repo)

<https://github.com/delivstat/swarmkit-skills> — MCP servers with the wiring already worked out: the `mcp_servers` block, the `permission` tier, an `effects` map per tool so `readonly` is enforceable, and `iam.required_scopes`. Organised as **bundles** — one server plus the skills that use it — because that is how a server is actually adopted, and it matches command packs.

**Each entry is started and asked, nightly.** A job launches every server and checks that the tool each skill names still exists, so an entry carries the date it last answered rather than a promise. Three states: `verified` (the server started and the tools were present, on that date), `broken` (a tool is gone — the entry stays visible, marked, with an issue filed), and `unverifiable` (needs a credential public CI cannot supply — reported honestly, because a green tick meaning "we did not look" is worth less than no tick). Only `broken` fails a run; failing on `unverifiable` would make every credentialed entry permanently red and teach everyone to ignore the check.

Bring one in with `swarmkit skill add <skill|bundle>` (1.235.0+; it shows the skill file and the `mcp_servers` entry before writing either, `--dry-run` prints them) or the portal's **Skills → Library** tab, then grant it. `swarmkit skill search <words>`, `show <id>`, `check` (start each server, confirm the tool still exists) and `remove` (refused while an agent holds the skill) are the rest of the command; `import <SKILL.md>` brings an Agent Skills file in as an `llm_prompt` skill. A `pack:` grant carries the bundle's READ skills — now and later — while a write is always named individually, so a bundle can never silently widen an agent that already holds it. Every skill declares `provenance.requires_runtime`, so one that outgrows your runtime is refused at workspace load naming both versions rather than failing mid-run. Design: `design/details/skill-catalogue.md`, `skill-catalogue-seed.md` and `skill-registry.md` (the command).

## `permission: readonly` needs declared `effects` (BREAKING, 1.199.0)

`readonly` used to decide write-ness by substring-scanning the **tool name** for `create|delete|update|write|put|post|set|add|remove|modify|edit|insert|drop|push|send`. It failed in both directions at once: `get_dataset` and `read_asset` matched **set**, `list_addresses` matched **add**, `get_post` matched **post** — ordinary reads, denied; while `truncate_table`, `purge_cache`, `revoke_token` and `wipe_db` matched nothing and were allowed. A longer word list was never the fix — the vocabulary of destructive verbs is unbounded and per-server.

Declare effects per tool instead:

```yaml
mcp_servers:
  - id: warehouse
    permission: readonly
    effects: { get_dataset: read, truncate_table: write }
```

Resolution order: the **declared entry wins** (it is the half the operator controls, and cannot change under them on a server upgrade), then the server's own `readOnlyHint` annotation, then `unknown`. **Under `readonly`, `unknown` is now DENIED** where it was previously allowed whenever the name missed the word list — the fail-closed direction, with a denial naming the tool and the field to add. Other tiers are untouched; `effects` is consulted only by `readonly`. Migration: `docs/notes/mcp-effects-migration.md`.

## Two checks before you run (reachability + verification)

Both are read-only, both come from **one compile of the workspace**, so they cannot disagree.

- `swarmkit validate --require` · `GET /workspace/reachability` — **configuration no code path can reach.** The recurring defect in this codebase was config that is declared, accepted, validated, displayed and loaded by *nothing*; the compiler now records what it actually built, on the line that builds it, and anything declared-but-unwired is reported by name.
- `swarmkit validate --require-verified` · `GET /workspace/verification` — **which topology roots produce an output nothing checks.** Strength counts *wired* funnel layers, never declared ones (counting a declaration would repeat the defect above); declared-but-inert layers are named. Only roots are findings — a leaf worker returning a fact to its parent is not producing a reviewable artifact.

```
verification: 15 topology root(s)
  deploy/deploy-coordinator (root): funnel deploy-approval — approve; declared but inert: validate
  oms-build-harness/builder (root): funnel oms-code-review — judge, approve
  12 topology root(s) produce an output that nothing checks — the run's answer is whatever the model said.
```

The two flags stay separate: "is my config wired" and "is my output checked" are different questions a CI job may want independently.

## Contract (integration contract between apps)

A **Contract** is a first-class artifact (`kind: Contract`) — the agreed interface between two (or more) applications, identified by id. It is what makes a sequencer's locks real: a lock **is** an integration contract, so `locks: [oms-web, oms-inventory]` mean "hold the OMS↔Web and OMS↔Inventory interfaces while I change them, so no concurrent requirement commits a conflicting version." Making the contract an artifact turns those lock ids from free-form strings (where a typo silently becomes a *different* lock and two requirements that should serialise don't) into a **checked, pickable vocabulary**: the resolver rejects a lock that names no contract, and the contention view ("which work fights over the same contract") is exact. A contract is **not executed** — your sequencer is the lock manager; the registry only makes the vocabulary real and records which apps each lock binds.

Fields: `parties` (required, ≥2 app ids — what makes it a contract, an interface *between* apps; drives the contention/ownership display; app ids are free strings, apps are not artifacts) and `interface` (optional — a pointer to where the interface spec itself lives, an API/event schema; not interpreted by core, it is documentation + a handle for reviewers). Core does not parse or diff the `interface` — identity + locking only, not interface compatibility.

Minimal:

```yaml
apiVersion: swarmkit/v1
kind: Contract
metadata:
  id: oms-web
  name: OMS ↔ Web order API
  description: The order-submission + status API OMS exposes to the Web storefront.
parties: [oms, web]            # the apps this contract binds (>= 2)
provenance:
  authored_by: human
  version: 1.0.0
```

Fuller (with an `interface` pointer):

```yaml
apiVersion: swarmkit/v1
kind: Contract
metadata:
  id: oms-inventory
  name: OMS ↔ Inventory reservation API
  description: The stock-reservation + release events OMS exchanges with Inventory.
parties: [oms, inventory]              # >= 2 app ids
interface: schemas/oms-inventory.json  # optional pointer to the interface spec; not parsed by core
provenance:
  authored_by: human
  version: 1.0.0
```

- [Contract reference](https://github.com/delivstat/swarmkit/blob/main/docs/site/reference/contract.md): the fields, the locking/contention framing, referenced-by-`locks`, and the runtime ref-check.
- [Integration-contract registry design note](https://github.com/delivstat/swarmkit/blob/main/design/details/contract-registry.md): why lock ids become a checked vocabulary, the lock ref-check, and the non-goals (no interface-content validation, no app artifacts, no new lock manager).

## Webhook ingress (the surviving inbound seam)

`swarmkit serve` is the front door. A `Trigger` (`type: webhook`) validates the signature, extracts an opaque `correlation_id` from the body by JSONPath, and delivers an `EventSignal` to whatever your application registered. What the event *means* — which topology it starts, whether it advances anything — is your application's decision, because that judgement is what left with the sequencer.

A trigger whose `credentials_ref` names an absent environment variable **refuses to start**: accepting unsigned requests because the secret is missing is a fail-open, and the old behaviour (warn, skip validation) was indistinguishable at runtime from a correctly configured trigger. `swarmkit serve` does not load a `.env` file.

`POST /run/{topology_name}` and `POST /hooks/{topology_name}` start runs directly; the reserved-scope discipline is unchanged — human-only authorities (`skills:activate`, `mcp_servers:deploy`, `topologies:modify`, `iam:modify`, `approvals:resolve`) are enforced by the policy engine and structurally un-grantable to an agent.

## Events (the outbound seam) and Connections

**The runtime tells an application what happened; the application decides who hears about it** (`docs/site/reference/events.md`, `design/details/extracting-the-channels.md`). `GET /events?after=<cursor>&types=…&run_id=…&limit=…` returns events in log order from a position — the durable log a consumer replays forward from where it stopped (`GET /audit` is the other direction, newest first, for a person). `events:` in workspace.yaml adds best-effort push sinks (`webhook` with `url` + `credentials_ref` bearer, or `stdout` for development), a latency optimisation over the pull, never a replacement. `gates.auto_resume` (default true) continues a run as soon as its gate is resolved, so an application does not have to call `POST /jobs/{id}/resume` and cannot forget to. `examples/event-consumer/` is the reference application: it hears a gate open, asks a human on its own channel, resolves the gate, and imports no runtime module. The Slack/Discord/Telegram providers that used to live in the runtime were removed in 1.216.0.

**Connections** (`docs/site/reference/connections.md`): `credentials` entries are references (`env`, `file`, `oauth`; the cloud sources are accepted by the schema and refused at resolution until a `SecretsProvider` exists), resolved by one `CredentialService` at every entry point. For a remote MCP server that speaks OAuth, the portal's Connections page runs the login — `GET /auth/mcp/probe`, `POST /api/oauth/login` (discovery, dynamic client registration, PKCE), `GET /auth/mcp/callback` — and stores the token encrypted (`SWARMKIT_OAUTH_KEY` or a generated `.swarmkit/oauth.key`), keyed by (credential, **owner**): a token belongs to the person who logged in. Refresh happens **before** a run whose window (`SWARMKIT_OAUTH_RUN_WINDOW_S`, 900 s) would outlive the access token, never mid-run; a refresh the provider refuses is `ConsentRequired` and needs a browser, not a retry. No endpoint returns a token.

## Human decisions at a gate (the approval API)

A gate is resolved by a **human identity**, and the decision is a record — not a boolean. This is the surface an external application integrates with (docs/site/reference/orchestrator-integration.md).

`POST /review/{item_id}/resolve` — `{"outcome": "approve" | "changes-requested" | "reject", "comment": "…"}`. The body carries **no identity**: the resolver is the authenticated caller (`request.state.identity.client_id`), because a body-supplied identity makes every membership and segregation-of-duties check self-asserted. The caller must hold `approvals:resolve`, a **reserved human-identity scope** a transport (API-key/JWT) token structurally cannot carry — so an agent or webhook integration can never satisfy an approval gate. Membership is checked against the workspace role registry before anything is recorded, and a non-member gets a 403 naming the reason.

`reject` ends the run. **`changes-requested` does not** — it re-runs the stage with the reviewer's comment in its input, via a `rework` event distinct from `gate`. `POST /review/{id}/approve|reject` and `/answer` also accept `{"comment": …}`; a §6.3 input answer plus its comment becomes the parked harness's resume statement, so a conditional approval ("yes, staging only") reaches the agent instead of flattening to `true`.

Review items serialize with `kind` = `permission` | `input` | `role_task` | `other`. A role-task carries `gate_id` (`<run_id>:<agent_id>`, where `run_id` is the job id — split on the LAST colon), `role`, `scope`, `rule_index`, `resolved_by`, `comment`, `artifact_ref` and `round`. Narrow the queue with `GET /review?kind=role_task&gate_id=…`.

**Rounds and staleness.** A rework loop re-opens the gate against a new artifact and advances the round; the ref is keyed on a **content digest**, so an identical re-run does not re-ask reviewers. **Only decisions about the current artifact count toward quorum** — earlier rounds are retained, returned by the read APIs and rendered as `STALE`, but an approval of v1 is not an approval of v3. An empty `artifact_ref` (externally-driven gate, or an item predating this) is unfiltered.

What the agent receives is a fenced, attributed, typed and versioned block — human text is untrusted model input, framed as review feedback rather than instructions:

```
<human-decisions gate="run-42:design" artifact="run-42/design/output#d564402e" round="1">
  [changes-requested] security-reviewer (alice), scope=security:approve
      round 0, on run-42/design/output#c75bc614   (STALE — written about an earlier revision)
    The retry loop has no backoff. Add exponential backoff.
</human-decisions>
```

`GET /whoami` returns the authenticated caller (`/auth-info` is public and describes the server, not the caller). Every attempt — allowed or denied — is audited as `approval.role_task_resolved`, with `approval.gate_opened` per round and `approval.changes_requested` on a rework.

## Case study: feature-flag cleanup (the DoorDash shape, as data)

[`examples/flag-cleanup/`](https://github.com/delivstat/swarmkit/tree/main/examples/flag-cleanup) is the SwarmKit form of DoorDash's agentic stale-flag cleanup ([case study](https://delivstat.github.io/swarmkit/case-studies/feature-flag-cleanup/)): Phase 1 a model agent triages a flag into a cleanup report a human confirms (`intake-review` funnel); Phase 2 a **harness** agent removes the flag in an isolated worktree under a budget and its diff faces the `cleanup-review` funnel — the `code-review` decision skill judges it (a finding routes the critique back to the harness) and a human on `flags:approve` signs off, the only exit. No orchestration code — the worktree isolation, the budget, the gate and the audit are the runtime's; the daily cross-repo fan-out is the calling application's. `just demo-flag-cleanup` runs it deterministically (real `claude-code` adapter over a scripted transcript through the real gate; no keys, no network).

## The SDLC example (the worked reference)

[`examples/sdlc-pipeline`](https://github.com/delivstat/swarmkit/tree/main/examples/sdlc-pipeline) is the largest worked workspace: a software-delivery lifecycle — **intake → design → build → sit → pt → security-review → deploy → support-handover** — as topologies, archetypes, funnels, contracts and rigs, carrying three multi-party human gates, a harness build/review node, and IAM-scoped agents. Its *sequencing* half (the stage graph, the saga controller, the Temporal adapter) was removed with the bundled pipeline in 1.189.0; the artifacts, the gates and the per-stage demos remain, and `examples/pipeline-orchestrator/` shows how an application drives them.

- [Video walkthrough](https://delivstat.github.io/swarmkit/sdlc-example/): every artifact toured on-screen. Recorded before the extraction — the artifact tour is current, the stage-graph/controller sections are historical.
- Run it: `just demo-sdlc-stage-run` (one gated stage) · `just demo-consolidated-design` · `just demo-harness-build` · `just demo-sit-pt` · `python examples/sdlc-pipeline/validate_library.py` (validate every artifact).
- Check it: `swarmkit validate --require --require-verified` in `examples/sdlc-pipeline/workspace` reports both the inert funnel layers and the roots nothing checks.

## Reference workspace

Production-quality topologies, archetypes, and skills under `reference/`:

- [Code Review Swarm topology](https://github.com/delivstat/swarmkit/blob/main/reference/topologies/code-review.yaml): 3 leaders (Engineering, QA, Ops), 10 agents.
- [Skill Authoring Swarm topology](https://github.com/delivstat/swarmkit/blob/main/reference/topologies/skill-authoring.yaml): 6 specialist agents.
- [Knowledge Curator topology](https://github.com/delivstat/swarmkit/blob/main/reference/topologies/knowledge-curator.yaml): curates governed memory — resolves quarantined contradictions through a human gate.
- [27 reference skills](https://github.com/delivstat/swarmkit/tree/main/reference/skills): GitHub MCP, decision, knowledge, coordination, persistence (incl. `governed-memory` + `memory-reconcile`).
- [16 archetypes](https://github.com/delivstat/swarmkit/tree/main/reference/archetypes): leaders, code review workers, authoring agents.

## Structured delegation (v1.2.0+)

Planner-driven task execution replaces simple sequential delegation. Coordinators call `create-task-plan` to generate a dependency-ordered task plan; the compiler executes tasks in parallel when independent, sequentially when dependent. Key tools:

- `create-task-plan`: coordinator produces a structured plan with tasks, dependencies, and assignments
- `update-task-plan`: modify an in-flight plan (add tasks, change assignments)
- `read-task-result`: retrieve completed task results (summary-first: 3-5 bullet key_findings, full results on disk)

Self-tasks let the coordinator do its own work (synthesis, diagrams). Plans are crash-resilient via `tasks.json` on disk — the CLI detects previous plans on fresh runs. Auto-fix adds missing dependencies and synthesis tasks.

## Sterling workspace (reference implementation)

Production-grade workspace under `examples/sterling-oms/` demonstrating enterprise-scale agent orchestration:

- 8 topologies, 12 archetypes, 75 skills
- Sub-agent architecture: root coordinator delegates to architect, which delegates to 6 focused workers (jira, config, docs, developer, log-analyst, document-writer)
- Atlassian wrapper MCP: structured JQL/CQL queries so models never write raw query syntax
- Log analyser MCP: SQLite-indexed log analysis handling 500MB+ log files, 9 tools including `get-timer-detail` drill-down
- Document writer with pandoc MCP for DOCX/PDF generation
- Per-agent model selection through OpenRouter: Kimi K2.5/K2.6 for the tool-heavy agents, Qwen3-235B and DeepSeek V4 Flash for workers, DeepSeek Chat V3 for writing

## Design notes

Per-feature design notes under `design/details/`:

- [MCP client](https://github.com/delivstat/swarmkit/blob/main/design/details/mcp-client.md): stdio + HTTP transports, workspace config, governance gating, inputSchema forwarding.
- [Knowledge MCP Server](https://github.com/delivstat/swarmkit/blob/main/design/details/knowledge-mcp-server.md): 11 tools for live docs search.
- [User Knowledge Server](https://github.com/delivstat/swarmkit/blob/main/design/details/user-knowledge-server.md): bootstrap-time knowledge wiring for user codebases.
- [Code Review Swarm](https://github.com/delivstat/swarmkit/blob/main/design/details/topology-code-review.md): agent tree, skill map, HITL gates.
- [Skill Authoring Swarm](https://github.com/delivstat/swarmkit/blob/main/design/details/topology-skill-authoring.md): multi-agent authoring + edit mode.
- [Governance provider](https://github.com/delivstat/swarmkit/blob/main/design/details/governance-provider-interface.md): AGT integration, policy evaluation, audit.
- [Model provider](https://github.com/delivstat/swarmkit/blob/main/design/details/model-provider-abstraction.md): the `ModelProvider` seam, per-agent model selection.
- [Declarative model providers](https://github.com/delivstat/swarmkit/blob/main/design/details/declarative-model-providers.md): a provider is YAML over a wire-format family; 12 bundled (`anthropic`, `openai`, `google`, `ollama`, `openrouter`, `groq`, `together`, `rkllama`, `llama-server`, `openvino-model-server`, `mlx-lm`, `lemonade`); `<workspace>/providers/*.yaml` adds or overrides; registration by readiness (key set, or no auth); `swarmkit providers list|show`.
- [Conversational authoring](https://github.com/delivstat/swarmkit/blob/main/design/details/conversational-authoring.md): swarmkit init/author/edit.
- [Structured output](https://github.com/delivstat/swarmkit/blob/main/design/details/structured-output-governance.md): schema-constrained generation + auto-correction.
- [LangGraph compiler](https://github.com/delivstat/swarmkit/blob/main/design/details/langgraph-compiler.md): topology → StateGraph translation, approval gate checkpointing.
- [DAG dependency graph](https://github.com/delivstat/swarmkit/blob/main/design/details/dag-dependency-graph.md): `depends_on` for parallel-with-dependencies execution.
- [Decision skills](https://github.com/delivstat/swarmkit/blob/main/design/details/decision-skills.md): LLM judge verdicts, confidence scores, multi-persona panels.
- [A2A federation](https://github.com/delivstat/swarmkit/blob/main/design/details/a2a-federation.md): SwarmKit-to-SwarmKit A2A returns the callee's run id, token/cost usage and an observability pointer; the card identifies it and the caller stitches it into its audit.
- [Audit event journal](https://github.com/delivstat/swarmkit/blob/main/design/details/audit-event-journal.md): write-through audit — each event durable when recorded, so a crashed run keeps its trail; the store dedups the end-of-run backstop.
- [Human interaction model](https://github.com/delivstat/swarmkit/blob/main/design/details/human-interaction-model.md): audit event schema, CLI primitives, the review queue (its notification layer left the runtime in 1.216.0 — see Events).
- [Skill registry](https://github.com/delivstat/swarmkit/blob/main/design/details/skill-registry.md): the `swarmkit skill` command over the catalogue — search, add (both fragments shown first), import SKILL.md, check, remove; shipped 1.235.0.
- [Memory by default](https://github.com/delivstat/swarmkit/blob/main/design/details/memory-by-default.md): the defaults a silent workspace gets, the `memory:` block, and the `memory.disabled-but-bound` error; shipped 1.233.0.
- [Fleet control plane](https://github.com/delivstat/swarmkit/blob/main/design/details/fleet-control-plane.md) and the [control-plane note series](https://github.com/delivstat/swarmkit/tree/main/design/details/control-plane): enrolment (Mode A/B), membership scopes `monitor|manage|approve-as`, signed deploys, delta sync, operator identity forwarding (28).
- [Knowledge Curator topology](https://github.com/delivstat/swarmkit/blob/main/design/details/knowledge-curator-topology.md): persistent wiki maintained by LLM agents.

## Harness executors (M19 — shipped, container sandbox included, opt-in)

Run a coding harness (Claude Code, opencode, or any subprocess emitting line-delimited JSON) as an agent node, alongside the `model` executor, under the same governance + observability. An archetype selects a harness with an `executor` block — canonical shape `executor: { kind: harness, ref: claude-code }` (`ref` is the adapter id; swap `claude-code`→`opencode`/`codex`/`gemini-cli` to change harness). A registered adapter id may also be named directly as the kind (`kind: claude-code`, legacy). Absent an `executor` block, a node is `kind: model` (unchanged). Harnesses are **data**: a declarative `adapter.yaml` interpreted by one engine — no per-harness Python — with a bundled library (claude-code + opencode verified e2e; codex + gemini-cli experimental). A harness runs in an ephemeral git worktree by default (produces a diff, never integrates). Mid-run out-of-grant permissions **relay** to a human inbox and resume (`swarmkit review`); repeated operator approvals **accrue** into a proposed allowlist changeset (`swarmkit trust list|apply|clear`, default N=5, one denial resets+blocks). An **opt-in container sandbox** adds real isolation — resource limits, enforced egress (`deny`/`allowlist`), extra `mounts`, and a `build` step that runs the harness with **no local install** (bring only your API key). Off by default; `SWARMKIT_DISABLE_CONTAINER_SANDBOX` always wins; a container with no runtime fails loud, never a silent unsandboxed run.

- [Executor abstraction design](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-abstraction.md): the provider seam (`model` | harness kinds), adapter tiers, relay, trust accrual, RFC decisions.
- [Container sandbox design](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-container-sandbox.md): opt-in container tier, disable switch, build-no-local-install, mounts, egress proxy.
- [Relay design](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-relay-plan.md) · [Input escalation](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-input-escalation-plan.md) · [Trust accrual](https://github.com/delivstat/swarmkit/blob/main/design/details/executor-trust-accrual-plan.md).
- [Authoring a harness adapter](https://github.com/delivstat/swarmkit/blob/main/docs/guides/authoring-harness-adapters.md): anatomy, event-map DSL, auth modes, launch review gate, sandbox block, DSL ceiling.
- Adapter fixtures: [executor-adapter](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/executor-adapter) (minimal, claude-code, relay, sandbox-container, sandbox-build).

## HTTP server, auth, and canary deployments (M10 — shipped)

- [Storage reference](https://github.com/delivstat/swarmkit/blob/main/docs/site/reference/storage.md): the six stores, `_URL` vs `_BACKEND`, and the step-by-step SQLite → Postgres migration runbook.
- [Storage service design](https://github.com/delivstat/swarmkit/blob/main/design/details/storage-service.md): one resolver for every store; why a misconfigured workspace used to look like an empty one.
- [Serve and auth design](https://github.com/delivstat/swarmkit/blob/main/design/details/serve-and-auth.md): FastAPI server, AuthProvider ABC, MCP endpoint, triggers, workspace.yaml `server:` block.
- [Canary deployments design](https://github.com/delivstat/swarmkit/blob/main/design/details/canary-deployments.md): weighted topology version routing, auto-promotion by error rate + drift, manual promote/rollback.
- [Serve CLI reference](https://github.com/delivstat/swarmkit/blob/main/docs/reference/serve-cli-tests.md): all endpoints with real curl output — health, topologies, jobs, SSE, webhooks, conversations, auth, MCP.
- [Canary deployments guide](https://github.com/delivstat/swarmkit/blob/main/docs/reference/canary-deployments.md): quick start, configuration reference, monitoring, auto-promotion, manual controls, common scenarios, real test outputs.

`swarmkit serve` endpoints — the essentials: `POST /run/{topology}` (async; `attachments`, `correlation_id`, `labels`), `GET /jobs/{id}` · `/stream` (SSE) · `/diff` · `POST /jobs/{id}/resume` · `/stop`, `GET /jobs/history`, `POST /hooks/{topology}` (signed webhook), `GET /events?after=<cursor>` (durable log) and `POST /events/signal`, `GET /review` · `POST /review/{id}/approve|reject|answer|resolve`, `GET /gates/{gate_id}`, `GET /artifacts` · `/artifacts/{ref}`, `GET|POST /memory` · `/memory/quarantine` · `GET /memory/config`, `GET /gaps`, `POST /conversations` · `/conversations/{id}/messages` (SSE), `GET /audit`, `GET /usage`, `GET /system` · `/storage` · `/capabilities` · `/whoami` · `/auth-info`, `GET /workspace/reachability` · `/verification`, `POST /mcp` (Streamable HTTP), `GET /.well-known/agent-card.json` + `POST /a2a` (A2A, opt-in), canary and fleet routes, and the portal's `/api/*` (artifact CRUD, workspace config, OAuth login, the skill catalogue at `/api/skill-catalogue` + `/api/skills/{add,import,check}`, `POST /reload`). **Every endpoint is generated from the server's OpenAPI document into [the HTTP API reference](https://delivstat.github.io/swarmkit/reference/http-api/)** — 100 operations; that page is the complete list.

**A2A server (1.221.0+, `server.a2a.enabled: true`; off by default):** `swarmkit serve` publishes an Agent Card at `/.well-known/agent-card.json` — one A2A skill per topology, `securitySchemes` derived from the auth provider actually running, `identity` (`name`/`description`/`url`/`organization`) from the workspace — and answers JSON-RPC 2.0 at `POST /a2a` (or `/a2a/{topology}`): `message/send`, `message/stream` (SSE; needs `Accept: text/event-stream`), `tasks/get`, `tasks/list`, `tasks/cancel`, `tasks/subscribe`. It is a **transport onto jobs**, not a second execution path: the task id IS the job id, `contextId` rides on `correlation_id`, file parts become attachments, the run carries `source: a2a`, and every gate and audit event applies unchanged. Job status → task state: `pending`→`submitted`, `running`→`working`, `deferred`→`input-required`, `completed`, `failed`, `stopped`→`canceled`. A run parked on a human gate reports `input-required` with the gate URL in the message, and a follow-up `message/send` on that task is **refused** (`UnsupportedOperationError`, -32004) — approval scopes are un-grantable to agents and an A2A client is an agent; a person resolves it through the review queue and a subscriber sees the task go `working`. Push notifications (-32003), gRPC and card signing are not implemented. `/capabilities` reports `features.a2a`. The client side is the `agent` skill type (1.222.0, its own section above): `design/details/a2a-interop.md`.

Storage (1.130.0+): **one service resolves every store** — nothing else opens a database (`design/details/storage-service.md`). Six kinds: `runtime` (jobs, conversations, usage), `audit`, `artifacts`, `memory`, `fleet`, `checkpoints` (the `saga` store went with the bundled pipeline). All of them follow `storage.runtime` unless they declare their own block, EXCEPT `checkpoints`, which follows only `storage.checkpoints` because the Postgres LangGraph checkpointer is a separate install (`pip install "swarmkit-runtime[postgres]"`). Asking for postgres checkpoints WITHOUT that extra degrades to the local SQLite checkpointer with a warning (1.131.1+) rather than refusing to start — the one place degrading is right, because checkpoints are disposable run state and a missing optional dep should not take down serve. (1.130.0 and 1.131.0 refused, which broke upgrades for workspaces that had been carrying that silently-ignored config.) Everything that holds RECORDS still fails closed. A per-store block inherits `storage.runtime.url` when it declares none. `${VAR}` and `${VAR:-default}` are expanded in these URLs.

`SWARMKIT_STORE_URL` vs `SWARMKIT_STORE_BACKEND`: **the URL alone is sufficient** — a URL names its own backend, so `SWARMKIT_STORE_URL=postgresql://…` selects Postgres with no other variable set. `DATABASE_URL` is the fallback when `SWARMKIT_STORE_URL` is unset. `SWARMKIT_STORE_BACKEND` (`sqlite`|`postgres`) is a rarely-needed override, mainly to force SQLite while a URL is present. Env is a global signal: it moves every store that follows `storage.runtime`, and never `checkpoints`. (Before 1.130.0, setting only `SWARMKIT_STORE_URL` was silently ignored, and `url: ${SWARMKIT_STORE_URL}` in workspace.yaml was never expanded — both wrote to SQLite while reporting success.)

A backend naming a real database with **no resolvable URL raises at startup** rather than degrading to SQLite — a run must not write to a different database than the one configured. If your application keeps its own sequencing state, keep it in *your* database: SwarmKit's store holds runs, audit, artifacts, memory, fleet and checkpoints, not your workflow.

Inspecting + migrating: `swarmkit storage status <ws>` prints one line per store — backend, location, and **which setting won**. `swarmkit storage migrate <ws> [--dry-run] [--yes]` copies local SQLite rows into the configured Postgres: additive, idempotent (`ON CONFLICT DO NOTHING`), and it never deletes the SQLite files. `swarmkit system <ws>` adds versions, `workspace.env.yaml` properties, and the environment. Same data at `GET /storage` and `GET /system`, and on the web UI's **System** page. Passwords are masked in every surface.

SQLite → Postgres runbook (`docs/site/reference/storage.md`): 1. create the database (no schema step — tables are created on connect); 2. set `storage.runtime.backend: postgres` + `url: ${SWARMKIT_STORE_URL}` and export the URL; 3. `swarmkit storage status` and confirm every store says postgres BEFORE moving data; 4. stop the runtime and `swarmkit storage migrate`; 5. verify counts, then archive `.swarmkit/*.sqlite` — leaving them is how a split brain starts; 6. restart serve with the same environment. Skipping step 4 abandons the audit trail, run history and governed memory.

Every audit event carries its `run_id` (1.130.0+; it was NULL on every row before, so `query(run_id=…)` always returned empty and reported it as "no events").

`workspace.env.yaml` should exist in every workspace — `swarmkit init` scaffolds it. A reserved top-level `secrets:` key lists dotted property paths whose values are never displayed (they render as `set` in `swarmkit system`, the System page, and CI logs); a name-based heuristic (`key`/`token`/`secret`/`password`/`credential`) is the fallback for undeclared ones. Declaring adds to the masked set and can never remove from it.

A webhook trigger whose `credentials_ref` names an absent environment variable **refuses the request (503)** instead of accepting it unsigned.

Auth providers: `NoneAuthProvider` (default, open access), `APIKeyAuthProvider` (Bearer token, env:VAR resolution, scopes), `JWTAuthProvider` (RS256/ES256, JWKS auto-discovery). Auth is perimeter; governance is internal policy.

Canary config in workspace.yaml: `server.canary.routes[].versions[].weight` (traffic percentage), `promote_when.min_runs`, `promote_when.error_rate_below`, `promote_when.drift_below`, `promote_when.window_minutes`. Version-qualified names (`topology@1.1.0`) bypass routing for direct testing.

Output validation: two layers, not interchangeable. `output_schema` on an agent checks SHAPE (free, deterministic, kills shape-level hallucination); a **decision skill** bound at `post_output` checks SEMANTICS (grounding, scope, contradiction) and costs an LLM call. Bind under `governance.decision_skills` in the topology (or workspace, where a topology must explicitly opt out with `required: false` — auditable). `scope` defaults to `*`, which fires after EVERY agent: name the root agent for "the topology's answer". Triggers: `pre_input` (reject before any LLM work), `post_output` (the answer), `checkpoint` (between task batches), `pre_synthesis` (task results BEFORE the coordinator launders them into a fluent summary; auto-loads scope.json).

A decision skill is NOT necessarily an LLM call: `implementation.type` may be `mcp_tool` (deterministic — a validator, linter, test run, schema check), `command` (a local binary), `llm_prompt`, or `composed` (`strategy: parallel-consensus` to require agreement). The binding is identical either way. Prefer `mcp_tool` wherever the question has a computable answer; `reference/skills/` ships `lint-check`, `run-tests`, `security-scan`, `validate-workspace`, `gate-validator`. A decision skill's `verdict` must be `pass` | `fail` | `needs-revision`. FORM is normalised from 1.131.0 (`FAIL`, `Fail`, ` fail `, `needs_revision` all read correctly — before that they were unrecognised and therefore silently PASSED); VOCABULARY is not (`rejected`, `invalid`, `false` stay unrecognised — guessing a synonym would invent a verdict the skill never gave). An absent or unrecognised verdict is read as **pass**, so a mis-mapped validator reports success on every rejection; both cases now log a warning naming the skill ("the check is not running") instead of failing silently. **`fail` does not block**: the runtime builds feedback from the failed results, asks the agent to revise (it still holds its context), and after `max_retries` returns the output ANNOTATED with `GOVERNANCE FLAGS` rather than dropping it. Write `reasoning`/`violations` as instructions to the agent that will act on them — that text is the retry prompt. A hard stop is an approval gate, not a decision skill. `config.max_retries` on the binding (default 4) is honoured from 1.131.0 — before that the whole `config:` block was accepted by the schema and read by nothing. **`enabled` and `required` are different questions and were one flag until 1.169.0**: `enabled: false` means the binding does not run (this is how a topology switches off a workspace binding); `required: false` means it DOES run and its `fail` is advisory — logged, not fatal. Before 1.169.0 a falsey `required` discarded the binding entirely, so an advisory binding was accepted, validated, displayed and never evaluated — which is why `memory-reader`, bound advisory by the docs, never ran. A topology override still spelling the old `required: false` now makes the binding advisory and warns, rather than silently changing from off to on. Guides: `docs/site/guides/validating-topology-output.md`, `docs/site/guides/memory-and-decision-skills.md`.

## Workspace memory (shipped)

Agents remember across conversations. Two decision skill bindings (`memory-reader` at `pre_input`, `memory-writer` at `post_output`) enable automatic insight extraction and context injection. Both bindings are on by default from 1.233.0 (see "Memory" below). Two backends: `MemoryStore` (the configured storage service — SQLite or Postgres, `workspace_memory` table, TF-IDF search; was `.swarmkit/memory.json` before 1.227.0) and `GBrainMemory` (GBrain MCP server, hybrid vector + keyword search, graph relationships, Supabase/Postgres).

- [Workspace memory reference](https://github.com/delivstat/swarmkit/blob/main/docs/reference/workspace-memory.md): setup, config, GBrain integration, examples, real test outputs.
- [Memory demo script](https://github.com/delivstat/swarmkit/blob/main/docs/examples/memory-demo.py): runnable demo covering CRUD, search, context injection, extraction, persistence, deletion.
- [Workspace memory design](https://github.com/delivstat/swarmkit/blob/main/design/details/workspace-memory.md): two-layer knowledge graph, decision skill hooks, GBrain integration, privacy.

## Governed memory (shipped)

Structured memory that **evolves in place over time** rather than piling up. An agent carrying the `governed-memory` persistence skill proposes facts as `{subject, attribute, value}` candidates; the runtime reconciles each against current memory — **new** / **reinforce** (identical restatement — no duplicate) / **update** (supersede the value in place) / **refine** (merge, via the `memory-reconcile` decision skill) / **contradict** (conflicts with a trusted memory → **quarantined** for a human curator, never silently overwritten). One canonical row per `(subject, attribute)` key plus an append-only change-log, so any fact is readable `as_of` a past time and update-in-place coexists with the append-only audit invariant (§8.3). Confidence decays with recency (stale facts rank down, never deleted); retrieval is relevance-ranked (local TF-IDF, or cosine similarity when an embedder is wired — no vendor lock-in). `swarmkit memory search|get|quarantine|resolve` and the serve `/memory` endpoints share one service seam; the `knowledge-curator` reference topology curates it.

- [Governed memory reference](https://delivstat.github.io/swarmkit/reference/governed-memory/): the persistence skill, reconcile ops, quarantine + curator gate, confidence decay, relevance retrieval, CLI + serve.
- [Governed memory design](https://github.com/delivstat/swarmkit/blob/main/design/details/governed-memory.md): the update-in-place model, current-state + append-only change-log, the governed write path, IAM.

Memory-writer extracts structured insights (topic, context, key_points, tags) via LLM after each turn. Memory-reader searches for relevant prior conversations and prepends context. The agent sees prior sessions naturally: "As we discussed previously..." Config: `governance.decision_skills[].config.search_scope` (user/shared/both), `max_results`, `min_output_length`.

## Observability (design notes; OTel, intent drift and the ring buffer shipped)

- [OpenTelemetry observability](https://github.com/delivstat/swarmkit/blob/main/design/details/opentelemetry-observability.md): OTel as the telemetry standard. Trace-per-run, span-per-agent-step, `swarmkit.*` semantic attributes, console + OTLP exporters to any collector.
- [Intent drift detection](https://github.com/delivstat/swarmkit/blob/main/design/details/intent-drift-detection.md): optional per-agent intent monitoring via embedding similarity. Log/warn/nudge strategies.
- [Telemetry reference](https://delivstat.github.io/swarmkit/reference/telemetry/): the `telemetry:` block, exporters, sampling, redaction; `swarmkit debug` is the local prompt ring buffer.
- Operations across many instances is the self-hosted fleet control plane ([Level 22](https://delivstat.github.io/swarmkit/tutorials/22-fleet/), `design/details/fleet-control-plane.md`).

## Images (two channels, for two callers)

**A caller holding the file attaches it to the run**: `swarmkit run … --attach <path>`, or `attachments: [{"path": …}]` / `[{"data": <base64>}]` on `POST /run/{topology}`. It reaches the **entry agent's first message and no downstream node** — one model call, no tool round-trip. The media type is read from the bytes, so there is one `--attach` and no `--image`/`--pdf`; sending a `type` field is a 422, and `url`/`stream` sources are refused (the runtime does not fetch caller-supplied addresses, and an attachment is re-read on every turn of a tool loop). A bad path or an uncarryable type fails the CALL, not the run; images only today (`image/png|jpeg|gif|webp`), 20 MB ceiling (`SWARMKIT_ATTACHMENT_MAX_BYTES`). Every attachment is audited as a `run.attachments` event carrying name, media type, size, SHA-256 and source path — never the bytes.

**An agent deciding mid-run what to look at still needs a skill.** A path in the prompt is text; base64 in the prompt is tokens. **That route is an MCP tool returning `ImageContent`**, which `langgraph_compiler/_skill_executor.py` converts to a `ContentBlock(type="image", …)` and the provider maps to its native image part (`image_url` data-URL for the `openai-compatible` family, incl. OpenRouter; `source.base64` for Anthropic). `docs-reader`'s `view_image` is the bundled tool that does this. Harness executors read image files from disk directly instead.

The failure mode is silent: a path that does not resolve makes the tool report "not found", and the model then describes the image from surrounding prose anyway — fluently and wrongly. Pass **absolute** paths, rewrite relative refs inside any document the agent reads, and instruct the agent to report a failed path rather than describe the image. Since 1.129.2 the paths must also sit under the docs-reader `--workspace` root, which now confines rather than merely resolves.

- [Getting an image to a model](https://delivstat.github.io/swarmkit/guides/getting-an-image-to-a-model/): the channel, the trap, and a measured before/after.

## Memory

There are TWO memory subsystems and they are not interchangeable. **Governed memory** (`governed_memory/`, shape `{subject, attribute, value, type, confidence}`) is curated: reconcile-on-write, contradictions quarantined rather than applied, resolution a human action, confidence decaying by recency. It is what `swarmkit memory search|get|quarantine|resolve`, the `/memory` page and the `governed-memory` persistence skill address. **Workspace memory** (`memory/_store.py`, shape `{topic, context, key_points, tags}`) is what a run recorded by itself, unreviewed, on the configured store (`workspace_memory` table) or GBrain. From 1.168.0 the `memory-reader` binding at `pre_input` reads BOTH and injects curated facts first, in a delimited `<curated-memory>` block.

**Since 1.233.0 memory is on by default** (`design/details/memory-by-default.md`). A workspace that says nothing gets: `memory-reader` bound at `pre_input` before every agent, `memory-writer` at `post_output` (advisory, `required: false`), and the bundled `governed-memory` + `memory-reconcile` skills loaded when the workspace defines no skill with those ids. The `memory:` block tunes it — `enabled: false` switches everything automatic off (and `memory.disabled-but-bound` refuses a workspace that disables memory yet binds a memory skill explicitly), `reader`/`writer` sub-blocks set `search_scope` (user/shared/both), `max_results`, `min_output_length`; an explicit binding in `governance.decision_skills` is used as written. Two things stay per-agent, deliberately: **writing** curated memory is a grant of the `governed-memory` skill (it carries `kb:write`; granting it to obtain reads is how a curated store stops being curated), and nothing is injected into an agent whose binding is switched off. Confirm with `GET /memory/config` (the effective block), the portal's Memory page, or the `Memory context injected for agent=…` log line — not with `swarmkit memory search`, which proves the fact exists, not that an agent can see it. Before 1.233.0 all three pieces had to be declared by hand and each failed silently alone; before 1.169.0 `required: false` discarded the binding entirely (see decision skills above). Guide: `docs/site/guides/memory-and-decision-skills.md`.

## Guides

- [Memory and decision-skill bindings](https://delivstat.github.io/swarmkit/guides/memory-and-decision-skills/): the two memories, turning governed memory on, and `enabled` vs `required`.
- [Building swarms — the complete playbook](https://delivstat.github.io/swarmkit/guides/building-swarms/): the ordered, step-by-step build recipe (one agent → governed multi-app delivery flow), with a runnable artifact at every step and the SDLC worked example.
- [Validating a topology's output](https://delivstat.github.io/swarmkit/guides/validating-topology-output/): `output_schema` (inline or a file path) for shape, decision skills for meaning, the retry loop and `GOVERNANCE FLAGS`.
- [Getting an image to a model](https://delivstat.github.io/swarmkit/guides/getting-an-image-to-a-model/): attachments vs the `view_image` tool, and the silent failure mode.
- [Sterling OMS workspace](https://github.com/delivstat/swarmkit/blob/main/docs/guides/sterling-oms-workspace.md): building domain-specific agent workspaces with knowledge bases.
- [Model selection](https://github.com/delivstat/swarmkit/blob/main/docs/guides/model-selection.md): pricing comparison, per-agent config, env vars.
- [Authoring a harness adapter](https://github.com/delivstat/swarmkit/blob/main/docs/guides/authoring-harness-adapters.md): declarative `adapter.yaml`, event-map DSL, auth, launch gate, opt-in container sandbox.
- [Serve CLI reference](https://github.com/delivstat/swarmkit/blob/main/docs/reference/serve-cli-tests.md): complete endpoint reference with real test outputs.
- [Canary deployments](https://github.com/delivstat/swarmkit/blob/main/docs/reference/canary-deployments.md): weighted version routing, auto-promotion, monitoring, rollback.
- [Workspace memory](https://github.com/delivstat/swarmkit/blob/main/docs/reference/workspace-memory.md): agents that remember across conversations. Setup, GBrain backend, real test outputs.

## Tutorials (22 levels, each runnable)

[Overview](https://delivstat.github.io/swarmkit/tutorials/). Levels 1–16 build one workspace up from a single agent to correlated, gated runs with contracts. Levels 17–22 each take one shipped capability and end in a `just demo-*` target that runs on the mock provider: [17 Harness executors](https://delivstat.github.io/swarmkit/tutorials/17-harness-executors/) (adapters, the governed gateway, relay + trust, sandbox), [18 Funnels & approval](https://delivstat.github.io/swarmkit/tutorials/18-funnels-approval/) (validate → judge → approve, role registry, `--require-verified`, `cited-change`, `stop`), [19 Command packs & attachments](https://delivstat.github.io/swarmkit/tutorials/19-command-packs-attachments/), [20 Agents calling agents](https://delivstat.github.io/swarmkit/tutorials/20-agents-calling-agents/) (`agent` skills, `pack:workspace`, A2A both ways, the portal's remote agents), [21 Providers, storage & operations](https://delivstat.github.io/swarmkit/tutorials/21-providers-storage-operations/) (declarative providers, storage status/migrate, `system`, `eval`, `knowledge-pack`), [22 Running a fleet](https://delivstat.github.io/swarmkit/tutorials/22-fleet/). **`just demo-capstone`** (`examples/capstone`) runs every HTTP-reachable feature in one workspace: attachment, command pack, agent skill, gate, A2A in and out, the record, a stop.

## Fixtures (valid artifact examples)

- [Topology fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/topology)
- [Skill fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/skill)
- [Archetype fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/archetype)
- [Workspace fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/workspace)
- [Trigger fixtures](https://github.com/delivstat/swarmkit/tree/main/packages/schema/tests/fixtures/trigger)

## Cross-cutting notes

- [Schema change discipline](https://github.com/delivstat/swarmkit/blob/main/docs/notes/schema-change-discipline.md)
- [Usability-first](https://github.com/delivstat/swarmkit/blob/main/docs/notes/usability-first.md)
- [LLM-friendly knowledge](https://github.com/delivstat/swarmkit/blob/main/docs/notes/llm-friendly-knowledge.md)
- [Observability discipline](https://github.com/delivstat/swarmkit/blob/main/docs/notes/observability.md)
- [Harness adapter discipline](https://github.com/delivstat/swarmkit/blob/main/docs/notes/harness-adapters.md): verify-against-real-binary, byte-identical bundled copies, sandbox-is-opt-in/never-silently-unsandboxed.

## Package docs

- [packages/runtime CLAUDE.md](https://github.com/delivstat/swarmkit/blob/main/packages/runtime/CLAUDE.md): Python runtime invariants.
- [packages/schema CLAUDE.md](https://github.com/delivstat/swarmkit/blob/main/packages/schema/CLAUDE.md): schema package invariants.