---
id: BB-PLAN-4H7KQ2ZP
type: plan
title: Toad Deployment API
spec: BB-SPEC-5TQ2M8HD
status: draft
---

# Toad Deployment API

## Outcome

`toad` runs on klops as a rootless Go service that owns release activation for enrolled Quadlet workloads.
A Luci job in an application repository can publish an image and deploy it by immutable digest, with probation, promotion, rollback, and durable status.

## Decisions

- Language: Go, matching Luci and one service toolchain.
- Identity: dedicated rootless `toad` user, own Podman socket, own systemd user scope, storage under `/data/toad`.
- Enrolled workloads run inside the `toad` user scope, so no root, sudo, polkit, or foreign session bus is needed.
- Durable state is file-backed with atomic rename, following Luci's storage approach; no database dependency.
- Toad runs containerized; it drives the machine through the Podman HTTP API and the systemd D-Bus interface over mounted sockets, so the image needs no `podman` or `systemctl` binary.
- The container uses `UserNS=keep-id`, so its process identity and bind-mounted files retain the dedicated host user's natural ownership; no `User=0` override.
- The API listens on a Unix socket for local administration and on loopback TCP for token-authenticated consumers.
- Credentials are runtime state owned by toad: `toad token issue|list|revoke`, hashed at rest, shown once, never in OpenTofu, state, or Git.
- Operator interface is the `toad` client in the same binary, used over existing klops SSH; no dedicated public SSH endpoint.
- The read-only web surface uses existing public Caddy Basic Auth, a shared Unix socket, a bundled stock-Caddy gateway, and the private Toad network; it introduces no host TCP listener.
- Deployments are operator-triggered through the CLI first; CI-driven deployment is a later decision.
- Application image building is out of scope here: Luci job containers have no Podman socket and no capabilities, so they cannot build images today.

## Slices

### 1. Promote path

Submit a digest for one enrolled test workload and reach a promoted revision.

- `services/toad` module, `toad serve`, loopback HTTP API, per-service token.
- Enrollment read from an operator-owned file; requests carry only service, digest, expected generation, idempotency key.
- Durable operation and service records with atomic writes.
- Rollout: pull and verify digest, stop and verify termination, activate unit, observe readiness, promote.
- Generation CAS, idempotency replay, single active rollout per service.
- Status and operation log endpoints.

### 2. Failure path

- Rollout deadline, continuous-ready stability window, restart budget across the whole attempt.
- Readiness attribution to expected unit invocation, container, and digest.
- Automatic rollback to the retained promoted revision, retained locally.
- Terminal, non-looping failed rollback.
- Post-promotion failures alert without rollback.

### 3. Credentials and transports

- Credential store with hashed tokens, `<id>.<secret>` format, and immediate revocation.
- Unix socket listener with permission-based local administration; loopback TCP for service tokens.
- Token administration refused over TCP.
- Enrollment drops all credential material.

### 4. Operator client

- `toad status|deploy|rollback|log` against the API with `TOAD_ENDPOINT` and a service token.
- `deploy` defaults to digest-and-generation idempotency, waits for a terminal phase, prints the operation log, and exits non-zero unless promoted.
- `rollback` aborts the active probation and waits for the restored revision.

### 5. Recovery

- Reconciliation of persisted intent against host state at start.
- Single-writer lock across processes and restarts.
- Deadlines that do not reset on restart; monitoring gaps that do not accumulate healthy time.
- Boot ordering that prevents an unreconciled candidate from running.
- Fault injection at each mutation boundary plus host reboot during probation.

### 6. Host layer on Podman and systemd APIs

- Replace binary execution with the Podman HTTP API for pull, image inspect, and container inspect.
- Replace `systemctl` with systemd D-Bus calls for reload, start, stop, and restart count.
- Keep the existing `rollout.Host` seam so rollout tests stay unchanged.

### 7. Host deployment

- OpenTofu: `toad` user setup mirroring `setup-ci-user.sh`, image build, Quadlet with the four mounts, enrollment rendering, host wrapper script.
- Fence existing automation: `rsync --delete`, group restarts, weekly GC, and `AutoUpdate` must not touch enrolled units or retained images.
- `just check`, `just tofu plan`, human-approved `just apply`.

### 8. CI handoff, deferred

Blocked on two prior decisions, and not required for the plan's outcome:

- where application images are built, since Luci cannot build them today;
- whether Luci gains one generic deploy adapter plus a service-scoped token, or deployment stays an operator action.

If taken, Luci speaks plain HTTP to the API, learns no application semantics, and surfaces only sanitized, operation-scoped output.

### 9. Enroll applications

- Enroll the first application, then a second one, to prove enrollment carries no per-application code.
- Each enrollment moves its workload into the `toad` scope with stable volumes and secret references.
- Record the backward-compatible migration rule every enrolled application accepts for the retained rollback window.
- Prove publish, deploy, probation, promotion, and rollback from each application's own repository.

### 10. Read-only web surface and ingress

- Responsive service cards and selectable detail for deployment state, live readiness, sanitized configuration, container state, systemd user-unit properties, recent operations, and highlighted operation logs.
- Nugu styling, bundled Nerd Font symbols, copy controls, and no mutation controls.
- Never expose raw Quadlets, environment values, host journal entries, generic host metrics, token material, or host paths.
- Bundled stock-Caddy gateway binds only the shared Unix socket, accepts only the configured hostname, routes Toad over the private network, and returns 404 otherwise.
- Existing public Caddy mounts only the shared socket directory read-only, applies a dedicated Basic Auth credential, and adds no relay port beyond existing HTTPS.

## Verification

Each slice ends with the complete `services/toad` Go test suite plus the behavior it introduces, exercised against a real rootless Podman where the behavior depends on it.
Slices 7 to 10 additionally require `just check`, a reviewed `just tofu plan`, and live deployment evidence before any claim of deployment.