just check modules/klops/quadlets

just generate zot-user admin

just generate basic-auth-user jan
just generate basic-auth-user oli

just deploy luci-image # creates ci user if needed and builds the image on klops
just plan
just apply

just deploy luci-image
just plan
just apply
# add .ci/*.kdl to any public repo, push, then check https://ci.bugabinga.net

# bugabinga.net Infrastructure Private Cloud

This repo hosts Infrastructure as Code (IaC) for the internal, private cloud
services of bugabinga.net. This infrastructure is hosted on hetzner.com and
provisioned via opentofu.

## Dependencies

Besides the obvious dependency on git, these Tools are needed to work with this
repo:

- [opentofu][opentofu]: IaC language
- [just][just]: command runner
- [lefthook][lefthook]: git commit hooks
- [gitleaks][gitleaks]: staged secret scanning
- `podman`: build and transfer the custom Luci image to klops
- `go` 1.26+: build and test Go services under `services/luci` and `services/toad`
- `rust` 1.98+ with rustfmt and Clippy: check `services/luigit`
- a C compiler with CGO enabled: required only for `go test -race`
- `caddy` and `jq`: generate and maintain bcrypt authentication entries
- `shellcheck` and `shfmt`: lint and format Bash automation
- `wrangler`: bootstrap R2 and run backend lock/recovery checks
- your favorite code editor

## Setup

Run `just setup` to verify required local tools.
Run `just setup hooks` to verify tools and install repository checks.
`just check [PATH...]` formats and validates all or selected configuration.
`just check read [PATH...]` performs the same selection without mutation.
Go paths select their owning modules, so `just check read services/luci/cmd/luci/main.go` checks Luci only and `just check read .` checks both Go modules.
Gitignored files plus `.git`, `.terraform`, `target`, `generated`, and `vendor` trees do not select or format Go code.
Nested Go modules own their source formatting, so parent-module checks never format child-module files.
Changing Go check automation selects its affected Go modules even without staged Go source.
Shell checks cover tracked plus unignored untracked `.sh` files, excluding `.apk`, `.git`, `.terraform`, `target`, `generated`, and `vendor`; explicit excluded shell paths are accepted without checking unrelated files.
The shell CI job bootstraps Bash, Git, Shfmt, and ShellCheck from Alpine 3.22 repositories into its writable workspace, prints their installed versions, then runs only `scripts/local/shell-check.sh read`.
The pre-commit hook passes added, copied, modified, and renamed staged paths as quoted arguments, preserving unstaged work without automatic staging.

Go commands set `GOTOOLCHAIN=local`, so the declared Go 1.26 toolchain is required without automatic runtime upgrades.
Go tool commands use versions pinned in each owning module through `go tool`; no global installation is needed.
`just verify go [PATH...]` runs race tests, Staticcheck, and Govulncheck for selected Go modules.
Use `just verify go-race`, `just verify go-staticcheck`, or `just verify go-vuln` to run one gate.
`just rust fmt`, `just rust clippy`, and `just rust test` run Luigit's locked Rust gates.
Govulncheck queries the Go vulnerability database, so it requires network access and may fail when that service is unavailable.
Race tests require CGO plus a working C compiler.
`just verify luci-bench COUNT OUTPUT` records repeated Luci web benchmark samples to an explicit untracked repository-local output file and fails if no benchmark result is produced.
`just verify luci-benchstat BASELINE CANDIDATE` rejects empty or non-Luci-benchmark inputs before comparing explicit repository-local benchmark files with Luci's pinned Benchstat.
Neither benchmark action defines a performance threshold.
`just verify luci-mutation [OUTPUT]` is opt-in, uses Luci's pinned Gremlins tool, and writes its report to an untracked path below this repository by default.
Staticcheck and Govulncheck tool closures raise indirect `golang.org/x/sys` to v0.48.0 in both modules; this changes Toad's resolved runtime dependency from v0.27.0, with no direct application dependency upgrade.
Luci retains `github.com/klauspost/compress` v1.18.6 pending a parent decision; Govulncheck reports non-reachable module advisory `GO-2026-5841`, fixed in v1.18.7, rather than zero module vulnerabilities.

## OpenTofu workflow

Run OpenTofu only through repository `just` recipes.
Credentialed recipes use Just's built-in dotenv loader for `.env`; unrelated
recipes do not inherit the R2 credentials.
`just verify infra` performs read-only infrastructure checks from an isolated source copy with an empty credential environment, private OpenTofu data and home directories, and `init -backend=false -input=false -lockfile=readonly`.
It runs recursive OpenTofu formatting and validation, Caddy adaptation with dummy basic-auth fixtures, static Quadlet generation, plus rendered non-secret Luci, DDNS, and Soft Serve Quadlet samples.
It does not read ignored variable files, state, saved plans, generated secret files, or `.env`; it does not plan, apply, contact the configured backend, or verify live deployment behavior.
Use `just verify infra tofu-route` or `just verify infra quadlets` for the corresponding CI surface.

Copy `.env.example` to `.env` and provide the bucket-scoped R2 access key and
secret.
Use `just tofu init` for a clean checkout.
For the one-time local-state migration:

1. Stop every other OpenTofu operation.
2. Run `just tofu migrate <protected-backup-path>`.
   The recipe creates a mode-0600 backup and rejects resource-address drift.
3. Run and review `just plan`; reject backend-induced infrastructure changes.

The backend uses private bucket `bugabinga-net-tofu-state`, key
`root/terraform.tfstate`, and native lockfile coordination.
OpenTofu 1.12.1 can leave an orphaned lock after an ambiguous conditional
write.
Only after confirming no operation is active and the reported lock belongs to
the failed operation, clear it with `just tofu unlock <lock-id>`.
Run `just tofu test lock` to verify contention against a disposable state key.
Run `just tofu test backend-failure` to verify backend outages fail closed.
Run `just tofu migrate-auth` once, then remove the migrated
`zot_htpasswd_content` and `hallucygenie_basic_auth_users` assignments from
`secrets.auto.tfvars` and confirm `just plan` shows no credential changes.

Preserve the independent backup until `just tofu test restore <backup-path>`
restores it to a disposable R2 key and verifies every state address.
Only then use the same recovery path before replacing the authoritative object.
Rotate R2 credentials in Cloudflare and `.env`; backend configuration and saved
plans contain no credential values.

## Secrets management

Since the services to manage secrets (e.g. Hashicorp's Vault) are created within
this project, these cannot be used here to manage secrets. Instead, secrets are
managed manually via files excluded from version control:

- `secrets.tf` -> version controlled, declares all secret variables and sets
  them as `sensitive`, as to prevent logging them by opentofu. Does **not**
  contain secret values.
- `secrets.auto.tfvars` -> ignored by version control. Contains manually managed
  secret values referenced by `secrets.tf`.
- `auth.auto.tfvars.json` -> ignored generated authentication values maintained
  by `just generate zot-user` and `just generate basic-auth-user`.

After checking out this repository, create `secrets.auto.tfvars` manually for
non-authentication values and generate the required authentication entries.
If referenced values are missing, OpenTofu prompts during planning.

OpenTofu state, saved plans, and generated files can contain plaintext secrets
(e.g. WireGuard private keys).
Authoritative state is stored only in the private R2 backend.
Keep local state copies, `tfplan`, `secrets.auto.tfvars`,
`auth.auto.tfvars.json`, and `modules/klops/generated/` private and do not sync
them elsewhere.

## Homelab Services (Podman/Quadlets)

The home server (klops) runs various services via Podman with systemd quadlets.
These are defined in `modules/klops/quadlets/` and deployed automatically via
OpenTofu.

### Quadlet & Homepage Management

Quadlets and Homepage config are deployed automatically when you run `just apply`.
Homepage config templates live under `modules/klops/templates/homepage/` and are
rendered into `modules/klops/generated/homepage/` before being synced to
`/data/Databases/homepage/config` on klops. Optional widget secrets are provided
via `homepage_jellyfin_api_key` and `homepage_paperless_api_token` in
`secrets.auto.tfvars`.

To check syntax locally:

```bash
just check modules/klops/quadlets
```

### Adding a New Service

1. Create a `.container` or `.pod` file in `modules/klops/quadlets/`
2. Run `just plan`, review `tfplan`, then run `just apply` to deploy to klops
3. Configure reverse proxy in Caddy (update
   `modules/klops/quadlets/caddy.container` or the Caddyfile on klops)

### Source/CI/Package Services

Soft Serve owns Git repositories and SSH access at `vcs.bugabinga.net:22`; its data lives in `/data/vcs/public`.
Luigit is a read-only web interface at `vcs.bugabinga.net` HTTPS, routed by Caddy to `luigit:8080`.
Luigit reads `/data/vcs/public/repos` read-only and advertises only `refs/heads/*`, `refs/tags/*`, and `refs/notes/*`; it does not own Git data or write refs.
The configured Caddy route and Quadlets describe intended infrastructure, not proof of currently deployed source or live behavior.
Luci is CI and artifact publication, served at `ci.bugabinga.net` under a dedicated `ci` user with its own rootless Podman socket.
Luci manual submissions pin a resolved revision at admission; registry publication reports an immutable manifest digest after successful publication.
Toad is a separate operator-controlled digest rollout service, not an automatic Luci deploy target.
Caddy terminates public HTTPS and authentication; for Toad it is the outer authenticated proxy to a private-network Unix-socket ingress gateway.
There is no CI-to-Toad credential or automatic deployment path.
Any future automated deployment requires explicit operator-issued service-scoped credentials plus trusted-ref/revision policy enforcement before Toad invocation.
Build the custom Luci image on klops with `just deploy luci-image`.
- `pkg.bugabinga.net` serves static package/release trees from `/data/pkg`; `/v2/*` proxies to zot for OCI clients.
- `registry.bugabinga.net` serves the zot web UI and registry API.
- `genie.bugabinga.net` serves HallucyGenie behind Caddy basic auth.
- `quest.bugabinga.net` serves Quest Log; deploy sibling source with `just deploy quest-log-image`.

`auth.auto.tfvars.json` stores `zot_htpasswd_content` with htpasswd-format
users (for example an `admin` user matching zot `adminPolicy`). Generate or
update entries with:

```bash
just generate zot-user admin
```

HallucyGenie needs `hallucygenie_minimax_api_key` in `secrets.auto.tfvars` and
stores `hallucygenie_basic_auth_users` in `auth.auto.tfvars.json`. Generate or
update basic-auth users with:

```bash
just generate basic-auth-user jan
just generate basic-auth-user oli
```

Luci image bootstrap order:

```bash
just deploy luci-image # creates ci user if needed and builds the image on klops
just plan
just apply
```

First Luci repo setup sequence:

```bash
just deploy luci-image
just plan
just apply
# add .ci/*.kdl to any public repo, push, then check https://ci.bugabinga.net
```

No per-repo hook install is needed. Luci polls public bare repo refs and
auto-discovers current and future repos. The SSH interface exposes dashboard,
repo, run, log, docs, and manual-run commands through `ci`.
Manual submissions resolve requested refs/revisions and validate jobs before enqueueing; execution uses the admitted revision rather than resolving a mutable ref again.

Luci supports opt-in push triggers with changed-path filters, five-field UTC
cron schedules, persistent keyed caches, and `registry`, `pkg`, and `site`
publish adapters. The live agent reference is served at `/llms.txt`.
Repository `.ci/go.kdl` runs Go vet, tests, race tests, Staticcheck, plus Luigit Rust formatting, Clippy, and tests on matching `trunk` pushes, while the networked Govulncheck job runs on a weekly UTC schedule.
Registry publishing reads operator-managed auth from
`/data/ci/secrets/zot-auth.json`.
Jobs must write a fresh OCI image archive, then declare `publish "registry"` with `from` archive path and `to` registry target.
Luci stages that archive and runs daemonless `skopeo copy`; legacy host-image `image` publishing is rejected.
Successful publication retains requested `destination`, logs `digest=sha256:<64 lowercase hex>`, and logs canonical `reference=<repository>@<digest>` without any requested tag.
Pass only strict bare `digest` field to operator's `toad deploy <service> <digest>` command.
Toad accepts only `sha256:<64 lowercase hex>` and supplies enrolled repository itself.
Toad supplies the enrolled image repository and validates the manifest digest; Luci does not invoke Toad or grant deployment authority.

Luci v1 treats every visible repo, log, and artifact as public-readable.
Known declared secret values are masked from ANSI-free logs best-effort;
masking is not a security boundary.

Deployment helpers are split by execution location:

- `scripts/local/klops.sh`: local SSH/SCP/rsync dispatcher used by OpenTofu.
- `scripts/local/build-remote-image.sh`: streams a context into a rootless klops build.
- `scripts/local/load-luci-image.sh`: prepares and restarts Luci around that build.
- `scripts/local/auth-user.sh`: local Zot and Caddy bcrypt user helper.
- `scripts/remote/`: streamed over SSH and executed on klops.

Storage is bounded automatically:

- Luci removes every completed run workspace and sweeps abandoned workspaces before startup.
- Weekly per-owner GC (`bugabinga-gc@oli|ci|toad.timer`) removes stopped containers older than seven days and dangling images older than seven days from each user's rootless Podman storage; tagged images such as the Luci rollback image are never pruned, and volumes are never pruned automatically.
- `podman auto-update` for `oli` no longer prunes images afterwards (drop-in clears the vendor `ExecStartPost`), so an update result is never masked by a prune failure.
- Luci logs expire after 90 days, journald keeps at most 2 GiB or 30 days, and rsyslog keeps seven compressed daily rotations capped at 100 MiB each.

Maintenance health is supervised:

- A weekly root-run check (`/usr/local/sbin/bugabinga-check-maintenance`, installed from `scripts/remote/check-maintenance.sh`) verifies GC freshness within 8 days, `oli` auto-update trigger within 25 hours, existence of the Luci rollback image, and `/data/ci` plus root disk thresholds; it prints one TSV line per check and exits nonzero on failure.
- Failed GC, host-cleanup, or maintenance units trigger `bugabinga-alert@%n.service`, which logs a critical journal entry and additionally mails `root` when `mailx` is installed on klops; without `mailx` the journal entry is the only notification.
- All timers are `Persistent=true`, so a host that was down catches up on missed runs.

---

## WireGuard Configuration

WireGuard is now managed by OpenTofu. The config is generated automatically when
you run `just apply`. The generated config is stored in:

- Server (alpha): `/etc/wireguard/wg0.conf` (via cloud-init)
- Client (klops): `modules/klops/generated/klops-wg0.conf`

[opentofu]: https://opentofu.org
[just]: https://just.systems
[lefthook]: https://lefthook.dev
[gitleaks]: https://gitleaks.io