Introduction

Website: varta.sh · Guides: comparisons & Prometheus setup

Varta is a zero-dependency, zero-allocation health protocol for distributed local agents and networked clusters. Agents emit a fixed 32-byte heartbeat over a Unix domain socket (or UDP, with AEAD); an observer (varta-watch) decodes the beats, detects stalls, fires recovery commands, and exposes Prometheus metrics — all from a single poll loop on a single thread, with no heap allocation on the steady-state path.

Should I use Varta?

Varta is not a tracing system, an APM, a load balancer, or a config-distribution tool. It carries one fact per beat: this PID is alive, and here’s how it feels about that.

The “Zero-Everything” philosophy

• Zero registry dependencies. Production crates (varta-client, varta-watch) have empty [dependencies] sections. The single exception is varta-vlp’s optional crypto feature.
• Zero heap allocation on the beat path after Varta::connect(). Every encode/decode operates on stack buffers. Enforced by a guard-allocator test in CI.
• Non-blocking I/O. The agent socket is non-blocking at connect time. WouldBlock is treated as Dropped, never as an error that stalls the caller.
• Zero unsafe in varta-vlp and varta-client; auditable, line-by- line opt-in unsafe in varta-watch for platform syscalls.

How it works

flowchart LR
    A1[agent 1]  -.beat 32B.-> S[(UDS /run/varta/varta.sock)]
    A2[agent 2]  -.beat 32B.-> S
    AN[agent N]  -.beat 32B.-> S
    S --> O{varta-watch<br/>poll loop}
    O -- per-pid state --> T[Tracker<br/>≤ 4096 pids]
    O -- stall detected --> R[Recovery<br/>exec, non-blocking spawn]
    O -- /metrics --> P[Prometheus]
    O -- TSV --> AL[Audit log]

1. Agents call Varta::connect() once, then beat(status, payload) on whatever cadence they like (typically every 100 ms – 1 s).
2. The observer polls the socket on a 100 ms read-timeout cadence, decodes frames on the stack, updates per-pid state, fires recovery commands for pids past their silence threshold, and serves /metrics. All on one thread.
3. Recovery is opt-in and gated. Only kernel-attested beats are eligible; UDP / socket-mode-only beats are refused with a labelled counter.

Where to next

Install

Five paste-paths covering every mainstream operator audience. Pick the one that matches your runtime. Every artifact in every path is built, signed, and published by the same workflow (.github/workflows/release.yml), so the supply chain is uniform regardless of how you fetch it.

Kubernetes adopters who want raw manifests over Helm: keep using observability/examples/kubernetes/. The Helm chart’s default render matches those files for container args, mounts, security context, ports, and probes — asserted by the helm-parity CI gate. Helm-standard labels (helm.sh/chart, app.kubernetes.io/managed-by) are the only deltas.

Bare metal / VM — install.sh

curl -fsSL https://varta.sh/install.sh | sh

Knobs (env vars):

The script:

1. Detects OS + arch and computes the matching release-asset triple.
2. Downloads the tarball + .sha256 and verifies integrity.
3. If cosign is on $PATH, verifies the keyless signature against the GitHub Actions OIDC issuer + the aramirez087/Varta certificate subject (otherwise prints a recommendation and continues).
4. Copies the binary to $INSTALL_DIR/varta-watch.
5. On a systemd host running as root: creates the varta user, generates /etc/varta/prom.token (mode 0400), installs the unit at /etc/systemd/system/varta-watch.service, and prints the systemctl enable --now varta-watch invocation. Does not start the service automatically (operator-confirmation gate).

The installer itself is published as a release asset; the URL https://varta.sh/install.sh redirects to the versioned, raw GitHub URL so the script you run is reproducible.

Docker

docker run -d --name varta-watch \
  --user 65532:65532 \
  --restart unless-stopped \
  --read-only \
  --tmpfs /tmp \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  -v /run/varta:/run/varta \
  -v /etc/varta/prom.token:/etc/varta/prom.token:ro \
  -p 127.0.0.1:9100:9100 \
  ghcr.io/aramirez087/varta-watch:0.3.0 \
  --socket=/run/varta/varta.sock \
  --prom-addr=0.0.0.0:9100 \
  --prom-token-file=/etc/varta/prom.token \
  --self-watchdog-secs=4

Verify the image before pulling in production:

cosign verify ghcr.io/aramirez087/varta-watch:0.3.0 \
  --certificate-identity-regexp '^https://github.com/aramirez087/Varta' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com'

See Container Image for the full reference.

Kubernetes (Helm)

helm install varta-watch \
  oci://ghcr.io/aramirez087/charts/varta-watch \
  --version 0.2.0 \
  --create-namespace \
  --namespace varta \
  --set prometheusToken.token=$(openssl rand -hex 32)

Then helm test varta-watch -n varta runs an in-cluster pod that scrapes /metrics with the bearer token and asserts the observer is emitting varta_iterations_total. See Helm Chart for the full values reference.

Rust developer

cargo binstall varta-watch          # fetches the signed release tarball
# or
cargo install --path crates/varta-watch --features prometheus-exporter

cargo binstall resolves the platform triple via [package.metadata.binstall] in crates/varta-watch/Cargo.toml, pulls the same signed .tar.gz the installer script uses, and verifies the sha256 sidecar before writing to $CARGO_HOME/bin. No source build needed.

Verifying the artifacts

Every release ships:

• varta-watch-vX.Y.Z-<triple>.tar.gz — the binary + LICENSE + systemd unit
• varta-watch-vX.Y.Z-<triple>.tar.gz.sha256 — checksum
• varta-watch-vX.Y.Z-<triple>.tar.gz.cosign.bundle — keyless cosign signature bundle
• varta-watch-vX.Y.Z-<triple>.sbom.cdx.json — CycloneDX SBOM
• SLSA L3 build provenance attached via GitHub’s attestation API
• install.sh — the installer script itself (so you can verify before running)

Verify any binary tarball:

cosign verify-blob \
  --bundle varta-watch-v0.2.0-linux-amd64-musl.tar.gz.cosign.bundle \
  --certificate-identity-regexp '^https://github.com/aramirez087/Varta' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  varta-watch-v0.2.0-linux-amd64-musl.tar.gz

Verify SLSA provenance (uses GitHub’s verify API; no install needed):

gh attestation verify varta-watch-v0.2.0-linux-amd64-musl.tar.gz \
  --repo aramirez087/Varta

Troubleshooting

• cosign not found — the install script prints a recommendation and continues with sha256-only verification, which is still strong given HTTPS + the GitHub release-asset URL. To require cosign, set VERIFY_COSIGN=1.
• systemd unit fails to start — journalctl -u varta-watch -e surfaces the real error. The most common cause is a missing /etc/varta/prom.token (the installer creates it; manual installs must do so before the first start).
• Container exits immediately — the binary fails fast on missing required flags. docker logs varta-watch shows the parse error. Compare against the canonical args in the Docker block above.
• Helm test pod fails — kubectl -n varta logs <pod> from the failing pod prints the exact curl output. The most common cause is a bearer token mismatch between the Secret and the binary’s --prom-token-file mount.

Guides

Practical comparisons and setup walkthroughs for teams evaluating Varta against familiar watchdog and observability patterns.

The full operator reference remains in Monitoring & Alerting. The loadable artefact bundle lives in observability/.

Varta vs the alternatives

If you run background workers, daemons, or edge agents, you already have some liveness story — systemd WatchdogSec, a supervisord/monit config, Kubernetes liveness probes, or an HTTP /health route a prober polls. Each is the right tool when its assumptions hold: one process per unit, one orchestrator, one HTTP server already in the binary. The question this page answers is narrower: when you have many local processes and don’t want to add a unit file, a pod, or an inbound TCP port to each one, what does adding varta-watch actually buy you — and where do the incumbents still win?

At a glance

Which one should you pick?

Pick systemd WatchdogSec if your processes are 1:1 with units, recovery is just Restart=on-failure, and you don’t need per-beat payload (queue depth, degraded mode) or native Prometheus metrics. It’s already on the host, costs nothing extra, and is the simplest correct answer for a handful of long-running daemons.

Pick supervisord / monit if you want a single process manager to start, restart, and check a modest set of children on a non-systemd host, and process-alive (plus an occasional check command) is a sufficient liveness signal. You get supervision and recovery without per-beat instrumentation or a metrics pipeline — and without each child needing to emit anything.

Pick k8s liveness probes if you’re already on Kubernetes and your workloads are containers the kubelet schedules. The probe + restart loop is built in, no extra component to run, and it’s the idiomatic choice — there’s no reason to add Varta for liveness of pods the orchestrator already restarts.

Pick HTTP /health polling if the workload is already an HTTP server, /health is one more route, and your orchestrator or load balancer only speaks HTTP. It’s also the right call when you need rich JSON diagnostics a human can curl. Note it signals liveness but doesn’t recover anything — recovery stays external.

Pick Varta when you have many local processes (tens to thousands) and adding a unit file, pod, or inbound TCP port per PID is operationally heavy or simply unavailable — embedded / no_std / non-systemd / inside a container with no orchestrator. It fits high-frequency liveness (100 ms–1 s) where p99 matters, polyglot fleets that need uniform metrics and labels, and safety-critical builds that must not ship an HTTP server or a shell. It’s niche and local-first by design; for a single 1:1 daemon, systemd is the better tool.

You don’t have to choose just one

These aren’t mutually exclusive, and the common production setup combines them: run varta-watch itself as a single Type=notify systemd unit (or one k8s sidecar), and let systemd/k8s watch the watcher while Varta watches the fleet. systemd or the kubelet restarts the one observer if it stalls; Varta restarts the many agents when their beats go silent. You get the incumbent’s battle-tested supervision for the single long-lived component, and Varta’s one-collector-many-agents model for everything that would otherwise need a unit, pod, or HTTP port each. See Varta vs systemd WatchdogSec for the hybrid systemd recipe and observability/examples/varta-watch.service for the unit file.

Next steps

• Install (Quickstart)
• Varta vs systemd WatchdogSec
• Varta vs HTTP /health checks
• Prometheus setup walkthrough

Varta vs systemd WatchdogSec

systemd’s per-unit watchdog is excellent when one service equals one unit and recovery is systemctl restart <unit>. Varta fits when many processes on a host must report liveness to one operator-facing pipeline (Prometheus, audit log, debounced recovery) without giving each agent an HTTP server or its own systemd unit.

Comparison

When systemd alone is enough

• A single long-running daemon with a 1:1 unit file.
• Recovery policy is entirely Restart=on-failure inside systemd.
• You do not need per-beat custom payload (queue depth, degraded mode) in metrics.

When to add Varta

• Agent fleets (tens–thousands of workers) where creating a unit per PID is operationally heavy.
• Uniform metrics: varta_beats_total, stall counters, recovery refusals with consistent labels across languages.
• Kernel-attested beats on Linux (peer credentials) with an explicit refusal to run recovery for unauthenticated origins — see Peer Authentication.
• Hospital / embedded profiles that must not ship an HTTP /metrics server in the safety binary — use prometheus-exporter only on a non–Class-A build.

Hybrid pattern (common in production)

Run varta-watch as a single Type=notify systemd service (see observability/examples/varta-watch.service) and keep application agents as lightweight processes that only call beat(). Let systemd restart the observer if it stalls; let Varta restart agents when their beats go silent.

Next steps

• Install (Quickstart)
• Prometheus setup walkthrough

Varta vs HTTP /health checks

HTTP health endpoints are the default in Kubernetes and many service meshes. They work well for request/response services behind a load balancer. Varta targets local agents and side processes that should not pay for an HTTP stack, TLS, JSON parsing, or a listening TCP port on every PID.

Comparison

When HTTP health is the right tool

• The workload is already an HTTP server and /health is one route.
• Your orchestrator only speaks HTTP (some LB health checks).
• You need rich JSON diagnostics readable by curl.

When Varta is the better fit

• High-frequency liveness (100ms–1s) on hot paths where p99 matters.
• Embedded / no_std agents using varta-vlp without an HTTP library.
• Bare-metal fleets monitored by Prometheus without kubelet probes.
• Safety-critical deployments where opening inbound TCP on every agent is unacceptable attack surface.

Migration sketch

1. Stand up varta-watch with --socket (and optional secure UDP) — Install.
2. Replace the periodic HTTP self-check loop with beat(Status::Ok, payload).
3. Point Prometheus at varta-watch instead of per-agent blackbox jobs.
4. Map old probe failures to alert rules in observability/alerts/varta.rules.yml.

Keep HTTP /health for external traffic if customers still need it; use Varta for internal liveness the platform owns.

Next steps

• Benchmark Results
• Prometheus setup walkthrough

Prometheus setup walkthrough

This page is the fastest path from zero to scraping varta-watch, importing the bundled Grafana dashboard, and paging on stalls. For alert semantics see Monitoring & Alerting.

Prerequisites

• varta-watch built with the default prometheus-exporter feature (not the Class-A excised profile).
• A bearer token file when --prom-addr is set.

openssl rand -hex 32 | sudo tee /etc/varta/prom.token >/dev/null
sudo chmod 0400 /etc/varta/prom.token

1. Start the observer

varta-watch \
  --socket /run/varta/varta.sock \
  --threshold-ms 2000 \
  --prom-addr 127.0.0.1:9100 \
  --prom-token-file /etc/varta/prom.token \
  --self-watchdog-secs 4

Verify metrics locally:

curl -s -H "Authorization: Bearer $(sudo cat /etc/varta/prom.token)" \
  http://127.0.0.1:9100/metrics | head

2. Configure Prometheus scrape

Copy the job from observability/examples/prometheus-scrape.yml into your prometheus.yml:

scrape_configs:
  - job_name: varta-watch
    scheme: http
    metrics_path: /metrics
    static_configs:
      - targets: ["127.0.0.1:9100"]
    authorization:
      type: Bearer
      credentials_file: /etc/varta/prom.token

Reload Prometheus (POST /-/reload or SIGHUP).

3. Install recording + alert rules

sudo cp observability/recording-rules/varta.rules.yml /etc/prometheus/rules.d/
sudo cp observability/alerts/varta.rules.yml          /etc/prometheus/rules.d/
curl -X POST http://localhost:9090/-/reload

Confirm under Status → Rules that groups varta-watch.sli and varta-watch.critical appear.

4. Import Grafana dashboard

Grafana → Dashboards → Import → Upload observability/dashboards/varta-health.json and select your Prometheus datasource (24 panels, stall/recovery focused).

5. Route Alertmanager

Merge observability/examples/alertmanager.yml into your Alertmanager config. Replace PagerDuty / Slack placeholders before reload.

Kubernetes (kube-prometheus)

Skip hand-edited prometheus.yml and apply:

kubectl apply -f observability/examples/kubernetes/varta-watch.deployment.yaml
kubectl apply -f observability/examples/kubernetes/varta-watch.servicemonitor.yaml

The release: label on the ServiceMonitor must match your Prometheus operator selector — see Deployment Patterns.

Smoke test with a live agent

cargo run -p varta-client --example basic

Watch varta_beats_total increment and confirm the Grafana Beats row moves.

Related

• Varta vs HTTP health checks
• Observability bundle README

VLP v0.2 — Base Frame Specification

Status: Normative. Wire version: 0x02. Document audience: anyone building a Varta-compatible client, observer, packet decoder, or fuzz harness in any language.

This document defines the on-wire layout of a Varta Lifeline Protocol (VLP) heartbeat frame. The Rust reference implementation lives at crates/varta-vlp, but no part of this specification depends on Rust. A conforming implementation in any language MUST produce and accept the byte sequences described below.

A conformance test-vector suite is published at tools/vlp-test-vectors.json; running an implementation against that file is the definition of conformance (see Conformance).

1. Conventions

The keywords MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are interpreted as in RFC 2119 / RFC 8174.

• Integers are unsigned, little-endian, of the width specified in each field row.
• Byte offsets are zero-based and inclusive on the left.
• || denotes byte concatenation.
• 0x.. is hexadecimal; 0b.. is binary; ASCII byte literals appear in double quotes ("VA").
• Capitalised type names (Status, Frame, DecodeError) refer to the conceptual entities defined in this document, not to types in any specific language.

Terminology

2. Frame Structure

A VLP frame is exactly 32 bytes:

offset │ size │ field      │ type   │ notes
───────┼──────┼────────────┼────────┼─────────────────────────────────────────
 0     │  2   │ magic      │ u8[2]  │ const [0x56, 0x41]  (ASCII "VA")
 2     │  1   │ version    │ u8     │ const 0x02  (this document)
 3     │  1   │ status     │ u8     │ Status enum, see §3.2
 4     │  4   │ pid        │ u32    │ LE — emitter's process id
 8     │  8   │ timestamp  │ u64    │ LE — emitter-local monotonic value
16     │  8   │ nonce      │ u64    │ LE — strictly increasing per session
24     │  4   │ payload    │ u32    │ LE — opaque application-defined context
28     │  4   │ crc32c     │ u32    │ LE — CRC-32C over bytes 0..28
───────┴──────┴────────────┴────────┴─────────────────────────────────────────
                                                              total 32 bytes

Senders MUST emit exactly 32 bytes per beat. Receivers MUST reject any datagram whose length is not exactly 32 bytes (or, for secure transports, the corresponding wrapped length defined in VLP — Secure Transport).

3. Field Semantics

3.1 Magic (offset 0, 2 bytes)

The literal bytes 0x56 0x41 (ASCII "VA"). Senders MUST emit exactly these two bytes. Receivers MUST reject any frame whose first two bytes differ, with a BadMagic diagnostic.

3.2 Version (offset 2, 1 byte)

The literal byte 0x02. Senders MUST emit 0x02. Receivers MUST reject any other version byte with a BadVersion diagnostic — including the legacy 0x01 (VLP v0.1). Implementations MUST NOT attempt to re-interpret a 0x01 frame; the v0.1 byte map is incompatible with v0.2.

3.3 Status (offset 3, 1 byte)

A single byte enumerating the agent’s last reported health:

Receivers MUST reject any other value with a BadStatus diagnostic.

3.4 PID (offset 4, 4 bytes, u32 little-endian)

The operating-system process identifier of the emitting agent, in the agent’s own PID namespace. Senders MUST set this to the OS-reported PID at frame-construction time, freshly per emit (caching across fork(2) is a wire-protocol bug — see §3.7).

Receivers MUST reject pid values 0 (kernel/scheduler) and 1 (init/systemd) with a BadPid diagnostic. No legitimate Varta agent runs at either; the rejection closes a “spoof init has stalled” recovery-trigger attack vector on unauthenticated transports.

3.5 Timestamp (offset 8, 8 bytes, u64 little-endian)

A monotonic timestamp chosen by the emitter, typically nanoseconds since an agent-local epoch. Observers do not interpret it; they only compare consecutive timestamps for the same pid.

Receivers MUST reject timestamp == 0xFFFFFFFFFFFFFFFF (the saturation sentinel) with a BadTimestamp diagnostic. Reaching the sentinel via real elapsed nanoseconds requires ~584 years, so the value is reserved.

3.6 Nonce (offset 16, 8 bytes, u64 little-endian)

A strictly increasing 64-bit counter, starting at 1 on the first beat after session establishment. On exhaustion at 0xFFFFFFFFFFFFFFFE, the counter wraps to 0 and continues. The sentinel value 0xFFFFFFFFFFFFFFFF (NONCE_TERMINAL) is reserved for panic frames (see §3.7); regular beats MUST NOT use it.

3.7 Reserved Values

nonce == 0xFFFFFFFFFFFFFFFF paired with status == Critical is the unique on-wire marker for a panic-terminated agent. Receivers MUST accept this pairing; downstream consumers (alert rules, recovery filters) MAY use it to distinguish panic-terminal from operational-critical frames.

The converse — that Critical always implies the terminal nonce — is not enforced. An agent emitting Status::Critical for operational alerts (queue full, shedding load) MAY use any regular nonce value.

3.8 Payload (offset 24, 4 bytes, u32 little-endian)

Application-defined opaque context. The protocol carries it through unchanged. Typical uses: queue depth, last error code, an index into a shared ring buffer. Implementations MUST NOT interpret this field at the protocol layer; downstream consumers define its meaning.

3.9 CRC-32C Trailer (offset 28, 4 bytes, u32 little-endian)

A CRC-32C (Castagnoli) checksum computed over bytes 0..28 of the frame. Wire layout is little-endian. See §4 for the parameter list and reference algorithm.

The CRC catches:

• Single-event-upset bit flips on non-ECC RAM (agent or observer host).
• NIC firmware corruption between RX queue and userspace.
• In-process memory corruption between encode and the transport write (or between transport read and decode).

For secure transports, the CRC is defence-in-depth against in-process corruption on either side of the AEAD seal/open boundary; AEAD tag failures surface as a transport-layer error (AuthError/AEAD failure), never as BadCrc.

Receivers MUST verify the CRC after the magic and version checks (so wrong-protocol bytes surface diagnostically as BadMagic / BadVersion, not BadCrc) and before any field-range check (so corruption cannot produce a “well-formed” frame with the wrong meaning).

4. CRC-32C

VLP uses CRC-32C (Castagnoli) — the iSCSI/SCTP/ext4/Btrfs CRC, not the IEEE 802.3 CRC-32. Parameters (Koopman notation CRC-32C/iSCSI):

4.1 Reference vectors

4.2 Byte-at-a-time algorithm

compute(bytes):
    table[0..256] := build_table()
    crc := 0xFFFFFFFF
    for each byte b in bytes:
        idx := (crc XOR b) AND 0xFF
        crc := table[idx] XOR (crc >> 8)
    return crc XOR 0xFFFFFFFF

build_table():
    for i in 0..256:
        c := i
        repeat 8 times:
            if c AND 1 != 0:
                c := (c >> 1) XOR 0x82F63B78
            else:
                c := c >> 1
        table[i] := c
    return table

The Rust reference implementation lives at crates/varta-vlp/src/crc32c.rs and is const fn-evaluable — the 256-entry table is built at compile time.

Hardware CRC-32C is available on x86_64 (SSE 4.2) and ARMv8.1+; conforming implementations MAY use hardware acceleration as long as the output matches the reference vectors above.

5. Decode Order

Receivers MUST validate fields in the following order, returning the first encountered failure:

1. Magic → BadMagic if bytes 0..2 ≠ 0x56 0x41.
2. Version → BadVersion if byte 2 ≠ 0x02.
3. CRC → BadCrc if compute(bytes[0..28]) ≠ read_u32_le(bytes[28..32]).
4. Status → BadStatus if byte 3 ∉ {0x00, 0x01, 0x02, 0x03}.
5. Stall-on-wire → StallOnWire if status byte 3 = 0x03.
6. PID range → BadPid if pid ∈ {0, 1}.
7. Timestamp range → BadTimestamp if timestamp == 0xFFFFFFFFFFFFFFFF.
8. Nonce/status pairing → BadNonce if nonce == 0xFFFFFFFFFFFFFFFF and status ≠ Critical.

The order is normative: operators rely on observing BadMagic versus BadCrc to distinguish “wrong protocol on this port” from “real VLP but corrupted in flight”.

6. Encode Procedure

encode(status, pid, timestamp, nonce, payload) -> [32]byte:
    out[0]      := 0x56            # 'V'
    out[1]      := 0x41            # 'A'
    out[2]      := 0x02            # version
    out[3]      := status_byte(status)
    out[4..8]   := u32_le(pid)
    out[8..16]  := u64_le(timestamp)
    out[16..24] := u64_le(nonce)
    out[24..28] := u32_le(payload)
    crc         := crc32c_compute(out[0..28])
    out[28..32] := u32_le(crc)
    return out

Senders MUST compute the CRC over the encoded bytes 0..28, after every preceding field has been written. Mutating any byte between encode and the transport write will cause the receiver to reject the frame as BadCrc.

7. Decode Procedure

decode(bytes: [32]byte) -> Frame | DecodeError:
    if bytes[0..2] != [0x56, 0x41]:
        return BadMagic
    if bytes[2] != 0x02:
        return BadVersion
    stored_crc   := u32_le(bytes[28..32])
    computed_crc := crc32c_compute(bytes[0..28])
    if stored_crc != computed_crc:
        return BadCrc(expected=computed_crc, actual=stored_crc)

    status_byte := bytes[3]
    if status_byte not in {0x00, 0x01, 0x02, 0x03}:
        return BadStatus(status_byte)
    if status_byte == 0x03:
        return StallOnWire

    pid       := u32_le(bytes[4..8])
    timestamp := u64_le(bytes[8..16])
    nonce     := u64_le(bytes[16..24])
    payload   := u32_le(bytes[24..28])

    if pid in {0, 1}:
        return BadPid(pid)
    if timestamp == 0xFFFFFFFFFFFFFFFF:
        return BadTimestamp(timestamp)
    if nonce == 0xFFFFFFFFFFFFFFFF and status_byte != 0x02:
        return BadNonce(nonce, status)

    return Frame{
        status, pid, timestamp, nonce, payload
    }

8. Worked Examples

8.1 Minimal Ok beat

status = Ok, pid = 2, timestamp = 0, nonce = 1, payload = 0:

56 41 02 00 02 00 00 00 00 00 00 00 00 00 00 00
01 00 00 00 00 00 00 00 00 00 00 00 e4 11 6b aa

8.2 Panic-hook terminal frame

status = Critical, pid = 2, timestamp = 999, nonce = 0xFFFF…FFFF, payload = 0:

56 41 02 02 02 00 00 00 e7 03 00 00 00 00 00 00
ff ff ff ff ff ff ff ff 00 00 00 00 a2 99 ee ed

Additional encoded golden frames live in tools/vlp-test-vectors.json.

9. Reference Implementations

Non-normative encoders/decoders in Python, C99, and Go live under tools/reference-implementations/. Each ships a verify_vectors driver that loads the conformance JSON and asserts every entry round-trips.

• Python — tools/reference-implementations/python/vlp.py (~80 lines, stdlib-only, Python 3.8+).
• C99 — tools/reference-implementations/c/vlp.c (~120 lines, <stdint.h> + <string.h> only).
• Go — tools/reference-implementations/go/vlp.go (~80 lines, encoding/binary only).

These snippets are examples, not part of the normative spec. The pseudocode in §6 and §7 is the authoritative reference.

10. Versioning Policy

The version byte 0x02 is the current wire format. Any change to field order, width, or semantics requires bumping the version byte; receivers emit BadVersion for unrecognised values. There is no silent backward compatibility — a single decoder implementation can support multiple versions only by branching on the byte at offset 2.

Future versions of this specification will increment VERSION and publish a new document; the test-vector file at tools/vlp-test-vectors.json carries a top-level spec_version field matching the wire version.

11. Conformance

Run the test-vector suite against your implementation:

1. Load tools/vlp-test-vectors.json.
2. For each entry in crc32c_vectors, run your CRC over input_hex (after hex decoding) and compare against expected_crc_hex.
3. For each encode_decode_roundtrip entry in frame_vectors, encode the inputs block and compare byte-for-byte against expected_wire_hex. Then decode expected_wire_hex and compare the recovered fields against inputs.
4. For each decode_error entry in frame_vectors, decode wire_hex and confirm the decoder returns expected_decode_error.

See Conformance & Test Vectors for the full JSON schema and language-by-language driver recipes.

See also

• VLP — Secure Transport — AEAD-wrapped frame for untrusted networks.
• Conformance & Test Vectors — JSON schema for the test-vector file.
• Rust implementation rationale — design choices in the reference implementation (#[repr(C, align(8))], zero-allocation policy, compile-time layout proofs). Not normative.

VLP Secure Transport Specification

Status: Normative. Frame versions: Shared-key 60-byte, Master-key 64-byte. Document audience: anyone implementing a Varta-compatible client or observer that crosses an untrusted network.

This document defines the encrypted wrapping of a VLP base frame (see VLP — Base Frame) for transmission over a network where the underlying transport offers no authentication or confidentiality (the typical example is UDP).

The base 32-byte VLP frame remains the plaintext; the secure transport wraps it in a ChaCha20-Poly1305 AEAD construction (RFC 8439). Two wrapped forms are defined:

• Shared-key (60 bytes) — a single pre-shared symmetric key is used by every agent.
• Master-key (64 bytes) — every agent has a key derived from a single master via HKDF-SHA256; the agent’s PID is bound into the AEAD AAD.

1. Conventions

The keywords MUST, MUST NOT, SHOULD, MAY follow RFC 2119 / RFC 8174. The base-frame conventions in VLP — Base Frame §1 apply unchanged.

In addition:

• Plaintext always refers to the canonical 32-byte VLP frame defined in VLP — Base Frame §2.
• Ciphertext is the AEAD-encrypted form of Plaintext, always 32 bytes.
• Tag is the 16-byte Poly1305 authentication tag.
• Nonce (in this document) refers to the 12-byte AEAD nonce, distinct from the 8-byte nonce field inside the plaintext base frame.

2. AEAD Primitive

All secure transports use ChaCha20-Poly1305 AEAD as defined in RFC 8439.

Implementations SHOULD use an externally-audited AEAD library (libsodium, BoringSSL, golang.org/x/crypto/chacha20poly1305, RustCrypto chacha20poly1305). Hand-rolled implementations of ChaCha20 or Poly1305 are strongly discouraged for production use.

The AEAD seal and open operations:

seal(key[32], nonce[12], aad[], plaintext[32]) -> (ciphertext[32], tag[16])
open(key[32], nonce[12], aad[], ciphertext[32], tag[16])
  -> Some(plaintext[32]) if tag verifies, else None

Senders MUST never reuse a (key, nonce) pair. Nonce-reuse under a fixed key catastrophically breaks both confidentiality and integrity (Poly1305 key recovery, plaintext-XOR leak). The nonce-construction rules in §5 prevent reuse for conforming implementations.

3. Shared-Key Wire Format (60 bytes)

A single 32-byte symmetric key K is provisioned out-of-band to every agent and observer. The wire frame is:

offset │ size │ field        │ notes
───────┼──────┼──────────────┼────────────────────────────────────────────
 0     │  8   │ iv_random    │ per-session OS entropy or HKDF derivation
 8     │  4   │ iv_counter   │ u32 LE — strictly increasing per emit
12     │ 32   │ ciphertext   │ AEAD(K, nonce, "", plaintext)
44     │ 16   │ tag          │ Poly1305 tag
───────┴──────┴──────────────┴────────────────────────────────────────────
                                                              total 60 bytes

• AEAD nonce = iv_random || iv_counter (12 bytes total).
• AAD = empty byte string.
• Plaintext = the 32-byte VLP base frame.

3.1 Encoder

encode_shared(key[32], iv_random[8], iv_counter: u32,
              plaintext[32]) -> [60]byte:
    nonce[0..8]  := iv_random
    nonce[8..12] := u32_le(iv_counter)
    (ct, tag) := seal(key, nonce, b"", plaintext)
    out[0..8]   := iv_random
    out[8..12]  := u32_le(iv_counter)
    out[12..44] := ct
    out[44..60] := tag
    return out

3.2 Decoder

decode_shared(key[32], wire[60]) -> Plaintext | AuthError:
    iv_random  := wire[0..8]
    iv_counter := u32_le(wire[8..12])
    ct         := wire[12..44]
    tag        := wire[44..60]
    nonce[0..8]  := iv_random
    nonce[8..12] := u32_le(iv_counter)
    plaintext := open(key, nonce, b"", ct, tag)
    if plaintext is None:
        return AuthError
    return plaintext  # caller MUST then run base-frame decode (§7 of VLP)

Observers MUST run the base-frame decoder (VLP — Base Frame §7) on the recovered plaintext before trusting any field.

4. Master-Key Wire Format (64 bytes)

A single 32-byte master key M is provisioned out-of-band. Every agent derives a per-agent key K_agent = HKDF(M, agent_pid) (see §6). The agent’s PID is bound into the AEAD AAD so an observer cannot accept a frame whose plaintext PID disagrees with the wire prefix.

offset │ size │ field        │ notes
───────┼──────┼──────────────┼────────────────────────────────────────────
 0     │  4   │ agent_pid    │ u32 LE — bound into AAD, NOT encrypted
 4     │  8   │ iv_random    │ per-session OS entropy or HKDF derivation
12     │  4   │ iv_counter   │ u32 LE — strictly increasing per emit
16     │ 32   │ ciphertext   │ AEAD(K_agent, nonce, agent_pid_bytes, pt)
48     │ 16   │ tag          │ Poly1305 tag
───────┴──────┴──────────────┴────────────────────────────────────────────
                                                              total 64 bytes

• AEAD nonce = iv_random || iv_counter (12 bytes).
• AAD = the 4-byte little-endian encoding of agent_pid (bytes 0..4 of the wire frame, verbatim).
• Plaintext = the 32-byte VLP base frame.
• K_agent = HKDF-SHA256(IKM=M, salt=empty, info="varta-agent-v1\0" || agent_pid_LE).

The plaintext frame still carries its own pid field at base-frame offset 4..8. Observers MUST verify that agent_pid (wire offset 0..4) equals the recovered plaintext pid and reject mismatches.

4.1 Encoder

encode_master(master[32], agent_pid: u32, iv_random[8], iv_counter: u32,
              plaintext[32]) -> [64]byte:
    K_agent := derive_agent_key(master, agent_pid)
    aad := u32_le(agent_pid)
    nonce[0..8]  := iv_random
    nonce[8..12] := u32_le(iv_counter)
    (ct, tag) := seal(K_agent, nonce, aad, plaintext)
    out[0..4]   := aad
    out[4..12]  := iv_random
    out[12..16] := u32_le(iv_counter)
    out[16..48] := ct
    out[48..64] := tag
    return out

4.2 Decoder

decode_master(master[32], wire[64]) -> Plaintext | AuthError:
    aad        := wire[0..4]               # agent_pid LE
    agent_pid  := u32_le(aad)
    iv_random  := wire[4..12]
    iv_counter := u32_le(wire[12..16])
    ct         := wire[16..48]
    tag        := wire[48..64]
    K_agent    := derive_agent_key(master, agent_pid)
    nonce[0..8]  := iv_random
    nonce[8..12] := u32_le(iv_counter)
    plaintext := open(K_agent, nonce, aad, ct, tag)
    if plaintext is None:
        return AuthError
    # Observer MUST also confirm plaintext.pid == agent_pid.
    return plaintext

5. Nonce Construction & Uniqueness

The AEAD nonce is the 12-byte concatenation iv_random || iv_counter.

• iv_random (8 bytes) MUST be drawn from cryptographic OS entropy at session start, or derived from a 16-byte session salt via the IV-prefix HKDF in §6.
• iv_counter (4 bytes, little-endian) MUST start at 0 for a new session and strictly increase by 1 per emit.
• The pair (iv_random, iv_counter) MUST NEVER be reused under the same key.

5.1 Counter exhaustion

A 32-bit iv_counter exhausts after 2^32 emits per session. Senders MUST rotate iv_random (and reset iv_counter to 0) before the counter wraps. Implementations MAY rotate proactively at any threshold; the canonical Rust transport rotates well under 2^32.

5.2 fork(2)

A child process inherits its parent’s iv_random and iv_counter after fork(2). If the child emits without first refreshing the IV, both processes will use the same (iv_random, iv_counter) under the same key — catastrophic nonce reuse. Senders MUST detect fork (e.g. by comparing the cached PID at session establishment against the current PID at emit time) and either:

1. Re-read iv_random from OS entropy and reset iv_counter = 0, or
2. Refuse to emit until the application explicitly re-establishes the session.

The Rust reference implementation auto-detects fork and rotates IV state transparently; see book/src/architecture/vlp-transports.md — “Fork-safety on secure-UDP” for the operational model.

6. HKDF Key Derivation

The master-key mode (and the IV-prefix rotation mode for both wire formats) uses HKDF-SHA256 as defined in RFC 5869. Three info strings are reserved, all suffixed with -v1 for future versioning:

6.1 Per-agent key derivation

derive_agent_key(master_key[32], agent_id: u32) -> [32]byte:
    info := b"varta-agent-v1\0" || u32_le(agent_id)
    # info length = 15 (the literal incl. NUL) + 4 = 19 bytes.
    return HKDF-SHA256(IKM=master_key, salt=empty, info=info, L=32)

• IKM = master key (32 B).
• salt = empty string (HKDF treats this as 0x00 * HashLen per RFC 5869 §2.2).
• info = the byte string "varta-agent-v1" (14 ASCII bytes) followed by one NUL byte (0x00), followed by agent_id in 4-byte little-endian encoding. Total info length: 19 bytes.
• L = 32 output bytes.

6.2 Per-session IV prefix derivation

derive_iv_prefix(session_salt[16], prefix_index: u32) -> [8]byte:
    info := b"varta-iv-prefix-v1\0" || u32_le(prefix_index)
    # info length = 19 + 4 = 23 bytes.
    return HKDF-SHA256(IKM=session_salt, salt=empty, info=info, L=8)

The session salt is the IKM; the HKDF-Extract salt argument is empty (RFC 5869 §2.2 treats empty salt as 0x00 * HashLen).

The Rust reference uses this to rotate iv_random over a session without re-reading OS entropy on the hot path.

6.3 Per-epoch key derivation (reserved)

derive_epoch_key(agent_key[32], epoch: u64) -> [32]byte:
    info := b"varta-epoch-v1\0" || u64_le(epoch)
    # info length = 15 + 8 = 23 bytes.
    return HKDF-SHA256(IKM=agent_key, salt=empty, info=info, L=32)

Epoch keys are reserved for forward compatibility — they are not currently used on the wire. Conforming implementations MAY skip this derivation.

6.4 Reference vectors

Full info byte strings (so external implementers can confirm endianness and the literal NUL):

7. Replay Protection

Observers MUST maintain per-sender state to reject replayed frames.

Known limitation (reference implementation). Per-sender replay state is reset when a PID is recycled and its session restarts after a silence gap, which opens a bounded window in which a captured frame of the dead session can be replayed — forging one liveness beat for the recycled PID (recovery commands remain gated and are not triggerable by the replay). A full fix needs a wire-level session/epoch identifier, deferred to a future VLP version. See vlp-transports.md — “Secure UDP — session-restart replay window (H5)” for the exact window, bound, and root cause.

Replay state MUST be keyed by the AEAD-authenticated sender identity, never by the UDP source address. A source address is neither stable — a legitimate reconnect changes the source port — nor authenticated — a replay attacker can resend a captured ciphertext from any port. A source-keyed scheme therefore both drops legitimate reconnects and admits a replayed frame from a fresh port. The sender identity below is bound by the Poly1305 tag, so it is the only identity an observer can trust.

7.1 Shared-key mode

Key by the VLP frame PID read from the decrypted plaintext; within that sender, track last_seen_counter per iv_random prefix. Accept a new frame only if iv_counter > last_seen_counter for its prefix; reject equal or lesser counters.

7.2 Master-key mode

Key by the on-wire agent_pid (which is also bound as the AEAD AAD), tracking the same per-prefix counter monotonicity rule. The UDP source address is not part of the key in either mode.

7.3 Bounded state

A real observer cannot retain unbounded per-sender state. The Rust reference implementation bounds per-sender records to 1024 simultaneous senders and refuses unknown senders at capacity after a stale-sender sweep; see book/src/architecture/vlp-transports.md — “Secure UDP — replay-state capacity boundary (H4)” for the precise capacity rule and the threat-model implication (loopback-default binding when secure-UDP is configured).

8. Worked Example — Shared-Key Seal

• Key: 0001020304050607 0809 0a0b0c0d0e0f 1011 1213 1415 1617 1819 1a1b 1c1d 1e1f (32 bytes)
• iv_random: 1122334455667788
• iv_counter: 0
• Plaintext (a base VLP frame, Status::Ok pid=2 ts=1000 nonce=1 payload=0): 5641020002000000e80300000000000001000000000000000000000055d0861c

Resulting 60-byte wire frame:

1122334455667788 00000000
bcba1b202190b688a08e0a7ac909da44a2023cb7a421fd6428453fd12141c257
b6fd638c55fddf5c621020de1327975a

9. Worked Example — Master-Key Seal

• Master key: 0001020304…1f (same as above, 32 bytes)
• agent_pid: 2 (0x02000000 LE)
• Derived agent key: db292f5843a0737aec785a9df270561b343d06e5fe8f89fce72f0869ba77afd5
• iv_random: 1122334455667788
• iv_counter: 0
• Plaintext: same as §8.

Resulting 64-byte wire frame:

02000000
1122334455667788 00000000
efe8fd8c226106641e01fc8fe649f79475e19b4f2093e063987f1c663a5d2f0b
73ba429fadc4c494e2723baff86af9cc

10. Stability

Implementations MUST NOT silently accept an unknown info-string version; any change to a derivation requires explicit re-keying across the deployment.

11. Reference Implementation

The Rust reference lives in crates/varta-vlp/src/crypto. The seal/open operations delegate to the RustCrypto chacha20poly1305 crate (NCC Group audit 2020); the HKDF derivations use the RustCrypto hkdf + sha2 crates. No hand-rolled ChaCha20, Poly1305, or KDF logic exists in the workspace.

Cross-language references (Python cryptography, C libsodium, Go golang.org/x/crypto/chacha20poly1305) live in tools/reference-implementations/.

12. Conformance

Run the test-vector suite against your implementation. The secure_frame_vectors array in tools/vlp-test-vectors.json contains:

• shared_key_seal — encode a 60-byte wire frame; compare to expected_wire_hex.
• master_key_seal — derive the agent key, encode a 64-byte wire frame; compare both the derived key and the wire bytes to the goldens.
• kdf_agent_key, kdf_iv_prefix, kdf_epoch_key — drive each HKDF derivation and compare to the published OKM.

See Conformance & Test Vectors for the JSON schema and end-to-end recipe.

See also

• VLP — Base Frame — the 32-byte plaintext layout this document wraps.
• Conformance & Test Vectors — JSON schema.
• Rust transport rationale — design trade-offs in the reference implementation (loopback default, bounded replay state, fork-safety auto-recovery). Not normative.

Conformance & Test Vectors

This document describes the cross-language conformance contract for the Varta Lifeline Protocol. An implementation in any language is conformant if and only if it agrees with the published test-vector suite at tools/vlp-test-vectors.json.

The Rust reference implementation regenerates this file via:

cargo run -p varta-vlp --example gen_test_vectors --features "std crypto"

and asserts byte-equality on every entry via the integration test crates/varta-vlp/tests/conformance_vectors.rs. The file is the source of truth; if it drifts from the Rust implementation, the test fails before CI passes.

1. JSON Schema

The top-level document is an object with these fields:

{
  "spec_version": "0.2",
  "description": "...",
  "magic_hex": "5641",
  "version_byte": 2,
  "nonce_terminal_hex": "ffffffffffffffff",
  "crc32c_vectors": [ ... ],
  "frame_vectors":  [ ... ],
  "secure_frame_vectors": [ ... ]
}

1.1 crc32c_vectors (array)

Reference CRC-32C (Castagnoli) inputs and outputs. Each entry:

{
  "id": "crc-rfc3720",
  "description": "RFC 3720 appendix B reference vector.",
  "input_hex": "313233343536373839",
  "expected_crc_hex": "e3069283"
}

• input_hex — the input byte string, lowercase hex.
• expected_crc_hex — the 4-byte CRC, lowercase hex, big-endian in the hex string (the leftmost hex pair is the most-significant CRC byte). On the wire, the CRC is little-endian; the JSON file presents the numeric value directly for readability.

1.2 frame_vectors (array)

Base-frame encode/decode goldens. Two kind values are defined.

Kind encode_decode_roundtrip:

{
  "id": "frame-ok-minimal",
  "description": "Minimum legal frame.",
  "kind": "encode_decode_roundtrip",
  "expected_decode_error": null,
  "inputs": {
    "status": "ok",
    "pid": 2,
    "timestamp": 0,
    "nonce": 1,
    "payload": 0
  },
  "expected_wire_hex": "56410200020000000000000000000000010000000000000000000000e4116baa"
}

• status is one of "ok", "degraded", "critical". ("stall" never appears as an input — Status::Stall is observer-synthesized only.)
• All numeric fields are unsigned decimal integers; widths match the base-frame spec (pid: u32, timestamp: u64, nonce: u64, payload: u32).
• expected_wire_hex is the canonical 32-byte wire frame, lowercase hex (64 characters).

A conformant implementation MUST:

1. Encode inputs and produce a byte sequence equal to expected_wire_hex.
2. Decode expected_wire_hex and recover fields equal to inputs.

Kind decode_error:

{
  "id": "frame-error-bad-magic",
  "description": "First byte 0x00 instead of 'V'.",
  "kind": "decode_error",
  "expected_decode_error": "BadMagic",
  "wire_hex": "00410200020000006400000000000000010000000000000000000000421795be"
}

• wire_hex — 32 bytes lowercase hex.
• expected_decode_error — one of BadMagic, BadVersion, BadCrc, BadStatus, StallOnWire, BadPid, BadTimestamp, BadNonce.

A conformant implementation MUST decode wire_hex and surface the named error variant.

1.3 secure_frame_vectors (array)

Secure-transport encode/decode + key-derivation goldens. Five kind values are defined.

Kind shared_key_seal:

{
  "id": "secure-shared-key-seal",
  "description": "Shared-key 60-byte secure frame.",
  "kind": "shared_key_seal",
  "key_hex": "0001020304…1f",
  "iv_random_hex": "1122334455667788",
  "iv_counter": 0,
  "plaintext_hex": "5641020002000000…1c",
  "expected_wire_hex": "11223344…975a"
}

A conformant implementation MUST:

1. Construct nonce = iv_random || iv_counter_LE.
2. AEAD-seal plaintext_hex under key_hex with empty AAD.
3. Concatenate iv_random || iv_counter_LE || ciphertext || tag and match expected_wire_hex exactly (60 bytes / 120 hex chars).

Kind master_key_seal:

{
  "id": "secure-master-key-seal",
  "kind": "master_key_seal",
  "master_key_hex": "0001…1f",
  "agent_pid": 2,
  "derived_agent_key_hex": "db29…afd5",
  "iv_random_hex": "1122334455667788",
  "iv_counter": 0,
  "plaintext_hex": "...",
  "expected_wire_hex": "..."
}

Steps: derive agent_key = HKDF-SHA256(master, info="varta-agent-v1\0" || agent_pid_LE); the derived key MUST match derived_agent_key_hex. Then seal under agent_key with AAD = agent_pid_LE and assemble the 64-byte wire frame.

Kinds kdf_agent_key, kdf_iv_prefix, kdf_epoch_key: HKDF goldens. Each carries the input material, the assembled info string (as info_hex), and the expected output. See VLP — Secure Transport §6 for the exact derivation procedure.

2. Running the Suite

2.1 Python (stdlib only)

import json, sys
import vlp  # your implementation

vectors = json.load(open("tools/vlp-test-vectors.json"))

for v in vectors["crc32c_vectors"]:
    got = vlp.crc32c(bytes.fromhex(v["input_hex"]))
    want = int(v["expected_crc_hex"], 16)
    assert got == want, f"{v['id']}: {got:08x} != {want:08x}"

for v in vectors["frame_vectors"]:
    if v["kind"] == "encode_decode_roundtrip":
        i = v["inputs"]
        wire = vlp.encode(i["status"], i["pid"], i["timestamp"],
                          i["nonce"], i["payload"])
        assert wire.hex() == v["expected_wire_hex"], v["id"]
    elif v["kind"] == "decode_error":
        try:
            vlp.decode(bytes.fromhex(v["wire_hex"]))
            assert False, f"{v['id']}: expected error"
        except vlp.DecodeError as e:
            assert e.kind == v["expected_decode_error"], v["id"]

A full runner is at tools/reference-implementations/python/verify_vectors.py.

2.2 Go (stdlib only)

import (
    "encoding/hex"
    "encoding/json"
    "os"
)

type Doc struct {
    Crc32cVectors []struct {
        ID, InputHex, ExpectedCRCHex string
    } `json:"crc32c_vectors"`
    // ...
}

doc := Doc{}
buf, _ := os.ReadFile("tools/vlp-test-vectors.json")
json.Unmarshal(buf, &doc)
for _, v := range doc.Crc32cVectors {
    input, _ := hex.DecodeString(v.InputHex)
    got := vlp.CRC32C(input)
    want, _ := strconv.ParseUint(v.ExpectedCRCHex, 16, 32)
    if uint64(got) != want { /* fail */ }
}

Full runner: tools/reference-implementations/go/verify_vectors.go.

2.3 C99 (libsodium or hand-rolled)

The C reference at tools/reference-implementations/c/ uses a minimal hand-rolled JSON tokenizer and libsodium for AEAD. Run:

cd tools/reference-implementations/c
make verify
./verify_vectors ../../vlp-test-vectors.json

2.4 Shell quick-spot via jq

# Spot-check one CRC vector:
jq -r '.crc32c_vectors[] | select(.id=="crc-rfc3720") | .input_hex' \
   tools/vlp-test-vectors.json \
| xxd -r -p \
| your-crc-tool

3. Canonical Vectors (Inline)

For readers who want to confirm the spec without fetching the JSON, the canonical vectors are reproduced here.

3.1 CRC-32C

3.2 Base frames — encode/decode round-trips

3.3 Base frames — decode errors

(Full wire bytes for the truncated rows live in the JSON file.)

3.4 Secure-frame goldens — summary

Full bytes live in tools/vlp-test-vectors.json; the structural summary:

The HKDF goldens published in VLP — Secure Transport §6.4 are sufficient to confirm an implementation’s HKDF info-string assembly is byte-correct.

4. What “Conformant” Means

An implementation is conformant if and only if:

1. It encodes every frame_vectors[].inputs to bytes equal to expected_wire_hex.
2. It decodes every expected_wire_hex back to fields equal to the corresponding inputs.
3. It rejects every decode_error entry with the named error variant.
4. For implementations that include secure transport: it reproduces every secure_frame_vectors[] byte-for-byte, including the HKDF derivations.

There is no partial conformance — BadMagic returned where BadCrc was expected is a non-conforming bug, even if the frame is correctly rejected.

5. Re-generating the file

The JSON file is checked into the repository so that:

• External implementers can clone the repo and run their conformance suite without any Rust toolchain.
• CI catches accidental wire-format drift via cargo test -p varta-vlp --features "std crypto" --test conformance_vectors.

The generator that emits the file is crates/varta-vlp/examples/gen_test_vectors.rs. After any change to Frame::encode, the CRC table, the AEAD parameters, or the HKDF info strings, regenerate the file:

cargo run -p varta-vlp --example gen_test_vectors --features "std crypto"
cargo test -p varta-vlp --features "std crypto" --test conformance_vectors

and commit the resulting JSON alongside the source change.

See also

• VLP — Base Frame
• VLP — Secure Transport

Go client

The Go client (go get github.com/aramirez087/Varta/clients/go) is a first-class peer of the Rust varta-client crate. It tracks the same wire-format contract, passes the same tools/vlp-test-vectors.json conformance suite, and interoperates with the same varta-watch observer binary.

Install

go get github.com/aramirez087/Varta/clients/go

Requires Go 1.21+. The base module (UDS + plaintext UDP) has no registry dependencies. ConnectSecureUDP adds golang.org/x/crypto.

20-line example

package main

import (
    "log"
    "time"

    varta "github.com/aramirez087/Varta/clients/go"
)

func main() {
    // Connect once. path must match --socket on your observer.
    agent, err := varta.Connect("/run/varta/varta.sock")
    if err != nil {
        log.Fatal(err)
    }
    defer agent.Close()

    for {
        if outcome := agent.Beat(varta.StatusOK, 0); outcome.IsDropped() {
            // Four-way taxonomy mirrors the Rust client:
            log.Printf("varta: dropped (%s)", outcome.DropReason())
        }
        time.Sleep(500 * time.Millisecond)
    }
}

For payload encoding, fork-safety, the panic-handler subpackage, the full transport comparison, and the complete API parity matrix see the package README: clients/go/README.md.

Transports

Stability

• Wire format: VLP v0.2, governed by the spec.
• Go API: independent semver, tracked in clients/go/CHANGELOG.md.

Source

• pkg.go.dev reference
• GitHub source
• Issue tracker

Python client

The Python client (pip install varta) is a first-class peer of the Rust varta-client crate. It tracks the same wire-format contract, passes the same tools/vlp-test-vectors.json conformance suite, and interoperates with the same varta-watch observer binary.

Install

pip install varta                # base client (stdlib only)
pip install 'varta[secure]'      # adds secure-UDP via `cryptography`

Requires Python 3.8+. Runs on Linux and macOS. The base install carries zero third-party dependencies; the secure extra pulls in cryptography for the ChaCha20-Poly1305 AEAD primitive.

20-line example

import time
from varta import Varta, Status, DropReason

with Varta.connect("/run/varta/varta.sock") as agent:
    while True:
        outcome = agent.beat(Status.OK)
        if outcome.is_dropped:
            # Four-way taxonomy mirrors the Rust client:
            assert outcome.reason in {
                DropReason.KERNEL_QUEUE_FULL,
                DropReason.NO_OBSERVER,
                DropReason.PEER_GONE,
                DropReason.STORAGE_FULL,
            }
        time.sleep(0.5)

For UDP and secure-UDP transports, fork-safety, the panic-hook family, and the full parity matrix see the package README in the repo: clients/python/README.md.

Transports

Stability

• Wire format: VLP v0.2, governed by the spec.
• Python API: independent semver, tracked in clients/python/CHANGELOG.md.

Source

• PyPI page
• GitHub source
• Issue tracker

Node.js client

The Node.js client (npm install @varta-health/client) is a first-class peer of the Rust varta-client crate. It tracks the same wire-format contract, passes the same tools/vlp-test-vectors.json conformance suite, and interoperates with the same varta-watch observer binary.

Install

npm install @varta-health/client

Requires Node.js 18 LTS or newer. ESM-only. Ships compiled JavaScript plus TypeScript declarations. The only registry dependency is an optional native addon (node-unix-socket) for the UDS transport; UDP and secure-UDP work without it. ChaCha20-Poly1305 and HKDF-SHA256 come from Node’s built-in node:crypto.

20-line example

import { Varta, Status, DropReason } from "@varta-health/client";

const agent = Varta.connectUds("/var/run/varta.sock");
setInterval(() => {
  const outcome = agent.beat(Status.Ok);
  if (outcome.kind === "dropped") {
    // Four-way taxonomy mirrors the Rust client:
    const _: DropReason = outcome.reason;
  } else if (outcome.kind === "failed") {
    console.error("varta beat failed:", outcome.error);
  }
}, 500);

For secure-UDP, fork-safety, the panic-hook family, and the full parity matrix see the package README in the repo: clients/node/README.md.

Transports

Stability

• Wire format: VLP v0.2, governed by the spec.
• Node API: independent semver, tracked in clients/node/CHANGELOG.md.

Source

• npm page
• GitHub source
• Issue tracker

.NET client

The .NET client (dotnet add package Varta.Client) is a first-class peer of the Rust varta-client crate. It tracks the same wire-format contract, passes the same tools/vlp-test-vectors.json conformance suite, and interoperates with the same varta-watch observer binary.

Install

dotnet add package Varta.Client

Targets net8.0 (LTS) and net10.0 (LTS). Pure managed code — ChaCha20-Poly1305 (System.Security.Cryptography.ChaCha20Poly1305), HKDF-SHA256 (System.Security.Cryptography.HKDF), and POSIX signals (System.Runtime.InteropServices.PosixSignalRegistration) all come from the BCL. Zero native dependencies.

20-line example

using Varta;

using var agent = global::Varta.Varta.Connect("/run/varta/observer.sock");

while (true)
{
    BeatOutcome outcome = agent.Beat(Status.Ok, payload: 0);
    if (outcome.IsDropped)
    {
        // Four-way taxonomy mirrors the Rust client:
        DropReason _ = outcome.Reason;
    }
    else if (outcome.IsFailed)
    {
        Console.Error.WriteLine($"varta beat failed: {outcome.Error}");
    }
    await Task.Delay(500);
}

For secure UDP, fork-safety, the signal-handler family, and the full parity matrix see the package README in the repo: clients/dotnet/README.md.

Transports

Stability

• Wire format: VLP v0.2, governed by the spec.
• .NET API: independent semver, tracked in clients/dotnet/CHANGELOG.md.

Source

• NuGet page
• GitHub source
• Issue tracker

JVM (Java) client

The JVM client (implementation("health.varta:varta-client:0.2.0")) is a first-class peer of the Rust varta-client crate. It tracks the same wire-format contract, passes the same tools/vlp-test-vectors.json conformance suite, and interoperates with the same varta-watch observer binary.

Install

// build.gradle.kts
dependencies {
    implementation("health.varta:varta-client:0.2.0")
    // UDS transport requires a SOCK_DGRAM AF_UNIX provider.
    runtimeOnly("com.kohlschutter.junixsocket:junixsocket-core:2.10.1")
}

Targets JDK 17 LTS minimum (Spring Boot 3.x baseline). The base jar has zero runtime dependencies — ChaCha20-Poly1305 comes from standard JCE (Cipher.getInstance("ChaCha20-Poly1305"), JDK 11+), HKDF-SHA256 is implemented in-process via javax.crypto.Mac, CRC-32C uses the hardware-accelerated java.util.zip.CRC32C (JDK 9+).

No shipping JDK exposes SOCK_DGRAM AF_UNIX in standard NIO, so the client probes the classpath at connect() time and uses either junixsocket (recommended) or a future zero-dep FFM module on JDK 22+.

20-line example

import health.varta.*;
import java.nio.file.Path;

try (Varta agent = Varta.connect(Path.of("/run/varta/observer.sock"))) {
    while (true) {
        BeatOutcome outcome = agent.beat(Status.OK, 0);
        if (outcome instanceof BeatOutcome.Dropped d) {
            // Four-way taxonomy mirrors the Rust client:
            DropReason _ = d.reason();
        } else if (outcome instanceof BeatOutcome.Failed f) {
            System.err.println("varta beat failed: " + f.error());
        }
        Thread.sleep(500);
    }
}

For secure UDP, fork-safety, the signal-handler family, and the full parity matrix see the package README in the repo: clients/java/README.md.

Transports

Stability

• Wire format: VLP v0.2, governed by the spec.
• JVM API: independent semver, tracked in clients/java/CHANGELOG.md.

Source

• Maven Central page
• GitHub source
• Issue tracker

Container Image

Registry

ghcr.io/aramirez087/varta-watch:<version>
ghcr.io/aramirez087/varta-watch:<major>.<minor>
ghcr.io/aramirez087/varta-watch:latest        # discouraged in production

Multi-arch: linux/amd64 + linux/arm64. Same digest for both — the image index resolves at pull time.

What’s inside

• Base: gcr.io/distroless/static-debian12:nonroot (pinned by digest; bumped via Renovate alongside the matching :debug-nonroot tag).
• Binary: varta-watch built with prometheus-exporter + json-log features. Class-A (compile-time-config) builds are deliberately not published as the public image — see Safety Profiles.
• User: nonroot (UID 65532, matches the Helm chart and the example DaemonSet).
• No shell, no apt, no busybox in the default tag. A :debug-<ver> tag with the busybox shell is published separately for triage.

Running

docker run -d --name varta-watch \
  --user 65532:65532 \
  --restart unless-stopped \
  --read-only \
  --tmpfs /tmp \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  -v /run/varta:/run/varta \
  -v /etc/varta/prom.token:/etc/varta/prom.token:ro \
  -p 127.0.0.1:9100:9100 \
  ghcr.io/aramirez087/varta-watch:0.3.0 \
  --socket=/run/varta/varta.sock \
  --prom-addr=0.0.0.0:9100 \
  --prom-token-file=/etc/varta/prom.token \
  --self-watchdog-secs=4

--self-watchdog-secs 4 is useful even without systemd — the in-process watchdog still aborts on wedge, and --restart unless-stopped brings the container back up.

Verifying the image

cosign verify ghcr.io/aramirez087/varta-watch:0.3.0 \
  --certificate-identity-regexp '^https://github.com/aramirez087/Varta' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com'

CycloneDX SBOM is attached via cosign attest --type cyclonedx:

cosign verify-attestation ghcr.io/aramirez087/varta-watch:0.3.0 \
  --type cyclonedx \
  --certificate-identity-regexp '^https://github.com/aramirez087/Varta' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  | jq -r '.payload | @base64d | fromjson | .predicate'

SLSA L3 build provenance is registered with GitHub:

gh attestation verify --repo aramirez087/Varta \
  oci://ghcr.io/aramirez087/varta-watch:0.3.0

Image labels

Every image carries the OCI standard label set:

org.opencontainers.image.title         = varta-watch
org.opencontainers.image.description   = Varta observer — receives VLP frames and surfaces stalls.
org.opencontainers.image.source        = https://github.com/aramirez087/Varta
org.opencontainers.image.documentation = https://varta.sh/book/operations/container.html
org.opencontainers.image.vendor        = Varta
org.opencontainers.image.licenses      = MIT OR Apache-2.0
org.opencontainers.image.version       = <tag>
org.opencontainers.image.revision      = <git sha>

Building locally

The source Dockerfile lives at the repo root and uses BuildKit’s $TARGETPLATFORM cross-compile pattern:

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  --tag varta-watch:local \
  --load .          # --push for registry publishing

CI smokes this on every PR (docker-build job in .github/workflows/ci.yml) so a typo in the Dockerfile fails fast instead of breaking adopters at the next tagged release.

Helm Chart

The chart at charts/varta-watch/ is published as an OCI artifact at oci://ghcr.io/aramirez087/charts/varta-watch. Default render matches the key observer-container fields from the raw manifests at observability/examples/kubernetes/ (a helm-parity CI gate fails the build on drift).

Install

helm install varta-watch \
  oci://ghcr.io/aramirez087/charts/varta-watch \
  --version 0.1.1 \
  --create-namespace \
  --namespace varta \
  --set prometheusToken.token=$(openssl rand -hex 32)

For production, source the token from your secret manager (SOPS, ESO, Vault Secrets Operator) and reference an existing Secret instead:

prometheusToken:
  existingSecret:
    name: varta-prom-token   # provisioned out-of-band
    key: token

Deployment modes

Switch via --set mode=…:

The chart resolves the UDS volume type per mode (host path vs. emptyDir) and emits exactly one of the two object kinds.

Each workload pod includes a uds-permissions init container that prepares the socket parent before varta-watch binds. It makes the directory owned by the configured observer UID/GID and mode 0755, so Kubernetes fsGroup handling cannot leave /run/varta in a group-writable state that the observer rejects.

Values reference

The full reference is the chart’s own values.yaml. Most-touched knobs:

Helm test

helm test varta-watch -n varta

Runs an in-cluster Pod that scrapes /metrics with the bearer token and asserts the observer is emitting varta_iterations_total. The pod is cleaned up after success (helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded).

Upgrading

helm upgrade varta-watch \
  oci://ghcr.io/aramirez087/charts/varta-watch \
  --version 0.2.0 \
  -n varta \
  --reuse-values

The chart follows SemVer independently from the varta-watch app version. Breaking template changes (e.g. renaming a values key) bump the chart’s major; bumping just the appVersion (a new binary release) bumps the chart patch.

Verifying the chart artifact

cosign verify oci://ghcr.io/aramirez087/charts/varta-watch:0.1.1 \
  --certificate-identity-regexp '^https://github.com/aramirez087/Varta' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com'

Loading the dashboard under kube-prometheus

dashboard.enabled: true ships a ConfigMap with the Grafana sidecar label (default grafana_dashboard: "1") — the kube-prometheus-stack Grafana sidecar auto-imports it. If you run a custom Grafana, point its sidecar at the chart’s namespace or override dashboard.label / dashboard.labelValue / dashboard.namespace.

Migrating from raw manifests

Adopters who previously used observability/examples/kubernetes/ can switch to the chart with no operational disruption: the rendered default keeps the observer container image repository, args, and mounts in sync with the raw manifest. The CI helm-parity job asserts this on every PR. Expected differences include Helm-standard labels (helm.sh/chart, app.kubernetes.io/managed-by) and chart-managed init containers for token staging and UDS directory preparation.

Deployment Patterns

Concrete recipes for the three most common varta-watch deployment targets. All three files referenced live under observability/examples/ and are CI-linted on every push.

Looking for the one-paste install paths? See Install (Quickstart) for curl | sh, cargo binstall, Helm, and Docker one-liners. This page is the reference on the underlying recipes those paths assemble.

Pre-flight: the bearer token

varta-watch requires --prom-token-file whenever --prom-addr is set. Generate the token once per host:

install -d -m 0750 -o varta -g varta /etc/varta
openssl rand -hex 32 | install -m 0400 -o varta -g varta /dev/stdin /etc/varta/prom.token

Anything that can read /etc/varta/prom.token can scrape /metrics. Mirror the token to the Prometheus scraper via your secret manager (Vault, SOPS, External Secrets Operator, etc.).

systemd (bare metal / VM)

Drop observability/examples/varta-watch.service into /etc/systemd/system/varta-watch.service, then:

useradd --system --no-create-home --shell /usr/sbin/nologin varta
systemctl daemon-reload
systemctl enable --now varta-watch
systemctl status varta-watch

Key bindings the unit gets for free:

• Type=notify + WatchdogSec=8s — systemd will SIGABRT-then-SIGKILL the process if WATCHDOG=1 doesn’t arrive every 4 seconds (half of WatchdogSec).
• --self-watchdog-secs 4 (passed in ExecStart) — auto-enables when $WATCHDOG_USEC is set; spawns the in-process watchdog thread that emits WATCHDOG=1 and calls process::abort() on wedge before systemd has to intervene.
• Hardening flags (ProtectSystem, NoNewPrivileges, MemoryDenyWriteExecute, etc.) — defence in depth; safe to leave on even if your kernel ignores some of them.

Validate the unit syntactically on Linux:

systemd-analyze verify observability/examples/varta-watch.service

Docker

docker run -d --name varta-watch \
  --user 65532:65532 \
  --restart unless-stopped \
  --read-only \
  --tmpfs /tmp \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  -v /run/varta:/run/varta \
  -v /etc/varta/prom.token:/etc/varta/prom.token:ro \
  -p 127.0.0.1:9100:9100 \
  ghcr.io/aramirez087/varta-watch:0.3.0 \
  --socket=/run/varta/varta.sock \
  --prom-addr=0.0.0.0:9100 \
  --prom-token-file=/etc/varta/prom.token \
  --self-watchdog-secs=4

Verify the image before pulling it into production:

cosign verify ghcr.io/aramirez087/varta-watch:0.3.0 \
  --certificate-identity-regexp '^https://github.com/aramirez087/Varta' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com'

The --self-watchdog-secs 4 flag stays useful even without systemd — the in-process watchdog still aborts on wedge, and Docker’s --restart unless-stopped brings the container back up.

Kubernetes (kube-prometheus)

The supported path is the Helm chart; the raw manifests remain for adopters who don’t want Helm and are CI-asserted to match the chart’s default render.

# Helm (recommended)
helm install varta-watch \
  oci://ghcr.io/aramirez087/charts/varta-watch \
  --version 0.2.0 \
  --namespace varta --create-namespace \
  --set prometheusToken.token=$(openssl rand -hex 32)

# Or raw manifests
kubectl apply -f observability/examples/kubernetes/varta-watch.deployment.yaml
kubectl apply -f observability/examples/kubernetes/varta-watch.servicemonitor.yaml

Two patterns supported by the bundle:

1. Per-node DaemonSet (default): one observer per node, agents share the UDS via hostPath:/run/varta. Easiest fit for existing workloads not deployed as pods of their own.
2. Sidecar: replace the DaemonSet with a Deployment and mount emptyDir: {} at /run/varta. Only agents inside the same pod can reach the observer; useful for strict isolation (one Varta per tenant).

ServiceMonitor vs. PodMonitor:

• Use ServiceMonitor (default) when you keep the headless Service. Discovery is via Service endpoints; Prometheus retrieves the pod list from the API server.
• Use PodMonitor (alternative manifest provided) when you remove the Service for strict-sidecar deployments. Discovery is direct pod enumeration.

The release: label on the CRD must match your Prometheus CR’s serviceMonitorSelector. The kube-prometheus-stack Helm chart defaults to release: <chart-release-name>; adjust accordingly.

Loading the dashboard under kube-prometheus

Two options:

1. ConfigMap with the sidecar label (recommended — Grafana auto-imports):

   kubectl create configmap varta-grafana-dashboard \
     --from-file=observability/dashboards/varta-health.json \
     -n monitoring
   kubectl label configmap varta-grafana-dashboard \
     grafana_dashboard="1" \
     -n monitoring
   

2. grafanaDashboards field on a GrafanaDashboard CR if you run the grafana-operator.

Class-A safety-critical builds

The compile-time-config Cargo feature structurally excises the Prometheus exporter from the binary (see Compile-Time Configuration and Safety Profiles). If you’re deploying a Class-A profile:

• The deployment recipes on this page do not apply — --prom-* flags are not recognised by the binary, and the CI strings audit rejects GET /metrics literals in the artifact.
• Use the file exporter (--export-file <path>) instead. The TSV schema is documented in crates/varta-watch/README.md.
• For audit-log integrity, treat the on-disk audit log as the source of truth; there is no /metrics endpoint to scrape.

Verifying a deployment

Sanity checks for any of the three deployment patterns:

# 1. Token authenticates.
curl -sS -H "Authorization: Bearer $(cat /etc/varta/prom.token)" \
  http://127.0.0.1:9100/metrics | head

# 2. Bad token rejected.
curl -i -H "Authorization: Bearer not-the-token" \
  http://127.0.0.1:9100/metrics
# Expect: HTTP/1.0 401 Unauthorized

# 3. Stable label set present (alert rules depend on it).
curl -sS -H "Authorization: Bearer $(cat /etc/varta/prom.token)" \
  http://127.0.0.1:9100/metrics \
  | grep -E '^varta_rate_limited_total\{reason='
# Expect: both reason="per_pid" and reason="global" lines, value 0

Monitoring & Alerting

Quick walkthrough: Prometheus setup guide

varta-watch ships with a turn-key Prometheus + Grafana + Alertmanager bundle. The artefacts live under observability/ in the repo; this chapter is the operator-facing prose tying them together.

If you just want files-on-disk, the bundle README is at observability/README.md. Come back here for the why behind each alert and dashboard panel.

10-minute setup

1. Run varta-watch with the prometheus-exporter feature enabled and a bearer token. The token file is mandatory whenever --prom-addr is set:

   openssl rand -hex 32 > /etc/varta/prom.token
   chmod 0400 /etc/varta/prom.token
   varta-watch \
     --socket /run/varta/varta.sock \
     --prom-addr 127.0.0.1:9100 \
     --prom-token-file /etc/varta/prom.token \
     --self-watchdog-secs 4
   

2. Copy the rule files into your Prometheus config directory and reload:

   cp observability/recording-rules/varta.rules.yml /etc/prometheus/rules.d/
   cp observability/alerts/varta.rules.yml          /etc/prometheus/rules.d/
   curl -X POST http://localhost:9090/-/reload
   

3. Paste observability/examples/prometheus-scrape.yml’s scrape_configs: block into your prometheus.yml, point credentials_file at the token from step 1, and reload again.

4. Import observability/dashboards/varta-health.json in Grafana, selecting the Prometheus datasource on the import dialog.

5. Wire observability/examples/alertmanager.yml into your Alertmanager. Replace the PagerDuty key and Slack webhook before reloading.

For Kubernetes / kube-prometheus operators, use the CRDs in observability/examples/kubernetes/ instead of editing prometheus.yml directly. See Deployment Patterns for full systemd / Docker / K8s recipes.

Severity model

Every Varta alert carries a severity label. Three values, three routes:

info alerts are not actionable per-event; they are the trend signal that earlier action is needed (auth-failure clustering ⇒ rotate token, namespace-conflict spikes ⇒ review --allow-cross-namespace-agents policy).

Metrics by subsystem

Every varta-watch metric is in one of nine subsystems. Stable label sets are emitted from the first scrape (every label value present at zero), so by (label) queries and absent() rules are safe day-one.

Beat path (5 metrics)

Decode / authentication (5 metrics)

Tracker / capacity (8 metrics)

Observer liveness (5 metrics)

Recovery (11 metrics)

Audit log (6 metrics)

Scrape (8 metrics)

Secure-UDP (4 metrics)

Observer metadata (5 metrics)

Alert catalogue

The complete rule file is observability/alerts/varta.rules.yml. Below is one line per alert linking the alert name (and its anchor) to the metric and the operator action. Section anchors are stable: every alert in the YAML references #<lowercased alertname> in its annotations.runbook_url.

Critical (page on-call)

VartaWatchStalled

• Metric: varta_watch_uptime_seconds
• Trigger: uptime gauge not advancing for 1 minute.
• Why critical: stall detection and recovery are not happening on this host; agents that go silent will not get killed-and-respawned.
• Action: check the host (is the process alive? kill -0 $(pidof varta-watch)); inspect kernel ring buffer for OOM kills; inspect the systemd journal for process::abort() from the self-watchdog thread; restart the unit if needed. The journal will name the stage that wedged.

VartaTrackerCapacityExceeded

• Metric: varta_tracker_capacity_exceeded_total
• Trigger: any non-zero increment over 5m.
• Why critical: new agents are not being added to the tracker — their beats are decoded and counted, but never reach the stall detector.
• Action: shard the deployment (see Deployment Ceiling & Sharding) or raise --tracker-capacity. Check varta_tracker_evicted_total rate first — if eviction is healthy and you’re still hitting the cap, sharding is mandatory.

VartaOutstandingProbeExhausted

• Metric: varta_recovery_outstanding_probe_exhausted_total
• Trigger: any non-zero increment over 5m.
• Why critical: the bounded OutstandingTable hit its probe budget; new recoveries cannot be tracked.
• Action: as above — shard or raise capacity.

VartaPidIndexProbeExhausted

• Metric: varta_tracker_pid_index_probe_exhausted_total
• Trigger: any non-zero increment over 5m.
• Why critical: the open-addressed PID hash blew its 64-probe budget.
• Action: as above. This usually fires before VartaTrackerCapacityExceeded on Linux hosts with high pid_max.

VartaAuditRecordDropped

• Metric: varta_recovery_audit_dropped_total
• Trigger: any non-zero increase over 5m.
• Why critical: for IEC 62304 Class C, every recovery decision must be auditable. A drop means at least one decision is unrecorded.
• Action: check disk latency, audit ring watermark history (varta_audit_ring_watermark_total), and the audit-flush budget; raise --audit-fsync-budget-ms if the disk is slow but still healthy.

VartaInvariantViolation

• Metrics: varta_tracker_invariant_violations_total, varta_recovery_invariant_violations_total
• Trigger: any non-zero rate over 5m on either metric.
• Why critical: this is the DO-178C “no unproven panics” counter — an unreachable defensive fall-through executed.
• Action: file a bug. Attach the metric values, the varta-watch version, and any nearby stage-budget alarms.

VartaIterationP99High

• Metric: varta_observer_iteration_seconds histogram
• Trigger: p99 over 5m exceeds 500 ms.
• Why critical: stall-detection latency is being burned.
• Action: inspect varta_observer_stage_seconds to find which phase contributes (likely serve_pending if it’s a scrape storm, or maintenance if it’s an audit-fsync stall).

VartaBeatPathP99High

• Recording rule: varta:beat_path_seconds:p99_5m
• Trigger: beat-path p99 (iteration − scrape) exceeds 200 ms.
• Why critical: the beat path itself is slow, independent of scrape pressure. Scrape-storm alarms route off varta_observer_serve_pending_seconds; this one is unaffected.
• Action: check audit fsync p99 first (varta:audit_fsync_seconds:p99_5m), then disk latency, then tracker eviction window.

Warning (investigate within working day)

Info (record for trend analysis)

Dashboard tour

The single dashboard varta-health.json has six rows. Each row maps 1:1 onto a subsystem; alerts you receive route back to the relevant row via the runbook_url annotation.

See also

• SLOs & Tuning — recommended SLO model and threshold tuning.
• Deployment Patterns — systemd / Docker / Kubernetes recipes.
• Stall Detection & Liveness — the rationale behind the iteration-budget mechanism.
• Deployment Ceiling & Sharding — what to do when the tracker capacity alarms fire.
• Audit Logging — the durability model behind VartaAuditRecordDropped and the ring watermark counters.

SLOs & Tuning

Service-level objectives for varta-watch. These are starting points; tune to your latency budget and audit-durability needs.

Recommended SLOs

SLO-1 — Beat-path latency

99% of poll-loop iterations complete within 100 ms, measured over a 30-day rolling window.

• SLI: varta:iteration_seconds:p99_5m (recording rule)
• Error budget: 1% of 30 days = ~7.2 h above 100 ms.
• Polices it: VartaIterationBudgetOverruns (warn at 10% of iterations >budget), VartaIterationP99High (page at p99 >500 ms).
• Why 100 ms: the documented worst-case iteration is ~310 ms (see Observer Liveness latency-budget table). 100 ms is well within that bound and leaves headroom for occasional fsync stalls.

SLO-2 — Recovery responsiveness

99% of recoveries fire within (stall_threshold + debounce_window + 100 ms) of stall detection, measured over a 7-day rolling window.

• SLI: varta_recovery_outcomes_total{outcome="spawned"} rate vs. varta_stalls_total rate, joined on pid.
• Error budget: 1% of 7 days = ~1.7 h of mis-budgeted recoveries.
• Polices it: VartaRecoveryReapTruncated (queue backlog), VartaBeatPathP99High (stall detection itself is slow).
• Why this matters: the whole product is the chain “agent stops → stall detected → recovery fires”. If recovery responsiveness regresses but the individual stages look fine, the queue-shape (debounce + outstanding tables) is the culprit.

SLO-3 — Audit durability

Zero varta_recovery_audit_dropped_total increments over any 7-day rolling window.

• SLI: increase(varta_recovery_audit_dropped_total[7d])
• Error budget: zero. This is a hard SLO for IEC 62304 Class C deployments; for other deployments, treat as warning.
• Polices it: VartaAuditRecordDropped (page on any non-zero), VartaAuditRingWatermarkCritical (warn at 95% fill).
• Why this matters: the audit log is the regulatory record. Any drop is a hole in the recovery decision trail.

SLO-4 — Scrape availability

99.9% of Prometheus scrapes complete within scrape_timeout (default 10s), measured over a 7-day rolling window.

• SLI: scraper-side up{job="varta-watch"} == 1 over total scrapes.
• Error budget: 0.1% of 7 days = ~10 minutes.
• Polices it: VartaScrapeStormPressure (warn at >10% serve overruns).

Tuning matrix

When an SLO is at risk, the dial that affects it most:

Sizing examples

50-agent single host

• --tracker-capacity 64 (default 256 is fine but tighter is cheaper).
• --max-beat-rate 100 (default).
• --scrape-budget-ms 250 (default).
• Single Prometheus scraping at 15s.

Iteration p99 ≤ 50 ms typical, p99.9 ≤ 200 ms under audit-flush bursts.

1000-agent shard

• --tracker-capacity 2048.
• --eviction-scan-window 512.
• --max-beat-rate 200 if your agents beat aggressively.
• --audit-fsync-budget-ms 100 (twice the default to absorb disk hiccups).
• Two-host HA scrape: stagger the two Prometheus scrapers by 7.5s so the per-IP rate-limit table doesn’t see them as a coordinated burst; raise --prom-rate-limit-burst to 20 as defence in depth.

4096-agent ceiling

This is the documented maximum per observer (deployment ceiling). Beyond this, shard. Two 2048-agent observers on the same host with separate UDS paths is cheaper than one tuned-up 4096-agent observer, because every linear scan (eviction_scan_window) and every probe budget (PidIndex) is duplicated rather than expanded.

Trend signals

Three info-tier alerts are not alerts in the operational sense — they are SLO erosion signals you should be watching in a dashboard, not paging on:

• VartaAuthFailureBurst — bearer-token scanning is happening. Rotate.
• VartaNamespaceConflict — agent placement is leaking into wrong PID namespaces; review the deployer.
• VartaFrameDecodeAnomaly{kind="bad_version"} — client/observer version skew on the wire format. Coordinate the upgrade.

The dashboard’s “Security & integrity” row surfaces all three.

Troubleshooting

Runtime issues after varta-watch is installed. For install-time issues (cosign, systemd unit, container start) see Install (Quickstart) → Troubleshooting.

The fastest debug surface is /metrics. If you can scrape it, varta-watch is alive and the metric values pinpoint the layer at fault. Every diagnostic below names the metric to check first.

My agent reports Sent but varta_beats_total{pid=…} stays at 0

The observer is rejecting the beat before counting it. Walk these in order:

1. varta_decode_errors_total — any kind is nonzero ⇒ wire-format problem. Most common: bad_version (agent on a different VLP release) or bad_magic (something other than a Varta agent is writing to the socket).

2. varta_frame_auth_failures_total — nonzero ⇒ peer-cred PID disagrees with the frame’s claimed PID. Either an actual spoof attempt or the agent is calling beat() from a thread/process whose PID differs from what was wired in. Fork without re-connect() is the usual culprit (v0.2.0+ auto-detects this; older clients do not).

3. varta_truncated_datagrams_total — nonzero ⇒ kernel handed the observer a wrong-sized datagram. UDS frames must be exactly 32 bytes; secure-UDP frames must be exactly 60 (shared-key) or 64 (master-key).

4. varta_tracker_capacity_exceeded_total — nonzero ⇒ the tracker is at --tracker-capacity (default 4096) and is dropping beats from new pids. Raise the cap or shard (see Deployment Ceiling & Sharding).

5. varta_rate_limited_total{reason="per_pid"|"global"} — nonzero ⇒ rate limit kicked in. Defaults are 100/s per pid and 5000/s globally; raise or disable as appropriate (see Upgrade Guide).

6. Socket path mismatch. The observer logs the path it bound to at startup. Confirm the agent’s Varta::connect() argument is byte- identical, including any container mount remapping.

varta_status{pid=…} shows 3 (Stall) but my process is healthy

The observer hasn’t received a beat from this PID in --threshold-ms (default depends on your config). Causes:

• Agent is blocked on something synchronous that doesn’t yield to the beat loop. The fix is on the agent — Varta is reporting the truth.
• Beat thread crashed silently. If the agent uses the panic-handler feature, a terminal Critical beat should have been emitted. Check varta_status history; if it skipped from 0 to 3 without 2, the panic-handler is missing or the process hard-died (segfault, kill -9, OOM).
• Threshold is too tight for your cadence. If your agent beats every 1 s but --threshold-ms 500, every minor scheduling hiccup trips a stall.

Recovery never fires even though stalls are surfaced

In order of likelihood:

1. No recovery command configured. Recovery is opt-in. Check the observer was started with --recovery-exec or --recovery-exec-file.

2. varta_recovery_refused_total{reason="…"} — non-zero on one of these labels:

   Label	What to do
   unauthenticated_transport	Beat origin is NetworkUnverified (plain UDP). Recovery requires kernel-attested origin. Use UDS, or secure-UDP with master-key.
   socket_mode_only	Platform without per-datagram peer-creds (e.g. OpenBSD). No fix — recovery is structurally refused here.
   cross_namespace_agent	PID namespace mismatch. Set --allow-cross-namespace-agents if you trust the source, or fix the deployment (Docker --pid=host, k8s hostPID: true).
   debounce_capacity	The per-pid debounce window suppressed a duplicate spawn. Expected.
   outstanding_capacity	The OutstandingTable is full (≥ --tracker-capacity). A previous recovery is still running for every slot. Investigate why children are not exiting.

3. varta_recovery_outcomes_total{outcome="spawn_failed"} is nonzero. Inspect observer stderr for the actual io::Error. Most common: ENOENT (program path wrong, or PATH doesn’t include it under the isolated env), EACCES (file not executable / wrong owner).

/metrics returns 401 even with the right token

• Token mismatch. Check the file the observer reads (--prom-token-file) and the value you’re sending byte-for-byte (xxd both; a trailing newline counts).
• Token file mode. The observer refuses to load tokens from a file with mode broader than 0400 / 0600 and owner other than the observer UID.
• Bearer header form. The exporter accepts Authorization: Bearer <hex> only. Token <hex>, Basic, query-string credentials are all rejected as 401.

/metrics is slow or times out under high pid count

• varta_observer_serve_pending_seconds — bimodal latency ⇒ the serve-pending stage is consuming budget. Raise --scrape-budget-ms or scrape less often.
• varta_scrape_budget_exhausted_total — incrementing ⇒ scrapes arriving faster than the observer can serve. The exporter falls back to a cached response (varta_scrape_skipped_total increments). This is a graceful degradation, not a bug.
• varta_prom_connections_dropped_total{reason="ip_table_full"} — the per-IP rate-limit table is full. Tune --prom-rate-limit-per-sec and --prom-rate-limit-burst.

varta_watch_uptime_seconds is frozen

The self-watchdog hasn’t kicked. Either the observer is wedged in a syscall or the metric is being read from a cached scrape. Order of investigation:

1. ps — is the process actually running?
2. kill -USR1 <pid> — observer logs current iteration state to stderr (if the signal handler is installed; default-on for direct mode, opt-out via --signal-handler-mode disabled).
3. If --hw-watchdog was passed, the kernel watchdog will reboot the host before this gauge can stay frozen long. Check dmesg for watchdog: BUG traces.
4. If --self-watchdog-secs was passed, the in-process watchdog SIGABRTs the process before systemd restarts it. Check journalctl -u varta-watch for the abort line.

Namespace conflict on every beat in a container

varta_tracker_namespace_conflict_total is incrementing and beats are refused. The observer container’s PID namespace differs from the agent container’s. Fixes:

• Docker / podman: run the observer with --pid=host.
• Kubernetes: set hostPID: true on the observer Pod, OR colocate observer + agents in the same Pod with shared shareProcessNamespace: true.
• Explicit override: pass --allow-cross-namespace-agents to accept beats (but recovery for those agents will still be refused unless --strict-namespace-check is left off; see Namespacing).

Where to file something the docs miss

• Wire-protocol surprises: open an issue against varta-vlp tagged protocol.
• Observer behaviour gaps: tag observer.
• Anything security-shaped: see Security for private disclosure.

Upgrade Guide

This page lists every change between Varta releases that requires operator action. For the human-readable release prose see RELEASES/; for the full diff see CHANGELOG.md.

The convention on this page: a checkbox per breaking change so you can walk the list and tick off what you’ve handled.

v0.1.x → v0.2.0

Client API

• BeatOutcome::Dropped now carries a DropReason. Match against the variant or use a _ placeholder. Variants: KernelQueueFull, NoObserver, PeerGone, StorageFull.

  #![allow(unused)]
  fn main() {
  // Before
  if let BeatOutcome::Dropped = outcome { … }
  
  // After
  if let BeatOutcome::Dropped(_) = outcome { … }
  }

Observer defaults (rate limiting)

The observer now ships with default-on rate limits. Reset all three to zero if you want the v0.1 unlimited behaviour:

• --max-beat-rate defaults to 100 (per-pid). Disable with --max-beat-rate 0.
• --global-beat-rate defaults to 5000 (process-wide). Disable with --global-beat-rate 0.
• --uds-rcvbuf-bytes defaults to 1048576 (1 MiB). Skip tuning with --uds-rcvbuf-bytes 0.

Shell-mode recovery removed

The /bin/sh recovery path is gone. Operators using shell templates must migrate to the exec path. Passing a removed flag is now a hard startup error.

• --recovery-cmd <TEMPLATE> → --recovery-exec <PROGRAM> [ARGS…]. The stalled pid is appended as the final argument.
• --recovery-cmd-file <PATH> → --recovery-exec-file <PATH>.
• --i-accept-shell-risk → remove; no replacement needed.

If your recovery logic relied on shell features (pipes, redirects), wrap them in a script and exec the script:

#!/bin/sh
# /usr/local/bin/varta-recovery
PID="$1"
systemctl restart "myapp-${PID}"

varta-watch --recovery-exec /usr/local/bin/varta-recovery …

Full migration reference and rationale: Shell-Mode Recovery Removal.

Recovery child environment is isolated by default

Pre-v0.2.0 recovery children inherited the observer’s full process environment. From v0.2.0 they see only PATH=/usr/bin:/bin plus any explicit --recovery-env KEY=VALUE entries.

This is the secure default; observer environments routinely carry credentials (AWS_*, OAuth tokens, Vault tokens) and inheriting them into a recovery child turns every recovery template into a credential exfiltration vector.

• Audit your recovery scripts for environment dependencies. If they need any inherited variable, add it explicitly:

  varta-watch --recovery-env HOME=/var/log/varta --recovery-env LANG=C …
  

• Or, as an escape hatch, restore full inheritance:

  varta-watch --recovery-inherit-env …
  

  This emits a one-shot stderr warning at startup so the choice is visible in syslog / SIEM.

Full configuration matrix: Recovery — Async Spawn → env policy.

Wire format

• The frame layout is unchanged between v0.1 and v0.2. Existing agents continue to work. Future protocol bumps (v0.3+) will be called out here.

Defaults that didn’t change but worth re-verifying

• --threshold-ms (silence detection) — unchanged.
• --recovery-debounce-ms — unchanged.
• UDS socket path conventions (/run/varta/varta.sock is the canonical recommendation; the binary takes whatever you pass via --socket).

New deployment surfaces

These are additive — adopt at your own pace:

• Container image — ghcr.io/aramirez087/varta-watch:0.3.0, cosign-signed with keyless OIDC, SLSA L3 provenance.
• Helm chart — oci://ghcr.io/aramirez087/charts/varta-watch --version 0.2.0 (the chart version is independent of the app version).
• curl | sh installer — https://varta.sh/install.sh.
• cargo binstall varta-watch.
• Python client — pip install varta.

Pick a path: Install (Quickstart).

Verifying the upgrade

After upgrading, run the standard smoke checks from Deployment Patterns → Verifying a deployment. The two most important signals:

1. varta_watch_uptime_seconds is advancing.
2. varta_beats_total{pid="<your_agent>"} is non-zero and increasing.

If either is wrong, head to Troubleshooting.

VLP Frame — Wire Layout (v0.2)

Audience: Rust contributors. This page documents the Rust implementation of the Varta Lifeline Protocol — design rationale, type-system choices, performance characteristics. If you are building a client in another language, start at the normative VLP specification — it is language-neutral and carries a published conformance-vector suite.

The Varta Lifeline Protocol carries a single message type: a 32-byte fixed-layout health frame. Every byte position is pinned at the protocol level so encode/decode is a handful of from_le_bytes / to_le_bytes calls and a single CRC-32C pass — nothing else.

Byte map

offset │ size │ field      │ notes
───────┼──────┼────────────┼──────────────────────────────────────────────
 0     │  2   │ magic      │ const [0x56, 0x41]  (ASCII "VA")
 2     │  1   │ version    │ const 0x02         (v0.1 → BadVersion)
 3     │  1   │ status     │ Status::{Ok=0, Degraded=1, Critical=2, Stall=3}
 4     │  4   │ pid        │ u32 little-endian — emitter's process id
 8     │  8   │ timestamp  │ u64 little-endian — emitter-local monotonic
16     │  8   │ nonce      │ u64 little-endian — strictly increasing
24     │  4   │ payload    │ u32 little-endian — opaque app context (v0.2)
28     │  4   │ crc32c     │ u32 LE CRC-32C over bytes 0..28        (v0.2)
───────┴──────┴────────────┴──────────────────────────────────────────────
                                                              total 32 bytes

Nonce semantics

The 8-byte nonce field at offset 16 carries two distinct kinds of value:

• Regular beats from varta_client::Varta::beat use a per-connection counter that starts at 1 on the first beat after Varta::connect and increments monotonically. On exhaustion the counter wraps at NONCE_TERMINAL - 1 → 0 — so the regular-beat stream cycles through 1, 2, 3, …, u64::MAX - 1, 0, 1, 2, … and structurally never emits NONCE_TERMINAL (== u64::MAX).
• Panic frames from varta_client::panic::install* hooks pin the nonce to NONCE_TERMINAL and the status to Status::Critical. This is the unique on-wire marker for a panic-terminated agent.

Frame::decode enforces the wire-side invariant nonce == NONCE_TERMINAL ⇒ status == Status::Critical; any other status paired with the sentinel is rejected as DecodeError::BadNonce. The Kani harness in crates/varta-vlp/src/proofs.rs proves this for every decodable byte pattern.

The converse — Status::Critical ⇒ nonce == NONCE_TERMINAL — is not enforced. Agents legitimately emit Status::Critical at regular nonce values for operational alerts (queue full, shedding load, etc.). Downstream consumers (alert rules, log dashboards, recovery filters) that need to distinguish “panic terminal” from “operational critical” must inspect both status and nonce:

Wrap to 0 is rare in practice (an agent emitting one beat per millisecond takes ~584 million years to consume u64::MAX - 1), but it is structurally correct and observable: an agent that does wrap will continue emitting without observer ambiguity, because nonce 0 paired with Critical is classified as “operational critical,” not “panic.” The Kani harness covers the wire boundary; the wrap arithmetic itself lives in varta-client and is straight-line code in Varta::beat.

v0.2 wire integrity (CRC-32C)

Bytes 28..32 carry a CRC-32C (Castagnoli, polynomial 0x1EDC6F41, init 0xFFFFFFFF, reflected, output-XOR 0xFFFFFFFF) computed over bytes 0..28. The CRC catches:

• Non-ECC RAM bit flips and cosmic-ray single-event upsets on the agent or the observer host.
• NIC firmware corruption between RX queue and userspace.
• In-process memory corruption between Frame::encode and the transport write (or between the transport read and Frame::decode), including the gap between crypto::seal / crypto::open and the frame-level codec on the secure-UDP transport. AEAD tag failures surface separately as crypto::AuthError; the CRC is the defence-in-depth catch for everything that AEAD does not (in-process corruption on either side of the seal/open boundary).

Decode order is fixed: magic → version → CRC → status → pid → timestamp → nonce. CRC verification sits between version and field-range checks so random bytes from a wrong-protocol sender still surface as BadMagic / BadVersion (preserving the “this isn’t even VLP” diagnostic) while a single-bit-flipped status byte surfaces as BadCrc, never as a valid frame with the wrong meaning.

Implementation: crates/varta-vlp/src/crc32c.rs carries a const-fn 256-entry lookup table; per-frame cost is ~28 cycles (~9 ns on Apple Silicon). Hardware CRC-32C is available on x86_64 (SSE 4.2) and ARMv8.1+ via core::arch intrinsics; a future target_feature cfg can drop the cost to ~1 cycle without changing the wire format.

The payload field shrank from u64 (v0.1) to u32 (v0.2) to make room for the CRC trailer inside the 32-byte budget. Agents needing more than 4 bytes of context should externalize the data and reference it from the payload (e.g. as a slot index into a shared ring buffer).

The two compile-time assertions in crates/varta-vlp/src/lib.rs lock this in:

#![allow(unused)]
fn main() {
const _: () = assert!(core::mem::size_of::<Frame>() == 32);
const _: () = assert!(core::mem::align_of::<Frame>() == 8);
}

A drift in field order, padding, or width breaks the build. The integration test frame_round_trip_matches_golden_bytes cross-checks a hand-computed golden byte array against Frame::encode, so the layout is also pinned at runtime.

Why #[repr(C, align(8))]

• repr(C) pins field order to declaration order. Without it the compiler is free to reorder fields, which would silently break a wire format consumed by any tool that decodes by offset (including varta-watch itself).
• align(8) makes the struct’s start address 8-byte aligned, matching the natural alignment of the three u64 fields. The first 8 bytes (magic + version + status + pid) total exactly 8 bytes, so once the struct is 8-aligned the u64 fields land on 8-byte boundaries with zero padding. size_of therefore equals the sum of the field widths (32), and the const-assert proves it.
• No unsafe is required at the encode/decode boundary because we never transmute the struct to or from [u8; 32]. The body of Frame::encode and Frame::decode is a sequence of to_le_bytes / from_le_bytes calls against fixed-length array slices, all of which are checked at the type system level.

Why little-endian on the wire

• Every tier-1 target Varta will plausibly run on (x86_64, aarch64) is little-endian natively, so to_le_bytes is a no-op copy on the hot path.
• Even on a hypothetical big-endian target the cost is one bswap-class instruction per integer field — a rounding error against UDS write/read.
• Pinning byte order in the spec means a frame captured on one host can be decoded byte-for-byte on another, which keeps the varta-watch recovery command testable in isolation.

Why zero-dependency

• The protocol crate is the foundation everything else links against. Any registry crate it pulls in (bytes, byteorder, zerocopy, …) becomes a transitive obligation for every agent that wants to integrate Varta. Keeping [dependencies] empty preserves the “drop in one path dep, get health signaling” contract.
• The whole crate is a struct, an enum, and four free functions. There is nothing here that core does not already provide.
• Empty deps also keep the audit surface minimal: the only unsafe in the workspace will live in varta-client and varta-watch (where required for UDS plumbing), never in the protocol crate itself.

Cross-references

• Acceptance contract: docs/acceptance/varta-v0-1-0.md
• Crate root: crates/varta-vlp/src/lib.rs
• Integration tests: crates/varta-vlp/tests/frame.rs

VLP Transports

Audience: Rust contributors. This page documents the Rust implementation of the VLP transport layer — trait shapes, feature-flag matrix, operational guidance. For the normative wire format of the base frame and the AEAD-wrapped secure frames, see the VLP specification and VLP secure-transport specification.

The Varta Lifeline Protocol (VLP) wire format is entirely transport-agnostic — a 32-byte, 8-byte-aligned #[repr(C)] frame. The transport layer is abstracted via traits that allow swapping out the underlying socket type without modifying the protocol core.

Architecture

graph TD
    VLP["<b>varta-vlp</b><br/>Frame (32 bytes) · Status · DecodeError<br/><i>Zero dependencies. Never changes.</i>"]

    VLP --> CLIENT
    VLP --> WATCH

    subgraph CLIENT["varta-client"]
        BT["BeatTransport"]
        BT --> UdsT["UdsTransport"]
        BT --> UdpT["UdpTransport<br/><i>(udp feat.)</i>"]
        BT --> SecT["SecureUdpTransport<br/><i>(secure-udp feat.)</i>"]
    end

    subgraph WATCH["varta-watch"]
        BL["BeatListener"]
        BL --> UdsL["UdsListener"]
        BL --> UdpL["UdpListener<br/><i>(udp feat.)</i>"]
        BL --> SecL["SecureUdpListener<br/><i>(secure-udp feat.)</i>"]
    end

Agent side (varta-client)

#![allow(unused)]
fn main() {
pub trait BeatTransport: Send + 'static {
    fn send(&mut self, buf: &[u8; 32]) -> io::Result<usize>;
    fn reconnect(&mut self) -> io::Result<()>;
}
}

Varta<T: BeatTransport> owns a transport and calls send(2) on every beat(). The default transport is UdsTransport (Unix Domain Socket). When the udp feature is enabled, UdpTransport is available via Varta::connect_udp(addr). When the secure-udp feature is enabled, SecureUdpTransport is available via Varta::connect_secure_udp(addr, key) — every beat is encrypted with ChaCha20-Poly1305 AEAD (RFC 8439).

Observer side (varta-watch)

#![allow(unused)]
fn main() {
pub trait BeatListener: Send + 'static {
    fn recv(&mut self) -> RecvResult;
    fn drain_decrypt_failures(&mut self) -> u64 { 0 }  // default = 0
    fn drain_truncated(&mut self) -> u64 { 0 }         // default = 0
}
}

The Observer holds a Vec<Box<dyn BeatListener>> and polls all listeners round-robin on each poll() call. When --udp-port is passed at the CLI, a UdpListener is added alongside the UDS listener.

Transport comparison

| | UDS (default) | UDP (feature = “udp”) | Secure UDP (feature = “secure-udp”) | |—|—|—|—|—| | Addressing | Filesystem path | IP:PORT | IP:PORT | | Encryption | None (kernel isolation) | None | ChaCha20-Poly1305 AEAD | | Authentication | Kernel PID + UID via SO_PASSCRED / SCM_CREDS where available; socket permissions only on macOS pathname UDS | None | Poly1305 tag + PID in IV prefix (master-key mode) — wire-content only, not the sending process | | Replay protection | None (local IPC) | None | Per-sender IV counter monotonicity | | Trust model | Filesystem permissions + kernel credential attestation on supported kernels | Network segmentation | 256-bit pre-shared or per-agent derived key | | Origin classification | KernelAttested on Linux / supported BSDs / illumos / Solaris; SocketModeOnly on macOS pathname UDS | NetworkUnverified | NetworkUnverified (cryptographic binding != kernel attestation) | | Recovery-eligible by default? | Yes where kernel-attested; no on socket-mode-only targets | No (see [peer-authentication.md → Recovery eligibility]) | No (same gate; even master-key derivation cannot replace kernel attestation) | | Frame size | 32 bytes | 32 bytes | 60 bytes (AEAD overhead) | | Socket cleanup | UdsListener::drop unlinks socket | Kernel reclaims port | Kernel reclaims port | | Use case | Local IPC, process monitoring | IoT/edge, microservices | Anything crossing untrusted networks |

Recovery-on-UDP is structurally rejected by default. Combining any recovery flag (--recovery-exec / --recovery-exec-file) with --udp-port is a startup hard-error unless the operator passes the transport-qualified accept flag for that listener — --secure-udp-i-accept-recovery-on-unauthenticated-transport for a secure-UDP listener, or --plaintext-udp-i-accept-recovery-on-unauthenticated-transport for a plaintext one. That flag stamps the listener’s beats OperatorAttestedTransport, which the runtime origin gate (Recovery::on_stall) accepts; without it, UDP beats stay NetworkUnverified and recovery is refused. It is the single switch — there is no separate runtime opt-in. See book/src/architecture/peer-authentication.md for the full threat model.

CLI additions

# Listen on UDS only (default)
varta-watch --socket /tmp/varta.sock --threshold-ms 500

# Listen on UDS + UDP (requires --features udp at build time)
varta-watch --socket /tmp/varta.sock --threshold-ms 500 \
            --udp-port 9000 --udp-bind-addr 0.0.0.0

# UDP-only (no UDS)
varta-watch --socket /tmp/varta.sock --threshold-ms 500 \
            --udp-port 9000

# UDP with ChaCha20-Poly1305 encryption
# Generate a 256-bit key (64 hex chars)
openssl rand -hex 32 > /tmp/varta.key

varta-watch --socket /tmp/varta.sock --threshold-ms 500 \
            --udp-port 9000 --key-file /tmp/varta.key

# Rotation: accept old key while transitioning to new key
openssl rand -hex 32 > /tmp/varta-new.key
varta-watch --socket /tmp/varta.sock --threshold-ms 500 \
            --udp-port 9000 --key-file /tmp/varta.key \
            --accepted-key-file /tmp/varta-new.key

# Per-agent key derivation from master key
# The observer derives agent-specific keys from the PID embedded in
# each frame's iv_random prefix. Compromise of one agent's key does
# not reveal other agents' keys or the master key.
openssl rand -hex 32 > /tmp/varta-master.key
varta-watch --socket /tmp/varta.sock --threshold-ms 500 \
            --udp-port 9000 --master-key-file /tmp/varta-master.key

The combined shared-key set from --key-file and --accepted-key-file is capped at 8 keys. The observer trials every shared key on every secure-UDP datagram to avoid leaking the active rotation slot through response timing, so the cap is also a poll-loop work bound.

Feature flags

Security

• UDS: On Linux, the kernel attests the sender’s PID and UID via SCM_CREDENTIALS. The observer rejects frames where frame.pid != peer_pid or peer_uid != observer_uid. Linux recovery eligibility also requires the observer to pin the sender’s /proc/<pid>/stat start-time generation before first contact can become KernelAttested; an unpinned first-contact beat is tracked as SocketModeOnly. On macOS pathname datagram sockets, LOCAL_PEERTOKEN requires a connected local socket and the observer falls back to --socket-mode 0600. On other platforms without per-datagram credentials, the only defence is --socket-mode.

• UDP (plaintext): No kernel credential mechanism exists. peer_pid is always 0, which causes the observer to skip PID verification. Trust must be established at the network layer — firewall rules, VPC boundaries.

• UDP (secure): Every frame is encrypted with ChaCha20-Poly1305 (RFC 8439) using a 256-bit key. Primitives are provided by the chacha20poly1305 crate (RustCrypto, NCC Group audit 2020) — no hand-rolled crypto. Key derivation uses HKDF-SHA256 (RFC 5869) via the hkdf + sha2 crates. Two key modes:

  • Shared key: A single pre-shared key for all agents (--key-file).
  • Master key: Per-agent keys derived from the agent’s PID via HKDF-SHA256 (--master-key-file). The PID is embedded in the iv_random prefix so the observer can derive the correct agent key before decryption. Compromise of one agent’s key does not reveal other agents’ keys or the master key. Note: the HKDF-based KDF is incompatible with the ChaCha20-PRF KDF used in earlier releases — agents must re-key when upgrading from a pre-RustCrypto build if master-key mode was in use.
  • Replay attacks are blocked by enforcing monotonic IV counters per sender. Key rotation is supported via --accepted-key-file (no downtime required).
  • Panic-hook entropy: install_panic_handler_secure_udp reads all IV material at install time and fails closed if the entropy chain (getrandom, getentropy, /dev/urandom) is unavailable. Forked-child panic prefixes are derived from that pre-read salt with HKDF, so the hook never calls the OS entropy chain. In chrooted environments without /dev, use install_panic_handler_secure_udp_accept_degraded_entropy to opt into a non-cryptographic fallback — see book/src/architecture/peer-authentication.md for the full nonce-reuse risk analysis.

• Recovery commands: Exec mode only (shell mode was permanently removed):

  • --recovery-exec: Command executed directly via execvp(2) with {pid} replaced in arguments; the pid is also appended as the final argument. No shell is involved.
  • --recovery-exec-file: Read the program + args from a hardened file with mandatory ownership/permission checks (UID match, mode ≤ 0600).

Container / PID-namespace semantics

Frame.pid carries the agent’s PID in the agent’s PID namespace. The observer’s kernel-attested peer PID (SO_PASSCRED / SCM_CREDS) is in the observer’s namespace. When the two namespaces differ:

• The pid in the frame cannot be used to identify a process the observer can kill(2) or systemctl restart — the same numeric PID refers to a different process in each namespace.
• The existing frame.pid == peer_pid check at observer ingress catches most cases (different namespaces usually produce different numeric pids), but same-pid collisions across containers (every container’s first process is PID 1) are invisible to that gate.

varta-watch therefore (Linux only):

1. Reads /proc/self/ns/pid once at startup and caches the inode as the observer’s namespace identity.
2. For every kernel-attested beat (UDS), reads /proc/<peer_pid>/ns/pid and compares the inode to the observer’s. Mismatch ⇒ drop the beat (varta_frame_namespace_mismatch_total++) and emit Event::NamespaceConflict.
3. Per-pid tracker slots pin the namespace inode at first beat; a later beat with a different Some(_) inode is rejected as Update::NamespaceConflict (varta_tracker_namespace_conflict_total++).
4. Recovery commands refuse to spawn for cross-namespace stalls and log an audit record with reason=cross_namespace_agent (varta_recovery_refused_total{reason="cross_namespace_agent"}++).

Escape hatch — --allow-cross-namespace-agents

When agents are intentionally run with --pid=host (containers sharing the host PID namespace), the observer’s namespace and the agents’ namespace agree at the kernel level — the gate above is a no-op.

For deployments where the agent runs in a private namespace and the operator has out-of-band PID translation (e.g. CNI metadata that lets a recovery script translate container pids to host pids), pass --allow-cross-namespace-agents. The audit log and metrics still fire, but beats are admitted and recovery is permitted.

--strict-namespace-check

Treat namespace mismatch as a fatal startup error: on the first Event::NamespaceConflict, the daemon logs a FATAL line and exits with a non-zero status. Used in environments where the operator wants the daemon to fail loudly rather than silently log audit refusals.

Non-Linux platforms

PID namespaces are a Linux kernel concept. On macOS and the BSDs, observer_pid_namespace_inode() returns None and all comparisons short-circuit to “match”. The CLI flags are accepted for portability but have no runtime effect.

UDP transports

UDP listeners (plain or secure) have no kernel peer-cred mechanism. peer_pid is 0; peer_pid_ns_inode is None. Recovery is already refused for NetworkUnverified origins by the existing transport gate — namespace mismatch adds nothing for UDP. See peer-authentication.md for the full trust model.

Secure UDP — replay-state capacity boundary (H4)

SecureUdpListener keeps per-sender replay state in a bounded table indexed by the authenticated VLP frame PID:

• Capacity: MAX_SENDER_STATES = 1024 simultaneously-tracked senders.
• Known senders can advance their replay state even while the table is full.
• Unknown senders are accepted only if a stale-sender sweep frees a slot.
• If the table remains full, the authenticated frame is consumed and refused; no live sender’s replay state is evicted to admit it.

This is a fail-closed replay posture. A reachable network can still create an availability event by sending authenticated traffic for enough unique PIDs to fill the table, but it cannot make the listener forget an existing sender’s last counter and then accept a captured older ciphertext for that sender.

Why the table refuses instead of evicting

Any finite eviction shadow can be rotated out by enough authenticated chaff. The only replay-safe bounded behavior is to preserve live replay state and refuse new identities at capacity. This trades admission of new senders for nonce monotonicity of already-tracked senders, which matches the safety profile used elsewhere in Varta: capacity pressure is visible and alertable, but replay state is not sacrificed silently.

Mitigation

varta-watch defaults --udp-bind-addr to 127.0.0.1 when secure-UDP keys are configured. Operators who genuinely need the listener to accept non-loopback peers must pass --i-accept-secure-udp-non-loopback explicitly — a CLI flag whose name signals the residual availability risk. When the flag is set, a high-visibility startup warning is emitted to stderr and the operator is expected to constrain network reach (firewall, private VLAN, mTLS-fronted tunnel) so that no untrusted host can reach the bound port.

The recovery gate on NetworkUnverified origins (see peer-authentication.md) remains independent of this flag — opting in to non-loopback secure-UDP does NOT enable recovery commands from UDP-origin beats. Those still require the separate --secure-udp-i-accept-recovery-on-unauthenticated-transport acknowledgement.

Secure UDP — session-restart replay window (H5)

SecureUdpListener resets a sender’s per-PID replay high-water when a recycled PID legitimately restarts its session. A process that dies and has its PID reused by a fresh agent reconnects with a new session salt, so its IV prefixes and VLP nonce sequence restart from the beginning; after SESSION_RESTART_GAP (= --threshold-ms) of silence the listener treats the first frame on an aged-out prefix as a session restart and resets the per-sender max_regular_nonce so the new low-nonce session is admitted.

That reset opens a bounded replay window. An on-path attacker who captured a frame from the dead session — one whose nonce is higher than the recycled session’s reset baseline; no key is needed to resend a captured ciphertext — can replay it in the window after the reset. Because the captured nonce exceeds the reset baseline it is accepted, forging a single liveness beat attributed to the recycled PID (and advancing the high-water to the replayed value).

Bounded by. The attack requires all of: (a) a captured dead-session frame, (b) the PID to be recycled, (c) SESSION_RESTART_GAP of silence, and (d) the replay to land before the recycled agent’s own nonce climbs past the captured value. The forged beat carries a NetworkUnverified origin (or, under the explicit opt-in, OperatorAttestedTransport): recovery commands remain gated behind --secure-udp-i-accept-recovery-on-unauthenticated-transport and are not triggerable by a replayed beat alone. The residual impact is therefore forged liveness (a recycled PID briefly appearing alive) plus a poisoned nonce high-water — not arbitrary command execution.

Root cause. Nonce-based replay protection cannot distinguish a recycled agent’s fresh-but-low-nonce session from a replayed high-nonce frame of the dead session, because the recycled agent derives a new session salt (and thus new IV prefixes) that the observer cannot bind to: VLP v0.2 carries no wire-level session/epoch identifier. Closing the window fully requires a protocol change — a per-session epoch in the frame that the observer binds into replay state — so it is deferred to a future VLP version rather than patched in the listener. Any listener-only heuristic either re-opens the window or rejects a legitimately fast-beating recycled agent (its nonce can outpace any fixed post-reset jump bound), so no clean implementation-only fix exists.

Mitigation. The loopback-default binding (H4) and the recovery-origin gate already constrain who can reach the port and what a replayed beat can do. Operators who require strict cross-session replay rejection should keep secure-UDP loopback-only or front it with an authenticated tunnel; a shorter --threshold-ms narrows (but does not eliminate) the window.

Fork-safety on secure-UDP

After fork(2), a child process inherits its parent’s SecureUdpTransport state — the 16-byte iv_session_salt, the iv_prefix_index, and the iv_counter. Three nominally-independent fields whose product defines the AEAD nonce. If the child ever calls Varta::beat() without intervention, it derives the same 12-byte ChaCha20-Poly1305 nonce its parent has already emitted under the same key — a catastrophic confidentiality and integrity failure (Poly1305 key recovery, plaintext XOR leak).

How Varta enforces fork-safety structurally

Varta::connect snapshots both std::process::id() and a process-lineage epoch maintained by a one-time pthread_atfork child callback. Every Varta::beat reads the current PID and epoch and compares both — on either mismatch, the wrapper invokes transport.reconnect() before building the frame. The epoch changes on every fork and therefore remains distinct even if a later descendant is assigned the original connect-time PID. SecureUdpTransport::reconnect() re-reads OS entropy into a fresh 16-byte session salt, recomputes the IV prefix, and resets the prefix index and counter to zero. The child’s first emitted frame therefore uses an IV prefix derived from independent entropy — nonce collision across the fork boundary is impossible.

Auto-recovery is silent: the caller observes BeatOutcome::Sent. The event is observable via Varta::fork_recoveries() -> u64 (suggested Prometheus name: varta_client_fork_recoveries_total). The local session epoch resets too — nonce → 0, start → Instant::now(), last_timestamp → 0, consecutive_dropped → 0 — so the child’s wire stream looks like a fresh session to the observer.

Observer view

The observer’s per-sender state in SecureUdpListener is keyed by the authenticated VLP frame PID, with a 1-deep IV-prefix history per sender (see H4 replay-state capacity above). When the forked child sends frames with a new IV prefix, the observer transitions its current state into the prev_* slots and accepts the new prefix as a fresh session — no replay error, no protocol-level signal required. Fork-recovery is entirely transparent to the wire format.

Advanced callers

Callers using SecureUdpTransport directly receive the same protection. The transport snapshots the process-lineage epoch at construction and checks it before every seal, reconnecting before inherited AEAD state can be used in a child.

Parent-pid stall window (transport-agnostic)

Auto-recovery handles the child. The parent does not get a free pass: if the parent forks and then exit(0)s (the daemonise pattern), its PID disappears from the kernel but the observer’s tracker slot for that PID keeps aging. After --threshold-ms the slot stalls; if recovery is configured for kernel-attested origins, the observer may fire a recovery command for a PID that no longer exists. This applies to every transport (UDS, plaintext UDP, secure UDP) — it is a property of the silence-equals-stall contract, not of any particular wire format.

The fix is on the agent side. The recommended pattern is to emit a final Status::Critical beat from the parent immediately before its terminal exit() — the observer records the critical frame and treats subsequent silence as expected closure rather than as a stall. See crates/varta-client/README.md — Fork recovery & tracker semantics for the operator-side patterns and the alternative --threshold-ms widening approach.

Panic-hook parallel

install_panic_handler_secure_udp caches an 8-byte IV prefix plus a 16-byte fork salt at install time to avoid non-async-signal-safe entropy reads inside the panic hook itself. The same fork hazard applies: a child that panics would otherwise emit (cached_iv, iv_counter=0) — colliding with the parent’s identical pair if the parent panicked too. The installer snapshots install_pid; if the hook later sees a different PID, it derives a child-specific IV prefix with HKDF-SHA256 over the pre-read fork salt, the panic PID, the panic timestamp, and the AEAD counter. No getrandom, getentropy, or /dev/urandom call happens from the hook body. The strict variant fails closed at install time when no entropy source is reachable; the accept-degraded-entropy variant falls back to fallback_iv_random() and fallback_iv_session_salt() per the documented degraded-entropy policy.

Cross-references

• Observer liveness — the watcher’s own liveness story: in-process self-watchdog, systemd sd_notify, hardware watchdog, and paired-observer pattern
• Safety profiles — compile-time vs. runtime feature gating for production-safe builds
• Peer authentication — kernel-level PID attestation and transport trust classification
• Namespaces — dedicated reference for cross-namespace deployments

Future transports

Additional transports can be implemented by implementing BeatTransport (agent side) and BeatListener (observer side) without touching the protocol core:

• Shared memory (memfd, shm) — Wasm plugins writing directly to a shared ring buffer
• Unix pipes (pipe, fifo) — stdin/stdout health frames for supervised processes
• WebSocket — for browser-based health dashboards

Varta Threat Model

This document outlines the formal threat model for the Varta health monitoring protocol. Varta is designed for high-assurance, zero-overhead health signalling in distributed systems and local IPC.

1. Scope and Assets

Scope

The scope of this threat model includes:

• varta-vlp: The 32-byte wire protocol and its cryptographic primitives.
• varta-client: The agent library integrated into application processes.
• varta-watch: The observer daemon that tracks agent health and triggers recovery.
• Transports: Unix Domain Sockets (UDS) and User Datagram Protocol (UDP).

Assets

1. Agent Liveness State: The true health status of an agent process.
2. Recovery Commands: The ability to execute privileged operations (e.g., systemctl restart) based on agent health.
3. Master/Session Keys: Cryptographic material used to secure UDP heartbeats.
4. Metrics Data: Operational visibility provided via the Prometheus /metrics endpoint.
5. System Availability: The continued operation of the observer itself.

2. Trust Boundaries

Varta operates across several trust boundaries:

3. Threat Analysis (STRIDE)

S: Spoofing (Impersonating an Agent)

• Threat: An attacker process sends forged heartbeats for a target PID to hide a stall or trigger a false recovery.
• Mitigation (UDS): Kernel PID Attestation where the OS supports it. The observer uses SO_PASSCRED (Linux) or SCM_CREDS / SCM_UCRED (supported BSD-family and illumos/Solaris targets) to verify the sender’s real PID. macOS pathname UDS is socket-mode-only and recovery-ineligible.
• Mitigation (Secure UDP): ChaCha20-Poly1305 AEAD. Every frame is cryptographically signed. Master-key mode derives per-agent keys from PIDs to prevent cross-agent spoofing.
• Mitigation (Network): UDP heartbeats are tagged NetworkUnverified and are ineligible for recovery commands by default.

T: Tampering (Modifying Heartbeats)

• Threat: An attacker modifies a heartbeat in transit to change its status (e.g., changing Ok to Critical).
• Mitigation (Wire): CRC-32C Integrity. Every frame includes a CRC-32C trailer to catch bit flips and in-process corruption.
• Mitigation (Secure UDP): Poly1305 MAC. Any modification to an encrypted frame causes a decryption failure, and the frame is dropped.

R: Repudiation (Audit Log Evasion)

• Threat: A recovery action occurs, but there is no record of why or which agent triggered it.
• Mitigation: Opt-in Recovery Auditing. When --recovery-audit-file <PATH> is configured, all recovery actions — including refusals, kills, and reaps — are logged to a structured TSV with kernel-attested PIDs where available. The audit-chain feature adds SHA-256 hash chaining for tamper evidence. Without the flag, recovery actions are visible only via the Prometheus varta_recovery_outcomes_total / varta_recovery_refused_total counters. For high-assurance deployments the audit file is strongly recommended and is required for IEC 62304 / DO-178C-grade installations.

I: Information Disclosure (Leaking Secrets)

• Threat: Cryptographic keys or Prometheus tokens are leaked via environment variables or insecure file permissions.
• Mitigation: Secret-File Hardening. Varta refuses to load keys from environment variables. Key files must be owned by the observer UID and have 0600 permissions.
• Mitigation (Memory): Zero-on-Drop. The Key type zeros its memory before being released. Panic hooks use a single-owner Box to minimize secret lifetime.

D: Denial of Service (Exhausting Observer Resources)

• Threat: An attacker floods the observer with connection requests or malformed frames to prevent it from monitoring legitimate agents.
• Mitigation (Metrics): Multi-layer DoS Protection. Rate limiting per source IP, connection budgets, and constant-time token comparison for the /metrics endpoint.
• Mitigation (Wire): Zero-Allocation Hot Path. The observer processes frames without allocating on the steady-state path, preventing heap exhaustion.

E: Elevation of Privilege (Exploiting Recovery)

• Threat: An attacker triggers a recovery command that executes an arbitrary shell script or targets a process outside its namespace.
• Mitigation: Exec-Only Recovery. Shell-mode recovery is removed. Commands are executed directly via execvp(2) with no shell interpolation.
• Mitigation (Namespace): Linux PID Namespace Gating. The observer verifies that the agent belongs to the same PID namespace before permitting recovery.
• Mitigation (Environment): Isolated Recovery Environment. Recovery children run with a sanitized, minimal environment to prevent LD_PRELOAD or PATH attacks.

4. Security Boundaries & Mitigations Summary

The “Kernel-First” Trust Model

For local IPC, Varta trusts the kernel over the wire format. A frame’s pid field is only used if the kernel attests that the sending process actually owns that PID.

Cryptographic Identity (Secure UDP)

For network communication, Varta uses a 256-bit key-based identity.

• Forward Secrecy: Deliberately not provided. A one-way unauthenticated-receiver heartbeat protocol has no handshake in which to negotiate ephemeral keys; adding a DH ratchet would require multi-round-trip session establishment, which contradicts Varta’s connectionless beat-and-forget model. Key rotation via --accepted-key-file (multiple accepted keys, time-bounded rollover) is the recommended mitigation; see Peer Authentication for the full key-loading model and rotation procedure.
• Replay Protection: Enforced via monotonic IV counters per sender.

Recovery Safety Gates

Recovery is the most privileged action Varta performs. It is guarded by:

1. Origin Gating: Recovery is disabled for NetworkUnverified (UDP) sources unless explicitly enabled with verbose CLI flags.
2. Platform Gating: On platforms without kernel PID attestation (e.g., OpenBSD), recovery is disabled.
3. Execution Safety: Commands are never passed to a shell.

5. Residual Risks

1. Compromised UID (Local): If an attacker gains the same UID as the observer, they can read the secret keys and potentially bypass socket-mode permissions.
2. Master Key Leak: A leak of the master key allows an attacker to derive all agent keys and spoof any agent on the network.
3. Clock Skew (UDP): Varta uses monotonic timestamps, but significant clock drift or resets on the agent side can lead to rejected heartbeats or false stall detections.
4. Namespace Mapping: In complex container environments, PID 1 in a container may map to a different host PID. varta-watch provides PID-namespace gating via its own --allow-cross-namespace-agents and --strict-namespace-check flags. For multi-container recovery the observer container typically needs the runtime’s host-PID share (e.g. Docker/podman --pid=host, Kubernetes hostPID: true) so that recovery targets and the observed PID namespace agree; otherwise namespace mismatches cause beats and recovery to be refused. See Namespacing.
5. No Forward Secrecy: As a one-way protocol, Varta does not provide forward secrecy. If a key is compromised, all past traffic encrypted with that key can be decrypted if captured.

Observer Liveness — “Who Watches the Watcher?”

varta-watch is the single observer for all agents on a host. If it crashes or its poll loop hangs, no agent gets a Stall event and no recovery fires — the entire monitoring layer fails silently. For life-support deployments this is the most critical functional gap.

This document describes four independent, layered defenses. Deploy as many as your environment supports; each catches failure modes the others cannot.

Threat model

*systemd detects a hang only if WATCHDOG=1 stops arriving; the self-watchdog ensures that also stops when the loop wedges.
†hardware watchdog fires when the kick loop stops; process crash achieves this.
‡since H5 the watchdog thread is the sole source of WATCHDOG=1; if it dies, the emission stream stops and systemd’s WatchdogSec= fires.

L1 — In-process self-watchdog (--self-watchdog-secs)

A background thread checks that the main poll loop has ticked at least once within the configured deadline. If not, it calls process::abort().

varta-watch --self-watchdog-secs 4 ...

• The background thread is the only non-main thread in the binary. The beat path and observer loop remain single-threaded.

• process::abort() produces SIGABRT, which appears in journalctl, enables core dumps, and triggers Restart=on-abort in systemd units.

• The deadline should be set to roughly 2× the expected worst-case poll latency (typically --threshold-ms + reaping time).

• Per-stage wedge detection (H6): in addition to the full-iteration check, the watchdog reads two atomics written by the main thread — CURRENT_STAGE (which of the six loop phases is running) and LAST_STAGE_ENTRY_NS (monotonic ns at which that phase started). Each stage has an independent hard abort threshold in STAGE_ABORT_NS (≥ 5× the stage’s soft budget):

  Stage	Hard abort threshold
  drain_pending	2 s
  poll	2 s
  maintenance	500 ms
  recovery_reap	1 s
  serve_pending	2 s
  housekeeping	1 s

  A stage wedge (e.g. an fdatasync blocking indefinitely, or a single waitpid hanging) trips the per-stage threshold long before the full-iteration watchdog fires. The watchdog logs which stage wedged and aborts. Between iterations CURRENT_STAGE is set to u8::MAX (idle sentinel) so the per-stage check is suppressed during throttle sleeps.

• H5 (post-2026-05-13): the watchdog thread is ALSO the sole emitter of systemd WATCHDOG=1. Emission used to live on the main loop, which left a silent-failure window: if the watchdog thread died while the main loop remained healthy, WATCHDOG=1 kept arriving from the main thread and systemd had no way to notice the in-process abort path was already gone. Now WATCHDOG=1 emission is moved to the watchdog thread (via a dup(2)-ed copy of the notify socket carved off SdNotify with take_watchdog_notifier). If the thread dies, the emission stream stops and WatchdogSec= fires. This is the only design where systemd can detect a dead watchdog while the main loop is still alive.

• Auto-enable: when $WATCHDOG_USEC is set by the service manager and --self-watchdog-secs is not passed, the watchdog thread is spawned unconditionally with a 4 s deadline. Operators with tighter WatchdogSec= values can override via the CLI. This collapses the L1+L2 layers structurally: enabling WatchdogSec= in the unit automatically buys both the in-process abort path and the WATCHDOG=1 emission stream.

L2 — systemd sd_notify watchdog integration

varta-watch speaks the sd_notify(3) protocol natively. Set Type=notify in the service unit and configure WatchdogSec=:

[Service]
Type=notify
NotifyAccess=main
WatchdogSec=5s
Restart=on-watchdog
RestartSec=1s
TimeoutStartSec=10s
ExecStart=/usr/bin/varta-watch \
    --socket /run/varta/agents.sock \
    --threshold-ms 5000 \
    --self-watchdog-secs 4 \
    --hw-watchdog /dev/watchdog \
    --heartbeat-file /run/varta/heartbeat

varta-watch sends:

• READY=1 after observer bind succeeds and all listeners are attached
• WATCHDOG=1 every WATCHDOG_USEC / 2 microseconds while the poll loop runs
• STOPPING=1 when the SHUTDOWN latch flips

If WATCHDOG=1 stops arriving, systemd kills and restarts the process. This catches both crashes (no more sends) and hangs (LAST_TICK_NS stops advancing, the self-watchdog aborts, systemd restarts).

$NOTIFY_SOCKET and $WATCHDOG_USEC are passed automatically by systemd; no extra flags are needed.

L3 — Hardware watchdog (--hw-watchdog)

On hosts with a kernel hardware watchdog (e.g. /dev/watchdog), varta-watch can kick it once per poll iteration. If the kick stops, the kernel reboots the host — even if the OS itself is wedged.

varta-watch --hw-watchdog /dev/watchdog ...

At startup, varta-watch verifies that the opened descriptor is a character device. A regular file, FIFO, or socket is rejected rather than silently accepting writes while providing no watchdog protection.

On Linux builds with a pinned watchdog ioctl ABI (x86_64, aarch64, and riscv64 today), varta-watch also verifies that the descriptor implements the standard watchdog ioctl API and has crash-close semantics. It reads WDIOC_GETSUPPORT and accepts the device only when it advertises WDIOF_MAGICCLOSE or sysfs reports nowayout=1 for the same character device. It then reads WDIOC_GETTIMEOUT, and if the current timeout is below 30 s it requests 30 s with WDIOC_SETTIMEOUT. Startup fails if the device is not a watchdog, close behavior cannot be proven safe, the timeout cannot be read, the kernel clamps the timeout below the 30 s floor, or the Linux target’s ioctl encoding has not yet been pinned.

Magic close: on a clean shutdown (SIGTERM/SIGINT followed by graceful exit) varta-watch writes the magic byte 'V' to disarm the watchdog before exiting. A crash or hang leaves the watchdog armed; the kernel reboots after its timeout. If startup validation rejects a device after opening it, varta-watch also best-effort writes 'V' before returning the startup error so a clean configuration failure does not leave an already-opened watchdog running. On Linux nowayout=1 devices the kernel deliberately overrules magic close; varta-watch accepts that mode because crash/hang protection is stronger, but a clean service stop cannot disarm the hardware watchdog.

The /dev/watchdog device is typically root-owned (mode 0600). Run varta-watch as root or grant the CAP_SYS_ADMIN capability, or use a watchdog daemon (e.g. watchdog(8)) for the actual device management.

L4 — Paired observers (operational)

A second monitoring process scrapes the first observer’s liveness signals and restarts it if they stall. This requires no code changes — use the existing --heartbeat-file and /metrics primitives.

Heartbeat-file poller

#!/bin/sh
HEARTBEAT=/run/varta/heartbeat
while :; do
    prev=$(awk '{print $1}' "$HEARTBEAT" 2>/dev/null || echo 0)
    sleep 5
    cur=$(awk '{print $1}' "$HEARTBEAT" 2>/dev/null || echo 0)
    if [ "$cur" -le "$prev" ]; then
        logger -t varta-watchdog "heartbeat stalled (loop_count=$prev); restarting"
        systemctl restart varta-watch
    fi
done

The first field in the heartbeat file is a monotonically increasing loop counter. If it stops advancing, the observer is wedged or dead. Each update uses an exclusively-created mode-0600 tempfile plus an atomic same-directory rename, so stale files and symlinks are never opened or truncated. The parent directory should still be writable only by the observer account.

Prometheus uptime scraper

/metrics exposes varta_watch_uptime_seconds. A second Prometheus instance (or Alertmanager rule) can alert when the gauge stops increasing. The canonical VartaWatchStalled rule ships in observability/alerts/varta.rules.yml; see Monitoring & Alerting for the operator-facing rationale.

Threading note

--self-watchdog-secs spawns one background thread. This is the only non-main thread in the varta-watch binary, and that property is a load-bearing architectural invariant, not an accident. All agent beat processing, stall detection, recovery spawning, and Prometheus serving happen on the main thread. The watchdog thread reads two atomics (SHUTDOWN and LAST_TICK_NS), calls process::abort() on wedge, and writes WATCHDOG=1 to its own dup(2)-ed UnixDatagram fd; it never touches shared mutable state. The dup-ed fd is independent kernel state — both threads own their own descriptor and there is no synchronisation between them on the notify path.

The single-threaded design is what lets the project preserve its zero-alloc, ABI-stable beat contract: a beat is decoded into a stack-allocated [u8; 32] and dispatched through the per-pid tracker without locking, because nothing else holds a reference. Moving any phase of the loop to a second thread would require a lock-free SPSC ring between threads at the ingress and break that contract. Stall-detection latency under scrape load is instead bounded by an explicit per-iteration latency budget — see below.

Why /metrics is on the poll thread

“Doesn’t scrape latency variance steal time from beat ingestion?”

It can, by up to ~200 ms per iteration — the structural cap of PromExporter::serve_pending (100 ms serve deadline + 100 ms drain deadline, see exporter.rs). The obvious mitigation is to spawn a second thread that owns serve_pending and reads tracker state through a shared snapshot. We deliberately do not do this. Three reasons:

1. The beat path would acquire a lock on every tick. Whether via Arc<Mutex<PromExporter>> or an SPSC snapshot ring, every record-side counter increment (pe.record_beat(...), pe.record_stall(...), pe.record_loop_tick(...) etc.) becomes either a mutex acquisition or a single-producer write into a wait-free queue. Neither is zero-overhead on the hot path, and both introduce per-architecture memory-ordering questions that the current &mut self model eliminates by construction.
2. The zero-allocation invariant becomes harder to enforce. The beat path is currently zero-alloc post-connect, enforced by the varta-tests guard allocator. A snapshot ring requires either a pre-sized arena (more state on the hot path) or per-snapshot allocation (kills the invariant). Both are worse than what we have.
3. The variance is already bounded and now observable. Scrape work per iteration is capped at ~200 ms by PROM_READ_DEADLINE = 10 ms, PROM_MAX_CONNECTIONS_PER_SERVE = 8, PROM_MAX_DRAIN_PER_SERVE = 50, the 100 ms serve deadline, and the per-IP token bucket. Operators see the variance through varta_observer_serve_pending_seconds (new — see “Observing scrape-induced latency” below); beat-path latency is iteration_seconds - serve_pending_seconds in PromQL.

Scrape-storm alarms and beat-path alarms therefore route off different metrics, and the load-bearing single-thread invariant is preserved.

Latency budget — worst-case poll iteration time

A bounded iteration time guarantees a bounded stall-detection latency. The table below names the phases of the poll loop in main.rs and the upper-bound source for each:

Two observations the table makes explicit:

• The UDS read-timeout is the idle floor: with no incoming beats and no scrape pressure, every iteration costs about read_timeout. This is intentional — it yields CPU between recvs without busy-spinning. Lower the floor by lowering --read-timeout-ms, at the cost of a tighter idle poll loop.
• The worst-case active iteration is bounded by read_timeout + serve_pending, since recv(2) returns early as soon as a frame arrives and serve_pending is the only other phase that can spend more than a few milliseconds.

The default soft budget is 250 ms (--iteration-budget-ms). Iterations exceeding it increment varta_observer_iteration_budget_exceeded_total and are visible in the varta_observer_iteration_seconds histogram. The budget is advisory: hard wedges (seconds, never returning) remain the responsibility of --self-watchdog-secs.

The idle sleep at the end of an iteration with no pending I/O (10 ms) is excluded from the histogram. Idle time is a throttling primitive, not work latency; including it would mask the bad iterations.

Tuning relationship

For a given --threshold-ms T, stall-detection latency is bounded by T + per_iteration_worst_case. With defaults (--threshold-ms 5000, --read-timeout-ms 100, default serve_pending bounds) the worst case is ~310 ms, so a stalled agent surfaces no later than ~5.31 s after its last beat.

The soft --iteration-budget-ms (default 250 ms) sits between the typical case (~100 ms idle floor) and the worst case (~310 ms under scrape storm) so the budget-exceeded counter fires only during real scrape pressure, not on every active iteration. Operators with higher --read-timeout-ms or multiple listeners should raise the budget proportionally (budget ≥ read_timeout × N_listeners + 150 ms).

--self-watchdog-secs should be set such that self_watchdog_secs × 1000 ≥ 4 × iteration_budget_ms so transient overruns during scrape bursts do not trigger false-positive aborts. The default guidance (--self-watchdog-secs 4 with --iteration-budget-ms 250) gives a 16× margin (4000 ms ÷ 250 ms), well above the worst-case ratio.

Observing scrape-induced latency

Three metrics together let an operator separate scrape pressure from beat-path slowness:

• varta_observer_iteration_seconds — wall time for the entire poll iteration (drain → poll → maintenance → recovery reap → serve_pending → heartbeat write → watchdog kicks). Bucketed by [0.001, 0.005, 0.01, 0.05, 0.1, 0.25, 0.5, 1.0, +Inf]. Includes serve_pending — unchanged contract.
• varta_observer_serve_pending_seconds — wall time for the serve_pending phase alone. Same bucket boundaries as iteration_seconds so the two are coherent. Configurable budget via --scrape-budget-ms (default 250 ms); values below the built-in structural cap also bound live scrape work, and overruns increment varta_observer_scrape_budget_exceeded_total.
• varta_observer_iteration_budget_exceeded_total — iterations exceeding --iteration-budget-ms (default 250 ms). Includes serve_pending time.

Beat-path latency is then a PromQL expression — the difference between iteration time and serve-pending time:

# P99 beat-path latency = P99(iteration_seconds) − P99(serve_pending_seconds).
# Note: subtracting quantiles is approximate (P99 of diff ≠ diff of P99s),
# but in practice serve_pending and the rest of the iteration are weakly
# correlated, so the approximation is monotonic with the true beat-path
# latency.  Use sum_by-(le) rate() if you want exact derived histograms
# (compute beat_path_seconds in a recording rule from the two histos).
histogram_quantile(0.99,
  sum by (le) (rate(varta_observer_iteration_seconds_bucket[5m])))
- histogram_quantile(0.99,
    sum by (le) (rate(varta_observer_serve_pending_seconds_bucket[5m])))

Alarms that should fire on beat-path slowness route off iteration_seconds - serve_pending_seconds or off iteration_budget_exceeded_total minus scrape_budget_exceeded_total when scrape overruns dominate the budget overruns.

Alarms that should fire on scrape-storm pressure route off scrape_budget_exceeded_total and serve_pending_seconds quantiles directly.

Per-stage histograms (varta_observer_stage_seconds)

Each of the six loop phases emits an independent histogram with the same bucket boundaries as varta_observer_iteration_seconds:

Every stage emits every bucket from the first scrape (stable label set) so absent() alert rules stay valid before the first observation.

Use rate(varta_observer_stage_seconds_sum[5m]) / rate(varta_observer_stage_seconds_count[5m]) per stage to isolate which phase is contributing to latency.

Audit-ring back-pressure metrics

The recovery audit log uses an in-memory ring (cap 256) to decouple fdatasync from the hot path. record_spawn / record_complete enqueue formatted lines; the maintenance phase drains them within a 10 ms budget.

Non-zero audit_dropped_total means audit records are being permanently lost — either the disk is too slow, the budget is too tight, or the event rate is unsustainably high. Non-zero audit_flush_budget_exceeded_total is a precursor: lines are accumulating faster than they drain, but no data is lost yet.

The canonical alert rules covering all three observer-liveness symptoms (VartaAuditRecordDropped, VartaAuditFlushBudgetPressure, VartaRecoveryReapTruncated) plus the iteration-budget / beat-path-latency family (VartaIterationBudgetOverruns, VartaIterationP99High, VartaScrapeStormPressure, VartaBeatPathP99High) live in observability/alerts/varta.rules.yml. The beat-path-latency derivation uses the varta:beat_path_seconds:p99_5m recording rule (defined in observability/recording-rules/varta.rules.yml) so dashboards and alerts read identical numbers.

Recommended Prometheus alerts

See Monitoring & Alerting for the catalogue with per-alert runbooks and severity routing.

Tracker bounded-work guarantee

Each beat frame triggers at most one call to find_evictable_slot when the tracker is at capacity. That call scans at most eviction_scan_window slots (default 256, configurable via --eviction-scan-window).

Per-frame slot reads ≤ eviction_scan_window.

A full table sweep — confirming every slot is ineligible — takes at most:

ceil(tracker_capacity / eviction_scan_window)

consecutive record() calls (the rotating cursor resumes where it stopped).

With defaults (capacity = 256, window = 256) this is 1 call. With --tracker-capacity 4096 --eviction-scan-window 16 the sweep takes 256 calls — each individual call still reads ≤ 16 slots, so the per-frame beat-path cost stays bounded.

The varta_tracker_eviction_scan_window_max gauge (set once at startup) exposes the configured window so dashboards can derive the worst-case sweep depth. Operators alert on varta_tracker_eviction_scan_truncated_total to detect when the cap engages under a unique-pid flood.

Combine this bound with the iteration-budget WCET derivation above:

iteration_max ≤ read_timeout × N_listeners + eviction_scan_window × slot_read_ns

Tick-latency budget and hardware-watchdog margin

Bench-derived p99 cap

Under the canonical stress profile — 4096-slot tracker, balanced eviction policy, 30 agents × 100 Hz (≈ 3 000 beats/s) over UDS — the varta_observer_iteration_seconds p99 is ≤ 5 ms.

Run the bench to reproduce the measurement on your hardware:

cargo build --workspace --release --features prometheus-exporter
cargo run -p varta-bench --release -- tick-distribution

The bench asserts p99 ≤ 5 ms and exits non-zero if the cap is breached, printing the full bucket distribution and observed percentiles for triage. It also reports varta_tracker_eviction_scan_truncated_total and varta_observer_iteration_budget_exceeded_total so you can confirm the eviction-scan cap engages under the test load without blowing the latency budget.

Soft iteration budget

--iteration-budget-ms (default 250 ms) is the soft per-iteration ceiling. Overruns increment varta_observer_iteration_budget_exceeded_total but do not abort the loop. The default 250 ms gives 50× headroom over the 5 ms p99 cap; overruns therefore indicate genuine scrape-storm pressure, not normal active-load variance. See the “Latency budget” section for the full derivation.

Hardware-watchdog timeout floor

Operators deploying --hw-watchdog /dev/watchdog need a kernel watchdog timeout of ≥ 30 s. On Linux builds with a pinned watchdog ioctl ABI, varta-watch enforces this at startup by querying the device and raising shorter timeouts when the driver supports WDIOC_SETTIMEOUT; on other Unix targets, configure the device externally before starting the observer. The derivation:

The observer kicks the hardware watchdog at the end of every poll iteration (after heartbeat-file write and sd_notify). A single missed kick cannot trip the device; a sustained stall of ≥ device-timeout will. The 30 s floor provides ample budget for:

• Audit-log filesystem stalls (varta_log_suppressed_total{kind="audit_io"} will show rate limiting if these recur)
• Prometheus scrape contention (serve_pending_seconds quantiles)
• The H5 self-watchdog’s 4 s deadline with ≥ 7× margin

Round-robin fairness bound

Observer::poll() rotates the next_listener_start cursor on every non-WouldBlock receive. Per-listener worst-case admission delay is therefore bounded by N_listeners × per-listener-recv-cost. Under the canonical bench profile (single UDS listener) this is simply the UDS recv latency; with N additional UDP listeners add N × ~10 µs per iteration.

Eviction scan under stress

The bench will record non-zero varta_tracker_eviction_scan_truncated_total when the tracker fills and the 256-slot eviction window exhausts without finding a stalled slot. This is expected and by design — the cap proves the per-frame cost stays bounded even under a unique-pid flood. The p99 assertion holds even when the truncation counter is non-zero.

Debounce table semantics under load

The Recovery runner keeps a per-pid ledger of the most recent recovery fire (LastFiredTable). Each subsequent stall for the same pid is gated on now - last_fired[pid] >= debounce; closer-than-debounce stalls return RecoveryOutcome::Debounced and never spawn a child.

Capacity and eviction policy

The ledger is a fixed-size, array-backed table with capacity MAX_LAST_FIRED_CAPACITY = 4096. Capacity is sized to make the M8 adversarial-burst pattern costly: 4096 distinct pids would have to stall faster than debounce cadence before the eviction policy is engaged. Per-slot cost is Option<LastFiredSlot> ≈ 24 bytes → ~96 KiB total — within budget for the observer.

When the table is full and a stall arrives for a new pid, the policy is fail-closed:

1. The oldest slot is identified by a single bounded linear scan.
2. If that slot’s age is at least debounce, it is evicted and the new pid takes its place. Per-pid debounce semantics are preserved because the evicted pid’s window has already elapsed. The eviction is counted in varta_recovery_last_fired_evictions_total (operators tune capacity on this signal).
3. If the oldest slot’s age is below debounce, the recovery is refused. The runner returns RecoveryOutcome::RefusedDebounceCapacity { pid }, emits a RefusedRecord { reason: "debounce_capacity" } to the audit log, and bumps both varta_recovery_outcomes_total{outcome="refused_debounce_capacity"} and varta_recovery_refused_total{reason="debounce_capacity"}.

Eviction is debounce-respecting churn; refusal is suppression. Operators tune capacity on the first signal and alert on the second.

Clock-regression defense

All age comparisons use Instant::saturating_duration_since, which returns Duration::ZERO on regression. ZERO-duration entries are treated as “not eligible for eviction” — preventing a backwards clock blip from auto-evicting the whole table.

Recommended alerts

# Alert immediately on any debounce-capacity refusal — this is either
# legitimate scale-out past 4096 concurrent stalls or the M8
# adversarial stall-burst pattern.  Either case warrants paging.
rate(varta_recovery_refused_total{reason="debounce_capacity"}[5m]) > 0

# Warn on sustained eviction churn — debounce semantics are still
# intact, but capacity is becoming a bottleneck under steady-state
# load.  Tune MAX_LAST_FIRED_CAPACITY or audit which pids are
# stalling.
rate(varta_recovery_last_fired_evictions_total[5m]) > 0.1

# Page on any non-zero invariant-violation count — the defensive
# fall-throughs in LastFiredTable should never fire in correct
# operation.  Non-zero values indicate a code bug, not load.
varta_recovery_invariant_violations_total > 0

For the deployment-side answer to exceeding the 4096-agent cap — running multiple varta-watch instances and fanning agents across them — see Deployment Ceiling & Sharding.

Bounded-WCET guarantee

Every LastFiredTable operation is a linear scan over a fixed-size backing store. The unit test last_fired_table_prune_bounded_wcet asserts the prune sweep completes in under 5 ms in debug builds at full capacity (a future refactor that reintroduces O(n²) behaviour disguised as “cleanup” is caught by this test).

The pre-M8 HashMap-based implementation was the source of the debounce-bypass bug closed by this section: reactive pruning at the top of on_stall (prune_threshold = debounce * 10) left the map full of fresh entries under adversarial load, and the at_capacity branch skipped the debounce check entirely. The new table never skips the check; capacity pressure surfaces as a refusal or an audited eviction.

Audit-log durability vs availability

The recovery audit log (crate::audit::RecoveryAuditLog) is the last synchronous-disk path on the poll thread. Three operator-controlled budgets keep a wedged filesystem (NFS stall, full disk, slow SSD garbage-collection) from blocking the poll loop:

The drain is deferral-aware: when one fsync exceeds --audit-fsync-budget-ms, the remaining records in the same drain are written to the BufWriter only — the fsync is reattempted on the next tick. This bounds the worst-case poll stall on a slow disk to one fsync per tick, while still progressing the audit chain on disk (records sit in the BufWriter, durable through process restart via the Drop impl’s best-effort flush + sync_data).

Rotation is a state machine (drive_audit_rotation): one sub-step per call (one rename, then the fresh fd open, then the v2 header write, then the chain-stitching boot record + fsync). Each call honours --audit-rotation-budget-ms; if exceeded, state is preserved on self and the next tick resumes from the same sub-step. Recursion through direct_write_line → maybe_rotate → emit_boot → direct_write_line is structurally impossible because the hot path never drives rotation directly — it only sets a needs_rotation flag for the main loop to consume.

The default configuration preserves IEC 62304 Class C durability byte-for-byte: --recovery-audit-sync-every=1 + --audit-sync-interval-ms=0 means every record fsyncs before the drain returns, and --audit-fsync-budget-ms=50 only ever takes effect when a single fsync exceeds 50 ms — i.e. when the disk is already stalling the poll loop. Operators who can accept relaxed durability (e.g. cloud SRE deployments, not safety-critical) set --recovery-audit-sync-every=64 --audit-sync-interval-ms=100 to amortise fsync cost over many records while still pinning a worst-case sync interval.

Audit-log observability

Four signals back the operator’s mental model:

• varta_audit_fsync_seconds (histogram, shares ITERATION_BUCKET_BOUNDS_S with iteration_seconds) — per-call wall time of each fdatasync(2) on the audit fd.
• varta_audit_fsync_budget_exceeded_total (counter) — fsync calls whose wall time exceeded --audit-fsync-budget-ms.
• varta_audit_rotation_budget_exceeded_total (counter) — rotation drive calls that ran out of budget and deferred to the next tick.
• varta_audit_ring_watermark_total{level="warn"|"critical"} (counter) — rising-edge transitions of the in-memory ring fill across 75% and 95% of AUDIT_RING_CAP (= 256). Counter increments once per excursion above each threshold; falling-edge re-arms the rising-edge trigger. Both label values are emitted from the first scrape (stable-label-set discipline).

Recommended alerts

# fsync wall-time is climbing past the budget — disk is degraded but
# the poll loop is still bounded.  Investigate before audit records
# start dropping.
rate(varta_audit_fsync_budget_exceeded_total[5m]) > 0.1

# Critical watermark crossed — drain has fallen far enough behind
# that records will start dropping if the trend continues.  Pages
# operator before audit_dropped_total increments.
rate(varta_audit_ring_watermark_total{level="critical"}[5m]) > 0

# p99 fsync wall-time exceeds 100 ms — disk is becoming a poll-loop
# bottleneck even under the deferral.
histogram_quantile(0.99, rate(varta_audit_fsync_seconds_bucket[5m])) > 0.1

The structural answer to “what happens when the disk is permanently slow” is visible degradation: every tick defers, both fsync and rotation budget-exceeded counters climb, ring watermarks fire, and operators see the regression before records start dropping or the self-watchdog aborts. The poll loop itself stays within budget.

Cross-references

• Safety profiles — compile-time vs. runtime feature gating for production-safe builds
• VLP transports — transport-level trust classification
• Peer authentication — kernel-level PID attestation
• Verification — symbolic verification of Frame::decode (M7) and the LastFiredTable invariants on the verification roadmap

Recovery — Non-Blocking Spawn / Async Reap

This page documents how varta-watch keeps the observer loop responsive while still firing recovery commands on stalled agents.

Implementation lives in crates/varta-watch/src/recovery/ and is wired into the poll loop from crates/varta-watch/src/main.rs.

Why this exists

varta-watch runs a single thread driving Observer::poll on a 100 ms read-timeout cadence. When a pid crosses its silence threshold the observer surfaces Event::Stall and the binary calls Recovery::on_stall(pid).

A naive implementation would block the calling thread on the recovery child until it exits. That would freeze the entire poll loop — beat decoding, exporter pumping, Prometheus serving, and stall detection for every other pid — for the duration of one recovery command. A slow recovery template would take the observer offline.

Instead, Recovery::on_stall performs a non-blocking spawn and returns immediately. Outstanding children are reaped (or killed past their deadline) on subsequent observer ticks.

Constraints

These follow from the workspace-wide hard rules (see CLAUDE.md):

• Zero registry dependencies in varta-watch (path-only).
• No new threads. No tokio, no executors.
• No unsafe.
• Library code does not print; diagnostics live in main.rs only.

Public API

#![allow(unused)]
fn main() {
use std::process::ExitStatus;
use std::time::Duration;

#[derive(Debug)]
pub enum RecoveryOutcome {
    /// A child process was forked and is now outstanding. The observer
    /// has NOT waited on it. Reap on a later tick via `try_reap`.
    Spawned { child_pid: u32 },

    /// Previous invocation for this pid is still inside the per-pid
    /// debounce window; nothing was spawned.
    Debounced,

    /// `Command::spawn` failed (e.g. fork failure, program not found).
    SpawnFailed(std::io::Error),

    /// A previously-`Spawned` child has exited and was reaped on this
    /// tick. The observer never blocks waiting for this transition.
    Reaped {
        child_pid: u32,
        status: ExitStatus,
        duration_ns: u64,
    },

    /// A previously-`Spawned` child was killed via `kill(2)` on this tick
    /// after a timeout or a recycled-pid stale-child reclaim.
    Killed { child_pid: u32 },

    /// `try_wait` or `kill` failed for an outstanding child. The pid is
    /// still tracked; the observer will retry on the next tick.
    ReapFailed(std::io::Error),

    /// A recycled pid's previous recovery child could not be killed, so the
    /// old outstanding slot is retained and the new lineage is not spawned.
    RefusedStaleChildKillFailed { pid: u32, error: std::io::Error },
}

impl Recovery {
    pub fn with_exec_and_timeout(
        program: String,
        args: Vec<String>,
        debounce: Duration,
        timeout: Option<Duration>,
    ) -> Self;

    /// Spawn the configured program with the stalled pid appended as
    /// the final argument. Returns immediately; never blocks.
    pub fn on_stall(&mut self, pid: u32) -> RecoveryOutcome;

    /// Drain completed (or deadline-exceeded) children for one tick.
    /// Returns one outcome per state transition; empty when no children
    /// have transitioned since the last call.
    pub fn try_reap(&mut self) -> Vec<RecoveryOutcome>;
}
}

Lifecycle of one recovery

                    debounce-suppressed
                ┌──────────────► Debounced
                │
  Event::Stall ─┤                                  spawn ok
                │                              ┌────────────► Outstanding
                └─► Recovery::on_stall(pid) ───┤
                                               │ spawn err
                                               └────────────► SpawnFailed
                                                              (terminal)

  on every Observer tick:
      Recovery::try_reap()
         │
         ├─► child exited ─────► Reaped { child_pid, status }   (terminal)
         │
         ├─► deadline exceeded ─► kill(2) ─► Killed { child_pid } (terminal)
         │
         └─► try_wait/kill errno ─► ReapFailed(io::Error)        (retry)

Outstanding-child storage

Outstanding records live in OutstandingTable, a BoundedIndex-backed slab keyed by stalled pid. The table is sized to tracker::MAX_CAPACITY = 4096 at construction (recovery/mod.rs:436), so the recovery system can never hold more outstanding children than the tracker can hold pids — both bounded collections share the same ceiling. Operators raise the cap with --tracker-capacity; see Deployment Ceiling & Sharding.

When the table is full a fresh on_stall returns the bounded equivalent of Debounced and increments varta_recovery_refused_total{reason="outstanding_capacity"} (recovery/mod.rs:786). See Bounded Collections for the table’s allocation proof and the static-allocation rationale.

One outstanding child per stalled pid; if the pid stalls again while a child is still outstanding, the per-pid debounce window suppresses a duplicate spawn regardless of the table state.

If the pid’s start-time generation proves the OS recycled that numeric pid while a previous recovery child is still outstanding, recovery first tries to kill the stale child and move it to the bounded orphan reaper. A new lineage is spawned only after that kill succeeds or the child has already exited. Any other kill(2) failure is fail-closed as RefusedStaleChildKillFailed: the old outstanding slot stays tracked, no new child is spawned for the recycled pid, and the audit log records stale_child_kill_failed.

Tick budget

Observer READ_TIMEOUT is 100 ms. try_reap is invoked once per Observer::poll iteration. Worst-case latencies:

These are additive with the observer’s normal stall-detection latency; they do not affect beat decoding or exporter throughput on the critical path.

Default behaviour when --recovery-timeout-ms is omitted

Config::recovery_timeout = None is the default. In that mode outstanding children are reaped on completion but never killed. This preserves long-running-recovery semantics (e.g. a restart that blocks on health checks).

Operators who want the kill-after behaviour set --recovery-timeout-ms <MS> explicitly. The accepted minimum is 100 ms: a value of 0 would make the reap gate kill every still-running child on the first reap tick (silently neutering recovery), so 0 and any sub-100 ms value are rejected at parse time. The kill is surfaced no faster than one tick after the deadline. The never-kill default is reached by omitting the flag, never by 0.

Concurrency model

• The Recovery struct is owned exclusively by the binary’s poll loop. It is !Send by virtue of holding std::process::Child values, which is fine since the observer is single-threaded.
• No locks anywhere on the recovery path.
• Debounce is per-pid; a repeat stall inside the debounce window returns Debounced regardless of whether a child is still outstanding.

Recovery child environment policy

Recovery subprocesses run with an isolated environment by default: the inherited observer environment is wiped, and the child only sees PATH=/usr/bin:/bin plus any explicit --recovery-env KEY=VALUE entries.

Rationale: observers typically run with secrets in their process environment — AWS_*, GOOGLE_APPLICATION_CREDENTIALS, OAuth bearer tokens, database URLs, Vault tokens. Inheriting that environment into a recovery child means any recovery template (or any binary on the recovery allowlist) becomes a credential-exfiltration vector. The blast radius is catastrophic and silent. The observer default-clears.

Configuration matrix:

Operators whose recovery templates relied on inherited variables (e.g. $HOME for log paths) have two options:

1. Preferred — allowlist explicitly: --recovery-env HOME=/var/log/varta.
2. Escape hatch — full inheritance: pass --recovery-inherit-env. The observer emits a one-shot stderr warning at startup naming the risk so the choice is visible in SIEM/syslog audit trails.

Enforcement is centralised in Recovery::apply_env (recovery/mod.rs); all exec-mode children flow through it.

Out of scope

• varta-vlp — frame ABI is frozen.
• varta-client — no agent-side change.
• Observer poll cadence — still 100 ms read timeout.
• Exporter line schema.
• Panic-handler feature.

See also

• Stall Detection & Liveness — how the observer surfaces Event::Stall in the first place.
• Bounded Collections — the static-allocation proof for OutstandingTable and Tracker.
• Deployment Ceiling & Sharding — what 4096 means in practice and how to scale past it.
• Audit Logging — every recovery decision (Spawned / Debounced / Refused / SpawnFailed / Skipped / Reaped / Killed / stale-child kill failure) emits a TSV record.

Shell-Mode Recovery Removal

Shell-mode recovery (RecoveryMode::Shell, --recovery-cmd, --recovery-cmd-file, --i-accept-shell-risk, Cargo feature unsafe-shell-recovery) has been permanently removed from all build profiles of varta-watch.

Rationale

The shell path (/bin/sh -c <template>) was a shell-injection surface. Even with the two-layer gate (compile feature + runtime flag), the presence of a /bin/sh invocation in the observer binary represents an unnecessary RCE vector. The exec path (--recovery-exec) subsumes every legitimate use case without /bin/sh involvement.

Specific risks that motivated the removal:

1. Shell injection — a template containing $1 and constructed from any operator-controlled input (config file, environment variable) can be weaponised to execute arbitrary commands with the observer’s authority.
2. Hardened container incompatibility — containers built with no-new-privs or seccomp profiles that block execve("/bin/sh", ...) would silently fail recovery without any error surfaced to the operator.
3. ABI assumption — the path /bin/sh is a POSIX assumption but not a guarantee. Musl-based or busybox-minimal images may place the shell elsewhere or omit it entirely.
4. Strings-audit regression — the presence of /bin/sh in the binary caused the Class-A profile strings audit to require a feature-conditional exemption. Removal makes the audit unconditional.

Migration guide

Before

varta-watch \
  --socket /run/varta/agents.sock \
  --threshold-ms 5000 \
  --recovery-cmd "systemctl restart myapp-\$1" \
  --i-accept-shell-risk

After

varta-watch \
  --socket /run/varta/agents.sock \
  --threshold-ms 5000 \
  --recovery-exec /usr/local/bin/restart-myapp

Where /usr/local/bin/restart-myapp is a script or binary that receives the stalled pid as $1 (passed directly by the observer, not via shell expansion).

Using a wrapper script

If your recovery logic requires shell features (pipes, conditionals), write a thin wrapper script and invoke it via --recovery-exec:

#!/bin/sh
# /usr/local/bin/varta-recovery
set -euo pipefail
PID="$1"
echo "stall detected for pid $PID" >> /var/log/varta-recovery.log
systemctl restart "myapp-${PID}"

varta-watch \
  --socket /run/varta/agents.sock \
  --threshold-ms 5000 \
  --recovery-exec /usr/local/bin/varta-recovery

The shell is now invoked once, inside a dedicated file that is under version control and auditable — not expanded from a command-line string at runtime.

Compile-time enforcement

Passing any of the removed flags now produces a hard error at startup:

error: --recovery-cmd has been removed; use --recovery-exec instead

This ensures operators whose deployment scripts still reference the old flags are notified immediately rather than silently running without recovery.

Cross-references

• Safety profiles — how /bin/sh absence is audited
• Peer authentication — why exec-only recovery is consistent with the KernelAttested recovery gate

Peer Authentication

Varta’s observer trusts the kernel, not the wire. Two layers of defence in-depth ensure that process identity cannot be spoofed by anything that can reach the Unix Domain Socket.

Layer 1: socket file permissions (--socket-mode)

During bind(2), the observer temporarily narrows its process umask so the socket file is created as 0600 by default (owner read and write only). It does not apply the mode with a post-bind pathname chmod(2); that avoids a time-of-check/time-of-use window where a replaceable parent directory could redirect the chmod to another file. The socket parent itself must be a real directory rather than a symlink, must be owned by the observer or root, and group/other-writable parents are accepted only when the sticky bit is set (/tmp-style semantics). Only processes running under the same UID as the observer can connect(2) to the default socket.

Layer 2: kernel credential verification

Linux

The observer sets SO_PASSCRED on the socket after binding. Every recvmsg(2) call then receives a SCM_CREDENTIALS ancillary message containing a struct ucred { pid, uid, gid } populated by the kernel. The observer compares ucred.pid against frame.pid from the VLP wire format. If they disagree the frame is silently dropped and varta_frame_auth_failures_total is incremented. The ucred.uid field is implicitly trusted by Layer 1 (--socket-mode 0600 already restricts access to the owning UID), but could be checked as a fail-safe if a permission bypass is ever discovered.

macOS

macOS exposes LOCAL_PEERTOKEN / LOCAL_PEERPID for connected local sockets, but Varta’s production UDS transport is a pathname SOCK_DGRAM observer socket receiving from unconnected clients. On that socket shape, Darwin returns ENOTCONN for the peer-token and peer-pid queries after recvmsg(2), so the observer cannot bind frame.pid to a kernel-attested sender.

The implementation still attempts the Darwin peer queries so connected test fixtures catch constant or ABI drift, but pathname UDS beats resolve to the PID-0 sentinel and are tagged BeatOrigin::SocketModeOnly. Recovery is intentionally refused for those beats.

The Darwin queries are:

1. LOCAL_PEERPID (0x0002) — returns the peer’s PID directly.
2. LOCAL_PEERCRED (0x0001) — returns a struct xucred with the peer’s UID in cr_uid.
3. LOCAL_PEERTOKEN (0x0006) — returns an audit_token_t containing the peer’s PID and effective UID on connected local sockets.

FreeBSD

On FreeBSD, the observer sets LOCAL_CREDS_PERSISTENT on the socket (value 0x0003, at SOL_LOCAL). Every recvmsg(2) then receives a SCM_CREDS2 ancillary message (type 0x08) containing a struct sockcred2 { sc_version, sc_pid, sc_uid, sc_euid, sc_gid, ... } populated by the kernel. The observer accepts version 0, extracts sc_pid and sc_euid, and performs the same PID + UID verification as on Linux.

Plain FreeBSD LOCAL_CREDS is intentionally not used: it emits struct sockcred, which does not contain a sender PID and therefore cannot enforce frame.pid == peer_pid.

DragonFly BSD

On DragonFly, the observer sets SO_PASSCRED on the socket (value 0x4000, at SOL_SOCKET). Every recvmsg(2) then receives a SCM_CREDS ancillary message (type 0x03) containing a struct cmsgcred { cmcred_pid, cmcred_uid, cmcred_euid, cmcred_gid, ... } populated by the kernel. The observer extracts cmcred_pid and cmcred_euid and performs the same PID + UID verification as on Linux.

NetBSD

NetBSD uses the same LOCAL_CREDS / SCM_CREDS concept, but not the FreeBSD ABI. The observer enables LOCAL_CREDS with value 0x0004; the kernel delivers SCM_CREDS type 0x10 containing a struct sockcred. The observer extracts sc_pid and sc_euid and performs the same PID + UID verification as on Linux.

The ancillary buffer is sized at 256 bytes — sufficient for FreeBSD’s 32-byte minimum sockcred2, DragonFly’s 84-byte cmsgcred, and NetBSD’s 28-byte minimum sockcred, with generous headroom for future kernel extensions.

illumos / Solaris

On illumos and Solaris the observer sets SO_RECVUCRED (value 0x0400, level SOL_SOCKET = 0xffff) on the socket. Every recvmsg(2) then receives a SCM_UCRED ancillary message whose payload is an opaque ucred_t. The layout of ucred_t is not a stable ABI contract, so field extraction goes through the ucred_getpid(3C) and ucred_geteuid(3C) accessor functions from the system’s C library rather than a direct struct cast.

This is a genuine per-datagram credential mechanism: the kernel attests the sender’s identity on each recvmsg(2), not just at connection time. The ancillary buffer is sized at 1 024 bytes to accommodate the additional audit and MAC-label attributes that illumos attaches under Trusted Solaris / labelled-zone configurations.

ucred_t lifetime invariant. Both ucred_getpid(3C) and ucred_geteuid(3C) are called inside extract_pid_uid while the cmsg buffer is still on the recv_authenticated stack frame. Neither call stores the pointer past its return; the buffer is released after the call completes.

Zone isolation. ucred_getzoneid(3C) is available for extracting the Solaris zone-id of the sender (analogous to the Linux PID-namespace inode). Cross-zone detection is a planned follow-up; for now, peer_pid_ns_inode is None on illumos/Solaris, which disables the cross-container recovery refusal. See book/src/architecture/namespaces.md for the planned gate.

Platform support summary

¹ Recycle-unverifiable recovery. These platforms attest the sender PID per datagram (minting KernelAttested, so recovery is eligible) but expose no /proc/<pid>/stat, so read_pid_start_time returns None and no start-time generation token can be pinned. A PID recycled inside the per-tick spawn-budget deferral window therefore cannot be distinguished from the original process. Rather than risk firing kill(2)/restart against an innocent recycled PID, the observer’s deferred-stall freshness re-check withholds recovery for a KernelAttested stall that carries no generation, surfacing it as varta_recovery_outcomes_total{outcome="skipped_stall_unverifiable"}. The immediate (same-tick) recovery path is unaffected — it has no deferral window for a recycle to occur in. On Linux a KernelAttested slot always carries a generation, so this skip never fires there.

² BSD field-validation status. The FreeBSD, DragonFly, and NetBSD constants and credential layouts are guarded by compile-time checks and fabricated-buffer tests, but the project has not yet run live varta-watch instances on those kernels against real kernel-delivered credentials after the latest ABI split. Treat the targets as ABI-verified and not yet field-validated until each host signs off one accepted beat.

Socket-mode-only fallback

On platforms that do not provide per-datagram kernel credential passing for Varta’s pathname UDS transport (currently: macOS, OpenBSD, and any other Unix not listed above), varta-watch falls back to a plain recv(2) receive path and tags each beat with BeatOrigin::SocketModeOnly.

Trust model for SocketModeOnly beats. The only defence is filesystem permissions (--socket-mode 0600, the default). Any process running under the same UID as the observer can reach the socket and forge frame.pid. Recovery commands must not and do not fire for these beats — the recovery gate at recovery.rs::on_stall refuses with RecoveryOutcome::RefusedSocketModeOnly and increments the Prometheus counter varta_recovery_refused_total{reason="socket_mode_only"}. The audit log records the refusal with reason=socket_mode_only.

At startup, varta-watch emits a warning on stderr when compiled for a socket-mode-only platform:

[WARN] per-datagram PID verification is unavailable on this platform.
The only defence is --socket-mode (default 0600). Any process under the
same UID can impersonate any PID. Beats will be tagged SocketModeOnly;
recovery commands will not fire.

UDP transport authentication

For network-based agents that emit beats over UDP, the trust model is cryptographic, not kernel-attested. UDP has no peer-credential mechanism on any platform — recvmsg(2) cannot tell the observer who sent a datagram, only where it claims to be from. Varta therefore requires authentication at the AEAD layer, and refuses to bind an unauthenticated UDP listener without two layers of explicit opt-in.

Compile-time features (crates/varta-watch/Cargo.toml)

A build that does not include unsafe-plaintext-udp cannot link the plaintext path at all. Passing --udp-port without keys to such a build hard-errors at startup; there is no warn-and-continue path.

Runtime selection rules

When --udp-port is set, the observer chooses exactly one listener:

1. If --features secure-udp is compiled in and --key-file, --accepted-key-file, or --master-key-file resolve to usable key material, bind SecureUdpListener.
2. Otherwise, only the plaintext path remains. It is bound only if both --features unsafe-plaintext-udp is compiled in and --i-accept-plaintext-udp was passed on the command line.
3. Any other configuration is a hard error (InvalidInput).

When the plaintext path is taken, a high-visibility varta_warn! is emitted at startup naming the bound address, so the choice appears in SIEM / syslog logs:

UDP on <addr> is running WITHOUT authentication (--i-accept-plaintext-udp). Any device with network reach to this port can inject heartbeats, suppress stall detection, or trigger false recovery commands. NOT for production / safety-critical use.

--i-accept-plaintext-udp is intentionally verbose: an operator who types it is making an explicit statement that this build is for development or testing, not for a hospital VLAN.

Why no kernel-level UDP credentials

Unix Domain Sockets can carry SCM_CREDENTIALS / SCM_CREDS / SCM_CREDS2 per-datagram on supported kernels. UDP carries none of those. Even on a single host where --udp-bind-addr 127.0.0.1 is used, any local process can send to that port — there is no equivalent of --socket-mode 0600 for network sockets. AEAD is the only durable defence.

Recovery eligibility and transport-origin gating

Recovery commands (--recovery-exec and --recovery-exec-file) take the stalled agent’s frame.pid and substitute it into the spawned process (kill -9 {pid}, systemctl restart agent@{pid}.service, etc.). That makes recovery a privileged action that targets an arbitrary process by id — and means the wire-level frame.pid must be tied back to the real sending process, not just to whoever holds an AEAD key.

The trust invariant

A recovery command MUST NEVER fire for a pid whose beat lifetime is not kernel-attested. In practice that means:

Internally each beat is tagged with a BeatOrigin (KernelAttested, OperatorAttestedTransport, SocketModeOnly, or NetworkUnverified). The tracker handles origin races monotonically by trust: a stronger origin can replace a weaker preemption attempt for the same pid, while weaker conflicting beats are rejected as Event::OriginConflict (counter: varta_origin_conflict_total). This prevents an attacker on an untrusted transport from pinning a pid before a kernel-attested agent can prove liveness.

Two-layer enforcement

1. Startup hard-error. If any --recovery-exec / --recovery-exec-file is configured and --udp-port is set, the daemon refuses to start with ConfigError::RecoveryRequiresAuthenticatedTransport. To proceed the operator must pass the transport-qualified accept flag for the listener in play — --secure-udp-i-accept-recovery-on-unauthenticated-transport for a secure-UDP listener, or --plaintext-udp-i-accept-recovery-on-unauthenticated-transport for a plaintext one. The flag is verbose by design (matches the --i-accept-<risk> convention) and shows up in cargo tree / startup banners.

2. Runtime origin gate. Recovery::on_stall spawns the recovery command only when the stalled slot’s pinned origin is KernelAttested or OperatorAttestedTransport; NetworkUnverified and SocketModeOnly origins are always refused. The transport-qualified accept flag from step 1 is exactly what stamps the listener’s beats as OperatorAttestedTransport — so the single flag that clears the startup hard-error is also what makes the runtime gate pass; there is no separate runtime opt-in. A refused UDP stall returns the typed RecoveryOutcome::RefusedUnauthenticatedSource { pid }, increments varta_recovery_refused_total{reason="unauthenticated_transport"}, and emits a structured refused record into the recovery audit log (--recovery-audit-file).

Why secure-UDP isn’t enough

The secure-UDP master-key mode binds frame.pid to the 4-byte PID prefix in iv_random[0..4] and derives a per-agent key from the master key. That is a useful cryptographic binding for the UDP threat model — a holder of a single derived agent key cannot forge frames for other pids. But the binding lives at the protocol layer, not at the kernel layer:

• A leak of the shared key lets anyone forge any pid.
• A leak of the master key lets anyone derive any agent key.
• A leak of any per-agent key still lets that agent forge its own pid to misbehave (e.g. stop sending → trigger recovery against its own pid during legitimate maintenance windows).

Kernel attestation has no such failure mode: the kernel knows which process owns the socket fd, and that knowledge cannot be forged by any amount of key material. This is why Varta never grants UDP variants (plain or secure) recovery eligibility on its own: UDP beats default to NetworkUnverified (recovery refused), and the only way to make them recovery-eligible is for the operator to explicitly vouch for the transport with the accept flag, which stamps the beats OperatorAttestedTransport — an operator attestation, never a kernel one.

Recovery command authentication boundary

--recovery-exec and --recovery-exec-file invoke the program directly via execvp(2) — no shell, no metacharacter interpretation, no injection surface. The stalled pid is appended as the final argument: never interpolated into a command string.

Shell-mode recovery (--recovery-cmd / --recovery-cmd-file) was permanently removed. All builds — SRE, Class-A, and default — use exec-only recovery. No opt-in flag is required; exec-mode is the only available recovery mode.

Prometheus /metrics endpoint exposure

The /metrics endpoint is HTTP/1.0 with mandatory bearer-token authentication. When --prom-addr is set, --prom-token-file is required: the observer refuses to start without it. Every scrape must send Authorization: Bearer <hex> where <hex> is the lowercase 64-byte hex form of the file’s 32 random bytes (the format produced by openssl rand -hex 32). Missing or wrong tokens get HTTP/1.0 401 Unauthorized and bump varta_prom_auth_failures_total.

The token file is loaded through the same hardened validator that guards --key-file (see “Secret-file validation” below): regular file, no symlinks, owned by the observer UID, mode 0o600 or stricter, opened with O_NOFOLLOW.

The endpoint also retains four DoS-protection layers from earlier work, so that a hostile scraper cannot exhaust file descriptors or starve the observer’s poll loop even before the auth check runs:

1. Serve budget — at most PROM_MAX_CONNECTIONS_PER_SERVE=8 accepted connections per outer poll tick, and a 100 ms wall-clock deadline.
2. Drain budget — after the serve budget is exhausted, an additional PROM_MAX_DRAIN_PER_SERVE=50 connections may be accepted and immediately closed, so the kernel accept queue does not back up.
3. Per-source-IP token bucket — every accepted connection (in both serve and drain phases) decrements a per-IP token bucket sized by --prom-rate-limit-burst (default 10) and refilled at --prom-rate-limit-per-sec (default 5). Connections from an IP whose bucket is empty are closed without serving and counted as varta_prom_connections_dropped_total{reason="rate_limit"}.
4. Per-IP table cap — the per-IP map is bounded to 1024 entries; when full, stale entries (no activity in 60 s) are evicted first, then if necessary the oldest entry is force-evicted and counted as varta_prom_connections_dropped_total{reason="ip_table_full"}.

Token comparison is constant-time

The exporter compares the presented and expected tokens via varta_vlp::ct_eq — the same constant-time XOR-and-OR routine that guards Poly1305 tag verification. This prevents byte-by-byte timing oracles from leaking the prefix of the token to a remote scraper.

Bind-address recommendation

The bearer token is the authoritative authentication boundary. Loopback bind (127.0.0.1:<port> or [::1]:<port>) behind a reverse proxy remains the recommended posture for defense in depth, but is no longer the only defense. The observer still emits a startup varta_warn! whenever the bound address is non-loopback, so the exposure is visible in audit logs.

Prometheus scrape config

The standard authorization: block injects the bearer token verbatim:

scrape_configs:
  - job_name: 'varta'
    static_configs:
      - targets: ['varta-host:9100']
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/varta-prom.token

The credentials_file should be the same content as --prom-token-file on the observer; Prometheus reads it with the same 0600-or-stricter expectation.

Secret-file validation

Every file containing key material — --key-file, --accepted-key-file, --master-key-file, and the new --prom-token-file — flows through validate_secret_file in varta-watch/src/config.rs. The validator enforces:

1. The path is not a symlink (symlink_metadata + is_symlink).
2. The path resolves to a regular file (not a directory, FIFO, block/char device, etc.).
3. The mode is 0o600 or stricter (mode & 0o077 == 0).
4. The file is owned by the observer’s UID (kernel-attested via stat.uid, not derived from the env).
5. The file is opened with O_NOFOLLOW to close the TOCTOU window between the metadata check and the read.

A failure on any of these aborts startup with a typed ConfigError naming the failing constraint (insecure permissions ..., must not be a symlink, owned by uid X, expected uid Y, etc.).

Why environment-variable keys are gone

Earlier releases offered --key-env <NAME> as a key-source fallback. That flag is removed. Passing it now returns ConfigError::RemovedFlag with an inline migration hint pointing at --key-file. The motivation:

• On Linux, /proc/<pid>/environ is readable by any process running under the same UID; a peer with a UDS connection to the observer (which already has UID-restricted access) can read the master key out of the observer’s own environment.
• In containers, docker inspect <container> exposes every environment variable to anyone with read access to the Docker socket — typically all members of the docker group, which is often a superset of the in-container UID.
• systemd-journald captures process environment on demand for crash reports; an env-var key ends up in /var/log/journal indefinitely.

File-based keys avoid all three exposures and slot into the same ownership/permission model as TLS private keys, SSH host keys, and any other long-lived secret an operator already knows how to manage.

The Key type in varta_vlp::crypto also lost its Copy derive and gained a Drop impl that volatile-zeros the secret bytes before the allocation is returned to the stack, closing a small but real leak surface in core dumps and ASLR-defeated speculative reads.

Shutdown grace and systemd

--shutdown-grace-ms (default 5000, range 100..60000) bounds the time Recovery::drop blocks waiting for outstanding recovery children to exit after issuing SIGKILL during shutdown. Children that outlive the grace are abandoned to PID 1 for reaping; the observer process exits either way, so the bound on shutdown latency is deterministic.

In a systemd unit, TimeoutStopSec must be at least shutdown_grace_ms + 2 s (roughly: grace + reap margin) to ensure that systemd does not SIGKILL the observer mid-grace and leak an unreaped recovery child:

[Service]
Environment=VARTA_SHUTDOWN_GRACE_MS=5000
ExecStart=/usr/local/bin/varta-watch --shutdown-grace-ms ${VARTA_SHUTDOWN_GRACE_MS} ...
TimeoutStopSec=7s
KillMode=mixed

KillMode=mixed is recommended: systemd sends SIGTERM to the main observer process only; the observer then runs its own Drop sequence to kill+reap any recovery children it had spawned. This is what the shutdown-grace tunable is designed around.

Recovery command environment isolation

When --recovery-env KEY=VALUE is specified (repeatable), the recovery child process runs with a sanitized environment:

1. The child’s environment is cleared entirely.
2. PATH is set to /usr/bin:/bin (sufficient to locate common tools).
3. Only the explicitly-listed KEY=VALUE pairs are exported.

Without --recovery-env, the child inherits PATH=/usr/bin:/bin only (secure default since 2026-05-14). This eliminates LD_PRELOAD, IFS, and other environment-injection vectors from recovery children entirely.

Exec safety

The {pid} substitution in --recovery-exec args is safe: a u32 PID formatted as a decimal string contains only the characters 0–9 and can never carry shell metacharacters (;, |, &, $, `, etc.). Furthermore, since exec-mode never passes arguments through a shell, metacharacter interpretation is structurally impossible.

Metrics

Trust model summary

 Process ── connect(2) to UDS ──┐
                                   ├─ [FAIL]  Kernel blocks (Layer 1: --socket-mode 0600, wrong UID)
                                   ├─ [PASS]  Layer 2: SO_PASSCRED → ucred.pid (Linux)
                                   │          Layer 2: LOCAL_CREDS_PERSISTENT → sockcred2.sc_pid (FreeBSD)
                                   │          Layer 2: SO_PASSCRED → cmsgcred.pid (DragonFly)
                                   │          Layer 2: LOCAL_CREDS → sockcred.sc_pid (NetBSD)
                                   │          ├─ [PID MISMATCH] → Drop frame + bump counter
                                   │          ├─ [UID MISMATCH] → Drop frame as IoError
                                   │          └─ [PID MATCH + UID MATCH] →
                                   ↓
                              [SUCCESS]  Observer trusts the PID → tracks,
                                         surfaces stalls, triggers --recovery-exec
                                         with {pid} as the final argument.

The trust boundary is the kernel: a frame is only accepted if the kernel attests that the sending process’s PID matches the one encoded in the VLP frame and that the sending process runs under the observer’s UID. On Linux this is enforced per-datagram via SO_PASSCRED; on FreeBSD via LOCAL_CREDS_PERSISTENT + SCM_CREDS2 + struct sockcred2; on DragonFly via SO_PASSCRED + SCM_CREDS + struct cmsgcred; on NetBSD via LOCAL_CREDS + SCM_CREDS + struct sockcred. macOS pathname datagram sockets and platforms without kernel-level credential passing fall back to --socket-mode 0600.

Security limitations

No forward secrecy

The KDF derives per-agent and per-epoch keys from a single master key. An epoch key can decrypt frames from past epochs if the agent key is compromised. True forward secrecy requires bidirectional ephemeral key exchange (e.g. X25519), which is incompatible with the connectionless, one-way heartbeat model.

When the master key is rotated, all agents must be updated atomically. The observer reads the master key once at startup from --master-key-file. To rotate keys, restart the observer with the new master key file. SIGHUP-based hot-reload is planned for a future release.

Panic-hook entropy policy (secure UDP)

install_panic_handler_secure_udp reads all IV material at install time (getrandom(2) on Linux, getentropy(3) on macOS/BSD, falling back to /dev/urandom): an 8-byte prefix for the installing process plus a 16-byte salt for forked children. If a forked child panics, the hook derives a child-specific IV prefix with HKDF-SHA256 over the pre-read salt, PID, timestamp, and AEAD counter. No file I/O or OS entropy call occurs inside the panic handler itself (async-signal-safety).

Fail-closed default: if the entropy chain fails — common in chrooted environments without a mounted /dev — the function returns Err(PanicInstallError::EntropyUnavailable) and the hook is NOT registered. This prevents a panic-time Critical frame from reusing a deterministic IV under the same AEAD key, which would be a catastrophic nonce-reuse failure.

Degraded-entropy opt-in: use install_panic_handler_secure_udp_accept_degraded_entropy to fall back to a non-cryptographic IV prefix and fork salt derived from PID, TID, monotonic time, and counters (SipHash-2-4). This always succeeds but accepts nonce-reuse risk if the fallback inputs collide. The verbose function name is intentional structural enforcement matching the project’s --i-accept-<risk> convention.

Little-endian only

The VLP wire format uses little-endian integer encoding natively. Protocol correctness depends on the host being little-endian (all tier-1 targets — x86_64 and aarch64 — satisfy this). Building on a big-endian host is a compile error. See book/src/architecture/vlp-frame.md for design rationale.

Panic-hook key lifetime — accepted residual

The secure-UDP panic handler (install_panic_handler_secure_udp, install_panic_handler_secure_udp_accept_degraded_entropy) captures a Key by move into a Box<dyn Fn> registered via std::panic::set_hook. The Box is the single owner of the captured Key for the lifetime of the process — Key is !Clone (see crates/varta-vlp/src/crypto/mod.rs), so no duplicate of the secret bytes can exist anywhere else in the address space.

The !Clone invariant pins the count of in-memory copies to one. The remaining concern is the lifetime of that one copy on process exit:

• Normal hook replacement (std::panic::take_hook): the prior Box is dropped, the captured Key’s ZeroizeOnDrop fires, and the 32 secret bytes are wiped before the heap page is returned to the allocator. OK.
• panic = "unwind" profile, normal process exit: the panic-hook Box is leaked by the runtime — Drop is not called on registry-held objects at exit. The captured Key bytes persist in heap memory until the kernel reclaims the page. Linux does not zero pages on reclaim (memory contents are reused; zero-on-allocation guarantees apply only to new allocations into the same process).
• panic = "abort" profile: the panic-hook closure never runs, but set_hook still owns the Box — same residual as the normal-exit case. Additionally, no Drop runs anywhere during abort().

This residual is accepted: there is no async-signal-safe mechanism that can reliably wipe a heap-resident secret at process exit. atexit handlers do not run on abort(), are not async-signal-safe, and race the panic hook firing. mlock / memfd_secret cannot prevent the kernel from copying the page during scheduler context switches or core dumps. The minimum-surface design is to keep the captured Key alive in a single Box and treat the OS process boundary as the security boundary: inspecting the memory of a live process requires ptrace or /proc/<pid>/mem privileges, at which point all in-memory secrets in any design are accessible.

Cross-references

• Safety profiles — compile-time feature gating for dangerous recovery paths; production-safe build verification recipe
• Observer liveness — defending against varta-watch itself crashing or hanging
• VLP transports — transport-level trust classification and BeatOrigin semantics

PID-namespace semantics

Varta agents and the varta-watch observer can run on the same host but in different Linux PID namespaces (typical when agents run in containers and the observer on the host, or vice-versa). This document defines what the protocol does in that case, why, and how operators configure it.

Problem statement

std::process::id() (called by Varta::beat()) returns the agent’s PID in the calling process’s PID namespace (see pid_namespaces(7)). The observer’s kernel-attested peer PID (SO_PASSCRED / SCM_CREDS / SCM_UCRED) is the PID as seen from the observer’s namespace.

Two consequences when namespaces differ:

1. The numeric pid is meaningless across the boundary. PID 17 in container A is a different process from PID 17 on the host. kill(2) against PID 17 in the observer’s namespace targets the observer-namespace process, not the agent.
2. Collisions are guaranteed. Every container’s first process is PID 1. Two containerized agents binding the same observer socket will both claim PID 1.

Threat model

Detection

On Linux, every process’s PID namespace has a unique inode exposed at /proc/<pid>/ns/pid (stat(1) it, or readlink(1) for the canonical pid:[NNNN] form). Two processes share a PID namespace iff their /proc/<pid>/ns/pid symlinks resolve to the same inode.

varta-watch caches its own inode at startup (crate::peer_cred::observer_pid_namespace_inode()) and, for every kernel-attested beat, reads the peer’s inode (crate::peer_cred::read_pid_namespace_inode(peer_pid)). Both helpers are allocation-free; the per-beat read is one readlink(2) syscall into a stack buffer (sub-microsecond on modern Linux).

Non-Linux platforms (macOS, BSD) return None from both helpers and the comparison short-circuits to “match”. UDP listeners set peer_pid_ns_inode = None because there is no kernel attestation; the existing UDP recovery refusal gate is the relevant protection there.

PID recycling within a namespace (generation token)

A PID is not a stable identity even within one namespace: the kernel recycles it once the holding process exits. If agent A (PID 1234) dies and the OS reuses 1234 for a fresh agent B, B’s first beat carries nonce = 1 while the observer’s slot for PID 1234 still holds A’s high-water nonce. Without extra signal, B’s low-nonce beats are rejected as out-of-order, the slot’s last_ns freezes, a false stall fires, and recovery is misdirected against the healthy new process — all with no attacker involved. On Linux UDS (KernelAttested) recovery is permitted, so this can kill or restart an unrelated bystander; on macOS UDS (SocketModeOnly) and all UDP (NetworkUnverified) recovery is already refused, so the residual there is limited to monitoring accuracy.

The fix binds slot identity to (pid, generation), where generation is the kernel-attested process start-time — field 22 (starttime) of /proc/<pid>/stat, read via crate::peer_cred::read_pid_start_time(peer_pid) (allocation-free, one open/read/close into a stack buffer, parsed from the last ) so a comm containing spaces or parentheses cannot fool it). Two processes that share a PID value cannot share a start-time, because the first holds the PID until it exits.

When a beat for an already-tracked pid carries a different Some(_) generation, the slot is reset to a fresh agent (nonce baseline, origin, namespace inode, and silence timer all re-pinned) and the event is counted as varta_tracker_pid_recycle_total. The generation check runs before the origin / namespace / nonce checks — a recycled process legitimately differs on all of them. A None on either side (“generation unknown”: non-Linux, UDP, or unreadable /proc) is treated leniently by the tracker and never triggers a reset, so prior PID-only behaviour is preserved exactly for non-recovery transports and for already-pinned slots whose peer vanished. When the slot’s pinned generation is None and a later beat carries Some(_) with an accepted nonce, the token is pinned in place (same rule as the namespace-inode None → Some upgrade) so a subsequent recycle can compare (Some(G1), Some(G2)) instead of staying stuck at None. Out-of-order frames must not pin generation. Replay protection is untouched: a low nonce under the same generation is still dropped as out-of-order.

For Linux UDS first contact, the observer requires that Some(generation) before the slot may pin recovery-eligible KernelAttested origin. If the sender exits after recvmsg(2) but before the /proc/<pid>/stat start-time read, the beat is still observable, but it is recorded as SocketModeOnly. A later accepted beat that can read a concrete generation may upgrade the slot to KernelAttested. This keeps transient dying-gasp frames from breaking monitoring while preventing an unpinned numeric PID from driving {pid} recovery after PID recycle.

The same generation is revalidated when a kernel-attested slot first becomes a stall candidate. If /proc/<pid>/stat now returns a concrete different generation, the old slot is retired and no Event::Stall is emitted. This closes the silent-death case where no new beat arrives to trigger the beat-time recycle gate: a dead agent’s stale KernelAttested origin must not drive recovery against a healthy process that inherited the PID. A missing generation read at stall time remains fail-open, because it may simply mean the original agent exited and recovery should still restart it.

Cost. Beat-time recycle detection requires re-reading the generation on every admitted KernelAttested beat — there is no way to observe PID reuse without re-stat-ing the peer. This adds one /proc/<pid>/stat open/read/close (three syscalls, allocation-free) per beat, on top of the existing /proc/<pid>/ns/pid namespace read. The read is deferred until after the global rate limiter admits the frame, so a flood cannot force a /proc read per packet. Stall-time revalidation adds the same stack-buffered read only for kernel-attested slots that have already crossed the stall threshold. Non-Linux and non-attested transports skip the read entirely (read_pid_start_time returns None).

Mitigation by deployment style

Audit and metrics inventory

All counters are emitted at every scrape even at zero, so absent() alert rules stay green-on-green until the first event.

API surface

• Observer::observer_pid_namespace_inode() -> Option<u64> — returns the observer’s cached PID-namespace inode (Linux only).
• Observer::with_allow_cross_namespace(bool) -> Self — opt out of the default refuse-and-audit behaviour. Wired from --allow-cross-namespace-agents.
• Observer::drain_cross_namespace_drops() -> u64 — counter drain.
• Observer::drain_namespace_conflicts() -> u64 — counter drain.
• Observer::drain_pid_recycles() -> u64 — counter drain (PID-recycle slot resets/retirements).
• Tracker::record_with_generation(frame, now_ns, threshold_ns, origin, peer_pid_ns_inode, peer_generation) — the generation-aware record path; Tracker::record(..) is a shim passing peer_generation = None.
• Tracker::pid_ns_inode_of(pid: u32) -> Option<Option<u64>> — observer-side introspection.
• Recovery::with_allow_cross_namespace(bool) -> Self — same opt-out at the recovery layer.
• Recovery::on_stall(pid, origin, cross_namespace_agent: bool) — caller-supplied cross-ns flag (typically derived from Event::Stall::pid_ns_inode vs Observer::observer_pid_namespace_inode()).
• Recovery::take_refused_cross_namespace() -> u64 — counter drain.
• RecoveryOutcome::RefusedCrossNamespace { pid } — refusal variant.

CLI flags

--allow-cross-namespace-agents   Permit beats and recovery for agents whose
                                 kernel-attested PID namespace differs from
                                 the observer's. Default off — beats dropped
                                 at receive (counted) and recovery refused
                                 (audit + counter).

--strict-namespace-check         Fatal startup error on first cross-namespace
                                 beat. Default off — log + counter only.

Edge cases

• /proc/<peer_pid>/ns/pid unreadable (ptrace_may_access denial, peer exited between recvmsg and readlink, /proc not mounted): the helper returns None. The tracker’s None → Some upgrade allows one-shot recovery so a transient /proc unavailability does not pin a slot as permanently unknown.
• /proc/<peer_pid>/stat unreadable on first contact: the helper returns None, so the beat is tracked as SocketModeOnly until a later accepted Linux UDS beat can pin Some(generation). Missing generation remains fail-open only after a slot already has recovery-eligible identity pinned.
• Existing frame.pid != peer_pid check fires first for most real cross-namespace traffic (the two namespaces almost always produce different numeric pids for the same process). The namespace gate is belt-and-suspenders for the surprising case where the pids happen to collide.
• unsafe_code = "deny" is workspace-wide. The new readlink FFI follows the established peer_cred.rs pattern (extern "C" + one-line unsafe { ... } blocks with a SAFETY comment).
• Frame ABI is unchanged — the 32-byte Frame is not touched. All state lives observer-side.

Cross-references

• vlp-transports.md — overall transport model.
• peer-authentication.md — kernel-attested PID and the BeatOrigin trust classification.
• pid_namespaces(7) and user_namespaces(7) man pages — kernel reference.

Recovery audit log (schema v2)

The recovery audit log (varta-watch/src/audit/) is the canonical forensic record of every recovery action the daemon took or refused. It exists to satisfy three operational requirements:

1. Traceability. For an IEC 62304 Class C device — or an aviation ground-station — every recovery action must be reconstructable after the fact: what was spawned, when, why, with what outcome.
2. Survivability. A power cut on the host must not silently drop the most recent audit records.
3. Tamper-evidence. A reviewer must be able to detect retroactive editing of historical records.

Schema v1 (the pre-2026 format) satisfied only the first of these. Schema v2 — the current format — satisfies all three when the daemon is built with the audit-chain feature.

File format

One file-level header line, then one record per line. Fields are tab-separated. Every record kind carries a leading seq column and a trailing chain column. Free-form fields (program paths, refusal reasons) have their \t, \n, and \r bytes replaced with a single space at write time so a maliciously-chosen argv[0] can never inject columns.

# varta-watch recovery audit v2

boot

seq    wallclock_ms    observer_ns    boot    daemon_pid    prev_chain|-    reason    chain

A boot record opens every audit-log session and every post-rotation generation. The reason column carries one of six stable tokens:

spawn

seq    wallclock_ms    observer_ns    spawn    agent_pid    child_pid    mode    program    source    template_len    chain

Emitted at the moment a recovery child is fork(2) + execvp(2)’d. mode is always "exec" (shell mode was permanently removed); program is argv[0]; source is either the literal "inline" (for --recovery-exec) or the path-string for --recovery-exec-file. The full argv is not logged — it may contain secrets, and the source path is already auditable.

complete

seq    wallclock_ms    observer_ns    complete    agent_pid    child_pid    outcome    exit_code|-    signal|-    duration_ns    stdout_len    stderr_len    truncated    chain

Emitted on reap, kill-after-timeout, or reap failure. outcome is one of reaped, killed, reap_failed. exit_code and signal are mutually exclusive: at most one is a number, the other is -.

refused

seq    wallclock_ms    observer_ns    refused    agent_pid    reason    chain

Emitted when a stall is detected but recovery is structurally declined (e.g. unauthenticated transport, cross-namespace agent). reason is a stable short token so SIEM consumers can alert on it without parsing free text.

Current reason tokens:

Sequencing

seq is a u64 starting at 1 on the first boot record. It is strictly monotonic within a daemon lifetime and across daemon restarts (the new daemon resumes from last_seq + 1 after parsing the existing tail). A consumer detects record loss as a gap: seq[i+1] - seq[i] > 1.

Durability cadence

Every record_* call is followed by BufWriter::flush() and File::sync_data() (= fdatasync(2) on Linux) at a configurable cadence controlled by --recovery-audit-sync-every <N>:

• N = 1 (default, IEC 62304 Class C-conforming): one fdatasync per record.
• N > 1: one fdatasync per N records. The daemon emits a startup warning and the build is not Class C-conforming. Up to N - 1 records can be lost on power cut.
• N = 0: rejected at parse time.

In addition, the daemon unconditionally syncs:

• Before every rotation rename.
• After writing the post-rotation boot record.
• In Drop (best-effort; not load-bearing for correctness).

Directory-entry durability

fsync(2) on the audit file does not persist the directory entry that names it. The daemon therefore also fsyncs the audit file’s parent directory: once at startup in create (a freshly-created file would otherwise vanish entirely on power cut — including records whose fdatasync had already returned), and once per rotation in a dedicated final SyncingDir state-machine stage covering the generation renames, the new live file’s create_new, and the EXDEV fallback’s copy/unlink pair. The stage runs behind its own budget check so the Finalizing tail keeps the exact two-fsync cost its --audit-rotation-budget-ms model is sized for. A directory-fsync failure is a soft durability degradation latched on the audit error channel, mirroring the UDS-bind posture.

File identity hardening

Audit startup opens the live path once with O_NOFOLLOW, verifies the opened inode is a regular file owned by the observer UID with exactly one hard link, then uses that same descriptor for tail recovery, optional truncation, and all later appends. Leaf symlinks and multiply-linked files are rejected before any audit bytes are changed.

Rotation creates every new live generation and EXDEV copy destination exclusively with mode 0600. The EXDEV fallback copies from a clone of the writer’s already-validated descriptor, not from a second pathname open, and keeps the exclusive destination descriptor as the new sink.

Tamper-evidence: the hash chain

When the daemon is built with --features audit-chain, every record’s trailing chain column is the lowercase-hex SHA-256 of:

DOMAIN || 0x00 || kind || 0x00 || prev_chain_raw || 0x00 || body_with_seq

where:

• DOMAIN = b"VARTA-AUDIT-v2". The trailing v2 is the schema version; a future v3 mandatorily bumps this so chains across schemas cannot be confused.
• kind is the bytes b"boot" / b"spawn" / b"complete" / b"refused".
• prev_chain_raw is the raw 32-byte prior chain hash (not its hex form), or [0u8; 32] for the very first record in a fresh file.
• body_with_seq is the TSV line from the seq column up to (but not including) the chain column — no trailing \n.
• Four 0x00 separators prevent field-boundary confusion: e.g. (kind="ab", body="cd") and (kind="abcd", body="") hash to distinct strings.

The construction is implemented once in crates/varta-vlp/src/crypto/hash.rs::audit_chain_hash so callers cannot accidentally drop the domain separation or transpose the input order.

What this detects

• Any byte edited in any historical record. The edited record’s own chain stops matching, and every subsequent chain also stops matching.
• Any record deleted. The chain breaks at the deletion point.
• Any record inserted. Same — the chain over the synthetic record cannot match the next legitimate record.
• Records reordered. The chain validates only in original order.

What this does NOT detect

A pure SHA-256 hash chain — without a secret key — can be recomputed end-to-end by an attacker with write access to the file. Tampering is only detectable when the latest chain head is verified against an externally trusted source. Operators in safety-critical deployments should periodically export tail -1 audit.log | cut -f<last> to a sealed log (Tang, AWS S3 with object-lock, a hardware HSM, etc.). The daemon does not do this — it is an operational policy decision.

A future HMAC-keyed mode is out of scope for v2 to avoid forcing a key-distribution workflow on every Class C deployment.

When audit-chain is disabled

If the daemon is built without --features audit-chain:

• The chain column is the literal string -.
• The daemon emits a startup warning explicitly stating that the build is not IEC 62304 Class C-conforming.
• seq and fdatasync cadence still work — record loss is detectable; power-cut durability is preserved; only tamper-evidence is absent.

The build remains zero-registry-dep (the audit-chain feature propagates the existing optional crypto deps in varta-vlp/crypto).

Rotation

When --recovery-audit-max-bytes <N> is set, the file rotates after any write that pushes it over the threshold: PATH → PATH.1 → … → PATH.5. Five generations are kept; the oldest is unlinked. The same generation count as the event-stream FileExporter.

The chain spans rotation: the first non-header record in the new generation is a boot with reason=rotation whose prev_chain column is the final chain of the just-rotated file. A reviewer who pieces generations together by seq order can replay-verify the chain across the entire history.

Verification recipe

# 1. Confirm seq is strictly monotonic across all generations.
cat audit.log.5 audit.log.4 audit.log.3 audit.log.2 audit.log.1 audit.log \
    | grep -v '^#' \
    | awk -F'\t' 'NR==1 { prev = $1; next } $1 != prev+1 { print "GAP at seq", $1; exit 1 } { prev = $1 }'

# 2. Confirm chain validates (requires the daemon's
# audit_chain_hash helper exposed in a verification tool — out of scope
# for the daemon binary itself, see book/src/architecture/peer-authentication.md
# for the pattern).

# 3. Cross-check that the chain head matches the latest sealed-log entry
# the operator exports to their trusted store.

CLI surface

Durability vs availability

The default configuration is unchanged from Class C semantics: --recovery-audit-sync-every=1 + --audit-sync-interval-ms=0 means every record fsyncs before the drain returns, and --audit-fsync-budget-ms=50 only ever takes effect when a single fsync exceeds 50 ms — i.e. when the disk is already stalling the poll loop. The new flag does not weaken durability for safety-critical operators; it provides the structural guarantee that the poll loop itself cannot block indefinitely on a wedged fsync.

Operators who can accept relaxed durability (e.g. cloud SRE deployments, not safety-critical) set --recovery-audit-sync-every=64 --audit-sync-interval-ms=100 to amortise fsync cost over many records while still pinning a worst-case sync interval.

See observer-liveness.md for the audit-log observability signals and recommended alerts.

Threat model

Compile-time Configuration (Class-A profile)

The Class-A safety-critical profile builds varta-watch with the compile-time-config Cargo feature. In this profile the runtime binary has no argv parser, no Prometheus HTTP exporter, and a single neutral --help body that mentions no flag names. Every operational knob is supplied at compile time by build.rs from a static KEY = VALUE file pointed to by the VARTA_CONFIG_FILE environment variable.

The Class-A binary is verified by the CI safety-profiles job:

B=target/release/varta-watch
strings "$B" | grep -E -- "(GET /metrics|HTTP/1\.|--[a-z])"
# expect: no output

When to use this profile

• Hospital VLAN deployments where every CVE surface is a liability.
• IEC 62304 Class C medical devices (insulin pumps, holter monitors, ventilators) where the host configuration is part of the validated firmware.
• Avionics / industrial-control systems where the binary must boot from a signed image and accept no operator input post-deployment.

For SRE / cloud deployments use the default-feature build (or --features prometheus-exporter for /metrics). The two profiles are mutually exclusive at compile time via a compile_error! guard in crates/varta-watch/src/lib.rs.

Build recipe

export VARTA_CONFIG_FILE=/etc/varta/varta.conf
cargo build -p varta-watch --release \
  --no-default-features --features secure-udp,compile-time-config

secure-udp is the recommended companion feature — Class-A almost always wants authenticated transport. Other features that combine cleanly with compile-time-config: audit-chain, json-log.

The prometheus-exporter feature is forbidden in combination with compile-time-config; cargo build fails with a clear compile_error! diagnostic.

File grammar

Plain text, UTF-8. Lines that begin with # or are entirely whitespace are ignored. Each remaining line is KEY = VALUE:

• The = separator may have any amount of whitespace on either side.
• KEY must be in the KNOWN_KEYS catalogue (see below).
• VALUE is the rest of the line after the first =, trimmed.
• Quoting is not supported — paths and strings are taken verbatim.
• Repeated singleton keys are a build error; repeated list keys (recovery_env) accumulate.
• Unknown keys are a build error that surfaces during cargo build.
• Cross-field safety rules are also build errors: udp_port requires a secure key source, recovery on secure UDP requires i_accept_recovery_on_secure_udp = true, and non-loopback secure UDP requires i_accept_secure_udp_non_loopback = true.

Example:

# /etc/varta/varta.conf

socket = /run/varta/varta.sock
threshold_ms = 5000
socket_mode = 0600

# Recovery: exec-mode only, never shell.
recovery_exec_cmd = /usr/local/sbin/varta-recover {pid}
recovery_audit_file = /var/log/varta/recovery.tsv
recovery_audit_sync_every = 1

# Authenticated UDP listener bound to loopback.
udp_port = 8443
udp_bind_addr = 127.0.0.1
secure_key_file = /etc/varta/agent.key

# Hospital deployment: medical-device clock semantics + strict mode.
clock_source = boottime
strict_namespace_check = true

Accepted keys

Operational contract

• --help (and any other argv) is rejected at startup. The binary exits non-zero with the neutral diagnostic “this binary was configured at compile time; refusing to accept command-line arguments”.
• Diagnostic messages in stderr / sd_notify use neutral wording — no --flag-name strings appear anywhere in the binary. See the cerebrum entry on pub const &str being unconditionally linked for the rationale.
• The configuration file is consumed once, at cargo build time. The resulting binary is immutable: redeployment requires a new build. This is the structural feature operators rely on for Class-A release-gating.

See also

• Safety profiles overview
• Peer authentication — key-file requirements
• Observer liveness — self-watchdog wiring

Safety Profiles

varta-watch ships with a two-layer gate for every structurally-dangerous capability: a compile-time Cargo feature that must be explicitly enabled, AND a runtime flag that must be passed by the operator. Neither layer alone is sufficient; both must be active.

This document defines what “production-safe” means for Varta and how to verify a binary before deploying it to a safety-critical environment.

Profile matrix

The two profiles are mutually exclusive: prometheus-exporter cannot combine with compile-time-config (a compile_error! in crates/varta-watch/src/lib.rs rejects the combination at build time). This is the structural guarantee Class-A builds rest on — the Class-A binary cannot ship with an HTTP server linked in.

Production-safe build

A production-safe varta-watch binary is built with default features only:

cargo build -p varta-watch --release

No --features argument is needed or wanted. Default features are empty.

What is absent from a production-safe build

Shell-mode recovery (/bin/sh -c) has been permanently removed from all build profiles. /bin/sh does not appear in any varta-watch binary, regardless of feature flags. Use --recovery-exec for all recovery configurations.

Without the compile-time feature, the code path is not linked into the binary. A misconfigured deployment cannot accidentally enable the dangerous path at runtime.

Verification recipe

cargo build -p varta-watch --release
strings target/release/varta-watch | grep -F "/bin/sh" && echo "FAIL" || echo "OK"

The strings check is belt-and-suspenders: because /bin/sh is structurally absent, the literal cannot appear in any binary regardless of features.

Unsafe features

unsafe-plaintext-udp

Compiles in the plaintext UdpListener transport. Any device with network access to the bound port can inject heartbeats, suppress stall detection, or trigger false recovery commands.

# varta-watch/Cargo.toml
[features]
unsafe-plaintext-udp = ["udp-core"]

Even with this feature, the listener will not bind unless --i-accept-plaintext-udp is also passed at runtime.

This feature is structurally unavailable in Class-A builds — compile-time-config + unsafe-plaintext-udp is rejected by a compile_error! in crates/varta-watch/src/lib.rs. Mission-critical deployments must use secure-udp (AEAD-authenticated, bounded per-sender replay state) for any UDP transport; plaintext UDP has no replay protection and can be used by a network attacker to suppress stall detection.

Class-A safety-critical features

prometheus-exporter (opt-in HTTP exposition)

The Prometheus /metrics endpoint, the bearer-token loader, the per-IP rate-limit table, and every --prom-* argv flag live behind this feature. When absent the binary contains zero HTTP / TCP-accept code and the only exporter linked is FileExporter (one-way append-only TSV sink — no listener, no network surface).

[features]
prometheus-exporter = []

Verification recipe (default build, feature off):

cargo build -p varta-watch --release
B=target/release/varta-watch
strings "$B" | grep -E -- "(GET /metrics|HTTP/1\.|WWW-Authenticate|Bearer realm)" \
  && echo "FAIL" || echo "OK"

compile-time-config (no argv parser, no runtime config)

Replaces the runtime argv parser with a build-script-generated constant populated from $VARTA_CONFIG_FILE (a KEY = VALUE text file). When the feature is on:

• Config::from_args is excluded from compilation; the 292-arm match block carrying every --flag-name literal is not linked.
• Config::HELP is a neutral one-liner that contains no flag names.
• The binary refuses any argv tokens with CompileTimeArgvForbidden.

Cannot be combined with prometheus-exporter, libc-signal-mode, or unsafe-plaintext-udp — each combination is rejected at compile time by a dedicated compile_error! in lib.rs. The Class-A binary is structurally free of the HTTP exporter, the libc-signal indirection, and the plaintext UDP listener; none of those code paths can be linked in.