Reliability & SRE

Antemure runs one agent through two lenses. The Cyber lens watches for threats and responds with containment; the SRE lens watches infrastructure vitals and service health and responds with recovery — restart, reload, reprovision. Same collectors cycle, same fleet-config push, same gated remediation executor; reliability-specific signals, automations and views.

Flip the Cyber / SRE switch in the header of both the fleet dashboard and the endpoint console. SRE mode re-skins the UI, scopes the AI analyst to reliability, and reveals the reliability panels. All of it is driven from Administration → Fleet config and the dashboard's Reliability tab and pushed to the fleet.

Two lenses over one pipeline

Reliability is not a separate engine. SRE evaluators emit a distinct Reliability event kind; node incidents aren't domain-tagged, so the hub classifies them heuristically (CPU, memory, OOM, disk, latency, saturation, crash-loop, 5xx, SLO, uptime…) and GET /api/fleet/incidents?domain=sre filters on that. The AI analyst switches to an SRE system prompt and SRE-only tools when its scope is sre. Every reliability collector and evaluator is registered unconditionally on the node and is a no-op until you configure something.

Metrics & EWMA baselines

A lightweight, always-on metrics collector samples each cycle — CPU%, memory%, root disk%, network rx/tx rates, per-process disk read/write rates, connection and process counts — and maintains EWMA baselines (so it learns a host's normal) plus a rolling trend window for the graphs. It also carries a blunt sustained-signal tripwire: >85% CPU for several consecutive polls emits a high-CPU signal (the cryptominer case the rate baselines would miss). These machine:metrics gauges are what the SLO, alert and correlation evaluators read.

Custom telemetry probes

Analysts define read-only probes in the hub that are pushed to the fleet and emit a named metric:<name> you can then alert on. There is deliberately no arbitrary-command probe. Kinds:

Kind	Emits
`tcp_port`	1/0 — can connect to `host:port`
`http_status` / `http_up` / `http_latency_ms`	status code / 1 if 2xx–3xx / round-trip ms
`process_count`	number of matching processes
`file_size_bytes` / `path_exists`	bytes (−1 if missing) / 1–0
`disk_free_pct`	0–100 for a mount (default `/`)
`log_match_rate`	count of new lines matching a pattern since last read (rotation-safe, 8 MiB/read cap)

Each probe has an enforced minimum interval of 10 s and a domain (default sre). Manage them under Reliability → Custom telemetry; API GET/POST /api/fleet/collectors.

SLOs

An SLO defines a target on a metric (cpu_pct, mem_pct, disk_pct, net_mbps, connections, processes, disk I/O, or any custom metric:<name>) with a comparator (lt = healthy below / gt = healthy above), a warn_at headroom fraction, a sustained-breach window_secs, a severity and an optional node scope. The node evaluator records per-SLO state (ok / warn / breach) for the fleet mesh and, on a sustained breach, emits a Reliability event scored by severity. API: hub GET/POST/DELETE /api/fleet/slos, daemon GET /api/slos.

SLOs and the alert warn band overlap: both express “how close to the edge.” The SLO subsystem ships end-to-end (model, evaluator, endpoints), but the dashboard's day-to-day SRE workflow is driven by Alerts (below), whose warn band folds in the SLO headroom colouring. Use SLOs via the API/mesh and Alerts for actionable thresholds.

Alerts & automated responses

An alert fires when a metric crosses a threshold and stays across it for a sustained window. Each alert can combine a typed remediation action, an AI remediation, a SOAR playbook and a channel notification. Schema:

Field	Meaning
`metric`, `comparator` (gt/lt), `threshold`	what and when it fires
`warn`	optional early-warning level on the same side — turns the node amber without firing
`window_secs` (def 60), `cooldown_secs` (def 300)	sustained-breach debounce and anti-flap
`severity`	info / minor / major / critical (sets risk)
`action` + `action_target`	bound remediation (e.g. `restart_service` / `nginx`); empty = detection-only
`remediation_prompt`	runs the SRE analyst to investigate and propose/apply a fix
`playbook`	a SOAR playbook to run
`notify_channel`	an integration to notify (Slack / Teams / Google Chat shape)
`nodes`	host scope (empty = fleet-wide)

When an alert fires on a node it escalates to the hub (so channel-notify, the analyst and any playbook fire) and runs its bound action through the normal gate: in Watch tier it observes only; with autonomy it executes now; otherwise it queues for admin approval (deduplicated per alert). Manage under Reliability → Alerts & automated responses; API GET/POST/DELETE /api/fleet/alerts.

This is distinct from the hub's telemetry alert rules engine (/api/alerts/rules, with a dry-run /preview), which aggregates events (count/sum/avg over a window, group-by) and leans security-side. The SRE Alerts above are per-metric→action and node-evaluated — don't conflate the two.

Correlation alerts

A correlation alert combines two or more metric conditions with ALL (AND) or ANY (OR), fired only when the combination holds for a sustained window. It carries the same action / notify / analyst / playbook options as a plain alert (a log-based condition is just a log_match_rate probe used as > 0). Build them under Reliability → Correlation alerts; API GET/POST/DELETE /api/fleet/correlation-alerts.

Scheduled runbooks

A schedule runs one typed action on a set of nodes on a cadence (every_secs, min 60) — e.g. a periodic service reload or cleanup cronjob. The hub dispatches it to each node's pre-approved action path, where it still passes the allowlist, self-protection, verify and rollback checks. API GET/POST/DELETE /api/fleet/schedules.

Inbound webhooks

An inbound webhook lets an external SRE signal (a pager, a monitor, CI) trigger a remediation by POSTing to a path-token URL — the token is the credential (the route is outside the admin session gate). The action and target are fixed in the webhook spec; the token selects which webhook. The generated URL is shown once at creation, so copy it then. API GET/POST/DELETE /api/fleet/webhooks and the fire endpoint POST /api/webhooks/:token. (This is inbound; SOAR's webhook.signed is the separate outbound capability.)

Reliability incidents, health & post-mortems

Infrastructure health — GET /api/fleet/health fans out to every node for per-host CPU/mem/disk/net plus a short trend, drawn as the health grid.
Kubernetes topology — GET /api/fleet/k8s aggregates node models into namespaces → pods with container and image counts.
Reliability incidents — the SRE-classified subset of fleet incidents, handled by the SRE remediation agent.
AI post-mortems — generate a blameless, structured post-mortem (Summary / Impact / Timeline / Root cause / Remediation / Action items) from an incident plus the node's recent audit trail; stored and retrievable. API POST/GET /api/fleet/postmortems.

Reliability remediation

SRE responses run through the same executor as security remediation — allowlist, self-protection, verify, rollback (see Investigation & Remediation). Two things are specific to the SRE side:

The four recoverable service actions — restart_service, start_service, reload_service, enable_service — always ship regardless of the configured allowlist, so a stale allowlist can't silently disable auto-recovery. Provisioning actions (create_service/remove_service, create_cronjob/remove_cronjob) still require allowlisting. All remain subject to self-protection: Antemure will never restart or disable its own agent, sshd, systemd, host networking, or the gateway/hub dependencies — but restarting a normal app service (nginx, your API) is allowed.
Agentic SRE remediation — an alert's remediation_prompt (or the operator in the SRE copilot) runs the SRE-scoped analyst, which investigates the host and proposes or dispatches a fix through the approval and self-protection gates.

The SRE consoles

Fleet dashboard — Reliability tab (header Cyber/SRE switch, copilot scoped to SRE): a Kubernetes topology tree; an Infrastructure health grid (per-node CPU/Mem/Disk/Net with thresholds and sparklines, “set a threshold for this host” shortcut); Reliability incidents & post-mortems (generate / open a post-mortem in a markdown viewer); Custom telemetry (add a probe: name, kind, target + remote Browse, pattern, metric, interval); Alerts & automated responses (the full alert form with edit/remove); Correlation alerts (a conditions builder with add/remove rows); Scheduled runbooks; and Inbound webhooks (create, copy the one-time URL).

Endpoint console — SRE domain (🛡 Cyber / 📈 SRE switch): an SRE hero with Node-health / Services / Reliability-incident KPIs; Application & system logs (pick a log, errors-only, manual path, tail with highlighting — restricted to allowed roots); a read-only Services list; Custom telemetry cards for this node's metrics; a live SRE agentic remediation stream; and an SRE audit trail with range filter and CSV export.