Operational Assurance

Decision provenance, circuit-breaker alerting, and Prometheus metrics with Grafana dashboards — every routing chain persisted from trigger to outcome.

What is it?

Operational Assurance (COS-002) is Warburton's system for ensuring that every decision the platform makes is recorded, traceable, and auditable. It provides three tiers of operational reliability: decision provenance, circuit-breaker alerting, and full telemetry across all subsystems.

Every decision leaves a paper trail. Every escalation is logged. Every routing chain is persisted. If something goes wrong, you don't have to reconstruct what happened from memory. The system already wrote it down.

Why does it exist?

AI systems that make decisions affecting real people need to be accountable for those decisions. When a moderation action is taken, when a welfare escalation is triggered, when a safety rule fires — someone needs to be able to answer "why did that happen?" with precision, not with guesswork.

Most community management tools treat logging as an afterthought. Operational Assurance treats it as the foundation. The ability to reconstruct exactly what happened, why it happened, and what policy applied is not a feature — it's a requirement for any system making decisions that affect human beings.

Why is it different?

Operational Assurance goes beyond simple event logging. It tracks the complete decision chain from trigger to outcome — not just "this action was taken" but "this input was received, this routing decision was made, this policy was applied, this outcome was produced, and here is the full chain connecting them."

The circuit-breaker system is proactive rather than reactive. When system health thresholds are breached, the platform automatically degrades gracefully rather than failing catastrophically. Operators are alerted before problems become outages.

How does it work?

Operational Assurance operates across three tiers:

OA1 — Decision Provenance

Every AI response is persisted with full routing provenance using a Context Accumulator Pattern — a mutable metadata object attached to the request context that is populated at each routing step. After response delivery, the accumulated provenance is written to the database fire-and-forget, so provenance failures never block response delivery.

Each response records 12 fields: the complete routing path (as a JSON array of steps tried, not a single string), fallback chain, model used, retrieval sources consulted, middleware intercepts fired, persona, prompt and completion token counts, cost, response latency, quality score, and a request ID for audit lookup.

A decision_reason classification field provides fast operator queries without parsing JSON. Twelve deterministic categories — including welfare_intervention, safety_intercept, learned_cache, substack_retrieval, and model_response — are set at routing time across 21 return points. An operator can answer "what percentage of responses are welfare interventions?" with a single indexed query.

OA2 — Circuit-Breaker Alerting

Critical operational events are wired to real-time admin alerts, not buried in logs. The system uses the existing admin alert infrastructure with a 30-minute per-category cooldown to prevent alert storms. Five specific alerting gaps were identified and closed:

Welfare and crisis events now trigger admin alerts (previously logged silently to the welfare audit trail)
Circuit breaker state transitions (OPEN, HALF, CLOSED) are wired to both admin alerts and Prometheus state metrics
Claude budget exhaustion triggers an admin alert (previously returned a fallback reply silently)
Provenance write failures trigger both an admin alert and an audit-write-failure metric
Health checks now cover Ollama, Claude circuit breaker status, and Redis connectivity (previously only knex/database)

OA3 — Prometheus Metrics & Grafana Dashboards

Every safety pipeline detection is visible to Prometheus. Nineteen recordSafetyTrigger call sites are wired across five moderation modules — scam detection (11 call sites covering pattern, impersonation, DM bait, and pump-dump categories), NSFW/advances (5 call sites), safeguarding, community standards, and welfare/distress. Operators can answer "how many scam detections happened this week?" from metrics alone, without querying logs.

Moderation action metrics are recorded consistently across all three platform adapters — Telegram, Discord, and WhatsApp — ensuring no platform is metrically invisible.

Example

Scenario: A member reports that their message was unfairly moderated. The operator queries the decision provenance by request ID and sees the complete chain: the message matched a scam detection pattern (dm_bait), the safety pipeline triggered at step 2, the detection was metered as recordSafetyTrigger('scam', 'dm_bait'), and the decision_reason was classified as safety_intercept. The operator can explain exactly why the action was taken — with the full routing path, not a guess.

Why should anyone care?

When a community member asks "why was my message moderated?" the operator needs a real answer — not "the AI decided." Operational Assurance provides that answer, every time, with the complete decision chain that led to the action.

For regulated industries, this level of auditability is not optional. For any community operator who takes their responsibility seriously, it's the difference between running a system you understand and running a system you hope is working correctly.