Anomaly detection

Static thresholds are hard to pick and drift out of date as traffic grows. Healthcheck learns each monitor's normal behaviour from a 7-day rolling baseline, then lets you either apply a suggested threshold (you approve the number) or alert on dynamic anomalies with the σ (sigma) operator — no fixed number required.

The baseline

For every active monitor Healthcheck computes a rolling baseline over the last 7 days and refreshes it hourly. It records, per metric:

Field	Meaning
mean (μ)	The typical value.
stddev (σ)	How much the value normally varies.
p95	The 95th-percentile value.
samples	How many checks fed the baseline.

Which metric is baselined depends on the monitor type:

• Elastic (WAF-log) monitors → the per-cycle request count.
• Protocol monitors (HTTP, TCP, DNS, TLS) → latency.

note

A brand-new monitor shows "Baseline accruing" until it has collected enough checks. The baseline and suggestions appear automatically once there are enough samples — no action needed.

Per-bucket baselines

A WAF-log monitor groups its traffic by a dimension (status code, HTTP method, country, mitigation…). Healthcheck baselines each bucket of that dimension separately, in addition to the total — so "normal" is learned per bucket, not just for total volume. That's what lets an alarm catch a surge in one specific bucket even while total traffic looks ordinary.

Monitor groups by	Buckets baselined	A σ alarm catches
HTTP status	per status code — 200, 301, 404, 502, 429…	a spike in one code, e.g. 502 surge (baseline ≈ 0) or a 429 rate-limit storm
HTTP method / country / …	per method (GET/POST), per country…	e.g. a sudden burst of POST, or traffic from one country
WAF mitigation	per mitigation category — WAFRULE, SEO user agent, ACL-WHITE-IP, EDGE…	a jump in WAFRULE blocks (attack) or bot hits, vs that category's normal
request count (total)	the whole monitor's per-cycle total	an overall traffic spike / drop (volumetric DDoS, outage)

High-cardinality dimensions (per-IP, per-URL) aren't bucketed — there'd be thousands of baselines — so those monitors keep just the total count baseline.

Baseline & suggested thresholds

Open a monitor's detail and find the Baseline & suggested thresholds panel. It shows the total baseline (μ ± σ, p95, sample count) as a chart of recent history against the baseline mean plus a status breakdown donut, and — for WAF-log monitors — a per-bucket baselines table, each bucket with its own suggested thresholds.

Two suggested thresholds are offered:

Suggestion	Value	Use for
Apply warning	mean + 2σ	An early heads-up.
Apply critical	mean + 3σ	A strong, rarely-false signal.

Selecting Apply warning or Apply critical opens the alarm form pre-filled with a fixed-threshold condition at that value. You review it, name the rule, pick a notification channel, and save — the number is never applied automatically, so you stay in control.

Dynamic anomalies: the sigma operator

Instead of a fixed number, an alarm condition can use the σ (sigma) operator to fire whenever the value is unusually high relative to the current baseline. In the condition builder, pick a baselined field (count for WAF-log monitors, latency_ms for protocol monitors), choose the σ anomaly operator, and enter a multiplier k (default 3).

The alarm fires when:

value > mean + k × stddev

So k = 3 fires only when traffic spikes more than 3 standard deviations above normal — a threshold that adjusts itself as the baseline moves. σ detects spikes (upper-tail anomalies); use a lower k to be more sensitive.

note

Until a baseline exists, a σ alarm is fail-safe: it never fires. This avoids false alarms on a monitor that hasn't learned its normal behaviour yet.

tip

Very bursty traffic has a large σ, so ordinary swings stay "normal" and only a genuine surge trips k = 3. That's by design — it keeps everyday noise from paging you.

Spikes, drops, and per-source anomalies

The operator dropdown offers two σ variants, plus derived per-source metrics for high-cardinality dimensions:

• σ anomaly ($sigma) — upper-tail only; fires on spikes (value > mean + k·stddev).
• σ anomaly (spike+drop) ($sigma_abs) — two-sided; fires when |value − mean| ≥ k·stddev, so it also catches drops — a normally-active signal falling toward zero (an outage).
• top source concentration / distinct sources — derived metrics on grouped WAF-log monitors. A σ alarm on top source concentration catches a single IP flooding (one source spikes); on distinct sources it catches a distributed scan (many IPs/URIs at once) — even when the total request count looks normal.

These are exactly what the Single-IP flood and Distributed scan monitor templates set up for you.

The anomaly notification

When a σ alarm fires, the notification carries a rich, SOC-style summary:

• A chart image — recent history with the baseline mean ± 1σ band and the current value marked.
• Finding — the z-score, e.g. count z=3.5 · current 641 (baseline 512±88).
• Status breakdown — 2xx · 3xx · 4xx · 5xx for WAF-log monitors.
• Top IP / Top URL — the top contributors, when the monitor groups by those dimensions.

The chart is attached on the first alert; periodic re-notifications while the incident stays open are text-only to keep the channel readable.

See also

• Alarms — build the rule and pick the operator.
• Notifications — where alerts are delivered.
• Incidents — the incident a fired alarm opens.