Deduplicating and Escalating Cost Alerts

A level-triggered quota breach re-evaluates every ingestion cycle, so without a shared suppression store every cycle becomes a page — this page shows the Redis-backed deduplication and the time-based escalation ladder that turn that stream of identical states into exactly one alert, escalated only when it is ignored.

Back to: Alert Routing & Escalation

Deduplication and escalation are two halves of one state machine keyed on a stable dedup_key. Deduplication answers “have we already alerted on this exact condition?” and escalation answers “has the alert we sent been acknowledged, and if not, who hears about it next?” Both need shared, atomic, expiring state, which is exactly what Redis provides through SET key value NX EX ttl. The dedup_key is the same identity-derived key the PagerDuty adapter sends and the Slack adapter stamps into its footer, and the whole mechanism plugs into the routing layer described in alert routing and escalation. This page covers the atomic dedup claim, an escalation ladder that promotes unacknowledged alerts after N minutes, and flap suppression with a hysteresis band.

The reason this belongs in Redis rather than process memory is blunt: the moment you run two alerter replicas, an in-process cache lets both fire the same page because neither sees the other’s suppression state. A shared store with an atomic claim is the only correct answer once there is more than one worker.

Prerequisites

Before running the suppression and escalation logic, confirm the following.

  • A reachable Redis instance. Version 6 or newer, so SET supports the NX and EX options in one atomic command. A single instance is fine for suppression state; for durability across restarts enable AOF persistence, though a lost dedup key at worst causes one duplicate alert, not a missed one.

    export REDIS_URL="rediss://:$REDIS_PASSWORD@cache.internal:6380/0"

    The connection credential is a secret governed by the same rotation and least-privilege posture as every other pipeline credential in security and access control for cost data.

  • Python: 3.10 or newer.

  • Libraries: the async Redis client (bundled in modern redis-py as redis.asyncio).

    pip install "redis>=5.0"

  • A stable dedup key. Every function here keys on the identity-derived dedup_key — a hash of (cost_center, quota_id, breach_kind) — never on the live spend figure, so repeated evaluations of the same breach map to one key.

Step-by-Step Implementation

The mechanism is three cooperating pieces of Redis state per alert: a suppression key that grants the first-send claim, an acknowledgement marker, and an escalation-tier counter advanced by a timer. Flap suppression adds a hysteresis band and a resolve delay on top.

Step 1 — Claim the first send atomically

SET key value NX EX ttl is atomic: exactly one caller — one replica, one evaluation cycle — sets the key and gets True; every subsequent caller within the TTL gets None. That single atomic operation is the entire deduplication primitive.

from redis.asyncio import Redis


async def first_seen(redis: Redis, dedup_key: str, ttl: int = 3600) -> bool:
    """Return True only for the first caller to claim this dedup_key.

    NX makes the SET succeed only if the key is absent; EX expires it
    after `ttl` seconds so a breach that recurs after recovery re-alerts.
    """
    # value is a claim timestamp for debugging; the win/lose is the return
    claimed = await redis.set(
        name=f"alert:dedup:{dedup_key}",
        value="claimed",
        nx=True,     # only set if not already present -> atomic first-claim
        ex=ttl,      # auto-expire so the condition can re-alert later
    )
    return bool(claimed)


# Expected:
#   await first_seen(r, "qb-analytics-9f2c")  -> True   (first cycle)
#   await first_seen(r, "qb-analytics-9f2c")  -> False  (every cycle within TTL)

Step 2 — Record acknowledgement and drive the escalation ladder

The ladder is a list of tiers, each with an unacknowledged wait. On each cycle a periodic sweep checks every open, unacknowledged alert; if its age has passed the current tier’s after_minutes, it advances to the next tier and dispatches to that tier’s channel. Acknowledgement — set by a webhook from PagerDuty or a Slack action — stops the ladder cold.

import time
from dataclasses import dataclass

ACK_TTL = 24 * 3600


@dataclass(frozen=True)
class Tier:
    after_minutes: int      # unacknowledged wait before this tier fires
    channel: str            # which route this tier notifies


# notify the primary immediately, escalate if unacknowledged
LADDER = [
    Tier(after_minutes=0,  channel="slack"),        # tier 0: primary, on first send
    Tier(after_minutes=10, channel="pagerduty"),    # tier 1: page primary on-call
    Tier(after_minutes=25, channel="pagerduty-lead"),  # tier 2: page the DBA lead
]


async def acknowledge(redis: Redis, dedup_key: str, who: str) -> None:
    """Mark an alert acknowledged; freezes the escalation ladder."""
    await redis.set(f"alert:ack:{dedup_key}", who, ex=ACK_TTL)


async def maybe_escalate(redis: Redis, dedup_key: str, fired_at: float,
                         dispatch) -> int | None:
    """Advance the ladder if the alert is still unacknowledged past a tier.

    Returns the tier index just fired, or None if nothing changed.
    """
    if await redis.exists(f"alert:ack:{dedup_key}"):
        return None                              # acknowledged: ladder frozen

    age_min = (time.time() - fired_at) / 60.0
    tier_key = f"alert:tier:{dedup_key}"
    current = int(await redis.get(tier_key) or 0)

    # find the highest tier whose wait has elapsed
    target = current
    for i in range(current + 1, len(LADDER)):
        if age_min >= LADDER[i].after_minutes:
            target = i

    if target > current:
        # advance atomically; only escalate if we actually moved the counter
        await redis.set(tier_key, target, ex=ACK_TTL)
        await dispatch(dedup_key, LADDER[target].channel)
        return target
    return None

Step 3 — Suppress flapping with hysteresis and a resolve delay

An alert whose spend hovers at the threshold will open and close every few minutes without hysteresis. The fix is a two-sided band: fire only when spend crosses above the limit, and auto-resolve only when it holds below a lower recovery threshold for a sustained resolve delay. The state below tracks how long recovery has held.

The state diagram traces an alert from firing through the escalation ladder, acknowledgement, and hysteresis-gated resolution.

Cost-alert dedup and escalation state machine: dedup gate, firing tiers on a timed ladder, acknowledgement freeze, and hysteresis-gated resolveA dedup gate admits only the first evaluation of a condition and suppresses duplicates. The admitted alert enters Firing at tier zero on Slack, advances to tier one on PagerDuty after ten unacknowledged minutes and tier two to the DBA lead after twenty-five. Acknowledgement freezes the ladder. Spend holding below the recovery threshold for the resolve delay moves the alert to Resolved; if spend rises again during the countdown, flap suppression returns it to Firing.DedupgateSuppressFiringtier 0 · Slack · t+0tier 1 · PagerDuty · t+10mtier 2 · DBA lead · t+25madvances while unacknowledgedAcknowledgedladder frozenResolve delaybelow recovery thresholdResolvedauto-close sentfirstduplicateackrecoversspend rises: flap
RECOVERY_HOLD_SECS = 600         # spend must hold below recovery threshold this long
RECOVERY_MARGIN = 0.95           # resolve only under 95% of the limit (hysteresis)


async def evaluate_resolution(redis: Redis, dedup_key: str,
                              current_spend: float, limit: float) -> str:
    """Gate auto-resolve behind a hysteresis band and a sustained hold."""
    hold_key = f"alert:recover_since:{dedup_key}"

    if current_spend >= limit:
        # still breached: cancel any in-progress recovery countdown -> flap guard
        await redis.delete(hold_key)
        return "firing"

    if current_spend > limit * RECOVERY_MARGIN:
        # in the hysteresis band: not breached, but not safely recovered either
        return "firing"

    # below the recovery threshold: start or continue the hold timer
    started = await redis.set(hold_key, time.time(), nx=True, ex=RECOVERY_HOLD_SECS * 2)
    since = float(await redis.get(hold_key))
    if time.time() - since >= RECOVERY_HOLD_SECS:
        # sustained recovery: clear all state and signal resolve
        await redis.delete(hold_key, f"alert:dedup:{dedup_key}",
                           f"alert:tier:{dedup_key}", f"alert:ack:{dedup_key}")
        return "resolved"
    return "recovering"

Expected output over the lifetime of one flapping-then-recovering alert:

first_seen -> True            # cycle 1: alert dispatched to tier 0 (Slack)
maybe_escalate -> 1           # +10m unacknowledged: escalated to PagerDuty
evaluate_resolution -> firing # spend bounced back above limit: flap suppressed
evaluate_resolution -> recovering  # spend below 95% of limit, hold timer running
evaluate_resolution -> resolved    # held 10m below threshold: auto-resolve fired

Verification

Confirm the suppression and ladder behave before trusting them with a real rotation.

  1. Prove the atomic claim. Fire first_seen concurrently from many coroutines against one key and assert exactly one wins:

    import asyncio
results = await asyncio.gather(*(first_seen(r, "qb-race-test", ttl=60)
                                 for _ in range(50)))
assert sum(results) == 1, "dedup claim is not atomic"

  2. Confirm the ladder advances only when unacknowledged. Set fired_at to eleven minutes ago and call maybe_escalate; it must return 1. Then acknowledge the key and call again with fired_at at twenty-six minutes ago; it must return None — the ack froze the ladder.

  3. Confirm hysteresis holds against a flap. Feed evaluate_resolution a sequence that dips below the limit, bounces back above it, then drops and holds below the recovery margin. It must never return resolved until the sustained hold elapses — proof the flap guard cancels the countdown on each bounce.

Gotchas & Edge Cases

  • Never build the dedup key from the live spend figure. A key like qb-analytics-1300 changes every cycle as spend moves, so NX always succeeds and every cycle pages. Key on identity — (cost_center, quota_id, breach_kind) — which is the same rule the PagerDuty adapter relies on for incident collapse.
  • Set the dedup TTL longer than a breach’s natural lifetime, shorter than forever. Too short and a still-active breach re-alerts mid-incident; too long and a genuinely new breach after recovery is silently suppressed. Tie the TTL to your recovery detection so the key clears exactly when the condition does, which evaluate_resolution does by deleting the key on sustained recovery.
  • SETNX (the legacy command) cannot set a TTL atomically. Using SETNX then EXPIRE opens a window where a crash between the two leaves a key that never expires — a permanently suppressed alert. Always use the unified SET ... NX EX, which first_seen does.
  • A missing acknowledgement must read as unacknowledged. If the ack webhook fails to write to Redis, maybe_escalate sees no ack key and correctly keeps escalating — fail toward paging, never toward silence. This is the delivery-side application of the resilience discipline in error handling in cost pipelines.
  • Clock skew breaks the ladder. The escalation timer compares fired_at against time.time(); if fired_at came from a worker with a skewed clock, tiers fire early or late. Stamp fired_at from the same monotonic authority the sweep reads, and keep all timestamps timezone-aware UTC.
  • Coalesce before you escalate. A single upstream event (a shared pool breaching) can spawn many per-tenant alerts; escalating each independently re-creates the storm dedup was meant to prevent. Group related keys under a parent condition so the ladder escalates the group once — pairs naturally with the digest batching in sending cost alerts to Slack with webhooks.
  • Hysteresis margin must sit below the alert threshold, not at it. A recovery margin equal to the limit still flaps because spend crossing the exact boundary toggles the state. A margin of 90–95 percent of the limit gives the band the width it needs to absorb noise, and the anomaly signals feeding this from cost anomaly detection benefit from the same buffer.

Frequently Asked Questions

Why Redis rather than an in-memory dict for deduplication?

An in-process dict works only with a single alerter instance. The moment you scale to two replicas for availability, each holds its own cache and both fire the same page, because neither sees the other’s suppression state. Redis gives all replicas one shared, atomic keyspace, so the first replica to run SET NX wins the send and every other replica and every re-evaluation within the TTL is a no-op.

How do I pick the dedup TTL?

Match it to the condition’s expected lifetime and your recovery-detection cadence. A monthly-budget breach might persist for hours, so a one-hour TTL that is refreshed while the breach is active, and deleted on confirmed recovery, keeps one incident open without re-paging. The anti-pattern is a fixed short TTL that lets a still-active breach re-alert, or an unbounded key that suppresses the next genuine breach forever.

What stops the escalation ladder from firing after someone acknowledges?

An acknowledgement writes an alert:ack:<dedup_key> marker to Redis, and maybe_escalate returns immediately if that key exists. The ack is set by a webhook — PagerDuty’s acknowledge event or a Slack interactive action — so a human tapping “ack” on the page freezes the ladder without any code deploy. If the ack write fails, the ladder keeps escalating, which is the safe direction.

How is flap suppression different from deduplication?

Deduplication stops identical repeated alerts for a condition that is continuously true. Flap suppression stops an alert that rapidly toggles between breached and recovered because spend hovers at the threshold. The fix is a hysteresis band plus a resolve delay: fire on crossing above the limit, but auto-resolve only after spend holds below a lower recovery threshold for a sustained period, so a brief dip never closes the incident prematurely.

Should the escalation tiers be code or configuration?

Configuration. The tier list — wait times and target channels — encodes on-call policy that changes far more often than the escalation engine. Keeping LADDER as data (loaded from a config store per service or per severity) means “critical also pages the DBA lead after 25 minutes” is a config edit, not a code change, which is the same declarative-routing principle the dispatcher uses for its severity-to-channel policy.

Back to: Alert Routing & Escalation