Routing Quota Breach Alerts to PagerDuty

When a database cost center crosses a hard quota, someone on call needs to be paged exactly once, at a severity that matches the money at stake, and the incident must close itself when spend recovers — this page shows the precise PagerDuty Events API v2 calls that make that happen.

Back to: Alert Routing & Escalation

PagerDuty’s Events API v2 is a single-endpoint contract: every notification is a JSON POST to https://events.pagerduty.com/v2/enqueue carrying a routing_key, an event_action of trigger, acknowledge, or resolve, and — critically — a dedup_key that PagerDuty uses to collapse repeated triggers onto one open incident. That last field is what turns a level-triggered quota breach, which re-evaluates every ingestion cycle, into exactly one page rather than a storm. The severity you send is derived from the breach ratio defined in database quota boundary design, and the whole call sits at the end of the routing layer described in alert routing and escalation. This page covers the trigger call, the auto-resolve call, breach-ratio severity mapping, and retry on throttling.

The Events API is deliberately narrow, which is what makes it reliable: there is no per-incident endpoint to look up, no incident ID to track. You own the dedup_key, PagerDuty owns the incident lifecycle keyed on it, and the two stay in sync as long as your key is stable across evaluations of the same condition.

Prerequisites

Before sending a single event, confirm the following.

  • A PagerDuty service with an Events API v2 integration. In the PagerDuty service directory, add an Events API v2 integration to the service that should receive database-cost incidents. That produces a 32-character Integration Key — this is the routing_key. It is a routing credential, not a personal token; treat it as a secret governed by the same rotation posture as every other pipeline credential in security and access control for cost data.

    # never hard-code the key; export it from your secret store
export PAGERDUTY_ROUTING_KEY="R0ABCD1234EFGH5678IJKL9012MNOP34"

  • Python: 3.10 or newer (the code uses modern match-free structural typing and httpx).

  • Libraries: install an HTTP client. Either httpx (async) or requests (sync) works; the examples below use httpx for its first-class timeout and async support.

    pip install "httpx>=0.27"

  • A canonical alert envelope. These calls consume the normalized CostAlert object produced by the routing layer — a dedup_key, a cost_center, a severity, a summary, and a details map. The adapter here only translates that envelope into the Events API shape; it never re-derives severity.

Step-by-Step Implementation

The adapter has three responsibilities: derive a stable dedup_key, build the Events API v2 payload with a breach-ratio severity, and POST it with retry on 429/5xx. Auto-resolve is the same call with event_action: "resolve" and no payload body.

Step 1 — Derive a stable dedup key

PagerDuty collapses triggers that share a dedup_key onto one open incident, so the key must encode the identity of the breach condition, never its live dollar value. Hashing (cost_center, quota_id) gives a key that is stable while the breach persists and distinct across genuinely different breaches.

import hashlib


def pd_dedup_key(cost_center: str, quota_id: str) -> str:
    """Stable PagerDuty dedup_key for one quota-breach condition.

    Keyed on identity, not on the current spend figure, so repeated
    evaluations of the same breach collapse onto a single incident.
    """
    raw = f"quota-breach:{cost_center}:{quota_id}"
    digest = hashlib.sha1(raw.encode("utf-8")).hexdigest()[:16]
    return f"qb-{cost_center}-{digest}"


# Expected:
#   pd_dedup_key("team-analytics", "monthly-usd")  -> "qb-team-analytics-9f2c1a7b6d4e8f03"
#   same inputs always return the same key; a different quota_id yields a different key

Step 2 — Map the breach ratio onto Events API severity

The Events API v2 payload.severity field accepts exactly critical, error, warning, or info. The breach ratio — spend divided by the limit — drives the mapping so a marginal soft-limit overshoot and a runaway hard-limit breach never land on the same rung.

from decimal import Decimal


def pd_severity(current_spend: Decimal, limit: Decimal) -> str:
    """Map spend/limit ratio onto a PagerDuty Events API severity."""
    if limit <= 0:
        return "critical"                 # an undefined limit is never safe
    ratio = current_spend / limit
    if ratio >= Decimal("1.5"):
        return "critical"                 # >=150% of limit: runaway spend
    if ratio >= Decimal("1.0"):
        return "error"                    # at or over the hard limit
    if ratio >= Decimal("0.9"):
        return "warning"                  # approaching the limit
    return "info"


# Expected:
#   pd_severity(Decimal("1600"), Decimal("1000"))  -> "critical"
#   pd_severity(Decimal("1050"), Decimal("1000"))  -> "error"
#   pd_severity(Decimal("920"),  Decimal("1000"))  -> "warning"

Step 3 — Trigger an incident with retry

The trigger call POSTs a trigger event carrying the routing_key, the dedup_key, and a payload whose custom_details carries the machine-readable breach context an on-call engineer triages on. PagerDuty returns 202 Accepted with the accepted dedup_key; it returns 429 under rate pressure and 5xx during transient trouble, both of which are retried with exponential backoff.

The sequence below traces a trigger through a 429 retry to acceptance, then a later resolve.

PagerDuty routing sequence: derive dedup key and severity, POST trigger with 429 backoff-retry, receive 202, then resolve on recoveryThe Router derives a stable dedup key and breach-ratio severity, then POSTs a trigger event to the PagerDuty Events API v2. A 429 response triggers an exponential-backoff sleep and a retry of the same trigger; a 202 Accepted response returns the dedup key. When spend later recovers, the Router POSTs a resolve event with the same dedup key and the API replies 202 Accepted, closing the incident.RouterEvents API v2/v2/enqueuederive dedup_key + severityalt[429 Too Many Requests]POST trigger { dedup_key, payload }429 rate limitedsleep 2ⁿ backoff, retryelse[accepted]202 Accepted { dedup_key }POST resolve { same dedup_key }202 Accepted — incident closed— later, when spend recovers —
import asyncio
import logging
import os
from decimal import Decimal

import httpx

ENQUEUE_URL = "https://events.pagerduty.com/v2/enqueue"
MAX_RETRIES = 4
BASE_DELAY = 1.5


async def _post_event(client: httpx.AsyncClient, body: dict, attempt: int = 0) -> dict:
    """POST one Events API v2 event, retrying on 429/5xx with backoff."""
    resp = await client.post(ENQUEUE_URL, json=body, timeout=10.0)
    if resp.status_code == 429 or resp.status_code >= 500:
        if attempt < MAX_RETRIES:
            delay = BASE_DELAY * (2 ** attempt)
            logging.warning("PagerDuty %s; retrying in %.1fs", resp.status_code, delay)
            await asyncio.sleep(delay)
            return await _post_event(client, body, attempt + 1)
        raise RuntimeError(f"PagerDuty enqueue failed after retries: {resp.status_code}")
    resp.raise_for_status()             # 400/402/403 are our bug: fail loudly
    return resp.json()


async def trigger_breach(alert, current_spend: Decimal, limit: Decimal) -> str:
    """Open (or update) a PagerDuty incident for a quota breach."""
    routing_key = os.environ["PAGERDUTY_ROUTING_KEY"]
    body = {
        "routing_key": routing_key,
        "event_action": "trigger",
        "dedup_key": alert.dedup_key,
        "payload": {
            "summary": alert.summary,                       # <=1024 chars, shown on the incident
            "source": alert.cost_center,                    # the entity that breached
            "severity": pd_severity(current_spend, limit),  # critical|error|warning|info
            "component": "database-cost-quota",
            "group": alert.cost_center,
            "class": "quota-breach",
            "custom_details": {
                "cost_center": alert.cost_center,
                "current_spend": str(current_spend),
                "limit": str(limit),
                "breach_ratio": f"{current_spend / limit:.2f}",
                "projected_overspend": str(alert.details.get("projected_overspend", "0")),
            },
        },
    }
    async with httpx.AsyncClient() as client:
        result = await _post_event(client, body)
    logging.info("triggered %s -> %s", alert.dedup_key, result.get("status"))
    return result["dedup_key"]


async def resolve_breach(dedup_key: str) -> None:
    """Close the incident for a breach whose spend has recovered."""
    body = {
        "routing_key": os.environ["PAGERDUTY_ROUTING_KEY"],
        "event_action": "resolve",
        "dedup_key": dedup_key,                             # must match a prior trigger
    }
    async with httpx.AsyncClient() as client:
        await _post_event(client, body)
    logging.info("resolved %s", dedup_key)

Expected output when a breach is triggered and later resolved:

INFO:root:triggered qb-team-analytics-9f2c1a7b6d4e8f03 -> success
INFO:root:resolved qb-team-analytics-9f2c1a7b6d4e8f03

The 202 response body carries {"status": "success", "message": "Event processed", "dedup_key": "..."}. Reading the returned dedup_key back — rather than trusting the one you sent — is deliberate: if you omit dedup_key on a trigger, PagerDuty generates one and returns it, and you must persist that value to resolve the incident later.

Verification

Confirm the round trip before wiring the adapter into the dispatcher.

  1. Fire a trigger and inspect the response. A quick smoke test against a test service:

    from types import SimpleNamespace
alert = SimpleNamespace(
    dedup_key="qb-smoke-test-0001",
    cost_center="smoke-test",
    summary="smoke-test at 130% of budget",
    details={"projected_overspend": "420.00"},
)
key = asyncio.run(trigger_breach(alert, Decimal("1300"), Decimal("1000")))
assert key == "qb-smoke-test-0001", "PagerDuty did not echo our dedup_key"

  2. Confirm collapse, not duplication. Fire the same trigger three times in a row and check the PagerDuty service — there must be exactly one open incident, not three. That single incident with an incrementing alert count is the proof your dedup_key is stable.

  3. Resolve and confirm closure. Call resolve_breach("qb-smoke-test-0001") and confirm the incident moves to Resolved. Resolving a dedup_key that was never triggered still returns 202 but is a no-op — harmless, but a sign your open-incident bookkeeping drifted.

Gotchas & Edge Cases

  • dedup_key must not encode the spend figure. If you build the key from the current dollar amount (qb-team-analytics-1300), every re-evaluation produces a new key and therefore a new incident — the exact pager storm on a sustained breach. Key on (cost_center, quota_id) identity only, as pd_dedup_key does.
  • A resolve before a trigger is silently accepted. The Events API returns 202 for a resolve of an unknown key, so a bug that resolves incidents that were never opened produces no error. Track which keys you have open (the deduplication store already holds this) and only resolve keys you know are live.
  • HTTP 400 means a malformed payload, not throttling. Missing routing_key, an unrecognized event_action, or a severity outside the four allowed values returns 400 — do not retry it. The code above deliberately lets raise_for_status() surface 4xx (other than 429) as a hard failure.
  • HTTP 429 is per-integration-key. Rate limits apply to the routing key, so a burst across many cost centers sharing one PagerDuty service can throttle collectively. Honor the backoff, and consider coalescing low-severity breaches rather than firing one event per cost center — the same throttling discipline used for sending cost alerts to Slack with webhooks.
  • custom_details is displayed but not indexed for grouping. Put anything you want PagerDuty to group or route on into group, class, or component, not custom_details; the latter is for human triage context only.
  • Severity is fixed at trigger time. If a breach escalates from error to critical between evaluations, re-sending a trigger with the same dedup_key and a higher severity updates the open incident — PagerDuty does not open a second one. Use that to reflect a worsening breach rather than opening a parallel incident.

Frequently Asked Questions

Should I use the Events API v2 or the REST API to page on quota breaches?

The Events API v2. It is purpose-built for machine-generated events: one endpoint, no incident IDs to track, and built-in deduplication keyed on your dedup_key. The REST API is for managing incidents, schedules, and services programmatically, and it requires a personal or service API token with far broader scope. For firing and resolving cost incidents, the Events API’s routing_key is both simpler and lower-privilege.

How do I stop a sustained breach from paging every evaluation cycle?

Two layers cooperate. PagerDuty collapses repeated triggers that share a dedup_key onto one incident, so even if you send a trigger every cycle you get one page. But the network-efficient pattern is to also suppress the repeat send upstream in a shared deduplication store, so you only POST once per condition — that is the subject of deduplicating and escalating cost alerts.

What severity should a soft-limit breach map to?

warning, not error or critical. A soft limit is an early heads-up that a cost center is approaching its budget, typically routed to chat rather than a page. Reserve error for at-or-over the hard limit and critical for runaway spend well past it — the pd_severity function encodes exactly that ladder from the breach ratio.

Does resolving an incident require the same routing key that triggered it?

Yes. The resolve event must carry the same routing_key (the integration on the service that holds the incident) and the same dedup_key that the trigger used. A resolve sent to a different service’s routing key cannot find the incident and is a silent no-op, which is a common cause of stale open incidents after a service is re-keyed.

How should I handle a PagerDuty outage during a real breach?

Retry 429/5xx with bounded exponential backoff, and if the enqueue still fails, fall back to a secondary channel rather than dropping the alert. Routing a critical breach to Slack when PagerDuty is unreachable keeps a human in the loop; the dispatcher’s independent-channel design means a PagerDuty failure never suppresses the parallel Slack send for the same alert.

Back to: Alert Routing & Escalation