Sending Cost Alerts to Slack with Webhooks

Chat is where cost awareness lives day to day, and this page shows the exact Slack incoming-webhook calls that turn a database cost signal into a readable, color-coded Block Kit message without tripping Slack’s rate limits.

Back to: Alert Routing & Escalation

A Slack incoming webhook is a per-channel URL that accepts a JSON POST and renders it as a message in one fixed channel. It is the right transport for the informational and warning end of the cost-severity spectrum — a soft-limit approach, a projected month-end overspend, a chargeback summary — where the goal is visibility for a team, not a paging incident for one on-call engineer. The severity that decides whether an alert reaches Slack at all comes from the routing policy in alert routing and escalation, and the higher-severity path to a pager is covered separately in routing quota breach alerts to PagerDuty. This page covers building a Block Kit message, color-coding by severity, throttling to stay under Slack’s limits, and honoring Retry-After on 429.

The webhook contract is intentionally minimal — a 200 with the body ok means delivered, anything else means it was not — so almost all of the engineering here is in shaping a message a human can triage at a glance and in pacing the sends so Slack does not start rejecting them.

Prerequisites

Before sending, set up the webhook and pin down the message shape.

  • A Slack incoming webhook URL. Create a Slack app, enable Incoming Webhooks, and add one to the target channel. Slack returns a URL of the form https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX. The channel is baked into the URL — a webhook cannot post elsewhere. Treat the URL as a secret: anyone holding it can post to your channel, so store it in your secret manager under the same controls as every other pipeline credential in security and access control for cost data.

    export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"

  • Python: 3.10 or newer.

  • Libraries: an async HTTP client. The examples use httpx; requests works for a synchronous variant.

    pip install "httpx>=0.27"

  • A canonical alert envelope. As with every channel adapter, this one consumes the normalized CostAlert — severity, summary, cost_center, projected_overspend, and a details map — and only translates it into Slack’s Block Kit shape.

Step-by-Step Implementation

The Slack adapter builds a Block Kit message, wraps it in a color-coded attachment, and posts it through a token-bucket throttle that respects Slack’s roughly one-message-per-second-per-webhook budget, retrying on 429 by honoring the Retry-After header.

Step 1 — Build a Block Kit message

Block Kit renders structured blocks — a header, a section with side-by-side fields, and a context footer — far more legibly than a wall of text. The color bar comes from wrapping the blocks in an attachment with a color, which Slack draws as a vertical stripe.

from decimal import Decimal

# Slack attachment color bar, keyed by canonical severity.
SEVERITY_COLOR = {
    "info": "#36a64f",       # green
    "warning": "#e8a317",    # amber
    "critical": "#d00000",   # red
}


def build_slack_message(alert) -> dict:
    """Render a CostAlert into a color-coded Block Kit attachment payload."""
    color = SEVERITY_COLOR.get(alert.severity, "#808080")
    blocks = [
        {
            "type": "header",
            "text": {"type": "plain_text", "text": f"Cost alert: {alert.cost_center}"},
        },
        {
            "type": "section",
            "fields": [
                {"type": "mrkdwn", "text": f"*Severity:*\n{alert.severity.upper()}"},
                {"type": "mrkdwn", "text": f"*Cost center:*\n{alert.cost_center}"},
                {"type": "mrkdwn", "text": f"*Summary:*\n{alert.summary}"},
                {"type": "mrkdwn",
                 "text": f"*Projected overspend:*\n${alert.projected_overspend:,.2f}"},
            ],
        },
        {
            "type": "context",
            "elements": [
                {"type": "mrkdwn",
                 "text": f"dedup `{alert.dedup_key}` · fired {alert.fired_at:%Y-%m-%d %H:%M UTC}"},
            ],
        },
    ]
    # attachments[].color draws the severity stripe; blocks carry the content
    return {"attachments": [{"color": color, "blocks": blocks}]}


# Expected shape (abridged):
#   {"attachments": [{"color": "#e8a317", "blocks": [{"type": "header", ...}, ...]}]}

Step 2 — Throttle sends with a token bucket

Slack rate-limits incoming webhooks to roughly one message per second per webhook, with short bursts tolerated. A token-bucket limiter smooths a burst of per-tenant alerts into a compliant stream without dropping any, which is what keeps you out of 429 territory in the first place.

import asyncio
import time


class TokenBucket:
    """Async token bucket: refill `rate` tokens/sec up to `capacity`."""

    def __init__(self, rate: float = 1.0, capacity: int = 3):
        self.rate = rate
        self.capacity = capacity
        self._tokens = float(capacity)
        self._updated = time.monotonic()
        self._lock = asyncio.Lock()

    async def acquire(self) -> None:
        """Block until a token is available, then consume one."""
        async with self._lock:
            while True:
                now = time.monotonic()
                self._tokens = min(
                    self.capacity, self._tokens + (now - self._updated) * self.rate
                )
                self._updated = now
                if self._tokens >= 1:
                    self._tokens -= 1
                    return
                deficit = (1 - self._tokens) / self.rate
                await asyncio.sleep(deficit)

Step 3 — Post with Retry-After-aware retry

Even with throttling, a shared webhook or a coordinated burst can still earn a 429. Slack returns a Retry-After header in seconds; honoring it exactly is the difference between recovering and being throttled harder. Everything else — a 200 with body ok — is success.

The flow below shows the throttle gate, a 429 that honors Retry-After, and the successful send.

Slack webhook delivery: render Block Kit, throttle, POST, honor Retry-After on 429, deliver on 200A cost alert is rendered into a Block Kit message, passed through a token-bucket throttle, and POSTed to the Slack incoming webhook. A 429 response waits the Retry-After seconds and loops back to the POST; a 200 ok response delivers the message to the channel.429: wait Retry-After seconds, resendRenderBlock KitToken-bucketthrottlePOST webhookhooks.slack.com200 ok —delivered to channel
import logging
import os

import httpx

_bucket = TokenBucket(rate=1.0, capacity=3)
MAX_RETRIES = 4


async def send_to_slack(alert, client: httpx.AsyncClient) -> bool:
    """Deliver one CostAlert to Slack, throttled and 429-aware."""
    url = os.environ["SLACK_WEBHOOK_URL"]
    payload = build_slack_message(alert)

    for attempt in range(MAX_RETRIES + 1):
        await _bucket.acquire()                     # pace under ~1 msg/sec/webhook
        resp = await client.post(url, json=payload, timeout=10.0)

        if resp.status_code == 200:
            return True                             # body is literally "ok"
        if resp.status_code == 429:
            # Slack tells us exactly how long to wait; obey it
            retry_after = float(resp.headers.get("Retry-After", "1"))
            logging.warning("Slack 429; honoring Retry-After=%.0fs", retry_after)
            await asyncio.sleep(retry_after)
            continue
        if resp.status_code >= 500:
            await asyncio.sleep(1.5 * (2 ** attempt))
            continue
        # 400 "invalid_payload" / 404 "channel_not_found" are our bug: do not retry
        logging.error("Slack rejected message: %s %s", resp.status_code, resp.text)
        return False

    logging.error("Slack delivery failed after retries for %s", alert.dedup_key)
    return False

Expected output for a burst of five alerts through the throttle, one of which hits a 429:

WARNING:root:Slack 429; honoring Retry-After=2s
INFO:root:delivered 5/5 cost alerts to #db-cost-alerts

Verification

Confirm rendering and pacing before enabling the adapter in production.

  1. Render without sending. Assert the payload shape so a Block Kit typo fails a test rather than a live message:

    from types import SimpleNamespace
from datetime import datetime, timezone
from decimal import Decimal
alert = SimpleNamespace(
    severity="warning", cost_center="team-analytics",
    summary="team-analytics at 96% of budget", dedup_key="qb-analytics-9f2c",
    projected_overspend=Decimal("318.40"),
    fired_at=datetime(2026, 7, 18, 9, 30, tzinfo=timezone.utc),
)
msg = build_slack_message(alert)
assert msg["attachments"][0]["color"] == "#e8a317"
assert msg["attachments"][0]["blocks"][0]["type"] == "header"

  2. Validate against Slack’s Block Kit Builder. Paste the blocks array into the Block Kit Builder in Slack to catch structural errors before runtime; an invalid block returns 400 invalid_blocks, which the adapter treats as a non-retryable failure.

  3. Confirm the throttle holds. Fire twenty alerts through a single httpx.AsyncClient and confirm from timestamps that sends are paced at roughly one per second with a short initial burst — proof the token bucket is doing its job and you are not relying on Slack’s 429 as your rate limiter.

Gotchas & Edge Cases

  • A webhook posts to exactly one channel. The channel is fixed in the URL; you cannot redirect a message by setting a channel field (modern webhooks ignore it). To route different cost centers to different channels, hold a map of cost_center -> webhook_url and select the URL per alert.
  • 429 is your rate limiter of last resort, not your first. Relying on Slack to throttle you means dropped or long-delayed alerts under load. Pace proactively with the token bucket and treat any 429 as a signal your send rate is mis-tuned — the same posture as honoring per-integration limits when routing to PagerDuty.
  • Coalesce related alerts. Ten cost centers breaching in the same minute should become one digest message with ten section blocks, not ten messages. Coalescing cuts both noise and request volume, and it is the natural pairing with the deduplication described in deduplicating and escalating cost alerts.
  • attachments is legacy for color, blocks are current for content. Slack’s color bar only exists on attachments, so the idiomatic pattern is a single attachment carrying color plus a blocks array — you get the stripe without the deprecated attachment fields.
  • A 3,000-character text limit per section. A section text or mrkdwn field is capped at 3,000 characters; a verbose custom_details dump will return 400. Keep the message a summary and link out to a dashboard for detail.
  • Slack markdown is mrkdwn, not standard Markdown. Bold is *single asterisks*, links are <url|label>, and there are no headings inside mrkdwn — use a header block instead. Feeding standard Markdown produces literal asterisks in the channel.
  • A recovery note is not a resolve. Slack has no incident lifecycle, so “the spend recovered” is just another message posting to the channel, ideally in the thread of the original alert via a stored ts. Do not expect Slack to close anything the way PagerDuty resolves an incident.

Frequently Asked Questions

Should I use an incoming webhook or the chat.postMessage Web API method?

For fire-and-forget alerts to a fixed channel, an incoming webhook is simpler — no OAuth token, no scopes, one URL. Reach for chat.postMessage only when you need capabilities the webhook lacks: posting to a channel chosen at runtime, threading replies by ts, updating a message in place, or reading the posted message back. Those extra powers cost you a bot token with chat:write scope and tighter rate accounting.

What are the real rate limits on an incoming webhook?

Slack documents incoming webhooks at roughly one message per second per webhook, tolerating short bursts. There is no published hard quota beyond that, and sustained oversending earns 429 responses with a Retry-After header. The token bucket in Step 2 targets that one-per-second budget with a small burst capacity, which keeps normal alerting well within limits.

How do I color-code alerts by severity in Block Kit?

Block Kit itself has no color property; the colored stripe comes from wrapping your blocks in a legacy attachment and setting its color to a hex value or one of good, warning, danger. The build_slack_message function maps the canonical severity onto a hex color so info shows green, warning amber, and critical red.

Why did my message post as raw JSON or literal asterisks?

Two common causes. Posting the payload as form-encoded instead of Content-Type: application/json makes Slack render the raw body — always send JSON. Literal asterisks mean you used standard Markdown in a field typed plain_text; switch the field type to mrkdwn so Slack interprets its *bold* and <url|label> syntax.

Should critical cost alerts go to Slack at all?

Slack is the right home for info and warning, but a truly critical breach needs an acknowledged page, not a channel message that can scroll past unseen. The routing policy sends critical alerts to both Slack (for team visibility) and PagerDuty (for an accountable page); Slack alone should never be the escalation path for spend that is actively running away.

Back to: Alert Routing & Escalation