Delivering Cost Reports to Slack and Email

This page is the delivery stage of a chargeback pipeline: take an already-generated CSV or PDF statement and get it to its owners over Slack and email reliably — with retry and backoff on transient failures, and an idempotent send guard so a retried delivery never posts the same statement twice.

Back to: Chargeback Reporting Automation

Delivery is where a chargeback pipeline earns or loses trust with the people it serves. The files themselves come from generating monthly chargeback reports with pandas, which produces reconciled artifacts and then stops; this stage does one thing — ship them — and must do it exactly once. The hard part is not the API call, it is the failure semantics around it: a Slack upload or an SES send can time out on the client after the server has already accepted it, and a naive retry then double-posts a financial statement. The defense is a persistent send guard keyed on the run, checked before every attempt and written before every success is returned. Because these credentials can post to internal channels and send mail as your domain, they are scoped and rotated under security and access control for cost data; the delivery worker needs Slack files:write and SES ses:SendRawEmail, and nothing else.

Prerequisites

  • Python: 3.10 or newer.

  • Libraries: the Slack SDK, boto3 for SES, and tenacity for declarative backoff.

    pip install "slack_sdk>=3.27" "boto3>=1.34" "tenacity>=8.2"

  • Slack: a bot token (xoxb-...) with the files:write scope, invited to the target channel. For a lightweight summary you can instead use an incoming webhook URL; the file upload path below uses the bot token because it attaches the actual statement.

  • AWS SES: a verified sender identity (domain or address) and, if the account is in the sandbox, verified recipients. The IAM policy needs only ses:SendRawEmail:

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ses:SendRawEmail",
      "Resource": "*",
      "Condition": {
        "StringEquals": { "ses:FromAddress": "finops@example.com" }
      }
    }
  ]
}

  • Secrets: read SLACK_BOT_TOKEN, the AWS credentials, and any webhook URL from os.environ — never hard-code them.

Step-by-Step Implementation

The flow is: check the send guard, upload to Slack, send the email, then record the send. Slack and email are attempted independently so a Slack outage does not block the invoice email, and each records its own guard entry.

The sequence below shows one channel’s deliver-and-confirm path, including the backoff loop and the guard that makes a retry safe.

Idempotent delivery sequence: guard check, retry-with-backoff upload, and a persisted sent markerThe Delivery worker asks the send guard whether this run and channel were already delivered; if so it returns done. Otherwise it loops: it uploads the file; a rate-limit or 5xx makes the channel return an error, the worker sleeps an exponential backoff honoring Retry-After and retries; a success makes the channel return ok, the worker persists a sent marker to the guard and returns delivered.DeliveryworkerSend guardChannelSlack / SESalready_sent(run, channel)?no — proceedloop[retry ≤ N]alt[429 / 5xx]else[ok]files_upload_v2 / send_raw_emailerror + Retry-Afteroksleep 2ⁿ backoffmark_sent(run, channel)→ delivered

Step 1 — Persist an idempotent send guard

The guard is a tiny keyed store — a database table or a Redis set — recording which (run_signature, channel) pairs have already been delivered. The key is the same run signature the audit ledger uses, so delivery state and generation state share one identity.

import redis  # redis.Redis; swap for a DB table in production

r = redis.Redis.from_url("redis://localhost:6379/0")

def already_sent(run_sig: str, channel: str) -> bool:
    """True if this run was already delivered to this channel."""
    return r.sismember("chargeback:sent", f"{run_sig}:{channel}")

def mark_sent(run_sig: str, channel: str) -> None:
    """Record a successful delivery so a retry becomes a no-op."""
    r.sadd("chargeback:sent", f"{run_sig}:{channel}")

# Expected:
#   >>> already_sent("a1b2c3d4e5f60718", "slack:#finops")
#   False
#   >>> mark_sent("a1b2c3d4e5f60718", "slack:#finops"); already_sent(...)
#   True

Step 2 — Upload the file to Slack with backoff

slack_sdk’s files_upload_v2 uploads the actual CSV or PDF and posts it into a channel with an initial comment. Slack returns HTTP 429 with a Retry-After header under load; tenacity handles the backoff declaratively, and the guard makes the whole call safe to retry.

import os
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

slack = WebClient(token=os.environ["SLACK_BOT_TOKEN"])

@retry(
    retry=retry_if_exception_type(SlackApiError),
    wait=wait_exponential(multiplier=1, max=30),   # 1s, 2s, 4s ... capped at 30s
    stop=stop_after_attempt(5),
    reraise=True,
)
def _upload_to_slack(channel: str, path: str, title: str, comment: str) -> str:
    resp = slack.files_upload_v2(
        channel=channel, file=path, title=title, initial_comment=comment,
    )
    return resp["file"]["id"]

def deliver_slack(run_sig: str, channel: str, path: str, period: str) -> str:
    """Upload once; a repeat call after success is a no-op."""
    key = f"slack:{channel}"
    if already_sent(run_sig, key):
        return "skipped"
    file_id = _upload_to_slack(
        channel, path, title=f"Chargeback {period}",
        comment=f"Chargeback statement for {period}.",
    )
    mark_sent(run_sig, key)
    return file_id

# Expected:
#   >>> deliver_slack("a1b2c3d4e5f60718", "#finops", "out/cc-payments.pdf", "2026-06")
#   'F07ABC123'                      # first call uploads
#   >>> deliver_slack("a1b2c3d4e5f60718", "#finops", "out/cc-payments.pdf", "2026-06")
#   'skipped'                        # second call is idempotent

For a slimmer footprint — a summary line rather than the attachment — an incoming webhook works: requests.post(os.environ["SLACK_WEBHOOK_URL"], json={"text": summary}). Use the webhook for the “your statement is ready” nudge and the bot-token upload when the file itself must land in the channel.

Step 3 — Send the report by email with a MIME attachment

SES’s send_raw_email takes a full MIME message, which is what lets you attach the CSV or PDF. Build a MIMEMultipart with a text body and the file part, then hand its bytes to boto3. The same tenacity policy retries SES Throttling and 5xx.

import boto3
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.application import MIMEApplication
from botocore.exceptions import ClientError

ses = boto3.client("ses", region_name="us-east-1")

def _build_message(sender: str, to: list[str], subject: str, body: str, path: str) -> MIMEMultipart:
    msg = MIMEMultipart()
    msg["Subject"], msg["From"], msg["To"] = subject, sender, ", ".join(to)
    msg.attach(MIMEText(body, "plain"))
    with open(path, "rb") as fh:
        part = MIMEApplication(fh.read())
    part.add_header("Content-Disposition", "attachment", filename=os.path.basename(path))
    msg.attach(part)
    return msg

@retry(
    retry=retry_if_exception_type(ClientError),
    wait=wait_exponential(multiplier=1, max=30),
    stop=stop_after_attempt(5),
    reraise=True,
)
def _send_ses(sender: str, to: list[str], raw: bytes) -> str:
    resp = ses.send_raw_email(
        Source=sender, Destinations=to, RawMessage={"Data": raw},
    )
    return resp["MessageId"]

def deliver_email(run_sig: str, to: list[str], path: str, period: str) -> str:
    """Send the statement as an attachment, exactly once per recipient set."""
    key = f"email:{','.join(sorted(to))}"
    if already_sent(run_sig, key):
        return "skipped"
    sender = os.environ["REPORT_SENDER"]      # e.g. finops@example.com
    msg = _build_message(
        sender, to, f"Chargeback statement — {period}",
        f"Attached is your chargeback statement for {period}.", path,
    )
    message_id = _send_ses(sender, to, msg.as_bytes())
    mark_sent(run_sig, key)
    return message_id

# Expected:
#   >>> deliver_email("a1b2c3d4e5f60718", ["owner@example.com"], "out/cc.csv", "2026-06")
#   '0100018f...-a1b2c3d4-000000'    # SES MessageId on first send
#   >>> deliver_email("a1b2c3d4e5f60718", ["owner@example.com"], "out/cc.csv", "2026-06")
#   'skipped'

Verification

Confirm both channels are idempotent and that a forced retry does not double-deliver.

def verify_idempotent(run_sig: str) -> None:
    """A second delivery attempt must be a no-op on every channel."""
    first_slack = deliver_slack(run_sig, "#finops", "out/cc-payments.pdf", "2026-06")
    second_slack = deliver_slack(run_sig, "#finops", "out/cc-payments.pdf", "2026-06")
    assert first_slack != "skipped" and second_slack == "skipped"
    assert already_sent(run_sig, "email:owner@example.com") in (True, False)

# Expected on a fresh run signature:
#   (no output; the assertion holds — first sends, second skips)

A closed-loop check against Slack confirms the file actually landed:

curl -s -H "Authorization: Bearer $SLACK_BOT_TOKEN" \
  "https://slack.com/api/files.info?file=F07ABC123" | \
  python -c "import sys,json; d=json.load(sys.stdin); print(d['ok'], d['file']['title'])"

True Chargeback 2026-06

Gotchas & Edge Cases

  • files_upload (v1) is deprecated. Use files_upload_v2, which uses Slack’s newer external-upload flow. The v1 method is being retired and will start failing; code written against it needs migration.
  • SES sandbox blocks unverified recipients. A brand-new SES account can only send to verified addresses. A MessageRejected: Email address is not verified error means you are still in the sandbox — request production access before wiring real recipients.
  • SES message size cap. send_raw_email rejects messages over 40 MB (10 MB after MIME encoding in older limits). A large PDF plus base64 overhead can exceed it; if a statement is huge, upload it to object storage and email a link instead of the attachment.
  • Retry after a client timeout that actually succeeded. This is the exact case the send guard exists for. Without it, a socket timeout on a request the server accepted causes tenacity to resend and double-post. Always write the sent marker before returning success, and check it before every attempt — the same idempotency discipline used in implementing retry logic for failed metric pulls.
  • Slack rate limits are per-method, not global. Bursting one statement per cost center through files_upload_v2 trips the method’s tier limit. Serialize uploads per channel and honor the Retry-After header rather than firing the whole batch at once.
  • Partial delivery. Slack can succeed while SES throttles. Because each channel has its own guard key, a re-run redelivers only the failed channel — never re-posting the one that already succeeded. This mirrors the graceful-degradation posture in graceful degradation when billing APIs are down.
  • Wrong file, right channel. The guard keys on the run signature, so regenerating a corrected statement produces a new signature and legitimately re-delivers. If you patch a file without changing its signature, the guard will suppress the corrected send — bump the signature whenever content changes.

Frequently Asked Questions

Should I use a Slack bot token or an incoming webhook?

Use the bot token with files_upload_v2 when the statement file itself must land in the channel — it attaches the CSV or PDF directly. Use an incoming webhook for a lightweight text summary or a “your report is ready” nudge that links elsewhere. Webhooks cannot upload files, so they are a complement to the bot-token path, not a replacement.

How does the send guard prevent double-delivery?

It records each successful (run_signature, channel) pair in a persistent store and is checked before every attempt. If a retry fires after a timeout on a call the server already accepted, the guard sees the pair as sent and returns a no-op instead of posting again. The marker is written only on confirmed success and keyed on the same signature the audit ledger uses.

Why send_raw_email instead of send_email?

send_email builds a simple message from subject and body fields and cannot carry a file attachment. send_raw_email accepts a full MIME message, so you can attach the CSV or PDF with a MIMEApplication part. For a chargeback statement the recipient needs the actual file, so raw email is the correct call.

What backoff strategy is right for delivery?

Bounded exponential backoff with a cap, honoring any Retry-After header the provider returns. Slack sends Retry-After on 429; SES throttling should back off similarly. Cap the wait (30 seconds here) and cap the attempts (five), then fail loudly so the run is flagged rather than retrying forever and hiding a real outage.

How do I deliver only the channel that failed on a re-run?

Give each channel its own guard key — slack:#finops, email:owner@example.com. Because the keys are independent, a re-run checks each separately and redelivers only the ones without a sent marker. The channel that already succeeded stays skipped, so a partial failure never re-posts the successful half.

Back to: Chargeback Reporting Automation