Implementing a Circuit Breaker for Cost Explorer

AWS Cost Explorer combines a low default request quota with occasional multi-second latency, so a naive retry loop around get_cost_and_usage can pin every worker in your cost pipeline against a failing endpoint — this page builds an async circuit breaker that trips fast and fails loudly instead.

Back to: Fallback Routing for Cost APIs

Cost Explorer (ce) is not a high-throughput API. It throttles aggressively, its get_cost_and_usage calls can take seconds under load, and its rate limits are shared across every caller in the account. When the endpoint degrades, blind retries make things worse: each stalled coroutine holds a connection, the retry storm amplifies the throttle, and your quota-enforcement loop blocks on a call that was never going to succeed. A circuit breaker inverts that behavior. After a run of failures it flips to OPEN and raises CircuitOpenError immediately — no network call — giving the downstream code a fast, deterministic signal to serve the last-good cached cost snapshot rather than hang. It complements the retry-with-backoff logic covered in error handling in cost pipelines: retries absorb transient blips, the breaker absorbs sustained outages.

The breaker moves through three states. Transitions are driven by two counters (consecutive failures and a monotonic timer) and one probe.

Circuit breaker state machine: CLOSED to OPEN on failure threshold, OPEN to HALF_OPEN after the reset timeout, HALF_OPEN to CLOSED on a successful probe or back to OPEN on a failed probeThe breaker begins in CLOSED where calls pass through. Reaching the failure threshold moves it to OPEN, where calls fail fast. After the reset timeout it moves to HALF_OPEN and admits one probe. A successful probe returns it to CLOSED; a failed probe returns it to OPEN and restarts the timer.failures ≥ thresholdreset timeout elapsedprobe failsprobe succeedsCLOSEDcalls pass throughOPENfail fast — CircuitOpenErrorHALF_OPENadmit one probe call

Prerequisites

Before wiring the breaker into a pipeline, confirm the following.

  • IAM permissions: the identity running the job needs read-only Cost Explorer access. ce:GetCostAndUsage is the only action this page calls; grant nothing broader. Scoping this down is part of least-privilege access control for cost data.

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadCostExplorer",
      "Effect": "Allow",
      "Action": ["ce:GetCostAndUsage"],
      "Resource": "*"
    }
  ]
}

    Cost Explorer does not support resource-level ARNs, so Resource must be *; constrain the principal instead of the resource.

  • Python: 3.10 or newer (the code uses asyncio locks, time.monotonic, and modern typing).

  • Libraries: install the async AWS client. aioboto3 pulls in aiobotocore and botocore transitively.

    pip install "aioboto3>=13.0"

  • Region: Cost Explorer is a global service addressed through us-east-1. Always construct the client with region_name="us-east-1" regardless of where your workloads run.

Step-by-Step Implementation

We model the states and the fast-fail error, build a reusable async breaker, teach it which Cost Explorer failures should count, and finally hand off to the cache layer when the circuit is open.

Step 1 — Model the states and the fast-fail error

Keep the state machine and its error in one small module so both the breaker and the calling code import the same symbols. CircuitOpenError is what lets the caller branch to a fallback without inspecting boto3 internals.

from enum import Enum


class CircuitState(str, Enum):
    CLOSED = "closed"        # normal operation; calls pass through
    OPEN = "open"            # tripped; calls fail fast
    HALF_OPEN = "half_open"  # trial window; one probe admitted


class CircuitOpenError(RuntimeError):
    """Raised immediately while the breaker is OPEN, without calling the API.

    Catch this in the caller to serve a cached snapshot instead of blocking
    on an endpoint that is known to be failing.
    """

Step 2 — Build the async circuit breaker

The breaker guards every state read and mutation with an asyncio.Lock so concurrent coroutines cannot corrupt the failure counter or race two probes through a half-open window. It records a failure only when a caller-supplied predicate says the exception is worth counting — a malformed request should never trip the circuit.

import asyncio
import time
from typing import Awaitable, Callable, TypeVar

T = TypeVar("T")


class AsyncCircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,
        reset_timeout: float = 30.0,
        half_open_max: int = 1,
        should_trip: Callable[[BaseException], bool] = lambda exc: True,
    ) -> None:
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout          # seconds to stay OPEN
        self.half_open_max = half_open_max          # probes allowed in HALF_OPEN
        self._should_trip = should_trip
        self._state = CircuitState.CLOSED
        self._failures = 0
        self._opened_at = 0.0                        # time.monotonic() timestamp
        self._half_open_calls = 0
        self._lock = asyncio.Lock()

    @property
    def state(self) -> CircuitState:
        return self._state

    async def call(self, fn: Callable[..., Awaitable[T]], *args, **kwargs) -> T:
        """Run fn() through the breaker, or fail fast if the circuit is OPEN."""
        async with self._lock:
            self._admit()          # may raise CircuitOpenError or flip to HALF_OPEN
        try:
            result = await fn(*args, **kwargs)
        except BaseException as exc:
            if self._should_trip(exc):
                async with self._lock:
                    self._record_failure()
            raise
        async with self._lock:
            self._record_success()
        return result

    def _admit(self) -> None:
        if self._state is CircuitState.OPEN:
            if time.monotonic() - self._opened_at >= self.reset_timeout:
                self._state = CircuitState.HALF_OPEN   # timer elapsed: try a probe
                self._half_open_calls = 0
            else:
                raise CircuitOpenError("circuit is OPEN; refusing Cost Explorer call")
        if self._state is CircuitState.HALF_OPEN:
            if self._half_open_calls >= self.half_open_max:
                raise CircuitOpenError("half-open probe budget exhausted")
            self._half_open_calls += 1

    def _record_success(self) -> None:
        # A success in any admitting state clears the fault and closes the circuit.
        self._failures = 0
        self._state = CircuitState.CLOSED

    def _record_failure(self) -> None:
        self._failures += 1
        # A failed probe in HALF_OPEN re-opens immediately; otherwise wait for threshold.
        if self._state is CircuitState.HALF_OPEN or self._failures >= self.failure_threshold:
            self._state = CircuitState.OPEN
            self._opened_at = time.monotonic()

Using time.monotonic() rather than time.time() is deliberate: the reset timer must not jump if NTP steps the wall clock during an incident.

Step 3 — Wrap the Cost Explorer call and classify failures

The actual work is a single get_cost_and_usage call. The classifier tells the breaker to count throttling, service errors, and connection faults, but to ignore client-side mistakes like a bad date range — those are bugs, not outages, and should not open the circuit for healthy callers.

import aioboto3
from botocore.exceptions import (
    ClientError,
    EndpointConnectionError,
    ConnectTimeoutError,
    ReadTimeoutError,
)

# Error codes that indicate the endpoint is degraded, not that we sent junk.
TRANSIENT_CODES = {
    "ThrottlingException",
    "TooManyRequestsException",
    "RequestLimitExceeded",
    "ServiceUnavailable",
    "InternalFailure",
}


def is_transient(exc: BaseException) -> bool:
    """Only endpoint-degradation faults should trip the breaker."""
    if isinstance(exc, (EndpointConnectionError, ConnectTimeoutError,
                        ReadTimeoutError, asyncio.TimeoutError)):
        return True
    if isinstance(exc, ClientError):
        return exc.response.get("Error", {}).get("Code") in TRANSIENT_CODES
    return False


_session = aioboto3.Session()
breaker = AsyncCircuitBreaker(
    failure_threshold=5,
    reset_timeout=30.0,
    should_trip=is_transient,
)


async def fetch_daily_cost(start: str, end: str) -> dict:
    """One raw Cost Explorer pull for [start, end) grouped by service."""
    async with _session.client("ce", region_name="us-east-1") as ce:
        return await ce.get_cost_and_usage(
            TimePeriod={"Start": start, "End": end},   # YYYY-MM-DD, end exclusive
            Granularity="DAILY",
            Metrics=["UnblendedCost"],
            GroupBy=[{"Type": "DIMENSION", "Key": "SERVICE"}],
        )


async def get_cost_guarded(start: str, end: str) -> dict:
    """Cost Explorer through the breaker; raises CircuitOpenError when tripped."""
    return await breaker.call(fetch_daily_cost, start, end)

Step 4 — Fall back to cached state when the circuit is open

The breaker’s whole value is the branch it enables. When get_cost_guarded raises CircuitOpenError (or a transient error that has not yet crossed the threshold), the caller reads the last-good snapshot instead of blocking. The cache layer itself — TTL, staleness flags, and the store/load calls — is built in serving cached cost state during provider outages.

from cost_cache import store_snapshot, load_snapshot  # sibling-page cache layer


async def get_cost_with_fallback(start: str, end: str) -> tuple[dict, bool]:
    """Return (snapshot, is_stale). is_stale=True means the provider was down."""
    try:
        raw = await get_cost_guarded(start, end)
        await store_snapshot(start, end, raw)          # keep the last-good copy
        return raw, False
    except (CircuitOpenError, ClientError, EndpointConnectionError,
            ConnectTimeoutError, ReadTimeoutError) as exc:
        cached = await load_snapshot(start, end)
        if cached is None:
            raise                                      # cold cache: nothing to serve
        return cached, True                            # serve stale, flagged


if __name__ == "__main__":
    snapshot, stale = asyncio.run(get_cost_with_fallback("2026-07-01", "2026-07-18"))
    print("breaker:", breaker.state.value, "stale:", stale)

Expected output when Cost Explorer is healthy on the first call:

breaker: closed stale: False

And after five consecutive throttles have tripped the circuit, the same call returns instantly from cache:

breaker: open stale: True

Verification

Exercise the state machine directly rather than waiting for a real outage. A fake call that fails on demand proves every transition without touching AWS.

import asyncio

async def _verify() -> None:
    calls = {"n": 0}

    async def flaky() -> str:
        calls["n"] += 1
        raise EndpointConnectionError(endpoint_url="https://ce.us-east-1.amazonaws.com")

    cb = AsyncCircuitBreaker(failure_threshold=3, reset_timeout=0.2,
                             should_trip=is_transient)

    # 3 real failures trip the circuit.
    for _ in range(3):
        try:
            await cb.call(flaky)
        except EndpointConnectionError:
            pass
    assert cb.state is CircuitState.OPEN

    # While OPEN, calls fail fast and never reach flaky().
    before = calls["n"]
    try:
        await cb.call(flaky)
    except CircuitOpenError:
        pass
    assert calls["n"] == before, "OPEN circuit must not invoke the target"

    # After the reset timeout a probe is admitted (HALF_OPEN) and re-opens on failure.
    await asyncio.sleep(0.25)
    try:
        await cb.call(flaky)
    except EndpointConnectionError:
        pass
    assert cb.state is CircuitState.OPEN
    print("all transitions verified")

asyncio.run(_verify())

Expected output:

all transitions verified

The load-bearing assertion is calls["n"] == before: it proves an OPEN breaker short-circuits without invoking the target, which is the entire point.

Gotchas & Edge Cases

  • Breaker state is per-process, not global. Each worker holds its own counters, so a fleet of ten pods needs ten failures-worth of throttling per pod before all trip. For a shared view, publish state to Redis or accept the per-process approximation; do not assume one open circuit protects the whole fleet.
  • One breaker per (account, region, API). Cost Explorer throttles per account, so a single account’s breaker instance is correct — but never share one instance across different AWS accounts. A failing payer account would otherwise open the circuit for healthy linked accounts.
  • Do not count ValidationException or DataUnavailableException. A bad TimePeriod or a period Cost Explorer has not finalized is a client-side condition. The is_transient predicate deliberately excludes them so a coding bug cannot open the circuit for everyone else — the same discrimination the retry logic for failed metric pulls applies before a retry.
  • Cost Explorer’s request quota is genuinely low. The account-wide limit is small and paginated calls consume it fast. Combine the breaker with client-side rate limiting; the breaker handles sustained failure, not steady-state throttle avoidance.
  • Half-open thundering herd. With half_open_max=1 per process, a large fleet can still send one probe each the instant the timer elapses. Add jitter to reset_timeout per instance so probes do not synchronize into a fresh throttle spike.
  • The breaker is not a retry. It intentionally fails fast. Layer a bounded retry with backoff inside the guarded function for transient blips, and let the breaker catch the case where retries keep losing — the two mechanisms compose, they do not replace each other.
  • Cold cache on first outage. If the circuit opens before any snapshot was ever stored, the fallback re-raises. Prime the cache on startup so the very first outage still has something conservative to serve.

Frequently Asked Questions

How is a circuit breaker different from retry with backoff?

Retries assume the next attempt might succeed and are correct for transient, self-healing blips. A circuit breaker assumes the endpoint is currently broken and stops calling it entirely for a cool-down window, which prevents a retry storm from amplifying an outage. Use both: retry inside the guarded call for short blips, and let the breaker trip when retries keep failing.

What failure threshold and reset timeout should I pick for Cost Explorer?

Start with a threshold of 5 consecutive transient failures and a 30-second reset timeout, then tune from incident data. Because Cost Explorer’s quota is low, a threshold that is too high leaves workers hammering a throttled endpoint; one that is too low trips on normal jitter. Add per-instance jitter to the reset timeout so a fleet does not probe in lockstep.

Should throttling exceptions trip the breaker?

Yes — ThrottlingException and TooManyRequestsException are exactly the sustained-degradation signal the breaker exists to catch. What should not trip it are client-side errors like ValidationException, which mean your request was malformed. The is_transient predicate encodes that split so a bad request never opens the circuit for healthy callers.

Does the breaker share state across processes or pods?

Not in this implementation — state lives in one process’s memory. Each worker maintains its own counters, which is simple and race-free but means the fleet trips gradually rather than all at once. If you need a shared circuit, publish the open/closed state to Redis and have workers read it before admitting a call, accepting the added latency and coupling.

What happens on the very first request if the API is already down?

The guarded call raises, the fallback tries the cache, and if nothing was ever stored it re-raises because there is genuinely no cost state to serve. Prime the cache at deploy time with a recent snapshot so the first outage after a cold start still degrades gracefully instead of erroring.

Back to: Fallback Routing for Cost APIs