Serving Cached Cost State During Provider Outages

When a cost provider API is throttled or down, a cost pipeline that has no cached fallback either blocks or returns nothing — this page builds the cache layer that keeps the last-good cost snapshot on hand and serves it with an explicit staleness flag and a bounded grace window.

Back to: Fallback Routing for Cost APIs

This is the layer a tripped breaker falls back to. When a circuit breaker for Cost Explorer raises CircuitOpenError, or when any provider call fails, the caller needs something to serve — and serving a stale-but-known number, clearly labelled as stale, is almost always better than serving an error to a dashboard or an enforcement loop. The design has three moving parts: a snapshot store that keeps the last successful cost pull, a freshness policy with a TTL and a hard maximum grace window, and a serving path that flags staleness and refreshes in the background. It sits alongside the broader graceful degradation strategy for billing APIs; where that page covers the whole degradation posture, this one is the concrete cache mechanism underneath it.

The decision flow below shows how a read resolves against the cache, when it serves fresh versus stale, and when it must fall through to the provider or a conservative hold.

Cached cost state serving decision: fresh within TTL, stale within grace with background revalidation, or provider fetch and conservative hold when expiredA cost read request performs a cache lookup, then a gate compares snapshot age to the TTL. Within TTL it serves fresh. Past TTL but within max grace it serves stale, flagged, and revalidates in the background. Beyond TTL plus grace, or on a cache miss, it fetches from the provider and, if that fails, applies a conservative enforcement hold.age ≤ TTLTTL < age ≤ +graceage > grace / missasyncon failurerefreshCost read requestCache lookupsnapshot agevs TTL?Serve freshServe stalestale=true flagFetch from providerRevalidatein backgroundConservativeenforcement hold

Prerequisites

The cache layer is provider-agnostic — it stores whatever your extraction path produced. This page uses AWS Cost Explorer snapshots for concreteness.

  • A source of snapshots: any successful cost pull. The examples assume the guarded fetch built in the circuit-breaker page supplies the raw Cost Explorer response.

  • Python: 3.10 or newer (dataclasses with | unions, asyncio.create_task).

  • Redis: a reachable instance for the shared cache; the local JSON store needs only a writable directory and no extra service.

    pip install "redis>=5.0"

  • Configuration via environment — never hard-code the connection string or the grace window:

    export REDIS_URL="redis://cache.internal:6379/0"
export SNAPSHOT_DIR="/var/lib/costcache"

Step-by-Step Implementation

We model a snapshot with a freshness policy, persist it to Redis (with a local JSON fallback), serve it with a staleness flag and stale-while-revalidate, and finally make enforcement treat stale data conservatively.

Step 1 — Model the snapshot and its freshness levels

The snapshot is an immutable record of one successful pull plus the wall-clock time it was fetched. Freshness is a pure function of age against two bounds: the TTL (below which data is fresh) and the maximum grace (beyond which stale data is unusable and must not be served).

import time
from dataclasses import dataclass, asdict
from enum import Enum


class Freshness(str, Enum):
    FRESH = "fresh"        # age <= ttl
    STALE = "stale"        # ttl < age <= ttl + max_grace
    EXPIRED = "expired"    # age > ttl + max_grace; never serve this


@dataclass(frozen=True)
class CostSnapshot:
    key: str                        # e.g. "ce:2026-07-01:2026-07-18"
    fetched_at: float               # epoch seconds (time.time)
    total_unblended: float
    by_service: dict[str, float]
    currency: str = "USD"

    def freshness(self, ttl: float, max_grace: float,
                  now: float | None = None) -> Freshness:
        age = (now if now is not None else time.time()) - self.fetched_at
        if age <= ttl:
            return Freshness.FRESH
        if age <= ttl + max_grace:
            return Freshness.STALE
        return Freshness.EXPIRED


# Expected:
#   snap.freshness(ttl=3600, max_grace=21600)  # 30 min old  -> Freshness.FRESH
#   snap.freshness(ttl=3600, max_grace=21600)  # 2 h old     -> Freshness.STALE
#   snap.freshness(ttl=3600, max_grace=21600)  # 8 h old     -> Freshness.EXPIRED

Step 2 — Persist and load the snapshot

Redis is the shared store so every worker sees the same last-good value. Set the key’s hard expiry to ttl + max_grace so an EXPIRED snapshot physically cannot be served — the key is already gone by the time it would qualify. The local JSON store is a drop-in for single-node deployments or a last-resort when Redis itself is unreachable.

import json
import logging
import os
from pathlib import Path
from redis import asyncio as aioredis

TTL_SECONDS = 3600              # snapshot is "fresh" for 1 hour
MAX_GRACE_SECONDS = 6 * 3600   # serve stale for up to 6 hours into an outage

_redis = aioredis.from_url(os.environ.get("REDIS_URL", "redis://localhost:6379/0"))
SNAPSHOT_DIR = Path(os.environ.get("SNAPSHOT_DIR", "/var/lib/costcache"))


async def store_snapshot(snap: CostSnapshot) -> None:
    """Write to Redis with a hard TTL, and mirror to local disk as a backstop."""
    payload = json.dumps(asdict(snap))
    # ex= guarantees the key evaporates once it is past the grace window.
    await _redis.set(snap.key, payload, ex=int(TTL_SECONDS + MAX_GRACE_SECONDS))
    SNAPSHOT_DIR.mkdir(parents=True, exist_ok=True)
    (SNAPSHOT_DIR / f"{snap.key.replace(':', '_')}.json").write_text(payload)


async def load_snapshot(key: str) -> CostSnapshot | None:
    """Prefer Redis; fall back to the local mirror if Redis is unreachable."""
    try:
        payload = await _redis.get(key)
    except aioredis.RedisError as exc:
        logging.warning("Redis unavailable (%s); reading local snapshot", exc)
        payload = None
    if payload is None:
        path = SNAPSHOT_DIR / f"{key.replace(':', '_')}.json"
        if not path.exists():
            return None
        payload = path.read_text()
    return CostSnapshot(**json.loads(payload))

Building the snapshot from a raw Cost Explorer response is a small adapter:

def snapshot_from_ce(key: str, raw: dict) -> CostSnapshot:
    """Reduce a get_cost_and_usage response into a compact snapshot."""
    by_service: dict[str, float] = {}
    for day in raw.get("ResultsByTime", []):
        for group in day.get("Groups", []):
            service = group["Keys"][0]
            amount = float(group["Metrics"]["UnblendedCost"]["Amount"])
            by_service[service] = by_service.get(service, 0.0) + amount
    return CostSnapshot(
        key=key,
        fetched_at=time.time(),
        total_unblended=round(sum(by_service.values()), 4),
        by_service=by_service,
    )

Step 3 — Serve with a staleness flag and background revalidation

The serving path is where the policy pays off. A FRESH snapshot is returned as-is. A STALE snapshot is returned immediately but flagged, and a background task tries to refresh it — the stale-while-revalidate pattern, so a slow provider never blocks the caller. Only an EXPIRED snapshot or a cold cache forces a synchronous provider fetch, which may itself fail.

import asyncio
from typing import Awaitable, Callable

_bg_tasks: set[asyncio.Task] = set()   # hold refs so tasks are not GC'd mid-flight


async def get_cost_state(
    key: str,
    fetch_fresh: Callable[[], Awaitable[CostSnapshot]],
) -> tuple[CostSnapshot, Freshness]:
    """Return (snapshot, freshness). Never blocks on the provider while STALE."""
    snap = await load_snapshot(key)
    if snap is not None:
        level = snap.freshness(TTL_SECONDS, MAX_GRACE_SECONDS)
        if level is Freshness.FRESH:
            return snap, level
        if level is Freshness.STALE:
            task = asyncio.create_task(_revalidate(key, fetch_fresh))
            _bg_tasks.add(task)
            task.add_done_callback(_bg_tasks.discard)
            return snap, level          # serve stale now; refresh happens off-path

    # EXPIRED or cold cache: we must go to the provider and store the result.
    fresh = await fetch_fresh()          # raises if the provider is down
    await store_snapshot(fresh)
    return fresh, Freshness.FRESH


async def _revalidate(key: str,
                      fetch_fresh: Callable[[], Awaitable[CostSnapshot]]) -> None:
    try:
        fresh = await fetch_fresh()
        await store_snapshot(fresh)
        logging.info("revalidated snapshot %s", key)
    except Exception as exc:
        # Provider still down: retain the existing snapshot until grace expires.
        logging.warning("revalidation failed for %s (%s); keeping stale", key, exc)

Expected output across an outage, with fetch_fresh wrapping the guarded Cost Explorer call:

INFO  revalidated snapshot ce:2026-07-01:2026-07-18        # provider healthy, FRESH
WARNING revalidation failed for ce:...:... (CircuitOpenError); keeping stale

Step 4 — Enforce conservatively on stale data

Quota enforcement must never relax a limit because it is looking at stale numbers. When the freshness is STALE, the safe assumption is that spend kept climbing since the snapshot was taken, so the enforcer inflates the cached total by a safety margin and, near the thresholds, holds new provisioning rather than allowing it. This is the discipline that connects the cache to database quota boundary design.

from decimal import Decimal


def enforce_with_staleness(
    snap: CostSnapshot,
    level: Freshness,
    soft: Decimal,
    hard: Decimal,
    stale_margin: Decimal = Decimal("1.10"),
) -> str:
    """Map a (possibly stale) snapshot to an enforcement action, erring safe."""
    total = Decimal(str(snap.total_unblended))
    if level is Freshness.STALE:
        # Blind to spend since the snapshot: assume it grew, never shrank.
        total *= stale_margin
    if total >= hard:
        return "throttle"                # block; a hard limit is never relaxed
    if total >= soft:
        return "alert"
    if level is Freshness.STALE:
        return "hold"                    # freeze new provisioning while blind
    return "ok"


# Expected, with soft=800, hard=1000 and a stale snapshot of 760:
#   inflated 760 * 1.10 = 836 >= soft  -> "alert"

Verification

Confirm the freshness boundaries and the conservative enforcement behave exactly at the edges, using a fixed now so the test is deterministic.

def _verify() -> None:
    base = 1_000_000.0
    snap = CostSnapshot(key="t", fetched_at=base,
                        total_unblended=760.0, by_service={"AmazonRDS": 760.0})

    # Boundary checks against a 1h TTL and 6h grace.
    assert snap.freshness(3600, 21600, now=base + 1800) is Freshness.FRESH
    assert snap.freshness(3600, 21600, now=base + 3 * 3600) is Freshness.STALE
    assert snap.freshness(3600, 21600, now=base + 8 * 3600) is Freshness.EXPIRED

    # Stale data must round UP toward the limit, never down.
    action = enforce_with_staleness(snap, Freshness.STALE,
                                    soft=Decimal("800"), hard=Decimal("1000"))
    assert action == "alert", action     # 760*1.10 = 836 crosses the soft limit
    print("cache freshness + conservative enforcement verified")

_verify()

Expected output:

cache freshness + conservative enforcement verified

A served snapshot handed to a dashboard or API should carry the flag explicitly so consumers can render a banner:

{
  "key": "ce:2026-07-01:2026-07-18",
  "total_unblended": 760.0,
  "currency": "USD",
  "freshness": "stale",
  "fetched_at": "2026-07-18T09:14:02Z"
}

Gotchas & Edge Cases

  • Never serve EXPIRED data. Past the grace window, a stale number is worse than an error because it looks authoritative while being arbitrarily wrong. The Redis ex= hard expiry enforces this at the storage layer; the local mirror check must re-verify freshness on load rather than trusting file existence.
  • Cache stampede on expiry. When a STALE snapshot tips to EXPIRED, every worker fetches the provider at once — the exact retry storm the circuit breaker is there to blunt. Add single-flight coordination (a short Redis lock, e.g. SET NX PX) so only one worker revalidates while the others wait or serve their last stale copy.
  • Background tasks get garbage-collected. asyncio.create_task returns a task the loop only weakly references; without holding it in _bg_tasks, a revalidation can be collected mid-await and silently vanish. Keep the strong reference and discard it in the done-callback.
  • Redis eviction can drop your snapshot early. On a shared instance with allkeys-lru, memory pressure can evict a cost snapshot well before its TTL. Use a dedicated logical database with noeviction, or the local JSON mirror as the guaranteed backstop.
  • Clock skew corrupts freshness. fetched_at and the reader’s now must come from the same trusted clock domain. Stamp fetched_at on the machine that made the successful pull and rely on NTP; a reader whose clock is fast will declare fresh data stale, and one whose clock is slow will serve expired data.
  • Currency and scope must match on hit. A snapshot keyed only by date range can collide across accounts or currencies. Encode account, region, currency, and period into the cache key so a fallback can never serve one tenant’s cost as another’s — a habit shared with graceful degradation for billing APIs.
  • Stale data is not fresh data with a warning. Downstream automation must branch on the freshness flag, not just log it. An auto-scaler or provisioning gate that ignores the flag will make irreversible decisions on numbers that stopped updating hours ago.

Frequently Asked Questions

How do I choose the TTL and the maximum grace window?

Set the TTL to how long a cost figure stays decision-useful — often 30-60 minutes, since provider cost data itself lags by hours. Set the maximum grace to how long you are willing to act on frozen numbers during an outage; 4-6 hours is common. Beyond grace, refusing to serve is safer than serving a figure that could be badly wrong.

What is the difference between TTL expiry and the grace window?

TTL is the line between fresh and stale: past it, data is still served but flagged and revalidated in the background. Grace is the line between stale and expired: past TTL plus grace, the snapshot is discarded and never served. TTL keeps data current in normal operation; grace bounds how long you will lean on the cache during an actual outage.

Should I use Redis or a local JSON snapshot?

Use Redis when multiple workers must share one last-good value and see consistent staleness; use the local JSON store for single-node jobs or as a backstop when Redis itself is down. The example does both: Redis is the primary store and the local mirror is the fallback the loader reaches for when Redis raises.

How should enforcement treat stale cost data?

Conservatively and asymmetrically: inflate the stale total by a safety margin, never relax a limit, and hold new provisioning near the thresholds. Because you cannot see spend since the snapshot, the safe assumption is that it grew. Allowing more provisioning on stale data risks blowing a real budget you simply cannot currently observe.

How does this relate to the circuit breaker?

They compose. The circuit breaker detects that the provider is failing and raises CircuitOpenError fast; this cache is what the caller serves in response. The breaker decides whether to call the provider, and the cache decides what to return when the answer is “do not call.” Together they turn a provider outage into a flagged, bounded degradation instead of an error.

Back to: Fallback Routing for Cost APIs