Fallback Routing for Cost APIs #

Keeping database cost attribution and quota enforcement deterministic when the upstream billing API rate-limits, returns stale data, or changes its schema underneath you.

Back to: Cloud Database Cost Fundamentals & Architecture

Cost attribution pipelines for managed database services run under strict expectations for data freshness, dimensional accuracy, and continuous quota enforcement. When an upstream billing API rate-limits a poll, ships a delayed aggregation manifest, or renames a billing dimension, the automated chargeback workflows and resource-boundary controllers downstream cannot afford to block synchronously or silently emit zeros. Fallback routing is the pattern that keeps metric extraction and quota evaluation deterministic while primary telemetry is degraded: instead of failing the run, the pipeline descends through a ranked set of alternative sources, records exactly which one served the value, and attaches a confidence score so every downstream consumer knows how much to trust it. This page covers the billing edge cases that force a fallback, how to extract and normalize metrics from each tier, the Python that wires it together, how the resulting signals feed quota boundary policies, and the failure modes to expect in production.

The decision tree below traces how a single cost-API request descends through the ranked tiers as each health signal fails.

Billing Model & Attribution Challenges #

Fallback routing exists because provider cost APIs are eventually-consistent financial systems, not real-time meters. Three failure classes force a route change, and each demands a different downstream path:

• Transient transport errors — HTTP 429 ThrottlingException, 500/503 from the provider gateway, or connection resets. These are retryable in place and only trigger a tier change once the retry budget is exhausted.
• Delayed aggregation — AWS Cost Explorer and the Cost and Usage Report (CUR) restate the last 24–72 hours as line items settle; Azure Cost Management amortizes reservation and savings-plan charges across the term; GCP BigQuery billing export lands partition-by-partition. A 200 OK here is structurally valid but temporally wrong, which is the most dangerous failure because it looks healthy.
• Structural schema drift — a deprecated dimension (UnblendedCost semantics shifting under new pricing), a renamed GroupBy key, or a new RECORD-typed column in the BigQuery export that breaks a flat-row assumption.

The blended-versus-disaggregated distinction matters at every tier. A primary API may hand back a blended figure that hides whether spend went to compute or storage; reconstructing that split is the job of the compute and storage cost breakdown logic, and a fallback source must preserve the same grain or the breakdown silently collapses. Reserved-instance and savings-plan amortization is another edge case: a warehouse cache built from unblended CUR rows will disagree with a primary API returning amortized cost unless both are normalized to the same cost metric before comparison.

The routing topology follows a ranked hierarchy, from most authoritative to most approximate:

1. Primary provider API — direct REST/GraphQL ingestion (Cost Explorer, Azure Cost Management, GCP billing) with a strict timeout budget and structured retries.
2. Regional / endpoint mirror — a geographically isolated API endpoint or a read replica of the export that draws from a separate rate-limit pool, so localized throttling does not stall the whole pipeline.
3. Warehouse cache fallback — pre-materialized Parquet/Avro extracts of the CUR or billing export in object storage or an analytical warehouse, refreshed on a schedule and queried when the live API is unavailable.
4. Deterministic synthetic estimation — a rule-based projection from the last-known-good baseline, used only when every richer source is exhausted and always flagged as low-confidence.

Transitions between tiers must be stateless, idempotent, and explicitly logged for FinOps audit trails, and driven by observed health signals rather than blind retries so that a fallback path never amplifies the upstream load that caused the degradation.

Telemetry Extraction & Metric Normalization #

Each tier speaks a different dialect, so the router’s contract is a single normalized cost record that every source must produce before its value is accepted. Normalization covers three axes: units (vCPU-hours, IOPS-seconds, TiB-months), cost metric (unblended, amortized, or net), and identity (tenant, project, cost_center, and resource tags). Aligning tags across sources is the same canonical-schema discipline described in multi-cloud cost normalization; a fallback that emits a differently-keyed record is worse than no data, because it corrupts aggregation silently.

The primary tier for AWS reads Cost Explorer with a bounded client and a fixed GroupBy, so the shape is identical regardless of which tier ultimately serves the value:

import boto3
from botocore.config import Config

# Adaptive retry mode backs off automatically on ThrottlingException;
# max_attempts caps in-place retries before the router escalates a tier.
_CE_CONFIG = Config(
    retries={"max_attempts": 4, "mode": "adaptive"},
    connect_timeout=3,
    read_timeout=10,
)


def fetch_primary(start: str, end: str) -> list[dict]:
    """Pull daily unblended cost grouped by tenant tag from Cost Explorer."""
    client = boto3.client("ce", config=_CE_CONFIG)
    response = client.get_cost_and_usage(
        TimePeriod={"Start": start, "End": end},  # end is exclusive
        Granularity="DAILY",
        Metrics=["UnblendedCost", "UsageQuantity"],
        GroupBy=[{"Type": "TAG", "Key": "tenant"}],
    )
    return response["ResultsByTime"]

Pagination is not optional at scale. Cost Explorer returns a NextPageToken, and dropping it truncates the ledger without raising an error, which reads downstream as a spurious spend drop. The extraction loop must exhaust the token before the record is considered complete:

def fetch_primary_paginated(client, start: str, end: str) -> list[dict]:
    results, token = [], None
    while True:
        kwargs = {
            "TimePeriod": {"Start": start, "End": end},
            "Granularity": "DAILY",
            "Metrics": ["UnblendedCost", "UsageQuantity"],
            "GroupBy": [{"Type": "TAG", "Key": "tenant"}],
        }
        if token:
            kwargs["NextPageToken"] = token
        page = client.get_cost_and_usage(**kwargs)
        results.extend(page["ResultsByTime"])
        token = page.get("NextPageToken")
        if not token:
            return results

The warehouse tier reads the pre-materialized extract instead of the live API. Because the columns were already flattened at ETL time, the read is a cheap columnar scan against object storage — the mechanics of building those extracts are covered in batch processing for historical metrics:

import awswrangler as wr


def fetch_warehouse(bucket: str, prefix: str, day: str) -> "pandas.DataFrame":
    """Read the daily CUR extract already normalized into Parquet."""
    df = wr.s3.read_parquet(path=f"s3://{bucket}/{prefix}/dt={day}/")
    # Project to the canonical record columns the router expects.
    return df[["tenant", "usage_type", "usage_unit", "quantity", "unit_cost"]]

Whatever the tier, every candidate value is validated against the same contract before the router accepts it. Strict typing with Pydantic v2 rejects a structurally-drifted payload at the boundary rather than letting a renamed dimension propagate into the ledger — the enforcement layer is detailed in schema validation for billing data and, concretely, in validating JSON billing payloads with Pydantic:

from decimal import Decimal
from pydantic import BaseModel, ConfigDict, field_validator


class CostRecord(BaseModel):
    model_config = ConfigDict(strict=True, frozen=True)

    tenant: str
    usage_type: str
    usage_unit: str
    quantity: Decimal
    unit_cost: Decimal
    source_tier: str          # primary | mirror | warehouse | synthetic
    confidence: float         # 1.0 for live, lower for cached/estimated

    @field_validator("quantity", "unit_cost")
    @classmethod
    def non_negative(cls, v: Decimal) -> Decimal:
        if v < 0:
            raise ValueError("cost dimensions must be non-negative")
        return v

A candidate that fails this validation never becomes a fallback value; it is quarantined and routed to a dead-letter queue for asynchronous reconciliation, exactly as the graceful degradation path when billing APIs are down prescribes.

Python Automation Patterns #

Two building blocks turn the tier list into a router: a circuit breaker that stops hammering a failing endpoint, and an ordered dispatch that walks the tiers and stamps each result with its provenance. Keeping routing logic decoupled from business logic lets the telemetry-ingestion and quota-evaluation services scale independently.

In-place retries against a single tier use exponential backoff with jitter. The Tenacity decorator expresses the retry budget declaratively so it stays consistent with the retry logic used across the metric pipeline:

from botocore.exceptions import ClientError
from tenacity import (
    retry, retry_if_exception, stop_after_attempt,
    wait_exponential_jitter,
)

_RETRYABLE = {"ThrottlingException", "TooManyRequestsException",
              "ServiceUnavailable", "RequestTimeout"}


def _is_retryable(exc: Exception) -> bool:
    return (
        isinstance(exc, ClientError)
        and exc.response["Error"]["Code"] in _RETRYABLE
    )


@retry(
    retry=retry_if_exception(_is_retryable),
    wait=wait_exponential_jitter(initial=0.5, max=20),
    stop=stop_after_attempt(4),
    reraise=True,
)
def call_with_retry(fn, *args, **kwargs):
    return fn(*args, **kwargs)

Once the in-place budget is spent, a circuit breaker prevents the router from re-attempting a tier that is systemically down. A minimal breaker tracks consecutive failures and enforces a cooldown before it allows a probe:

import time


class CircuitBreaker:
    def __init__(self, threshold: int = 5, cooldown: float = 60.0):
        self.threshold = threshold
        self.cooldown = cooldown
        self._failures = 0
        self._opened_at = 0.0

    def is_open(self) -> bool:
        if self._failures < self.threshold:
            return False
        # Half-open: allow a single probe once the cooldown elapses.
        if time.monotonic() - self._opened_at >= self.cooldown:
            return False
        return True

    def record_success(self) -> None:
        self._failures = 0

    def record_failure(self) -> None:
        self._failures += 1
        if self._failures >= self.threshold:
            self._opened_at = time.monotonic()

The router composes these into an ordered walk. Each tier is a callable paired with a breaker and a confidence weight; the first tier that returns a value passing validation wins, and its provenance is stamped onto the record:

import logging

log = logging.getLogger("cost.fallback")


def route_cost(tiers, record_fields) -> CostRecord:
    """Walk tiers in priority order; return the first validated result."""
    for name, fetch, breaker, confidence in tiers:
        if breaker.is_open():
            log.warning("tier %s open, skipping", name)
            continue
        try:
            raw = call_with_retry(fetch)
            breaker.record_success()
            record = CostRecord(
                **record_fields(raw),
                source_tier=name,
                confidence=confidence,
            )
            log.info("served from tier=%s confidence=%.2f", name, confidence)
            return record
        except Exception as exc:  # noqa: BLE001 - classified below
            breaker.record_failure()
            log.warning("tier %s failed: %s", name, exc)
    raise RuntimeError("all cost tiers exhausted")

For high-fanout extraction — many tenants, many accounts — the same router runs under async with a bounded semaphore so a fallback storm cannot open thousands of concurrent connections at once; that concurrency discipline is the subject of the async usage-parsing workflows, and pairing it with aioboto3 keeps the event loop from blocking on a slow tier.

When every richer tier is exhausted, the synthetic tier projects from the last-known-good baseline. A defensible estimate scales the trailing mean cost by the ratio of current to trailing-mean usage, so a tenant whose usage spiked is not billed a flat average:

The estimate is emitted with a confidence well below 1.0, tagged source_tier="synthetic", and marked provisional so the reconciliation job overwrites it once the primary API restates the true figure.

Quota Enforcement Integration #

Fallback routing exists to keep the quota controller running, so the contract between them is the confidence score. When the primary tier serves a value, the controller enforces hard limits normally. When a fallback tier serves it, the controller shifts to a more conservative posture: it may enforce soft limits, widen alert thresholds, or freeze increases to allocations rather than acting on numbers it cannot fully trust. Translating these normalized cost signals into hard and soft limits is the responsibility of database quota boundary design, and the confidence field is what lets that layer degrade gracefully instead of enforcing on estimates.

def enforce(record: CostRecord, hard_limit: Decimal) -> str:
    projected = record.quantity * record.unit_cost
    if record.confidence >= 0.99:
        return "block" if projected > hard_limit else "allow"
    # Degraded telemetry: never hard-block on a low-confidence estimate;
    # alert and hold instead so a stale reading cannot halt provisioning.
    if projected > hard_limit:
        return "alert_and_hold"
    return "allow"

Aligning fallback payloads to the same baselines used by query execution cost modeling lets the controller project consumption trends and adjust dynamic throttling thresholds even while the live feed is dark, so enforcement continuity does not depend on the billing API being reachable at the moment a limit is tested.

Fallback data must never become a way to bypass policy. A degraded value flows through the same authorization and redaction path as a live one: raw, unvalidated billing payloads are never exposed to downstream consumers, and the confidence score is itself an audited field. Keeping the fallback path inside the controls described in security and access control for cost data ensures that degraded telemetry cannot be used to sneak an over-budget allocation past enforcement.

Failure Modes & Troubleshooting #

• ThrottlingException storms during a fallback cascade. A failing primary tier that is retried too aggressively pushes the whole pipeline into the mirror tier at once, which then throttles too. Fix by using mode="adaptive" on the botocore Config, capping max_attempts, and letting the circuit breaker open so retries stop feeding the fire.
• Stale-but-valid cache served as fresh. The warehouse tier returns a structurally perfect record from a manifest that is hours old. Guard it with an explicit freshness window: compare the extract’s dt partition against datetime.now(timezone.utc) and treat anything past the reconciliation window as a miss, not a hit, so the router falls through to synthetic estimation instead of trusting stale numbers.
• Schema drift passing validation. A renamed dimension can still deserialize if the model is permissive. ConfigDict(strict=True) plus explicit field validators reject the drift at the boundary; pin the expected column set and fail closed when an unexpected RECORD column appears in a BigQuery export.
• Dimensional parity loss across tiers. The warehouse extract groups by a different tag than the live API, so tenant totals do not reconcile. Enforce one canonical GroupBy key everywhere and assert the tenant set matches the known roster before accepting the record.
• Blended vs amortized mismatch. Primary returns amortized cost while the cache holds unblended rows, producing a phantom variance at month boundaries. Normalize both to a single declared cost metric before any comparison, and record which metric was used on the cost record.
• Silent pagination truncation. A dropped NextPageToken looks like a spend drop, not an error. Always exhaust the token and assert the returned period count matches the requested range before the record is marked complete.

Broader retry, dead-letter, and degradation strategy for the whole ingestion path lives in error handling in cost pipelines; this page covers the routing decision specifically, and defers the general resilience taxonomy there.

Related #

• Database Quota Boundary Design — translate normalized cost signals into hard and soft limits, using the confidence score to degrade enforcement safely.
• Multi-Cloud Cost Normalization — the canonical schema every fallback tier must emit so records stay comparable across providers.
• Compute vs Storage Cost Breakdowns — preserving the compute/storage grain through a fallback so the breakdown does not collapse.
• Query Execution Cost Modeling — the consumption baselines that anchor deterministic synthetic estimation.
• Error Handling in Cost Pipelines — the wider retry, dead-letter, and graceful-degradation strategy this routing pattern plugs into.

Back to: Cloud Database Cost Fundamentals & Architecture