Multi-Cloud Cost Normalization

Mapping AWS, GCP, and Azure database billing units onto a single canonical compute- and storage-equivalent schema so cost signals compare, aggregate, and enforce identically across every provider.

Back to: Cloud Database Cost Fundamentals & Architecture

Database cost attribution fractures the moment a team runs more than one cloud. AWS meters RDS and Aurora in instance-hours, provisioned IOPS, and ACU-hours; GCP prices Cloud SQL by machine type and BigQuery by slot-time; Azure bills DTUs and vCores with the SQL Server license folded into the compute charge. Each provider names the same physical resource differently and prices it on a different clock, so a raw sum of three invoices is not a number any budget, chargeback report, or quota policy can trust. Normalization is the layer that fixes this: it extracts usage at the finest available granularity, translates every heterogeneous unit into a shared denominator, and emits deterministic cost signals that downstream policy engines consume without knowing or caring which cloud produced them. This is the canonical schema that the parent cost fundamentals and attribution architecture depends on, and the same discipline that lets separating compute from storage cost vectors work uniformly regardless of provider.

The diagram below traces how divergent provider billing metrics converge into one canonical cost model that feeds quota enforcement.

Divergent provider billing units converging into one canonical cost schemaAWS, GCP and Azure each expose database usage in incompatible native units. Every source maps its units into a shared Normalization Engine, which classifies each row, applies a performance weight and pins a single currency. The engine outputs a Canonical Cost Schema that separates into a compute-equivalent metric (CEM) and a storage-equivalent metric (SEM); both provider-neutral signals feed the Policy and Quota Engine.map unitsapply weightsAWS RDS / Aurorainstance-hrs · IOPS · ACUGCP Cloud SQL / BigQuerymachine-type · slot-timeAzure SQL DatabaseDTUs · vCore-hoursNormalizationEngineclassify · weightpin one currencyCanonicalCost SchemaCEM + SEMCEMcompute-equivalentSEMstorage-equivalentPolicy &Quota Engine

Billing Model & Attribution Challenges

The core difficulty is that no two providers expose a comparable primitive, and several deliberately blend distinct cost vectors into a single meter. Before any Python touches an SDK, the model has to name the incompatibilities it is normalizing away.

AWS disaggregates cleanly but verbosely. A single Aurora resource emits InstanceUsage:db.r6g.xlarge for provisioned compute, Aurora:ServerlessUsage (ACU-hours) for serverless compute, Aurora:StorageUsage for the distributed volume, Aurora:StorageIOUsage for per-request I/O, and RDS:ChargedBackupUsage for snapshots — five USAGE_TYPE values on one ARN, on daily granularity, with the trailing days restated as charges finalize.

GCP prices Cloud SQL by machine type (which bundles vCPU and memory) and separately by SSD/HDD storage GB and network egress, while BigQuery bills on-demand by bytes scanned or by reserved slot-hours. A slot is not a vCPU: it is a unit of analytical parallelism, so it must be weighted far below a general-purpose core when translated to a compute-equivalent value.

Azure is the most conflated. In the vCore purchasing model the compute charge includes the SQL Server license, so an Azure vCore-hour is not directly comparable to an AWS vCPU-hour without adjusting for the embedded license cost — the same bundling that makes splitting the vCore license from the managed disk tier a prerequisite. The DTU model collapses compute, memory, and I/O into a single opaque throughput unit that can only be normalized statistically, against a benchmarked DTU-to-vCore ratio, never deterministically.

Three attribution edge cases recur across all three clouds and must be handled explicitly rather than discovered in production:

  • Baseline vs ephemeral conflation. Reserved instances, committed use discounts, and savings plans apply at the account or billing-account level, not the resource level. Folding a committed-use credit into a per-resource compute-equivalent value silently understates the normalized cost of every other workload.
  • Storage double-reading. On Aurora and Cloud SQL the compute and storage meters share a resource id. A naive GROUP BY resource sums the compute line and the storage line into one figure, so storage-heavy analytical workloads inflate the compute quota they never touched.
  • Currency and cadence drift. Cost Explorer is daily and unblended; Azure Cost Management can return the enterprise-agreement blended rate; GCP billing export is per-line-item with its own currency field. Normalization must pin one currency, one granularity, and UnblendedCost-equivalent semantics before any weighting is applied.

Formally, normalization resolves the provider-specific quantities into two additive equivalents — a compute-equivalent metric (CEM) keyed to a reference vCPU-hour, and a storage-equivalent metric (SEM) keyed to a canonical GB-month:

CEM=∑iqi⋅ωf(i),SEM=∑jsj⋅σt(j)\text{CEM} = \sum_{i} q_i \cdot \omega_{f(i)}, \qquad \text{SEM} = \sum_{j} s_j \cdot \sigma_{t(j)}

Here q_i is a raw compute quantity in its native unit, ω is the performance weight for its instance family f(i), s_j is a raw storage quantity, and σ is the weight for its storage class t(j). Every downstream comparison operates on CEM and SEM, never on the provider units.

Telemetry Extraction & Metric Normalization

Extraction pulls each provider’s finest-granularity usage records, and normalization maps them into the canonical record before aggregation. The extractors are thin, deterministic wrappers over the provider SDK; the intelligence lives in the classifier and the weight table, both of which must be version-controlled and tested because they are the load-bearing pieces of the pipeline.

The AWS extractor groups by USAGE_TYPE and paginates on NextPageToken, because a wide account easily exceeds one Cost Explorer response page:

import boto3
from datetime import date

ce = boto3.client("ce", region_name="us-east-1")


def fetch_aws_db_usage(start: date, end: date) -> list[dict]:
    """Pull RDS/Aurora usage grouped by USAGE_TYPE; one row per (day, usage_type)."""
    rows: list[dict] = []
    next_token: str | None = None
    while True:
        kwargs = dict(
            TimePeriod={"Start": start.isoformat(), "End": end.isoformat()},
            Granularity="DAILY",
            Metrics=["UnblendedCost", "UsageQuantity"],
            Filter={"Dimensions": {"Key": "SERVICE",
                                   "Values": ["Amazon Relational Database Service"]}},
            GroupBy=[{"Type": "DIMENSION", "Key": "USAGE_TYPE"}],
        )
        if next_token:
            kwargs["NextPageToken"] = next_token
        resp = ce.get_cost_and_usage(**kwargs)
        for period in resp["ResultsByTime"]:
            day = period["TimePeriod"]["Start"]
            for g in period["Groups"]:
                rows.append({
                    "provider": "aws",
                    "day": day,
                    "usage_type": g["Keys"][0],                       # e.g. "USW2-Aurora:ServerlessUsage"
                    "quantity": float(g["Metrics"]["UsageQuantity"]["Amount"]),
                    "unit": g["Metrics"]["UsageQuantity"]["Unit"],
                    "cost_usd": float(g["Metrics"]["UnblendedCost"]["Amount"]),
                })
        next_token = resp.get("NextPageToken")
        if not next_token:
            break
    return rows

The Azure extractor issues a Cost Management query.usage call scoped to the subscription and grouped by MeterSubCategory, which is where Azure encodes the compute-vs-storage distinction:

from azure.identity import DefaultAzureCredential
from azure.mgmt.costmanagement import CostManagementClient
from azure.mgmt.costmanagement.models import (
    QueryDefinition, QueryDataset, QueryAggregation, QueryGrouping, QueryTimePeriod,
)


def fetch_azure_db_usage(subscription_id: str, start: str, end: str) -> list[dict]:
    """Daily SQL Database cost grouped by MeterSubCategory (compute vs storage vs I/O)."""
    client = CostManagementClient(DefaultAzureCredential())
    scope = f"/subscriptions/{subscription_id}"
    query = QueryDefinition(
        type="ActualCost",
        timeframe="Custom",
        time_period=QueryTimePeriod(from_property=start, to=end),
        dataset=QueryDataset(
            granularity="Daily",
            aggregation={"totalCost": QueryAggregation(name="Cost", function="Sum")},
            grouping=[QueryGrouping(type="Dimension", name="MeterSubCategory")],
        ),
    )
    result = client.query.usage(scope=scope, parameters=query)
    cols = [c.name for c in result.columns]
    rows = []
    for raw in result.rows:
        rec = dict(zip(cols, raw))
        rows.append({
            "provider": "azure",
            "meter": rec.get("MeterSubCategory", "unknown"),
            "cost_usd": float(rec.get("Cost", 0.0)),
        })
    return rows

The GCP extractor reads the standard BigQuery billing export with parameterized dates, filtering to the database services and grouping by SKU so slot-hours and storage GB stay distinct:

from google.cloud import bigquery


def fetch_gcp_db_usage(billing_table: str, start: str, end: str) -> list[dict]:
    """Read Cloud SQL and BigQuery SKU costs from the standard billing export."""
    client = bigquery.Client()
    sql = f"""
        SELECT
          service.description AS service,
          sku.description     AS sku,
          usage.unit          AS unit,
          SUM(usage.amount)   AS quantity,
          SUM(cost)           AS cost_usd
        FROM `{billing_table}`
        WHERE DATE(usage_start_time) BETWEEN @start AND @end
          AND service.description IN ('Cloud SQL', 'BigQuery')
        GROUP BY service, sku, unit
    """
    job = client.query(
        sql,
        job_config=bigquery.QueryJobConfig(query_parameters=[
            bigquery.ScalarQueryParameter("start", "DATE", start),
            bigquery.ScalarQueryParameter("end", "DATE", end),
        ]),
    )
    return [
        {"provider": "gcp", "service": r["service"], "sku": r["sku"],
         "unit": r["unit"], "quantity": float(r["quantity"] or 0.0),
         "cost_usd": float(r["cost_usd"] or 0.0)}
        for r in job.result()
    ]

Because each provider restates trailing days and the three exports land on different cadences, ingestion must be idempotent: records are upserted on (provider, day, resource_or_sku, dimension) rather than appended, and every row is run through strict schema validation on billing records so rows missing an allocation tag or a recognizable unit are rejected before they poison an aggregate. When BigQuery slot reservations are in play, per-query cost is amortized rather than billed as flat reservation overhead, using the fraction of the reservation window a query consumed:

uq=slot_msqslot_mswindowu_q = \frac{\text{slot\_ms}_q}{\text{slot\_ms}_{\text{window}}}

That fractional utilization maps directly to a CEM value, so a single expensive analytical query — traced back to its plan through query execution cost modeling — is attributed the compute cost it actually caused instead of an even split of the reservation.

Python Automation Patterns

The three extractors feed one normalization function. The classifier maps a raw provider row onto a canonical dimension and weight key; the weight table converts native units to CEM and SEM. Keeping the weights in data (not code branches) is what makes the model auditable and testable.

import re
from dataclasses import dataclass

# Compute weights are calibrated to a reference on-demand vCPU-hour (CEM = 1.0).
COMPUTE_WEIGHTS = {
    "aws:instance":    1.00,   # provisioned RDS vCPU-hour
    "aws:serverless":  0.90,   # Aurora ACU-hour ≈ 0.9 vCPU-hour
    "azure:vcore":     0.85,   # net of the embedded SQL Server license
    "gcp:vcpu":        0.98,
    "gcp:slot":        0.25,   # a BigQuery slot is not a full vCPU
}
# Storage weights are calibrated to a canonical provisioned GB-month (SEM = 1.0).
STORAGE_WEIGHTS = {
    "provisioned-gb": 1.00,
    "snapshot-gb":    0.50,
    "io-1k-requests": 0.02,
}

_COMPUTE = re.compile(r"(InstanceUsage|ServerlessUsage|vCore|Compute|CPU|slot)", re.I)
_STORAGE = re.compile(r"(Storage|Volume|Snapshot|Backup|IO|IOPS|Disk)", re.I)


def classify(row: dict) -> tuple[str, str] | None:
    """Return (canonical_dimension, weight_key) or None if the row is unattributable."""
    text = " ".join(str(row.get(k, "")) for k in ("usage_type", "meter", "sku"))
    if _STORAGE.search(text):
        key = "snapshot-gb" if re.search(r"snap|backup", text, re.I) else "provisioned-gb"
        return "storage", key
    if _COMPUTE.search(text):
        if re.search(r"slot", text, re.I):
            return "compute", f"{row['provider']}:slot"
        if re.search(r"serverless|ACU", text, re.I):
            return "compute", f"{row['provider']}:serverless"
        return "compute", f"{row['provider']}:vcore" if row["provider"] == "azure" \
            else f"{row['provider']}:instance" if row["provider"] == "aws" \
            else f"{row['provider']}:vcpu"
    return None


@dataclass
class CanonicalCost:
    provider: str
    cem: float          # compute-equivalent units
    sem: float          # storage-equivalent units
    cost_usd: float
    unattributed_usd: float


def normalize(rows: list[dict]) -> CanonicalCost:
    cem = sem = attributed = unattributed = 0.0
    provider = rows[0]["provider"] if rows else "mixed"
    for row in rows:
        verdict = classify(row)
        cost = float(row.get("cost_usd", 0.0))
        if verdict is None:
            unattributed += cost          # feeds the schema-drift alarm below
            continue
        dimension, key = verdict
        qty = float(row.get("quantity", 0.0))
        if dimension == "compute":
            cem += qty * COMPUTE_WEIGHTS.get(key, 1.0)
        else:
            sem += qty * STORAGE_WEIGHTS.get(key, 1.0)
        attributed += cost
    return CanonicalCost(provider, round(cem, 4), round(sem, 4),
                         round(attributed, 2), round(unattributed, 2))

Cost Explorer permits only a low steady-state request rate and each call is billable, so the fan-out that pulls all three providers is concurrency-capped with an async semaphore and wraps each call in a throttle-aware retry — the same async semaphore-controlled concurrency pattern the extraction pillar standardizes on. Never remove the cap to “go faster”; that is what trips the throttle in the first place.

import asyncio
from functools import wraps
from botocore.exceptions import ClientError

_THROTTLE_CODES = {"ThrottlingException", "LimitExceededException", "TooManyRequestsException"}


def retry_throttled(max_attempts: int = 5, base: float = 0.5):
    """Exponential-backoff retry that only catches provider throttling errors."""
    def decorator(fn):
        @wraps(fn)
        async def wrapper(*args, **kwargs):
            for attempt in range(max_attempts):
                try:
                    return await fn(*args, **kwargs)
                except ClientError as exc:
                    code = exc.response["Error"]["Code"]
                    if code not in _THROTTLE_CODES or attempt == max_attempts - 1:
                        raise
                    await asyncio.sleep(base * 2 ** attempt)   # 0.5s, 1s, 2s, 4s …
        return wrapper
    return decorator


async def gather_providers(tasks: list, concurrency: int = 4) -> list:
    """Run provider pulls under a hard concurrency ceiling."""
    sem = asyncio.Semaphore(concurrency)

    async def guarded(coro):
        async with sem:
            return await coro

    return await asyncio.gather(*(guarded(t) for t in tasks))

When a provider endpoint degrades or throttles past the retry budget, the pipeline should not emit zeros — it degrades to the last good cached normalization using the fallback routing pattern for cost APIs, so a transient outage never manifests as a false “spend dropped to nothing” signal that would relax a quota exactly when it should hold.

Quota Enforcement Integration

Normalized CEM and SEM are the only inputs a cross-cloud quota policy needs, because they are already provider-neutral. A budget is expressed once, in compute-equivalent units, and every cloud’s consumption is measured against it on the same scale. The enforcement function turns aggregated equivalents into a discrete action that the control plane can act on:

def to_quota_signal(costs: list[CanonicalCost], budget_cem: float,
                    soft_ratio: float = 0.8) -> dict:
    """Collapse normalized costs into a single cross-cloud quota decision."""
    total_cem = sum(c.cem for c in costs)
    total_sem = sum(c.sem for c in costs)
    utilization = total_cem / budget_cem if budget_cem else 0.0
    if utilization >= 1.0:
        action = "hard_deny"        # block new provisioning / throttle queries
    elif utilization >= soft_ratio:
        action = "soft_alert"       # notify owners, keep serving
    else:
        action = "ok"
    return {
        "cem": round(total_cem, 3),
        "sem": round(total_sem, 3),
        "utilization": round(utilization, 3),
        "action": action,
    }

These decisions become the inputs to hard and soft enforcement tiers: the soft_alert band fires owner notifications and scaling advisories, while hard_deny drives provisioning denial and query queueing. That mapping from a normalized signal to an enforceable boundary is exactly the contract described in translating normalized cost signals into hard and soft limits, and because the signal is already normalized, a single policy governs a workload that spans AWS and GCP without per-cloud special cases. The service accounts that read cost data and write throttle actions must run under the least-privilege access model for cost data: a compromised cost-reader identity should never be able to raise or remove a quota, and every normalization transformation should leave an immutable audit trail.

Normalized CEM utilization crossing the soft and hard thresholds into ok, soft_alert and hard_deny actionsA single utilization bar runs from zero to 1.20. The zone below 0.80 is the ok band. The narrow zone from 0.80 to 1.00 is the soft_alert band. The zone at or above 1.00 is the hard_deny band. Each band drops down to an action card: ok keeps serving, soft_alert notifies owners while continuing to serve, and hard_deny blocks provisioning and queues queries.utilization = total_cem / budget_cem0.80 · soft_ratio1.00 · budgetok bandsofthardokutilization < 0.80keep servingsoft_alert0.80 ≤ util < 1.00notify owners, keep servinghard_denyutil ≥ 1.00block provisioning · queue queries

Failure Modes & Troubleshooting

The normalization pipeline fails in a small set of characteristic ways. Recognizing the signature is most of the fix.

  • ThrottlingException / LimitExceededException from Cost Explorer. The account exceeded Cost Explorer’s low steady-state QPS, or too many providers were pulled at once. Resolution: the retry_throttled decorator plus the semaphore ceiling in gather_providers; treat rate-limit handling as a first-class concern, as covered in handling rate limits when pulling database metrics.
  • unattributed_usd climbing over time. A provider shipped a new USAGE_TYPE, MeterSubCategory, or SKU that the classifier’s regex table does not match — classic schema drift. Resolution: alarm when unattributed / (attributed + unattributed) crosses ~2%, dump the offending keys, and extend the weight table with a test rather than silently folding the spend into compute or storage.
  • CEM that does not reconcile to the invoice. Usually account-level credits, reserved instances, committed-use discounts, or savings plans applied above the resource. Resolution: normalize UnblendedCost-equivalent figures, keep committed-use credits in a separate baseline dimension, and reconcile monthly against each provider’s detailed export, wiring failures through error handling in cost pipelines.
  • Storage inflating the compute quota. A resource-scoped sum on Aurora or Cloud SQL read the shared storage line into the compute figure. Resolution: classify every row before summing and keep CEM and SEM strictly separate, exactly as the weighting model above enforces.
  • Azure vCore-hours over-weighted. Comparing an Azure vCore-hour directly to an AWS vCPU-hour double-counts the embedded SQL Server license. Resolution: use the license-adjusted azure:vcore weight and split the license out at extraction time.
  • Empty or zeroed output during an outage. A provider API was unavailable and the extractor returned nothing, collapsing the normalized total. Resolution: degrade to the last good cached breakdown via graceful degradation when billing APIs are down rather than emitting zeros that would falsely relax a quota.

Back to: Cloud Database Cost Fundamentals & Architecture