Mapping Provider SKUs to Canonical Compute Units

AWS, Azure, and GCP each name managed-database compute with an incompatible SKU grammar — db.r6g.xlarge, GP_Gen5_4, db-custom-4-16384 — and this page builds the deterministic classifier that collapses all three into one canonical compute unit of normalized vCPU and memory.

Back to: Multi-Cloud Cost Normalization

Any cross-provider cost comparison is meaningless until an Aurora db.r6g.xlarge, an Azure SQL GP_Gen5_4, and a Cloud SQL db-custom-4-16384 can be measured on the same axis. A per-hour price alone tells you nothing — one SKU might pack twice the memory per vCPU of another — so the unit you actually compare on is capacity, not a provider string. The classifier here parses each provider’s instance-family grammar, resolves it against a lookup table to a (vCPU, memory_GB) pair, and folds that pair into a single canonical compute unit. It is the resolution step that sits directly downstream of schema validation for billing data: validation guarantees a record’s SKU field is well-typed, and this mapping guarantees that field resolves to comparable capacity before any multi-cloud normalization rollup runs.

We reduce two dimensions to one with a reference memory ratio. Given a SKU’s vCPU count and memory in GB, and a reference density $\rho_{\text{ref}}$ (GB of memory per vCPU on the baseline family), the canonical compute unit is:

CCU=vCPU+mem_GBρref\text{CCU} = \text{vCPU} + \frac{\text{mem\_GB}}{\rho_{\text{ref}}}

With $\rho_{\text{ref}} = 4$ GB/vCPU, a general-purpose 4-vCPU/16-GB instance scores exactly 4 + 16/4 = 8 CCU, while a memory-optimized 4-vCPU/32-GB instance scores 4 + 32/4 = 12 CCU — the extra memory it bills for is now visible in the unit itself.

Prerequisites

The mapping runs entirely offline against a static lookup table; no cloud credentials are required to classify a SKU, only to source the SKU strings from billing exports in the first place.

  • Read access to the billing exports that carry the SKU field. On AWS that is the Cost and Usage Report product/instanceType / lineItem/UsageType; on Azure the Cost Management meterName / serviceTier; on GCP the BigQuery billing export sku.description. Sourcing those follows the least-privilege posture in security and access control for cost data — the classifier itself needs no live API.

  • Python: 3.10 or newer (the code uses match statements and modern typing).

  • Libraries: pandas for the batch join, plus the standard-library decimal and re.

    pip install "pandas>=2.1"

Step-by-Step Implementation

The design keeps three concerns separate: a static lookup table of known capacities, a set of pure per-provider parsers that turn a raw SKU into a (vCPU, memory_GB) pair, and a dispatcher that routes by provider and quarantines anything it cannot resolve. Everything downstream of the parsers is Decimal, never float, because a fractional CCU feeds directly into per-unit cost math.

Step 1 — Define the canonical unit and the capacity lookup

The lookup table encodes the two facts a SKU string cannot always self-describe: how many GB a memory-optimized family packs per vCPU, and the vCPU count implied by each named size. AWS sizes follow a strict doubling grammar (large = 2 vCPU, xlarge = 4, Nxlarge = N×4), so we compute those rather than enumerate them.

from decimal import Decimal, getcontext

getcontext().prec = 28  # ample headroom for capacity math

REF_DENSITY_GB_PER_VCPU = Decimal("4")  # rho_ref: baseline general-purpose density

# Memory-per-vCPU density by AWS RDS family prefix.
AWS_FAMILY_DENSITY: dict[str, Decimal] = {
    "t3": Decimal("4"), "t4g": Decimal("4"),   # burstable, general purpose
    "m5": Decimal("4"), "m6g": Decimal("4"), "m6i": Decimal("4"),
    "r5": Decimal("8"), "r6g": Decimal("8"), "r6i": Decimal("8"),  # memory optimized
    "x2g": Decimal("16"),                       # extra memory optimized
}

# vCPU per AWS named size that is not part of the N-xlarge doubling grammar.
AWS_BASE_SIZE_VCPU: dict[str, Decimal] = {
    "medium": Decimal("1"), "large": Decimal("2"), "xlarge": Decimal("4"),
}

# Azure vCore -> GB density for the General Purpose Gen5 tier.
AZURE_TIER_DENSITY: dict[str, Decimal] = {
    "GP_Gen5": Decimal("5.1"),   # General Purpose, Gen5 hardware
    "BC_Gen5": Decimal("5.1"),   # Business Critical, Gen5 hardware
    "HS_Gen5": Decimal("5.1"),   # Hyperscale, Gen5 hardware
}


def canonical_compute_unit(vcpu: Decimal, mem_gb: Decimal) -> Decimal:
    """Collapse (vCPU, memory) into a single canonical compute unit."""
    return vcpu + (mem_gb / REF_DENSITY_GB_PER_VCPU)


# Expected:
#   canonical_compute_unit(Decimal("4"), Decimal("16")) -> Decimal("8")
#   canonical_compute_unit(Decimal("4"), Decimal("32")) -> Decimal("12")

Step 2 — Parse each provider’s instance-family grammar

Each parser is pure and returns either a (vCPU, memory_GB) tuple or None, so an unresolvable string is a value the caller routes, not an exception that unwinds a batch. The three grammars are genuinely different: AWS is db.<family><gen>.<size>, Azure is <tier>_<hardware>_<vcores>, and GCP db-custom-<vcpu>-<mem_mib> states both numbers outright.

import re
from typing import Optional

AWS_RE = re.compile(r"^db\.(?P<family>[a-z0-9]+?)\.(?P<size>\d*x?large|medium)$")
AZURE_RE = re.compile(r"^(?P<tier>[A-Z]{2}_Gen\d)_(?P<vcores>\d+)$")
GCP_CUSTOM_RE = re.compile(r"^db-custom-(?P<vcpu>\d+)-(?P<mem_mib>\d+)$")

Capacity = tuple[Decimal, Decimal]  # (vCPU, memory_GB)


def _aws_size_to_vcpu(size: str) -> Optional[Decimal]:
    if size in AWS_BASE_SIZE_VCPU:
        return AWS_BASE_SIZE_VCPU[size]
    m = re.fullmatch(r"(\d+)xlarge", size)          # 2xlarge, 4xlarge, 24xlarge...
    return Decimal(m.group(1)) * Decimal("4") if m else None


def parse_aws(sku: str) -> Optional[Capacity]:
    """db.r6g.xlarge -> (4 vCPU, 32 GB)."""
    m = AWS_RE.match(sku)
    if not m:
        return None
    density = AWS_FAMILY_DENSITY.get(m.group("family"))
    vcpu = _aws_size_to_vcpu(m.group("size"))
    if density is None or vcpu is None:
        return None                                  # known grammar, unknown family/size
    return vcpu, vcpu * density


def parse_azure(sku: str) -> Optional[Capacity]:
    """GP_Gen5_4 -> (4 vCore, 20.4 GB)."""
    m = AZURE_RE.match(sku)
    if not m:
        return None
    density = AZURE_TIER_DENSITY.get(m.group("tier"))
    if density is None:
        return None
    vcpu = Decimal(m.group("vcores"))
    return vcpu, vcpu * density


def parse_gcp(sku: str) -> Optional[Capacity]:
    """db-custom-4-16384 -> (4 vCPU, 16 GB). Memory is stated in MiB."""
    m = GCP_CUSTOM_RE.match(sku)
    if not m:
        return None
    vcpu = Decimal(m.group("vcpu"))
    mem_gb = Decimal(m.group("mem_mib")) / Decimal("1024")
    return vcpu, mem_gb


# Expected:
#   parse_aws("db.r6g.xlarge")     -> (Decimal('4'), Decimal('32'))
#   parse_azure("GP_Gen5_4")       -> (Decimal('4'), Decimal('20.4'))
#   parse_gcp("db-custom-4-16384") -> (Decimal('4'), Decimal('16'))
#   parse_aws("db.z9z.nano")       -> None   (unknown family)

Step 3 — Dispatch, classify, and quarantine unknowns

The dispatcher is the single entry point. It selects a parser by provider, and on a None result it does not guess: the record is stamped with a quarantine reason and diverted, exactly the dead-letter discipline used across the normalization pipeline. A silently mis-mapped SKU corrupts every downstream comparison, so an unresolved string must fail loudly rather than resolve to zero.

The flow below shows how one raw SKU string is routed to a provider parser, resolved against the lookup table, and either scored to a CCU or quarantined.

SKU-to-CCU mapping flow: provider dispatch to family parsers, capacity lookup, resolution gate, and CCU emit or quarantineA raw SKU record enters a provider dispatcher that routes to the AWS, Azure, or GCP parser. Each parser resolves against the capacity lookup table and reaches a "Resolved?" gate. Resolved records are folded into a canonical compute unit and emitted to the normalized cost join; unresolved records are diverted to the unknown-SKU quarantine, which feeds a lookup-table patch and replay.resolvedunknownRaw SKUrecordProviderdispatchAWS parserAzure parserGCP parserCapacitylookupResolved?Canonical unitto cost joinUnknown-SKUquarantineTable patch& replay
from dataclasses import dataclass

PARSERS = {"aws": parse_aws, "azure": parse_azure, "gcp": parse_gcp}


@dataclass(frozen=True)
class Resolved:
    provider: str
    sku: str
    vcpu: Decimal
    memory_gb: Decimal
    ccu: Decimal


def resolve_sku(provider: str, sku: str) -> tuple[Optional[Resolved], Optional[dict]]:
    """Return (Resolved, None) on success or (None, quarantine_record) on failure."""
    parser = PARSERS.get(provider)
    if parser is None:
        return None, {"provider": provider, "sku": sku, "reason": "unknown_provider"}
    cap = parser(sku)
    if cap is None:
        return None, {"provider": provider, "sku": sku, "reason": "unparsed_sku"}
    vcpu, mem_gb = cap
    ccu = canonical_compute_unit(vcpu, mem_gb)
    return Resolved(provider, sku, vcpu, mem_gb, ccu), None

Step 4 — Apply the mapping across a billing frame with pandas

In production the SKUs arrive as a column in a billing export. The classifier joins onto that frame row by row, splitting resolved rows from quarantined ones so a single unmappable SKU never poisons the aggregate. The resolved frame carries ccu, which lets you compute a directly comparable cost-per-CCU across providers.

import pandas as pd

def map_frame(df: pd.DataFrame) -> tuple[pd.DataFrame, pd.DataFrame]:
    """Split a billing frame into (resolved rows with CCU, quarantined rows)."""
    resolved_rows, dead_rows = [], []
    for row in df.itertuples(index=False):
        rec, dead = resolve_sku(row.provider, row.sku)
        if rec is not None:
            resolved_rows.append({
                "provider": rec.provider, "sku": rec.sku,
                "vcpu": rec.vcpu, "memory_gb": rec.memory_gb, "ccu": rec.ccu,
                "cost": Decimal(str(row.cost)),               # Decimal from the wire
            })
        else:
            dead_rows.append(dead)
    resolved = pd.DataFrame(resolved_rows)
    if not resolved.empty:
        # cost-per-CCU is the cross-provider comparable unit price
        resolved["cost_per_ccu"] = resolved.apply(
            lambda r: r["cost"] / r["ccu"], axis=1
        )
    return resolved, pd.DataFrame(dead_rows)


if __name__ == "__main__":
    frame = pd.DataFrame([
        {"provider": "aws",   "sku": "db.r6g.xlarge",     "cost": "182.50"},
        {"provider": "azure", "sku": "GP_Gen5_4",         "cost": "210.24"},
        {"provider": "gcp",   "sku": "db-custom-4-16384", "cost": "168.00"},
        {"provider": "aws",   "sku": "db.z9z.nano",       "cost": "9.99"},  # unknown
    ])
    resolved, quarantined = map_frame(frame)
    print(resolved[["provider", "sku", "vcpu", "memory_gb", "ccu", "cost_per_ccu"]])
    print("quarantined:", quarantined.to_dict("records"))

Expected output:

  provider              sku vcpu memory_gb   ccu       cost_per_ccu
0      aws    db.r6g.xlarge    4        32    12  15.20833333333...
1    azure        GP_Gen5_4    4      20.4  9.10  23.10329670329...
2      gcp  db-custom-4-16384    4        16     8              21.00
quarantined: [{'provider': 'aws', 'sku': 'db.z9z.nano', 'reason': 'unparsed_sku'}]

Verification

Confirm the mapping is total and lossless before any cross-provider report consumes it.

  1. Assert the quarantine is empty for a known estate. Every SKU your accounts actually run must resolve; a non-empty quarantine is a coverage gap, not a warning to ignore.

    resolved, quarantined = map_frame(frame)
assert quarantined.empty, f"unmapped SKUs: {quarantined['sku'].tolist()}"

  2. Round-trip a spot-check against provider docs. Pick one SKU per provider and confirm the resolved (vcpu, memory_gb) matches the published instance spec. db.r6g.xlarge is 4 vCPU / 32 GiB; db-custom-4-16384 is 4 vCPU / 16 GiB by construction.

  3. Check CCU monotonicity. Sort the resolved frame by ccu and confirm larger sizes score higher — a size-parsing bug usually shows up as a 2xlarge scoring below an xlarge.

Gotchas & Edge Cases

  • AWS size grammar has irregular members. db.t3.medium is 1 vCPU and db.r6g.16xlarge is 64 vCPU, but metal and some .large burstable SKUs break the doubling rule. Keep the base-size table authoritative and let only the Nxlarge pattern compute — never extrapolate a size you have not verified.
  • Azure serverless has no fixed vCore count. GP_S_Gen5_2 is a serverless tier that autoscales between a min and max vCore; a single SKU string cannot state the billed capacity, so resolve serverless against actual vCore-seconds from the meter rather than the SKU, the same discipline used when separating compute and storage costs in Azure SQL.
  • GCP shared-core tiers are not db-custom. db-f1-micro and db-g1-small predate the custom grammar and carry fractional vCPU; add them to an explicit lookup entry rather than forcing them through the db-custom-* regex, which will simply return None and quarantine them.
  • GiB versus GB. GCP states memory in MiB, and RDS specs are in GiB, while pricing pages sometimes quote GB. Pick one — this pipeline treats all memory as binary GiB after the /1024 conversion — and document it, because a 7% base-2/base-10 error compounds through every CCU.
  • Never coerce cost through float. Read the cost column as a string and wrap it in Decimal at the boundary; a float cost divided by a Decimal CCU raises TypeError, and worse, a float-sourced Decimal carries binary rounding noise. This is the same money-typing rule enforced in schema validation for billing data.
  • A growing quarantine signals SKU drift. Providers ship new families (Graviton4, Gen6 hardware) continuously. Treat the quarantine count as a monitored metric; a step change means a new family launched and the lookup table needs a patch-and-replay, not a silent drop.

Frequently Asked Questions

Why normalize to a synthetic CCU instead of just comparing vCPU counts?

Because vCPU alone hides the memory a SKU bills for. A 4-vCPU memory-optimized instance and a 4-vCPU general-purpose instance have identical vCPU counts but very different capacity and price; comparing on vCPU makes the memory-optimized SKU look artificially expensive per core. Folding memory into the unit with a reference density makes cost-per-CCU an apples-to-apples figure across families and providers.

How do I choose the reference density rho_ref?

Set it to the memory-per-vCPU of your most common general-purpose family — 4 GB/vCPU is typical for m-class AWS and Azure General Purpose Gen5. The absolute value does not matter for ranking as long as it is constant across every SKU; it only rescales the unit. Keep it in one named constant so a change re-scores the whole estate consistently.

What should happen to a SKU the parser cannot resolve?

It goes to quarantine with a reason code, never to a default or zero. A mis-mapped SKU silently corrupts every cross-provider comparison downstream, whereas a quarantined one is a visible, countable gap you patch in the lookup table and replay. Defaulting an unknown SKU to zero capacity would make its cost-per-CCU infinite and poison any ranking.

Does this handle serverless and autoscaling tiers?

Not from the SKU string alone — serverless capacity is not fixed at provision time. For Aurora Serverless v2 (ACUs) and Azure serverless (min/max vCores), resolve capacity from the billed vCore-seconds or ACU-hours in the usage record instead of the SKU, then feed that measured capacity into the same canonical_compute_unit function.

Why Decimal rather than float for the capacity math?

The CCU is a divisor for cost-per-unit, and that quotient flows into chargeback ledgers where sub-cent drift accumulates across millions of rows. Decimal with a set precision gives exact, reproducible arithmetic; float introduces binary rounding that makes two runs of the same data disagree in the last cents. The typing discipline is identical to the one at the validation boundary.

Back to: Multi-Cloud Cost Normalization