CPU, GPU, or TPU: Choosing Compute by Workload, Not by Benchmark

What this is really about

The CPU-versus-GPU-versus-TPU question is usually framed as a ranking, with the TPU as the exotic, most-powerful option at the top. That framing is wrong, and it leads to expensive mistakes. These three are not rungs on a ladder; they are points on a single spectrum that trades flexibility for specialization. A CPU is fully general. A GPU is a general-purpose parallel processor. A TPU is a fixed-function matrix engine that does one kind of math extremely well and little else.

So the real question is not "which is fastest." It is "what shape is my workload, which ecosystem am I willing to commit to, and where am I willing to be locked in." Peak throughput on a spec sheet is the least useful number in the decision.

Why it matters

For anything AI-adjacent, compute is often the single largest infrastructure line item, and the most wasted. The failure modes are symmetrical: run the wrong workload on a GPU and you pay premium prices for silicon that sits idle; reach for a TPU because it benchmarks well and you can inherit a portability problem that outlives the performance win.

Choosing compute is a multi-year commitment to a programming model, a supply chain, and a pricing relationship, not just a chip. That is why the decision deserves a framework rather than a leaderboard.

What each one actually is

Strip away the marketing and the three are architecturally distinct answers to "how much do we specialize."

• CPU: a handful of large, powerful cores optimized for latency and control flow. Deep caches, branch prediction, and out-of-order execution make it the best processor for serial, branchy, unpredictable work: orchestration, data preparation, business logic, and classical machine learning. It does math on a few things at a time, very quickly, and handles "if this then that" better than anything else. Modern server CPUs also add matrix extensions (Intel AMX, Arm SME) that narrow the gap for light inference.
• GPU: thousands of smaller cores running the same instruction across many data elements at once (SIMT). That design is built for throughput, not latency: it applies one operation to enormous batches of numbers, which is exactly the shape of the dense linear algebra behind neural networks, simulation, and graphics. Modern GPUs also embed dedicated matrix units (NVIDIA Tensor Cores, AMD Matrix Cores), so they are no longer purely general parallel processors; they are general parallel processors with a matmul accelerator built in. Blackwell (B200 and B300) is today's installed base and AMD's Instinct MI355X its main rival; NVIDIA's Vera Rubin begins production shipments this fall and AMD's MI450-series Helios racks ramp from the second half of 2026. Each generation pushes matrix throughput and on-package memory further, but the defining trait holds: they stay programmable across training, inference, HPC, and rendering.
• TPU: an ASIC, an application-specific integrated circuit that Google designed to do one thing, large tensor (matrix) operations. The current generation is Ironwood (TPU v7), generally available since April 2026 and the first TPU Google describes as built primarily for inference, with 192 GB of HBM per chip. Its core is a systolic array, a grid of multiply-accumulate units that streams data through in lockstep, achieving very high matmul throughput per watt with little control overhead. That efficiency is the reward for giving up generality: a TPU is programmed through the XLA compiler and is happiest with JAX and TensorFlow. PyTorch historically ran through a compatibility layer (PyTorch/XLA); Google's TorchTPU, confirmed in April 2026 and moving to an open-source release, gives PyTorch a native TPU backend (still compiled by XLA underneath), and serving engines such as vLLM and SGLang are gaining TPU backends. That meaningfully lowers the porting friction. What it does not change is where the silicon lives: overwhelmingly, in Google Cloud.

The through-line: as you move CPU to GPU to TPU, you gain throughput on dense matrix math and give up generality. Everything below is about deciding how far along that trade you should go.

1. Workload shape: what math are you actually running?

The first and most important factor is the shape of the work, not the size of the model. Match it honestly:

• Serial, branchy, latency-sensitive logic — request routing, feature engineering, tree-based models, small-batch or low-QPS inference: CPU. This work is dominated by control flow, and control flow is what a CPU is best at.
• Dense, parallel matrix and vector math at high throughput, across a changing mix of models and frameworks — most training and a large share of inference: GPU. The parallelism maps onto the hardware, and the flexibility covers whatever you run next.
• Massive, fixed-shape tensor operations at large batch and scale, especially large neural-network training and high-volume inference that stays matmul-bound: TPU. This is the narrow case the systolic array was built for.

The tell is how much of your runtime is spent in big matrix multiplies versus control flow. The more your work collapses to dense matmul, the further toward specialization you can afford to move. If a meaningful fraction is branching, data wrangling, or glue code, that fraction wants a CPU no matter what the model is.

2. Ecosystem and portability: the real lock-in

The most underweighted factor is software, not silicon. CUDA is the reason the GPU is the default: more than a decade of libraries, frameworks, drivers, and operational tooling target NVIDIA first. That ecosystem, not raw performance, is the moat, and it is why a GPU is rarely the wrong answer even when it isn't the fastest.

TPUs trade that breadth for efficiency. JAX and TensorFlow are first-class; PyTorch is improving fast with TorchTPU but is the newest path, so expect rough edges in custom kernels and less-common operators; and for almost every buyer the hardware lives inside Google Cloud. Framework portability is getting better; platform portability mostly isn't. Code and pipelines tuned for TPU are still not a drop-in move to anywhere else. CPUs are the universal fallback at the other end: every framework runs on them and portability is total, but heavy training performance is not there.

Ask a blunt question before committing: if this vendor doubled the price at renewal, what would it cost me to leave? For CPUs, effectively nothing. For GPUs, a toolchain migration to another accelerator's stack. For TPUs, potentially a re-platforming and a change of cloud.

Exit cost is future leverage. Estimate it before you commit, not after you discover it at renewal.

3. Availability and operations: where can you even get it?

A chip you can't source is not an option, and this is where the three diverge sharply:

• CPU: everywhere — on-prem and every cloud, at every size.
• GPU: available on-prem and from every major cloud, though supply, quota, and price are frequently the real constraint rather than capability.
• TPU: Google Cloud, with one new and narrow exception. In 2026 Google began delivering TPU systems into a small set of customers' own data centers (it has cited AI labs, capital-markets firms, and HPC users), but supply is allocated case by case and Google has said its own frontier work gets first call. It is not a product you can order the way you order a GPU server. (The Edge TPU remains a separate, tiny inference device for embedded use.) For practically everyone, choosing a TPU is still choosing a cloud.

That last point quietly turns a chip decision into a sourcing and even a cloud-strategy decision. A TPU can be the right engine on the merits and still be the wrong choice if it forces a single-cloud commitment you otherwise wouldn't make, or if it can't live next to data that has to stay on-prem. The on-prem door opening is worth watching, but it only matters today if you buy at the scale of an AI lab.

4. Cost and efficiency at scale: perf-per-watt, not peak TOPS

Specialized silicon wins on efficiency for the workload it was built for. For large, matmul-bound training and inference at scale, TPUs can deliver strong performance per watt and per dollar; that is the entire economic argument for an ASIC. GPUs cost more per unit of pure matmul but earn it back in flexibility: one fleet serves training, inference, HPC, and whatever architecture ships next quarter. CPUs are the cheapest and most available, and the wrong tool for heavy training, yet often the most cost-effective for orchestration, data prep, and low-QPS inference, because you are not paying accelerator prices for work that does not need an accelerator.

One caveat sharpens the point: large-language-model inference is usually bound by memory bandwidth, not compute. Serving a big model is largely a race to move its weights out of high-bandwidth memory (HBM) fast enough, so HBM capacity and bandwidth, not peak TOPS, frequently decide real inference throughput. It is another reason the headline compute number is the wrong thing to optimize for, and why two accelerators with similar TOPS can serve the same model at very different speeds. The vendors have noticed: Ironwood's headline feature is its memory, and each new GPU generation leads with HBM capacity and bandwidth as loudly as with FLOPS.

The economics that actually bite are utilization, not sticker price. The most expensive processor is the one you bought and left idle. A cheaper, well-utilized GPU usually beats a faster TPU you can only keep busy part of the time, and a right-sized CPU beats an accelerator you rented for work that never needed one.

Where each one wins

Put the factors together and the mapping is clear. This is a starting point, not a law, but it is the right default:

Notice the pattern: the CPU owns the edges (control-heavy and light work), the GPU owns the broad, changing middle, and the TPU owns a narrow, deep corner — big, stable, matmul-bound work for teams already inside its ecosystem.

The bigger picture: what changed in 2026

CPU, GPU, and TPU are the canonical three, but the TPU is really a stand-in for a whole class of domain-specific AI accelerators. AWS's Trainium3 has been generally available since late 2025, alongside Inferentia for inference; a wave of startups ship inference-focused ASICs and novel dataflow and wafer-scale designs aimed at low-latency LLM serving; AMD (Instinct on ROCm) is landing rack-scale MI450 systems with frontier-lab customers; Intel (Gaudi) pushes alternatives too.

Three shifts this year sharpen the framework rather than overturn it:

• Inference became the center of gravity. Google built Ironwood primarily for inference and has split its eighth-generation TPU into separate training (8t) and inference (8i) chips. When the most specialized vendor decides one chip can't serve both jobs well, that is the flexibility-versus-specialization curve in action.
• The walls around specialized silicon got lower, not removed. TorchTPU brings native PyTorch to TPUs, vLLM and SGLang are gaining TPU backends, and Google has started placing TPU systems in a handful of customer data centers. Porting is cheaper than it was a year ago; changing clouds is not.
• Rivals now share plumbing. AWS's next-generation Trainium4 is slated to use NVIDIA's NVLink Fusion interconnect. The competition is moving from "whose chip" to "whose rack, whose network, and whose software stack."

The categories are also converging from both ends: GPUs added matrix units to behave more like ASICs, and CPUs added matrix extensions to handle light inference. The useful mental model is not three boxes but one axis, general to specialized, with new silicon landing at different points on it. The decision framework doesn't change with the brand: workload shape, ecosystem lock-in, availability, and utilization economics apply to whatever accelerator is being pitched this quarter.

Trade-offs & risks

No option is free; naming the honest cost of each is the whole point.

• CPU buys universality, low cost, and total portability, at the cost of throughput. It is simply the wrong engine for large-scale training.
• GPU buys flexibility and the deepest ecosystem, at the cost of a higher price per unit of raw matmul and, in practice, supply constraints and lock-in to one vendor's software stack.
• TPU buys efficiency at scale, at the cost of portability and (for almost all buyers) single-cloud dependence. Framework freedom is improving with TorchTPU, but it is the newest and least battle-tested path.

The subtle risk sits with the most specialized option. A fixed-function accelerator is efficient only as long as the dominant model architectures keep looking like the ones it was tuned for. Google splitting its next TPU into training and inference variants is a reminder of how quickly "the workload" can fork. A general GPU is a hedge against the field changing under you, and in machine learning it always does. Specialization is leverage when the workload is stable and scale is real; it is a liability when either one moves.

IT Intelligence's take

Decide in this order: workload shape first, ecosystem and portability second, raw performance third. That is the exact inverse of how these decisions are usually made, and the inversion is the point.

For most organizations the honest default is the GPU, not because it tops every benchmark but because it wins the ecosystem and hedges against change: one flexible fleet that runs today's models and next year's. Keep the CPU for what it is genuinely best at — orchestration, data preparation, classical ML, and latency-sensitive or low-volume inference — and refuse to pay accelerator prices for work that doesn't need an accelerator. Reach for a TPU when three things are true at once: the workload is large and firmly matmul-bound, you are already committed to Google Cloud (JAX and TensorFlow remain the smoothest path, with PyTorch via TorchTPU now a credible one to pilot), and the efficiency gain is big enough to justify the portability you give up. If high-volume inference is the target, benchmark on memory-bound serving of your own model, not on vendor training numbers.

The mistake to avoid is treating "more specialized" as "better." In compute, as in the rest of infrastructure, the fastest chip on paper is rarely the one that turns out cheapest in production. That is almost always the one you can keep busy and can afford to leave.

Glossary

Plain-language definitions for the terms above.

• ASIC: application-specific integrated circuit, a chip hard-wired for one job (a TPU is an ASIC for tensor math).
• SIMT: single instruction, multiple threads, the GPU model of running the same operation across many data elements at once.
• Systolic array: the grid of multiply-accumulate units at the heart of a TPU that streams data through in lockstep for high matmul throughput.
• Tensor Core / Matrix Core: dedicated matrix-multiply units inside modern GPUs that accelerate the same math a TPU specializes in.
• Matmul: matrix multiplication, the dominant operation in neural-network training and inference.
• TOPS / FLOPS: tera-operations (or floating-point operations) per second, a peak-throughput spec that rarely predicts real-world performance on its own.
• XLA: the compiler that turns TensorFlow/JAX (and PyTorch/XLA) programs into optimized code for TPUs and other targets.
• TorchTPU: Google's native PyTorch backend for TPUs; it captures PyTorch graphs, lowers them to StableHLO, and hands them to XLA.
• Rack-scale system: an accelerator sold as a whole rack wired into one large domain (NVIDIA's NVL72, AMD's Helios, Google's TPU pods) rather than as individual cards.
• AMX / SME: matrix extensions added to modern CPUs (Intel AMX, Arm SME) to speed up light inference.
• HBM: high-bandwidth memory, the fast on-package memory that feeds accelerators; often the real bottleneck for large models.
• Training vs inference: training builds a model from data (heaviest, most parallel); inference runs the finished model to answer requests (latency- and cost-sensitive).

Reference & sources

This analysis is IT Intelligence's own, synthesized from vendors' official documentation. No vendor text or diagrams are reproduced, and all product names belong to their owners. Product status is current as of October 2026. Architectures, availability, and pricing change; verify current details against the official docs before designing around them.

• NVIDIA: CUDA, Tensor Core, and data-center GPU documentation (docs.nvidia.com); Vera Rubin production announcement (May 2026).
• Google Cloud: Cloud TPU architecture, Ironwood, XLA, and TPU/JAX/TensorFlow documentation (cloud.google.com/tpu); TorchTPU announcement (developers.googleblog.com); Alphabet Q1 and Q2 2026 earnings commentary on TPU system sales (abc.xyz/investor).
• AMD and Intel: ROCm, Instinct, and Helios documentation (rocm.docs.amd.com, amd.com); Intel AMX and Gaudi documentation (intel.com).
• AWS: Trainium and Inferentia documentation (aws.amazon.com/machine-learning/accelerators).