Compare GPUs for AI, side by side
VRAM, tensor throughput, memory bandwidth, power and price for the GPUs people actually train and serve models on. Pick up to four, or start from a preset.
Filters
Choose up to 4 GPUs
Side by side
At a glance
Our read
Where to rent them
Live per-GPU prices from the providers we track. Spot instances cost less but can be interrupted at any time.
Rent it instead of buying it
TokensAI tracks hourly prices for these GPUs across RunPod, Vast.ai, Lambda Labs, CoreWeave and more, and one credit wallet pays for all of them.
Every GPU in the comparator
Uses the filters above. Click a row to add it to the comparison.
| GPU | VRAM | FP16 tensor | Memory bandwidth | Power (TDP) | Price (indicative) | Cloud rental, from | + |
|---|---|---|---|---|---|---|---|
| RTX 3060 12GBNVIDIA · Ampere | 12 GB | 25.4 TFLOPS | 360 GB/s | 170 W | $329 | — | |
| RTX 3090 24GBNVIDIA · Ampere | 24 GB | 71 TFLOPS | 936 GB/s | 350 W | $1,499 | $0.20/hr | |
| RTX 4070 Ti 12GBNVIDIA · Ada Lovelace | 12 GB | 80.2 TFLOPS | 504 GB/s | 285 W | $799 | — | |
| RTX 4080 16GBNVIDIA · Ada Lovelace | 16 GB | 97.5 TFLOPS | 717 GB/s | 320 W | $1,199 | — | |
| RTX 4090 24GBNVIDIA · Ada Lovelace | 24 GB | 165.2 TFLOPS | 1,008 GB/s | 450 W | $1,599 | $0.44/hr | |
| RTX 5090 32GBNVIDIA · Blackwell | 32 GB | 209.5 TFLOPS | 1,792 GB/s | 575 W | $1,999 | — | |
| RTX 6000 Ada 48GBNVIDIA · Ada Lovelace | 48 GB | 364.2 TFLOPS | 960 GB/s | 300 W | $6,800 | $1.36/hr | |
| L4 24GBNVIDIA · Ada Lovelace | 24 GB | 121 TFLOPS | 300 GB/s | 72 W | $3,000 | — | |
| L40S 48GBNVIDIA · Ada Lovelace | 48 GB | 362 TFLOPS | 864 GB/s | 350 W | $8,000 | $1.77/hr | |
| A100 40GBNVIDIA · Ampere | 40 GB | 312 TFLOPS | 1,555 GB/s | 400 W | $10,000 | $1.52/hr | |
| A100 80GBNVIDIA · Ampere | 80 GB | 312 TFLOPS | 2,039 GB/s | 400 W | $15,000 | $1.10/hr | |
| H100 SXM 80GBNVIDIA · Hopper | 80 GB | 989 TFLOPS | 3,350 GB/s | 700 W | $30,000 | $2.06/hr | |
| H200 141GBNVIDIA · Hopper | 141 GB | 989 TFLOPS | 4,800 GB/s | 700 W | $35,000 | $5.66/hr | |
| B200 180GBNVIDIA · Blackwell | 180 GB | 2250 TFLOPS | 7,700 GB/s | 1000 W | $40,000 | $8.07/hr | |
| MI300X 192GBAMD · CDNA 3 | 192 GB | 1307 TFLOPS | 5,300 GB/s | 750 W | $15,000 | — |
How to read the numbers
FP16 tensor TFLOPS
The headline number for modern training and inference. We quote dense tensor-core throughput without sparsity, and use the FP32-accumulate rate on consumer cards so they compare fairly with data-center parts.
INT8 TOPS
What quantized inference runs on. On most recent GPUs it is roughly double the FP16 figure.
VRAM
Decides what fits. A model needs about 2 bytes per parameter in FP16 and half a byte in 4-bit, plus headroom for activations and the KV cache. The "largest model" rows use that rule of thumb with about 20% overhead.
Memory bandwidth
Often the real bottleneck for LLM inference, since every generated token streams the weights through memory. HBM data-center cards win big here.
Efficiency
FP16 TFLOPS per watt of TDP. It drives the electricity bill and how many cards a rack can power.
Value
FP16 TFLOPS per $1,000 of indicative price. Consumer cards usually lead; data-center cards pay you back in memory, interconnect and reliability instead.
Which GPU for which job
Development & prototyping
12–24 GB consumer cards (RTX 3060 to 4090) cover prototyping, LoRA fine-tuning of 7–8B models and running quantized models locally.
Production inference
For serving, look at bandwidth and efficiency first: L4 for small models at low power, L40S or A100 for 30–70B, H100, H200 or MI300X for the largest open models.
Training & fine-tuning
Full fine-tuning and pre-training want HBM and fast interconnect: A100 80GB as the workhorse, H100 and H200 for speed, B200 at the frontier.
Research
Labs usually pair a few large-memory data-center GPUs for big runs with cheaper consumer cards for day-to-day experiments.
FAQ
Where do the numbers come from?
Specs come from vendor datasheets. Purchase prices are indicative: launch MSRP for consumer cards, typical market price for data-center parts. Cloud prices come live from providers' public APIs where we track them (marked "live") and are indicative otherwise.
Why is the RTX 4090's FP16 lower than figures I've seen elsewhere?
NVIDIA quotes GeForce cards with FP16 accumulate, which doubles the number. We use FP32 accumulate, which is what mixed-precision training actually runs at, so a 4090 and an A100 are measured the same way.
Should I buy or rent?
Rent while you are experimenting or when usage is bursty; buy when a card will stay busy most of the day for a year or more. The break-even row divides the purchase price by the hourly rate.
Can I compare more than four?
The side-by-side view stops at four so it stays readable. The full table lists every GPU and follows the same filters and sort.
Specs from vendor datasheets. Prices are indicative and change often.