GPU capacity watch
When are cloud GPUs cheap and actually obtainable as spot, and when is on-demand the better buy? Every 30 minutes we record spot and on-demand prices on AWS and Google Cloud, how likely a spot request is to be filled, and what happened when we really tried to launch.
Right now
AWS: spot is discounted (up to 76%, 4× L4 24GB), but the placement score is 3 or lower in every zone, so requests are unlikely to be filled; our rule buys on-demand everywhere. Google Cloud: spot is the better buy for A100 40GB, A100 80GB, H100 80GB and L4 24GB.
| Machine | Spot | On-demand | Discount | Spot availability | Our rule says | Discount and availability, 7 days |
|---|---|---|---|---|---|---|
| AWS · us-east-1 (N. Virginia) · availability is the spot placement score, 1 to 10 | ||||||
| A10G 24GB g5.2xlarge |
$0.689 us-east-1c |
$1.212 | 43% | 1 / 10 | On-demand placement score 3 or less in every zone |
discountscore |
| L4 24GB g6.2xlarge |
$0.863 us-east-1d |
$0.978 | 12% | 1 / 10 | On-demand placement score 3 or less in every zone |
discountscore |
| L40S 48GB g6e.xlarge |
$1.833 us-east-1d |
$1.861 | 1% | 1 / 10 | On-demand placement score 3 or less in every zone |
discountscore |
| L40S 48GB g6e.2xlarge |
$2.210 us-east-1c |
$2.242 | 1% | 1 / 10 | On-demand placement score 3 or less in every zone |
discountscore |
| 4× A10G 24GB g5.12xlarge |
$2.878 us-east-1c |
$5.672 | 49% | 1 / 10 | On-demand no quota placement score 3 or less in every zone |
discountscore |
| 4× L4 24GB g6.12xlarge |
$1.105 us-east-1f |
$4.602 | 76% | 1 / 10 | On-demand no quota placement score 3 or less in every zone |
discountscore |
| 4× L40S 48GB g6e.12xlarge |
$5.961 us-east-1b |
$10.49 | 43% | 1 / 10 | On-demand no quota placement score 3 or less in every zone |
discountscore |
| 8× A100 40GB p4d.24xlarge |
$16.81 us-east-1b |
$21.96 | 23% | 1 / 10 | On-demand no quota placement score 3 or less in every zone |
discountscore |
| 8× H100 80GB p5.48xlarge |
$28.42 us-east-1f |
$55.04 | 48% | 1 / 10 | On-demand no quota placement score 3 or less in every zone |
discountscore |
| Google Cloud · us-central1 (Iowa), europe-west4 (Netherlands) · availability is spot obtainability, 0 to 1 | ||||||
| L4 24GB g2-standard-8 · us-central1 |
$0.486 us-central1-c |
$0.854 | 43% | 0.90 ~1 min before preemption |
Spot price under 85% of on-demand (GCP: price rule only) |
discountobtain. |
| A100 40GB a2-highgpu-1g · us-central1 |
$2.204 us-central1-c |
$3.673 | 40% | 0.90 ~10 min before preemption |
Spot price under 85% of on-demand (GCP: price rule only) |
discountobtain. |
| A100 80GB a2-ultragpu-1g · us-central1 |
$3.017 us-central1-a |
$5.028 | 40% | 0.90 ~1 min before preemption |
Spot price under 85% of on-demand (GCP: price rule only) |
discountobtain. |
| H100 80GB a3-highgpu-1g · us-central1 |
$6.243 us-central1-a |
$10.98 | 43% | 0.90 ~10 min before preemption |
Spot price under 85% of on-demand (GCP: price rule only) |
discountobtain. |
| L4 24GB g2-standard-8 · europe-west4 |
$0.538 | $0.897 | 40% | not offered | Spot price under 85% of on-demand (GCP: price rule only) |
discountobtain. |
| A100 40GB a2-highgpu-1g · europe-west4 |
$2.136 europe-west4-b |
$3.748 | 43% | 0.90 ~1 h before preemption |
Spot price under 85% of on-demand (GCP: price rule only) |
discountobtain. |
| A100 80GB a2-ultragpu-1g · europe-west4 |
$2.912 europe-west4-a |
$5.536 | 47% | 0.50 ~1 min before preemption |
Spot price under 85% of on-demand (GCP: price rule only) |
discountobtain. |
| H100 80GB a3-highgpu-1g · europe-west4 |
$8.383 europe-west4-c |
$13.98 | 40% | 0.90 ~10 min before preemption |
Spot price under 85% of on-demand (GCP: price rule only) |
discountobtain. |
Spot is the cheapest zone's price; the zone under it is where our rule would try first (AWS zone letters are our account's names). Discount = 1 − spot ÷ on-demand. “No quota”: our account cannot launch that machine in that market today. Google Cloud's capacity advice does not cover every machine yet (“not offered”).
Over time
Spot price against on-demand
US$ per hour. The solid line is the cheapest zone's spot price; the dashed line is the on-demand list price; the dotted line is 85% of on-demand, where our rule stops buying spot. Spot climbing towards on-demand means spare capacity is scarce.
Spot availability
AWS spot placement score, 1 to 10, best zone, for one machine. Our rule wants more than 3.
Day by day
AWS spot placement score, daily mean of the best zone
One row per machine, one column per day (UTC). Darker is easier to get.
How to read this
- Spot placement score (AWS)
- AWS's own estimate, from 1 to 10, of how likely a spot request of a given size is to succeed right now. We ask for one machine in a single zone, so the score is per zone; we show the best. 10 means very likely; 1 means you will probably be told there is no capacity. It is a hint, not a promise, and it moves within the hour.
- Spot obtainability (Google Cloud)
- Google's equivalent, from 0 to 1, from its capacity advice (in preview), with an estimated uptime: roughly how long most spot machines of that type run before Google takes them back (1, 10 or 60 minutes or more). It is on a different scale from the AWS score, so we never put the two on one axis.
- Why spot near on-demand means scarcity
- Spot prices are what spare capacity costs. When a type is plentiful its spot price sits far below on-demand; when the spare pool runs dry the price drifts up towards the on-demand price, and requests start being refused. A price at 98% of on-demand is the market saying “there is almost none left”.
- Our rule
- For a one-GPU training job we buy spot first, unless (a) spot costs 85% of on-demand or more, or (b) the placement score is 3 or lower in every zone, or (c) a faster machine on-demand is cheaper per unit of work than the best spot offer. Then we buy on-demand. Spot can be taken back at any time, so a small discount is not worth the restarts; our training jobs checkpoint and resume, which is what makes spot usable at all. On Google Cloud only the price part applies for now.
- Our launch refusals
- Every time our launcher asks a zone for a machine and is told there is no capacity, that is a refusal; asking the next zone is a new attempt. These are real requests from our own training and OCR jobs, so a day with no jobs has no data. Interruptions are spot machines taken back while running.
On-demand prices are list prices (Linux, no reservations, no discounts). Google Cloud spot prices change at most about once a month. AWS spot prices before our first sample come from AWS's spot price history; scores start with our first sample.