f10_slp1_uni8k_d700m_plus_all_v2_3x
d700m (704.1M parameters) on plus_all v2 slice (489.9M words, 3 passes), trained on A100 40GB (Google Cloud, spot). Started 5 Oct 2026, ended 7 Oct 2026.
Hypothesis
A 700M model (twice F9's size) on the filtered plus_all v2 slice, with the weight decay found on 125M proxies, beats F9.
What we learned
Winner: ex-Gita 0.5307 (4.3% better than F9), clean_v1 0.5541 (4.0% better), every held-out set better. Three passes (96,822 steps) on a spot A100, no preemptions. Size, slice filter and weight decay changed together, so it does not say which helped most.
Scores
Bits per byte, lower is better; change against the parent run, F9.
| Test set | F10 | F9 (parent) | Change |
|---|---|---|---|
| DCS gold (classical) | 0.5513 | 0.5687 | −3.1% |
| Bhagavad-gītā (memorisation) | 0.0758 | 0.1367 | −44.6% |
| Out of domain | 0.5286 | 0.5542 | −4.6% |
| Prose | 0.5186 | 0.5406 | −4.1% |
| Vedic (Ṛgveda) | 0.6671 | 0.6892 | −3.2% |
Curves
Training loss
Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.
Held-out bits per byte
The validation split, evaluated during training, by step. Lower is better.
Throughput
Tokens per second, by step.
Model
- Preset
- d700m
- Parameters
- 704,128,512
- Outside embeddings
- 679,552,512
- Layers · heads · width
- 24 · 12 · 1,536
- Tokenizer
- SLP1 unigram 8k
Data
- Slice
- plus_all v2 slice
- Words
- 489,850,000
- Training tokens
- 1,586,326,034
- Passes
- 3 passes
- Tokens seen
- 4,758,994,944
Compute
- GPU
- A100 40GB
- Where
- Google Cloud (spot)
- Steps
- 96,822 / 96,822
- GPU hours
- 40.45
- Cost
- $52.59
- Spot restarts
- 0
Lineage
gcpa100spotd700m