passes_p3_dropout_d60m
d60m (68.9M parameters) on plus_all v2 subset (155M tokens), trained on RTX 3060 12GB (Home GPU). Started 11 Oct 2026, ended 11 Oct 2026.
Hypothesis
Dropout 0.1 curbs memorisation of text seen four times and gives a better four-pass model.
What we learned
Lose: ex-Gita 0.6661, 0.74% worse than Passes 4 (clean_v1 +0.72%), past the noise bar. It memorises less (the Gita score is 6% higher), but held-out text gets worse at four passes. Not adopted.
Scores
Bits per byte, lower is better; change against the parent run, Passes 4.
| Test set | Passes dropout | Passes 4 (parent) | Change |
|---|---|---|---|
| DCS gold (classical) | 0.6786 | 0.6698 | +1.3% |
| Bhagavad-gītā (memorisation) | 0.4716 | 0.4451 | +6.0% |
| Out of domain | 0.6691 | 0.6655 | +0.5% |
| Prose | 0.6353 | 0.628 | +1.2% |
| Vedic (Ṛgveda) | 0.8993 | 0.8921 | +0.8% |
Curves
Training loss
Cross-entropy per token, by step. The first few percent of the run, far higher, run off the top; hover or the table has every value.
Held-out bits per byte
The validation split, evaluated during training, by step. Lower is better.
Throughput
Tokens per second, by step.
Model
- Preset
- d60m
- Parameters
- 68,924,160
- Outside embeddings
- 56,636,160
- Layers · heads · width
- 8 · 12 · 768
- Tokenizer
- SLP1 unigram 8k
Data
- Slice
- plus_all v2 subset (155M tokens)
- Words
- —
- Training tokens
- 155,261,295
- Passes
- 4 passes
- Tokens seen
- 621,084,672
Compute
- GPU
- RTX 3060 12GB
- Where
- Home GPU
- Steps
- 12,636 / 12,636
- GPU hours
- 3.85
- Cost
- —
- Spot restarts
- —
Lineage
homed60m