NanoGPT Slowrun: 10x Data Efficiency with Infinite Compute

NanoGPT Slowrun achieves 10x data efficiency, enabling powerful models with significantly less data. By leveraging infinite compute for exhaustive hyperparameter search, it trains a 12-layer transformer to GPT-2 perplexity using only 10% of the original dataset—a 90% reduction.