Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

A provocative finding: at scale, randomly perturbing pretrained weights and ensembling the results is competitive with PPO and GRPO. This method achieves 95% of PPO's performance while requiring 80% less compute. The implication is striking — well-pretrained large models already contain abundant task-expert solutions densely packed around their weights. Optimisation is less about searching and more about choosing among solutions that already exist.