non-linguistic prior

Teaching AI to Learn Like Humans Through Synthetic Language Data

Most machine learning models can't learn from new data they haven't seen during training. Prior-fitted networks changed that for tables, and now a 300M-parameter transformer extends the idea to language itself. Trained only on synthetic sequences, it learns real languages in context, improving its predictions the more it reads. It even counts and predicts primes. It won't rival trillion-token models, but a synthetic non-linguistic prior producing genuine in-context learning is worth exploring.