memorization

When a Tiny GPT Knows the Facts but Misses the Connection

A tiny GPT can memorize every fact you hand it and still fail when the answer requires connecting two of them. That gap is fascinating, and frustrating. I tested the one-line fix from a recent paper, and it only half worked. The problem isn't recall; it's reasoning across what's already known. If you're exploring how small models actually think, this is a worthwhile stop. For a deeper look at structured decision-making, our piece on decision models pairs nicely with this one.