3-bit Quantization

Finding the sweet spot between model scale and quantization precision

The sweet spot for quantizing LLMs keeps shifting, and the old 4-bit consensus is fading fast. New research points to a more aggressive target, where a 2-bit 70B model can genuinely outpace a 4-bit 35B one, given a fixed memory budget. This is about maximizing capability, not preserving a single model's fidelity. The degradation from quantization is real, but it's often outweighed by the sheer benefit of more parameters. It's a trade-off worth exploring, especially for those running models locally.