Ternary LLMs: Breaking the 1.58-bit Barrier
Sources
We're seeing a push toward "ternary" weights—restricting model weights to just three values: -1, 0, and 1. This effectively crushes the VRAM requirement and replaces expensive floating-point multiplications with simple additions and subtractions.
The Architecture Impact
The shift to 1.58-bit (ternary) isn't just about compression; it's about the compute kernel. By eliminating the need for matrix-multiply units (MMUs) in the traditional sense, you can theoretically push throughput to the limits of memory bandwidth. The weights are no longer "stored" so much as they are "indexed."
The "Catch": The Quantization Tax
The trade-off is a significant "intelligence tax" on smaller models. While frontier models can absorb this quantization without losing much perplexity, 7B or 13B models start to lose the "nuance" required for complex reasoning. You get a model that is incredibly fast and light, but prone to "repetitive looping" and loss of coherence in long-context windows.
Who Should Skip
Avoid this for high-precision tasks (medical, legal, code synthesis) where a single bit-flip in logic is a failure. This is for high-throughput, low-latency "edge" applications where "mostly right and instant" beats "perfect and slow."
Verdict
The death of FP16 for edge AI. It's a brutal optimization that works, provided you have enough parameters to mask the precision loss.