16GB RAM? Going to be running 2-bit quantized teraslop Qwen hallucinating 24/7.
- 16GB RAM? Going to be running 2-bit quantized teraslop Qwen hallucinating 24/7.
Qwen 9B quantized for serialized tasks perfectly capable
There are sizable Mixture of Experts models can run comfortably on 16GB with the right settings and 4-bit. I'm doing so right now on a miniPC.
Give it time, but yes. It's too soon for this one.
