There are sizable Mixture of Experts models can run comfortably on 16GB with the right settings and 4-bit. I'm doing so right now on a miniPC.
16GB RAM? Going to be running 2-bit quantized teraslop Qwen hallucinating 24/7.- There are sizable Mixture of Experts models can run comfortably on 16GB with the right settings and 4-bit. I'm doing so right now on a miniPC.
Not really, good models start at 18gb
Setup?
which model are u using