Supporting quantized (GGUF) models for resource constraint environments

[bhupesh-sf]

Hey,

Currently the repo uses only safetensor models and I was wondering if we can support quantized (GGUF) models for resource constraint environments. I had to deploy use it in a an CPU only env with low resources and it takes lots memory for such environment.