feat: add vocabulary quantization

[stephantul]

This PR adds vocabulary quantization.

The quantization itself is handled by scikit-learn ,so that has been added as a dependency to quantization. There's a helper function to quantize already existing models, which can be imported directly:

from model2vec import quantize_model

model = quantize_model(model, vocabulary_quantization=1024)

The quantization itself necessitated a lot of internal changes. Most prominently, the embeddings and token weights are now decoupled. Every token still has a unique weight, but shares embeddings with tokens in the same cluster.
Old models can still be created and loaded, so there are no breaking changes. Quantized models can't be loaded using the old model2vec, however.

[codecov]

Codecov Report

❌ Patch coverage is 92.00000% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
model2vec/distill/distillation.py	72.72%	3 Missing ⚠️
model2vec/model.py	92.30%	3 Missing ⚠️
model2vec/train/base.py	85.71%	2 Missing ⚠️
model2vec/vocabulary_quantization.py	90.47%	2 Missing ⚠️

Files with missing lines	Coverage Δ
model2vec/__init__.py	100.00% <100.00%> (ø)
model2vec/distill/inference.py	97.50% <100.00%> (+0.03%)	⬆️
model2vec/hf_utils.py	79.46% <100.00%> (+3.22%)	⬆️
model2vec/inference/model.py	92.66% <100.00%> (+0.27%)	⬆️
model2vec/quantization.py	96.77% <100.00%> (+0.22%)	⬆️
model2vec/tokenizer/normalizer.py	95.23% <100.00%> (ø)
model2vec/train/classifier.py	97.56% <100.00%> (ø)
model2vec/train/base.py	97.89% <85.71%> (-2.11%)	⬇️
model2vec/vocabulary_quantization.py	90.47% <90.47%> (ø)
model2vec/distill/distillation.py	86.20% <72.72%> (-2.41%)	⬇️
... and 1 more

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[stephantul]

This new version stores all the relevant information (i.e., mappings, weights etc.) in the safetensors file.

[Pringled]

LGTM, tested it on a few models and it works well I think. Only thing missing is some documentation; can probably add a small snippet in https://github.com/MinishLab/model2vec/blob/main/docs/usage.md?

[Pringled]

I think this is quite slow if the vocab is very large, is MiniBatchKmeans an option (or an optinal arg) that could help?

[stephantul]

Maybe, we could consider that for a follow up if you think that's interesting