Skip to content

fix: padding token not recognized, update transformers #265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
stephantul merged 1 commit into from
Jul 14, 2025
Merged

fix: padding token not recognized, update transformers #265

stephantul merged 1 commit into from
Jul 14, 2025

Conversation

@stephantul
Copy link
Contributor

@stephantul stephantul commented Jul 13, 2025

This PR:

  • Updates transformers
  • This moved an import, so fixed that
  • Fixes a bug with padding token not being registered
@stephantul stephantul changed the title fix: padding token not recognized, update transfrormers Jul 13, 2025
@stephantul stephantul requested a review from Pringled July 13, 2025 19:52
@codecov
Copy link

codecov bot commented Jul 13, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines Coverage Δ
model2vec/tokenizer/tokenizer.py 90.53% <100.00%> (+0.11%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
@stephantul stephantul merged commit e0fe9d1 into main Jul 14, 2025
6 checks passed
@stephantul stephantul deleted the fix-token-registration branch July 14, 2025 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants

@stephantul @Pringled