Skip to content

Feat add local inference #89

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
codelion merged 13 commits into from
Nov 13, 2024
Merged

Feat add local inference #89

codelion merged 13 commits into from
Nov 13, 2024

Conversation

@codelion
Copy link
Member

@codelion codelion commented Nov 10, 2024 •
edited
Loading

  • Ability to use local built-in inference server
  • Allows logprobs in output responses (this is not supported in ollama)
  • Allows multiple response sampling (this is not supported in ollama)
  • Supports multiple LoRAs (this is not supported in ollama)
  • Supports prompt caching
  • Supports alternative decoding techniques like cot_decoding and entropy_decoding
codelion added 13 commits November 8, 2024 13:47
@codelion
initial implementation
23e48ef 
- allow loading any model from hf and any lora adapter
- support caching
- support batchs
- support optimized attention
- add dynamic temperature
@codelion
return logprobs and toplogprobs
60a84ef 
add support for logprobs
@codelion
Update README.md
08cdffd
@codelion
Update README.md
87c5f3a
@codelion
Add none approach
2c66fc2 
fix logprobs return
@codelion
Merge branch 'feat-add-local-inference' of https://github.com/codelio…
9c2426a 
…n/optillm into feat-add-local-inference
@codelion
Update README.md
795556f
@codelion
Update inference.py
26030d0 
fix multiple loras loading and setting of adapters
@codelion
Merge branch 'feat-add-local-inference' of https://github.com/codelio…
24b559d 
…n/optillm into feat-add-local-inference
@codelion
add support for cot_decoding and entropy_decoding in local inference …
e6d61b1 
…server
@codelion
Aggregate paths in cot decoding by default
6a3ffa7
@codelion
Update README.md
3ff58a3
@codelion
Update setup.py
ad90fd8 
bump version for new release
@codelion codelion merged commit 7381008 into main Nov 13, 2024
@codelion codelion deleted the feat-add-local-inference branch November 13, 2024 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants

@codelion