Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
f61cc21
Created litellm client
kevinmessiaen Nov 5, 2024
e3844ff
Updated documentation
kevinmessiaen Nov 5, 2024
c524217
Added litellm embedding
kevinmessiaen Nov 5, 2024
d6b032f
Code improvement
kevinmessiaen Nov 5, 2024
e31cdfa
Added deprecated warnings
kevinmessiaen Nov 5, 2024
0f5ade7
Fixed typo
kevinmessiaen Nov 5, 2024
3045060
Improved documentation and llm setup
kevinmessiaen Nov 7, 2024
f49a2bc
Added back fastembed as default
kevinmessiaen Nov 7, 2024
1157bda
Removed todo: LiteLLM does not support embeddings for Gemini and Ollama
kevinmessiaen Nov 7, 2024
10e4113
Typo
kevinmessiaen Nov 7, 2024
5fb6a78
Fixed embeddings
kevinmessiaen Nov 7, 2024
f897262
Default model to gpt-4o
kevinmessiaen Nov 7, 2024
2657b19
Code cleanup
kevinmessiaen Nov 7, 2024
b04f126
Code cleanup
kevinmessiaen Nov 7, 2024
4633aa4
Skip LiteLLM tests with pydantic < 2
kevinmessiaen Nov 8, 2024
63ace19
Added test for custom client
kevinmessiaen Nov 8, 2024
1b382ee
Added test for embedding
kevinmessiaen Nov 8, 2024
713f0b0
Fixed tests
kevinmessiaen Nov 8, 2024
deca09a
Merge branch 'main' into feature/litellm
henchaves Nov 14, 2024
e54c414
Merge branch 'main' into feature/litellm
henchaves Nov 14, 2024
dee0e83
Reintroduced old way to set LLM models
kevinmessiaen Nov 15, 2024
7703d51
Reintroduced old way to set LLM models
kevinmessiaen Nov 15, 2024
5349fc2
Reintroduced old clients
kevinmessiaen Nov 15, 2024
6458f97
Merge branch 'main' into feature/litellm
kevinmessiaen Nov 15, 2024
2756e27
Fixed OpenAI embeddings
kevinmessiaen Nov 15, 2024
2b88ed3
Update Setting up the LLM client docs
henchaves Nov 15, 2024
1dc73d9
Update Setting up the LLM client docs pt 2
henchaves Nov 15, 2024
b39731e
Update testset generation docs
henchaves Nov 15, 2024
7eaf007
Update scan llm docs
henchaves Nov 15, 2024
5f51327
Merge branch 'main' into feature/litellm
henchaves Nov 18, 2024
39c4fa9
Removed response_format with ollama models due to issue in litellm
kevinmessiaen Nov 19, 2024
b09d266
Added dumb trim
kevinmessiaen Nov 19, 2024
911d6e5
Fixed output
kevinmessiaen Nov 19, 2024
40bede9
Add _parse_json_output to LiteLLM
henchaves Nov 19, 2024
77e6a4f
Added way to disable structured output
kevinmessiaen Nov 20, 2024
cab45a1
Fix test_litellm_client
henchaves Nov 21, 2024
5f39da1
Merge branch 'main' into feature/litellm
henchaves Nov 21, 2024
78dd03e
Check if format is json before calling _parse_json_output
henchaves Nov 21, 2024
82712c7
Set LITELLM_LOG as error level
henchaves Nov 21, 2024
a61e4b2
Add `disable_structured_output` to bedrock examples
henchaves Nov 21, 2024
3d33028
Format files
henchaves Nov 21, 2024
a571312
Fix sonar issues
henchaves Nov 21, 2024
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update Setting up the LLM client docs
  • Loading branch information
henchaves committed Nov 15, 2024
commit 2b88ed3e51e163e0d165ad240a233816faf99362
37 changes: 23 additions & 14 deletions docs/open_source/setting_up/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,6 @@ giskard.llm.set_llm_model("azure/<your_deployment_name>", api_base="", api_versi
giskard.llm.set_embedding_model("azure/<your_deployment_name>", api_base="", api_version="", azure_ad_token="")
```


## Mistral Client Setup

More information on [LiteLLM documentation](https://docs.litellm.ai/docs/providers/mistral)
Expand All @@ -79,7 +78,7 @@ More information on [LiteLLM documentation](https://docs.litellm.ai/docs/provide
import os
import giskard

os.environ['MISTRAL_API_KEY'] = ""
os.environ['MISTRAL_API_KEY'] = "" # "my-mistral-api-key"

giskard.llm.set_llm_model("mistral/mistral-tiny")
giskard.llm.set_embedding_model("mistral/mistral-embed")
Expand All @@ -95,7 +94,16 @@ More information on [LiteLLM documentation](https://docs.litellm.ai/docs/provide
```python
import giskard

giskard.llm.set_llm_model("ollama/llama2", api_base="http://localhost:11434") # See supported models here: https://docs.litellm.ai/docs/providers/ollama#ollama-models
# See supported models here: https://docs.litellm.ai/docs/providers/ollama#ollama-models
giskard.llm.set_llm_model("ollama/llama3", api_base="http://localhost:11434")
giskard.llm.set_embedding_model("ollama/nomic-embed-text", api_base="http://localhost:11434")
```

If you encounter errors with the embedding model in a Jupyter notebook, run this code:

```python
import nest_asyncio
nest_asyncio.apply()
```

## AWS Bedrock Client Setup
Expand All @@ -108,12 +116,12 @@ More information on [LiteLLM documentation](https://docs.litellm.ai/docs/provide
import os
import giskard

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""
os.environ["AWS_ACCESS_KEY_ID"] = "" # "my-aws-access-key"
os.environ["AWS_SECRET_ACCESS_KEY"] = "" # "my-aws-secret-access-key"
os.environ["AWS_REGION_NAME"] = "" # "us-west-2"

giskard.llm.set_llm_model("bedrock/anthropic.claude-3-sonnet-20240229-v1:0")
giskard.llm.set_embedding_model("bedrock/amazon.titan-embed-text-v1")
giskard.llm.set_embedding_model("bedrock/amazon.titan-embed-image-v1")
```

## Gemini Client Setup
Expand All @@ -126,14 +134,15 @@ More information on [LiteLLM documentation](https://docs.litellm.ai/docs/provide
import os
import giskard

os.environ["GEMINI_API_KEY"] = "your-api-key"
os.environ["GEMINI_API_KEY"] = "" # "my-gemini-api-key"

giskard.llm.set_llm_model("gemini/gemini-pro")
giskard.llm.set_embedding_model("gemini/text-embedding-004")
```

## Custom Client Setup

More information on [LiteLLM documentation](https://docs.litellm.ai/docs/providers/custom_llm_server )
More information on [LiteLLM documentation](https://docs.litellm.ai/docs/providers/custom_llm_server)

```python
import requests
Expand All @@ -148,23 +157,23 @@ class MyCustomLLM(litellm.CustomLLM):
api_key = api_key or os.environ.get('MY_SECRET_KEY')
if api_key is None:
raise litellm.AuthenticationError("Api key is not provided")
response = requests.post('https://www.my-fake-llm.ai/chat/completion', json={

response = requests.post('https://www.my-custom-llm.ai/chat/completion', json={
'messages': messages
}, headers={'Authorization': api_key})

return litellm.ModelResponse(**response.json())


my_custom_llm = MyCustomLLM()

litellm.custom_provider_map = [ # 👈 KEY STEP - REGISTER HANDLER
{"provider": "my-custom-llm", "custom_handler": my_custom_llm}
{"provider": "my-custom-llm-endpoint", "custom_handler": my_custom_llm}
]

api_key = os.environ['MY_SECRET_KEY']

giskard.llm.set_llm_model("my-custom-llm/my-fake-llm-model", api_key=api_key)
giskard.llm.set_llm_model("my-custom-llm-endpoint/my-custom-model", api_key=api_key)
```

If you run into any issues configuring the LLM client, don't hesitate to [ask us on Discord](https://discord.com/invite/ABvfpbu69R) or open a new issue on [our GitHub repo](https://github.com/Giskard-AI/giskard).
Loading