Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/integrations/mlflow/mlflow-llm-example.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@
},
"outputs": [],
"source": [
"%pip install langchain-openai openai pypdf faiss-cpu tiktoken pyngrok -q"
"%pip install langchain langchain-openai openai \"pypdf<=3.17.0\" faiss-cpu tiktoken pyngrok -q"
]
},
{
Expand All @@ -91,6 +91,7 @@
"outputs": [],
"source": [
"import mlflow\n",
"import giskard\n",
"\n",
"mlflow.models.list_evaluators() # ['default', 'giskard']"
]
Expand Down
29 changes: 15 additions & 14 deletions docs/integrations/mlflow/mlflow-tabular-example.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@
"cells": [
{
"cell_type": "markdown",
"metadata": {
"collapsed": false
},
"source": [
"# MLFlow Example - Tabular\n",
"## Detecting tabular ML models vulnerabilities in MLflow with Giskard\n",
Expand All @@ -11,17 +14,18 @@
"|----------|------------------------------------------------------------------------|-----------------|\n",
"| `model1` | A simple sklearn `LogisticRegression` model trained only for 5 epochs. | Titanic dataset |\n",
"| `model2` | A simple sklearn `LogisticRegression` model trained for 100 epochs. | Titanic dataset |"
],
"metadata": {
"collapsed": false
}
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import mlflow\n",
"import giskard\n",
"\n",
"from giskard import demo\n",
"model1, df = demo.titanic(max_iter=5)\n",
Expand All @@ -33,13 +37,13 @@
" with mlflow.start_run(run_name=model_name) as run:\n",
" model_uri = mlflow.sklearn.log_model(model, model_name, pyfunc_predict_fn=\"predict_proba\").model_uri\n",
" mlflow.evaluate(model=model_uri, model_type=\"classifier\", data=df, targets=\"Survived\", evaluators=\"giskard\", evaluator_config={\"model_config\": {\"classification_labels\": [\"no\", \"yes\"]}})"
],
"metadata": {
"collapsed": false
}
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": false
},
"source": [
"After completing the previous steps, you can run mlflow ui from the directory where the mlruns folder is located, which will enable you to visualize the results. By accessing http://127.0.0.1:5000, you will be presented with the interface. There, you will find the two LLMs logged as separate runs for comparison and analysis.\n",
"<img src=\"../../assets/integrations/mlflow/tabular/table_view.png\">\n",
Expand All @@ -52,17 +56,14 @@
"\n",
"A scan summary: After each model evaluation, a scan-summary.json file is created, enabling a comparison of vulnerabilities and metrics for each model in the Artifact view.\n",
"<img src=\"../../assets/integrations/mlflow/tabular/scan-summary.png\">"
],
"metadata": {
"collapsed": false
}
]
},
{
"cell_type": "markdown",
"source": [],
"metadata": {
"collapsed": false
}
},
"source": []
}
],
"metadata": {
Expand Down