Merge pull request #2151 from GTimothee/qatest_push_to_hub

QATestset - push to hub integration

Author: henchaves

docs/integrations/huggingface/QATestset.md

@@ -0,0 +1,53 @@
+# 📤 Push a QATestset to the Hugging Face Hub
+
+**Learn how to upload and manage your QATestset on the Hugging Face Hub using the `push_to_hf_hub` feature.**
+
+This tutorial will guide you through the steps to push a dataset to the Hugging Face Hub and load it back for reuse.
+
+## Install Required Dependencies
+
+Before you begin, ensure you have the necessary libraries installed. Run the following command to install the `datasets` and `huggingface_hub` packages:
+
+```bash
+pip install datasets huggingface_hub
+```
+
+## Authenticate with Hugging Face
+
+To enable access to your account, set your Hugging Face authentication token (`HF_TOKEN`). You can generate your token from your [Hugging Face account settings](https://huggingface.co/settings/tokens).
+
+## Push Your Dataset to the Hub
+
+Use the `push_to_hf_hub` method to upload your dataset to the Hugging Face Hub. Replace `<username>` with your Hugging Face username and `<dataset_name>` with the desired name for your dataset:
+
+This example demonstrates how to load a `QATestset` from the file `test_set.jsonl` and push it to the Hugging Face Hub:
+
+```python
+from giskard.rag.testset import QATestset
+test_set = QATestset.load("test_set.jsonl")
+test_set.push_to_hf_hub("<username>/<dataset_name>")
+```
+
+Once the dataset is successfully pushed, it will be available on your Hugging Face profile.
+
+## Load the Dataset from the Hub
+
+To reuse the dataset, you can load it back using the `load_from_hf_hub` method. This example demonstrates how to load the dataset and convert it to a pandas DataFrame for inspection:
+
+```python
+from giskard.rag.testset import QATestset
+dset = QATestset.load_from_hf_hub("<username>/<dataset_name>")
+dset.to_pandas().head()
+```
+
+Replace `<username>` and `<dataset_name>` with the appropriate values.
+
+## Benefits of Using the Hugging Face Hub
+
+By leveraging this integration, you can:
+
+- Seamlessly share datasets across projects and collaborators.
+- Reuse datasets without the need for manual file transfers.
+- Access datasets directly from the Hugging Face Hub for streamlined workflows.
+
+Start pushing your datasets today and take advantage of the collaborative power of the Hugging Face Hub!

docs/integrations/huggingface/index.md

@@ -8,6 +8,7 @@
 :hidden:
 
 ./evaluator.md
+./QATestset.md
 
 ```
 
@@ -17,3 +18,8 @@
 :text-align: center
 :link: ./evaluator.md
 ::::
+
+::::{grid-item-card} <br/><h3>📤 Push a QATestset to the Hugging Face Hub</h3>
+:text-align: center
+:link: ./QATestset.md
+::::

docs/open_source/testset_generation/testset_generation/index.md

@@ -279,6 +279,8 @@ from giskard.rag import QATestset
 loaded_testset = QATestset.load("my_testset.jsonl")
 ```
 
+You can push your generated test set to the Hugging Face Hub or load an existing dataset from it using [`QATestset.push_to_hf_hub`](giskard.rag.QATestset.push_to_hf_hub) and [`QATestset.load_from_hf_hub`](giskard.rag.QATestset.load_from_hf_hub). This allows you to share and reuse datasets easily. For detailed instructions, refer to the [Hugging Face Integration Documentation](../../../integrations/huggingface/QATestset.md).
+
 You can also convert it to a pandas DataFrame, for quick inspection or further processing:
 
 ```py

giskard/llm/client/base.py

@@ -31,3 +31,8 @@ def complete(
         format=None,
     ) -> ChatMessage:
         ...
+
+    @abstractmethod
+    def get_config(self) -> dict:
+        """Return the configuration of the LLM client."""
+        ...

giskard/llm/client/bedrock.py

@@ -61,6 +61,10 @@ def complete(
 
         return self._parse_completion(completion, caller_id)
 
+    def get_config(self) -> dict:
+        """Return the configuration of the LLM client."""
+        return {"client_type": self.__class__.__name__, "model": self.model}
+
 
 @deprecated("ClaudeBedrockClient is deprecated: https://docs.giskard.ai/en/latest/open_source/setting_up/index.html")
 class ClaudeBedrockClient(BaseBedrockClient):

giskard/llm/client/gemini.py

@@ -57,6 +57,10 @@ def __init__(self, model: str = "gemini-pro", _client=None):
         self.model = model
         self._client = _client or genai.GenerativeModel(self.model)
 
+    def get_config(self) -> dict:
+        """Return the configuration of the LLM client."""
+        return {"client_type": self.__class__.__name__, "model": self.model}
+
     def complete(
         self,
         messages: Sequence[ChatMessage],

giskard/llm/client/litellm.py

@@ -151,3 +151,12 @@ def complete(
                     continue
 
         return ChatMessage(role=response_message.role, content=response_message.content)
+
+    def get_config(self) -> dict:
+        """Return the configuration of the LLM client."""
+        return {
+            "client_type": self.__class__.__name__,
+            "model": self.model,
+            "disable_structured_output": self.disable_structured_output,
+            "completion_params": self.completion_params,
+        }

giskard/llm/client/mistral.py

@@ -24,6 +24,10 @@ def __init__(self, model: str = "mistral-large-latest", client: Mistral = None):
         self.model = model
         self._client = client or Mistral(api_key=os.getenv("MISTRAL_API_KEY", ""))
 
+    def get_config(self) -> dict:
+        """Return the configuration of the LLM client."""
+        return {"client_type": self.__class__.__name__, "model": self.model}
+
     def complete(
         self,
         messages: Sequence[ChatMessage],

giskard/llm/client/openai.py

@@ -37,6 +37,14 @@ def __init__(
         self._client = client or openai.OpenAI()
         self.json_mode = json_mode if json_mode is not None else _supports_json_format(model)
 
+    def get_config(self) -> dict:
+        """Return the configuration of the LLM client."""
+        return {
+            "client_type": self.__class__.__name__,
+            "model": self.model,
+            "json_mode": self.json_mode,
+        }
+
     def complete(
         self,
         messages: Sequence[ChatMessage],

giskard/rag/dataset_card_template.md

@@ -0,0 +1,46 @@
+---
+tags:
+- giskard
+- synthetic
+
+task_categories:
+- text-generation
+- text2text-generation
+---
+
+# Dataset Card for {repo_id}
+This dataset was created using the [giskard](https://github.com/Giskard-AI/giskard) library, an open-source Python framework designed to evaluate and test AI systems. Giskard helps identify performance, bias, and security issues in AI applications, supporting both LLM-based systems like RAG agents and traditional machine learning models for tabular data.
+
+This dataset is a QA (Question/Answer) dataset, containing {num_items} pairs.
+
+## Usage
+
+You can load this dataset using the following code:
+
+```python
+from giskard.rag.testset import QATestset
+test_set = QATestset.load_from_hub("{repo_id}")
+```
+
+Refer to the following tutorial to use it for evaluating your RAG engine: [RAG evaluation tutorial](https://docs.giskard.ai/en/stable/open_source/testset_generation/rag_evaluation/index.html).
+
+## Configuration
+
+The configuration relative to the dataset generation:
+
+```bash
+{config}
+```
+
+---
+
+<h2 style="text-align: center;">
+  <span style="display: inline-flex; align-items: center; gap: 8px;">
+    Built with 
+    <a href="https://giskard.ai" target="_blank" style="display: inline-flex;">
+      <img src="https://cdn.prod.website-files.com/601d6f7d0b9c984f07bf10bc/62983fa8ef716259c397a57d_logo.svg" 
+             alt="Giskard Logo" 
+             width="100">
+    </a>
+  </span>
+</h2>