Merge pull request #2 from AnuradhaKaruppiah/eval-doc-fixes

Update evaluation docs

Author: AnuradhaKaruppiah

docs/source/concepts/evaluate.md

@@ -29,15 +29,15 @@ aiq eval --config_file=examples/simple/configs/eval_config.yml
 ```
 
 ## Using Datasets
-Run and  evaluate the workflow on a specified dataset. The dataset files types are `json`, `jsonl`, `csv`, `xls`, or `parquet`.
+Run and evaluate the workflow on a specified dataset. The dataset files types are `json`, `jsonl`, `csv`, `xls`, or `parquet`.
 
 Download and use datasets provided by AgentIQ examples by running the following.
 
 ```bash
 git lfs fetch
 git lfs pull
 ```
- The dataset used for evaluation is specified in the `config.yml` file  via `eval.general.dataset`. For example, to use the `langsmith.json` dataset, the configuration is as follows:
+ The dataset used for evaluation is specified in the configuration file  via `eval.general.dataset`. For example, to use the `langsmith.json` dataset, the configuration is as follows:
 ```yaml
 eval:
   general:
@@ -246,11 +246,20 @@ aiq eval --config_file=examples/simple/configs/eval_config.yml --skip_completed_
 ## Running evaluation offline
 You can evaluate a dataset with previously generated answers via the `--skip_workflow` option. In this case the dataset has both the expected `answer` and the `generated_answer`.
 ```bash
-aiq eval --config_file=examples/simple/configs/config.yml --skip_workflow
+aiq eval --config_file=examples/simple/configs/eval_config.yml --skip_workflow --dataset=.tmp/aiq/examples/simple/workflow_output.json
 ```
+This assumes that the workflow output is previously generated and stored in the `.tmp/aiq/examples/simple/workflow_output.json` file.
 
 ## Running the workflow over a dataset without evaluation
-You can do this via a config.yml file that has no `evaluators`.
+You can do this by running `aiq eval` with a workflow configuration file that includes an `eval` section with no `evaluators`.
+```yaml
+eval:
+  general:
+    output_dir: ./.tmp/aiq/examples/simple/
+    dataset:
+      _type: json
+      file_path: examples/simple/data/langsmith.json
+```
 
 ## Evaluation output
 The output of the workflow is stored as `workflow_output.json` in the `output_dir` provided in the config.yml -

docs/source/guides/custom-evaluator.md

@@ -44,15 +44,17 @@ The following is an example of an evaluator configuration and evaluator function
 
 `examples/simple/src/aiq_simple/evaluator_register.py`:
 ```python
+from pydantic import Field
+
 from aiq.builder.builder import EvalBuilder
 from aiq.builder.evaluator import EvaluatorInfo
 from aiq.cli.register_workflow import register_evaluator
 from aiq.data_models.evaluator import EvaluatorBaseConfig
 
 
 class SimilarityEvaluatorConfig(EvaluatorBaseConfig, name="similarity"):
-    '''Configuration for custom evaluator'''
-    similarity_type: str = "cosine"
+    '''Configuration for custom similarity evaluator'''
+    similarity_type: str = = Field(description="Similarity type to be computed", default="cosine")
 
 
 @register_evaluator(config_type=SimilarityEvaluatorConfig)
@@ -72,6 +74,7 @@ The `register_similarity_evaluator` function is used to register the evaluator w
 
 To ensure that evaluator is registered the evaluator function is imported, but not used, in the simple example's `register.py`
 
+`examples/simple/src/aiq_simple/register.py`:
 ```python
 from .evaluator_register import register_similarity_evaluator  # pylint: disable=unused-import
 ```
@@ -80,9 +83,9 @@ from .evaluator_register import register_similarity_evaluator  # pylint: disable
 The asynchronous evaluate method provide by the custom evaluator takes an `EvalInput` object as input and returns an `EvalOutput` object as output.
 
 `EvalInput` is a list of `EvalInputItem` objects. Each `EvalInputItem` object contains the following fields:
-- `id`: The unique identifier for the item.
-- `input_obj`: This is typically the question. It can be a string or any serializable object.
-- `expected_output_obj`: The expected answer for the question. This can be a string or any serializable object.
+- `id`: The unique identifier for the item. It is defined in the dataset file and can be an integer or a string.
+- `input_obj`: This is typically the question. It is derived from the dataset file and can be a string or any serializable object.
+- `expected_output_obj`: The expected answer for the question. It is derived from the dataset file and can be a string or any serializable object.
 - `output_obj`: The answer generated by the workflow for the question. This can be a string or any serializable object.
 - `trajectory`: List of intermediate steps returned by the workflow. This is a list of `IntermediateStep` objects.
 
@@ -97,8 +100,8 @@ The evaluate method computes the score for each item in the evaluation input and
 
 ### Similarity Evaluator
 Similarity evaluator is used as an example to demonstrate the process of creating and registering a custom evaluator with AgentIQ. We add this code to a new `similarity_evaluator.py` file in the simple example directory for testing purposes.
-`examples/simple/src/aiq_simple/similarity_evaluator.py`:
 
+`examples/simple/src/aiq_simple/similarity_evaluator.py`:
 ```python
 import asyncio
 
@@ -152,7 +155,7 @@ class SimilarityEvaluator:
         sample_scores, sample_reasonings = zip(*results) if results else ([], [])
 
         # Compute average score
-        avg_score = sum(sample_scores) / len(sample_scores) if sample_scores else 0.0
+        avg_score = round(sum(sample_scores) / len(sample_scores), 2) if sample_scores else 0.0
 
         # Construct EvalOutputItems
         eval_output_items = [
@@ -208,7 +211,7 @@ The results of each evaluator is stored in a separate file with name `<keyword>_
 `examples/simple/.tmp/aiq/examples/simple/similarity_eval_output.json`:
 ```json
 {
-  "average_score": 0.6333333333333334,
+  "average_score": 0.63,
   "eval_output_items": [
     {
       "id": 1,

docs/source/guides/evaluate.md

@@ -68,6 +68,12 @@ The dataset file provides a list of questions and expected answers. The followin
 ## Understanding the Evaluator Configuration
 The evaluators section specifies the evaluators to use for evaluating the workflow output. The evaluator configuration includes the evaluator type, the metric to evaluate, and any additional parameters required by the evaluator.
 
+### Display all evaluators
+To display all existing evaluators, run the following command:
+```bash
+aiq info components -t evaluator
+```
+
 ### Ragas Evaluator
 [RAGAS](https://docs.ragas.io/) is an OSS evaluation framework that enables end-to-end
 evaluation of RAG workflows. AgentIQ provides an interface to RAGAS to evaluate the performance