Skip to content

Conversation

@GTimothee
Copy link
Contributor

Description

Documentation on how to implement custom metrics with raget.

Related Issue

#2140

Type of Change

  • 📚 Examples / docs / tutorials / dependencies update
  • 🔧 Bug fix (non-breaking change which fixes an issue)
  • 🥂 Improvement (non-breaking change which improves an existing feature)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to change)
  • 🔐 Security fix

Checklist

  • I've read the CODE_OF_CONDUCT.md document.
  • I've read the CONTRIBUTING.md guide.
  • I've written tests for all new methods and classes that I created.
  • I've written the docstring in Google format for all the methods and classes that I used.
  • I've updated the pdm.lock running pdm update-lock (only applicable when pyproject.toml has been
    modified)
@davidberenstein1957 davidberenstein1957 self-requested a review April 8, 2025 07:28
Copy link
Member

@davidberenstein1957 davidberenstein1957 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! Left some minor remarks. I think there can be some more guidance and nuance overall (not too much but a bit more) but it is looking great :)

@GTimothee
Copy link
Contributor Author

GTimothee commented Apr 8, 2025

I did an important update, I hope it is better like this.

@davidberenstein1957 davidberenstein1957 added safe for build Lockfile Temporary label to update pdm.lock labels Apr 9, 2025
Copy link
Member

@davidberenstein1957 davidberenstein1957 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, some final remarks. After, we can merge


```python
# Instantiate the custom correctness metric
correctness_score = CorrectnessScoreMetric(name="correctness_score")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we emphasise something about the "correctness_score" name?

Copy link
Contributor Author

@GTimothee GTimothee Apr 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidberenstein1957 I am not sure what to say about that. From what I tested you can use whatever name you want it does not seem to be used anywhere. I tried setting a random name as the metric name and there is no error, there is still the correctness_score in the RAGReport and when we print the report as html it is correctness_score that shows up in the "selected metrics" panel. Edit: .name attribute seems mostly used by the RagasMetric metric; they use the .name as a key in the dict that is returned by the metric.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, let's keep this for a more serious overhaul then.

@henchaves henchaves added safe for build and removed safe for build Lockfile Temporary label to update pdm.lock labels Apr 9, 2025
@henchaves henchaves changed the title Doc update - how to create a custom metrics Apr 10, 2025
@davidberenstein1957 davidberenstein1957 merged commit c13fbce into Giskard-AI:main Apr 14, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

3 participants