Skip to content

KeyError: 'correctness' when running evaluate in giskard.rag #2111

@kunjanshah0811

Description

@kunjanshah0811

When running evaluate from giskard.rag with the specified metrics ragas_context_recall and ragas_context_precision, the following error occurs

from giskard.rag import evaluate, RAGReport, AgentAnswer
from giskard.rag.metrics.ragas_metrics import ragas_context_recall, ragas_context_precision

def answer_fn(question, history=None):

    answer = chat_engine.chat(question, chat_history=[])

    return AgentAnswer(
        message=answer.response,
        documents=[source.content for source in answer.sources]
    )

report = evaluate(answer_fn, 
                testset=testset, 
                knowledge_base=knowledge_base,
                metrics=[ragas_context_recall, ragas_context_precision])
report.save("test_report")

ERROR

KeyError                                  Traceback (most recent call last)
File c:\Users\KUNJAN SHAH\AppData\Local\Programs\Python\Python311\Lib\site-packages\pandas\core\indexes\base.py:3805, in Index.get_loc(self, key)
   3804 try:
-> 3805     return self._engine.get_loc(casted_key)
   3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas\\_libs\\hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas\\_libs\\hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'correctness'```

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions