Skip to content
This repository was archived by the owner on Oct 25, 2024. It is now read-only.

[NeuralChat] RAG evaluation #1333

Open
wants to merge 158 commits into
base: main
Choose a base branch
from
Open

[NeuralChat] RAG evaluation #1333

wants to merge 158 commits into from

Conversation

Liangyx2
Copy link
Contributor

@Liangyx2 Liangyx2 commented Mar 1, 2024 •
edited
Loading

  1. Data augmentation: retrieval dataset construction, include (1) Context to Question and Mine Hard Negatives, (2) Context, Question to Ground Truth.
  2. Retrieval evaluation: MRR (Mean reciprocal rank) and Hit (Hit Ratio) under different vector database, embedding model, retrieval type, search type.
  3. RAG evaluation: answer relevancy, faithfulness, context recall, context precision using Ragas framework.
@Liangyx2 Liangyx2 added the draft label Mar 1, 2024
@Liangyx2 Liangyx2 requested a review from lvliang-intel as a code owner March 1, 2024 03:12
@Liangyx2 Liangyx2 requested a review from XuhuiRen March 1, 2024 06:05
Liangyx2 and others added 24 commits March 1, 2024 14:58
@Liangyx2
Update llm_generate_raw_data.py
5ef0332 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
ee1db83 
…context_utils.py

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
89597f2 
for more information, see https://pre-commit.ci
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
b132d66 
…html_parser.py

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
update
8e955ce
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
635b906 
for more information, see https://pre-commit.ci
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
d7d3d03 
…hn_mine.py

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
c9fec02 
…llm_generate_raw_data.py

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
5e32113 
…mine_hard_negatives_check_similarity.py

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
f67622c 
…requirements.txt

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Delete intel_extension_for_transformers/neural_chat/tools/evaluation/…
f2e344a 
…retrieval_dataset_construction.py

Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update prompt.py
383e5b3 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
81014d1 
for more information, see https://pre-commit.ci
@Liangyx2
Update llm_generate_raw_data.py
4b7bec7 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update llm_generate_raw_data.py
0df51a6 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update retrieval_dataset_construction.py
95b16bd 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update llm_generate_raw_data.py
80dd21b 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update mine_hard_negatives_check_similarity.py
f495b22 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
add test_evaluation.py to nightly test
593dee3 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update and rename requirements.txt to requirements_cpu.txt
cf59b18 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Create requirements_cuda.txt
40e0b0e 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update requirements.txt
bf1b1aa 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update retrieval_dataset_construction.py
5552ebc 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update llm_generate_raw_data.py
d3b7579 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Liangyx2 added 3 commits March 28, 2024 15:13
@Liangyx2
Update README.md
8957b18 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update README.md
96f477c 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update README.md
c99856d 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
Liangyx2 added 8 commits April 1, 2024 10:59
@Liangyx2
Update README.md
19dfb93 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update README.md
99940f3 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update README.md
464d52b 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update README.md
da2e829 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update README.md
3ce2cb2 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update README.md
268d89c 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update README.md
40fc2e9 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update README.md
13bb3b8 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2 Liangyx2 removed the draft label Apr 3, 2024
xmx-521 and others added 3 commits April 10, 2024 17:36
@xmx-521
add config file form rag evaluation
763bd1d 
Signed-off-by: xmx-521 <1426356297@qq.com>
@xmx-521
complete config superbenchmark
092e951 
Signed-off-by: lvliang-intel <1426356297@qq.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
e931143 
for more information, see https://pre-commit.ci
@wenhuach21 wenhuach21 self-requested a review May 7, 2024 08:32
XuhuiRen and others added 6 commits May 8, 2024 09:41
@XuhuiRen
Merge branch 'main' into yuxiang/evaluation
f0a0cd6 
Signed-off-by: XuhuiRen <44249229+XuhuiRen@users.noreply.github.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
895075b 
for more information, see https://pre-commit.ci
@Liangyx2
Create test_evaluation.py in CI
6b60154 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Update requirements.txt
c73a68f 
Signed-off-by: Liangyx2 <yuxiang.liang@intel.com>
@Liangyx2
Merge branch 'main' into yuxiang/evaluation
c6f8906
@VincyZhang
Merge branch 'main' into yuxiang/evaluation
7c80ce2
@VincyZhang
Copy link
Contributor

@Liangyx2 @XuhuiRen any update?
@Liangyx2
Copy link
Contributor Author

@Liangyx2 @XuhuiRen any update?

No, only to solve the CI error if occurs.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
NeuralChat
6 participants
@Liangyx2 @VincyZhang @hshen14 @XuhuiRen @XuehaoSun @xmx-521