Note: This is not an officially supported Google product.

This repository contains code for evaluating LLMs on various natural language tasks in three Semitic Languages: Modern Hebrew, Modern Standard Arabic and Levantine Arabic. The tasks are detailed below. This page includes instructions for downloading, processing, fine-tuning, running inference, and evaluating models in a fine-tuning setting.

We include scripts for fine-tuning encoder-decoder and decoder LLMs, and for generating test set predictions using both Huggingface transformers and T5X. The evaluation script calculates performance metrics from these predictions. In this page we provide Baseline results on all tasks using mt5-XL.

The following tasks are supported:

Note that this package requires Python 3.10 or higher.

First, clone the repository:

git clone https://github.com/google-research/mrl_eval.git

Then, install the requirements, preferably in a new virtual environment.

pip install torch --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

Download and preprocess raw data for all tasks (done once per dataset):

bash mrl_eval/datasets/download_raw_data.sh
bash mrl_eval/datasets/ingest_all_datasets.sh

To prepare the dataset for the OntoNotes task (End-to-end and Gold Mentions variants) follow these steps:

1. Download the OntoNotes 5.0 dataset: Obtain the dataset from the LDC website (LDC2013T19).
2. Generate CoNLL 2012 Data Split in JSONL format: Run this script to generate the jsonl files for the Arabic subset of the dataset: setup_training.sh.
3. Run the data ingestion script: Copy the generated files to the data directory mrl_eval_data/ontonotes/ and mrl_eval_data/ontonotes_gold_mentions/, and run the ingestion script:

python -m mrl_eval.datasets.ingest_dataset ontonotes
python -m mrl_eval.datasets.ingest_dataset ontonotes_gold_mentions

To finetune on a specific dataset using Huggingface, run:

python -m mrl_eval.hf.finetune --dataset {dataset}

The options for dataset are:

• hebnli
• hesentiment
• heq
• heq_question_gen
• hesum
• hebsummaries
• nemo_token
• nemo_morph
• hebco
• hebco_gold_mentions
• arabic_nli
• msa_sentiment
• arq_MSA
• arq_MSA_question_gen
• artydiqa
• artydiqa_question_gen
• ar_xlsum
• asas
• iahlt_ner
• wojood_msa
• wojood_full
• ontonotes
• ontonotes_gold_mentions
• arsentiment
• arq_spoken
• arq_spoken_question_gen
• wojood_spoken
• shamner
• arcoref
• arcoref_gold_mentions

By default, the mt5-xl model will be trained. To train a different model (e.g. a decoder LLM) specify its HF model name as follows:

python -m mrl_eval.hf.finetune --dataset {dataset} --model "google/gemma-2-9b"

Decoder models will be trained by default with LORA using half precision.

Once the training is done, the script will print the path to the best checkpoint.

To generate responses for the inputs of the test set, run:

python -m mrl_eval.hf.generate --dataset {dataset} --checkpoint_path path/to/checkpoint

Alternatively, the T5X framework can also be used for running training and inference with T5-family models. For more details, see models/T5X.md.

To evaluate the score of model predictions, run:

python -m mrl_eval.evaluation.evaluate --dataset {dataset} --predictions_path path/to/prediction/file

The options for dataset are the same as above.

Your predictions file is expected to be a jsonl file in the following format:

{"input": {"id": "example_id_1"}, "prediction": "prediction1"}
{"input": {"id": "example_id_2"}, "prediction": "prediction2"}
...

We finetune mT5-xl model per task as the first baseline. Results are shown in the table below.

This framework includes an evaluation module that leverages a Large Language Model (LLM) for assessing the quality of question generation and summarization tasks - offering evaluation beyond word matching metrics. The specific model used for this evaluation is Gemini 2.5 Flash, accessed via the Vertex AI platform on Google Cloud.

More details about autoraters can be found in evaluation/autoraters/README.md.

For question generation the Autorater evaluates answerability which is the percentage of generated questions that can be accurately and directly answered by the provided reference answer/passage.

The following command can be used to run autorater evaluation on question generation datasets:

python -m mrl_eval.evaluation.autoraters.qg_autorater_main \
    --project_id={cloud-project-id} \
    --location={project-location} \
    --model_name="gemini-2.5-flash" \
    --dataset={question-generation-dataset} \
    --dataset_split=test \
    --predictions_path=path/to/prediction/file.jsonl \
    --output_path=path/to/output/file.jsonl

where question-generation-dataset can be one of:

• arq_MSA_question_gen
• arq_spoken_question_gen
• artydiqa_question_gen
• heq_question_gen

For summarisation, the autorater evaluates three dimensions:

• Factual grounding: all the information in the summary should be grounded in the original article.
• Relevance or Recall: every single fact of the summary should be “salient” enough to appear in the summary.
• Quality: the summary should be fluent and coherent. Each will be rated on a scale of 1-5.

The following command can be used to run autorater evaluation on summarisation datasets:

python -m mrl_eval.evaluation.autoraters.summarization_autoraters_main \
  --project_id={cloud-project-id} \
  --location={project-location} \
  --model_name="gemini-2.5-flash" \
  --dataset={summarisation-dataset} \
  --dataset_split="test" \
  --predictions_path=path/to/prediction/file.jsonl \
  --output_path=path/to/output/file.jsonl

where summarisation-dataset can be one of:

• hesum
• hebsummaries
• ar_xlsum
• asas