ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models
Abstract
The rapid advancements in Large Language Models (LLMs) have led to significant improvements in various natural language processing tasks. However, the evaluation of LLMs’ legal knowledge, particularly in non-English languages such as Arabic, remains under-explored. To address this gap, we introduce ArabLegalEval
, a multitask benchmark dataset for assessing the Arabic legal knowledge of LLMs.
Inspired by the MMLU
and LegalBench
datasets, ArabLegalEval
consists of multiple tasks sourced from Saudi legal documents and synthesized questions. In this work, we aim to analyze the capabilities required to solve legal problems in Arabic and benchmark the performance of state-of-the-art LLMs. We explore the impact of in-context learning and investigate various evaluation methods. Additionally, we explore workflows for generating questions with automatic validation to enhance the dataset’s quality. We benchmark multilingual and Arabic-centric LLMs, such as GPT-4
and Jais
, respectively. We also share our methodology for creating the dataset and validation, which can be generalized to other domains.
We hope to accelerate AI research in the Arabic Legal domain by releasing the ArabLegalEval dataset and code: https://github.com/Thiqah/ArabLegalEval
utf8
ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models
Faris Hijazi1, Somayah AlHarbi1, Abdulaziz AlHussein1, Harethah Abu Shairah2, Reem AlZahrani2, Hebah AlShamlan1, Omar Knio2, George Turkiyyah2 1THIQAH, 2KAUST
1 Introduction
The development of LLMs has revolutionized various fields by enhancing natural language understanding and generation capabilities. However, the applicability and performance of these models in specialized domains, such as legal contexts, specially in low- and medium-resource languages such as Arabic, remain active research areas. In this paper, we report on ongoing work for evaluating the proficiency of large language models in understanding and processing Arabic legal texts. Given the complexity and richness of legal language, especially in Arabic, it is crucial to develop benchmarks that accurately assess the models’ capabilities in this domain in order to guide model development.
One of the key motivations for this work and benchmark is to find out the current state of LLMs in the Arabic Legal domain. Thus, we benchmark a wide range of LLMs from proprietary multilingual LLMs, such as GPT-4
OpenAI (2024a), to open-source Arabic-centric LLMs, such as Jais
Sengupta et al. (2023).
![Refer to caption](extracted/5790735/latex/assets/ArabLegalEval.png)
There are perhaps two broad categories of evaluation criteria that are useful for assessing the performance of legal LLMs. The first category is the ability of a model to use specific regulations, facts, and data that are relevant to a particular conversation. This can be achieved by having the LLM memorize specific facts, presumably by finetuning it on particular legal corpora, or more conveniently by using a Retrieval-Augmented Generation (RAG) system to retrieve information relevant to the context at hand. The second broad category of assessment criteria is related to the model’s ability to exhibit logical reasoning, understand relationships between entities and events, and apply these skills to answer questions.
In this initial release of ArabLegalEval
, we include tasks to assess the legal reasoning capabilities in Arabic as well as tasks to measure the ability to recall and use legal knowledge embedded in the finetuned models.
Much like ArabicMMLU
Koto et al. (2024) was designed to test the general reasoning capabilities of Arabic LLMs but includes a mix of tasks with some requiring previous knowledge, the ArabLegalEval
benchmark seeks to develop Arabic legal tasks and questions derived from original Arabic legal sources, in consultation with legal professionals, to test the legal reasoning capabilities of Arabic LLMs. The benchmark also includes some high quality translations, verified by legal experts, of tasks from English legal benchmarks (LegalBench
) Guha et al. (2023).
The benchmark includes Arabic legal Multiple Choice Questions (MCQs), Question & Answer (QA) pairs where the relevant Saudi regulations are included in the question, as well as Arabic translations of tasks from LegalBench
.
The benchmark does not assess the ability of models to retrieve specific facts and laws relevant to a particular context from external knowledge bases. While important for the successful deployment of a legal LLM, these retrieval abilities will be assessed in future tasks.
The primary contributions of this paper are: the development of a novel methodology for generating a legal QA dataset that can be adapted to other domains, such as finance; and ArabLegalEval
, an initial dataset specifically designed for assessing Arabic legal knowledge in large language models.
2 Related work
2.1 Arabic and multilingual reasoning capabilities in LLMs
The success of LLMs, such as GPT-4
and GPT-4o
OpenAI (2024a, b), Claude-3
Anthropic (2024), Command R
and Command R Plus
Cohere (2024a, b), and Llama3
LLaMA (2024), in exhibiting general purpose reasoning abilities - when queried in English - have naturally led to the development of models trained on Arabic content as well as benchmarks to evaluate the quality of reasoning in Arabic. Jais
Sengupta et al. (2023),
for example, trained on native Arabic and quality translations, has shown better performance than other open-weight models on an Arabic version of the MMLU
multiple-choice school exam questions benchmark Koto et al. (2024).
Arabic content and questions are also part of many broadly used multilingual benchmarks for assessing general reasoning. For example, MLQA Lewis et al. (2020) assesses reading comprehension and question answering capabilities in languages including Arabic.
While substantial experimentation is ongoing for evaluating reasoning and knowledge-based tasks in multilingual models, there are some model scale effects that appear to be emerging. On one hand, smaller models and models trained with limited data do not generally perform very well. Even GPT-3.5
Ouyang et al. (2022) performs worse when responding to queries in Arabic and Non-English languages as compared to English on various tasks Koto et al. (2024); Jin et al. (2023). On the other hand, the performance generally improves steadily with increasing model scale Shi et al. (2023). GPT-4
, for example, exhibits remarkable performance on all sections of ArabicMMLU
surpassing all other models including Arabic-centric ones, even when few-shot prompts are used. It appears that
multilingual reasoning is an emergent ability of large language models. Beyond a certain (task-dependent) scale, LLMs have evidently strong multilingual reasoning abilities.
2.2 LLMs and evaluation benchmarks in legal domains
Given the importance of natural language to law, the advancing capabilities of large language models have been very quickly recognized and used for performing legals tasks. Current efforts are ongoing to explore whether current LLMs can be used as legal assistants for producing background research, drafting initial documents, summarizing contracts, answering questions about reports, and handling related tasks Nay et al. (2023); Perlman (2022); Goth (2023). The announcement that GPT-4
has “passed the bar exam” Katz et al. (2024) has shown potential that intelligent legal advisors are not too far off. In the area of tax law for example, LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy Nay et al. (2023). Systems for Arabic court rulings and QA related to Legal Palestinian cooperative maters were presented in Ammar et al. (2024); Maree et al. (2024).
However, closer inspection has revealed that certification exams or narrow domains are not always representative of the actual use-cases in law practice or for LLMs Fei et al. (2023). As a result, a number of efforts have been ongoing to develop models, agents, and benchmarks that use tasks and
general conceptual frameworks similar to those used in the legal profession. LegalBench
and a Chinese counterpart, LawBench
Fei et al. (2023), have sought to collect and categorize a broad set of tasks that appear in law practice, from simple tasks requiring rule-recall or issue recognition to sophisticated tasks that require interpretation, drawing conclusions, and multi-hop reasoning. As many of the LegalBench tasks, particularly those requiring interpretation, appear to involve reasoning skills that are not particular to US law, it seems reasonable to use translated versions of these tasks for testing LLMs in different languages.
3 Data Sources
3.1 Raw Arabic data sources
Obtaining comprehensive Arabic legal data is challenging. After consulting Saudi legal experts, we identified key sources that we have started to incorporate (Figure 1). These include scanned documents from the Ministry of Justice (MoJ, \< >) and the Board of Experts (BoE, \< >). MoJ documents cover 67 regulations (5720 subjects) and 388 circulars, while BoE documents include 448 rules (14134 subjects). See Appendix B for samples.
Both sets of documents contain regulations and statutes, they differ primarily in the topics they cover. MoJ documents specialize in topics issued by the Ministry, providing a comprehensive database of judicial regulations and legislation essential for legal research and practice.
Note that all data sources used are open and publicly available and scraped from official websites with no confidential information. These were the raw data sources used to generated the benchmark
3.1.1 Preparation Steps
For both BoE and MoJ documents, we followed a systematic data preparation process to ensure that the data is rich and easy to work with. The data was scraped from the web to capture all the regulations while preserving all metadata to allow for careful filtering later.
An example of a processed, structured, MoJ data document is shown in Figure 10.
3.1.2 Frequently asked Questions (FAQs)
In addition to these raw sources, we also rely on human-written FAQs that are publicly available. The questions and answers in this data are generally available in the BoE and MoJ documents, and we use them to build an open-ended question answering task in the benchmark, which we call NajizQA. A sample of this data can be found in Figure 11.
3.2 LegalBench
In addition to native Arabic sources, we rely on the translation of English legal documents.
Legalbench
is a benchmark for legal reasoning in English LLMs Guha et al. (2023). We selected four datasets from it and translated them from English to Arabic.
These were specifically chosen because they have fewer localization requirements and are more universal. This makes them ideal for assessing the ability of LLMs to understand and interpret legal clauses and contracts in Arabic.
Consumer Contracts QA: this dataset consists of 400 yes/no questions about the rights and obligations outlined in online terms of service agreements.
Contracts QA: this dataset consists contract clauses and questions about these contracts. It has 88 examples, with 80 examples for testing and 8 examples for training.
Privacy Policy QA: the dataset consists of questions and corresponding clauses from a privacy policy. It consists of a total of 10,931 instances, with 8 examples for training and 10,923 for testing.
Privacy Policy Entailment: this dataset has 4385 examples in training and testing, each example is a privacy policy clause and a description. The goal is to determine if the description for the clause is correct or not.
3.3 ArabicMMLU
ArabicMMLU
Koto et al. (2024), an Arabic knowledge evaluation benchmark constructed from human-written school exams from Arabic-speaking countries, served as one of the inspirations for this work. With a subset of its samples focused on the legal domain, ArabicMMLU
provided a valuable starting point for us to generate our MCQs.
4 Benckmark Tasks
In this section, we describe the three broad task categories in the benchmark, including 10,000+ MCQs from the native Arabic MoJ and BoE documents, a set of QA from these documents, and a quality translation of a subset of the English LegalBench
benchmark related to consumer contracts and privacy policies. See Figure 1. We believe that the mixture of questions from native Arabic documents and translated questions gives us a somewhat diverse set of tasks and allows the benchmark to test a broader set of capabilities. In addition, this allows us to test the observation that, with increasing model scale, multilingual LLMs can display reasoning abilities and semantic judgment in Arabic as well as in they do in English Shi et al. (2023).
The quantitative details and human performance baseline are presented in Appendix J.
4.1 MCQs
One standard method of benchmarking reasoning and memorization capabilities in neural networks are MCQs, such as MMLU
Hendrycks et al. (2021). It is easy to verify the correctness of the answer using exact matching or regular expressions making MCQs ideal for automatic evaluation.
Given the availability of a large collection of raw legal documents from the MoJ and BoE, we aim to generate synthetic MCQs from them, using them the documents as context.
Generating MCQs poses two main challenges: formulating questions and generating options (correct answers and plausible distractors).
We approached this using a robust LLM and experimented with three methods: 1. QA to MCQ, 2. Chain of Thought (CoT), and 3. Retrieval-based in-context learning. See Figure 2.
In all cases, we prompt the model to synthesize questions in the same format and style as MMLU
.
![Refer to caption](extracted/5790735/latex/assets/fig2.png)
4.1.1 QA to MCQ
Here we use a two-step prompt. Given a legal document, the model is prompted to generate both a question and its answer, then a follow-up prompt to convert the question into a Multiple Choice Question (MCQ) by rephrasing the answer and generating appropriate distractors,
where is a language model that is given some legal context , and then prompted to generate a QA pair , and is another instance of the model that is given the QA pair and is tasked to generate a set of distractors for the question.
4.1.2 QA to MCQ with CoT
CoT is a relatively recent method of prompting LLMs where instead of directly producing the answer, the model is given space to reason and “think out loud” before answering, essentially providing itself with more context. This simple technique has led to relatively huge gains in performance, in a variety of tasks Wei et al. (2023). We want to utilize this idea to generate reasoning-based plausible distractors. This approach can be formulated as:
where is a language model prompted with some context from the scraped MoJ documents to produce a question and its answer , these are then fed to another instance of the model with CoT prompt to produce a set of plausible distractors for the question.
4.1.3 Direct MCQs generation with in-context examples
Since the inspirations of this work are popular knowledge evaluation benchmarks such as MMLU
and ArabicMMLU
, our goal is to generate questions in a similar format. We begin by taking a subset of ArabicMMLU
questions that have ‘Law’ as their subject tag (about 300 examples). For each document in the MoJ dataset, we perform a semantic similarity search Risch et al. (2021) to retrieve the top examples from these questions. These examples are then added to the prompt to guide the model in generating questions of a similar style.
The MoJ context is augmented with a set of ArabicMMLU
examples that are fed to a model to generate a question , where is the number of retrieved examples. These examples provide context for the model for in-context learning so that the generated answer has the same style and format as the ArabicMMLU
examples. From this, the model generated a question and all of the distractors in one go.
4.1.4 MCQ filtering and curation
Each of the above techniques was tested and then had the results manually reviewed and inspected by the legal experts according to the metrics in Appendix 3.1
It was concluded that the best method for generating MCQs was in-context examples with . Based on this, we decided to use this technique to generate all the MCQs.
After generating the MCQs dataset (approximately 12k samples), we did some automatic filtering using GPT-4
Chiang and yi Lee (2023), where it was prompted to check if each sample satisfied our criteria (see appendix subsection 3.1). Any sample that failed to satisfy all of the above were removed, which left us with 10,583 MCQs for our benchmark. At the end, we extracted a random subset of the dataset for a final manual inspection.
4.1.5 Models for Generation
![Refer to caption](extracted/5790735/latex/barplot.png)
It has been observed that models tend to perform better on synthetic data generated by themselves as apposed to another model Huang et al. (2024).
To mitigate this unfair advantage, we split our documents and alternate between two state of the art models: Claude-3-opus
and GPT-4
. We make sure a model isn’t evaluated on questions generated by itself.
4.2 QA
This dataset includes filtered legal QAs from a publicly available human written set of FAQs. The questions include only those referencing specific statues and regulations and articles in the MoJ and BoE Arabic documents, and are therefore particularly valuable for evaluating Arabic legal LLMs. See figure LABEL:fig:NajizQAproc. A sample of the data shown in the Appendix B.
Embedding techniques were used in the semantic similarity matching phase (Figure LABEL:fig:NajizQAproc). Specifically, we utilized text-embedding-3-small
OpenAI (2024c) to generate embeddings for both the questions and contextual information. Subsequently, cosine similarity
was employed to identify relevant texts corresponding to each question. This methodological framework was selected to evaluate the models’ reasoning capabilities with and without context.
![Refer to caption](extracted/5790735/latex/assets/fig4.png)
4.3 Arabic translation of LegalBench
4.3.1 Translation Strategy
We evaluated three different machine translation models, Azure Translation Services Microsoft (n.d.), Google API Translation Google (n.d.), and Opus MT Tiedemann and Thottingal (2020) along with GPT-4
to determine the best translator for a publicly available dataset with legal context. The dataset used in these experiments is the United Nations Parallel Corpus Ziemski et al. (2016), which consists of UN documents in the six official UN languages.
We focused solely on the English and Arabic datasets, which comprise 20 million rows.
However, to expedite the experimentation process, we utilized a subset of 14,000 examples from this dataset. This subset was used to evaluate the performance of the selected models without any preprocessing. Rouge metrics were used in evaluating the translation quality.
By comparing the results obtained from the different models, we aimed to identify the one that consistently produced the best translation from English to Arabic in the legal domain. The results of our experiments on this subset of data, without any prepossessing, are shown in Table 3 in Appendix D. Overall, the Opus model outperformed the other models across all the metrics. The Opus model achieved a ROUGE-1 score of 0.52, a ROUGE-2 score of 0.3, and a ROUGE-L score of 0.51. In addition to this, we had a sample of the translations manually reviewed by legal experts.
These findings established a basis for selecting Opus as the translator for our main task: translating selected datasets from LegalBench
from English to Arabic.
4.3.2 Evaluation of Translated Output
To ensure the highest translation quality, we employed the three models from our initial experiments to translate the “Consumer Contract QA” dataset. We conducted a manual inspection of the overall results and, to further enhance the quality assessment, engaged both GPT-4
and legal experts as evaluators for the translated content.
GPT-4 Evaluation
We tasked GPT-4
with evaluating a sample of the translated text alongside the original text, asking it to rate the translation quality on a scale from 1 to 5. Interestingly, GPT-4
consistently rated the translations as either 4 or 5 out of 5, indicating a high level of perceived quality, see Appendix D for GPT-4
evaluation prompt and result example.
Human Evaluation
We selected five examples from the translated texts, each consisting of a contract text and a corresponding question. To evaluate the performance of the three translation models — Azure Translation Services, Google API Services, and Opus MT — we asked legal experts to assess each contract and question separately. The experts provided scores for each model’s based on the translation quality and how well the legal context was preserved in the translation. The overall scores for all three models were relatively close, but Opus MT consistently achieved the highest score among them:
-
•
Google API: average score of 3.3 out of 5
-
•
Azure: average score of 3.6 out of 5
-
•
Opus MT: Average score of 4 out of 5
In addition to the overall scores, we asked the experts to rank the best model for each example. Opus MT was chosen as the best model 60% of the time, while Azure Translation Services was selected 40% of the time. Interestingly, Google API Translation was never selected as the best model for any of the examples.
5 Evaluation and results
5.1 MCQ Evaluation
In this section, we evaluate the performance of language models on our synthetic MCQs dataset using tailored prompts, where the instructions in the prompt are provided in English for each model.
5.1.1 Experiment Setup
We aim to improve the models’ capabilities by modifying the given prompt. Different parts of the prompt can be optimized according to a given metric, and in this evaluation, we started with optimizing the instruction and few-shot examples to determine which method is more effective. Unfortunately, instruction optimization yielded no significant performance gain. On the other hand, few-shot optimization boosted the performance of many models. Hence, we decided to focus on few-shot optimization. ArabicMMLU
is a benchmark to assess the capabilities of models, similar to MMLU
benchmark, but with localized data in Arabic. A subset relevant to the legal domain of ArabicMMLU
was sampled resulting in a total size of 524 questions after filtering questions that require context. Out of those 524 samples, 314 and 210 are law and political science, respectively. We optimize the prompts on this subset to use it for evaluating our generated MCQs.
DSPy Khattab et al. (2024) is a Language Model (LM) programming framework to optimize LM prompts and weights automatically by recompiling the entire pipeline to optimize it on a specific task. We relied on this framework for prompt optimization. Initially, all of the models were given a zero-shot prompt with an English answer instruction and the input-output format. Then, this zero-shot prompt is optimized for each model to achieve a higher performance by augmenting it with either plain few-shot examples or few-shot with reasoning demonstrations using CoT. Teacher and student models were used to create few-shot examples with CoT demonstrations, where the teacher is either a clone of the student or another model. Figure 5 demonstrates DSPy’s bootstrapped few-shot optimization.
![Refer to caption](extracted/5790735/latex/bootstrap2.png)
Original | Few-shot | Few-shot (GPT-4 Teacher) | CoT Few-shot | CoT Few-shot (GPT-4 Teacher) | |
GPT-4o | 76.80% | 79.10% | - | 76.50% | - |
Command R | 68.10% | 71.40% | 72.30% | 71.00% | 70.09% |
Command R+ | 68.10% | 71.40% | 72.30% | 73.10% | 69.86% |
Llama3 (8B) | 68.20% | 71.20% | 72.30% | 73.60% | 70.45% |
Llama3 (70B) | 68.17% | 71.47% | 72.34% | 71.93% | 70.53% |
5.1.2 MCQ Results Analysis
The MCQ dataset contains 10,583 questions, with 5,544 generated by GPT-4
and 5,039 by Claude-3-opus
. We evaluated several LMs on this dataset, including Command R
, Command R Plus
, and Llama3 (8B and 70B)
. However, GPT-4o
was tested only on Claude-3-opus
subset to mitigate potential bias towards its own generated questions Panickssery et al. (2024). Our evaluation metric assesses LM performance by comparing the selected answer with the correct one for each question. For further ArabicMMLU
detailed results with optimized prompts, refer to appendix G.
Table 1 summarizes the performance of LMs on our generated MCQs using prompts optimized with DSPy on ArabicMMLU
’s legal subsets. Interestingly, many of the optimized few-shot prompts shared identical examples, suggesting that certain examples play a more significant role in improving LMs’ performance than others. In addition, few-shot examples coupled with CoT reasoning boosted the capabilities of the models. For further testing, we employed GPT-4
as a teacher model for smaller LMs in both plain and CoT few-shot prompting. Surprisingly, these smaller LMs demonstrated greater performance in few-shot CoT when the teacher was a clone of themselves, rather than the more advanced GPT-4
. This unexpected result suggests that LMs may have a better grasp of their own reasoning.
We observed that the choice of language plays a crucial role in the reasoning abilities of smaller models. In many cases, these LMs generated answers without providing accompanying reasoning. Figure 6 shows the differences in reasoning language for Command R Plus
, revealing a degradation of the model’s reasoning capability when the language choice is Arabic.
![Refer to caption](extracted/5790735/latex/assets/cot_example.png)
Table 1 shows that GPT-4o
demonstrated superior performance across all prompting methods, achieving 79.10% accuracy with few-shot prompting. Llama3 (8B)
achieved a 5% increase using few-shot prompting with CoT reasoning. Similarly, the other LMs, except Command R
, obtained better results when the prompt included their own CoT reasoning. Our findings reveal potential for improving LMs’ question-answering capabilities through task-specific optimized prompts.
5.2 Open ended QA Evaluation
5.2.1 Experiment Setup
In open-ended QA—contrary to closed-set QA where the possible answers are fixed—the answer can be expressed in natural language, which is harder to evaluate since it relies on semantics and meaning. We observe that English prompts for the exam-taker LLM and the judge-LLM outperform Arabic prompts, so we use those English prompts for the instructions.
Traditional evaluation of QA models uses metrics like Exact Match (EM), F1-score, and top-n-accuracy, focusing on lexical matching. These metrics often miss semantically similar answers that use different words.
We use use an LLM-as-a-judge Huang et al. (2024), Kim et al. (2023), Zheng et al. (2023) to rate the answer similarity on a scale from 0 to 5 given the generated answer and the reference ground truth answer.
In our case, GPT-4
is the judge, and we refer to the output score as the answer similarity metric, see figure LABEL:fig:llm-as-a-judge.png for details. We notice that the judge LLMs give a lower score to answers that are in a different language than that of the reference answer even when the content is correct. To mitigate this, we prompt the models to output answers in the same language as the reference answer (which is Arabic in our dataset).
We run the evaluation for each LLM with two cases: one where it is given the question and the needed context, and one where it is given only the question. We also add a "golden model" that always answers the perfect ground truth answer, just so that we can compare against the upper bound of what the judge-LLM is going to score.
5.2.2 QA Results
We run our experiments on the 79 NajizQA pairs that have been filtered and verified by legal experts.
![Refer to caption](extracted/5790735/latex/QA_results_newllms.png)
From the original 1358 QA pairs, we chose the filtered 79 QA + context sets. We run them with and without context and show result in Figure 7. We notice that all models perform significantly better when the context is provided, and on closer inspection (Figure 11), we can see that the answer can easily be extracted from the context, making the context a giveaway of the answer which only measures the model’s ability to recall and extract information and not legal reasoning.
5.3 Arabic LegalBench Evaluation
5.3.1 Experiments Setup
In total, we carried out 96 unique experiments with different prompting techniques to assess the performance of the models in legal reasoning. All prompts were in English, as this yielded better model performance. Appendix I provides prompt examples for each technique used.
For the Contract QA and Consumer Contract QA datasets, we utilized the entire testing data in our experiments. However, for the larger datasets, we selected a representative sample. For each technique and dataset, we created tailored prompts. The training examples in one-shot and few-shot learning were fixed across all models for each specific dataset. We then benchmarked the performance of all models using the following four approaches: zero-shot learning with simple prompts where models are asked straightforward questions without extensive instructions; zero-shot learning with detailed instructions in the prompt; one-shot learning; and few shot learning.
5.3.2 Results Analysis
A comprehensive overview of model performance across all tasks and learning techniques is presented in Table 6 in Appendix H. This table provides a summary of F1 scores for each model and task combination, offering a comparative analysis of the various approaches evaluated in this study.
The consumer contract QA task can assess the models’ ability to answer questions based on a long context, in this case consumer contracts. GPT-4
with one-shot learning, and Llama3 (70B)
, with a zero-shot basic prompt, achieved the highest F1 score of 90%. This suggests that both models can extract relevant information from consumer contracts and provide answers, even when training examples are not available or limited.
We observed that most of the models performed well on the Contract QA task, which assessed the models capabilities to answer questions related to contracts. Command R Plus
, using few-shot learning, achieved the highest F1 score of 99%. This high score indicates that the model can accurately understand and respond to questions about contracts when provided with a small number of training examples. However, this task proved to be the least challenging among the four tasks.
On the other hand,the privacy policy entailment task proved to be the most challenging for the LLMs across all techniques, highlighting the complexity of this task. Command R Plus
, using one-shot learning, achieved the best F1 score of 66%. This result suggests that while the models struggle with this task, Command R Plus
is more capable of understanding of privacy policies when given a single training example.
In the final task, Privacy Policy QA, which evaluated the models’ ability to answer questions based on privacy policies, GPT-4
with one-shot learning achieved the highest performance. This result demonstrates GPT-4
’s strong capability in extracting relevant information from privacy policies and providing accurate answers when given a single training example.
Overall, one-shot learning achieved the best results for most of the models across the various tasks. This finding suggests that providing a single example can significantly improve the models’ performance in understanding and responding to questions related to legal documents such as consumer contracts and privacy policies.
![Refer to caption](extracted/5790735/latex/assets/LBData.png)
6 Limitations
The ArabLegalEval
benchmark currently relies heavily on Saudi Arabian legal documents, with some tasks translated from universal benchmarks. Including documents from more Arabic-speaking countries would improve geographic representation. Our study did not evaluate all models, which limits generalizability; future work should include a broader range of models. Limited access to legal experts affected validation depth; involving more experts would improve quality control. The dataset lacks granular categorization, such as task-specific prior knowledge, document origin, and AI-generated content labels. Adding more granular metadata and task categories would aid nuanced model training and evaluation.
7 Conclusions and Future Work
We are developing an Arabic benchmark to evaluate LLMs’ legal reasoning, using Saudi regulations and translated LegalBench problems. Future plans include adding more KSA regulatory documents, court cases, and judicial decisions to enhance this benchmark and promote advancements in Arabic legal AI.
Acknowledgments
We would like to express our sincere gratitude to Yusra Alonaizan for her contributions in envisioning this project, and dedicated team of six members—Lama Alsaif, Alanoud Alangari, Alaa Bazaid, and Faisal Alessa for gathering and curating the data sources. Additionally, we extend our thanks to the legal team at THIQAH for their valuable time in reviewing the labeled data, we are grateful to Norah AlHussein, an external legal expert, for her meticulous double-review of the labeled data and the translated LegalBench dataset, as well as for her clarifications and support in explaining the legal data.
References
- Abdelali et al. (2024) Ahmed Abdelali, Hamdy Mubarak, Shammur Absar Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Yousseif Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with Large Language Models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, Malta. Association for Computational Linguistics.
- Alghamdi et al. (2023) Emad A. Alghamdi, Jezia Zakraoui, and Fares A. Abanmy. 2023. Domain adaptation for arabic machine translation: The case of financial texts. Preprint, arXiv:2309.12863.
- Alghamdi et al. (2022) Reem Alghamdi, Zhenwen Liang, and Xiangliang Zhang. 2022. ArMATH: a dataset for solving Arabic math word problems. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 351–362, Marseille, France. European Language Resources Association.
- Ammar et al. (2024) Adel Ammar, Anis Koubaa, Bilel Benjdira, Omer Nacar, and Serry Sibaee. 2024. Prediction of arabic legal rulings using large language models. Electronics, 13(4):764.
- Anthropic (2024) Anthropic. 2024. Claude 3 haiku: Our fastest model yet. https://www.anthropic.com/news/claude-3-haiku. Accessed: 2024-06-26.
- Chiang and yi Lee (2023) Cheng-Han Chiang and Hung yi Lee. 2023. Can large language models be an alternative to human evaluations? Preprint, arXiv:2305.01937.
- Cohere (2024a) Cohere. 2024a. Command r. https://docs.cohere.com/docs/command-r. Accessed: 2024-06-26.
- Cohere (2024b) Cohere. 2024b. Command r+. https://docs.cohere.com/docs/command-r-plus. Accessed: 2024-06-26.
- Dalvi et al. (2024) Fahim Dalvi, Maram Hasanain, Sabri Boughorbel, Basel Mousi, Samir Abdaljalil, Nizi Nazar, Ahmed Abdelali, Shammur Absar Chowdhury, Hamdy Mubarak, Ahmed Ali, Majd Hawasly, Nadir Durrani, and Firoj Alam. 2024. LLMeBench: A flexible framework for accelerating llms benchmarking.
- Fei et al. (2023) Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. LawBench: Benchmarking Legal Knowledge of Large Language Models. arXiv.
- Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023. A framework for few-shot language model evaluation.
- Google (n.d.) Google. n.d. Google cloud translation api.
- Goth (2023) Gregory Goth. 2023. Why are lawyers afraid of AI? Commun. ACM, https://doi.org/10.1145/3631935, 67(1):14–16.
- Guha et al. (2023) Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li. 2023. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. arXiv.
- Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Huang et al. (2024) Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. 2024. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task-specific classifiers.
- Jin et al. (2023) Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, and Srijan Kumar. 2023. Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries. arXiv.
- Katz et al. (2024) Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(20230254).
- Khattab et al. (2024) Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. In 12th International Conference on Learning Representations, ICLR 2024, Messe Wien Exhibition and Congress Center, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Kim et al. (2023) Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2023. Prometheus: Inducing fine-grained evaluation capability in language models.
- Kolt (2022) Noam Kolt. 2022. Predicting consumer contracts. Berkeley Tech. LJ, 37:71.
- Koto et al. (2024) Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in arabic. Preprint, arXiv:2402.12840.
- Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315–7330, Online. Association for Computational Linguistics.
- LLaMA (2024) Meta LLaMA. 2024. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/. Accessed: 2024-06-26.
- Maree et al. (2024) Mohammed Maree, Rabee Al-Qasem, and Banan Tantour. 2024. Transforming legal text interactions: leveraging natural language processing and large language models for legal support in palestinian cooperatives. International Journal of Information Technology, 16(1):551–558.
- Microsoft (n.d.) Microsoft. n.d. Azure translation services.
- Mukherjee et al. (2021) Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christopher Meek, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021. Few-shot learning evaluation in natural language understanding. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
- Nay et al. (2023) John J. Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi, and Jungo Kasai. 2023. Large language models as tax attorneys: A case study in legal capabilities emergence. arXiv, https://arxiv.org/abs/2306.07075.
- OpenAI (2024a) OpenAI. 2024a. Gpt-4 technical report. Preprint, arXiv:2303.08774.
- OpenAI (2024b) OpenAI. 2024b. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2024-06-26.
- OpenAI (2024c) OpenAI. 2024c. New embedding models and api updates. https://openai.com/index/new-embedding-models-and-api-updates/. Accessed: 2024-07-02.
- Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc.
- Panickssery et al. (2024) Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076.
- Perlman (2022) Andrew Perlman. 2022. The implications of chatgpt for legal services and society. Suffolk University Law School Research Paper, (22-14). Available at SSRN: http://dx.doi.org/10.2139/ssrn.4294197.
- Risch et al. (2021) Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. 2021. Semantic answer similarity for evaluating question answering models. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 149–157, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Sengupta et al. (2023) Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Alham Fikri Aji, Zhengzhong Liu, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models. arXiv, https://arxiv.org/abs/2308.16149.
- Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language Models are Multilingual Chain-of-Thought Reasoners. ICLR.
- Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS-MT — Building open translation services for the World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal.
- Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
- Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.
- Ziemski et al. (2016) Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).
Appendix A LMs descriptions
Model | Language | # parameters |
GPT 4o | Multilingual | Undisclosed |
GPT 4 (0125-preview) | Multilingual | Undisclosed |
GPT 3.5 turbo 16k (0613) | Multilingual | Undisclosed |
meta-llama/Meta-Llama-3-8B-Instruct | Multilingual | 8B |
meta-llama/Meta-Llama-3-70B-Instruct | Multilingual | 70B |
CohereForAI/c4ai-command-r-v01 | Multilingual | 35B |
CohereForAI/c4ai-command-r-plus | Multilingual | 104B |
CohereForAI/aya-101 | Multilingual | 13B |
Claude 3 opus (20240229) | Multilingual | 137B |
Appendix B Examples of Data Sources
![Refer to caption](extracted/5790735/latex/BOE-data-example.png)
![Refer to caption](extracted/5790735/latex/MOJ-data-example.png)
![Refer to caption](extracted/5790735/latex/assets/najizqa_table.png)
Appendix C MCQs Generation
3.1 MCQs Evaluation Criteria
![Refer to caption](extracted/5790735/latex/assets/mcq_eval_prompt.png)
3.2 MCQs Example
![Refer to caption](extracted/5790735/latex/assets/mcq_example.png)
3.3 Prompts
![Refer to caption](extracted/5790735/latex/assets/q_gen_prompt.png)
![Refer to caption](extracted/5790735/latex/assets/q_answer_prompt.png)
![Refer to caption](extracted/5790735/latex/assets/mcq_in_context_gen.png)
![Refer to caption](extracted/5790735/latex/assets/qa_to_mcq_cot.png)
Appendix D Translation
4.1 ROUGE scores for UN English-Arabic translation
Model | Rouge-1 | Rouge-2 | Rouge-L |
Azure Translation Services | 0.446 | 0.242 | 0.435 |
Google Translation API | 0.408 | 0.207 | 0.399 |
Opus MT | 0.519 | 0.308 | 0.509 |
GPT 4 Turbo | 0.327 | 0.138 | 0.316 |
4.2 GPT-4 Translation Evaluation Prompt
![Refer to caption](extracted/5790735/latex/assets/GPT4_evalPrompt.png)
4.3 GPT-4 Translation Result Evaluation
![Refer to caption](extracted/5790735/latex/assets/GPT4_TranslationEval.png)
4.4 Sample of the Translated Data
![Refer to caption](extracted/5790735/latex/assets/TranslatedSample.png)
Appendix E QA
![Refer to caption](extracted/5790735/latex/assets/QA_model_prompt.png)
![Refer to caption](extracted/5790735/latex/assets/llm_judge_prompt.png)
![Refer to caption](extracted/5790735/latex/llm-judge.png)
Appendix F Data statistics
![Refer to caption](extracted/5790735/latex/assets/data_sources_chart.png)
Source | Category | Value |
LegalBench | Privacy Policy Entailment | 4385 |
Privacy Policy QA | 10931 | |
Contracts QA | 88 | |
Consumer Contracts QA | 400 | |
BOE | Rules Count | 448 |
Rules Count - Subject | 14134 | |
MOJ | Regulations | 67 |
Regulations - Subjects | 5720 | |
Circular | 388 | |
FAQ Najiz | FAQ Najiz | 492 |
ArabicMMLU | ArabicMMLU | 800 |
Appendix G MCQs Evaluation
7.1 DSPy Signatures
![Refer to caption](extracted/5790735/latex/dspy_few.png)
![Refer to caption](extracted/5790735/latex/dspy_cot.png)
7.2 ArabicMMLU Results with DSPy optimization
Table 5 displays the performance of models on ArabicMMLU’s law and political science subsets with optimized prompts. The prompts were optimized with 15 examples as labeled inputs from law and political science subsets, respectively. Therefore, the accuracies were computed using the remaining 299 and 195 samples from the respective law and political science sets.
Original | Few-shot | Few-shot (GPT-4 Teacher) | CoT Few-shot | CoT Few-shot (GPT-4 Teacher) | ||
GPT-4 | Law | 73.60% | 74.90% | - | 77.30% | - |
Political Science | 70.80% | 75.90% | - | 70.80% | - | |
GPT-4o | Law | 68.20% | 82.90% | - | 81.90% | - |
Political Science | 71.30% | 74.90% | - | 73.80% | - | |
Command R | Law | 56.90% | 61.50% | 65.20% | 69.20% | 69.90% |
Political Science | 65.10% | 67.20% | 67.70% | 72.80% | 66.70% | |
Command R+ | Law | 56.50% | 61.20% | 65.60% | 72.20% | 68.20% |
Political Science | 65.10% | 66.70% | 67.70% | 72.30% | 66.20% | |
Llama3 8B | Law | 56.90% | 61.50% | 65.90% | 73.20% | 69.20% |
Political Science | 65.10% | 67.20% | 67.20% | 73.30% | 65.10% | |
Llama3 70B | Law | 56.60% | 61.50% | 65.90% | 73.90% | 67.60% |
Political Science | 65.10% | 67.20% | 67.70% | 70.30% | 66.70% | |
Aya101 | Law | 45.80% | 42.80% | 19.10% | 21.70% | 47.10% |
Political Science | 47.20% | 50.80% | 51.30% | 51.80% | 26.10% |
Appendix H Details for Arabic LegalBench Experiments Results
Technique | Model | consumer_contract_qa | contract_qa | privacy_policy_entailment | privacy_policy_qa |
few_shot | Cohere-command-r | 43% | 89% | 61% | 64% |
Cohere-command-r-plus | 35% | 99% | 41% | 68% | |
gpt-35-turbo-16k-2023-03-15-preview | 51% | 89% | 50% | 66% | |
gpt4-2024-02-15-preview | 62% | 97% | 62% | 67% | |
Meta-Llama-3-70B-Instruct | 89% | 62% | 49% | 40% | |
Meta-Llama-3-8B-Instruct | 35% | 1% | 0% | 14% | |
One_Shot | Cohere-command-r | 81% | 95% | 61% | 66% |
Cohere-command-r-plus | 89% | 94% | 66% | 68% | |
gpt-35-turbo-16k-2023-03-15-preview | 73% | 92% | 47% | 64% | |
gpt4-2024-02-15-preview | 90% | 96% | 60% | 74% | |
Meta-Llama-3-70B-Instruct | 86% | 96% | 50% | 60% | |
Meta-Llama-3-8B-Instruct | 65% | 56% | 4% | 62% | |
Zero_shot_basic | Cohere-command-r | 88% | 94% | 44% | 65% |
Cohere-command-r-plus | 59% | 50% | 54% | 43% | |
gpt-35-turbo-16k-2023-03-15-preview | 82% | 50% | 36% | 30% | |
gpt4-2024-02-15-preview | 31% | 92% | 37% | 60% | |
Meta-Llama-3-70B-Instruct | 90% | 90% | 38% | 35% | |
Meta-Llama-3-8B-Instruct | 75% | 26% | 12% | 35% | |
Zero_shot_detailed | Cohere-command-r | 55% | 62% | 44% | 63% |
Cohere-command-r-plus | 43% | 29% | 52% | 65% | |
gpt-35-turbo-16k-2023-03-15-preview | 77% | 59% | 35% | 62% | |
gpt4-2024-02-15-preview | 57% | 92% | 43% | 55% | |
Meta-Llama-3-70B-Instruct | 89% | 59% | 44% | 62% | |
Meta-Llama-3-8B-Instruct | 45% | 18% | 13% | 23% |
Appendix I Araic LegalBench Prompts
![Refer to caption](extracted/5790735/latex/assets/ZeroShot_Basic.png)
![Refer to caption](extracted/5790735/latex/assets/Zeroshot_Detailed.png)
![Refer to caption](extracted/5790735/latex/assets/OneShot.png)
![Refer to caption](extracted/5790735/latex/assets/FewShots.png)
Appendix J Dataset Quantitative Analysis
10.1 MCQs
![Refer to caption](extracted/5790735/latex/assets/options_freq.png)
![Refer to caption](extracted/5790735/latex/assets/correct_answer_freq.png)
Min Length | Max Length | Avg Length | |
Context | 4 | 1497 | 46.0 |
Question | 2 | 48 | 12.0 |
![Refer to caption](extracted/5790735/latex/assets/contexg_words_freq.png)
![Refer to caption](extracted/5790735/latex/assets/context_questions_length_freq.png)
10.2 QA
![Refer to caption](extracted/5790735/latex/qa_context_freq.png)
![Refer to caption](extracted/5790735/latex/qa_length_dist.png)
Min Length | Max Length | Avg Length | |
Context | 11 | 284 | 64.0 |
Question | 4 | 35 | 11.0 |
10.3 Arabic LegalBench
![Refer to caption](extracted/5790735/latex/assets/QuantArabicLegalBench.png)
10.4 Experts Performance on the Tasks
To establish a human performance baseline and provide a meaningful comparison for the LLMs’ performance, a sample of the data was presented to legal experts for evaluation. These experts, with their specialized knowledge and experience, were tasked with solving the same problems that were presented to the LLMs. Table 9 summarize the results of the experts assessment.
MCQs | NajezQA | Arabic LegalBench | |
F1 Score | - | - | 0.73 |
Accuracy | 0.9 | 0.65 | 0.8 |