African Languages Lag in LLM Benchmarks, Highlighting Critical Gaps

[{"type":"prose","html":"

LLM CAPABILITIES
High-resource languages dominate AI advancements

\n

Large language models (LLMs) have demonstrated impressive capabilities, yet these advancements are disproportionately concentrated in a few high-resource languages, predominantly English. This linguistic bias limits the global reach and equitable application of AI technologies. For many of the world's languages, particularly those in Africa, the development and evaluation of LLMs lag considerably behind.

\n

The scarcity of appropriate and comprehensive benchmarks outside of high-resource languages means that African languages are often relegated to basic text classification tasks. This limited scope fails to capture the full potential or identify the specific weaknesses of LLMs when applied to the linguistic diversity of the continent.

"},{"type":"prose","html":"

AFRICAN LANGUAGE BENCHMARKS
Tools for assessing LLM performance across the continent

\n

Addressing the deficit in evaluation tools, new benchmarks are emerging to provide a more nuanced understanding of LLM performance in African languages. IrokoBench, for instance, is a human-translated benchmark dataset encompassing 17 typologically diverse low-resource African languages. It covers three critical tasks: natural language inference (AfriXNLI), mathematical reasoning (AfriMGSM), and multi-choice knowledge-based question answering (AfriMMLU).

\n

Similarly, the Uhura benchmark focuses on scientific question answering and truthfulness in six African languages. Uhura-ARC-Easy, composed of multiple-choice science questions, and Uhura-TruthfulQA, a safety benchmark testing model truthfulness across health, law, finance, and politics, highlight the challenges in creating benchmarks with highly technical content for these languages. These efforts are essential for moving beyond basic evaluations and assessing LLMs on more complex cognitive tasks.

"},{"type":"prose","html":"

PERFORMANCE DISPARITIES
Gaps between languages, models, and translation strategies

\n

Evaluations using these new benchmarks reveal significant performance gaps. A consistent finding is the substantial difference between high-resource languages, such as English and French, and low-resource African languages. This disparity is evident across various tasks, indicating that current LLMs are not equally adept at processing and generating content in these diverse linguistic contexts.

\n

Furthermore, a notable performance gap exists between open and proprietary models. The highest-performing open model, Gemma 2 27B, achieved only 63% of the performance of the best-performing proprietary model, GPT-4o, on the IrokoBench dataset. This suggests that while open models are advancing, proprietary solutions currently hold a lead in capabilities relevant to African languages.

\n

Machine translating test sets from African languages into English before evaluation offers a partial mitigation strategy. This approach helped to close the performance gap for larger, English-centric models like Gemma 2 27B and LLaMa 3.1 70B. However, it also underscores a reliance on English as a lingua franca for LLM evaluation, potentially masking nuances specific to African languages.

"},{"type":"prose","html":"

SPECIFIC TASK CHALLENGES
Truthfulness and reasoning present particular hurdles

\n

LLMs exhibit particular struggles with knowledge-intensive tasks and factual accuracy when applied to low-resource African languages. The Uhura benchmark, for example, highlights that all evaluated models perform better in English than in the African languages tested. This indicates a fundamental challenge in the models' ability to accurately process and respond to information in these languages.

\n

Specifically, LLMs are more prone to generating false claims in low-resource African languages, particularly on topics related to science, health, law, finance, and politics. This lack of truthfulness is a critical concern for the safe and reliable deployment of AI in real-world applications across the continent. The necessity for continuous improvement of multilingual LLM capabilities in low-resource settings is therefore strongly underscored.

"},{"type":"prose","html":"

BROADER EVALUATION EFFORTS
Machine translation and sentiment analysis add context

\n

Beyond comprehensive benchmarks, other research areas are contributing to the evaluation of LLMs for African languages. Efforts in machine translation evaluation, using models like NLLB-200 and M2M-100, explore benchmark quality and metrics such as BLEU, ChrF, BERTScore, and COMET. This work reveals discrepancies in reference datasets; for instance, in isiZulu, 22.23% of sentences differed between original and corrected FLORES references, impacting translation score reliability.

\n

Sentiment analysis research, while historically fragmented due to data limitations, has seen progress with initiatives like the AfriSenti-SemEval Shared Task 12, which provided annotated sentiment data for 14 African languages. Benchmarking transformer models across 12 languages showed that models explicitly developed for African languages outperform general multilingual models. However, no single model-fits-all solution exists, and for some languages with smaller sample sizes, larger multilingual models may offer better performance.

\n

Further exploration into safety evaluation, such as the TUKABENCH benchmark for jailbreaking, indicates that prompting in African languages can reduce refusal rates compared to English, with culturally adapted prompts yielding the least refusal. However, this also surfaces structural limitations like model comprehension failures and reduced LLM-as-a-judge reliability in low-resource settings.

"},{"type":"prose","html":"

REALITY VERSUS THE PITCH
Bridging the divide requires dedicated resources

\n

The verified claims highlight a critical gap: LLM capabilities, while remarkable, remain largely confined to high-resource languages, leaving African languages significantly underserved. The lack of appropriate benchmarks has historically limited evaluations to basic tasks, failing to capture the complexity of these languages.

\n

New benchmarks like IrokoBench and Uhura are actively working to rectify this by providing human-translated datasets for diverse tasks, including natural language inference, mathematical reasoning, scientific question answering, and truthfulness. These efforts are crucial for a more accurate assessment of LLMs in African contexts. The evidence clearly indicates that all models perform better in English than in African languages, and a significant performance gap exists between open and proprietary models, with the latter generally outperforming open-source alternatives.

"},{"type":"prose","html":"

WHAT WOULD CHANGE THE PICTURE
Beyond evaluation: fostering true linguistic parity

\n

The current landscape, characterized by performance gaps and reliance on English-centric evaluations, would change if significant investments were made in developing foundational LLMs trained natively on diverse African language corpora. Such models, alongside robust, contextually relevant benchmarks that go beyond translation, would provide a more accurate measure of true capabilities.

\n

Furthermore, increased availability of computational resources within Africa and a concerted effort to build local AI research capacity would accelerate progress. The open-sourcing of benchmarks and platforms, like the Uhura Benchmark and Platform, is a positive step, but sustained, collaborative efforts are needed to ensure that LLM development truly serves the linguistic and informational needs of the entire continent.

"}]