Afri-MCQA: Multimodal Cultural Question Answering for African Languages | AGIDaily
AGI Daily
资讯日报词条论文人物新创
返回论文列表

暂无章节

返回论文列表
目录

Afri-MCQA: Multimodal Cultural Question Answering for African Languages

收录于 ACL 2026 Best Social Impact Paper;Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers);DOI 10.18653/v1/2026.acl-long.1869被引 7 次
打开论文来源

Afri-MCQA: Multimodal Cultural Question Answering for African Languages

Afri-MCQA:面向非洲语言的多模态文化问答

Atnafu Lambebo Tonja1,∗,†^{1,\ast,\dagger}1,∗,†, Srija Anand2,∗,†^{2,\ast,\dagger}2,∗,†, Emilio Villa-Cueva3,∗^{3,*}3,∗, Israel Abebe Azime4^44, Jesujoba O. Alabi4^44, Muhidin A. Mohamed5^55, Debela Desalegn Yadeta6^66, Negasi Haile Abadi7^77, Abigail Oppong8^88, Nnaemeka Casmir Obiefuna9^99, Idris Abdulmumin10^{10}10, Naome A. Etori11^{11}11, Eric Peter Wairagala12^{12}12, Kanda Patrick Tshinu13^{13}13, Imanigirimbabazi Emmanuel14^{14}14, Gabofetswe Malema15^{15}15, Alham Fikri Aji3^33, David Ifeoluwa Adelani16^{16}16, Thamar Solorio3^33

1^11UCL, 2^22AI4Bharat, Indian Institute of Technology, Madras, 3^33 MBZUAI, 4^44Saarland University, 5^55Aston University, 6^66Addis Ababa University, 7^77Lesan AI, 8^88Independent, 9^99Friedrich-Alexander University, 10^{10}10University of Pretoria, 11^{11}11University of Minneosta -Twin Cities, 12^{12}12Lelapa AI,13^{13}13Tshwane University of Technology, 14^{14}14Digital Umuganda, 15^{15}15University of Botswana, 16^{16}16Mila, McGill University & Canada CIFAR AI Chair

∗^*∗ Equal contribution
†^\dagger† Work done while the authors were at MBZUAI

∗^*∗ 同等贡献
†^\dagger† 作者在 MBZUAI 期间完成此项工作

Abstract

摘要

Africa is home to over one-third of the world's languages, yet remains underrepresented in AI research. We introduce Afri-MCQA, the first Multilingual Cultural Question-Answering benchmark covering 7.5k Q&A pairs across 15 African languages from 12 countries. The benchmark offers parallel English-African language Q&A pairs across text and speech modalities and was entirely created by native speakers. Benchmarking large language models (LLMs) on Afri-MCQA shows that open-weight models show poor performance across evaluated cultures, with near-zero accuracy on open-ended VQA when queried using native language or speech. To evaluate linguistic competence, we include control experiments meant to assess this specific aspect separate from cultural knowledge, and we observe significant performance gaps between native languages and English for both text and speech. These findings underscore the need for speech-first approaches, culturally grounded pretraining, and cross-lingual cultural transfer. We will release Afri-MCQA to support more inclusive multimodal AI development.

非洲拥有全球三分之一以上的语言,但在人工智能研究中的代表性仍然不足。我们提出 Afri-MCQA,这是首个多语种文化问答(Multilingual Cultural Question-Answering)基准,覆盖来自 12 个国家、15 种非洲语言的 7500 条问答对。该基准提供英语与非洲语言平行的文本和语音问答,并全部由母语者创建。在 Afri-MCQA 上评测大语言模型(LLM)后可以看到,开放权重模型在所评估的文化上表现较差;一旦用母语或语音做开放式视觉问答,准确率接近零。为了把语言能力从文化知识中单独拿出来评估,我们设置了控制实验,结果在文本和语音两种模态上,母语与英语之间都存在明显差距。这些发现说明,有必要采取语音优先的路径、融入文化内容的预训练,以及跨语言的文化迁移。我们将发布 Afri-MCQA,以支持更具包容性的多模态人工智能发展。

1 Introduction

1 引言

Africa is one of the most culturally diverse and rapidly growing regions in the world. It is home to more than one-third of the world's languages (Hammarström, 2018) and a population exceeding 1.3 billion, projected to surpass 2.5 billion by 2050 (Simane et al., 2025). African languages have linguistic features that are very different from many high-resource languages represented in LLMs, including rich morphology, use of noun classes, tonality, and serial verb constructions, among others (Nurse and Philippson, 2006; Adebara and Abdul-Mageed, 2022). Additionally, many African languages are primarily spoken, with literacy often occurring in a colonial or foreign language. This makes speech-based applications such as Multimodal large language models (MLLMs) particularly relevant for enabling access to technology.

非洲是世界上文化最多样、增长也最快的地区之一。这里拥有全球三分之一以上的语言(Hammarström, 2018),人口已超过 13 亿,并预计在 2050 年突破 25 亿(Simane et al., 2025)。非洲语言的许多特征与大语言模型中常见的高资源语言很不一样,包括丰富的形态变化、名词类、声调和连动结构等(Nurse and Philippson, 2006; Adebara and Abdul-Mageed, 2022)。此外,许多非洲语言主要以口语使用,书面读写往往发生在殖民语言或外语中。因此,多模态大语言模型(MLLM)这类能够处理语音的应用,对扩大技术可及性尤为重要。

Examples of Afri-MCQA datapoints with parallel Hausa and Amharic text and speech questions grounded in culturally relevant images

Figure 1: Examples of Afri-MCQA datapoints, containing parallel text and speech QA pairs grounded in culturally relevant images across English and native African languages.

图 1:Afri-MCQA 数据示例,在英语和非洲母语中提供基于文化相关图像的平行文本与语音问答对。

MLLMs perform well on tasks that require reasoning over visual, spoken, and textual inputs to follow user instructions (Liu et al., 2021a; Kamath et al., 2025; Abdin et al., 2024). However, these systems are developed primarily for high-resource, mostly Western-centric, or as Mihalcea et al. (2025) put it, they have a 'WEIRD' coverage. As a result, their knowledge and reasoning abilities tend to favor the languages and cultures represented in their training data (Mihalcea et al., 2025).

在需要结合视觉、语音和文本输入来执行用户指令的任务上,多模态大语言模型已经表现不错(Liu et al., 2021a; Kamath et al., 2025; Abdin et al., 2024)。不过,这些系统主要面向高资源、且大多以西方为中心的场景来开发;用 Mihalcea et al. (2025) 的话说,它们的覆盖面是“WEIRD”的。于是,模型的知识和推理能力往往偏向训练数据里出现过的语言与文化(Mihalcea et al., 2025)。

Although most evaluation datasets for low-resource languages are translated (NLLB Team, 2022; Adelani et al., 2024, 2025; Azime et al., 2024; Alabi et al., 2025), researchers have recently made promising progress in creating culturally relevant benchmarks for African languages across various NLP tasks (Adelani et al., 2023; Azime et al., 2025; Muhammad et al., 2025; Yu et al., 2025; Tonja et al., 2024b,a). However, these evaluations are designed for text-only tasks. In the language-vision space, recent benchmarks assess global cultural knowledge through question answering (Romero et al., 2024; Vayani et al., 2025a; Winata et al., 2025), yet their coverage of African languages and contexts remains limited.

尽管面向低资源语言的评测集大多是翻译而来(NLLB Team, 2022; Adelani et al., 2024, 2025; Azime et al., 2024; Alabi et al., 2025),研究者近来已在多种自然语言处理任务上,为非洲语言建立起更贴近文化的基准,并取得了可喜进展(Adelani et al., 2023; Azime et al., 2025; Muhammad et al., 2025; Yu et al., 2025; Tonja et al., 2024b,a)。不过,这些评测都是为纯文本任务设计的。在语言–视觉方向,最近的基准通过问答来衡量全球文化知识(Romero et al., 2024; Vayani et al., 2025a; Winata et al., 2025),但对非洲语言和场景的覆盖仍然有限。

Africa's rich cultural diversity demands AI systems that can effectively serve its communities. A critical step toward this goal is evaluating how well current MLLMs understand and reason about African cultural knowledge in multimodal settings. To enable this, we introduce Afri-MCQA, the first multilingual cultural VQA dataset supporting text and speech modalities (as shown in Figure 1). The dataset consists of ≈\approx≈7.5k Q&A pairs (in English and native languages) across 15 African languages from 12 countries. Our data collection involves native speakers as annotators, who reside in the countries where their respective languages are spoken. Each language covers ≈\approx≈500 image-grounded Q&A samples, with both text and spoken audio in the native language and English. We evaluate multiple MLLMs across various setups to answer the following research questions:

非洲丰富的文化多样性,要求人工智能系统真正服务当地社区。迈向这一目标的关键一步,是评估当前多模态大语言模型在多模态设定下,能在多大程度上理解并推理非洲文化知识。为此,我们提出 Afri-MCQA,这是首个同时支持文本和语音模态的多语种文化视觉问答数据集(见图 1)。数据包含来自 12 个国家、15 种非洲语言的约 7500 条问答对(英语与母语)。采集工作由居住在相应语言使用国的母语者完成。每种语言约有 500 条以图像为根据的问答样本,并同时提供母语和英语的文本与口语录音。我们在多种设定下评测多个多模态大语言模型,以回答下列研究问题:

RQ1: How well do MLLMs understand African cultural contexts in visually-grounded QA?

RQ1:多模态大语言模型在以视觉为根据的问答中,对非洲文化语境的理解有多好?

RQ2: How does input modality (text vs. speech) affect performance?

RQ2:输入模态(文本与语音)会怎样影响表现?

RQ3: How does query language (native vs. English) affect performance, and do differences reflect language understanding or cultural knowledge gaps?

RQ3:提问语言(母语与英语)会怎样影响表现,其间的差异反映的是语言理解问题,还是文化知识缺口?

RQ4: How does task format (Multiple-Choice vs. Open QA) affect accuracy?

RQ4:任务形式(选择题与开放式问答)会怎样影响准确率?

Our contributions are: (1) We introduce Afri-MCQA, the first large-scale multilingual visual cultural QA benchmark for 15 African languages across 12 countries, with parallel text and speech QA created by native speakers. (2) We demonstrate that text-based multilingual ability does not transfer to speech understanding, emphasizing the need for more African language representation in multimodal training. (3) We release Afri-MCQA to advance multimodal research for Africa's diverse languages and cultures.1^11

我们的贡献是:(1)提出 Afri-MCQA,这是面向 12 个国家、15 种非洲语言的首个大规模多语种视觉文化问答基准,并包含由母语者撰写的平行文本与语音问答。(2)我们证明,基于文本的多语能力并不能迁移到语音理解上,从而强调多模态训练需要纳入更多非洲语言。(3)我们发布 Afri-MCQA,以推动面向非洲多样语言与文化的多模态研究。1^11

1^11https://huggingface.co/datasets/Atnafu/Afri-MCQA

2 Related work

2 相关工作

With the growing popularity of LLMs and MLLMs, cultural and multilingual multimodal evaluation has received greater attention. For instance, Liu et al. (2021b) tested models on verifying statements about pairs of culturally related images, but this was framed merely as a binary classification task. CVQA (Romero et al., 2024) and CulturalVQA (Nayak et al., 2024) explicitly target cultural knowledge through human-written questions, yet they are limited to a small set of languages per continent. ALM-bench (Vayani et al., 2025a) expands the language coverage, but relies heavily on LLM-generated questions and web-sourced images. Additionally, a common limitation across these benchmarks is that they query models only through text, overlooking the importance of speech, an important modality for communities where language is mostly spoken.

随着大语言模型和多模态大语言模型日益普及,文化和多语种的多模态评测也受到更多关注。例如,Liu et al. (2021b) 让模型判断成对文化相关图像上的陈述是否成立,但这一设定只是二分类。CVQA(Romero et al., 2024)和 CulturalVQA(Nayak et al., 2024)用人工撰写的问题直接考察文化知识,可是每个大洲只覆盖少数语言。ALM-bench(Vayani et al., 2025a)扩大了语言覆盖,却在很大程度上依赖大语言模型生成的问题和来自网络的图像。这些基准还有一个共同局限:它们只用文本向模型提问,忽略了语音的重要性——而对语言主要以口语使用的社区来说,语音是关键模态。

Previous work on African languages has focused mainly on text-based benchmarks. While some multilingual resources, such as GlobalMMLU (Singh et al., 2024), include a small subset of African languages, other region-specific benchmarks provide broader coverage. Examples of these include MasakhaNEWS (Adelani et al., 2023) that evaluates text classification, AfriQA (Ogundepo et al., 2023) for cross-lingual question answering, or larger suites such as AfroBench (Ojo et al., 2025) and IrokoBench (Adelani et al., 2025) that cover multiple text tasks. Across these efforts, a consistent finding is that models perform poorly on African languages, with open-weight models showing a significant gap to proprietary systems.

以往关于非洲语言的工作,主要集中在文本基准上。GlobalMMLU(Singh et al., 2024)这类多语资源只包含一小部分非洲语言,而一些面向区域的基准覆盖更宽。例如,MasakhaNEWS(Adelani et al., 2023)评估文本分类,AfriQA(Ogundepo et al., 2023)做跨语言问答,更大规模的套件如 AfroBench(Ojo et al., 2025)和 IrokoBench(Adelani et al., 2025)则覆盖多项文本任务。这些工作有一个一致的发现:模型在非洲语言上表现较差,开放权重模型与专有系统之间存在明显差距。

Despite this progress, all of these evaluations remain text-only, while visual knowledge and multimodal reasoning for African contexts are still largely untested. Existing multi-region multimodal datasets (Romero et al., 2024; Vayani et al., 2025b) include only a few African languages and do not adequately capture the region's diversity. Afri-MCQA addresses these gaps with three key contributions: (1) broader coverage with 15 languages across 12 countries, (2) inclusion of speech modality with parallel native and African-accented English audio, and (3) diagnostic control experiments that separate linguistic competence from cultural knowledge limitations.

尽管已有这些进展,上述评测仍全部限于文本;面向非洲场景的视觉知识和多模态推理,在很大程度上还没有被检验。现有的跨区域多模态数据集(Romero et al., 2024; Vayani et al., 2025b)只包含少数非洲语言,不足以反映该地区的多样性。Afri-MCQA 从三个方面弥补这些缺口:(1)覆盖更广,包含 12 个国家的 15 种语言;(2)纳入语音模态,并提供平行的母语录音和带非洲口音的英语录音;(3)通过诊断性控制实验,把语言能力与文化知识的局限分开来看。

3 Dataset

3 数据集

To create Afri-MCQA, we selected 15 widely spoken languages in sub-Saharan Africa (by number of speakers, according to Ethnologue2^22) across 12 countries, representing an aggregate speaker population of approximately 392.6 million (see Table 2). We hired native language speakers as annotators through Upwork.3^33 The selection criteria were based on (i) fluency in English, (ii) prior annotation or data collection experience, (iii) high project completion rate on Upwork, and (iv) residence in a country where the target language is spoken. After the selection process, we divided the annotation into two phases designed to ensure quality.

为构建 Afri-MCQA,我们依据 Ethnologue2^22 的使用人数,在撒哈拉以南非洲选取了 15 种使用较广的语言,覆盖 12 个国家,合计使用者约 3.926 亿(见表 2)。我们通过 Upwork 聘请母语者担任标注员。3^33 筛选标准包括:(i)英语流利;(ii)此前有标注或数据采集经验;(iii)在 Upwork 上的项目完成率高;(iv)居住在目标语言的使用国。完成筛选后,我们将标注分成两个阶段,以便保证质量。

Table 1: Data statistics for Afri-MCQA compared to existing VQA datasets that include African languages.

Datasets # African Lang # Countries QA categories # QA per langs Audio QA Parallel Data
CVQA (Romero et al., 2024) 5 4 10 200 × ✓
WC-VQA (Winata et al., 2025) 1 1 1 – × ✓
M5 (Schneider and Sitaram, 2024) 4 4 – – × ×
HaVQA (Parida et al., 2023) 1 1 – 6,200 × ✓
Afri-MCQA (ours) 15 12 10 500 ✓ ✓

表 1:Afri-MCQA 与已有包含非洲语言的视觉问答数据集的数据统计对照。

数据集 非洲语言数 国家数 问答类别数 每语言问答数 语音问答 平行数据
CVQA (Romero et al., 2024) 5 4 10 200 × ✓
WC-VQA (Winata et al., 2025) 1 1 1 – × ✓
M5 (Schneider and Sitaram, 2024) 4 4 – – × ×
HaVQA (Parida et al., 2023) 1 1 – 6,200 × ✓
Afri-MCQA (ours) 15 12 10 500 ✓ ✓

3.1 Dataset collection

3.1 数据采集

Guideline and Platform Preparation We followed the annotation guidelines of Romero et al. (2024), including image categories, question templates, and distractors, and extended them to include audio recording instructions (for both native language and English) and added a two-step review and verification process.4^44 Full annotation guidelines are provided in Appendix J.

指南与平台准备 我们沿用 Romero et al. (2024) 的标注指南,包括图像类别、问题模板和干扰项,并加以扩展,补充了母语和英语的录音说明,还加入了两步审核与核验流程。4^44 完整标注指南见附录 J。

Training and Screening Phase The first phase consisted of a small-scale pilot designed to train and screen annotators. We provided detailed guidelines and training to ensure annotators understood the task criteria and quality standards. After training, each annotator submitted 50 QA samples. We reviewed these submissions to verify alignment with the guidelines. Based on this review, we provided detailed feedback and, when necessary, scheduled follow-up meetings to clarify issues. Only annotators whose submissions met our quality criteria and successfully incorporated feedback were selected for the next phase. Approved samples from this screening phase were included in the final dataset.

培训与筛选阶段 第一阶段是小规模试点,用来培训并筛选标注员。我们提供了详细指南和培训,确保标注员理解任务标准和质量要求。培训结束后,每位标注员提交 50 条问答样本。我们审阅这些提交,核对其是否符合指南。根据审阅结果,我们给出详细反馈,必要时再安排跟进会议澄清问题。只有提交达到质量标准、并能落实反馈的标注员,才会进入下一阶段。筛选阶段获准的样本,也收入了最终数据集。

Main Annotation Phase In this phase, the remaining 450 items per language were collected by annotators selected in the first phase. To ensure quality across the large volume of submissions, we involved language coordinators, experienced native speakers with strong linguistic and cultural knowledge for each language. Coordinators reviewed all submissions for linguistic accuracy, cultural appropriateness, adherence to guidelines, and audio quality. Review guidelines are provided in Appendix O. When issues arose, coordinators discussed them directly with annotators to resolve them. Unresolved disagreements were then raised with the project team for a final decision. Language coordinators are co-authors of this paper and played a major role in the quality assurance process. After their approval, the project team conducted a final review to ensure overall data quality.

主标注阶段 在这一阶段,每种语言余下的 450 条由第一阶段入选的标注员采集。为了在大量提交中保证质量,我们为每种语言安排了语言协调员,他们是熟悉该语言和相关文化的资深母语者。协调员审阅全部提交,检查语言是否准确、文化是否得当、是否符合指南,以及音频质量。审核指南见附录 O。出现问题时,协调员直接与标注员讨论并解决;仍无法达成一致的,再提交项目组做最终裁定。语言协调员是本文的共同作者,在质量保证中发挥了重要作用。他们批准之后,项目组还会做一次终审,以确保整体数据质量。

2^22https://www.ethnologue.com/insights/ethnologue200/

3^33https://www.upwork.com/

4^44This study received approval from the Institutional Review Board (IRB) at MBZUAI prior to data collection.

4^44本研究在数据采集前已获得 MBZUAI 机构审查委员会(IRB)批准。

3.2 Dataset Composition

3.2 数据构成

Each data point in Afri-MCQA includes an image and a set of carefully constructed multiple-choice questions in both text and speech modalities, both in English and the native language. Below, we describe each of these components.

Afri-MCQA 中的每条数据都包含一张图像,以及一组仔细编写的选择题;这些问题同时提供文本和语音两种模态,并分别有英语和母语版本。下面分别说明这些组成部分。

Image / Category Selection To build the image set for Afri-MCQA, we encouraged annotators to contribute their own images whenever possible. When self-sourcing was not feasible, we permitted the use of open-license images from websites we provided (see Appendix J for list of websites). All collected images were categorized into the 10 classes defined by Romero et al. (2024). Figure 2 shows the distributions of images per category (See Appendix A for category distributions across languages).

图像 / 类别选择 为构建 Afri-MCQA 的图像集,我们鼓励标注员尽可能提供自己拍摄的照片。在无法自行取材时,允许使用我们指定网站上的开放许可图像(网站清单见附录 J)。全部采集到的图像都归入 Romero et al. (2024) 定义的 10 个类别。图 2 给出了各类图像的分布(各语言的类别分布见附录 A)。

Donut chart of image category distributions in the Afri-MCQA dataset

Figure 2: Image categories in our dataset and their distributions.

图 2:本数据集中的图像类别及其分布。

Question & Answer Generation For each image, annotators wrote up to 3 multiple-choice QA triplets (question + 1 correct answer + 3 distractors) in both their native language and English. To ensure the benchmark remains challenging, we instructed annotators to design complex questions that require reasoning to answer correctly (See annotation guidelines in Appendix J).

问题与答案撰写 对每张图像,标注员最多撰写 3 组选择题问答(问题 + 1 个正确答案 + 3 个干扰项),并同时给出母语和英语版本。为使基准保持难度,我们要求标注员设计需要推理才能答对的复杂问题(标注指南见附录 J)。

To assess question clarity and unambiguity, we recruited one independent native speaker who was not involved in the creation of the original dataset and conducted a small-scale validation study in five languages (Amharic, Oromo, Tigrinya, Somali, and Luganda). For each language, we randomly sampled 20 questions and provided an image-question pair without the correct answer to validators, and asked them to provide their own answers to the questions given the image. Then we compared their responses using a binary match against the gold-standard answers (match = 1, no match = 0). Across all languages, the validators' answers matched our gold-standard answers, confirming that the questions are clear, unambiguous, and well-formed.

为检查问题是否清楚、是否没有歧义,我们另聘了一位未参与原始数据集创建的独立母语者,并在五种语言(阿姆哈拉语、奥罗莫语、提格利尼亚语、索马里语和卢干达语)中做了小规模核验。每种语言随机抽取 20 道题,只向核验者提供图像和问题、不给出正确答案,请他们根据图像自行作答。随后用是否匹配标准答案做二元对照(匹配为 1,不匹配为 0)。在所有语言上,核验者的答案都与标准答案一致,说明这些问题清楚、没有歧义,并且形式规整。

Audio Recording To investigate spoken language understanding capabilities, we instructed annotators to record audio for each question and its corresponding answers, reading them clearly in both the native language and English. Therefore, Afri-MCQA includes audio recordings for both questions and answers in native language and African-accented English.

音频录制 为考察口语理解能力,我们要求标注员为每道题及其对应答案录音,并用母语和英语清楚朗读。因此,Afri-MCQA 同时包含问题和答案的母语录音,以及带非洲口音的英语录音。

Table 2: Overview of languages in Afri-MCQA. #spk = estimated L1 & L2 speakers, #eng (h) = hours of accented English audio, #nat (h) = hours of native language audio. – indicates English audio not collected for that language.

Lang-Country Family / Branch Reg #spk #QA #eng(h) #nat(h)
Akan/Twi-Ghana Niger-Congo / Volta-Niger West 9M 537 2.41 2.43
Amharic-Ethiopia Afro-Asiatic / Ethio-Semitic East 57M 500 1.56 1.51
Chichewa-Malawi Niger-Congo / Bantu S&E 12M 501 1.41 1.50
Hausa-Nigeria Afro-Asiatic / Chadic West 77M 496 2.80 3.04
Igbo-Nigeria Niger-Congo / Volta-Niger West 31M 501 1.61 1.59
Kikuyu-Kenya Niger-Congo / Bantu East 8.1M 495 1.66 1.72
Kinyarwanda-Rwanda Niger-Congo / Bantu East 18M 501 2.73 2.67
Luganda-Uganda Niger-Congo / Bantu East 10M 500 2.30 2.42
Oromo-Ethiopia Afro-Asiatic / Cushitic East 37M 512 2.20 2.30
Setswana-Botswana Niger-Congo / Bantu South 14M 502 1.89 2.39
Somali-Somalia Afro-Asiatic / Cushitic East 22M 501 1.96 1.99
Tigrinya-Eritrea Afro-Asiatic / Ethio-Semitic East 9M 537 2.13 2.31
Yoruba-Nigeria Niger-Congo / Volta-Niger West 46M 498 1.98 2.06
Sesotho-Lesotho Niger-Congo / Bantu South 13.5M 533 – 1.90
Zulu-S.Africa Niger-Congo / Bantu South 28M 528 – 1.37

表 2:Afri-MCQA 所含语言一览。#spk = 估计的母语及二语使用人数,#eng (h) = 带口音英语音频时长(小时),#nat (h) = 母语音频时长(小时)。– 表示该语言未采集英语音频。

语言-国家 语系 / 语支 区域 #spk #QA #eng(h) #nat(h)
Akan/Twi-Ghana Niger-Congo / Volta-Niger West 9M 537 2.41 2.43
Amharic-Ethiopia Afro-Asiatic / Ethio-Semitic East 57M 500 1.56 1.51
Chichewa-Malawi Niger-Congo / Bantu S&E 12M 501 1.41 1.50
Hausa-Nigeria Afro-Asiatic / Chadic West 77M 496 2.80 3.04
Igbo-Nigeria Niger-Congo / Volta-Niger West 31M 501 1.61 1.59
Kikuyu-Kenya Niger-Congo / Bantu East 8.1M 495 1.66 1.72
Kinyarwanda-Rwanda Niger-Congo / Bantu East 18M 501 2.73 2.67
Luganda-Uganda Niger-Congo / Bantu East 10M 500 2.30 2.42
Oromo-Ethiopia Afro-Asiatic / Cushitic East 37M 512 2.20 2.30
Setswana-Botswana Niger-Congo / Bantu South 14M 502 1.89 2.39
Somali-Somalia Afro-Asiatic / Cushitic East 22M 501 1.96 1.99
Tigrinya-Eritrea Afro-Asiatic / Ethio-Semitic East 9M 537 2.13 2.31
Yoruba-Nigeria Niger-Congo / Volta-Niger West 46M 498 1.98 2.06
Sesotho-Lesotho Niger-Congo / Bantu South 13.5M 533 – 1.90
Zulu-S.Africa Niger-Congo / Bantu South 28M 528 – 1.37

4 Experimental setup

4 实验设置

Our experimental design is organized to address each research question as follows:

实验设计按各个研究问题组织如下:

Models: To assess how well current MLLMs understand African cultural contexts (RQ1), we selected models based on two criteria: a) support for both image and audio input, enabling multimodal evaluation, and (b) availability of different model sizes within each family to assess scaling effects. From open-weight MLLMs, we selected Qwen 2.5-Omni (3B & 7B) (Jin Xu, 2025) and Gemma-3n-(2B & 4B)-it (Kamath et al., 2025). For text-only baselines, we include Gemma3 (12B & 27B)-it (Kamath et al., 2025). For comparison with closed-source models, we include Gemini-2.5 Pro (Comanici et al., 2025), which supports audio, text, and vision inputs.

模型: 为评估当前多模态大语言模型对非洲文化语境的理解(RQ1),我们按两条标准选模型:a)同时支持图像和音频输入,以便做多模态评测;b)同一模型家族内有不同规模,以便观察缩放效应。开放权重的多模态大语言模型中,我们选用 Qwen 2.5-Omni(3B 与 7B)(Jin Xu, 2025)和 Gemma-3n-(2B 与 4B)-it(Kamath et al., 2025)。纯文本基线包括 Gemma3(12B 与 27B)-it(Kamath et al., 2025)。为与闭源模型比较,我们纳入支持音频、文本和视觉输入的 Gemini-2.5 Pro(Comanici et al., 2025)。

Query Modality: To explore how input modality affects performance (RQ2), we evaluate models using both text and audio modalities. For text evaluation, models receive the written version of the question. For audio evaluation, we use native speaker recordings of the same questions and options, allowing us to assess how well current MLLMs handle spoken language inputs both in African languages and accented English. This comparison shows whether model performance generalizes from text to speech. Since we evaluate VQA, all settings include the image related to the question.

提问模态: 为考察输入模态对表现的影响(RQ2),我们同时用文本和音频评测模型。文本评测中,模型收到书面问题。音频评测中,我们使用母语者对同一问题和选项的录音,以衡量当前多模态大语言模型处理非洲语言口语和带口音英语的能力。这一对照可以看出模型表现能否从文本推广到语音。由于评测的是视觉问答,所有设定都包含与问题相关的图像。

Query Languages: To explore how query languages affect models' performance and whether gaps reflect linguistic or cultural limitations (RQ3), each question is presented in native language and English. This setup allows us to compare model behavior between English and native languages.

提问语言: 为考察提问语言对模型表现的影响,以及差距反映的是语言局限还是文化局限(RQ3),每道题都分别用母语和英语给出。这一设定便于比较模型在英语和母语下的行为。

Task Format: To understand how task format affects model performance (RQ4), we evaluate models on both Multiple-Choice VQA (MC-VQA) and Open-ended VQA. MC-VQA provides answer options, while Open-ended VQA requires answer generation. Comparing these formats shows whether strong MC-VQA performance reflects actual cultural understanding or simply selecting from provided options. For each task format, we use the same prompt templates across models and languages.

任务形式: 为了解任务形式如何影响模型表现(RQ4),我们同时评测选择题视觉问答(MC-VQA)和开放式视觉问答。MC-VQA 提供备选项,开放式视觉问答则要求生成答案。比较这两种形式,可以看出较强的选择题表现究竟反映了真正的文化理解,还只是在给定选项中挑选。对每种任务形式,我们在各模型和各语言上使用相同的提示模板。

Prompt: We evaluated all settings and models using a Location-aware prompt (adding location/country as a context). We chose this setting as it performed better than image-only prompts (without providing context). We provide results for Image-only prompts and language-wise results in Appendix C, and the prompts used are in Appendix B.

提示: 所有设定和模型都使用带地点信息的提示(把地点/国家作为上下文加入)。我们选择这一设定,是因为它比只给图像、不提供上下文的提示表现更好。仅图像提示的结果以及分语言结果见附录 C,所用提示见附录 B。

5 Evaluation

5 评测

This section describes our two evaluation setups and the metrics used in this study.

本节说明本研究所用的两套评测设定及指标。

5.1 Cultural VQA Evaluation

5.1 文化视觉问答评测

We evaluate models on visually-grounded cultural QA using Afri-MCQA in both modalities. For all tasks, models are provided with an image and must use visual information to answer the question.

我们用 Afri-MCQA 在两种模态上评测模型的视觉依据文化问答。所有任务都会向模型提供图像,模型必须利用视觉信息来回答问题。

Text-based VQA: For the text modality, we use two evaluation formats: (1) MC-VQA: Models are given an image, a text question, and four answer options. They must select the correct option by reasoning over the image and question. (2) Open-ended VQA: Models receive an image and a text question without answer options, and are required to generate the correct answer. This tests their ability to retrieve and reason about cultural knowledge without potential hints in the answer set.

基于文本的视觉问答: 在文本模态上,我们使用两种评测形式:(1)MC-VQA:向模型提供图像、文本问题和四个选项,模型须结合图像和问题进行推理并选出正确选项。(2)开放式视觉问答:模型只收到图像和文本问题,没有选项,须生成正确答案。这一设定检验模型在没有选项提示的情况下检索并推理文化知识的能力。

Audio-based VQA: Similarly, we evaluate audio modality using two formats: (1) Audio MC-VQA: Same setting as MC-VQA described above, but models are queried through African-accented English and native language speech. (2) Audio open-ended VQA: Given an image and the question in speech format without answer options and models required to generate the correct answer in text.

基于音频的视觉问答: 音频模态同样用两种形式评测:(1)音频 MC-VQA:设定与上面的 MC-VQA 相同,但问题以带非洲口音的英语和母语语音提出。(2)音频开放式视觉问答:给定图像和语音形式的问题,不提供选项,模型须用文本生成正确答案。

5.2 Control Experiments

5.2 控制实验

While it is technically challenging to determine whether prediction failures on Afri-MCQA arise from limitations in language understanding or gaps in cultural knowledge, we conduct control experiments on easy tasks that primarily require language understanding in either text or speech form. These evaluations provide evidence to understand the extent to which language-understanding limitations may contribute to the observed failures.

要准确判断 Afri-MCQA 上的预测失败究竟来自语言理解的不足,还是来自文化知识的缺口,在技术上并不容易。为此,我们在主要考察语言理解的较容易任务上做了控制实验,分别覆盖文本和语音。这些评测可以帮助判断,语言理解方面的局限在多大程度上造成了我们所看到的失败。

Text-based experiments: To probe text understanding, we evaluate on two benchmarks: (1) AfriXNLI (Adelani et al., 2025): natural language inference, and (2) AfriMMLU (Adelani et al., 2025): general knowledge QA. By analyzing performance on these text-only tasks and comparing it with results on Afri-MCQA, we obtain an approximate measure of the models' baseline linguistic competence on the studied languages.

基于文本的实验: 为探测文本理解,我们在两个基准上评测:(1)AfriXNLI(Adelani et al., 2025):自然语言推理;(2)AfriMMLU(Adelani et al., 2025):通识问答。分析这些纯文本任务上的表现,并与 Afri-MCQA 的结果对照,可以大致衡量模型在所研究语言上的基线语言能力。

Audio-based experiments: To probe audio understanding, we conduct two tasks: (1) ASR: transcribing spoken African language audio to text, assessing whether models can accurately capture spoken content as a prerequisite for answering questions. (2) Language Identification (LID): identifying which of the 15 languages is spoken, testing the model's ability to recognize spoken languages. These tasks reveal whether poor audio VQA results from speech-processing failures or cultural reasoning limitations.

基于音频的实验: 为探测音频理解,我们做两项任务:(1)自动语音识别(ASR):把非洲语言口语转写成文本,检验模型能否准确抓住口语内容,这是答题的前提。(2)语种识别(LID):判断 15 种语言中哪一种正在被说,检验模型识别口语语种的能力。这些任务可以揭示,较差的音频视觉问答究竟来自语音处理失败,还是来自文化推理的局限。

5.3 Evaluation metrics

5.3 评测指标

We evaluate models in a zero-shot setting using automatic metrics across all tasks, with additional human evaluation for open-ended VQA (text).

我们在零样本设定下,对所有任务使用自动指标评测模型,并在开放式视觉问答(文本)上另做人工评测。

Automatic Evaluation: We report accuracy scores for MC-VQA and classification tasks, and use GPT-4o-mini (Hurst et al., 2024) as a judge for Open-ended VQA. For Open-ended QA, we additionally compute chrF++ (Popović, 2015) scores and present them in Appendix C. We report Word Error Rate (WER) for ASR.

自动评测: 我们对 MC-VQA 和分类任务报告准确率,并用 GPT-4o-mini(Hurst et al., 2024)作为开放式视觉问答的评审。对开放式问答,我们还计算 chrF++(Popović, 2015)分数,结果见附录 C。自动语音识别则报告词错误率(WER)。

Human Evaluation: We evaluate 50 randomly sampled questions per language on the best-performing model per family. Bilingual native speakers rated whether model outputs matched the gold answer or were valid alternatives.

人工评测: 我们对每个模型家族中表现最好的模型,按语言随机抽取 50 道题进行评测。由双语母语者判断模型输出是否与标准答案一致,或是否属于可接受的等效答案。

6 Results

6 结果

In this Section, we present results organized by evaluation type, beginning with cultural VQA performance across MC-VQA and Open-ended VQA tasks, followed by control experiments.

本节按评测类型汇报结果,先看 MC-VQA 和开放式视觉问答上的文化视觉问答表现,再报告控制实验。

6.1 Cultural VQA Results

6.1 文化视觉问答结果

We show evaluations on MC-VQA and Open-ended VQA tasks, comparing performance across text and audio modalities in both English and native languages.

我们给出 MC-VQA 和开放式视觉问答的评测,并比较文本与音频模态、以及英语与母语下的表现。

6.1.1 Text-based QA

6.1.1 基于文本的问答

Figure 3 depicts MC-VQA average accuracy (3(a)) and LLM-as-a-judge scores (3(b)) for open-ended QA in both English and native African languages using location-aware prompts. In figure 5, we report language-wise accuracy for text-based VQA on both English and African-language questions.

图 3 给出了在带地点信息的提示下,英语和非洲母语的 MC-VQA 平均准确率(3(a)),以及开放式问答的 LLM 评审分数(3(b))。图 5 则报告了基于文本的视觉问答在英语和非洲语言问题上的分语言准确率。

Open-weight models consistently perform better when the question is in English compared to native languages across both tasks. We also observe a significant performance gap between MC-VQA and Open-ended QA, where all models, including Gemini-2.5 Pro, show major drops when tasked with answering in an open-ended setting, even in English. This suggests that generating culturally grounded responses is more challenging than selecting from predefined options, particularly for native-language queries, where performance degrades significantly compared to English queries.

在两项任务上,开放权重模型在问题为英语时都稳定地好于母语。我们还看到 MC-VQA 与开放式问答之间存在明显差距:包括 Gemini-2.5 Pro 在内的所有模型,一旦改为开放式作答,即便是英语也会大幅下降。这说明,生成有文化依据的回答,要比从既定选项中挑选更难,在母语提问时尤其如此,相对英语提问会明显变差。

We also observe that increasing the model size of open-weight models does not necessarily translate into improved performance in low-resource languages. Larger variants show limited or no improvement over smaller counterparts in native language QA, indicating that model scaling alone is insufficient to address low-resource challenges. Notably, some smaller models achieve near-zero accuracy for native languages for Open-ended QA. In contrast, Gemini-2.5-Pro outperforms all open-weight models across both tasks while maintaining comparable performance between English and native languages, highlighting the current gap between proprietary and open-weight models.

我们还发现,把开放权重模型做大,并不一定会提高低资源语言上的表现。在母语问答中,更大的变体相对较小模型只有有限提升,甚至没有提升,说明仅靠扩大规模并不足以应对低资源挑战。值得注意的是,部分较小模型在母语开放式问答上的准确率接近零。相比之下,Gemini-2.5-Pro 在两项任务上都超过所有开放权重模型,并且英语与母语上的表现比较接近,凸显了当前专有模型与开放权重模型之间的差距。

Human evaluations on Open-Ended VQA (in Figure 6), conducted on a random subset of 50 samples across seven languages, are consistent with the trends observed in LLM-as-a-Judge evaluations, with Gemini-2.5 Pro performs the best when queried in native languages compared to English. In Table 3, we report inter-rater agreement (Krippendorff, 2011) between LLM-as-a-Judge and human evaluation on a random subset of 50 samples across seven languages for four models. As shown in Table 3, LLM-as-a-Judge shows strong inter-rater agreement with human evaluation on a small subset of data in both setups, demonstrating its reliability for open-ended generation in low-resource languages. We added per-language scores in Appendix D.

对开放式视觉问答的人工评测(见图 6)在 7 种语言上各随机抽取 50 条样本,结果与 LLM 评审看到的趋势一致;其中 Gemini-2.5 Pro 在母语提问时比英语更好。表 3 报告了四种模型上,LLM 评审与人工评测在 7 种语言、各 50 条随机样本上的评分者间一致性(Krippendorff, 2011)。如表 3 所示,在两种设定的小规模子集上,LLM 评审与人工评测都有较强的一致性,说明它在低资源语言的开放式生成评测中是可靠的。分语言分数见附录 D。

Table 3: Average per-model inter-rater agreement (LLM-as-a-Judge vs Human evaluator) for English and Native on a random subset of 50 samples.

Setup Language Krippendorff's Alpha
English Gemini-2.5-pro 0.77
English Qwen2.5-Omni-7B 0.84
English Gemma-3-27b 0.76
English Gemma-3n-E4B 0.88
English Average 0.81
Native Gemini-2.5-pro 0.78
Native Qwen2.5-Omni-7B 0.91
Native Gemma-3-27b 0.91
Native Gemma-3n-E4B 0.67
Native Average 0.81

表 3:英语和母语设定下,各模型的平均评分者间一致性(LLM 评审对人工评测),样本为每组随机 50 条。

设定 模型 Krippendorff's Alpha
English Gemini-2.5-pro 0.77
English Qwen2.5-Omni-7B 0.84
English Gemma-3-27b 0.76
English Gemma-3n-E4B 0.88
English Average 0.81
Native Gemini-2.5-pro 0.78
Native Qwen2.5-Omni-7B 0.91
Native Gemma-3-27b 0.91
Native Gemma-3n-E4B 0.67
Native Average 0.81

6.1.2 Audio-based QA

6.1.2 基于音频的问答

Figure 4 shows MC-VQA accuracy (4(a)) and LLM-as-a-judge scores for Open-ended QA (4(b)) on audio inputs in both English and native African languages. Similar to text-based evaluation, open-weight models achieve higher performance when queried in English compared to native languages across both tasks. For open-weight models, audio modality is significantly more difficult than text modality, with notable performance degradation across both tasks. Gemini-2.5-Pro, however, demonstrates robust multimodal capabilities, with consistent performance across modalities. The performance drop from MC-VQA to Open-ended QA is also noticeable in the audio modality, with nearly zero accuracy in spoken native languages. These results align with our LID and ASR analyses (Section 6.2.2). Open-weight models demonstrate poor LID capabilities, especially the Qwen variants, which exhibit near-random accuracy and significantly compromised native ASR performance compared to English. These failures also affect performance models in downstream tasks, such as Open-ended VQA. Similar to text evaluations, we find that scaling up model size shows little improvement in native language understanding.

图 4 给出了英语和非洲母语的音频输入上,MC-VQA 准确率(4(a))以及开放式问答的 LLM 评审分数(4(b))。与文本评测类似,开放权重模型在两项任务上用英语提问都好于母语。对开放权重模型而言,音频模态明显难于文本模态,两项任务都出现明显下降。不过 Gemini-2.5-Pro 表现出更稳健的多模态能力,跨模态表现较为一致。从 MC-VQA 到开放式问答的下降在音频模态同样明显,母语口语上的准确率几乎为零。这些结果与我们的语种识别和语音识别分析(第 6.2.2 节)一致。开放权重模型的语种识别能力较差,Qwen 系列尤其如此,准确率接近随机,母语语音识别相对英语也明显受损。这些失败还会影响到下游任务中的模型表现,例如开放式视觉问答。与文本评测类似,我们看到扩大模型规模对母语理解几乎没有带来提升。

Bar charts comparing text-based MC-VQA accuracy and open-ended VQA scores in English and native languages

Figure 3: Performance comparison of models on text-based question answering tasks: (a) Text MC-VQA (Multiple Choice) and (b) Text Open-ended QA in English and Native languages.

图 3:各模型在基于文本的问答任务上的表现对照:(a)文本 MC-VQA(选择题);(b)英语和母语上的文本开放式问答。

Bar charts comparing audio-based MC-VQA accuracy and open-ended VQA scores in English and native languages

Figure 4: Performance comparison of models on audio-based question answering tasks: (a) Audio MC-VQA (Multiple Choice) and (b) Audio Open-ended VQA in English and Native languages.

图 4:各模型在基于音频的问答任务上的表现对照:(a)音频 MC-VQA(选择题);(b)英语和母语上的音频开放式视觉问答。

Heatmaps of language-wise text VQA accuracy for seven multimodal LLMs on English and native-language questions

Figure 5: Language-wise accuracy of seven multimodal LLMs on the text-based VQA task, evaluated separately on English and native African language questions.

图 5:七个多模态大语言模型在基于文本的视觉问答任务上的分语言准确率,分别在英语和非洲母语问题上评测。

Heatmap of human evaluation English minus native accuracy gaps for text open-ended VQA

Figure 6: Human Evaluation for Text Open-ended VQA. Accuracy across best-performing models for English and Native. We observed that, while most models perform best in the English setting, Gemini-2.5 Pro seems to perform better in the Native language.

图 6:文本开放式视觉问答的人工评测。给出各家族中表现最好的模型在英语和母语上的准确率。我们看到,多数模型在英语设定下最好,但 Gemini-2.5 Pro 在母语上似乎更好。

6.2 Results for Control Experiments

6.2 控制实验结果

We present results from control experiments on established benchmarks alongside our cultural QA task, aiming to probe the linguistic competence of the tested models and to provide pointers on why models fail on Afri-MCQA.

我们把控制实验在既有基准上的结果,与文化问答任务放在一起报告,目的是探测受测模型的语言能力,并为模型为何在 Afri-MCQA 上失败提供线索。

6.2.1 Text-based Experiments

6.2.1 基于文本的实验

In Table 4, we report the performance of the models on the control-experiment benchmarks in addition to Afri-MCQA. All models showed performance degradation from English to native languages across all benchmarks. Gemini-2.5-Pro maintains the smallest gaps, particularly on Afri-MCQA, where the drop is minimal. Open-weight models show a big drop, with Qwen variants showing the most severe gaps on AfriXNLI and AfriMMLU.

表 4 报告了各模型在控制实验基准以及 Afri-MCQA 上的表现。在所有基准上,所有模型从英语到母语都出现下降。Gemini-2.5-Pro 的差距最小,在 Afri-MCQA 上的下降尤其有限。开放权重模型下降幅度较大,其中 Qwen 系列在 AfriXNLI 和 AfriMMLU 上的差距最为严重。

When comparing AfriMMLU with Afri-MCQA, open-weight models show considerably higher AfriMMLU scores in English than Afri-MCQA scores, suggesting that while models possess general factual knowledge, they lack Africa-specific cultural understanding. Gemini-2.5-Pro shows a smaller gap between these benchmarks, indicating it has acquired more African cultural knowledge during training. AfriXNLI exposes the most severe cross-lingual gaps, particularly for Qwen models, suggesting that linguistic reasoning tasks are more sensitive to the availability of language resources than factual retrieval tasks.

把 AfriMMLU 与 Afri-MCQA 对照时,开放权重模型的英语 AfriMMLU 分数明显高于 Afri-MCQA,说明模型虽有一般事实知识,却缺乏针对非洲的文化理解。Gemini-2.5-Pro 在这两个基准之间的差距较小,表明它在训练中获得了更多非洲文化知识。AfriXNLI 暴露出最严重的跨语言差距,对 Qwen 模型尤其如此,说明语言推理任务比事实检索任务对语言资源是否充足更为敏感。

We additionally compute Spearman rank correlations between Afri-MCQA and our text control datasets (see Appendix E). We observe strong and statistically significant correlations between AfriXNLI and AfriMMLU performance for several models. In contrast, correlations between Afri-MCQA and AfriXNLI or AfriMMLU are generally weaker and not statistically significant. Hence, while limitations in language understanding may play a large role in explaining why models fail in Afri-MCQA, other factors related to African cultural and visual knowledge may be important as well.

我们还计算了 Afri-MCQA 与文本控制数据集之间的 Spearman 等级相关(见附录 E)。在若干模型上,AfriXNLI 与 AfriMMLU 的表现呈强且统计显著的相关。相比之下,Afri-MCQA 与 AfriXNLI 或 AfriMMLU 之间的相关通常更弱,而且不具统计显著性。因此,语言理解的局限也许能在很大程度上解释模型为何在 Afri-MCQA 上失败,但与非洲文化和视觉知识有关的其他因素,可能同样重要。

Table 4: Text-based performance (%). Eng = English accuracy, Nat = Native accuracy, Δ\DeltaΔ = English −-− Native gap (negative = drop).

AfriXNLI AfriXNLI AfriXNLI AfriMMLU AfriMMLU AfriMMLU Afri-MCQA Afri-MCQA Afri-MCQA
Model Eng Nat Δ\DeltaΔ Eng Nat Δ\DeltaΔ Eng Nat Δ\DeltaΔ
Gemini-2.5-Pro 89.66 76.3 -13.36 94 83.46 -10.54 78.68 76.27 -2.4
Gemma3n-4B 78.33 51.24 -27.09 62.2 37.56 -24.64 55.23 41.49 -13.7
Gemma3n-2B 80.66 51.47 -29.19 53.6 35.74 -17.86 51.54 40.32 -11.2
Qwen-7B 81.11 34.33 -46.78 73.8 36.10 -37.70 49.75 31.30 -18.46
Qwen-3B 65.66 36.7 -28.96 65.8 34.72 -31.08 50.68 31.00 -19.65

表 4:基于文本的表现(%)。Eng = 英语准确率,Nat = 母语准确率,Δ\DeltaΔ = 英语减母语的差距(负值表示下降)。

AfriXNLI AfriXNLI AfriXNLI AfriMMLU AfriMMLU AfriMMLU Afri-MCQA Afri-MCQA Afri-MCQA
模型 Eng Nat Δ\DeltaΔ Eng Nat Δ\DeltaΔ Eng Nat Δ\DeltaΔ
Gemini-2.5-Pro 89.66 76.3 -13.36 94 83.46 -10.54 78.68 76.27 -2.4
Gemma3n-4B 78.33 51.24 -27.09 62.2 37.56 -24.64 55.23 41.49 -13.7
Gemma3n-2B 80.66 51.47 -29.19 53.6 35.74 -17.86 51.54 40.32 -11.2
Qwen-7B 81.11 34.33 -46.78 73.8 36.10 -37.70 49.75 31.30 -18.46
Qwen-3B 65.66 36.7 -28.96 65.8 34.72 -31.08 50.68 31.00 -19.65

6.2.2 Audio-based Experiments

6.2.2 基于音频的实验

Figure 7 shows three evaluations on native African language audio. LID: Gemini-2.5-Pro achieves near-perfect LID accuracy, while Gemma variants show moderate performance. Qwen models perform at near random levels, indicating minimal exposure to African language audio during pretraining. ASR Performance: WER shows a similar pattern. Gemini-2.5-Pro maintains reasonable native ASR, whereas Gemma models exhibit substantial degradation. Qwen models produce high error rates, indicating hallucinations rather than meaningful transcription. This aligns with observed results for open-ended VQA (Audio). Gemini-2.5-Pro achieves moderate accuracy, whereas open-weight models score near zero. In light of these results, it is likely that models fail on open-ended VQA (Audio) because they cannot properly identify or transcribe the tested spoken languages.

图 7 给出了在非洲母语音频上的三项评测。语种识别: Gemini-2.5-Pro 的语种识别准确率接近满分,Gemma 系列表现中等,Qwen 模型则接近随机,说明预训练中几乎没有接触过非洲语言音频。语音识别表现: 词错误率呈现出类似规律。Gemini-2.5-Pro 的母语识别尚可,Gemma 模型则明显变差。Qwen 模型错误率很高,更像是在产生幻觉,而不是在做有意义的转写。这与开放式视觉问答(音频)上看到的结果一致。Gemini-2.5-Pro 准确率中等,开放权重模型则接近零。综合这些结果可以认为,模型在开放式视觉问答(音频)上失败,是因为它们无法正确识别或转写所测试的口语语言。

Hence, poor Open-ended VQA (Audio) results come from errors at each step: (1) Open models often fail at identifying African languages, reflecting fundamental gaps in audio representation, (2) they show high ASR errors, suggesting that most spoken content is not perceived adequately for understanding and answering a question (3) these errors can carry over to open-ended VQA when models receive wrong transcriptions, as they cannot answer correctly even if they know the cultural content. These findings demonstrate that foundational speech processing capabilities are prerequisites for meaningful evaluation of cultural reasoning in African languages.

因此,较差的开放式视觉问答(音频)结果来自每一个步骤上的错误:(1)开放模型常常识别不出非洲语言,反映出音频表征上的根本缺口;(2)它们的语音识别错误率很高,说明大部分口语内容并没有被充分感知,因而无法理解并回答问题;(3)这些错误会带到开放式视觉问答中——一旦模型得到错误转写,即便它本来知道相关文化内容,也无法答对。这些发现说明,基础的语音处理能力,是有意义地评测非洲语言文化推理的前提。

Audio probing bar charts for language ID, ASR word error rate, and open-ended audio VQA

Figure 7: Audio probing results on native African languages: (a) LID (↑\uparrow↑higher is better), (b) ASR (↓\downarrow↓lower is better), and (c) Open-ended VQA (↑\uparrow↑higher is better). (+) means WER is more than 100%.

图 7:非洲母语上的音频探测结果:(a)语种识别(↑\uparrow↑越高越好);(b)自动语音识别(↓\downarrow↓越低越好);(c)开放式视觉问答(↑\uparrow↑越高越好)。(+)表示词错误率超过 100%。

7 Discussion

7 讨论

We now organize our findings to address each research question, summarizing key patterns observed across models, modalities, and languages.

下面我们按各个研究问题整理发现,并概括在模型、模态和语言上看到的主要规律。

RQ1: How well do MLLMs understand African cultural contexts in visually-grounded QA? MLLMs show limited understanding of African cultural contexts. As shown in Figures 3 and 4, even the best performing model (Gemini-2.5-Pro) achieves only 78% on MC-VQA and 38% on Open-VQA (text-based, English). Smaller models (Gemma3, Qwen2.5) perform substantially worse, ranging from 50–59% on MC-VQA and 9–24% on Open-VQA, indicating significant room for improvement.

RQ1:多模态大语言模型在以视觉为根据的问答中,对非洲文化语境的理解有多好? 多模态大语言模型对非洲文化语境的理解仍然有限。如图 3 和图 4 所示,即便是表现最好的模型(Gemini-2.5-Pro),在基于文本的英语设定下,MC-VQA 也只有 78%,开放式视觉问答只有 38%。较小的模型(Gemma3、Qwen2.5)明显更差,MC-VQA 约在 50–59%,开放式视觉问答约在 9–24%,仍有很大改进空间。

RQ2: How does input modality (text vs. speech) affect performance? Performance degrades when switching from text to speech input, particularly for smaller models. As shown in Figures 3 and 4, on MC-VQA, Qwen models drop by approximately 1–2% while Gemma models show mixed results. The gap is more visible in Open-VQA, where audio-based native language queries yield near-zero accuracy (2–5%) for smaller models. Control experiments for audio show that this comes from poor language identification (2–4% for Qwen) and high ASR word error rates (85–100%+ for non-Gemini models).

RQ2:输入模态(文本与语音)会怎样影响表现? 从文本改成语音输入后,表现会下降,较小模型尤其明显。如图 3 和图 4 所示,在 MC-VQA 上,Qwen 模型大约下降 1–2 个点,Gemma 模型则结果不一。差距在开放式视觉问答上更清楚:较小模型在基于音频的母语提问上准确率接近零(2–5%)。音频控制实验表明,这来自较差的语种识别(Qwen 为 2–4%)以及很高的语音识别词错误率(非 Gemini 模型为 85–100%+)。

RQ3: How does query language (native vs. English) affect performance, and do differences reflect language understanding or cultural knowledge gaps? English queries consistently outperform native language queries across all models and settings. As shown in Figure 3, the gap ranges from 2% (Gemini-2.5-Pro on MC-VQA) to 19% (Qwen on MC-VQA). Control experiments on AfriXNLI and AfriMMLU (Table 4) show that language understanding gaps (Δ=13\Delta = 13Δ=13–474747%) are substantially larger than cultural knowledge gaps (Δ=2\Delta = 2Δ=2–191919% on Afri-MCQA), suggesting language understanding is the dominant limitation. However, models also struggle with cultural QA in English, indicating that both linguistic and cultural limitations contribute to poor performance.

RQ3:提问语言(母语与英语)会怎样影响表现,其间的差异反映的是语言理解问题,还是文化知识缺口? 在所有模型和设定上,英语提问都稳定地好于母语提问。如图 3 所示,差距从 2%(Gemini-2.5-Pro 的 MC-VQA)到 19%(Qwen 的 MC-VQA)不等。AfriXNLI 和 AfriMMLU 上的控制实验(表 4)显示,语言理解差距(Δ=13\Delta = 13Δ=13–474747%)明显大于文化知识差距(Afri-MCQA 上为 Δ=2\Delta = 2Δ=2–191919%),说明语言理解仍是主要限制。不过,模型在英语文化问答上同样吃力,表明语言和文化两方面的局限都造成了较差的表现。

RQ4: How does task format (Multiple-Choice vs. Open QA) affect accuracy? As shown in Figures 3 and 4, models perform significantly better on MC-VQA than Open-VQA. Gemini-2.5-Pro achieves 78% on MC-VQA vs. 38% on Open-VQA (a 40% gap). Smaller models show even larger relative drops, with some scoring less than 10% on Open-VQA. This performance gap suggests that MC-VQA benefits from simplified answer selection, while Open-VQA exposes true limitations. Models struggle to generate culturally grounded responses even in English, with performance degrading further for native languages.

RQ4:任务形式(选择题与开放式问答)会怎样影响准确率? 如图 3 和图 4 所示,模型在 MC-VQA 上明显好于开放式视觉问答。Gemini-2.5-Pro 的 MC-VQA 为 78%,开放式视觉问答为 38%(相差 40 个点)。较小模型的相对下降更大,有的在开放式视觉问答上不到 10%。这一差距说明,MC-VQA 受益于简化的选项选择,而开放式视觉问答才暴露出真正的不足。即便用英语,模型也难以生成有文化依据的回答,到了母语上还会进一步变差。

8 Conclusion

8 结论

We introduced Afri-MCQA, the first large-scale multilingual and multimodal benchmark for African cultural visual QA, covering 15 languages across 12 countries with over 7.5k Q&A pairs in text and speech modalities. Our evaluation shows: 1) MLLMs struggle significantly with African cultural knowledge, 2) speech processing presents a critical bottleneck, and 3) gaps persist across languages and task formats.

我们提出了 Afri-MCQA,这是首个面向非洲文化视觉问答的大规模多语种、多模态基准,覆盖 12 个国家的 15 种语言,包含 7500 余条文本与语音问答对。评测表明:1)多模态大语言模型在非洲文化知识上明显吃力;2)语音处理是关键瓶颈;3)差距在不同语言和任务形式上持续存在。

These findings motivate several research directions: (1) speech-first approaches: Many African languages are primarily oral, yet current open-weight models lack basic LID and ASR capabilities for these languages; (2) culturally-grounded pretraining: The gap between AfriMMLU and Afri-MCQA performance suggests language data alone is insufficient; models need explicit exposure to African cultural content; and (3) cross-lingual cultural transfer: Models may "know" cultural facts in English but cannot access them through native language queries, motivating research into cross-lingual knowledge retrieval.

这些发现引出几条研究方向:(1)语音优先的路径: 许多非洲语言主要以口语使用,但当前开放权重模型仍缺乏对这些语言的基本语种识别和语音识别能力;(2)有文化依据的预训练: AfriMMLU 与 Afri-MCQA 之间的表现差距说明,仅有语言数据还不够,模型需要明确接触非洲文化内容;(3)跨语言文化迁移: 模型也许在英语中“知道”某些文化事实,却无法通过母语提问调用它们,这促使人们研究跨语言的知识检索。

We release Afri-MCQA to provide both a benchmark and a foundation for building more inclusive, culturally aware multimodal systems that better represent African languages and cultures.

我们发布 Afri-MCQA,既作为评测基准,也为构建更具包容性、更有文化意识、更能代表非洲语言与文化的多模态系统提供基础。

9 Limitations

9 局限

We believe Afri-MCQA represents an important step toward more inclusive evaluation by foregrounding African languages and cultural contexts that have long been overlooked in existing benchmarks. Although the dataset spans 15 languages across 12 countries, Africa is home to thousands of languages and cultural groups, many of which remain unrepresented. Furthermore, while our question categories aim to reflect culturally grounded knowledge, culture itself is fluid, subjective, and deeply contextual. Our formulation inevitably abstracts away from finer-grained variations such as regional, generational, or community-LLM involvement t shape cultural understanding. As with most human-curated datasets, potential biases in data collection remain. Annotators' backgrounds and interpretations of "cultural relevance" may influence the formulation of questions or the selection of images. Additionally, due to computational and financial constraints, we evaluate only a limited set of open- and closed-source models, so the reported performance gaps may not fully capture the broader landscape. Regarding the use of an LLM-as-a-judge for low-resource languages: we did not use GPT-4o-mini to generate questions, assess cultural validity, or provide subjective scoring. Instead, it was used strictly to verify semantic equivalence between a model's prediction and a gold reference answer. This is a constrained matching task, not an open-ended evaluation, and therefore the risk of introducing bias is narrow. Moreover, we supplemented LLM-as-a-judge accuracy with chrF++ scores and human evaluation on a subset of the data. The consistency across these measures suggests that the reported trends are not artifacts of LLM-based scoring. Finally, Afri-MCQA is , who are native speakers,hout the involvement of LLMs and with minimal reliance on web-sourced images. Because this process is inherently time-consuming, the dataset is of moderate size and intended as an evaluation benchmark rather than a pretraining or fine-tuning resource, as its moderate size would likely cause overfitting.

我们认为,Afri-MCQA 把长期被现有基准忽视的非洲语言和文化语境放到前景位置,是迈向更具包容性评测的重要一步。尽管数据集覆盖 12 个国家的 15 种语言,非洲仍有成千上万种语言和文化群体,其中许多尚未被纳入。此外,我们的问题类别意在反映有文化依据的知识,但文化本身是流动的、主观的,并且高度依赖语境。我们的表述难免会抽掉更细的差异,例如地域、代际,或社区层面那些会塑造文化理解的因素。与大多数由人工整理的数据集一样,采集过程中仍可能存在偏差。标注员的背景,以及对“文化相关性”的理解,可能影响问题写法或图像选择。另外,受计算和经费限制,我们只评测了有限的开源与闭源模型,因此所报告的表现差距未必能完整反映更广的格局。关于在低资源语言上使用 LLM 评审:我们没有用 GPT-4o-mini 来出题、判断文化是否成立,或做主观打分。它只被用来核验模型预测与标准参考答案之间的语义是否等价。这是一项受限的匹配任务,而不是开放式评判,因此引入偏差的风险较窄。此外,我们还用 chrF++ 分数,以及在数据子集上的人工评测,来补充 LLM 评审的准确率。这些度量彼此一致,说明所报告的趋势并非 LLM 评分造成的假象。最后,Afri-MCQA 由母语者完成,没有使用大语言模型,也尽量少依赖来自网络的图像。由于这一过程本身耗时,数据集规模中等,只宜作为评测基准,而不宜用作预训练或微调资源,因为中等规模很容易导致过拟合。

10 Ethical Considerations

10 伦理考虑

Our work involves the collection of culturally grounded question-answer pairs in 15 African languages, annotated, spoken and reviewed by native speakers. All annotators participated voluntarily and were compensated fairly for their work in accordance with local wage standards on the Upwork platform. Before beginning annotation, contributors were informed about the goals of the project, the intended use of the dataset for research and evaluation purposes, and their right to withdraw from participation at any stage.

本研究采集了 15 种非洲语言中有文化依据的问答对,由母语者标注、朗读并审核。所有标注员均自愿参与,并按 Upwork 平台上的当地薪酬标准获得合理报酬。开始标注前,参与者已获知项目目标、数据集用于研究与评测的用途,以及他们可在任何阶段退出的权利。

We took several steps to ensure cultural sensitivity and respect throughout the data creation process. Question formulation guidelines were designed to avoid harmful stereotypes, offensive content, or culturally inappropriate framing. All annotations were reviewed by language coordinators who are themselves native speakers to check for accuracy, contextual appropriateness, and respectful representation. Despite these efforts, we acknowledge that culture is deeply complex and subjective, and that our dataset may still reflect certain biases or oversimplifications.

在数据创建过程中,我们采取了多项措施以保障文化敏感与尊重。问题撰写指南旨在避免有害刻板印象、冒犯性内容或不恰当的文化表述。全部标注都由本身即为母语者的语言协调员审阅,检查是否准确、是否合乎语境,以及是否以尊重的方式呈现。尽管做了这些努力,我们仍承认文化极为复杂且主观,本数据集仍可能带有某些偏差或过度简化。

References

参考文献

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 others. 2024. Phi-4 technical report. Preprint, arXiv:2412.08905.

Ife Adebara and Muhammad Abdul-Mageed. 2022. Towards afrocentric NLP for African languages: Where we are and where we can go. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3814–3841, Dublin, Ireland. Association for Computational Linguistics.

David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 226–245, St. Julian’s, Malta. Association for Computational Linguistics.

David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, and 46 others. 2023. MasakhaNEWS: News topic classification for African languages. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 144–159, Nusa Dua, Bali. Association for Computational Linguistics.

David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Ijeoma Chukwuneke, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson KoffiKalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, and 8 others. 2025. IrokoBench: A new benchmark for African languages in the age of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2732–2757, Albuquerque, New Mexico. Association for Computational Linguistics.

Jesujoba Oluwadara Alabi, Michael A. Hedderich, David Ifeoluwa Adelani, and Dietrich Klakow. 2025. Charting the landscape of African NLP: Mapping progress and shaping the road ahead. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 27807–27841, Suzhou, China. Association for Computational Linguistics.

Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Yonas Chanie, Bontu Fufa Balcha, Negasi Haile Abadi, Henok Biadglign Ademtew, Mulubrhan Abebe Nerea, Debela Desalegn Yadeta, Derartu Dagne Geremew, Assefa Atsbiha Tesfu, Philipp Slusallek, Thamar Solorio, and Dietrich Klakow. 2025. ProverbEval: Exploring LLM evaluation challenges for low-resource language understanding. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6250–6266, Albuquerque, New Mexico. Association for Computational Linguistics.

Israel Abebe Azime, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Mitiku Yohannes Fuge, Aman Kassahun Wassie, Eyasu Shiferaw Jada, Yonas Chanie, Walelign Tewabe Sewunetie, and Seid Muhie Yimam. 2024. Walia-LLM: Enhancing Amharic-LLaMA by integrating task-specific and generative datasets. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 432–444, Miami, Florida, USA. Association for Computational Linguistics.

Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261.

Harald Hammarström. 2018. A survey of african languages.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Jinzheng He Hangrui Hu Ting He Shuai Bai Keqin Chen Jialin Wang Yang Fan Kai Dang Bin Zhang Xiong Wang Yunfei Chu Junyang Lin Jin Xu, Zhifang Guo. 2025. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215.

Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, and 196 others. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786.

Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability.

Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021a. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021b. Visually grounded reasoning across languages and cultures. arXiv preprint arXiv:2109.13238.

Rada Mihalcea, Oana Ignat, Longju Bai, Angana Borah, Luis Chiruzzo, Zhijing Jin, Claude Kwizera, Joan Nwatu, Soujanya Poria, and Thamar Solorio. 2025. Why ai is weird and shouldnt be this way: Towards ai for everyone, with everyone, by everyone.

Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, David Ifeoluwa Adelani, Ibrahim Said Ahmad, Saminu Mohammad Aliyu, Paul Röttger, Abigail Oppong, Andiswa Bukula, Chiamaka Ijeoma Chukwuneke, Ebrahim Chekol Jibril, Elyas Abdi Ismail, Esubalew Alemneh, Hagos Tesfahun Gebremichael, Lukman Jibril Aliyu, Meriem Beloucif, Oumaima Hourrane, Rooweither Mabuya, Salomey Osei, and 8 others. 2025. AfriHate: A multilingual collection of hate speech and abusive language datasets for African languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1854–1871, Albuquerque, New Mexico. Association for Computational Linguistics.

Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Aishwarya Agrawal, and 1 others. 2024. Benchmarking vision language models for cultural understanding. arXiv preprint arXiv:2407.10920.

James Cross Onur Çelebi Maha Elbayad Kenneth Heafield Kevin Heffernan Elahe Kalbassi Janice Lam Daniel Licht Jean Maillard Anna Sun Skyler Wang Guillaume Wenzek Al Youngblood Bapi Akula Loic Barrault Gabriel Mejia Gonzalez Prangthip Hansanti John Hoffman Semarley Jarrett Kaushik Ram Sadagopan Dirk Rowe Shannon Spruit Chau Tran Pierre Andrews Necip Fazil Ayan Shruti Bhosale Sergey Edunov Angela Fan Cynthia Gao Vedanuj Goswami Francisco Guzmán Philipp Koehn Alexandre Mourachko Christophe Ropers Safiyyah Saleem Holger Schwenk Jeff Wang NLLB Team, Marta R. Costa-jussà. 2022. No language left behind: Scaling human-centered machine translation.

Derek Nurse and Gerard Philippson, editors. 2006. The Bantu Languages. Routledge Language Family Series. Routledge, London, England.

Odunayo Ogundepo, Tajuddeen R Gwadabe, Clara E Rivera, Jonathan H Clark, Sebastian Ruder, David Ifeoluwa Adelani, Bonaventure FP Dossou, Abdou Aziz Diop, Claytone Sikasote, Gilles Hacheme, and 1 others. 2023. Afriqa: Cross-lingual open-retrieval question answering for african languages. arXiv preprint arXiv:2305.06897.

Jessica Ojo, Odunayo Ogundepo, Akintunde Oladipo, Kelechi Ogueji, Jimmy Lin, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. Afrobench: how good are large language models on african languages? In Findings of the Association for Computational Linguistics: ACL 2025, pages 19048–19095.

Shantipriya Parida, Idris Abdulmumin, Shamsuddeen Hassan Muhammad, Aneesh Bose, Guneet Singh Kohli, Ibrahim Said Ahmad, Ketan Kotwal, Sayan Deb Sarkar, Ondˇrej Bojar, and Habeebah Adamu Kakudi. 2023. Havqa: A dataset for visual question answering and multimodal research in hausa language. arXiv preprint arXiv:2305.17690.

Maja Popovi´c. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395.

David Romero, Chenyang Lyu, Haryo Wibowo, Santiago Góngora, Aishik Mandal, Sukannya Purkayastha, Jesus-German Ortiz-Barajas, Emilio Cueva, Jinheon Baek, Soyeong Jeong, and 1 others. 2024. Cvqa: Culturally-diverse multilingual visual question answering benchmark. Advances in Neural Information Processing Systems, 37:11479–11505.

Florian Schneider and Sunayana Sitaram. 2024. M5–a diverse benchmark to assess the performance of large multimodal models across multilingual and multicultural vision-language tasks. arXiv preprint arXiv:2407.03791.

Belay Simane, Thandi Kapwata, Natasha Naidoo, Guéladio Cissé, Caradee Y. Wright, and Kiros Berhane. 2025. Ensuring africas food security by 2050: The role of population growth, climateresilient strategies, and putative pathways to resilience. Foods, 14(2):262.

Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I Adelani, Jian Gang Ngui, Daniel VilaSuero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, and 1 others. 2024. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. arXiv preprint arXiv:2412.03304.

Atnafu Lambebo Tonja, Israel Abebe Azime, Tadesse Destaw Belay, Mesay Gemeda Yigezu, Moges Ahmed Ah Mehamed, Abinew Ali Ayele, Ebrahim Chekol Jibril, Michael Melese Woldeyohannis, Olga Kolesnikova, Philipp Slusallek, and 1 others. 2024a. Ethiollm: Multilingual large language models for ethiopian languages with task evaluation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6341–6352.

Atnafu Lambebo Tonja, Bonaventure FP Dossou, Jessica Ojo, Jenalea Rajab, Fadel Thior, Eric Peter Wairagala, Anuoluwapo Aremu, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, and 1 others. 2024b. Inkubalm: A small language model for low-resource african languages. arXiv preprint arXiv:2408.17024.

Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadglign Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kukreja, and 1 others. 2025a. All languages matter: Evaluating lmms on culturally diverse 100 languages. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19565–19575.

Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadglign Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kukreja, and 1 others. 2025b. All languages matter: Evaluating lmms on culturally diverse 100 languages. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19565–19575.

Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Cheng Ching Lam, Daud Abolade, Emmanuele Chersoni, and 32 others. 2025. WorldCuisines: A massive-scale benchmark for multilingual and multicultural visual question answering on global cuisines. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3242–3264, Albuquerque, New Mexico. Association for Computational Linguistics.

Hao Yu, Jesujoba Oluwadara Alabi, Andiswa Bukula, Jian Yun Zhuang, En-Shiun Annie Lee, Tadesse Kebede Guge, Israel Abebe Azime, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson KoffiKalipe, Jonathan Mukiibi, Salomon Kabongo Kabenamualu, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, and 2 others. 2025. INJONGO: A multicultural intent detection and slot-filling dataset for 16 African languages. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9429–9452, Vienna, Austria. Association for Computational Linguistics.

A Image Categories

A 图像类别

A) Language wise category distribution

A) 分语言的类别分布

Stacked bar chart of image category counts by language in Afri-MCQA

Figure 8: Language-wise distribution of the categories.

图 8:各类别在不同语言中的分布。

B) Image category distribution

B) 图像类别分布

Horizontal bar charts of the top six image categories by language in Afri-MCQA

Figure 9: Top 6 Image category distribution

图 9:排名前 6 的图像类别分布

B Prompt

B 提示

For each QA format, we use the same prompt templates to ensure consistency across models and languages. We evaluated models under two distinct prompt conditions to assess the impact of visual and contextual grounding:

对每种问答形式,我们使用相同的提示模板,以保证跨模型和跨语言的一致性。我们在两种不同的提示条件下评测模型,以衡量视觉依据和上下文依据的影响:

  • Image-grounded: [Image] Question: {q} Options: {opts}
  • 以图像为根据:[Image] Question: {q} Options: {opts}
  • Image + Location: [Image] Location: {country}. Question: {q} Options: {opts}
  • 图像 + 地点:[Image] Location: {country}. Question: {q} Options: {opts}

Figures 10 & 11 show the location-aware prompts we used in both setups for audio and text modalities.

图 10 和图 11 给出了我们在音频和文本两种设定中使用的带地点信息的提示。

Location-aware text prompts for multiple-choice and open-ended visual question answering

Figure 10: Location-aware text prompts.

图 10:带地点信息的文本提示。

Location-aware audio prompts for multiple-choice and open-ended visual question answering

Figure 11: Location-aware audio prompts.

图 11:带地点信息的音频提示。

C Additional Experiments results

C 补充实验结果

C.1 Text-based QA

C.1 基于文本的问答

In this section we report detailed results for location-aware and Image only prompts and per country-language results.

本节报告带地点信息提示与仅图像提示的详细结果,以及按国家–语言划分的结果。

Table 5: Text Prompt - Location/Language Aware–English

Country-Lang G-12B G-27B G-3n-2B G-3n-4B Q-2.5 3B Q-2.5 7B Gemni-2.5-pro
Ethiopia-amh 50.85 50.34 41.23 40.54 42.61 41.23 76.11
Nigeria-hau 63.17 62.47 55.14 58.21 54.79 57.2 81.16
Nigeria-ibo 50.19 49.8 43.72 40.39 41.37 42.15 79.8
Uganda-lug 61.65 62.23 56.92 60.48 49.32 53.75 84.15
Ethiopia-orm 55.03 54.48 44.76 51.5 45.93 45.73 76.35
Rwanda-kin 67.97 68.16 56.92 60.48 55.05 55.61 82.27
Kenya-kik 58.83 58.38 47.67 54.34 46.63 47.67 87.87
Somali-som 56.36 56.23 50 54.33 50.27 45.52 68.02
Eritrea-tir 57.54 57.89 45.05 51.22 50.17 50.52 76.66
Ghana-twi 62.99 63.15 56.93 62.27 54.7 56.14 81.33
Nigeria-yor 55.61 55.24 49.9 53.63 48.68 46.62 82.95
Botswana-tsn 60.41 61.74 50.94 55.3 53.21 53.03 82.54
Malawi-nya 47.91 47.43 40.7 43.42 41.98 37.98 66.02
S.Africa-zul 58.9 59.97 59.13 58.66 55.55 51.49 67.38
Lesotho-sot 74.47 74.63 74.63 74.47 70.08 61.62 87.64
Average 58.79 58.80 51.54 55.23 50.68 49.75 78.66

表 5:文本提示——带地点/语言信息,英语

Country-Lang G-12B G-27B G-3n-2B G-3n-4B Q-2.5 3B Q-2.5 7B Gemni-2.5-pro
Ethiopia-amh 50.85 50.34 41.23 40.54 42.61 41.23 76.11
Nigeria-hau 63.17 62.47 55.14 58.21 54.79 57.2 81.16
Nigeria-ibo 50.19 49.8 43.72 40.39 41.37 42.15 79.8
Uganda-lug 61.65 62.23 56.92 60.48 49.32 53.75 84.15
Ethiopia-orm 55.03 54.48 44.76 51.5 45.93 45.73 76.35
Rwanda-kin 67.97 68.16 56.92 60.48 55.05 55.61 82.27
Kenya-kik 58.83 58.38 47.67 54.34 46.63 47.67 87.87
Somali-som 56.36 56.23 50 54.33 50.27 45.52 68.02
Eritrea-tir 57.54 57.89 45.05 51.22 50.17 50.52 76.66
Ghana-twi 62.99 63.15 56.93 62.27 54.7 56.14 81.33
Nigeria-yor 55.61 55.24 49.9 53.63 48.68 46.62 82.95
Botswana-tsn 60.41 61.74 50.94 55.3 53.21 53.03 82.54
Malawi-nya 47.91 47.43 40.7 43.42 41.98 37.98 66.02
S.Africa-zul 58.9 59.97 59.13 58.66 55.55 51.49 67.38
Lesotho-sot 74.47 74.63 74.63 74.47 70.08 61.62 87.64
Average 58.79 58.80 51.54 55.23 50.68 49.75 78.66

Table 6: Text Prompt - Location/Language Aware–native

Country-Lang G-12B G-27B G-3n-2B G-3n-4B Q-2.5 3B Q-2.5 7B Gemni-2.5-pro
Ethiopia-amh 44.5 43.64 42.09 42.78 30.93 29.38 78.17
Nigeria-hau 47.12 49.38 40.31 42.58 28.44 29.12 81.67
Nigeria-ibo 41.17 41.56 38.43 36.86 30.39 31.56 79.41
Uganda-lug 48.36 48.16 44.89 43.54 35.26 35.26 81.31
Ethiopia-orm 39.34 38.56 35.65 36.24 33.13 34.49 75.77
Rwanda-kin 57.11 58.05 53.55 53.18 39.7 39.13 82.58
Kenya-kik 44.42 44.64 40.6 41.41 35.25 36.76 85.05
Somali-som 40.37 41.19 43.22 42.68 27.5 25.06 66.12
Eritrea-tir 47.01 46.84 41.57 40.52 32.28 34.56 75.96
Ghana-twi 45.13 45.61 37.79 37.48 29.5 29.34 77.51
Nigeria-yor 44.19 43.82 38.36 38.36 31.08 31.27 81.27
Botswana-tsn 39.77 40.15 36.93 37.87 30.3 32.38 83.52
Malawi-nya 36.85 35.73 34.15 53.18 28.36 28.68 68.58
S.Africa-zul 38.23 39.9 39.18 39.42 27.8 22.46 61.88
Lesotho-sot 37.39 37.07 38.21 36.26 25.56 29.91 65.36
Average 43.39 43.62 40.49 41.49 31.03 31.29 76.27

表 6:文本提示——带地点/语言信息,母语

Country-Lang G-12B G-27B G-3n-2B G-3n-4B Q-2.5 3B Q-2.5 7B Gemni-2.5-pro
Ethiopia-amh 44.5 43.64 42.09 42.78 30.93 29.38 78.17
Nigeria-hau 47.12 49.38 40.31 42.58 28.44 29.12 81.67
Nigeria-ibo 41.17 41.56 38.43 36.86 30.39 31.56 79.41
Uganda-lug 48.36 48.16 44.89 43.54 35.26 35.26 81.31
Ethiopia-orm 39.34 38.56 35.65 36.24 33.13 34.49 75.77
Rwanda-kin 57.11 58.05 53.55 53.18 39.7 39.13 82.58
Kenya-kik 44.42 44.64 40.6 41.41 35.25 36.76 85.05
Somali-som 40.37 41.19 43.22 42.68 27.5 25.06 66.12
Eritrea-tir 47.01 46.84 41.57 40.52 32.28 34.56 75.96
Ghana-twi 45.13 45.61 37.79 37.48 29.5 29.34 77.51
Nigeria-yor 44.19 43.82 38.36 38.36 31.08 31.27 81.27
Botswana-tsn 39.77 40.15 36.93 37.87 30.3 32.38 83.52
Malawi-nya 36.85 35.73 34.15 53.18 28.36 28.68 68.58
S.Africa-zul 38.23 39.9 39.18 39.42 27.8 22.46 61.88
Lesotho-sot 37.39 37.07 38.21 36.26 25.56 29.91 65.36
Average 43.39 43.62 40.49 41.49 31.03 31.29 76.27

Table 7: Prompt - Image only–English(text-based)

Country-Lang G-12B G-27B G-3n-2B G-3n-4B Q-2.5 3B Q-2.5 7B Gemni-2.5-pro
Ethiopia-amh 50.34 50.51 42.09 44.5 42.09 40.72 72.23
Nigeria-hau 64.92 65.44 56.36 57.06 57.59 58.87 77.31
Nigeria-ibo 48.43 47.05 44.11 47.64 37.64 40.98 77.45
Uganda-lug 61.65 60.88 56.84 58.76 49.51 53.56 80.73
Ethiopia-orm 52.32 50.58 46.51 50.96 44.18 44.37 77.32
Rwanda-kin 68.53 67.6 57.86 60.48 54.86 56.92 78.27
Kenya-kik 56.56 55.75 49.09 54.14 42.02 47.47 86.46
Somali-som 56.36 57.99 50.81 54.06 49.86 46.06 64.09
Eritrea-tir 57.01 56.49 41.01 47.89 48.94 50 73.15
Ghana-twi 63.31 63.15 59.01 63.95 55.34 56.45 81.49
Nigeria-yor 55.8 57.11 52.14 53.44 47.37 47 82.2
Botswana-tsn 61.74 61.17 53.21 55.11 51.7 52.46 80.8
Malawi-nya 46.15 46.79 43.75 45.19 42.14 37.82 65.54
S.Africa-zul 64.63 66.06 65.23 64.99 48.5 53.52 88.76
Lesotho-sot 75.28 74.95 75.28 74.63 53.82 60.16 92.5
Average 58.86 58.76 53.28 55.52 48.37 49.75 78.56

表 7:仅图像提示——英语(基于文本)

Country-Lang G-12B G-27B G-3n-2B G-3n-4B Q-2.5 3B Q-2.5 7B Gemni-2.5-pro
Ethiopia-amh 50.34 50.51 42.09 44.5 42.09 40.72 72.23
Nigeria-hau 64.92 65.44 56.36 57.06 57.59 58.87 77.31
Nigeria-ibo 48.43 47.05 44.11 47.64 37.64 40.98 77.45
Uganda-lug 61.65 60.88 56.84 58.76 49.51 53.56 80.73
Ethiopia-orm 52.32 50.58 46.51 50.96 44.18 44.37 77.32
Rwanda-kin 68.53 67.6 57.86 60.48 54.86 56.92 78.27
Kenya-kik 56.56 55.75 49.09 54.14 42.02 47.47 86.46
Somali-som 56.36 57.99 50.81 54.06 49.86 46.06 64.09
Eritrea-tir 57.01 56.49 41.01 47.89 48.94 50 73.15
Ghana-twi 63.31 63.15 59.01 63.95 55.34 56.45 81.49
Nigeria-yor 55.8 57.11 52.14 53.44 47.37 47 82.2
Botswana-tsn 61.74 61.17 53.21 55.11 51.7 52.46 80.8
Malawi-nya 46.15 46.79 43.75 45.19 42.14 37.82 65.54
S.Africa-zul 64.63 66.06 65.23 64.99 48.5 53.52 88.76
Lesotho-sot 75.28 74.95 75.28 74.63 53.82 60.16 92.5
Average 58.86 58.76 53.28 55.52 48.37 49.75 78.56

Table 8: Image only-native (text-based)

Country-Lang G-12B G-27B G-3n-2B G-3n-4B Q-2.5 3B Q-2.5 7B Gemni-2.5-pro
Ethiopia-amh 43.81 41.76 43.29 43.47 29.73 29.89 76.8
Nigeria-hau 46.94 46.77 43.45 44.5 30.89 32.16 81.32
Nigeria-ibo 40.58 41.76 38.43 37.64 30 32.94 79.01
Uganda-lug 50.86 50.48 45.08 44.31 35.07 36.22 81.11
Ethiopia-orm 39.53 38.56 34.68 37.2 30.04 31 76.55
Rwanda-kin 56.92 57.67 55.24 55.24 34.83 37.82 80.71
Kenya-kik 45.85 45.45 40.6 41.61 33.74 37.17 84.84
Somali-som 41.19 40.78 43.9 45.12 27.77 24.93 65.62
Eritrea-tir 47.01 47.01 39.82 41.05 28.07 33.5 73.28
Ghana-twi 45.45 45.61 36.52 31.57 29.18 29.66 75.91
Nigeria-yor 44.19 44 39.29 30.71 32.02 32.95 81.23
Botswana-tsn 42.04 40.34 38.06 31.43 28.4 31.81 81.06
Malawi-nya 38.94 37.17 36.21 35.09 28.68 28.84 70.19
S.Africa-zul 39.66 39.54 39.78 39.3 29.39 22.46 40
Lesotho-sot 37.07 37.72 38.37 37.2 28.34 30.56 47.64
Average 44.00 43.64 41.05 41.14 30.27 31.46 73.01

表 8:仅图像——母语(基于文本)

Country-Lang G-12B G-27B G-3n-2B G-3n-4B Q-2.5 3B Q-2.5 7B Gemni-2.5-pro
Ethiopia-amh 43.81 41.76 43.29 43.47 29.73 29.89 76.8
Nigeria-hau 46.94 46.77 43.45 44.5 30.89 32.16 81.32
Nigeria-ibo 40.58 41.76 38.43 37.64 30 32.94 79.01
Uganda-lug 50.86 50.48 45.08 44.31 35.07 36.22 81.11
Ethiopia-orm 39.53 38.56 34.68 37.2 30.04 31 76.55
Rwanda-kin 56.92 57.67 55.24 55.24 34.83 37.82 80.71
Kenya-kik 45.85 45.45 40.6 41.61 33.74 37.17 84.84
Somali-som 41.19 40.78 43.9 45.12 27.77 24.93 65.62
Eritrea-tir 47.01 47.01 39.82 41.05 28.07 33.5 73.28
Ghana-twi 45.45 45.61 36.52 31.57 29.18 29.66 75.91
Nigeria-yor 44.19 44 39.29 30.71 32.02 32.95 81.23
Botswana-tsn 42.04 40.34 38.06 31.43 28.4 31.81 81.06
Malawi-nya 38.94 37.17 36.21 35.09 28.68 28.84 70.19
S.Africa-zul 39.66 39.54 39.78 39.3 29.39 22.46 40
Lesotho-sot 37.07 37.72 38.37 37.2 28.34 30.56 47.64
Average 44.00 43.64 41.05 41.14 30.27 31.46 73.01

Table 9: Location/Language Aware-English(Open-endded VQA -text based)

Country-Lang G-12B chrf++ G-12B LLM G-27B chrf++ G-27B LLM Q-2.5 3B chrf++ Q-2.5 3B LLM Q-2.5 7B chrf++ Q-2.5 7B LLM G-3n-2B chrf++ G-3n-2B LLM G-3n-4B chrf++ G-3n-4B LLM Gemni chrf++ Gemni LLM
Ethiopia-amh 27 26.2 27.55 26.8 17.69 15 20.7 19.2 20.73 16.4 31.53 30.4 47.88 60.2
Nigeria-hau 7.28 12.63 7.47 13.43 7.73 7.41 7.7 7.82 10.82 8.45 13.43 11.02 14.63 25.5
Nigeria-ibo 10.41 10.54 11.04 12.13 10.31 10.14 11.8 8.55 9.41 7.75 15.65 15.9 25.61 36.18
Uganda-lug 20.64 25.45 22.37 27.04 15.38 16.1 17.51 18.29 17.1 18.89 27.91 33.2 35.7 44.33
Ethiopia-orm 18.44 17.19 19.24 19.34 16.86 11.72 20.08 17.58 19.2 16.41 27.38 25 34.01 38.09
Rwanda-kin 12.51 18.76 12.06 18.36 11.37 10.18 12.51 12.38 10.82 16.38 16.37 22.92 18.92 36.53
Kenya-kik 17.77 21.82 18.2 23.64 14.42 11.31 16.54 12.12 16.73 15.15 30.33 31.11 41.2 49.49
Somali-som(so) 13.77 15.94 13.4 16.93 12.59 13.55 15.03 17.73 13.55 16.33 17.04 18.73 18.62 25.5
Eritrea-tir 20.11 28.75 20.47 29.29 14.82 19.17 17.87 22.24 14.64 19.23 24.22 30.02 35.75 49.01
Ghana-twi 20.37 29.05 21.01 29.24 16.83 23.46 18.87 23.09 19.41 24.58 25.71 32.4 25.71 34.64
Nigeria-yor 10.2 17.27 9.38 17.87 9.13 13.86 9.92 12.25 10.12 15 13.13 20.6 20.61 43.98
Botswana-tsn 16.14 20.96 15.16 19.66 14.93 17.37 16.34 20.56 15.55 18.56 20.59 27.94 22.98 28.14
Malawi-nya 16.41 15.17 15.56 15.57 11.08 17.37 13.39 12.77 13.42 18.56 20.72 22.75 25.24 30.34
S.Africa-zul 18.48 15.72 17.94 16.1 11.48 8.7 12.05 9.28 17.49 15.72 18.2 16.67 34.87 39.58
Lesotho-sot 12.56 11.26 14.01 11.82 11.49 8.44 10.69 8.63 13.37 11.44 14.16 12.01 24.7 25.7
Average 16.13 19.14 16.32 19.81 13.07 13.58 14.73 8.56 14.61 15.83 20.75 23.54 28.43 37.80

表 9:带地点/语言信息——英语(开放式视觉问答,基于文本)

Country-Lang G-12B chrf++ G-12B LLM G-27B chrf++ G-27B LLM Q-2.5 3B chrf++ Q-2.5 3B LLM Q-2.5 7B chrf++ Q-2.5 7B LLM G-3n-2B chrf++ G-3n-2B LLM G-3n-4B chrf++ G-3n-4B LLM Gemni chrf++ Gemni LLM
Ethiopia-amh 27 26.2 27.55 26.8 17.69 15 20.7 19.2 20.73 16.4 31.53 30.4 47.88 60.2
Nigeria-hau 7.28 12.63 7.47 13.43 7.73 7.41 7.7 7.82 10.82 8.45 13.43 11.02 14.63 25.5
Nigeria-ibo 10.41 10.54 11.04 12.13 10.31 10.14 11.8 8.55 9.41 7.75 15.65 15.9 25.61 36.18
Uganda-lug 20.64 25.45 22.37 27.04 15.38 16.1 17.51 18.29 17.1 18.89 27.91 33.2 35.7 44.33
Ethiopia-orm 18.44 17.19 19.24 19.34 16.86 11.72 20.08 17.58 19.2 16.41 27.38 25 34.01 38.09
Rwanda-kin 12.51 18.76 12.06 18.36 11.37 10.18 12.51 12.38 10.82 16.38 16.37 22.92 18.92 36.53
Kenya-kik 17.77 21.82 18.2 23.64 14.42 11.31 16.54 12.12 16.73 15.15 30.33 31.11 41.2 49.49
Somali-som(so) 13.77 15.94 13.4 16.93 12.59 13.55 15.03 17.73 13.55 16.33 17.04 18.73 18.62 25.5
Eritrea-tir 20.11 28.75 20.47 29.29 14.82 19.17 17.87 22.24 14.64 19.23 24.22 30.02 35.75 49.01
Ghana-twi 20.37 29.05 21.01 29.24 16.83 23.46 18.87 23.09 19.41 24.58 25.71 32.4 25.71 34.64
Nigeria-yor 10.2 17.27 9.38 17.87 9.13 13.86 9.92 12.25 10.12 15 13.13 20.6 20.61 43.98
Botswana-tsn 16.14 20.96 15.16 19.66 14.93 17.37 16.34 20.56 15.55 18.56 20.59 27.94 22.98 28.14
Malawi-nya 16.41 15.17 15.56 15.57 11.08 17.37 13.39 12.77 13.42 18.56 20.72 22.75 25.24 30.34
S.Africa-zul 18.48 15.72 17.94 16.1 11.48 8.7 12.05 9.28 17.49 15.72 18.2 16.67 34.87 39.58
Lesotho-sot 12.56 11.26 14.01 11.82 11.49 8.44 10.69 8.63 13.37 11.44 14.16 12.01 24.7 25.7
Average 16.13 19.14 16.32 19.81 13.07 13.58 14.73 8.56 14.61 15.83 20.75 23.54 28.43 37.80

Table 10: Location/Language Aware-native(Open-ended VQA -text based)

Country-Lang G-12B chrf++ G-12B LLM G-27B chrf++ G-27B LLM Q-2.5 3B chrf++ Q-2.5 3B LLM Q-2.5 7B chrf++ Q-2.5 7B LLM G-3n-2B chrf++ G-3n-2B LLM G-3n-4B chrf++ G-3n-4B LLM Gemni chrf++ Gemni LLM
Ethiopia-amh 2.47 22.8 2.91 22.6 0.11 6.6 1.01 4.2 0 1 0.71 20.4 42.17 55.4
Nigeria-hau 4.23 3.01 4.31 3.21 2.79 0.6 2.9 0.6 1.41 2.4 3.83 2.6 15.02 30.26
Nigeria-ibo 5.93 5.96 5.93 5.96 3.65 0.99 4.97 0.99 0.44 0.99 5.42 2.98 24.11 33
Uganda-lug 9.48 9.96 9.22 11.16 5.69 3.19 4.27 3.19 0.5 1.79 8.45 8.76 30.33 40.44
Ethiopia-orm 6.56 6.45 6.35 5.86 5.26 6.05 3.26 1.76 0.68 0.59 6.27 4.69 37.92 39.84
Rwanda-kin 8.31 9.38 8.23 9.58 4.8 3.39 3.65 3.02 0.39 0.2 6.24 7.19 25.27 34.13
Kenya-kik 7.47 3.03 7.82 30.3 5.81 1.21 4.03 0.61 1.84 1.21 9.31 3.03 39.17 47.37
Somali-som 7.85 7.57 7.9 7.37 3.72 1.39 4.42 2.19 2.17 1.99 5.64 4.78 22.59 30.48
Eritrea-tir 0.85 3.98 0.95 4.34 0.04 5.79 0.43 5.79 0.01 0.18 0.13 14.1 27.23 37.43
Ghana-twi 6.19 5.77 6.3 6.89 4.92 2.42 4.43 2.61 1.88 0.74 6.89 5.21 31.63 33.89
Nigeria-yor 3.91 6.43 3.72 5.62 2.25 1 2.3 1.2 1.22 0.6 4.3 7.8 22.33 46.18
Botswana-tsn 7.91 4.39 7.11 4.19 4.4 2.79 4.15 2 2.04 0.6 7.26 5.99 34.41 39.75
Malawi-nya 8.99 5.19 8.44 4.99 4.23 0.4 3.43 0.8 1.71 0.4 9.33 3.59 31.5 34.13
S.Africa-zul 13.54 10.04 12.72 8.33 4.68 1.33 5.21 4.88 14.32 10.42 14.75 10.42 41.13 44.51
Lesotho-sot 7.6 3 7.15 2.81 4.73 3 8.08 4.88 7.77 3.38 8 3 28.67 29.83
Average 6.75 7.13 6.60 6.88 3.80 2.67 3.73 2.68 2.42 1.76 6.43 6.93 30.22 38.43

表 10:带地点/语言信息——母语(开放式视觉问答,基于文本)

Country-Lang G-12B chrf++ G-12B LLM G-27B chrf++ G-27B LLM Q-2.5 3B chrf++ Q-2.5 3B LLM Q-2.5 7B chrf++ Q-2.5 7B LLM G-3n-2B chrf++ G-3n-2B LLM G-3n-4B chrf++ G-3n-4B LLM Gemni chrf++ Gemni LLM
Ethiopia-amh 2.47 22.8 2.91 22.6 0.11 6.6 1.01 4.2 0 1 0.71 20.4 42.17 55.4
Nigeria-hau 4.23 3.01 4.31 3.21 2.79 0.6 2.9 0.6 1.41 2.4 3.83 2.6 15.02 30.26
Nigeria-ibo 5.93 5.96 5.93 5.96 3.65 0.99 4.97 0.99 0.44 0.99 5.42 2.98 24.11 33
Uganda-lug 9.48 9.96 9.22 11.16 5.69 3.19 4.27 3.19 0.5 1.79 8.45 8.76 30.33 40.44
Ethiopia-orm 6.56 6.45 6.35 5.86 5.26 6.05 3.26 1.76 0.68 0.59 6.27 4.69 37.92 39.84
Rwanda-kin 8.31 9.38 8.23 9.58 4.8 3.39 3.65 3.02 0.39 0.2 6.24 7.19 25.27 34.13
Kenya-kik 7.47 3.03 7.82 30.3 5.81 1.21 4.03 0.61 1.84 1.21 9.31 3.03 39.17 47.37
Somali-som 7.85 7.57 7.9 7.37 3.72 1.39 4.42 2.19 2.17 1.99 5.64 4.78 22.59 30.48
Eritrea-tir 0.85 3.98 0.95 4.34 0.04 5.79 0.43 5.79 0.01 0.18 0.13 14.1 27.23 37.43
Ghana-twi 6.19 5.77 6.3 6.89 4.92 2.42 4.43 2.61 1.88 0.74 6.89 5.21 31.63 33.89
Nigeria-yor 3.91 6.43 3.72 5.62 2.25 1 2.3 1.2 1.22 0.6 4.3 7.8 22.33 46.18
Botswana-tsn 7.91 4.39 7.11 4.19 4.4 2.79 4.15 2 2.04 0.6 7.26 5.99 34.41 39.75
Malawi-nya 8.99 5.19 8.44 4.99 4.23 0.4 3.43 0.8 1.71 0.4 9.33 3.59 31.5 34.13
S.Africa-zul 13.54 10.04 12.72 8.33 4.68 1.33 5.21 4.88 14.32 10.42 14.75 10.42 41.13 44.51
Lesotho-sot 7.6 3 7.15 2.81 4.73 3 8.08 4.88 7.77 3.38 8 3 28.67 29.83
Average 6.75 7.13 6.60 6.88 3.80 2.67 3.73 2.68 2.42 1.76 6.43 6.93 30.22 38.43

Table 11: Image only-english(Open-endded VQA -text based)

Country-Lang G-12B chrf++ G-12B LLM G-27B chrf++ G-27B LLM Q-2.5 3B chrf++ Q-2.5 3B LLM Q-2.5 7B chrf++ Q-2.5 7B LLM G-3n-2B chrf++ G-3n-2B LLM G-3n-4B chrf++ G-3n-4B LLM Gemni chrf++ Gemni LLM
Ethiopia-amh 19.86 15.8 20.04 15.8 10.38 6.6 11.76 9.2 15.42 10 24.62 20.8 40.41 43.8
Nigeria-hau 6.28 12.63 6.53 11.62 7.98 6.8 7.98 7.01 7.36 8.02 7.35 12.02 9.37 16.83
Nigeria-ibo 10.57 8.55 9.95 8.15 9.78 3.76 10.5 6.76 8.97 5.96 13.27 11.13 24.63 30.82
Uganda-lug 20.59 24.25 19.68 22.47 13.85 13.92 15.89 15.9 16.43 18.49 27.36 30.2 49.04 48.21
Ethiopia-orm 18.79 15.04 19.01 14.45 12.29 6.84 14.68 11.33 14.08 8.01 23.32 19.14 37.92 36.52
Rwanda-kin 11.41 15.97 11.22 15.37 10.29 7.39 11.38 10.18 9.41 10.78 15.15 18.16 23.07 32.93
Kenya-kik 18.55 20.81 18.2 20.2 12.03 7.68 15.97 11.11 15.12 13.94 28.99 28.48 49.78 59.19
Somali-som 14.79 18.53 15.4 19.12 8.03 14.94 12.45 17.13 12.97 15.34 18.04 21.71 26.79 32.07
Eritrea-tir 14.71 16.09 14.64 15.19 8 6.87 9.84 7.59 10.71 7.78 18.09 12.84 31.98 40.14
Ghana-twi 21.95 29.8 20.66 29.24 8.85 18.44 15.78 21.23 26.57 24.95 25.57 33.33 36.76 47.87
Nigeria-yor 10.87 17.87 10.61 17.47 7.02 11.45 8.89 11.24 9.25 14.4 13.48 21 23.2 45.38
Botswana-tsn 15.75 22.95 16.14 22.75 9.27 17.76 14.95 22.16 13.77 18.36 19.43 25.35 30.27 39.72
Malawi-nya 12.25 9.78 12.97 11.38 8.7 6.19 9.88 6.19 15.33 8.18 15.33 12.77 26.35 26.35
S.Africa-zul 16.12 13.25 15.77 11.93 10.61 7.1 10.7 8.52 15.65 12.69 16.36 13.07 40.02 40.34
Lesotho-sot 11.99 9.19 11.75 9.57 10.59 6.94 10.55 9.01 12.71 9.94 12.33 9.38 23.31 23.83
Average 14.97 16.7 14.84 16.31 9.84 9.51 12.08 11.64 13.58 12.46 19.1 19.29 31.53 37.6

表 11:仅图像——英语(开放式视觉问答,基于文本)

Country-Lang G-12B chrf++ G-12B LLM G-27B chrf++ G-27B LLM Q-2.5 3B chrf++ Q-2.5 3B LLM Q-2.5 7B chrf++ Q-2.5 7B LLM G-3n-2B chrf++ G-3n-2B LLM G-3n-4B chrf++ G-3n-4B LLM Gemni chrf++ Gemni LLM
Ethiopia-amh 19.86 15.8 20.04 15.8 10.38 6.6 11.76 9.2 15.42 10 24.62 20.8 40.41 43.8
Nigeria-hau 6.28 12.63 6.53 11.62 7.98 6.8 7.98 7.01 7.36 8.02 7.35 12.02 9.37 16.83
Nigeria-ibo 10.57 8.55 9.95 8.15 9.78 3.76 10.5 6.76 8.97 5.96 13.27 11.13 24.63 30.82
Uganda-lug 20.59 24.25 19.68 22.47 13.85 13.92 15.89 15.9 16.43 18.49 27.36 30.2 49.04 48.21
Ethiopia-orm 18.79 15.04 19.01 14.45 12.29 6.84 14.68 11.33 14.08 8.01 23.32 19.14 37.92 36.52
Rwanda-kin 11.41 15.97 11.22 15.37 10.29 7.39 11.38 10.18 9.41 10.78 15.15 18.16 23.07 32.93
Kenya-kik 18.55 20.81 18.2 20.2 12.03 7.68 15.97 11.11 15.12 13.94 28.99 28.48 49.78 59.19
Somali-som 14.79 18.53 15.4 19.12 8.03 14.94 12.45 17.13 12.97 15.34 18.04 21.71 26.79 32.07
Eritrea-tir 14.71 16.09 14.64 15.19 8 6.87 9.84 7.59 10.71 7.78 18.09 12.84 31.98 40.14
Ghana-twi 21.95 29.8 20.66 29.24 8.85 18.44 15.78 21.23 26.57 24.95 25.57 33.33 36.76 47.87
Nigeria-yor 10.87 17.87 10.61 17.47 7.02 11.45 8.89 11.24 9.25 14.4 13.48 21 23.2 45.38
Botswana-tsn 15.75 22.95 16.14 22.75 9.27 17.76 14.95 22.16 13.77 18.36 19.43 25.35 30.27 39.72
Malawi-nya 12.25 9.78 12.97 11.38 8.7 6.19 9.88 6.19 15.33 8.18 15.33 12.77 26.35 26.35
S.Africa-zul 16.12 13.25 15.77 11.93 10.61 7.1 10.7 8.52 15.65 12.69 16.36 13.07 40.02 40.34
Lesotho-sot 11.99 9.19 11.75 9.57 10.59 6.94 10.55 9.01 12.71 9.94 12.33 9.38 23.31 23.83
Average 14.97 16.7 14.84 16.31 9.84 9.51 12.08 11.64 13.58 12.46 19.1 19.29 31.53 37.6

Table 12: Image only-native(Open-endded VQA -text based)

Country-Lang G-12B chrf++ G-12B LLM G-27B chrf++ G-27B LLM Q-2.5 3B chrf++ Q-2.5 3B LLM Q-2.5 7B chrf++ Q-2.5 7B LLM G-3n-2B chrf++ G-3n-2B LLM G-3n-4B chrf++ G-3n-4B LLM Gemni chrf++ Gemni LLM
Ethiopia-amh 4.65 17.2 3.78 17.6 3.5 3.2 0.68 2.4 1.51 14.6 1.21 13.8 42.59 54.8
Nigeria-hau 3.42 3.01 3.83 2 3.12 3.41 3.12 1.4 3.77 3.81 3.44 3.81 14.67 29.66
Nigeria-ibo 5.21 4.17 5.31 2.98 2.53 2.78 5.32 0.99 5.41 2.58 5.67 2.19 23.2 29.62
Uganda-lug 9.17 10.96 8.14 9.96 4.28 4.38 4.72 4.58 8.1 8.96 7.95 7.57 31.54 39.04
Ethiopia-orm 6.28 7.03 5.94 6.45 2.01 2.15 4.68 2.93 5.67 3.32 5.54 2.73 29.32 38.48
Rwanda-kin 7.66 9.98 7.59 9.78 2.36 2.59 3.98 2.2 6.69 8.58 6.19 7.39 25.51 35.33
Kenya-kik 7.58 3.23 7.57 3.03 4.53 4.04 6.11 1.62 8.7 2.22 8.9 2.63 39.23 48.08
Somali-som 7.56 8.17 7.3 8.76 3.29 2.39 6.21 3.78 5.73 5.78 5.7 4.98 25.02 32.07
Eritrea-tir 1.14 3.62 1.21 3.8 0.12 1.63 0.19 0.72 0.74 4.16 0.69 4.7 27.03 35.62
Ghana-twi 6.52 6.15 6.24 7.08 5.03 2.42 5.74 4.28 6.44 4.66 6.21 4.66 29.44 32.22
Nigeria-yor 3.76 6.22 3.94 6.92 2.3 0.6 3.33 0.8 4.3 8.2 4.22 8 20.6 43.73
Botswana-tsn 6.74 4.99 6.48 5.59 2.97 2.99 5.86 8.58 6.6 4.19 6.95 4.59 34.32 36.93
Malawi-nya 8.78 5.99 8.3 1.2 3.31 1 5.36 2.79 8.35 3.79 8.33 3.19 29.83 31.94
S.Africa-zul 14.11 9.66 14.68 10.61 3.44 2.65 3.72 1.52 13.78 10.23 14.06 9.66 40.69 42.8
Lesotho-sot 7.28 3.75 7.76 4.69 3.13 2.25 6.88 10.69 7.55 3.19 7.04 3.19 24.95 25.7
Average 6.66 6.94 6.54 6.7 2.97 2.57 4.39 3.29 6.22 5.88 6.14 5.54 29.20 37.07

表 12:仅图像——母语(开放式视觉问答,基于文本)

Country-Lang G-12B chrf++ G-12B LLM G-27B chrf++ G-27B LLM Q-2.5 3B chrf++ Q-2.5 3B LLM Q-2.5 7B chrf++ Q-2.5 7B LLM G-3n-2B chrf++ G-3n-2B LLM G-3n-4B chrf++ G-3n-4B LLM Gemni chrf++ Gemni LLM
Ethiopia-amh 4.65 17.2 3.78 17.6 3.5 3.2 0.68 2.4 1.51 14.6 1.21 13.8 42.59 54.8
Nigeria-hau 3.42 3.01 3.83 2 3.12 3.41 3.12 1.4 3.77 3.81 3.44 3.81 14.67 29.66
Nigeria-ibo 5.21 4.17 5.31 2.98 2.53 2.78 5.32 0.99 5.41 2.58 5.67 2.19 23.2 29.62
Uganda-lug 9.17 10.96 8.14 9.96 4.28 4.38 4.72 4.58 8.1 8.96 7.95 7.57 31.54 39.04
Ethiopia-orm 6.28 7.03 5.94 6.45 2.01 2.15 4.68 2.93 5.67 3.32 5.54 2.73 29.32 38.48
Rwanda-kin 7.66 9.98 7.59 9.78 2.36 2.59 3.98 2.2 6.69 8.58 6.19 7.39 25.51 35.33
Kenya-kik 7.58 3.23 7.57 3.03 4.53 4.04 6.11 1.62 8.7 2.22 8.9 2.63 39.23 48.08
Somali-som 7.56 8.17 7.3 8.76 3.29 2.39 6.21 3.78 5.73 5.78 5.7 4.98 25.02 32.07
Eritrea-tir 1.14 3.62 1.21 3.8 0.12 1.63 0.19 0.72 0.74 4.16 0.69 4.7 27.03 35.62
Ghana-twi 6.52 6.15 6.24 7.08 5.03 2.42 5.74 4.28 6.44 4.66 6.21 4.66 29.44 32.22
Nigeria-yor 3.76 6.22 3.94 6.92 2.3 0.6 3.33 0.8 4.3 8.2 4.22 8 20.6 43.73
Botswana-tsn 6.74 4.99 6.48 5.59 2.97 2.99 5.86 8.58 6.6 4.19 6.95 4.59 34.32 36.93
Malawi-nya 8.78 5.99 8.3 1.2 3.31 1 5.36 2.79 8.35 3.79 8.33 3.19 29.83 31.94
S.Africa-zul 14.11 9.66 14.68 10.61 3.44 2.65 3.72 1.52 13.78 10.23 14.06 9.66 40.69 42.8
Lesotho-sot 7.28 3.75 7.76 4.69 3.13 2.25 6.88 10.69 7.55 3.19 7.04 3.19 24.95 25.7
Average 6.66 6.94 6.54 6.7 2.97 2.57 4.39 3.29 6.22 5.88 6.14 5.54 29.20 37.07

Table 13: Audio-Loc-Lang Aware(MC-VQA)

Country-Lang Eng Gemma3n-2B Eng Gemma3n-4B Eng Qwen-3B Eng Qwen-7B Eng Gemini-2.5 Pro Nat Gemma3n-2B Nat Gemma3n-4B Nat Qwen-3B Nat Qwen-7B Nat Gemini-2.5 Pro
Ghana-twi 37.78 44.35 58.96 62.83 87.01 28.83 32.11 31.48 31.27 69.12
Ethiopia-amh 34.38 36.17 42.43 43.60 83.52 29.48 32.97 30.03 28.63 82.46
Malawi-nya 27.60 33.17 42.96 44.30 80.09 26.52 27.98 29.62 32.08 70.80
Nigeria-hau 36.96 41.46 62.74 62.53 83.61 27.95 32.29 29.93 27.51 82.32
Nigeria-ibo 26.85 30.09 40.60 44.50 77.15 27.18 26.18 27.15 26.08 56.50
Kenya-kik 27.12 28.72 44.41 45.98 83.28 26.86 25.00 31.90 33.15 72.34
Rwanad-kin 45.78 51.40 59.07 63.06 85.16 30.69 32.07 29.31 31.44 78.13
Uganda-lug 39.64 46.10 47.65 50.56 81.73 33.87 31.81 34.39 38.38 72.48
Ethiopia-orm 37.55 39.20 41.23 42.35 74.52 29.11 28.87 26.49 27.06 71.46
Botswana-tsn 35.86 40.92 51.06 54.60 83.15 25.05 26.11 27.17 19.70 74.46
Somali-som 26.24 30.73 49.17 56.05 82.14 24.76 26.42 24.88 24.14 75.41
Eritrea-tir 36.55 44.30 46.98 52.30 75.48 28.27 29.76 31.26 34.33 72.35
Nigeria-yor 31.17 35.49 45.56 47.58 84.13 23.64 25.53 26.31 26.68 76.54
Lesotho-sot – – – – – 29.77 30.00 34.00 31.12 59.15
S.Africa-zul – – – – – 27.04 26.63 28.89 31.06 74.18
Average 34.11 38.62 48.68 51.56 81.61 27.86 29.01 29.52 29.51 73.41

表 13:音频——带地点/语言信息(MC-VQA)

Country-Lang Eng Gemma3n-2B Eng Gemma3n-4B Eng Qwen-3B Eng Qwen-7B Eng Gemini-2.5 Pro Nat Gemma3n-2B Nat Gemma3n-4B Nat Qwen-3B Nat Qwen-7B Nat Gemini-2.5 Pro
Ghana-twi 37.78 44.35 58.96 62.83 87.01 28.83 32.11 31.48 31.27 69.12
Ethiopia-amh 34.38 36.17 42.43 43.60 83.52 29.48 32.97 30.03 28.63 82.46
Malawi-nya 27.60 33.17 42.96 44.30 80.09 26.52 27.98 29.62 32.08 70.80
Nigeria-hau 36.96 41.46 62.74 62.53 83.61 27.95 32.29 29.93 27.51 82.32
Nigeria-ibo 26.85 30.09 40.60 44.50 77.15 27.18 26.18 27.15 26.08 56.50
Kenya-kik 27.12 28.72 44.41 45.98 83.28 26.86 25.00 31.90 33.15 72.34
Rwanad-kin 45.78 51.40 59.07 63.06 85.16 30.69 32.07 29.31 31.44 78.13
Uganda-lug 39.64 46.10 47.65 50.56 81.73 33.87 31.81 34.39 38.38 72.48
Ethiopia-orm 37.55 39.20 41.23 42.35 74.52 29.11 28.87 26.49 27.06 71.46
Botswana-tsn 35.86 40.92 51.06 54.60 83.15 25.05 26.11 27.17 19.70 74.46
Somali-som 26.24 30.73 49.17 56.05 82.14 24.76 26.42 24.88 24.14 75.41
Eritrea-tir 36.55 44.30 46.98 52.30 75.48 28.27 29.76 31.26 34.33 72.35
Nigeria-yor 31.17 35.49 45.56 47.58 84.13 23.64 25.53 26.31 26.68 76.54
Lesotho-sot – – – – – 29.77 30.00 34.00 31.12 59.15
S.Africa-zul – – – – – 27.04 26.63 28.89 31.06 74.18
Average 34.11 38.62 48.68 51.56 81.61 27.86 29.01 29.52 29.51 73.41

Table 14: MC-VQA Audio - Image Only

Country-Lang Eng Gemma3n-2B Eng Gemma3n-4B Eng Qwen-3B Eng Qwen-7B Eng Gemini-2.5 Pro Nat Gemma3n-2B Nat Gemma3n-4B Nat Qwen-3B Nat Qwen-7B Nat Gemini-2.5 Pro
Ghana-twi 33.67 46.40 57.23 62.19 87.75 27.61 24.74 32.55 32.51 65.16
Ethiopia-amh 31.01 35.05 45.18 44.58 79.72 28.17 28.38 31.66 29.91 79.69
Malawi-nya 25.18 32.44 41.81 48.50 77.72 28.71 29.44 37.07 26.86 68.87
Nigeria-hau 36.49 42.18 63.30 65.41 84.28 27.95 30.60 25.24 24.80 81.36
Nigeria-ibo 23.37 27.08 37.65 39.71 76.62 27.93 23.44 21.85 27.17 55.50
Kenya-kik 25.26 30.31 45.65 45.96 83.73 25.53 25.00 27.74 35.04 71.81
Rwanda-kin 44.24 48.59 57.54 61.33 81.79 28.28 28.97 30.83 31.00 79.24
Uganda-lug 36.74 44.09 45.49 45.94 78.65 32.49 31.12 35.64 35.20 69.27
Ethiopia-orm 34.50 32.62 37.40 42.50 71.15 27.68 28.16 26.01 26.78 71.22
Botswana-tsn 35.86 40.29 50.85 54.36 85.31 23.99 25.69 27.89 25.00 70.82
Somali-som 24.11 29.78 49.64 55.34 80.42 26.65 22.17 25.30 24.64 72.64
Eritrea-tir 33.11 36.34 47.29 47.67 74.51 26.98 27.19 32.10 31.97 70.39
Nigeria-yor 23.50 35.25 47.07 48.92 85.99 24.82 25.77 28.01 26.68 77.96
Lesotho-sot – – – – – – – 31.12 28.29 59.11
S.Africa-zul – – – – – – – 29.04 32.09 72.07
Average 31.31 36.96 48.16 50.02 80.59 27.45 26.98 29.47 29.20 71.01

表 14:音频 MC-VQA——仅图像

Country-Lang Eng Gemma3n-2B Eng Gemma3n-4B Eng Qwen-3B Eng Qwen-7B Eng Gemini-2.5 Pro Nat Gemma3n-2B Nat Gemma3n-4B Nat Qwen-3B Nat Qwen-7B Nat Gemini-2.5 Pro
Ghana-twi 33.67 46.40 57.23 62.19 87.75 27.61 24.74 32.55 32.51 65.16
Ethiopia-amh 31.01 35.05 45.18 44.58 79.72 28.17 28.38 31.66 29.91 79.69
Malawi-nya 25.18 32.44 41.81 48.50 77.72 28.71 29.44 37.07 26.86 68.87
Nigeria-hau 36.49 42.18 63.30 65.41 84.28 27.95 30.60 25.24 24.80 81.36
Nigeria-ibo 23.37 27.08 37.65 39.71 76.62 27.93 23.44 21.85 27.17 55.50
Kenya-kik 25.26 30.31 45.65 45.96 83.73 25.53 25.00 27.74 35.04 71.81
Rwanda-kin 44.24 48.59 57.54 61.33 81.79 28.28 28.97 30.83 31.00 79.24
Uganda-lug 36.74 44.09 45.49 45.94 78.65 32.49 31.12 35.64 35.20 69.27
Ethiopia-orm 34.50 32.62 37.40 42.50 71.15 27.68 28.16 26.01 26.78 71.22
Botswana-tsn 35.86 40.29 50.85 54.36 85.31 23.99 25.69 27.89 25.00 70.82
Somali-som 24.11 29.78 49.64 55.34 80.42 26.65 22.17 25.30 24.64 72.64
Eritrea-tir 33.11 36.34 47.29 47.67 74.51 26.98 27.19 32.10 31.97 70.39
Nigeria-yor 23.50 35.25 47.07 48.92 85.99 24.82 25.77 28.01 26.68 77.96
Lesotho-sot – – – – – – – 31.12 28.29 59.11
S.Africa-zul – – – – – – – 29.04 32.09 72.07
Average 31.31 36.96 48.16 50.02 80.59 27.45 26.98 29.47 29.20 71.01

Table 15: Audio Open-ended for English (Loc-Lang Aware)

Language G3n-2B chrf++ G3n-2B acc. G3n-4B chrf++ G3n-4B acc. Q-3B chrf++ Q-3B acc. Q-7B chrf++ Q-7B acc. Gemini-2.5 Pro chrf++ Gemini-2.5 Pro acc.
Ghana-twi 19.21 29.36 21.61 31.01 19.16 29.98 21.11 33.06 31.92 45.59
Ethiopia-amh 17.86 18.65 27.61 24.94 18.5 19.05 21.14 22.56 49.39 64.04
Malawi-nya 13.05 18.16 14.55 20.82 13.05 21.22 18.04 24.81 27.9 35.35
Nigeria-hau 13.02 14.89 11.34 18.68 13.26 14.42 17.28 19.76 17.99 31.21
Nigeria-ibo 12.31 6.94 12.09 9.03 11.26 7.89 14.48 12.6 30.9 42.13
Kenya-kik 17.87 14.47 18.89 18.16 15.86 14.47 19.21 18.23 44.58 55.26
Rwanda-kin 15.99 17.14 15.09 17.39 15.59 16.25 16.49 19.48 25.82 43.99
Uganda-lug 16.37 18.71 19.13 21.60 16.14 19.6 17.09 20.33 30.9 52.34
Ethiopia-orm 18.72 17.80 23.03 22.25 17.02 15.49 20.62 20.54 38.46 45.43
Botswana-tsn 18.15 24.10 19.27 25.16 16.48 23.33 17.61 23.56 26.79 36.81
Somali-som 14.39 21.51 16.39 23.17 15.02 18.29 17.54 23.99 23.05 32.39
Eritrea-tir 16.3 21.08 17.89 21.94 15.93 21.59 19.39 27.61 35.64 53.33
Nigeria-yor 11.55 16.31 11.91 18.71 11.3 16.63 12.81 18.99 25.07 49.64
Average 15.75 18.39 17.58 20.99 15.27 18.31 17.91 21.96 32.11 45.19

表 15:音频开放式问答,英语(带地点/语言信息)

Language G3n-2B chrf++ G3n-2B acc. G3n-4B chrf++ G3n-4B acc. Q-3B chrf++ Q-3B acc. Q-7B chrf++ Q-7B acc. Gemini-2.5 Pro chrf++ Gemini-2.5 Pro acc.
Ghana-twi 19.21 29.36 21.61 31.01 19.16 29.98 21.11 33.06 31.92 45.59
Ethiopia-amh 17.86 18.65 27.61 24.94 18.5 19.05 21.14 22.56 49.39 64.04
Malawi-nya 13.05 18.16 14.55 20.82 13.05 21.22 18.04 24.81 27.9 35.35
Nigeria-hau 13.02 14.89 11.34 18.68 13.26 14.42 17.28 19.76 17.99 31.21
Nigeria-ibo 12.31 6.94 12.09 9.03 11.26 7.89 14.48 12.6 30.9 42.13
Kenya-kik 17.87 14.47 18.89 18.16 15.86 14.47 19.21 18.23 44.58 55.26
Rwanda-kin 15.99 17.14 15.09 17.39 15.59 16.25 16.49 19.48 25.82 43.99
Uganda-lug 16.37 18.71 19.13 21.60 16.14 19.6 17.09 20.33 30.9 52.34
Ethiopia-orm 18.72 17.80 23.03 22.25 17.02 15.49 20.62 20.54 38.46 45.43
Botswana-tsn 18.15 24.10 19.27 25.16 16.48 23.33 17.61 23.56 26.79 36.81
Somali-som 14.39 21.51 16.39 23.17 15.02 18.29 17.54 23.99 23.05 32.39
Eritrea-tir 16.3 21.08 17.89 21.94 15.93 21.59 19.39 27.61 35.64 53.33
Nigeria-yor 11.55 16.31 11.91 18.71 11.3 16.63 12.81 18.99 25.07 49.64
Average 15.75 18.39 17.58 20.99 15.27 18.31 17.91 21.96 32.11 45.19

Table 16: Audio Open-ended for Native Audio(Loc-Lang Aware)

Language G3n-2B chrf++ G3n-2B acc. G3n-4B chrf++ G3n-4B acc. Q-3B chrf++ Q-3B acc. Q-7B chrf++ Q-7B acc. Gemini-2.5 Pro chrf++ Gemini-2.5 Pro acc.
Ghana-twi 12.14 11.75 12.24 13.25 5.98 3.92 5.98 4.10 45.72 57.06
Ethiopia-amh 0.58 24.85 1.53 25.85 0.00 2.61 0.00 2.61 47.43 79.30
Malawi-nya 16.03 10.98 17.32 13.77 6.31 5.39 6.31 5.59 58.40 63.05
Nigeria-hau 9.52 13.10 12.53 16.94 2.87 0.40 2.87 0.60 63.06 81.65
Nigeria-ibo 9.42 9.90 9.01 10.71 4.22 2.83 4.22 2.83 41.81 52.08
Kenya-kik 20.13 12.96 21.02 14.40 5.54 1.66 5.54 1.66 54.02 63.09
Rwanda-kin 12.81 14.20 11.84 17.40 2.48 1.20 2.48 1.20 67.48 73.68
Uganda-lug 15.13 18.49 15.41 18.29 4.96 3.59 4.96 3.78 56.59 64.26
Ethiopia-orm 6.22 10.60 5.80 9.40 2.90 2.93 3.27 3.72 51.83 50.55
Botswana-tsn 17.03 15.23 14.58 12.22 4.88 3.04 4.88 3.04 71.22 74.39
Somali-som 6.78 8.58 6.61 11.38 3.00 2.00 3.00 2.00 59.27 69.17
Eritrea-tir 0.10 22.35 0.14 17.32 0.01 7.08 0.01 7.26 47.97 68.63
Nigeria-yor 8.92 19.00 8.30 19.40 2.76 1.60 2.76 1.20 33.51 72.98
Lesotho-sot 6.46 3.56 5.30 4.13 5.14 1.27 6.47 2.11 50.94 50.84
S.Africa-zul 19.12 18.75 22.44 16.86 8.78 4.20 15.36 9.24 69.90 71.43
Average 10.69 14.29 10.94 14.75 3.99 2.91 4.54 3.40 54.61 66.14

表 16:音频开放式问答,母语音频(带地点/语言信息)

Language G3n-2B chrf++ G3n-2B acc. G3n-4B chrf++ G3n-4B acc. Q-3B chrf++ Q-3B acc. Q-7B chrf++ Q-7B acc. Gemini-2.5 Pro chrf++ Gemini-2.5 Pro acc.
Ghana-twi 12.14 11.75 12.24 13.25 5.98 3.92 5.98 4.10 45.72 57.06
Ethiopia-amh 0.58 24.85 1.53 25.85 0.00 2.61 0.00 2.61 47.43 79.30
Malawi-nya 16.03 10.98 17.32 13.77 6.31 5.39 6.31 5.59 58.40 63.05
Nigeria-hau 9.52 13.10 12.53 16.94 2.87 0.40 2.87 0.60 63.06 81.65
Nigeria-ibo 9.42 9.90 9.01 10.71 4.22 2.83 4.22 2.83 41.81 52.08
Kenya-kik 20.13 12.96 21.02 14.40 5.54 1.66 5.54 1.66 54.02 63.09
Rwanda-kin 12.81 14.20 11.84 17.40 2.48 1.20 2.48 1.20 67.48 73.68
Uganda-lug 15.13 18.49 15.41 18.29 4.96 3.59 4.96 3.78 56.59 64.26
Ethiopia-orm 6.22 10.60 5.80 9.40 2.90 2.93 3.27 3.72 51.83 50.55
Botswana-tsn 17.03 15.23 14.58 12.22 4.88 3.04 4.88 3.04 71.22 74.39
Somali-som 6.78 8.58 6.61 11.38 3.00 2.00 3.00 2.00 59.27 69.17
Eritrea-tir 0.10 22.35 0.14 17.32 0.01 7.08 0.01 7.26 47.97 68.63
Nigeria-yor 8.92 19.00 8.30 19.40 2.76 1.60 2.76 1.20 33.51 72.98
Lesotho-sot 6.46 3.56 5.30 4.13 5.14 1.27 6.47 2.11 50.94 50.84
S.Africa-zul 19.12 18.75 22.44 16.86 8.78 4.20 15.36 9.24 69.90 71.43
Average 10.69 14.29 10.94 14.75 3.99 2.91 4.54 3.40 54.61 66.14

Table 17: Audio Open-ended for English (Image only Audio)

Language G3n-2B chrf++ G3n-2B acc. G3n-4B chrf++ G3n-4B acc. Q-3B chrf++ Q-3B acc. Q-7B chrf++ Q-7B acc. Gemini-2.5 Pro chrf++ Gemini-2.5 Pro acc.
Ghana-twi 18.91 28.54 21.52 29.98 19.30 29.36 20.01 29.55 35.14 57.49
Ethiopia-amh 13.41 10.56 15.65 13.26 13.42 11.24 14.90 15.04 40.36 49.44
Malawi-nya 11.83 9.44 11.71 9.69 12.57 14.58 13.30 14.13 25.41 32.69
Nigeria-hau 11.58 13.95 11.02 13.48 12.67 14.89 15.60 15.73 23.14 37.35
Nigeria-ibo 11.36 4.86 11.66 6.25 12.72 5.79 12.72 6.02 26.90 34.95
Kenya-kik 16.33 13.68 16.77 16.58 15.60 13.68 18.32 17.24 44.25 57.11
Rwanda-kin 14.19 13.81 13.96 12.02 15.71 14.87 15.41 14.92 25.34 33.5
Uganda-lug 14.81 16.7 16.32 19.38 15.25 19.15 15.76 18.18 37.54 49.22
Ethiopia-orm 14.36 9.6 17.3 11.94 13.85 10.07 15.75 14.67 38.42 42.62
Botswana-tsn 15.82 20.93 17.02 23.89 14.70 22.03 18.99 24.6 34.18 49.89
Somali-som 13.62 19.39 14.7 22.22 14.63 20.33 16.56 22.8 25.76 37.35
Eritrea-tir 12.36 8.39 12.33 10.11 12.37 8.62 12.62 10.87 29.77 39.57
Nigeria-yor 10.67 13.91 10.65 14.39 10.81 13.43 11.40 17.46 27.21 49.4
Average 13.79 14.13 14.66 15.63 14.12 15.25 15.49 17.01 31.80 43.89

表 17:音频开放式问答,英语(仅图像音频)

Language G3n-2B chrf++ G3n-2B acc. G3n-4B chrf++ G3n-4B acc. Q-3B chrf++ Q-3B acc. Q-7B chrf++ Q-7B acc. Gemini-2.5 Pro chrf++ Gemini-2.5 Pro acc.
Ghana-twi 18.91 28.54 21.52 29.98 19.30 29.36 20.01 29.55 35.14 57.49
Ethiopia-amh 13.41 10.56 15.65 13.26 13.42 11.24 14.90 15.04 40.36 49.44
Malawi-nya 11.83 9.44 11.71 9.69 12.57 14.58 13.30 14.13 25.41 32.69
Nigeria-hau 11.58 13.95 11.02 13.48 12.67 14.89 15.60 15.73 23.14 37.35
Nigeria-ibo 11.36 4.86 11.66 6.25 12.72 5.79 12.72 6.02 26.90 34.95
Kenya-kik 16.33 13.68 16.77 16.58 15.60 13.68 18.32 17.24 44.25 57.11
Rwanda-kin 14.19 13.81 13.96 12.02 15.71 14.87 15.41 14.92 25.34 33.5
Uganda-lug 14.81 16.7 16.32 19.38 15.25 19.15 15.76 18.18 37.54 49.22
Ethiopia-orm 14.36 9.6 17.3 11.94 13.85 10.07 15.75 14.67 38.42 42.62
Botswana-tsn 15.82 20.93 17.02 23.89 14.70 22.03 18.99 24.6 34.18 49.89
Somali-som 13.62 19.39 14.7 22.22 14.63 20.33 16.56 22.8 25.76 37.35
Eritrea-tir 12.36 8.39 12.33 10.11 12.37 8.62 12.62 10.87 29.77 39.57
Nigeria-yor 10.67 13.91 10.65 14.39 10.81 13.43 11.40 17.46 27.21 49.4
Average 13.79 14.13 14.66 15.63 14.12 15.25 15.49 17.01 31.80 43.89

Table 18: Audio Open-ended for Native (Image only Audio)

Language G3n-2B chrf++ G3n-2B acc. G3n-4B chrf++ G3n-4B acc. Q-3B chrf++ Q-3B acc. Q-7B chrf++ Q-7B acc. Gemini-2.5 Pro chrf++ Gemini-2.5 Pro acc.
Ghana-twi 6.96 2.45 6.89 4.70 9.83 6.75 9.99 5.52 15.72 18.40
Ethiopia-amh 0.08 3.28 0.15 6.11 6.74 1.31 5.27 1.09 28.23 58.52
Malawi-nya 8.44 2.68 7.75 2.43 9.68 3.65 9.38 2.43 20.66 21.90
Nigeria-hau 7.45 4.58 7.35 3.86 10.69 3.86 9.85 4.58 22.09 36.63
Nigeria-ibo 6.39 0.25 5.51 1.00 7.23 0.50 6.64 0.75 11.23 15.25
Kenya-kik 9.07 2.13 9.14 2.93 9.03 2.13 9.09 1.86 20.14 23.73
Rwanda-kin 8.11 5.17 7.55 5.17 9.84 5.86 9.47 3.79 25.57 41.03
Uganda-lug 8.21 4.35 7.51 5.49 5.94 2.75 5.26 1.37 24.11 31.65
Ethiopia-orm 9.04 6.44 8.46 8.11 4.67 0.72 8.15 0.95 27.87 40.19
Botswana-tsn 6.97 2.12 6.89 3.40 9.69 4.67 9.89 3.82 24.55 31.62
Somali-som 6.65 4.25 5.81 7.55 11.33 10.38 11.52 10.14 17.59 29.48
Eritrea-tir 0.02 3.21 0.04 4.28 4.67 0.43 5.35 0.00 15.71 36.70
Nigeria-yor 4.18 1.42 4.00 3.31 6.84 2.36 7.39 2.84 15.73 35.63
Lesotho-sot 9.01 2.89 8.41 4.22 8.70 1.56 8.94 2.00 13.68 20.67
S.Africa-zul 9.65 5.12 10.54 6.76 7.88 2.66 7.67 2.25 34.04 40.37
Average 6.68 3.36 6.40 4.62 8.18 3.31 8.26 2.89 21.13 32.12

表 18:音频开放式问答,母语(仅图像音频)

Language G3n-2B chrf++ G3n-2B acc. G3n-4B chrf++ G3n-4B acc. Q-3B chrf++ Q-3B acc. Q-7B chrf++ Q-7B acc. Gemini-2.5 Pro chrf++ Gemini-2.5 Pro acc.
Ghana-twi 6.96 2.45 6.89 4.70 9.83 6.75 9.99 5.52 15.72 18.40
Ethiopia-amh 0.08 3.28 0.15 6.11 6.74 1.31 5.27 1.09 28.23 58.52
Malawi-nya 8.44 2.68 7.75 2.43 9.68 3.65 9.38 2.43 20.66 21.90
Nigeria-hau 7.45 4.58 7.35 3.86 10.69 3.86 9.85 4.58 22.09 36.63
Nigeria-ibo 6.39 0.25 5.51 1.00 7.23 0.50 6.64 0.75 11.23 15.25
Kenya-kik 9.07 2.13 9.14 2.93 9.03 2.13 9.09 1.86 20.14 23.73
Rwanda-kin 8.11 5.17 7.55 5.17 9.84 5.86 9.47 3.79 25.57 41.03
Uganda-lug 8.21 4.35 7.51 5.49 5.94 2.75 5.26 1.37 24.11 31.65
Ethiopia-orm 9.04 6.44 8.46 8.11 4.67 0.72 8.15 0.95 27.87 40.19
Botswana-tsn 6.97 2.12 6.89 3.40 9.69 4.67 9.89 3.82 24.55 31.62
Somali-som 6.65 4.25 5.81 7.55 11.33 10.38 11.52 10.14 17.59 29.48
Eritrea-tir 0.02 3.21 0.04 4.28 4.67 0.43 5.35 0.00 15.71 36.70
Nigeria-yor 4.18 1.42 4.00 3.31 6.84 2.36 7.39 2.84 15.73 35.63
Lesotho-sot 9.01 2.89 8.41 4.22 8.70 1.56 8.94 2.00 13.68 20.67
S.Africa-zul 9.65 5.12 10.54 6.76 7.88 2.66 7.67 2.25 34.04 40.37
Average 6.68 3.36 6.40 4.62 8.18 3.31 8.26 2.89 21.13 32.12

D Per language inter-rater agreement

D 分语言的评分者间一致性

Table 19: Average per-language inter-rater agreement (LLM-as-a-Judge vs Human evaluator) for English and Native

Setup Language Krippendorff's Alpha
English Ghana-twi 0.68
English Ethiopia-amh 0.96
English Eritrea-tir 0.76
English Ethiopia-orm 0.70
English Nigeria-yor 0.67
English Nigeria-igb 0.76
English Somalia-som 0.79
English Average 0.76
Native Ghana-twi 0.78
Native Ethiopia-amh 0.95
Native Eritrea-tir 0.74
Native Ethiopia-orm 0.98
Native Nigeria-yor 0.79
Native Nigeria-igb 0.92
Native Somalia-som 0.78
Native Average 0.84

表 19:英语和母语设定下,分语言的平均评分者间一致性(LLM 评审对人工评测),样本为随机 50 条。

Setup Language Krippendorff's Alpha
English Ghana-twi 0.68
English Ethiopia-amh 0.96
English Eritrea-tir 0.76
English Ethiopia-orm 0.70
English Nigeria-yor 0.67
English Nigeria-igb 0.76
English Somalia-som 0.79
English Average 0.76
Native Ghana-twi 0.78
Native Ethiopia-amh 0.95
Native Eritrea-tir 0.74
Native Ethiopia-orm 0.98
Native Nigeria-yor 0.79
Native Nigeria-igb 0.92
Native Somalia-som 0.78
Native Average 0.84

E Correlation statistics

E 相关统计

Table 20: Correlation statistics of controlled experiments for text

Gemma3n-2B Gemma3n-4B Qwen2.5-3B Qwen2.5-7B Gemini-2.5 Pro
XNLI - MMLU 0.684 (p=0.029) 0.88 (p=0.001) 0.254 (p=0.479) 0.426 (p=0.22) 0.831 (p=0.003)
XNLI - Afri-MCQA 0.261 (p=0.498) 0.636 (p=0.065) -0.388 (p=0.302) 0.316 (p=0.407) -0.248 (p=0.519)
MMLU - Afri-MCQA 0.373 (p=0.288) 0.463 (p=0.178) 0.236 (p=0.511) 0.479 (p=0.162) -0.222 (p=0.537)

表 20:文本控制实验的相关统计

Gemma3n-2B Gemma3n-4B Qwen2.5-3B Qwen2.5-7B Gemini-2.5 Pro
XNLI - MMLU 0.684 (p=0.029) 0.88 (p=0.001) 0.254 (p=0.479) 0.426 (p=0.22) 0.831 (p=0.003)
XNLI - Afri-MCQA 0.261 (p=0.498) 0.636 (p=0.065) -0.388 (p=0.302) 0.316 (p=0.407) -0.248 (p=0.519)
MMLU - Afri-MCQA 0.373 (p=0.288) 0.463 (p=0.178) 0.236 (p=0.511) 0.479 (p=0.162) -0.222 (p=0.537)

F Annotation guideline

F 标注指南

Guidelines for Annotators

标注员指南

The following shows detailed requirements and rules for dataset creation when annotating the dataset.

以下给出标注本数据集时,在数据创建方面的详细要求与规则。

Image Selection:

图像选择:

  • Collect images that represent the diverse cultural aspects and the specific cultural background of your country. The image must fall into one of the categories below. Pick one of the most relevant categories (more later):
  • 采集能够反映本国多样文化侧面及具体文化背景的图像。图像必须属于下列类别之一。请选择一个最相关的类别(后文详述):
  • Images should be relevant to your culture or country.
  • 图像应与你的文化或国家相关。
  • Ensure that images are relevant to the questions being posed. In other words, the image is needed to answer the question.
  • 确保图像与所提问题相关。也就是说,回答该问题需要这张图像。
  • If the image contains the answer's text, you can blur/crop the image so that it does not contain the answer.
  • 如果图像中含有答案文字,可以模糊或裁切图像,使其不再包含答案。
  • Image source:
  • 图像来源:
  • Self/personal picture (highly preferable). Ask your family or friends to donate their photos, if possible.
  • 自己/个人拍摄的照片**(最为优先)**。如有可能,也可请家人或朋友提供照片。
  • We also accept external images from:
  • 我们也接受来自下列来源的外部图像:
  • Flickr: https://www.flickr.com/explore (please make sure the associated license to the image is Creative Commons), this can be selected at the top left of Flickr ("Any License").
  • Flickr:https://www.flickr.com/explore(**请确保图像所附许可为 Creative Commons**),可在 Flickr 左上角选择("Any License")。
  • Wikimedia Commons: https://commons.wikimedia.org/wiki/Main_Page (here you do not need to select any license for the images),
  • Wikimedia Commons:https://commons.wikimedia.org/wiki/Main_Page(**此处无需为图像选择许可**),
  • Unsplash: https://unsplash.com/ (please make sure to search the image first and select the license: Free). More details (Tutorial) at the end of this document.
  • Unsplash:https://unsplash.com/(**请先搜索图像并选择许可:Free**)。更详细的说明(教程)见本文档末尾。

G Annotation guideline

G 标注指南

  • Dollar Street: https://www.gapminder.org/dollar-street (here you do not need to select any license for the images). This webpage has images only from some countries; please make sure to select your country to find images if applicable.
  • Dollar Street:https://www.gapminder.org/dollar-street(**此处无需为图像选择许可**)。该网页只包含部分国家的图像;如适用,请先选择你的国家再查找图像。

More detailed instructions for each web page are shown at the end of this document.

各网页的更详细说明见本文档末尾。

  • If you use an external image, you'll need to put the URL of the original image.
  • 如果使用外部图像,需要填写原图的 URL。
  • The image must be of reasonable quality (not pixelated or blurry) and understandable. You can upload images of any ratio as long as it is not too tall or wide (e.g.: don't submit panorama pictures).
  • 图像必须有合理的质量(不能有明显像素感或模糊),并且能够看懂。可以上传任意比例的图像,但不要过长或过宽(例如:不要提交全景照片)。
  • Do not show personally identifiable information (PII) such as faces, car plates, house addresses. Faces of public figures or fictional characters are ok. Also, please be sure to blur text in the image that will leak the answer. "PicdeFacer" can be used for blurring: https://picdefacer.com/en/. Tutorial on using PicdeFacer is shown at the end of this document.
  • 不要展示可识别个人身份的信息(PII),例如面孔、车牌、住址。公众人物或虚构角色的面孔可以保留。另外,请务必模糊图像中会泄露答案的文字。可使用 "PicdeFacer" 进行模糊:https://picdefacer.com/en/。PicdeFacer 的使用教程见本文档末尾。

Question and Answer Creation:

问题与答案撰写:

After finding the image, you must formulate 1-3 questions + answers from that image. Specifically:

找到图像后,必须根据该图撰写 1–3 道问题及答案。具体要求如下:

  • The question must be answerable only by looking at the image.
  • 问题必须只看图像就能回答。
  • Ensure that the questions are culturally relevant and specific to the image content.
  • 确保问题与文化相关,并且紧扣图像内容。
  • Provide answers that are concise, accurate, and directly related to the question.
  • 答案应简洁、准确,并与问题直接相关。
  • You must also provide one correct option and three other incorrect options (distractors). For the distractors, choose relevant options, not obvious wrong answers.
  • 还必须提供 1 个正确选项和 3 个错误选项(干扰项)。干扰项应选择相关选项,而不是一眼就能看出的错误答案。
  • The question must be answerable even without multiple-choice questions. Example of the invalid question: ("What song is not performed by this musician" – not answerable if you don't know the choices)
  • 即便没有选择题选项,问题也必须能够回答。 无效问题的例子:("这位音乐家没有演唱哪首歌"——如果不知道选项就无法回答)
  • Make sure the questions are written fluently in both the local language and English. Use a grammar checker if needed i.e. if you are not fluent in English.
  • 确保问题用当地语言和英语都写得通顺。 如有需要(例如英语不够流利),可使用语法检查工具。
  • Be mindful of cultural sensitivities and avoid stereotyping or misrepresenting cultural aspects.
  • 注意文化敏感,避免刻板印象或对文化内容的错误呈现。
  • Complexity Levels: Include a variety of question types:
  • 复杂度层次:应包含多种问题类型:
  • Multi-hop reasoning questions
  • 多跳推理问题
  • Counting or estimation questions
  • 计数或估计问题
  • Questions requiring local common sense knowledge
  • 需要当地常识才能回答的问题

H Annotation guideline

H 标注指南

  • Cultural inference questions
  • 文化推断问题

Ensure there are variations on your question. Create complex questions (for example, multi-hop reasoning, counting, referencing, or questions that require local commonsense knowledge to be answered).

确保问题有变化。请撰写复杂问题(例如多跳推理、计数、指代,或需要当地常识才能回答的问题)。

  • Multi-hop (e.g., Q: What is the name of the brand that the person in the picture is leading as CEO? A: Nando's B: Shoprite C: Woolworths D: Checkers)
  • 多跳(例如,Q:图中人物担任 CEO 的品牌叫什么?A: Nando's B: Shoprite C: Woolworths D: Checkers)

I Annotation guideline

I 标注指南

e.g., Q: Where was the person in the picture born? A: Ruanda-Urundi, B…..)

例如,Q:图中人物出生在哪里?A: Ruanda-Urundi,B……)

  • Intersection (e.g, what national dish is often prepared with the above ingredients in _____)
  • 交叉型(例如,上述食材常被用来做哪道全国性菜肴,在 _____)

Audio Recording Requirements

录音要求

Recording Specifications

录音规格

  • Format: MP3 or WAV (MP3 preferred for file size)
  • 格式:MP3 或 WAV(为控制文件大小,优先使用 MP3)
  • Environment: Record in a quiet space with minimal background noise
  • 环境:在安静、背景噪声尽量少的空间录音

What to Record

需要录制的内容

For each question-answer pair, record:

对每组问答,请录制:

  1. Question in Native Language
  1. 母语问题
  1. Question in English
  1. 英语问题
  1. Options in Native Language
  1. 母语选项
  1. Options in English
  1. 英语选项

Recording Guidelines

录音指南

  • Speaking Style:
  • 说话方式:

J Annotation guideline

J 标注指南

  • Speak clearly and at a natural pace
  • 口齿清楚,语速自然
  • Use standard pronunciation for your language/dialect
  • 使用所属语言/方言的标准发音
  • Avoid rushed or overly slow speech
  • 避免过快或过慢
  • Consistency: Maintain consistent volume and tone across recordings
  • 一致性:各条录音的音量和语气应保持一致
  • Multiple Takes: Record multiple takes and select the clearest version
  • 多次试录:多录几遍,选择最清楚的一版
  • Native Speakers Only: Questions should be recorded by native speakers of the language
  • 仅限母语者:问题应由该语言的母语者录制

Category Definition

类别定义

When selecting a category, pick one of the most relevant. Please follow the guidelines:

选择类别时,请挑一个最相关的。请遵循以下指南:

  • Vehicles and Transportation: Local public transport, local vehicles.
  • 车辆与交通: 当地公共交通、当地车辆。
  • Objects, Materials, Clothing: Questions about local/traditional clothes. Unique/local tools or items.
  • 物品、材料、服饰: 关于当地/传统服装的问题。独特或当地的工具或物品。
  • Cooking and Food: Local dishes and food/drink. This category includes native fruits in the context of the image if that fruit is served as a food/drink.
  • 烹饪与食物: 当地菜肴和食品/饮品。若图像语境中的本地水果是作为食品/饮品出现的,也归入此类。
  • Geography, Buildings, Landmarks: Popular/common landmarks, local architecture/buildings. Local monuments.
  • 地理、建筑、地标: 常见或知名地标、当地建筑。当地纪念物。
  • Plants and Animals: Plants and animals commonly found in the region.
  • 植物与动物: 该地区常见的植物和动物。
  • Brands, Products, and Companies: Questions about understanding local yet popular brands or companies. Even if the brand is about food/transportation, if the main focus of the question is the brand recognition itself, then it should be under this category.
  • 品牌、产品与公司: 考察对当地但广为人知的品牌或公司的理解。即便该品牌与食品/交通有关,只要问题的重点是品牌识别本身,就应归入此类。
  • Sports & Recreation: Local sports and fun activities. Focuses on the activity itself rather than the location (in that case, it goes to the 'landmark' category).
  • 体育与娱乐: 当地的体育和娱乐活动。侧重活动本身,而不是地点(若侧重地点,则归入“地标”类)。
  • Tradition, Art, History: Local ceremonies/festivals/events, local dance/music, folklores. Historical artifacts.
  • 传统、艺术、历史: 当地仪式/节日/活动、当地舞蹈/音乐、民间传说。历史文物。
  • People & Everyday Life: Focuses on the people themselves: i.e., common habits/customs, common occupations and jobs, routine religious activities, everyday activities/routines.
  • 人物与日常生活: 侧重人本身:例如常见习惯/习俗、常见职业与工作、日常宗教活动、日常活动/作息。
  • Public Figures & Pop Culture: Questions on the understanding of common public figures (e.g., politicians, artists, musicians, etc.). Common pop culture such as movies and games. If the category is still ambiguous to you, pick the one you think is the most appropriate.
  • 公众人物与流行文化: 考察对常见公众人物(如政治人物、艺术家、音乐家等)的了解,以及电影、游戏等常见流行文化。若类别仍难以判断,请选择你认为最合适的一项。

K Annotator Demography

K 标注员人口统计

Table 21: Annotator Demographics

Annotator ID Gender Age Group Resides in Africa
Ethiopia-amh M 18-30 ✓
Nigeria-hau F 30-40 ✓
Nigeria-ibo F 0-40 ✓
Uganda-lug F 18-30 ✓
Ethiopia-orm M 30-40 ✓
Rwanda-kin F 18-30 ✓
Kenya-kik F 30-40 ✓
Somali-som F 18-30 ✓
Eritrea-tir M 18-30 ✓
Ghana-twi F 18-30 ✓
Nigeria-yor M 30-40 ✓
Botswana-tsn M 30-40 ✓
Malawi-nya F 18-30 ✓
S.Africa-zul F 18-30 ✓
Lesotho-sot F 30-40 ✓

表 21:标注员人口统计

Annotator ID Gender Age Group Resides in Africa
Ethiopia-amh M 18-30 ✓
Nigeria-hau F 30-40 ✓
Nigeria-ibo F 0-40 ✓
Uganda-lug F 18-30 ✓
Ethiopia-orm M 30-40 ✓
Rwanda-kin F 18-30 ✓
Kenya-kik F 30-40 ✓
Somali-som F 18-30 ✓
Eritrea-tir M 18-30 ✓
Ghana-twi F 18-30 ✓
Nigeria-yor M 30-40 ✓
Botswana-tsn M 30-40 ✓
Malawi-nya F 18-30 ✓
S.Africa-zul F 18-30 ✓
Lesotho-sot F 30-40 ✓

L Review Guidelines

L 审核指南

Afri-MC-VQA Dataset Review Guidelines

Afri-MC-VQA 数据集审核指南

Overview

概述

This document provides comprehensive guidelines for reviewers to ensure quality control and consistency across all submissions for the Afri-MC-VQA dataset. Reviewers must carefully evaluate each submission against these criteria to ensure the integrity of the dataset.

本文件为审核人员提供全面指南,以确保 Afri-MC-VQA 数据集全部提交的质量控制与一致性。审核人员必须按这些标准仔细评估每份提交,以保障数据集的完整性。

  1. Image Review Criteria
  1. 图像审核标准

1.1 Privacy and Anonymization

1.1 隐私与匿名化

All images must protect personal privacy

所有图像都必须保护个人隐私

  • Faces: All private individuals' faces must be blurred (public figures exempt)
  • 面孔:所有私人的面孔都必须模糊(公众人物除外)
  • Personal Information: Must be completely obscured:
  • 个人信息:必须完全遮盖:
  • Vehicle registration numbers
  • 车辆登记号码
  • Phone numbers
  • 电话号码
  • Home addresses
  • 家庭住址
  • Personal identification documents
  • 个人身份证件
  • Credit card information
  • 信用卡信息
  • Verification: Use image zoom to check for any missed PII
  • 核验:放大图像,检查是否仍有遗漏的个人身份信息

1.2 Cultural Relevance

1.2 文化相关性

  • The image must clearly represent aspects of local culture
  • 图像必须清楚呈现当地文化的某些方面
  • Should be meaningful to the specific country/region
  • 应对特定国家/地区具有意义
  • Must align with one of the designated cultural categories
  • 必须归入指定的文化类别之一
  • Reject generic images that could be from anywhere
  • 拒绝那些看不出具体来源地的泛化图像

1.3 Image Uniqueness

1.3 图像独特性

  • Check for duplicate images within the same batch
  • 检查同一批次中是否有重复图像
  • Verify against existing submissions to avoid repetition
  • 与已有提交对照,避免重复
  • Similar scenes from different angles are acceptable if they enable different questions
  • 若能支撑不同的问题,同一场景的不同角度可以接受
  • Flag any stock photos or overly generic content
  • 标记任何图库照片或过于泛化的内容
  1. Question Quality Standards
  1. 问题质量标准

2.1 Language Quality

2.1 语言质量

M Review Guidelines

M 审核指南

  • English Version:
  • 英语版本:
  • Must be grammatically correct
  • 必须语法正确
  • Natural phrasing (not awkward translations)
  • 表述自然(不要别扭的翻译腔)
  • Clear and unambiguous
  • 清楚且没有歧义
  • Native Language Version:
  • 母语版本:
  • Fluent and natural in the local language
  • 当地语言流利、自然
  • Appropriate register and tone
  • 语域和语气得当
  • Correct spelling and diacritics
  • 拼写和变音符号正确

2.2 Translation Accuracy

2.2 翻译准确性

  • Semantic Equivalence: Both versions must convey identical meaning
  • 语义对等:两个版本必须表达相同的意思
  • Not Word-for-Word: Translations should be idiomatic, not literal
  • 不要逐词对译:译文应合乎习惯,而不是字面直译
  • Cultural Adaptation: Acceptable to adapt phrasing for cultural context while maintaining meaning
  • 文化适配:可以按文化语境调整措辞,但须保持原意

2.3 Question Independence

2.3 问题独立性

  • Stand-alone: Question must be answerable from the image alone
  • 独立成立:只凭图像就应能回答问题
  • No Option Dependency: Remove questions like "Which of these is NOT..." that require seeing options
  • 不依赖选项:删除诸如 "Which of these is NOT..." 这类必须看到选项才能回答的问题
  • Single Clear Answer: Only one option should be definitively correct
  • 唯一明确答案:只有一个选项应能被明确判定为正确
  • Test: Cover the options - can the question still be answered?
  • 检验:遮住选项后,问题是否仍能回答?

2.4 Clarity and Specificity

2.4 清楚与具体

  • No Vagueness: Questions must be precise
  • 避免含糊:问题必须精确
  • Object Specification: When multiple objects present, clearly indicate which one
  • 对象指明:出现多个对象时,须清楚指明是哪一个

2.5 Formatting

2.5 格式

  • No Numbering: Questions should not start with "1.", "Q:", etc.
  • 不要编号:问题不要以 "1."、"Q:" 等开头
  • Proper Punctuation: End with a question mark
  • 标点正确:以问号结尾
  • Capitalization: Follow standard rules for both languages
  • 大小写:两种语言都遵循常规规则
  1. Multiple Choice Options Review
  1. 选择题选项审核

3.1 Plausibility Check

3.1 合理性检查

  • All Options Believable: Someone unfamiliar with the culture should find all options possible
  • 所有选项都应可信:不熟悉该文化的人应觉得每个选项都有可能
  • No Obvious Eliminations: Avoid options that are clearly wrong
  • 不能一眼排除:避免明显错误的选项

N Review Guidelines

N 审核指南

  • Context-Appropriate: All options should fit the question's context
  • 合乎语境:所有选项都应适合问题的语境

3.2 Format Consistency

3.2 格式一致

  • Uniform Structure: All options should have similar length and detail
  • 结构统一:各选项的长度和细节程度应相近
  • Parallel Grammar: Same grammatical structure across options
  • 语法并列:各选项使用相同的语法结构

3.3 Prohibited Answer Types

3.3 禁止的答案类型

  • Never Accept:
  • 一律不接受:
  • "None of the above"
  • "No answer"
  • "Nothing"
  • "All of the above"
  • "Not applicable"

3.4 Objectivity Requirements

3.4 客观性要求

  • No Subjective Adjectives: Avoid beautiful, ugly, nice, bad, etc.
  • 不要主观形容词:避免 beautiful、ugly、nice、bad 等
  • Factual Descriptions Only: Focus on observable characteristics
  • 只做事实描述:关注可以观察到的特征
  1. Audio Review Requirements
  1. 音频审核要求

4.1 Transcript Accuracy

4.1 转写准确性

Audio must match text EXACTLY

音频必须与文本完全一致

  • Word-for-Word: No deviations, additions, or omissions
  • 逐字对应:不得有偏离、增添或遗漏
  • Pronunciation: Clear and accurate for the language
  • 发音:对该语言而言清楚、准确
  • Verification Method: Follow along with text while listening
  • 核验方法:边听边对照文本

4.2 Audio Structure

4.2 音频结构

  • Question Format: Natural reading with appropriate intonation
  • 问题形式:朗读自然,语调得当
  • Option Spacing: Brief pause (0.5-1 second) between each option
  • 选项间隔:各选项之间稍作停顿(0.5–1 秒)
  • Answer Position: Correct answer must ALWAYS be the first option spoken
  • 答案位置:正确答案必须始终是朗读的第一个选项

4.3 Audio Quality

4.3 音频质量

  • Clarity: No background noise or distortion
  • 清晰度:没有背景噪声或失真
  • Volume: Consistent throughout recording
  • 音量:整段录音保持一致
  • Speed: Natural speaking pace (not rushed or too slow)
  • 语速:自然语速(不要过快或过慢)
  • Complete: Full question and all options included
  • 完整:包含完整问题和全部选项

O Review Guidelines

O 审核指南

  1. Critical Review Points
  1. 关键审核要点

5.1 Question-Option Alignment

5.1 问题与选项对齐

  • Check: Ensure question type matches option format
  • 检查:确保问题类型与选项形式匹配
  • Common Issue: Yes/No question with descriptive options
  • 常见问题:是非问句却配了描述性选项
  • Verify: Question grammar aligns with option grammar
  • 核验:问题语法与选项语法一致

5.2 Cultural Sensitivity

5.2 文化敏感性

  • No Stereotypes: Avoid reinforcing negative cultural stereotypes
  • 没有刻板印象:避免强化负面的文化刻板印象
  • Respectful Representation: Ensure dignified portrayal of cultural elements
  • 尊重的呈现:确保以有尊严的方式表现文化元素
  • Balanced View: Include both traditional and modern aspects
  • 平衡视角:同时包含传统与现代的方面

5.3 Language Quality Control

5.3 语言质量控制

  • Spelling: Check both languages thoroughly
  • 拼写:两种语言都要仔细检查
  • Grammar: Ensure proper sentence structure
  • 语法:确保句子结构正确
  • Punctuation: Consistent and correct usage
  • 标点:用法一致且正确
  • Diacritics: Verify correct marks for languages that require them
  • 变音符号:对需要变音符号的语言,核对其是否正确

5.4 Content Diversity

5.4 内容多样性

  • Avoid Repetition: Similar questions across different images
  • 避免重复:不同图像上不要反复出现相似问题
  • Varied Complexity: Mix of simple and complex questions
  • 复杂度有变化:简单问题与复杂问题搭配
  • Different Question Types: Not all "What is..." questions
  • 问题类型多样:不要全是 "What is..." 一类的问题
  • Topic Distribution: Balanced across cultural categories
  • 主题分布:在各文化类别之间保持均衡

5.5 Question Complexity

5.5 问题复杂度

  • Avoid Trivial Questions:
  • 避免琐碎问题:
  • Require Cultural Knowledge: Questions should test understanding, not just observation
  • 需要文化知识:问题应考察理解,而不仅仅是观察
  • Informative Value: Each question should teach something about the culture
  • 信息价值:每道题都应当让人了解到该文化的某些内容

P Examples

P 示例

Example items from the Afri-MCQA benchmark with images, categories, and English and native-language questions

Table 22: Examples from the Afri-MCVQA Benchmark

表 22:Afri-MCVQA 基准中的示例