Shared task at the BLP Workshop, NAACL 2027

Multilingual Cultural Question Answering

Build systems that answer real, culturally grounded questions from native speakers in Bangla, English, Assamese and Hindi, in the language they were asked.

Join on EvalAI Get the data and code

বাংলাBangla, from Dhaka and Kolkata

EnglishEnglish, from Dhaka and Doha

অসমীয়াAssamese, from Assam

हिन्दीHindi, from Delhi

training QA pairs: 37,301
languages, from five regions: 4
tracks, enter any mix: 8
new blind test questions: 2,000

Where we are now

Loading…

News

Ask in Bangla, answer in Bangla

Questions come from real native-speaker queries about everyday life, across 18 topics such as food and drinks, travel, religion, tradition, education, names and weather. Your system answers each one in the language it was asked.

Example from the data: English, Doha, immigration

Can I extend my tourist visa in Qatar?

Subtask 2A

Monolingual QA

One model per language. Each model only ever sees its own language.

বাংলাmodel bnবাংলা Englishmodel enEnglish অসমীয়াmodel asঅসমীয়া हिन्दीmodel hiहिन्दी

Subtask 2B

Multilingual QA

One model trained on all four languages, including low-resource Assamese, answers every question.

বাংলা English অসমীয়া हिन्दी one model4 languages বাংলা English অসমীয়া हिन्दी

Eight tracks: enter one, a few, or all

Every track has its own leaderboard.

BanglaEnglishAssameseHindi 2AMonolingual 2A-bnবাংলা 2A-enEnglish 2A-asঅসমীয়া 2A-hiहिन्दी 2BMultilingual 2B-bnবাংলা 2B-enEnglish 2B-asঅসমীয়া 2B-hiहिन्दी

One submission file

All your tracks go into a single task.jsonl. Every track you include must answer all test questions of that language.

2A model rule

One model per language. Different 2A languages may use different models.

2B model rule

One multilingual model, trained on all four languages, for every 2B language you submit, even if you submit only some.

Rankings

Eight per-track leaderboards ranked by BERTScore F1, plus 2B All-Languages for teams covering all four languages in 2B.

Any pre-trained model or publicly available data may be used and must be described in your system paper. The MultiNativQA test split is our dev-test set and must not be used for training.

Important dates

Current stage

Native, culturally aligned questions with manually annotated answers. Final rankings use a new blind test set of 500 annotated questions per language.

Get the data on GitHub

বাংলাDhaka, Kolkata

13,909 train

Dev: 1,883
Dev-test: 3,667
Blind test: 500

EnglishDhaka, Doha

12,973 train

Dev: 1,820
Dev-test: 3,435
Blind test: 500

অসমীয়াAssam

1,131 train

Dev: 157
Dev-test: 545
Blind test: 500

हिन्दीDelhi

9,288 train

Dev: 1,286
Dev-test: 2,745
Blind test: 500

Total: 37,301 train, 5,146 dev, 10,392 dev-test and 2,000 blind test questions.

Get the files

Clone the repository and build the data files. They are written per language and combined, in JSON Lines format.

git clone https://github.com/AridHasan/blp27_task2.git
cd blp27_task2
pip install -r requirements.txt
python data/prepare_data.py

One line per question

Each record has an id, language, region, topic, question and reference answer. The blind test set has no answers.

{"id": "en_qa_cf92ec1e…",
 "lang": "en", "region": "qa",
 "category": "immigration",
 "question": "Can I extend my tourist visa in Qatar?",
 "answer": "If you would like to extend your visa, …"}

How to submit

Submissions are made on EvalAI. Use one EvalAI team for all of your team's runs; your latest submission for each track counts in the evaluation phase.

Register your team
Sign up on EvalAI, create a participant team and join the challenge.
Open the challenge on EvalAI
Build your system
Start from the mT5 baseline or the LoRA fine-tuning script in the repository. Both write a ready-to-submit file.
Browse the repository
Check the file
Run the format checker with the phase's question file. It confirms every track you submitted is complete.
Upload task.jsonl
Upload the file, or a zip containing it, on the Submit tab of the current phase.

Submission format

One JSON object per line. If you enter 2A and 2B for the same language, each question appears twice, once per subtask.

{"id": "bn_bd_0a1b2c…", "subtask": "2A", "lang": "bn",
 "answer": "ঢাকার বিখ্যাত খাবারের মধ্যে কাচ্চি বিরিয়ানি অন্যতম।",
 "model": "banglat5-ft"}

subtask is 2A or 2B; lang is bn, en, as or hi.
2A uses one model name per language; 2B uses one model name for all its rows.

Check before uploading

python format_checker/task.py \
  -p task.jsonl \
  --test-file data/test/all.jsonl

The checker lists the tracks it found, for example:

Tracks found -> 2A (Monolingual QA): Assamese;
2B (Multilingual QA): English, Assamese

Ranked by BERTScore F1, with a native encoder per language

Greedy cosine matching of contextual embeddings from layer 9, with no IDF weighting or rescaling. Inputs are truncated to 512 tokens and Bangla text is normalised before scoring.

বাংলাcsebuetnlp/banglabert

Englishbert-base-uncased

অসমীয়াai4bharat/IndicBERTv2-MLM-only

हिन्दीai4bharat/IndicBERTv2-MLM-only

Also reported, not used for ranking: BERTScore precision and recall ROUGE-1, ROUGE-2, ROUGE-L METEOR BLEU

Baseline: mT5-base

BERTScore F1 on the dev-test set. One mT5 model per language for 2A, one multilingual model for 2B.

Track	Bangla	English	Assamese	Hindi	All

Same numbers, locally

The official scorer in the repository is the one EvalAI runs, so you can reproduce every leaderboard score on your own machine.

python scorer/task.py \
  --gold-file-path data/dev_test/all.jsonl \
  --pred-file-path task.jsonl

Leaderboard

Scores are pulled from EvalAI every hour. Each team's ranked submission is shown per track.

Open on EvalAI

2A Monolingual

2B Multilingual

Subtask 2A, Bangla

System papers

Teams that submit during the evaluation phase and appear on a leaderboard are invited to describe their system in a short paper.

Title format

AlphaX at BLP-2027 Task 2: Culturally-Aware LLM Fine-tuning for Multilingual Question Answering

Length

Up to four pages of content, plus references.

Template

The official ACL two-column templates, unmodified. Open on Overleaf

Format check

Run your PDF through the ACL pubcheck tool; non-conforming papers are rejected without review.

Please cite

@inproceedings{hasan-etal-2025-nativqa,
    title = "{N}ativ{QA}: Multilingual Culturally-Aligned Natural Query for {LLM}s",
    author = "Hasan, Md. Arid and Hasanain, Maram and Ahmad, Fatema and Laskar, Sahinur Rahman and Upadhyay, Sunaya and Sukhadia, Vrunda N and Kutlu, Mucahid and Chowdhury, Shammur Absar and Alam, Firoj",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-acl.770/",
    doi = "10.18653/v1/2025.findings-acl.770",
    pages = "14886--14909",
}

@article{alam2025nativqa,
    title = {{NativQA Framework:} Enabling LLMs with Native, Local, and Everyday Knowledge},
    author = {Alam, Firoj and Hasan, Md Arid and Laskar, Sahinur Rahman and Kutlu, Mucahid and Darwish, Kareem and Chowdhury, Shammur Absar},
    journal = {arXiv preprint arXiv:2504.05995},
    year = {2025},
}