cheRAGh · چراغ · Reranker Benchmark

Persian RAG
Reranker Evaluation

cheRAGh — Benchmarking Suite for Persian RAG Systems

چراغِ جست‌وجو افروختیم در شامِ پرسش‌ها

که پیدا گردد از نورش، رهِ پاسخ زِ چالش‌ها

We lit the lamp of search in the night of questions,
so its light may reveal the path of answers from challenges.

cheRAGh (چراغ) is a unified benchmarking suite for Persian Retrieval-Augmented Generation (RAG) systems, covering embedding models, rerankers, retrieval quality, tool calling, and large language model performance across diverse Persian-language datasets from General, Scientific, Education, Legal, and Religious domains.

This report presents the evaluation results of reranker models in the cheRAGh benchmark. Each configuration pairs a base embedding model's retrieval with a reranker pass; performance is measured using Recall@5, MRR, and Δ MRR — the gain (or loss) the reranker contributes over base embedding-only retrieval.

Configurations

Domains

0.870

Avg MRR

0.928

Avg Recall@5

+0.006

Avg Δ MRR

Benchmark Datasets

Five Persian-language domains — Legal, Religious, General, Scientific, and Education — each containing queries paired with a single ground-truth document. The base embedding model retrieves top-k candidates, the reranker re-orders them, and we compare ranking quality and retrieval coverage before and after reranking.

📚

Education

Official high-school textbook passages

🌐

General

Open-domain Q&A platform logs, everyday topics

⚖️

Legal

Legal statutes, court decisions & regulatory texts

🕌

Religious

Islamic texts, scholarly commentary & theological Q&A

🔬

Scientific

Peer-reviewed scientific paper abstracts

Leaderboard

Green ≥ 0.75 · Orange 0.50–0.74 · Red < 0.50 · Bold green = best in column · Δ MRR: green = improved, red = hurt by reranking

# ↕	Reranker ↕	Architecture	Params ↕	Size	Max Len ↕	Base Model	Education ↕			General ↕			Legal ↕			Religious ↕			Scientific ↕			Average ↕
# ↕	Reranker ↕	Architecture	Params ↕	Size	Max Len ↕	Base Model	MRR ↕	Recall@5 ↕	Δ MRR ↕	MRR ↕	Recall@5 ↕	Δ MRR ↕	MRR ↕	Recall@5 ↕	Δ MRR ↕	MRR ↕	Recall@5 ↕	Δ MRR ↕	MRR ↕	Recall@5 ↕	Δ MRR ↕	MRR ↕	Recall@5 ↕	Δ MRR ↕

Visualisations

MRR by Dataset

Mean Reciprocal Rank

Recall@5 by Dataset

Fraction of queries where ground-truth is in top-k

Δ MRR by Dataset

Reranker lift over base embedding retrieval

Radar — MRR

Multi-domain strength profile

Radar — Recall@5

Multi-domain coverage profile

Performance Heatmap

All configurations × datasets — scroll to explore · hover for value

Worst

Best

About the Metrics

MRR — Mean Reciprocal Rank

Averages 1/rank across all queries. Rank 1 every time → 1.0; rank 5 every time → 0.20. Rewards getting the right answer to the top.

Recall@5

Fraction of queries where the correct document appears in the top-k results.

Δ MRR — Reranker Lift

MRR after reranking minus MRR of the base embedding-only retrieval. Positive means the reranker improved ranking quality; negative means it hurt it.

About cheRAGh

cheRAGh (چراغ) is a unified benchmarking suite for Persian RAG systems, covering embeddings, rerankers, retrieval quality, tool calling, and end-to-end pipeline performance.

# ⇅	Reranker ⇅	Rerank Type	Rerank Params	Rerank Size	Rerank Max Len	Rerank Base	Education ⇅			General ⇅			Legal ⇅			Religious ⇅			Scientific ⇅			Avg ⇅
# ⇅	Reranker ⇅	Rerank Type	Rerank Params	Rerank Size	Rerank Max Len	Rerank Base	MRR	Recall@5	Δ MRR	MRR	Recall@5	Δ MRR	MRR	Recall@5	Δ MRR	MRR	Recall@5	Δ MRR	MRR	Recall@5	Δ MRR	MRR	Recall@5	Δ MRR
🥇	Qwen/Qwen3-Reranker-8B on BAAI/bge-m3	Cross-Encoder (LLM-based)	8B	16.1 GB	32768	Qwen3-8B	0.9691	1.0000	+0.1635	0.8667	0.9380	+0.0640	0.9963	0.9980	+0.0197	0.8545	0.9303	+0.0974	0.9930	1.0000	+0.0117	0.9359	0.9733	+0.0713
🥈	Qwen/Qwen3-Reranker-4B on BAAI/bge-m3	Cross-Encoder (LLM-based)	4B	8.06 GB	32768	Qwen3-4B	0.9591	1.0000	+0.1535	0.8468	0.9180	+0.0441	0.9960	1.0000	+0.0194	0.7770	0.8625	+0.0186	0.9942	1.0000	+0.0129	0.9146	0.9561	+0.0497
🥉	BAAI/bge-reranker-v2-m3 on BAAI/bge-m3	Cross-Encoder	568M	1.14 GB	8192	bge-m3	0.9396	0.9951	+0.1340	0.8493	0.9200	+0.0466	0.9951	1.0000	+0.0185	0.7628	0.8526	+0.0044	0.9990	1.0000	+0.0177	0.9092	0.9535	+0.0442
4	nvidia/llama-nemotron-rerank-1b-v2 on BAAI/bge-m3	Cross-Encoder (LLM-based)	1B	2.2 GB	8192	Llama-Nemotron	0.9396	1.0000	+0.1340	0.8256	0.9180	+0.0229	0.9972	1.0000	+0.0206	0.7200	0.8267	-0.0384	0.9960	1.0000	+0.0147	0.8957	0.9489	+0.0308
5	BAAI/bge-reranker-v2-gemma on BAAI/bge-m3	Cross-Encoder (LLM-based)	2.6B	5.2 GB	8192	gemma-2b	0.9371	0.9951	+0.1315	0.8232	0.9060	+0.0205	0.9883	0.9980	+0.0117	0.6893	0.8088	-0.0691	0.9955	1.0000	+0.0142	0.8867	0.9416	+0.0218
6	jinaai/jina-reranker-v2-base-multilingual on BAAI/bge-m3	Cross-Encoder	278M	1.08 GB	8192	JinaBERT	0.9399	0.9951	+0.1343	0.8109	0.8960	+0.0082	0.9868	1.0000	+0.0102	0.6671	0.7351	-0.0913	0.9953	0.9980	+0.0140	0.8800	0.9248	+0.0151
7	jinaai/jina-reranker-v3 on BAAI/bge-m3	Listwise (Late-but-not-late interaction)	0.6B	1.52 GB	131072	Qwen3-0.6B	0.9353	0.9805	+0.1297	0.8048	0.9040	+0.0021	0.9733	0.9920	-0.0033	0.6515	0.7530	-0.1069	0.9883	0.9960	+0.0070	0.8706	0.9251	+0.0057
8	BAAI/bge-m3 (base — no reranker)	—	—	—	—	—	0.8056	0.9220	0.0000	0.8027	0.8740	0.0000	0.9766	0.9940	0.0000	0.7584	0.8287	0.0000	0.9813	0.9960	0.0000	0.8649	0.9229	0.0000
9	mixedbread-ai/mxbai-rerank-large-v2 on BAAI/bge-m3	Cross-Encoder (LLM-based)	1.5B	3.2 GB	8192	Qwen2.5-1.5B	0.8784	0.9707	+0.0728	0.7875	0.8540	-0.0152	0.9811	0.9940	+0.0045	0.6280	0.7351	-0.1304	0.9947	1.0000	+0.0134	0.8539	0.9108	-0.0110
10	Qwen/Qwen3-Reranker-0.6B on BAAI/bge-m3	Cross-Encoder (LLM-based)	0.6B	1.52 GB	32768	Qwen3-0.6B	0.9229	0.9854	+0.1173	0.7691	0.8760	-0.0336	0.9723	0.9900	-0.0043	0.6116	0.7590	-0.1468	0.9924	0.9980	+0.0111	0.8537	0.9217	-0.0113
11	tomaarsen/Qwen3-Reranker-0.6B-seq-cls on BAAI/bge-m3	Sequence Classification	0.6B	1.52 GB	32768	Qwen3-0.6B	0.9202	0.9854	+0.1146	0.7643	0.8760	-0.0384	0.9734	0.9900	-0.0032	0.6167	0.7610	-0.1417	0.9924	0.9980	+0.0111	0.8534	0.9221	-0.0115
12	BAAI/bge-reranker-large on BAAI/bge-m3	Cross-Encoder	335M	1.34 GB	512	bge-large	0.8405	0.9268	+0.0349	0.7554	0.8560	-0.0473	0.9480	0.9860	-0.0286	0.6142	0.7151	-0.1442	0.9905	0.9980	+0.0092	0.8297	0.8964	-0.0352
13	Alibaba-NLP/gte-multilingual-reranker-base on BAAI/bge-m3	Cross-Encoder	305M	1.22 GB	8192	mGTE (BERT+RoPE+GLU)	0.8966	0.9854	+0.0910	0.7512	0.8620	-0.0515	0.9608	0.9880	-0.0158	0.5800	0.6873	-0.1784	0.9287	0.9821	-0.0526	0.8234	0.9009	-0.0415
14	mixedbread-ai/mxbai-rerank-base-v2 on BAAI/bge-m3	Cross-Encoder (LLM-based)	494M	1.3 GB	8192	Qwen2.5-0.5B	0.8740	0.9902	+0.0684	0.7161	0.8100	-0.0867	0.9602	0.9860	-0.0164	0.5203	0.6514	-0.2380	0.9851	0.9980	+0.0038	0.8111	0.8871	-0.0538

Persian RAGReranker Evaluation

بنچمارک RAG فارسیارزیابی مدل‌های Reranker

Persian RAG
Reranker Evaluation

بنچمارک RAG فارسی
ارزیابی مدل‌های Reranker