Open Persian LLM Alignment Leaderboard

Developed by MCILAB in collaboration with the Machine Learning Laboratory at Sharif University of Technology , this benchmark is based on the open-source ELAB where presents a comprehensive evaluation framework for assessing the alignment of Persian Large Language Models (LLMs) with critical ethical dimensions, including safety, fairness, and social norms. Addressing the gaps in existing LLM evaluation frameworks, this benchmark is specifically tailored to Persian linguistic and cultural contexts.

It combines three types of Persian-language benchmarks:

1. Translated datasets (adapted from established English benchmarks)
2. Synthetically generated data (newly created for Persian LLMs)
3. Naturally collected data (reflecting indigenous cultural nuances)

Key Datasets in the Benchmark

The benchmark integrates the following datasets to ensure a robust evaluation of Persian LLMs:

Translated Datasets

• Anthropic-fa
• AdvBench-fa
• HarmBench-fa
• DecodingTrust-fa

Newly Developed Persian Datasets

• ProhibiBench-fa: Evaluates harmful and prohibited content in Persian culture.
• SafeBench-fa: Assesses safety in generated outputs.
• FairBench-fa: Measures bias mitigation in Persian LLMs.
• SocialBench-fa: Evaluates adherence to culturally accepted behaviors.

Naturally Collected Persian Dataset

• GuardBench-fa: A large-scale dataset designed to align Persian LLMs with local cultural norms.

A Unified Framework for Persian LLM Evaluation

By combining these datasets, our work establishes a culturally grounded alignment evaluation framework, enabling systematic assessment across three key aspects:

- Safety:  Avoiding harmful or toxic content.
- Fairness:  Mitigating biases in model outputs.
- Social Norms:  Ensuring culturally appropriate behavior.

This benchmark not only fills a critical gap in Persian LLM evaluation but also provides a standardized leaderboard to track progress in developing aligned, ethical, and culturally aware Persian language models.

Download Dataset

The full dataset is not publicly accessible; however, you can download a sample of 1,500 entries here. The distribution of this sample is as follows: