CMMLU is a comprehensive Chinese large model evaluation benchmark, covering multiple choice questions in multiple fields such as humanities, social sciences, science and engineering, and Chinese-specific knowledge. It is used to measure the knowledge reserve and reasoning capabilities of large language models in the Chinese context. The data set and evaluation code are open source on GitHub.

CMMLU

Core parameters and statistics of CMMLU

CMMLU (Chinese Massive Multitask Language Understanding) is a set of multi-domain knowledge evaluation benchmarks for Chinese large language models, which will be published by an academic research team in 2023. It does not provide chat or generation functions for end users, but provides researchers and model development teams with a standardized Chinese knowledge question bank covering 67 disciplines to measure the factual knowledge reserve and reasoning accuracy of the model in the Chinese context.

Parameter item	Value
Full name of benchmark	CMMLU (Chinese Massive Multitask Language Understanding)
Total number of questions	~11,500 multiple choice questions
Subject coverage	67 disciplines, divided into four categories: humanities, social sciences, science and engineering, and Chinese unique knowledge
Option structure	Choose one of four (A/B/C/D)
Question type	Knowledge-based multiple choice questions (fact memory + reasoning and judgment)
Output indicators	Accuracy of each subject (Accuracy) and overall average accuracy
License Agreement	Open Source (GitHub Public Repository)
First paper	arXiv 2023
Evaluation method	Local inference + script scoring, supporting few-shot and zero-shot settings
Dependencies	PyTorch / Transformers + Evaluation Script

Interpretation of parameter meaning: The division granularity of 67 subjects is much finer than that of MMLU (57 subjects) and C-Eval (52 subjects), which means that CMMLU provides more fine-grained ability diagnostic capabilities in the subject dimension. Although the total number of about 11,500 questions is not as large as some tens of thousands of questions, the average density of about 170 questions per subject is enough to support statistically significant subject-level accuracy comparisons. The four-choice format reduces the random guessing baseline (25%), so that the 30%-40% accuracy interval still has discriminating value, which is suitable for locating the shortcomings of the model in difficult subjects.

Scale comparison with similar benchmarks:

Benchmark	Number of subjects	Number of questions	Language	China-specific knowledge
MMLU	57	~14,000	English	None
C-Eval	52	~13,900	Chinese	Partial
CMMLU	67	~11,500	Chinese	Key Coverage
AGIEval	20+	~8,000	Chinese/English	Partial

Type determination: CMMLU belongs to the model evaluation infrastructure under the category of "Basic Large Model/API Infrastructure". The core deliverable is a set of standardized data sets + evaluation protocols + baseline scores. It does not involve continuously running API services, but it is an indispensable quality gate control in the model development chain.

Users and market recognition of CMMLU

• Academic citations and influence: After the CMMLU paper was made public on arXiv, it was cited by multiple Chinese large model teams in technical reports and papers as evidence of capabilities, including the releases of Qwen, Yi, Baichuan, InternLM and other series of models that all reported CMMLU scores. As of mid-2026, its GitHub repository has received thousands of stars, and the community fork and issue discussions are active.

• Industrial Adoption Model: Large domestic model manufacturers generally use CMMLU and C-Eval as the standard evaluation combination for Chinese knowledge ability. In the model capability comparison table, the overall accuracy of CMMLU and the accuracy of each subject are one of the most frequently cited Chinese benchmark indicators. Some teams incorporate CMMLU into daily regression testing pipelines to detect capability degradation during training.

• Positioning difference with C-Eval: C-Eval focuses on difficult questions for China’s college entrance examination/postgraduate entrance examination in general subjects, while CMMLU focuses more on the coverage of China’s unique knowledge (such as Chinese history, geography, law, literature). There is about 60% overlap between the two in overall subject distribution, but the depth of CMMLU in China-specific knowledge categories is significantly higher. Many teams report scores on both benchmarks simultaneously to obtain a more complete picture of Chinese proficiency.

• Boundary Note: The multiple-choice question benchmark has an inherent risk of data contamination - the model may have seen the original text of the question during the pre-training stage, resulting in falsely high scores. At present, the CMMLU team has not established an independent hold-out privacy set or dynamic question generation mechanism, so when citing its scores, it needs to be combined with the MMLU/C-Eval trend cross-validation of the same batch of models. In addition, high scores only reflect the amount of factual knowledge and do not represent the model’s depth of reasoning, conversation quality, or instruction following ability.

Cost Advantages of CMMLU

C side/individual researcher:

• Zero direct cost: The data set and evaluation script are completely open source, and the GitHub repository can be directly cloned and used without paying any licensing fees or subscription fees.
• Running Cost: Evaluation requires loading the model and performing inference on your own hardware or cloud instance. Taking a 7B parameter scale model as an example, it takes about 30-60 minutes to complete inference for all 67 subjects on a single A100, and the corresponding cloud computing cost is about $1-3 (estimated based on on-demand pricing). The computing power required for a 70B+ scale model increases linearly, and the estimated cost of a single full-scale evaluation is in the range of $10-30.

API Developer/Model Service Provider:

• Self-built alternative cost comparison: If you build a Chinese assessment set from scratch covering 67 subjects and about 170 questions in each subject, you need to hire subject experts to develop, review, test, and calibrate the questions. The estimated labor cost is hundreds of thousands of yuan. CMMLU reduces this cost to zero by providing validated question banks and baselines directly at zero licensing fees.
• Integration cost: The evaluation script is based on the Hugging Face transformers and datasets libraries, and is naturally compatible with mainstream model frameworks without the need for additional adaptation. The engineering cost of integrating CMMLU evaluation into a model training pipeline is typically within 1-2 man-days.

Business/Institution:

• Private deployment: The data set can be used completely offline and does not rely on any external API to meet data security and compliance requirements. The evaluation results do not leave the local area and are suitable for industries such as finance, medical, and government affairs that have strict restrictions on data export.
• Hidden Cost - Evaluation protocol consistency: Small differences between different teams in prompt template few-shot sample selection and answer extraction logic may lead to score deviations of 3-5 percentage points. In order to obtain horizontally comparable results, the officially recommended evaluation parameters (zero-shot or 5-shot, unified prompt format) need to be strictly implemented. When protocols are not harmonized, reported scores may not be comparable.

Main functions of CMMLU

• Multi-disciplinary standardized question bank: Covers 67 subjects and is organized into four categories: humanities (Chinese literature, history, philosophy, etc.), social sciences (law, economics, education, etc.), science and engineering (mathematics, physics, computers, etc.), and Chinese-specific knowledge (Chinese politics, Chinese geography, Chinese folklore, etc.). There are approximately 170 questions in each subject, four-choice format, and a random baseline of 25%. Use value: Provide a standardized measurement scale for Chinese knowledge ability, making different models comparable on the same scale.

• Flexible evaluation framework: Supports two evaluation settings: zero-shot and few-shot (default 5-shot). The prompt template, number and order of examples can be adjusted to adapt to the optimal performance range of different models. Usage value: The team can choose the most appropriate evaluation settings based on the model type (base vs. command fine-tuning) to avoid deviations caused by a single style.

• Open source baseline results: The warehouse has disclosed the accuracy baselines of multiple mainstream models in various disciplines, including GPT-4, Claude, Qwen, Baichuan, InternLM, Yi and other series. Use value: The new model team can directly judge the level of capabilities against the baseline without establishing a reference system from scratch.

• Subject-level ability diagnosis: The evaluation script outputs the independent accuracy rate of each subject, and can automatically generate a radar chart or histogram for horizontal comparison. Use value: Precisely locate the subjects in which the model is strong (such as Chinese history, literature) and weak (such as advanced mathematics, physics), and provide quantitative basis for data matching and training strategy adjustment.

• Community scalability: The data set and code are completely open to the public. The research team can add or delete subjects, modify the questions, add new model baselines, or extract interfaces from the evaluation scripts and embed them into custom test pipelines. Use value: Teams targeting specific industries (such as law, medicine) can build vertical field evaluation sets based on CMMLU and reuse the ready-made evaluation tool chain.

Expert opinion: CMMLU's "subject-level diagnosis" and "baseline comparison" are related - the baseline provides a frame of reference, and the subject score locates shortcomings. The combination of the two enables the model team to no longer rely solely on the overall score to measure the quality of the model, but can be precise as to "which subject is worse by X percentage points." This link has direct guiding significance for the optimization of training data ratio, design of course learning strategies, and data selection for domain enhancement and fine-tuning.

Model and version evolution of CMMLU

Mainline release

• First version of the paper (2023-06): Along with the arXiv paper public data set v1.0 and the evaluation agreement, 67 subject divisions and 5-shot evaluation specifications are established. The paper also reports the baseline results of GPT-4, ChatGPT and multiple early Chinese models. This is the initial milestone for CMMLU and all subsequent updates are based on this release.

• Dataset Update (~2024): Based on community feedback, question errors and answer labeling issues in some subjects have been corrected, and a small number of new subject questions have been added. The official version number is not listed separately, and the update record is subject to the GitHub commit log. Changes: About dozens of errors have been corrected, and the total number of subjects remains unchanged at 67.

• Continuous maintenance period (2024 to present): The warehouse uses issue to manage problem reports and suggestions fed back by the community, and regularly merges and corrects PRs. The baseline score table is continuously updated as new models are released, but structural changes to the data set ontology tend to be stable. Change Points: The number of baseline models continues to increase, and the core structure of the data set has not been significantly changed.

Key points of version context

The version evolution feature of CMMLU is the academic benchmark model of "one-time release of data set + continuous update of baseline", which is different from the semantic version management of commercial products. The data set itself has only been revised at the errata level since its release, and has not experienced structural expansion or version number jumps. When citing, the publication date of the paper and the GitHub commit hash shall prevail.

Version Milestone	Date	Core Changes
First release of paper	~2023-06	Public 67 subject data set v1.0, evaluation script, baseline scores
Errata update	~2024	Corrected some question errors, supplemented subjects, no version number jump
Continuous maintenance	2024 to present	The baseline model continues to expand and the data set structure remains stable

Technical advantages of CMMLU

Mechanism: CMMLU adopts the offline evaluation architecture of "static data set + local evaluation script". The data set stores the question stem, four options, and correct answer index of each question in JSON format; the evaluation script sequentially reads the questions, constructs prompts, calls model reasoning, compares answers, and counts accuracy by subject.

Effect:

• Completely reproducible offline: All evaluation logic is executed locally and does not rely on any external services. Any team using the same version of the data set and script can obtain consistent results. This eliminates the interference on scores caused by uncontrollable factors such as API version differences, service updates, and random sampling in online reviews.

• Fine-grained subject diagnosis: The granularity of 67 subjects is at the forefront of the Chinese evaluation benchmarks. Comparing MMLU's 57 subjects with C-Eval's 52 subjects, CMMLU has added a large number of subdivisions (such as Chinese political system, Chinese folklore, ancient Chinese history, etc.) to the "Chinese-specific knowledge" category, so that the model's mastery of Chinese local knowledge can be measured individually.

• Complementary design with MMLU: The subject setting of CMMLU and MMLU are not simply related to Chinese-English translation. The disciplines covered by MMLU are based on the Western knowledge system, while CMMLU adds a large number of knowledge and evaluation dimensions unique to China. By running MMLU and CMMLU on the same model at the same time, the difference in the two dimensions of "general knowledge ability" and "Chinese contextual knowledge ability" can be separated. For example, a model that performs well on MMLU may score low on the China-specific knowledge category of CMMLU, suggesting that the model has insufficient coverage of Chinese culture in the Chinese pre-training data.

Applicable scenarios: CMMLU is more suitable as a capability diagnostic tool in the model development stage, rather than as a final publicity indicator before release. Regularly running CMMLU evaluation during the training process can promptly detect the degradation of the model's ability in certain subjects (such as the forgetting of factual knowledge caused by over-alignment). For publishing reports, it is recommended to combine CMMLU with C-Eval, MMLU, AGIEval, etc. to form an evaluation suite to avoid over-interpretation of the scores of a single benchmark.

How to use CMMLU

Entrance control:

Entrance	Purpose	Cost
GitHub repository (haonan-li/CMMLU)	Data set download, evaluation script, baseline results, documentation	Free
arXiv paper	Method description, subject definition, early baseline	Free
Hugging Face Datasets	Directly load CMMLU through the datasets library	Free

Typical steps:

1. Get the dataset: git clone https://github.com/haonan-li/CMMLU.git, or load it directly through datasets.load_dataset("haonan-li/cmmlu").
2. Prepare the model: Ensure that the model to be evaluated can be loaded through AutoModelForCausalLM of Hugging Face transformers, or implement a compatible inference interface.
3. Run Evaluation: Execute the evaluation script, specify the model path, evaluation settings (zero-shot or 5-shot), output directory and other parameters. Example command: python eval.py --model Qwen/Qwen2-7B --shot 5 --output ./results.
4. View results: The script outputs the accuracy rate of each subject and the overall average accuracy rate, and also generates a visual comparison chart between subjects.

Adaptation Tips:

• prompt format must be unified: Different prompt templates (such as whether to add a role description, whether to use Chinese command prefixes) may cause the score to fluctuate by 2-5 percentage points. The version of prompt used must be noted when reporting scores, otherwise the results will not be comparable.
• Randomness of few-shot samples: In 5-shot evaluation, the order in which the samples are selected will affect the model performance. It is recommended to fix the random seed and average it over multiple runs to reduce the variance of a single sampling.
• Decoding Parameter Control: Evaluation should use greedy decoding (temperature=0) to avoid randomness introduced by the sampling strategy from interfering with accuracy statistics.
• Model tokenizer compatibility: The tokenizers of some models segment Chinese characters in different ways, which may affect the actual token length of the prompt and the model's understanding of the topic. It is recommended to verify that prompt can be correctly encoded on a small sample before formal evaluation.

Product Pricing for CMMLU

• Billing Model: Completely free and open source. No licensing fees, no subscription fees, no pay-per-use.
• Included costs: The computing power required for evaluation is borne by the user. Depending on the model scale, the computing power cost for a single full-scale evaluation is between $1-30 (based on mainstream cloud GPU on-demand pricing).
• Enterprise Privatization: Data sets can be run completely offline, with no data transfer outbound and no additional SaaS fees.
• No Hidden Terms: The GitHub open source license does not restrict commercial use, but when quoting CMMLU scores, the data version and evaluation settings must be indicated to keep the results reproducible.

Application scenarios of CMMLU

• Capability diagnosis in the model development stage: Run CMMLU evaluation regularly during the pre-training or fine-tuning process to monitor the changing trends of the model's capabilities in different subjects. Actual benefits: Capability degradation in certain subjects can be detected early (for example, the model forgets Chinese historical knowledge after over-alignment), and the training data ratio or training strategy can be adjusted in a timely manner. Acceptance focus: Set a subject-level accuracy threshold (for example, each subject should not be lower than 90% of the baseline level) as an access condition for whether checkpoint can enter the next stage.

• Competency Statement at Model Release: Report CMMLU scores in technical reports or release announcements as quantitative proof of Chinese knowledge proficiency. Actual benefits: Provide a comparable capability scale within the industry and reduce information asymmetry when users make selections. Acceptance concerns: When reporting scores, the evaluation settings (zero-shot/5-shot, prompt version, decoding parameters) must also be noted, and C-Eval and MMLU scores under the same conditions must be supplemented for cross-validation.

• Academic research and benchmark comparison: NLP researchers use CMMLU as an evaluation tool for Chinese language understanding, report the performance of the model on this benchmark in papers, or conduct evaluation method research (such as prompt sensitivity analysis, data pollution detection, etc.) based on CMMLU. Actual benefits: Provide a standardized evaluation platform for Chinese NLP research and reduce the evaluation cost of new models and new methods.

• Vertical field model selection evaluation: When selecting large models in Chinese industries (law, medical, finance, etc.), the scores of the corresponding disciplines (such as law, medicine, economics) in CMMLU are used as one of the reference dimensions for selection. Actual benefits: From the perspective of knowledge coverage, models with obvious advantages in the target field can be quickly selected and the scope of testing can be reduced. Acceptance Concerns: Only relevant subject scores shall be referred to, and the overall average score shall not be used as a substitute for field-specific assessment.

Applicable groups of CMMLU

• Large Model Algorithm Engineer and Researcher: Standardized Chinese knowledge evaluation is needed to verify the model training effect, detect capability degradation, and compare the impact of different training strategies. The subject-level diagnosis provided by CMMLU can directly guide data allocation adjustment and course learning design. Prerequisites: Possess model reasoning and basic Python script execution capabilities.

• NLP academic researcher: Engaged in research on Chinese language understanding, evaluation methodology, data pollution detection, etc., a set of publicly reproducible benchmarks are needed as an experimental platform. The open source nature and fine-grained subject division of CMMLU provide flexibility in experimental design. Prerequisite: Familiar with the Hugging Face ecosystem and standard evaluation process.

• Model Selection and Procurement Decision Maker: When enterprises purchase large models or select open source model bases, use the CMMLU score as one of the quantitative reference indicators for Chinese knowledge capabilities. Prerequisite: It is necessary to understand the limitations of the multiple-choice question benchmark and not use it as the only basis for decision-making.

• ❌ Not suitable for people:

  • End users who want direct access to available features such as conversation, writing, translation, etc. - CMMLU is a review tool and not an application product.
  • Scenarios that require assessing the depth of model reasoning, creativity, and quality of multiple rounds of dialogue—dimensions that the multiple-choice question format cannot measure.
  • Decision makers who only refer to a single overall score for model selection - the multiple-choice question benchmark is affected by data contamination, and the overall score may cover up serious shortcomings in key subjects. Comprehensive judgment needs to be made in conjunction with other evaluations and actual measurements.

Summary and Outlook

Core Competencies: CMMLU has become an indispensable part of the Chinese large model evaluation landscape with its fine-grained division of 67 disciplines, key coverage of China-specific knowledge, and completely offline reproducible open source architecture. It complements C-Eval - the former focuses on the difficulty gradient of general subjects, and the latter focuses on the depth of Chinese local knowledge - and the combination of the two can more completely characterize the model's Chinese knowledge ability structure.

Current Limitations:

• Data pollution risk: Static public data sets are easily "seen in advance" by the model during the pre-training stage, resulting in falsely high scores. Currently, CMMLU does not have a dynamic question generation or regular refresh mechanism, and long-term reliance on fixed versions will face more and more pollution problems.
• Single question type: There are only four multiple-choice questions, which cannot evaluate the model's generative reasoning, multi-step reasoning, open question and answer capabilities. High scores only reflect the amount of factual knowledge memorized and do not guarantee performance in practical applications.
• There are still gaps in subject coverage: Although the 67 subjects are already high-density, there is insufficient coverage in emerging interdisciplinary subjects (such as AI ethics, computational social sciences) and some vertical industries (such as pharmaceuticals, financial engineering).
• Insufficient standardization of evaluation protocols: Differences in prompt template few-shot settings and answer extraction between different teams still result in scores that are not directly comparable. The community needs a more stringent standard evaluation protocol.

Follow-up observation points:

• Whether CMMLU will launch a dynamic evaluation version of "Privacy Set" to combat data pollution.
• Can it be related and transformed with real task evaluations (such as Baichuan-TextEval, SuperCLUE) to help users understand the actual meaning of multiple-choice test scores in application scenarios.
• Whether the community will build vertical industry extension sets (such as CMMLU-Med, CMMLU-Law) based on the CMMLU framework.
• After the scores of mainstream models converge on CMMLU, whether the benchmark needs to increase difficulty or introduce adversarial samples to maintain discrimination.

Procurement and Adoption Risk Assessment: For teams planning to incorporate CMMLU into the evaluation system, it is recommended to take the following steps - short-term (1-2 weeks) integrate it into the existing evaluation pipeline, run it simultaneously with MMLU and C-Eval, and establish a baseline comparison; mid-term (1-3 months) focus on the subject-level diagnostic output of CMMLU to guide pre-training data ratio adjustment; long-term (more than 6 months) closely track CMMLU Data contamination trends, such as observed community-reported score inflation exceeding a reasonable range, should be considered to supplement other heterogeneous evaluation indicators. For the capability statement of a commercial model, it is recommended to always disclose the MMLU and C-Eval scores under the same conditions in addition to the CMMLU, and clearly indicate the evaluation settings (shot number, prompt version, decoding parameters) to avoid misleading model selection due to inconsistent evaluations.

Related tools: hugging-face, replicate

Version Info

• CMMLU evaluation benchmark (GitHub open source) ：The open-source comprehensive Chinese knowledge assessment benchmark covers multiple-choice questions in humanities, social sciences, science and engineering, and Chinese-specific knowledge. It provides data sets, evaluation scripts, and baseline scores. The official version is based on the warehouse and the paper. The consecutive minor version numbers are not listed separately. The dates are approximate. （~2023-06）
• CMMLU paper and data set first released ：The data set and evaluation protocol will be disclosed with the paper, and the subject division and baseline evaluation method will be established. The follow-up will mainly focus on warehouse updates and result supplements. The official has not disclosed the precise date, and the date is an approximation. （~2023-06）