C-Eval is a Chinese comprehensive assessment suite for large language models. It contains 13,948 multiple-choice questions, covering 52 subjects and four difficulty levels. It also provides public rankings and is one of the commonly used academic benchmarks to measure the ability of basic Chinese models.

C-Eval

Core parameters and statistics of C-Eval

C-Eval is not an application for end users, but a set of Chinese large model evaluation benchmarks for researchers and model teams. Its core value is "to use a unified and comparable question bank to measure the Chinese knowledge and reasoning ability of the model." The pain point it solves is the lack of standardized, interdisciplinary horizontal evaluation scales in the Chinese scenario.

Projects	Public Information
Benchmark Type	Chinese Large Model Multidisciplinary Evaluation Kit
Question size	13948 multiple choice questions
Subject coverage	52 subjects
Difficulty levels	Four difficulty levels (Basic, Professional, Postgraduate, Comprehensive)
Question source	Chinese University Examination, Graduate Entrance Examination, Vocational Qualification Examination, Open Competition
Format	Public Question Bank + Online Ranking
Openness	Dataset and code GitHub open source (MIT license)
Billing Model	Free (Academic Benchmark)
Evaluation protocol	5-shot and 0-shot settings
Ranking scale	Continuously includes the results of dozens of mainstream models

Parameter meaning: 13948 questions covering the entire spectrum from basic subjects (mathematics, physics, chemistry) to professional fields (medicine, law, computer) to advanced reasoning (logic, ethics). The tiered design of 52 subjects enables the model not only to produce an overall score but also to break down competency profiles by subject. The four difficulty levels (basic, professional, graduate, and comprehensive) further distinguish the gap between models in simple common sense and high-order reasoning - the same model may be close to full score at the basic level, but drop significantly at the graduate level. This gap can better reflect the true depth of the model's capabilities than the total score.

Evaluation Protocol: C-Eval provides two evaluation settings of 5-shot (5 examples) and 0-shot (no examples) by default. The 5-shot results usually reflect the model's performance under a small amount of context learning (in-context learning), while the 0-shot results are closer to the model's "naked knowledge" mastery. The difference between the two can help determine how dependent the model is on the sample format.

Benchmark comparison: Compared with similar Chinese evaluation benchmarks, C-Eval has obvious differences in subject breadth and difficulty stratification.

Benchmark	Number of questions	Subject/area	Difficulty level	Organizational form	Language
C-Eval	13948	52 subjects	Four levels (Basic/Professional/Graduate/Comprehensive)	Multiple choice	Chinese
MMLU	~15900	57 subjects	No clear stratification	Multiple choice	English
CMMLU	~10000	67 subjects	No clear stratification	Multiple choice	Chinese
AGIEval	~6800	20+ exam types	Score by exam source	Multiple choice + fill in the blank	Chinese-English
GAOKAO	~4500	9 subject college entrance examination questions	divided by grade	multiple choice + fill in the blanks	Chinese

Publicity Verification: The official website clearly positions it as "a multi-level and multi-disciplinary Chinese assessment suite suitable for large language models (2023)". Key parameters such as 13948 questions and 52 disciplines, four levels of difficulty, and open source protocols can be directly verified on the official website and the GitHub repository.

User and market recognition of C-Eval

• Research and Industry Adoption: C-Eval has become the de facto "entrance test" for Chinese large models. Almost all large domestic models (including DeepSeek, Tongyi Qianwen, Wenxinyiyan ChatGLM, Baichuan, Yi series, etc.) will list C-Eval scores when releasing technical reports or publicity as a key reference indicator of Chinese ability. According to public technical report statistics, there are more than 15 models with a total score of more than 90% in C-Eval's five classification accuracy rates (as of mid-2026), reflecting the continued attention of this benchmark in model iterations.
• Verifiable data: 13,948 questions, 52 subjects, and four difficulty levels are all publicly available on the official website. The rankings are continuously updated on the official website, and the score entries submitted by each model are accompanied by submission time and setting instructions (such as 5-shot / 0-shot).
• Academic citations: C-Eval’s original paper has been cited more than 2,000 times on Google Scholar (as of mid-2026), becoming one of the most highly cited benchmark papers in the field of Chinese NLP evaluation.
• Community Ecology: The GitHub repository (hkust-nlp/ceval) publicly displays 400+ stars, 100+ forks, and active Issue and PR discussions, reflecting the academic community's concern for this standard and its willingness to continuously improve.
• Boundary Description: Ranking scores are submitted by each model team or obtained by third-party evaluation. There are differences in evaluation settings (changes in Prompt wording, batch size, output decoding strategy, etc.). As a single-choice question benchmark, C-Eval does not cover dimensions such as generation quality, instruction following, and multiple rounds of dialogue, and cannot fully represent the model's comprehensive performance in real tasks. In addition, since the question bank is a static public question bank, there is a risk of being "memorized" by pre-training data (data pollution), that is, the model may have seen some questions during the training phase, thereby obtaining falsely high scores.

Cost Advantages of C-Eval

C-Eval itself does not charge any usage fees, but the complete cost of the evaluation needs to be broken down from the three levels of "individual researcher", "academic/model team" and "enterprise industrialization evaluation".

C-side/Personal researcher

• Tool Cost: Zero cost. The data set and evaluation code are completely open source and can be downloaded directly from GitHub and used.
• Computing power cost: Bring your own GPU. A single complete evaluation of 13948 questions takes about 1-4 hours on a single card A100-80G (depending on the model size and decoding efficiency). For small models with less than 7B parameters, a single consumer-grade GPU (such as RTX 4090) can complete the evaluation.
• Hidden Cost: You need to correctly configure the evaluation context (Python, PyTorch and transformers dependencies) and understand the evaluation protocol (5-shot template format, answer extraction rules). Newbies may spend 1-3 days on this section.

API Developers and Model Team

• Tool Cost: Zero cost.
• Integration cost: Integrating the C-Eval evaluation script into the model training pipeline (such as automatically executing after each checkpoint is saved) requires engineering investment. The official Python evaluation script is provided, but advanced requirements such as batch model version comparison, result visualization, and anomaly detection need to be developed by yourself.
• Comparison Cost: If you need to make a fair comparison with competing models under the same settings, you need to prepare your own inference context or API access for the competing models. This part of the cost falls on the model calling side, not C-Eval itself.

Enterprise and large-scale evaluation needs

• Tool Cost: Zero cost. There are no commercial licensing fees, and the MIT open source license allows commercial use and redistribution.
• Scale Evaluation Cost: If an enterprise needs to conduct multiple evaluations on dozens of model variants (such as controlled experiments with different fine-tuning strategies), the machine cost will increase linearly. The official script supports parallel evaluation, but large-scale solutions require the team to build a distributed evaluation framework by themselves.
• Compliance Cost: None. The C-Eval question bank comes from public academic materials and examination questions, and the data set license is MIT, with no additional compliance risks. However, if an enterprise uses its own assessment set or expanded questions, it needs to confirm the rights by itself.

Comparison dimensions: Compared with self-built evaluation sets, C-Eval provides a ready-made standardized question bank and unified list, saving construction (usually requiring 2-5 man-months) and alignment costs. Compared with using English MMLU for evaluation, C-Eval is oriented to Chinese scenarios, covers the Chinese subject system, and can more accurately reflect the true level of the model in Chinese knowledge and reasoning.

Main functions of C-Eval

• Standardized question bank: 13948 multiple-choice questions covering 52 subjects, each question has 4 options, unified format, and supports automatic scoring. Subjects are divided into four categories: humanities and social sciences (such as history, philosophy, economics), STEM (such as mathematics, physics, computer science), medicine (such as clinical medicine, pharmacy, basic medicine) and others (such as logic, ethics, law), covering the core knowledge system of China's higher education and vocational qualification examinations.
• Multiple Difficulty Stratifications: Four difficulty levels - Basic (corresponding to basic undergraduate knowledge), Professional (corresponding to in-depth knowledge in professional fields), Postgraduate (corresponding to advanced concepts at the graduate level), Comprehensive (interdisciplinary comprehensive application). This stratification can not only measure the "breadth" of the model, but also distinguish the "depth" - a model that scores 90% at the basic level and only 50% at the graduate level, and a model that scores 80% at both levels, may correspond to completely different research and development paths (the former needs to enhance high-order reasoning, and the latter needs to repair knowledge blind spots).
• Public Ranking: The official website continuously collects the 5-shot and 0-shot scores of each model, and sorts them by the overall average score. The ranking list comes with subject breakdown, submission date and evaluation settings for each score, making it easy to refer horizontally. There are no submission process restrictions for the rankings, but submitters are required to indicate the evaluation configuration to maintain basic comparability.
• Open source code and data: The GitHub repository provides a complete data set (JSON format), evaluation script (Python, based on lm_eval framework), Prompt template and answer extraction logic. Supports custom model access, just implement the standard text generation interface (input→output→scoring). The community already has many interpretation articles and third-party improved versions (such as multi-language expansion, adaptive evaluation difficulty adjustment, etc.).
• Subject Level Diagnostic Report: C-Eval’s subject classification granularity supports the ability to generate models

"radar chart". For example, a model might score highly in Computer Science and Mathematics, but score significantly low in Forensic Science or Ethics. This granular diagnosis can guide targeted training data supplementation better than a single total score.

C-Eval’s model and version evolution

As an academic benchmark, C-Eval's "version iteration" is mainly reflected in three levels: data set maintenance, ranking list update and evaluation protocol optimization, rather than the increment of software version number in the traditional sense.

1. Initial release (2023-05)

• First published with the paper "C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Large Language Models".
• Establish a core structure of 13948 questions in 52 subjects and four difficulty levels.
• The initial ranking list is made public, including the results of the first batch of models such as GPT-4, ChatGPT, and Claude.
• When the paper was published, the total score of GPT-4 under the 5-shot setting was about 68%, while the highest score of the Chinese open source model at that time was about 45%-50%, reflecting the significant gap in Chinese knowledge between Chinese and foreign models.

2. Data set improvement and protocol fine-tuning (2023-06 to 2024-06)

• Driven by community feedback, we will gradually fix incorrect answers to some questions and deviations in subject classification (GitHub Issue can be traced).
• 5-shot is clearly recommended as the default evaluation protocol and the Prompt template is standardized to improve the comparability of results.
• Added compatibility testing for different model series (MoE architecture, reinforcement learning fine-tuning models, etc.).
• The number of models included in the rankings has been expanded from the initial dozen to dozens, covering mainstream open source and closed source models.

3. Continued influence expansion (2024-06 to present)

• C-Eval score has become the "standard" indicator for Chinese large model technical reports.
• Subsequent derivative work (such as C-Eval Hard, C-Eval Zero) attempts to increase the difficulty of adversarial sample or zero sample evaluation based on C-Eval.
• The original paper has become a highly cited benchmark paper in the field of Chinese NLP evaluation.
• Facing the challenge of data contamination: As the original data set is widely disclosed, subsequent training models may be passively exposed to questions through training data leakage, resulting in falsely high scores. The community is already discussing building a dynamic question bank with invisible questions as an alternative.

Technical advantages of C-Eval

• Dual-dimensional design of subject and difficulty: The core innovation of C-Eval is to organize questions from two orthogonal dimensions of "subject area" and "difficulty level" at the same time. This makes the evaluation output not just a total score, but a 52×4 capability matrix, with each grid representing the model's performance on a specific difficulty level in a specific subject. In the context of "wide caliber and thick foundation" in China's higher education, this granularity is better able to capture the shortcomings of the model's knowledge structure than MMLU's single subject classification.
• Chinese Reproducible Evaluation Protocol: C-Eval clearly stipulates the prompt format, answer extraction rules and scoring standards for 5-shot and 0-shot. Compared with some benchmarks that only provide data sets but do not strictly agree on evaluation protocols, C-Eval's high degree of normalization ensures that the results of different teams and different time points are basically horizontally comparable. This is the key engineering reason for its widespread adoption.
• Examination-driven question design: The questions are sourced from real Chinese examinations (university finals, postgraduate entrance examinations, professional qualification examinations), not self-compiled by researchers. This means that the questions naturally have: (1) The language is real and natural, without the artificial feeling of "dataset language"; (2) The answers have clear authoritative standards and there is no subjective ambiguity; (3) They are aligned with the Chinese educational knowledge system and can directly reflect the knowledge level of the model in the Chinese context. This also brings limitations - the timeliness of the questions is limited by the examination time, and the coverage of emerging fields (such as AI ethics, quantum computing and other subjects that have not yet entered standardized examinations) is low.
• Evaluation - Diagnosis - Closed: The technical value of C-Eval is not just "out of a score", but lies in its diagnostic capabilities at subject granularity. After receiving the C-Eval results, a model team can immediately locate weak subjects and then supplement the pre-training data or fine-tuning data in that subject in a targeted manner. This process from evaluation to improvement to the next round of evaluation,

Make C-Eval a dashboard for model capability iteration rather than a finish line.

How to use C-Eval

Entrance comparison

Purpose	Entry	Description
View description and rankings	https://cevalbenchmark.com/	Official website, browse positioning, rankings, subject classifications
Download data set and code	https://github.com/hkust-nlp/ceval	GitHub repository, complete data set and evaluation script
Read the paper	https://arxiv.org/abs/2305.10957	arXiv, understand the design principles and baseline results
Community discussion	GitHub Issues, Zhihu	Feedback issues, discussion and evaluation settings

Standard evaluation steps

1. Boundary preparation: Clone the repository and install Python dependencies (PyTorch, transformers, datasets, etc.).
2. Data loading: Use the ceval/ data directory provided by the warehouse, or load ceval/ceval through Hugging Face datasets.
3. Configure Evaluation: Set evaluation parameters - model name or path, evaluation settings (5-shot or 0-shot), maximum generation length, etc.
4. Run evaluation: Execute the evaluation script, and the script will automatically load the model → perform reasoning and answer → extract the answer → compare with the standard answer → calculate the accuracy of each subject and the overall accuracy.
5. Result Analysis: Check the output results (accuracy rate of each subject and overall average), and compare with the rankings to locate the gap.

Key Notes

• Prompt Consistency: Small changes in the prompt template (such as "Answer is" vs. "Answer:") can cause scores to fluctuate by 1-3 percentage points. Make sure the prompt used is consistent with the official default template, otherwise the results will not be comparable.
• Decoding Strategy: It is recommended to use greedy decoding (temperature=0) or consistent settings. Strategies such as beam search and sampling will cause unstable output and affect reproducibility.
• Subject Weighting: The overall average score in the rankings adopts the subject weighted average (rather than a simple arithmetic average). The specific weight varies depending on the number of subject questions. When comparing the rankings, you should confirm that the weighting methods are consistent.

Product Pricing for C-Eval

User Level	Cost Item	Description
C-side/individual researchers	Free	The data set and code are completely open source, MIT license, zero licensing fee
Academic/Model Team	Free	No subscription fees or pay-per-view, unlimited reviews
Enterprise	Free (tool layer)	Zero cost for tool layer; large-scale evaluation requires self-prepared computing cluster
Commercial use	Free	MIT License permits commercial use and redistribution without additional permission

Included costs: The main cost of the evaluation is not the tool itself, but the model inference computing power and engineering integration. The computing power consumption of a complete C-Eval evaluation (13948 questions, 5-shot) depends on the model size - a 7B model requires about 0.5-1 GPU hours (A100), a 70B model requires about 4-8 GPU hours, and a 700B+ model may require more than 24 hours. If frequent evaluation is required (such as after each training checkpoint), the total computing power cost will accumulate quickly.

Application scenarios of C-Eval

• Model development self-test: When training or fine-tuning a large Chinese model, perform C-Eval evaluation after each key checkpoint to verify whether the iterative direction of knowledge and reasoning capabilities is correct. Acceptance Points: Compare the subject-level changes between the two evaluations before and after - if the total score increases but a certain subject drops significantly, it may mean catastrophic forgetting, and the training data ratio needs to be checked.
• External Capability Disclosure: When the model is released or upgraded, C-Eval results will be listed in the technical report or public page as a third-party reference for Chinese proficiency. Key points for acceptance: Clearly mark the evaluation settings (5-shot/0-shot, Prompt version, scoring method) to ensure alignment with the rankings; if non-standard settings are used, the scores under the standard settings should also be given for horizontal comparison.
• Subject Diagnosis and Training Guidance: Locate the model's capability shortcomings through C-Eval's subject granularity results to guide subsequent data collection and training strategies. For example, if the model continues to perform poorly in the disciplines of "Law" and "Medicine", you can consider adding Chinese corpus of the corresponding disciplines in the pre-training stage, or supplementing the question-answer pairs of the disciplines in the SFT stage. Acceptance key: The diagnostic conclusion needs to be cross-validated with the training data distribution - a low score in a certain subject may be due to insufficient knowledge training in the subject, or it may be a generalization problem caused by the inconsistency between the C-Eval question expression method and the training data distribution.
• Academic Research and Benchmark Construction: As a reference baseline for Chinese NLP evaluation, used to compare newly proposed evaluation methods or data sets, or as a standardized platform for cross-model capability comparison. Acceptance Point: Newly proposed evaluation methods should be verified on C-Eval for correlation with the actual capabilities of the model.

Applicable groups of C-Eval

• Large Model Researchers and Algorithm Engineers: Standardized Chinese evaluation is needed to quantify model iteration effects and guide training strategies through subject granular diagnosis. Prerequisites: Have basic model reasoning and contextual construction capabilities, and understand the Prompt project and evaluation protocol.
• Model Evaluation Engineer: Responsible for building an automated evaluation pipeline for the team, integrating C-Eval into the CI/CD process, and continuously tracking the capability changes of model versions. Prerequisites: Familiar with Python and mainstream large model inference frameworks (Hugging Face Transformers, vLLM, etc.), and able to handle distributed evaluation scenarios.
• Academic Researchers: Engaged in Chinese NLP evaluation, benchmark research and cross-model analysis, using C-Eval as the baseline to compare new methods or new data sets. Prerequisite: Understand the evaluation methodology and pay attention to the impact of data contamination issues on experimental results.
• Technical Selection Evaluator (CTO/Technical VP): Incorporate C-Eval scores into the evaluation matrix as a quantitative dimension of Chinese proficiency before purchasing or introducing base models. Prerequisites: Recognize the limitations of a single benchmark and make comprehensive judgments based on scenario-based assessments (such as product functional testing, small-scale grayscale verification).
• Unfit Boundary: (1) Ordinary users who need to directly obtain "available product features" - C-Eval is an evaluation benchmark rather than an interactive application and cannot be used directly; (2) Teams who need to evaluate the model's performance on open tasks such as dialogue quality, security, and idea generation - C-Eval The multiple-choice question format cannot cover the measurement of generative ability; (3) Teams worried about data contamination - the static question bank may be covered by pre-training data, it is recommended to combine other invisible question banks (such as internal self-built evaluation sets) for cross-validation; (4) Teams that need to evaluate models in non-Chinese scenarios - C-Eval is completely oriented to Chinese and Chinese subject systems, and English model evaluation should be optimized

Start by using MMLU or an equivalent English benchmark.

Summary and Outlook

With 13,948 questions in 52 subjects and four difficulty levels, C-Eval constitutes one of the most systematic Chinese large-scale model evaluation benchmarks. Its core competitiveness lies in the refined design of the dual dimensions of subject and difficulty, the reproducible evaluation protocol, and the continuously updated rankings - these three together elevate C-Eval from a data set to a standardized evaluation ecosystem. For the Chinese basic model team, C-Eval is both an "entrance test" and a "diagnostic tool": the total score measures the level of proficiency, and the subject-level scores reveal the direction of shortcomings.

Current limitations: (1) The static question bank faces the risk of data contamination. As the question bank is widely disclosed, the post-training model may passively obtain questions through training data leakage, resulting in falsely high scores and reducing the credibility of the benchmark; (2) The multiple-choice question format cannot cover the evaluation needs of generative tasks (such as translation quality, code function correctness, creative writing); (3) Subject coverage is biased towards traditional knowledge systems, which is not good for AI Forward-looking subjects such as ethics, quantum computing, and emerging engineering fields are insufficiently covered; (4) The submission and review of the rankings relies on an autonomous mechanism, and the comparability of scores among various models is affected by differences in evaluation settings.

Future Directions: The community is already exploring the dynamic question bank version of C-Eval (such as non-public question pool, adaptive question setting) to alleviate the problem of data pollution; at the same time, the multi-modal version and Chinese-English bilingual expansion are also natural extension directions. Continuous community maintenance and topic updates are key for C-Eval to maintain its benchmark status.

Procurement/Adoption Risk Assessment: For teams that are evaluating basic models or self-developed large models, C-Eval is a low-cost, high-signal-to-noise ratio entry-level evaluation tool - zero licensing fee, open source and reproducible, and subject-granular diagnostic functions can directly guide training. But it should not be used as the only evaluation criterion. It is recommended to use a multi-dimensional evaluation combination of "C-Eval + several scenario-based internal evaluations + small-scale manual evaluation" during model selection or R&D iterations to reduce the selection risk caused by a single benchmark deviation. When teams use C-Eval scores for horizontal comparison, they must confirm that all scores are obtained under the same evaluation settings (5-shot/0-shot, the same Prompt template) to avoid misleading conclusions due to differences in settings.

Related tools: hugging-face, replicate

Version Info

• C-Eval Evaluation Kit (2023) ：The publicly released Chinese large model multi-disciplinary assessment kit contains 13,948 multiple-choice questions in 52 subjects and four difficulty levels, and provides online rankings to continuously record model results. The official page is marked with the year 2023, and the exact date is subject to the paper and warehouse. （~2023-05）
• C-Eval paper and data set first released ：C-Eval will be released for the first time with papers and data sets, a question bank, subject division and evaluation protocol will be established, and rankings will be launched online. There is no official official list of subsequent minor version numbers, and iteration is mainly reflected in the continuous updating of the rankings. （~2023-05）