EduBenchEvaluator

This is a fine-tuned evaluator designed to assess LLM on the EduBench benchmark.

• 📄 Paper
• 💻 GitHub Repository

Model Details

• Model Name: EduBenchEvaluator
• Model Type: Fine-tuned language model (0.6B parameters)
• Base Model: Qwen3-Reranker-0.6B

Training & Methodology

The base model, Qwen3-Reranker-0.6B, was fine-tuned to align with human evaluations on the EduBench dataset.

We approached the fine-tuning process as a text classification task. The model evaluates a given response by taking a <question, answer, metric> triplet as input. Based on this context, it is trained to output a precise evaluation score ranging from 1 to 5.

This evaluator is specifically constructed to measure an LLM's capability across the diverse educational tasks presented in EduBench.

Performance

• Accuracy: The model achieves a satisfactory accuracy of 75.28% on the test set.
• Human Alignment: In addition to standard accuracy, we calculated the correlation between the model's predictions and actual human scorers, demonstrating that the model closely mirrors human judgment.

Note: Further evaluation results and comparisons are reported on our GitHub.

🫣 Citation

If you find our benchmark, evaluation pipeline, or models useful or interesting, please cite our paper:

@misc{xu2025edubenchcomprehensivebenchmarkingdataset,
      title={EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios}, 
      author={Bin Xu and Yu Bai and Huashan Sun and Yiguan Lin and Siming Liu and Xinyue Liang and Yaolin Li and Yang Gao and Heyan Huang},
      year={2025},
      eprint={2505.16160},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={[https://arxiv.org/abs/2505.16160](https://arxiv.org/abs/2505.16160)}, 
}