Article
Introducing the Open Leaderboard for Japanese LLMs!
- +5
AkimfromParis, miyao-yusuke, namgiH, t0-0, sh1gechan, hysts, clefourrier
โข โข 39Akim Mousterou
AkimfromParis
AI & ML interests
NLP for French, English, and Japanese - Opinion mining, dataset, evaluation, and quantitative research.
Recent Activity
posted an update 21 days ago
๐ธ ๐๐ฅ๐๐ฃ ๐
๐๐ฅ๐๐ฃ๐๐จ๐ ๐๐๐ ๐๐๐๐๐๐ง๐๐ค๐๐ง๐ ๐2 ๐ค๐ฃ ๐๐ช๐๐๐๐ฃ๐ ๐๐๐๐ ๐ฏ๐ต // ๐ธ ใใฎใณใฐใใงใคใน็ใ ๐ข๐ฝ๐ฒ๐ป ๐๐ฎ๐ฝ๐ฎ๐ป๐ฒ๐๐ฒ ๐๐๐ ๐๐ฒ๐ฎ๐ฑ๐ฒ๐ฟ๐ฏ๐ผ๐ฎ๐ฟ๐ฑ ๐ฉ๐ฎ ใๅ
ฌ้ ๐ฏ๐ต
I am thrilled to announce the launch of version 2 of the ๐๐ฅ๐๐ฃ ๐
๐๐ฅ๐๐ฃ๐๐จ๐ ๐๐๐ ๐๐๐๐๐๐ง๐๐ค๐๐ง๐. This initiative is driven by the "Fine-tuning and Evaluation" team, led by Professor Miyao at the The University of Tokyo, under the Research and Development Center for Large Language Models (LLMC) at Japanโs National Institute of Informatics (NII).
๐๐ฉ๐ง๐๐ฉ๐๐๐๐ ๐๐ฃ๐ ๐ฉ๐๐๐๐ฃ๐๐๐๐ก ๐ช๐ฅ๐๐ง๐๐๐๐จ:
- Our new backend features eight A100 GPUs, enabling the evaluation of open-source models of more than 100B parameters.
- Submissions now require a Hugging Face Hub login to ensure accountability.
- We have added metrics for evaluation time, COโ emissions (thx to Code Carbon ๐ฑ ), alongside reasoning capabilities.
๐ฟ๐๐ฉ๐๐จ๐๐ฉ๐จ ๐๐ฃ๐ ๐๐ซ๐๐ก๐ช๐๐ฉ๐๐ค๐ฃ ๐จ๐ฉ๐๐ฃ๐๐๐ง๐๐จ:
- New datasets cover reasoning, mathematics, exams, and instruction following.
- Math evaluations now span from grade-school levels to expert-tier challenges (GSM8K, PolyMath, AIME).
- While integrating English-heavy and multilingual benchmarks (including Humanityโs Last Exam, GPQA, and BBH in both English and Japanese), we continue to prioritize unique Japanese cultural datasets.
https://huggingface.co/spaces/llm-jp/open-japanese-llm-leaderboard-v2
ใฉใใใ้กใ่ดใใพใ๏ผ๐ upvoted a paper about 2 months ago
Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document CollectionsOrganizations
published an article over 1 year ago
published an article over 1 year ago
Article
Introducing the Open Leaderboard for Japanese LLMs!
- +5
AkimfromParis, miyao-yusuke, namgiH, t0-0, sh1gechan, hysts, clefourrier
โข โข 39