---
title: README
emoji: 📈
colorFrom: red
colorTo: red
sdk: static
pinned: false
---

# 🇩🇪 German Tokenizer Benchmark

A curated collection of German datasets for comprehensive tokenizer evaluation across diverse domains and text types.

This organization hosts datasets used by the [GerTokEval](https://github.com/stefan-it/german-tokenizer-benchmark) framework to evaluate German tokenizers with standardized metrics and fairness analysis.

## 🔎 Datasets

The following datasets are currently supported in the main framework:

* [GermEval 2014](https://huggingface.co/datasets/german-tokenizer-benchmark/germeval14)
* [BIOfid](https://huggingface.co/datasets/german-tokenizer-benchmark/biofid)
* [CO-Fun](https://huggingface.co/datasets/german-tokenizer-benchmark/co-funer)
* [German LER](https://huggingface.co/datasets/german-tokenizer-benchmark/german-ler)
* [DFKI MobIE](https://huggingface.co/datasets/german-tokenizer-benchmark/mobie)
* [UD HDT](https://huggingface.co/datasets/german-tokenizer-benchmark/ud-hdt)

The main goal for choosing these datasets is to evaluate tokenizers on a broad range of domains.

# ❤️ Acknowledgements

Many thanks to [Clara Meister](https://github.com/cimeister) for releasing the amazing [TokEval framework](https://github.com/cimeister/tokenizer-analysis-suite)!