| --- |
| library_name: transformers |
| license: apache-2.0 |
| --- |
| |
|
|
| See [transformers で複数のトークナイザーを一つのプロセッサーで扱う](https://zenn.dev/platina/articles/732feb7c3e9852). |
|
|
| https://zenn.dev/platina/articles/732feb7c3e9852 |
|
|
|
|
| ## Example usage |
|
|
| ```py |
| from transformers import AutoProcessor |
| |
| processor = AutoProcessor.from_pretrained( |
| "p1atdev/multi-tokenizers-processor-sample", |
| trust_remote_code=True, |
| commit_hash="111e8a30609fb5bc13e16d08f7c49196b23d5056" |
| ) |
| |
| print(processor( |
| text_1="テキスト1", |
| text_2="テキスト2", |
| )) |
| # {'input_ids': tensor([[ 1, 43412, 28745]]), 'attention_mask': tensor([[1, 1, 1]]), 'input_ids_2': tensor([[56833, 61803, 70534, 17]]), 'attention_mask_2': tensor([[1, 1, 1, 1]])} |
| ``` |
|
|
|
|
|
|
|
|
|
|
