Galunid daab3d7f45

Add more tokenizer tests (#3742) ...

* Add more tokenizer tests

* Add starcoder

* Update test vocab files

* Restrict bpe tokenizer tests to unicode planes

* Update comment

* Comment cosmetics

* Remove bloom vocab/test

2023-10-24 09:17:17 +02:00

1.6 MiB

Raw History

View raw