llama.cpp/models/ggml-vocab-mpt.gguf
goerch 9e70cc0322
Add test for MPT tokenization (#3728)
* Add test for MPT tokenization

* Revert code motion

* Remove unnecessary restriction in test case

* Clarify logic in conversion
2023-10-22 21:21:42 +02:00

1.7 MiB