Kerfuffle a5e7dbd614 llama : validate special token ids are in range when loading GGUF model (#3635) ... * Add validation for special token ids to llama.cpp Small optimization for llama_byte_to_token SPM mode * Fix BPE newline check, only I could break something so simple * Killll meeeeee * Account for GGUF_KEY_KEY only setting when the key exists * Minor code cleanups. * Fix convert.py error msg when added tokens are out of range * Make gguf SpecialVocab vocab size-aware Update conversion scripts accordingly * Avoid a string copy Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>		2023-10-22 21:14:56 +03:00
..
__init__.py	gguf : export objects to user code (#2780)	2023-08-25 12:43:41 +03:00
gguf.py	llama : validate special token ids are in range when loading GGUF model (#3635)	2023-10-22 21:14:56 +03:00
py.typed	convert : various script cleanups/fixes + merges and special token handling (#2842)	2023-08-30 11:25:50 +03:00