llama.cpp/tests
Qin Yue Chen 8cf19d60dc
gguf : support big endian platform (#3552)
* check whether platform is 390x if yes->do not import immintrin.h

* support s390x big endian

* support --bigendian option for s390x
1. verified with baichuan7b-chat with float 16 on s390x
2. verified with baichuan7b-chat
3. verified with chinese-alpaca-2-13b-f16

* update format based on editor-config checker result

* Update convert-baichuan-hf-to-gguf.py

* 1. check in ggml.c if endianess is not match
2. update GGUF version
3. change get_pack_prefix to property
4. update information log

* always use "GGUF" as beginng of GGUF file

* Compare "GGUF" with file header char by char
1.  Set GGUF_MAGIC to "GGUF" string instead of int value
2. Compare "GGUF" char by char to ensure its byte order
3. Move bytes swap code from convert.py to gguf.py write_tensor_data

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
2023-10-20 14:19:40 +03:00
..
CMakeLists.txt Work on the BPE tokenizer (#3252) 2023-10-03 09:16:26 +02:00
test-c.c tests : add a C compliance test (#2848) 2023-08-30 09:20:26 +03:00
test-double-float.cpp gguf : support big endian platform (#3552) 2023-10-20 14:19:40 +03:00
test-grad0.cpp sync : ggml (conv 1d + 2d updates, UB fixes) (#3468) 2023-10-04 15:29:58 +03:00
test-grammar-parser.cpp gguf : new file format with flexible meta data (beta) (#2398) 2023-08-21 23:07:43 +03:00
test-llama-grammar.cpp gguf : new file format with flexible meta data (beta) (#2398) 2023-08-21 23:07:43 +03:00
test-opt.cpp sync : ggml (conv 1d + 2d updates, UB fixes) (#3468) 2023-10-04 15:29:58 +03:00
test-quantize-fns.cpp check C++ code with -Wmissing-declarations (#3184) 2023-09-15 15:38:27 -04:00
test-quantize-perf.cpp sync : ggml (conv 1d + 2d updates, UB fixes) (#3468) 2023-10-04 15:29:58 +03:00
test-rope.cpp llama : custom attention mask + parallel decoding + no context swaps (#3228) 2023-09-28 19:04:36 +03:00
test-sampling.cpp check C++ code with -Wmissing-declarations (#3184) 2023-09-15 15:38:27 -04:00
test-tokenizer-0-falcon.cpp Minor improvements in GPT2 tokenizer (#3567) 2023-10-10 18:59:52 +02:00
test-tokenizer-0-falcon.py Minor improvements in GPT2 tokenizer (#3567) 2023-10-10 18:59:52 +02:00
test-tokenizer-0-llama.cpp Minor improvements in GPT2 tokenizer (#3567) 2023-10-10 18:59:52 +02:00
test-tokenizer-0-llama.py Minor improvements in GPT2 tokenizer (#3567) 2023-10-10 18:59:52 +02:00
test-tokenizer-1-bpe.cpp Work on the BPE tokenizer (#3252) 2023-10-03 09:16:26 +02:00
test-tokenizer-1-llama.cpp Work on the BPE tokenizer (#3252) 2023-10-03 09:16:26 +02:00