ff5a3f0c09
* Work on the BPE tokenizer Tokenizer tests work for Falcon-7B * Try to fix build problem * Fix debug assertion failure * Fix MSVC Unicode BOM problem * Cleanup and an improvement * Fix compiler warning * Cleanup * Test doesn't work over the full range of Unicodes * Update .gitignore and Makefile * Another Makefile rule * Testing Aquila * Moving byte decoding back to `token_to_piece` ... ... because everyone is using it. * Guarding some unusable code pathes * Streamlining code and adding some more assertions Important change: I'm classifying added tokens as control tokens now for BPE. * Adding a comment * Adding another assertion * Fixed vocabulary guarding assertions * Fix PR for recent change * Fix PR for recent change * Fix for compiler warning * Fix PR for recent change * Fix PR for recent change * Fix PR for recent change * Fix for compiler warning * Fixes for more compiler warnings * Remove unused code * Fix initialization of static maps * Add scores and token types back, adapt gptneox * Update llama.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update unicode.h Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update unicode.h Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Ported Starcoder and added some assertions * Fix coding style * Apply @jploski 's fix for missing tokens --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
96 lines
1 KiB
Plaintext
96 lines
1 KiB
Plaintext
*.o
|
|
*.a
|
|
*.so
|
|
*.gguf
|
|
*.bin
|
|
*.exe
|
|
*.dll
|
|
*.log
|
|
*.gcov
|
|
*.gcno
|
|
*.gcda
|
|
*.dot
|
|
.DS_Store
|
|
.build/
|
|
.cache/
|
|
.direnv/
|
|
.envrc
|
|
.swiftpm
|
|
.venv
|
|
.clang-tidy
|
|
.vs/
|
|
.vscode/
|
|
|
|
lcov-report/
|
|
gcovr-report/
|
|
|
|
build*/
|
|
out/
|
|
tmp/
|
|
|
|
models/*
|
|
models-mnt
|
|
|
|
/Pipfile
|
|
/baby-llama
|
|
/beam-search
|
|
/benchmark-matmult
|
|
/convert-llama2c-to-ggml
|
|
/embd-input-test
|
|
/embedding
|
|
/gguf
|
|
/gguf-llama-simple
|
|
/infill
|
|
/libllama.so
|
|
/llama-bench
|
|
/main
|
|
/metal
|
|
/perplexity
|
|
/q8dot
|
|
/quantize
|
|
/quantize-stats
|
|
/result
|
|
/save-load-state
|
|
/server
|
|
/simple
|
|
/batched
|
|
/export-lora
|
|
/finetune
|
|
/speculative
|
|
/parallel
|
|
/train-text-from-scratch
|
|
/vdot
|
|
build-info.h
|
|
arm_neon.h
|
|
compile_commands.json
|
|
CMakeSettings.json
|
|
|
|
__pycache__
|
|
dist
|
|
|
|
zig-out/
|
|
zig-cache/
|
|
|
|
ppl-*.txt
|
|
qnt-*.txt
|
|
perf-*.txt
|
|
|
|
examples/jeopardy/results.txt
|
|
|
|
poetry.lock
|
|
poetry.toml
|
|
|
|
# Test binaries
|
|
tests/test-grammar-parser
|
|
tests/test-llama-grammar
|
|
tests/test-double-float
|
|
tests/test-grad0
|
|
tests/test-opt
|
|
tests/test-quantize-fns
|
|
tests/test-quantize-perf
|
|
tests/test-sampling
|
|
tests/test-tokenizer-0-llama
|
|
tests/test-tokenizer-0-falcon
|
|
tests/test-tokenizer-1-llama
|
|
tests/test-tokenizer-1-bpe
|