45 lines
566 B
Markdown
45 lines
566 B
Markdown
# quantize
|
|
|
|
TODO
|
|
|
|
## Llama 2 7B
|
|
|
|
Quantization | Bits per Weight (BPW)
|
|
-- | --
|
|
Q2_K | 3.35
|
|
Q3_K_S | 3.50
|
|
Q3_K_M | 3.91
|
|
Q3_K_L | 4.27
|
|
Q4_K_S | 4.58
|
|
Q4_K_M | 4.84
|
|
Q5_K_S | 5.52
|
|
Q5_K_M | 5.68
|
|
Q6_K | 6.56
|
|
|
|
## Llama 2 13B
|
|
Quantization | Bits per Weight (BPW)
|
|
-- | --
|
|
Q2_K | 3.34
|
|
Q3_K_S | 3.48
|
|
Q3_K_M | 3.89
|
|
Q3_K_L | 4.26
|
|
Q4_K_S | 4.56
|
|
Q4_K_M | 4.83
|
|
Q5_K_S | 5.51
|
|
Q5_K_M | 5.67
|
|
Q6_K | 6.56
|
|
|
|
# Llama 2 70B
|
|
|
|
Quantization | Bits per Weight (BPW)
|
|
-- | --
|
|
Q2_K | 3.40
|
|
Q3_K_S | 3.47
|
|
Q3_K_M | 3.85
|
|
Q3_K_L | 4.19
|
|
Q4_K_S | 4.53
|
|
Q4_K_M | 4.80
|
|
Q5_K_S | 5.50
|
|
Q5_K_M | 5.65
|
|
Q6_K | 6.56
|