99009e72f8
* Starting to add k-quantization to ggml I think it is better to have quantization separate from ggml. For now just adding the k-quants there, but it would be better to also factor out the existing ggml quantizations. * Adding Q3_K and Q8_K (de)-quantization * Q3_K now working on CUDA and AVX2/scalar CUDA is not ideal - ~50% slower than Q4_0 for single token prediction, about the same in batch mode (perplexity). CPU single token is ~55 ms (on Ryzen 7950X). * Some improvement for Q3_K on CUDA It is now ~22.5 ms/token on my GPU, so ~30% slower than Q4_0. * Some more CUDA optimizations for Q3_K Single token is now 20.5 ms/token (~20% slower than Q4_0). Perplexity is on par with Q4_0. * Adding Q4_K - scalar, AVX2, CUDA Performance is the same or perhaps very slightly better than Q4_0 on the CPU. On the GPU, single token prediction is ~10% better than Q4_0, batch mode (perplexity is about the same). * Adding Q6_K - scalar, AVX2, CUDA Performance is ~40% lower compared to Q4_K on the CPU. This is to be expected, considering that we are memory bound on the CPU and the 6-bit model is ~44% larger than the 4-bit. On the GPU, single token prediction is ~6% lower than Q4_0, batch mode (perplexity) is even closer (but still slower). * Adding Q5_K - scalar, AVX2, CUDA Performance is ~20% lower compared to Q4_K on the CPU. This is to be expected, considering that we are memory bound on the CPU and the 5-bit model is ~22% larger than the 4-bit. On the GPU, single token prediction is about the same as Q4_0 for both, single token and batch prediction. * Per convention, all QX_K quantizations use Q5_K for output.weight * Adding quantization mixes * Quantization mixes: didn't quite get what I wanted in the last commit * Q4_K dot product for ARM_NEON * Q6_K dot product for ARM_NEON * Q5_K dot product for ARM_NEON * Adding Q3_K dot for ARM_NEON It is 22% slower than Q4_K, despite the smaller model size. On x86_64, where we are memory bound, the Q3_K model is quite a bit faster than Q4_K. * A very slightly faster ARM_NEON Q3_K dot * Adding Q2_K - just CUDA for now Token prediction is pretty good - about 15.5 ms on a RTX 4080. Perplexity is about the same as Q4_K. * Adding scalar and AVX2 Q2_K dot * Adding ARM_NEON Q2_K dot About the same performance as Q4_K. * A slightly faster ARM_NEON Q2_K dot Single token prediction is now ~36 ms on M2 Max. The code is much simpler too. * Fixed bug in Q2_K CUDA dot product kernel Stranegly enough, for the few prompts I tried with the 7B model the responses looked perfectly reasonable. Only realized something is not quite right when I tried the larger models and started getting nonse back. In any case, Q2_K single token evaluation time on an RTX 4080 in a Ryzen7950X box iusing CUDA and model fully loaded on the GPU are ~15.5 ms for 7B, ~25.4 ms for 13B, and ~55.8 ms for 30B. The max number of layers that fit in VRAM for The 65B is 32. With that, we get ~330 ms per token, which is not that much faster than just running on the CPU (~470 ms per token). * Don't print zeros/NaNs when no count histogram has been collected * A 10% faster CUDA vector dot kernel for Q3_K Q3_K is now running at ~18.5 ms / token on CUDA, so the gap to Q4_0 is only 10%. It seems memory acccess pattern is more important for performance than the amount of computation the kernel does. * A slightly daster Q4_K AVX2 dot product For perplexity, where we are less memory bound, time per pass drops by ~5%. Barely measurable difference for single token prediction. * A slightly faster ARM_NEON A4_K dot product * Minor * Fix quantization error test We cannot possibly be expecting rmse < 0.002 for 2- and 3-bit quantization variants. * Fix docker build I have been sloppy with vector reinterpret casts on ARM_NEON. It seems clang is very forgiving in that regard. * Added forgotten ggml.o dependence on k_quants.h to the Makefile * Had unintentionally committed the Makefile with -Ofast enabled * ggml : rename k_quants -> ggml-quants-k, use lowercase in code --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
123 lines
5.7 KiB
C
123 lines
5.7 KiB
C
#pragma once
|
|
|
|
#include "ggml.h"
|
|
|
|
#include <stdint.h>
|
|
#include <assert.h>
|
|
#include <stddef.h>
|
|
|
|
// Super-block size
|
|
#define QK_K 256
|
|
|
|
//
|
|
// Super-block quantization structures
|
|
//
|
|
|
|
// 2-bit quantization
|
|
// weight is represented as x = a * q + b
|
|
// 16 blocks of 16 elemenets each
|
|
// Effectively 2.5625 bits per weight
|
|
typedef struct {
|
|
uint8_t scales[QK_K/16]; // scales and mins, quantized with 4 bits
|
|
uint8_t qs[QK_K/4]; // quants
|
|
ggml_fp16_t d; // super-block scale for quantized scales
|
|
ggml_fp16_t dmin; // super-block scale for quantized mins
|
|
} block_q2_k;
|
|
static_assert(sizeof(block_q2_k) == 2*sizeof(ggml_fp16_t) + QK_K/16 + QK_K/4, "wrong q2_k block size/padding");
|
|
|
|
// 3-bit quantization
|
|
// weight is represented as x = a * q
|
|
// 16 blocks of 16 elemenets each
|
|
// Effectively 3.4375 bits per weight
|
|
typedef struct {
|
|
uint8_t hmask[QK_K/8]; // quants - high bit
|
|
uint8_t qs[QK_K/4]; // quants - low 2 bits
|
|
uint8_t scales[3*QK_K/64]; // scales, quantized with 6 bits
|
|
ggml_fp16_t d; // super-block scale
|
|
} block_q3_k;
|
|
static_assert(sizeof(block_q3_k) == sizeof(ggml_fp16_t) + QK_K / 4 + 11 * QK_K / 64, "wrong q3_k block size/padding");
|
|
|
|
// 4-bit quantization
|
|
// 16 blocks of 32 elements each
|
|
// weight is represented as x = a * q + b
|
|
// Effectively 4.5 bits per weight
|
|
typedef struct {
|
|
ggml_fp16_t d; // super-block scale for quantized scales
|
|
ggml_fp16_t dmin; // super-block scale for quantized mins
|
|
uint8_t scales[3*QK_K/64]; // scales and mins, quantized with 6 bits
|
|
uint8_t qs[QK_K/2]; // 4--bit quants
|
|
} block_q4_k;
|
|
static_assert(sizeof(block_q4_k) == 2*sizeof(ggml_fp16_t) + 3*QK_K/64 + QK_K/2, "wrong q4_k block size/padding");
|
|
|
|
// 5-bit quantization
|
|
// 16 blocks of 32 elements each
|
|
// weight is represented as x = a * q + b
|
|
// Effectively 5.5 bits per weight
|
|
typedef struct {
|
|
ggml_fp16_t d; // super-block scale for quantized scales
|
|
ggml_fp16_t dmin; // super-block scale for quantized mins
|
|
uint8_t scales[3*QK_K/64]; // scales and mins, quantized with 6 bits
|
|
uint8_t qh[QK_K/8]; // quants, high bit
|
|
uint8_t qs[QK_K/2]; // quants, low 4 bits
|
|
} block_q5_k;
|
|
static_assert(sizeof(block_q5_k) == 2*sizeof(ggml_fp16_t) + 3*QK_K/64 + QK_K/2 + QK_K/8, "wrong q5_k block size/padding");
|
|
|
|
// 6-bit quantization
|
|
// weight is represented as x = a * q
|
|
// 16 blocks of 16 elemenets each
|
|
// Effectively 6.5625 bits per weight
|
|
typedef struct {
|
|
uint8_t ql[QK_K/2]; // quants, lower 4 bits
|
|
uint8_t qh[QK_K/4]; // quants, upper 2 bits
|
|
int8_t scales[QK_K/16]; // scales, quantized with 8 bits
|
|
ggml_fp16_t d; // super-block scale
|
|
} block_q6_k;
|
|
static_assert(sizeof(block_q6_k) == sizeof(ggml_fp16_t) + QK_K / 16 + 3*QK_K/4, "wrong q6_k block size/padding");
|
|
|
|
// This is only used for intermediate quantization and dot products
|
|
typedef struct {
|
|
float d; // delta
|
|
int8_t qs[QK_K]; // quants
|
|
int16_t bsums[QK_K/16]; // sum of quants in groups of 16
|
|
} block_q8_k;
|
|
static_assert(sizeof(block_q8_k) == sizeof(float) + QK_K + QK_K/16*sizeof(int16_t), "wrong q8_k block size/padding");
|
|
|
|
|
|
// Quantization
|
|
void quantize_row_q2_k_reference(const float * restrict x, block_q2_k * restrict y, int k);
|
|
void quantize_row_q3_k_reference(const float * restrict x, block_q3_k * restrict y, int k);
|
|
void quantize_row_q4_k_reference(const float * restrict x, block_q4_k * restrict y, int k);
|
|
void quantize_row_q5_k_reference(const float * restrict x, block_q5_k * restrict y, int k);
|
|
void quantize_row_q6_k_reference(const float * restrict x, block_q6_k * restrict y, int k);
|
|
void quantize_row_q8_k_reference(const float * restrict x, block_q8_k * restrict y, int k);
|
|
|
|
void quantize_row_q2_k(const float * restrict x, void * restrict y, int k);
|
|
void quantize_row_q3_k(const float * restrict x, void * restrict y, int k);
|
|
void quantize_row_q4_k(const float * restrict x, void * restrict y, int k);
|
|
void quantize_row_q5_k(const float * restrict x, void * restrict y, int k);
|
|
void quantize_row_q6_k(const float * restrict x, void * restrict y, int k);
|
|
void quantize_row_q8_k(const float * restrict x, void * restrict y, int k);
|
|
|
|
// Dequantization
|
|
void dequantize_row_q2_k(const block_q2_k * restrict x, float * restrict y, int k);
|
|
void dequantize_row_q3_k(const block_q3_k * restrict x, float * restrict y, int k);
|
|
void dequantize_row_q4_k(const block_q4_k * restrict x, float * restrict y, int k);
|
|
void dequantize_row_q5_k(const block_q5_k * restrict x, float * restrict y, int k);
|
|
void dequantize_row_q6_k(const block_q6_k * restrict x, float * restrict y, int k);
|
|
void dequantize_row_q8_k(const block_q8_k * restrict x, float * restrict y, int k);
|
|
|
|
// Dot product
|
|
void ggml_vec_dot_q2_k_q8_k(int n, float * restrict s, const void * restrict vx, const void * restrict vy);
|
|
void ggml_vec_dot_q3_k_q8_k(int n, float * restrict s, const void * restrict vx, const void * restrict vy);
|
|
void ggml_vec_dot_q4_k_q8_k(int n, float * restrict s, const void * restrict vx, const void * restrict vy);
|
|
void ggml_vec_dot_q5_k_q8_k(int n, float * restrict s, const void * restrict vx, const void * restrict vy);
|
|
void ggml_vec_dot_q6_k_q8_k(int n, float * restrict s, const void * restrict vx, const void * restrict vy);
|
|
|
|
// Quantization with histogram collection
|
|
size_t ggml_quantize_q2_k(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
size_t ggml_quantize_q3_k(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
size_t ggml_quantize_q4_k(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
size_t ggml_quantize_q5_k(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
size_t ggml_quantize_q6_k(const float * src, void * dst, int n, int k, int64_t * hist);
|
|
|