alessandroberna · 3 days ago

Seems like it wouldn't work for different branches.

It probably needs to store the full F16 version for each model variant, even if the variants share the same base model

adrian17 · 3 days ago

The catch is that it's not really doing anything. It "stores" 16GB of quants by... storing the base 2GB model and quantizing it on user's machine on demand.

The extra diff files are only there because the quantization is not fully reproducible*, and author's 3-line llama.cpp fix PR** supposedly fixes this. With it fixed, then in my understanding the author's tool is literally just a wrapper around llama-quantize.

* I also wonder if this even matters for quant quality. In fact, isn't it possible for contracted FMAs to be more numerically accurate (even if less correct from IEEE pov) than non-contracted math?

** I'd give the PR like 50% chances of not surviving the day, since AFAIK on llama.cpp, vibed PR descriptions usually end up rejected as a rule. Maybe the diff being super trivial could save it from this fate, dunno.

alessandroberna · 3 days ago

It's just a wrapper around llama-quantize, i don't see why it wouldn't work.

Yet it's only useful if one wants to have both the F16 original model plus all the other GGUF quants at a time and doesn't mind spending compute every time they need a specific quant again.

It doesn't do much for a consumer who only needs one or a few specific quants that work on their hardware and I'd wager it's not something that multi billion dollar companies need. At least as far as I'm aware major llm providers aren't using gguf and llama.cpp to serve their inference