master
remove-vzip
readme
ci_cublas
fix-eval-bos
q4_3-range-fix
ik/rmse_quantization
q4_0-q4_2-range-fix
gg/rmse_quantization
quant-attn
mmap-pages-stats
flash-attn
mmap
q4_1_more_accel
q4_1_more_accel_kahan
q4_1_more_accel_loopsplit
tcp_server
dev
ci_cublas-31ff9e2
ci_cublas-44286d3
ci_cublas-45d94c8
master-018f227
master-01a297b
master-02c5b27
master-02d6988
master-03f7e33
master-04aaae1
master-04c6f5e
master-074bea2
master-084e2f0
master-09aecbf
master-0ad9646
master-0b2da20
master-0b366e7
master-0b5a935
master-0ba76c1
master-0c44427
master-0c56923
master-0d054e2
master-0e018fe
master-0e07e6a
master-0e6cbff
master-0f07cac
master-0f1b21c
master-106faaf
master-10f19c1
master-11d9023
master-12b5900
master-13b0c68
master-1481a9c
master-1623a6e
master-180b693
master-1972616
master-1bfc153
master-1d08882
master-1f0414f
master-2005469
master-20e1e84
master-20fbf2a
master-214b6a3
master-22213a1
master-2456837
master-2485d7a
master-25d7abb
master-2663d2c
master-29b7baa
master-2a2e63c
master-2a98bc1
master-2bb992f
master-2bdc096
master-2d099e5
master-2d3481c
master-2e17dfd
master-2e664f1
master-2ec8342
master-2edbdb0
master-2f7bf7d
master-2f7c8e0
master-305ba6f
master-305eb5a
master-31572d9
master-315a95a
master-3173a62
master-334637e
master-33e35b8
master-34ab526
master-34c1072
master-34d9f22
master-3525899
master-368d0c8
master-36b4f7e
master-36d0753
master-36d19a6
master-38de86a
master-3bcc129
master-3cd8dde
master-3d59769
master-3e5aa8a
master-3e6e70d
master-4122dff
master-4274722
master-436e561
master-437e778
master-459e93c
master-461ba9e
master-4640eff
master-47f61aa
master-481044d
master-483bab2
master-4870e45
master-489537e
master-4953e90
master-4b8efff
master-502a400
master-50a8a2a
master-50cb666
master-50fae10
master-53c8434
master-53dbba7
master-54bb60e
master-55390bc
master-55bc5f0
master-563cdc3
master-56e659a
master-574406d
master-585d91a
master-58b367c
master-58e6c9f
master-5a5f8b1
master-5a8c4f6
master-5addcb1
master-5af8e32
master-5b70e7d
master-5c19c70
master-5d5817c
master-5ecff35
master-6232f2d
master-62cfc54
master-6667401
master-66aab46
master-67c7779
master-684da25
master-698f7b5
master-69b7402
master-69c9229
master-6a9661e
master-6b6dbc8
master-6bc4400
master-6c24870
master-6f1ee4b
master-6f79699
master-70269ca
master-70f01cb
master-7296c96
master-76a8849
master-77a7340
master-77efdf5
master-799fdc1
master-7a32fcb
master-7a87d31
master-7a9b6c3
master-7b8dbcb
master-7e312f1
master-7f4c5c6
master-7fc50c0
master-7ff0dcd
master-81040f1
master-83df563
master-8520fc3
master-857308d
master-859fee6
master-863f65e
master-8687c1f
master-872c365
master-87a6f84
master-884e7d7
master-8944a13
master-8a0f867
master-8a1756a
master-8b67998
master-8c2ec5e
master-8c3ffc2
master-8c9be35
master-8cda5c9
master-8cf9f34
master-8d4a855
master-90b19bd
master-9190e8e
master-928480e
master-92a6e13
master-93265e9
master-939ad2d
master-9411288
master-94c5652
master-957c8ae
master-95ea26f
master-96f9c05
master-9794052
master-986b6ce
master-99c5b27
master-9b0a4d4
master-9cbc404
master-9daff41
master-9e17072
master-9ff334f
master-a140219
master-a316a42
master-a3a2a0e
master-a4755cf
master-a5c42c4
master-a5d30b1
master-a6bdc47
master-a791a68
master-aa485ce
master-aaf3b23
master-ad072fc
master-ad5fd5b
master-ae44e23
master-afd220d
master-b1ee8f5
master-b391579
master-b3f460e
master-b51c717
master-b6e7f9b
master-b925f1f
master-be87b6e
master-bf4b22f
master-c0bb1d3
master-c12b14b
master-c1f8850
master-c2b25b6
master-c3ac702
master-c3ca7a5
master-c494ed5
master-c4f89d8
master-c4fe84f
master-c50b628
master-c56b715
master-c5aa5e5
master-c5d70f5
master-c85e03d
master-c8c2c52
master-c9a59b7
master-c9e2c26
master-cc0bb72
master-cc9cee8
master-cd7fa95
master-cea1c85
master-d0aaff5
master-d3f202d
master-d40fded
master-d502bc7
master-d5850c5
master-d7def1a
master-d990e3f
master-d9a239c
master-da5303c
master-db10808
master-dcdd65e
master-dd0eabc
master-dd7eff5
master-e0305ea
master-e216aa0
master-e2cd506
master-e4412b4
master-e4422e2
master-e4cf982
master-e6c9e09
master-e7f6997
master-e899bf5
master-e8c0516
master-e95b655
master-e986f94
master-ea10d3d
master-ea3a0ad
master-eb17a02
master-ec728e4
master-ec9cdb6
master-ecbe466
master-ed3c680
master-ee0c40d
master-eeaa7b0
master-efd0564
master-f0d70f1
master-f121705
master-f202ada
master-f266259
master-f2d1c47
master-f3d4edf
master-f4cef87
master-f4d277a
master-f5a77a6
master-f647ce0
master-f7d0509
master-f7dc43b
master-fbd4d38
${ noResults }
4 Commits (9190e8eac8bdc108c40d2d7505e9b45fa773251f)
Author | SHA1 | Message | Date |
---|---|---|---|
|
cbef542879 |
py : cleanup the code
- use f-strings where possible - drop first param of encode/decode functions since "utf-8" is the default |
1 year ago |
|
78ca9838ee |
Make loading weights 10-100x faster
This is a breaking change that's going to give you three benefits: 1. Your inference commands should load 100x faster 2. You may be able to safely load models 2x larger 3. You can run many concurrent inference processes This was accomplished by changing the file format so we can mmap() weights directly into memory without having to read() or copy them thereby ensuring the kernel can make its file cache pages directly accessible to our inference processes; and secondly, that the file cache pages are much less likely to get evicted (which would force loads to hit disk) because they're no longer competing with memory pages that were needlessly created by gigabytes of standard i/o. The new file format supports single-file models like LLaMA 7b, and it also supports multi-file models like LLaMA 13B. Our Python tool now merges the foo.1, foo.2, etc. files back into a single file so that the C++ code which maps it doesn't need to reshape data every time. That's made llama.cpp so much simpler. Much of its load code has now been deleted. Furthermore, this change ensures that tensors are aligned properly on a 32-byte boundary. That opens the door to seeing if we can get additional performance gains on some microprocessors, by using ops that require memory alignment. Lastly note that both POSIX and the Windows platform are supported Fixes #91 |
1 year ago |
|
20a1a4e09c
|
Fix GPTQ converter (#423)
* Fix GPTQ converter * Fix comment --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
1 year ago |
|
16ffc013c6
|
Importer for GPTQ quantized LLaMA models (#301)
* [WIP, broken] Importer for GPTQ quantized LLaMA models Based on: https://github.com/qwopqwop200/GPTQ-for-LLaMa Current status: Something is busted. The output starts out decent, but quickly degrades into gibberish. This doesn't happen with either the original GPTQ-for-LLaMa using the same weights, or llama.cpp when using weights quantized by its own quantizer. Is there a bug in the conversion script that somehow only comes into play with a large context size? I did notice one potential issue. It's clearly not the main cause of the gibberish, since it doesn't happen when using q4_1 weights quantized by llama.cpp itself, but it seems concerning. When doing a matrix multiplication of f16 * f32 => f32 or q4_1 * f32 => f32, at least when the multiplication is not done with BLAS, the intermediate results are stored in the smaller format rather than f32. This seems like an unnecessary waste of precision, especially in the q4_1 case. I was originally hoping to validate the results by matching the Python implementation's output exactly, but precision and non-associativity issues make this very difficult, including when performing matrix multiplications and, especially, computing norms. Anyway, design details: The models being imported store per-layer weights in essentially q4_1 format, although the addend and scale are shared across an entire row rather than every group of 32 weights. This script duplicates the addend and scale to match ggml's expectations, at the cost of wasting some memory. However, there are two differences which I accommodated changing the output format (and adding corresponding support to main.cpp) rather than having the script match the existing one: - The tok_embeddings and output weights (i.e. the weights that aren't per-layer) are f16 instead of q4_1. They could be converted to q4_1, and the impact of the loss of precision would probably be low, but this would rule out exactly matching the Python implementation's output for validation. - There is no sharding, since the input doesn't have it, and for a CPU-only implementation it seems more useful to avoid having to deal with multiple files. The new format is differentiated from existing q4_1 format by changing the 'f16' header flag to a new value, 4. That said, I think a cleaner approach would be to change main.cpp to support loading each tensor with an arbitrary sharding configuration and type rather than hardcoding specific combinations of types. So far I've wasted too much time debugging to try implementing this... * Add missing permutation. Now it works. --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
1 year ago |