Commit Graph

425 Commits (q4_0-q4_2-range-fix)
 

Author SHA1 Message Date
Kawrakow 5ecff35151
Adding a simple program to measure speed of dot products (#1041)
On my Mac, the direct Q4_1 product is marginally slower
(~69 vs ~55 us for Q4_0). The SIMD-ified ggml version
is now almost 2X slower (~121 us).

On a Ryzen 7950X CPU, the direct product for Q4_1 quantization
is faster than the AVX2 implementation (~60 vs ~62 us).

---------

Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
1 year ago
Georgi Gerganov 7faa7460f0
readme : update hot topics about new LoRA functionality 1 year ago
Georgi Gerganov 5af8e32238
ci : do not run on drafts 1 year ago
Ivan Komarov 42747220b4
Do not close file after mmap (Windows version) (#1034) 1 year ago
Atsushi Tatsuma e9298af389
readme : add Ruby bindings (#1029) 1 year ago
Cameron 4ad73137a1
add 4_0 to default outfile namestr dict (#1031)
this came up when trying to convert the gpt4all-lora-unfiltered-quantized.bin file
1 year ago
slaren 315a95a4d3
Add LoRA support (#820) 1 year ago
Arik Poznanski efd05648c8
llama : well-defined static initialization of complex objects (#927)
* Replaced static initialization of complex objects with a initialization on first use. This prevents an undefined behavior on program run, for example, crash in Release build, works in Debug build

* replaced use of auto with exact type to avoid using -std=c++14

* Made the assessors functions for static maps be static const
1 year ago
Georgi Gerganov eb17a026fd
quantize-stats : fix bug in --type argument 1 year ago
Georgi Gerganov 69b740289f
ggml : avoid using ggml_fp16_to_fp32() and ggml_fp32_to_fp16() in ggml.c 1 year ago
Ivan Komarov f266259ad9
Speedup the AVX-512 implementation of ggml_vec_dot_q4_0() (#933) 1 year ago
slaren 47f61aaa5f
Fix: do not close file on mmap (#1017) 1 year ago
Georgi Gerganov 3173a62eb9
stdout : vertical align outputs for better readibility 1 year ago
Pavol Rusnak 489537e6cf
examples: add missing <ctime> include for time() (#1011) 1 year ago
nanahi 2d3481c721
Fix msys2 build error and warnings (#1009) 1 year ago
comex 74f5899df4
convert.py: Fix loading safetensors and ggml format on Windows (#991)
Calling `mmap.mmap` on Windows apparently resets the file offset of the
raw file object (and makes the BufferedReader return a *negative* file
offset).  For safetensors, avoid using the file offset after calling
mmap.  For GGML format, explicitly save and restore the offset.

Fixes #966.
1 year ago
Stephan Walter 2f7c8e014e
Fix potential int8 overflow in non-SIMD vec_dot (#986) 1 year ago
Stephan Walter 0ad964631f
Refactor ggml.c for future tensor types (#1001) 1 year ago
Georgi Gerganov e95b6554b4
ggml : add Q8_0 quantization for intermediate results (#951)
* ggml : add Q8_0 quantization for intermediate results

* quantize-stats : fix test + add it to Makefile default

* Q8: use int8_t, AVX/AVX2 optimizations

* ggml : fix quantize_row_q8_0() ARM_NEON rounding

* minor : updates after rebase to latest master

* quantize-stats : delete obsolete strings

* ggml : fix q4_1 dot func

---------

Co-authored-by: Stephan Walter <stephan@walter.name>
1 year ago
Georgi Gerganov aa485cee33
ggml : use posix_memalign on non-Windows env 1 year ago
Ivan Komarov c12b14b77f
benchmark : fix result validation in benchmark-q4_0-matmult (#987) 1 year ago
katsu560 106faaf297
cmake : add finding the OpenBLAS header file (#992) 1 year ago
Pavol Rusnak c85e03d12e
Revert "main : alternative instruct mode (Vicuna support, etc.) (#863)" (#982)
This reverts commit f4d277ae17.
1 year ago
Pavol Rusnak 489093548c
py : bump sentencepiece to 0.1.98 to support Python 3.11 (#976) 1 year ago
Stephan Walter 93265e988a
make : fix dependencies, use auto variables (#983) 1 year ago
Pavol Rusnak c56b715269
Expose type name from ggml (#970)
Avoid duplication of type names in utils

Co-authored-by: Håkon H. Hitland <haakon@likedan.net>
1 year ago
Tomáš Pazdiora f4d277ae17
main : alternative instruct mode (Vicuna support, etc.) (#863)
* Add support for configs, add configurable prefixes / suffixes, deprecate instruct mode, add stop prompt

* Add multiline mode, update text input.

* bugfix

* update implementation

* typos

* Change --multiline implementation to be toggled by EOF.

* bugfix

* default multiline mode

* add more configs

* update formating

* update formatting

* apply suggestions
1 year ago
Kerfuffle c9a59b70a5
ggml : add unary and binary map operations (#874)
* GGML map ops proof of concept.

* Various cleanups.

Add handling for task setting.

Add handling for ggml_compute_backward.

Rename functions to ggml_map_unary_f32 and ggml_map_binary_f32

Fix compiler warnings related to casting function pointers and `void *`

Reorder functions and definitions based on the GGML op number.

Use typedefs for map op function pointer types.

* Fix position of map ops cases in ggml_compute_forward
1 year ago
Pavol Rusnak a32f7acc9f
py : cleanup dependencies (#962)
after #545 we do not need torch, tqdm and requests in the dependencies
1 year ago
Pavol Rusnak 43ffdefb74
py : fix flake8 and isort nitpicks (#960) 1 year ago
Georgi Gerganov 1623a6e9b4
ggml : minor 1 year ago
Georgi Gerganov c14e0d2f23
ggml : always allocate buffers with size multiple of GGML_MEM_ALIGN 1 year ago
comex 723dac55fa
py : new conversion script (#545)
Current status: Working, except for the latest GPTQ-for-LLaMa format
  that includes `g_idx`.  This turns out to require changes to GGML, so
  for now it only works if you use the `--outtype` option to dequantize it
  back to f16 (which is pointless except for debugging).

  I also included some cleanup for the C++ code.

  This script is meant to replace all the existing conversion scripts
  (including the ones that convert from older GGML formats), while also
  adding support for some new formats.  Specifically, I've tested with:

  - [x] `LLaMA` (original)
  - [x] `llama-65b-4bit`
  - [x] `alpaca-native`
  - [x] `alpaca-native-4bit`
  - [x] LLaMA converted to 'transformers' format using
        `convert_llama_weights_to_hf.py`
  - [x] `alpaca-native` quantized with `--true-sequential --act-order
        --groupsize 128` (dequantized only)
  - [x] same as above plus `--save_safetensors`
  - [x] GPT4All
  - [x] stock unversioned ggml
  - [x] ggmh

  There's enough overlap in the logic needed to handle these different
  cases that it seemed best to move to a single script.

  I haven't tried this with Alpaca-LoRA because I don't know where to find
  it.

  Useful features:

  - Uses multiple threads for a speedup in some cases (though the Python
    GIL limits the gain, and sometimes it's disk-bound anyway).

  - Combines split models into a single file (both the intra-tensor split
    of the original and the inter-tensor split of 'transformers' format
    files).  Single files are more convenient to work with and more
    friendly to future changes to use memory mapping on the C++ side.  To
    accomplish this without increasing memory requirements, it has some
    custom loading code which avoids loading whole input files into memory
    at once.

  - Because of the custom loading code, it no longer depends in PyTorch,
    which might make installing dependencies slightly easier or faster...
    although it still depends on NumPy and sentencepiece, so I don't know
    if there's any meaningful difference.  In any case, I also added a
    requirements.txt file to lock the dependency versions in case of any
    future breaking changes.

  - Type annotations checked with mypy.

  - Some attempts to be extra user-friendly:

      - The script tries to be forgiving with arguments, e.g. you can
        specify either the model file itself or the directory containing
        it.

      - The script doesn't depend on config.json / params.json, just in
        case the user downloaded files individually and doesn't have those
        handy.  But you still need tokenizer.model and, for Alpaca,
        added_tokens.json.

      - The script tries to give a helpful error message if
        added_tokens.json is missing.
1 year ago
Georgi Gerganov 0f07cacb05
ggml : fix q4_1 dot product types 1 year ago
Howard Su c5d70f5c9e
ggml : optimize rope function to avoid call powf in the tight loop (#807) 1 year ago
Gary Linscott be87b6ed20
perplexity : add support for batch size to `--perplexity` (#407)
* Add support to batch size for perplexity

* Revert "Fix memory allocation issues and seg faults"

This reverts commit 4870e455b3.

* update from merge

* Remove perplexity from main

* updates

* Update batch size for efficiency
1 year ago
CRD716 0e07e6a839
common : remove unnecessary includes (#947) 1 year ago
Georgi Gerganov a3a2a0eda8
ggml : add GGML_DEFAULT_N_THREADS 1 year ago
Georgi Gerganov d990e3fffc
ggml : speed-up ggml_vec_dot_q4_1() ARM_NEON + 32-bit ARM support (#900)
* ggml : speed-up q4_1 ARM_NEON by ~5%

* ggml : implement vaddvq when missing

* ggml : implement vminvq and vmaxvq when missing

* ggml : implement vzip when missing

* ggml : fix comment

* ggml : try to use correct ifdef
1 year ago
Georgi Gerganov 9190e8eac8
llama : merge llama_internal.h into llama.h
Hide it behind an #ifdef
1 year ago
Georgi Gerganov c85980acd0
gitignore : benchmark 1 year ago
Stephan Walter 6232f2d7fd
ggml : optimize non-SIMD Q4_0 vector dot product (#703) 1 year ago
Pavol Rusnak 6c248707f5
ggml : introduce GGML_ALIGNED_MALLOC/GGML_ALIGNED_FREE macros (#884)
which allows us to use aligned_alloc or _aligned_malloc functions
1 year ago
CRD716 8cda5c981d
fix whitespace (#944) 1 year ago
CRD716 ec29272175
readme : remove python 3.10 warning (#929) 1 year ago
Genkagaku.GPT 7e941b95eb
readme : llama node binding (#911)
* chore: add nodejs binding

* chore: add nodejs binding
1 year ago
Pavol Rusnak c729ff730a
flake.nix: add all binaries from bin (#848) 1 year ago
Judd 4579af95e8
zig : update build.zig (#872)
* update

* update readme

* minimize the changes.

---------

Co-authored-by: zjli2019 <zhengji.li@ingchips.com>
1 year ago
Vladimir 8c3ffc2f04
ggml : update cblas_sgemm columns var to be more reasonable (#838) 1 year ago
niansa/tuxifan 107980d970
examples : add -n to alpaca and gpt4all scripts (#706) 1 year ago