Skip to content

quantizing-models-bitsandbytes

from Orchestra-Research/AI-research-SKILLs

Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 fo

v1.0.0MIT
412
Lines
1,169
Words
23
Code Blocks

Languages

bashpython
10-optimization/bitsandbytes/SKILL.md