TurboQuant
TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration
About
Implementation of TurboQuant KV cache compression (ICLR 2026, arXiv:2504.19874) with vLLM integration. Tested on dense and MoE architectures across RTX 3090 and RTX 5090 GPUs.
Open Source Health
- Stars
- 1,767
- Forks
- 196
- License
- GPL-3.0
- Last commit
- 1 months ago
Related Categories
Vendor
0xSero
Orchestrator
Quick Links
Open Source
More by 0xSero
Related Products
CHaiDNN
HLS based Deep Neural Network Accelerator Library for Xilinx Ultrascale+ MPSoCs
Top category match
SoulX-FlashTalk
SoulX-FlashTalk is the first 14B model to achieve sub-second start-up latency (0.87s) while maintaining a real-time throughput of 32 FPS on an 8xH800 node.
Top category match
Ts Pattern
The exhaustive Pattern Matching library for TypeScript, with smart type inference.
Top category match
Nobodywho
NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.
Top category match
ds4
DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm
Top category match