Quest
[ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
Model DeploymentMLOps & Experiment TrackingOpen Source
About
Quest is an efficient long-context LLM inference framework that leverages query-aware sparsity in KV cache to reduce memory movement during attention and thus boost throughput.
Open Source Health
- Stars
- 406
- Forks
- 49
- License
- MIT
- Last commit
- 1 years ago
Cuda
Alternatives to Quest
Once For Allby MIT HAN Lab[ICLR 2020] Once for All: Train One Network and Specialize it for Efficient Deployment
CHaiDNNby XilinxHLS based Deep Neural Network Accelerator Library for Xilinx Ultrascale+ MPSoCs- SoulX-FlashTalkby Soul AI LabSoulX-FlashTalk is the first 14B model to achieve sub-second start-up latency (0.87s) while maintaining a real-time throughput of 32 FPS on an 8xH800 node.
Ts Patternby Gabriel VergnaudThe exhaustive Pattern Matching library for TypeScript, with smart type inference.
Nobodywhoby NobodyWhoNobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.
Resources & Links
Related Categories
Using Quest?
Track its cost next to the rest of your stack and get a reminder before it renews.
Add to my stackVendor
MIT HAN Lab
Efficient AI Computing. PI: Song Han
