Minbpe
byKarpathy
Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.
LLM OrchestrationOpen Source AI & Machine LearningOpen Source
About
Minimal, clean code for the (byte-level) Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. The BPE algorithm is "byte-level" because it runs on UTF-8 encoded strings.
Open Source Health
- Stars
- 10,716
- Forks
- 1,097
- License
- MIT
- Last commit
- 2 years ago
Python
Alternatives to Minbpe
The Algorithmby TwitterSource code for the X Recommendation Algorithm
Recommendersby TensorFlowTensorFlow Recommenders is a library for building recommender system models using TensorFlow.- RPG_KDD2025by FacebookresearchThis repository provides the code for implementing RPG described in our KDD'25 paper "Generating Long Semantic IDs in Parallel for Recommendation".
- Pairecby AlibabaA Go web framework for quickly building recommendation online services based on JSON configuration.
Textrankby Summa NLPTextRank implementation for Python 3.
Resources & Links
Related Categories
Using Minbpe?
Track its cost next to the rest of your stack and get a reminder before it renews.
Add to my stackVendor
Karpathy
Publisher of Nn Zero To Hero, Char Rnn and Llm.C