SentencePiece
Unsupervised text tokenizer for Neural Network-based text generation.
About
SentencePiece is a fast, lightweight, and unsupervised text tokenizer and detokenizer designed for neural network-based text generation systems (such as Large Language Models) where the vocabulary size is fixed prior to training.
Open Source Health
- Stars
- 12,075
- Forks
- 1,379
- License
- Apache-2.0
- Last commit
- 26 days ago
Related Categories
Vendor
Organize the world's information and make it universally accessible.
Quick Links
Open Source
More by Google
Related Products
Textrank
TextRank implementation for Python 3.
Top category match
KoBERT
Korean BERT pre-trained cased (KoBERT)
Top category match
TAADpapers
Must-read Papers on Textual Adversarial Attack and Defense
Top category match
AutoPhrase
AutoPhrase: Automated Phrase Mining from Massive Text Corpora
Top category match
Recognizers-Text
Recognizers-Text offers entity recognition and resolution for numbers, dates, and units in multiple languages.
Top category match