Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
About
This repo contains Typescript and C# implementation of byte pair encoding(BPE) tokenizer for OpenAI LLMs, it's based on open sourced rust implementation in the OpenAI tiktoken. Both implementation are valuable to run prompt tokenization in Nodejs and .NET environment before feeding prompt into a LLM.
Open Source Health
- Stars
- 212
- Forks
- 37
- License
- MIT
- Last commit
- 29 days ago
Related Categories
Vendor
Microsoft
Empower every person and organization to achieve more.
Quick Links
Open Source
More by Microsoft
Related Products
Recognizers-Text
Recognizers-Text offers entity recognition and resolution for numbers, dates, and units in multiple languages.
More from this vendor
Language Server Protocol
Defines a common protocol for language servers.
More from this vendor
Azure AI Speech
Azure Speech in Foundry Tools
More from this vendor
Lamar Benchmark
Source code for the ECCV 2022 paper "Benchmarking Localization and Mapping for Augmented Reality".
More from this vendor
Pyright
Static Type Checker for Python
More from this vendor