marcelroed/gigatoken

From the README

Gigatoken ~1000x faster than HuggingFace's tokenizers, drop-in replacement. *Tokenize your text data at GB/s!* Note that both HF tokenizers and tiktoken are already running multithreaded Rust!

Setup prompt

Paste this into Cursor or Claude Code. It only includes install lines found in the README, and it asks the agent to review the code first.

Set up the GitHub repository marcelroed/gigatoken (Language model tokenization at GB/s).
Repository: https://github.com/marcelroed/gigatoken

Official install commands found in the README:
pip install gigatoken

Clone the repository and follow those steps. Do not invent package names or extra install commands.
Review the code before running anything.

Buzz by window

Same mentions, three half-lives. This is not a star history. Star gain appears after a second daily snapshot.

• Today0.00
• Week0.00
• Month0.01

Spotted on

• HN @syrusakbaryJul 22, 2026

Related

• deepseek-ai/DeepInfer

  DeepInfer: High-performance, large-scale inference framework for DeepSeek models

• ayghri/i-have-adhd

  A skill to stop your coding agent from burying the answer. ADHD-friendly output.

• antirez/h3.c

  MiniMax H3 inference engine for Mac computers

• angelos-p/llm-from-scratch

  Train Your Own LLM From Scratch A hands-on workshop where you write every piece of a GPT training pipeline yourself, understanding what each component does…