github:zechenzhangAGI/AI-research-SKILLs · first seen 2026-10-31 · source · badge
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
Listed, not yet scored. This skill has no public model test yet. It is in the scoring queue; submitted skills go first, then the most-installed.