github:zechenzhangAGI/AI-research-SKILLs · first seen 2026-10-31 · source · badge
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
Listed, not yet scored. This skill has no public model test yet. It is in the scoring queue; submitted skills go first, then the most-installed.