nemo-curator

github:zechenzhangAGI/AI-research-SKILLs · first committed 2025-11-06 · source · badge

GPU-accelerated data curation for LLM training. Supports text/image/video/audio. Features fuzzy deduplication (16× faster), quality filtering (30+ heuristics), semantic deduplication, PII redaction, NSFW detection. Scales across GPUs with RAPIDS. Use for preparing high-quality training datasets, cleaning web data, or deduplicating large corpora.

Skill Health: not yet scored

Listed, not yet scored. This skill has no public model test yet. It is in the scoring queue; submitted skills go first, then the most-installed.

Version history

8 commits to this file on GitHub (newest first).

2025-11-20 · commit 0a9ded4 · Standardize skill metadata: Add category tags and update authors
2025-11-20 · commit 72c8b2c · Trigger upload of remaining 26 skills
2025-11-20 · commit 85a6d14 · Trigger upload of remaining 12 skills with large file fix
2025-11-20 · commit f38509b · Sync first 20 skills to Orchestra marketplace
2025-11-19 · commit bd4461f · Standardize YAML frontmatter for all 54 skills
2025-11-09 · commit 096b8f0 · feat: Add 8 comprehensive AI research skills (inference, optimization, tokenization, data)
2025-11-06 · commit f0eb7ea · refactor: Delete 9 low-quality skills, establish quality standards
2025-11-06 · commit 4a393a6 · feat: Add 16 comprehensive AI research skills with project infrastructure

Archived snapshots