Governed AI Training Datasets Directory

FineWeb implements automated deduplication, heuristic quality scoring, PII redaction filters, and toxic content removal, satisfying Article 10 requirements for training data curation.

Deployers must apply post-ingestion copyright opt-out filtering under Article 53, synthetic PII scrubbing, and statistical bias mitigation before using raw Common Crawl partitions.

The Pile contains copyrighted academic and bibliographic sub-corpora requiring proof of text-and-data mining (TDM) opt-out compliance under the EU DSM Directive.

RedPajama provides full per-token quality annotations, deduplication lineage, and explicit source domain categorizations aligned with Article 10 technical documentation.

C4 applies aggressive English-language heuristic filters and basic PII scrubbing, though deployers in the EU must verify GDPR compliance for persistent entity references.

LAION-5B contains unfiltered web image-text pairs requiring comprehensive biometric privacy scrubbing and copyright verification under Article 10 and GDPR Article 9.

GSM8K consists exclusively of grade-school math problem formulations synthesized without personal identity references, ensuring zero PII and full Article 10 compliance.

HumanEval uses hand-written Python docstrings and unit tests with complete copyright clearance, fulfilling Article 10 data provenance benchmarks for coding models.

MMLU tests cross-disciplinary knowledge across 57 subjects, providing standardized accuracy benchmarks mandated for systemic risk model filings with the EU AI Office.

SQuAD 2.0 is licensed under CC BY-SA 4.0, requiring downstream share-alike compliance or commercial clean-room adaptations satisfying Article 53 documentation.

Under Article 10(5) and GDPR Article 9, clinical datasets must maintain strict de-identification, institutional review board (IRB) approvals, and encrypted audit trails.

Deployers must document known representation gaps under Article 10(2)(f) and apply balancing transformations before deploying vision systems in Annex III high-risk domains.

BookCorpus contains full-text fiction books scraped without author consent, presenting extreme copyright infringement liability and non-compliance with EU TDM opt-out rules.

BigCode provides an automated GitHub repository opt-out portal and strict permissive license filtering, serving as a model implementation for EU AI Act Article 53 compliance.

CodeContests provides automated unit test suites and complexity boundaries, ensuring high data integrity and reproducible evaluation metrics under Article 10.

Every contributor signs an explicit open-source agreement with GDPR consent waivers, fulfilling Article 10 data collection provenance requirements.

Synthetic datasets derived from proprietary LLMs carry contractual non-compete terms and require transparency labeling under EU AI Act Article 50.

Transcripts often contain accidental personal secrets and PII requiring automated entity redaction before being permissible for Article 10 compliant training.

Evol-Instruct algorithmically broadens instruction diversity across complexity dimensions, mitigating dataset homogeneity under Article 10 requirements.

Being entirely synthetically generated from web seed outlines, Cosmopedia has zero PII, zero copyright infringement liability, and verified data provenance.