wikipedia-paragraphs is a dataset generated from Wikipedia, designed for natural language processing (NLP) research.
The dataset is available on Hugging Face Hub.
This repository contains the code used to create the dataset. For details about the dataset itself, including its specifications and usage examples, please refer to the Hugging Face Hub page.
The following steps outline the procedure for creating the **wiki-20260607-v1.2.1 configuration of the dataset, which is available on Hugging Face Hub.
In this context, ** represents the language codes for the 11 target languages: ar, de, en, es, fr, it, ja, ko, pt, ru, and zh.
The process was run on a Linux machine with 128GB of RAM.
For the 11 target languages, download the following dump files and save them to the data/ directory:
- Wikipedia Cirrussearch dumps:
- Download the
**wiki_content-20260607-*****.json.bz2and**wiki_general-20260607-*****.json.bz2files.
- Download the
- Wikimedia Enterprise API (sign-up required):
- Use the Snapshot API to download the
**wiki_namespace_0and**wiki_namespace_14snapshots (saved as**wiki_namespace_0.tar.gzand**wiki_namespace_14.tar.gz, respectively).
- Use the Snapshot API to download the
- Wikimedia Enterprise HTML dumps:
- Download the
**wiki-NS10-20250320-ENTERPRISE-HTML.json.tar.gzfile.
- Download the
uv (v0.10.8) is used.
uv sync --lockedImportant
This step queries the MediaWiki Action API to retrieve category aliases for each language's Wikipedia. Please be sure to consult the API's web page for its terms of use.
Note
Please refer to this page for setting a valid user agent.
for WIKILANG in ar de en es fr it ja ko pt ru zh; do \
uv run create_page_dataset.py \
--cirrussearch_files \
data/${WIKILANG}wiki_content-20260607-*.json.bz2 \
data/${WIKILANG}wiki_general-20260607-*.json.bz2 \
--output_file work/${WIKILANG}wiki-20260607-pages.jsonl.gz \
--lang ${WIKILANG} \
--user_agent <valid user agent> \
--num_processes 16; \
doneImportant
This step queries the Wikimedia REST API to fetch article HTMLs missing from the dump files. Please be sure to consult the API's web page for its terms of use and check the latest information on rate limits.
Note
Please refer to this page for setting a valid user agent.
for WIKILANG in ar de en es fr it ja ko pt ru zh; do \
uv run create_page_html_dataset.py \
--page_dataset_file work/${WIKILANG}wiki-20260607-pages.jsonl.gz \
--output_file work/${WIKILANG}wiki-20260607-page-htmls.jsonl.gz \
--enterprise_html_files \
data/${WIKILANG}wiki_namespace_0.tar.gz \
data/${WIKILANG}wiki-NS10-20250320-ENTERPRISE-HTML.json.tar.gz \
data/${WIKILANG}wiki_namespace_14.tar.gz \
--get_page_html \
--transform_wikitext_to_html \
--user_agent <valid user agent> \
--get_request_batch_size 3 \
--transform_request_batch_size 3; \
donefor WIKILANG in ar de en es fr it ja ko pt ru zh; do \
uv run create_paragraph_dataset.py \
--page_dataset_files work/*wiki-20260607-pages.jsonl.gz \
--page_html_dataset_files work/${WIKILANG}wiki-20260607-page-htmls.jsonl.gz \
--output_dir work/${WIKILANG}wiki-20260607-paragraphs \
--paragraph_tags p li dt dd blockquote \
--article_redirects_file work/article-redirects-20260607.pkl \
--num_processes 4; \
donefor WIKILANG in ar de en es fr it ja ko pt ru zh; do \
uv run merge_page_paragraph_datasets.py \
--page_dataset_file work/${WIKILANG}wiki-20260607-pages.jsonl.gz \
--paragraph_dataset_dir work/${WIKILANG}wiki-20260607-paragraphs \
--output_dir outputs/${WIKILANG}wiki-20260607 \
--cache_dir cache \
--num_processes 4 \
&& rm -r cache; \
doneNote
For primary languages like en, the cache/ directory can grow to hundreds of gigabytes in size.
for WIKILANG in ar de en es fr it ja ko pt ru zh; do \
uv run push_to_hub.py \
--dataset_dir outputs/${WIKILANG}wiki-20260607 \
--repo_id <your dataset repo id (e.g., singletongue/wikipedia-paragraphs)> \
--config_name ${WIKILANG}wiki-20260607-v1.2.1; \
done- Code: Apache 2.0
- Datasets: CC-BY-SA 4.0 and GFDL
