wikipedia-english-ir-simulated-search-queries

An English Wikipedia retrieval dataset with more than 29 million simulated search queries and paragraph-level positive targets.

This dataset contains 29,366,101 English query-document pairs derived from Wikipedia. Each row is designed so that the associated Wikipedia paragraph is a positive retrieval target for the generated query. The queries were generated with a Qwen3.5-35B-A3B model quantized to NVFP4, and were intentionally constructed to resemble realistic user search behavior rather than a single idealized query form.

The dataset is intended for information retrieval research and engineering. It mixes clean, direct information-seeking queries with noisier variants such as typo-containing queries, keyword-style queries, rough search-log-like formulations, shorthand, abbreviations, and broader or only partially specified queries. The goal is to approximate the range of queries that a retrieval system may encounter in practice, rather than producing only polished benchmark-style questions. By simulating both clean and search-log-like query behavior, the dataset provides supervision for learning retrieval over a broader range of query styles while still pairing each query with a positive target document.

The data is built from an English Wikipedia retrieval corpus derived from Wikipedia paragraph data. The original source family is Wikipedia/Wikimedia content; the multilingual paragraph resource used by related Hotchpotch Wikipedia datasets is singletongue/wikipedia-paragraphs. Operationally, the generation run consumed 29,568,452 source rows, where each row contained a title-like anchor and a paragraph-level text field. For this release, the paragraph becomes the positive retrieval text as document, while the source-side title or title-section anchor is preserved as title_section.

Loading the dataset

from datasets import load_dataset

ds = load_dataset(
    "hotchpotch/wikipedia-english-ir-simulated-search-queries",
    split="train",
    streaming=True,
)
row = next(iter(ds))
print(row)

Note that categorical columns are stored as Hugging Face ClassLabel, so rows may be returned with integer IDs in some loading modes.

License

This dataset is derived from Wikipedia text and follows Wikipedia licensing:

• Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
• GNU Free Documentation License (GFDL), where applicable

The generated query fields are distributed together with Wikipedia-derived positive documents; downstream users should preserve attribution/share-alike obligations for the Wikipedia-derived text.

---

Technical report

Dataset structure

The dataset uses the following top-level columns, in this exact order.

query is the user-like search input, document is the positive target text, and title_section is a preserved source-side anchor from the source corpus. In practice, title_section corresponds to the Wikipedia page title together with the relevant section name when applicable. For lead passages, it is typically just the page title.

Example row:

Control semantics

clean_or_noisy

mode

The clean-oriented modes are direct, partial, and tangential. The noisy-oriented modes are rough, typo, keyword, and vague.

focus_segment

asker_level

answer_type

Empirical distribution

The following counts describe the released dataset after filtering.

clean_or_noisy

mode

focus_segment

asker_level

answer_type

Constraint between clean_or_noisy and mode

mode is not independent of clean_or_noisy in this release. The allowed combinations are intentionally partitioned as follows.

Observed pair counts:

Construction methodology

This section is written so that a senior data scientist or engineer can reproduce a close variant of the dataset design. The resulting generations will not be bitwise identical without the original prompts, seeds, and infrastructure, but the methodological recipe is intended to be clear.

Step 1. Define the source retrieval corpus

Start from an English Wikipedia retrieval corpus. For the release used here, the source corpus contained 29,568,452 rows. Each row provides a title-like anchor together with a paragraph-level document field.

The key design decision is to treat the paragraph, not the title, as the retrieval target. The generation problem is therefore asymmetric: given a paragraph, generate a search query for which that paragraph would be a plausible positive hit. This differs from ordinary paraphrase generation because the output query should reflect user search behavior rather than sentence-level rewriting.

Step 2. Define the generation objective

For each source paragraph, ask a Qwen3.5-35B-A3B model quantized to NVFP4 to generate one English search query. The target behavior is a query that a user might realistically type into a search engine when looking for information that the paragraph can satisfy, support, or hint at. The generator should not be restricted to one idealized style. Instead, it should cover a controlled mixture of direct fact-seeking queries, partially specified queries, topic-adjacent queries, typo-containing queries, rough short queries, and keyword-heavy queries.

The underlying design assumption is that large-scale retrieval supervision becomes more useful when it includes both strong and imperfect signals. A purely clean synthetic dataset often makes retrieval look easier than it is in production. This dataset therefore treats stylistic and noise diversity as a first-class design goal rather than a side effect.

Step 3. Sample categorical controls before generation

Before generating the query for a paragraph, sample five categorical control variables. These variables govern the intended style of the generated query, the amount of surface noise, where the supporting evidence should come from inside the paragraph, the expected sophistication of the wording, and how directly the paragraph should answer the query.

The controls used in the release are described earlier in this README:

• clean_or_noisy
• mode
• focus_segment
• asker_level
• answer_type

The control distribution is deliberately non-uniform. In this release, clean_or_noisy is sampled with a target clean ratio of 0.75, and mode is then sampled conditionally on that choice. The remaining three controls are sampled independently of one another.

Step 4. Generate one candidate query per retained source row

Condition the language model on the paragraph and the sampled controls, then generate a single English search query. At minimum, the model input should contain the source paragraph and the selected control values for mode, focus_segment, asker_level, and answer_type, together with instructions to write the output as a search query in English. The prompt should instruct the model to behave like a realistic search user rather than a benchmark annotator. In practice, this means encouraging terse phrasings when appropriate, tolerating spelling noise in designated cases, and allowing some queries to be broader or only partially anchored to the paragraph.

The paragraph should remain the target of retrieval supervision. That is, even when the query is tangential, noisy, or only indirectly answerable, the associated paragraph should still be a reasonable positive retrieval example. This requirement keeps the dataset useful for IR training while still broadening the query distribution beyond exact-answer lookups.

The generation pipeline used for this release also applies a lightweight validity check after each candidate query is produced. The model is asked whether the generated query is valid for the input paragraph using a three-way decision: valid, unknown, or invalid. Candidates that fail this check can be retried a small number of times before being discarded.

Step 5. Remove failed or low-quality generations

Not every raw generation should be exported. Rows with explicit generation errors, empty generations, or other unsuccessful outcomes are discarded. Among successful generations, apply an additional conservative hard filter to exclude outputs that are obviously poor retrieval queries, such as extreme length outliers, heavily digit-dominated strings, or strings with insufficient alphabetic content.

In practice, the generation pipeline started from 29,568,452 source rows and retained 29,366,101 rows after filtering. This corresponds to a retention rate of 99.3%. The final release contains 29,366,101 query-document pairs.

Step 6. Export the final retrieval dataset

Export the retained rows as a flat retrieval dataset. Each row contains one generated query, one positive paragraph, one inherited title-like field, and five categorical controls stored as top-level fields. Flattening the controls is useful for downstream filtering, grouped evaluation, stratified sampling, and curriculum-style training because the supervision attributes are directly accessible without nested metadata.

Acknowledgements

Thank you for publishing the very useful dataset singletongue/wikipedia-paragraphs. This dataset made it possible to build this IR-focused dataset quickly.

Author

• Yuichi Tateno (@hotchpotch)