arxiv-ir-simulated-search-queries

An arXiv retrieval dataset with more than 2.8 million simulated specialist search queries and paper-level positive targets.

This dataset contains 2,875,637 query-document pairs derived from arXiv title-and-abstract records. Each row is designed so that the associated arXiv paper record is a positive retrieval target for the generated query. The queries were generated with a Qwen3.5-35B-A3B model quantized to NVFP4, and were intentionally constructed to resemble realistic specialist search behavior rather than a single idealized query form.

The dataset is intended for information retrieval research and engineering over scientific text. It mixes more direct paper-finding queries with broader, more indirect discovery-oriented queries. It also mixes fluent semantic search formulations with compressed keyword-style queries. The goal is to approximate the range of queries that a technical search system may encounter in practice, rather than producing only polished benchmark-style questions.

The source corpus is the Open arXiv dataset, where each record contains a paper identifier, a title, and an abstract. For this release, the retrieval text is stored as text, which is the original generation source text built from the paper title followed by a newline and the abstract. The original arXiv identifier is preserved as source_id so each row can be traced back to the underlying paper record.

Loading the dataset

from datasets import load_dataset

ds = load_dataset(
    "hotchpotch/arxiv-ir-simulated-search-queries",
    split="train",
    streaming=True,
)
row = next(iter(ds))
print(row)

License

This dataset is derived from Open arXiv, using a snapshot of that upstream dataset from early April 2026. The upstream dataset card declares the source data license as:

• CC0 1.0 Universal (cc0-1.0)

This README follows that upstream declaration. If you need to validate redistribution or downstream commercial use for a specific workflow, check the upstream dataset card as well.

---

Technical report

Dataset structure

The dataset uses the following top-level columns, in this exact order.

query is the simulated user search input and text is the positive retrieval target text. source_id is retained so downstream users can join back to external metadata or recover the original paper record when needed.

Example row:

Control semantics

directness

query_style

Empirical distribution

The following counts describe the released dataset.

directness

query_style

Construction methodology

This section is written so that a senior data scientist or engineer can reproduce a close variant of the dataset design. The resulting generations will not be bitwise identical without the original prompts, seeds, and infrastructure, but the methodological recipe is intended to be clear.

Step 1. Define the source retrieval corpus

Start from an arXiv paper metadata corpus containing a stable paper identifier, a title, and an abstract for each record. For the release used here, the generation run considered 2,989,022 source rows.

The key design decision is to treat the title-plus-abstract text as the retrieval target. The generation problem is therefore asymmetric: given a paper record, generate a specialist search query for which that record would be a plausible positive hit. This differs from ordinary paraphrase generation because the output query should reflect user search behavior rather than sentence-level rewriting.

Step 2. Define the generation objective

For each paper record, ask a Qwen3.5-35B-A3B model quantized to NVFP4 to generate one search query in English. The target behavior is a query that a researcher or technically literate user might realistically type when trying to find that paper or closely related work. The generator should not be restricted to one idealized style. Instead, it should cover both more direct known-item-like queries and more indirect discovery-oriented queries, together with both semantic and keyword-heavy query forms.

The underlying design assumption is that retrieval supervision becomes more useful when it includes both strong and imperfect signals. A purely clean synthetic dataset often makes retrieval easier than it is in production scientific search. This dataset therefore treats variation in directness and surface form as a first-class design goal.

Step 3. Sample categorical controls before generation

Before generating the query for a paper record, sample two categorical control variables. These variables govern how directly the generated query should target the source paper and whether the surface form should be semantic or keyword-like.

The controls used in the release are described earlier in this README:

• directness
• query_style

The control distribution is deliberately non-uniform. In this release, both controls were sampled with a 3:1 bias toward the more standard setting.

Step 4. Generate one candidate query per retained source row

Condition the language model on the paper title and abstract together with the sampled controls, then generate a single search query. The prompt should instruct the model to behave like a realistic specialist search user rather than a benchmark annotator. In practice, this means allowing both direct paper-finding queries and broader related-work queries, while also allowing compressed keyword forms when requested.

The paper record should remain the target of retrieval supervision. That is, even when the query is indirect or keyword-heavy, the associated title-and-abstract text should still be a reasonable positive retrieval example. This requirement keeps the dataset useful for IR training while broadening the query distribution beyond exact-title or exact-topic lookups.

The generation pipeline used for this release also applies a validity check after each candidate query is produced. The model is asked whether the generated query is valid for finding the paper or strongly related papers from the title and abstract, with explicit instructions to reject paper-metadata questions and generic textbook questions.

Step 5. Remove failed or low-quality generations

Not every raw generation should be exported. Rows with explicit generation errors, invalid outputs, or unsuccessful outcomes are discarded. Rows below the abstract-length threshold are also excluded. For the released dataset, the final export keeps rows that passed the generation checks and validity filter.

In practice, the generation run started from 2,989,022 source rows and the final export retained 2,875,637 rows. This corresponds to a retention rate of 96.2% relative to the input rows considered by the run.

Step 6. Export the final retrieval dataset

Export the retained rows as a flat retrieval dataset. Each row contains one generated query, one positive paper text, one raw-data reference key, and two top-level generation-control fields. Keeping the control fields flat is useful for downstream filtering, grouped evaluation, stratified sampling, and ablation work.

Author

• Yuichi Tateno (@hotchpotch)