← All posts

SPECTRA: synthetic IR collections to stress-test search before human labels

arXiv preprint: reproducible corpus generator up to 60k docs — near-linear speed, controlled distractors, BM25 nDCG@10 drops from 1.00 to 0.43 as noise grows.

Tags
Performanceresearch
SPECTRA: synthetic IR collections to stress-test search before human labels
Contents

In brief

Scaling information retrieval tests needs large corpora and expensive human relevance judgments. SPECTRA generates synthetic text collections with deterministic relevance oracles — a diagnostic complement to TREC-style evaluation, not a replacement for human assessment.

What they studied

The framework separates latent topics, surface text, metadata, query intent, and graded labels. A Python prototype built up to 60,000 documents (~9.6M tokens) and 96 queries.

Generation is near-linear at ~12–14k documents/sec. As cross-topic distractor text increases, BM25 nDCG@10 falls from 1.00 (2% distractors) to 0.43 (36%) — a knob for ranking stress tests.

Key findings

  • Lightweight synthetic corpora can expose index build, query routing, and eval pipeline issues before human labeling spend.
  • Controlled long-tail vocabulary (Zipf slope ~0.86) helps test rare-term behavior.
  • Complements Cranfield/TREC; does not replace human quality judgment.

What this means for developers

  1. Before scaling Elasticsearch/OpenSearch, run a synthetic stress corpus with known oracles.
  2. Add CI cases with rising off-topic noise to catch ranking regressions early.
  3. Split infra tests (index speed) from quality tests (nDCG on labels).

Limitations

Synthetic text misses real user query nuance. Prototype is single-process; distributed generation is not production-ready in the paper.