QueryGym Leaderboard

Query Reformulation Methods Performance Evaluation

The leaderboard below provides comprehensive performance metrics for QueryGym query reformulation methods across different LLM configurations and retrieval backends. Each row represents a unique method configuration. Click on any metric column header (⇅) to sort methods by that metric. Click on any row to expand and view detailed commands for reproducing the results.

Performance Metrics

Configuration	Method	TREC DL 2019				TREC DL 2020				DL HARD
		AP	nDCG@10	R@1K		AP	nDCG@10	R@1K		AP	nDCG@10	R@1K
GPT-4.1-mini temp=1.0, max_tokens=128 BM25 k1=0.9, b=0.4	Query2Doc (ZS)	0.4508	0.6709	0.8746		0.4325	0.6323	0.8864		0.2276	0.3303	0.7704
	Query2Doc (FS)	0.4418	0.6532	0.8521		0.4051	0.6111	0.8869		0.2260	0.3388	0.7842
	Query2Doc (CoT)	0.4145	0.6128	0.8495		0.3801	0.5894	0.8846		0.2225	0.3191	0.7524
	Command to reproduce results on TREC DL 2019: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --output-dir outputs/dl19_query2doc_zs` Command to reproduce results on TREC DL 2020: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --output-dir outputs/dl20_query2doc_zs` Command to reproduce results on DL HARD: `python examples/querygym_pyserini/pipeline.py \ --queries-file path/to/queries.tsv \ --qrels-file path/to/qrels.trec \ --index-name msmarco-v1-passage \ --method query2doc \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --output-dir outputs/dlhard_query2doc_zs`
	Command to reproduce results on TREC DL 2019: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"dataset_type":"msmarco",' \ '"collection_path":"path/to/collection.tsv",' \ '"train_queries_path":"path/to/queries.train.tsv","train_qrels_path":"path/to/qrels.train.tsv"}' \ --output-dir outputs/dl19_query2doc` Command to reproduce results on TREC DL 2020: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"dataset_type":"msmarco",' \ '"collection_path":"path/to/collection.tsv",' \ '"train_queries_path":"path/to/queries.train.tsv","train_qrels_path":"path/to/qrels.train.tsv"}' \ --output-dir outputs/dl20_query2doc` Command to reproduce results on DL HARD: `python examples/querygym_pyserini/pipeline.py \ --queries-file path/to/queries.tsv \ --qrels-file path/to/qrels.trec \ --index-name msmarco-v1-passage \ --method query2doc \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"dataset_type":"msmarco",' \ '"collection_path":"path/to/collection.tsv",' \ '"train_queries_path":"path/to/queries.train.tsv","train_qrels_path":"path/to/qrels.train.tsv"}' \ --output-dir outputs/dlhard_query2doc`
	Command to reproduce results on TREC DL 2019: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"mode":"cot"}' \ --output-dir outputs/dl19_query2doc_cot` Command to reproduce results on TREC DL 2020: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"mode":"cot"}' \ --output-dir outputs/dl20_query2doc_cot` Command to reproduce results on DL HARD: `python examples/querygym_pyserini/pipeline.py \ --queries-file path/to/queries.tsv \ --qrels-file path/to/qrels.trec \ --index-name msmarco-v1-passage \ --method query2doc \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"mode":"cot"}' \ --output-dir outputs/dlhard_query2doc_cot`
	Query2E (ZS)	0.3709	0.5679	0.8384		0.3436	0.5624	0.8373		0.1845	0.3179	0.7642
Command to reproduce results on TREC DL 2019: `python examples/querygym_pyserini/pipeline.py \ --queries-file path/to/original_queries.tsv \ --qrels-file path/to/qrels.trec \ --index-name msmarco-v1-passage \ --method query2e \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"mode":"zs"}' \ --output-dir outputs/dl19_query2e_zs` Command to reproduce results on TREC DL 2020: `python examples/querygym_pyserini/pipeline.py \ --queries-file path/to/original_queries.tsv \ --qrels-file path/to/qrels.trec \ --index-name msmarco-v1-passage \ --method query2e \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"mode":"zs"}' \ --output-dir outputs/dl20_query2e_zs` Command to reproduce results on DL HARD: `python examples/querygym_pyserini/pipeline.py \ --queries-file path/to/original_queries.tsv \ --qrels-file path/to/qrels.trec \ --index-name msmarco-v1-passage \ --method query2e \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"mode":"zs"}' \ --output-dir outputs/dlhard_query2e_zs`
CSQE	0.4007	0.5962	0.8506		0.3542	0.5298	0.8431		0.2170	0.3194	0.7276
Command to reproduce results on TREC DL 2019: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method csqe \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 1024 \ --method-params '{"retrieval_k":10,"gen_num":2}' \ --output-dir outputs/dl19_csqe` Command to reproduce results on TREC DL 2020: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method csqe \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 1024 \ --method-params '{"retrieval_k":10,"gen_num":2}' \ --output-dir outputs/dl20_csqe` Command to reproduce results on DL HARD: `python examples/querygym_pyserini/pipeline.py \ --queries-file path/to/original_queries.tsv \ --qrels-file path/to/qrels.trec \ --index-name msmarco-v1-passage \ --method csqe \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 1024 \ --method-params '{"retrieval_k":10,"gen_num":2}' \ --output-dir outputs/dlhard_csqe`
LameR	0.4185	0.6587	0.8611		0.4432	0.6353	0.8839		0.2562	0.3623	0.7887
Command to reproduce results on TREC DL 2019: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"retrieval_k":10,"gen_passages":5}' \ --output-dir outputs/dl19_lamer` Command to reproduce results on TREC DL 2020: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"retrieval_k":10,"gen_passages":5}' \ --output-dir outputs/dl20_lamer` Command to reproduce results on DL HARD: `python examples/querygym_pyserini/pipeline.py \ --queries-file path/to/original_queries.tsv \ --qrels-file path/to/qrels.trec \ --index-name msmarco-v1-passage \ --method lamer \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"retrieval_k":10,"gen_passages":5}' \ --output-dir outputs/dlhard_lamer`
MuGI	0.4766	0.6903	0.8822		0.4353	0.6300	0.8985		0.2393	0.3515	0.7974
Command to reproduce results on TREC DL 2019: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"num_docs":5,"parallel":true,"mode":"zs"}' \ --output-dir outputs/dl19_mugi_zs` Command to reproduce results on TREC DL 2020: `python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"num_docs":5,"parallel":true,"mode":"zs"}' \ --output-dir outputs/dl20_mugi_zs` Command to reproduce results on DL HARD: `python examples/querygym_pyserini/pipeline.py \ --queries-file path/to/original_queries.tsv \ --qrels-file path/to/qrels.trec \ --index-name msmarco-v1-passage \ --method mugi \ --model gpt-4.1-mini \ --temperature 1.0 \ --max-tokens 128 \ --method-params '{"num_docs":5,"parallel":true,"mode":"zs"}' \ --output-dir outputs/dlhard_mugi_zs`

Notes

AP: Average Precision (AP@100)
nDCG@10: Normalized Discounted Cumulative Gain at rank 10
R@1K: Recall at rank 1000
All results are reproducible using the commands shown in the expanded rows
Configuration column shows the LLM and retrieval backend used for all methods in that group

Programmatic Execution

All experimental runs shown in the above table can be programmatically executed using the QueryGym pipeline scripts. The pipeline command runs the complete workflow: reformulation, retrieval, and evaluation.

Example pipeline command:

python scripts/querygym_pyserini/pipeline.py \
  --dataset msmarco-v1-passage.dev \
  --method query2doc \
  --model gpt-4 \
  --base-url https://api.openai.com/v1 \
  --api-key YOUR_API_KEY \
  --output-dir runs/query2doc-gpt4-bm25

For more information, see the QueryGym Pyserini Pipeline documentation.