Assessing Reproducibility in Quantum Computing Research: 64.5% of Shared Code Fails to Run

A comprehensive study by researchers at Leibniz Universität Hannover has found that nearly two-thirds of the quantum code shared alongside research papers fails to execute in new environments. The analysis of nearly 5,000 papers reveals that unrecorded development environment details and undocumented dependencies are major barriers to verifying scientific claims. This lack of reproducibility poses a significant challenge for the quantum computing sector, where independent validation is essential for building upon complex software-driven advancements.

Dominik Köster and a team of researchers at Leibniz Universität Hannover conducted a dual-method assessment of reproducibility in quantum computing research, involving a manual evaluation of 127 papers and an automated screening of nearly 5,000 publications. Their findings indicate that while code sharing is a recognized goal, only 24.4% of manually reviewed papers actually provided code artifacts, a figure mirrored by a 26.8% availability rate in the larger automated sample. Most critically, 64.5% of the publicly accessible code failed to run successfully in a clean computing environment, suggesting that many research artifacts may never have been fully reproducible outside of their original development setups.

The study identifies that these failures are rarely due to simple code errors or outdated software; instead, they stem from incomplete or implicit environment specifications. Approximately one-third of the papers with accessible code lacked machine-readable environment details, often omitting unreferenced locally installed dependencies or undocumented configuration steps. This lack of precise documentation creates a significant hurdle for independent researchers attempting to validate reported results, as the complex software stacks required for quantum experiments often rely on specific, unstated configurations that are not captured by standard code-sharing practices.

To address these systemic issues, the authors advocate for the use of declarative environment specifications through tools such as Nix and devenv.sh. These platforms enable researchers to create explicit and reproducible definitions of their entire software environment, reducing the reliance on undocumented assumptions and ensuring long-term reproducibility even as hardware and dependencies evolve. The researchers emphasize that while providing code is a commendable and necessary precondition for scientific progress, the quantum computing industry must prioritize the inclusion of complete execution contexts to bridge the gap between published claims and verifiable reality.