Mandatory Research Artifacts: Should Every Paper Include Data and Code?

Getting your Trinity Audio player ready...

Scientific papers traditionally present conclusions, methods, and selected results. Yet in many fields, the paper itself is no longer enough to evaluate the research. The underlying datasets, source code, computational environments, protocols, and analysis scripts may contain much of the information needed to verify the authors’ claims.

As a general rule, every empirical or computational paper should provide the artifacts necessary to evaluate its central claims. However, this should be a requirement to disclose or justify—not an inflexible demand to publish every file without regard to privacy, security, licensing, cost, or research ethics.

The practical standard should be:

Authors must make relevant research artifacts accessible unless they provide a specific, reviewable reason why an artifact cannot be shared.

This model makes artifact availability the default while recognizing that responsible science sometimes requires controlled access, partial disclosure, synthetic data, delayed publication, or other alternatives.

What Are Research Artifacts?

A research artifact is any material used to produce, validate, or reproduce a scientific result beyond the prose of the paper itself.

Depending on the discipline, artifacts may include:

• raw and processed data;
• source code and analysis scripts;
• software dependencies and environment specifications;
• trained machine-learning models;
• experimental protocols;
• mathematical proofs or proof-assistant files;
• laboratory notebooks;
• questionnaires and interview instruments;
• simulation configurations;
• hardware designs;
• negative results and failed experimental runs;
• instructions for reproducing figures and tables.

Not every paper produces every type of artifact. A theorem in pure mathematics may have no dataset, while an experimental biology paper may depend on specimens, laboratory procedures, imaging files, and statistical code. Artifact requirements must therefore follow the actual claims being made.

Why Papers Alone Are Often Insufficient

A methods section compresses a complex research process into a limited number of pages. Important operational decisions may disappear during that compression.

A statistical result can depend on undocumented exclusions, variable transformations, random seeds, software versions, preprocessing steps, or parameter choices. A systems paper may depend on configuration files and benchmarks that cannot be reconstructed from its textual description. A machine-learning paper may describe an architecture without providing enough information to recreate the training process.

Releasing the relevant artifacts can help other researchers:

• inspect how reported results were produced;
• detect implementation or data-processing errors;
• distinguish conceptual failures from software bugs;
• reproduce tables and figures;
• test the method on new datasets;
• reuse expensive scientific resources;
• compare competing methods under common conditions;
• build directly on previous research.

The US National Institutes of Health states that scientific-data sharing can improve research rigor, reproducibility, accessibility, and reuse. The US National Science Foundation similarly requires data-management and sharing plans in research proposals and expects investigators to address access, preservation, privacy, security, intellectual property, and reuse.

Artifacts therefore serve at least three different purposes:

1. Verification: Can reviewers inspect the evidence behind the claims?
2. Reproduction: Can another researcher obtain substantially the same result using the supplied materials?
3. Reuse: Can the artifact support new research beyond the original paper?

These purposes should not be conflated. A dataset may be available but poorly documented. Code may execute but fail to reproduce the published figures. An artifact may reproduce a result while being legally unavailable for unrestricted reuse.

USENIX Security 2026: A Stronger Open-Science Model

USENIX Security 2026 provides a significant example of artifact availability moving from an optional benefit toward a publication expectation.

Its call for papers states that authors are required to share research artifacts openly at submission time as part of the conference’s open-science policy. Relevant artifacts may include datasets, scripts, binaries, and source code. The artifacts can be evaluated for availability, with optional further evaluation of functionality and reproducibility.

The policy does not treat openness as an excuse to ignore ethical constraints. Some security artifacts could expose vulnerable systems, facilitate abuse, violate licenses, reveal confidential information, or endanger research subjects. USENIX therefore permits justified restrictions and requires authors to discuss open-science and ethical considerations explicitly.

This is an important distinction. The emerging norm is not simply “upload everything.” It is:

Share what can responsibly be shared, identify what cannot be shared, and explain the restriction in enough detail for reviewers to assess whether it is legitimate.

That principle can extend well beyond computer security.

Should Data and Code Be Mandatory for Every Paper?

Data and code should be mandatory when they are necessary to evaluate the paper’s claims and can be shared responsibly. But the universal requirement should concern artifact accountability, not identical artifact publication.

A useful policy would place papers into four categories.

Papers With Fully Public Artifacts

This should be the default for most non-sensitive computational and empirical research.

Authors should deposit artifacts in a persistent repository rather than relying only on a personal website or temporary cloud link. The deposit should include a stable identifier, a license, documentation, and the version associated with the published paper.

Where practical, the package should also contain:

• a README file;
• installation and execution instructions;
• dependency versions;
• expected outputs;
• scripts that recreate the main tables and figures;
• machine-readable metadata;
• a citation file;
• an explicit reuse license.

Simply releasing an undocumented folder should not qualify as adequate artifact sharing.

Papers With Controlled-Access Artifacts

Some data can be shared only with qualified researchers or under specific conditions.

Examples include human-subject data, medical records, educational records, commercially sensitive datasets, and information governed by consent agreements. In these cases, authors could provide:

• an application procedure;
• access criteria;
• a data-use agreement;
• an independent data custodian;
• a secure analysis environment;
• anonymized or aggregated subsets;
• synthetic data for testing the code.

“Available upon reasonable request” is generally too vague unless “reasonable,” the decision-maker, the conditions, and the expected response process are clearly defined.

Papers With Partially Shareable Artifacts

Sometimes code can be published while the data cannot. In other cases, derived data can be released but raw data cannot. A commercial dataset may be inaccessible, but analysis code, schema descriptions, and acquisition instructions may still be shareable.

Authors should disclose artifacts component by component rather than treating the entire research package as either open or closed.

For example:

Artifact	Availability	Explanation
Analysis code	Public	Archived repository with permanent identifier
Raw participant data	Restricted	Consent and privacy limitations
De-identified aggregate data	Public	Sufficient for reproducing primary statistics
Proprietary software	Not redistributable	Version and configuration documented
Reproduction instructions	Public	Includes expected outputs and checksums

This gives reviewers and readers a precise view of what can actually be examined.

Papers With No Relevant Data or Code

Some papers genuinely do not depend on data or software. Conceptual philosophy papers, certain theoretical works, literature reviews, and handwritten mathematical proofs may fall into this category.

Such papers should not be forced to create meaningless repositories merely to satisfy a checklist. Authors can instead provide a short artifact statement explaining that no external dataset or computational code is required to support the claims.

In mathematics, however, supplementary artifacts may still be valuable. Long computer-assisted proofs should include source files, proof-assistant code, certificates, or verification instructions where applicable.

Legitimate Exceptions to Public Artifact Sharing

A strong artifact policy must recognize real constraints without allowing generic excuses.

Privacy and Research-Subject Protection

Removing names from a dataset may not eliminate re-identification risk. Highly dimensional medical, genomic, geographic, or behavioral information can remain sensitive even after conventional anonymization.

Ethical commitments to participants take priority over unrestricted openness. Reviewers should evaluate whether controlled access, aggregation, synthetic data, or confidential independent verification can provide accountability without exposing participants.

Cybersecurity and Adversarial Risk

Security research may produce exploit code, vulnerability details, malware samples, or techniques that could be misused. Immediate public release may create risks before affected systems can be repaired.

Possible alternatives include coordinated disclosure, delayed artifact release, access for vetted reviewers, publication of defensive components, or a reduced demonstration that verifies the scientific claim without operationalizing the harmful capability.

The restriction should be proportionate. “Security concerns” should not become a blanket exemption from scrutiny.

Intellectual Property and Licensing

Researchers may use proprietary datasets, licensed databases, commercial laboratory systems, or third-party code that they are not legally permitted to redistribute.

They should still disclose the source, license conditions, version, acquisition procedure, relevant configuration, and any shareable transformation or analysis code. A reproducibility limitation caused by proprietary inputs should be visible to readers and may appropriately reduce the strength of the paper’s reproducibility claims.

Indigenous, Community, and Cultural Data Governance

Legal permission does not always settle the ethical question of data publication. Some communities have legitimate governance interests over data derived from their members, territories, traditional knowledge, or cultural resources.

Artifact policy should therefore distinguish open access from responsible stewardship. A controlled repository governed with the affected community may be more ethical than unrestricted public release.

Excessive Size or Preservation Cost

Some fields generate enormous datasets that are expensive to host and transfer. Nevertheless, cost should normally change the method of access rather than eliminate accountability.

Researchers might share representative subsets, processed outputs, metadata, checksums, query access, cloud-hosted analysis environments, or instructions for regenerating the dataset. Funders should also recognize long-term artifact preservation as a legitimate research expense.

Why Mandatory Uploads Are Not Enough

A repository link does not prove that a result is reproducible.

Artifacts can be incomplete, undocumented, corrupted, dependent on unavailable software, or incompatible with current systems. Code may require hidden files, private credentials, undocumented manual steps, or hardware that reviewers cannot access.

Artifact policy should therefore separate several evaluations:

• Available: The artifact can be accessed.
• Complete: The components needed to evaluate the main claims are present.
• Documented: A competent researcher can understand how to use it.
• Functional: The artifact executes or behaves as described.
• Reproducible: It can regenerate the reported result within defined tolerances.
• Reusable: It is sufficiently licensed, structured, and documented for further work.
• Preserved: A stable version is archived for long-term access.

USENIX Security’s distinction between availability verification and optional functionality or reproducibility evaluation illustrates why a single “artifact included” checkbox is inadequate.

Artifact Review Requires Time and Expertise

Mandatory artifacts create work for authors, reviewers, repositories, and institutions. That work is scientifically valuable, but it is not free.

Running unfamiliar software, inspecting datasets, rebuilding environments, and validating outputs can require hours or days. Reviewers may need domain expertise, specialized hardware, access to secure facilities, or knowledge of particular programming languages.

Artifact review should therefore become a recognized and, where possible, paid scientific activity. It should not simply be added to the unpaid workload of conventional peer reviewers.

A mature system could assign separate reviewers to:

• the scientific argument;
• the dataset and its documentation;
• the software implementation;
• statistical reproducibility;
• ethical and legal restrictions;
• long-term preservation.

This division of labor aligns with the broader case for paying scientific reviewers for specific verification work rather than treating peer review as one undifferentiated judgment.

Artifacts Should Receive Separate Scientific Credit

A dataset can be valuable even when the accompanying hypothesis is later rejected. Well-engineered research software may support dozens of later studies. A formal proof library, benchmark suite, protocol, or replication package may have more lasting utility than the original article.

Research assessment should therefore avoid treating artifacts as mere appendices. Data, code, proofs, replications, and explanatory papers are different outputs and should receive separate credit.

This matters because the incentives for publishing a paper are currently much stronger than the incentives for cleaning code, documenting data, maintaining repositories, or making experiments reproducible. A policy that mandates artifacts without rewarding the work required to produce them risks encouraging minimal, low-quality compliance.

The better model combines requirements with incentives:

Require artifact disclosure, evaluate artifacts independently, and reward each useful output according to its verified scientific value.

How AIIM Could Evaluate Research Artifacts

The AI Internet Meritocracy could evaluate data, code, proofs, and replications separately from the article that introduces them.

An AI-assisted artifact process could begin with mechanical checks:

• Does the repository exist?
• Is the referenced version archived?
• Are files accessible?
• Is a license specified?
• Are dependencies declared?
• Do documented commands execute?
• Can the main tables or figures be regenerated?
• Are checksums and expected outputs supplied?
• Does the artifact contain obvious personal or confidential information?
• Do the paper’s artifact claims match the repository contents?

AI systems could also compare a manuscript with its artifacts, identify undocumented parameters, locate missing files, flag discrepancies between reported and generated results, and prepare a structured report for human reviewers.

However, AI should not make final decisions about privacy, consent, security risk, community governance, or the scientific meaning of discrepancies without accountable human oversight. These questions require contextual judgment and sometimes confidential information unavailable to an automated evaluator.

AIIM could assign different rewards for:

• making an artifact publicly available;
• improving its documentation;
• reproducing the original result;
• finding a defect;
• repairing broken research software;
• preserving an abandoned dataset;
• creating a reusable benchmark;
• producing an independent replication;
• verifying that a claimed exception to sharing is legitimate.

This would turn artifact review from a ceremonial publication badge into an ongoing scientific market for verification, maintenance, and reuse.

A Practical Mandatory-Artifact Policy

Journals, conferences, and funding systems could implement the following standard.

1. Require an Artifact Declaration

Every submitted paper should state:

• which artifacts exist;
• which claims depend on them;
• where they can be accessed;
• under what license or conditions;
• which artifacts are unavailable;
• why any restrictions are necessary.

2. Make Sharing the Default

Artifacts necessary to support central claims should be deposited in a persistent repository unless a justified exception applies.

3. Evaluate Restrictions

Editors or dedicated artifact reviewers should examine whether privacy, security, licensing, or ethical restrictions are specific and proportionate.

4. Require the Best Available Alternative

When full publication is impossible, authors should provide the strongest safe alternative: controlled access, aggregate data, synthetic data, executable access, confidential review, delayed release, detailed metadata, or reproduction instructions.

5. Assess Artifact Quality Separately

Availability, functionality, reproducibility, reuse, and preservation should receive distinct assessments rather than one binary badge.

6. Publish the Evaluation

Readers should be able to see what reviewers actually tested. A reproducibility badge should identify the tested environment, commands, outputs, tolerances, and limitations.

7. Preserve Versioned Artifacts

The evaluated artifact must be frozen and linked to the paper. Continued development can occur in a separate repository, but readers must retain access to the exact reviewed version.

8. Reward Artifact Producers and Reviewers

Researchers who create reusable data and software—and reviewers who verify them—should receive citations, reputation, and financial rewards proportionate to their contributions.

Could Mandatory Artifacts Exclude Underfunded Researchers?

Artifact preparation can disadvantage researchers who lack software engineers, data curators, repository funding, or institutional support. Poorly designed mandates may favor wealthy laboratories that can afford polished research infrastructure.

The answer is not to abandon artifact requirements. It is to fund compliance.

Grant budgets and retroactive funding systems should support:

• data cleaning and documentation;
• software packaging;
• repository fees;
• privacy review;
• long-term storage;
• reproducible computational environments;
• professional artifact evaluation.

Automated tools can also reduce the burden by generating environment files, checking dependencies, creating metadata, testing installation procedures, and detecting missing documentation.

Open science should broaden participation, not create another gatekeeping mechanism.

The Right Rule: Artifacts or an Accountable Explanation

Not every scientific paper can publish all its data and code. Some papers have no such artifacts, while others involve legitimate privacy, security, legal, ethical, or practical restrictions.

Nevertheless, every paper can provide an accountable artifact declaration.

The strongest general policy is therefore:

Every paper should provide the artifacts necessary to evaluate its central claims—or clearly explain, component by component, why access must be restricted and what alternative verification is available.

This approach avoids both extremes. It rejects a publication culture in which readers must trust undocumented results, but it also rejects careless openness that can expose research subjects or create other harms.

Research artifacts should become first-class scientific outputs: identifiable, versioned, reviewed, preserved, credited, and rewarded. Combined with open-science funding and independent artifact evaluation, mandatory disclosure can make scientific publishing more verifiable without reducing research integrity to a rigid upload requirement.