Artifact submissions

CCGrid 2026 will award 3 IEEE reproducibility badges to accepted main research track papers:

Submission guidelines

Authors seeking any badge must submit a 2-page artifact description including a brief artifact overview and review details:

• Software: Including link, language; build or run environment (tools, binaries, hardware), input dataset, expected output or metrics, estimated times for build or run.
• Data: Include a link and documentation per the criteria above.

When submitting:

• Clearly map each artifact to specific parts of the accepted paper.
• Cite artifacts with a persistent identifier, such as a DOI from Zenodo or Figshare. Hosting software and data on GitHub or GitLab isn't sufficient. We recommend repositories like Zenodo, Dryad, or figshare to promote FAIR principles.
• Optionally add a development URL, such as GitHub.
• Use the IEEE conference template.

Artifact evaluation is post-acceptance only and doesn't influence acceptance decisions.

If artifacts pass evaluation, update the 2-page description and include it as an appendix to the camera-ready paper, along with the badge logo. This step is required for publication with a badge.

Submission notes

• Badging summary:

  • ORO = DOI'd archive + license + docs
  • ROR = successful install or run + complete documents or tests
  • ROR-R = reviewers reproduce main behaviour or metrics.

• Commodity hardware: Artifacts should run on typical laptops or desktops. This depends mainly on mainstream packages.

• Packaging: Preferred container or VM for nontrivial dependencies, such as a Docker image or OVF VM, and to keep images small.

• Containerisation (recommended): For ROR and ROR-R, provide Docker Compose build and Docker Compose run (or Apptainer), bind-mount outputs to ./out/, include a 30 to 60 min smoke test with expected outputs (files, metrics, hashes, plots).

• Versions and runtime:

  • State the exact CUDA, driver and toolchain versions
  • Pin packages (lockfile).
  • Target ≤1h install + ≤2h minimal run.
  • If a special HW or cloud is needed, provide temporary access with full step-by-step instructions.

• Specialised environments: If tightly coupled to HP, cloud or accelerators, authors must provide reviewer access. State this in both the artifact submission and EasyChair abstract, and share credentials with AEC as directed.

Timelines

• Deadline for artifact submission for accepted papers: 20 February
• Artifact review midpoint check-in: 25 February
• Artifact review results announcements to authors: 10 March
• Camera-ready with 2-page appendix for awarded artifacts: 15 March

We'll hold a midpoint check-in to catch easy issues early, such as missing files or imports.

Artifact badges

1. Open Research Objects (ORO)

Indicates that author-created digital objects (data or code) are:

• permanently stored in a public repository with a globally unique identifier, guaranteed persistence, and an open license to maximise access
• aligned with the Association for Computing Machinery (ACM) 'Artifacts Available' and COS 'Open Data or Materials' for digital objects.

The AEC and authors decide what objects are 'relevant', and provide enough documentation for reviewers to understand core functionality and data context.

Review criteria

• Availability: Public repository with global identifier and guaranteed persistence.

• License: Open license maximising accessibility, such as OSI, CC, public domain.

• Authorship: Reasonable and complete authorship attached to the archived artifact.

• Documentation:

  • for software, core functionality overview
  • for data, context, description, source, and usage in the paper
  • metadata accessible for relevant physical objects.

2. Research Objects Reviewed (ROR)

Higher-level than ORO, and requires ORO.

Corresponds to the IEEE 'Code Reviewed'.

All author-created digital objects used in the research (data and software) are reviewed.

Software review criteria

• Documentation: Statement of need or function, install instructions, usage examples, API documents.
• Functionality: Reviewers can install and verify core functionality on commodity hardware with step-by-step instructions.
• Testing: Documented manual checks, such as sample input and expected output, and ideally an automated test suite with CI, such as GitHub Actions.

Data review criteria

• Documentation: Dataset context, description, source, and how it's used in the paper.
• Functionality: Enable reuse beyond the paper; for proprietary formats, include code to access data programmatically.

Additional requirements

Paper has already qualified for ORO.

3. Results Reproduced (ROR-R)

Highest tier, awarded when evaluators reproduce the paper's key results using the authors' objects, methods, code, and analysis conditions. The focus is on reproducing behaviour or claims, not exact values (especially with hardware dependence).

Review criteria

• Reproduce behaviour: Attempt on the same hardware where feasible. Otherwise, collaborate to establish equivalent behaviour on commodity hardware.
• Reproduce central results: AEC identifies key results or claims. The badge is based on reproducing these behaviours.

Additional requirements

• Paper has already qualified for ROR and ORO.
• Authors assist with materials and clarifications.

Process

1. Authors submit a comprehensive reproduction plan and research objects.
2. Independent attempt to reproduce by evaluators.
3. Author–evaluator collaboration to resolve discrepancies.
4. Final verification by the AEC. The ROR-R badge appears next to ROR in proceedings.