Nix Build Reproducibility Verification With Diffoscope
Detect nondeterministic Nix builds by comparing outputs with content-aware diffing.
Nix build reproducibility verification with diffoscope tests a claim that sandboxing and hashing alone cannot establish. The same derivation must produce the same bytes. A workflow combining rebuild checks, preserved outputs, and diffoscope reports turns that requirement into evidence that maintainers can inspect and CI can act on.
Why Nix's sandboxing and hashing are not the same as bitwise reproducibility
Nix controls the declared inputs, but a build can still do things that are not deterministic. Sandboxing blocks network access and unspecified dependencies, while input-addressed hashing pins dependency references. Neither prevents a derivation from writing $RANDOM to its output. Such a derivation can build cleanly under Nix’s model and produce a different file every time. A valid build therefore establishes something narrower than bitwise reproducibility.
Malka’s 2025 study rebuilt packages from nixpkgs revisions spanning 2017–2023 and found reproducibility rates ranging from roughly two-thirds to just over nine-tenths. Those results show that Nix supports reproducibility at scale without making it automatic. The live NixOS GNOME ISO tracker likewise exposes unreproduced paths rather than treating a completed image as proof of identical rebuilds. Maintainers need to identify which uncontrolled value reached the output and how to remove it.
The recurring causes of non-reproducibility inside a Nix sandbox
Four recurring causes explain the identifiable failures, namely embedded build dates, embedded uname output, embedded environment variables, and non-deterministic build IDs. Malka et al. identified embedded build dates as the largest cause, despite timestamp removal being the reproducible-builds community’s oldest recommendation. Compilers, documentation generators, and archive tools can write the current time unless they are told to leave it out. The .pyc mod-date stamps in nixpkgs issue #566878, the immich unreproducibility example recorded as of September 2026, are an example of this problem.
Some build systems query the running kernel through uname and embed the result in binaries or metadata, although the kernel is outside the sandbox’s control. Environment variables can also carry facts about the host, such as its CPU core count. A four-core laptop and a 32-core CI runner can consequently produce different output from the same derivation. The immich artifact .pnpm-workspace-state-v1.json illustrates why generated state files deserve attention as well as compiled binaries.
Build IDs add another source of divergence that can be less obvious than a date or host value. Ecosystems such as Go derive IDs from build inputs, including source, compiler, flags, and dependencies. If uncontrolled inputs such as absolute file paths differ, the ID can differ even when the source is the same. The immich issue also contains plugin.wasm, a non-deterministic build artifact. Together, these files show why a package-level failure needs separate explanations for the artifacts inside it rather than a single assumption about the whole build.
What diffoscope does that a plain diff cannot
A standard recursive diff can show that two store paths differ without showing the structure inside the files where they differ. Diffoscope makes that structure visible through content-aware, recursive comparison. It descends into archive formats, ELF sections, Java class files, WebAssembly modules, and Python bytecode. Instead of stopping at an opaque binary mismatch, it can expose the embedded field or section that changed.
The immich HTML report identified plugin.wasm, .pyc timestamps, and .pnpm-workspace-state-v1.json as separate sources of divergence. The report’s value lies in attribution, because knowing which field changed narrows the search for the process that wrote it. A timestamp field and a WebAssembly section may both break reproducibility, but they do not imply the same remedy.
Diffoscope also emits JSON, allowing CI tooling to process the same differences that maintainers inspect. A report can serve as a debugging attachment and as input to a pipeline failure, an issue, or a dashboard. This machine-readable form connects diagnosis to repeatable verification. The tool’s ecosystem-specific decoders let automation work with the fields and sections that changed, beyond the fact that some bytes differ.
Running the verification workflow: --rebuild, --check --keep-failed, and diff-hook
The workflow begins by triggering another build of an existing result. With the new CLI, run nix build nixpkgs#hello --rebuild --keep-failed. The --rebuild flag requests the rebuild, while --keep-failed preserves the diverging result for inspection. The classic equivalent is nix-build '<nixpkgs>' -A hello --check --keep-failed. Here, --check compares a rebuild against the output already present on the system.
The Nix reference manual says that a deterministic build exits 0 and needs no further comparison. A differing result exits 1 with the message “derivation may not be deterministic: output <Y> differs from <Z>”. The classic --check operation requires the derivation to have been built already. If that prerequisite is missing, Nix refuses the check, so build first without the flag and then repeat with it. Fixed-output derivations, including fetchers, require a different protocol from this check.
Reproducible.nixos.org specifies four builds for fixed-output derivations, alternating between an empty outputHash and the hash produced in the prior build. Preserving outputs is equally important for ordinary derivations. With --keep-failed, Nix writes the second output to a .check path such as /nix/store/<hash>-unstable.check. The Nix reference manual warns that garbage collection can remove these paths, which are sure to stay alive only while the diff-hook runs. Automated tooling must therefore handle a comparison path that has already been collected.
For the comparison itself, run nix run nixpkgs#diffoscopeMinimal -- --exclude-directory-metadata recursive <Y> <Z>. The --exclude-directory-metadata recursive option suppresses directory metadata differences, including directory mtime, so filesystem timestamp noise does not obscure content differences. Diffoscope accepts --json-output <file> to write machine-readable results in addition to its default human-readable output. This invocation provides the comparison tool without a separate installation step. The two paths supplied to it are the original and rebuilt outputs identified by Nix.
To automate the comparison, set diff-hook = /path/to/script and run-diff-hook = true in nix.conf. The hook receives the two paths as positional arguments and runs as the user who initiated the build. It can read and report on the store paths, but it has no write access to them. Crucially, Nix invokes the hook only when results differ, so it diagnoses failure. A hook that calls diffoscope and writes JSON to a known path gives downstream CI a report without having to build again.
Reading a diffoscope report: tracing the immich failure from symptom to root cause
A diffoscope report is a recursive tree, and its leaf-level differences are the starting points for diagnosis. Each leaf identifies a change within an enclosing file or artifact. Reading from that change back to its container connects a visible symptom to the build process responsible for it. The immich report demonstrates three different levels of repair difficulty. Its generated state file offers the most local starting point, followed by Python timestamps and then the WebAssembly dependency.
The .pnpm-workspace-state-v1.json difference concerns a pnpm-generated state file that embeds build-time metadata. A separate nixpkgs PR, #556271, addressed this problem. This is the configuration-level end of the investigation, where the generated metadata directs attention to package-side handling. The .pyc differences are in the Python bytecode mod-date stamps. Their timestamp fields point toward passing SOURCE_DATE_EPOCH or preventing node-gyp from emitting .pyc files, with a PR and commits referenced in the issue.
The plugin.wasm divergence reaches further upstream. Diffoscope flags a section-level difference inside the WebAssembly module, while the root cause is wizer’s non-reproducible build output, tracked in bytecodealliance/wizer#153. Fixing that difference takes an upstream patch or a pinned pre-built artifact. The report separates these repair paths instead of implying that all three failures can be fixed locally.
That distinction also makes automated classification useful. JSON reports can be parsed for known patterns, including timestamp fields, random IDs, and environment variable strings. A CI job can then route a recognized failure to the appropriate team or issue template. Routing gives the team a place to start, but it still needs to find the root cause. The earlier failure categories become useful because they correspond to evidence within the report.
Scaling verification: monitoring across the package set and integrating into CI
A successful check of one package is only a spot-check. Continuous verification must cover the full dependency closure if it is to become a supply chain control. Reproducible.nixos.org tracks the Minimal ISO and GNOME ISO runtime and build-time dependency sets. Its GNOME ISO image figure is 95.18% reproducible paths at the time of writing, with per-path diffoscope links for unreproduced paths.
Nixpkgs records known failures under the tag 6.topic: reproducible builds, creating a queryable backlog. In CI, the corresponding pattern is a diff-hook that writes JSON to a known artifact path, followed by a step that fails the pipeline when that file is non-empty. The CI check must look in the same location where the hook writes its output file.
The JSON report can be attached to the CI run, linked in a PR comment, or parsed to file a nixpkgs issue using the standard reproducibility issue template. Malka et al. also released a dataset of non-reproducibility events with full diffoscopes. Teams building heuristic classifiers can use that dataset to tune pattern matching against known failure signatures, so they have more to work with than their own failures.
The broader direction is enforcement beyond CI. Repro-threshold intercepts package downloads through an APT transport and queries independent rebuilders for signed attestations before permitting installation. Per-build comparison remains the foundation, but the resulting evidence can support controls at other points in delivery.
Why reproducibility verification strengthens the supply chain argument, not just the build log
Bitwise reproducibility makes a delivered binary independently verifiable rather than solely dependent on trust in its build server. Rebuilding and comparing establish whether declared inputs converge on the delivered artifact. Diffoscope explains a mismatch when that comparison fails. This is a stronger claim than successful compilation or a provenance statement alone. It is also the reason Nix usage must include active verification.
Malka’s analysis argues that proactive rebuild-and-compare verification within the nixpkgs bootstrap could have mechanically detected the xz compromise. An independently sourced tarball would have produced a binary different from the one produced by the build server. NixOS did ship the malicious xz builds, but the backdoor did not activate because it targeted RPM/Debian packaging hooks specifically. That outcome is distinct from detecting the compromise. Passive usage did not supply the comparison that the proactive workflow could have provided.
An SBOM signed by a build server attests provenance but does not prove that the delivered artifact matches its declared inputs. Consumers can check that match by reproducing the artifact and comparing digests. A proposed SBOM integrity verification framework combines authority-generated reference digests with consumer-side reproduction, with diffoscope suited to diagnosing differences on the consumer side. The DevGuard project demonstrates a related approach through reproducible Nix builds compared across GitHub Actions and sovereign GitLab infrastructure on openCode, signed with Sigstore/Cosign. Independent digest agreement strengthens the integrity claim beyond provenance attestations alone.
Reproducibility does not, however, establish that software is free of known vulnerabilities. A reproducible artifact carrying known CVEs remains a liability. Input-derived hashes attached to deployments support the separate task of identifying which deployed environments contain a vulnerable artifact without rescanning images.
Embedding reproducibility verification in a shared environment so the whole team benefits
A workflow kept in shell history relies on what each person does, while a versioned shared environment gives the team control over it. The verification stack has dependencies of its own, including diffoscope’s decoders, the nix CLI, jq for JSON parsing, and CI integration scripts. Pinning these tools together lets local runs and CI use the same versions. A declarative manifest can also encode hook scripts and output conventions. Any team member or CI runner can then get the same verification setup when they need it.
Versioning matters when decoder compatibility changes as well as when contributors join. Diffoscope’s June 2026 development covered versions 319–323, cross-distribution compatibility fixes, apktool version 3 support, and Androguard version 4 support. With a shared environment, rollback after a compatibility break is an atomic operation, with no need to hunt down dependencies by hand. New contributors and CI agents receive the full stack in one step.
