100% Reproducible OpenWrt

This one has been cooking for a long time.

tl;dr: OpenWrt (SNAPSHOT) is now 100% reproducible.

I planned a blog post like this back in 2019 when I first worked on the reproducibility of OpenWrt. The work was initially fueled by h01ger, who kindly invited me to the Reproducible Builds Summit 2019 in Marrakesh. Discussions at the conference motivated me to look into rebuilding firmware rather than building twice in different environments. Previous OpenWrt CI tests would build the same OpenWrt commit twice on two hosts with different times, different build paths, different GCC versions etc., whereas I wanted to check if I could fully recreate the officially published OpenWrt firmware images, therefore “re-building”. In other words, it’s a verification build to prove that whatever was published is really based on the source. While sources for this claim are missing, kpcyrd and I think the term rebuilder was coined at that conference.

openwrt-rebuilder

So it needed a rebuilder for OpenWrt, which became small software that consumes the OpenWrt buildinfo files. They contain the version (version.buildinfo), the build configuration (config.buildinfo) and the package feeds with their respective Git commit (feeds.buildinfo). When running, all files are put in place and parsed, and the official OpenWrt repository is cloned and built.

Initial versions of openwrt-rebuilder compared created artifacts on its own, ran diffoscope and were triggered via GitHub CI. Instead of doing the tracking via CI scripts, the tool was rewritten to be run by rebuilderd. rebuilderd parses upstream, finds new versions and offers a queue for attached workers. The integration bits are still pending, however I’m optimistic about getting this upstream eventually.

Making sense of diffs

Diffoscope is an excellent tool. You hand it two files and it tells you what actually makes them different. A long list of archive formats and binary decoders is supported, so the output is as human readable as possible.

In cases where diffoscope doesn’t know how to proceed, it shows a hexdump. While various AIs are surprisingly good at understanding hex dumps and figuring out what causes reproducibility issues, I’m not. As a consequence, I added APKv3 (not the Android stuff) support to diffoscope, neatly printing issues. Once APK was supported, I struggled with further hexdumps and therefore added further patches. Below is an example of how diffoscope drills through the image and uncovers GNU Build ID as the villain.

gzip: /tmp/rebuilderdaGqkBb/inputs/openwrt-x86-64-generic-squashfs-combined.img.gz: decompression OK, trailing garbage ignored
--- /tmp/rebuilderdaGqkBb/inputs/openwrt-x86-64-generic-squashfs-combined.img.gz
+++ /tmp/rebuilderdaGqkBb/out/openwrt-x86-64-generic-squashfs-combined.img.gz
├── openwrt-x86-64-generic-squashfs-combined.img
│ ├── part1
│ │ ├── part1.tar
│ │ │ ├── ./boot/vmlinuz
│ │ │ │ ├── filetype from file(1)
│ │ │ │ │ @@ -1 +1 @@
│ │ │ │ │ -Linux kernel x86 boot executable, bzImage, version 6.18.38 (builder@buildhost) #0 SMP Fri Jul 17 08:02:59 2026, RO-rootFS, Normal VGA, setup size 512*31, syssize 0x6c040, jump 0x26c 0x8cd88ec0fc8cd239 instruction, protocol 2.15, from protected-mode code at offset 0x2c4 0x6b252c bytes XZ compressed, relocatable, handover offset 0x6ba2cf, legacy 64-bit entry point, can be above 4G, 64-bit EFI handoff entry point, EFI kexec boot support, xloadflags bit 5, max cmdline size 2047, init_size 0x1c6c000
│ │ │ │ │ +Linux kernel x86 boot executable, bzImage, version 6.18.38 (builder@buildhost) #0 SMP Fri Jul 17 08:02:59 2026, RO-rootFS, Normal VGA, setup size 512*31, syssize 0x6c040, jump 0x26c 0x8cd88ec0fc8cd239 instruction, protocol 2.15, from protected-mode code at offset 0x2c4 0x6b1d54 bytes XZ compressed, relocatable, handover offset 0x6b9b0f, legacy 64-bit entry point, can be above 4G, 64-bit EFI handoff entry point, EFI kexec boot support, xloadflags bit 5, max cmdline size 2047, init_size 0x1c6c000
│ │ │ │ ├── vmlinux
│ │ │ │ │ ├── readelf --wide --notes {}
│ │ │ │ │ │ @@ -1,11 +1,11 @@
│ │ │ │ │ │  
│ │ │ │ │ │  Displaying notes found in: .notes
│ │ │ │ │ │    Owner                Data size 	Description
│ │ │ │ │ │ -  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)	    Build ID: d981a9bbe8fa43067fc8e64806852f45d6c395ba
│ │ │ │ │ │ +  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)	    Build ID: 925f8055192d0534838d8269bc355d288d8bb7bb
│ │ │ │ │ │    Linux                0x00000004	func	   description data: 00 00 00 00 
│ │ │ │ │ │    Linux                0x00000001	OPEN	   description data: 00 
│ │ │ │ │ │    Xen                  0x0000000c	Unknown note type: (0x00000013)	   description data: 00 00 00 01 00 00 00 01 ff ff ff 1f 
│ │ │ │ │ │    Xen                  0x00000008	Unknown note type: (0x00000012)	   description data: 70 e1 74 02 00 00 00 00 
│ │ │ │ │ │    Xen                  0x00000006	Unknown note type: (0x00000006)	   description data: 6c 69 6e 75 78 00 
│ │ │ │ │ │    Xen                  0x00000004	Unknown note type: (0x00000007)	   description data: 32 2e 36 00 
│ │ │ │ │ │    Xen                  0x00000008	Unknown note type: (0x00000005)	   description data: 78 65 6e 2d 33 2e 30 00
│ │ │ │ │ ├── readelf --wide --decompress --hex-dump=.rodata {}
│ │ │ │ │ │ @@ -339287,16 +339287,16 @@
│ │ │ │ │ │    0xffffffff8252d540 00000000 00000000 00000000 00000000 ................
│ │ │ │ │ │    0xffffffff8252d550 65787061 6e642033 322d6279 7465206b expand 32-byte k
│ │ │ │ │ │    0xffffffff8252d560 38040000 00000000 00000000 00000000 8...............
│ │ │ │ │ │    0xffffffff8252d570 00000000 00000000 06000000 04000000 ................
│ │ │ │ │ │    0xffffffff8252d580 00000000 4c696e75 78000000 26120600 ....Linux...&...
│ │ │ │ │ │    0xffffffff8252d590 06000000 01000000 00010000 4c696e75 ............Linu
│ │ │ │ │ │    0xffffffff8252d5a0 78000000 00000000 04000000 14000000 x...............
│ │ │ │ │ │ -  0xffffffff8252d5b0 03000000 474e5500 b7e0b5f4 12822e0e ....GNU.........
│ │ │ │ │ │ -  0xffffffff8252d5c0 cf6f5eeb 812d8ff2 d080dee4 011b033b .o^..-.........;
│ │ │ │ │ │ +  0xffffffff8252d5b0 03000000 474e5500 c1bc1435 c2eb2e79 ....GNU....5...y
│ │ │ │ │ │ +  0xffffffff8252d5c0 d2dfa1b5 22029299 4a103c2c 011b033b ...."...J.<,...;
│ │ │ │ │ │    0xffffffff8252d5d0 50000000 09000000 b4010000 6c000000 P...........l...
│ │ │ │ │ │    0xffffffff8252d5e0 04020000 80000000 84020000 94000000 ................
│ │ │ │ │ │    0xffffffff8252d5f0 f4030000 c8000000 14040000 dc000000 ................
│ │ │ │ │ │    0xffffffff8252d600 b4050000 0c010000 14060000 24010000 ............$...
│ │ │ │ │ │    0xffffffff8252d610 54060000 3c010000 640b0000 8c010000 T...<...d.......
│ │ │ │ │ │    0xffffffff8252d620 14000000 00000000 017a5200 01781001 .........zR..x..
│ │ │ │ │ │    0xffffffff8252d630 1b0c0708 90010000 10000000 1c000000 ................

Making sense of results

The patched rebuilderd did a great job at fetching and checking firmwares, however it lacked a convenient way to inspect the results. Inspired by Reproducible Archlinux, I asked an AI to create an OpenWrt page with the old design. The result was this reproducible wireless freedom page and with further inspiration from Debian’s stats, the gamification to reach 100% reproducibility got started.

The OpenWrt frontend is entirely AI generated, pull requests for a better UI/UX are very welcome.

The image below shows the current UI. If a firmware or package isn’t reproducible, a diffoscope link is offered. For all successful rebuilds, an attest is offered, a cryptographically signed document, more on that later.

Lastly, the UI invites you to rebuild the target locally. This can be helpful in case diffoscope couldn’t fully unpack the image and you want to manually investigate. To save space, artifacts aren’t kept on the server itself.

May the patching begin

This effort started long ago and I’d really like to thank lynxis for all his work on this topic in the past. Reproducibility patches can be fun and easy; often it’s more a matter of diligence than complex problem solving. For inspiration, a very incomplete list of (recent) patches that got us to 100%.

Keeping 100%

The page is nice and 100% feels good, however it’s unclear how to keep it that way. I’m thinking of an automated system to create issues or notify interested developers directly. It’s always a continuous effort, OpenWrt keeps on adding more devices and so more corner cases may introduce unreproducible quirks.

Furthermore, I want to upstream my patches to rebuilderd and diffoscope. Only that way are the next steps really possible.

Independent Rebuilders

When I started with rebuilders, I envisioned that multiple entities, private or institutional, would spin up their own rebuilders and track OpenWrt reproducibility. From one of my other projects, the OpenWrt sysupgrade server, I learned that people do want to help and host those things on their own, but don’t have infinite time to figure out why their local setup doesn’t work. Before working on yet another tool around OpenWrt, I’ll try to make the rebuilder easy to deploy.

Once more people run a rebuilder and generate attests (cryptographic signatures for successful rebuilds based on in-toto), the logs could automatically be stored and warnings triggered, if upstream doesn’t match. This supply chain security of OpenWrt is a bigger undertaking, however I’m looking forward to working on that more in the future.