dash/contrib/guix/README.md

349 lines
13 KiB
Markdown
Raw Normal View History

Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
# Bootstrappable Bitcoin Core Builds
This directory contains the files necessary to perform bootstrappable Bitcoin
Core builds.
[Bootstrappability][b17e] furthers our binary security guarantees by allowing us
to _audit and reproduce_ our toolchain instead of blindly _trusting_ binary
downloads.
We achieve bootstrappability by using Guix as a functional package manager.
## Requirements
Conservatively, a x86_64 machine with:
- 16GB of free disk space on the partition that /gnu/store will reside in
- 8GB of free disk space per platform triple you're planning on building (see
the `HOSTS` environment variable description)
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
## Setup
### Installing Guix
If you're just testing this out, you can use the
[Dockerfile][fanquake/guix-docker] for convenience. It automatically speeds up
your builds by [using substitutes](#speeding-up-builds-with-substitute-servers).
If you don't want this behaviour, refer to the [next
section](#choosing-your-security-model).
Otherwise, follow the [Guix installation guide][guix/bin-install].
> Note: For those who like to keep their filesystems clean, Guix is designed to
> be very standalone and _will not_ conflict with your system's package
> manager/existing setup. It _only_ touches `/var/guix`, `/gnu`, and
> `~/.config/guix`.
### Choosing your security model
Guix allows us to achieve better binary security by using our CPU time to build
everything from scratch. However, it doesn't sacrifice user choice in pursuit of
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
this: users can decide whether or not to bootstrap and to use substitutes
(pre-built packages).
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
After installation, you may want to consider [adding substitute
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
servers](#speeding-up-builds-with-substitute-servers) from which to download
pre-built packages to speed up your build if that fits your security model (say,
if you're just testing that this works). Substitute servers are set up by
default if you're using the [Dockerfile][fanquake/guix-docker].
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
If you prefer not to use any substitutes, make sure to supply `--no-substitutes`
like in the following snippet. The first build will take a while, but the
resulting packages will be cached for future builds.
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
```sh
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes'
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
```
Likewise, to perform a bootstrapped build (takes even longer):
```sh
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes' ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap'
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
```
### Using a version of Guix with `guix time-machine` capabilities
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
> Note: This entire section can be skipped if you are already using a version of
> Guix that has [the `guix time-machine` command][guix/time-machine].
Once Guix is installed, if it doesn't have the `guix time-machine` command, pull
the latest `guix`.
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
```sh
guix pull --max-jobs=4 # change number of jobs accordingly
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
```
Make sure that you are using your current profile. (You are prompted to do this
at the end of the `guix pull`)
```bash
export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
```
Merge #21375: guix: Misc feedback-based fixes + hier restructuring 7476b46f1893a4858616d2a8456a7c43238851ed guix: Build dmg as a static binary (Carl Dong) 06d6cf6784421290e6235fe8684d5e08ed6f1b62 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong) 65176ab5730dff34466caaecdd292625ef8294fc guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong) ca85679eb43b8375a95d82101977829d08fb1e1b guix: Use clang-toolchain instead of clang (Carl Dong) 1aec0eda8fd31a57b0621eea616398017c2ead98 guix: Fallback to local build for substitute-enabled Guix users (Carl Dong) 1742f8e12d163852df09575e03edcd3db73198ee guix: Add early health check for guix-daemon (Carl Dong) c1ae726a13ecfa5e7e9fdc3030a8110b8bb263f8 guix: More thoroughly control native toolchain (Carl Dong) 39741128d3775d198dbee34dc827353bfd18acd8 guix: Supply --link-profile (Carl Dong) d55a1056ee565afed64e42d6f6efb6b0adc5599b guix: Add troubleshooting documentation entries (Carl Dong) 7f401c953f8bb3574cec48561e13ef3b47dedc6e guix: Adapt guix-build to prelude, restructure hier (Carl Dong) 4eccf063b252bfe256cf72d363a24cf0183e926e guix: Remove guix-build.sh filename extension (Carl Dong) 7753357a7bae98ec775c707b9dec4cea1e945802 guix: Add source-able bash prelude and utils (Carl Dong) e5b49a01f5d0f631e7f08f86ca8a2c2b8213319f guix: Create windeploy inside distsrc-* (Carl Dong) 3e9982ab3877eb8fe0a8c0cb3d847ac0913c7336 contrib: Silence git-describe when looking for tag (Carl Dong) d5a71e97853ea9e1b879e8c76bfb01d4bef33172 guix: Use --cores instead of --max-jobs (Carl Dong) Pull request description: This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts: - Resolves confusion between `--cores=` and `--max-jobs=` - `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var - `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that - `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people) - Add troubleshooting documentation to `README.md` - Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon` - Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally - `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple` A few robustness changes are also included: - We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile` - We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug - We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages. Finally, we restructure the guix building hierarchy such that it looks something like: ``` guix-build-<short-hash-or-version-tag> ├── distsrc-<short-hash-or-version-tag>-${HOST} │ ├── contrib │ ├── depends │ ├── src │ └── ... ├── distsrc-<short-hash-or-version-tag>-... └── output ├── dist-archive │ └── bitcoin-<short-hash-or-version-tag>.tar.gz ├── *-linux-* │ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz │ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz ├── x86_64-apple-darwin18 │ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz │ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg │ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz └── x86_64-w64-mingw32 ├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip ├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe ├── bitcoin-<short-hash-or-version-tag>-win64.zip └── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz ``` Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts. ACKs for top commit: sipa: ACK 7476b46f1893a4858616d2a8456a7c43238851ed laanwj: ACK 7476b46f1893a4858616d2a8456a7c43238851ed Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
2021-04-06 00:59:11 +02:00
### Controlling the number of threads used by `guix` build commands
By default, the scripts under `./contrib/guix` will invoke all `guix` build
commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
specified. However, astute manual readers will also notice that there is a
`--max-jobs=` flag (which defaults to 1 if unspecified).
Here is the difference between `--cores=` and `--max-jobs=`:
> Note: When I say "derivation," think "package"
`--cores=`
- controls the number of CPU cores to build each derivation. This is the value
passed to `make`'s `--jobs=` flag.
`--max-jobs=`
- controls how many derivations can be built in parallel
- defaults to 1
Therefore, the default is for `guix` build commands to build one derivation at a
time, utilizing `$JOBS` threads.
Specifying the `$JOBS` environment variable will only modify `--cores=`, but you
can also modify the value for `--max-jobs=` by specifying
`$ADDITIONAL_GUIX_COMMON_FLAGS`. For example, if you have a LOT of memory, you
may want to set:
```sh
export ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
```
Which allows for a maximum of 8 derivations to be built at the same time, each
utilizing `$JOBS` threads.
Or, if you'd like to avoid spurious build failures caused by issues with
parallelism within a single package, but would still like to build multiple
packages when the dependency graph allows for it, you may want to try:
```sh
export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
```
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
## Usage
### As a Tool for Deterministic Builds
From the top of a clean Bitcoin Core repository:
```sh
./contrib/guix/guix-build
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
```
After the build finishes successfully (check the status code please), compare
hashes:
```sh
find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
```
#### Recognized environment variables
* _**HOSTS**_
Override the space-separated list of platform triples for which to perform a
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
bootstrappable build. _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf
aarch64-linux-gnu riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
x86\_64-w64-mingw32 x86\_64-apple-darwin18")_
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
* _**SOURCES_PATH**_
Set the depends tree download cache for sources. This is passed through to the
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary redownloading of package sources.
* _**BASE_CACHE**_
Set the depends tree cache for built packages. This is passed through to the
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary building of packages.
* _**SDK_PATH**_
Set the path where _extracted_ SDKs can be found. This is passed through to
the depends tree. Note that this is should be set to the _parent_ directory of
the actual SDK (e.g. SDK_PATH=$HOME/Downloads/macOS-SDKs instead of
$HOME/Downloads/macOS-SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers).
Merge #21375: guix: Misc feedback-based fixes + hier restructuring 7476b46f1893a4858616d2a8456a7c43238851ed guix: Build dmg as a static binary (Carl Dong) 06d6cf6784421290e6235fe8684d5e08ed6f1b62 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong) 65176ab5730dff34466caaecdd292625ef8294fc guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong) ca85679eb43b8375a95d82101977829d08fb1e1b guix: Use clang-toolchain instead of clang (Carl Dong) 1aec0eda8fd31a57b0621eea616398017c2ead98 guix: Fallback to local build for substitute-enabled Guix users (Carl Dong) 1742f8e12d163852df09575e03edcd3db73198ee guix: Add early health check for guix-daemon (Carl Dong) c1ae726a13ecfa5e7e9fdc3030a8110b8bb263f8 guix: More thoroughly control native toolchain (Carl Dong) 39741128d3775d198dbee34dc827353bfd18acd8 guix: Supply --link-profile (Carl Dong) d55a1056ee565afed64e42d6f6efb6b0adc5599b guix: Add troubleshooting documentation entries (Carl Dong) 7f401c953f8bb3574cec48561e13ef3b47dedc6e guix: Adapt guix-build to prelude, restructure hier (Carl Dong) 4eccf063b252bfe256cf72d363a24cf0183e926e guix: Remove guix-build.sh filename extension (Carl Dong) 7753357a7bae98ec775c707b9dec4cea1e945802 guix: Add source-able bash prelude and utils (Carl Dong) e5b49a01f5d0f631e7f08f86ca8a2c2b8213319f guix: Create windeploy inside distsrc-* (Carl Dong) 3e9982ab3877eb8fe0a8c0cb3d847ac0913c7336 contrib: Silence git-describe when looking for tag (Carl Dong) d5a71e97853ea9e1b879e8c76bfb01d4bef33172 guix: Use --cores instead of --max-jobs (Carl Dong) Pull request description: This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts: - Resolves confusion between `--cores=` and `--max-jobs=` - `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var - `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that - `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people) - Add troubleshooting documentation to `README.md` - Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon` - Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally - `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple` A few robustness changes are also included: - We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile` - We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug - We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages. Finally, we restructure the guix building hierarchy such that it looks something like: ``` guix-build-<short-hash-or-version-tag> ├── distsrc-<short-hash-or-version-tag>-${HOST} │ ├── contrib │ ├── depends │ ├── src │ └── ... ├── distsrc-<short-hash-or-version-tag>-... └── output ├── dist-archive │ └── bitcoin-<short-hash-or-version-tag>.tar.gz ├── *-linux-* │ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz │ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz ├── x86_64-apple-darwin18 │ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz │ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg │ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz └── x86_64-w64-mingw32 ├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip ├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe ├── bitcoin-<short-hash-or-version-tag>-win64.zip └── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz ``` Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts. ACKs for top commit: sipa: ACK 7476b46f1893a4858616d2a8456a7c43238851ed laanwj: ACK 7476b46f1893a4858616d2a8456a7c43238851ed Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
2021-04-06 00:59:11 +02:00
* _**JOBS**_
Override the number of jobs to run simultaneously, you might want to do so on
a memory-limited machine. This may be passed to:
- `guix` build commands as in `guix environment --cores="$JOBS"`
- `make` as in `make --jobs="$JOBS"`
- `xargs` as in `xargs -P"$JOBS"`
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
Merge #21375: guix: Misc feedback-based fixes + hier restructuring 7476b46f1893a4858616d2a8456a7c43238851ed guix: Build dmg as a static binary (Carl Dong) 06d6cf6784421290e6235fe8684d5e08ed6f1b62 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong) 65176ab5730dff34466caaecdd292625ef8294fc guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong) ca85679eb43b8375a95d82101977829d08fb1e1b guix: Use clang-toolchain instead of clang (Carl Dong) 1aec0eda8fd31a57b0621eea616398017c2ead98 guix: Fallback to local build for substitute-enabled Guix users (Carl Dong) 1742f8e12d163852df09575e03edcd3db73198ee guix: Add early health check for guix-daemon (Carl Dong) c1ae726a13ecfa5e7e9fdc3030a8110b8bb263f8 guix: More thoroughly control native toolchain (Carl Dong) 39741128d3775d198dbee34dc827353bfd18acd8 guix: Supply --link-profile (Carl Dong) d55a1056ee565afed64e42d6f6efb6b0adc5599b guix: Add troubleshooting documentation entries (Carl Dong) 7f401c953f8bb3574cec48561e13ef3b47dedc6e guix: Adapt guix-build to prelude, restructure hier (Carl Dong) 4eccf063b252bfe256cf72d363a24cf0183e926e guix: Remove guix-build.sh filename extension (Carl Dong) 7753357a7bae98ec775c707b9dec4cea1e945802 guix: Add source-able bash prelude and utils (Carl Dong) e5b49a01f5d0f631e7f08f86ca8a2c2b8213319f guix: Create windeploy inside distsrc-* (Carl Dong) 3e9982ab3877eb8fe0a8c0cb3d847ac0913c7336 contrib: Silence git-describe when looking for tag (Carl Dong) d5a71e97853ea9e1b879e8c76bfb01d4bef33172 guix: Use --cores instead of --max-jobs (Carl Dong) Pull request description: This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts: - Resolves confusion between `--cores=` and `--max-jobs=` - `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var - `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that - `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people) - Add troubleshooting documentation to `README.md` - Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon` - Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally - `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple` A few robustness changes are also included: - We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile` - We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug - We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages. Finally, we restructure the guix building hierarchy such that it looks something like: ``` guix-build-<short-hash-or-version-tag> ├── distsrc-<short-hash-or-version-tag>-${HOST} │ ├── contrib │ ├── depends │ ├── src │ └── ... ├── distsrc-<short-hash-or-version-tag>-... └── output ├── dist-archive │ └── bitcoin-<short-hash-or-version-tag>.tar.gz ├── *-linux-* │ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz │ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz ├── x86_64-apple-darwin18 │ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz │ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg │ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz └── x86_64-w64-mingw32 ├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip ├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe ├── bitcoin-<short-hash-or-version-tag>-win64.zip └── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz ``` Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts. ACKs for top commit: sipa: ACK 7476b46f1893a4858616d2a8456a7c43238851ed laanwj: ACK 7476b46f1893a4858616d2a8456a7c43238851ed Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
2021-04-06 00:59:11 +02:00
_(defaults to the value of `nproc` outside the container)_
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
* _**SOURCE_DATE_EPOCH**_
Override the reference UNIX timestamp used for bit-for-bit reproducibility,
the variable name conforms to [standard][r12e/source-date-epoch]. _(defaults
to the output of `$(git log --format=%at -1)`)_
* _**V**_
If non-empty, will pass `V=1` to all `make` invocations, making `make` output
verbose.
Note that any given value is ignored. The variable is only checked for
emptiness. More concretely, this means that `V=` (setting `V` to the empty
string) is interpreted the same way as not setting `V` at all, and that `V=0`
has the same effect as `V=1`.
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
* _**SUBSTITUTE_URLS**_
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
A whitespace-delimited list of URLs from which to download pre-built packages.
A URL is only used if its signing key is authorized (refer to the [substitute
servers section](#speeding-up-builds-with-substitute-servers) for more
details).
* _**ADDITIONAL_GUIX_COMMON_FLAGS**_
Additional flags to be passed to all `guix` commands. For a fully-bootstrapped
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
build, set this to `--bootstrap --no-substitutes` (refer to the [security
model section](#choosing-your-security-model) for more details). Note that a
fully-bootstrapped build will take quite a long time on the first run.
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
* _**ADDITIONAL_GUIX_TIMEMACHINE_FLAGS**_
Additional flags to be passed to `guix time-machine`.
* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
Additional flags to be passed to the invocation of `guix environment` inside
`guix time-machine`.
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
## Tips and Tricks
### Speeding up builds with substitute servers
_This whole section is automatically done in the convenience
[Dockerfiles][fanquake/guix-docker]_
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
For those who are used to life in the fast _(and trustful)_ lane, you can
specify [substitute servers][guix/substitutes] from which to download pre-built
packages.
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
> For those who only want to use substitutes from the official Guix build farm
> and have authorized the build farm's signing key during Guix's installation,
> you don't need to do anything.
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
#### Step 1: Authorize the signing keys
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
For the official Guix build farm at https://ci.guix.gnu.org, run as root:
```
guix archive --authorize < ~root/.config/guix/current/share/guix/ci.guix.gnu.org.pub
```
For dongcarl's substitute server at https://guix.carldong.io, run as root:
```sh
wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
```
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
#### Step 2: Specify the substitute servers
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
The official Guix build farm at https://ci.guix.gnu.org is automatically used
unless the `--no-substitutes` flag is supplied.
This can be overridden for all `guix` invocations by passing the
`--substitute-urls` option to your invocation of `guix-daemon`. This can also be
overridden on a call-by-call basis by passing the same `--substitute-urls`
option to client tools such at `guix environment`.
To use dongcarl's substitute server for Bitcoin Core builds after having
[authorized his signing key](#authorize-the-signing-keys):
```
Merge #20619: guix: Quality of life improvements 570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong) 2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong) 0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong) d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong) 57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong) 38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong) 036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong) 34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong) 402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong) cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong) 93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong) 0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong) 444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong) Pull request description: After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them. Notable changes: 1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment 2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules 3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process. 4. README cleanup ACKs for top commit: fanquake: ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward. Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
2021-01-12 11:52:42 +01:00
export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org'
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
```
Merge #21375: guix: Misc feedback-based fixes + hier restructuring 7476b46f1893a4858616d2a8456a7c43238851ed guix: Build dmg as a static binary (Carl Dong) 06d6cf6784421290e6235fe8684d5e08ed6f1b62 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong) 65176ab5730dff34466caaecdd292625ef8294fc guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong) ca85679eb43b8375a95d82101977829d08fb1e1b guix: Use clang-toolchain instead of clang (Carl Dong) 1aec0eda8fd31a57b0621eea616398017c2ead98 guix: Fallback to local build for substitute-enabled Guix users (Carl Dong) 1742f8e12d163852df09575e03edcd3db73198ee guix: Add early health check for guix-daemon (Carl Dong) c1ae726a13ecfa5e7e9fdc3030a8110b8bb263f8 guix: More thoroughly control native toolchain (Carl Dong) 39741128d3775d198dbee34dc827353bfd18acd8 guix: Supply --link-profile (Carl Dong) d55a1056ee565afed64e42d6f6efb6b0adc5599b guix: Add troubleshooting documentation entries (Carl Dong) 7f401c953f8bb3574cec48561e13ef3b47dedc6e guix: Adapt guix-build to prelude, restructure hier (Carl Dong) 4eccf063b252bfe256cf72d363a24cf0183e926e guix: Remove guix-build.sh filename extension (Carl Dong) 7753357a7bae98ec775c707b9dec4cea1e945802 guix: Add source-able bash prelude and utils (Carl Dong) e5b49a01f5d0f631e7f08f86ca8a2c2b8213319f guix: Create windeploy inside distsrc-* (Carl Dong) 3e9982ab3877eb8fe0a8c0cb3d847ac0913c7336 contrib: Silence git-describe when looking for tag (Carl Dong) d5a71e97853ea9e1b879e8c76bfb01d4bef33172 guix: Use --cores instead of --max-jobs (Carl Dong) Pull request description: This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts: - Resolves confusion between `--cores=` and `--max-jobs=` - `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var - `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that - `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people) - Add troubleshooting documentation to `README.md` - Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon` - Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally - `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple` A few robustness changes are also included: - We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile` - We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug - We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages. Finally, we restructure the guix building hierarchy such that it looks something like: ``` guix-build-<short-hash-or-version-tag> ├── distsrc-<short-hash-or-version-tag>-${HOST} │ ├── contrib │ ├── depends │ ├── src │ └── ... ├── distsrc-<short-hash-or-version-tag>-... └── output ├── dist-archive │ └── bitcoin-<short-hash-or-version-tag>.tar.gz ├── *-linux-* │ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz │ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz ├── x86_64-apple-darwin18 │ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz │ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg │ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz └── x86_64-w64-mingw32 ├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip ├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe ├── bitcoin-<short-hash-or-version-tag>-win64.zip └── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz ``` Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts. ACKs for top commit: sipa: ACK 7476b46f1893a4858616d2a8456a7c43238851ed laanwj: ACK 7476b46f1893a4858616d2a8456a7c43238851ed Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
2021-04-06 00:59:11 +02:00
## Troubleshooting
### Derivation failed to build
When you see a build failure like below:
```
building /gnu/store/...-foo-3.6.12.drv...
/ 'check' phasenote: keeping build directory `/tmp/guix-build-foo-3.6.12.drv-0'
builder for `/gnu/store/...-foo-3.6.12.drv' failed with exit code 1
build of /gnu/store/...-foo-3.6.12.drv failed
View build log at '/var/log/guix/drvs/../...-foo-3.6.12.drv.bz2'.
cannot build derivation `/gnu/store/...-qux-7.69.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-bar-3.16.5.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-baz-2.0.5.drv': 1 dependencies couldn't be built
guix time-machine: error: build of `/gnu/store/...-baz-2.0.5.drv' failed
```
It means that `guix` failed to build a package named `foo`, which was a
dependency of `qux`, `bar`, and `baz`. Importantly, note that the last "failed"
line is not necessarily the root cause, the first "failed" line is.
Most of the time, the build failure is due to a spurious test failure or the
package's build system/test suite breaking when running multi-threaded. To
rebuild _just_ this derivation in a single-threaded fashion:
```sh
$ guix build --cores=1 /gnu/store/...-foo-3.6.12.drv
```
If the single-threaded rebuild did not succeed, you may need to dig deeper.
Merge #21375: guix: Misc feedback-based fixes + hier restructuring 7476b46f1893a4858616d2a8456a7c43238851ed guix: Build dmg as a static binary (Carl Dong) 06d6cf6784421290e6235fe8684d5e08ed6f1b62 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong) 65176ab5730dff34466caaecdd292625ef8294fc guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong) ca85679eb43b8375a95d82101977829d08fb1e1b guix: Use clang-toolchain instead of clang (Carl Dong) 1aec0eda8fd31a57b0621eea616398017c2ead98 guix: Fallback to local build for substitute-enabled Guix users (Carl Dong) 1742f8e12d163852df09575e03edcd3db73198ee guix: Add early health check for guix-daemon (Carl Dong) c1ae726a13ecfa5e7e9fdc3030a8110b8bb263f8 guix: More thoroughly control native toolchain (Carl Dong) 39741128d3775d198dbee34dc827353bfd18acd8 guix: Supply --link-profile (Carl Dong) d55a1056ee565afed64e42d6f6efb6b0adc5599b guix: Add troubleshooting documentation entries (Carl Dong) 7f401c953f8bb3574cec48561e13ef3b47dedc6e guix: Adapt guix-build to prelude, restructure hier (Carl Dong) 4eccf063b252bfe256cf72d363a24cf0183e926e guix: Remove guix-build.sh filename extension (Carl Dong) 7753357a7bae98ec775c707b9dec4cea1e945802 guix: Add source-able bash prelude and utils (Carl Dong) e5b49a01f5d0f631e7f08f86ca8a2c2b8213319f guix: Create windeploy inside distsrc-* (Carl Dong) 3e9982ab3877eb8fe0a8c0cb3d847ac0913c7336 contrib: Silence git-describe when looking for tag (Carl Dong) d5a71e97853ea9e1b879e8c76bfb01d4bef33172 guix: Use --cores instead of --max-jobs (Carl Dong) Pull request description: This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts: - Resolves confusion between `--cores=` and `--max-jobs=` - `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var - `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that - `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people) - Add troubleshooting documentation to `README.md` - Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon` - Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally - `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple` A few robustness changes are also included: - We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile` - We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug - We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages. Finally, we restructure the guix building hierarchy such that it looks something like: ``` guix-build-<short-hash-or-version-tag> ├── distsrc-<short-hash-or-version-tag>-${HOST} │ ├── contrib │ ├── depends │ ├── src │ └── ... ├── distsrc-<short-hash-or-version-tag>-... └── output ├── dist-archive │ └── bitcoin-<short-hash-or-version-tag>.tar.gz ├── *-linux-* │ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz │ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz ├── x86_64-apple-darwin18 │ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz │ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg │ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz └── x86_64-w64-mingw32 ├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip ├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe ├── bitcoin-<short-hash-or-version-tag>-win64.zip └── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz ``` Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts. ACKs for top commit: sipa: ACK 7476b46f1893a4858616d2a8456a7c43238851ed laanwj: ACK 7476b46f1893a4858616d2a8456a7c43238851ed Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
2021-04-06 00:59:11 +02:00
You may view `foo`'s build logs in `less` like so (please replace paths with the
path you see in the build failure output):
```sh
$ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
```
`foo`'s build directory is also preserved and available at
`/tmp/guix-build-foo-3.6.12.drv-0`. However, if you fail to build `foo` multiple
times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
failure output for the most accurate, up-to-date information.
#### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
which rejects characters not present in the UTF-8 character code set. An example
is ZFS with the utf8only=on option set.
More information: https://bugs.python.org/issue37584
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
## FAQ
### How can I trust the binary installation?
As mentioned at the bottom of [this manual page][guix/bin-install]:
> The binary installation tarballs can be (re)produced and verified simply by
> running the following command in the Guix source tree:
>
> make guix-binary.x86_64-linux.tar.xz
### Is Guix packaged in my operating system?
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
Guix is shipped starting with [Debian Bullseye][debian/guix-bullseye] and
[Ubuntu 21.04 "Hirsute Hippo"][ubuntu/guix-hirsute]. Other operating systems
are working on packaging Guix as well.
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
[b17e]: http://bootstrappable.org/
[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
[guix/install.sh]: https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
[guix/bin-install]: https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
[guix/env-setup]: https://www.gnu.org/software/guix/manual/en/html_node/Build-Environment-Setup.html
[guix/substitutes]: https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html
[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html
[guix/time-machine]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
[debian/guix-bullseye]: https://packages.debian.org/bullseye/guix
[ubuntu/guix-hirsute]: https://packages.ubuntu.com/hirsute/guix
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix