dash/contrib/guix/manifest.scm

198 lines
7.4 KiB
Scheme
Raw Normal View History

Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
(use-modules (gnu)
(gnu packages)
(gnu packages autotools)
(gnu packages base)
(gnu packages bash)
(gnu packages check)
(gnu packages commencement)
(gnu packages compression)
(gnu packages cross-base)
(gnu packages file)
(gnu packages gawk)
(gnu packages gcc)
Merge #17595: guix: Enable building for `x86_64-w64-mingw32` target a35e3235891d35daa167116cc70340140e883f06 guix: Appease travis. (Carl Dong) 0b66d22da5f53640e22f05adf880782c613e6d0f guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong) ba0b99bdd613ba7f17c6247ece3001e1b44759a3 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong) 93439a71eda49fb69f1e82966a23a946733aa6fa guix: Bump to upstream commit with mingw-w64 changes (Carl Dong) 35a96792dda9e78165b1598aeac7b2ab759e7be5 guix: Check mingw symbols, improve SSP fix docs (Carl Dong) 449d8fe25bbe25daacfc67aa89ca32b0a3254c5a guix: Expand on INT trap message (Carl Dong) 3f1f03c67a8e9edf487f08d272adb18b0a3942c8 guix: Spelling fixes (Carl Dong) ff821dd2a1c600488d11e7d9a20e9179ecc9144b guix: Reinstate make-ssp-fixed-gcc (Carl Dong) 360a9e0ad50a36ec79a1a160dbed3966689fd41c guix: Bump time-machine for mingw-w64 patches (Carl Dong) 93e41b7e3b54c17fd1b4c61ee95fc0dc2827e954 guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong) ef4f7e4c45c60a69406134122f091c77c6ef740f guix: Set the well-known timezone env var (Carl Dong) acf4b3b3b5accf60a19441a0298ef27001b78e72 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong) c4cce00eac691625b78b92f7dba0b7f57def19e5 guix: Remove dead links from README. (Carl Dong) df953a4c9a6143f45864757b706c88b6fa70545a guix: Appease shellcheck. (Carl Dong) 91897c95e191d293eb27d8af15cbeafc5b8f3895 guix: Improve guix-build.sh documentation (Carl Dong) 570d769c6c59b9f6d1a2b95b2ed60432cb33b3ba guix: Build support for Windows (Carl Dong) Pull request description: ~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~ Based on: #17933 (Time Machines are... shall we say... superior :grin:) This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon) ACKs for top commit: fanquake: ACK a35e3235891d35daa167116cc70340140e883f06 2/3 Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
2020-04-12 12:08:46 +02:00
(gnu packages installers)
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
(gnu packages linux)
Merge #17595: guix: Enable building for `x86_64-w64-mingw32` target a35e3235891d35daa167116cc70340140e883f06 guix: Appease travis. (Carl Dong) 0b66d22da5f53640e22f05adf880782c613e6d0f guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong) ba0b99bdd613ba7f17c6247ece3001e1b44759a3 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong) 93439a71eda49fb69f1e82966a23a946733aa6fa guix: Bump to upstream commit with mingw-w64 changes (Carl Dong) 35a96792dda9e78165b1598aeac7b2ab759e7be5 guix: Check mingw symbols, improve SSP fix docs (Carl Dong) 449d8fe25bbe25daacfc67aa89ca32b0a3254c5a guix: Expand on INT trap message (Carl Dong) 3f1f03c67a8e9edf487f08d272adb18b0a3942c8 guix: Spelling fixes (Carl Dong) ff821dd2a1c600488d11e7d9a20e9179ecc9144b guix: Reinstate make-ssp-fixed-gcc (Carl Dong) 360a9e0ad50a36ec79a1a160dbed3966689fd41c guix: Bump time-machine for mingw-w64 patches (Carl Dong) 93e41b7e3b54c17fd1b4c61ee95fc0dc2827e954 guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong) ef4f7e4c45c60a69406134122f091c77c6ef740f guix: Set the well-known timezone env var (Carl Dong) acf4b3b3b5accf60a19441a0298ef27001b78e72 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong) c4cce00eac691625b78b92f7dba0b7f57def19e5 guix: Remove dead links from README. (Carl Dong) df953a4c9a6143f45864757b706c88b6fa70545a guix: Appease shellcheck. (Carl Dong) 91897c95e191d293eb27d8af15cbeafc5b8f3895 guix: Improve guix-build.sh documentation (Carl Dong) 570d769c6c59b9f6d1a2b95b2ed60432cb33b3ba guix: Build support for Windows (Carl Dong) Pull request description: ~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~ Based on: #17933 (Time Machines are... shall we say... superior :grin:) This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon) ACKs for top commit: fanquake: ACK a35e3235891d35daa167116cc70340140e883f06 2/3 Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
2020-04-12 12:08:46 +02:00
(gnu packages mingw)
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
(gnu packages perl)
(gnu packages pkg-config)
(gnu packages python)
(gnu packages shells)
(gnu packages bison)
Merge #17595: guix: Enable building for `x86_64-w64-mingw32` target a35e3235891d35daa167116cc70340140e883f06 guix: Appease travis. (Carl Dong) 0b66d22da5f53640e22f05adf880782c613e6d0f guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong) ba0b99bdd613ba7f17c6247ece3001e1b44759a3 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong) 93439a71eda49fb69f1e82966a23a946733aa6fa guix: Bump to upstream commit with mingw-w64 changes (Carl Dong) 35a96792dda9e78165b1598aeac7b2ab759e7be5 guix: Check mingw symbols, improve SSP fix docs (Carl Dong) 449d8fe25bbe25daacfc67aa89ca32b0a3254c5a guix: Expand on INT trap message (Carl Dong) 3f1f03c67a8e9edf487f08d272adb18b0a3942c8 guix: Spelling fixes (Carl Dong) ff821dd2a1c600488d11e7d9a20e9179ecc9144b guix: Reinstate make-ssp-fixed-gcc (Carl Dong) 360a9e0ad50a36ec79a1a160dbed3966689fd41c guix: Bump time-machine for mingw-w64 patches (Carl Dong) 93e41b7e3b54c17fd1b4c61ee95fc0dc2827e954 guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong) ef4f7e4c45c60a69406134122f091c77c6ef740f guix: Set the well-known timezone env var (Carl Dong) acf4b3b3b5accf60a19441a0298ef27001b78e72 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong) c4cce00eac691625b78b92f7dba0b7f57def19e5 guix: Remove dead links from README. (Carl Dong) df953a4c9a6143f45864757b706c88b6fa70545a guix: Appease shellcheck. (Carl Dong) 91897c95e191d293eb27d8af15cbeafc5b8f3895 guix: Improve guix-build.sh documentation (Carl Dong) 570d769c6c59b9f6d1a2b95b2ed60432cb33b3ba guix: Build support for Windows (Carl Dong) Pull request description: ~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~ Based on: #17933 (Time Machines are... shall we say... superior :grin:) This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon) ACKs for top commit: fanquake: ACK a35e3235891d35daa167116cc70340140e883f06 2/3 Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
2020-04-12 12:08:46 +02:00
(guix build-system gnu)
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
(guix build-system trivial)
(guix gexp)
(guix packages)
(guix profiles)
(guix utils))
Merge #17595: guix: Enable building for `x86_64-w64-mingw32` target a35e3235891d35daa167116cc70340140e883f06 guix: Appease travis. (Carl Dong) 0b66d22da5f53640e22f05adf880782c613e6d0f guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong) ba0b99bdd613ba7f17c6247ece3001e1b44759a3 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong) 93439a71eda49fb69f1e82966a23a946733aa6fa guix: Bump to upstream commit with mingw-w64 changes (Carl Dong) 35a96792dda9e78165b1598aeac7b2ab759e7be5 guix: Check mingw symbols, improve SSP fix docs (Carl Dong) 449d8fe25bbe25daacfc67aa89ca32b0a3254c5a guix: Expand on INT trap message (Carl Dong) 3f1f03c67a8e9edf487f08d272adb18b0a3942c8 guix: Spelling fixes (Carl Dong) ff821dd2a1c600488d11e7d9a20e9179ecc9144b guix: Reinstate make-ssp-fixed-gcc (Carl Dong) 360a9e0ad50a36ec79a1a160dbed3966689fd41c guix: Bump time-machine for mingw-w64 patches (Carl Dong) 93e41b7e3b54c17fd1b4c61ee95fc0dc2827e954 guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong) ef4f7e4c45c60a69406134122f091c77c6ef740f guix: Set the well-known timezone env var (Carl Dong) acf4b3b3b5accf60a19441a0298ef27001b78e72 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong) c4cce00eac691625b78b92f7dba0b7f57def19e5 guix: Remove dead links from README. (Carl Dong) df953a4c9a6143f45864757b706c88b6fa70545a guix: Appease shellcheck. (Carl Dong) 91897c95e191d293eb27d8af15cbeafc5b8f3895 guix: Improve guix-build.sh documentation (Carl Dong) 570d769c6c59b9f6d1a2b95b2ed60432cb33b3ba guix: Build support for Windows (Carl Dong) Pull request description: ~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~ Based on: #17933 (Time Machines are... shall we say... superior :grin:) This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon) ACKs for top commit: fanquake: ACK a35e3235891d35daa167116cc70340140e883f06 2/3 Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
2020-04-12 12:08:46 +02:00
(define (make-ssp-fixed-gcc xgcc)
"Given a XGCC package, return a modified package that uses the SSP function
from glibc instead of from libssp.so. Our `symbol-check' script will complain if
we link against libssp.so, and thus will ensure that this works properly.
Taken from:
http://www.linuxfromscratch.org/hlfs/view/development/chapter05/gcc-pass1.html"
(package
(inherit xgcc)
(arguments
(substitute-keyword-arguments (package-arguments xgcc)
((#:make-flags flags)
`(cons "gcc_cv_libc_provides_ssp=yes" ,flags))))))
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
(define (make-gcc-rpath-link xgcc)
"Given a XGCC package, return a modified package that replace each instance of
-rpath in the default system spec that's inserted by Guix with -rpath-link"
(package
(inherit xgcc)
(arguments
(substitute-keyword-arguments (package-arguments xgcc)
((#:phases phases)
`(modify-phases ,phases
(add-after 'pre-configure 'replace-rpath-with-rpath-link
(lambda _
(substitute* (cons "gcc/config/rs6000/sysv4.h"
(find-files "gcc/config"
"^gnu-user.*\\.h$"))
(("-rpath=") "-rpath-link="))
#t))))))))
(define (make-cross-toolchain target
base-gcc-for-libc
base-kernel-headers
base-libc
base-gcc)
"Create a cross-compilation toolchain package for TARGET"
(let* ((xbinutils (cross-binutils target))
;; 1. Build a cross-compiling gcc without targeting any libc, derived
;; from BASE-GCC-FOR-LIBC
(xgcc-sans-libc (cross-gcc target
#:xgcc base-gcc-for-libc
#:xbinutils xbinutils))
;; 2. Build cross-compiled kernel headers with XGCC-SANS-LIBC, derived
;; from BASE-KERNEL-HEADERS
(xkernel (cross-kernel-headers target
base-kernel-headers
xgcc-sans-libc
xbinutils))
;; 3. Build a cross-compiled libc with XGCC-SANS-LIBC and XKERNEL,
;; derived from BASE-LIBC
(xlibc (cross-libc target
base-libc
xgcc-sans-libc
xbinutils
xkernel))
;; 4. Build a cross-compiling gcc targeting XLIBC, derived from
;; BASE-GCC
(xgcc (cross-gcc target
#:xgcc base-gcc
#:xbinutils xbinutils
#:libc xlibc)))
;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
;; XGCC
(package
(name (string-append target "-toolchain"))
(version (package-version xgcc))
(source #f)
(build-system trivial-build-system)
(arguments '(#:builder (begin (mkdir %output) #t)))
(propagated-inputs
`(("binutils" ,xbinutils)
("libc" ,xlibc)
("libc:static" ,xlibc "static")
("gcc" ,xgcc)))
(synopsis (string-append "Complete GCC tool chain for " target))
(description (string-append "This package provides a complete GCC tool
chain for " target " development."))
(home-page (package-home-page xgcc))
(license (package-license xgcc)))))
(define* (make-bitcoin-cross-toolchain target
#:key
(base-gcc-for-libc gcc-5)
(base-kernel-headers linux-libre-headers-4.19)
(base-libc glibc-2.27)
(base-gcc (make-gcc-rpath-link gcc-9)))
"Convenience wrapper around MAKE-CROSS-TOOLCHAIN with default values
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
desirable for building Bitcoin Core release binaries."
(make-cross-toolchain target
base-gcc-for-libc
base-kernel-headers
base-libc
base-gcc))
Merge #17595: guix: Enable building for `x86_64-w64-mingw32` target a35e3235891d35daa167116cc70340140e883f06 guix: Appease travis. (Carl Dong) 0b66d22da5f53640e22f05adf880782c613e6d0f guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong) ba0b99bdd613ba7f17c6247ece3001e1b44759a3 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong) 93439a71eda49fb69f1e82966a23a946733aa6fa guix: Bump to upstream commit with mingw-w64 changes (Carl Dong) 35a96792dda9e78165b1598aeac7b2ab759e7be5 guix: Check mingw symbols, improve SSP fix docs (Carl Dong) 449d8fe25bbe25daacfc67aa89ca32b0a3254c5a guix: Expand on INT trap message (Carl Dong) 3f1f03c67a8e9edf487f08d272adb18b0a3942c8 guix: Spelling fixes (Carl Dong) ff821dd2a1c600488d11e7d9a20e9179ecc9144b guix: Reinstate make-ssp-fixed-gcc (Carl Dong) 360a9e0ad50a36ec79a1a160dbed3966689fd41c guix: Bump time-machine for mingw-w64 patches (Carl Dong) 93e41b7e3b54c17fd1b4c61ee95fc0dc2827e954 guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong) ef4f7e4c45c60a69406134122f091c77c6ef740f guix: Set the well-known timezone env var (Carl Dong) acf4b3b3b5accf60a19441a0298ef27001b78e72 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong) c4cce00eac691625b78b92f7dba0b7f57def19e5 guix: Remove dead links from README. (Carl Dong) df953a4c9a6143f45864757b706c88b6fa70545a guix: Appease shellcheck. (Carl Dong) 91897c95e191d293eb27d8af15cbeafc5b8f3895 guix: Improve guix-build.sh documentation (Carl Dong) 570d769c6c59b9f6d1a2b95b2ed60432cb33b3ba guix: Build support for Windows (Carl Dong) Pull request description: ~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~ Based on: #17933 (Time Machines are... shall we say... superior :grin:) This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon) ACKs for top commit: fanquake: ACK a35e3235891d35daa167116cc70340140e883f06 2/3 Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
2020-04-12 12:08:46 +02:00
(define (make-gcc-with-pthreads gcc)
(package-with-extra-configure-variable gcc "--enable-threads" "posix"))
(define (make-mingw-pthreads-cross-toolchain target)
"Create a cross-compilation toolchain package for TARGET"
(let* ((xbinutils (cross-binutils target))
(pthreads-xlibc mingw-w64-x86_64-winpthreads)
(pthreads-xgcc (make-gcc-with-pthreads
(cross-gcc target
#:xgcc (make-ssp-fixed-gcc gcc-9)
#:xbinutils xbinutils
#:libc pthreads-xlibc))))
;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
;; XGCC
(package
(name (string-append target "-posix-toolchain"))
(version (package-version pthreads-xgcc))
(source #f)
(build-system trivial-build-system)
(arguments '(#:builder (begin (mkdir %output) #t)))
(propagated-inputs
`(("binutils" ,xbinutils)
("libc" ,pthreads-xlibc)
("gcc" ,pthreads-xgcc)))
(synopsis (string-append "Complete GCC tool chain for " target))
(description (string-append "This package provides a complete GCC tool
chain for " target " development."))
(home-page (package-home-page pthreads-xgcc))
(license (package-license pthreads-xgcc)))))
Merge #15277: contrib: Enable building in Guix containers 751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong) 8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011) [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210) [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f laanwj: ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
(packages->manifest
Merge #17595: guix: Enable building for `x86_64-w64-mingw32` target a35e3235891d35daa167116cc70340140e883f06 guix: Appease travis. (Carl Dong) 0b66d22da5f53640e22f05adf880782c613e6d0f guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong) ba0b99bdd613ba7f17c6247ece3001e1b44759a3 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong) 93439a71eda49fb69f1e82966a23a946733aa6fa guix: Bump to upstream commit with mingw-w64 changes (Carl Dong) 35a96792dda9e78165b1598aeac7b2ab759e7be5 guix: Check mingw symbols, improve SSP fix docs (Carl Dong) 449d8fe25bbe25daacfc67aa89ca32b0a3254c5a guix: Expand on INT trap message (Carl Dong) 3f1f03c67a8e9edf487f08d272adb18b0a3942c8 guix: Spelling fixes (Carl Dong) ff821dd2a1c600488d11e7d9a20e9179ecc9144b guix: Reinstate make-ssp-fixed-gcc (Carl Dong) 360a9e0ad50a36ec79a1a160dbed3966689fd41c guix: Bump time-machine for mingw-w64 patches (Carl Dong) 93e41b7e3b54c17fd1b4c61ee95fc0dc2827e954 guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong) ef4f7e4c45c60a69406134122f091c77c6ef740f guix: Set the well-known timezone env var (Carl Dong) acf4b3b3b5accf60a19441a0298ef27001b78e72 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong) c4cce00eac691625b78b92f7dba0b7f57def19e5 guix: Remove dead links from README. (Carl Dong) df953a4c9a6143f45864757b706c88b6fa70545a guix: Appease shellcheck. (Carl Dong) 91897c95e191d293eb27d8af15cbeafc5b8f3895 guix: Improve guix-build.sh documentation (Carl Dong) 570d769c6c59b9f6d1a2b95b2ed60432cb33b3ba guix: Build support for Windows (Carl Dong) Pull request description: ~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~ Based on: #17933 (Time Machines are... shall we say... superior :grin:) This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon) ACKs for top commit: fanquake: ACK a35e3235891d35daa167116cc70340140e883f06 2/3 Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
2020-04-12 12:08:46 +02:00
(append
(list ;; The Basics
bash-minimal
which
coreutils
util-linux
;; File(system) inspection
file
grep
diffutils
findutils
;; File transformation
patch
gawk
sed
;; Compression and archiving
tar
bzip2
gzip
xz
zlib
;; Build tools
gnu-make
libtool
autoconf
automake
pkg-config
bison
;; Scripting
perl
python-3.7
;; Native gcc 9 toolchain targeting glibc 2.27
(make-gcc-toolchain gcc-9 glibc-2.27))
(let ((target (getenv "HOST")))
(cond ((string-suffix? "-mingw32" target)
;; Windows
(list zip (make-mingw-pthreads-cross-toolchain "x86_64-w64-mingw32") nsis-x86_64))
((string-contains target "riscv64-linux-")
(list (make-bitcoin-cross-toolchain "riscv64-linux-gnu"
#:base-gcc-for-libc gcc-7)))
((string-contains target "-linux-")
(list (make-bitcoin-cross-toolchain target)))
(else '())))))