Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( use-modules ( gnu )
( gnu packages )
( gnu packages autotools )
( gnu packages base )
( gnu packages bash )
2021-05-24 15:31:05 +02:00
( gnu packages certs )
2021-01-22 04:32:24 +01:00
( gnu packages cdrom )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( gnu packages check )
2021-01-22 04:32:24 +01:00
( gnu packages cmake )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( gnu packages commencement )
( gnu packages compression )
( gnu packages cross-base )
2021-05-24 15:31:05 +02:00
( gnu packages curl )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( gnu packages file )
( gnu packages gawk )
( gnu packages gcc )
2021-01-22 04:32:24 +01:00
( gnu packages gnome )
( gnu packages image )
( gnu packages imagemagick )
2020-04-12 12:08:46 +02:00
( gnu packages installers )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( gnu packages linux )
2021-01-22 04:32:24 +01:00
( gnu packages llvm )
2020-04-12 12:08:46 +02:00
( gnu packages mingw )
Merge bitcoin/bitcoin#22182: guix: Overhaul how guix-{attest,verify} works and hierarchy
e2c40a4ed5272d72fea997bd936fba28bb753226 guix-attest: Error out if SHA256SUMS is unexpected (Carl Dong)
4cc35daed557f38b080360a89036b2e97a6f78c2 Rewrite guix-{attest,verify} for new hier (Carl Dong)
28a9c9b83924f585b397f0f3b8e9e73780ac0ad6 Make SHA256SUMS fragment right after build (Carl Dong)
Pull request description:
Based on: #22075
Code reviewers: I recommend reading the new `guix-{attest,verify}` files instead of trying to read the diff
The following changes resolve many usability improvements which were pointed out to me:
1. Some maintainers like to extract their "uncodesigned tarball" inside the `output/` directory, resulting in the older `guix-attest` mistakenly attesting to the extracted contents
2. Maintainers whose GPG keys reside on an external smartcard often need to physically interact with the smartcard as a way to approve the signing operation, having one signature per platform means a lot of fidgeting
3. Maintainers wishing to sign on a separate machine now has the option of transferring only a subtree of `output/`, namely `output/*/SHA256SUMS.part`, in order to perform a signature (you may need to specify an `$OUTDIR_BASE` env var)
4. An `all.SHA256SUMS` file should be usable as the base `SHA256SUMS` in bitcoin core torrents and on the release server.
For those who sign on an separate machine than the one you do builds on, the following steps will work:
1. `env GUIX_SIGS_REPO=/home/achow101/guix.sigs SIGNER=achow101 NO_SIGN=1 ./contrib/guix/guix-attest`
2. Copy `/home/achow101/guix.sigs/<tag>/achow101` (which does not yet have signatures) to signing machine
3. Sign the `SHA256SUMS` files:
```bash
for i in "<path-to-achow101>/*.SHA256SUMS"; do
gpg --detach-sign --local-user "<your-key-here>" --armor --output "$i"{.asc,}
done
```
5. Upload `<path-to-achow101>` (now with signatures) to `guix.sigs`
-----
After this change, output directories will now include a `SHA256SUMS.part` fragment, created immediately after a successful build:
```
output
└── x86_64-w64-mingw32
├── bitcoin-4e069f7589da-win64-debug.zip
├── bitcoin-4e069f7589da-win64-setup-unsigned.exe
├── bitcoin-4e069f7589da-win64.zip
├── bitcoin-4e069f7589da-win-unsigned.tar.gz
└── SHA256SUMS.part
```
These `SHA256SUMS.part` fragments look something like:
```
3ebd7262b1a0a5bb757fef1f70e7e14033c70f98c059bc4dbfee5d1992b25825 dist-archive/bitcoin-4e069f7589da.tar.gz
def2e7d3de5ab3e3f955344e75151df4f33713f9101f5295bd13c9375bdf633b x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-debug.zip
643049fe3ee4a4e83a1739607e67b11b7c9b1a66208a6f35a9ff634ba795500e x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-setup-unsigned.exe
a247a1ccec0ccc2e138c648284bd01f6a761f2d8d6d07d91b5b4a6670ec3f288 x86_64-w64-mingw32/bitcoin-4e069f7589da-win-unsigned.tar.gz
fab76a836dcc592e39c04fd2396696633fb6eb56e39ecbf6c909bd173ed4280c x86_64-w64-mingw32/bitcoin-4e069f7589da-win64.zip
```
Meaning that they are valid `SHA256SUMS` files when `sha256sum --check`'d at the `guix-build-*/output` directory level
When `guix-attest` is invoked, these `SHA256SUMS.part` files are combined and sorted (by `-k2`, `LC_ALL=C`) to create:
1. `noncodesigned.SHA256SUMS` for a manifest of all non-codesigned outputs, and
3. `all.SHA256SUMS` for a manifest of all outputs including non-codesigned outputs
Then both files are signed, resulting in the following `guix.sigs` hierarchy:
```
4e069f7589da/
└── dongcarl
├── all.SHA256SUMS
├── all.SHA256SUMS.asc
├── noncodesigned.SHA256SUMS
└── noncodesigned.SHA256SUMS.asc
```
ACKs for top commit:
achow101:
ACK e2c40a4ed5272d72fea997bd936fba28bb753226
hebasto:
ACK e2c40a4ed5272d72fea997bd936fba28bb753226, tested on Linux Mint 20.1 (x86_64) with and w/o `NO_SIGN=1`. Changes in `contrib/guix/libexec/codesign.sh` and `contrib/guix/guix-verify` are reviewed only.
Tree-SHA512: 618aacefb0eb6595735a9ab6a98ea6598fce65f9ccf33fa1e7ef93bf140c0f6cfc16e34870c6aa3e4777dd3f004b92a82a994141879870141742df948ec59c1f
2021-06-17 07:10:29 +02:00
( gnu packages moreutils )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( gnu packages perl )
( gnu packages pkg-config )
( gnu packages python )
2021-05-24 15:31:05 +02:00
( gnu packages python-web )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( gnu packages shells )
( gnu packages bison )
2021-05-24 15:31:05 +02:00
( gnu packages tls )
2020-05-06 06:51:18 +02:00
( gnu packages version-control )
2021-01-22 04:32:24 +01:00
( guix build-system font )
2020-04-12 12:08:46 +02:00
( guix build-system gnu )
2021-05-24 15:31:05 +02:00
( guix build-system python )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( guix build-system trivial )
2021-01-22 04:32:24 +01:00
( guix download )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( guix gexp )
2021-05-24 15:31:05 +02:00
( guix git-download )
2021-01-22 04:32:24 +01:00
( ( guix licenses ) # :prefix license: )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( guix packages )
( guix profiles )
( guix utils ) )
2021-01-20 00:43:38 +01:00
( define-syntax-rule ( search-our-patches file-name . . . )
" Return the list of absolute file names corresponding to each
FILE-NAME found in . /patches relative to the current file . "
( parameterize
( ( %patch-path ( list ( string-append ( dirname ( current-filename ) ) "/patches" ) ) ) )
( list ( search-patch file-name ) . . . ) ) )
2020-04-12 12:08:46 +02:00
( define ( make-ssp-fixed-gcc xgcc )
" Given a XGCC package, return a modified package that uses the SSP function
from glibc instead of from libssp . so . Our ` symbol-check ' script will complain if
we link against libssp . so, and thus will ensure that this works properly .
Taken from:
http://www . linuxfromscratch . org/hlfs/view/development/chapter05/gcc-pass1 . html "
( package
2021-05-24 15:31:05 +02:00
( inherit xgcc )
( arguments
( substitute-keyword-arguments ( package-arguments xgcc )
( ( # :make-flags flags )
` ( cons "gcc_cv_libc_provides_ssp=yes" , flags ) ) ) ) ) )
2020-04-12 12:08:46 +02:00
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( define ( make-gcc-rpath-link xgcc )
" Given a XGCC package, return a modified package that replace each instance of
-rpath in the default system spec that 's inserted by Guix with -rpath-link "
( package
2021-05-24 15:31:05 +02:00
( inherit xgcc )
( arguments
( substitute-keyword-arguments ( package-arguments xgcc )
( ( # :phases phases )
` ( modify-phases , phases
( add-after 'pre-configure 'replace-rpath-with-rpath-link
( lambda _
( substitute* ( cons "gcc/config/rs6000/sysv4.h"
( find-files "gcc/config"
"^gnu-user.*\\.h$" ) )
( ( "-rpath=" ) "-rpath-link=" ) )
#t ) ) ) ) ) ) ) )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( define ( make-cross-toolchain target
base-gcc-for-libc
base-kernel-headers
base-libc
base-gcc )
"Create a cross-compilation toolchain package for TARGET"
( let* ( ( xbinutils ( cross-binutils target ) )
;; 1. Build a cross-compiling gcc without targeting any libc, derived
;; from BASE-GCC-FOR-LIBC
( xgcc-sans-libc ( cross-gcc target
# :xgcc base-gcc-for-libc
# :xbinutils xbinutils ) )
;; 2. Build cross-compiled kernel headers with XGCC-SANS-LIBC, derived
;; from BASE-KERNEL-HEADERS
( xkernel ( cross-kernel-headers target
base-kernel-headers
xgcc-sans-libc
xbinutils ) )
;; 3. Build a cross-compiled libc with XGCC-SANS-LIBC and XKERNEL,
;; derived from BASE-LIBC
( xlibc ( cross-libc target
base-libc
xgcc-sans-libc
xbinutils
xkernel ) )
;; 4. Build a cross-compiling gcc targeting XLIBC, derived from
;; BASE-GCC
( xgcc ( cross-gcc target
# :xgcc base-gcc
# :xbinutils xbinutils
# :libc xlibc ) ) )
;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
;; XGCC
( package
( name ( string-append target "-toolchain" ) )
( version ( package-version xgcc ) )
( source #f )
( build-system trivial-build-system )
( arguments ' ( # :builder ( begin ( mkdir %output ) #t ) ) )
( propagated-inputs
` ( ( "binutils" , xbinutils )
( "libc" , xlibc )
( "libc:static" , xlibc "static" )
2021-02-19 15:16:02 +01:00
( "gcc" , xgcc )
( "gcc-lib" , xgcc "lib" ) ) )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( synopsis ( string-append "Complete GCC tool chain for " target ) )
( description ( string-append " This package provides a complete GCC tool
chain for " target " development . " ) )
( home-page ( package-home-page xgcc ) )
( license ( package-license xgcc ) ) ) ) )
2021-05-24 15:31:05 +02:00
( define base-gcc
( package-with-extra-patches gcc-8
( search-our-patches "gcc-8-sort-libtool-find-output.patch" ) ) )
2021-07-05 16:40:08 +02:00
;; Building glibc with stack smashing protector first landed in glibc 2.25, use
;; this function to disable for older glibcs
;;
;; From glibc 2.25 changelog:
;;
;; * Most of glibc can now be built with the stack smashing protector enabled.
;; It is recommended to build glibc with --enable-stack-protector=strong.
;; Implemented by Nick Alcock (Oracle).
( define ( make-glibc-without-ssp xglibc )
( package-with-extra-configure-variable
( package-with-extra-configure-variable
xglibc "libc_cv_ssp" "no" )
"libc_cv_ssp_strong" "no" ) )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( define* ( make-bitcoin-cross-toolchain target
2021-05-24 15:31:05 +02:00
# :key
( base-gcc-for-libc gcc-7 )
2021-07-20 06:20:34 +02:00
( base-kernel-headers linux-libre-headers-4 . 9 )
2021-07-05 16:40:08 +02:00
( base-libc ( make-glibc-without-ssp glibc-2 . 24 ) )
2021-05-24 15:31:05 +02:00
( base-gcc ( make-gcc-rpath-link base-gcc ) ) )
2019-11-04 14:01:16 +01:00
" Convenience wrapper around MAKE-CROSS-TOOLCHAIN with default values
2023-03-31 12:01:55 +02:00
desirable for building Dash Core release binaries . "
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( make-cross-toolchain target
2021-05-24 15:31:05 +02:00
base-gcc-for-libc
base-kernel-headers
base-libc
base-gcc ) )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
2020-04-12 12:08:46 +02:00
( define ( make-gcc-with-pthreads gcc )
( package-with-extra-configure-variable gcc "--enable-threads" "posix" ) )
( define ( make-mingw-pthreads-cross-toolchain target )
"Create a cross-compilation toolchain package for TARGET"
( let* ( ( xbinutils ( cross-binutils target ) )
( pthreads-xlibc mingw-w64-x86_64-winpthreads )
( pthreads-xgcc ( make-gcc-with-pthreads
( cross-gcc target
2021-05-24 15:31:05 +02:00
# :xgcc ( make-ssp-fixed-gcc base-gcc )
2020-04-12 12:08:46 +02:00
# :xbinutils xbinutils
# :libc pthreads-xlibc ) ) ) )
;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
;; XGCC
( package
( name ( string-append target "-posix-toolchain" ) )
( version ( package-version pthreads-xgcc ) )
( source #f )
( build-system trivial-build-system )
( arguments ' ( # :builder ( begin ( mkdir %output ) #t ) ) )
( propagated-inputs
` ( ( "binutils" , xbinutils )
( "libc" , pthreads-xlibc )
2021-02-19 15:16:02 +01:00
( "gcc" , pthreads-xgcc )
( "gcc-lib" , pthreads-xgcc "lib" ) ) )
2020-04-12 12:08:46 +02:00
( synopsis ( string-append "Complete GCC tool chain for " target ) )
( description ( string-append " This package provides a complete GCC tool
chain for " target " development . " ) )
( home-page ( package-home-page pthreads-xgcc ) )
( license ( package-license pthreads-xgcc ) ) ) ) )
2021-01-20 00:43:38 +01:00
( define ( make-nsis-with-sde-support base-nsis )
( package-with-extra-patches base-nsis
( search-our-patches "nsis-SConstruct-sde-support.patch" ) ) )
2021-01-22 04:32:24 +01:00
( define-public font-tuffy
( package
2021-05-24 15:31:05 +02:00
( name "font-tuffy" )
( version "20120614" )
( source
( origin
( method url-fetch )
( uri ( string-append "http://tulrich.com/fonts/tuffy-" version ".tar.gz" ) )
( file-name ( string-append name "-" version ".tar.gz" ) )
( sha256
( base32
"02vf72bgrp30vrbfhxjw82s115z27dwfgnmmzfb0n9wfhxxfpyf6" ) ) ) )
( build-system font-build-system )
( home-page "http://tulrich.com/fonts/" )
( synopsis "The Tuffy Truetype Font Family" )
( description
"Thatcher Ulrich's first outline font design. He started with the goal of producing a neutral, readable sans-serif text font. There are lots of \"expressive\" fonts out there, but he wanted to start with something very plain and clean, something he might want to actually use. " )
( license license:public-domain ) ) )
( define osslsigncode
( package
( name "osslsigncode" )
( version "2.0" )
( source ( origin
( method url-fetch )
( uri ( string-append "https://github.com/mtrojnar/"
name "/archive/" version ".tar.gz" ) )
( sha256
( base32
"0byri6xny770wwb2nciq44j5071122l14bvv65axdd70nfjf0q2s" ) ) ) )
( build-system gnu-build-system )
( native-inputs
` ( ( "pkg-config" , pkg-config )
( "autoconf" , autoconf )
( "automake" , automake )
( "libtool" , libtool ) ) )
( inputs
` ( ( "openssl" , openssl ) ) )
( arguments
` ( # :configure-flags
` ( "--without-gsf"
"--without-curl"
"--disable-dependency-tracking" ) ) )
( home-page "https://github.com/mtrojnar/osslsigncode" )
( synopsis "Authenticode signing and timestamping tool" )
( description " osslsigncode is a small tool that implements part of the
functionality of the Microsoft tool signtool . exe - more exactly the Authenticode
signing and timestamping . But osslsigncode is based on OpenSSL and cURL, and
thus should be able to compile on most platforms where these exist . " )
( license license:gpl3+ ) ) ) ; license is with openssl exception
( define-public python-asn1crypto
( package
( name "python-asn1crypto" )
( version "1.4.0" )
( source
( origin
( method git-fetch )
( uri ( git-reference
( url "https://github.com/wbond/asn1crypto" )
( commit version ) ) )
( file-name ( git-file-name name version ) )
( sha256
( base32
"19abibn6jw20mzi1ln4n9jjvpdka8ygm4m439hplyrdfqbvgm01r" ) ) ) )
( build-system python-build-system )
( arguments
' ( # :phases
( modify-phases %standard-phases
( replace 'check
( lambda _
( invoke "python" "run.py" "tests" ) ) ) ) ) )
( home-page "https://github.com/wbond/asn1crypto" )
( synopsis "ASN.1 parser and serializer in Python" )
( description " asn1crypto is an ASN . 1 parser and serializer with definitions
for private keys, public keys, certificates, CRL, OCSP, CMS, PKCS # 3 , PKCS # 7 ,
PKCS # 8 , PKCS # 12 , PKCS # 5 , X . 509 and TSP . " )
( license license:expat ) ) )
( define-public python-elfesteem
( let ( ( commit "87bbd79ab7e361004c98cc8601d4e5f029fd8bd5" ) )
( package
( name "python-elfesteem" )
( version ( git-version "0.1" "1" commit ) )
( source
( origin
( method git-fetch )
( uri ( git-reference
( url "https://github.com/LRGH/elfesteem" )
( commit commit ) ) )
( file-name ( git-file-name name commit ) )
( sha256
( base32
"1nyvjisvyxyxnd0023xjf5846xd03lwawp5pfzr8vrky7wwm5maz" ) ) ) )
( build-system python-build-system )
;; There are no tests, but attempting to run python setup.py test leads to
;; PYTHONPATH problems, just disable the test
( arguments ' ( # :tests? #f ) )
( home-page "https://github.com/LRGH/elfesteem" )
( synopsis "ELF/PE/Mach-O parsing library" )
( description "elfesteem parses ELF, PE and Mach-O files." )
( license license:lgpl2 . 1 ) ) ) )
( define-public python-oscrypto
( package
( name "python-oscrypto" )
( version "1.2.1" )
( source
( origin
( method git-fetch )
( uri ( git-reference
( url "https://github.com/wbond/oscrypto" )
( commit version ) ) )
( file-name ( git-file-name name version ) )
( sha256
( base32
"1d4d8s4z340qhvb3g5m5v3436y3a71yc26wk4749q64m09kxqc3l" ) )
( patches ( search-our-patches "oscrypto-hard-code-openssl.patch" ) ) ) )
( build-system python-build-system )
( native-search-paths
( list ( search-path-specification
( variable "SSL_CERT_FILE" )
( file-type 'regular )
( separator #f ) ;single entry
( files ' ( "etc/ssl/certs/ca-certificates.crt" ) ) ) ) )
( propagated-inputs
` ( ( "python-asn1crypto" , python-asn1crypto )
( "openssl" , openssl ) ) )
( arguments
` ( # :phases
( modify-phases %standard-phases
( add-after 'unpack 'hard-code-path-to-libscrypt
( lambda* ( # :key inputs # :allow-other-keys )
( let ( ( openssl ( assoc-ref inputs "openssl" ) ) )
( substitute* "oscrypto/__init__.py"
( ( "@GUIX_OSCRYPTO_USE_OPENSSL@" )
( string-append openssl "/lib/libcrypto.so" "," openssl "/lib/libssl.so" ) ) )
#t ) ) )
( add-after 'unpack 'disable-broken-tests
( lambda _
;; This test is broken as there is no keyboard interrupt.
( substitute* "tests/test_trust_list.py"
( ( "^(.*)class TrustListTests" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) ) )
( substitute* "tests/test_tls.py"
( ( "^(.*)class TLSTests" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) ) )
#t ) )
( replace 'check
( lambda _
( invoke "python" "run.py" "tests" )
#t ) ) ) ) )
( home-page "https://github.com/wbond/oscrypto" )
( synopsis "Compiler-free Python crypto library backed by the OS" )
( description "oscrypto is a compilation-free, always up-to-date encryption library for Python." )
( license license:expat ) ) )
( define-public python-oscryptotests
( package ( inherit python-oscrypto )
( name "python-oscryptotests" )
( arguments
` ( # :tests? #f
# :phases
( modify-phases %standard-phases
( add-after 'unpack 'hard-code-path-to-libscrypt
( lambda* ( # :key inputs # :allow-other-keys )
( chdir "tests" )
#t ) ) ) ) ) ) )
( define-public python-certvalidator
( let ( ( commit "e5bdb4bfcaa09fa0af355eb8867d00dfeecba08c" ) )
( package
( name "python-certvalidator" )
( version ( git-version "0.1" "1" commit ) )
( source
( origin
( method git-fetch )
( uri ( git-reference
( url "https://github.com/achow101/certvalidator" )
( commit commit ) ) )
( file-name ( git-file-name name commit ) )
( sha256
( base32
"18pvxkvpkfkzgvfylv0kx65pmxfcv1hpsg03cip93krfvrrl4c75" ) ) ) )
( build-system python-build-system )
( propagated-inputs
` ( ( "python-asn1crypto" , python-asn1crypto )
( "python-oscrypto" , python-oscrypto )
( "python-oscryptotests" , python-oscryptotests ) ) ) ;; certvalidator tests import oscryptotests
( arguments
` ( # :phases
( modify-phases %standard-phases
( add-after 'unpack 'disable-broken-tests
( lambda _
( substitute* "tests/test_certificate_validator.py"
( ( "^(.*)class CertificateValidatorTests" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) ) )
( substitute* "tests/test_crl_client.py"
( ( "^(.*)def test_fetch_crl" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) ) )
( substitute* "tests/test_ocsp_client.py"
( ( "^(.*)def test_fetch_ocsp" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) ) )
( substitute* "tests/test_registry.py"
( ( "^(.*)def test_build_paths" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) ) )
( substitute* "tests/test_validate.py"
( ( "^(.*)def test_revocation_mode_hard" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) ) )
2022-02-09 10:29:50 +01:00
( substitute* "tests/test_validate.py"
( ( "^(.*)def test_revocation_mode_soft" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) ) )
2021-05-24 15:31:05 +02:00
#t ) )
( replace 'check
( lambda _
( invoke "python" "run.py" "tests" )
#t ) ) ) ) )
( home-page "https://github.com/wbond/certvalidator" )
( synopsis "Python library for validating X.509 certificates and paths" )
( description " certvalidator is a Python library for validating X . 509
certificates or paths . Supports various options, including: validation at a
specific moment in time, whitelisting and revocation checks . " )
( license license:expat ) ) ) )
( define-public python-requests-2 . 25.1
( package ( inherit python-requests )
( version "2.25.1" )
( source ( origin
( method url-fetch )
( uri ( pypi-uri "requests" version ) )
( sha256
( base32
"015qflyqsgsz09gnar69s6ga74ivq5kch69s4qxz3904m7a3v5r7" ) ) ) ) ) )
( define-public python-altgraph
( package
( name "python-altgraph" )
( version "0.17" )
( source
( origin
( method git-fetch )
( uri ( git-reference
( url "https://github.com/ronaldoussoren/altgraph" )
( commit ( string-append "v" version ) ) ) )
( file-name ( git-file-name name version ) )
( sha256
( base32
"09sm4srvvkw458pn48ga9q7ykr4xlz7q8gh1h9w7nxpf001qgpwb" ) ) ) )
( build-system python-build-system )
( home-page "https://github.com/ronaldoussoren/altgraph" )
( synopsis "Python graph (network) package" )
( description " altgraph is a fork of graphlib: a graph ( network ) package for
constructing graphs, BFS and DFS traversals, topological sort, shortest paths,
etc . with graphviz output . " )
( license license:expat ) ) )
( define-public python-macholib
( package
( name "python-macholib" )
( version "1.14" )
( source
( origin
( method git-fetch )
( uri ( git-reference
( url "https://github.com/ronaldoussoren/macholib" )
( commit ( string-append "v" version ) ) ) )
( file-name ( git-file-name name version ) )
( sha256
( base32
"0aislnnfsza9wl4f0vp45ivzlc0pzhp9d4r08700slrypn5flg42" ) ) ) )
( build-system python-build-system )
( propagated-inputs
` ( ( "python-altgraph" , python-altgraph ) ) )
( arguments
' ( # :phases
( modify-phases %standard-phases
( add-after 'unpack 'disable-broken-tests
( lambda _
;; This test is broken as there is no keyboard interrupt.
( substitute* "macholib_tests/test_command_line.py"
( ( "^(.*)class TestCmdLine" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) ) )
( substitute* "macholib_tests/test_dyld.py"
( ( "^(.*)def test_\\S+_find" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) )
( ( "^(.*)def testBasic" line indent )
( string-append indent
"@unittest.skip(\"Disabled by Guix\")\n"
line ) )
)
#t ) ) ) ) )
( home-page "https://github.com/ronaldoussoren/macholib" )
( synopsis "Python library for analyzing and editing Mach-O headers" )
( description " macholib is a Macho-O header analyzer and editor . It 's
typically used as a dependency analysis tool, and also to rewrite dylib
references in Mach-O headers to be @executable_path relative . Though this tool
targets a platform specific file format, it is pure python code that is platform
and endian independent . " )
( license license:expat ) ) )
( define-public python-signapple
2021-06-11 03:09:18 +02:00
( let ( ( commit "b084cbbf44d5330448ffce0c7d118f75781b64bd" ) )
2021-05-24 15:31:05 +02:00
( package
( name "python-signapple" )
( version ( git-version "0.1" "1" commit ) )
( source
( origin
( method git-fetch )
( uri ( git-reference
2021-06-11 03:09:18 +02:00
( url "https://github.com/achow101/signapple" )
2021-05-24 15:31:05 +02:00
( commit commit ) ) )
( file-name ( git-file-name name commit ) )
( sha256
( base32
2021-06-11 03:09:18 +02:00
"0k7inccl2mzac3wq4asbr0kl8s4cghm8982z54kfascqg45shv01" ) ) ) )
2021-05-24 15:31:05 +02:00
( build-system python-build-system )
( propagated-inputs
` ( ( "python-asn1crypto" , python-asn1crypto )
( "python-oscrypto" , python-oscrypto )
( "python-certvalidator" , python-certvalidator )
( "python-elfesteem" , python-elfesteem )
( "python-requests" , python-requests-2 . 25.1 )
( "python-macholib" , python-macholib )
( "libcrypto" , openssl ) ) )
;; There are no tests, but attempting to run python setup.py test leads to
;; problems, just disable the test
( arguments ' ( # :tests? #f ) )
( home-page "https://github.com/achow101/signapple" )
( synopsis "Mach-O binary signature tool" )
( description " signapple is a Python tool for creating, verifying, and
inspecting signatures in Mach-O binaries . " )
( license license:expat ) ) ) )
2020-04-12 12:08:46 +02:00
2021-07-05 16:40:08 +02:00
( define-public glibc-2 . 24
( package
( inherit glibc )
( version "2.24" )
( source ( origin
( method git-fetch )
( uri ( git-reference
( url "https://sourceware.org/git/glibc.git" )
( commit "0d7f1ed30969886c8dde62fbf7d2c79967d4bace" ) ) )
( file-name ( git-file-name "glibc" "0d7f1ed30969886c8dde62fbf7d2c79967d4bace" ) )
( sha256
( base32
"0g5hryia5v1k0qx97qffgwzrz4lr4jw3s5kj04yllhswsxyjbic3" ) )
( patches ( search-our-patches "glibc-ldd-x86_64.patch"
"glibc-versioned-locpath.patch"
"glibc-2.24-elfm-loadaddr-dynamic-rewrite.patch"
"glibc-2.24-no-build-time-cxx-header-run.patch" ) ) ) ) ) )
( define glibc-2 . 27 /bitcoin-patched
( package-with-extra-patches glibc-2 . 27
( search-our-patches "glibc-2.27-riscv64-Use-__has_include__-to-include-asm-syscalls.h.patch" ) ) )
Merge #15277: contrib: Enable building in Guix containers
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14)
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
2019-07-12 19:23:30 +02:00
( packages->manifest
2020-04-12 12:08:46 +02:00
( append
( list ;; The Basics
2021-01-22 04:32:24 +01:00
bash
2020-04-12 12:08:46 +02:00
which
coreutils
util-linux
;; File(system) inspection
file
grep
diffutils
findutils
;; File transformation
patch
gawk
sed
Merge bitcoin/bitcoin#22182: guix: Overhaul how guix-{attest,verify} works and hierarchy
e2c40a4ed5272d72fea997bd936fba28bb753226 guix-attest: Error out if SHA256SUMS is unexpected (Carl Dong)
4cc35daed557f38b080360a89036b2e97a6f78c2 Rewrite guix-{attest,verify} for new hier (Carl Dong)
28a9c9b83924f585b397f0f3b8e9e73780ac0ad6 Make SHA256SUMS fragment right after build (Carl Dong)
Pull request description:
Based on: #22075
Code reviewers: I recommend reading the new `guix-{attest,verify}` files instead of trying to read the diff
The following changes resolve many usability improvements which were pointed out to me:
1. Some maintainers like to extract their "uncodesigned tarball" inside the `output/` directory, resulting in the older `guix-attest` mistakenly attesting to the extracted contents
2. Maintainers whose GPG keys reside on an external smartcard often need to physically interact with the smartcard as a way to approve the signing operation, having one signature per platform means a lot of fidgeting
3. Maintainers wishing to sign on a separate machine now has the option of transferring only a subtree of `output/`, namely `output/*/SHA256SUMS.part`, in order to perform a signature (you may need to specify an `$OUTDIR_BASE` env var)
4. An `all.SHA256SUMS` file should be usable as the base `SHA256SUMS` in bitcoin core torrents and on the release server.
For those who sign on an separate machine than the one you do builds on, the following steps will work:
1. `env GUIX_SIGS_REPO=/home/achow101/guix.sigs SIGNER=achow101 NO_SIGN=1 ./contrib/guix/guix-attest`
2. Copy `/home/achow101/guix.sigs/<tag>/achow101` (which does not yet have signatures) to signing machine
3. Sign the `SHA256SUMS` files:
```bash
for i in "<path-to-achow101>/*.SHA256SUMS"; do
gpg --detach-sign --local-user "<your-key-here>" --armor --output "$i"{.asc,}
done
```
5. Upload `<path-to-achow101>` (now with signatures) to `guix.sigs`
-----
After this change, output directories will now include a `SHA256SUMS.part` fragment, created immediately after a successful build:
```
output
└── x86_64-w64-mingw32
├── bitcoin-4e069f7589da-win64-debug.zip
├── bitcoin-4e069f7589da-win64-setup-unsigned.exe
├── bitcoin-4e069f7589da-win64.zip
├── bitcoin-4e069f7589da-win-unsigned.tar.gz
└── SHA256SUMS.part
```
These `SHA256SUMS.part` fragments look something like:
```
3ebd7262b1a0a5bb757fef1f70e7e14033c70f98c059bc4dbfee5d1992b25825 dist-archive/bitcoin-4e069f7589da.tar.gz
def2e7d3de5ab3e3f955344e75151df4f33713f9101f5295bd13c9375bdf633b x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-debug.zip
643049fe3ee4a4e83a1739607e67b11b7c9b1a66208a6f35a9ff634ba795500e x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-setup-unsigned.exe
a247a1ccec0ccc2e138c648284bd01f6a761f2d8d6d07d91b5b4a6670ec3f288 x86_64-w64-mingw32/bitcoin-4e069f7589da-win-unsigned.tar.gz
fab76a836dcc592e39c04fd2396696633fb6eb56e39ecbf6c909bd173ed4280c x86_64-w64-mingw32/bitcoin-4e069f7589da-win64.zip
```
Meaning that they are valid `SHA256SUMS` files when `sha256sum --check`'d at the `guix-build-*/output` directory level
When `guix-attest` is invoked, these `SHA256SUMS.part` files are combined and sorted (by `-k2`, `LC_ALL=C`) to create:
1. `noncodesigned.SHA256SUMS` for a manifest of all non-codesigned outputs, and
3. `all.SHA256SUMS` for a manifest of all outputs including non-codesigned outputs
Then both files are signed, resulting in the following `guix.sigs` hierarchy:
```
4e069f7589da/
└── dongcarl
├── all.SHA256SUMS
├── all.SHA256SUMS.asc
├── noncodesigned.SHA256SUMS
└── noncodesigned.SHA256SUMS.asc
```
ACKs for top commit:
achow101:
ACK e2c40a4ed5272d72fea997bd936fba28bb753226
hebasto:
ACK e2c40a4ed5272d72fea997bd936fba28bb753226, tested on Linux Mint 20.1 (x86_64) with and w/o `NO_SIGN=1`. Changes in `contrib/guix/libexec/codesign.sh` and `contrib/guix/guix-verify` are reviewed only.
Tree-SHA512: 618aacefb0eb6595735a9ab6a98ea6598fce65f9ccf33fa1e7ef93bf140c0f6cfc16e34870c6aa3e4777dd3f004b92a82a994141879870141742df948ec59c1f
2021-06-17 07:10:29 +02:00
moreutils
2020-04-12 12:08:46 +02:00
;; Compression and archiving
tar
bzip2
gzip
xz
zlib
2021-04-06 00:59:11 +02:00
( list zlib "static" )
2020-04-12 12:08:46 +02:00
;; Build tools
gnu-make
libtool
autoconf
automake
pkg-config
bison
;; Scripting
perl
2021-02-19 15:16:02 +01:00
python-3
2020-05-06 06:51:18 +02:00
;; Git
git
2021-01-22 04:32:24 +01:00
;; Native gcc 7 toolchain
2021-04-06 00:59:11 +02:00
gcc-toolchain-7
( list gcc-toolchain-7 "static" ) )
2020-04-12 12:08:46 +02:00
( let ( ( target ( getenv "HOST" ) ) )
( cond ( ( string-suffix? "-mingw32" target )
;; Windows
2021-01-20 00:43:38 +01:00
( list zip
( make-mingw-pthreads-cross-toolchain "x86_64-w64-mingw32" )
2021-05-24 15:31:05 +02:00
( make-nsis-with-sde-support nsis-x86_64 )
osslsigncode ) )
2020-04-12 12:08:46 +02:00
( ( string-contains target "-linux-" )
2021-07-05 16:40:08 +02:00
( list ( cond ( ( string-contains target "riscv64-" )
2021-07-20 06:20:34 +02:00
( make-bitcoin-cross-toolchain target
# :base-libc glibc-2 . 27 /bitcoin-patched
# :base-kernel-headers linux-libre-headers-4 . 19 ) )
2021-07-05 16:40:08 +02:00
( else
( make-bitcoin-cross-toolchain target ) ) ) ) )
2021-01-22 04:32:24 +01:00
( ( string-contains target "darwin" )
2021-05-24 15:31:05 +02:00
( list clang-toolchain-10 binutils imagemagick libtiff librsvg font-tuffy cmake xorriso python-signapple ) )
2020-04-12 12:08:46 +02:00
( else ' ( ) ) ) ) ) )