sikkienl/dash
1
0
Fork 0
mirror of https://github.com/dashpay/dash.git synced 2024-12-27 04:52:59 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
401b55e5c4
dash/contrib/guix/README.md
fanquake 7d78f06a92 partial Merge bitcoin/bitcoin#19817: build: macOS toolchain bump 
a5550f877a2c46d01bb620ae051c0c8ed0fecd0b build: use -stdlib++-isystem with Clang 10 (fanquake)
51d9d1607f2e9d593693ca799393f068192e41aa guix: use Clang 10 for the macOS cross compile (fanquake)
b80a6af9e55325d444e117e85bbfc76d88d898a8 build: no longer patch threading out of ld64 (fanquake)
c29cba44b3706e0a2035e440e560f2d15d50433b build: Xcode 12.1, macOS SDK 10.15.6 (fanquake)
9ed2f19d385aa95f65807999bba2e18417b143dc build: native cctools 973.0.1, ld64 609 (fanquake)
f48f187cce7fa43646fb0d796c244e1515e763ec build: Clang 10.0.1 (Hennadii Stepanov)
9b193cd2a3ca20917611fbed56dfbcd8a39aeab8 build: libtapi 1100.0.11 (fanquake)

Pull request description:

  Bumps our macOS toolchain to be using the following:
  * Clang 10.0.1 (gitian) & Clang 10.0.0 (Guix)
  * ld64 609
  * libtapi 1100.0.11
  * cctools  973.0.1
  * Xcode 12.1
  * macOS SDK 10.15.6

  which are currently the most recent releases available as open source. See upstream [`cctools`](https://github.com/tpoechtrager/cctools-port/commits/973.0.1-ld64-609) and [`libtapi`](https://github.com/tpoechtrager/apple-libtapi/tree/1100.0.11).

  This should improve the possibility of Apple ARM cross-compilation in depends.

  This also removes our [patching out of pthreads usage](https://github.com/bitcoin/bitcoin/blob/master/depends/patches/native_cctools/ld64_disable_threading.patch) in `ld64`. There have been multiple changes since `ld64 450.3`, which have likely fixed the non-determinism we were working around. i.e from [InputFiles.cpp](https://opensource.apple.com/source/ld64/ld64-609/src/ld/InputFiles.cpp.auto.html):
  ```cpp
  // <rdar://problem/15002251> make implicit dylib order be deterministic by sorting by install_name
  std::sort(implicitDylibs.begin(), implicitDylibs.end(), DylibByInstallNameSorter());
  ```

  ```cpp
  // <rdar://problem/42675402> ld64 output is not deterministic due to dylib processing order
  std::sort(unprocessedDylibs.begin(), unprocessedDylibs.end(), [](const ld::dylib::File* lhs, const ld::dylib::File* rhs) {
  return strcmp(lhs->path(), rhs->path()) < 0;
  });
  ```

  Guix Build:
  ```bash
  find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
  f6c3817b8fe5f7370299d1ae2533e4a3acd313ba9f9aa8d423a8956117e52dd5  guix-build-a5550f877a2c/output/dist-archive/bitcoin-a5550f877a2c.tar.gz
  4954dcf563c2d496b8d9fecd48f8e3f7fba2f319ffa254a5bc8ee12cfee6acf0  guix-build-a5550f877a2c/output/x86_64-apple-darwin18/bitcoin-a5550f877a2c-osx-unsigned.dmg
  8f6095b445c7f1a8e6accd86bb7f0696d5849402084927d2b726b7d557831c3a  guix-build-a5550f877a2c/output/x86_64-apple-darwin18/bitcoin-a5550f877a2c-osx-unsigned.tar.gz
  cc40f25477b4defc1617ae694313d80f307ddf6742fe6cc85c6bc0e215ef8be0  guix-build-a5550f877a2c/output/x86_64-apple-darwin18/bitcoin-a5550f877a2c-osx64.tar.gz
  ```

  Gitian Build:
  ```bash
  Generating report
  506a8abdefe559999b43dd9f14905b9b2b5a3363b1cd013d45ae47acc1f7ef6c  bitcoin-a5550f877a2c-osx-unsigned.dmg
  f606997f74026dd12d110d683c6f116b40df324836904ef507dd7ac787e6ebe2  bitcoin-a5550f877a2c-osx-unsigned.tar.gz
  5b495ef15f2c3260c2950921b61326912a9bf533cccd51e13818809fd225489e  bitcoin-a5550f877a2c-osx64.tar.gz
  f6c3817b8fe5f7370299d1ae2533e4a3acd313ba9f9aa8d423a8956117e52dd5  src/bitcoin-a5550f877a2c.tar.gz
  9eb0221e962d2839770963bd03c6c9e98e8bf3078566bee2ae42f06233a710fa  bitcoin-core-osx-22-res.yml
  Done.
  ```

ACKs for top commit:
  hebasto:
    ACK a5550f877a2c46d01bb620ae051c0c8ed0fecd0b

Tree-SHA512: 504c4b0f9cd3b939714a322298320c5bde07e9356a48a9a000060b36f8dce4d6134ed60c3a5188810476a28ec5b108733eabbc6fb8053231b9ea8a494cc91b12
2023-03-26 16:50:26 -05:00

346 lines
12 KiB
Markdown
Raw Blame History

# Bootstrappable Bitcoin Core Builds
This directory contains the files necessary to perform bootstrappable Bitcoin
Core builds.
[Bootstrappability][b17e] furthers our binary security guarantees by allowing us
to _audit and reproduce_ our toolchain instead of blindly _trusting_ binary
downloads.
We achieve bootstrappability by using Guix as a functional package manager.
## Requirements
Conservatively, a x86_64 machine with:
- 16GB of free disk space on the partition that /gnu/store will reside in
- 8GB of free disk space per platform triple you're planning on building (see
the `HOSTS` environment variable description)
## Setup
### Installing Guix
If you're just testing this out, you can use the
[Dockerfile][fanquake/guix-docker] for convenience. It automatically speeds up
your builds by [using substitutes](#speeding-up-builds-with-substitute-servers).
If you don't want this behaviour, refer to the [next
section](#choosing-your-security-model).
Otherwise, follow the [Guix installation guide][guix/bin-install].
> Note: For those who like to keep their filesystems clean, Guix is designed to
> be very standalone and _will not_ conflict with your system's package
> manager/existing setup. It _only_ touches `/var/guix`, `/gnu`, and
> `~/.config/guix`.
### Choosing your security model
Guix allows us to achieve better binary security by using our CPU time to build
everything from scratch. However, it doesn't sacrifice user choice in pursuit of
this: users can decide whether or not to bootstrap and to use substitutes
(pre-built packages).
After installation, you may want to consider [adding substitute
servers](#speeding-up-builds-with-substitute-servers) from which to download
pre-built packages to speed up your build if that fits your security model (say,
if you're just testing that this works). Substitute servers are set up by
default if you're using the [Dockerfile][fanquake/guix-docker].
If you prefer not to use any substitutes, make sure to supply `--no-substitutes`
like in the following snippet. The first build will take a while, but the
resulting packages will be cached for future builds.
```sh
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes'
```
Likewise, to perform a bootstrapped build (takes even longer):
```sh
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes' ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap'
```
### Using a version of Guix with `guix time-machine` capabilities
> Note: This entire section can be skipped if you are already using a version of
> Guix that has [the `guix time-machine` command][guix/time-machine].
Once Guix is installed, if it doesn't have the `guix time-machine` command, pull
the latest `guix`.
```sh
guix pull --max-jobs=4 # change number of jobs accordingly
```
Make sure that you are using your current profile. (You are prompted to do this
at the end of the `guix pull`)
```bash
export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
```
### Controlling the number of threads used by `guix` build commands
By default, the scripts under `./contrib/guix` will invoke all `guix` build
commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
specified. However, astute manual readers will also notice that there is a
`--max-jobs=` flag (which defaults to 1 if unspecified).
Here is the difference between `--cores=` and `--max-jobs=`:
> Note: When I say "derivation," think "package"
`--cores=`
- controls the number of CPU cores to build each derivation. This is the value
passed to `make`'s `--jobs=` flag.
`--max-jobs=`
- controls how many derivations can be built in parallel
- defaults to 1
Therefore, the default is for `guix` build commands to build one derivation at a
time, utilizing `$JOBS` threads.
Specifying the `$JOBS` environment variable will only modify `--cores=`, but you
can also modify the value for `--max-jobs=` by specifying
`$ADDITIONAL_GUIX_COMMON_FLAGS`. For example, if you have a LOT of memory, you
may want to set:
```sh
export ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
```
Which allows for a maximum of 8 derivations to be built at the same time, each
utilizing `$JOBS` threads.
Or, if you'd like to avoid spurious build failures caused by issues with
parallelism within a single package, but would still like to build multiple
packages when the dependency graph allows for it, you may want to try:
```sh
export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
```
## Usage
### As a Tool for Deterministic Builds
From the top of a clean Bitcoin Core repository:
```sh
./contrib/guix/guix-build
```
After the build finishes successfully (check the status code please), compare
hashes:
```sh
find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
```
#### Recognized environment variables
* _**HOSTS**_
Override the space-separated list of platform triples for which to perform a
bootstrappable build. _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf
aarch64-linux-gnu riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
x86\_64-w64-mingw32 x86\_64-apple-darwin18")_
* _**SOURCES_PATH**_
Set the depends tree download cache for sources. This is passed through to the
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary redownloading of package sources.
* _**BASE_CACHE**_
Set the depends tree cache for built packages. This is passed through to the
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary building of packages.
* _**SDK_PATH**_
Set the path where _extracted_ SDKs can be found. This is passed through to
the depends tree. Note that this is should be set to the _parent_ directory of
the actual SDK (e.g. SDK_PATH=$HOME/Downloads/macOS-SDKs instead of
$HOME/Downloads/macOS-SDKs/Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers).
* _**JOBS**_
Override the number of jobs to run simultaneously, you might want to do so on
a memory-limited machine. This may be passed to:
- `guix` build commands as in `guix environment --cores="$JOBS"`
- `make` as in `make --jobs="$JOBS"`
- `xargs` as in `xargs -P"$JOBS"`
_(defaults to the value of `nproc` outside the container)_
* _**SOURCE_DATE_EPOCH**_
Override the reference UNIX timestamp used for bit-for-bit reproducibility,
the variable name conforms to [standard][r12e/source-date-epoch]. _(defaults
to the output of `$(git log --format=%at -1)`)_
* _**V**_
If non-empty, will pass `V=1` to all `make` invocations, making `make` output
verbose.
Note that any given value is ignored. The variable is only checked for
emptiness. More concretely, this means that `V=` (setting `V` to the empty
string) is interpreted the same way as not setting `V` at all, and that `V=0`
has the same effect as `V=1`.
* _**SUBSTITUTE_URLS**_
A whitespace-delimited list of URLs from which to download pre-built packages.
A URL is only used if its signing key is authorized (refer to the [substitute
servers section](#speeding-up-builds-with-substitute-servers) for more
details).
* _**ADDITIONAL_GUIX_COMMON_FLAGS**_
Additional flags to be passed to all `guix` commands.
* _**ADDITIONAL_GUIX_TIMEMACHINE_FLAGS**_
Additional flags to be passed to `guix time-machine`.
* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
Additional flags to be passed to the invocation of `guix environment` inside
`guix time-machine`.
## Tips and Tricks
### Speeding up builds with substitute servers
_This whole section is automatically done in the convenience
[Dockerfiles][fanquake/guix-docker]_
For those who are used to life in the fast _(and trustful)_ lane, you can
specify [substitute servers][guix/substitutes] from which to download pre-built
packages.
> For those who only want to use substitutes from the official Guix build farm
> and have authorized the build farm's signing key during Guix's installation,
> you don't need to do anything.
#### Step 1: Authorize the signing keys
For the official Guix build farm at https://ci.guix.gnu.org, run as root:
```
guix archive --authorize < ~root/.config/guix/current/share/guix/ci.guix.gnu.org.pub
```
For dongcarl's substitute server at https://guix.carldong.io, run as root:
```sh
wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
```
#### Step 2: Specify the substitute servers
The official Guix build farm at https://ci.guix.gnu.org is automatically used
unless the `--no-substitutes` flag is supplied.
This can be overridden for all `guix` invocations by passing the
`--substitute-urls` option to your invocation of `guix-daemon`. This can also be
overridden on a call-by-call basis by passing the same `--substitute-urls`
option to client tools such at `guix environment`.
To use dongcarl's substitute server for Bitcoin Core builds after having
[authorized his signing key](#authorize-the-signing-keys):
```
export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org'
```
## Troubleshooting
### Derivation failed to build
When you see a build failure like below:
```
building /gnu/store/...-foo-3.6.12.drv...
/ 'check' phasenote: keeping build directory `/tmp/guix-build-foo-3.6.12.drv-0'
builder for `/gnu/store/...-foo-3.6.12.drv' failed with exit code 1
build of /gnu/store/...-foo-3.6.12.drv failed
View build log at '/var/log/guix/drvs/../...-foo-3.6.12.drv.bz2'.
cannot build derivation `/gnu/store/...-qux-7.69.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-bar-3.16.5.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-baz-2.0.5.drv': 1 dependencies couldn't be built
guix time-machine: error: build of `/gnu/store/...-baz-2.0.5.drv' failed
```
It means that `guix` failed to build a package named `foo`, which was a
dependency of `qux`, `bar`, and `baz`. Importantly, note that the last "failed"
line is not necessarily the root cause, the first "failed" line is.
Most of the time, the build failure is due to a spurious test failure or the
package's build system/test suite breaking when running multi-threaded. To
rebuild _just_ this derivation in a single-threaded fashion:
```sh
$ guix build --cores=1 /gnu/store/...-foo-3.6.12.drv
```
If the single-threaded rebuild did not succeed, you may need to dig deeper.
You may view `foo`'s build logs in `less` like so (please replace paths with the
path you see in the build failure output):
```sh
$ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
```
`foo`'s build directory is also preserved and available at
`/tmp/guix-build-foo-3.6.12.drv-0`. However, if you fail to build `foo` multiple
times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
failure output for the most accurate, up-to-date information.
#### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
which rejects characters not present in the UTF-8 character code set. An example
is ZFS with the utf8only=on option set.
More information: https://bugs.python.org/issue37584
## FAQ
### How can I trust the binary installation?
As mentioned at the bottom of [this manual page][guix/bin-install]:
> The binary installation tarballs can be (re)produced and verified simply by
> running the following command in the Guix source tree:
>
> make guix-binary.x86_64-linux.tar.xz
### Is Guix packaged in my operating system?
Guix is shipped starting with [Debian Bullseye][debian/guix-bullseye] and
[Ubuntu 21.04 "Hirsute Hippo"][ubuntu/guix-hirsute]. Other operating systems
are working on packaging Guix as well.
[b17e]: http://bootstrappable.org/
[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
[guix/install.sh]: https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
[guix/bin-install]: https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
[guix/env-setup]: https://www.gnu.org/software/guix/manual/en/html_node/Build-Environment-Setup.html
[guix/substitutes]: https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html
[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html
[guix/time-machine]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html
[debian/guix-bullseye]: https://packages.debian.org/bullseye/guix
[ubuntu/guix-hirsute]: https://packages.ubuntu.com/hirsute/guix
[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix
Reference in New Issue View Git Blame Copy Permalink