sikkienl/dash
1
0
Fork 0
mirror of https://github.com/dashpay/dash.git synced 2024-12-24 19:42:46 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge #15277: contrib: Enable building in Guix containers

Browse Source 
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)

Pull request description:

  ~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~

  Please read the `README.md`.

  -----

  ### Guix Introduction

  This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.

  ### Guix Build Walkthrough

  Please read the `README.md`.

  [Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)

  [Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)

  [Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)

  <details>
  <summary>Old instructions no. 1</summary>
  In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.

  We can then invoke
  ```
  guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
  ```
  To have Guix:
  1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
  2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
  3. Drop you into a shell in that container.

  > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`

  Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:

  ```
  make -C depends -j"$(nproc)" download && \
      cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
  ```

  We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
  </details>

  ### Rationale

  I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:

  1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](72bd4ab867/contrib/gitian-descriptors/gitian-linux.yml (L10))), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
  2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
  3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).

  ### Steps to Completion

  - [x] Successfully build bitcoin inside the Guix environment
  - [x] Make `check-symbols` pass
  - [x] Do the above but without nasty hacks
  - [x] Solve some of the more innocuous hacks
  - [ ] Make it cross-compile (HELP WANTED HERE)
    - [x] Linux
      - [x] x86_64-linux-gnu
      - [x] i686-linux-gnu
      - [x] aarch64-linux-gnu
      - [x] arm-linux-gnueabihf
      - [x] riscv64-linux-gnu
    - [ ] OS X
      - [ ] x86_64-apple-darwin14
    - [ ] Windows
      - [ ] x86_64-w64-mingw32
  - [ ] Maybe make importer for depends syntax
  - [ ] Document build process for future releases
  - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)

  ### Help Wanted

  [Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)

  [Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)

  <details>
  <summary>Old content no. 1</summary>
  As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.

  Here's what `ldd src/bitcoind` looks like when built in a Guix container:
  ```
  	linux-vdso.so.1 (0x00007ffcc2d90000)
  	libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
  	librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
  	libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
  	libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
  	libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
  	libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
  	libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
  	/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
  ```

  And here's what it looks in one of our releases:
  ```
  	linux-vdso.so.1 (0x00007ffff52cd000)
  	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
  	librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
  	libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
  	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
  	libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
  	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
  ```

  ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.

  Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](d6e700e40f/contrib/gitian-descriptors/gitian-win-signer.yml (L14))
  </details>

  ### How to Help

  1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
  2. Try out my branch and the command described above!

ACKs for top commit:
  MarcoFalke:
    Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
  laanwj:
    ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f

Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
This commit is contained in:
Wladimir J. van der Laan 2019-07-12 19:23:30 +02:00 committed by PastaPastaPasta
parent 9f4d431b52
commit aec7441ac2
5 changed files with 637 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

229
contrib/guix/README.md Normal file
View File

@ -0,0 +1,229 @@
# Bootstrappable Bitcoin Core Builds
This directory contains the files necessary to perform bootstrappable Bitcoin
Core builds.
[Bootstrappability][b17e] furthers our binary security guarantees by allowing us
to _audit and reproduce_ our toolchain instead of blindly _trusting_ binary
downloads.
We achieve bootstrappability by using Guix as a functional package manager.
## Requirements
Conservatively, a x86_64 machine with:
- 2 or more logical cores
- 4GB of free disk space on the partition that /gnu/store will reside in
- 24GB of free disk space on the partition that the Bitcoin Core git repository
resides in
> Note: these requirements are slightly less onerous than those of Gitian builds
## Setup
### Installing Guix
If you're just testing this out, you can use the
[Dockerfile][fanquake/guix-docker] for convenience. It automatically speeds up
your builds by [using substitutes](#speeding-up-builds-with-substitute-servers).
If you don't want this behaviour, refer to the [next
section](#choosing-your-security-model).
Otherwise, follow the [Guix installation guide][guix/bin-install].
> Note: For those who like to keep their filesystems clean, Guix is designed to
> be very standalone and _will not_ conflict with your system's package
> manager/existing setup. It _only_ touches `/var/guix`, `/gnu`, and
> `~/.config/guix`.
### Choosing your security model
Guix allows us to achieve better binary security by using our CPU time to build
everything from scratch. However, it doesn't sacrifice user choice in pursuit of
this: users can decide whether or not to bootstrap and to use substitutes.
After installation, you may want to consider [adding substitute
servers](#speeding-up-builds-with-substitute-servers) to speed up your build if
that fits your security model (say, if you're just testing that this works).
This is skippable if you're using the [Dockerfile][fanquake/guix-docker].
If you prefer not to use any substitutes, make sure to set
`ADDITIONAL_GUIX_ENVIRONMENT_FLAGS` like the following snippet. The first build
will take a while, but the resulting packages will be cached for future builds.
```sh
export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--no-substitutes'
```
Likewise, to perform a bootstrapped build (takes even longer):
```sh
export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap --no-substitutes'
```
### Using the right Guix
Once Guix is installed, deploy our patched version into your current Guix
profile. The changes there are slowly being upstreamed.
```sh
guix pull --url=https://github.com/dongcarl/guix.git \
--commit=82c77e52b8b46e0a3aad2cb12307c2e30547deec \
--max-jobs=4 # change accordingly
```
Make sure that you are using your current profile. (You are prompted to do this
at the end of the `guix pull`)
```bash
export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
```
> Note: There is ongoing work to eliminate this entire section using Guix
> [inferiors][guix/inferiors] and [channels][guix/channels].
## Usage
### As a Development Environment
For a Bitcoin Core depends development environment, simply invoke
```sh
guix environment --manifest=contrib/guix/manifest.scm
```
And you'll land back in your shell with all the build dependencies required for
a `depends` build injected into your environment.
### As a Tool for Deterministic Builds
From the top of a clean Bitcoin Core repository:
```sh
./contrib/guix/guix-build.sh
```
After the build finishes successfully (check the status code please), compare
hashes:
```sh
find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
```
#### Recognized environment variables
* _**HOSTS**_
Override the space-separated list of platform triples for which to perform a
bootstrappable build. _(defaults to "i686-linux-gnu x86\_64-linux-gnu
arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu")_
> Windows and OS X platform triplet support are WIP.
* _**SOURCES_PATH**_
Set the depends tree download cache for sources. This is passed through to the
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary redownloading of package sources.
* _**MAX_JOBS**_
Override the maximum number of jobs to run simultaneously, you might want to
do so on a memory-limited machine. This may be passed to `make` as in `make
--jobs="$MAX_JOBS"` or `xargs` as in `xargs -P"$MAX_JOBS"`. _(defaults to the
value of `nproc` outside the container)_
* _**SOURCE_DATE_EPOCH**_
Override the reference UNIX timestamp used for bit-for-bit reproducibility,
the variable name conforms to [standard][r12e/source-date-epoch]. _(defaults
to the output of `$(git log --format=%at -1)`)_
* _**V**_
If non-empty, will pass `V=1` to all `make` invocations, making `make` output
verbose.
* _**ADDITIONAL_GUIX_ENVIRONMENT_FLAGS**_
Additional flags to be passed to `guix environment`. For a fully-bootstrapped
build, set this to `--bootstrap --no-substitutes` (refer to the [security
model section](#choosing-your-security-model) for more details). Note that a
fully-bootstrapped build will take quite a long time on the first run.
## Tips and Tricks
### Speeding up builds with substitute servers
_This whole section is automatically done in the convenience
[Dockerfiles][fanquake/guix-docker]_
For those who are used to life in the fast _(and trustful)_ lane, you can use
[substitute servers][guix/substitutes] to enable binary downloads of packages.
> For those who only want to use substitutes from the official Guix build farm
> and have authorized the build farm's signing key during Guix's installation,
> you don't need to do anything.
#### Authorize the signing keys
For the official Guix build farm at https://ci.guix.gnu.org, run as root:
```
guix archive --authorize < ~root/.config/guix/current/share/guix/ci.guix.gnu.org.pub
```
For dongcarl's substitute server at https://guix.carldong.io, run as root:
```sh
wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
```
#### Use the substitute servers
The official Guix build farm at https://ci.guix.gnu.org is automatically used
unless the `--no-substitutes` flag is supplied.
This can be overridden for all `guix` invocations by passing the
`--substitute-urls` option to your invocation of `guix-daemon`. This can also be
overridden on a call-by-call basis by passing the same `--substitute-urls`
option to client tools such at `guix environment`.
To use dongcarl's substitute server for Bitcoin Core builds after having
[authorized his signing key](#authorize-the-signing-keys):
```
export ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--substitute-urls="https://guix.carldong.io https://ci.guix.gnu.org"'
```
## FAQ
### How can I trust the binary installation?
As mentioned at the bottom of [this manual page][guix/bin-install]:
> The binary installation tarballs can be (re)produced and verified simply by
> running the following command in the Guix source tree:
>
> make guix-binary.x86_64-linux.tar.xz
### When will Guix be packaged in debian?
Vagrant Cascadian has been making good progress on this
[here][debian/guix-package]. We have all the pieces needed to put up an APT
repository and will likely put one up soon.
[b17e]: http://bootstrappable.org/
[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
[guix/install.sh]: https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
[guix/bin-install]: https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
[guix/env-setup]: https://www.gnu.org/software/guix/manual/en/html_node/Build-Environment-Setup.html
[guix/substitutes]: https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html
[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html
[guix/inferiors]: https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html
[guix/channels]: https://www.gnu.org/software/guix/manual/en/html_node/Channels.html
[debian/guix-package]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=850644
[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix

39
contrib/guix/guix-build.sh Executable file
View File

@ -0,0 +1,39 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
# Determine the maximum number of jobs to run simultaneously (overridable by
# environment)
MAX_JOBS="${MAX_JOBS:-$(nproc)}"
# Download the depends sources now as we won't have internet access in the build
# container
make -C "${PWD}/depends" -j"$MAX_JOBS" download ${V:+V=1} ${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"}
# Determine the reference time used for determinism (overridable by environment)
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
# Deterministically build Bitcoin Core for HOSTs (overriable by environment)
for host in ${HOSTS=i686-linux-gnu x86_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu riscv64-linux-gnu}; do
# Display proper warning when the user interrupts the build
trap 'echo "** INT received while building ${host}, you may want to clean up the relevant output and distsrc-* directories before rebuilding"' INT
# Run the build script 'contrib/guix/libexec/build.sh' in the build
# container specified by 'contrib/guix/manifest.scm'
# shellcheck disable=SC2086
guix environment --manifest="${PWD}/contrib/guix/manifest.scm" \
--container \
--pure \
--no-cwd \
--share="$PWD"=/bitcoin \
${SOURCES_PATH:+--share="$SOURCES_PATH"} \
${ADDITIONAL_GUIX_ENVIRONMENT_FLAGS} \
-- env HOST="$host" \
MAX_JOBS="$MAX_JOBS" \
SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:?unable to determine value}" \
${V:+V=1} \
${SOURCES_PATH:+SOURCES_PATH="$SOURCES_PATH"} \
bash -c "cd /bitcoin && bash contrib/guix/libexec/build.sh"
done

207
contrib/guix/libexec/build.sh Normal file
View File

@ -0,0 +1,207 @@
#!/usr/bin/env bash
export LC_ALL=C
set -e -o pipefail
# Check that environment variables assumed to be set by the environment are set
echo "Building for platform triple ${HOST:?not set} with reference timestamp ${SOURCE_DATE_EPOCH:?not set}..."
echo "At most ${MAX_JOBS:?not set} jobs will run at once..."
#####################
# Environment Setup #
#####################
# The depends folder also serves as a base-prefix for depends packages for
# $HOSTs after successfully building.
BASEPREFIX="${PWD}/depends"
# Setup an output directory for our build
OUTDIR="${OUTDIR:-${PWD}/output}"
[ -e "$OUTDIR" ] || mkdir -p "$OUTDIR"
# Setup the directory where our Bitcoin Core build for HOST will occur
DISTSRC="${DISTSRC:-${PWD}/distsrc-${HOST}}"
if [ -e "$DISTSRC" ]; then
echo "DISTSRC directory '${DISTSRC}' exists, probably because of previous builds... Aborting..."
exit 1
else
mkdir -p "$DISTSRC"
fi
# Given a package name and an output name, return the path of that output in our
# current guix environment
store_path() {
grep --extended-regexp "/[^-]{32}-${1}-cross-${HOST}-[^-]+${2:+-${2}}" "${GUIX_ENVIRONMENT}/manifest" \
| head --lines=1 \
| sed --expression='s|^[[:space:]]*"||' \
--expression='s|"[[:space:]]*$||'
}
# Determine output paths to use in CROSS_* environment variables
CROSS_GLIBC="$(store_path glibc)"
CROSS_GLIBC_STATIC="$(store_path glibc static)"
CROSS_KERNEL="$(store_path linux-libre-headers)"
CROSS_GCC="$(store_path gcc)"
# Set environment variables to point Guix's cross-toolchain to the right
# includes/libs for $HOST
export CROSS_C_INCLUDE_PATH="${CROSS_GCC}/include:${CROSS_GLIBC}/include:${CROSS_KERNEL}/include"
export CROSS_CPLUS_INCLUDE_PATH="${CROSS_GCC}/include/c++:${CROSS_GLIBC}/include:${CROSS_KERNEL}/include"
export CROSS_LIBRARY_PATH="${CROSS_GLIBC}/lib:${CROSS_GLIBC_STATIC}/lib:${CROSS_GCC}/lib:${CROSS_GCC}/${HOST}/lib:${CROSS_KERNEL}/lib"
# Disable Guix ld auto-rpath behavior
export GUIX_LD_WRAPPER_DISABLE_RPATH=yes
# Make /usr/bin if it doesn't exist
[ -e /usr/bin ] || mkdir -p /usr/bin
# Symlink file and env to a conventional path
[ -e /usr/bin/file ] || ln -s --no-dereference "$(command -v file)" /usr/bin/file
[ -e /usr/bin/env ] || ln -s --no-dereference "$(command -v env)" /usr/bin/env
# Determine the correct value for -Wl,--dynamic-linker for the current $HOST
glibc_dynamic_linker=$(
case "$HOST" in
i686-linux-gnu) echo /lib/ld-linux.so.2 ;;
x86_64-linux-gnu) echo /lib64/ld-linux-x86-64.so.2 ;;
arm-linux-gnueabihf) echo /lib/ld-linux-armhf.so.3 ;;
aarch64-linux-gnu) echo /lib/ld-linux-aarch64.so.1 ;;
riscv64-linux-gnu) echo /lib/ld-linux-riscv64-lp64d.so.1 ;;
*) exit 1 ;;
esac
)
# Environment variables for determinism
export QT_RCC_TEST=1
export QT_RCC_SOURCE_DATE_OVERRIDE=1
export TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --mtime='@${SOURCE_DATE_EPOCH}' --sort=name"
export TZ="UTC"
####################
# Depends Building #
####################
# Build the depends tree, overriding variables that assume multilib gcc
make -C depends --jobs="$MAX_JOBS" HOST="$HOST" \
${V:+V=1} \
${SOURCES_PATH+SOURCES_PATH="$SOURCES_PATH"} \
i686_linux_CC=i686-linux-gnu-gcc \
i686_linux_CXX=i686-linux-gnu-g++ \
i686_linux_AR=i686-linux-gnu-ar \
i686_linux_RANLIB=i686-linux-gnu-ranlib \
i686_linux_NM=i686-linux-gnu-nm \
i686_linux_STRIP=i686-linux-gnu-strip \
x86_64_linux_CC=x86_64-linux-gnu-gcc \
x86_64_linux_CXX=x86_64-linux-gnu-g++ \
x86_64_linux_AR=x86_64-linux-gnu-ar \
x86_64_linux_RANLIB=x86_64-linux-gnu-ranlib \
x86_64_linux_NM=x86_64-linux-gnu-nm \
x86_64_linux_STRIP=x86_64-linux-gnu-strip \
qt_config_opts_i686_linux='-platform linux-g++ -xplatform bitcoin-linux-g++'
###########################
# Source Tarball Building #
###########################
# Create the source tarball and move it to "${OUTDIR}/src" if not already there
if [ -z "$(find "${OUTDIR}/src" -name 'bitcoin-*.tar.gz')" ]; then
./autogen.sh
env CONFIG_SITE="${BASEPREFIX}/${HOST}/share/config.site" ./configure --prefix=/
make dist GZIP_ENV='-9n' ${V:+V=1}
mkdir -p "${OUTDIR}/src"
mv "$(find "${PWD}" -name 'bitcoin-*.tar.gz')" "${OUTDIR}/src/"
fi
# Determine the full path to our source tarball
SOURCEDIST="$(find "${OUTDIR}/src" -name 'bitcoin-*.tar.gz')"
# Determine our distribution name (e.g. bitcoin-0.18.0)
DISTNAME="$(basename "$SOURCEDIST" '.tar.gz')"
###########################
# Binary Tarball Building #
###########################
# Create a spec file to normalize ssp linking behaviour
spec_file="$(mktemp)"
cat << EOF > "$spec_file"
*link_ssp:
%{fstack-protector|fstack-protector-all|fstack-protector-strong|fstack-protector-explicit:}
EOF
# Similar flags to Gitian
CONFIGFLAGS="--enable-glibc-back-compat --enable-reduce-exports --disable-bench --disable-gui-tests"
HOST_CFLAGS="-O2 -g -specs=${spec_file} -ffile-prefix-map=${PWD}=."
HOST_CXXFLAGS="-O2 -g -specs=${spec_file} -ffile-prefix-map=${PWD}=."
HOST_LDFLAGS="-Wl,--as-needed -Wl,--dynamic-linker=$glibc_dynamic_linker -static-libstdc++"
# Make $HOST-specific native binaries from depends available in $PATH
export PATH="${BASEPREFIX}/${HOST}/native/bin:${PATH}"
(
cd "$DISTSRC"
# Extract the source tarball
tar --strip-components=1 -xf "${SOURCEDIST}"
# Configure this DISTSRC for $HOST
# shellcheck disable=SC2086
env CONFIG_SITE="${BASEPREFIX}/${HOST}/share/config.site" \
./configure --prefix=/ \
--disable-ccache \
--disable-maintainer-mode \
--disable-dependency-tracking \
${CONFIGFLAGS} \
CFLAGS="${HOST_CFLAGS}" \
CXXFLAGS="${HOST_CXXFLAGS}" \
LDFLAGS="${HOST_LDFLAGS}"
sed -i.old 's/-lstdc++ //g' {./,src/dashbls/,src/secp256k1/}{config.status,libtool}
# Build Bitcoin Core
make --jobs="$MAX_JOBS" ${V:+V=1}
# Perform basic ELF security checks on a series of executables.
make -C src --jobs=1 check-security ${V:+V=1}
# Check that executables only contain allowed gcc, glibc and libstdc++
# version symbols for Linux distro back-compatibility.
make -C src --jobs=1 check-symbols ${V:+V=1}
# Setup the directory where our Bitcoin Core build for HOST will be
# installed. This directory will also later serve as the input for our
# binary tarballs.
INSTALLPATH="${PWD}/installed/${DISTNAME}"
mkdir -p "${INSTALLPATH}"
# Install built Bitcoin Core to $INSTALLPATH
make install DESTDIR="${INSTALLPATH}" ${V:+V=1}
(
cd installed
# Prune libtool and object archives
find . -name "lib*.la" -delete
find . -name "lib*.a" -delete
# Prune pkg-config files
rm -r "${DISTNAME}/lib/pkgconfig"
# Split binaries and libraries from their debug symbols
{
find "${DISTNAME}/bin" -type f -executable -print0
find "${DISTNAME}/lib" -type f -print0
} | xargs -0 -n1 -P"$MAX_JOBS" -I{} "${DISTSRC}/contrib/devtools/split-debug.sh" {} {} {}.dbg
cp "${DISTSRC}/doc/README.md" "${DISTNAME}/"
# Finally, deterministically produce {non-,}debug binary tarballs ready
# for release
find "${DISTNAME}" -not -name "*.dbg" -print0 \
| sort --zero-terminated \
| tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \
| gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST}.tar.gz" \
|| ( rm -f "${OUTDIR}/${DISTNAME}-${HOST}.tar.gz" && exit 1 )
find "${DISTNAME}" -name "*.dbg" -print0 \
| sort --zero-terminated \
| tar --create --no-recursion --mode='u+rw,go+r-w,a+X' --null --files-from=- \
| gzip -9n > "${OUTDIR}/${DISTNAME}-${HOST}-debug.tar.gz" \
|| ( rm -f "${OUTDIR}/${DISTNAME}-${HOST}-debug.tar.gz" && exit 1 )
)
)

160
contrib/guix/manifest.scm Normal file
View File

@ -0,0 +1,160 @@
(use-modules (gnu)
(gnu packages)
(gnu packages autotools)
(gnu packages base)
(gnu packages bash)
(gnu packages check)
(gnu packages commencement)
(gnu packages compression)
(gnu packages cross-base)
(gnu packages file)
(gnu packages gawk)
(gnu packages gcc)
(gnu packages linux)
(gnu packages perl)
(gnu packages pkg-config)
(gnu packages python)
(gnu packages shells)
(gnu packages bison)
(guix build-system trivial)
(guix gexp)
(guix packages)
(guix profiles)
(guix utils))
(define (make-ssp-fixed-gcc xgcc)
"Given a XGCC package, return a modified package that uses the SSP function
from glibc instead of from libssp.so. Taken from:
http://www.linuxfromscratch.org/hlfs/view/development/chapter05/gcc-pass1.html"
(package
(inherit xgcc)
(arguments
(substitute-keyword-arguments (package-arguments xgcc)
((#:make-flags flags)
`(cons "gcc_cv_libc_provides_ssp=yes" ,flags))))))
(define (make-gcc-rpath-link xgcc)
"Given a XGCC package, return a modified package that replace each instance of
-rpath in the default system spec that's inserted by Guix with -rpath-link"
(package
(inherit xgcc)
(arguments
(substitute-keyword-arguments (package-arguments xgcc)
((#:phases phases)
`(modify-phases ,phases
(add-after 'pre-configure 'replace-rpath-with-rpath-link
(lambda _
(substitute* (cons "gcc/config/rs6000/sysv4.h"
(find-files "gcc/config"
"^gnu-user.*\\.h$"))
(("-rpath=") "-rpath-link="))
#t))))))))
(define (make-cross-toolchain target
base-gcc-for-libc
base-kernel-headers
base-libc
base-gcc)
"Create a cross-compilation toolchain package for TARGET"
(let* ((xbinutils (cross-binutils target))
;; 1. Build a cross-compiling gcc without targeting any libc, derived
;; from BASE-GCC-FOR-LIBC
(xgcc-sans-libc (cross-gcc target
#:xgcc base-gcc-for-libc
#:xbinutils xbinutils))
;; 2. Build cross-compiled kernel headers with XGCC-SANS-LIBC, derived
;; from BASE-KERNEL-HEADERS
(xkernel (cross-kernel-headers target
base-kernel-headers
xgcc-sans-libc
xbinutils))
;; 3. Build a cross-compiled libc with XGCC-SANS-LIBC and XKERNEL,
;; derived from BASE-LIBC
(xlibc (cross-libc target
base-libc
xgcc-sans-libc
xbinutils
xkernel))
;; 4. Build a cross-compiling gcc targeting XLIBC, derived from
;; BASE-GCC
(xgcc (cross-gcc target
#:xgcc base-gcc
#:xbinutils xbinutils
#:libc xlibc)))
;; Define a meta-package that propagates the resulting XBINUTILS, XLIBC, and
;; XGCC
(package
(name (string-append target "-toolchain"))
(version (package-version xgcc))
(source #f)
(build-system trivial-build-system)
(arguments '(#:builder (begin (mkdir %output) #t)))
(propagated-inputs
`(("binutils" ,xbinutils)
("libc" ,xlibc)
("libc:static" ,xlibc "static")
("gcc" ,xgcc)))
(synopsis (string-append "Complete GCC tool chain for " target))
(description (string-append "This package provides a complete GCC tool
chain for " target " development."))
(home-page (package-home-page xgcc))
(license (package-license xgcc)))))
(define* (make-bitcoin-cross-toolchain target
#:key
(base-gcc-for-libc gcc-5)
(base-kernel-headers linux-libre-headers-4.19)
(base-libc glibc-2.27)
(base-gcc (make-gcc-rpath-link
(make-ssp-fixed-gcc gcc-9))))
"Convienience wrapper around MAKE-CROSS-TOOLCHAIN with default values
desirable for building Bitcoin Core release binaries."
(make-cross-toolchain target
base-gcc-for-libc
base-kernel-headers
base-libc
base-gcc))
(packages->manifest
(list ;; The Basics
bash-minimal
which
coreutils
util-linux
;; File(system) inspection
file
grep
diffutils
findutils
;; File transformation
patch
gawk
sed
;; Compression and archiving
tar
bzip2
gzip
xz
zlib
;; Build tools
gnu-make
libtool
autoconf
automake
pkg-config
bison
;; Scripting
perl
python-3.7
;; Native gcc 9 toolchain targeting glibc 2.27
(make-gcc-toolchain gcc-9 glibc-2.27)
;; Cross gcc 9 toolchains targeting glibc 2.27
(make-bitcoin-cross-toolchain "i686-linux-gnu")
(make-bitcoin-cross-toolchain "x86_64-linux-gnu")
(make-bitcoin-cross-toolchain "aarch64-linux-gnu")
(make-bitcoin-cross-toolchain "arm-linux-gnueabihf")
;; The glibc 2.27 for riscv64 needs gcc 7 to successfully build (see:
;; https://www.gnu.org/software/gcc/gcc-7/changes.html#riscv). The final
;; toolchain is still a gcc 9 toolchain targeting glibc 2.27.
(make-bitcoin-cross-toolchain "riscv64-linux-gnu"
#:base-gcc-for-libc gcc-7)))

3
depends/packages/qt.mk
View File

@ -249,7 +249,8 @@ define $(package)_preprocess_cmds
echo "!host_build: QMAKE_LFLAGS += $($(package)_ldflags)" >> qtbase/mkspecs/common/gcc-base.conf && \
sed -i.old "s|QMAKE_CC = \$$$$\$$$${CROSS_COMPILE}clang|QMAKE_CC = $($(package)_cc)|" qtbase/mkspecs/common/clang.conf && \
sed -i.old "s|QMAKE_CXX = \$$$$\$$$${CROSS_COMPILE}clang++|QMAKE_CXX = $($(package)_cxx)|" qtbase/mkspecs/common/clang.conf && \
sed -i.old "s/error(\"failed to parse default search paths from compiler output\")/\!darwin: error(\"failed to parse default search paths from compiler output\")/g" qtbase/mkspecs/features/toolchain.prf
sed -i.old "s/error(\"failed to parse default search paths from compiler output\")/\!darwin: error(\"failed to parse default search paths from compiler output\")/g" qtbase/mkspecs/features/toolchain.prf && \
sed -i.old "s/LIBRARY_PATH/(CROSS_)?\0/g" qtbase/mkspecs/features/toolchain.prf
endef
define $(package)_config_cmds