Merge bitcoin/bitcoin#21711: guix: Add full installation and usage documentation

fac4814106c796b8786dd90053513cc35142dfe5 doc/release-process: Add torrent creation details (Carl Dong)
5d24cc3d82dad6812f8370c3ccc7c2b5a6c12c11 guix/INSTALL: Guix installs init scripts in libdir (Carl Dong)
5da2ee49d5b44de803b671aedbdd14e5c1d71ea9 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting (Carl Dong)
318c60700b7bbb7ec09a29bf037e7c2787646be6 guix: Adapt release-process.md to new Guix process (Carl Dong)
fcab35b2292f9221eaba521740e8b3b2511a8b78 guix-attest: Produce and sign normalized documents (Carl Dong)
c2541fd0ca99481a5a792a8f2772925d64fb0491 guix: Overhaul README (Carl Dong)
46ce6ce3782dfbd8f9d26dc2ba0f284755e75f2d tree-wide: Rename gitian-keys to builder-keys (Carl Dong)
fc4f8449f34e32b0b9ac9d218d6c3264b02467ba guix: Update various check_tools lists (Carl Dong)
263220a85c1df218431fafbda07c8b23ccc4ce4d guix: Check for a sane services database (Carl Dong)

Pull request description:

  Based on: #21462

  Keeping the README in one file so that it's easy to search through. Will add more jumping links later so navigation is easier.

  Current TODOs:
  - [x] Shell installer option: prompt user to re-login for `/etc/profile.d` entry to be picked up
  - [x] Binary tarball option: prompt user to create `/etc/profile.d` entry and re-login
  - [x] Fanquake docker option: complete section
  - [x] Arch Linux AUR option: prompt to start `guix-daemon-latest` unit after finishing "optional setup" section
  - [x] Building from source option: Insert dependency tree diagram that I made
  - [x] Building from source option: redo sectioning, kind of a mess right now
  - [x] Optional setup: make clear which parts are only needed if building from source
  - [x] Workaround 1 for GnuTLS: perhaps mention how to remove Guix build farm's key
  - [x] Overall (after everything): Make the links work.

  Note to self: wherever possible, tell user how to check that something is true rather than branching by installation option.

ACKs for top commit:
  fanquake:
    ACK fac4814106c796b8786dd90053513cc35142dfe5 - going to go ahead and merge this now. It's a lot of documentation, and could probably be nit-picked / improved further, however, that can continue over the next few weeks. I'm sure more (backportable) improvements / clarifications will be made while we progress through RCs towards a new release.

Tree-SHA512: dc46c0ecdfc67c7c7743ca26e4a603eb3f54adbf81be2f4c1f4c20577ebb84b5250b9c9ec89c0e9860337ab1c7cff94d7963c603287267deecfe1cd987fa070a
This commit is contained in:
fanquake 2021-07-20 11:07:18 +08:00 committed by PastaPastaPasta
parent 5b027e6611
commit f54ec9cde7
15 changed files with 1229 additions and 310 deletions

View File

@ -27,8 +27,8 @@ for Debian-based Linux systems. If you compile dashd/dash-qt yourself, there are
### [Gitian-descriptors](/contrib/gitian-descriptors) ###
Notes on getting Gitian builds up and running using KVM.
### [Gitian-keys](/contrib/gitian-keys)
PGP keys used for signing Dash Core [Gitian release](/doc/release-process.md) results.
### [Builder keys](/contrib/builder-keys)
PGP keys used for signing Dash Core [release](/doc/release-process.md) results.
### [MacDeploy](/contrib/macdeploy) ###
Scripts and notes for Mac builds.

View File

@ -3,8 +3,7 @@ PGP keys
This folder contains the public keys of developers and active contributors.
The keys are mainly used to sign git commits or the build results of Gitian
builds.
The keys are mainly used to sign git commits or the build results of builds.
You can import the keys into gpg as follows. Also, make sure to fetch the
latest version from the key server to see if any key was revoked in the

813
contrib/guix/INSTALL.md Normal file
View File

@ -0,0 +1,813 @@
# Guix Installation and Setup
This only needs to be done once per machine. If you have already completed the
installation and setup, please proceed to [perform a build](./README.md).
Otherwise, you may choose from one of the following options to install Guix:
1. Using the official **shell installer script** [⤓ skip to section][install-script]
- Maintained by Guix developers
- Easiest (automatically performs *most* setup)
- Works on nearly all Linux distributions
- Only installs latest release
- Binary installation only, requires high level of trust
- Note: The script needs to be run as root, so it should be inspected before it's run
2. Using the official **binary tarball** [⤓ skip to section][install-bin-tarball]
- Maintained by Guix developers
- Normal difficulty (full manual setup required)
- Works on nearly all Linux distributions
- Installs any release
- Binary installation only, requires high level of trust
3. Using fanquake's **Docker image** [↗︎ external instructions][install-fanquake-docker]
- Maintained by fanquake
- Easy (automatically performs *some* setup)
- Works wherever Docker images work
- Installs any release
- Binary installation only, requires high level of trust
4. Using a **distribution-maintained package** [⤓ skip to section][install-distro-pkg]
- Maintained by distribution's Guix package maintainer
- Normal difficulty (manual setup required)
- Works only on distributions with Guix packaged, see: https://repology.org/project/guix/versions
- Installs a release decided on by package maintainer
- Source or binary installation depending on the distribution
5. Building **from source** [⤓ skip to section][install-source]
- Maintained by you
- Hard, but rewarding
- Can be made to work on most Linux distributions
- Installs any commit (more granular)
- Source installation, requires lower level of trust
## Options 1 and 2: Using the official shell installer script or binary tarball
The installation instructions for both the official shell installer script and
the binary tarballs can be found in the GNU Guix Manual's [Binary Installation
section](https://guix.gnu.org/manual/en/html_node/Binary-Installation.html).
Note that running through the binary tarball installation steps is largely
equivalent to manually performing what the shell installer script does.
Note that at the time of writing (July 5th, 2021), the shell installer script
automatically creates an `/etc/profile.d` entry which the binary tarball
installation instructions do not ask you to create. However, you will likely
need this entry for better desktop integration. Please see [this
section](#add-an-etcprofiled-entry) for instructions on how to add a
`/etc/profile.d/guix.sh` entry.
Regardless of which installation option you chose, the changes to
`/etc/profile.d` will not take effect until the next shell or desktop session,
so you should log out and log back in.
## Option 3: Using fanquake's Docker image
Please refer to fanquake's instructions
[here](https://github.com/fanquake/core-review/tree/master/guix).
Note that the `Dockerfile` is largely equivalent to running through the binary
tarball installation steps.
## Option 4: Using a distribution-maintained package
Note that this section is based on the distro packaging situation at the time of
writing (July 2021). Guix is expected to be more widely packaged over time. For
an up-to-date view on Guix's package status/version across distros, please see:
https://repology.org/project/guix/versions
### Debian 11 (Bullseye)/Ubuntu 21.04 (Hirsute Hippo)
Guix v1.2.0 is available as a distribution package starting in [Debian
11](https://packages.debian.org/bullseye/guix) and [Ubuntu
21.04](https://packages.ubuntu.com/hirsute/guix).
Note that if you intend on using Guix without using any substitutes (more
details [here][security-model]), v1.2.0 has a known problems when building
GnuTLS from source. Solutions and workarounds are documented
[here](#gnutls-test-suite-fail-status-request-revoked).
To install:
```sh
sudo apt install guix
```
For up-to-date information on Debian and Ubuntu's release history:
- [Debian release history](https://www.debian.org/releases/)
- [Ubuntu release history](https://ubuntu.com/about/release-cycle)
### Arch Linux
Guix is available in the AUR as
[`guix`](https://aur.archlinux.org/packages/guix/), please follow the
installation instructions in the Arch Linux Wiki ([live
link](https://wiki.archlinux.org/index.php/Guix#AUR_Package_Installation),
[2021/03/30
permalink](https://wiki.archlinux.org/index.php?title=Guix&oldid=637559#AUR_Package_Installation))
to install Guix.
At the time of writing (2021/03/30), the `check` phase will fail if the path to
guix's build directory is longer than 36 characters due to an anachronistic
character limit on the shebang line. Since the `check` phase happens after the
`build` phase, which may take quite a long time, it is recommended that users
either:
1. Skip the `check` phase
- For `makepkg`: `makepkg --nocheck ...`
- For `yay`: `yay --mflags="--nocheck" ...`
- For `paru`: `paru --nocheck ...`
2. Or, check their build directory's length beforehand
- For those building with `makepkg`: `pwd | wc -c`
## Option 5: Building from source
Building Guix from source is a rather involved process but a rewarding one for
those looking to minimize trust and maximize customizability (e.g. building a
particular commit of Guix). Previous experience with using autotools-style build
systems to build packages from source will be helpful. *hic sunt dracones.*
I strongly urge you to at least skim through the entire section once before you
start issuing commands, as it will save you a lot of unncessary pain and
anguish.
### Installing common build tools
There are a few basic build tools that are required for most things we'll build,
so let's install them now:
Text transformation/i18n:
- `autopoint` (sometimes packaged in `gettext`)
- `help2man`
- `po4a`
- `texinfo`
Build system tools:
- `g++` w/ C++11 support
- `libtool`
- `autoconf`
- `automake`
- `pkg-config` (sometimes packaged as `pkgconf`)
- `make`
- `cmake`
Miscellaneous:
- `git`
- `gnupg`
- `python3`
### Building and Installing Guix's dependencies
In order to build Guix itself from source, we need to first make sure that the
necessary dependencies are installed and discoverable. The most up-to-date list
of Guix's dependencies is kept in the ["Requirements"
section](https://guix.gnu.org/manual/en/html_node/Requirements.html) of the Guix
Reference Manual.
Depending on your distribution, most or all of these dependencies may already be
packaged and installable without manually building and installing.
For reference, the graphic below outlines Guix v1.3.0's dependency graph:
![boostrap map](https://user-images.githubusercontent.com/6399679/125064185-a9a59880-e0b0-11eb-82c1-9b8e5dc9950d.png)
#### Guile
##### Choosing a Guile version and sticking to it
One of the first things you need to decide is which Guile version you want to
use: Guile v2.2 or Guile v3.0. Unlike the python2 to python3 transition, Guile
v2.2 and Guile v3.0 are largely compatible, as evidenced by the fact that most
Guile packages and even [Guix
itself](https://guix.gnu.org/en/blog/2020/guile-3-and-guix/) support running on
both.
What is important here is that you **choose one**, and you **remain consistent**
with your choice throughout **all Guile-related packages**, no matter if they
are installed via the distribution's package manager or installed from source.
This is because the files for Guile packages are installed to directories which
are separated based on the Guile version.
###### Example: Checking that Ubuntu's `guile-git` is compatible with your chosen Guile version
On Ubuntu Focal:
```sh
$ apt show guile-git
Package: guile-git
...
Depends: guile-2.2, guile-bytestructures, libgit2-dev
...
```
As you can see, the package `guile-git` depends on `guile-2.2`, meaning that it
was likely built for Guile v2.2. This means that if you decided to use Guile
v3.0 on Ubuntu Focal, you would need to build guile-git from source instead of
using the distribution package.
On Ubuntu Hirsute:
```sh
$ apt show guile-git
Package: guile-git
...
Depends: guile-3.0 | guile-2.2, guile-bytestructures (>= 1.0.7-3~), libgit2-dev (>= 1.0)
...
```
In this case, `guile-git` depends on either `guile-3.0` or `guile-2.2`, meaning
that it would work no matter what Guile version you decided to use.
###### Corner case: Multiple versions of Guile on one system
It is recommended to only install one version of Guile, so that build systems do
not get confused about which Guile to use.
However, if you insist on having both Guile v2.2 and Guile v3.0 installed on
your system, then you need to **consistently** specify one of
`GUILE_EFFECTIVE_VERSION=3.0` or `GUILE_EFFECTIVE_VERSION=2.2` to all
`./configure` invocations for Guix and its dependencies.
##### Installing Guile
Guile is most likely already packaged for your distribution, so after you have
[chosen a Guile version](#choosing-a-guile-version-and-sticking-to-it), install
it via your distribution's package manager.
If your distribution splits packages into `-dev`-suffixed and
non-`-dev`-suffixed sub-packages (as is the case for Debian-derived
distributions), please make sure to install both. For example, to install Guile
v2.2 on Debian/Ubuntu:
```sh
apt install guile-2.2 guile-2.2-dev
```
#### Mixing distribution packages and source-built packages
At the time of writing, most distributions have _some_ of Guix's dependencies
packaged, but not all. This means that you may want to install the distribution
package for some dependencies, and manually build-from-source for others.
Distribution packages usually install to `/usr`, which is different from the
default `./configure` prefix of source-built packages: `/usr/local`.
This means that if you mix-and-match distribution packages and source-built
packages and do not specify exactly `--prefix=/usr` to `./configure` for
source-built packages, you will need to augment the `GUILE_LOAD_PATH` and
`GUILE_LOAD_COMPILED_PATH` environment variables so that Guile will look
under the right prefix and find your source-built packages.
For example, if you are using Guile v2.2, and have Guile packages in the
`/usr/local` prefix, either add the following lines to your `.profile` or
`.bash_profile` so that the environment variable is properly set for all future
shell logins, or paste the lines into a POSIX-style shell to temporarily modify
the environment variables of your current shell session.
```sh
# Help Guile v2.2.x find packages in /usr/local
export GUILE_LOAD_PATH="/usr/local/share/guile/site/2.2${GUILE_LOAD_PATH:+:}$GUILE_LOAD_PATH"
export GUILE_LOAD_COMPILED_PATH="/usr/local/lib/guile/2.2/site-ccache${GUILE_LOAD_COMPILED_PATH:+:}$GUILE_COMPILED_LOAD_PATH"
```
Note that these environment variables are used to check for packages during
`./configure`, so they should be set as soon as possible should you want to use
a prefix other than `/usr`.
<!-- ##### Example: Consistently using Guile 3.0 on Ubuntu -->
<!-- For example, on Ubuntu, if you choose to use Guile 3.0 and install the -->
<!-- `guile-3.0` package, you want to make sure that if you also want to install the -->
<!-- `guile-git` package with `apt` that said `guile-git` package was built for Guile -->
<!-- v3.0. This can be checked by invoking the following: -->
<!-- ``` -->
<!-- apt update -->
<!-- apt show guile-git -->
<!-- ``` -->
#### Building and installing source-built packages
***IMPORTANT**: A few dependencies have non-obvious quirks/erratas which are documented in the
sub-sections immediately below. Please read these sections before proceeding to
build and install these packages.*
Although you should always refer to the README or INSTALL files for the most
accurate information, most of these dependencies use autoconf-style build
systems (check if there's a `configure.ac` file), and will likely do the right
thing with the following:
Clone the repository and check out the latest release:
```sh
git clone <git-repo-of-dependency>/<dependency>.git
cd <dependency>
git tag -l # check for the latest release
git checkout <latest-release>
```
For autoconf-based build systems (if `./autogen.sh` or `configure.ac` exists at
the root of the repository):
```sh
./autogen.sh || autoreconf -vfi
./configure --prefix=<prefix>
make
sudo make install
```
For CMake-based build systems (if `CMakeLists.txt` exists at the root of the
repository):
```sh
mkdir build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=<prefix>
sudo cmake --build . --target install
```
If you choose not to specify exactly `--prefix=/usr` to `./configure`, please
make sure you've carefully read the [previous section] on mixing distribution
packages and source-built packages.
##### Binding packages require `-dev`-suffixed packages
Relevant for:
- Everyone
When building bindings, the `-dev`-suffixed version of the original package
needs to be installed. For example, building `Guile-zlib` on Debian-derived
distributions requires that `zlib1g-dev` is installed.
When using bindings, the `-dev`-suffixed version of the original package still
needs to be installed. This is particularly problematic when distribution
packages are mispackaged like `guile-sqlite3` is in Ubuntu Focal such that
installing `guile-sqlite3` does not automatically install `libsqlite3-dev` as a
dependency.
Below is a list of relevant Guile bindings and their corresponding `-dev`
packages in Debian at the time of writing.
| Guile binding package | -dev Debian package |
|-----------------------|---------------------|
| guile-gcrypt | libgcrypt-dev |
| guile-git | libgit2-dev |
| guile-lzlib | liblz-dev |
| guile-ssh | libssh-dev |
| guile-sqlite3 | libsqlite3-dev |
| guile-zlib | zlib1g-dev |
##### `guile-git` actually depends on `libgit2 >= 1.1`
Relevant for:
- Those building `guile-git` from source against `libgit2 < 1.1`
- Those installing `guile-git` from their distribution where `guile-git` is
built against `libgit2 < 1.1`
As of v0.4.0, `guile-git` claims to only require `libgit2 >= 0.28.0`, however,
it actually requires `libgit2 >= 1.1`, otherwise, it will be confused by a
reference of `origin/keyring`: instead of interpreting the reference as "the
'keyring' branch of the 'origin' remote", the reference is interpreted as "the
branch literally named 'origin/keyring'"
This is especially notable because Ubuntu Focal packages `libgit2 v0.28.4`, and
`guile-git` is built against it.
Should you be in this situation, you need to build both `libgit2 v1.1.x` and
`guile-git` from source.
Source: http://logs.guix.gnu.org/guix/2020-11-12.log#232527
##### `{scheme,guile}-bytestructures` v1.0.8 and v1.0.9 are broken for Guile v2.2
Relevant for:
- Those building `{scheme,guile}-bytestructures` from source against Guile v2.2
Commit
[707eea3](https://github.com/TaylanUB/scheme-bytestructures/commit/707eea3a85e1e375e86702229ebf73d496377669)
introduced a regression for Guile v2.2 and was first included in v1.0.8, this
was later corrected in commit
[ec9a721](https://github.com/TaylanUB/scheme-bytestructures/commit/ec9a721957c17bcda13148f8faa5f06934431ff7)
and included in v1.1.0.
TL;DR If you decided to use Guile v2.2, do not use `{scheme,guile}-bytestructures` v1.0.8 or v1.0.9.
### Building and Installing Guix itself
Start by cloning Guix:
```
git clone https://git.savannah.gnu.org/git/guix.git
cd guix
```
You will likely want to build the latest release, however, if the latest release
when you're reading this is still 1.2.0 then you may want to use 95aca29 instead
to avoid a problem in the GnuTLS test suite.
```
git branch -a -l 'origin/version-*' # check for the latest release
git checkout <latest-release>
```
Bootstrap the build system:
```
./bootstrap
```
Configure with the recommended `--localstatedir` flag:
```
./configure --localstatedir=/var
```
Note: If you intend to hack on Guix in the future, you will need to supply the
same `--localstatedir=` flag for all future Guix `./configure` invocations. See
the last paragraph of this
[section](https://guix.gnu.org/manual/en/html_node/Requirements.html) for more
details.
Build Guix (this will take a while):
```
make -j$(nproc)
```
Install Guix:
```
sudo make install
```
### Post-"build from source" Setup
#### Creating and starting a `guix-daemon-original` service with a fixed `argv[0]`
At this point, guix will be installed to `${bindir}`, which is likely
`/usr/local/bin` if you did not override directory variables at
`./configure`-time. More information on standard Automake directory variables
can be found
[here](https://www.gnu.org/software/automake/manual/html_node/Standard-Directory-Variables.html).
However, the Guix init scripts and service configurations for Upstart, systemd,
SysV, and OpenRC are installed (in `${libdir}`) to launch
`${localstatedir}/guix/profiles/per-user/root/current-guix/bin/guix-daemon`,
which does not yet exist, and will only exist after [`root` performs their first
`guix pull`](#guix-pull-as-root).
We need to create a `-original` version of these init scripts that's pointed to
the binaries we just built and `make install`'ed in `${bindir}` (normally,
`/usr/local/bin`).
Example for `systemd`, run as `root`:
```sh
# Create guix-daemon-original.service by modifying guix-daemon.service
libdir=# set according to your PREFIX (default is /usr/local/lib)
bindir="$(dirname $(command -v guix-daemon))"
sed -E -e "s|/\S*/guix/profiles/per-user/root/current-guix/bin/guix-daemon|${bindir}/guix-daemon|" "${libdir}"/systemd/system/guix-daemon.service > /etc/systemd/system/guix-daemon-original.service
chmod 664 /etc/systemd/system/guix-daemon-original.service
# Make systemd recognize the new service
systemctl daemon-reload
# Make sure that the non-working guix-daemon.service is stopped and disabled
systemctl stop guix-daemon
systemctl disable guix-daemon
# Make sure that the working guix-daemon-original.service is started and enabled
systemctl enable guix-daemon-original
systemctl start guix-daemon-original
```
#### Creating `guix-daemon` users / groups
Please see the [relevant
section](https://guix.gnu.org/manual/en/html_node/Build-Environment-Setup.html)
in the Guix Reference Manual for more details.
## Optional setup
At this point, you are set up to [use Guix to build Bitcoin
Core](./README.md#usage). However, if you want to polish your setup a bit and
make it "what Guix intended", then read the next few subsections.
### Add an `/etc/profile.d` entry
This section definitely does not apply to you if you installed Guix using:
1. The shell installer script
2. fanquake's Docker image
3. Debian's `guix` package
#### Background
Although Guix knows how to update itself and its packages, it does so in a
non-invasive way (it does not modify `/usr/local/bin/guix`).
Instead, it does the following:
- After a `guix pull`, it updates
`/var/guix/profiles/per-user/$USER/current-guix`, and creates a symlink
targeting this directory at `$HOME/.config/guix/current`
- After a `guix install`, it updates
`/var/guix/profiles/per-user/$USER/guix-profile`, and creates a symlink
targeting this directory at `$HOME/.guix-profile`
Therefore, in order for these operations to affect your shell/desktop sessions
(and for the principle of least astonishment to hold), their corresponding
directories have to be added to well-known environment variables like `$PATH`,
`$INFOPATH`, `$XDG_DATA_DIRS`, etc.
In other words, if `$HOME/.config/guix/current/bin` does not exist in your
`$PATH`, a `guix pull` will have no effect on what `guix` you are using. Same
goes for `$HOME/.guix-profile/bin`, `guix install`, and installed packages.
Helpfully, after a `guix pull` or `guix install`, a message will be printed like
so:
```
hint: Consider setting the necessary environment variables by running:
GUIX_PROFILE="$HOME/.guix-profile"
. "$GUIX_PROFILE/etc/profile"
Alternately, see `guix package --search-paths -p "$HOME/.guix-profile"'.
```
However, this is somewhat tedious to do for both `guix pull` and `guix install`
for each user on the system that wants to properly use `guix`. I recommend that
you instead add an entry to `/etc/profile.d` instead. This is done by default
when installing the Debian package later than 1.2.0-4 and when using the shell
script installer.
#### Instructions
Create `/etc/profile.d/guix.sh` with the following content:
```sh
# _GUIX_PROFILE: `guix pull` profile
_GUIX_PROFILE="$HOME/.config/guix/current"
if [ -L $_GUIX_PROFILE ]; then
export PATH="$_GUIX_PROFILE/bin${PATH:+:}$PATH"
# Export INFOPATH so that the updated info pages can be found
# and read by both /usr/bin/info and/or $GUIX_PROFILE/bin/info
# When INFOPATH is unset, add a trailing colon so that Emacs
# searches 'Info-default-directory-list'.
export INFOPATH="$_GUIX_PROFILE/share/info:$INFOPATH"
fi
# GUIX_PROFILE: User's default profile
GUIX_PROFILE="$HOME/.guix-profile"
[ -L $GUIX_PROFILE ] || return
GUIX_LOCPATH="$GUIX_PROFILE/lib/locale"
export GUIX_PROFILE GUIX_LOCPATH
[ -f "$GUIX_PROFILE/etc/profile" ] && . "$GUIX_PROFILE/etc/profile"
# set XDG_DATA_DIRS to include Guix installations
export XDG_DATA_DIRS="$GUIX_PROFILE/share:${XDG_DATA_DIRS:-/usr/local/share/:/usr/share/}"
```
Please note that this will not take effect until the next shell or desktop
session (log out and log back in).
### `guix pull` as root
Before you do this, you need to read the section on [choosing your security
model][security-model] and adjust `guix` and `guix-daemon` flags according to
your choice, as invoking `guix pull` may pull substitutes from substitute
servers (which you may not want).
As mentioned in a previous section, Guix expects
`${localstatedir}/guix/profiles/per-user/root/current-guix` to be populated with
`root`'s Guix profile, `guix pull`-ed and built by some former version of Guix.
However, this is not the case when we build from source. Therefore, we need to
perform a `guix pull` as `root`:
```sh
sudo --login guix pull --branch=version-<latest-release-version>
# or
sudo --login guix pull --commit=<particular-commit>
```
`guix pull` is quite a long process (especially if you're using
`--no-substitute`). If you encounter build problems, please refer to the
[troubleshooting section](#troubleshooting).
Note that running a bare `guix pull` with no commit or branch specified will
pull the latest commit on Guix's master branch, which is likely fine, but not
recommended.
If you installed Guix from source, you may get an error like the following:
```sh
error: while creating symlink '/root/.config/guix/current' No such file or directory
```
To resolve this, simply:
```
sudo mkdir -p /root/.config/guix
```
Then try the `guix pull` command again.
After the `guix pull` finishes successfully,
`${localstatedir}/guix/profiles/per-user/root/current-guix` should be populated.
#### Using the newly-pulled `guix` by restarting the daemon
Depending on how you installed Guix, you should now make sure that your init
scripts and service configurations point to the newly-pulled `guix-daemon`.
##### If you built Guix from source
If you followed the instructions for [fixing argv\[0\]][fix-argv0], you can now
do the following:
```sh
systemctl stop guix-daemon-original
systemctl disable guix-daemon-original
systemctl enable guix-daemon
systemctl start guix-daemon
```
##### If you installed Guix via the Debian/Ubuntu distribution packages
You will need to create a `guix-daemon-latest` service which points to the new
`guix` rather than a pinned one.
```sh
# Create guix-daemon-latest.service by modifying guix-daemon.service
sed -E -e "s|/usr/bin/guix-daemon|/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon|" /etc/systemd/system/guix-daemon.service > /lib/systemd/system/guix-daemon-latest.service
chmod 664 /lib/systemd/system/guix-daemon-latest.service
# Make systemd recognize the new service
systemctl daemon-reload
# Make sure that the old guix-daemon.service is stopped and disabled
systemctl stop guix-daemon
systemctl disable guix-daemon
# Make sure that the new guix-daemon-latest.service is started and enabled
systemctl enable guix-daemon-latest
systemctl start guix-daemon-latest
```
##### If you installed Guix via lantw44's Arch Linux AUR package
At the time of writing (July 5th, 2021) the systemd unit for "updated Guix" is
`guix-daemon-latest.service`, therefore, you should do the following:
```sh
systemctl stop guix-daemon
systemctl disable guix-daemon
systemctl enable guix-daemon-latest
systemctl start guix-daemon-latest
```
##### Otherwise...
Simply do:
```sh
systemctl restart guix-daemon
```
### Checking everything
If you followed all the steps above to make your Guix setup "prim and proper,"
you can check that you did everything properly by running through this
checklist.
1. `/etc/profile.d/guix.sh` should exist and be sourced at each shell login
2. `guix describe` should not print `guix describe: error: failed to determine
origin`, but rather something like:
```
Generation 38 Feb 22 2021 16:39:31 (current)
guix f350df4
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: version-1.2.0
commit: f350df405fbcd5b9e27e6b6aa500da7f101f41e7
```
3. `guix-daemon` should be running from `${localstatedir}/guix/profiles/per-user/root/current-guix`
# Troubleshooting
## Derivation failed to build
When you see a build failure like below:
```
building /gnu/store/...-foo-3.6.12.drv...
/ 'check' phasenote: keeping build directory `/tmp/guix-build-foo-3.6.12.drv-0'
builder for `/gnu/store/...-foo-3.6.12.drv' failed with exit code 1
build of /gnu/store/...-foo-3.6.12.drv failed
View build log at '/var/log/guix/drvs/../...-foo-3.6.12.drv.bz2'.
cannot build derivation `/gnu/store/...-qux-7.69.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-bar-3.16.5.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-baz-2.0.5.drv': 1 dependencies couldn't be built
guix time-machine: error: build of `/gnu/store/...-baz-2.0.5.drv' failed
```
It means that `guix` failed to build a package named `foo`, which was a
dependency of `qux`, `bar`, and `baz`. Importantly, note that the last "failed"
line is not necessarily the root cause, the first "failed" line is.
Most of the time, the build failure is due to a spurious test failure or the
package's build system/test suite breaking when running multi-threaded. To
rebuild _just_ this derivation in a single-threaded fashion (please don't forget
to add other `guix` flags like `--no-substitutes` as appropriate):
```sh
$ guix build --cores=1 /gnu/store/...-foo-3.6.12.drv
```
If the single-threaded rebuild did not succeed, you may need to dig deeper.
You may view `foo`'s build logs in `less` like so (please replace paths with the
path you see in the build failure output):
```sh
$ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
```
`foo`'s build directory is also preserved and available at
`/tmp/guix-build-foo-3.6.12.drv-0`. However, if you fail to build `foo` multiple
times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
failure output for the most accurate, up-to-date information.
### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
which rejects characters not present in the UTF-8 character code set. An example
is ZFS with the utf8only=on option set.
More information: https://bugs.python.org/issue37584
### GnuTLS: test-suite FAIL: status-request-revoked
*The derivation is likely identified by: `/gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv`*
This unfortunate error is most common for non-substitute builders who installed
Guix v1.2.0. The problem stems from the fact that one of GnuTLS's tests uses a
hardcoded certificate which expired on 2020-10-24.
What's more unfortunate is that this GnuTLS derivation is somewhat special in
Guix's dependency graph and is not affected by the package transformation flags
like `--without-tests=`.
The easiest solution for those encountering this problem is to install a newer
version of Guix. However, there are ways to work around this issue:
#### Workaround 1: Using substitutes for this single derivation
If you've authorized the official Guix build farm's key (more info
[here](./README.md#step-1-authorize-the-signing-keys)), then you can use
substitutes just for this single derivation by invoking the following:
```sh
guix build --substitute-urls="https://ci.guix.gnu.org" /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv
```
See [this section](./README.md#removing-authorized-keys) for instructions on how
to remove authorized keys if you don't want to keep the build farm's key
authorized.
#### Workaround 2: Temporarily setting the system clock back
This workaround was described [here](https://issues.guix.gnu.org/44559#5).
Basically:
1. Turn off networking
2. Turn off NTP
3. Set system time to 2020-10-01
4. guix build --no-substitutes /gnu/store/vhphki5sg9xkdhh2pbc8gi6vhpfzryf0-gnutls-3.6.12.drv
5. Set system time back to accurate current time
6. Turn NTP back on
7. Turn networking back on
### coreutils: FAIL: tests/tail-2/inotify-dir-recreate
The inotify-dir-create test fails on "remote" filesystems such as overlayfs
(Docker's default filesystem) due to the filesystem being mistakenly recognized
as non-remote.
A relatively easy workaround to this is to make sure that a somewhat traditional
filesystem is mounted at `/tmp` (where `guix-daemon` performs its builds). For
Docker users, this might mean [using a volume][docker/volumes], [binding
mounting][docker/bind-mnt] from host, or (for those with enough RAM and swap)
[mounting a tmpfs][docker/tmpfs] using the `--tmpfs` flag.
Please see the following links for more details:
- An upstream coreutils bug has been filed: [debbugs#47940](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47940)
- A Guix bug detailing the underlying problem has been filed: [guix-issues#47935](https://issues.guix.gnu.org/47935)
- A commit to skip this test in Guix has been merged into the core-updates branch:
[savannah/guix@6ba1058](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6ba1058df0c4ce5611c2367531ae5c3cdc729ab4)
[install-script]: #options-1-and-2-using-the-official-shell-installer-script-or-binary-tarball
[install-bin-tarball]: #options-1-and-2-using-the-official-shell-installer-script-or-binary-tarball
[install-fanquake-docker]: #option-3-using-fanquakes-docker-image
[install-distro-pkg]: #option-4-using-a-distribution-maintained-package
[install-source]: #option-5-building-from-source
[fix-argv0]: #creating-and-starting-a-guix-daemon-original-service-with-a-fixed-argv0
[security-model]: ./README.md#choosing-your-security-model
[docker/volumes]: https://docs.docker.com/storage/volumes/
[docker/bind-mnt]: https://docs.docker.com/storage/bind-mounts/
[docker/tmpfs]: https://docs.docker.com/storage/tmpfs/

View File

@ -9,83 +9,171 @@ downloads.
We achieve bootstrappability by using Guix as a functional package manager.
## Requirements
# Requirements
Conservatively, a x86_64 machine with:
- 16GB of free disk space on the partition that /gnu/store will reside in
- 8GB of free disk space per platform triple you're planning on building (see
the `HOSTS` environment variable description)
- 8GB of free disk space **per platform triple** you're planning on building
(see the `HOSTS` [environment variable description][env-vars-list])
## Setup
# Installation and Setup
### Installing Guix
If you don't have Guix installed and set up, please follow the instructions in
[INSTALL.md](./INSTALL.md)
If you're just testing this out, you can use the
[Dockerfile][fanquake/guix-docker] for convenience. It automatically speeds up
your builds by [using substitutes](#speeding-up-builds-with-substitute-servers).
If you don't want this behaviour, refer to the [next
section](#choosing-your-security-model).
# Usage
Otherwise, follow the [Guix installation guide][guix/bin-install].
If you haven't considered your security model yet, please read [the relevant
section](#choosing-your-security-model) before proceeding to perform a build.
> Note: For those who like to keep their filesystems clean, Guix is designed to
> be very standalone and _will not_ conflict with your system's package
> manager/existing setup. It _only_ touches `/var/guix`, `/gnu`, and
> `~/.config/guix`.
## Making the Xcode SDK available for macOS cross-compilation
### Choosing your security model
In order to perform a build for macOS (which is included in the default set of
platform triples to build), you'll need to extract the macOS SDK tarball using
tools found in the [`macdeploy` directory](../macdeploy/README.md).
Guix allows us to achieve better binary security by using our CPU time to build
everything from scratch. However, it doesn't sacrifice user choice in pursuit of
this: users can decide whether or not to bootstrap and to use substitutes
(pre-built packages).
After installation, you may want to consider [adding substitute
servers](#speeding-up-builds-with-substitute-servers) from which to download
pre-built packages to speed up your build if that fits your security model (say,
if you're just testing that this works). Substitute servers are set up by
default if you're using the [Dockerfile][fanquake/guix-docker].
If you prefer not to use any substitutes, make sure to supply `--no-substitutes`
like in the following snippet. The first build will take a while, but the
resulting packages will be cached for future builds.
You can then either point to the SDK using the `SDK_PATH` environment variable:
```sh
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes'
# Extract the SDK tarball to /path/to/parent/dir/of/extracted/SDK/Xcode-<foo>-<bar>-extracted-SDK-with-libcxx-headers
tar -C /path/to/parent/dir/of/extracted/SDK -xaf /path/to/Xcode-<foo>-<bar>-extracted-SDK-with-libcxx-headers.tar.gz
# Indicate where to locate the SDK tarball
export SDK_PATH=/path/to/parent/dir/of/extracted/SDK
```
Likewise, to perform a bootstrapped build (takes even longer):
or extract it into `depends/SDKs`:
```sh
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes' ADDITIONAL_GUIX_ENVIRONMENT_FLAGS='--bootstrap'
mkdir -p depends/SDKs
tar -C depends/SDKs -xaf /path/to/SDK/tarball
```
### Using a version of Guix with `guix time-machine` capabilities
## Building
> Note: This entire section can be skipped if you are already using a version of
> Guix that has [the `guix time-machine` command][guix/time-machine].
*The author highly recommends at least reading over the [common usage patterns
and examples](#common-guix-build-invocation-patterns-and-examples) section below
before starting a build. For a full list of customization options, see the
[recognized environment variables][env-vars-list] section.*
Once Guix is installed, if it doesn't have the `guix time-machine` command, pull
the latest `guix`.
To build Bitcoin Core reproducibly with all default options, invoke the
following from the top of a clean repository:
```sh
guix pull --max-jobs=4 # change number of jobs accordingly
./contrib/guix/guix-build
```
Make sure that you are using your current profile. (You are prompted to do this
at the end of the `guix pull`)
## Codesigning build outputs
The `guix-codesign` command attaches codesignatures (produced by codesigners) to
existing non-codesigned outputs. Please see the [release process
documentation](/doc/release-process.md) for more context.
It respects many of the same environment variable flags as `guix-build`, with 2
crucial differences:
1. Since only Windows and macOS build outputs require codesigning, the `HOSTS`
environment variable will have a sane default value of `x86_64-w64-mingw32
x86_64-apple-darwin18` instead of all the platforms.
2. The `guix-codesign` command ***requires*** a `DETACHED_SIGS_REPO` flag.
* _**DETACHED_SIGS_REPO**_
Set the directory where detached codesignatures can be found for the current
Bitcoin Core version being built.
_REQUIRED environment variable_
An invocation with all default options would look like:
```bash
export PATH="${HOME}/.config/guix/current/bin${PATH:+:}$PATH"
```
env DETACHED_SIGS_REPO=<path/to/bitcoin-detached-sigs> ./contrib/guix-codesign
```
## Cleaning intermediate work directories
By default, `guix-build` leaves all intermediate files or "work directories"
(e.g. `depends/work`, `guix-build-*/distsrc-*`) intact at the end of a build so
that they are available to the user (to aid in debugging, etc.). However, these
directories usually take up a large amount of disk space. Therefore, a
`guix-clean` convenience script is provided which cleans the current `git`
worktree to save disk space:
```
./contrib/guix/guix-clean
```
## Attesting to build outputs
Much like how Gitian build outputs are attested to in a `gitian.sigs`
repository, Guix build outputs are attested to in the [`guix.sigs`
repository](https://github.com/bitcoin-core/guix.sigs).
After you've cloned the `guix.sigs` repository, to attest to the current
worktree's commit/tag:
```
env GUIX_SIGS_REPO=<path/to/guix.sigs> SIGNER=<gpg-key-name> ./contrib/guix/guix-attest
```
See `./contrib/guix/guix-attest --help` for more information on the various ways
`guix-attest` can be invoked.
## Verifying build output attestations
After at least one other signer has uploaded their signatures to the `guix.sigs`
repository:
```
git -C <path/to/guix.sigs> pull
env GUIX_SIGS_REPO=<path/to/guix.sigs> ./contrib/guix/guix-verify
```
## Common `guix-build` invocation patterns and examples
### Keeping caches and SDKs outside of the worktree
If you perform a lot of builds and have a bunch of worktrees, you may find it
more efficient to keep the depends tree's download cache, build cache, and SDKs
outside of the worktrees to avoid duplicate downloads and unnecessary builds. To
help with this situation, the `guix-build` script honours the `SOURCES_PATH`,
`BASE_CACHE`, and `SDK_PATH` environment variables and will pass them on to the
depends tree so that you can do something like:
```sh
env SOURCES_PATH="$HOME/depends-SOURCES_PATH" BASE_CACHE="$HOME/depends-BASE_CACHE" SDK_PATH="$HOME/macOS-SDKs" ./contrib/guix/guix-build
```
Note that the paths that these environment variables point to **must be
directories**, and **NOT symlinks to directories**.
See the [recognized environment variables][env-vars-list] section for more
details.
### Building a subset of platform triples
Sometimes you only want to build a subset of the supported platform triples, in
which case you can override the default list by setting the space-separated
`HOSTS` environment variable:
```sh
env HOSTS='x86_64-w64-mingw32 x86_64-apple-darwin18' ./contrib/guix/guix-build
```
See the [recognized environment variables][env-vars-list] section for more
details.
### Controlling the number of threads used by `guix` build commands
Depending on your system's RAM capacity, you may want to decrease the number of
threads used to decrease RAM usage or vice versa.
By default, the scripts under `./contrib/guix` will invoke all `guix` build
commands with `--cores="$JOBS"`. Note that `$JOBS` defaults to `$(nproc)` if not
specified. However, astute manual readers will also notice that there is a
`--max-jobs=` flag (which defaults to 1 if unspecified).
specified. However, astute manual readers will also notice that `guix` build
commands also accept a `--max-jobs=` flag (which defaults to 1 if unspecified).
Here is the difference between `--cores=` and `--max-jobs=`:
@ -124,30 +212,18 @@ packages when the dependency graph allows for it, you may want to try:
export JOBS=1 ADDITIONAL_GUIX_COMMON_FLAGS='--max-jobs=8'
```
## Usage
See the [recognized environment variables][env-vars-list] section for more
details.
### As a Tool for Deterministic Builds
From the top of a clean Bitcoin Core repository:
```sh
./contrib/guix/guix-build
```
After the build finishes successfully (check the status code please), compare
hashes:
```sh
find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
```
#### Recognized environment variables
## Recognized environment variables
* _**HOSTS**_
Override the space-separated list of platform triples for which to perform a
bootstrappable build. _(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf
aarch64-linux-gnu riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
bootstrappable build.
_(defaults to "x86\_64-linux-gnu arm-linux-gnueabihf aarch64-linux-gnu
riscv64-linux-gnu powerpc64-linux-gnu powerpc64le-linux-gnu
x86\_64-w64-mingw32 x86\_64-apple-darwin18")_
* _**SOURCES_PATH**_
@ -156,18 +232,27 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary redownloading of package sources.
The path that this environment variable points to **must be a directory**, and
**NOT a symlink to a directory**.
* _**BASE_CACHE**_
Set the depends tree cache for built packages. This is passed through to the
depends tree. Setting this to the same directory across multiple builds of the
depends tree can eliminate unnecessary building of packages.
The path that this environment variable points to **must be a directory**, and
**NOT a symlink to a directory**.
* _**SDK_PATH**_
Set the path where _extracted_ SDKs can be found. This is passed through to
the depends tree. Note that this is should be set to the _parent_ directory of
the actual SDK (e.g. SDK_PATH=$HOME/Downloads/macOS-SDKs instead of
$HOME/Downloads/macOS-SDKs/Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers).
the actual SDK (e.g. `SDK_PATH=$HOME/Downloads/macOS-SDKs` instead of
`$HOME/Downloads/macOS-SDKs/Xcode-12.1-12A7403-extracted-SDK-with-libcxx-headers`).
The path that this environment variable points to **must be a directory**, and
**NOT a symlink to a directory**.
* _**JOBS**_
@ -178,13 +263,17 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
- `make` as in `make --jobs="$JOBS"`
- `xargs` as in `xargs -P"$JOBS"`
See [here](#controlling-the-number-of-threads-used-by-guix-build-commands) for
more details.
_(defaults to the value of `nproc` outside the container)_
* _**SOURCE_DATE_EPOCH**_
Override the reference UNIX timestamp used for bit-for-bit reproducibility,
the variable name conforms to [standard][r12e/source-date-epoch]. _(defaults
to the output of `$(git log --format=%at -1)`)_
the variable name conforms to [standard][r12e/source-date-epoch].
_(defaults to the output of `$(git log --format=%at -1)`)_
* _**V**_
@ -200,8 +289,7 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
A whitespace-delimited list of URLs from which to download pre-built packages.
A URL is only used if its signing key is authorized (refer to the [substitute
servers section](#speeding-up-builds-with-substitute-servers) for more
details).
servers section](#option-1-building-with-substitutes) for more details).
* _**ADDITIONAL_GUIX_COMMON_FLAGS**_
@ -216,119 +304,169 @@ find output/ -type f -print0 | sort -z | xargs -r0 sha256sum
Additional flags to be passed to the invocation of `guix environment` inside
`guix time-machine`.
## Tips and Tricks
# Choosing your security model
### Speeding up builds with substitute servers
No matter how you installed Guix, you need to decide on your security model for
building packages with Guix.
_This whole section is automatically done in the convenience
[Dockerfiles][fanquake/guix-docker]_
Guix allows us to achieve better binary security by using our CPU time to build
everything from scratch. However, it doesn't sacrifice user choice in pursuit of
this: users can decide whether or not to use **substitutes** (pre-built
packages).
For those who are used to life in the fast _(and trustful)_ lane, you can
specify [substitute servers][guix/substitutes] from which to download pre-built
packages.
## Option 1: Building with substitutes
> For those who only want to use substitutes from the official Guix build farm
> and have authorized the build farm's signing key during Guix's installation,
> you don't need to do anything.
### Step 1: Authorize the signing keys
#### Step 1: Authorize the signing keys
Depending on the installation procedure you followed, you may have already
authorized the Guix build farm key. In particular, the official shell installer
script asks you if you want the key installed, and the debian distribution
package authorized the key during installation.
For the official Guix build farm at https://ci.guix.gnu.org, run as root:
You can check the current list of authorized keys at `/etc/guix/acl`.
At the time of writing, a `/etc/guix/acl` with just the Guix build farm key
authorized looks something like:
```lisp
(acl
(entry
(public-key
(ecc
(curve Ed25519)
(q #8D156F295D24B0D9A86FA5741A840FF2D24F60F7B6C4134814AD55625971B394#)
)
)
(tag
(guix import)
)
)
)
```
If you've determined that the official Guix build farm key hasn't been
authorized, and you would like to authorize it, run the following as root:
```
guix archive --authorize < ~root/.config/guix/current/share/guix/ci.guix.gnu.org.pub
guix archive --authorize < /var/guix/profiles/per-user/root/current-guix/share/guix/ci.guix.gnu.org.pub
```
If
`/var/guix/profiles/per-user/root/current-guix/share/guix/ci.guix.gnu.org.pub`
doesn't exist, try:
```sh
guix archive --authorize < <PREFIX>/share/guix/ci.guix.gnu.org.pub
```
Where `<PREFIX>` is likely:
- `/usr` if you installed from a distribution package
- `/usr/local` if you installed Guix from source and didn't supply any
prefix-modifying flags to Guix's `./configure`
For dongcarl's substitute server at https://guix.carldong.io, run as root:
```sh
wget -qO- 'https://guix.carldong.io/signing-key.pub' | guix archive --authorize
```
#### Step 2: Specify the substitute servers
#### Removing authorized keys
The official Guix build farm at https://ci.guix.gnu.org is automatically used
unless the `--no-substitutes` flag is supplied.
To remove previously authorized keys, simply edit `/etc/guix/acl` and remove the
`(entry (public-key ...))` entry.
This can be overridden for all `guix` invocations by passing the
`--substitute-urls` option to your invocation of `guix-daemon`. This can also be
overridden on a call-by-call basis by passing the same `--substitute-urls`
option to client tools such at `guix environment`.
### Step 2: Specify the substitute servers
To use dongcarl's substitute server for Bitcoin Core builds after having
[authorized his signing key](#authorize-the-signing-keys):
Once its key is authorized, the official Guix build farm at
https://ci.guix.gnu.org is automatically used unless the `--no-substitutes` flag
is supplied. This default list of substitute servers is overridable both on a
`guix-daemon` level and when you invoke `guix` commands. See examples below for
the various ways of adding dongcarl's substitute server after having [authorized
his signing key](#authorize-the-signing-keys).
Change the **default list** of substitute servers by starting `guix-daemon` with
the `--substitute-urls` option (you will likely need to edit your init script):
```sh
guix-daemon <cmd> --substitute-urls='https://guix.carldong.io https://ci.guix.gnu.org'
```
Override the default list of substitute servers by passing the
`--substitute-urls` option for invocations of `guix` commands:
```sh
guix <cmd> --substitute-urls='https://guix.carldong.io https://ci.guix.gnu.org'
```
For scripts under `./contrib/guix`, set the `SUBSTITUTE_URLS` environment
variable:
```sh
export SUBSTITUTE_URLS='https://guix.carldong.io https://ci.guix.gnu.org'
```
## Troubleshooting
## Option 2: Disabling substitutes on an ad-hoc basis
### Derivation failed to build
When you see a build failure like below:
```
building /gnu/store/...-foo-3.6.12.drv...
/ 'check' phasenote: keeping build directory `/tmp/guix-build-foo-3.6.12.drv-0'
builder for `/gnu/store/...-foo-3.6.12.drv' failed with exit code 1
build of /gnu/store/...-foo-3.6.12.drv failed
View build log at '/var/log/guix/drvs/../...-foo-3.6.12.drv.bz2'.
cannot build derivation `/gnu/store/...-qux-7.69.1.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-bar-3.16.5.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/...-baz-2.0.5.drv': 1 dependencies couldn't be built
guix time-machine: error: build of `/gnu/store/...-baz-2.0.5.drv' failed
```
It means that `guix` failed to build a package named `foo`, which was a
dependency of `qux`, `bar`, and `baz`. Importantly, note that the last "failed"
line is not necessarily the root cause, the first "failed" line is.
Most of the time, the build failure is due to a spurious test failure or the
package's build system/test suite breaking when running multi-threaded. To
rebuild _just_ this derivation in a single-threaded fashion:
If you prefer not to use any substitutes, make sure to supply `--no-substitutes`
like in the following snippet. The first build will take a while, but the
resulting packages will be cached for future builds.
For direct invocations of `guix`:
```sh
$ guix build --cores=1 /gnu/store/...-foo-3.6.12.drv
guix <cmd> --no-substitutes
```
If the single-threaded rebuild did not succeed, you may need to dig deeper.
You may view `foo`'s build logs in `less` like so (please replace paths with the
path you see in the build failure output):
For the scripts under `./contrib/guix/`:
```sh
$ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
export ADDITIONAL_GUIX_COMMON_FLAGS='--no-substitutes'
```
`foo`'s build directory is also preserved and available at
`/tmp/guix-build-foo-3.6.12.drv-0`. However, if you fail to build `foo` multiple
times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
failure output for the most accurate, up-to-date information.
## Option 3: Disabling substitutes by default
#### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
`guix-daemon` accepts a `--no-substitutes` flag, which will make sure that,
unless otherwise overridden by a command line invocation, no substitutes will be
used.
This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
which rejects characters not present in the UTF-8 character code set. An example
is ZFS with the utf8only=on option set.
If you start `guix-daemon` using an init script, you can edit said script to
supply this flag.
More information: https://bugs.python.org/issue37584
## FAQ
# Purging/Uninstalling Guix
### How can I trust the binary installation?
In the extraordinarily rare case where you messed up your Guix installation in
an irreversible way, you may want to completely purge Guix from your system and
start over.
As mentioned at the bottom of [this manual page][guix/bin-install]:
1. Uninstall Guix itself according to the way you installed it. (e.g. `sudo apt
purge guix` for Ubuntu packaging, `sudo make uninstall` for
built-from-source).
2. Remove all build users and groups
> The binary installation tarballs can be (re)produced and verified simply by
> running the following command in the Guix source tree:
>
> make guix-binary.x86_64-linux.tar.xz
You may check for relevant users and groups using:
### Is Guix packaged in my operating system?
```
getent passwd | grep guix
getent group | grep guix
```
Guix is shipped starting with [Debian Bullseye][debian/guix-bullseye] and
[Ubuntu 21.04 "Hirsute Hippo"][ubuntu/guix-hirsute]. Other operating systems
are working on packaging Guix as well.
Then, you may remove users and groups using:
```
sudo userdel <user>
sudo groupdel <group>
```
3. Remove all possible Guix-related directories
- `/var/guix/`
- `/var/log/guix/`
- `/gnu/`
- `/etc/guix/`
- `/home/*/.config/guix/`
- `/home/*/.cache/guix/`
- `/home/*/.guix-profile/`
- `/root/.config/guix/`
- `/root/.cache/guix/`
- `/root/.guix-profile/`
[b17e]: http://bootstrappable.org/
[r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
@ -343,3 +481,5 @@ are working on packaging Guix as well.
[debian/guix-bullseye]: https://packages.debian.org/bullseye/guix
[ubuntu/guix-hirsute]: https://packages.ubuntu.com/hirsute/guix
[fanquake/guix-docker]: https://github.com/fanquake/core-review/tree/master/guix
[env-vars-list]: #recognized-environment-variables

View File

@ -18,7 +18,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
# Required non-builtin commands should be invokable
################
check_tools cat env basename mkdir xargs find
check_tools cat env basename mkdir diff sort
if [ -z "$NO_SIGN" ]; then
check_tools gpg
fi
@ -159,6 +159,20 @@ Hint: You may wish to remove the existing attestations and their signatures by
EOF
}
# Given a document with unix line endings (just <LF>) in stdin, make all lines
# end in <CR><LF> and make sure there's no trailing <LF> at the end of the file.
#
# This is necessary as cleartext signatures are calculated on text after their
# line endings are canonicalized.
#
# For more information:
# 1. https://security.stackexchange.com/a/104261
# 2. https://datatracker.ietf.org/doc/html/rfc4880#section-7.1
#
rfc4880_normalize_document() {
sed 's/$/\r/' | head -c -2
}
echo "Attesting to build outputs for version: '${VERSION}'"
echo ""
@ -174,7 +188,8 @@ mkdir -p "$outsigdir"
cat "${noncodesigned_fragments[@]}" \
| sort -u \
| sort -k2 \
> "$temp_noncodesigned"
| rfc4880_normalize_document \
> "$temp_noncodesigned"
if [ -e noncodesigned.SHA256SUMS ]; then
# The SHA256SUMS already exists, make sure it's exactly what we
# expect, error out if not
@ -201,7 +216,9 @@ mkdir -p "$outsigdir"
cat "${sha256sum_fragments[@]}" \
| sort -u \
| sort -k2 \
> "$temp_codesigned"
| sed 's/$/\r/' \
| rfc4880_normalize_document \
> "$temp_codesigned"
if [ -e codesigned.SHA256SUMS ]; then
# The SHA256SUMS already exists, make sure it's exactly what we
# expect, error out if not
@ -226,6 +243,7 @@ mkdir -p "$outsigdir"
for i in *.SHA256SUMS; do
if [ ! -e "$i".asc ]; then
gpg --detach-sign \
--digest-algo sha256 \
--local-user "$gpg_key_name" \
--armor \
--output "$i".asc "$i"

View File

@ -18,7 +18,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/libexec/prelude.bash"
# Required non-builtin commands should be invocable
################
check_tools cat mkdir make git guix
check_tools cat mkdir make getent curl git guix
################
# GUIX_BUILD_OPTIONS should be empty
@ -186,6 +186,29 @@ fi
#
# However, the internal API is likely to change more than the CLI invocation
################
# Services database must have basic entries
################
if ! getent services http https ftp; then
cat << EOF
ERR: Your system's C library can not find service database entries for at least
one of the following services: http, https, ftp.
Hint: Most likely, /etc/services does not exist yet (common for docker images
and minimal distros), or you don't have permissions to access it.
If /etc/services does not exist yet, you may want to install the
appropriate package for your distro which provides it.
On Debian/Ubuntu: netbase
On Arch Linux: iana-etc
For more information, see: getent(1), services(5)
EOF
fi
#########
# SETUP #

View File

@ -24,18 +24,17 @@ Before every major release:
* Update [`src/chainparams.cpp`](/src/chainparams.cpp) m_assumed_blockchain_size and m_assumed_chain_state_size with the current size plus some overhead (see [this](#how-to-calculate-m_assumed_blockchain_size-and-m_assumed_chain_state_size) for information on how to calculate them).
* Update `src/chainparams.cpp` chainTxData with statistics about the transaction count and rate. Use the output of the RPC `getchaintxstats`, see
[this pull request](https://github.com/bitcoin/bitcoin/pull/12270) for an example. Reviewers can verify the results by running `getchaintxstats <window_block_count> <window_last_block_hash>` with the `window_block_count` and `window_last_block_hash` from your output.
* Update version of `contrib/gitian-descriptors/*.yml`: usually one'd want to do this on master after branching off the release - but be sure to at least do it before a new major release
### First time / New builders
If you're using the automated script (found in [contrib/gitian-build.py](/contrib/gitian-build.py)), then at this point you should run it with the "--setup" command. Otherwise ignore this.
Install Guix using one of the installation methods detailed in
[contrib/guix/INSTALL.md](/contrib/guix/INSTALL.md).
Check out the source code in the following directory hierarchy.
cd /path/to/your/toplevel/build
git clone https://github.com/dashpay/gitian.sigs.git
git clone https://github.com/dashpay/guix.sigs.git
git clone https://github.com/dashpay/dash-detached-sigs.git
git clone https://github.com/devrandom/gitian-builder.git
git clone https://github.com/dashpay/dash.git
### Dash Core maintainers/release engineers, suggestion for writing release notes
@ -52,114 +51,56 @@ Tag version (or release candidate) in git
git tag -s v(new version, e.g. 0.12.3)
### Setup and perform Gitian builds
### Setup and perform Guix builds
If you're using the automated script (found in [contrib/gitian-build.py](/contrib/gitian-build.py)), then at this point you should run it with the "--build" command. Otherwise ignore this.
Checkout the Dash Core version you'd like to build:
Setup Gitian descriptors:
```sh
pushd ./dash
export SIGNER='(your builder key, ie UdjinM6, Pasta, etc)'
export VERSION='(new version, e.g. 20.0.0)'
git fetch "v${VERSION}"
git checkout "v${VERSION}"
popd
```
pushd ./dash
export SIGNER="(your Gitian key, ie UdjinM6, Pasta, etc)"
export VERSION=(new version, e.g. 0.12.3)
git fetch
git checkout v${VERSION}
popd
Ensure your guix.sigs are up-to-date if you wish to `guix-verify` your builds
against other `guix-attest` signatures.
Ensure your gitian.sigs are up-to-date if you wish to gverify your builds against other Gitian signatures.
```sh
git -C ./guix.sigs pull
```
pushd ./gitian.sigs
git pull
popd
### Create the macOS SDK tarball: (first time, or when SDK version changes)
Ensure gitian-builder is up-to-date:
Create the macOS SDK tarball, see the [macOS build
instructions](build-osx.md#deterministic-macos-dmg-notes) for
details.
pushd ./gitian-builder
git pull
popd
### Build and attest to build outputs:
Follow the relevant Guix README.md sections:
- [Performing a build](/contrib/guix/README.md#performing-a-build)
- [Attesting to build outputs](/contrib/guix/README.md#attesting-to-build-outputs)
### Fetch and create inputs: (first time, or when dependency versions change)
### Verify other builders' signatures to your own. (Optional)
pushd ./gitian-builder
mkdir -p inputs
wget -O inputs/osslsigncode-2.0.tar.gz https://github.com/mtrojnar/osslsigncode/archive/2.0.tar.gz
echo '5a60e0a4b3e0b4d655317b2f12a810211c50242138322b16e7e01c6fbb89d92f inputs/osslsigncode-2.0.tar.gz' | sha256sum -c
popd
Add other builders keys to your gpg keyring, and/or refresh keys: See `../dash/contrib/builder-keys/README.md`.
Create the macOS SDK tarball, see the [macOS build instructions](build-osx.md#deterministic-macos-dmg-notes) for details, and copy it into the inputs directory.
### Optional: Seed the Gitian sources cache and offline git repositories
NOTE: Gitian is sometimes unable to download files. If you have errors, try the step below.
By default, Gitian will fetch source files as needed. To cache them ahead of time, make sure you have checked out the tag you want to build in dash, then:
pushd ./gitian-builder
make -C ../dash/depends download SOURCES_PATH=`pwd`/cache/common
popd
Only missing files will be fetched, so this is safe to re-run for each build.
NOTE: Offline builds must use the --url flag to ensure Gitian fetches only from local URLs. For example:
pushd ./gitian-builder
./bin/gbuild --url dash=/path/to/dash,signature=/path/to/sigs {rest of arguments}
popd
The gbuild invocations below <b>DO NOT DO THIS</b> by default.
### Build and sign Dash Core for Linux, Windows, and macOS:
pushd ./gitian-builder
./bin/gbuild --num-make 2 --memory 3000 --commit dash=v${VERSION} ../dash/contrib/gitian-descriptors/gitian-linux.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-linux --destination ../gitian.sigs/ ../dash/contrib/gitian-descriptors/gitian-linux.yml
mv build/out/dash-*.tar.gz build/out/src/dash-*.tar.gz ../
./bin/gbuild --num-make 2 --memory 3000 --commit dash=v${VERSION} ../dash/contrib/gitian-descriptors/gitian-win.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-win-unsigned --destination ../gitian.sigs/ ../dash/contrib/gitian-descriptors/gitian-win.yml
mv build/out/dash-*-win-unsigned.tar.gz inputs/dash-win-unsigned.tar.gz
mv build/out/dash-*.zip build/out/dash-*.exe ../
./bin/gbuild --num-make 2 --memory 3000 --commit dash=v${VERSION} ../dash/contrib/gitian-descriptors/gitian-osx.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-unsigned --destination ../gitian.sigs/ ../dash/contrib/gitian-descriptors/gitian-osx.yml
mv build/out/dash-*-osx-unsigned.tar.gz inputs/dash-osx-unsigned.tar.gz
mv build/out/dash-*.tar.gz build/out/dash-*.dmg ../
popd
Build output expected:
1. source tarball (`dash-${VERSION}.tar.gz`)
2. linux 32-bit and 64-bit dist tarballs (`dash-${VERSION}-linux[32|64].tar.gz`)
3. windows 32-bit and 64-bit unsigned installers and dist zips (`dash-${VERSION}-win[32|64]-setup-unsigned.exe`, `dash-${VERSION}-win[32|64].zip`)
4. macOS unsigned installer and dist tarball (`dash-${VERSION}-osx-unsigned.dmg`, `dash-${VERSION}-osx64.tar.gz`)
5. Gitian signatures (in `gitian.sigs/${VERSION}-<linux|{win,osx}-unsigned>/(your Gitian key)/`)
### Verify other gitian builders signatures to your own. (Optional)
Add other gitian builders keys to your gpg keyring, and/or refresh keys.
gpg --import dash/contrib/gitian-keys/*.pgp
gpg --refresh-keys
Verify the signatures
pushd ./gitian-builder
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-linux ../dash/contrib/gitian-descriptors/gitian-linux.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-unsigned ../dash/contrib/gitian-descriptors/gitian-win.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-unsigned ../dash/contrib/gitian-descriptors/gitian-osx.yml
popd
Follow the relevant Guix README.md sections:
- [Verifying build output attestations](/contrib/guix/README.md#verifying-build-output-attestations)
### Next steps:
Commit your signature to gitian.sigs:
Commit your signature to guix.sigs:
pushd gitian.sigs
git add ${VERSION}-linux/"${SIGNER}"
git add ${VERSION}-win-unsigned/"${SIGNER}"
git add ${VERSION}-osx-unsigned/"${SIGNER}"
git commit -a
git push # Assuming you can push to the gitian.sigs tree
popd
```sh
pushd guix.sigs
git add "${VERSION}/${SIGNER}/noncodesigned.SHA256SUMS{,.asc}"
git commit -a
git push # Assuming you can push to the guix.sigs tree
popd
```
Codesigner only: Create Windows/macOS detached signatures:
- Only one person handles codesigning. Everyone else should skip to the next step.
@ -171,7 +112,7 @@ Codesigner only: Sign the macOS binary:
tar xf dashcore-osx-unsigned.tar.gz
./detached-sig-create.sh -s "Key ID" -o runtime
Enter the keychain password and authorize the signature
Move signature-osx.tar.gz back to the gitian host
Move signature-osx.tar.gz back to the guix-build host
Codesigner only: Sign the windows binaries:
@ -182,84 +123,69 @@ Codesigner only: Sign the windows binaries:
Codesigner only: Commit the detached codesign payloads:
cd ~/dashcore-detached-sigs
checkout the appropriate branch for this release series
rm -rf *
tar xf signature-osx.tar.gz
tar xf signature-win.tar.gz
git add -A
git commit -m "point to ${VERSION}"
git tag -s v${VERSION} HEAD
git push the current branch and new tag
```sh
pushd ~/dashcore-detached-sigs
# checkout the appropriate branch for this release series
rm -rf *
tar xf signature-osx.tar.gz
tar xf signature-win.tar.gz
git add -A
git commit -m "point to ${VERSION}"
git tag -s "v${VERSION}" HEAD
git push the current branch and new tag
popd
```
Non-codesigners: wait for Windows/macOS detached signatures:
- Once the Windows/macOS builds each have 3 matching signatures, they will be signed with their respective release keys.
- Detached signatures will then be committed to the [dash-detached-sigs](https://github.com/dashpay/dash-detached-sigs) repository, which can be combined with the unsigned apps to create signed binaries.
Create (and optionally verify) the signed macOS binary:
pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../dash/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-osx-signed --destination ../gitian.sigs/ ../dash/contrib/gitian-descriptors/gitian-osx-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-osx-signed ../dash/contrib/gitian-descriptors/gitian-osx-signer.yml
mv build/out/dash-osx-signed.dmg ../dash-${VERSION}-osx.dmg
popd
Create (and optionally verify) the signed Windows binaries:
pushd ./gitian-builder
./bin/gbuild -i --commit signature=v${VERSION} ../dash/contrib/gitian-descriptors/gitian-win-signer.yml
./bin/gsign --signer "$SIGNER" --release ${VERSION}-win-signed --destination ../gitian.sigs/ ../dash/contrib/gitian-descriptors/gitian-win-signer.yml
./bin/gverify -v -d ../gitian.sigs/ -r ${VERSION}-win-signed ../dash/contrib/gitian-descriptors/gitian-win-signer.yml
mv build/out/dash-*win64-setup.exe ../dash-${VERSION}-win64-setup.exe
popd
Create (and optionally verify) the codesigned outputs:
- [Codesigning](/contrib/guix/README.md#codesigning)
Commit your signature for the signed macOS/Windows binaries:
pushd gitian.sigs
git add ${VERSION}-osx-signed/"${SIGNER}"
git add ${VERSION}-win-signed/"${SIGNER}"
git commit -m "Add ${SIGNER} ${VERSION} signed binaries signatures"
git push # Assuming you can push to the gitian.sigs tree
popd
```sh
pushd ./guix.sigs
git add "${VERSION}/${SIGNER}"/all.SHA256SUMS{,.asc}
git commit -m "Add ${SIGNER} ${VERSION} signed binaries signatures"
git push # Assuming you can push to the guix.sigs tree
popd
```
### After 3 or more people have gitian-built and their results match:
### After 3 or more people have guix-built and their results match:
- Create `SHA256SUMS.asc` for the builds, and GPG-sign it:
Combine `all.SHA256SUMS` and `all.SHA256SUMS.asc` into a clear-signed
`SHA256SUMS.asc` message:
```sh
echo -e "-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA256\n\n$(cat all.SHA256SUMS)\n$(cat filename.txt.asc)" > SHA256SUMS.asc
```
Here's an equivalent, more readable command if you're confident that you won't
mess up whitespaces when copy-pasting:
```bash
sha256sum * > SHA256SUMS
cat << EOF > SHA256SUMS.asc
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
$(cat all.SHA256SUMS)
$(cat all.SHA256SUMS.asc)
EOF
```
The list of files should be:
```
dash-${VERSION}-aarch64-linux-gnu.tar.gz
dash-${VERSION}-riscv64-linux-gnu.tar.gz
dash-${VERSION}-x86_64-linux-gnu.tar.gz
dash-${VERSION}-osx64.tar.gz
dash-${VERSION}-osx.dmg
dash-${VERSION}.tar.gz
dash-${VERSION}-win64-setup.exe
dash-${VERSION}-win64.zip
```
The `*-debug*` files generated by the Gitian build contain debug symbols
for troubleshooting by developers. It is assumed that anyone that is interested
in debugging can run Gitian to generate the files for themselves. To avoid
end-user confusion about which file to pick, as well as save storage
space *do not upload these to the dash.org server*.
- Upload to the dash.org server:
1. The contents of /dash/guix-build-${VERSION}/output`, except for
`*-debug*` files.
- GPG-sign it, delete the unsigned file:
```
gpg --digest-algo sha256 --clearsign SHA256SUMS # outputs SHA256SUMS.asc
rm SHA256SUMS
```
(the digest algorithm is forced to sha256 to avoid confusion of the `Hash:` header that GPG adds with the SHA256 used for the files)
Note: check that SHA256SUMS itself doesn't end up in SHA256SUMS, which is a spurious/nonsensical entry.
- Upload zips and installers, as well as `SHA256SUMS.asc` from last step, to the dash.org server
- Update dash.org
The `*-debug*` files generated by the guix build contain debug symbols
for troubleshooting by developers. It is assumed that anyone that is
interested in debugging can run guix to generate the files for
themselves. To avoid end-user confusion about which file to pick, as well
as save storage space *do not upload these to the dash.org server*.
2. The combined clear-signed message you just created `SHA256SUMS.asc`
- Announce the release: