fac4814106c796b8786dd90053513cc35142dfe5 doc/release-process: Add torrent creation details (Carl Dong)
5d24cc3d82dad6812f8370c3ccc7c2b5a6c12c11 guix/INSTALL: Guix installs init scripts in libdir (Carl Dong)
5da2ee49d5b44de803b671aedbdd14e5c1d71ea9 guix/INSTALL: Add coreutils/inotify-dir-recreate troubleshooting (Carl Dong)
318c60700b7bbb7ec09a29bf037e7c2787646be6 guix: Adapt release-process.md to new Guix process (Carl Dong)
fcab35b2292f9221eaba521740e8b3b2511a8b78 guix-attest: Produce and sign normalized documents (Carl Dong)
c2541fd0ca99481a5a792a8f2772925d64fb0491 guix: Overhaul README (Carl Dong)
46ce6ce3782dfbd8f9d26dc2ba0f284755e75f2d tree-wide: Rename gitian-keys to builder-keys (Carl Dong)
fc4f8449f34e32b0b9ac9d218d6c3264b02467ba guix: Update various check_tools lists (Carl Dong)
263220a85c1df218431fafbda07c8b23ccc4ce4d guix: Check for a sane services database (Carl Dong)
Pull request description:
Based on: #21462
Keeping the README in one file so that it's easy to search through. Will add more jumping links later so navigation is easier.
Current TODOs:
- [x] Shell installer option: prompt user to re-login for `/etc/profile.d` entry to be picked up
- [x] Binary tarball option: prompt user to create `/etc/profile.d` entry and re-login
- [x] Fanquake docker option: complete section
- [x] Arch Linux AUR option: prompt to start `guix-daemon-latest` unit after finishing "optional setup" section
- [x] Building from source option: Insert dependency tree diagram that I made
- [x] Building from source option: redo sectioning, kind of a mess right now
- [x] Optional setup: make clear which parts are only needed if building from source
- [x] Workaround 1 for GnuTLS: perhaps mention how to remove Guix build farm's key
- [x] Overall (after everything): Make the links work.
Note to self: wherever possible, tell user how to check that something is true rather than branching by installation option.
ACKs for top commit:
fanquake:
ACK fac4814106c796b8786dd90053513cc35142dfe5 - going to go ahead and merge this now. It's a lot of documentation, and could probably be nit-picked / improved further, however, that can continue over the next few weeks. I'm sure more (backportable) improvements / clarifications will be made while we progress through RCs towards a new release.
Tree-SHA512: dc46c0ecdfc67c7c7743ca26e4a603eb3f54adbf81be2f4c1f4c20577ebb84b5250b9c9ec89c0e9860337ab1c7cff94d7963c603287267deecfe1cd987fa070a
a5550f877a2c46d01bb620ae051c0c8ed0fecd0b build: use -stdlib++-isystem with Clang 10 (fanquake)
51d9d1607f2e9d593693ca799393f068192e41aa guix: use Clang 10 for the macOS cross compile (fanquake)
b80a6af9e55325d444e117e85bbfc76d88d898a8 build: no longer patch threading out of ld64 (fanquake)
c29cba44b3706e0a2035e440e560f2d15d50433b build: Xcode 12.1, macOS SDK 10.15.6 (fanquake)
9ed2f19d385aa95f65807999bba2e18417b143dc build: native cctools 973.0.1, ld64 609 (fanquake)
f48f187cce7fa43646fb0d796c244e1515e763ec build: Clang 10.0.1 (Hennadii Stepanov)
9b193cd2a3ca20917611fbed56dfbcd8a39aeab8 build: libtapi 1100.0.11 (fanquake)
Pull request description:
Bumps our macOS toolchain to be using the following:
* Clang 10.0.1 (gitian) & Clang 10.0.0 (Guix)
* ld64 609
* libtapi 1100.0.11
* cctools 973.0.1
* Xcode 12.1
* macOS SDK 10.15.6
which are currently the most recent releases available as open source. See upstream [`cctools`](https://github.com/tpoechtrager/cctools-port/commits/973.0.1-ld64-609) and [`libtapi`](https://github.com/tpoechtrager/apple-libtapi/tree/1100.0.11).
This should improve the possibility of Apple ARM cross-compilation in depends.
This also removes our [patching out of pthreads usage](https://github.com/bitcoin/bitcoin/blob/master/depends/patches/native_cctools/ld64_disable_threading.patch) in `ld64`. There have been multiple changes since `ld64 450.3`, which have likely fixed the non-determinism we were working around. i.e from [InputFiles.cpp](https://opensource.apple.com/source/ld64/ld64-609/src/ld/InputFiles.cpp.auto.html):
```cpp
// <rdar://problem/15002251> make implicit dylib order be deterministic by sorting by install_name
std::sort(implicitDylibs.begin(), implicitDylibs.end(), DylibByInstallNameSorter());
```
```cpp
// <rdar://problem/42675402> ld64 output is not deterministic due to dylib processing order
std::sort(unprocessedDylibs.begin(), unprocessedDylibs.end(), [](const ld::dylib::File* lhs, const ld::dylib::File* rhs) {
return strcmp(lhs->path(), rhs->path()) < 0;
});
```
Guix Build:
```bash
find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
f6c3817b8fe5f7370299d1ae2533e4a3acd313ba9f9aa8d423a8956117e52dd5 guix-build-a5550f877a2c/output/dist-archive/bitcoin-a5550f877a2c.tar.gz
4954dcf563c2d496b8d9fecd48f8e3f7fba2f319ffa254a5bc8ee12cfee6acf0 guix-build-a5550f877a2c/output/x86_64-apple-darwin18/bitcoin-a5550f877a2c-osx-unsigned.dmg
8f6095b445c7f1a8e6accd86bb7f0696d5849402084927d2b726b7d557831c3a guix-build-a5550f877a2c/output/x86_64-apple-darwin18/bitcoin-a5550f877a2c-osx-unsigned.tar.gz
cc40f25477b4defc1617ae694313d80f307ddf6742fe6cc85c6bc0e215ef8be0 guix-build-a5550f877a2c/output/x86_64-apple-darwin18/bitcoin-a5550f877a2c-osx64.tar.gz
```
Gitian Build:
```bash
Generating report
506a8abdefe559999b43dd9f14905b9b2b5a3363b1cd013d45ae47acc1f7ef6c bitcoin-a5550f877a2c-osx-unsigned.dmg
f606997f74026dd12d110d683c6f116b40df324836904ef507dd7ac787e6ebe2 bitcoin-a5550f877a2c-osx-unsigned.tar.gz
5b495ef15f2c3260c2950921b61326912a9bf533cccd51e13818809fd225489e bitcoin-a5550f877a2c-osx64.tar.gz
f6c3817b8fe5f7370299d1ae2533e4a3acd313ba9f9aa8d423a8956117e52dd5 src/bitcoin-a5550f877a2c.tar.gz
9eb0221e962d2839770963bd03c6c9e98e8bf3078566bee2ae42f06233a710fa bitcoin-core-osx-22-res.yml
Done.
```
ACKs for top commit:
hebasto:
ACK a5550f877a2c46d01bb620ae051c0c8ed0fecd0b
Tree-SHA512: 504c4b0f9cd3b939714a322298320c5bde07e9356a48a9a000060b36f8dce4d6134ed60c3a5188810476a28ec5b108733eabbc6fb8053231b9ea8a494cc91b12
09b3e468480538977d2844ba083c66197c9d1d08 doc: remove boostrap info from GUIX_COMMON_FLAGS doc (fanquake)
Pull request description:
Passing `ADDITIONAL_GUIX_COMMON_FLAGS="--no-substitutes --bootstrap"` as suggested doesn't work:
```bash
...outputting in: '/bitcoin/guix-build-a1f0b8b62eb8/output/x86_64-linux-gnu'
...bind-mounted in container to: '/outdir-base/x86_64-linux-gnu'
guix time-machine: error: bootstrap: unrecognized option
```
and I think bootstrapping is more than covered in the preceding "Choose your security model" section.
ACKs for top commit:
dongcarl:
ACK 09b3e468480538977d2844ba083c66197c9d1d08
jarolrod:
ACK 09b3e468480538977d2844ba083c66197c9d1d08
laanwj:
Documentation review ACK 09b3e468480538977d2844ba083c66197c9d1d08
Tree-SHA512: e533a0b925a2ec091884ec04313f56376b4e85f615c8853dd51840181493d61bd01fee1c019e34880c32d7896e4871ea77e63398ba02b022f614c03bb16531aa
7476b46f1893a4858616d2a8456a7c43238851ed guix: Build dmg as a static binary (Carl Dong)
06d6cf6784421290e6235fe8684d5e08ed6f1b62 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong)
65176ab5730dff34466caaecdd292625ef8294fc guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong)
ca85679eb43b8375a95d82101977829d08fb1e1b guix: Use clang-toolchain instead of clang (Carl Dong)
1aec0eda8fd31a57b0621eea616398017c2ead98 guix: Fallback to local build for substitute-enabled Guix users (Carl Dong)
1742f8e12d163852df09575e03edcd3db73198ee guix: Add early health check for guix-daemon (Carl Dong)
c1ae726a13ecfa5e7e9fdc3030a8110b8bb263f8 guix: More thoroughly control native toolchain (Carl Dong)
39741128d3775d198dbee34dc827353bfd18acd8 guix: Supply --link-profile (Carl Dong)
d55a1056ee565afed64e42d6f6efb6b0adc5599b guix: Add troubleshooting documentation entries (Carl Dong)
7f401c953f8bb3574cec48561e13ef3b47dedc6e guix: Adapt guix-build to prelude, restructure hier (Carl Dong)
4eccf063b252bfe256cf72d363a24cf0183e926e guix: Remove guix-build.sh filename extension (Carl Dong)
7753357a7bae98ec775c707b9dec4cea1e945802 guix: Add source-able bash prelude and utils (Carl Dong)
e5b49a01f5d0f631e7f08f86ca8a2c2b8213319f guix: Create windeploy inside distsrc-* (Carl Dong)
3e9982ab3877eb8fe0a8c0cb3d847ac0913c7336 contrib: Silence git-describe when looking for tag (Carl Dong)
d5a71e97853ea9e1b879e8c76bfb01d4bef33172 guix: Use --cores instead of --max-jobs (Carl Dong)
Pull request description:
This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts:
- Resolves confusion between `--cores=` and `--max-jobs=`
- `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var
- `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that
- `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people)
- Add troubleshooting documentation to `README.md`
- Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon`
- Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally
- `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple`
A few robustness changes are also included:
- We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile`
- We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug
- We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages.
Finally, we restructure the guix building hierarchy such that it looks something like:
```
guix-build-<short-hash-or-version-tag>
├── distsrc-<short-hash-or-version-tag>-${HOST}
│ ├── contrib
│ ├── depends
│ ├── src
│ └── ...
├── distsrc-<short-hash-or-version-tag>-...
└── output
├── dist-archive
│ └── bitcoin-<short-hash-or-version-tag>.tar.gz
├── *-linux-*
│ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz
│ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz
├── x86_64-apple-darwin18
│ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz
│ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg
│ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz
└── x86_64-w64-mingw32
├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip
├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe
├── bitcoin-<short-hash-or-version-tag>-win64.zip
└── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz
```
Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts.
ACKs for top commit:
sipa:
ACK 7476b46f1893a4858616d2a8456a7c43238851ed
laanwj:
ACK 7476b46f1893a4858616d2a8456a7c43238851ed
Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
a0a7a4337d06553ab625bbd66ed4198c4bf7f18c guix, doc: Update default HOSTS value (Hennadii Stepanov)
Pull request description:
This is a #21089 follow up.
ACKs for top commit:
fanquake:
ACK a0a7a4337d06553ab625bbd66ed4198c4bf7f18c
Tree-SHA512: c1813cc2b9212a79fd34d4e25cd0816b58264e1890daf777cd59411bd20fcc9affe312871d06fab1308b8f55c1a78ac1101e631882c18360a4709ecef4529f05
13a9fd11a507fd3398bc2c0a0575bdc81579243f guix: Passthrough SDK_PATH into container (Carl Dong)
Pull request description:
This is a usability improvement for Guix builders so that they don't have to extract the Xcode tarball into `depends/SDKs` every time.
Inspiration: https://github.com/bitcoin/bitcoin/pull/21089#issuecomment-778639698
ACKs for top commit:
laanwj:
Tested ACK 13a9fd11a507fd3398bc2c0a0575bdc81579243f
Tree-SHA512: 63392d537e48a0da9f0ee04a929613b139bef1ac5643187871c9ea5376afd2a3d95df0f5e0950ae0eccd2813b166667be98401e5a248ae9c187fe4e84e54d427
95990b9f3278360b63e79d6975af4ab5009c66ba guix: Update conservative space requirements (Carl Dong)
5e6df1132656995ce5b9ce279d5a9808ea52ab32 guix: Add support for powerpc64{,le} (Carl Dong)
Pull request description:
```
The new time-machine commit contains a few small changes that make the
powerpc cross-toolchain work.
```
See this compare to review my custom patches to Guix: 7d6bd44da5...6c9d16db96
ACKs for top commit:
fanquake:
ACK 95990b9f3278360b63e79d6975af4ab5009c66ba
Tree-SHA512: 464b0fb93d65962d8c27499293edb618d13d18f40d44e3eed96935e86d430666dfb1c5b8a30f99ffdfd17b44514ad88e358977390b689a2e3831d521f6f7b86a
fa051c23860bcdcc871db5ad6b51b8d9ca88da35 doc: Guix is shipped in Debian and Ubuntu (MarcoFalke)
Pull request description:
ACKs for top commit:
practicalswift:
ACK fa051c23860bcdcc871db5ad6b51b8d9ca88da35 🚀
Tree-SHA512: f72f546cfc20cf1cc0c26c2306ac06416ada87661596fe811b497cce646aa286dc4aee832145bf838b13fbd3c5f064519eb8c0b4525eb562f2f04f20e2876ffc
f1694757ddbcb3635213b085e864851e285c8c12 guix: Fix typo (Carl Dong)
771c4b98a8693eee642f2b118b3193fe6e022291 guix: README: Add darwin HOSTS entry (Carl Dong)
8dbf18cb1d3260d34ba822ceb12e67b1f124ea13 guix: Check for macOS SDK before building anything (Carl Dong)
34b23f597ec52efb795d72e9e5620712d0010edd guix: Set ZERO_AR_DATE for darwin build determinism (Carl Dong)
f3835dc6a3732dcd4afbb5987f84dc27f2bf55af build: Make xorrisofs reproducible with -volume_date (Carl Dong)
c9eb4cf3a0f81bfd72f06fd43b5610f0a4f5e804 guix: Add support for darwin builds (Carl Dong)
37fe73a092b08fe9d7ce636a1021429de6cda757 build: Add var printing target to src/Makefile.am (Carl Dong)
Pull request description:
This PR brings our Guix builds on par with Gitian in terms of supported architectures.
Reviewers: if you run a build, please submit:
```
find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
```
So that we can compare hashes and ensure reproducibility!
ACKs for top commit:
fanquake:
ACK f1694757ddbcb3635213b085e864851e285c8c12 - I think we can make some small usability improvements, but this is ok to merge now.
Tree-SHA512: 4af2b71654a9736467dcc681d10601c6eee37800d7847011a50585455b67b55d61742ca5604585f310a2fd75335b674e5e27dfb5169cb2f26e112aa4c411d8be
570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong)
2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong)
0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong)
d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong)
57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong)
38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong)
036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong)
34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong)
402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong)
cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong)
93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong)
0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong)
444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong)
Pull request description:
After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them.
Notable changes:
1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment
2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules
3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process.
4. README cleanup
ACKs for top commit:
fanquake:
ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward.
Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
f852761aec81ed23c7b9e4546c08d1ef303f2507 guix: Add clarifying documentation for V env var (Carl Dong)
85f4a4b0822e3aa10310c4623eff719f301e9263 guix: Make V=1 more powerful for debugging (Carl Dong)
Pull request description:
```
- Print commands in both unexpanded and expanded forms
- Set VERBOSE=1 for CMake
```
Ping MarcoFalke hopefully you use `V=1` already for the Guix builds on DrahtBot?
ACKs for top commit:
fanquake:
ACK f852761aec81ed23c7b9e4546c08d1ef303f2507. Ran a Windows Guix build and compared the output from master and this PR when using `V=1`. i.e `HOSTS=x86_64-w64-mingw32 PATH="/root/.config/guix/current/bin${PATH:+:}$PATH" V=1 ./contrib/guix/guix-build.sh`.
Tree-SHA512: 8bc466fa7b869618bbd5a0a91c6b23d4785009289f8dfb93b0349317463a9ab9ece128c72436e02a0819722a63e703100aed15807867a716fda891292fcb9d9d
bfe1ba2f5b36056e0c41edf8206b93d3d83098df rel-builds: Specify core.abbrev for git-rev-parse (Carl Dong)
27e63e01cce368d67092de8f0c736927d6f6aa69 build: Accomodate makensis v2.x (Carl Dong)
1f2c39a30e0f82046c7aecddfda3eb99cb536816 guix: Remove logical cores requirement (Carl Dong)
a4f6ffa71e335d4b2a6bf525b7f416968f9cd9f7 lint: Also enable source statements for non-gitian (Carl Dong)
d256f91cb1b0d6ff5170106b99b0266cbe51f5a2 rel-builds: Directly deploy win installer to OUTDIR (Carl Dong)
fa791da02f9684e3fd554b687fb692ae6a23d65a nsis: Specify OutFile path only once (Carl Dong)
14701604d0904bc5bbf1c67de08f8ee6d3215523 guix: Expose GIT_COMMON_DIR in container as readonly (Carl Dong)
f5a6ac4f48b18f93050d77bcb23f9cf45ec34647 guix: Make source tarball using git-archive (Carl Dong)
395c1137f630dc495ffb2752a23bc1dfd470ee53 gitian: Limit sourced script to just assignments (Carl Dong)
Pull request description:
Based on: #18556
Related: https://github.com/bitcoin/bitcoin/pull/17595#discussion_r399728721
ACKs for top commit:
fanquake:
ACK bfe1ba2f5b36056e0c41edf8206b93d3d83098df - I agree with Carl, and am going to merge this. I'd like for Linux Guix builds to be working again, and we can rebase #18818.
Tree-SHA512: c87ada7e3de17ca0b692a91029b86573442ded5780fc081c214773f6b374a0cdbeaf6f6898c36669c2e247ee32aa7f82defb1180f8decac52c65f0c140f18674
a35e3235891d35daa167116cc70340140e883f06 guix: Appease travis. (Carl Dong)
0b66d22da5f53640e22f05adf880782c613e6d0f guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong)
ba0b99bdd613ba7f17c6247ece3001e1b44759a3 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong)
93439a71eda49fb69f1e82966a23a946733aa6fa guix: Bump to upstream commit with mingw-w64 changes (Carl Dong)
35a96792dda9e78165b1598aeac7b2ab759e7be5 guix: Check mingw symbols, improve SSP fix docs (Carl Dong)
449d8fe25bbe25daacfc67aa89ca32b0a3254c5a guix: Expand on INT trap message (Carl Dong)
3f1f03c67a8e9edf487f08d272adb18b0a3942c8 guix: Spelling fixes (Carl Dong)
ff821dd2a1c600488d11e7d9a20e9179ecc9144b guix: Reinstate make-ssp-fixed-gcc (Carl Dong)
360a9e0ad50a36ec79a1a160dbed3966689fd41c guix: Bump time-machine for mingw-w64 patches (Carl Dong)
93e41b7e3b54c17fd1b4c61ee95fc0dc2827e954 guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong)
ef4f7e4c45c60a69406134122f091c77c6ef740f guix: Set the well-known timezone env var (Carl Dong)
acf4b3b3b5accf60a19441a0298ef27001b78e72 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong)
c4cce00eac691625b78b92f7dba0b7f57def19e5 guix: Remove dead links from README. (Carl Dong)
df953a4c9a6143f45864757b706c88b6fa70545a guix: Appease shellcheck. (Carl Dong)
91897c95e191d293eb27d8af15cbeafc5b8f3895 guix: Improve guix-build.sh documentation (Carl Dong)
570d769c6c59b9f6d1a2b95b2ed60432cb33b3ba guix: Build support for Windows (Carl Dong)
Pull request description:
~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~
Based on: #17933 (Time Machines are... shall we say... superior 😁)
This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon)
ACKs for top commit:
fanquake:
ACK a35e3235891d35daa167116cc70340140e883f06 2/3
Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
fae9084ac5b10f94bdee54853d307838c4254e9c build: Skip i686 build by default in guix and gitian (MarcoFalke)
fa55a2554c2661b8f2a759044d5ac85c9979d9ca depends: Remove reference to win32 (MarcoFalke)
Pull request description:
Closes#17504
Now that we no longer provide downloads for i686 on our website (https://bitcoincore.org/en/download/), there is no need to build them by default.
i686 can still be built in depends (tested by ci/travis) and in guix/gitian by setting the appropriate `HOSTS`.
ACKs for top commit:
practicalswift:
ACK fae9084ac5b10f94bdee54853d307838c4254e9c -- patch looks correct
dongcarl:
ACK fae9084ac5b10f94bdee54853d307838c4254e9c patch looks correct
laanwj:
Code review ACK fae9084ac5b10f94bdee54853d307838c4254e9c
hebasto:
ACK fae9084ac5b10f94bdee54853d307838c4254e9c, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: b000c19a2cd2a596a52028fa298c4022c24cfdfc1bdb3795a90916d0a00a32e4dd22278db93790b6a11724e08ea8451f4f05c77bc40d1664518e11a8c82d6e29
88c83636d5a56bd9551577139786bdd3e74852c2 guix: Update documentation for time-machine (Carl Dong)
e6050884fdabfa6e51e6afce2041d91e60a5adec guix: Pin Guix using `guix time-machine` (Carl Dong)
Pull request description:
An alternative to #16519, pinning our version of Guix and eliminating a `guix pull` and changing the default Guix profile of builders.
I think this method might be superior, as it:
- Eliminates the possibility of future changes to the `guix environment` command line interface breaking our builds
- Eliminates the need to set up a separate channel repo
It is a more general pinning solution than #16519.
-----
The reason why I didn't originally propose this is because `guix time-machine` is a recent addition to Guix, only available since `f675f8dec73d02e319e607559ed2316c299ae8c7`
ACKs for top commit:
fanquake:
ACK 88c83636d5a56bd9551577139786bdd3e74852c2
Tree-SHA512: 85e03b0987ffa86da73e02801e1cd8b7622698d70c4ba4e60561611be1e9717d661c2811a59b3e137b1b8eef2d0ba37c313867d035ebc89c3bd06a23a078064a
751549b52a9a4cd27389d807ae67f02bbb39cd7f contrib: guix: Additional clarifications re: substitutes (Carl Dong)
cd3e947f50db7cfe05c05b368c25742193729a62 contrib: guix: Various improvements. (Carl Dong)
8dff3e48a9e03299468ed3b342642f01f70da9db contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong)
3e80ec3ea9691c7c89173de922a113e643fe976b contrib: Add deterministic Guix builds. (Carl Dong)
Pull request description:
~~**This post is kept updated as this project progresses. Use this [latest update link](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718) to see what's new.**~~
Please read the `README.md`.
-----
### Guix Introduction
This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin.
### Guix Build Walkthrough
Please read the `README.md`.
[Old instructions no. 4](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-497303718)
[Old instructions no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-493827011)
[Old instructions no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old instructions no. 1</summary>
In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment.
We can then invoke
```
guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
To have Guix:
1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details).
2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in.
3. Drop you into a shell in that container.
> Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>`
Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following:
```
make -C depends -j"$(nproc)" download && \
cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes
```
We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing).
</details>
### Rationale
I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because:
1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](72bd4ab867/contrib/gitian-descriptors/gitian-linux.yml (L10))), because Guix builds everything about the container, we can perform this on almost any Linux distro/system.
2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)).
3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler).
### Steps to Completion
- [x] Successfully build bitcoin inside the Guix environment
- [x] Make `check-symbols` pass
- [x] Do the above but without nasty hacks
- [x] Solve some of the more innocuous hacks
- [ ] Make it cross-compile (HELP WANTED HERE)
- [x] Linux
- [x] x86_64-linux-gnu
- [x] i686-linux-gnu
- [x] aarch64-linux-gnu
- [x] arm-linux-gnueabihf
- [x] riscv64-linux-gnu
- [ ] OS X
- [ ] x86_64-apple-darwin14
- [ ] Windows
- [ ] x86_64-w64-mingw32
- [ ] Maybe make importer for depends syntax
- [ ] Document build process for future releases
- [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html)
### Help Wanted
[Old content no. 3](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-483318210)
[Old content no. 2](https://github.com/bitcoin/bitcoin/pull/15277#issuecomment-471658439)
<details>
<summary>Old content no. 1</summary>
As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be.
Here's what `ldd src/bitcoind` looks like when built in a Guix container:
```
linux-vdso.so.1 (0x00007ffcc2d90000)
libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000)
librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000)
libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000)
libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000)
libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000)
libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000)
libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000)
/gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000)
```
And here's what it looks in one of our releases:
```
linux-vdso.so.1 (0x00007ffff52cd000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000)
```
~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely.
Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](d6e700e40f/contrib/gitian-descriptors/gitian-win-signer.yml (L14))
</details>
### How to Help
1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation)
2. Try out my branch and the command described above!
ACKs for top commit:
MarcoFalke:
Thanks for the replies. ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
laanwj:
ACK 751549b52a9a4cd27389d807ae67f02bbb39cd7f
Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
