66c4b58e518aff08030b3c879c44af7716110619 guix: switch from guix environment to guix shell (fanquake)
Pull request description:
See https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-environment.html.
> Deprecation warning: The guix environment command is deprecated
in favor of guix shell, which performs similar functions but is more convenient to use. See Invoking guix shell.
> Being deprecated, guix environment is slated for eventual removal,
but the Guix project is committed to keeping it until May 1st, 2023. Please get in touch with us at guix-devel@gnu.org if you would like to discuss it.
See also https://guix.gnu.org/blog/2021/from-guix-environment-to-guix-shell/ for a blog post and additional details.
Guix `shell` was added to Guix ~1 year ago, in this commit, https://git.savannah.gnu.org/cgit/guix.git/commit/?id=80edb7df6586464aa40e84e103f0045452de95db, which isn't part of the 1.3.0 release binaries out of the box, but invoking a `guix pull`, and updating will make it available. i.e:
```bash
bash-5.1# guix --version
guix (GNU Guix) 1.3.0
Copyright (C) 2021 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
bash-5.1# guix shell
guix: shell: command not found
Try 'guix --help' for more information.
bash-5.1# guix pull
Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...
Authenticating channel 'guix', commits 9edb3f6 to 7a980bb (6,278 new commits)...
Building from this channel:
guix https://git.savannah.gnu.org/git/guix.git7a980bb
< snip >
building /gnu/store/2wwwsczxcw61m05p4mv0kf0advx4fqsb-inferior-script.scm.drv...
building package cache...
building profile with 1 package...
New in this revision:
6,866 new packages: a2jmidid, abjad,
bash-5.1# guix help shell
Usage: guix shell [OPTION] PACKAGES... [-- COMMAND...]
Build an environment that includes PACKAGES and execute COMMAND or an
interactive shell in that environment.
```
ACKs for top commit:
TheCharlatan:
ACK 66c4b58e518aff08030b3c879c44af7716110619
Tree-SHA512: caa3fd2ca8d0f261c50ecdda3728a75389d24d89b51293dedc704ee77ab1342b2bb08ca8c871dcb4646229f056ec86cb15500934ded1b0c501a3ffc25aaa8ae6
eff19fa1c8d736d30dea937ebce0b372c19b7559 build, macos: Drop unused `osx_volname` target (Hennadii Stepanov)
Pull request description:
The `osx_volname` makefile target was introduced in https://github.com/bitcoin/bitcoin/pull/7192 and was used to pass the package name to Gitian scripts as a content of the `osx_volname` file.
With the current Guix scripts, the `osx_volname` file is never read. Therefore, its creation might be omitted.
My Guix builds:
```
x86_64
5e2d254e207d53784621c8df331c9bf4a969da667d185992402f48a5ac49f563 guix-build-eff19fa1c8d7/output/arm64-apple-darwin/SHA256SUMS.part
089dba70685893aca5e7c8ce1d53a07380e87ca50eda8b3a2a75aeaeb1d28e48 guix-build-eff19fa1c8d7/output/arm64-apple-darwin/bitcoin-eff19fa1c8d7-arm64-apple-darwin-unsigned.tar.gz
390c57197c6ab4aefdde1c665d5e4ebdfb4ae5e553f8f93b017f2fad1093d110 guix-build-eff19fa1c8d7/output/arm64-apple-darwin/bitcoin-eff19fa1c8d7-arm64-apple-darwin-unsigned.zip
e1edde7ca28bf26aea8d956b1d3c1725a475f2a9c148f5c36b651db4b814091c guix-build-eff19fa1c8d7/output/arm64-apple-darwin/bitcoin-eff19fa1c8d7-arm64-apple-darwin.tar.gz
d0096ea73a5f75cc4d3cef4ef1761ae3e48c8a63aff918f07371c5c88896e4e6 guix-build-eff19fa1c8d7/output/dist-archive/bitcoin-eff19fa1c8d7.tar.gz
51b4affb9fd6f8aea05b7d25d29f017d0a0a145395f457caa14b9af9646b035b guix-build-eff19fa1c8d7/output/x86_64-apple-darwin/SHA256SUMS.part
b1df081ecf636a92754e673e5388d1d988653d4646f0b0446a4c9f14d865a265 guix-build-eff19fa1c8d7/output/x86_64-apple-darwin/bitcoin-eff19fa1c8d7-x86_64-apple-darwin-unsigned.tar.gz
62e09926029d176da950d3e3db7ff8ae6cbe4c0b2ea17b084fc1d28565f91475 guix-build-eff19fa1c8d7/output/x86_64-apple-darwin/bitcoin-eff19fa1c8d7-x86_64-apple-darwin-unsigned.zip
477dcb2382cbd447bd88a3b644b4bd736f5b67d66d42cb73fe31ffc153d3e181 guix-build-eff19fa1c8d7/output/x86_64-apple-darwin/bitcoin-eff19fa1c8d7-x86_64-apple-darwin.tar.gz
```
ACKs for top commit:
Empact:
ACK eff19fa1c8
Sjors:
tACK eff19fa1c8d736d30dea937ebce0b372c19b7559
TheCharlatan:
ACK eff19fa1c8d736d30dea937ebce0b372c19b7559
Tree-SHA512: 29714be5c58caa07b3eb99846d71bb83366dade769af022059bb3c499878adcd34cdf03b006c5da561291f373ccc59abdb83c925057ec0049465eaa8dd6ef4e1
46b025e00df40724175735eb5606ac73067cb3b8 test: add new python linter to check file names and permissions (windsok)
6f6bb3ebc7cb8e17a5dfc8ef55aa2d3f2dc6bdea test: fix file permissions on various scripts (windsok)
Pull request description:
Adds a new python linter test which tests for correct filenames and file permissions in the repository.
Replaces the existing tests in the `test/lint/lint-filenames.sh` and `test/lint/lint-shebang.sh` linter tests, as well as adding some new and increased testing. This increased coverage is intended to catch issues such as in #21728 and https://github.com/bitcoin/bitcoin/pull/16807/files#r345547050
Summary of tests:
* Checks every file in the repository against an allowed regexp to make sure only lowercase or uppercase alphanumerics (a-zA-Z0-9), underscores (_), hyphens (-), at (@) and dots (.) are used in repository filenames.
* Checks only source files (*.cpp, *.h, *.py, *.sh) against a stricter allowed regexp to make sure only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.) are used in source code filenames. Additionally there is an exception regexp for directories or files which are excepted from matching this regexp (This should replicate the existing `test/lint/lint-filenames.sh` test)
* Checks all files in the repository match an allowed executable or non-executable file permission octal. Additionally checks that for executable files, the file contains a shebang line.
* Checks that for executable `.py` and `.sh` files, the shebang line used matches an allowable list of shebangs (This should replicate the existing `test/lint/lint-shebang.sh` test)
* Checks every file that contains a shebang line to ensure it has an executable permission
Additionally updates the permissions on various files to comply with the new tests.
Fixes#21729
ACKs for top commit:
practicalswift:
cr re-ACK 46b025e00df40724175735eb5606ac73067cb3b8: patch still looks correct
kiminuo:
code review ACK 46b025e00df40724175735eb5606ac73067cb3b8 if `contrib/gitian-descriptors/assign_DISTNAME` permission change is deemed OK.
laanwj:
Code review ACK 46b025e00df40724175735eb5606ac73067cb3b8
Tree-SHA512: 1c8201a2cee0d9cbce15652b68cec9a6458a8b493fcd5392f98560aca0b1a12e668baab65a47100f116f626dadc3f591deb47f7368468c6a46c6c712c2533455
4133c8104f522c403c55d26bd03436a8149ff106 guix: use gcc tool wrappers (fanquake)
Pull request description:
This way, correct `--plugin` arguments are passed through.
This is a prerequisite for LTO (see #25391). Split out, to try move things along, as this change is isolated, and should be straight-forward.
ACKs for top commit:
TheCharlatan:
ACK [4133c81](4133c8104f)
hebasto:
ACK 4133c8104f522c403c55d26bd03436a8149ff106
Tree-SHA512: 4311a72a613cf027bd4490caa29604c985ed455589acd972285f13cbdf4806d2184a4dc6f20cb6f47c3fa751d58bfd0bacc257b87d4a804bf5ecf5b240e4a757
## Issue being fixed or feature implemented
We had this in Gitian
https://github.com/dashpay/dash/blob/master/contrib/gitian-descriptors/gitian-win.yml#L38.
We also had it for macos
https://github.com/dashpay/dash/blob/master/contrib/gitian-descriptors/gitian-osx.yml#L42
but it looks like it's no longer an issue there (or at least I did not
see anyone complaining about it).
## What was done?
tweak `CONFIGFLAGS` for `mingw` host
## How Has This Been Tested?
n/a
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
979271a5d9ff887cb2efb199feaf9602c9b2086d macdeploy: remove unused detached-sig-apply (fanquake)
Pull request description:
Signature application is now done with signapple.
8435d7f11a/contrib/guix/libexec/codesign.sh (L84-L85)
ACKs for top commit:
laanwj:
ACK 979271a5d9ff887cb2efb199feaf9602c9b2086d
gruve-p:
ACK 979271a5d9
achow101:
ACK 979271a5d9ff887cb2efb199feaf9602c9b2086d
hebasto:
ACK 979271a5d9ff887cb2efb199feaf9602c9b2086d, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: ab51a609d00cead4f33bcfc5b5ff1008ee02363ab1f4c4bf9544631069c237bfa92eac4dfa231bff8a1d702bda6cc92b4151361f74f58e77b595e0cb82a8391a
## Issue being fixed or feature implemented
make it possible to run `./contrib/guix/guix-build` without specifying
`CONFIGFLAGS`
## What was done?
## How Has This Been Tested?
run `./contrib/guix/guix-build` w/ and w/out this patch
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
## Issue being fixed or feature implemented
Add debug symbols for Darwin
## What was done?
Added Darwin debug symbols and combine them as output
## How Has This Been Tested?
guix build
## Breaking Changes
_Please describe any breaking changes your code introduces_
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that
apply._
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
---------
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
## Issue being fixed or feature implemented
Make it possible to pass additional configure params into Guix. This
could be used to setup various sets of nightly/debug builds which could
then be deployed automagically to catch potential issues early.
## What was done?
## How Has This Been Tested?
`CONFIGFLAGS="--enable-debug" HOSTS="x86_64-linux-gnu"
./contrib/guix/guix-build`
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
c73415bc10c1baa7988e1c55a0e9201df73a6c22 build: Fix xargs warnings for Guix builds (Hennadii Stepanov)
Pull request description:
On master (e3ce019667fba2ec50a59814a26566fb67fa9125) there are warnings in `./contrib/guix/guix-build` logs:
```
xargs: warning: options --max-args and --replace/-I/-i are mutually exclusive, ignoring previous --max-args value
```
This PR fixes such warnings.
ACKs for top commit:
prusnak:
utACK c73415b
Tree-SHA512: a7b55f59afbb19b78f795cea64acacf29903cfcd5fd7c37a771b073c1f2ff54555a26f3d00c1c73a8ef588396217ddf598e32b2ae961559042cc051b0aad162a
9f01feda0ad99d8600f9edee1f37594c5166a7b8 guix/build: Remove vestigial SKIPATTEST.TAG (Carl Dong)
Pull request description:
No longer needed or referenced by anything. A relic from prior to the great hierarchy overhaul of #22182
ACKs for top commit:
achow101:
ACK 9f01feda0ad99d8600f9edee1f37594c5166a7b8
fanquake:
ACK 9f01feda0ad99d8600f9edee1f37594c5166a7b8
Tree-SHA512: a94cf63f0c5cb8dbacf1025b6c0e81b219c2a3c93b3cbcefc239ccde29e602ecd4b717b1d93dbe53cb791a5017236fb09823c034aec42b0c31894fc9e0ab8b21
1edddf5de41b053049ce0b0bdbc39c2fbb743c40 Avoid GCC 7.1 ABI change warning in guix build (Pieter Wuille)
Pull request description:
The arm-linux-gnueabihf guix build output is littered with warnings like:
```
/gnu/store/7a96hdqdb2qi8a39f09n84xjy2hr23rs-gcc-cross-arm-linux-gnueabihf-8.4.0/include/c++/bits/stl_vector.h:1085:4: note:
parameter passing for argument of type '__gnu_cxx::__normal_iterator<CRecipient*, std::vector<CRecipient> >' changed in GCC 7.1
```
These are irrelevant for us. Disable them using `-Wno-psabi`.
ACKs for top commit:
laanwj:
ACK 1edddf5de41b053049ce0b0bdbc39c2fbb743c40
hebasto:
ACK 1edddf5de41b053049ce0b0bdbc39c2fbb743c40, after thorough reading related materials, I agree this change can be merged. As I mentioned above, I have been compiling my arm-32bit binaries with `-Wno-psabi` flag for two years, and no related flaws were observed.
Tree-SHA512: 485c7500547ac5da567ad23847341c18ff832607f5a1002676404cc647e437cf3445b6894ecff5b52929ca52bea946c06bd90eace1997c895e56204e787065e4
797b3ed9090030f32fade81803b580562d4a90a3 script: remove gitian reference from symbol-check.py (fanquake)
15fc9a0299091bfeb3370f993ad95ff638f6ba8c guix: add additional documentation to patches (fanquake)
4516e5ec9223486fe2eba7f4320d786d074a58fd lint: exclude Guix patches from spell-checking (fanquake)
de6ca41a52d2646598daae5f4620bbe766757e21 guix: no-longer pass --enable-glibc-back-compat to Guix (fanquake)
84dd81fb5bf7308b8070b53520266854fb6efad3 build: remove glibc backcompat requirement for Linux symbol checks (fanquake)
Pull request description:
Now that our Guix toolchains are based on glibc 2.24 and 2.27 (RISCV), we don't need to use the `--enable-glibc-back-compat` option to produce binaries that don't use any symbols from glibc 2.17 and 2.27 or later.
This also adds additional documentation to some Guix patches (pointed out in #22365) and removes Guix patches from the spelling linter, because that isn't our spelling.
Symbol usage: https://gist.github.com/fanquake/d15604fc580718444c5aa4b3c3c75fdc.
Guix Builds:
```bash
bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
ed54e6a6cf4fab328557c0c72eb08c73f2a58c6c70959544cf4b1882e75ea69e guix-build-797b3ed90900/output/aarch64-linux-gnu/SHA256SUMS.part
83bd9dadc59f89f848d143fa4fc3964f16fe0b4bdf35e5093b577ff2c4bd1f43 guix-build-797b3ed90900/output/aarch64-linux-gnu/bitcoin-797b3ed90900-aarch64-linux-gnu-debug.tar.gz
94cb8c35281f12dec6ea5b390b66cad5e27ac8c45a30c42c8d38c438695d54c0 guix-build-797b3ed90900/output/aarch64-linux-gnu/bitcoin-797b3ed90900-aarch64-linux-gnu.tar.gz
7318b63d65c0aa52d2446de8e1f40658d2e47ab8fb0268820c3b7585d140fb23 guix-build-797b3ed90900/output/arm-linux-gnueabihf/SHA256SUMS.part
95e1ffb372964b73f539653ca703b70cf0c018801a9c4c0ffc46a0b63539253c guix-build-797b3ed90900/output/arm-linux-gnueabihf/bitcoin-797b3ed90900-arm-linux-gnueabihf-debug.tar.gz
039d3842e6499626cf955ae0a7590dd6b3d0935cdc217c98aaf9d156b0ebd3b4 guix-build-797b3ed90900/output/arm-linux-gnueabihf/bitcoin-797b3ed90900-arm-linux-gnueabihf.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 guix-build-797b3ed90900/output/dist-archive/SKIPATTEST.TAG
2c4e7b6e7aff63ba811e5bf59362d16866c3a358f8844fba8739a61192870622 guix-build-797b3ed90900/output/dist-archive/bitcoin-797b3ed90900.tar.gz
955029b949c368eabd517dd33040d2f01e2ac6a55e7b4f9107907a7c6e0c6060 guix-build-797b3ed90900/output/powerpc64-linux-gnu/SHA256SUMS.part
fd6d6b137f8efedf58a879d11205b1d4649e1f97d7f91e193239ef206fcc285d guix-build-797b3ed90900/output/powerpc64-linux-gnu/bitcoin-797b3ed90900-powerpc64-linux-gnu-debug.tar.gz
51736ac8e77737999f1b5bd4c381b0016f19a8d5e40e786fe941ff04e84c11c9 guix-build-797b3ed90900/output/powerpc64-linux-gnu/bitcoin-797b3ed90900-powerpc64-linux-gnu.tar.gz
8c244c16bfa46c1efdb120e1d91fdd14d3f14eefee8d7e1fbb0a9b4664a5c315 guix-build-797b3ed90900/output/powerpc64le-linux-gnu/SHA256SUMS.part
704ee593251a1b1c65a5bebeef93b23f266af4e8cbf8ae556150c3b2e8f06a6c guix-build-797b3ed90900/output/powerpc64le-linux-gnu/bitcoin-797b3ed90900-powerpc64le-linux-gnu-debug.tar.gz
0ec06ae7d344de20d61e3965d8b383747ef20b0e9d93a3165733ea23bdf2ead8 guix-build-797b3ed90900/output/powerpc64le-linux-gnu/bitcoin-797b3ed90900-powerpc64le-linux-gnu.tar.gz
2dd6c6ecc67b0ea40ca9c43f92efca81ccd054b8db8c197ad84ad9674d510a25 guix-build-797b3ed90900/output/riscv64-linux-gnu/SHA256SUMS.part
5ebb27a855a677f7a188d83995be6b2a3ea8606be152abb7fc7832713fb0677a guix-build-797b3ed90900/output/riscv64-linux-gnu/bitcoin-797b3ed90900-riscv64-linux-gnu-debug.tar.gz
bdaf1783f5e1861597afa37c1880364e118d9a7a7af8017302d82202791019f6 guix-build-797b3ed90900/output/riscv64-linux-gnu/bitcoin-797b3ed90900-riscv64-linux-gnu.tar.gz
726c9092b60ac2e7d7e14b2c24467fcf276a6f89170a871ddab9dce6ac230699 guix-build-797b3ed90900/output/x86_64-apple-darwin18/SHA256SUMS.part
2af4d709b44952654f3c08c86593bf2ccc9a44ed422783a1b95b8a199a894db2 guix-build-797b3ed90900/output/x86_64-apple-darwin18/bitcoin-797b3ed90900-osx-unsigned.dmg
fd49ba445aa6cf3d8c47019a05e9e5740cb0f53349344dd80671297127f49f1a guix-build-797b3ed90900/output/x86_64-apple-darwin18/bitcoin-797b3ed90900-osx-unsigned.tar.gz
3f51cbf8cf18420d4be70e656aa993675cf5e828a255c2030047ae2e059ed5b7 guix-build-797b3ed90900/output/x86_64-apple-darwin18/bitcoin-797b3ed90900-osx64.tar.gz
afd1edee1447bb88d81e972abfae4c4e065b5b1827769f033cff9472084c7c1b guix-build-797b3ed90900/output/x86_64-linux-gnu/SHA256SUMS.part
ec468ef886d25e685f4f7a18b4f7d497dedf757495e0d5beb72c23cc32ab69b5 guix-build-797b3ed90900/output/x86_64-linux-gnu/bitcoin-797b3ed90900-x86_64-linux-gnu-debug.tar.gz
1934d7294f0c9e083d38a3f68d4a61cd679defa79ce0a89f77386978692b9b18 guix-build-797b3ed90900/output/x86_64-linux-gnu/bitcoin-797b3ed90900-x86_64-linux-gnu.tar.gz
94c11c328a628052eb6f50e9816aa768f87ea7acfbbbafdab60f6928da766811 guix-build-797b3ed90900/output/x86_64-w64-mingw32/SHA256SUMS.part
fd371922ba93d81bd4a2b711d617af6756f9f0494db6d83aa0e5f491a24168ef guix-build-797b3ed90900/output/x86_64-w64-mingw32/bitcoin-797b3ed90900-win-unsigned.tar.gz
4e4ad976bc029bbbf9596ad8493accaaba8b0d5c598dd342f8da330609bbdf21 guix-build-797b3ed90900/output/x86_64-w64-mingw32/bitcoin-797b3ed90900-win64-debug.zip
3a89a16b9101e9a17d98efb9234b5bdd264c0bba2c6326511017730e1a08311f guix-build-797b3ed90900/output/x86_64-w64-mingw32/bitcoin-797b3ed90900-win64-setup-unsigned.exe
e285ab737e3c843fd3f1c26c2f053e421a3c39b33995747ce48281884d3f28d1 guix-build-797b3ed90900/output/x86_64-w64-mingw32/bitcoin-797b3ed90900-win64.zip
```
ACKs for top commit:
sipa:
utACK 797b3ed9090030f32fade81803b580562d4a90a3
hebasto:
ACK 797b3ed9090030f32fade81803b580562d4a90a3
Tree-SHA512: 3a569702d8832c155c5ce8d2f6d823f7f12603885576078bc5192bc9038a48261ecb541800f79d1e9bc86d71fa640265c5b8b89df9d8bb680b3bb05d9d78a666
647f7e5f1da1089d451f3c431efc635b8e87b064 guix: Also sort SHA256SUMS.part (Carl Dong)
dc4137a60c99979b89f75d2bddba96d043f387b8 guix: Build depends/qt with our platform definition (Carl Dong)
16b0a936e15b81710755303e11ef51f608b61475 guix: Rebase toolchain on glibc 2.24 (2.27 for riscv64) (Carl Dong)
Pull request description:
After this PR, we'll have the following:
- riscv64 -> build with a toolchain targeting glibc 2.27
- everything else -> builds with a toolchain targeting glibc 2.24, but will not have symbols > 2.17 (checked by `symbol-check.py`)
ACKs for top commit:
achow101:
reACK 647f7e5f1da1089d451f3c431efc635b8e87b064
hebasto:
ACK 647f7e5f1da1089d451f3c431efc635b8e87b064
MarcoFalke:
review ACK 647f7e5f1da1089d451f3c431efc635b8e87b064
fanquake:
ACK 647f7e5f1da1089d451f3c431efc635b8e87b064 - documentation can be fixed shortly.
Tree-SHA512: ddff57a5d7c053687b0a273720d4ad7d28c6fc8816226d4304869284d017af5e3630d4b57565d91e74f2e1b7583c9c83ee8b2e5e70e41d619ab618e602c97a94
e2c40a4ed5272d72fea997bd936fba28bb753226 guix-attest: Error out if SHA256SUMS is unexpected (Carl Dong)
4cc35daed557f38b080360a89036b2e97a6f78c2 Rewrite guix-{attest,verify} for new hier (Carl Dong)
28a9c9b83924f585b397f0f3b8e9e73780ac0ad6 Make SHA256SUMS fragment right after build (Carl Dong)
Pull request description:
Based on: #22075
Code reviewers: I recommend reading the new `guix-{attest,verify}` files instead of trying to read the diff
The following changes resolve many usability improvements which were pointed out to me:
1. Some maintainers like to extract their "uncodesigned tarball" inside the `output/` directory, resulting in the older `guix-attest` mistakenly attesting to the extracted contents
2. Maintainers whose GPG keys reside on an external smartcard often need to physically interact with the smartcard as a way to approve the signing operation, having one signature per platform means a lot of fidgeting
3. Maintainers wishing to sign on a separate machine now has the option of transferring only a subtree of `output/`, namely `output/*/SHA256SUMS.part`, in order to perform a signature (you may need to specify an `$OUTDIR_BASE` env var)
4. An `all.SHA256SUMS` file should be usable as the base `SHA256SUMS` in bitcoin core torrents and on the release server.
For those who sign on an separate machine than the one you do builds on, the following steps will work:
1. `env GUIX_SIGS_REPO=/home/achow101/guix.sigs SIGNER=achow101 NO_SIGN=1 ./contrib/guix/guix-attest`
2. Copy `/home/achow101/guix.sigs/<tag>/achow101` (which does not yet have signatures) to signing machine
3. Sign the `SHA256SUMS` files:
```bash
for i in "<path-to-achow101>/*.SHA256SUMS"; do
gpg --detach-sign --local-user "<your-key-here>" --armor --output "$i"{.asc,}
done
```
5. Upload `<path-to-achow101>` (now with signatures) to `guix.sigs`
-----
After this change, output directories will now include a `SHA256SUMS.part` fragment, created immediately after a successful build:
```
output
└── x86_64-w64-mingw32
├── bitcoin-4e069f7589da-win64-debug.zip
├── bitcoin-4e069f7589da-win64-setup-unsigned.exe
├── bitcoin-4e069f7589da-win64.zip
├── bitcoin-4e069f7589da-win-unsigned.tar.gz
└── SHA256SUMS.part
```
These `SHA256SUMS.part` fragments look something like:
```
3ebd7262b1a0a5bb757fef1f70e7e14033c70f98c059bc4dbfee5d1992b25825 dist-archive/bitcoin-4e069f7589da.tar.gz
def2e7d3de5ab3e3f955344e75151df4f33713f9101f5295bd13c9375bdf633b x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-debug.zip
643049fe3ee4a4e83a1739607e67b11b7c9b1a66208a6f35a9ff634ba795500e x86_64-w64-mingw32/bitcoin-4e069f7589da-win64-setup-unsigned.exe
a247a1ccec0ccc2e138c648284bd01f6a761f2d8d6d07d91b5b4a6670ec3f288 x86_64-w64-mingw32/bitcoin-4e069f7589da-win-unsigned.tar.gz
fab76a836dcc592e39c04fd2396696633fb6eb56e39ecbf6c909bd173ed4280c x86_64-w64-mingw32/bitcoin-4e069f7589da-win64.zip
```
Meaning that they are valid `SHA256SUMS` files when `sha256sum --check`'d at the `guix-build-*/output` directory level
When `guix-attest` is invoked, these `SHA256SUMS.part` files are combined and sorted (by `-k2`, `LC_ALL=C`) to create:
1. `noncodesigned.SHA256SUMS` for a manifest of all non-codesigned outputs, and
3. `all.SHA256SUMS` for a manifest of all outputs including non-codesigned outputs
Then both files are signed, resulting in the following `guix.sigs` hierarchy:
```
4e069f7589da/
└── dongcarl
├── all.SHA256SUMS
├── all.SHA256SUMS.asc
├── noncodesigned.SHA256SUMS
└── noncodesigned.SHA256SUMS.asc
```
ACKs for top commit:
achow101:
ACK e2c40a4ed5272d72fea997bd936fba28bb753226
hebasto:
ACK e2c40a4ed5272d72fea997bd936fba28bb753226, tested on Linux Mint 20.1 (x86_64) with and w/o `NO_SIGN=1`. Changes in `contrib/guix/libexec/codesign.sh` and `contrib/guix/guix-verify` are reviewed only.
Tree-SHA512: 618aacefb0eb6595735a9ab6a98ea6598fce65f9ccf33fa1e7ef93bf140c0f6cfc16e34870c6aa3e4777dd3f004b92a82a994141879870141742df948ec59c1f
108a6be92adc1e80839d90b552e72b8142140f6c guix: Check for disk space availability before building (Carl Dong)
d7dec89091ee4a456ff64ad7ce675ae6813668f1 guix: Remove dest if OUTDIR mv fails (Carl Dong)
Pull request description:
There seems to be some corner cases that can be hit when guix scripts unexpectedly fail in the middle of operation, see: https://gnusha.org/bitcoin-builds/2021-05-24.log
- Perform an early disk space check for `guix-build`
- Overwrite existing output directory after a successful build (the existing one might be malformed), and cleanup output directory if the `mv` somehow fails
ACKs for top commit:
laanwj:
Tested ACK 108a6be92adc1e80839d90b552e72b8142140f6c
achow101:
ACK 108a6be92adc1e80839d90b552e72b8142140f6c
Tree-SHA512: cf6438317da40bf55714cd2d8cce859b3d435cc66cabefe8d4a53552d7880966acfe84ffe8fadf1c80e368ae6b037992258a6d409df85ffc6ce8bf780e98e2e5
a58868d201cb6d263aa552815f7f86562c1ca9a5 build: Makes rcc output always deterministic (Hennadii Stepanov)
Pull request description:
The Qt Resource Compiler ([rcc](https://doc.qt.io/qt-5/rcc.html)) has a command-line option `--format-version` which has the [default value](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/main.cpp?h=5.12.10#n172) 2.
The only difference from `--format-version 1` is adding a [last modified timestamp](https://code.qt.io/cgit/qt/qtbase.git/tree/src/tools/rcc/rcc.cpp?h=5.12.10#n207) to the output file ([credits](https://github.com/bitcoin/bitcoin/pull/21654#issuecomment-819198228) to **fanquake**). That, in turn, forces us to use `QT_RCC_SOURCE_DATE_OVERRIDE=1` to get deterministic builds (#13732).
This change makes rcc output always deterministic by using `--format-version 1` option that makes usage of the
`QT_RCC_SOURCE_DATE_OVERRIDE` needless.
---
Also it improves interaction with ccache:
On master (f6c44e999b7d1d9a0de5d678ac8f1679aa271f65):
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:45:43 2021
stats zeroed Sun Apr 11 15:45:05 2021
cache hit (direct) 638
cache hit (preprocessed) 0
cache miss 1
cache hit rate 99.84 %
called for link 10
cleanups performed 0
files in cache 20023
cache size 13.2 GB
max cache size 15.0 GB
```
The missed file is always `qt/libbitcoinqt_a-qrc_bitcoin_locale.o`.
With this PR:
```
$ make && make clean && ccache --zero-stats && make && ccache --show-stats
...
cache directory /home/hebasto/.ccache
primary config /home/hebasto/.ccache/ccache.conf
secondary config (readonly) /etc/ccache.conf
stats updated Sun Apr 11 15:28:46 2021
stats zeroed Sun Apr 11 15:28:21 2021
cache hit (direct) 639
cache hit (preprocessed) 0
cache miss 0
cache hit rate 100.00 %
called for link 10
cleanups performed 0
files in cache 20012
cache size 13.2 GB
max cache size 15.0 GB
```
ACKs for top commit:
fanquake:
ACK a58868d201cb6d263aa552815f7f86562c1ca9a5
Tree-SHA512: 52f4a3267f41883d13025c0de79b6da22e92d60c729e01b986935c6812bbfe7fadc40b742bd715bfdf09df94af6838d4fbbe8208c6123f366108e38c8e1121c5
d420e5c1c015f58d07aca4d6a805086488f74d03 guix-attest: Avoid incomplete sigdirs with ERR traps (Carl Dong)
feda2c8e3180cb983c35976d4440cea23a155b7f guix: Skip attesting to dist-archive (Carl Dong)
d522d8006b891eccd7901faf391f9c041ddf8e38 guix: Attest to inputs in inputs.SHA256SUMS (Carl Dong)
f9e2960c018103be756a7f8a506816b49d662514 guix: Construct $OUTDIR in ${DISTSRC}/output (Carl Dong)
022abc85fc7e711a900fed8e5071919a151c0a63 guix: Minor quoting fix in libexec/build.sh (Carl Dong)
c83c4fa5b78aef33bba36b3a0d273422297bd630 guix-attest: Allow skipping GPG signing with NO_SIGN (Carl Dong)
0e1c2e448c25568f276e4f022128870c76ca216b guix-attest: Use ascii-armor signatures (Carl Dong)
b5fd89c4c89136007429688601ce4fa497f5f09e guix-attest: Only use cross-platform flags for find+xargs (Carl Dong)
5926432ba68ba154df6c8eaa74adb18cc0123167 guix: Add guix-verify script (Carl Dong)
30daf76a97c57a5f74c8dad1da282dcc0ff8b3fb guix: Add guix-attest script (Carl Dong)
Pull request description:
Adds replacements for `gsign` and `gverify`.
Personally I'm not a big fan of using the word "sign" as it's been used to refer to both codesigning and GPG signing.
ACKs for top commit:
laanwj:
Code review and tested ACK d420e5c1c015f58d07aca4d6a805086488f74d03
Tree-SHA512: 93d82d201f4596eaea0e3825aa55b013dfb91790e6ccee79893833d37921513d7b4e735f0641103e1e2ea8308abe4cb6218b73160924708802f2e0e3f7f6caf1
c90f6e51094a1ba4fb2aab35b78f23b6fda645d0 guix: Consistently use gcc-8 for $HOST (Carl Dong)
Pull request description:
Only non-base commit is the last commit: b5abb07d0d
Right now, here's what we use in Gitian:
- Linux: Focal's [`g++-8-<arch>-linux-gnu`](https://packages.ubuntu.com/focal/g++-8-aarch64-linux-gnu) (`8.4.0-3ubuntu1cross1`)
- MinGW-w64: Focal's [`g++-mingw-w64`](https://packages.ubuntu.com/focal/g++-mingw-w64) (`9.3.0-7ubuntu1+22~exp1ubuntu4`)
In Guix right now we use `gcc-9` across the board.
I think it makes more sense to use `gcc-8` across the board, as it doesn't suffer from the `memcmp` bug, and is what debian buster (stable) does, meaning it will be well tested ([`g++-mingw-w64`](https://packages.debian.org/buster/g++-mingw-w64), [`g++-aarch64-linux-gnu`](https://packages.debian.org/buster/g++-aarch64-linux-gnu)).
We can accomplish this somewhat easily using Guix as we have tighter control over the toolchain (see: b5abb07d0d).
Let me know your thoughts!
ACKs for top commit:
MarcoFalke:
Approach ACK c90f6e51094a1ba4fb2aab35b78f23b6fda645d0, haven't reviewed
laanwj:
Code review ACK c90f6e51094a1ba4fb2aab35b78f23b6fda645d0
hebasto:
ACK c90f6e51094a1ba4fb2aab35b78f23b6fda645d0, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 3e5b9297305232273323aa745ec417ed1be2418ead0e432db7742f5d5f45efe6e4a2ed44328731512cff4bfde80e5f2dc350a131b8b8fb9207a2ef66bce27ed2
c799a19b4bb8d0cc5ffd9b49746b5b267745c9b5 build, qt: No longer need to set QT_RCC_TEST=1 for determinism (Hennadii Stepanov)
Pull request description:
The Qt Resource Compiler (rcc) output order relies on [`QHash`](https://doc.qt.io/qt-5/qhash.html):
> This randomization of `QHash` is enabled by default. Even though programs should never depend on a particular `QHash` ordering, there may be situations where you temporarily need deterministic behavior, for example for debugging or regression testing. To disable the randomization, define the environment variable `QT_HASH_SEED` to have the value 0.
Since #3620 we use `QT_RCC_TEST=1` to achieve a deterministic output.
Since Qt 5.3.1 hash seeding is disabled for all of the bootstrapped tools, including rcc. Therefore, `QT_RCC_TEST=1` is no longer needed.
See commit [5283a6c87beac5a43f612786fefd6e43f2c70bf6](5283a6c87b).
ACKs for top commit:
fanquake:
ACK c799a19b4bb8d0cc5ffd9b49746b5b267745c9b5
Tree-SHA512: 9d116ac1e8c605ee3e8ed7f618586f0de85d8b06bbbb70fe8c298939ce203d2a7e97264a9afac037179993ab54c5f69a65ebb9ab27ca7f45acb963011bd45743
7476b46f1893a4858616d2a8456a7c43238851ed guix: Build dmg as a static binary (Carl Dong)
06d6cf6784421290e6235fe8684d5e08ed6f1b62 depends: libdmg-hfsplus: Skip CMake RPATH patching (Carl Dong)
65176ab5730dff34466caaecdd292625ef8294fc guix: Remove codesign_allocate+pagestuff from unsigned tarball (Carl Dong)
ca85679eb43b8375a95d82101977829d08fb1e1b guix: Use clang-toolchain instead of clang (Carl Dong)
1aec0eda8fd31a57b0621eea616398017c2ead98 guix: Fallback to local build for substitute-enabled Guix users (Carl Dong)
1742f8e12d163852df09575e03edcd3db73198ee guix: Add early health check for guix-daemon (Carl Dong)
c1ae726a13ecfa5e7e9fdc3030a8110b8bb263f8 guix: More thoroughly control native toolchain (Carl Dong)
39741128d3775d198dbee34dc827353bfd18acd8 guix: Supply --link-profile (Carl Dong)
d55a1056ee565afed64e42d6f6efb6b0adc5599b guix: Add troubleshooting documentation entries (Carl Dong)
7f401c953f8bb3574cec48561e13ef3b47dedc6e guix: Adapt guix-build to prelude, restructure hier (Carl Dong)
4eccf063b252bfe256cf72d363a24cf0183e926e guix: Remove guix-build.sh filename extension (Carl Dong)
7753357a7bae98ec775c707b9dec4cea1e945802 guix: Add source-able bash prelude and utils (Carl Dong)
e5b49a01f5d0f631e7f08f86ca8a2c2b8213319f guix: Create windeploy inside distsrc-* (Carl Dong)
3e9982ab3877eb8fe0a8c0cb3d847ac0913c7336 contrib: Silence git-describe when looking for tag (Carl Dong)
d5a71e97853ea9e1b879e8c76bfb01d4bef33172 guix: Use --cores instead of --max-jobs (Carl Dong)
Pull request description:
This PR addresses a few hiccups encountered by the brave souls who've been experimenting with the Guix scripts:
- Resolves confusion between `--cores=` and `--max-jobs=`
- `guix`'s `--cores=` actually corresponds to make's `--jobs=`, so let's just control `--cores=` with our overridable env var
- `git-describe` will scream `fatal: no tag exactly matches '<hash>'` when looking for a tag, but we don't care, so silence that
- `windeploy/unsigned` should be inside `distsrc-*` and created idempotently (sorry I know this one annoyed people)
- Add troubleshooting documentation to `README.md`
- Add early health check for `guix-daemon` in case user forgot to start a `guix-daemon`
- Depending on configuration, a `--fallback` flag may be needed to tell Guix to not fail if substitutes fail but fallback to building locally
- `codesign_allocate` and `pagestuff` are now unnecessary for codesigning as we're now using `signapple`
A few robustness changes are also included:
- We supply the `--link-profile` flag, as some Guix packages may expect the profile to be available under `$HOME/.guix-profile`
- We now clear and manually set all toolchain-related env vars (e.g. `C*_INCLUDE_PATH`) ourselves, after patching a Qt::moc bug
- We use the native `clang-toolchain` package for darwin builds instead of `clang`, lining up with all our other toolchain packages.
Finally, we restructure the guix building hierarchy such that it looks something like:
```
guix-build-<short-hash-or-version-tag>
├── distsrc-<short-hash-or-version-tag>-${HOST}
│ ├── contrib
│ ├── depends
│ ├── src
│ └── ...
├── distsrc-<short-hash-or-version-tag>-...
└── output
├── dist-archive
│ └── bitcoin-<short-hash-or-version-tag>.tar.gz
├── *-linux-*
│ ├── bitcoin-<short-hash-or-version-tag>-*-linux-*-debug.tar.gz
│ └── bitcoin-<short-hash-or-version-tag>-*-linux-*.tar.gz
├── x86_64-apple-darwin18
│ ├── bitcoin-<short-hash-or-version-tag>-osx64.tar.gz
│ ├── bitcoin-<short-hash-or-version-tag>-osx-unsigned.dmg
│ └── bitcoin-<short-hash-or-version-tag>-osx-unsigned.tar.gz
└── x86_64-w64-mingw32
├── bitcoin-<short-hash-or-version-tag>-win64-debug.zip
├── bitcoin-<short-hash-or-version-tag>-win64-setup-unsigned.exe
├── bitcoin-<short-hash-or-version-tag>-win64.zip
└── bitcoin-<short-hash-or-version-tag>-win-unsigned.tar.gz
```
Separating guix builds by their version identifier (basically namespacing them) allows us to change the layout in the future without worry about potential naming conflicts.
ACKs for top commit:
sipa:
ACK 7476b46f1893a4858616d2a8456a7c43238851ed
laanwj:
ACK 7476b46f1893a4858616d2a8456a7c43238851ed
Tree-SHA512: 0e899aa941aafdf552b2a7e8a08131ee9283180bbef7334439e2461a02aa7235ab7b9ca9c149b80fc5d0a9f4bbd35bc80fcee26197c0836ba8eaf2d86ffa0386
c967fb7fb97a86fc434912d0fff621f9f50dfe80 guix: Remove libcap from manifest (Hennadii Stepanov)
7bbb409314c63750a9731dedaaf6dd6ecd17d988 guix: Update darwin native packages dependencies (Hennadii Stepanov)
Pull request description:
It is a #20470 follow up.
ACKs for top commit:
fanquake:
ACK c967fb7fb97a86fc434912d0fff621f9f50dfe80
Tree-SHA512: 66ce05770f578ba61a44c58747c5a2669f425a989ed987838058bd86e3b49e342ac5a4f8852fc49f2b3a86b58fb6a340fdf3e34c1fc19bdab910729febba4bc7
fbbb2d4fc13971c98c83a51635166ac532e71a32 lint: Fix spelling errors in comments (fyquah)
Pull request description:
Found some spelling errors while running spelling linter https://github.com/bitcoin/bitcoin/pull/21245
This PR fixes them.
ACKs for top commit:
fanquake:
ACK fbbb2d4fc13971c98c83a51635166ac532e71a32 - I thought we just fixed all of these.
Tree-SHA512: 95525040001f94e899b778c616cb66ebafb679dff88835b66fccf6349d8eb942d6b7374c536a44e393f13156bce9a32ed57e6a82bb02074d2b3cddb2696addb2
13a9fd11a507fd3398bc2c0a0575bdc81579243f guix: Passthrough SDK_PATH into container (Carl Dong)
Pull request description:
This is a usability improvement for Guix builders so that they don't have to extract the Xcode tarball into `depends/SDKs` every time.
Inspiration: https://github.com/bitcoin/bitcoin/pull/21089#issuecomment-778639698
ACKs for top commit:
laanwj:
Tested ACK 13a9fd11a507fd3398bc2c0a0575bdc81579243f
Tree-SHA512: 63392d537e48a0da9f0ee04a929613b139bef1ac5643187871c9ea5376afd2a3d95df0f5e0950ae0eccd2813b166667be98401e5a248ae9c187fe4e84e54d427
d98f4593cf00ab2973f8113e30506861b24383bc guix: Explicitly set umask in build container (Carl Dong)
Pull request description:
Opened as a separate PR to fix non-reproducibility found through testing here: https://github.com/bitcoin/bitcoin/pull/21089#issuecomment-783549633
Many thanks to everyone who helped find this!
ACKs for top commit:
laanwj:
ACK d98f4593cf00ab2973f8113e30506861b24383bc
fanquake:
ACK d98f4593cf00ab2973f8113e30506861b24383bc - I'm seeing matching hashes.
Tree-SHA512: ea339c3902f2f4dea32e8ef5cc675a1df0679530881260ae999aaaf7339d5b12c46e01e58677cbb079f33e573ad105e2b443a835f3e944ef8e943a25f83027f1
95990b9f3278360b63e79d6975af4ab5009c66ba guix: Update conservative space requirements (Carl Dong)
5e6df1132656995ce5b9ce279d5a9808ea52ab32 guix: Add support for powerpc64{,le} (Carl Dong)
Pull request description:
```
The new time-machine commit contains a few small changes that make the
powerpc cross-toolchain work.
```
See this compare to review my custom patches to Guix: 7d6bd44da5...6c9d16db96
ACKs for top commit:
fanquake:
ACK 95990b9f3278360b63e79d6975af4ab5009c66ba
Tree-SHA512: 464b0fb93d65962d8c27499293edb618d13d18f40d44e3eed96935e86d430666dfb1c5b8a30f99ffdfd17b44514ad88e358977390b689a2e3831d521f6f7b86a
d02076b8852d8faae95cee6e3de434460c07412a guix: Jump forwards in time-machine and adapt (Carl Dong)
f8ca8c5c28d3050b780e67d47a50ac65fc2dc3ad guix: Supply --keep-failed for debugging (Carl Dong)
Pull request description:
```
The new time-machine commit is Guix v1.2.0 with a yet-unupstreamed patch
for NSIS.
A few important changes:
1. Guix switched back from using CPATH to C{,PLUS}_INCLUDE_PATH as the
way to indicate #include search paths.
2. GCC's library is now split into a separate output, whereas before it
was included in the default output. This means that our gcc toolchain
packages need to propagate that output.
3. A few package versions were bumped
```
See this compare to review my custom patches to Guix: https://github.com/dongcarl/guix/compare/version-1.2.0...7d6bd44da57926e0d4af25eba723a61c82beef98
ACKs for top commit:
laanwj:
ACK d02076b8852d8faae95cee6e3de434460c07412a
Tree-SHA512: 896d5bf1b6e5fda2f0106013c568c119bbbb86cb31a8c0a22432bada9b7da51678b96374bf8fd7c15353698ba47ac9dd39874d40c39001281471db7c78bf1705
cb151b797aeabb376d37ef058aee34c06d36d487 build: Disable --disable-fuzz-binary for guix builds (Hennadii Stepanov)
fd7caae35fd3f83175247a79f7573e374779a851 build: Disable --disable-fuzz-binary for gitian builds (Hennadii Stepanov)
Pull request description:
Fuzz binary is not shipped to users.
This PR saves hundreds MB of the disk space for containers.
ACKs for top commit:
MarcoFalke:
review ACK cb151b797aeabb376d37ef058aee34c06d36d487
fanquake:
ACK cb151b797aeabb376d37ef058aee34c06d36d487
Tree-SHA512: 858e3816576c307b47915bb05de79a28029beaef8835c01f1bd6a764a0cf7f7f63ef8c2dc2c5944cb36cc9f4788d9b0590b8a5dda96940167252ba371cdbd078
f1694757ddbcb3635213b085e864851e285c8c12 guix: Fix typo (Carl Dong)
771c4b98a8693eee642f2b118b3193fe6e022291 guix: README: Add darwin HOSTS entry (Carl Dong)
8dbf18cb1d3260d34ba822ceb12e67b1f124ea13 guix: Check for macOS SDK before building anything (Carl Dong)
34b23f597ec52efb795d72e9e5620712d0010edd guix: Set ZERO_AR_DATE for darwin build determinism (Carl Dong)
f3835dc6a3732dcd4afbb5987f84dc27f2bf55af build: Make xorrisofs reproducible with -volume_date (Carl Dong)
c9eb4cf3a0f81bfd72f06fd43b5610f0a4f5e804 guix: Add support for darwin builds (Carl Dong)
37fe73a092b08fe9d7ce636a1021429de6cda757 build: Add var printing target to src/Makefile.am (Carl Dong)
Pull request description:
This PR brings our Guix builds on par with Gitian in terms of supported architectures.
Reviewers: if you run a build, please submit:
```
find output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
```
So that we can compare hashes and ensure reproducibility!
ACKs for top commit:
fanquake:
ACK f1694757ddbcb3635213b085e864851e285c8c12 - I think we can make some small usability improvements, but this is ok to merge now.
Tree-SHA512: 4af2b71654a9736467dcc681d10601c6eee37800d7847011a50585455b67b55d61742ca5604585f310a2fd75335b674e5e27dfb5169cb2f26e112aa4c411d8be
570e43fe72e13e0a82e25f7145704f62b2c2cc52 guix: Print build params inside/outside of container (Carl Dong)
2f9d1fdde66f4713351905ec73487e5288d20f8f guix: Move DISTSRC determination to guix-build.sh (Carl Dong)
0b7cd07bb56baa112ffa596fb23a905871031a36 guix: Move OUTDIR determination+creation to guix-build.sh (Carl Dong)
d27ff8b86aa66acec63b5713912bd4ad9470e66f guix: Add more sanity checks to guix-build.sh (Carl Dong)
57f95331464f097261c63fd1b6040536c58a03fa guix: Add section headings to guix-build.sh (Carl Dong)
38b7b2ed72b1f0f57bd9800c7fbb7b7c98a20ed0 genbuild: Specify rev-parse length (Carl Dong)
036dc740da3239cdcc13e0f299ab95b456f7118b docs: Point to contrib/guix/README.md in doc/guix.md (Carl Dong)
34f0fda2d31d2ada632ca1165b82aebdfd342efe guix: Small updates to README wording (Carl Dong)
402e3a5b1ed9de7057ce9955ea792ad1c2b9f2b5 guix: Update HOSTS README entry for new architectures (Carl Dong)
cfa7ceb21b14d1fa24c2541bf242a0ed539b9e1b guix: Remove README development environment section (Carl Dong)
93b6a8544a03d13733ca2ef769f76df587ad86c8 guix: Add ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS options (Carl Dong)
0f31e24703e25698d2d41fb54e30ec75a4a80943 guix: Add SUBSTITUTE_URLS option (Carl Dong)
444fcfca907d46cfeb52001599966cce25bdf54e guix: Make guix honor MAX_JOBS setting (Carl Dong)
Pull request description:
After live-demo-ing a Guix build (which completed successfully!) on achow101's stream, I realized there were a few quality of life improvements which can be made to improve the user experience of our Guix build process. Here are a few of them.
Notable changes:
1. When `MAX_JOBS` is specified, both `guix time-machine` and `guix environment` will now build up to `MAX_JOBS` packages at a time when creating the build environment
2. The instructions for using substitutes were incorrect, and has now been replaced with a `SUBSTITUTE_URLS` environment variable, which works well with shell's IFS splitting rules
3. New `ADDITIONAL_GUIX_{COMMON,TIMEMACHINE}_FLAGS` options, for more granular customization of the build process.
4. README cleanup
ACKs for top commit:
fanquake:
ACK 570e43fe72e13e0a82e25f7145704f62b2c2cc52 - lets move this forward.
Tree-SHA512: 4e8ab560522ade5efb5e8736aec0fb1a3f19ae9deb586c1ab87020816876f3f466a950b3f8c04d9fa1d072ae5ee780038c5c9063577049bdd9db17978e11c328
faa2f06f5eaf8578873495f44603ee74d7a1abf4 scripted-diff: [build] Ensure source tarball has leading directory name (MarcoFalke)
Pull request description:
This has been fixed in 0.20, so it needs to be fixed on master as well to avoid a regression
#18945
ACKs for top commit:
laanwj:
ACK faa2f06f5eaf8578873495f44603ee74d7a1abf4
hebasto:
ACK faa2f06f5eaf8578873495f44603ee74d7a1abf4, tested gitian builds only.
promag:
ACK faa2f06f5eaf8578873495f44603ee74d7a1abf4.
Tree-SHA512: e3b025c29c45b025002abc35262bb5d771f6cbd807f1c256c477c243685e93cd43ad9f642b38e3cf218590912abe6ea0ddfec3bfbef36f99080aad74ed6cc0af
f852761aec81ed23c7b9e4546c08d1ef303f2507 guix: Add clarifying documentation for V env var (Carl Dong)
85f4a4b0822e3aa10310c4623eff719f301e9263 guix: Make V=1 more powerful for debugging (Carl Dong)
Pull request description:
```
- Print commands in both unexpanded and expanded forms
- Set VERBOSE=1 for CMake
```
Ping MarcoFalke hopefully you use `V=1` already for the Guix builds on DrahtBot?
ACKs for top commit:
fanquake:
ACK f852761aec81ed23c7b9e4546c08d1ef303f2507. Ran a Windows Guix build and compared the output from master and this PR when using `V=1`. i.e `HOSTS=x86_64-w64-mingw32 PATH="/root/.config/guix/current/bin${PATH:+:}$PATH" V=1 ./contrib/guix/guix-build.sh`.
Tree-SHA512: 8bc466fa7b869618bbd5a0a91c6b23d4785009289f8dfb93b0349317463a9ab9ece128c72436e02a0819722a63e703100aed15807867a716fda891292fcb9d9d
bfe1ba2f5b36056e0c41edf8206b93d3d83098df rel-builds: Specify core.abbrev for git-rev-parse (Carl Dong)
27e63e01cce368d67092de8f0c736927d6f6aa69 build: Accomodate makensis v2.x (Carl Dong)
1f2c39a30e0f82046c7aecddfda3eb99cb536816 guix: Remove logical cores requirement (Carl Dong)
a4f6ffa71e335d4b2a6bf525b7f416968f9cd9f7 lint: Also enable source statements for non-gitian (Carl Dong)
d256f91cb1b0d6ff5170106b99b0266cbe51f5a2 rel-builds: Directly deploy win installer to OUTDIR (Carl Dong)
fa791da02f9684e3fd554b687fb692ae6a23d65a nsis: Specify OutFile path only once (Carl Dong)
14701604d0904bc5bbf1c67de08f8ee6d3215523 guix: Expose GIT_COMMON_DIR in container as readonly (Carl Dong)
f5a6ac4f48b18f93050d77bcb23f9cf45ec34647 guix: Make source tarball using git-archive (Carl Dong)
395c1137f630dc495ffb2752a23bc1dfd470ee53 gitian: Limit sourced script to just assignments (Carl Dong)
Pull request description:
Based on: #18556
Related: https://github.com/bitcoin/bitcoin/pull/17595#discussion_r399728721
ACKs for top commit:
fanquake:
ACK bfe1ba2f5b36056e0c41edf8206b93d3d83098df - I agree with Carl, and am going to merge this. I'd like for Linux Guix builds to be working again, and we can rebase #18818.
Tree-SHA512: c87ada7e3de17ca0b692a91029b86573442ded5780fc081c214773f6b374a0cdbeaf6f6898c36669c2e247ee32aa7f82defb1180f8decac52c65f0c140f18674
f2b5b0a3b48f9241971c14238040048f8b630811 build: add linker optimization flags to guix (fanquake)
b8b050a8d642e38c81d1e451750c2c8db92fee5e build: add linker optimization flags to gitian descriptors (fanquake)
Pull request description:
This PR adds `-Wl,O2` to our gitian and guix LDFLAGS. This makes the linker perform certain optimisations (and is different from LTO).
Any -O argument will enable optimizations in GNU ld. We can use -O2 here, as this matches our compile flags. Note that this would also enable additional optimizations if using the lld or gold linkers, when compared to -O0.
A nice writeup + diagrams of some of these optimizations is available here: http://lwn.net/Articles/192624/.
#### master
```bash
# bitcoind
Histogram for `.gnu.hash' bucket list length (total of 3 buckets)
Length Number % of total Coverage
0 1 ( 33.3%) 0.0%
1 0 ( 0.0%) 0.0%
2 1 ( 33.3%) 40.0%
3 1 ( 33.3%) 100.0%
```
```bash
# bitcoin-qt
Histogram for `.gnu.hash' bucket list length (total of 3 buckets)
Length Number % of total Coverage
0 0 ( 0.0%) 0.0%
1 1 ( 33.3%) 10.0%
2 0 ( 0.0%) 10.0%
3 0 ( 0.0%) 10.0%
4 1 ( 33.3%) 50.0%
5 1 ( 33.3%) 100.0%
```
#### this PR:
```bash
# bitcoind
Histogram for `.gnu.hash' bucket list length (total of 8 buckets)
Length Number % of total Coverage
0 3 ( 37.5%) 0.0%
1 5 ( 62.5%) 100.0%
```
```bash
# bitcoin-qt
Histogram for `.gnu.hash' bucket list length (total of 19 buckets)
Length Number % of total Coverage
0 9 ( 47.4%) 0.0%
1 10 ( 52.6%) 100.0%
```
#### GNU ld -O
> If level is a numeric values greater than zero ld optimizes the output. This might take significantly longer and therefore probably should only be enabled for the final binary. At the moment this option only affects ELF shared library generation. Future releases of the linker may make more use of this option. Also currently there is no difference in the linker’s behaviour for different non-zero values of this option. Again this may change with future releases.
#### lld -O
> Optimize output file size
ACKs for top commit:
dongcarl:
ACK f2b5b0a3b48f9241971c14238040048f8b630811
laanwj:
ACK f2b5b0a3b48f9241971c14238040048f8b630811
Tree-SHA512: e53f3a4338317dbec65d3a93b57b5a6204aabdf9ac82d99447847a3c8627facc53c58c2cf947376f13edd979fc8129a80f18d9ebeccd191a576c83f1dad5c513
a35e3235891d35daa167116cc70340140e883f06 guix: Appease travis. (Carl Dong)
0b66d22da5f53640e22f05adf880782c613e6d0f guix: Use gcc-9 for mingw-w64 instead of 8 (Carl Dong)
ba0b99bdd613ba7f17c6247ece3001e1b44759a3 guix: Don't set MINGW_HAS_SECURE_API CFLAG in depends (Carl Dong)
93439a71eda49fb69f1e82966a23a946733aa6fa guix: Bump to upstream commit with mingw-w64 changes (Carl Dong)
35a96792dda9e78165b1598aeac7b2ab759e7be5 guix: Check mingw symbols, improve SSP fix docs (Carl Dong)
449d8fe25bbe25daacfc67aa89ca32b0a3254c5a guix: Expand on INT trap message (Carl Dong)
3f1f03c67a8e9edf487f08d272adb18b0a3942c8 guix: Spelling fixes (Carl Dong)
ff821dd2a1c600488d11e7d9a20e9179ecc9144b guix: Reinstate make-ssp-fixed-gcc (Carl Dong)
360a9e0ad50a36ec79a1a160dbed3966689fd41c guix: Bump time-machine for mingw-w64 patches (Carl Dong)
93e41b7e3b54c17fd1b4c61ee95fc0dc2827e954 guix: Use gcc-8 for mingw-w64 instead of 7 (Carl Dong)
ef4f7e4c45c60a69406134122f091c77c6ef740f guix: Set the well-known timezone env var (Carl Dong)
acf4b3b3b5accf60a19441a0298ef27001b78e72 guix: Make x86_64-w64-mingw32 builds reproducible (Carl Dong)
c4cce00eac691625b78b92f7dba0b7f57def19e5 guix: Remove dead links from README. (Carl Dong)
df953a4c9a6143f45864757b706c88b6fa70545a guix: Appease shellcheck. (Carl Dong)
91897c95e191d293eb27d8af15cbeafc5b8f3895 guix: Improve guix-build.sh documentation (Carl Dong)
570d769c6c59b9f6d1a2b95b2ed60432cb33b3ba guix: Build support for Windows (Carl Dong)
Pull request description:
~~Based on: https://github.com/bitcoin/bitcoin/pull/16519~~
Based on: #17933 (Time Machines are... shall we say... superior 😁)
This PR allows us to perform Guix builds for the `x86_64-w64-mingw32` target. We do this _without_ splitting up the build script like we do in Gitian by using this newfangled alien technology called `case` statements. (This is WIP and might be changed to `if` statements soon)
ACKs for top commit:
fanquake:
ACK a35e3235891d35daa167116cc70340140e883f06 2/3
Tree-SHA512: c471951c23eb2cda919a71285d8b8f2580cb20f09d5db17b53e13dbd8813e01b3e7a83ea848e4913fd0f2bc12c6c133c5f76b54e65c0d89fed4dfd2e0be19875
