c0f5cc14ef9fae2b2de4222ee061729629ebb6b4 build: Fix `libmultiprocess` cross-compiling to Linux hosts (Hennadii Stepanov)
Pull request description:
To successfully call the [`capnp_generate_cpp()`](d576d975de/CMakeLists.txt (L45)) function, the `libmultiprocess` build system must be provided with paths to the native `capnp` and `capnpc-c++` tools.
This [comment](https://github.com/bitcoin/bitcoin/issues/24387#issuecomment-1054776195) points the same:
> I think `packages/libmultiprocess.mk` probably needs to be passing a `-DCAPNP_EXECUTABLE=.../depends/arm-linux-gnueabihf/native/bin/capnp` argument to cmake. Also the package should have dependencies on both `capnp` and `native_capnp`.
Fixesbitcoin/bitcoin#24387.
ACKs for top commit:
ryanofsky:
Code review ACK c0f5cc14ef9fae2b2de4222ee061729629ebb6b4
Tree-SHA512: 2986d8bf98d2761eceba21b1897145c5185a0922d4c2084e8812d4d07dc94237e5c2809036641c4f7c491a3414727fff328cba91ce138b89e37ec5cba61d8f61
735610940c0cac72343ca247eb3062b2e09512a0 build: set --build when configuring packages in depends (fanquake)
Pull request description:
After reading [#Specifying-Target-Triplets](https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/autoconf.html#Specifying-Target-Triplets) in the autoconf manuel, my understanding is that this change should mostly be a no-op, as `--build` defaults to the output of [`config.guess`](https://github.com/bitcoin/bitcoin/blob/master/depends/config.guess), however, this may be slightly more correct
> For historical reasons, whenever you specify --host, be sure to specify --build too; this will be fixed in the future.
and will quell some warnings in depends (#16354), i.e:
> configure: WARNING: If you wanted to set the --build type, don't use --host.
If a cross compiler is detected then cross compile mode will be used.
If anything, this also explicitly enables cross-compilation mode when `--host` differs from `--build`. As for "fixed in the future", this is the case for Autoconf 2.70+.
If we don't make this change, I think we should consider closing #16354. We've now got a good handle on our depends configure flags, and we can just leave the last few warnings as they are. It's also unlikely that we are going to add the `xorg-macros` package to depends.
Guix builds at 735610940c0cac72343ca247eb3062b2e09512a0:
```bash
bash-5.1# find output -type f -name *$(git rev-parse --short HEAD)*.* -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
afabf99650d57abd3df2c7e6253a3281f9814ada77d01c28ffb57c740b14fd8e output/bitcoin-735610940c0c-aarch64-linux-gnu-debug.tar.gz
6e9dedced61489f7734c5387ee05823db68eb4b890fa05f11541c1b966576446 output/bitcoin-735610940c0c-aarch64-linux-gnu.tar.gz
5333a3c5978a61966251e56fedab8e059d9d8a1a7231a0908095bb373204a81f output/bitcoin-735610940c0c-arm-linux-gnueabihf-debug.tar.gz
d28e1c166b96e58cf40b3f9801c3050c2785b25eeb261d44eeb31a32a75e134c output/bitcoin-735610940c0c-arm-linux-gnueabihf.tar.gz
e00b0149c52e9207be1cfb56ec3b715b55f41691401095f7a7038ac124f08d80 output/bitcoin-735610940c0c-osx-unsigned.dmg
ae5052ad1a05916bc6656b034c169097471fd9bdba16b606e2bcb592c9395544 output/bitcoin-735610940c0c-osx-unsigned.tar.gz
8586a9b8b82a8832397fcd0756aa9649adc3e61ae579fb370b99fe406298199f output/bitcoin-735610940c0c-osx64.tar.gz
d560a8781bae856b6593e9b731fa99de0a4ad7bd3e03084166dc98904ece9e7b output/bitcoin-735610940c0c-powerpc64-linux-gnu-debug.tar.gz
a24cf28b7efd5ebdc892e7e8809a05b3405bb4222f7d2eaa32de265214c78f28 output/bitcoin-735610940c0c-powerpc64-linux-gnu.tar.gz
ad29421842dbbe96526eb597a7ff2970f0acae8c3e75dc6c2100716a2ba20606 output/bitcoin-735610940c0c-powerpc64le-linux-gnu-debug.tar.gz
ee7a8ccfd68cc2297bb96eaaacf7b3eb939a5e39cecc3c710dbe828518f1bfbd output/bitcoin-735610940c0c-powerpc64le-linux-gnu.tar.gz
185bf4ea3b35072a8dd54aedbafc0892ec1e486c8ebd311f05f0fb47a19058cf output/bitcoin-735610940c0c-riscv64-linux-gnu-debug.tar.gz
803078b15a9bbaa567176827639895e762edbe80844b80400f8a9581e5311d32 output/bitcoin-735610940c0c-riscv64-linux-gnu.tar.gz
d920327a32a42a5abadd247f0d3f3ee0ab0488a28892c11838d99bede6456723 output/bitcoin-735610940c0c-win-unsigned.tar.gz
9f1741a37e294af153d57ad4a6dad3e5f1cdf75f1588c86cb467abf0c177ae27 output/bitcoin-735610940c0c-win64-debug.zip
2d1a91552c0dd7bb400ce6c71d2e98de702743a97afe5f4b0945865f251f3aea output/bitcoin-735610940c0c-win64-setup-unsigned.exe
8a70020bb0cb68516acf4b2cc89832d72373f31611bd075826a30a817a071cd2 output/bitcoin-735610940c0c-win64.zip
93b03837679474905538e96c44ba81ee5653795ad333dc00569b92c82dc6ea31 output/bitcoin-735610940c0c-x86_64-linux-gnu-debug.tar.gz
6ccd046a83ef5089667ad426875319beb83e2b2a5fb4d4459d5c5ce8d75f1a31 output/bitcoin-735610940c0c-x86_64-linux-gnu.tar.gz
12771acb17de84cda8e81c59704c00b1d4725ff51fa84c9067dc61a8dc795bdc output/src/bitcoin-735610940c0c.tar.gz
```
Gitian builds:
```bash
# macOS:
7d087749f610e5e3eca91c730d41afd4d404808e276242d05cedb89403de247f bitcoin-735610940c0c-osx-unsigned.dmg
96d68b4f0042dc40f5e8e362fa84dcf8e122dcb566889feb48ad8a982847dd8b bitcoin-735610940c0c-osx-unsigned.tar.gz
2dea72cbe5c3e228babd88b516a6f43b0e040e9510ea95b62f07fc0815bf8d22 bitcoin-735610940c0c-osx64.tar.gz
12771acb17de84cda8e81c59704c00b1d4725ff51fa84c9067dc61a8dc795bdc src/bitcoin-735610940c0c.tar.gz
478dbda9d551a8bed985c61c625df3f543afab26989a52bc44d5a4745715af81 bitcoin-core-osx-22-res.yml
# Windows:
c31a5118dfa21259b9868e2c4858c635becb786c2ee8af92302da7811a1efe9b bitcoin-735610940c0c-win-unsigned.tar.gz
5c171ec3aee830abc8ae843d674d5fe7a57420e782ee765b6992914ae5b7671a bitcoin-735610940c0c-win64-debug.zip
ca010ced982f75d111e01f3759b961deb83e7bf511b1ee9c9041f4830570f2a0 bitcoin-735610940c0c-win64-setup-unsigned.exe
62514b01281f81af6057fb12cc5e3813c1d8f6532d0d8ac8f6f2909a595aa1dd bitcoin-735610940c0c-win64.zip
12771acb17de84cda8e81c59704c00b1d4725ff51fa84c9067dc61a8dc795bdc src/bitcoin-735610940c0c.tar.gz
e26b55d8302a419f594396db6ff441aaeb5e275e02fde7b1b4eb092270f902e7 bitcoin-core-win-22-res.yml
# Linux:
e5be4deb91cb372de484468c0575a04f907f982e35a9841659bb213540f770d0 bitcoin-735610940c0c-aarch64-linux-gnu-debug.tar.gz
6169e822679ece2bb6af7027362a6f373eb939eef9d89edc58c21190b4083c7d bitcoin-735610940c0c-aarch64-linux-gnu.tar.gz
aee5308767c5c930853a80ab94325b2a4ab3dbe9a7a20c7a9d60d57af2b5383f bitcoin-735610940c0c-arm-linux-gnueabihf-debug.tar.gz
a9cf6e56d859fffd5a2b5f1e6360515bc6b06b048776b93cc08954859457a251 bitcoin-735610940c0c-arm-linux-gnueabihf.tar.gz
7617d85c358e6614fd40026a8d346f9edba5b5139e3daade2ec2aec72aed3a1f bitcoin-735610940c0c-powerpc64-linux-gnu-debug.tar.gz
f76e46a8b130812541ebcaaba173b4cbe2ec1781e348579b44f96e9244cc2475 bitcoin-735610940c0c-powerpc64-linux-gnu.tar.gz
19ada7abb801422bd7772d7ee559004cab11a46141a747aabd27085e8c3d200d bitcoin-735610940c0c-powerpc64le-linux-gnu-debug.tar.gz
f3a6bbefe9f47b3960e3b9eeb3f740ecddf8773ae76961ced6a3cdf82503e37f bitcoin-735610940c0c-powerpc64le-linux-gnu.tar.gz
42ca333e8f82b4ec0b2354efc7c4f1c27ed32d60c35b88e33d26833419f19e69 bitcoin-735610940c0c-riscv64-linux-gnu-debug.tar.gz
dde930d9134fb5be1374014db3f6f503b614ae801a1d77f6f4b82a5f11eb2518 bitcoin-735610940c0c-riscv64-linux-gnu.tar.gz
82e1b4cc8aac309a99458cc27fd0285215a8af53780094c4990303ce8948cf0c bitcoin-735610940c0c-x86_64-linux-gnu-debug.tar.gz
ee8caad2ccddd8eaf8aa2d19040d859130168183355b09a2a0f77a59a97fcaee bitcoin-735610940c0c-x86_64-linux-gnu.tar.gz
12771acb17de84cda8e81c59704c00b1d4725ff51fa84c9067dc61a8dc795bdc src/bitcoin-735610940c0c.tar.gz
d90545b13226a338c994d9e538f468a6fafd694273bf71e14614feeaf5727ad4 bitcoin-core-linux-22-res.yml
```
ACKs for top commit:
dongcarl:
Code Review ACK 735610940c0cac72343ca247eb3062b2e09512a0
hebasto:
ACK 735610940c0cac72343ca247eb3062b2e09512a0, tested on Linux Mint 20.1 (x86_64).
Tree-SHA512: 13ce05269309a12d03b81a539ba5585003785f7b1bb7894a9353385a6e9c709f920cc4f79322f28d7a256809754f201e70be6b917915c7b14e1461530075781d
7d0271b5c30f86e4af175a5ab7df5e593cd85195 depends: Set CMAKE_INSTALL_RPATH for native packages (Russell Yanofsky)
Pull request description:
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).
---
After #19685 started setting `LDFLAGS`, the `INSTALL_RPATH_USE_LINK_PATH` cmake option used in the libmultiprocess build no longer works, so it is neccessary to set `CMAKE_INSTALL_RPATH` as a fallback.
It's unclear currently whether the bad interaction between `INSTALL_RPATH_USE_LINK_PATH` and `LDFLAGS` is a bug, but the issue is reported:
- https://github.com/bitcoin/bitcoin/issues/19981#issuecomment-696680877
- https://discourse.cmake.org/t/install-rpath-use-link-path-not-working-when-cmake-exe-linker-flags-ldflags-is-set/1892Fixes#19981
ACKs for top commit:
fanquake:
ACK 7d0271b5c30f86e4af175a5ab7df5e593cd85195 - I haven't looked in depth, but I've re-read through #19981 and checked the failure by testing #19160 (with this reverted):
dongcarl:
ACK 7d0271b Looked into this a bit, it makes sense that for the things we build in depends, we want the library search to start in depends. It seems reasonable to expect this to happen automatically when `CMAKE_INSTALL_PREFIX` and `INSTALL_RPATH_USE_LINK_PATH` are set, but oh well...
Tree-SHA512: 97cc5801c3204c14cd33004423631456ca0701e2127ee5146810a76e2f4aac9de1f4b5437402a4329cda54e022dc99270fee7e38c2995765f36b3848215fa78e
b8936883573708059357a66f67fad9dc77a8bade depends: Specify LDFLAGS to cmake as well (Carl Dong)
b3f541f618fe1f3d44baf6a0dd4299173c81f752 depends: Prepend CPPFLAGS to C{,XX}FLAGS for CMake (Carl Dong)
8e121e550953711cd03d7b6c221afd065c325c5e depends: Cleanup CMake invocation (Carl Dong)
8c7cd0c6d9f295bcb6913e3c69c9dac4ce2b25ce depends: More robust cmake invocation (Carl Dong)
3ecf0eca634601da216b06f091f95456c047f39c depends: Use $($(package)_cmake) instead of cmake (Carl Dong)
Pull request description:
- Use `$($(package)_cmake)` instead of invoking `cmake` directly
- Use well-known env vars instead of overriding CMake variables
ACKs for top commit:
ryanofsky:
Code review ACK b8936883573708059357a66f67fad9dc77a8bade. Only changes since last review are new commits adding whitespace, cppflags and ldflags to cmake invocation
Tree-SHA512: cfcd8cc9dcd0b336cf48b82fca9fe4bbc7930ed397cb7a68a07066680eb4c1906a6a9b5bd2589b4b4999e8f16232fa30ee9b376b60f4456d0fff931fbf9cc19a
e2bab2aa162ae38b2bf8195b577c982402fbee9d multiprocess: add multiprocess travis configuration (Russell Yanofsky)
603fd6a2e708c04ef6c9880f89d0a4cbaa6fc7c5 depends: add MULTIPROCESS depends option (Russell Yanofsky)
5d1377b52bfcd4edf8553aaf332bfeb92fc554cc build: multiprocess autotools changes (Russell Yanofsky)
Pull request description:
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).
---
This PR consists of build changes only. It adds an `--enable-multiprocess` autoconf option (off by default and marked experimental), that builds new `bitcoin-node` and `bitcoin-gui` binaries. These currently function the same as existing `bitcoind` and `bitcoin-qt` binaries, but are extended in #10102 with IPC features to execute node, wallet, and gui functions in separate processes.
In addition to adding the `--enable-multiprocess` config flag, it also adds a depends package and autoconf rules to build with the [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) library, and it adds new travis configuration to exercise the build code and run functional tests with the new binaries.
The changes in this PR were originally part of #10102 but were moved into #16367 to be able to develop and review the multiprocess build changes independently of the code changes. #16367 was briefly merged and then reverted in #18588. Only change since #16367 has been dropping the `native_boost.mk` depends package which was pointed out to be no longer necessary in https://github.com/bitcoin/bitcoin/pull/16367#issuecomment-596484337 and https://github.com/bitcoin/bitcoin/pull/18588#pullrequestreview-391765649
ACKs for top commit:
practicalswift:
ACK e2bab2aa162ae38b2bf8195b577c982402fbee9d
Sjors:
tACK e2bab2aa162ae38b2bf8195b577c982402fbee9d on macOS 10.15.4
hebasto:
ACK e2bab2aa162ae38b2bf8195b577c982402fbee9d, tested on Linux Mint 19.3 (x86_64):
Tree-SHA512: b5a76eab5abf63d9d8b6d628cbdff4cc1888eef15cafa0a5d56369e2f9d02595fed623f4b74b2cf2830c42c05a774f0943e700f9c768a82d9d348cad199e135c
e644591426fb4ee1bba5c4bfdde85eb378258272 build, refactor: Drop useless `call` Make function (Hennadii Stepanov)
Pull request description:
Using the [`call`](https://www.gnu.org/software/make/manual/html_node/Call-Function.html) function with `$(package)_*_cmds` is effectively noop because the latter, which could be found in `<package>.mk` files, do not use temporary `$(1)` variable at all.
This PR removes useless calls of the `call` function, and makes code more readable and easier to reason about.
No change in resulted dependency binaries could be easy verified with bitcoin/bitcoin/#21995.
ACKs for top commit:
laanwj:
Code review ACK e644591426fb4ee1bba5c4bfdde85eb378258272
shaavan:
Code review ACK e644591426fb4ee1bba5c4bfdde85eb378258272
Tree-SHA512: 8481fa0dc5bbf7dd6a180f7fae5a2ccc07f85b50c7a966bceb2d7e010e07e5f211ee3f74f8ac79bc5acfde5f0764264d599d959ff3ebb8511b1b4a33f79509bd
8494dcae0e32716fd7cc7abeacf0a795a1303e6a Replace $(AT) with .SILENCE. (Dmitry Goncharov)
Pull request description:
This reduces the amount of syntax noise in the makefiles.
Setting V=1 still enables verbose logging.
The only noticeable difference in behavior is that, unless V=1 is specified, make won't print its own messages like
make: Nothing to be done for 'all', make: 'all' is up to date, or touch <file>, if -t is specified.
ACKs for top commit:
laanwj:
Tested ACK 8494dcae0e32716fd7cc7abeacf0a795a1303e6a
Tree-SHA512: 66b9111229995aa54a9e87f4571648727d89b8529caec651063cdfe5c00a64341371b648701d192b2334df0614617a00c28eaa56c7f08ee9c00127cada0293ab
fa872c9af397837bb17859b5f43adec71239682a depends: Fix id_string invocations (Carl Dong)
Pull request description:
Closes: #21242
```
Reproduced from depends/Makefile comment:
When invoking a shell, GNU Make special-cases exit code 127 (command not
found) by not capturing the output but instead passing it through. This
is not done for any other exit code.
Therefore, we require a "|| true" to avoid this behaviour when in an
environment where the build_* or host_* may not exist yet.
```
ACKs for top commit:
laanwj:
Concept and light code review ACK fa872c9af397837bb17859b5f43adec71239682a
Tree-SHA512: 9ce88381aec579d956572cf70c4f69dc5a3873f0d2af14a71cf24814192a89452b8280258bed8cca804e4bd2644db056d213ab733df46a10560a47079524d8ac
1ba49bcdc216162f513e74e371ee9f26418a390e build: pass --enable-option-checking to applicable packages (fanquake)
bcff8e21b1bb360c3cf1e1d2e1a2a54ec7581720 build: only pass --disable-dependency-tracking to packages that understand it (fanquake)
Pull request description:
By blanket passing `--disable-dependency-tracking` to all depends packages we end up with warnings (i.e in `bdb` or `freetype`) like:
```bash
configure: WARNING: unrecognized options: --disable-dependency-tracking
```
Instead, only pass it to packages that actually understand it. Related to https://github.com/bitcoin/bitcoin/issues/16354.
More info on `--disable-dependency-tracking` available [here](https://www.gnu.org/software/automake/manual/html_node/Dependency-Tracking.html).
This PR also adds `--enable-option-checking` as a configure option to all applicable packages.
ACKs for top commit:
laanwj:
ACK 1ba49bcdc216162f513e74e371ee9f26418a390e
theuni:
ACK 1ba49bcdc216162f513e74e371ee9f26418a390e
Tree-SHA512: 6d3143ad5f5d1abed5e0a0b2ffbb4323f21c7bf24b0b8df26fb1b3cd16cf5309bbb830aa5aaec99164d5bbe8e9c62b97aa3e97ee1ddc2c7612bf8ff88a63885e
ef77549000ac46eededa3e4a89466024db111524 build: Include native_X.mk before X.mk (Hennadii Stepanov)
Pull request description:
Unfortunately, our depends build system still lacks 100% reliability.
On master (57982f419e36d0023c83af2dd0d683ca3160dc2a):
```
$ make --no-print-directory -C depends print-capnp_version MULTIPROCESS=1
capnp_version=
```
This PR fixes this issue:
```
$ make --no-print-directory -C depends print-capnp_version MULTIPROCESS=1
capnp_version=0.7.0
```
This PR split off from bitcoin/bitcoin#22552, bitcoin/bitcoin#22555 and bitcoin/bitcoin#22708.
ACKs for top commit:
Zero-1729:
tACK ef77549000ac46eededa3e4a89466024db111524
fanquake:
ACK ef77549000ac46eededa3e4a89466024db111524
Tree-SHA512: 9818277c82fcef778282a9fe854e48c6b6118cc757fcffa6fd4a2418e3de8b2a2b9d54e31804d22a53b28a28ffa5b8610c983add1969be0c4b3e13e0f2267271
7a89f2e6c539a54bcaa24bff41aae3910244ad3d build: Fix target name (Hennadii Stepanov)
Pull request description:
It seems like a typo :)
This PR:
- fixes errors when building a package in depends for `HOST=x86_64-apple-darwin16` (fix#19799)
- is a correct alternative to d25e0e308f from #19764
ACKs for top commit:
icota:
tACK 7a89f2e6c5
dongcarl:
Code Review ACK 7a89f2e6c539a54bcaa24bff41aae3910244ad3d
theuni:
ACK 7a89f2e6c539a54bcaa24bff41aae3910244ad3d.
Tree-SHA512: a0bcbc6805d3450e201476ef1e22e0eb53903db1586c5515314c19afd337bded887e56de0fbe62feaf359b2de15dbccd49a44f1a8b566b4c64f5ae3d94a2ab6d
6d44c5ebf9 depends: Add commands for each package for each stage (Carl Dong)
80f0e05b70 depends: Preprocessing doesn't care about deps (Carl Dong)
Pull request description:
Adds make targets for each package for each stage, e.g.
```sh
make zeromq_configured
```
ACKs for commit 6d44c5:
MarcoFalke:
ACK 6d44c5ebf97af4b357079fe4bc2130f98e1d0fd2 (Haven't looked at the code changes, but adding this feature makes sense)
ryanofsky:
ACK 6d44c5ebf97af4b357079fe4bc2130f98e1d0fd2
Tree-SHA512: f1ac0aecfd2372aed09ca63603e2634552cb3f6ff9d610f958e2a66952d7d9e870b4c32b7d996886879e6d3016532272e8b1a10c13ed7b31009c6c96f786db9f
f7696e6183 depends: qt: Don't hardcode pwd path (Carl Dong)
89bee1bdbf depends: tar: Always extract as yourself (Carl Dong)
340ef50772 depends: Defer to Python detected by autoconf (Carl Dong)
Pull request description:
Removes some implicit assumptions that the depends system has about its environment and, as a side-effect, makes it possible to build the depends tree under severely privilege-limited environments such as containers built by Guix.
Tree-SHA512: e8618f9310a0deae864b44f9b60baa29e6225ba16817973ff7830b55798ebd4343aa06da6c1f92682a7afb709d26f80d6ee794a139d4d44c27caf4f0c8fe95fc
* Retry downloading of depends sources 3 times
* Manually invoke apt-get update and install instead of using addon
This allows us to use travis_retry on these operations.
* Add exception for depends/Makefile in .gitignore
* Remove --retry from curl call
This avoids future confusion as we don't rely on curl anymore to properly
retry downloads. curls does only retry on transient errors, while Travis
might cause all kinds of errors which circumvent the retry logic of curl.
In some cases (Travis), sources and build caches may be moved around in-between
builds, and we can't necessarily trust that everything is still intact.
This introduces pre-build checks that verify against stashed checksums.
Note that this will cause all sources to be re-downloaded, since cached sources
weren't trustworthy before this.
Broken hash logic caused all depends on some platforms (osx at least) to end up
with the same build-id. Without this fix, nothing will be rebuilt when recipes
or dependencies change.
Since the last commit will force rebuilds of all depends, take the opportunity
to clean up a few other things that would trigger rebuilds as well.
- Move source stamps to the sources dir so that SOURCES_PATH is respected for
"make download".
- Only print "fetching..." when actually downloading a file.
- Avoid using non-deterministic paths for the recipe hash (patch location).
This should ensure that all builders get the same resulting build-ids.
- Use a per-package source paths. This will allow for removing old source files
in the future.
- Use a host-agnostic path for downloads which gets cleaned up properly.
Some sources are renamed after download, since the filenames don't play nice
with (for example) gitian. This fixes the rename.
Needed for OSX build as it renames a file.
