e2bab2aa162ae38b2bf8195b577c982402fbee9d multiprocess: add multiprocess travis configuration (Russell Yanofsky)
603fd6a2e708c04ef6c9880f89d0a4cbaa6fc7c5 depends: add MULTIPROCESS depends option (Russell Yanofsky)
5d1377b52bfcd4edf8553aaf332bfeb92fc554cc build: multiprocess autotools changes (Russell Yanofsky)
Pull request description:
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).
---
This PR consists of build changes only. It adds an `--enable-multiprocess` autoconf option (off by default and marked experimental), that builds new `bitcoin-node` and `bitcoin-gui` binaries. These currently function the same as existing `bitcoind` and `bitcoin-qt` binaries, but are extended in #10102 with IPC features to execute node, wallet, and gui functions in separate processes.
In addition to adding the `--enable-multiprocess` config flag, it also adds a depends package and autoconf rules to build with the [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) library, and it adds new travis configuration to exercise the build code and run functional tests with the new binaries.
The changes in this PR were originally part of #10102 but were moved into #16367 to be able to develop and review the multiprocess build changes independently of the code changes. #16367 was briefly merged and then reverted in #18588. Only change since #16367 has been dropping the `native_boost.mk` depends package which was pointed out to be no longer necessary in https://github.com/bitcoin/bitcoin/pull/16367#issuecomment-596484337 and https://github.com/bitcoin/bitcoin/pull/18588#pullrequestreview-391765649
ACKs for top commit:
practicalswift:
ACK e2bab2aa162ae38b2bf8195b577c982402fbee9d
Sjors:
tACK e2bab2aa162ae38b2bf8195b577c982402fbee9d on macOS 10.15.4
hebasto:
ACK e2bab2aa162ae38b2bf8195b577c982402fbee9d, tested on Linux Mint 19.3 (x86_64):
Tree-SHA512: b5a76eab5abf63d9d8b6d628cbdff4cc1888eef15cafa0a5d56369e2f9d02595fed623f4b74b2cf2830c42c05a774f0943e700f9c768a82d9d348cad199e135c
060a2a64d40d75fecb60b7d2b9946a67e46aa6fc ci: remove boost thread installation (fanquake)
06e1d7d81d5a56d136c6fc88f09a2b0654a164f9 build: don't build or use Boost Thread (fanquake)
7097add83c8596f81be9edd66971ffd2486357eb refactor: replace Boost shared_mutex with std shared_mutex in sigcache (fanquake)
8e55981ef834490c438436719f95cbaf888c4914 refactor: replace Boost shared_mutex with std shared_mutex in cuckoocache tests (fanquake)
Pull request description:
This replaces `boost::shared_mutex` and `boost::unique_lock` with [`std::shared_mutex`](https://en.cppreference.com/w/cpp/thread/shared_mutex) & [`std::unique_lock`](https://en.cppreference.com/w/cpp/thread/unique_lock).
Even though [some concerns were raised](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-726214696) in #16684 with regard to `std::shared_mutex` being unsafe to use across some glibc versions, I still think this change is an improvement. As I mentioned in #21022, I also think trying to restrict standard library feature usage based on bugs in glibc is not only hard to do, but it's not currently clear exactly how we do that in practice (does it also extend to patching out use in our dependencies, should we be implementing more runtime checks for features we are using, when do we consider an affected glibc "old enough" not to worry about? etc). If you take a look through the [glibc bug tracker](https://sourceware.org/bugzilla/describecomponents.cgi?product=glibc) you'll no doubt find plenty of (active) bug reports for standard library code we already using. Obviously not to say we shouldn't try and avoid buggy code where possible.
Two other points:
[Cory mentioned in #21022](https://github.com/bitcoin/bitcoin/pull/21022#issuecomment-769274179):
> It also seems reasonable to me to worry that boost hits the same underlying glibc bug, and we've just not happened to trigger the right conditions yet.
Moving away from Boost to the standard library also removes the potential for differences related to Boosts configuration. Boost has multiple versions of `shared_mutex`, and what you end up using, and what it's backed by depends on:
* The version of Boost.
* The platform you're building for.
* Which version of `BOOST_THREAD_VERSION` is defined: (2,3,4 or 5) default=2. (see [here](https://www.boost.org/doc/libs/1_70_0/doc/html/thread/build.html#thread.build.configuration) for some of the differences).
* Is `BOOST_THREAD_V2_SHARED_MUTEX` defined? (not by default). If so, you might get the ["less performant, but more robust"](https://github.com/boostorg/thread/issues/230#issuecomment-475937761) version of `shared_mutex`.
A lot of these factors are eliminated by our use of depends, but users will have varying configurations. It's also not inconceivable to think that a distro, or some package manager might start defining something like `BOOST_THREAD_VERSION=3`. Boost tried to change the default from 2 to 3 at one point.
With this change, we no longer use Boost Thread, so this PR also removes it from depends, the build system, CI etc.
Previous similar PRs were #19183 & #20922. The authors are included in the commits here.
Also related to #21022 - pthread sanity checking.
ACKs for top commit:
laanwj:
Code review ACK 060a2a64d40d75fecb60b7d2b9946a67e46aa6fc
vasild:
ACK 060a2a64d40d75fecb60b7d2b9946a67e46aa6fc
Tree-SHA512: 572d14d8c9de20bc434511f20d3f431836393ff915b2fe9de5a47a02dca76805ad5c3fc4cceecb4cd43f3ba939a0508178c4e60e62abdbaaa6b3e8db20b75b03
f13e642c831c5689cb2bb7f5c4f9cb4c0c03ef21 build: Disable valgrind when building zeromq package in depends (Hennadii Stepanov)
b970f03beae0f3ae6a796f0e3b97732fc579f6aa build: Disable libbsd when building zeromq package in depends (Hennadii Stepanov)
77899991b1e29a45bc377b21330148cb7cc08923 build: Update netbsd_kevent_void.patch (Hennadii Stepanov)
Pull request description:
Since v4.3.3 (068385c951) `libzmq` uses `libbsd` by default.
This PR disables `libbsd` explicitly, as it's not a part of our depends. Zeromq will fallback to its internal `strlcpy` implementation.
Otherwise, on systems with installed `libbsd-dev` package the `zeromq` package build system erroneously detects `libbsd` package from the host system:
```diff
--- a/libzmq.pc
+++ b/libzmq.pc
@@ -8,5 +8,5 @@
Version: 4.3.4
Libs: -L${libdir} -lzmq
Libs.private: -lpthread
-Requires.private:
+Requires.private: libbsd
Cflags: -I${includedir}
```
This causes the `configure` fails to detect the `zeromq` package:
```
configure: WARNING: libzmq version 4.x or greater not found, disabling
```
---
Other minor improvements:
- fixed `netbsd_kevent_void.patch` offset
- disabled valgrind as it's used in unit tests which we do not run:
```diff
--- a/zmq-configure-output
+++ b/zmq-configure-output
@@ -119,11 +119,6 @@
checking whether the g++ -m64 linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... (cached) GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
-checking for valgrind... valgrind
-checking for Valgrind tool memcheck... memcheck
-checking for Valgrind tool helgrind... helgrind
-checking for Valgrind tool drd... drd
-checking for Valgrind tool exp-sgcheck... exp-sgcheck
checking linker version script flag... --version-script
checking if version scripts can use complex wildcards... yes
checking for working posix_memalign... yes
```
ACKs for top commit:
fanquake:
ACK f13e642c831c5689cb2bb7f5c4f9cb4c0c03ef21
Tree-SHA512: d4c86d4a841eb6e7c32157e84972243072f905496c2a4c14ec6f6ab4216df6695cbf29baa2233ce27eaede35d1e250ad2f9975b16f570d01509f0c5da4597cad
6897c4bdf51a4aa74320ebfffa9467db14670109 build: patch depends zeromq to fix building on NetBSD Current (fanquake)
ce6dd2f1a2e2c56d86d00e0eeec34c9982017416 zeromq 4.3.4 (fanquake)
Pull request description:
This is a prerequisite for #23955. It updates zeromq to the latest available version, and adds a patch, [that I've sent upstream](https://github.com/zeromq/libzmq/pull/4326), to fix building on NetBSD Current (10).
ACKs for top commit:
hebasto:
ACK 6897c4bdf51a4aa74320ebfffa9467db14670109, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: d05d9753630faebe842e1ca70c8c4af660a38e7331a9d95e84df3a3b14564c5118ca41c4fc49fb71dfee563b63e1014e5a3f8874d652e26de59e8e188a12970e
continued from a2687d4594 from dash#4379
includes:
- 4af59a407a207e7de327c1bb145d68ebc8ddedd8
inapplicable:
- 335bd7f8bc415d6419763e17b013f94e5f0e5689
above commit is inapplicable as boost 1.71 bump did away with the patch,
backported as 6a367fc198 in dash#4356
we're currently using boost 1.77, this replacement was dropped in its
patch form in bitcoin#19764 (backported as 6a367fc1 in dash#4356), the
comment was dropped in bitcoin#19761 but the change is in this commit
instead due to direct relevance.
339b4a51f6d3558c3489b14efe0c8c195295cf86 build: don't install deprecated libevent headers (fanquake)
Pull request description:
We don't use the deprecated headers now, and never should do in the
future, so there is no need for them to exist in depends.
The headers themselves are just full of includes for the newer headers.
ACKs for top commit:
hebasto:
ACK 339b4a51f6d3558c3489b14efe0c8c195295cf86
Tree-SHA512: 736fd9e3b22212da462cc05203dd253806dc59f973090357b705f2742ed4a3b8c3cc44b3173d706527f60ad93e95cf4143ec6b7db4233a489890a98f8e5c8f07
bcd36e14f0a2b89445f1436094de40696d6ad76c build: correct depends FreeBSD C{XX}FLAGS (fanquake)
7b06ffce9c50110b475c722918c55a14402346a5 build: add NetBSD support to depends (fanquake)
Pull request description:
Similar to #23948. Doesn't build the Qt package; I haven't looked at doing that yet, but have an assumption that it's going to fail out of the box similar to the FreeBSD build.
Guix Build:
```bash
```
ACKs for top commit:
theuni:
ACK bcd36e14f0a2b89445f1436094de40696d6ad76c
Tree-SHA512: 9a0946cefbcb9a92dd730b885463f3213e304c8d4b39fea8d831fc013a73d2ef998ca84e384bf45a01fa1449cf5a35eaffaa5b57a9062c2cdda34312d33ec3fc
edc9a6afdc6926b40de2fed897b9b866d58f28d2 build, refactor: Reuse expat package version in its download path (Hennadii Stepanov)
4bb7821ab243325467a89ff8ffc1bed290f2cb58 build, refactor: Use conventional version notation for boost package (Hennadii Stepanov)
Pull request description:
`boost` package:
- `.` is used as a separator in versions of other depends packages.
`expat` package:
- reuse package version in its download path
---
The straightforward way to verify this PR:
```
$ cd depends
$ make clean-all
$ make boost_fetched
$ make expat_fetched
```
ACKs for top commit:
prusnak:
ACK edc9a6a
shaavan:
ACK edc9a6afdc6926b40de2fed897b9b866d58f28d2
Tree-SHA512: c15d672fe34ac59850425d3d6a6eee5f720e16d227aad1332a563b218465879b7ee6fb865dd1bac06aedf356f9bb1c67112d9d88da8f877f04838b50a9dc97be
## Issue being fixed or feature implemented
fix 3 gui issues reported about beta2
## What was done?
pls see individual commits
## How Has This Been Tested?
to test ad0c0eb591 (there were no css
issues on my machine with locally compiled binaries): compile, run
to test 09800cff8f: run beta2 guix
compiled binaries with css from this PR e.g. on macos:
`/path/to/Dash-Qt.app/Contents/MacOS/Dash-Qt --regtest --debug-ui
--custom-css-dir=/path/to/dash/src/qt/res/css`
before 139d70701c:
<img width="224" alt="Screenshot 2023-09-30 at 22 02 32"
src="https://github.com/dashpay/dash/assets/1935069/b7a26ae9-f87f-4dde-b1f6-9eb796c22495">
after 139d70701c:
<img width="229" alt="Screenshot 2023-09-30 at 22 02 59"
src="https://github.com/dashpay/dash/assets/1935069/57d92e97-b25d-4035-9d1b-da373e51a574">
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
