31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0 [util] add RunCommandParseJSON (Sjors Provoost)
c17f54ee535faaedf9033717403e1f775b5f1530 [ci] use boost::process (Sjors Provoost)
32128ba682033560d6eb2e4848a9f77a842016d2 [doc] include Doxygen comments for HAVE_BOOST_PROCESS (Sjors Provoost)
3c84d85f7d218fa27e9343c5cd1a55e519218980 [build] msvc: add boost::process (Sjors Provoost)
c47e4bbf0b44f2de1278f9538124ec98ee0815bb [build] make boost-process opt-in (Sjors Provoost)
929cda5470f98d1ef85c05b1cad4e2fb9227e3b0 configure: add ax_boost_process (Sjors Provoost)
8314c23d7b39fc36dde8b40b03b6efbe96f85698 [depends] boost: patch unused variable in boost_process (Sjors Provoost)
Pull request description:
Prerequisite for external signer support in #16546. Big picture overview in [this gist](https://gist.github.com/Sjors/29d06728c685e6182828c1ce9b74483d).
This adds a new dependency [boost process](https://github.com/boostorg/process/tree/boost-1.64.0). This is part of Boost since 1.64 which is part of `depends`. Because the minimum Boost version is 1.47, this functionality is skipped for older versions of Boost.
Use `./configure --with-boost-process` to opt in, which checks for the presence of Boost::Process.
We add `UniValue runCommandParseJSON(const std::string& strCommand)` to `system.{h,cpp}` which calls an arbitrary command and processes the JSON returned by it. This is currently only called by the test suite.
~For testing purposes this adds a new regtest-only RPC method `runcommand`, as well as `test/mocks/command.py` used by functional tests.~ (this is no longer the case)
TODO:
- [ ] review boost process in #15440
ACKs for top commit:
achow101:
ACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0
hebasto:
re-ACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0, only rebased (verified with `git range-diff`) and removed an unintentional tab character since the [previous](https://github.com/bitcoin/bitcoin/pull/15382#pullrequestreview-458371035) review.
meshcollider:
Very light utACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0, although I am not very confident with build stuff.
promag:
Code review ACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0, don't mind the nit.
ryanofsky:
Code review ACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0. I left some comments below that could be ignored or followed up later. The current change is clean and comprehensive.
Tree-SHA512: c506e747014b263606e1f538ed4624a8ad7bcf4e025cb700c12cc5739964e254dc04a2bbb848996b170e2ccec3fbfa4fe9e2b3976b191222cfb82fc3e6ab182d
060a2a64d40d75fecb60b7d2b9946a67e46aa6fc ci: remove boost thread installation (fanquake)
06e1d7d81d5a56d136c6fc88f09a2b0654a164f9 build: don't build or use Boost Thread (fanquake)
7097add83c8596f81be9edd66971ffd2486357eb refactor: replace Boost shared_mutex with std shared_mutex in sigcache (fanquake)
8e55981ef834490c438436719f95cbaf888c4914 refactor: replace Boost shared_mutex with std shared_mutex in cuckoocache tests (fanquake)
Pull request description:
This replaces `boost::shared_mutex` and `boost::unique_lock` with [`std::shared_mutex`](https://en.cppreference.com/w/cpp/thread/shared_mutex) & [`std::unique_lock`](https://en.cppreference.com/w/cpp/thread/unique_lock).
Even though [some concerns were raised](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-726214696) in #16684 with regard to `std::shared_mutex` being unsafe to use across some glibc versions, I still think this change is an improvement. As I mentioned in #21022, I also think trying to restrict standard library feature usage based on bugs in glibc is not only hard to do, but it's not currently clear exactly how we do that in practice (does it also extend to patching out use in our dependencies, should we be implementing more runtime checks for features we are using, when do we consider an affected glibc "old enough" not to worry about? etc). If you take a look through the [glibc bug tracker](https://sourceware.org/bugzilla/describecomponents.cgi?product=glibc) you'll no doubt find plenty of (active) bug reports for standard library code we already using. Obviously not to say we shouldn't try and avoid buggy code where possible.
Two other points:
[Cory mentioned in #21022](https://github.com/bitcoin/bitcoin/pull/21022#issuecomment-769274179):
> It also seems reasonable to me to worry that boost hits the same underlying glibc bug, and we've just not happened to trigger the right conditions yet.
Moving away from Boost to the standard library also removes the potential for differences related to Boosts configuration. Boost has multiple versions of `shared_mutex`, and what you end up using, and what it's backed by depends on:
* The version of Boost.
* The platform you're building for.
* Which version of `BOOST_THREAD_VERSION` is defined: (2,3,4 or 5) default=2. (see [here](https://www.boost.org/doc/libs/1_70_0/doc/html/thread/build.html#thread.build.configuration) for some of the differences).
* Is `BOOST_THREAD_V2_SHARED_MUTEX` defined? (not by default). If so, you might get the ["less performant, but more robust"](https://github.com/boostorg/thread/issues/230#issuecomment-475937761) version of `shared_mutex`.
A lot of these factors are eliminated by our use of depends, but users will have varying configurations. It's also not inconceivable to think that a distro, or some package manager might start defining something like `BOOST_THREAD_VERSION=3`. Boost tried to change the default from 2 to 3 at one point.
With this change, we no longer use Boost Thread, so this PR also removes it from depends, the build system, CI etc.
Previous similar PRs were #19183 & #20922. The authors are included in the commits here.
Also related to #21022 - pthread sanity checking.
ACKs for top commit:
laanwj:
Code review ACK 060a2a64d40d75fecb60b7d2b9946a67e46aa6fc
vasild:
ACK 060a2a64d40d75fecb60b7d2b9946a67e46aa6fc
Tree-SHA512: 572d14d8c9de20bc434511f20d3f431836393ff915b2fe9de5a47a02dca76805ad5c3fc4cceecb4cd43f3ba939a0508178c4e60e62abdbaaa6b3e8db20b75b03
fa4632c41714dfaa699bacc6a947d72668a4deef test: Move boost/stdlib includes last (MarcoFalke)
fa488f131fd4f5bab0d01376c5a5013306f1abcd scripted-diff: Bump copyright headers (MarcoFalke)
fac5c373006a9e4bcbb56843bb85f1aca4d87599 scripted-diff: Sort test includes (MarcoFalke)
Pull request description:
When writing tests, often includes need to be added or removed. Currently the list of includes is not sorted, so developers that write tests and have `clang-format` installed will either have an unrelated change (sorting) included in their commit or they will have to manually undo the sort.
This pull preempts both issues by just sorting all includes in one commit.
Please be aware that this is **NOT** a change to policy to enforce clang-format or any other developer guideline or process. Developers are free to use whatever tool they want, see also #18651.
Edit: Also includes a commit to bump the copyright headers, so that the touched files don't need to be touched again for that.
ACKs for top commit:
practicalswift:
ACK fa4632c41714dfaa699bacc6a947d72668a4deef
jonatack:
ACK fa4632c41714dfaa, light review and sanity checks with gcc build and clang fuzz build
Tree-SHA512: 130a8d073a379ba556b1e64104d37c46b671425c0aef0ed725fd60156a95e8dc83fb6f0b5330b2f8152cf5daaf3983b4aca5e75812598f2626c39fd12b88b180
Used to avoid the following error:
```
In file included from test/fuzz/addition_overflow.cpp:7:
In file included from ./test/fuzz/util.h:26:
In file included from ./test/util/setup_common.h:16:
In file included from ./txmempool.h:31:
In file included from /builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/optional.hpp:15:
In file included from /builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/optional/optional.hpp:47:
In file included from /builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/type_traits/is_nothrow_move_assignable.hpp:16:
/builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/type_traits/has_nothrow_assign.hpp:65:7: error: builtin __has_nothrow_assign is deprecated; use __is_nothrow_assignable instead [-Werror,-Wdeprecated-builtins]
BOOST_HAS_NOTHROW_ASSIGN(T)
^
/builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/type_traits/intrinsics.hpp:205:43: note: expanded from macro 'BOOST_HAS_NOTHROW_ASSIGN'
^
/builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/serialization/serialization.hpp:61:1: note: in instantiation of template class 'boost::has_nothrow_assign<unsigned int>' requested here
BOOST_STRONG_TYPEDEF(unsigned int, version_type)
^
/builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/serialization/strong_typedef.hpp:42:57: note: expanded from macro 'BOOST_STRONG_TYPEDEF'
D& operator=(const D& rhs) BOOST_NOEXCEPT_IF(boost::has_nothrow_assign<T>::value) {t = rhs.t; return *this;} \
```
clang warnings are quite noisy and easily cause gitlab's logs
to spill over, preventing logging of dash-specific warnings and errors
from making it to the CI logs.
our dependencies mostly track upstream so regardless, we cannot
act upon those warnings, so it's better to just suppress them if they're
too noisy.
faa9491870bf29583a989c407694d01f49b65bec ci: Use Focal for fuzzers (MarcoFalke)
Pull request description:
This gives us access to clang-10, as well as a newer version of valgrind
ACKs for top commit:
fanquake:
ACK faa9491870bf29583a989c407694d01f49b65bec - [Clang 10](https://packages.ubuntu.com/focal/clang) and [valgrind 3.15](https://packages.ubuntu.com/focal/valgrind).
practicalswift:
ACK faa9491870bf29583a989c407694d01f49b65bec -- diff looks correct & contemporary clang is better than vintage clang
Tree-SHA512: 0e67232673434c0309db79c1054e3e981115083585945967e346f4d58792635832100f89911428aab928155e44e5f401207a023681ae008fdb5280cf02c4d427
9e111db088e4137865ae068d206c769994ea0a29 test: set a name for CI Docker containers (fanquake)
Pull request description:
Addresses one part of #16664, by making it easier to identify CI containers that are running locally. By default Docker will generate random names, like `peaceful_rubin`, with this change, we explicitly set names for all containers.
ACKs for top commit:
MarcoFalke:
ACK 9e111db088e4137865ae068d206c769994ea0a29
Tree-SHA512: 0a29ada0d8cf6b0e9ae7a35f4f6df7a3dcc448523ceaed01371124360d6e3d1bf351172104a5fb629488eeaa57994ba04134dcb83c261eb1dfd2f0d73edf5f60
420fa0770f37619bfa29898d59dac45b6a477abb fuzz: use std::optional for sep_pos variable (Harris)
Pull request description:
This PR changes the original `size_t sep_pos` to `std::optional<size_t> sep_post_opt` to remove the warning when compiling fuzz tests.
```shell
warning: variable 'sep_pos' may be uninitialized when used here [-Wconditional-uninitialized]
```
Also, it adds `--enable-c++17` flag to CI fuzz scripts.
ACKs for top commit:
practicalswift:
ACK 420fa0770f37619bfa29898d59dac45b6a477abb
MarcoFalke:
ACK 420fa07
Tree-SHA512: e967d5d8ab8ee7394b243ff5b28bac72d30bd14774e4a206f8c87474fad22769da76e4ba4e03cbef83b8f60e5293e9d9293b613e2e2e59e187d4e59ae6b874ca
* Squashed 'src/dashbls/' content from commit 66ee820fbc
git-subtree-dir: src/dashbls
git-subtree-split: 66ee820fbc9e3b97370db8c164904af48327a124
* build: stop tracking build-system generated relic_conf.h.in
* build: add support for building bls-signatures from local subtree
* build: add exclusions to linting scripts and filters
* build: drop bls-signatures (bls-dash) from depends
Unfortunately, we need -Wno-unused-command-line-argument as configure will fail to run
with -Werror due to Dash's current build system utilizing -static-libstdc++ indiscriminately,
resulting in Clang warnings of "unused arguments".
