8c3ff618d3 chore: apply some `clang-format-diff.py` suggestions (Kittywhiskers Van Gogh)
aa1f56f126 merge bitcoin#22626: Remove txindex migration code (Kittywhiskers Van Gogh)
145d94d700 merge bitcoin#23636: Remove GetAdjustedTime from init.cpp (Kittywhiskers Van Gogh)
150ca008fe merge bitcoin#23683: valid but different LockPoints after a reorg (Kittywhiskers Van Gogh)
e85862ba11 merge bitcoin#23649: circular dependency followups (Kittywhiskers Van Gogh)
8ab99290f9 merge bitcoin#22677: cut the validation <-> txmempool circular dependency (Kittywhiskers Van Gogh)
ee49383cd6 merge bitcoin#23211: move `update_*` structs from txmempool.h to .cpp file (Kittywhiskers Van Gogh)
3d769c7a64 merge bitcoin#23249: ParseByteUnits - Parse a string with suffix unit (Kittywhiskers Van Gogh)
edd0bab6b5 chore: remove superfluous `ParseHDKeypath` definition (Kittywhiskers Van Gogh)
0073b66aaa refactor: migrate some Dash code to use `ChainstateManager::ProcessTransaction` (Kittywhiskers Van Gogh)
c8571c0956 merge bitcoin#23173: Add `ChainstateManager::ProcessTransaction` (Kittywhiskers Van Gogh)
a21bfd02e9 merge bitcoin#23157: improve performance of check() and remove dependency on validation (Kittywhiskers Van Gogh)
b35dc7236d merge bitcoin#23185: Add ParseMoney and ParseScript tests (Kittywhiskers Van Gogh)
7c03133be3 merge bitcoin#22987: Fix "RuntimeError: Event loop is closed" on Windows (Kittywhiskers Van Gogh)
ba60d5459e merge bitcoin#22772: hasher cleanup (follow-up to bitcoin#19935) (Kittywhiskers Van Gogh)
Pull request description:
## Additional Information
* When backporting [bitcoin#23173](https://github.com/bitcoin/bitcoin/pull/23173), `bypass_limits` had to be extended to `ChainstateManager::ProcessTransaction()` as Dash allows the `sendrawtransaction` RPC to bypass limits with the optional `bypasslimits` boolean (introduced in [dash#2110](https://github.com/dashpay/dash/pull/2110)).
The bool arguments are not in alphabetical order to prevent breakage with Bitcoin code that expects `bypass_limits` to always be `false`.
## Breaking Changes
None expected.
## Checklist
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
- [x] I have added or updated relevant unit/integration/functional/e2e tests
- [x] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
UdjinM6:
utACK 8c3ff618d3
PastaPastaPasta:
utACK 8c3ff618d3
Tree-SHA512: ea1eaad7279b6608a07c1175e5c3b44385d42e33afa8ce5160d871fc9b37a014e9981eafca379ac3ad6dc141b5fda6f1e676b4cc9658a6d1775fe929a134ff67
e1749b50a3 exclude x11 dir from linting (pasta)
dc1f566fce fmt: run clang-format on hash_x11.h (pasta)
bd8aa04d44 refactor: segregate x11 hashing (pasta)
Pull request description:
## Issue being fixed or feature implemented
Refactor x11 hashing out into it's own header
## What was done?
move x11 hashing out of hash.h, into hash_x11.h and also move those headers crypto/x11 folder
## How Has This Been Tested?
Building
## Breaking Changes
None
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
UdjinM6:
utACK e1749b50a3
Tree-SHA512: c5e9ac9995608b1963494a4e6d870bb452972d6c09387c06546623e25c8be26fd2918918fb00fbef0c0356c61f1bed127a469c5adae252792f25bc419c73100b
c77216ea80 docs: explain meaning of MessageProcessingResult's members (Konstantin Akimov)
d0f17788fc refactor: drop dependency of EhfSignals on PeerManager (Konstantin Akimov)
1d13f010d0 refactor: remove dependency of QuorumBlockProcessor on PeerManager (Konstantin Akimov)
f1c6d17879 refactor: remove dependency of chainlocks on PeerManager (Konstantin Akimov)
538342138c refactor: HandleNewRecoveredSig return PostProcessingMessage (Konstantin Akimov)
09565fe6cc refactor: new type of message processing result for resolving circular dependency over PeerManager (Konstantin Akimov)
d54b3eeb7b refactor: move ChainLocksSigningEnabled from header to cpp file as static function (Konstantin Akimov)
cab700aae3 refactor: remove unused include spork.h from validation.cpp (Konstantin Akimov)
Pull request description:
## Issue being fixed or feature implemented
It reduces circular dependencies and simplify code.
## What was done?
Removed circular dependency over PeerManager for llmq::QuorumBlockProcessor, llmq::CEhfSignals, llmq::ChainLocks, and several extra useful refactorings: see commits.
## How Has This Been Tested?
Run `test/lint/lint-circular-dependencies.sh`
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e tests
- [x] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
UdjinM6:
utACK c77216ea80
PastaPastaPasta:
utACK c77216ea80
Tree-SHA512: e01829a694c9bfbbe70bc346ef5949b5e9e4532560f4c40ee292952f05f0fd23ecf4bd978e918f74dd3422c1b90231fd7d9984f491f3ab8f7eb08072540406b4
cc1a75ab3a docs: add release notes (Kittywhiskers Van Gogh)
39625f16f0 stats: drop copyright notice from `stats/client.cpp` (Kittywhiskers Van Gogh)
18a2e48eb9 stats: rename `statsns` to clearer `statsprefix` (Kittywhiskers Van Gogh)
42918c2cdc stats: rename `statshostname` to more appropriate `statssuffix` (Kittywhiskers Van Gogh)
f3a4844b0a stats: implicitly treat stats as enabled if `statshost` is specified (Kittywhiskers Van Gogh)
69603a83fa stats: miscellaneous changes and housekeeping (Kittywhiskers Van Gogh)
3e12ac0e09 stats: deduplicate `send` and `sendDouble` logic (Kittywhiskers Van Gogh)
bf44fc3bf6 feat(stats): introduce support for batching messages (Kittywhiskers Van Gogh)
38b1643fe6 feat(stats): introduce support for queuing messages (Kittywhiskers Van Gogh)
fc4a736e2a stats: move message sending logic to `RawSender` (Kittywhiskers Van Gogh)
92690685be stats: move `statsd_client` to `stats` directory (Kittywhiskers Van Gogh)
f782dfd562 stats: remove double indentation in header file (Kittywhiskers Van Gogh)
Pull request description:
## Motivation
This pull request achieves the goal originally set out in [dash#5167](https://github.com/dashpay/dash/pull/5167), to migrate the base of our Statsd client implementation to one that is actively maintained. Statoshi ([source](54c3ffdcf0/src/statsd_client.cpp)) utilizes [talebook/statsd-client-cpp](https://github.com/talebook/statsd-client-cpp), which in turn is inherited by Dash.
As Statsd is the only cross-platform reporting mechanism available (USDT requires a Linux host with superuser privileges and RPCs don't provide as much flexibility as desired), emphasis is placed on using Statsd as the primary way for the Dash daemon to report metrics related to node and network health.
As part of maintaining our Statsd client, this PR aims to migrate the base of our implementation to [vthiery/cpp-statsd-client](https://github.com/vthiery/cpp-statsd-client), a streamlined implementation that embraces C++11 with a thread-safe implementation of queueing and batching, which reduces the number of packets used to transmit stats.
These capabilities are optional and users can opt not to use them by setting `-statsduration=0` to disable queueing (which will also disable batching) or `-statsbatchsize=0`, which will disable batching (but will not disable queueing unless requested explicitly).
## Additional Information
* Dependent on https://github.com/dashpay/dash/pull/5167
* `RawSender` (and by extension, `RawMessage`) strive to remain as unopinionated as possible, moving the responsibility to construct valid Statsd messages onto `StatsdClient`. This is to ensure that `RawSender` can be reused down the line or independently extended without impacting the Statsd client.
* `RawMessage` exists to provide extensions to `std::vector<uint8_t>` that make it easier to abstract away strings while also implementing some of its semantics like `append()` (and its alias, `+=`).
* `InitStatsClient()` was introduced to keep `StatsdClient` indifferent to _how_ arguments are obtained and sanitized before they're supplied to the constructor. This is to keep it indifferent to all the backwards-compatibility code for deprecated arguments still work.
* When constructing the Statsd message, we can use `%f` without having to specify a precision as tinyformat automatically assumes a precision of 6 ([source](17110f50b3/src/tinyformat.h (L673))) and problems don't seem to be observed when using `%f` with integers ([source](https://github.com/dashpay/dash/pull/6267#issuecomment-2345592051)).
* As a guardrail, there is a `static_assert` to ensure that a specialization of `send()` involving a non-arithmetic type will raise alarm when compiling ([source](a0ce720207/src/stats/client.cpp (L145))).
## Breaking changes
* `-statsenabled` (replaced with specifying `-statshost`), `-statshostname` (replaced by `-statssuffix`) and `-statsns` (replaced by `-statsprefix`) have been deprecated and will be removed in a future release.
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
- [x] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
knst:
utACK cc1a75ab3a
UdjinM6:
utACK cc1a75ab3a
Tree-SHA512: b038419f2b6d807dac40a04d23c5046fbaa95beedb88f5a9e4c06a7042c2f5da7e01c72c4a2744bce10878cafc747136d6599dcd86ae1be0782ad4194d5b7bec
This marks the completion of our transition from code based on
`talebook/statsd-client-cpp` to code based on `vthiery/cpp-statsd-client`.
Also, we long stopped using `snprintf`, remove it from exclusions list.
8f06ac9dfa Merge bitcoin/bitcoin#22172: doc: update tor.md, release notes with removal of tor v2 support (W. J. van der Laan)
9b22501a4d Merge bitcoin/bitcoin#22122: ci: Bump macOS image to big-sur-xcode-12.5 (MarcoFalke)
3b05a99b50 Merge bitcoin/bitcoin#22106: refactor: address ProcessNewBlock comments from #21713 (fanquake)
c8725560c9 Merge bitcoin/bitcoin#21856: doc: add OSS-Fuzz section to fuzzing.md doc (MarcoFalke)
facf685285 Merge bitcoin/bitcoin#22261: [p2p/mempool] Two small fixes to node broadcast logic (fanquake)
1430897fc4 Merge bitcoin/bitcoin#22445: fuzz: Move implementations of non-template fuzz helpers from util.h to util.cpp (MarcoFalke)
f0c62d50a5 Merge bitcoin/bitcoin#22447: test: whitelist rpc_rawtransaction peers to speed up tests (fanquake)
b609514142 Merge #22381: guix: Test security-check sanity before performing them (Carl Dong)
9ef68d1905 Merge bitcoin/bitcoin#22061: ci: Bump multiprocess memory (fanquake)
Pull request description:
## Issue being fixed or feature implemented
Regular backports from bitcoin v22
## What was done?
See commits
## How Has This Been Tested?
Run unit and functional tests
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
UdjinM6:
utACK 8f06ac9dfa
PastaPastaPasta:
utACK 8f06ac9dfa
Tree-SHA512: f800b7ca8d357f2d02ce5cb1fc4951c2765242676c5494efd5e22e8f6d41d889e1fa2f888930f72aded75813413c6488d8a7d96baa2cf4820e3461464708658e
6431f71b3a Merge bitcoin-core/gui#393: Fix regression in "Encrypt Wallet" menu item (Hennadii Stepanov)
fc900a8aea Merge bitcoin-core/gui#333: refactor: Signal-slot connections cleanup (Hennadii Stepanov)
9ca2aad0b3 Merge bitcoin-core/gui#164: Handle peer addition/removal in a right way (Hennadii Stepanov)
7d9ce32562 Merge bitcoin-core/gui#29: refactor: Optimize signal-slot connections logic (Hennadii Stepanov)
3be79a9ed9 Merge bitcoin-core/gui#256: Save/restore column sizes of the tables in the Peers tab (Hennadii Stepanov)
f4fccd31cb Merge bitcoin-core/gui#329: Make console buttons look clickable (Hennadii Stepanov)
5a0d524506 Merge bitcoin-core/gui#123: rpc: Do not accept command while executing another one (Hennadii Stepanov)
19310646e0 Merge bitcoin-core/gui#331: Make RPC console welcome message translation-friendly (Hennadii Stepanov)
69a1305978 Merge bitcoin-core/gui#309: Add access to the Peers tab from the network icon (Hennadii Stepanov)
c858325d40 Merge bitcoin-core/gui#346: English translations update (Hennadii Stepanov)
412445afb5 Merge bitcoin-core/gui#313: qt: Optimize string concatenation by default (W. J. van der Laan)
Pull request description:
## Issue being fixed or feature implemented
Gui related backports from bitcoin v22
## What was done?
See commits
## How Has This Been Tested?
Run unit/functional tests
See also:
<img alt="right menu" src="https://user-images.githubusercontent.com/32963518/116794314-d64b9b80-aad4-11eb-89ca-7f75c7442ba8.gif"/>
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
UdjinM6:
light ACK 6431f71b3a
PastaPastaPasta:
utACK 6431f71b3a
Tree-SHA512: bb14de71c9375b10da695db6c521c26686815b8b5ca2748bfe3bd2eafa9d332acd60acd85a1f2eed3aa831d16e5741ecc7570130ce9cf5bff011c065b55d62b2
62cb8d98d27e7f316f01f177f35ad0ed6f8cd9ce qt: Drop BitcoinGUI* WalletFrame data member (Hennadii Stepanov)
f73e5c972ab096e0f80cb9e753fa221d17313358 qt: Move CreateWalletActivity connection from WalletFrame to BitcoinGUI (Hennadii Stepanov)
20e2e24e90d782219e853ef0676ac66dc6a9de6a qt: Move WalletView connections from WalletFrame to BitcoinGUI (Hennadii Stepanov)
Pull request description:
This PR:
- implements an idea from https://github.com/bitcoin/bitcoin/pull/17937#issuecomment-575991765
- simplifies `WalletFrame` class interface
- as a side effect, removes `bitcoingui` -> `walletframe` -> `bitcoingui` circular dependency
- is an alternative to https://github.com/bitcoin/bitcoin/pull/17500
ACKs for top commit:
promag:
Tested ACK 62cb8d98d27e7f316f01f177f35ad0ed6f8cd9ce on macos 11.2.3 with depends build.
jarolrod:
ACK 62cb8d98d27e7f316f01f177f35ad0ed6f8cd9ce
Tree-SHA512: 633b526a8499ba9ab4b16928daf4de4f6d610284bb9fa51891cad35300a03bde740df3466a71b46e87a62121330fcc9e606eac7666ea5e45fa6d5785b60dcbbd
BACKPORT NOTICE: it includes missing commit: d6ef3543ae16847d5a91fa9271acee9bd2164b32
lint: Run mypy with --show-error-codes
When using mypy ignore directives, the error code needs to be specified.
Somehow mypy doesn't print it by default...
In upcoming commits, message sending will be split off into a separate
file and stats capabilities will be fleshed out. Prepare for that by
giving it its own directory.
Also get rid of `statsd` namespace, it is entirely unnecessary.
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
cc998abec1 fmt: apply `clang-format-diff.py` suggestions (Kittywhiskers Van Gogh)
0401c581eb stats: `const`-ify variables and arguments (Kittywhiskers Van Gogh)
9f96723774 stats: stop using error codes, switch over to `bool` (Kittywhiskers Van Gogh)
1a81979c1e stats: initialize socket after we have a valid socket address (Kittywhiskers Van Gogh)
dbbfc8d766 stats: use `Socks` wrapper, use `CService` to generate our `sockaddr` (Kittywhiskers Van Gogh)
2def905044 stats: move init logic into constructor (Kittywhiskers Van Gogh)
4bc727cd6c stats: clean up randomization code, move `FastRandomContext` inward (Kittywhiskers Van Gogh)
840241eefd stats: cleanup error logging, improve code sanity (Kittywhiskers Van Gogh)
85890ddb13 docs: add copyright notice to source file, update notice in header (Kittywhiskers Van Gogh)
a9d1b1494d stats: move `_StatsdClientData` variables into `StatsdClient` (Kittywhiskers Van Gogh)
30c30c1397 stats: fetch all arguments needed when constructing `g_stats_client` (Kittywhiskers Van Gogh)
5133d88415 stats: s/statsClient/g_stats_client/g (Kittywhiskers Van Gogh)
f81951dd00 stats: make `statsClient` a `std::unique_ptr`, denote as global variable (Kittywhiskers Van Gogh)
Pull request description:
## Additional Information
Support for transmitting stats to a Statsd server has been courtesy of Statoshi ([repo](https://github.com/jlopp/statoshi)), implemented Dec, 2020 by [dash#2515](https://github.com/dashpay/dash/pull/2515) but since then, it hasn't gotten much attention aside from benefiting from codebase-wide changes and the occasional compiler appeasement. This pull request aims to give our statistics code some TLC.
Changes include:
* Limiting initialization to solely during construction and moving the responsibility of fetching arguments outside of `statsd::StatsdClient`.
* Using the RAII `Socks` wrapper as early as possible (we still need to construct a raw socket ourselves but this is done in the initializer and control is moved to the wrapper and everywhere else, the wrapper is used)
* Utilizing existing networking code to generate the socket address
* This lets us trivially allow IPv6 connections as the responsibility to construct it safely is moved to `CService`.
* Using `std::string` and our string manipulation capabilities (replacing `snprintf` with `strprintf`), replacing platform-specific types (replacing `short` with `uint16_t`).
## Breaking Changes
None observed.
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
- [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
- [x] I have made corresponding changes to the documentation **(note: N/A)**
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
PastaPastaPasta:
utACK [cc998ab](cc998abec1)
UdjinM6:
utACK cc998abec1
Tree-SHA512: 433c92160d6ac7ebb8582ada3cbb65ead7913618266b773619a528c90dfe0e286aafa46dc3b0bca62f246938e5948a732080e2cddba942d3627f007ca6efcc1f
- Use `uint16_t` instead of `short`, `int64_t` instead of `size_t`
- Get rid of the `errmsg` buffer and use `LogPrintf` to report errors
- Use `strprintf` instead of `snprintf`
- Rephrase networking error logs to allow inclusion of error strings
DSQ messages are 142 bytes.
Previously, assuming a relatively highly connected masternode hosting 100 connection, each round of coinjoin will result in 14.2KB (100*142) of inbound and outbound traffic each.
Now, using the inventory system, a message will first use 36 bytes per peer (sending and receiving), plus the size of a `getdata` message and the actual message itself. As a result, bandwidth usage for 1 round of mixing would be closer to 36 * 100 + 142 (dsq) + 36 (getdata) = ~3.8KB, a reduction of around ~73%
BACKPORT NOTICE:
we keep and maintain cppcheck linter as lint-cppcheck-dash.sh
efae252f3072da598160670691757a0d60b9beb4 test: Remove extended lint (cppcheck) (laanwj)
Pull request description:
These are unreferenced in the CI and documentation, and have been since 2019 (see #17549).
I'm not sure the cppcheck is worthwhile. It takes a long time to run (I think this is why it isn't in the normal lints), and right
now it only appears to find implicit constructors. The list of exceptions is out of date. But if anyone wants to bring it back at any
time in the future they can do so from git history (and port it to Python).
ACKs for top commit:
fanquake:
ACK efae252f3072da598160670691757a0d60b9beb4
Tree-SHA512: 1a770b5d20ff1199d0d6bc471ae3d2c3438f0f0b169ce8d2fe73480daf8d3a7146c066b799afc90aa7898982c5fee79c1daca10e16e2bff0a7b38850aedd55b2
f4cb0fbfe1 fix: no need to relay quorum commitment in case of block undo (Konstantin Akimov)
0431a33919 fix: follow-up changes for bitcoin#14193. (Konstantin Akimov)
86b76d19b6 Merge bitcoin/bitcoin#21812: ci: Enable D_GLIBCXX_DEBUG for multiprocess task (fanquake)
334496ea7e Merge bitcoin/bitcoin#21775: p2p: Limit m_block_inv_mutex (MarcoFalke)
23b83109ea Merge bitcoin/bitcoin#21750: net: remove unnecessary check of CNode::cs_vSend (MarcoFalke)
b34514191f Merge bitcoin/bitcoin#21738: test: Use clang-12 for ASAN, Add missing suppression (fanquake)
3411577473 Merge bitcoin/bitcoin#19160: multiprocess: Add basic spawn and IPC support (W. J. van der Laan)
970048d917 fix: missing changes from bitcoin#19267 - run multiprocess on CI (Konstantin Akimov)
f2b7ee73db fix: follow-up bitcoin#15402 - removed dead code (Konstantin Akimov)
274068cdbc fix: follow-up bitcoin/bitcoin#21732 - minor missing typo (MarcoFalke)
e9450a8b36 Merge #21669: test: Remove spurious double lock tsan suppressions by bumping to clang-12 (MarcoFalke)
ef92c3065c Merge #21663: ci: Fix macOS brew install command (W. J. van der Laan)
Pull request description:
## Issue being fixed or feature implemented
Just regular backports from v22
## What was done?
See commits for backports.
Also there're 2 bugs are fixed which became visible after backporting bitcoin#21775 - both are related to possible deadlocks in net_processing
## How Has This Been Tested?
Run unit and functional tests. Enabled multiprocess builds on CI
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
UdjinM6:
utACK f4cb0fbfe1
PastaPastaPasta:
utACK f4cb0fbfe1
Tree-SHA512: 3204c2aa243fa4834ccf4ff4672d183cf9b35f87b857df8543572cd685729e15fca39f85b27194233e57cbc1746e36b556efab95ce20d0aa0a7d4476a9f3c6c0
84934bf70e11fe4cda1cfda60113a54895d4fdd5 multiprocess: Add echoipc RPC method and test (Russell Yanofsky)
7d76cf667eff512043a28d4407cc89f58796c42b multiprocess: Add comments and documentation (Russell Yanofsky)
ddf7ecc8dfc64cf121099fb047e1ac871de94f4c multiprocess: Add bitcoin-node process spawning support (Russell Yanofsky)
10afdf0280fa93bfffb0a7665c60dc155cd84514 multiprocess: Add Ipc interface implementation (Russell Yanofsky)
745c9cebd50fea1664efef571dc1ee1bddc96102 multiprocess: Add Ipc and Init interface definitions (Russell Yanofsky)
5d62d7f6cd48bbc4e9f37ecc369f38d5e1e0036c Update libmultiprocess library (Russell Yanofsky)
Pull request description:
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).
---
This PR adds basic process spawning and IPC method call support to `bitcoin-node` executables built with `--enable-multiprocess`[*].
These changes are used in https://github.com/bitcoin/bitcoin/pull/10102 to let node, gui, and wallet functionality run in different processes, and extended in https://github.com/bitcoin/bitcoin/pull/19460 and https://github.com/bitcoin/bitcoin/pull/19461 after that to allow gui and wallet processes to be started and stopped independently and connect to the node over a socket.
These changes can also be used to implement new functionality outside the `bitcoin-node` process like external indexes or pluggable transports (https://github.com/bitcoin/bitcoin/pull/18988). The `Ipc::spawnProcess` and `Ipc::serveProcess` methods added here are entry points for spawning a child process and serving a parent process, and being able to make bidirectional, multithreaded method calls between the processes. A simple example of this is implemented in commit "Add echoipc RPC method and test."
Changes in this PR aside from the echo test were originally part of #10102, but have been split and moved here for easier review, and so they can be used for other applications like external plugins.
Additional notes about this PR can be found at https://bitcoincore.reviews/19160
[*] Note: the `--enable-multiprocess` feature is still experimental, and not enabled by default, and not yet supported on windows. More information can be found in [doc/multiprocess.md](https://github.com/bitcoin/bitcoin/blob/master/doc/multiprocess.md)
ACKs for top commit:
fjahr:
re-ACK 84934bf70e11fe4cda1cfda60113a54895d4fdd5
ariard:
ACK 84934bf. Changes since last ACK fixes the silent merge conflict about `EnsureAnyNodeContext()`. Rebuilt and checked again debug command `echoipc`.
Tree-SHA512: 52a948b5e18a26d7d7a09b83003eaae9b1ed2981978c36c959fe9a55abf70ae6a627c4ff913a3428be17400a3dace30c58b5057fa75c319662c3be98f19810c6
1bf0bf492f merge bitcoin#24515: Only load BlockMan in BlockMan member functions (Kittywhiskers Van Gogh)
5c1eb67c42 merge bitcoin#24050: Give m_block_index ownership of CBlockIndexes (Kittywhiskers Van Gogh)
c440304c85 merge bitcoin#22932: Add CBlockIndex lock annotations, guard nStatus/nFile/nDataPos/nUndoPos by cs_main (Kittywhiskers Van Gogh)
e303a4ec45 merge bitcoin#23974: Make blockstorage globals private members of BlockManager (Kittywhiskers Van Gogh)
301163c65e merge bitcoin#23581: Move BlockManager to node/blockstorage (Kittywhiskers Van Gogh)
732e871a6b merge bitcoin#23785: Move stuff to ChainstateManager (Kittywhiskers Van Gogh)
b402fd57fa merge bitcoin#23174: have LoadBlockIndex account for snapshot use (Kittywhiskers Van Gogh)
a08f2f48bf merge bitcoin#21526: UpdateTip/CheckBlockIndex assumeutxo support (Kittywhiskers Van Gogh)
472caa048a merge bitcoin#22371: Move pblocktree global to BlockManager (Kittywhiskers Van Gogh)
d69ca833df merge bitcoin#21727: Move more stuff to blockstorage (Kittywhiskers Van Gogh)
6df927fc60 chore: exclude underscore placeholder from shadowing linter warnings (Kittywhiskers Van Gogh)
Pull request description:
## Additional Information
* Dependent on https://github.com/dashpay/dash/pull/6078
* Dependent on https://github.com/dashpay/dash/pull/6074
* Dependent on https://github.com/dashpay/dash/pull/6083
* Dependent on https://github.com/dashpay/dash/pull/6119
* Dependency for https://github.com/dashpay/dash/pull/6138
* In [bitcoin#24050](https://github.com/bitcoin/bitcoin/pull/24050), `BlockMap` is given ownership of the `CBlockIndex` instance contained within the `unordered_map`. The same has not been done for `PrevBlockMap` as `PrevBlockMap` is populated with `pprev` pointers and doing so seems to break validation logic.
* Dash has a specific linter for all Dash-specific code present in Core. The introduction of `util/translation.h` into `validation.h` has caused the linter to trigger shadowing warnings due to a conflict between the common use of `_` as a placeholder/throwaway name ([source](37e026a038/src/spork.cpp (L44))) and upstream's usage of it to process translatable strings ([source](37e026a038/src/util/translation.h (L55-L62))).
Neither C++17 nor C++20 have an _official_ placeholder/throwaway term or annotation for structured bindings (which cannot use `[[maybe_unused]` or `std::ignore`) but [P2169](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2169r4.pdf) is a proposal put forth to make it the official placeholder, in that light, the linter will silence shadowing warnings involving an underscore.
## Breaking Changes
None expected
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
- [x] I have added or updated relevant unit/integration/functional/e2e tests
- [x] I have made corresponding changes to the documentation **(note: N/A)**
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
UdjinM6:
utACK 1bf0bf492f (with one nit)
knst:
utACK 1bf0bf492f
PastaPastaPasta:
utACK 1bf0bf492f
Tree-SHA512: 875fff34fe91916722f017526135697466e521d7179c473a5c0c444e3aa873369019b804dee9f5f795fc7ebed5c2481b5ce2d895b2950782a37de7b098157ad4
Bitcoin uses underscore in util/translation.h for translatable strings
but underscores are also a placeholder used in multiple languages, incl.
C++. The next commit will be introducing backports, one of them will be
placing util/translation.h into validation.h, which is a header used by
Dash-specific code.
This causes a conflict between the normal usage of underscore as a
placeholder and Bitcoin's usage of it as a function name, which is
reported by the Dash-specific linter. We need to exclude shadowing
warnings from the linter to account for this.
fa04f9b4ddffc5ef23c2ee7f3cc72a7c2ae49204 rpc: Remove duplicate name and argNames from CRPCCommand (MarcoFalke)
fa92912b4bb4629addcbfdfb7cc000be701614af rpc: Use RPCHelpMan for check-rpc-mappings linter (MarcoFalke)
faf835680be39811827504f77005b6603165f53e rpc: [refactor] Use concise C++11 code in CRPCConvertTable constructor (MarcoFalke)
Pull request description:
Currently, the RPC argument names are specified twice to simplify consistency linting. To avoid having to specify the argnames twice when adding new arguments, remove the linter and add an equivalent test based on RPCHelpMan.
ACKs for top commit:
laanwj:
ACK fa04f9b4ddffc5ef23c2ee7f3cc72a7c2ae49204
Tree-SHA512: 3f5f32f5a09b22d879f24aa67031639d2612cff481d6aebc6cfe6fd757cafb3e7bf72120b30466f59292a260747b71e57322c189d5478b668519b9f32fcde31a
faaf9c58e4aa809019d4ca12747dd47411988e37 remove CRPCCommand constructor that takes rpcfn_type function pointer (MarcoFalke)
fa19bb2cd8c575593583138a84e6bb3444d6196d remove dead rpc code (MarcoFalke)
Pull request description:
Remove the CRPCCommand arguments, now that they are asserted to be equal and thus redundant
### Future work
> Here or follow up, makes sense to also assert type of returned UniValue?
Sure, but let's not get ahead of ourselves. I am going to submit any further works as follow-ups, including:
* Removing all python regex linters on the args, now that RPCMan can be used to generate any output, including the cli.cpp table
* Auto-formatting and sanity checking the RPCExamples with RPCMan
* Checking passed-in json in self-check. Removing redundant checks
* Checking returned json against documentation to avoid regressions or false documentation
* Compile the RPC documentation at compile-time to ensure it doesn't change at runtime and is completely static
### Bugs found
* The assert identified issue #18607
* The changes itself fixed bug #19250
ACKs for top commit:
fjahr:
tested ACK faaf9c58e4aa809019d4ca12747dd47411988e37
promag:
Tested ACK faaf9c58e4aa809019d4ca12747dd47411988e37.
ryanofsky:
Code review ACK faaf9c58e4aa809019d4ca12747dd47411988e37. Two obviously good simplifications.
Tree-SHA512: 5de3b440f7b2ed2c3e86655d4f0e2e5df9c67e8ce3c7817d5ea5311d1a38690f2f3e28fab41aad6936be9fc884326d037e5f19e85d4d2fe281474dada13911ee
fbffe06dad fix: suppress lint warnings for edge due to both missing epoll and kpoll (Konstantin Akimov)
b799683d60 fix: pass reference instead copy of argument in util/edge (Konstantin Akimov)
9d941aacb9 fix: removed unused assigned (Konstantin Akimov)
d9e2e47685 fix: add a missing file util/wpipe to linter lists (Konstantin Akimov)
Pull request description:
## Issue being fixed or feature implemented
Some source files in src/util is missing to specify as dash specific for linters
## What was done?
Added to list of dash's linters
## How Has This Been Tested?
Run `test/lint/lint-all.sh`
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
UdjinM6:
utACK fbffe06dad
PastaPastaPasta:
utACK fbffe06dad
kwvg:
utACK fbffe06dad
Tree-SHA512: 05bd40f987a93b47ca939f37d6e5b62e2044f2acce7a6ea6c361594eace808da17b17e0149095daecd0fb5418c6cbedbc70ad07ac3b7a1156a94188b04c2fd00
172c2333f03aecb4c347c791537e13c296adbde2 Porting lint-assertions.sh to lint-assertions.py (hiago)
Pull request description:
This PR is converting `test/lint/lint-assertions.sh` to `test/lint/lint-assertions.py`. It's an item of #24783.
ACKs for top commit:
laanwj:
Tested ACK 172c2333f03aecb4c347c791537e13c296adbde2
Tree-SHA512: 94d5b03acfeaf2303fad95d489d6c3aa7bd655889ddaa807cc97e0613b8eb8f5ef094feee2a98d974606890deb554e76490a5c523d64eb5bc55afa6a43221aae
908fb7e2ec37fe68675d38dbfee4df9f861bb2b5 test: Use permissions from git in `lint-files.py` (laanwj)
48d2e80a7479a44b0ab09e87542c8cb7a8f72223 test: Don't use shell=True in `lint-files.py` (laanwj)
Pull request description:
Improvements to the `lint-files.py` script:
- Avoid use of `shell=True`.
- Check the permissions in git's metadata instead of in the filesystem. This stops the umask or filesystem from interfering. It's also more efficient as it only needs a single call to `git ls-files`.
(what triggered this change was `File "..." contains a shebang line, but has the file permission 775 instead of the expected executable permission 755.` errors running the script locally).
ACKs for top commit:
vincenzopalazzo:
re-tACK 908fb7e2ec
Tree-SHA512: 2eaf868c55a9c3508b12658a5b3ac429963fd0551e645332d0ac54be56fefccee95115e4667386df24b46b545593cb0d0bf8c6cecab73f9cb19d37ddf704c614
fae211c0ae0dd90876a3390eb21449b7b0bb45c4 lint: Start to use py lint scripts (MarcoFalke)
fa82e890e7950fe5ba6d4fa88fcd922cc929dc47 Move lint script and data file to avoid lint- prefix (MarcoFalke)
Pull request description:
ACKs for top commit:
fjahr:
tACK fae211c0ae0dd90876a3390eb21449b7b0bb45c4
Tree-SHA512: f8272a1bab9efb8203cac121710baae68f01f79e520ad71ff15aa516d19763d61c088b411b019de105a6a30e7ee3c274814d59963f6ac22ba1084560fb601f45
2227fc4e6203064b14e99bcf453601bd263a0196 test: minor fixes & improvements for files linter test (windsok)
Pull request description:
Couple of minor fixes & improvements for files linter test added in #21740
- Use a context manager when opening files, so that files are closed are we are done with them
- Use the `-z` flag when shelling out to `git ls-files` so that we can catch newlines and other weird control characters in filenames.
From the `git ls-files` manpage:
```
-z \0 line termination on output and do not quote filenames. See OUTPUT below for more information.
Without the -z option, pathnames with "unusual" characters are quoted as explained for the configuration variable
core.quotePath (see git-config(1)). Using -z the filename is output verbatim and the line is terminated by a NUL byte.
```
ACKs for top commit:
MarcoFalke:
cr ACK 2227fc4e6203064b14e99bcf453601bd263a0196
practicalswift:
cr ACK 2227fc4e6203064b14e99bcf453601bd263a0196: patch looks correct
Tree-SHA512: af059a805f4a7614162de85dea856052a45ab531895cb0431087e7fc9e037513fa7501bb5eb2fe43238adf5f09e77712ebdbb15b1486983359ad3661a3da0c60
46b025e00df40724175735eb5606ac73067cb3b8 test: add new python linter to check file names and permissions (windsok)
6f6bb3ebc7cb8e17a5dfc8ef55aa2d3f2dc6bdea test: fix file permissions on various scripts (windsok)
Pull request description:
Adds a new python linter test which tests for correct filenames and file permissions in the repository.
Replaces the existing tests in the `test/lint/lint-filenames.sh` and `test/lint/lint-shebang.sh` linter tests, as well as adding some new and increased testing. This increased coverage is intended to catch issues such as in #21728 and https://github.com/bitcoin/bitcoin/pull/16807/files#r345547050
Summary of tests:
* Checks every file in the repository against an allowed regexp to make sure only lowercase or uppercase alphanumerics (a-zA-Z0-9), underscores (_), hyphens (-), at (@) and dots (.) are used in repository filenames.
* Checks only source files (*.cpp, *.h, *.py, *.sh) against a stricter allowed regexp to make sure only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.) are used in source code filenames. Additionally there is an exception regexp for directories or files which are excepted from matching this regexp (This should replicate the existing `test/lint/lint-filenames.sh` test)
* Checks all files in the repository match an allowed executable or non-executable file permission octal. Additionally checks that for executable files, the file contains a shebang line.
* Checks that for executable `.py` and `.sh` files, the shebang line used matches an allowable list of shebangs (This should replicate the existing `test/lint/lint-shebang.sh` test)
* Checks every file that contains a shebang line to ensure it has an executable permission
Additionally updates the permissions on various files to comply with the new tests.
Fixes#21729
ACKs for top commit:
practicalswift:
cr re-ACK 46b025e00df40724175735eb5606ac73067cb3b8: patch still looks correct
kiminuo:
code review ACK 46b025e00df40724175735eb5606ac73067cb3b8 if `contrib/gitian-descriptors/assign_DISTNAME` permission change is deemed OK.
laanwj:
Code review ACK 46b025e00df40724175735eb5606ac73067cb3b8
Tree-SHA512: 1c8201a2cee0d9cbce15652b68cec9a6458a8b493fcd5392f98560aca0b1a12e668baab65a47100f116f626dadc3f591deb47f7368468c6a46c6c712c2533455
7cc77f3a30 Merge #21373: test: generate fewer blocks in feature_nulldummy to fix timeouts, speed up (MarcoFalke)
a933a60b1a feat: new command line argument -bip147height for bitcoin#21373 (Konstantin Akimov)
51911388f2 Merge #21377: Speedy trial support for versionbits (fanquake)
ecade9bc39 Merge bitcoin/bitcoin#21749: test: Bump shellcheck version (W. J. van der Laan)
eeec2f2799 Merge bitcoin/bitcoin#21884: fuzz: Remove unused --enable-danger-fuzz-link-all option (fanquake)
51633d70ea Merge bitcoin/bitcoin#21874: fuzz: Add WRITE_ALL_FUZZ_TARGETS_AND_ABORT (MarcoFalke)
a02a2c0322 Merge bitcoin/bitcoin#21681: validation: fix ActivateSnapshot to use hardcoded nChainTx (MarcoFalke)
71f23d6e33 Merge bitcoin/bitcoin#21814: test: Fix feature_config_args.py intermittent issue (MarcoFalke)
de4d2a839d Merge bitcoin/bitcoin#21714: refactor: Drop CCoinControl::SetNull (MarcoFalke)
a63f9c31cc Merge bitcoin-core/gui#284: refactor: Simplify SendCoinsDialog::updateCoinControlState (Hennadii Stepanov)
b2d889380c Merge bitcoin/bitcoin#21792: test: Fix intermittent issue in p2p_segwit.py (MarcoFalke)
Pull request description:
## Issue being fixed or feature implemented
Regular batch of backports from bitcoin v22
## What was done?
Implemented new commandline argument `-bip147height` for RegTest.
Backports:
- bitcoin/bitcoin#21377
- bitcoin/bitcoin#21792
- bitcoin-core/gui#284
- bitcoin/bitcoin#21714
- bitcoin/bitcoin#21373
- bitcoin/bitcoin#21814
- bitcoin/bitcoin#21681
- bitcoin/bitcoin#21874
- bitcoin/bitcoin#21884
- bitcoin/bitcoin#21749
## How Has This Been Tested?
Run unit/functional tests
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
PastaPastaPasta:
utACK 7cc77f3a30
Tree-SHA512: d46218667430af19c445d67d6411b1e7f19920e85f86e6e74ae7b9062aeb3763637a8613587c203ba8d285ccc3ee755f936141010944cfae8627397e8b8584d3
