* instantsend: Avoid writing IS locks for unknown txes
* instantsend: Allow a competing tx into mempool if there is an islock waiting for it
* use try_emplace
* Hold cs_main while calling ResetBlockFailureFlags
* fix: replace seemingly buggy loop with std::adjacent_find
* Remove redundant variable declaration
* use std::fill instead of a loop
* remove a few raw for loops
* use unique_ptr instead of shared
Signed-off-by: pasta <pasta@dashboost.org>
* unique_ptr over shared_ptr
Signed-off-by: pasta <pasta@dashboost.org>
* remove unneeded ptr
Signed-off-by: pasta <pasta@dashboost.org>
* Adjust IsTxSafeForMining checks
Signed-off-by: pasta <pasta@dashboost.org>
* use const ref
Signed-off-by: pasta <pasta@dashboost.org>
* add a todo
Signed-off-by: pasta <pasta@dashboost.org>
* use optional instead of magic max value
fixes a hypothetical bug where myIdx is not "initialized" (ie max), and we sleep forever
Signed-off-by: pasta <pasta@dashboost.org>
* simplify relay check
Signed-off-by: pasta <pasta@dashboost.org>
* use count_if instead of a loop
Signed-off-by: pasta <pasta@dashboost.org>
* add a few vector reserves
Signed-off-by: pasta <pasta@dashboost.org>
* Handle attempts to read non-existent records from isdb properly
* Do not reject blocks that conflict with islocks while still syncing
Otherwise you can stuck with no new blocks/headers which means you won't be able to verify new chainlocks that might override stored islocks
* Handle duplicates/conflicting islocks better
* More constness
519b0bc5dc5155b6f7e2362c2105552bb7618ad0 Make last disconnected block BLOCK_FAILED_VALID, even when aborted (Pieter Wuille)
8d220417cd7bc34464e28a4861a885193ec091c2 Optimization: don't add txn back to mempool after 10 invalidates (Pieter Wuille)
9ce9c37004440d6a329874dbf66b51666d497dcb Prevent callback overruns in InvalidateBlock and RewindBlockIndex (Pieter Wuille)
9bb32eb571a846b66ed3bac493f55cee11a3a1b9 Release cs_main during InvalidateBlock iterations (Pieter Wuille)
9b1ff5c742dec0a6e0d6aab29b0bb771ad6d8135 Call InvalidateBlock without cs_main held (Pieter Wuille)
241b2c74ac8c4c3000e778554da1271e3f293e5d Make RewindBlockIndex interruptible (Pieter Wuille)
880ce7d46b51835c00d77a366ec28f54a05239df Call RewindBlockIndex without cs_main held (Pieter Wuille)
436f7d735f1c37e77d42ff59d4cbb1bd76d5fcfb Release cs_main during RewindBlockIndex operation (Pieter Wuille)
1d342875c21b5d0a17cf4d176063bb14b35b657e Merge the disconnection and erasing loops in RewindBlockIndex (Pieter Wuille)
32b2696ab4b079db736074b57bbc24deaee0b3d9 Move erasure of non-active blocks to a separate loop in RewindBlockIndex (Pieter Wuille)
9d6dcc52c6cb0cdcda220fddccaabb0ffd40068d Abstract EraseBlockData out of RewindBlockIndex (Pieter Wuille)
Pull request description:
This PR makes a number of improvements to the InvalidateBlock (`invalidateblock` RPC) and RewindBlockIndex functions, primarily around breaking up their long-term cs_main holding. In addition:
* They're made safely interruptible (`bitcoind` can be shutdown, and no progress in either will be lost, though if incomplete, `invalidateblock` won't continue after restart and will need to be called again)
* The validation queue is prevented from overflowing (meaning `invalidateblock` on a very old block will not drive bitcoind OOM) (see #14289).
* `invalidateblock` won't bother to move transactions back into the mempool after 10 blocks (optimization).
This is not an optimal solution, as we're relying on the scheduler call sites to make sure the scheduler doesn't overflow. Ideally, the scheduler would guarantee this directly, but that needs a few further changes (moving the signal emissions out of cs_main) to prevent deadlocks.
I have manually tested the `invalidateblock` changes (including interrupting, and running with -checkblockindex and -checkmempool), but haven't tried the rewinding (which is probably becoming increasingly unnecessary, as very few pre-0.13.1 nodes remain that would care to upgrade).
Tree-SHA512: 692e42758bd3d3efc2eb701984a8cb5db25fbeee32e7575df0183a00d0c2c30fdf72ce64c7625c32ad8c8bdc56313da72a7471658faeb0d39eefe39c4b8b8474
* replace raw owning ptr with unique ptr
Signed-off-by: pasta <pasta@dashboost.org>
* Add GUARDED_BY annotation to llmq_versionbitscache
Signed-off-by: pasta <pasta@dashboost.org>
* limit scope of locking cs_llmq_vbc
Signed-off-by: pasta <pasta@dashboost.org>
* use llmq_versionbitscache instead of versionbitscache in UpdatedBlockTip to avoid cs_main locking
Signed-off-by: pasta <pasta@dashboost.org>
* drop unneeded cs_main ::mempool.cs
* lock cs_main and mempool.cs in Db::Upgrade
* include adjustments
* fix macOs build failure
Signed-off-by: pasta <pasta@dashboost.org>
* expcitly include array in spork.h
* sort includes in most files
* merge bitcoin#15855: Add missing LockAnnotation for cs_main
* mutex: update cs_main locks, assertions and annotations
This commit is a squash between 8c98823 and 90d0535
bf2e01097 uint256: Remove unnecessary crypto/common.h use (Karl-Johan Alm)
Pull request description:
This is an alternative to #13242 which keeps the `ReadLE64` part, but moves the `crypto/common.h` dependency into `crypto/common.h` as a function outside of `uint256`.
**Reason:** this change will remove dependencies for `uint256` to `crypto/common.h`, `compat/endian.h`, and `compat/byteswap.h`.
This PR removes the need to update tests to be endian-aware/-independent, but keeps the (arguably dubious) `ReadLE64` part (which was only introduced to fix the tests, not for any functionality).
Tree-SHA512: 78b35123cdb185b3b3ec59aba5ca8a5db72624d147f2d6a5484ffa5ce626a72f782a01dc6893fc8f5619b03e2eae7b5a03b0df5d43460f3bda428e719e188aec
* refactor: introduce PendingSignatureData
* refactor: explicitly capture in lambdas
* refactor: use structured bindings
* refactor: use more const
* refactor: remove unused include header
* refactor: use if-init
* refactor: add nodiscard
* refactor: initialize llmqType
* refactor: add override
* refactor: remove redundant specifiers
* refactor: prevent shadowing
* refactor: use try_emplace where possible
* refactor: use more accurate name
string_view is needed for compile time interpretation (constexpr) of the LLMQParams
constexpr is good to add such that we aren't initializing in static scope, but instead at compile time
Sadly using string_view will require that we construct a std::string when passing into univalue, however, this isn't in a performance important section (rpc) and as such another allocation is an okay trade off for the benefits of constexpr
Also, now we aren't having extern variables, which should make the linker mildly happier
Signed-off-by: pasta <pasta@dashboost.org>
When receiving an islock, propagate it as islock.
When creating/receiving and isdlock, propagate it as isdlock to peers which support it and as islock to peers which don't.
Functional tests to cover both islock and isdlock scenarios.
* bls: use constexpr int instead of #define
* lint: bump c++ version to 17
* test: use BOOST_CHECK_EQUAL instead of BOOST_ASSERT, call boost assert in another location
* coinjoin: fix typo
* drop redundant LLMQType cast
* use numeric_limits instead of magic value
* net_processing.cpp whitespace fixes
* Add some const
* use std::all_of instead of raw for loop
* Introduce UNINITIALIZED_SESSION_ID and use it instead of a magic number
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Don't lock instantsend cs where not needed (db interactions)
* Introduce cs_db to CInstantSendDb
All private members are GUARDED_BY or EXCLUSIVE_LOCKS_REQUIRED w/ cs_db
All public functions lock cs_db immediately
* adjust cs usage a bit more in CInstantSendManager
* Use WITH_LOCK in a few spots
* Make AddNonLockedTx lock internally instead of externally
* instantsend adjust formatting
* instantsend move IsInstantSendEnabled enabled check up
This backport does not include changes that depend on bitcoin pr 18037
70a6b529f306ff72ea1badf25e970a92b2b17ab3 lint-cppcheck: Remove -DHAVE_WORKING_BOOST_SLEEP_FOR (Anthony Towns)
294937b39de5924e772f8ed90d35c53290c8acab scheduler_tests: re-enable mockforward test (Anthony Towns)
cea19f685915be8affb2203184a549576194413f Drop unused reverselock.h (Anthony Towns)
d0ebd93270758ea97ea956b8821e17a2d001ea94 scheduler: switch from boost to std (Anthony Towns)
b9c426012770d166e6ebfab27689be44e6e89aa5 sync.h: add REVERSE_LOCK (Anthony Towns)
306f71b4eb4a0fd8e64f47dc008bc235b80b13d9 scheduler: don't rely on boost interrupt on shutdown (Anthony Towns)
Pull request description:
Replacing boost functionality with C++11 stuff.
Motivated by #18227, but should stand alone. Changing from `boost::condition_var` to `std::condition_var` means `threadGroup.interrupt_all` isn't enough to interrupt `serviceQueue` anymore, so that means calling `stop()` before `join_all()` is needed. And the existing reverselock.h code doesn't work with sync.h's DebugLock code (because the reversed lock won't be removed from `g_lockstack` which then leads to incorrect potential deadlock warnings), so I've replaced that with a dedicated class and macro that's aware of our debug lock behaviour.
Fixes#16027, Fixes#14200, Fixes#18227
ACKs for top commit:
laanwj:
ACK 70a6b529f306ff72ea1badf25e970a92b2b17ab3
Tree-SHA512: d1da13adeabcf9186d114e2dad9a4fdbe2e440f7afbccde0c13dfbaf464efcd850b69d3371c5bf8b179d7ceb9d81f4af3cc22960b90834e41eaaf6d52ef7d331
# Conflicts:
# src/reverselock.h
# src/rpc/misc.cpp
# src/scheduler.cpp
# src/scheduler.h
# src/sync.cpp
# src/sync.h
# src/test/reverselock_tests.cpp
# src/test/scheduler_tests.cpp
# src/test/test_dash.cpp
# test/lint/extended-lint-cppcheck.sh
* Avoid locking cs_main before logging, instead log, then lock where possible
* use WITH_LOCK in a number of locations as a replacement of simple scopes
* add EXCLUSIVE_LOCKS_REQUIRED
* minimize locked scope in rpc
* fix macOS build error
Signed-off-by: pasta <pasta@dashboost.org>
* add missing EXCLUSIVE_LOCKS_REQUIRED(cs_main)
Signed-off-by: pasta <pasta@dashboost.org>
* revert an rpc scope reduction
* revert an rpc scope reduction
77777c5624 log: Construct global logger on first use (MarcoFalke)
Pull request description:
The (de)initialization order is not well defined in C++, so generally it is not safe to use globals as the (de/con)structor of one global could use the (de/con)structor of another global before/after it has been (con/de)structed.
Specifically this fixes:
* `g_logger` might not be initialized on the first use, so do that. (Fixes#15111)
Tree-SHA512: eb9c22f4baf31ebc5b0b9ee6a51d1354bae1f0df186cc0ce818b4483c7b5a7f90268d2b549ee96b4c57f8ef36ab239dc6497f74f3e2ef166038f7437c368297d
* refactor: Add `const` qualifier to various llmq related functions
* refactor: Add thread safety annotations to various llmq related functions
And around them where it's required
Fixes
```
POTENTIAL DEADLOCK DETECTED
Previous lock order was:
(1) cs llmq/quorums_instantsend.cpp:1071 (in thread )
(2) cs_main llmq/quorums_instantsend.cpp:1405 (in thread )
Current lock order is:
(2) cs_main net_processing.cpp:2535 (in thread )
(1) cs llmq/quorums_instantsend.cpp:1559 (in thread )
```
* Merge #13743: refactor: Replace boost::bind with std::bind
cb53b825c2 scripted-diff: Replace boost::bind with std::bind (Chun Kuan Lee)
2196c51821 refactor: Use boost::scoped_connection in signal/slot, also prefer range-based loop instead of std::transform (Chun Kuan Lee)
Pull request description:
Replace boost::bind with std::bind
- In `src/rpc/server.cpp`, replace `std::transform` with simple loop.
- In `src/validation.cpp`, store the `boost::signals2::connection` object and use it to disconnect.
- In `src/validationinterface.cpp`, use 2 map to store the `boost::signals2::scoped_connection` object.
Tree-SHA512: 6653cbe00036fecfc495340618efcba6d7be0227c752b37b81a27184433330f817e8de9257774e9b35828026cb55f11ee7f17d6c388aebe22c4a3df13b5092f0
* Replace boost::bind with std::bind and remove Boost.Bind includes
Co-authored-by: MarcoFalke <falke.marco@gmail.com>
* llmq: Split "isdb" out of "llmq" db
We write and erase islocks a lot. It probably makes sense to have a standalone db with a larger cache for this.
* llmq: Split "recsigdb" out of "llmq" db
* llmq: Migrate recovered sigs
* llmq: Split "dkgdb" out of "llmq" db
* llmq: Migrate DKG db
* llmq: Wipe llmq
* Make constructors explicit
Signed-off-by: pasta <pasta@dashboost.org>
* static analysis fixes
Signed-off-by: pasta <pasta@dashboost.org>
* Make pFrom nullptr check it's own
Signed-off-by: pasta <pasta@dashboost.org>
* revert std thread changes and hasOperatorKey
Signed-off-by: pasta <pasta@dashboost.org>
* refactor/bls: misc refactoring and spelling/grammar fixes in bls code
* refactor/evo: misc refactoring and spelling/grammar fixes in evo code
* refactor: some include changes
* refactor: remove redundant `public`
* fix linter
Signed-off-by: pasta <pasta@dashboost.org>
* Sort includes
* Move `class CTxDSIn;`
* Drop unused functions in CBLSWorker
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
Function CWallet::KeepKey requires locking as it has concurrent access to database and member nKeysLeftSinceAutoBackup.
Avoid data race when reading setInventoryTxToSend size by locking the read. If locking happens after the read, the size may change.
Lock cs_mnauth when reading verifiedProRegTxHash.
Make fRPCRunning atomic as it can be read/written from different threads simultaneously.
Make m_masternode_iqr_connection atomic as it can be read/written from different threads simultaneously.
Use a recursive mutex to synchronize concurrent access to quorumVvec.
Make m_masternode_connection atomic as it can be read/written from different threads simultaneously.
Make m_masternode_probe_connection atomic as it can be read/written from different threads simultaneously.
Use a recursive mutex in order to lock access to activeMasterNode.
Use a recursive mutex to synchronize concurrent access to skShare.
Guarded all mnauth fields of a CNode.
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
Removes all (besides one) usages of "Params().GetConsensus().llmqs.at" and instead uses the wrapper in quorum_utils.cpp
Rename all params to llmq_params for consistency and not conflict with non-llmq params
make some llmq_params const where possible
remove unneeded llmq_params variables where it's only used once
Signed-off-by: pasta <pasta@dashboost.org>
Change ctpl implementation to use STL queue & mutex.
Use ctpl synchronized queue instead of boost lockfree queue in bls worker aggregator.
Use smart pointers for memory management of Aggregator and VectorAggregator. With 'delete this;' the objects are prone to data race on the delete operator.
Use smart pointers for memory management of ContributionVerifier.
Pass shared_ptr by value to other threads via worker pool.
1fabd59e7 Break circular dependency: init -> * -> init by extracting shutdown.h (Ben Woosley)
e62fdfeea Drop unused init.h includes (Ben Woosley)
Pull request description:
Most includers just wanted to react to pending shutdown.
This isolates access to `fRequestShutdown` and limits access to the shutdown api functions, including the new `CancelShutdown` for setting it to `false`.
Tree-SHA512: df42f75dfbba163576710e9a67cf1228531fd99d70a2f187bfba0bcc476d6749cf88180a97e66a81bb5b6c3c7f0917de7402d26039ba7b644cb7509b02f7e267
* instantsend: make stuff const where possible
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: remove unused `params`
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: combine two nested if's into one
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: use auto in spots where possible and clear
Signed-off-by: pasta <pasta@dashboost.org>
* coinjoin: make IsValidStructure const
Signed-off-by: pasta <pasta@dashboost.org>
* coinjoin: divide by an integer to avoid double implicit conversions
Signed-off-by: pasta <pasta@dashboost.org>
* coinjoin: make unused parameter unnamed
Signed-off-by: pasta <pasta@dashboost.org>
* quorums.* use const and use references where possible
Signed-off-by: pasta <pasta@dashboost.org>
* quorums.h don't return const when returning by value
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_blockprocessor.cpp remove redundant casts and combine two if statements
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_blockprocessor.cpp make values const
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_chainlocks.cpp access static function statically
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_chainlocks.h remove commented out include
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_commitment.cpp remove redundant casts
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_debug.cpp remove redundant casts and add const
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_dkgsession.cpp use const where possible
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_dkgsessionhandler.cpp use const where possible
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_dkgsessionhandler.cpp don't use std::move which apparently prevents "copy elision" in this instance
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_dkgsessionhandler.cpp use const
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_dkgsessionmgr.cpp misc refactoring
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_signing.* misc refactoring
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_signing_shares.* misc refactoring
Signed-off-by: pasta <pasta@dashboost.org>
* quorums_utils.cpp misc refactoring
Signed-off-by: pasta <pasta@dashboost.org>
232f96f5c8a3920c09db92f4dbac2ad7d10ce8cf doc: Add release notes for -avoidpartialspends (Karl-Johan Alm)
e00b4699cc6d2ee5697d38dd6607eb2631c9b77a clean-up: Remove no longer used ivars from CInputCoin (Karl-Johan Alm)
43e04d13b1ffc02b1082176e87f420198b40c7b1 wallet: Remove deprecated OutputEligibleForSpending (Karl-Johan Alm)
0128121101fb3ee82f3abd3973a967a4226ffe0e test: Add basic testing for wallet groups (Karl-Johan Alm)
59d6f7b4e2f847ec1f2ff46c84e6157655984f85 wallet: Switch to using output groups instead of coins in coin selection (Karl-Johan Alm)
87ebce25d66952f5ce565bb5130dcf5e24049872 wallet: Add output grouping (Karl-Johan Alm)
bb629cb9dc567cc819724d9f4852652926e60cbf Add -avoidpartialspends and m_avoid_partial_spends (Karl-Johan Alm)
65b3eda458221644616d0fdd6ba0fe01bdbce893 wallet: Add input bytes to CInputCoin (Karl-Johan Alm)
a443d7a0ca333b0bae63e04b5d476f9ad9c7aeac moveonly: CoinElegibilityFilter into coinselection.h (Karl-Johan Alm)
173e18a289088c6087ba6fac708e322aa63b7a94 utils: Add insert() convenience templates (Karl-Johan Alm)
Pull request description:
This PR adds an optional (off by default) `-avoidpartialspends` flag, which changes coin select to use output groups rather than outputs, where each output group corresponds to all outputs with the same destination.
It is a privacy improvement, as each time you spend some output, any other output that is publicly associated with the destination (address) will also be spent at the same time, at the cost of fee increase for cases where coin select without group restriction would find a more optimal set of coins (see example below).
For regular use without address reuse, this PR should have no effect on the user experience whatsoever; it only affects users who, for some reason, have multiple outputs with the same destination (i.e. address reuse).
Nodes with this turned off will still try to avoid partial spending, if the fee of the resulting transaction is not greater than the fee of the original transaction.
Example: a node has four outputs linked to two addresses `A` and `B`:
* 1.0 btc to `A`
* 0.5 btc to `A`
* 1.0 btc to `B`
* 0.5 btc to `B`
The node sends 0.2 btc to `C`. Without `-avoidpartialspends`, the following coin selection will occur:
* 0.5 btc to `A` or `B` is picked
* 0.2 btc is output to `C`
* 0.3 - fee is output to (unique change address)
With `-avoidpartialspends`, the following will instead happen:
* Both of (0.5, 1.0) btc to `A` or `B` is picked (one or the other pair)
* 0.2 btc is output to `C`
* 1.3 - fee is output to (unique change address)
As noted, the pro here is that, assuming nobody sends to the address after you spend from it, you will only ever use one address once. The con is that the transaction becomes slightly larger in this case, because it is overpicking outputs to adhere to the no partial spending rule.
This complements #10386, in particular it addresses @luke-jr and @gmaxwell's concerns in https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-300667926 and https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-302361381.
Together with `-avoidreuse`, this fully addresses the concerns in #10065 I believe.
Tree-SHA512: 24687a4490ba59cf4198ed90052944ff4996653a4257833bb52ed24d058b3e924800c9b3790aeb6be6385b653b49e304453e5d7ff960e64c682fc23bfc447621
# Conflicts:
# src/Makefile.am
# src/bench/coin_selection.cpp
# src/wallet/coincontrol.h
# src/wallet/coinselection.cpp
# src/wallet/coinselection.h
# src/wallet/init.cpp
# src/wallet/test/coinselector_tests.cpp
# src/wallet/wallet.cpp
# src/wallet/wallet.h
# test/functional/test_runner.py
* instantsend: remove unused variable sum of input value
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: Access static member statically
Signed-off-by: pasta <pasta@dashboost.org>
* refac: make functions private as possible
8c2d695c4a util: Store debug log file path in BCLog::Logger member. (Jim Posen)
8e7b961388 scripted-diff: Rename BCLog::Logger member variables. (Jim Posen)
1eac317f25 util: Refactor GetLogCategory. (Jim Posen)
3316a9ebb6 util: Encapsulate logCategories within BCLog::Logger. (Jim Posen)
6a6d764ca5 util: Move debug file management functions into Logger. (Jim Posen)
f55f4fcf05 util: Establish global logger object. (Jim Posen)
Pull request description:
This is purely a refactor with no behavior changes.
This creates a new class `BCLog::Logger` to encapsulate all global logging configuration and state.
Tree-SHA512: b34811f54a53b7375d7b6f84925453c6f2419d21179379ee28b3843d0f4ff8e22020de84a5e783453ea927e9074e32de8ecd05a6fa50d7bb05502001aaed8e53
9b2704777c [doc] Include txindex changes in the release notes. (Jim Posen)
ed77dd6b30 [test] Simple unit test for TxIndex. (Jim Posen)
6d772a3d44 [rpc] Public interfaces to GetTransaction block until synced. (Jim Posen)
a03f804f2a [index] Move disk IO logic from GetTransaction to TxIndex::FindTx. (Jim Posen)
e0a3b80033 [validation] Replace tx index code in validation code with TxIndex. (Jim Posen)
8181db88f6 [init] Initialize and start TxIndex in init code. (Jim Posen)
f90c3a62f5 [index] TxIndex method to wait until caught up. (Jim Posen)
70d510d93c [index] Allow TxIndex sync thread to be interrupted. (Jim Posen)
94b4f8bbb9 [index] TxIndex initial sync thread. (Jim Posen)
34d68bf3a3 [index] Create new TxIndex class. (Jim Posen)
c88bcec93f [db] Migration for txindex data to new, separate database. (Jim Posen)
0cb8303241 [db] Create separate database for txindex. (Jim Posen)
Pull request description:
I'm re-opening #11857 as a new pull request because the last one stopped loading for people
-------------------------------
This refactors the tx index code to be in it's own class and get built concurrently with validation code. The main benefit is decoupling and moving the txindex into a separate DB. The primary motivation is to lay the groundwork for other indexers that might be desired (such as the [compact filters](https://github.com/bitcoin/bips/pull/636)). The basic idea is that the TxIndex spins up its own thread, which first syncs the txindex to the current block index, then once in sync the BlockConnected ValidationInterface hook writes new blocks.
### DB changes
At the suggestion of some other developers, the txindex has been split out into a separate database. A data migration runs at startup on any nodes with a legacy txindex. Currently the migration blocks node initialization until complete.
### Open questions
- Should the migration of txindex data from the old DB to the new DB block in init or should it happen in a background thread? The downside to backgrounding it is that `getrawtransaction` would return an error message saying the txindex is syncing while the migration is running.
### Impact
In a sample size n=1 test where I synced nodes from scratch, the average time [Index writing](https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L1903) was 3.36ms in master and 1.72ms in this branch. The average time between `UpdateTip` log lines for sequential blocks between 400,000 and IBD end on mainnet was 0.297204s in master and 0.286134s in this branch. Most likely this is just variance in IBD times, but I can try with some more trials if people want.
Tree-SHA512: 451fd7d95df89dfafceaa723cdf0f7b137615b531cf5c5035cfb54e9ccc2026cec5ac85edbcf71b7f4e2f102e36e9202b8b3a667e1504a9e1a9976ab1f0079c4
* instantsend: Upgrade IS db on dip0020 activation
* instantsend: Do not remove islocks for txes removed from mempool until dip0020 is activated
* refactor: introduce fUpgradedDB to avoid excessive locking and checking the state
* llmq: Decouple `fUpgradedDB` from `cs`
* Update src/llmq/quorums_instantsend.cpp
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* Fix deadlock
Upgrade locks cs_main via GetTransaction
Co-authored-by: xdustinface <xdustinfacex@gmail.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* instantsend: Resolve block conflicts first and take care of mempool ones later
* refactor: Rename RemoveChainLockConflictingLock -> RemoveConflictingLock
* instantsend: Handle transaction removal from mempool (for all reasons besides inclusion in blocks)
* instantsend: Remove old islocks with no known txes from db (once)
* refactor: Replace magic number with CURRENT_VERSION
* fix: Do not remove islocks for (yet) valid orphans
* Apply suggestions from code review
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* tests: Check that CLs override ISes which invalidated non-CLed blocks earlier
* partial revert 3987: Do not mark blocks which conflict with ISes as "conflicting"
1527015 Avoid std::locale/imbue in DateTimeStrFormat (Pieter Wuille)
Pull request description:
And replace them with just hardcoded ISO8601 strings and `gmtime_r`.
Pointed out by @laanwj here: https://github.com/bitcoin/bitcoin/pull/12970#issuecomment-380962488
Tree-SHA512: a459758b42ca56f8462115aefe8e6377c1319fce509ea64dbb767f3f087c9b848335954cb684e5896c38008847684045505a3e1559fb3e83b8e80e10b003d1e7
* Simple changes, mostly just includes
* Continued include changes combined with using pointers to avoid including more than necessary in headers
* remove dup include
* masternode-utils.cpp include net.h in all builds
* resolve linter -7 +11
* drop quorums.h from dkgsessionhandler.cpp
* Add `<utilstrencodings.h>`
* Initialize lastMNListForVotingKeys
* Refactor GetMinedCommitment to return sharedptr
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* remove unused includes in validation.cpp
circular depends -2
* use more specific include in quorums.h
circular depends -1
* Remove unneeded include from quorums.cpp
circular depends -1
* llmq: Return bool in EnsureQuorumConnections
* llmq: Call RemoveMasternodeQuorumNodes directly
Instead of removing the good nodes from the vector in first loop and
call it with the bad ones in the second loop.
* refactor: Introduce and use CLLMQUtils::IsWatchQuorumsEnabled
* llmq: Move IsWatchQuorumsEnabled into EnsureQuorumConnections
* Fix extra line break in CommitTransaction log message
Introduced in #3680ebe7e80a49 (diff-b2bb174788c7409b671c46ccc86034bdR4113)
* doc: Fix `quorum sign` help
* doc: Add `getdata` to quorum commands list help
* doc: Drop "P2WSH" from signrawtransactionwithkey help
* trivial: Replace "push_back(Pair(..))" with "pushKV"
* trivial: Reorder wallet cmd-line options
* git: Add macos debug simbols to .gitignore
* trivial: Fix typos and whitespaces, drop unused stuff
* llmq: Protect nodeStates in CSigSharesManager::Cleanup()
* refactor: Tweak nodeState cleanup in CSigSharesManager::Cleanup()
* llmq: Add a couple of AssertLockHeld-s to ensure nodeStates protection
Instead of calling `CLLMQUtils::GetAllQuorumMembers` before each `CFinalCommitment::Verify` call to get the quorum members, just get them inside `CFinalCommitment::Verify`.
This avoids race condition for "spork19 off, spork 19 on" situations when some nodes might call CheckActiveState already, created new clsig and sent it so that other nodes processed it too already but they will call CheckActiveState afterwards (because it's done every 5 sec) and reset best chainlock to null.
* llmq|init|test: Add "mode" to -llmq-qvvec-sync parameter
This changes the paramter from `-llmq-qvvec-sync=<quorum_name>` to `-llmq-qvvec-sync=<quorum_name:mode>`
With the following definitions:
- `quorum_name`: Internal name of the quorum type
- `mode=0` - Sync always from all quorums of the type defined by `quorum_name`
- `mode=1` - Sync only if member of any from all other quorum of the type defined by `quorum_name`
`-llmq-qvvec-sync=llmq_100_67:0` To always request qvvec's from all `LLMQ_100_67`.
`-llmq-qvvec-sync=llmq_100_67:1` Only request if type member.
This means, if platform enables this on all MNs with `mode=0` we will
have all nodes asking new quorum for their verification vector instead
of only `24*100` at max.
* llmq: Adjust GetQuorumRecoveryStartOffset to use all MNs
* Turn `QvvecSyncMode` into `enum class`
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* llmq: Let return ProcessTx void instead of bool
The returned boolean is never used..
* llmq: Combine some "bail out early" cases
* llmq: Explicitly check for nullptr
* instantsend: refactor input locking into it's own method
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: introduce spork 24 `SPORK_24_INSTANTSEND_SIGNING_ENABLED`
This spork tells masternodes to refuse to lock transactions in mempool. Only transactions included in a block should be retroactively signed.
Signed-off-by: pasta <pasta@dashboost.org>
add spork defenition
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: refactor `sed -i 's/allowReSigning/fRetroactive/g' src/llmq/*`
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: adjust comments
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend/tests: implement Spork 24 support in tests, and test it's usage
Signed-off-by: pasta <pasta@dashboost.org>
* fix feature_llmq_is_retroactive.py
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* drop Spork 24 and use Spork 2 value 1 as being no mempool signing
Signed-off-by: pasta <pasta@dashboost.org>
* fix spork check
Signed-off-by: pasta <pasta@dashboost.org>
* Fix tests
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* Change comment
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* IsInstantSendSigningEnabled -> IsInstantSendMempoolSigningEnabled
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* instantsend: keep islocks cache when removing confirmed islocks from db
unordered_lru_cache should truncate it automagically
* instantsend: Batched write/erase for connected/disconnected blocks
* docs: Add help for -recsigsmaxage option
* Make `-llmq-data-recovery` help string dynamic
* Make devnet/regtest specific help strings dynamic
Only show devnet related help strings on devnet.
* Make linter happy
* Skip FindDevNetGenesisBlock when creating devnet params for help purposes only
* llmq: Use thread pool for quorum cache and data recovery threads
Move quorum data and cache thread handling into CQuorumManager.
* llmq: Fix explicit capture list
* llmq: Directly push the lamdas to workerPool
Co-authored-by: xdustinface <xdustinfacex@gmail.com>
* instantsend: Bail out early on disabled IS in more places
* instantsend: Disable InstantSend while reindexing and importing blocks
* Drop extra checks in GetInstantSendLockHashByTxid, GetInstantSendLockByInput and GetInstantSendLockByHash
* instantsend: Introduce (and use) KnownInstantSendLock
Check islock hash against both current and archived islocks
* instantsend: Optimize ProcessInstantSendLock, check for known islocks first
Two reasons:
1. GetTransaction is a potentially much heavier one
2. Doesn't make much sense to look for a tx if we have a known islock already
* Drop extra pendingInstantSendLocks.count(hash)
* Check pindexMined != nullptr
* Update src/llmq/quorums_instantsend.cpp
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* Update src/llmq/quorums_instantsend.h
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
* llmq: Drop InternalHasConflictingChainLock in ProcessNewChainLock
* llmq: Directly use clsig.blockHash instead of copying it into msgHash
* llmq: Reuse CInv(MSG_CLSIG, hash)
* llmq: Add const in two places
* instantsend: Mark a block with IS-locks which conflict with txes in a CL-ed block as conflicting and not as invalid
* tests: Tweak feature_llmq_is_cl_conflicts.py to test CL overriding a block with conflicting IS-locks
* instantsend|sigs: Sleep when there is no more work
Instead of sleeping only when no work has been done.
Avoids useless cycles, improves batching.
* llmq: Add and use nMaxBatchSize
* llmq: Compare to what we got in return, not what we verified at the end
It might happen that we get 32 pending but do only verify less than 32 and in this case we would assume there is no more work but it could still be more in the pipeline from my understanding.
* llmq: Rename more_work -> fMoreWork
* llmq: Be consistent with the other fMoreWork initialization
Co-authored-by: xdustinface <xdustinfacex@gmail.com>
* llmq: Avoid writing commitments to evodb and altering caches when all we want is to check block candidate validity
* tests: call `getblocktemplate` to trigger `CreateNewBlock` before quorum commitment is mined
* llmq: Implement automated DKG recovery threads
* llmq: Implement quorum verification vector sync
* init: Validiate quorum data recovery related command line parameter
* test: Add quorum_data_request_timeout_seconds in DashTestFramework
* test: Test quorum data recovery in feature_llmq_data_recovery.py
* test: Add feature_llmq_data_recovery.py to BASE_SCRIPTS
* test: Fix quorum_data_request_expiration_timeout in wait_for_quorum_data
* test: Always test the existence of secretKeyShare in test_mn_quorum_data
With this change it also validates that "secretKeyShare" is not in `quorum_info` if its not expected to be in there. Before this was basically just not tested.
* llmq|test: Use bool as argument type for -llmq-data-recovery
* llmq: Always set nTimeLastSuccess to 0
* test: Set -llmq-data-recovery=0 in p2p_quorum_data.py
* test: Simplify test_mns
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* refactor: pass CQuorumCPtr to StartQuorumDataRecoveryThread
* test: Fix thread name in comment
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* version: Bump PROTOCOL_VERSION and MIN_MASTERNODE_PROTO_VERSION
* version: Introduce LLMQ_DATA_MESSAGES_VERSION for QGETDATA/QDATA support
* test: Bump MY_VERSION to 70219 (LLMQ_DATA_MESSAGES_VERSION)
* llmq: Introduce CQuorumDataRequest as wrapper for QGETDATA requests
* llmq: Implement CQuorum::{SetVerificationVector, SetSecretKeyShare}
* llmq|net|protocol: Implement QGETDATA/QDATA P2P messages
* llmq: Restrict processing QGETDATA/QDATA to masternodes only
* llmq: Implement request limiting for QGETDATA/QDATA
* llmq: Implement CQuorumManger::RequestQuorumData
* rpc: Implement "quorum getdata" as wrapper around QGETDATA
Allows to trigger sending QGETDATA messages to connected peers by RPC.
* test: Handle QGETDATA/QDATA messages in mininode
* test: Add data structures to support QGETDATA/QDATA
* test: Add some helper in test_framework.py
* test: Implement tests for QGETDATA/QDATA in p2p_quorum_data.py
* test: Add p2p_quorum_data.py to BASE_SCRIPTS
* llmq|test: Add QWATCH support for QGETDATA/QDATA
* llmq: Store CQuorumPtr in cache, not CQuorumCPtr
* llmq: Fix cache usage after recent changes
* Use uacomment to create/find specific p2ps
* No need to use network adjusted time here, GetTime should be enough
* rpc: check proTxHash
* minor tweaks
* test: Adjustments after 4e27d6513e
* llmq: Rename and improve error lambda in CQuorumManager::ProcessMessage
* llmq: Process QDATA if -watchquorums is enabled
* test: Handle qwatch messages in mininode
* test: Add test for -watchquorums support
* test: Just some empty lines
* test: Properly stop the p2p network thread at the end of the test
* rpc: Adjust "quorum getdata" parameter descriptions
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
* rpc: Fix optionality of proTxHash in "quorum getdata" command
* test: Test optionality of proTxHash for "quorum getdata" command
* test: Be more specific about imports in p2p_quorum_data.py
* llmq|rpc: Add some comments about the request.GetDataMask checks
* test: Some more empty lines
* rpc: One more parameter description
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
* test: Unify assert statements / drop parentheses for all of them
* fix typo
Signed-off-by: pasta <pasta@dashboost.org>
* adjust some line wrapping to 80 chars
Signed-off-by: pasta <pasta@dashboost.org>
* tests: Seperate out into dif atomic methods, add logging
Signed-off-by: pasta <pasta@dashboost.org>
* test: Avoid restarting masternodes, just let available requests expire
Just takes a lot time and isn't required imo.
* test: Drop redundant code/tests after separation
This was introduced in 9e224ec2f2
* test: Merge three tests
"test_mnauth_restriction", "test_invalid_messages" and "test_invalid_unexpected_qdata" with the resulting name "test_basics" because i don't feel like DKG recovery thing should be part of a test called "test_invalid_messages" and giving it an own test probably wouldn't make a lot sense because it would still depend on "test_invalid_messages". I also think there is no need for a separated "test_invalid_unexpected_qdata".
* test: Rename test_ratelimiting_banscore -> test_request_limit
* test: Apply python style
* test: Wrap all at 120 characters
Thats the default "draw annoying warnings" setting for PyCharm (and IMO a reasonable line length).
* test: Move some variables
* test: Optimize for speed
* tests: use wait_until in get_mininode_id
* test: Don't use `!=` to check for `None`
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
Co-authored-by: pasta <pasta@dashboost.org>
* llmq: Detach dash-q-cachepop from caller
There should be no reason to keep this tread attached
to its parent, if so, let me know.
* llmq: Avoid nullptr access for pindexStart in ScanQuorums
* llmq: Add cacheKey in ProcessCommitment
* llmq: Erase minable commitments if they have been processed
* llmq: Add CLLMQUtils::InitQuorumsCache
* llmq: Use unordered_lru_cache for quorumsCache and rename it
* llmq: Use unordered_lru_cache for hasMinedCommitmentCache and rename it
* llmq: Drop redundant check
* llmq: Rename nMaxCount2 -> nScanCommitments
* llmq: Refactor storeCache -> fCacheExists
* llmq: Rename maxCount -> nCountRequested
* llmq: Rename result -> vecResultQuorums
* llmq: Return an empty vector if the are zero elements requested
* unordered_lru_cache: Add max_size()
* llmq: Partially reuse existing cache if more than max is requested
* llmq: std::map<LLMQType, unordered_lru_cache<...>> for scanQuoumsCache
* llmq: Drop params
* llmq: Only emplace to cache if there is something available
* llmq: Add CDKGSessionManager::WriteEncryptedContributions
Allows to store each member's encrypted contributions of the DKG.
* llmq: Store each member's contributions in the llmq database
* llmq: Add CDKGSessionManager::GetEncryptedContributions
I decided to don't cache here since its probably very unlikely this is called twice in a short period with what we have planed for it so far. We can add caching if the requirement for it changes at some point?
* Use single-threaded scheduler for IS, CL and Governance notifications
* Pass shared_ptr-s instead of objects themselves for CL, IS and Governance notifiers in CMainSignals/CValidatibnInterface
* llmq: Create shared_ptr for clsig at the root of its lifetime
* llmq: Create shared_ptr for islock clsig at the root of its lifetime
* llmq: Create shared_ptr for recSig at the root of its lifetime
Co-authored-by: xdustinface <xdustinfacex@gmail.com>
* llmq: Refactor CQuorumManager::{BuildQuorumFromCommitment, GetQuorum}
Construct and cache new quorums inside BuildQuorumFromCommitment
* llmq: Make all methods of CQuorumManager const
* More accurate handling of the BLOCK_CONFLICT_CHAINLOCK flag
* Update test/functional/feature_llmq_chainlocks.py
Co-authored-by: thephez <thephez@users.noreply.github.com>
* tests: make sure that previous tip on the reorged node is marked conflicting after chainlock
* Apply suggestions from code review
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: thephez <thephez@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* llmq: Split CSigShare creation/processing in CSigSharesManager
* rpc: Add "submit" parameter to "quorum sign"
* test: Add CSigShare and msg_qsigshare to messages.py
* test: Test the optional "submit" parameter of "quorum sign"
* random: Introduce std::shuffle alternative for FastRandomContext
3db746beb4
* random: change std::random_shuffle calls to std::shuffle
https://en.cppreference.com/w/cpp/algorithm/random_shuffle (deprecated in c++14)
* random: change FastRandomContext std::random_shuffle calls to shuffle
* random: change last std::shuffle calls to Shuffle
std::shuffle doesn't accept only two arguments so we use FastRandomContext()
* llmq: use inherited FastRandomContext
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* llmq: use inherited FastRandomContext
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Make the linter happy :)
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* Do not store hashes in ProcessPendingMessageBatch and do not pass them to ReceiveMessage-s
* follow up: refactor/unify code in ProcessPendingMessageBatch
* llmq: Add an optional quorum hash to CSigningManager::AsyncSignIfMember
Allows to select the quorum to sign by its hash.
* rpc: Fix quorum selection of "quorum sign"
* test: Test the optional "quorumHash" parameter of "quorum sign"
* llmq: Move quorum checks up to avoid calling WriteVoteForId if they fail
* llmq: Drop hash parameter in PreVerifyMessage methods
* llmq: Drop some unused variables
* rpc: Drop unused variable
* llmq|net: Drop some unused CConnman parameter
* llmq: Drop some unused quorum parameter
* llmq: Drop some unused nodeId parameter
* Drop unused variables
* llmq: Drop more
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* llmq: Fix GetVerifiedContribution to return false in case of failure
* llmq: Move GetVerifiedContribution into GetVerifiedContributions
* llmq: Drop GetVerifiedContribution
* llmq: Keep cache locked while building GetVerifiedContributions result
* llmq: Read from DB into vvecPtr directly
Rename `sigSharesToAnnounce` class member in `CSigSharesManager` to `sigSharesQueuedToAnnounce` to avoid confusion with local variables/function args with the same name
* llmq: Restrict `ShouldSimulateError` to trigger for LLMQ_TEST only
Current `develop` tests fail. This was basically introduced by dashpay#3844 but it didn't come up before dashpay#3853 because the `v17` fork wasn't activated in `feature_llmq_dkgerrors.py`.
After dashpay#3853 `dip0008` activation takes [200 blocks](b95cf017c3 (diff-4a04bc0b355c780033960e8c261ee9b6d3c452897e1dcd88a15d272512266c76R539)) which was normally activated after [10 blocks](b95cf017c3 (diff-b92fa0fafafa27172736ebc88f9f9b658b1160caca512a318eefb7d93d22bf3cL18)) in `feature_llmq_dkgerrors.py`. Now with the 200 blocks `v17` gets activated during test which then leads to MN1, MN2 banning MN0 because it lies in DKG of `LLMQ_TEST` and `LLMQ_TEST_V17`.
There are other ways to solve it, like enabling `dip0008` earlier or enable `v17` later but IMO its anyway better to restrict `ShouldSimulateError` to only trigger for `LLMQ_TEST`.
* Revert "llmq: Restrict `ShouldSimulateError` to trigger for LLMQ_TEST only"
This reverts commit ec42d86126.
* llmq: Restrict `ShouldSimulateError` to trigger for LLMQ_TEST only (alternative)
Move ShouldSimulateError into CDKGSession
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Add LLMQ_100_67 quorums
* Re-use DEPLOYMENT_V17 bit to activate LLMQ_100_67 quorums
* Add LLMQ_TEST_NEW quorum and test its activation
* Tweak mine_quorum to work correctly with multiple quorum types
And to avoid a potentialy endless "while" loop
* llmq: Rename IsQuorumTypeEnabledAtBlock -> IsQuorumTypeEnabled
* chainparams|test: Rename llmq_test_new -> llmq_test_v17
* chainparams|consensus|llmq: Rename LLMQ_TEST_NEW -> LLMQ_TEST_V17
* Tweak few strings and the name of the test
* llmq: Make GetEnabledQuorumTypes return a vector of LLMQTypes, introduce GetLLMQParams
Signed-off-by: pasta <pasta@dashboost.org>
* Tweak minSize
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
* Exclude LLMQ_100_67 from Concentrated Recovery
* Update test/functional/feature_new_quorum_type_activation.py
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
Co-authored-by: xdustinface <xdustinfacex@gmail.com>
Co-authored-by: pasta <pasta@dashboost.org>
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
* Send islock notifications for txes received after their islocks were received
Also drop UpdateWalletTransaction - its name makes no sense and it's only used once.
* tests: early islocks should trigger notifications once a corresponding tx is received
* Tweak tests
- fail if an unexpected islock is received
- drop unused variable
* llmq: Drop `c_str()` in two log statements
* test: Move create_islock to DashTestFramework in test_framework.py
Just because it's used the same way in two files
* test: Simplify send tx in zmq test
* format
Co-authored-by: xdustinface <xdustinfacex@gmail.com>
* Fix potential deadlock in `CSporkManager::UpdateSpork()`
* Protect `inputRequestIds` with cs lock
* Protect `curDBTransaction` in `CEvoDB::CommitRootTransaction()`
* Check for `AssertLockNotHeld` in `EnforceBestChainLock()` instead of just having a comment in code
* Protect spork maps on (de)serialization
* Handle situations when we have evodb records but no blocks anymore
* Use IsEmpty
* Apply suggestions from code review
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* Always check for previous quorum set in llmq IS
* Refactor SelectQuorumForSigning and related code
Should have no changes in behaviour
* Do not use SIGN_HEIGHT_OFFSET when checking pending IS locks, use actual chain tip
This commit actually changes the behaviour
It seems like the idea here was to avoid announcing already available sigshares for a signhash to everyone if concentrated recovery is enabled. But by returning there if `IsAllMembersConnectedEnabled` equals to `false` we disable the re-announcements prior to the spork activation, where we want to still have them and re-enable if the spork is enabled where we don't want to have them.
* Rename CollectSigSharesToSend to CollectSigSharesToSendConcentrated
* Allow to disable concentrated recovery and new connection handling for large LLMQs
* Drop dead code in DoInvalidateBlock
* Let ActivateBestChain skip SyncWithValidationInterfaceQueue when called from IS or CL threads
* Use CL's own scheduler instead of a global one
* Revert "Let ActivateBestChain skip SyncWithValidationInterfaceQueue when called from IS or CL threads"
This reverts commit 1c9f6da50a.
* Return invalid state for GetTxPayload failures instead of asserting
* Throw an error when internal `find` fails instead of asserting
* Unify mnlist checks for lists created from blocks and lists created via updates of older lists with diffs
* Wrap evo block processing in try..catch to make sure all earlier introduced throws are caught
* evo/llmq: Adjust reject reasons for CFinalCommitmentTxPayload
In CheckLLMQCommitment this reject string was used so i adjusted the
other one in CQuorumBlockProcessor::GetCommitmentsFromBlock and the new
introduced in PR #3506
* return state.DoS instead of asserting in cbtx.cpp
Signed-off-by: pasta <pasta@dashboost.org>
Co-authored-by: xdustinface <xdustinfacex@gmail.com>
Co-authored-by: pasta <pasta@dashboost.org>
* llmq: Add more information to the result of CFinalCommitment::ToJson
Example output:
```
{
"version": 1,
"llmqType": 1,
"quorumHash":
"000002a20a35757fc467f0ebe8663f76e4c5df74411cc5f1735cfd92278071fe",
"signersCount": 49,
"signers": "03fbffffffffff",
"validMembersCount": 49,
"validMembers": "03fbffffffffff",
"quorumPublicKey":
"8db1108b50dd741076b577f6798f0bf382ca4d11dafd55ed14aaaf613f604aa55bf1da1f0469f75105376e440381ee23",
"quorumVvecHash":
"feedb15d5a7ca25a68d9c51036973d37925d8439d4769d7dd7bace7440089071",
"quorumSig":
"1879f6260ef427dc9542088b39e02a8e73e34c0484269cbc12108ab991d438f81e5be80a5b4b5595b6634113fdf060060c8fe190677627fb2d25f6e052c7f93598d36dd415f8f84c693c6a5943098616a1e135522852d8f572bb6e1470df4a94",
"membersSig":
"8ef7d881fcc1e8111dd82c556ab62dad56ea5ecb47a9ac067eb2057d92d27e39530630a7f77d19adc857d33eb485a1b601b2d78ac4c73742f2036089bfb1f25441f1dfcd3cfd1eed6e7d664c0e97bf75b7d7097ebd647791fbcef070fe3e3e91"
}
```
* llmq: Changed the byte order in ToHexStr(const std::vector<bool>& vBits)
* 🪲 improve evodb consistency recovering from dbcrash
* Adjust the fix
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Fix it
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Disable recovery from a crash during a fork and a corresponding part of dbcrash.py
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Skip some checks in CQuorumBlockProcessor when replaying blocks after the crash
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Process special txes in RollforwardBlock
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Update src/init.cpp
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
This was only used in only one remaining place and only to ensure that
reject messages are sent before closing sockets. This is solved by the
previous commit now.
The new way does not require the quorum hash to be included in the
calculation, which allows us to have determinism globally and not only
for single LLMQs.
Instead of propagating all sig shares to all LLMQ members, this will now
make all members send their individual sig share to a single member, which
is then responsible for the recovery and propagation of the recovered
signature. This process is repeated by all members every second for another
target/recovering member, until a recovered signature appears.
When taking the proTxHash naively, we might end up with a few unlucky MNs
which always have to perform most of the outbound connections while other
unlucky MNs would always have to wait for inbound connections. Hashing
the proTxHash with the quorum hash makes this more random.
Instead of only doing it the first time. This is a preparation for
the new intra-quorum connection system (which connects all members to all
other members)
* scripted-diff: Replace #include "" with #include <> (ryanofsky)
-BEGIN VERIFY SCRIPT-
for f in \
src/*.cpp \
src/*.h \
src/bench/*.cpp \
src/bench/*.h \
src/compat/*.cpp \
src/compat/*.h \
src/consensus/*.cpp \
src/consensus/*.h \
src/crypto/*.cpp \
src/crypto/*.h \
src/crypto/ctaes/*.h \
src/policy/*.cpp \
src/policy/*.h \
src/primitives/*.cpp \
src/primitives/*.h \
src/qt/*.cpp \
src/qt/*.h \
src/qt/test/*.cpp \
src/qt/test/*.h \
src/rpc/*.cpp \
src/rpc/*.h \
src/script/*.cpp \
src/script/*.h \
src/support/*.cpp \
src/support/*.h \
src/support/allocators/*.h \
src/test/*.cpp \
src/test/*.h \
src/wallet/*.cpp \
src/wallet/*.h \
src/wallet/test/*.cpp \
src/wallet/test/*.h \
src/zmq/*.cpp \
src/zmq/*.h
do
base=${f%/*}/ relbase=${base#src/} sed -i "s:#include \"\(.*\)\"\(.*\):if test -e \$base'\\1'; then echo \"#include <\"\$relbase\"\\1>\\2\"; else echo \"#include <\\1>\\2\"; fi:e" $f
done
-END VERIFY SCRIPT-
Signed-off-by: Pasta <pasta@dashboost.org>
* scripted-diff: Replace #include "" with #include <> (Dash Specific)
-BEGIN VERIFY SCRIPT-
for f in \
src/bls/*.cpp \
src/bls/*.h \
src/evo/*.cpp \
src/evo/*.h \
src/governance/*.cpp \
src/governance/*.h \
src/llmq/*.cpp \
src/llmq/*.h \
src/masternode/*.cpp \
src/masternode/*.h \
src/privatesend/*.cpp \
src/privatesend/*.h
do
base=${f%/*}/ relbase=${base#src/} sed -i "s:#include \"\(.*\)\"\(.*\):if test -e \$base'\\1'; then echo \"#include <\"\$relbase\"\\1>\\2\"; else echo \"#include <\\1>\\2\"; fi:e" $f
done
-END VERIFY SCRIPT-
Signed-off-by: Pasta <pasta@dashboost.org>
* build: Remove -I for everything but project root
Remove -I from build system for everything but the project root,
and built-in dependencies.
Signed-off-by: Pasta <pasta@dashboost.org>
# Conflicts:
# src/Makefile.test.include
* qt: refactor: Use absolute include paths in .ui files
* qt: refactor: Changes to make include paths absolute
This makes all include paths in the GUI absolute.
Many changes are involved as every single source file in
src/qt/ assumes to be able to use relative includes.
Signed-off-by: Pasta <pasta@dashboost.org>
# Conflicts:
# src/qt/dash.cpp
# src/qt/optionsmodel.cpp
# src/qt/test/rpcnestedtests.cpp
* test: refactor: Use absolute include paths for test data files
* Recommend #include<> syntax in developer notes
* refactor: Include obj/build.h instead of build.h
* END BACKPORT #11651 Remove trailing whitespace causing travis failure
* fix backport 11651
Signed-off-by: Pasta <pasta@dashboost.org>
* More of 11651
* fix blockchain.cpp
Signed-off-by: pasta <pasta@dashboost.org>
* Add missing "qt/" in includes
* Add missing "test/" in includes
* Fix trailing whitespaces
Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com>
Co-authored-by: Russell Yanofsky <russ@yanofsky.org>
Co-authored-by: MeshCollider <dobsonsa68@gmail.com>
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Do not sleep at the last block of the phase, it's not safe
* Refactor it a bit to make it clearer what's going on here
* Stop sleeping if blocks came faster than we expected
* Pass nAcceptTime via TransactionAddedToMempool and use it for ChainLocks
Otherwise the "first seen" time is way off after node restart
* Don't skip TransactionAddedToMempool for chainlocks while blockchain is not synced yet
Otherwise txes from mempool.dat won't be processed there
This should make code a little bit cleaner, should be no changes in the actual behaviour: non-members do not sleep already due to `sleepTime` being negative for them and `phaseTime = 0` does the same on regtest but for everyone.
* Implement re-signing of InstantSend inputs when TXs come in via blocks
* Use GetAdjustedTime instead of GetTimeMillis in CSigSharesManager
This allows use of mocktime in tests.
* Expose verifiedProRegTxHash in getpeerinfo and implement wait_for_mnauth
* Allow to wait for IS and CL to NOT happen
* Bump timeout for wait_for_instantlock
* Implement tests for retroactive signing of IS and CLs
* Add wait_for_tx function to DashTestFramework
* Add -whitelist=127.0.0.1 to node0
* Use node3 for isolated block generation
* Don't test for non-receival of TXs on node4/node5
