9b2704777c [doc] Include txindex changes in the release notes. (Jim Posen)
ed77dd6b30 [test] Simple unit test for TxIndex. (Jim Posen)
6d772a3d44 [rpc] Public interfaces to GetTransaction block until synced. (Jim Posen)
a03f804f2a [index] Move disk IO logic from GetTransaction to TxIndex::FindTx. (Jim Posen)
e0a3b80033 [validation] Replace tx index code in validation code with TxIndex. (Jim Posen)
8181db88f6 [init] Initialize and start TxIndex in init code. (Jim Posen)
f90c3a62f5 [index] TxIndex method to wait until caught up. (Jim Posen)
70d510d93c [index] Allow TxIndex sync thread to be interrupted. (Jim Posen)
94b4f8bbb9 [index] TxIndex initial sync thread. (Jim Posen)
34d68bf3a3 [index] Create new TxIndex class. (Jim Posen)
c88bcec93f [db] Migration for txindex data to new, separate database. (Jim Posen)
0cb8303241 [db] Create separate database for txindex. (Jim Posen)
Pull request description:
I'm re-opening #11857 as a new pull request because the last one stopped loading for people
-------------------------------
This refactors the tx index code to be in it's own class and get built concurrently with validation code. The main benefit is decoupling and moving the txindex into a separate DB. The primary motivation is to lay the groundwork for other indexers that might be desired (such as the [compact filters](https://github.com/bitcoin/bips/pull/636)). The basic idea is that the TxIndex spins up its own thread, which first syncs the txindex to the current block index, then once in sync the BlockConnected ValidationInterface hook writes new blocks.
### DB changes
At the suggestion of some other developers, the txindex has been split out into a separate database. A data migration runs at startup on any nodes with a legacy txindex. Currently the migration blocks node initialization until complete.
### Open questions
- Should the migration of txindex data from the old DB to the new DB block in init or should it happen in a background thread? The downside to backgrounding it is that `getrawtransaction` would return an error message saying the txindex is syncing while the migration is running.
### Impact
In a sample size n=1 test where I synced nodes from scratch, the average time [Index writing](https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L1903) was 3.36ms in master and 1.72ms in this branch. The average time between `UpdateTip` log lines for sequential blocks between 400,000 and IBD end on mainnet was 0.297204s in master and 0.286134s in this branch. Most likely this is just variance in IBD times, but I can try with some more trials if people want.
Tree-SHA512: 451fd7d95df89dfafceaa723cdf0f7b137615b531cf5c5035cfb54e9ccc2026cec5ac85edbcf71b7f4e2f102e36e9202b8b3a667e1504a9e1a9976ab1f0079c4
* instantsend: Upgrade IS db on dip0020 activation
* instantsend: Do not remove islocks for txes removed from mempool until dip0020 is activated
* refactor: introduce fUpgradedDB to avoid excessive locking and checking the state
* llmq: Decouple `fUpgradedDB` from `cs`
* Update src/llmq/quorums_instantsend.cpp
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* Fix deadlock
Upgrade locks cs_main via GetTransaction
Co-authored-by: xdustinface <xdustinfacex@gmail.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* instantsend: Resolve block conflicts first and take care of mempool ones later
* refactor: Rename RemoveChainLockConflictingLock -> RemoveConflictingLock
* instantsend: Handle transaction removal from mempool (for all reasons besides inclusion in blocks)
* instantsend: Remove old islocks with no known txes from db (once)
* refactor: Replace magic number with CURRENT_VERSION
* fix: Do not remove islocks for (yet) valid orphans
* Apply suggestions from code review
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* tests: Check that CLs override ISes which invalidated non-CLed blocks earlier
* partial revert 3987: Do not mark blocks which conflict with ISes as "conflicting"
1527015 Avoid std::locale/imbue in DateTimeStrFormat (Pieter Wuille)
Pull request description:
And replace them with just hardcoded ISO8601 strings and `gmtime_r`.
Pointed out by @laanwj here: https://github.com/bitcoin/bitcoin/pull/12970#issuecomment-380962488
Tree-SHA512: a459758b42ca56f8462115aefe8e6377c1319fce509ea64dbb767f3f087c9b848335954cb684e5896c38008847684045505a3e1559fb3e83b8e80e10b003d1e7
* Simple changes, mostly just includes
* Continued include changes combined with using pointers to avoid including more than necessary in headers
* remove dup include
* masternode-utils.cpp include net.h in all builds
* resolve linter -7 +11
* drop quorums.h from dkgsessionhandler.cpp
* Add `<utilstrencodings.h>`
* Initialize lastMNListForVotingKeys
* Refactor GetMinedCommitment to return sharedptr
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* remove unused includes in validation.cpp
circular depends -2
* use more specific include in quorums.h
circular depends -1
* Remove unneeded include from quorums.cpp
circular depends -1
* llmq: Return bool in EnsureQuorumConnections
* llmq: Call RemoveMasternodeQuorumNodes directly
Instead of removing the good nodes from the vector in first loop and
call it with the bad ones in the second loop.
* refactor: Introduce and use CLLMQUtils::IsWatchQuorumsEnabled
* llmq: Move IsWatchQuorumsEnabled into EnsureQuorumConnections
* Fix extra line break in CommitTransaction log message
Introduced in #3680ebe7e80a49 (diff-b2bb174788c7409b671c46ccc86034bdR4113)
* doc: Fix `quorum sign` help
* doc: Add `getdata` to quorum commands list help
* doc: Drop "P2WSH" from signrawtransactionwithkey help
* trivial: Replace "push_back(Pair(..))" with "pushKV"
* trivial: Reorder wallet cmd-line options
* git: Add macos debug simbols to .gitignore
* trivial: Fix typos and whitespaces, drop unused stuff
* llmq: Protect nodeStates in CSigSharesManager::Cleanup()
* refactor: Tweak nodeState cleanup in CSigSharesManager::Cleanup()
* llmq: Add a couple of AssertLockHeld-s to ensure nodeStates protection
Instead of calling `CLLMQUtils::GetAllQuorumMembers` before each `CFinalCommitment::Verify` call to get the quorum members, just get them inside `CFinalCommitment::Verify`.
This avoids race condition for "spork19 off, spork 19 on" situations when some nodes might call CheckActiveState already, created new clsig and sent it so that other nodes processed it too already but they will call CheckActiveState afterwards (because it's done every 5 sec) and reset best chainlock to null.
* llmq|init|test: Add "mode" to -llmq-qvvec-sync parameter
This changes the paramter from `-llmq-qvvec-sync=<quorum_name>` to `-llmq-qvvec-sync=<quorum_name:mode>`
With the following definitions:
- `quorum_name`: Internal name of the quorum type
- `mode=0` - Sync always from all quorums of the type defined by `quorum_name`
- `mode=1` - Sync only if member of any from all other quorum of the type defined by `quorum_name`
`-llmq-qvvec-sync=llmq_100_67:0` To always request qvvec's from all `LLMQ_100_67`.
`-llmq-qvvec-sync=llmq_100_67:1` Only request if type member.
This means, if platform enables this on all MNs with `mode=0` we will
have all nodes asking new quorum for their verification vector instead
of only `24*100` at max.
* llmq: Adjust GetQuorumRecoveryStartOffset to use all MNs
* Turn `QvvecSyncMode` into `enum class`
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* llmq: Let return ProcessTx void instead of bool
The returned boolean is never used..
* llmq: Combine some "bail out early" cases
* llmq: Explicitly check for nullptr
* instantsend: refactor input locking into it's own method
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: introduce spork 24 `SPORK_24_INSTANTSEND_SIGNING_ENABLED`
This spork tells masternodes to refuse to lock transactions in mempool. Only transactions included in a block should be retroactively signed.
Signed-off-by: pasta <pasta@dashboost.org>
add spork defenition
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: refactor `sed -i 's/allowReSigning/fRetroactive/g' src/llmq/*`
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend: adjust comments
Signed-off-by: pasta <pasta@dashboost.org>
* instantsend/tests: implement Spork 24 support in tests, and test it's usage
Signed-off-by: pasta <pasta@dashboost.org>
* fix feature_llmq_is_retroactive.py
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* drop Spork 24 and use Spork 2 value 1 as being no mempool signing
Signed-off-by: pasta <pasta@dashboost.org>
* fix spork check
Signed-off-by: pasta <pasta@dashboost.org>
* Fix tests
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* Change comment
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* IsInstantSendSigningEnabled -> IsInstantSendMempoolSigningEnabled
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* instantsend: keep islocks cache when removing confirmed islocks from db
unordered_lru_cache should truncate it automagically
* instantsend: Batched write/erase for connected/disconnected blocks
* docs: Add help for -recsigsmaxage option
* Make `-llmq-data-recovery` help string dynamic
* Make devnet/regtest specific help strings dynamic
Only show devnet related help strings on devnet.
* Make linter happy
* Skip FindDevNetGenesisBlock when creating devnet params for help purposes only
* llmq: Use thread pool for quorum cache and data recovery threads
Move quorum data and cache thread handling into CQuorumManager.
* llmq: Fix explicit capture list
* llmq: Directly push the lamdas to workerPool
Co-authored-by: xdustinface <xdustinfacex@gmail.com>
* instantsend: Bail out early on disabled IS in more places
* instantsend: Disable InstantSend while reindexing and importing blocks
* Drop extra checks in GetInstantSendLockHashByTxid, GetInstantSendLockByInput and GetInstantSendLockByHash
* instantsend: Introduce (and use) KnownInstantSendLock
Check islock hash against both current and archived islocks
* instantsend: Optimize ProcessInstantSendLock, check for known islocks first
Two reasons:
1. GetTransaction is a potentially much heavier one
2. Doesn't make much sense to look for a tx if we have a known islock already
* Drop extra pendingInstantSendLocks.count(hash)
* Check pindexMined != nullptr
* Update src/llmq/quorums_instantsend.cpp
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
* Update src/llmq/quorums_instantsend.h
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com>
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
