sikkienl/dash
1
0
Fork 0
mirror of https://github.com/dashpay/dash.git synced 2024-12-27 04:52:59 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
14c631691f
dash/test/lint/lint-circular-dependencies.sh
Pieter Wuille 7ff6515c88
Merge #13033: Build txindex in parallel with validation 
9b2704777c [doc] Include txindex changes in the release notes. (Jim Posen)
ed77dd6b30 [test] Simple unit test for TxIndex. (Jim Posen)
6d772a3d44 [rpc] Public interfaces to GetTransaction block until synced. (Jim Posen)
a03f804f2a [index] Move disk IO logic from GetTransaction to TxIndex::FindTx. (Jim Posen)
e0a3b80033 [validation] Replace tx index code in validation code with TxIndex. (Jim Posen)
8181db88f6 [init] Initialize and start TxIndex in init code. (Jim Posen)
f90c3a62f5 [index] TxIndex method to wait until caught up. (Jim Posen)
70d510d93c [index] Allow TxIndex sync thread to be interrupted. (Jim Posen)
94b4f8bbb9 [index] TxIndex initial sync thread. (Jim Posen)
34d68bf3a3 [index] Create new TxIndex class. (Jim Posen)
c88bcec93f [db] Migration for txindex data to new, separate database. (Jim Posen)
0cb8303241 [db] Create separate database for txindex. (Jim Posen)

Pull request description:

  I'm re-opening #11857 as a new pull request because the last one stopped loading for people

  -------------------------------

  This refactors the tx index code to be in it's own class and get built concurrently with validation code. The main benefit is decoupling and moving the txindex into a separate DB. The primary motivation is to lay the groundwork for other indexers that might be desired (such as the [compact filters](https://github.com/bitcoin/bips/pull/636)). The basic idea is that the TxIndex spins up its own thread, which first syncs the txindex to the current block index, then once in sync the BlockConnected ValidationInterface hook writes new blocks.

  ### DB changes

  At the suggestion of some other developers, the txindex has been split out into a separate database. A data migration runs at startup on any nodes with a legacy txindex. Currently the migration blocks node initialization until complete.

  ### Open questions

  - Should the migration of txindex data from the old DB to the new DB block in init or should it happen in a background thread? The downside to backgrounding it is that `getrawtransaction` would return an error message saying the txindex is syncing while the migration is running.

  ### Impact

  In a sample size n=1 test where I synced nodes from scratch, the average time [Index writing](https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L1903) was 3.36ms in master and 1.72ms in this branch. The average time between `UpdateTip` log lines for sequential blocks between 400,000 and IBD end on mainnet was 0.297204s in master and 0.286134s in this branch. Most likely this is just variance in IBD times, but I can try with some more trials if people want.

Tree-SHA512: 451fd7d95df89dfafceaa723cdf0f7b137615b531cf5c5035cfb54e9ccc2026cec5ac85edbcf71b7f4e2f102e36e9202b8b3a667e1504a9e1a9976ab1f0079c4
2021-05-25 13:48:04 +03:00

174 lines
10 KiB
Bash
Executable File
Raw Blame History

#!/usr/bin/env bash
#
# Copyright (c) 2018 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Check for circular dependencies
export LC_ALL=C
EXPECTED_CIRCULAR_DEPENDENCIES=(
"chainparamsbase -> util -> chainparamsbase"
"checkpoints -> validation -> checkpoints"
"index/txindex -> validation -> index/txindex"
"policy/fees -> txmempool -> policy/fees"
"policy/policy -> validation -> policy/policy"
"qt/addresstablemodel -> qt/walletmodel -> qt/addresstablemodel"
"qt/bantablemodel -> qt/clientmodel -> qt/bantablemodel"
"qt/bitcoingui -> qt/utilitydialog -> qt/bitcoingui"
"qt/bitcoingui -> qt/walletframe -> qt/bitcoingui"
"qt/bitcoingui -> qt/walletview -> qt/bitcoingui"
"qt/clientmodel -> qt/peertablemodel -> qt/clientmodel"
"qt/paymentserver -> qt/walletmodel -> qt/paymentserver"
"qt/recentrequeststablemodel -> qt/walletmodel -> qt/recentrequeststablemodel"
"qt/transactiontablemodel -> qt/walletmodel -> qt/transactiontablemodel"
"qt/walletmodel -> qt/walletmodeltransaction -> qt/walletmodel"
"rpc/rawtransaction -> wallet/rpcwallet -> rpc/rawtransaction"
"txmempool -> validation -> txmempool"
"validation -> validationinterface -> validation"
"wallet/fees -> wallet/wallet -> wallet/fees"
"wallet/wallet -> wallet/walletdb -> wallet/wallet"
"policy/fees -> policy/policy -> validation -> policy/fees"
"qt/addressbookpage -> qt/bitcoingui -> qt/walletview -> qt/addressbookpage"
"txmempool -> validation -> validationinterface -> txmempool"
"qt/addressbookpage -> qt/bitcoingui -> qt/walletview -> qt/receivecoinsdialog -> qt/addressbookpage"
"qt/addressbookpage -> qt/bitcoingui -> qt/walletview -> qt/signverifymessagedialog -> qt/addressbookpage"
"qt/addressbookpage -> qt/bitcoingui -> qt/walletview -> qt/sendcoinsdialog -> qt/sendcoinsentry -> qt/addressbookpage"
# Dash
"coinjoin/coinjoin-server -> init -> coinjoin/coinjoin-server"
"coinjoin/coinjoin-server -> net_processing -> coinjoin/coinjoin-server"
"coinjoin/coinjoin-client -> init -> dsnotificationinterface -> coinjoin/coinjoin-client"
"coinjoin/coinjoin-client -> init -> masternode/masternode-utils -> coinjoin/coinjoin-client"
"evo/cbtx -> evo/simplifiedmns -> evo/cbtx"
"evo/cbtx -> evo/specialtx -> evo/cbtx"
"evo/deterministicmns -> evo/providertx -> evo/deterministicmns"
"evo/deterministicmns -> evo/simplifiedmns -> evo/deterministicmns"
"evo/deterministicmns -> evo/specialtx -> evo/deterministicmns"
"evo/deterministicmns -> llmq/quorums_commitment -> evo/deterministicmns"
"evo/deterministicmns -> llmq/quorums_utils -> evo/deterministicmns"
"evo/deterministicmns -> validation -> evo/deterministicmns"
"evo/mnauth -> net_processing -> evo/mnauth"
"evo/specialtx -> llmq/quorums_blockprocessor -> evo/specialtx"
"evo/specialtx -> llmq/quorums_commitment -> evo/specialtx"
"evo/specialtx -> validation -> evo/specialtx"
"governance/governance -> governance/governance-classes -> governance/governance"
"governance/governance -> governance/governance-object -> governance/governance"
"governance/governance -> init -> governance/governance"
"governance/governance -> masternode/masternode-sync -> governance/governance"
"governance/governance -> net_processing -> governance/governance"
"governance/governance-object -> governance/governance-validators -> governance/governance-object"
"init -> masternode/masternode-sync -> init"
"init -> masternode/masternode-utils -> init"
"init -> net_processing -> init"
"init -> netfulfilledman -> init"
"init -> rpc/server -> init"
"init -> txdb -> init"
"init -> validation -> init"
"init -> validationinterface -> init"
"llmq/quorums -> llmq/quorums_utils -> llmq/quorums"
"llmq/quorums_blockprocessor -> net_processing -> llmq/quorums_blockprocessor"
"llmq/quorums_chainlocks -> llmq/quorums_instantsend -> llmq/quorums_chainlocks"
"llmq/quorums_chainlocks -> net_processing -> llmq/quorums_chainlocks"
"llmq/quorums_dkgsessionmgr -> net_processing -> llmq/quorums_dkgsessionmgr"
"llmq/quorums_instantsend -> net_processing -> llmq/quorums_instantsend"
"llmq/quorums_instantsend -> txmempool -> llmq/quorums_instantsend"
"llmq/quorums_instantsend -> validation -> llmq/quorums_instantsend"
"llmq/quorums_signing -> llmq/quorums_signing_shares -> llmq/quorums_signing"
"llmq/quorums_signing -> net_processing -> llmq/quorums_signing"
"llmq/quorums_signing_shares -> net_processing -> llmq/quorums_signing_shares"
"logging -> util -> logging"
"masternode/masternode-payments -> validation -> masternode/masternode-payments"
"net -> netmessagemaker -> net"
"net_processing -> spork -> net_processing"
"netaddress -> netbase -> netaddress"
"qt/appearancewidget -> qt/guiutil -> qt/appearancewidget"
"qt/bitcoinaddressvalidator -> qt/guiutil -> qt/bitcoinaddressvalidator"
"qt/bitcoingui -> qt/guiutil -> qt/bitcoingui"
"qt/guiutil -> qt/optionsdialog -> qt/guiutil"
"qt/guiutil -> qt/qvalidatedlineedit -> qt/guiutil"
"core_io -> evo/cbtx -> evo/deterministicmns -> core_io"
"core_io -> evo/cbtx -> evo/simplifiedmns -> core_io"
"dsnotificationinterface -> governance/governance -> init -> dsnotificationinterface"
"evo/simplifiedmns -> llmq/quorums_blockprocessor -> net_processing -> evo/simplifiedmns"
"httprpc -> httpserver -> init -> httprpc"
"httpserver -> init -> httpserver"
"init -> llmq/quorums_init -> llmq/quorums_signing_shares -> init"
"llmq/quorums_dkgsession -> llmq/quorums_dkgsessionmgr -> llmq/quorums_dkgsessionhandler -> llmq/quorums_dkgsession"
"logging -> util -> random -> logging"
"logging -> util -> sync -> logging"
"logging -> util -> stacktraces -> logging"
"coinjoin/coinjoin-client -> coinjoin/coinjoin-util -> wallet/wallet -> coinjoin/coinjoin-client"
"qt/appearancewidget -> qt/guiutil -> qt/optionsdialog -> qt/appearancewidget"
"qt/bitcoinaddressvalidator -> qt/guiutil -> qt/qvalidatedlineedit -> qt/bitcoinaddressvalidator"
"qt/guiutil -> qt/optionsdialog -> qt/optionsmodel -> qt/guiutil"
"bloom -> evo/cbtx -> evo/simplifiedmns -> merkleblock -> bloom"
"bloom -> evo/cbtx -> llmq/quorums_blockprocessor -> net -> bloom"
"evo/deterministicmns -> evo/simplifiedmns -> llmq/quorums_blockprocessor -> llmq/quorums_debug -> evo/deterministicmns"
"evo/deterministicmns -> evo/simplifiedmns -> llmq/quorums_blockprocessor -> net_processing -> evo/deterministicmns"
"evo/specialtx -> llmq/quorums_blockprocessor -> net_processing -> txmempool -> evo/specialtx"
"evo/providertx -> evo/specialtx -> llmq/quorums_blockprocessor -> net_processing -> txmempool -> evo/providertx"
"coinjoin/coinjoin-client -> net_processing -> coinjoin/coinjoin-client"
"llmq/quorums -> net_processing -> llmq/quorums"
"llmq/quorums_commitment -> llmq/quorums_utils -> llmq/quorums_commitment"
"llmq/quorums_dkgsession -> llmq/quorums_dkgsessionmgr -> llmq/quorums_dkgsession"
"coinjoin/coinjoin -> masternode/activemasternode -> net -> coinjoin/coinjoin"
"evo/deterministicmns -> validationinterface -> txmempool -> evo/deterministicmns"
"llmq/quorums -> validation -> llmq/quorums_chainlocks -> llmq/quorums"
"llmq/quorums_chainlocks -> llmq/quorums_instantsend -> validation -> llmq/quorums_chainlocks"
"evo/deterministicmns -> evo/simplifiedmns -> llmq/quorums_blockprocessor -> net_processing -> masternode/masternode-payments -> evo/deterministicmns"
"core_io -> evo/cbtx -> llmq/quorums_blockprocessor -> net_processing -> masternode/masternode-payments -> governance/governance-classes -> core_io"
"evo/deterministicmns -> evo/simplifiedmns -> llmq/quorums_blockprocessor -> net_processing -> masternode/masternode-payments -> masternode/activemasternode -> evo/deterministicmns"
"evo/deterministicmns -> evo/simplifiedmns -> llmq/quorums_blockprocessor -> net -> masternode/masternode-sync -> evo/deterministicmns"
"governance/governance -> net_processing -> masternode/masternode-payments -> governance/governance"
"core_io -> evo/cbtx -> llmq/quorums_blockprocessor -> net_processing -> masternode/masternode-payments -> governance/governance-classes -> governance/governance-object -> core_io"
"evo/deterministicmns -> evo/simplifiedmns -> llmq/quorums_blockprocessor -> net_processing -> masternode/masternode-payments -> governance/governance-classes -> governance/governance-object -> evo/deterministicmns"
"evo/deterministicmns -> evo/simplifiedmns -> llmq/quorums_blockprocessor -> net -> evo/deterministicmns"
"evo/deterministicmns -> evo/simplifiedmns -> llmq/quorums_blockprocessor -> net_processing -> masternode/masternode-payments -> governance/governance-classes -> governance/governance-object -> governance/governance-vote -> evo/deterministicmns"
"index/txindex -> init -> index/txindex"
"index/txindex -> init -> llmq/quorums_init -> llmq/quorums_instantsend -> index/txindex"
"index/txindex -> init -> net_processing -> index/txindex"
"index/txindex -> init -> rpc/blockchain -> index/txindex"
)
EXIT_CODE=0
CIRCULAR_DEPENDENCIES=()
IFS=$'\n'
for CIRC in $(cd src && ../contrib/devtools/circular-dependencies.py {*,*/*,*/*/*}.{h,cpp} | sed -e 's/^Circular dependency: //'); do
CIRCULAR_DEPENDENCIES+=($CIRC)
IS_EXPECTED_CIRC=0
for EXPECTED_CIRC in "${EXPECTED_CIRCULAR_DEPENDENCIES[@]}"; do
if [[ "${CIRC}" == "${EXPECTED_CIRC}" ]]; then
IS_EXPECTED_CIRC=1
break
fi
done
if [[ ${IS_EXPECTED_CIRC} == 0 ]]; then
echo "A new circular dependency in the form of \"${CIRC}\" appears to have been introduced."
echo
EXIT_CODE=1
fi
done
for EXPECTED_CIRC in "${EXPECTED_CIRCULAR_DEPENDENCIES[@]}"; do
IS_PRESENT_EXPECTED_CIRC=0
for CIRC in "${CIRCULAR_DEPENDENCIES[@]}"; do
if [[ "${CIRC}" == "${EXPECTED_CIRC}" ]]; then
IS_PRESENT_EXPECTED_CIRC=1
break
fi
done
if [[ ${IS_PRESENT_EXPECTED_CIRC} == 0 ]]; then
echo "Good job! The circular dependency \"${EXPECTED_CIRC}\" is no longer present."
echo "Please remove it from EXPECTED_CIRCULAR_DEPENDENCIES in $0"
echo "to make sure this circular dependency is not accidentally reintroduced."
echo
EXIT_CODE=1
fi
done
exit ${EXIT_CODE}
Reference in New Issue View Git Blame Copy Permalink