This avoids sorting before looping through it to figure out what to
request. The assumption that sorting would be cheap when vecAskFor is
already mostly sorted (only unsorted at the end) turned out to be false.
In reality, ~50% of CPU time was consumed by the sort when a lot of traffic
(thousands of TXs) happen.
This makes orphan processing work like handling getdata messages:
After every actual transaction validation attempt, interrupt
processing to deal with messages arriving from other peers.
* Ignore recent rejects filter for locked txes
If we had a conflicting tx in the mempool before the locked tx arrived and the locked one arrived before the corresponding islock (i.e. we don't really know it's the one that should be included yet), the locked one is going to be rejected due to a mempool conflict. The old tx is going to be removed from the mempool by an incoming islock a bit later, however, we won't be able to re-request the locked tx until the tip changes because of the recentRejects filter. This patch fixes it.
* Add some explanation
6262915 Add unit test for stale tip checking (Suhas Daftuar)
83df257 Add CConnmanTest to mutate g_connman in tests (João Barbosa)
ac7b37c Connect to an extra outbound peer if our tip is stale (Suhas Daftuar)
db32a65 Track tip update time and last new block announcement from each peer (Suhas Daftuar)
2d4327d net: Allow connecting to extra outbound peers (Suhas Daftuar)
Pull request description:
This is an alternative approach to #11534. Rather than disconnect an outbound peer when our tip looks stale, instead try to connect to an additional outbound peer.
Periodically, check to see if we have more outbound peers than we target (ie if any extra peers are in use), and if so, disconnect the one that least recently announced a new block (breaking ties by choosing the newest peer that we connected to).
Tree-SHA512: 8f19e910e0bb36867f81783e020af225f356451899adfc7ade1895d6d3bd5afe51c83759610dfd10c62090c4fe404efa0283b2f63fde0bd7da898a1aaa7fb281
f3d4adf Make p2p-acceptablock not an extended test (Matt Corallo)
00dcda6 [qa] test that invalid blocks on an invalid chain get a disconnect (Matt Corallo)
015a525 Reject headers building on invalid chains by tracking invalidity (Matt Corallo)
932f118 Accept unrequested blocks with work equal to our tip (Matt Corallo)
3d9c70c Stop always storing blocks from whitelisted peers (Matt Corallo)
3b4ac43 Rewrite p2p-acceptblock in preparation for slight behavior changes (Matt Corallo)
Pull request description:
@sdaftuar pointed out that the version in #11487 was somewhat DoS-able as someone could feed you a valid chain that forked off the the last checkpoint block and force you to do lots of work just walking backwards across blocks for each new block they gave you. We came up with a few proposals but settled on the one implemented here as likely the simplest without obvious DoS issues. It uses our existing on-load mapBlockIndex walk to make sure everything that descends from an invalid block is marked as such, and then simply caches blocks which we attempted to connect but which were found to be invalid. To avoid DoS issues during IBD, this will need to depend on #11458.
Includes tests from #11487.
Tree-SHA512: 46aff8332908e122dae72ceb5fe8cd241902c2281a87f58a5fb486bf69d46458d84a096fdcb5f3e8e07fbcf7466232b10c429f4d67855425f11b38ac0bf612e1
2530bf2 net: Add missing lock in ProcessHeadersMessage(...) (practicalswift)
Pull request description:
Add missing lock in `ProcessHeadersMessage(...)`.
Reading the variable `mapBlockIndex` requires holding the mutex `cs_main`.
The new "Disconnect outbound peers relaying invalid headers" code added in commit 37886d5e2f and merged as part of #11568 two days ago did not lock `cs_main` prior to accessing `mapBlockIndex`.
Tree-SHA512: b799c234be8043d036183a00bc7867bbf3bd7ffe3baa94c88529da3b3cd0571c31ed11dadfaf29c5b8498341d6d0a3c928029a43b69f3267ef263682c91563a3
37886d5e2 Disconnect outbound peers relaying invalid headers (Suhas Daftuar)
4637f1852 moveonly: factor out headers processing into separate function (Suhas Daftuar)
Pull request description:
Alternate to #11446.
Disconnect outbound (non-manual) peers that serve us block headers that are already known to be invalid, but exempt compact block announcements from such disconnects.
We restrict disconnection to outbound peers that are using up an outbound connection slot, because we rely on those peers to give us connectivity to the honest network (our inbound peers are not chosen by us and hence could all be from an attacker/sybil). Maintaining connectivity to peers that serve us invalid headers is sometimes desirable, eg after a soft-fork, to protect unupgraded software from being partitioned off the honest network, so we prefer to only disconnect when necessary.
Compact block announcements are exempted from this logic to comply with BIP 152, which explicitly permits nodes to relay compact blocks before fully validating them.
Tree-SHA512: 3ea88e4ccc1184f292a85b17f800d401d2c3806fefc7ad5429d05d6872c53acfa5751e3df83ce6b9c0060ab289511ed70ae1323d140ccc5b12e3c8da6de49936
e065249 Add unit test for outbound peer eviction (Suhas Daftuar)
5a6d00c Permit disconnection of outbound peers on bad/slow chains (Suhas Daftuar)
c60fd71 Disconnecting from bad outbound peers in IBD (Suhas Daftuar)
Pull request description:
The first commit will disconnect an outbound peer that serves us a headers chain with insufficient work while we're in IBD.
The second commit introduces a way to disconnect outbound peers whose chains fall out of sync with ours:
For a given outbound peer, we check whether their best known block (which is known from the blocks they announce to us) has at least as much work as our tip. If it doesn't, we set a 20 minute timeout, and if we still haven't heard about a block with as much work as our tip had when we set the timeout, then we send a single getheaders message, and wait 2 more minutes. If after two minutes their best known block has insufficient work, we disconnect that peer.
We protect 4 of our outbound peers (who provide some "good" headers chains, ie a chain with at least as much work as our tip at some point) from being subject to this logic, to prevent excessive network topology changes as a result of this algorithm, while still ensuring that we have a reasonable number of nodes not known to be on bogus chains.
We also don't require our peers to be on the same chain as us, to prevent accidental partitioning of the network in the event of a chain split. Note that if our peers are ever on a more work chain than our tip, then we will download and validate it, and then either reorg to it, or learn of a consensus incompatibility with that peer and disconnect. This PR is designed to protect against peers that are on a less work chain which we may never try to download and validate.
Tree-SHA512: 2e0169a1dd8a7fb95980573ac4a201924bffdd724c19afcab5efcef076fdbe1f2cec7dc5f5d7e0a6327216f56d3828884f73642e00c8534b56ec2bb4c854a656
2525b972a net: stop both net/net_processing before destroying them (Cory Fields)
80e2e9d0c net: drop unused connman param (Cory Fields)
8ad663c1f net: use an interface class rather than signals for message processing (Cory Fields)
28f11e940 net: pass CConnman via pointer rather than reference (Cory Fields)
Pull request description:
See individual commits.
Benefits:
- Allows us to begin moving stuff out of CNode and into CNodeState (after #10652 and follow-ups)
- Drops boost dependency and overhead
- Drops global signal registration
- Friendlier backtraces
Tree-SHA512: af2038c959dbec25f0c90c74c88dc6a630e6b9e984adf52aceadd6954aa463b6aadfccf979c2459a9f3354326b5077ee02048128eda2a649236fadb595b66ee3
15f5d3b17 Switch DNSSeed-needed metric to any-automatic-nodes, not services (Matt Corallo)
5ee88b4bd Clarify docs for requirements/handling of addnode/connect nodes (Matt Corallo)
57edc0b0c Rename fAddnode to a more-descriptive "manual_connection" (Matt Corallo)
44407100f Replace relevant services logic with a function suite. (Matt Corallo)
Pull request description:
This was mostly written as a way to clean things up so that the NETWORK_LIMITED PR (#10387) can be simplified a ton, but its also a nice standalone cleanup that will also require a bit of review because it tweaks a lot of stuff across net. The new functions are fine in protocol.h right now since they're straight-forward, but after NETWORK_LIMITED will really want to move elsewhere after @theuni moves the nServices-based selection to addrman from connman.
Adds HasAllRelevantServices and GetRelevantServices, which check
for NETWORK|WITNESS.
This changes the following:
* Removes nRelevantServices from CConnman, disconnecting it a bit
more from protocol-level logic.
* Replaces our sometimes-connect-to-!WITNESS-nodes logic with
simply always requiring WITNESS|NETWORK for outbound non-feeler
connections (feelers still only require NETWORK).
* This has the added benefit of removing nServicesExpected from
CNode - instead letting net_processing's VERSION message
handling simply check HasAllRelevantServices.
* This implies we believe WITNESS nodes to continue to be a
significant majority of nodes on the network, but also because
we cannot sync properly from !WITNESS nodes, it is strange to
continue using our valuable outbound slots on them.
* In order to prevent this change from preventing connection to
-connect= nodes which have !WITNESS, -connect nodes are now
given the "addnode" flag. This also allows outbound connections
to !NODE_NETWORK nodes for -connect nodes (which was already true
of addnodes).
* Has the (somewhat unintended) consequence of changing one of the
eviction metrics from the same
sometimes-connect-to-!WITNESS-nodes metric to requiring
HasRelevantServices.
This should make NODE_NETWORK_LIMITED much simpler to implement.
Tree-SHA512: 90606896c86cc5da14c77843b16674a6a012065e7b583d76d1c47a18215358abefcbab44ff4fab3fadcd39aa9a42d4740c6dc8874a58033bdfc8ad3fb5c649fc
01b52ce Add comment explaining forced processing of compact blocks (Suhas Daftuar)
08fd822 qa: add test for minchainwork use in acceptblock (Suhas Daftuar)
ce8cd7a Don't process unrequested, low-work blocks (Suhas Daftuar)
Pull request description:
A peer could try to waste our resources by sending us unrequested blocks with
low work (eg to fill up our disk). Since e265200 we no longer request blocks until we
know we're on a chain with more than nMinimumChainWork (our anti-DoS
threshold), but we would still process unrequested blocks that had more work
than our tip (which generally has low-work during IBD), even though we may not
yet have found a headers chain with sufficient work.
Fix this and add a test.
Tree-SHA512: 1a4fb0bbd78054b84683f995c8c3194dd44fa914dc351ae4379c7c1a6f83224f609f8b9c2d9dde28741426c6af008ffffea836d21aa31a5ebaa00f8e0f81229e
eac64bb7a [qa] Test nMinimumChainWork (Suhas Daftuar)
0311836f6 Allow setting nMinimumChainWork on command line (Suhas Daftuar)
Pull request description:
As discussed briefly here: https://botbot.me/freenode/bitcoin-core-dev/2017-02-28/?msg=81712308&page=4
This adds a hidden command line option for setting `nMinimumChainWork`, which allows us to test this parameter in our functional tests, as well as allowing for niche use cases like syncing nodes that are otherwise disconnected from the network.
See also #10345, which proposes a new use of `nMinimumChainWork`.
Tree-SHA512: fe4d8f4f289697615c98d8760f1cc74c076110310ea0b5b875fcab78c127da9195b4eb84148aebacc7606c246e5773d3f13bd5d9559d0a8bffac20a3a28c62df
3f8fa7f Make sure to clean up mapBlockSource if we've already seen the block (Cory Fields)
Pull request description:
Otherwise we may leave them dangling.
Credit TheBlueMatt.
Tree-SHA512: 8be77e08ebfc4f5b206d5ee7cfbe87f92c1eb5bc2b412471993658fe210306789aaf0f3d1454c635508a7d8effede2cf5ac144d622b0157b872733d9661d65c3
176c021 [qa] Test non-atomic chainstate writes (Suhas Daftuar)
d6af06d Dont create pcoinsTip until after ReplayBlocks. (Matt Corallo)
eaca1b7 Random db flush crash simulator (Pieter Wuille)
0580ee0 Adapt memory usage estimation for flushing (Pieter Wuille)
013a56a Non-atomic flushing using the blockchain as replay journal (Pieter Wuille)
b3a279c [MOVEONLY] Move LastCommonAncestor to chain (Pieter Wuille)
Tree-SHA512: 47ccc62303f9075c44d2a914be75bd6969ff881a857a2ff1227f05ec7def6f4c71c46680c5a28cb150c814999526797dc05cf2701fde1369c06169f46eccddee
5a9b508 [trivial] Add end of namespace comments (practicalswift)
Tree-SHA512: 92b0fcae4d1d3f4da9e97569ae84ef2d6e09625a5815cd0e5f0eb6dd2ecba9852fa85c184c5ae9de5117050330ce995e9867b451fa8cd5512169025990541a2b
* Remove ppszTypeName from protocol.cpp and reimplement GetCommand
This removes the need to carefully maintain ppszTypeName, which required
correct order and also did not allow to permanently remove old message
types.
To get the command name for an INV type, GetCommandInternal uses a switch
which needs to be maintained from now on.
The way this is implemented also resembles the way it is implemented in
Bitcoin today, but it's not identical. The original PR that introduced the
switch case in Bitcoin was part of the Segwit changes and thus never got
backported. I decided to implement it in a slightly different way that
avoids throwing exceptions when an unknown INV type is encountered.
IsKnownType will now also leverage GetCommandInternal() to figure out if
the INV type is known locally. This has the side effect of old/legacy
message types to return false from now on. We will depend on this side
effect in later commits when we remove legacy InstantSend code.
* Stop handling/relaying legacy IX messages
When we receive an IX message, we simply treat it as a regular TX and relay
it as such.
We'll however still request IX messages when they are announced to us. We
can't simply revert to requesting TX messages in this case as it might
result in the other peer not answering due to the TX not being in mapRelay
yet. We should at some point in the future completely drop handling of IX
messages instead.
* Remove IsNewInstantSendEnabled() and only use IsInstantSendEnabled()
* Remove legacy InstantSend from GUI
* Remove InstantSend from Bitcoin/Dash URIs
* Remove legacy InstantSend from RPC commands
* Remove legacy InstantSend from wallet
* Remove legacy instantsend.h include
* Remove legacy InstantSend from validation code
* Completely remove remaining legacy InstantSend code
* Remove now unused spork
* Fix InstantSend related test failures
* Remove now obsolete auto IS tests
* Make spork2 and spork3 disabled by default
This should have no influence on mainnet as these sporks are actually set
there. This will however affect regtest, which shouldn't have LLMQ based
InstantSend enabled by default.
* Remove instantsend tests from dip3-deterministicmns.py
These were only testing legacy InstantSend
* Fix .QCheckBox#checkUsePrivateSend styling a bit
* s/TXLEGACYLOCKREQUEST/LEGACYTXLOCKREQUEST/
* Revert "verified via InstantSend" back to "verified via LLMQ based InstantSend"
* Use cmd == nullptr instead of !cmd
* Remove last parameter from AvailableCoins call
This was for fUseInstantSend which is not present anymore since rebase
211adc0 Use range-based for loops (C++11) when looping over vector elements (practicalswift)
Tree-SHA512: 0e007f20dcef99d3c7a1036265e00f689d69f42e02fd82dd8389f45b52d31947e5f9388de2610d3d9bd9f554915ce0d35ebce561e5ae3a9013956d0ee4937145
0f3471f net: make CNode's id private (Cory Fields)
9ff0a51 scripted-diff: net: Use accessor rather than node's id directly (Cory Fields)
e50c33e devtools: add script to verify scriptable changes (Cory Fields)
skipped travis changes
Tree-SHA512: a0ff50f4e1d38a2b63109b4996546c91b3e02e00d92c0bf04f48792948f78b1f6d9227a15d25c823fd4723a0277fc6a32c2c1287c7abbb7e50fd82ffb0f8d994
pnode->id to pnode->GetId()
Signed-off-by: Pasta <pasta@dashboost.org>
* Generalize CBLSLazyWrapper so that it can be used of pubkeys and secret keys
* Implement == and != operators for CBLSLazyWrapper
* Implement cached hash for CBLSLazyWrapper
* Use CBLSLazyPublicKey for CDeterministicMNState::pubKeyOperator
* Speed up GetProjectedMNPayees by sorting the MN list by last paid
Instead of updating a temporary list for each projected height and calling
GetMNPayee() on it.
* Cache intermediate lists in GetListForBlock
This avoids re-loading and applying diffs again and again.
* Only update masternode list UI max once every 3 seconds
This avoids updating the UI on every block, which turned out to be very
expensive.
* Fix compilation
* Drop time restrictions for mn list update in ClientModel
They are fully handled by MasternodeList now.
b1a6d4c Take a CTransactionRef in AddToWalletIfInvolvingMe to avoid a copy (Matt Corallo)
1c95e2f Use std::shared_ptr instead of boost::shared_ptr in ScriptForMining (Matt Corallo)
91f1e6c Remove dead-code tracking of requests for blocks we generated (Matt Corallo)
acad82f Add override to functions using CValidationInterface methods (Matt Corallo)
e6d5e6c Hold cs_wallet for whole block [dis]connection processing (Matt Corallo)
461e49f SyncTransaction->TxAddedToMempool/BlockConnected/Disconnected (Matt Corallo)
f404334 Handle SyncTransaction in ActivateBestChain instead of ConnectTrace (Matt Corallo)
a147687 Keep conflictedTxs in ConnectTrace per-block (Matt Corallo)
d3167ba Handle conflicted transactions directly in ConnectTrace (Matt Corallo)
29e6e23 Make ConnectTrace::blocksConnected private, hide behind accessors (Matt Corallo)
822000c Add pblock to connectTrace at the end of ConnectTip, not start (Matt Corallo)
f5e9a01 Include missing #include in zmqnotificationinterface.h (Matt Corallo)
Tree-SHA512: 8893d47559da3b28d2ef7359768547cba8a4b43b6f891d80f5848f995a84b1517bfb0f706fdc8cd43f09a1350349eb440d9724a59363ab517dfcc4fcb31b2018
* Tighten rules for dstx
* Tighten rules for dsvin
* NULL -> nullptr
* Make `ConsumeCollateral()` a private function instead of a lamda and reuse it in `Charge*Fees()`
* Make sure inputs and outputs are of the same size
Introduces new response ERR_SIZE_MISMATCH, old clients will simply bail out.
* Drop now redundant vecTxOut.size() check
* Check max inputs size
* Fix log category
* Contains dashification. disables `-debug dash`
Merge #9424: Change LogAcceptCategory to use uint32_t rather than sets of strings.
6b3bb3d Change LogAcceptCategory to use uint32_t rather than sets of strings. (Gregory Maxwell)
Tree-SHA512: ebb5bcf9a7d00a32dd1390b727ff4d29330a038423611da01268d8e1d2c0229e52a1098e751d4e6db73ef4ae862e1e96d38249883fcaf12b68f55ebb01035b34
Signed-off-by: Pasta <Pasta@dash.org>
31 -> 32
Signed-off-by: Pasta <Pasta@dash.org>
* Merge #10123: Allow debug logs to be excluded from specified component
3bde556 Add -debugexclude option to switch off logging for specified components (John Newbery)
Tree-SHA512: 30202e3f2085fc2fc5dd4bedb92988f4cb162c612a42cf8f6395a7da326f34975ddc347f82bc4ddca6c84c438dc0cc6e87869f90c7ff88105dbeaa52a947fa43
* bump to uint64_t due to added Dash codes
Signed-off-by: Pasta <Pasta@dash.org>
* bump to uint64_t due to added Dash codes cont.
Signed-off-by: Pasta <Pasta@dash.org>
* string -> BCLog format
Signed-off-by: Pasta <Pasta@dash.org>
* uint32_t -> uint64_t
Signed-off-by: Pasta <Pasta@dash.org>
* Fix CBatchedLogger
* Fix most fDebug-s
* Fix `debug` rpc
* Fix BENCH and RAND conflicts
* Add ALERT and use it
* Update LogPrint-s in dash-specific code
* Tweak few log categories
Specifically:
- use PRIVATESEND in `CPrivateSendClientManager::GetRandomNotUsedMasternode()`
- use ZMQ in `CZMQPublishRawGovernanceVoteNotifier::NotifyGovernanceVote()` and `CZMQPublishRawGovernanceObjectNotifier::NotifyGovernanceObject()`
* Drop no longer used MASTERNODE category
* Merge #10153: logging: Fix off-by-one for shrinkdebugfile default
faab624 logging: Fix off-by-one for shrinkdebugfile (MarcoFalke)
Tree-SHA512: d6153e06067906172ff0611af9e585a3ecf0a7d56925b6ad7c12e75aa802441047059b9b6f6c78e79916c3f2abc8f1998bfd2d5b84201ec6421f727c08da3c21
* Shift dash-specific log categories to start from `1ul << 32` to avoid potential future conflicts with bitcoin ones
* Fix `dash` category
* remove debugCategories
Signed-off-by: Pasta <Pasta@dash.org>
* Prepend "std::" to find call
* Check for BCLog::PRIVATESEND instead of logCategories != BCLog::NONE
* Use BCLog::MNPAYMENTS category instead of checking for logCategories != BCLog::NONE
* Move "End Dash" comment below "ALERT"
When adding new entries here, we'll otherwise get confused with ordering
and might end up forgetting that adding something Dash specific must
continue with the bit after 43.
* rename instantx.* file to instantsend.*
* Reorganize Dash Specific code into folders
add to privatesend folder the header files
Signed-off-by: Pasta <Pasta@dash.org>
add "masternode" to imports
Signed-off-by: Pasta <Pasta@dash.org>
add "masternode" to imports
Signed-off-by: Pasta <Pasta@dash.org>
add "masternode" to imports pt 2
Signed-off-by: Pasta <Pasta@dash.org>
* Remove code for QDEBUGSTATUS propagation
This turned out to be too expensive and could easily take the network
down by bringing all nodes to 100% CPU usage. Better to fully remove this
functionality.
* Apply suggestions from code review
Co-Authored-By: codablock <ablock84@gmail.com>
* Update src/rpc/rpcquorums.cpp
Co-Authored-By: codablock <ablock84@gmail.com>
Observed on testnet that MNs tend to re-request the same objects multiple
times when load becomes high, which results in the same objects being
received multiple times.
