in 6148, I broke the functionality where a peer must opt in / opt out of DSQUEUE messages. This was mostly ok, and not immediately detected, as with this bug, simply everyone would receive DSQ messages over inventory (or classically, old proto versions were not affected by this bug). But this still would result in quite a bit of wasted bandwidth for peers which may not care about DSQ at all.
This commit should restore the prior functionality, where a node should send the SENDDSQUEUE message if they wish to receive DSQs. Once they've sent that, depending on their protocol version, they will either have the messages pushed to them as available, or on modern protocols, they will thereafter receive DSQs over the inventory system.
NOTE: I also refactor the code in this commit, moving some network proccessing into.... wait for it... net_processing.cpp! This allowed us to remove some dependencies in coinjoin.h. DSQ messages are now relayed to peers by calling peer_manager.RelayDSQ
86e92c376a refactor: drop unused CConnman from CSigningManager (Konstantin Akimov)
4668db60a2 refactor: create helper function RelayRecoveredSig inside peerman (pasta)
Pull request description:
## Issue being fixed or feature implemented
High m_nodes_mutex lock contention during high load
## What was done?
this commit should have a few benefits:
1. previous logic using ForEachNode results in locking m_nodes_mutex, a highly contended RecursiveMutex, AND m_peer_mutex(in GetPeerRef)
2. prior also resulted in calling .find over the m_peer_map for each node. Basically old logic was (probably) O(n(nlogn) the new logic results in acquiring m_peer_mutex once and looping over the list of peers, (probably) O(n)
3. Moves networking logic out of llmq/ and into actual net_processing.cpp
## How Has This Been Tested?
Hasn't really yet; it builds, but I need to run tests / maybe deploy to testnet mn
## Breaking Changes
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
knst:
utACK 86e92c376a
UdjinM6:
utACK 86e92c376a
Tree-SHA512: ca9d6ac22f8b72b117188147044c499ae62722283c6291633067b99726e6a6abc52e5c8cf3bdcd0d8fed0ad8d9086b000f628c9a932dfe89153e912b563eda5a
3ba602672c refactor: use self.wait_until in all the dash specific "wait_until_x" logic in order to actually apply the timeout scaling settings (pasta)
Pull request description:
## Issue being fixed or feature implemented
Currently we use the raw helper, but that means that timeout scaling isn't applying.. I think this may be a cause of a lot of the functional test failures that we see in tsan / ubsan.
## What was done?
## How Has This Been Tested?
hasn't; wait for CI
## Breaking Changes
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
knst:
utACK 3ba602672c
UdjinM6:
utACK 3ba602672c
Tree-SHA512: 935498f4b296b1abcac8be686cce396b61b654ef62da46de9a23a0f24ad31254f4938a581a6a4e2533576db0e0120861fd690bd9019e893b30990f21d1e48168
**This does change the logic!** We no longer prioritize asking MNs. This is probably fine? I don't specifically recall why we wanted to ask MNs besides potentially that they may be higher performing or better connected? We can potentially restore this logic once we bring masternode connection logic into Peer
Does also change logic, by short-circuiting once peersToAsk is full.
This commit has the added benefit of reducing contention on m_nodes_mutex due to no-longer calling connman.ForEachNode not once but twice
This may slightly increase contention on m_peer_mutex; but that should be an ok tradeoff for not only removing dependencies, but also reducing contention on a much more contested RecursiveMutex
3d67771f89 refactor: add in quorumBaseBlockIndexCache to reduce cs_main contention (pasta)
Pull request description:
## Issue being fixed or feature implemented
subset of https://github.com/dashpay/dash/pull/6418; only includes the new quorumBaseBlockIndexCache, doesn't include the caching of the chain-tip, as that introduced regressions I'm still debugging.
## What was done?
introduce a LRU cache for quorumHash -> const CBlockIndex*; this should significantly reduce cs_main contention during high transaction load.
## How Has This Been Tested?
Ran tests locally; let's see CI happy, and I also intend to run this on a testnet MN first and see the level of contention reduction
## Breaking Changes
None
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
UdjinM6:
utACK 3d67771f89
knst:
utACK 3d67771f89
Tree-SHA512: dbb4bdafed095397ca0e12dbd8bba25c108d199538387c71b1ff4285af821f9d9ad0ad4426407a015528270f3c163fa66ce91755efb1c8a7a90fd7cb70a918bc
this commit should have a few benefits:
1. previous logic using ForEachNode results in locking m_nodes_mutex, a highly contended RecursiveMutex, AND m_peer_mutex(in GetPeerRef)
2. prior also resulted in calling .find over the m_peer_map for each node. Basically old logic was (probably) O(n(nlogn) the new logic results in acquiring m_peer_mutex once and looping over the list of peers, (probably) O(n)
3. Moves networking logic out of llmq/ and into actual net_processing.cpp
b65f0bab7f refactor: move expensive CInv initialization out of hot loop (pasta)
Pull request description:
## Issue being fixed or feature implemented
Not sure how we introduced this one, but it appears we are calling a CInv construction (pretty cheap) and a GetDSTX ((relatively) EXPENSIVE) for each peer we are connected to in RelayTransaction... I can hope and pray that the compiler somehow was magically optimizing this for us, but I really doubt it
## What was done?
Move the initialization out of the loop
## How Has This Been Tested?
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
knst:
utACK b65f0bab7f
UdjinM6:
utACK b65f0bab7f
kwvg:
utACK b65f0bab7f
Tree-SHA512: f556789042ab9265d5c4d87c3ba2910138ce43ffa69c90ed208c9a3bcd861343f201ce2f00aeb541f345c9ca686dac7227df8d4833cf7fbdf61c36260f627864
31243ca313 chore: update nMinimumChainWork, defaultAssumeValid, checkpointData, chainTxData for mainnet and testnet (pasta)
Pull request description:
## Issue being fixed or feature implemented
Bump chainparams in prep for v22
## What was done?
## How Has This Been Tested?
## Breaking Changes
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
UdjinM6:
utACK 31243ca313
Tree-SHA512: 6a01bbbaefb69437e053340b968e0ce68e2bd9e5e5fd2900864d88ffc21c5cd3f9c2a50d8f5b19f16515cd4c62a4a8fc2d981dd6d0456e86ef96f40e350e786a
a34caecdff refactor: replace infinite loop with break to while in forcemninfo (Konstantin Akimov)
4ec385d020 refactor: simplify loop of node connection in test_framework (Konstantin Akimov)
fa4ba4d169 refactor: moved functions do_connect, remove_masternode closer to theirs usages (Konstantin Akimov)
8eb5d852c7 refactor: drop dead constant SPECIALTX_TYPE from quorum commitment (Konstantin Akimov)
Pull request description:
## What was done?
see each commit
## How Has This Been Tested?
Run unit and functional tests
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
UdjinM6:
utACK a34caecdff
Tree-SHA512: d998839c992ab1ff36c88ef10236b2a7811b2edb81a9211a659065bf06bebc2f9174ef3268b1508b489a7b9b29a3f3e0b9fc096ff055cce6c4cacbfaf5877cd2
397a157e8d refactor: introduce cs_pendingSigns (pasta)
Pull request description:
## Issue being fixed or feature implemented
Much more minimal version of https://github.com/dashpay/dash/pull/6410; data from testnet while MNs were doing lots of instantsend locking showed minimal contention over signing_shares cs, however, the contention that did exist, most was a result of AsyncSign conflicting with something else. This should resolve the vast majority of contention in signing_shares while minimizing the complication / risk of introducing dead locks compared to 6410
## What was done?
introduce cs_pendingSigns
## How Has This Been Tested?
Built locally
## Breaking Changes
None
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
knst:
utACK 397a157e8d
UdjinM6:
utACK 397a157e8d
Tree-SHA512: aac741c274449cf9c22625ac95e964d275dd1454647eaa2fc064c90b44a215e5509c9f9b8aceccbd49002edc21cbea883900f202204aa2138c104f4989ae5e26
c48efdac83 fix(qt): avoid potential precision loss in amounts on Governance Tab (UdjinM6)
Pull request description:
## Issue being fixed or feature implemented
https://github.com/dashpay/dash/pull/6403#discussion_r1847834174
## What was done?
## How Has This Been Tested?
## Breaking Changes
## Checklist:
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [ ] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
knst:
utACK c48efdac83
PastaPastaPasta:
utACK c48efdac83
Tree-SHA512: 705bc154e10150e32e3ff04d4a90de14e29ffd195cc9f94a753dd1b4588f2730d6526cd66d851005b956e0d074d39250d142baa946bea8df95dbc92718931762
5078baea2b fix(test): wait for chainlock before mining a block we expect to include said chainlock (pasta)
f39c1e6f4c fix: guard m_can_tx_relay behind m_tx_relay_mutex; make it private; add additional annotations (pasta)
Pull request description:
## Issue being fixed or feature implemented
See each commit; fixes two bugs, both discovered while running feature_llmq_chainlocks.py with tsan / debug.
one a datarace in net_processing.cpp and the other in the test I was using to ensure this fix was correct, feature_llmq_chainlocks
## What was done?
### net_processing.cpp
You can see the datarace here: https://gist.github.com/PastaPastaPasta/c966a9f805758b34524085e3d52ea7f8
We simply guard it with an existing mutex that is always locked in close proximity.
### feature_llmq_chainlocks.py
Most of the time, while generating the cycle quorum, there is sufficient time to generate a chainlock; however, this is racey, and I've observed locally where the block gets generated before a chainlock is present and as such `test_coinbase_best_cl` fails. We should instead wait for the chainlock first, and then mine the block. This was we can ensure the mined block will include that chainlock.
This was observed locally maybe 1/10 times or so
## How Has This Been Tested?
ran feature_llmq_chainlocks.py ~40 times locally with tsan / debug
## Breaking Changes
None
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
kwvg:
utACK 5078baea2b
knst:
utACK 5078baea2b
UdjinM6:
utACK 5078baea2b
Tree-SHA512: b346fc60809df72d0161f625073dce7062bd2641d35e4f80160fac9afeec63707de552e2856940ac2604875908ae3b98a225d352de36bfbfc6ee3fbe1e1538ff
5741d5da28 chore: update seeds (Kittywhiskers Van Gogh)
2d732fc66e chore: drop defunct onion seeds, add new existing onion hosts as seeds (pasta)
Pull request description:
## Issue being fixed or feature implemented
Update onion seeds for upcoming version
## What was done?
Update onion seeds to reflect latest status
## How Has This Been Tested?
As of today, all of these are able to be connected to on port 9999; I've not actually connected to all of them and verified they're on latest core or something like that; but their addresses my core knows about, and are able to be trivially connected to
## Breaking Changes
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
kwvg:
utACK 5741d5da28
UdjinM6:
utACK 5741d5da28
Tree-SHA512: 541bcc510b2ebf6de08ac91f2b7e5b1c910536dca9bab0b38930d6e5cfa1cd05e8c014ba4c74c14c43ed211cda3275fffb5baaf1489bbd1221567130d804b0ec
d728879329 chore: bump develop to be 22.1.0 (pasta)
Pull request description:
## Issue being fixed or feature implemented
Develop is now for 22.1
## What was done?
## How Has This Been Tested?
## Breaking Changes
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
kwvg:
utACK d728879329
UdjinM6:
utACK d728879329
Tree-SHA512: 3ce6d37c43925c59a581cb7d518279c23cbf412b5511a3fa46dc1dd479da8a5d6bd3c40615f88e3db5bd6452ad3d9246901057055db91e3387e9d7c452e5c730
0072318421 refactor: drop unused parameter mninfos_valid in test framework mine_cycle_quorum (pasta)
Pull request description:
## Issue being fixed or feature implemented
This param is unused
## What was done?
Removed
## How Has This Been Tested?
see ci
## Breaking Changes
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
knst:
utACK 0072318421
UdjinM6:
utACK 0072318421
Tree-SHA512: e66e2b23b95b98e843fdb1d58419ba0798e8ba4115ef6c52e2e4681b04ea3e43efe73cee0ac96758560914567f822d107169ca0d6212210e07cc81b317a3e2eb
As of today, all of these are able to be connected to on port 9999; I've not actually connected to all of them and verified they're on latest core or something like that; but their addresses my core knows about, and are able to be trivially connected to
3f2e064b18 refactor: set `const auto& cbTx` to avoid using optional throughout method (pasta)
0c0d91e491 fix: functional test feature_llmq_chainlocks.py should activate MN_RR instead v20 (Konstantin Akimov)
af93e877f2 refactor: removed pre-MN_RR logic of validation of CL (Konstantin Akimov)
Pull request description:
## Issue being fixed or feature implemented
The fork MN_RR is active awhile on testnet and mainnet and no more need legacy check
## What was done?
Removes legacy version of checks for CL and related functional tests
## How Has This Been Tested?
Run unit / functional test - DONE
Reindex testnet - DONE
Reindex mainnet - DONE
## Breaking Changes
Removed pre-mn_rr version of checks for CL.
It's no more relevant on mainnet and testnet.
It affects behavior on new devnets and regtest for pre-mn_rr activation blocks.
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e tests
- [x] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
UdjinM6:
utACK 3f2e064b18
Tree-SHA512: 8b4b3a20a54602f4df9d98e17c79004214493b15c0bce9c08c68d667a5cba86b817947f008d646c48ef9f2f86676c02085c7d0ed36e83548ef5425b64faffb89
Most of the time, while generating the cycle quorum, there is sufficient time to generate a chainlock; however, this is racey, and I've observed locally where the block gets generated before a chainlock is present and as such `test_coinbase_best_cl` fails. We should instead wait for the chainlock first, and then mine the block. This was we can ensure the mined block will include that chainlock.
This was observed locally maybe 1/10 times or so
Originally introduced in pr 6365; this datarace was discovered using tsan locally, and running feature_llmq_chainlocks 5 times. 1 out of 5 times failed
ba7e500062 refactor: more of 24306 (GetPathArg in Dash-specific code) (UdjinM6)
562e3f7b18 Merge bitcoin/bitcoin#24371: util: Fix ReadBinaryFile reading beyond maxsize (MarcoFalke)
3383b79049 Merge bitcoin/bitcoin#24312: addrman: Log too low compat value (MarcoFalke)
d96983a327 Merge bitcoin/bitcoin#24306: util: Make ArgsManager::GetPathArg more widely usable (MarcoFalke)
Pull request description:
## Issue being fixed or feature implemented
A few simple backports
## What was done?
## How Has This Been Tested?
Built locally; see CI
## Breaking Changes
None
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
UdjinM6:
utACK ba7e500062
kwvg:
utACK ba7e500062
Tree-SHA512: 42f41cb04945af15bfa4581ae53354eec88d4cdec44ab7c0100edcfe2fa71a95e05073e76b8d3935bd05cd5ee1f5cd5b34d2ed99e7eb551475abdc77f1ffff7c
2f18c1af30 ci: deduplicate depends building (pasta)
Pull request description:
## Issue being fixed or feature implemented
Currently we build the same host / options multiple times. Don't do this!
## What was done?
Now building depends only twice
## How Has This Been Tested?
Local CI appears to be working
## Breaking Changes
None
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
kwvg:
utACK 2f18c1af30
UdjinM6:
utACK 2f18c1af30
Tree-SHA512: 67460508a2e9458152f7c8bb5f4a1a786aedcfded0e5c54fb03d85010ba8bb87362b66a0c322b51aeba75752e36418fc235d8dc4197ef10695e234ccc5a00a39
