Commit Graph

26553 Commits

Author SHA1 Message Date
Konstantin Akimov
4d9837c21e
refactor: add a new flag disable_mocktime to set_test_params()
It's better than re-implement setup_nodes each time when you want just disable mocktime.
It seems more error prune and the code is shorter
2024-09-01 18:27:19 +07:00
pasta
7a9e475c26
Merge #6231: backport: merge bitcoin#19572, #20953, #20523, #21008, #21310, #22079, #23471, #24218 (zmq backports)
b75e83b298 merge bitcoin#24218: Fix implicit-integer-sign-change (Kittywhiskers Van Gogh)
8ecc22f51f merge bitcoin#23471: Improve ZMQ documentation (Kittywhiskers Van Gogh)
2965093c4a merge bitcoin#22079: Add support to listen on IPv6 addresses (Kittywhiskers Van Gogh)
3ac3714957 merge bitcoin#21310: fix sync-up by matching notification to generated block (Kittywhiskers Van Gogh)
7b0c725c59 merge bitcoin#21008: fix zmq test flakiness, improve speed (Kittywhiskers Van Gogh)
5e87efd04b merge bitcoin#20523: deduplicate 'sequence' publisher message creation/sending (Kittywhiskers Van Gogh)
99c730f0f3 merge bitcoin#20953: dedup zmq test setup code (node restart, topics subscription) (Kittywhiskers Van Gogh)
982c1f03d4 merge bitcoin#19572: Create "sequence" notifier, enabling client-side mempool tracking (Kittywhiskers Van Gogh)
b0b4e0fa7f zmq: Make `g_zmq_notification_interface` a smart pointer (Kittywhiskers Van Gogh)
0a1ffd30b9 zmq: extend appending address to log msg for Dash-specific notifications (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * [bitcoin#19572](https://github.com/bitcoin/bitcoin/pull/19572) introduces tests in `interface_zmq.py` that validate newly introduced "sequence" reporting of all message types (`C`, `D`, `R` and `A`). The `R` message type (removed from mempool) is tested by leveraging the RBF mechanism, which isn't present in Dash.

    In order to allow the tests to successfully pass, all fee bumping and RBF-specific code had to be removed from the tests. This test also involves creating a new block to test the `C` message (connected block) that would return a `time-too-new` error and fail unless mocktime has been disabled (which has been done in this test).

  * When backporting [bitcoin#18309](https://github.com/bitcoin/bitcoin/pull/18309) ([dash#5728](https://github.com/dashpay/dash/pull/5728)), Dash-specific ZMQ notifications did not have those changes applied to them and that particular backport was merged upstream *after* [bitcoin#19572](https://github.com/bitcoin/bitcoin/pull/19572), meaning, for completion, the changes from [bitcoin#18309](https://github.com/bitcoin/bitcoin/pull/18309) have been integrated into [bitcoin#19572](https://github.com/bitcoin/bitcoin/pull/19572)'s backport.

    As for the Dash-specific notifications, they have been covered to ensure uniformity in a separate commit.

  * The ZMQ notification interface has been converted to a smart pointer in the interest of safety (this change is eventually done upstream in 8ed4ff8e05d61a8e954d72cebdc2e1d1ab24fb84 ([bitcoin#27125](https://github.com/bitcoin/bitcoin/pull/27125)))

  ## Breaking Changes

  None expected.

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  UdjinM6:
    utACK b75e83b298
  PastaPastaPasta:
    utACK b75e83b298
  knst:
    utACK b75e83b298

Tree-SHA512: 9f860d1203bebe0914a5102f101f646873d14754830d651fb91ed0d1285a6c1a58ffc492b07d4768324d94f53171c9a4da974cf4a0b1e5c665979eace289f6f0
2024-08-30 22:18:37 -05:00
pasta
630254e00c
Merge #6238: refactor: drop some unneeded this and const
5aefd44ea3 fmt: run clang-format (pasta)
7e9dec290f refactor: drop some unneeded `this` and `const` (pasta)

Pull request description:

  ## Issue being fixed or feature implemented
  Neither identifies are used for anything, the const is always gonna be discarded as it's on a value return value, and the `this` is not used, so should be dropped.

  This fixes some spammy warnings for me locally

  ## What was done?

  ## How Has This Been Tested?
  Compiled in CI

  ## Breaking Changes
  None

  ## Checklist:
    _Go over all the following points, and put an `x` in all the boxes that apply._
  - [ ] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  knst:
    utACK 5aefd44ea3
  UdjinM6:
    utACK 5aefd44ea3

Tree-SHA512: 9ebfae9bccc8893dc79c25d2b824680f63d27ab2da82089c96660eacb5891577e00145f6b70a20b161fc6fda1b6e5cfd6897502ffd356f8febcec76844623613
2024-08-30 22:13:05 -05:00
pasta
d16e95352c
Merge #6233: fix: release unused memory in CNetMsgMaker::Make()
c2c4b2b794 fix: release unused memory in `CNetMsgMaker::Make()` (UdjinM6)

Pull request description:

  ## Issue being fixed or feature implemented
  We reserve capacity for large messages in `CNetMsgMaker::Make()` but most messages are small yet we never release unused memory here. Discovered while debugging 28165 backport issues.

  ## What was done?

  ## How Has This Been Tested?

  ## Breaking Changes
  n/a

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have assigned this pull request to a milestone

ACKs for top commit:
  kwvg:
    ACK c2c4b2b794
  PastaPastaPasta:
    utACK c2c4b2b794
  knst:
    utACK c2c4b2b794

Tree-SHA512: 72a3728e316fb76ca135fbfff4f15b80c60b048d7e916d2f5dbcde4b64dce7177af80f8f24153636ccfab2f5d5c7d9edc5b3bdba121f64c8da03117a98fd7411
2024-08-30 22:08:23 -05:00
pasta
cddbc2a87b
Merge #6240: backport: partial bitcoin#27920, #28799, #29007 - to improve CI experience
2eadcf2f68 partial Merge bitcoin/bitcoin#29007: test: create deterministic addrman in the functional tests (stratospher)
f95ca4ed3e fix: unify with bitcoin: removed requirement of txindex=0 (Konstantin Akimov)
bf46e7a8e1 partial Merge bitcoin/bitcoin#28799: wallet: cache descriptor ID to avoid repeated descriptor string creation (Andrew Chow)
05e5966199 partial Merge bitcoin/bitcoin#27920: wallet: bugfix, always use apostrophe for spkm descriptor ID (furszy)

Pull request description:

  ## Issue being fixed or feature implemented
  Lately our CI for tsan is flapping many functional tests and take long times.

  This PR has several important changes backported from the latest bitcoin's version to improve CI experience

  ## What was done?
  This PR has several backports that improved CI experience drastically.

  **Firstly, it aims to fix flapping test p2p_node_network_limited.py**

  For example: https://gitlab.com/dashpay/dash/-/jobs/7692635307

  <details>

  <summary>p2p_node_network_limited.py                        | ✖ Failed  | 28 s</summary>

  ```
   test  2024-08-29T02:50:53.929000Z TestFramework (ERROR): Assertion failed
  Traceback (most recent call last):
  File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/test_framework/test_framework.py", line 158, in main
    self.run_test()
  File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/p2p_node_network_limited.py", line 79, in run_test
    self.nodes[0].disconnect_p2ps()
  File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/test_framework/test_node.py", line 611, in disconnect_p2ps
    wait_until_helper(check_peers, timeout=5)
  File "/builds/dashpay/dash/build-ci/dashcore-linux64_tsan/test/functional/test_framework/util.py", line 262, in wait_until_helper
    raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout))
    AssertionError: Predicate ''''
         def check_peers():
             for p in self.getpeerinfo():
                 for p2p in self.p2ps:
                     if p['subver'] == p2p.strSubVer:
                         return False
             return True
  ''' not true after 5.0 seconds
  ```
  </details>

  **Secondly, it improves performance of Descriptor wallets significantly for case of `tsan` CI**. It is tiny improvement for Release build and local runs, but some fucnctional tests run as fast as twice:

  https://gitlab.com/dashpay/dash/-/jobs/7694458953
  https://gitlab.com/dashpay/dash/-/jobs/7665132625

  ```
  wallet_create_tx.py --descriptors                  | ✓ Passed  | 236 s <-- new version
  wallet_create_tx.py --legacy-wallet                | ✓ Passed  | 108 s
  wallet_basic.py --descriptors                      | ✓ Passed  | 135 s <---- new version
  wallet_basic.py --legacy-wallet                    | ✓ Passed  | 97 s

  wallet_create_tx.py --descriptors                  | ✓ Passed  | 456 s <-- old version
  wallet_create_tx.py --legacy-wallet                | ✓ Passed  | 98 s
  wallet_basic.py --descriptors                      | ✓ Passed  | 189 s <--- old version
  wallet_basic.py --legacy-wallet                    | ✓ Passed  | 131 s
  ```
  See performance investigation here: https://github.com/dashpay/dash/pull/6226

  ## How Has This Been Tested?
  Run unit/functional tests

  ## Breaking Changes
  N/A

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone

ACKs for top commit:
  PastaPastaPasta:
    utACK 2eadcf2f68
  UdjinM6:
    utACK 2eadcf2f68

Tree-SHA512: 127fbaa65c160aa95e2145a6b40d3811f7c42e36fbee9ce98a9ac021abd9cbe6edc7791870b331a54855ba891e3804885db7936ef212647b693f50f79a60d232
2024-08-30 10:32:20 -05:00
pasta
e3a5afd6a6
Merge #6064: backport: Merge bitcoin#22568, 21800, (partial)22707
bda74458b5 (partial) Merge bitcoin/bitcoin#22707: test: refactor use of getrawmempool in functional tests for efficiency (MarcoFalke)
1a8268770c Merge bitcoin/bitcoin#21800: mempool/validation: mempool ancestor/descendant limits for packages (fanquake)
97fd2b2842 Merge bitcoin/bitcoin#22568: test: add addr-fetch peer connection state and timeout coverage (MarcoFalke)

Pull request description:

  bitcoin backports

ACKs for top commit:
  UdjinM6:
    utACK bda74458b5
  PastaPastaPasta:
    utACK bda74458b5

Tree-SHA512: a001f790b36df8e6c640c383a302ce47a6add63f39561596a670fd8261b9a461d71b31b6457a78706a2f596134705879f5f4ba142a79ee587eba02302f57be71
2024-08-29 12:44:50 -05:00
stratospher
2eadcf2f68
partial Merge bitcoin/bitcoin#29007: test: create deterministic addrman in the functional tests
BACKPORT NOTICE
It includes only this commit: be25ac3092b7755e26e1ec6c33a27cd0e3dd9eac

[init] Remove -addrmantest command line arg

-addrmantest is only used in `p2p_node_network_limited.py` test to
test if the node self-advertises a hard-coded local address
(which wouldn't be advertised in the tests because it's unroutable
without the test-only code path) to check pruning-related services
are correct in that addr.

Remove -addrmantest because the self advertisement happens because
of hard coded test path logic, and expected services are nominal
due to how easily the test-only code could diverge from mainnet
logic. It's also being used only in 1 test.
2024-08-29 17:23:40 +07:00
Konstantin Akimov
f95ca4ed3e
fix: unify with bitcoin: removed requirement of txindex=0 2024-08-29 17:23:40 +07:00
Andrew Chow
bf46e7a8e1
partial Merge bitcoin/bitcoin#28799: wallet: cache descriptor ID to avoid repeated descriptor string creation
BACKPORT NOTICE:
It doesn't include changes related to wallet_miniscript.py

5e6bc6d830664a5afeb5d5bd7e7b3818a01376b7 test: remove custom rpc timeout for `wallet_miniscript.py`, reorder in test_runner (Sebastian Falbesoner)
f811a24421fe102a96ab8f75427cc6a3c5503dc3 wallet: cache descriptor ID to avoid repeated descriptor string creation (Sebastian Falbesoner)

Pull request description:

  Right now a wallet descriptor is converted to its string representation (via `Descriptor::ToString`) repeatedly at different instances:
  - on finding a `DescriptorScriptPubKeyMan` for a given descriptor (`CWallet::GetDescriptorScriptPubKeyMan`, e.g. used by the `importdescriptors` RPC); the string representation is created once for each spkm in the wallet and at each iteration again for the searched descriptor (`DescriptorScriptPubKeyMan::HasWalletDescriptor`)
  - whenever `DescriptorScriptPubKeyMan::GetID()` is called, e.g. in `TopUp` or any instances where a descriptor is written to the DB to determine the database key, also at less obvious places like `FastWalletRescanFilter` etc.

  As there is no good reason to calculate a fixed descriptor's string/ID more than once, add the ID as a field to `WalletDescriptor` and calculate it immediately at initialization (or deserialization). `HasWalletDescriptor` is changed to compare the spkm's and searched descriptor's ID instead of the string to take use of that.

  This speeds up the functional test `wallet_miniscript.py` by a factor of 5-6x on my machine (3m30.95s on master vs. 0m38.02s on PR). The recently introduced "max-size TapMiniscript" test-case introduced a descriptor that takes 2-3 seconds to create a string representation, so the repeated calls to that were significantly hurting the performance.

  Fixes https://github.com/bitcoin/bitcoin/issues/28800.

ACKs for top commit:
  Sjors:
    ACK 5e6bc6d830664a5afeb5d5bd7e7b3818a01376b7
  S3RK:
    Code Review ACK 5e6bc6d830664a5afeb5d5bd7e7b3818a01376b7
  achow101:
    ACK 5e6bc6d830664a5afeb5d5bd7e7b3818a01376b7
  BrandonOdiwuor:
    ACK 5e6bc6d830664a5afeb5d5bd7e7b3818a01376b7

Tree-SHA512: 98b43963a5dde6055bb26cecd3b878dadd837d6226af4c84142383310495da80b3c4bd552e73b9107f2f2ff1c11f5e18060c6fd3d9e44bbd5224114c4d245c1c
2024-08-29 17:23:40 +07:00
furszy
05e5966199
partial Merge bitcoin/bitcoin#27920: wallet: bugfix, always use apostrophe for spkm descriptor ID
BACKPORT NOTICE

It includes only this commit: 97a965d98f1582ea1b1377bd258c9088380e1b8b
refactor: extract descriptor ID calculation from spkm GetID()

This allows us to verify the descriptor ID on the descriptors
unit tests in different software versions without requiring to
use the entire DescriptorScriptPubKeyMan machinery.

Note:
The unit test changes are introduced after the bugfix commit
but this commit + the unit test commit can be cherry-picked
on top of the v25 branch to verify IDs correctness. IDs must
be the same for v25 and after the bugfix commit.
2024-08-29 17:23:40 +07:00
MarcoFalke
bda74458b5
(partial) Merge bitcoin/bitcoin#22707: test: refactor use of getrawmempool in functional tests for efficiency
47c48b5f35b4910fcf87caa6e37407e67d879c80 test: only use verbose for getrawmempool when necessary in functional tests (Michael Dietz)
77349713b189e80f2c140db4df50177353a1cb83 test: use getmempoolentry instead of getrawmempool in functional tests when appropriate (Michael Dietz)
86dbd54ae8a8f9c693c0ea67114bbff24a0754df test: improve mempool_updatefrom efficiency by using getmempoolentry for specific txns (Michael Dietz)

Pull request description:

  I don't think this changes the intention of the test. But it does shave ~30 seconds off the time it takes to run. From what I've seen our CI `macOS 11 native [gui] [no depends]` runs `mempool_updatefrom.py` in ~135 seconds. After this PR it should run in ~105 seconds

  I noticed this improvement should probably be made when testing performance/runtimes of https://github.com/bitcoin/bitcoin/pull/22698. But I wanted to separate this out from that PR so the affects of each is decoupled

  Edit: The major change in this PR is improving mempool_updatefrom.py's runtime as this is a very long running test. Then made the same efficiency improvements across all the functional tests as it made since to do that here

ACKs for top commit:
  theStack:
    Tested ACK 47c48b5f35b4910fcf87caa6e37407e67d879c80

Tree-SHA512: 40f553715f3d4649dc18c2738554eafaca9ea800c4b028c099217896cc1c466ff457ae814d59cf8564c782a8964d8fac3eda60c1b6ffb08bbee1439b2d34434b
Signed-off-by: Vijay <vijaydas.mp@gmail.com>
2024-08-29 10:01:34 +05:30
fanquake
1a8268770c
Merge bitcoin/bitcoin#21800: mempool/validation: mempool ancestor/descendant limits for packages
accf3d5868460b4b14ab607fd66ac985b086fbb3 [test] mempool package ancestor/descendant limits (glozow)
2b6b26e57c24d2f0abd442c1c33098e3121572ce [test] parameterizable fee for make_chain and create_child_with_parents (glozow)
313c09f7b7beddfdb74c284720d209c81dfdb94f [test] helper function to increase transaction weight (glozow)
f8253d69d6f02850995a11eeb71fedc22e6f6575 extract/rename helper functions from rpc_packages.py (glozow)
3cd663a5d33aa7ef87994e452bced7f192d021a0 [policy] ancestor/descendant limits for packages (glozow)
c6e016aa139c8363e9b38bbc1ba0dca55700b8a7 [mempool] check ancestor/descendant limits for packages (glozow)
f551841d3ec080a2d7a7988c7b35088dff6c5830 [refactor] pass size/count instead of entry to CalculateAncestorsAndCheckLimits (glozow)
97dd1c729d2bbedf9527b914c0cc8267b8a7c21b MOVEONLY: add helper function for calculating ancestors and checking limits (glozow)
f95bbf58aaf72aab8a9c5827b1f162f3b8ac38f4 misc package validation doc improvements (glozow)

Pull request description:

  This PR implements a function to calculate mempool ancestors for a package and enforces ancestor/descendant limits on them as a whole. It reuses a portion of `CalculateMemPoolAncestors()`; there's also a small refactor to move the reused code into a generic helper function. Instead of calculating ancestors and descendants on every single transaction in the package and their ancestors, we use a "worst case" heuristic, treating every transaction in the package as each other's ancestor and descendant. This may overestimate everyone's counts, but is still pretty accurate in the our main package use cases, in which at least one of the transactions in the package is directly related to all the others (e.g. 1 parent + 1 child, multiple parents with 1 child, or chains).

  Note on Terminology: While "package" is often used to describe groups of related transactions _within_ the mempool, here, I only use package to mean the group of not-in-mempool transactions we are currently validating.

  #### Motivation

  It would be a potential DoS vector to allow submission of packages to mempool without a proper guard for mempool ancestors/descendants. In general, the purpose of mempool ancestor/descendant limits is to limit the computational complexity of dealing with families during removals and additions. We want to be able to validate multiple transactions on top of the mempool, but also avoid these scenarios:

  - We underestimate the ancestors/descendants during package validation and end up with extremely complex families in our mempool (potentially a DoS vector).
  - We expend an unreasonable amount of resources calculating everyone's ancestors and descendants during package validation.

ACKs for top commit:
  JeremyRubin:
    utACK accf3d5
  ariard:
    ACK accf3d5.

Tree-SHA512: 0d18ce4b77398fe872e0b7c2cc66d3aac2135e561b64029584339e1f4de2a6a16ebab3dd5784f376e119cbafc4d50168b28d3bd95d0b3d01158714ade2e3624d
Signed-off-by: Vijay <vijaydas.mp@gmail.com>
2024-08-29 10:01:28 +05:30
MarcoFalke
97fd2b2842
Merge bitcoin/bitcoin#22568: test: add addr-fetch peer connection state and timeout coverage
f8d8eb5fdaa93b6e5b77fd901b94927dc3a0473e test: add addr-fetch timeout connection coverage in p2p_addrfetch.py (Jon Atack)
9321086af7b75be536767d25abef4d7e02ca416a test: add assert_getpeerinfo method and coverage in p2p_addrfetch.py (Jon Atack)

Pull request description:

  This patch adds additional addr-fetch peer connection state and timeout coverage as a follow-up to #22096.

ACKs for top commit:
  Saviour1001:
    Tested ACK <code>[f8d8eb5](f8d8eb5fda)</code>
  mzumsande:
    Code review ACK f8d8eb5fdaa93b6e5b77fd901b94927dc3a0473e

Tree-SHA512: 9a13a705d1da6b308d6dcbc6930575205e2e88bfe9f2e7cb4e0c4c40d26538430e6b02c6c772d0cee64e534777348291469a139f99afbf9d4f93f31b9e7b0818
Signed-off-by: Vijay <vijaydas.mp@gmail.com>
2024-08-29 08:07:41 +05:30
pasta
8d3f31306e
Merge #6039: fix: optimize compilation time of gsl library
dc59cbc315 fix: optimize includes in gsl/pointer.h to speed up compile time (Konstantin Akimov)
d5e32e1b79 fix: disable gsl iostream feature for faster compile time (Konstantin Akimov)

Pull request description:

  ## Issue being fixed or feature implemented
  This header `gsl/pointers.h` is included in multiple other headers all over codebase.
  Any extra line of code inside gsl/pointers.h makes all project to compile slower.

  ## What was done?
  Removed headers `<algorithm>`, `<system_error>`, `<iosfwd>` from gsl/pointers.h

  ## How Has This Been Tested?
  Run command 5 times, takes minimum time:

  - baseline
  ```
  $ time g++ -o /tmp/a.out gsl/pointers.h -O2 -g -I.
  real    0m0,572s
  user    0m0,461s
  sys     0m0,108s
  ```

   - removed algorithm:
  ```
  $ time g++ -o /tmp/a.out gsl/pointers.h -O2 -g -I.
  real    0m0,505s
  user    0m0,398s
  sys     0m0,107s
  ```

   - removed algorithm and system_error:
  ```
  real    0m0,332s
  user    0m0,265s
  sys     0m0,067s
  ```

   - disabled iostream:
  ```
  $ time g++ -o /tmp/a.out gsl/pointers.h -O2 -g -I. -D GSL_NO_IOSTREAMS
  real    0m0,316s
  user    0m0,256s
  sys     0m0,060s
  ```

  as a result, overall project compilation time is also improved: `make clean ; sleep 3s ; time make -j20`
  ```
  real    5m42,934s
  user    80m35,127s
  sys     6m40,735s
  ```
  ```
  real    5m28,862s
  user    75m31,931s
  sys     6m32,591s
  ```

  ## Breaking Changes
  N/A

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone

ACKs for top commit:
  UdjinM6:
    utACK dc59cbc315

Tree-SHA512: 353d79c0e297c92e823972b53daaaed42494554ce550ea800ce8aa6990c704cedfbcb922e1d735eb8fc01ad14ffa873d86cdbc4d3731d841496392bb74286d33
2024-08-28 19:31:55 -05:00
pasta
a40802b1fc
Merge #6156: fix: let internal govobj/vote inv request trackers expire after 60 sec
1f4e1a17ed test: add test for governance inv expiration (UdjinM6)
06b4dba0d3 feat: make CInv python implementation aware of governance invs (UdjinM6)
c7c930ece6 fix: use correct condition in logs (UdjinM6)
d41d87a5be fix: use `std::chrono::seconds` (UdjinM6)
d6fe7146ff chore: make clang-format and linter happy (UdjinM6)
b57a9220c1 refactor: sqash 2 hash maps into one, use proper naming (UdjinM6)
4116ba3253 fix: let internal govobj/vote inv request trackers expire after 60 sec (UdjinM6)

Pull request description:

  ## Issue being fixed or feature implemented
  Another issue noticed during recent sb voting period. Lots of inv are coming from peers that never send the object/vote they announced. Let's not wait for these forever, 60 seconds should be enough.

  ## What was done?
  Add a "timer" to each request.

  ## How Has This Been Tested?
  Run tests, run a MN on mainnet and check logs.

  ## Breaking Changes
  n/a

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  knst:
    utACK 1f4e1a17ed

Tree-SHA512: f5c5a896f61b5aed27c6f42c926254156c604edb4efd2a3cd3738a7e8d1ed7bafffadc584148e4a5c1a172c2b3d61a06884a1f6d56eb4ebe37a4b0dbc050edd6
2024-08-28 19:29:36 -05:00
pasta
6dbed2ae75
Merge #6192: fix: protect CoinJoinWalletManager::m_wallet_manager_map with Mutex, avoid data race, partial bitcoin#22868
87d775d27c partial bitcoin#22868: Call load handlers without cs_wallet locked (Kittywhiskers Van Gogh)
c602ca15e1 coinjoin: protect `m_wallet_manager_map` with `cs_wallet_manager_map` (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  This pull request aims to deal with regressions ([build](https://gitlab.com/dashpay/dash/-/jobs/7550859495)) spotted in `develop` after the merger of [dash#6143](https://github.com/dashpay/dash/pull/6143), namely, a failure to build the multiprocess variant and two data races, one involving `ChainstateManager::m_best_header` and another involving `CoinJoinWalletManager::m_wallet_manager_map`.

  * Fixes for the build failure and the first data race (b136742f98c6792735232cc170123f849fd13bad and 9e7c6850139e69cb64cfe89813e6f0e4b1e626e1), have been spun off into [dash#6199](https://github.com/dashpay/dash/pull/6199) and merged

  * The second data race is being avoided by protected `m_wallet_manager_map` with a new `RecursiveMutex`, `cs_wallet_manager_map` (the contents of this PR). ~~A version of these changes are available using a regular `Mutex` but prove far more cumbersome ([source](b89457dc9a)) when taking practicalities into account ([comment](https://github.com/dashpay/dash/pull/6192#discussion_r1713207664)).~~ A `Mutex` version is now available courtesy of UdjinM6 (see [patch](ecf5c1ba76)), thanks!

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation **(note: N/A)**
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  UdjinM6:
    light-ACK 87d775d27c

Tree-SHA512: 52545a1547357a45fb6bf4d3948fc75a5e88f1d86e9809b1060007cd74dd383249691d8d9777f647c7534d458fe126ae818fa2b47f0e5b0cee78a469af1ed0c9
2024-08-28 19:28:22 -05:00
pasta
5aefd44ea3
fmt: run clang-format 2024-08-28 17:04:47 -05:00
pasta
7e9dec290f
refactor: drop some unneeded this and const 2024-08-28 17:04:13 -05:00
pasta
9bcda44b2e
Merge #6130: backport: Merge bitcoin/bitcoin#22337
1973532372 Merge bitcoin/bitcoin#22337: wallet: Use bilingual_str for errors (Samuel Dobson)

Pull request description:

  bitcoin backport

Top commit has no ACKs.

Tree-SHA512: ec638bde047788824961c230804461cee96ddcc0dbe1928bfb8c675c7f7f8c0c104850676a70fb8c9380401ead6d5986022ae3698a2bbc84f679531b11349580
2024-08-28 12:20:51 -05:00
pasta
b9a2f08338
Merge #6071: backport: Merge bitcoin#22619, 22593
ad840ec31a Merge bitcoin#22593: remove `hex_str_to_bytes` helper (Kittywhiskers Van Gogh)
cbc4c63f58 Merge bitcoin/bitcoin#22619: test: refactor: use consistent bytes <-> hex-string conversion in functional test framework (MarcoFalke)

Pull request description:

  bitcoin backports

ACKs for top commit:
  knst:
    utACK ad840ec31a
  kwvg:
    utACK ad840ec31a

Tree-SHA512: 48b4137683daab01a1bf51493d082ec359f80be7a9930b3423476e9ac5f4e73035d0f64d0f8e9b6b0c61b3e06efb648f9cc6bd620088c8cb5f27830157440adb
2024-08-28 12:17:01 -05:00
UdjinM6
c2c4b2b794
fix: release unused memory in CNetMsgMaker::Make() 2024-08-27 20:45:36 +03:00
Samuel Dobson
1973532372
Merge bitcoin/bitcoin#22337: wallet: Use bilingual_str for errors
92993aa5cf37995e65e68dfd6f129ecaf418e01c Change SignTransaction's input_errors to use bilingual_str (Andrew Chow)
171366e89b828a557f8262d9dc14ff7a03f813f7 Use bilingual_str for address fetching functions (Andrew Chow)
9571c69b51115454c6a699be9492024f7b46c2b4 Add bilingual_str::clear() (Andrew Chow)

Pull request description:

  In a couple of places in the wallet, errors are `std::string`. In order for these errors to be translated, change them to use `bilingual_str`.

ACKs for top commit:
  hebasto:
    re-ACK 92993aa5cf37995e65e68dfd6f129ecaf418e01c, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/22337#pullrequestreview-694542729) review, verified with
  klementtan:
    Code review ACK 92993aa5cf37995e65e68dfd6f129ecaf418e01c
  meshcollider:
    Code review ACK 92993aa5cf37995e65e68dfd6f129ecaf418e01c

Tree-SHA512: 5400e419dd87db8c49b67ed0964de2d44b58010a566ca246f2f0760ed9ef6a9b6f6df7a6adcb211b315b74c727bfe8c7d07eb5690b5922fa5828ceef4c83461f
Signed-off-by: Vijay <vijaydas.mp@gmail.com>
2024-08-27 20:08:25 +05:30
Kittywhiskers Van Gogh
87d775d27c
partial bitcoin#22868: Call load handlers without cs_wallet locked
excludes changes in `wallet/context.h` due to `::vpwallets` (and thus,
`cs_wallets`) not being deglobalized yet
2024-08-26 15:55:38 +00:00
Kittywhiskers Van Gogh
c602ca15e1
coinjoin: protect m_wallet_manager_map with cs_wallet_manager_map
Avoid TSan-reported data race

```
WARNING: ThreadSanitizer: data race (pid=374820)
  Read of size 8 at 0x7b140002ce10 by thread T12:
    #0 _M_ptr /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:154:42 (dashd+0xb58e08) (BuildId: c3fdce9f7e778985a4fb0968ff4506d9ad24d408)
    #1 get /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:361:21 (dashd+0xb58e08)
    #2 operator-> /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:355:9 (dashd+0xb58e08)
    #3 CoinJoinWalletManager::DoMaintenance() /src/dash/src/coinjoin/client.cpp:1907:9 (dashd+0xb58e08)
    [...]
  Previous write of size 8 at 0x7b140002ce10 by thread T17 (mutexes: write M0):
    #0 operator new(unsigned long) <null> (dashd+0x162657) (BuildId: c3fdce9f7e778985a4fb0968ff4506d9ad24d408)
    #1 allocate /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/ext/new_allocator.h:114:27 (dashd+0xb772b4) (BuildId: c3fdce9f7e778985a4fb0968ff4506d9ad24d408)
    #2 allocate /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/alloc_traits.h:443:20 (dashd+0xb772b4)
    #3 _M_get_node /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_tree.h:580:16 (dashd+0xb772b4)
    [...]
    #8 CoinJoinWalletManager::Add(CWallet&) /src/dash/src/coinjoin/client.cpp:1898:26 (dashd+0xb58c73) (BuildId: c3fdce9f7e778985a4fb0968ff4506d9ad24d408)
[...]
SUMMARY: ThreadSanitizer: data race [...]
```
2024-08-26 15:55:37 +00:00
Kittywhiskers Van Gogh
b75e83b298
merge bitcoin#24218: Fix implicit-integer-sign-change 2024-08-26 15:35:13 +00:00
Kittywhiskers Van Gogh
8ecc22f51f
merge bitcoin#23471: Improve ZMQ documentation 2024-08-26 15:35:13 +00:00
Kittywhiskers Van Gogh
2965093c4a
merge bitcoin#22079: Add support to listen on IPv6 addresses 2024-08-26 15:35:13 +00:00
Kittywhiskers Van Gogh
3ac3714957
merge bitcoin#21310: fix sync-up by matching notification to generated block 2024-08-26 15:35:13 +00:00
Kittywhiskers Van Gogh
7b0c725c59
merge bitcoin#21008: fix zmq test flakiness, improve speed 2024-08-26 15:35:13 +00:00
Kittywhiskers Van Gogh
5e87efd04b
merge bitcoin#20523: deduplicate 'sequence' publisher message creation/sending 2024-08-26 15:35:12 +00:00
Kittywhiskers Van Gogh
99c730f0f3
merge bitcoin#20953: dedup zmq test setup code (node restart, topics subscription) 2024-08-26 15:35:12 +00:00
Kittywhiskers Van Gogh
982c1f03d4
merge bitcoin#19572: Create "sequence" notifier, enabling client-side mempool tracking 2024-08-26 15:35:12 +00:00
Kittywhiskers Van Gogh
b0b4e0fa7f
zmq: Make g_zmq_notification_interface a smart pointer
courtesy of 8ed4ff8e from bitcoin#27125
2024-08-26 14:06:05 +00:00
Konstantin Akimov
dc59cbc315
fix: optimize includes in gsl/pointer.h to speed up compile time
it removes:
 - algorithm: `forward` is a part of utility not algorithm
 - system_error: `hash` is already declared in memory
2024-08-26 16:16:31 +07:00
Konstantin Akimov
d5e32e1b79
fix: disable gsl iostream feature for faster compile time
We do not use this feature and implementation is trivial: just call `get()`
2024-08-26 16:16:31 +07:00
Kittywhiskers Van Gogh
0a1ffd30b9
zmq: extend appending address to log msg for Dash-specific notifications
Continuation of e66870c5 from bitcoin#18309
2024-08-25 12:15:29 +00:00
pasta
f675fcbd25
Merge #6230: fix: resolve GitHub workflow warnings
88bf7a8cb4 fix: resolve GitHub workflow warnings (UdjinM6)

Pull request description:

  ## Issue being fixed or feature implemented
  https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

  ## What was done?

  ## How Has This Been Tested?
  Check results for CI and Guix actions (in future PRs? 🤷‍♂️ )

  ## Breaking Changes
  n/a

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

Top commit has no ACKs.

Tree-SHA512: 91464dbb85767e27bd26484cf597f7794e450d56901a407864ce9d3880fb15118047ee040f4e4ab19ea2a0a6dc8aaa6239353427f35f9c7500a6df0fdff6fb6f
2024-08-24 21:28:44 -05:00
Kittywhiskers Van Gogh
ad840ec31a
Merge bitcoin#22593: remove hex_str_to_bytes helper
Signed-off-by: Vijay <vijaydas.mp@gmail.com>
2024-08-25 07:55:20 +05:30
MarcoFalke
cbc4c63f58
Merge bitcoin/bitcoin#22619: test: refactor: use consistent bytes <-> hex-string conversion in functional test framework
5a1bef60a03b57de708a1a751bd90b8245fd8b83 test: refactor: remove binascii from test_framework (Zero-1729)

Pull request description:

  This PR continues the work started in PR #22593, regarding using the `bytes` built-in module. In this PR specifically, instances of `binascii`'s methods `hexlify`, `unhexlify`,  and `a2b_hex` have been replaced with the build-in `bytes` module's `hex` and `fromhex` methods where appropriate to make bytes <-> hex-string conversions consistent across the functional test files and test_framework.

  Additionally, certain changes made are based on the following assumption:

  ```
  bytes.hex(data) == binascii.hexlify(data).decode()
  bytes.hex(data).encode() == binascii.hexlify(data)
  ```

  Ran the functional tests to ensure behaviour is still consistent and changes didn't break existing tests.

  closes #22605

ACKs for top commit:
  theStack:
    Code-review ACK 5a1bef60a03b57de708a1a751bd90b8245fd8b83 🔢

Tree-SHA512: 8f28076cf0580a0d02a156f3e1e94c9badd3d41c3fbdfb2b87cd8a761dde2c94faa5f4c448d6747b1ccc9111c3ef1a1d7b42a11c806b241fa0410b7529e2445f
Signed-off-by: Vijay <vijaydas.mp@gmail.com>
2024-08-25 07:55:13 +05:30
pasta
114d7877d3
Merge #6098: backport: merge bitcoin#22278, #24103, #24235, #24177, #24299, #24917, #22564, #25349, #25571, #17487, #26999, #27011 (blockstorage backports: part 2)
b65863783f merge bitcoin#27011: Add simulation-based `CCoinsViewCache` fuzzer (Kittywhiskers Van Gogh)
1d0e410280 merge bitcoin#26999: A few follow-ups to bitcoin#17487 (Kittywhiskers Van Gogh)
7d837ea5c5 merge bitcoin#17487: allow write to disk without cache drop (Kittywhiskers Van Gogh)
2c758f4ba2 merge bitcoin#25571: Make mapBlocksUnknownParent local, and rename it (Kittywhiskers Van Gogh)
70a91e1d3f merge bitcoin#25349: CBlockIndex/CDiskBlockIndex improvements for safety, consistent behavior (Kittywhiskers Van Gogh)
eca0a64ea1 merge bitcoin#22564: Move mutable globals cleared in ::UnloadBlockIndex to BlockManager (Kittywhiskers Van Gogh)
916b3f0041 merge bitcoin#24917: Make BlockManager::LoadBlockIndex private (Kittywhiskers Van Gogh)
e10ca27fa6 merge bitcoin#24299: UnloadBlockIndex and ChainstateManager::Reset thread safety cleanups (Kittywhiskers Van Gogh)
18aa55be1e merge bitcoin#24177: add missing thread safety lock assertions (Kittywhiskers Van Gogh)
678e67c505 merge bitcoin#24235: use stronger EXCLUSIVE_LOCKS_REQUIRED() (Kittywhiskers Van Gogh)
edc665c80d merge bitcoin#24103: Replace RecursiveMutex m_cs_chainstate with Mutex, and rename it (Kittywhiskers Van Gogh)
d19ffd613a merge bitcoin#22278: Add LIFETIMEBOUND to CScript where needed (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * Dependent on https://github.com/dashpay/dash/pull/6097
  * Dependency for https://github.com/dashpay/dash/pull/6067
  * Test failures in `linux64_multiprocess-build` ([build](https://gitlab.com/dashpay/dash/-/jobs/7550924662)) and `linux64_tsan-test` ([build](https://gitlab.com/dashpay/dash/-/jobs/7550924666)) do not stem from this PR but are pre-existing failures in `develop` ([build](https://gitlab.com/dashpay/dash/-/jobs/7550859495), [build](https://gitlab.com/dashpay/dash/-/jobs/7550859499)). A fix for the build failures has been opened as a separate PR.

  ## Breaking Changes

  None observed.

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation **(note: N/A)**
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  UdjinM6:
    LGTM, utACK b65863783f
  PastaPastaPasta:
    utACK b65863783f

Tree-SHA512: 1a9c3a41617af274db169db47a9c9fce7ba7ce0b2d68aa75617640a55da11a0fa095cb25ce6a2d38f06d3f6a6cc4c08cb4cf82dca4bdc74192e8882fd5f7052f
2024-08-24 21:12:29 -05:00
UdjinM6
88bf7a8cb4
fix: resolve GitHub workflow warnings 2024-08-24 01:15:31 +03:00
Kittywhiskers Van Gogh
b65863783f
merge bitcoin#27011: Add simulation-based CCoinsViewCache fuzzer 2024-08-23 07:00:39 +00:00
Kittywhiskers Van Gogh
1d0e410280
merge bitcoin#26999: A few follow-ups to bitcoin#17487 2024-08-23 06:57:38 +00:00
Kittywhiskers Van Gogh
7d837ea5c5
merge bitcoin#17487: allow write to disk without cache drop 2024-08-23 06:57:37 +00:00
Kittywhiskers Van Gogh
2c758f4ba2
merge bitcoin#25571: Make mapBlocksUnknownParent local, and rename it 2024-08-23 06:57:37 +00:00
Kittywhiskers Van Gogh
70a91e1d3f
merge bitcoin#25349: CBlockIndex/CDiskBlockIndex improvements for safety, consistent behavior 2024-08-23 06:57:37 +00:00
Kittywhiskers Van Gogh
eca0a64ea1
merge bitcoin#22564: Move mutable globals cleared in ::UnloadBlockIndex to BlockManager
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2024-08-23 06:57:37 +00:00
Kittywhiskers Van Gogh
916b3f0041
merge bitcoin#24917: Make BlockManager::LoadBlockIndex private 2024-08-23 06:57:37 +00:00
Kittywhiskers Van Gogh
e10ca27fa6
merge bitcoin#24299: UnloadBlockIndex and ChainstateManager::Reset thread safety cleanups 2024-08-23 06:57:37 +00:00
Kittywhiskers Van Gogh
18aa55be1e
merge bitcoin#24177: add missing thread safety lock assertions 2024-08-23 06:57:37 +00:00