fae211c0ae0dd90876a3390eb21449b7b0bb45c4 lint: Start to use py lint scripts (MarcoFalke)
fa82e890e7950fe5ba6d4fa88fcd922cc929dc47 Move lint script and data file to avoid lint- prefix (MarcoFalke)
Pull request description:
ACKs for top commit:
fjahr:
tACK fae211c0ae0dd90876a3390eb21449b7b0bb45c4
Tree-SHA512: f8272a1bab9efb8203cac121710baae68f01f79e520ad71ff15aa516d19763d61c088b411b019de105a6a30e7ee3c274814d59963f6ac22ba1084560fb601f45
2227fc4e6203064b14e99bcf453601bd263a0196 test: minor fixes & improvements for files linter test (windsok)
Pull request description:
Couple of minor fixes & improvements for files linter test added in #21740
- Use a context manager when opening files, so that files are closed are we are done with them
- Use the `-z` flag when shelling out to `git ls-files` so that we can catch newlines and other weird control characters in filenames.
From the `git ls-files` manpage:
```
-z \0 line termination on output and do not quote filenames. See OUTPUT below for more information.
Without the -z option, pathnames with "unusual" characters are quoted as explained for the configuration variable
core.quotePath (see git-config(1)). Using -z the filename is output verbatim and the line is terminated by a NUL byte.
```
ACKs for top commit:
MarcoFalke:
cr ACK 2227fc4e6203064b14e99bcf453601bd263a0196
practicalswift:
cr ACK 2227fc4e6203064b14e99bcf453601bd263a0196: patch looks correct
Tree-SHA512: af059a805f4a7614162de85dea856052a45ab531895cb0431087e7fc9e037513fa7501bb5eb2fe43238adf5f09e77712ebdbb15b1486983359ad3661a3da0c60
46b025e00df40724175735eb5606ac73067cb3b8 test: add new python linter to check file names and permissions (windsok)
6f6bb3ebc7cb8e17a5dfc8ef55aa2d3f2dc6bdea test: fix file permissions on various scripts (windsok)
Pull request description:
Adds a new python linter test which tests for correct filenames and file permissions in the repository.
Replaces the existing tests in the `test/lint/lint-filenames.sh` and `test/lint/lint-shebang.sh` linter tests, as well as adding some new and increased testing. This increased coverage is intended to catch issues such as in #21728 and https://github.com/bitcoin/bitcoin/pull/16807/files#r345547050
Summary of tests:
* Checks every file in the repository against an allowed regexp to make sure only lowercase or uppercase alphanumerics (a-zA-Z0-9), underscores (_), hyphens (-), at (@) and dots (.) are used in repository filenames.
* Checks only source files (*.cpp, *.h, *.py, *.sh) against a stricter allowed regexp to make sure only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.) are used in source code filenames. Additionally there is an exception regexp for directories or files which are excepted from matching this regexp (This should replicate the existing `test/lint/lint-filenames.sh` test)
* Checks all files in the repository match an allowed executable or non-executable file permission octal. Additionally checks that for executable files, the file contains a shebang line.
* Checks that for executable `.py` and `.sh` files, the shebang line used matches an allowable list of shebangs (This should replicate the existing `test/lint/lint-shebang.sh` test)
* Checks every file that contains a shebang line to ensure it has an executable permission
Additionally updates the permissions on various files to comply with the new tests.
Fixes#21729
ACKs for top commit:
practicalswift:
cr re-ACK 46b025e00df40724175735eb5606ac73067cb3b8: patch still looks correct
kiminuo:
code review ACK 46b025e00df40724175735eb5606ac73067cb3b8 if `contrib/gitian-descriptors/assign_DISTNAME` permission change is deemed OK.
laanwj:
Code review ACK 46b025e00df40724175735eb5606ac73067cb3b8
Tree-SHA512: 1c8201a2cee0d9cbce15652b68cec9a6458a8b493fcd5392f98560aca0b1a12e668baab65a47100f116f626dadc3f591deb47f7368468c6a46c6c712c2533455
bd8b5d4007 net: add more details to log information in ETE and `WakeupPipes` (Kittywhiskers Van Gogh)
ec99294976 net: restrict access `EdgeTriggerEvents` members (Kittywhiskers Van Gogh)
f24520a3a2 net: log `close` failures in `EdgeTriggerEvents` and `WakeupPipe` (Kittywhiskers Van Gogh)
b8c3b480eb refactor: introduce `WakeupPipe`, move wakeup select pipe logic there (Kittywhiskers Van Gogh)
ed7d976c3e refactor: move wakeup pipe (de)registration to ETE (Kittywhiskers Van Gogh)
f50c710028 refactor: move `CConnman::`(`Un`)`registerEvents` to ETE (Kittywhiskers Van Gogh)
3a9f386138 refactor: move `SOCKET` addition/removal from interest list to ETE (Kittywhiskers Van Gogh)
212df0677f refactor: introduce `EdgeTriggeredEvents`, move {epoll, kqueue} fd there (Kittywhiskers Van Gogh)
3b11ef9b89 refactor: move `CConnman::SocketEventsMode` to `util/sock.h` (Kittywhiskers Van Gogh)
Pull request description:
## Motivation
`CConnman` is an entity that contains a lot of platform-specific implementation logic, both inherited from upstream and added upon by Dash (support for edge-triggered socket events modes like `epoll` on Linux and `kqueue` on FreeBSD/Darwin).
Bitcoin has since moved to strip down `CConnman` by moving peer-related logic to the `Peer` struct in `net_processing` (portions of which are backported in #5982 and friends, tracking efforts from https://github.com/bitcoin/bitcoin/issues/19398) and moving socket-related logic to `Sock` (portions of which are aimed to be backported in #6004, tracking efforts from https://github.com/bitcoin/bitcoin/pull/21878).
Due to the direction being taken and the difference in how edge-triggered events modes operate (utilizing interest lists and events instead of iterating over each socket) in comparison to level-triggered modes (which are inherited from upstream), it would be reasonable to therefore, isolate Dash-specific code into its own entities and minimize the information `CConnman` has about its internal workings.
One of the visible benefits of this approach is comparing `develop` (as of this writing, d44b0d5dcb) and this pull request for interactions between wakeup pipes logic and {`epoll`, `kqueue`} logic.
This is what construction looks like:
d44b0d5dcb/src/net.cpp (L3358-L3397)
But, if we segment wakeup pipes logic (that work on any platform with POSIX APIs and excludes Windows) and {`epoll`, `kqueue`} logic (calling them `EdgeTriggeredEvents` instead), construction looks different:
907a351517/src/util/wpipe.cpp (L12-L38)
Now wakeup pipes logic doesn't need to know what socket events mode is being used nor are the implementation aspects of (de)registering it its concern, that is now `EdgeTriggeredEvents` problem.
## Additional Information
* This pull request will need testing on macOS (FreeBSD isn't a tier-one target) to ensure that lack of breakage in `kqueue`-specific logic.
## Breaking Changes
* Dependency for https://github.com/dashpay/dash/pull/6018
* More logging has been introduced and existing log messages have been made more exhaustive. If there is parsing that relies on a particular template, they will have to be updated.
* If `EdgeTriggeredEvents` or `WakeupPipes` fail to initialize or are incorrectly initialized and not destroyed immediately, any further attempts at calling any of its functions will result in an `assert`-induced crash. Earlier behavior may have allowed for silent failure but segmentation of logic from `CConnman` means the newly created instances must only exist if the circumstances needed for it to initialize correctly are present.
This is to ensure that `CConnman` doesn't have to concern itself with internal workings of either entities.
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
- [x] I have made corresponding changes to the documentation **(note: N/A)**
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
PastaPastaPasta:
utACK bd8b5d4007
Tree-SHA512: 8f793d4b4f2d8091e05bb9cc108013e924bbfbf19081290d9c0dfd91b0f2c80652ccf853f1596562942b4433509149c526e111396937988db605707ae1fe7366
- File descriptor creation and destruction are handled within ETE, no
reason to expose read-write access to it.
- (Un)registerPipe should only be used within WakeupPipe, remove from
public view.
a33dcb3283 fix: CheckWalletOwnsScript/CheckWalletOwnsKey to use wallet instead of SPK (Konstantin Akimov)
b2ede8bfee feat: update list of tests that still doesn't support descriptor wallets (Konstantin Akimov)
838d06f2fd feat: enable descriptor wallets for more tests (Konstantin Akimov)
5ab108c982 feat: implementation of RPC 'protx register' for descriptor wallets (Konstantin Akimov)
Pull request description:
## Issue being fixed or feature implemented
Many rpc such as `protx register` uses forcely LegacyScriptPubKeyMan instead using CWallet's interface.
It causes a failures such as
```
test_framework.authproxy.JSONRPCException: This type of wallet does not support this command (-4)
```
for all functional tests that uses Masternodes/evo nodes.
See https://github.com/dashpay/dash-issues/issues/59 to track progress
## What was done?
Some direct usages of LegacyScriptPubKeyMan refactored to use CWallet's functionality.
There are still 4 functional tests that doesn't work for descriptor wallets:
- feature_dip3_deterministicmns.py (no rpc `protx updateregistar`)
- feature_governance.py: no rpc for `governance votemany` and `governance votealias`
- interface_zmq_dash.py (see governance)
That's part I of changes, other changes are not PR-ready yet, WIP.
## How Has This Been Tested?
Firstly, the flag `--legacy-wallets` are removed for many functional tests.
Secondly, the flag `--descriptors` is inverted in default value:
```
diff --git a/test/functional/test_framework/test_framework.py b/test/functional/test_framework/test_framework.py
index 585a6a74d6..9ad5fd1daa 100755
--- a/test/functional/test_framework/test_framework.py
+++ b/test/functional/test_framework/test_framework.py
@@ -242,10 +242,10 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
if self.options.descriptors is None:
# Prefer BDB unless it isn't available
- if self.is_bdb_compiled():
- self.options.descriptors = False
- elif self.is_sqlite_compiled():
+ if self.is_sqlite_compiled():
self.options.descriptors = True
+ elif self.is_bdb_compiled():
+ self.options.descriptors = False
else:
# If neither are compiled, tests requiring a wallet will be skipped and the value of self.options.descriptors won't matter
# It still needs to exist and be None in order for tests to work however.
```
## Breaking Changes
N/A, descriptor wallets have not been publicly released yet
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e tests
- [x] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
PastaPastaPasta:
utACK a33dcb3283
Tree-SHA512: 24e22ac91e30a3804d1ac9af864eb9d073208bbb6d1f3326c7f0438f3ccce2b553aa46450989e48cb5c6e0d838ff1c88c6522f195e7aa2bd89342710f3ecef77
acd0f49d7b refactor: significant Mutex refactoring (pasta)
Pull request description:
## Issue being fixed or feature implemented
Don't use generic names; recursive mutexes where not needed; etc
## What was done?
Includes:
RecursiveMutex -> Mutex,
renaming of `cs` to something more meaningful,
usage of atomics where trivially possible,
introduce a method CQuorum::SetVerificationVector to avoid needing to lock an internal mutex externally
## How Has This Been Tested?
Compiling
## Breaking Changes
None
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
Top commit has no ACKs.
Tree-SHA512: 76d0ee37e348680bdcd8f03237d3fc1febbf908a9c13e6ddea7be52a35adfca35cde3001ce6ecb140d7dba950ad19519d34d137de17a073306e3e7b26cb95b70
5dde8e7b33 merge bitcoin#25109: Strengthen AssertLockNotHeld assertions (Kittywhiskers Van Gogh)
a1f005ee71 merge bitcoin#24157: Replace RecursiveMutex cs_totalBytesSent with Mutex and rename it (Kittywhiskers Van Gogh)
de4b4bf9ee merge bitcoin#24108: Replace RecursiveMutex cs_addrLocal with Mutex, and rename it (Kittywhiskers Van Gogh)
2f7a138452 merge bitcoin#24079: replace RecursiveMutex cs_SubVer with Mutex (and rename) (Kittywhiskers Van Gogh)
23b152cd37 merge bitcoin#22829: various RecursiveMutex replacements in CConnman (Kittywhiskers Van Gogh)
362e3101ad merge bitcoin#21943: Dedup and RAII-fy the creation of a copy of CConnman::vNodes (Kittywhiskers Van Gogh)
bf98ad6a42 merge bitcoin#22782: Remove unused MaybeSetAddrName (Kittywhiskers Van Gogh)
2b65526818 merge bitcoin#21167: make CNode::m_inbound_onion public, initialize explicitly (Kittywhiskers Van Gogh)
Pull request description:
## Additional Information
* Dependent on https://github.com/dashpay/dash/pull/6001
* Dependency for https://github.com/dashpay/dash/pull/6018
* Partially reverts ff69e0d575 from https://github.com/dashpay/dash/pull/5336 due to `Span<CNode*>`'s incompatibility with `CConnman::NodesSnapshot::Snap()` (returning `const std::vector<CNode*>&`)
```
masternode/sync.cpp:147:18: error: no matching member function for call to 'RequestGovernanceObjectVotes'
m_govman.RequestGovernanceObjectVotes(snap.Nodes(), connman);
~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
./governance/governance.h:360:9: note: candidate function not viable: no known conversion from 'const
std::vector<CNode *>' to 'CNode &' for 1st argument
int RequestGovernanceObjectVotes(CNode& peer, CConnman& connman) const;
^
./governance/governance.h:361:9: note: candidate function not viable: no known conversion from 'const std::vector<CNode *>' to 'Span<CNode *>' for 1st argument
int RequestGovernanceObjectVotes(Span<CNode*> vNodesCopy, CConnman& connman) const;
^
1 error generated.
```
* Dash already implements its own `CNode*` iteration logic in [dash#1382](https://github.com/dashpay/dash/pull/1382) and implemented additional capabilities in [dash#1575](https://github.com/dashpay/dash/pull/1575), which meant backporting [bitcoin#21943](https://github.com/bitcoin/bitcoin/pull/21943) involved migrating Dash-specific code to upstream logic that needed to be modified to implement expected functionality.
* Unlike Bitcoin, Dash maintains a map of every raw `SOCKET` corresponding to a pointer of their `CNode` instance and uses it to translate socket sets to their corresponding `CNode*` sets. This is done to accommodate for edge-triggered modes which have an event-socket relationship, as opposed to level-triggered modes, which have a socket-event relationship.
This means that `CConnman::SocketHandlerConnected()` doesn't require access to a vector of all `CNode` pointers and therefore, the argument `nodes` has been omitted.
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
- [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
- [x] I have made corresponding changes to the documentation **(note: N/A)**
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
PastaPastaPasta:
utACK 5dde8e7b33
Tree-SHA512: 5685d8ebb4fa1f10d018e60d9b0efc3100ea13ac437e7892a09ad3f86d6ac6756e4b5a08ebe70de2eabb27740678e10b975d319f2d553ae5b27dafa71dba0a9f
d3ad11d056 chore: add release notes for sethdseed RPC (Konstantin Akimov)
ad25d54300 Merge bitcoin/bitcoin#21329: descriptor wallet: Cache last hardened xpub and use in normalized descriptors (Samuel Dobson)
24b1f6bb27 Merge #21238: A few descriptor improvements to prepare for Taproot support (W. J. van der Laan)
14ac2b77f3 Merge #21127: wallet: load flags before everything else (Wladimir J. van der Laan)
19b2b27785 Merge #20403: wallet: upgradewallet fixes, improvements, test coverage (MarcoFalke)
708586c77e Merge #18836: wallet: upgradewallet fixes and additional tests (Andrew Chow)
752e4ca048 Merge #19490: wallet: Fix typo in comments; Simplify assert (Samuel Dobson)
63895fde23 Merge #19046: Replace CWallet::Set* functions that use memonly with Add/Load variants (Andrew Chow)
2c0d5b7c71 refactor: rename hdChain to m_hd_chain (Konstantin Akimov)
266aefc544 feat: sethdseed rpc added. Based on bitcoin#12560 and the newest related changes (Konstantin Akimov)
Pull request description:
## Issue being fixed or feature implemented
Next batch of backports related to descriptor wallets.
See related issue to track a progress: https://github.com/dashpay/dash-issues/issues/59
Changes in this PR also used in "hardware signing" feature (coming later)
## What was done?
1. It implements a new rpc `sethdseed` that is based on bitcoin#12560 and newer related changes.
2. refactoring to rename `hdChain` to `m_hd_chain` (see bitcoin#17681 which is DNM).
3. Bitcoin backports (some of them uses sethdseed, and requires m_hd_chain to reduce conflicts):
- bitcoin/bitcoin#19046
- bitcoin/bitcoin#19490
- bitcoin/bitcoin#18836
- bitcoin/bitcoin#20403
- bitcoin/bitcoin#21127
- bitcoin/bitcoin#21238
- bitcoin/bitcoin#21329
## How Has This Been Tested?
Run unit/functional tests.
The backports #18836 and #20403 are heavily modified to adopt `wallet_upgradewallet.py` to our codebase.
## Breaking Changes
N/A
Though, `sethdseed` implementation is not a final version as it is now, can be removed (superseeded by `upgradetohd` or got mnemonic-feature and super-seed `upgradetohd`).
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
PastaPastaPasta:
utACK d3ad11d056
Tree-SHA512: 02182182ec7a5f89eb7d3bc34072d894a86cc89c5eea124e702cc5ed527f76863469b1fd9313b3ea643a8774a358031be927d7b78ec7cd39df0a9ca77559d66d
e6cf0ed92de31a5ac35a271b0da8f0a8364d1175 wallet, rpc: listdescriptors does not need unlocked (Andrew Chow)
3280704886b60644d103a5eb310691c003a39328 Pass in DescriptorCache to ToNormalizedString (Andrew Chow)
7a26ff10c2f2e139fbc63e2f37fb33ea4efae088 Change DescriptorImpl::ToStringHelper to use an enum (Andrew Chow)
75530c93a83f3e94bcb78b6aa463c5570c1e737e Remove priv option for ToNormalizedString (Andrew Chow)
74fede3b8ba69e2cc82c617cdf406ab79df58825 wallet: Upgrade existing descriptor caches (Andrew Chow)
432ba9e5434da90d2cf680f23e8c7b7164c9f945 wallet: Store last hardened xpub cache (Andrew Chow)
d87b544b834077f102724415e0fada6ee8b2def2 descriptors: Cache last hardened xpub (Andrew Chow)
cacc3910989c4f3d7afa530dbab042461426abce Move DescriptorCache writing to WalletBatch (Andrew Chow)
0b4c8ef75cd03c8f0a8cfadb47e0fbcabe3c5e59 Refactor Cache merging and writing (Andrew Chow)
976b53b085d681645fd3a008fe382de85647e29f Revert "Cache parent xpub inside of BIP32PubkeyProvider" (Andrew Chow)
Pull request description:
Currently fetching a normalized descriptor requires the wallet to be unlocked as it needs the private keys to derive the last hardened xpub. This is not very user friendly as normalized descriptors shouldn't require and don't involve the private keys except for derivation. We solve this problem by caching the last hardened xpub (which has to be derived at some point when generating the address pool).
However the last hardened xpub was not already being cached. We only cached the immediate parent xpub and derived child keys. For example, with a descriptor derivation path of `/84'/0'/0'/0/*`, the parent xpub that is cached is `m/84'/0'/0'/0`, and the child keys of `m/84'/0'/0'/0/i` (note that child keys would not be cached in this case). This parent xpub is not suitable for the normalized descriptor form as we want the key at `m/84'/0'/0'`. So this PR adds another field to `DescriptorCache` to cache the last hardened xpub so that we can use them for normalized descriptors.
Since `DescriptorCache` is changing, existing descriptor wallets need to be upgraded to use this new cache. The upgrade will occur in the background either at loading time (if the wallet is not encrypted) or at unlocking time in the same manner that `UpgradeKeyMetadata` operates. It will use a new wallet flag `WALLET_FLAG_LAST_HARDENED_XPUB_CACHED` to indicate whether the descriptor wallet has the last hardened xpub cache.
Lastly `listdescriptors` will not require the wallet to be locked and `getaddressinfo`'s `parent_desc` will always be output (assuming the upgrade has occurred).
ACKs for top commit:
fjahr:
tACK e6cf0ed92de31a5ac35a271b0da8f0a8364d1175
S3RK:
reACK e6cf0ed
jonatack:
Semi ACK e6cf0ed92de31a5ac35a271b0da8f0a8364d1175 reviewed, debug-built and ran unit tests and some of the descriptor functional tests at each commit. I'm not very familiar with this code and it could be clearer to the uninitiated IMHO, so I'm not confident enough to give a full ACK. Various minor suggestions follow, most of them for readability, feel free to pick and choose.
meshcollider:
Code review + functional test run ACK e6cf0ed92de31a5ac35a271b0da8f0a8364d1175
Tree-SHA512: ac27aade8644525cd65bfcaf27ff32afb974085b1451faf4ff68c6671a690bd6a41d4f39a33cbf461ae0fbe85995c0a4c08dbd36171da1c1d2a1d00053ad298d
0b188b751f970027c52729e0c223cc9257669322 Clean up context dependent checks in descriptor parsing (Pieter Wuille)
33275a96490445e293c322a29a3b146ccb91083c refactor: move uncompressed-permitted logic into ParsePubkey* (Pieter Wuille)
17e006ff8d5e42f22474c5191d1b745bbc97571f refactor: split off subscript logic from ToStringHelper (Pieter Wuille)
6ba5dda0c9de75196c6a427d9e59d39e5a41bff7 Account for key cache indices in subexpressions (Pieter Wuille)
4441c6f3c046c0b28ce3f0ca6d938af71d797586 Make DescriptorImpl support multiple subscripts (Pieter Wuille)
a917478db0788b244c0c799b98bf67a94df7035e refactor: move population of out.scripts from ExpandHelper to MakeScripts (Pieter Wuille)
84f3939ece9f4901141b28fd2dd6e3899d01d66e Remove support for subdescriptors expanding to multiple scripts (Pieter Wuille)
Pull request description:
These are a few refactors and non-invasive improvements to the descriptors code to prepare for adding Taproot descriptors.
None of the commits change behavior in any way, except the last one which improves error reporting a bit.
ACKs for top commit:
S3RK:
reACK 0b188b7
Sjors:
re-ACK 0b188b7
achow101:
re-ACK 0b188b751f970027c52729e0c223cc9257669322
Tree-SHA512: cb4e999134aa2bace0e13d4883454c65bcf1369e1c8585d93cc6444ddc245f3def5a628d58af7dab577e9d5a4a75d3bb46f766421fcc8cc5c85c01a11f148b3f
9305862f71189d47c873d366bf976622447e18af wallet: load flags before everything else (Sjors Provoost)
Pull request description:
Load and set wallet flags before processing other records. That way we can take them into account while processing those other records.
Suggested here:
https://github.com/bitcoin/bitcoin/pull/16546#discussion_r572334983
ACKs for top commit:
laanwj:
Code review ACK 9305862f71189d47c873d366bf976622447e18af
gruve-p:
ACK 9305862f71
achow101:
ACK 9305862f71189d47c873d366bf976622447e18af
Tree-SHA512: 7104523e369ce3c670571fe5e8b52c67b9ca92b8e36a2da5eb6f9f8bf8ed0544897007257204b68f6f371d682b3ef0d0635d36e6e8416ac74af1999d9fbc869c
3eb6f8b2e61c24a22ea9396d86672307845f35eb wallet (not for backport): improve upgradewallet error messages (Jon Atack)
ca8cd893bb56bf5d455154b0498b1f58f77d20ed wallet: fix and improve upgradewallet error responses (Jon Atack)
99d56e357159c7154f69f28cb5587c5ca20d6594 wallet: fix and improve upgradewallet result responses (Jon Atack)
2498b04ce88696a3216fc38b7d393906b733e8b1 Don't upgrade to HD split if it is already supported (Andrew Chow)
c46c18b788cb0862aafbb116fd37936cbed6a431 wallet: refactor GetClosestWalletFeature() (Jon Atack)
Pull request description:
This follows up on #18836 and #20282 to fix and improve the as-yet unreleased `upgradewallet` feature and also implement review follow-up in https://github.com/bitcoin/bitcoin/pull/18836#discussion_r519328607.
This PR fixes 4 upgradewallet issues:
- this bug: https://github.com/bitcoin/bitcoin/pull/20403#discussion_r526063920
- it returns nothing in the absence of an RPC error, which isn't reassuring for users
- it returns the same thing both in the case of a successful upgrade and when no upgrade took place
- the error message object is currently dead code
This PR fixes the above and provides:
...user feedback to not silently return without upgrading
```
{
"wallet_name": "disable private keys",
"previous_version": 169900,
"current_version": 169900,
"result": "Already at latest version. Wallet version unchanged."
}
```
...better feedback after successfully upgrading
```
{
"wallet_name": "watch-only",
"previous_version": 159900,
"current_version": 169900,
"result": "Wallet upgraded successfully from version 159900 to version 169900."
}
```
...helpful error responses
```
{
"wallet_name": "blank",
"previous_version": 169900,
"current_version": 169900,
"error": "Cannot downgrade wallet from version 169900 to version 159900. Wallet version unchanged."
}
{
"wallet_name": "blank",
"previous_version": 130000,
"current_version": 130000,
"error": "Cannot upgrade a non HD split wallet from version 130000 to version 169899 without upgrading to support pre-split keypool. Please use version 169900 or no version specified."
}
```
updated help:
```
upgradewallet ( version )
Upgrade the wallet. Upgrades to the latest version if no version number is specified.
New keys may be generated and a new wallet backup will need to be made.
Arguments:
1. version (numeric, optional, default=169900) The version number to upgrade to. Default is the latest wallet version.
Result:
{ (json object)
"wallet_name" : "str", (string) Name of wallet this operation was performed on
"previous_version" : n, (numeric) Version of wallet before this operation
"current_version" : n, (numeric) Version of wallet after this operation
"result" : "str", (string, optional) Description of result, if no error
"error" : "str" (string, optional) Error message (if there is one)
}
```
ACKs for top commit:
achow101:
ACK 3eb6f8b
MarcoFalke:
review ACK 3eb6f8b2e61c24a22ea9396d86672307845f35eb 🛡
Tree-SHA512: b767314069e26b5933b123acfea6aa40708507f504bdb22884da020a4ca1332af38a7072b061e36281533af9f4e236d94d3c129daf6fe5b55241127537038eed
5f9c0b6360215636cfa62a70d3a70f1feb3977ab wallet: Remove -upgradewallet from dummywallet (MarcoFalke)
a314271f08215feba53ead27096ac7fda34acb3c test: Remove unused wallet.dat (MarcoFalke)
bf7635963c03203e7189ddaa56c6b086a0108cbf tests: Test specific upgradewallet scenarios and that upgrades work (Andrew Chow)
4b418a9decc3e855ee4b0bbf9e61121c8e9904e5 test: Add test_framework/bdb.py module for inspecting bdb files (Andrew Chow)
092fc434854f881330771a93a1280ac67b1d3549 tests: Add a sha256sum_file function to util (Andrew Chow)
0bd995aa19be65b0dd23df1df571c71428c2bc32 wallet: upgrade the CHDChain version number when upgrading to split hd (Andrew Chow)
8e32e1c41c995e832e643f605d35a7aa112837e6 wallet: remove nWalletMaxVersion (Andrew Chow)
bd7398cc6258c258e9f4411c50630ec4a552341b wallet: have ScriptPubKeyMan::Upgrade check against the new version (Andrew Chow)
5f720544f34dedf75b063b962845fa8eca604514 wallet: Add GetClosestWalletFeature function (Andrew Chow)
842ae3842df489f1b8d68e67a234788966218184 wallet: Add utility method for CanSupportFeature (Andrew Chow)
Pull request description:
This PR cleans up the wallet upgrade mechanism a bit, fixes some probably bugs, and adds more test cases.
The `nWalletMaxVersion` member variable has been removed as it made `CanSupportFeature` unintuitive and was causing a couple of bugs. The reason this was introduced originally was to allow a wallet upgrade to only occur when the new feature is first used. While this makes sense for the old `-upgradewallet` option, for an RPC, this does not quite make sense. It's more intuitive for an upgrade to occur if possible if the `upgradewallet` RPC is used as that's an explicit request to upgrade a particular wallet to a newer version. `nWalletMaxVersion` was only relevant for upgrades to `FEATURE_WALLETCRYPT` and `FEATURE_COMPRPUBKEY` both of which are incredibly old features. So for such wallets, the behavior of `upgradewallet` will be that the feature is enabled immediately without the wallet needing to be encrypted at that time (note that `FEATURE_WALLETCRYPT` indicates support for encryption, not that the wallet is encrypted) or for a new key to be generated.
`CanSupportFeature` would previously indicate whether we could upgrade to `nWalletMaxVersion` not just whether the current wallet version supported a feature. While this property was being used to determine whether we should upgrade to HD and HD chain split, it was also causing a few bugs. Determining whether we should upgrade to HD or HD chain split is resolved by passing into `ScriptPubKeyMan::Upgrade` the version we are upgrading to and checking against that. By removing `nWalletMaxVersion` we also fix a bug where you could upgrade to HD chain split without the pre-split keypool.
`nWalletMaxVersion` was also the version that was being reported by `getwalletinfo` which meant that the version reported was not always consistent across restarts as it depended on whether `upgradewallet` was used. Additionally to make the wallet versions consistent with actually supported versions, instead of just setting the wallet version to whatever is given to `upgradewallet`, we normalize the version number to the closest supported version number. For example, if given 150000, we would store and report 139900.
Another bug where CHDChain was not being upgraded to the version supporting HD chain split is also fixed by this PR.
Lastly several more tests have been added. Some refactoring to the test was made to make these tests easier. These tests check specific upgrading scenarios, such as from non-HD (version 60000) to HD to pre-split keypool. Although not specifically related to `upgradewallet`, `UpgradeKeyMetadata` is now being tested too.
Part of the new tests is checking that the wallet files are identical before and after failed upgrades. To facilitate this, a utility function `sha256sum_file` has been added. Another part of the tests is to examine the wallet file itself to ensure that the records in the wallet.dat file have been correctly modified. So a new `bdb.py` module has been added to deserialize the BDB db of the wallet.dat file. This format isn't explicitly documented anywhere, but the code and comments in BDB's source code in file `dbinc/db_page.h` describe it. This module just dumps all of the fields into a dict.
ACKs for top commit:
MarcoFalke:
approach ACK 5f9c0b6360
laanwj:
Code review ACK 5f9c0b6360215636cfa62a70d3a70f1feb3977ab
jonatack:
ACK 5f9c0b6360215636cfa62a70d3a70f1feb3977ab, approach seems fine, code review, only skimmed the test changes but they look well done, rebased on current master, debug built and verified the `wallet_upgradewallet.py` test runs green both before and after running `test/get_previous_releases.py -b v0.19.1 v0.18.1 v0.17.2 v0.16.3 v0.15.2`
Tree-SHA512: 7c4ebf420850d596a586cb6dd7f2ef39c6477847d12d105fcd362abb07f2a8aa4f7afc5bfd36cbc8b8c72fcdd1de8d2d3f16ad8e8ba736b6f4f31f133fe5feba
facd7dd3d1f9d51e1133974ff69eeb48f5ae282b wallet: Fix typo in comments; Simplify assert (MarcoFalke)
Pull request description:
Follow up to https://github.com/bitcoin/bitcoin/pull/19046#discussion_r443783101 and https://github.com/bitcoin/bitcoin/pull/19046#discussion_r443793690
ACKs for top commit:
practicalswift:
ACK facd7dd3d1f9d51e1133974ff69eeb48f5ae282b
jonatack:
ACK facd7dd3d1f9d51e1133974ff69eeb48f5ae282b
hebasto:
ACK facd7dd3d1f9d51e1133974ff69eeb48f5ae282b, spelling verified with `test/lint/lint-spelling.sh`: all remaining warnings are false positive.
Tree-SHA512: 2b185d138058840db56726bb6bcc42e5288a954e2a410c49e04806a047fbbdaf0bb2decc70ecf7613c69caa766655705ca44151613e7ea5015b386d1e726d870
3a9aba21a49a6d80bd187940d5e26893937b6832 Split SetWalletFlags into Add/LoadWalletFlags (Andrew Chow)
d9cd095b5965fc20c09f401370e7ba99446663e3 Split SetActiveScriptPubKeyMan into Add/LoadActiveScriptPubKeyMan (Andrew Chow)
0122fbab4c340b23ae56173de6c5ab866ba25ab8 Split SetHDChain into AddHDChain and LoadHDChain (Andrew Chow)
Pull request description:
`SetHDChaiin`, `SetActiveScriptPubKeyMan`, and `SetWalletFlags` have a `memonly` argument which is kind of confusing, as noted in https://github.com/bitcoin/bitcoin/pull/17681#discussion_r427633081. This PR replaces those functions with `Add*` and `Load*` variants so that they follow the pattern used elsewhere in the wallet.
`AddHDChain`, `AddActiveScriptPubKeyMan`, and `AddWalletFlags` both set their respective variables in `CWallet` and writes them to disk. These functions are used by the actions which modify the wallet such as `sethdseed`, `importdescriptors`, and creating a new wallet.
`LoadHDChain`, `LoadActiveScriptPubKeyMan`, and `LoadWalletFlags` just set the `CWallet` variables. These functions are used by `LoadWallet` when loading the wallet from disk.
ACKs for top commit:
jnewbery:
Code review ACK 3a9aba21a49a6d80bd187940d5e26893937b6832
ryanofsky:
Code review ACK 3a9aba21a49a6d80bd187940d5e26893937b6832. Only changes since last review tweaks making m_wallet_flags updates more safe
meshcollider:
utACK 3a9aba21a49a6d80bd187940d5e26893937b6832
Tree-SHA512: 365aeaafc5ba42879c0eb797ec3beb29ab70e27f917dc880763f743420b3be6ddf797240996beed8a9ad70fb212c2590253c6b44c9dc244529c3939d9538983f
This commit helps to unify our implementation with bitcoin which has
the similar refactoring in bitcoin#17681 but we DNM it due to difference in
sethdseed behavior: we do not replace seed ever
The key difference between bitcoin's and dash's implementation that sethdseed
does not update existing seed for wallet. Seed can be set only once.
It behave similarly to `upgradetohd` rpc, but since v20.1 all wallets are HD and
the name `upgradetohd` is not relevant more.
b9a2ce74c0 fix: remove stretching from Overview page when it's not needed (Konstantin Akimov)
Pull request description:
## Issue being fixed or feature implemented
Overview page has strange stretching on sides, which make balance moving left-right when window is scalled.
## What was done?
Removed stretches for Overview Page from the sides, it makes window a little more balanced. This PR lets to do backport of bitcoin-core/gui#176 which improves behavior further more be increasing size of "transaction list" part of window.
## How Has This Been Tested?
See screenshot. Resized window with/without patch, in "Discreet mode" off and on. Both looks not perfect but better than before.
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
Top commit has no ACKs.
Tree-SHA512: f599f4f40986b37c3c9a8c36e2146bdf7b8e38d15a6f65f26fec4e30440ab54d07ea1f36a2b11e48a6dc3bcd64ececd3c4da6255fc6c9f58fba2a4abb2221b8d
94a8e1a713 Merge bitcoin/bitcoin#28097: depends: xcb-proto 1.15.2 (fanquake)
Pull request description:
## Issue being fixed or feature implemented
Resolves build failures with Python 3.12, i.e building on rawhide:
```bash
make -C depends -j9
...
make[3]: Nothing to be done for 'install-exec-am'.
/usr/bin/mkdir -p '/bitcoin/depends/work/staging/aarch64-unknown-linux-gnu/xcb_proto/1.14.1-4a91ac9dc41/bitcoin/depends/aarch64-unknown-linux-gnu/lib/python3.12/site-packages/xcbgen'
/usr/bin/install -c -m 644 __init__.py error.py expr.py align.py matcher.py state.py xtypes.py '/bitcoin/depends/work/staging/aarch64-unknown-linux-gnu/xcb_proto/1.14.1-4a91ac9dc41/bitcoin/depends/aarch64-unknown-linux-gnu/lib/python3.12/site-packages/xcbgen'
Traceback (most recent call last):
File "<string>", line 2, in <module>
ModuleNotFoundError: No module named 'imp'
make[3]: *** [Makefile:271: install-pkgpythonPYTHON] Error 1
```
`imp` was removed in 3.12: https://docs.python.org/3/library/imp.html.
## What was done?
Bitcoin backport bitcoin#28097
## How Has This Been Tested?
Run build `depends` with clean - it doesn't fail
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
Top commit has no ACKs.
Tree-SHA512: 61415fcc0c86f57d6b124907505102ae5b319c95d6ee0b531ce75fa2370a82bafd7a40b21e24eb7d1ac38ba93d59d9e7b89829667c61f4f4caf46029e6dd1cf7
b2910fba02 fix: resolve potential deadlocks in CJ (UdjinM6)
Pull request description:
## Issue being fixed or feature implemented
```
POTENTIAL DEADLOCK DETECTED
Previous lock order was:
(2) 'cs_wallet' in wallet/wallet.cpp:3826 (in thread 'qt-init')
(2) 'pwallet->cs_wallet' in wallet/walletdb.cpp:705 (in thread 'qt-init')
(1) 'cs_KeyStore' in wallet/scriptpubkeyman.cpp:971 (in thread 'qt-init')
Current lock order is:
'cs_deqsessions' in coinjoin/client.cpp:261 (in thread 'main')
(1) 'cs_KeyStore' in wallet/scriptpubkeyman.cpp:1522 (in thread 'main')
(2) 'cs_wallet' in wallet/wallet.cpp:1629 (in thread 'main')
```
This one is for `ResetPool()`.
## What was done?
Lock `cs_wallet` when calling `keyHolderStorage.ReturnAll()` in some places* to ensure the right order of locks.
(*In other places `cs_wallet` is held already, no need to double lock).
## How Has This Been Tested?
Mixing
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
PastaPastaPasta:
utACK b2910fba02
Tree-SHA512: 8be98df021f7683cd496ebe095dd7b32ebea76c7f9c7c085af3bc0a6901d9cfd4d4624e20a2eee1f3b0d69fd711f8fceb9a91c386b9bf02475632a23b3a0f09a
That are:
- feature_dip3_deterministicmns.py
- interface_zmq_dash.py
- feature_governance.py
- wallet_upgradetohd.py (as expected to be implemented for legacy-only wallets)
- p2p_timeouts.py (why? can not understand it)
This partially reverts commit b20f812674.
04ba164064 refactor: immediately update lastCleanupTime (pasta)
d0d2641154 refactor: minimize locking of cs_main in chainlocks.cpp (pasta)
Pull request description:
## Issue being fixed or feature implemented
Simple changes, just look at the two commits: first we minimize scope of cs_main to what actually needs it. Then we change where we update the `lastCleanupTime` to right after we check it to minimize any chance of two threads entering at the same time.
## What was done?
## How Has This Been Tested?
Built, ran for a bit
## Breaking Changes
None
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that apply._
- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
Top commit has no ACKs.
Tree-SHA512: 63432ccc16a67e6478e578c96fe809cf33d08e5068285c21e65ccc175c0c9e82c23519d57f0c4ebac162a094bfc20eb44df30cbe85b26815d02eaac06ceb66ad
0bba55f6af spork: replace `LOCKS_EXCLUDED` with negative `EXCLUSIVE_LOCKS_REQUIRED` (Kittywhiskers Van Gogh)
d657951f90 masternode: replace `LOCKS_EXCLUDED` with negative `EXCLUSIVE_LOCKS_REQUIRED` (Kittywhiskers Van Gogh)
8b1d3b55ab llmq: replace `LOCKS_EXCLUDED` with negative `EXCLUSIVE_LOCKS_REQUIRED` (Kittywhiskers Van Gogh)
cceff152ef coinjoin: replace `LOCKS_EXCLUDED` with negative `EXCLUSIVE_LOCKS_REQUIRED` (Kittywhiskers Van Gogh)
Pull request description:
## Additional Information
With the exception some usages of `cs_main` and a few (`Recursive`)`Mutex`es, Bitcoin has replaced their usage of `LOCKS_EXCLUDED(cs)` with `EXCLUSIVE_LOCKS_REQUIRED(!cs)` due to the stricter enforcement that negative locking brings with Clang (and it having a trickle-up effect caused by needing lock annotations on calling functions as well).
Dash intensively uses `LOCKS_EXCLUDED` for Dash-specific logic and moving it over also required updating (or adding) lock annotations for calling functions.
This pull request is being opened due to an upcoming pull request that includes https://github.com/bitcoin/bitcoin/pull/25109, which requires all `AssertLockNotHeld` usage to accompany a negative lock annotation.
## Breaking Changes
None expected. (Negative) lock enforcement has been made stricter but no new locks should be introduced by changes.
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
- [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
- [x] I have made corresponding changes to the documentation **(note: N/A)**
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
PastaPastaPasta:
utACK 0bba55f6af
Tree-SHA512: 54c6b80f6ad8dc821cbe9250f8e8f111bc49fb7b38ef0de10423110c066c3e7ea4b93dc61580f319837ac4690ed21912f3ec43aaa1d3cd954043798a675226ee
10869fff3b fix: order of locks cs_wallet and cs_main in rpc/evo (Konstantin Akimov)
Pull request description:
## Issue being fixed or feature implemented
Changing order of locking `cs_main` and `cs_wallet` to prevent a deadlock.
There's call `protx_list` -> `BuildDMNListEntry` -> `CheckWalletOwnsScript`:
```
return WITH_LOCK(pwallet->cs_wallet, return pwallet->IsMine(script)) == isminetype::ISMINE_SPENDABLE;
```
It can cause a deadlock due to wrong order of locks (cs_wallet supposed to be blocked in prior of cs_main)
## What was done?
This PR adds and extra lock of cs_wallet and reduce scope of cs_main for a bit
## How Has This Been Tested?
Deadlock warning is reproduced with this PR: https://github.com/dashpay/dash/pull/6003
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
Top commit has no ACKs.
Tree-SHA512: 6d166fe7c47067c1c6d889d87e015ff3bc43aa9f66234341840cc8465ce00a79e7140bc09cbdb2fd08feaae5463b320e0b66bbe410422783f86cbc9d616af6b3
```
POTENTIAL DEADLOCK DETECTED
Previous lock order was:
(2) 'cs_wallet' in wallet/wallet.cpp:3826 (in thread 'qt-init')
(2) 'pwallet->cs_wallet' in wallet/walletdb.cpp:705 (in thread 'qt-init')
(1) 'cs_KeyStore' in wallet/scriptpubkeyman.cpp:971 (in thread 'qt-init')
Current lock order is:
'cs_deqsessions' in coinjoin/client.cpp:261 (in thread 'main')
(1) 'cs_KeyStore' in wallet/scriptpubkeyman.cpp:1522 (in thread 'main')
(2) 'cs_wallet' in wallet/wallet.cpp:1629 (in thread 'main')
```
Includes:
RecursiveMutex -> Mutex,
renaming of `cs` to something more meaningful,
usage of atomics where trivially possible,
introduce a method CQuorum::SetVerificationVector to avoid needing to lock an internal mutex externally
fix: avoid cs_vvec_shShare double-lock
Co-authored-by: UdjinM6 <udjinm6@users.noreply.github.com>
