dash/test/functional/wallet_importdescriptors.py

457 lines
25 KiB
Python
Raw Normal View History

Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
#!/usr/bin/env python3
# Copyright (c) 2019-2020 The Bitcoin Core developers
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""Test the importdescriptors RPC.
Test importdescriptors by generating keys on node0, importing the corresponding
descriptors on node1 and then testing the address info for the different address
variants.
- `get_generate_key()` is called to generate keys and return the privkeys,
pubkeys and all variants of scriptPubKey and address.
- `test_importdesc()` is called to send an importdescriptors call to node1, test
success, and (if unsuccessful) test the error code and error message returned.
- `test_address()` is called to call getaddressinfo for an address on node1
and test the values returned."""
from test_framework.address import key_to_p2pkh
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
from test_framework.test_framework import BitcoinTestFramework
from test_framework.descriptors import descsum_create
from test_framework.util import (
assert_equal,
assert_raises_rpc_error,
find_vout_for_address,
)
from test_framework.wallet_util import (
get_generate_key,
test_address,
)
class ImportDescriptorsTest(BitcoinTestFramework):
def set_test_params(self):
self.num_nodes = 2
self.extra_args = [[],
["-keypool=5"]
]
self.setup_clean_chain = True
Merge #18788: tests: Update more tests to work with descriptor wallets c7b7e0a69265946aecc885be911c7650911ba2e3 tests: Make only desc wallets for wallet_multwallet.py --descriptors (Andrew Chow) d4b67ad214ada7645c4ce2d5ec336fe5c3f7f7ca Avoid creating legacy wallets in wallet_importdescriptors.py (Andrew Chow) 6c9c12bf87f95066acc28ea2270a00196eb77703 Update feature_backwards_compatibility for descriptor wallets (Andrew Chow) 9a4c631e1c00eb1661c000978b133d7aa0226290 Update wallet_labels.py to not require descriptors=False (Andrew Chow) 242aed7cc1d003e8fed574bbebd19c7e54e23402 tests: Add a --legacy-wallet that is mutually exclusive with --descriptors (Andrew Chow) 388053e1722632c2e485c56a444bc75cf0152188 Disable some tests for tool_wallet when descriptors (Andrew Chow) 47d3243160fdec7e464cfb8f869be7f5d4ee25fe Make raw multisig tests legacy wallet only in rpc_rawtransaction.py (Andrew Chow) 59d3da5bce4ebd9c2291d8f201a53ee087938b21 Do addmultisigaddress tests in legacy wallet mode in wallet_address_types.py (Andrew Chow) 25bc5dccbfd52691adca6edd418dd54290300c28 Use importdescriptors when in descriptor wallet mode in wallet_createwallet.py (Andrew Chow) 0bd1860300b13b12a25d330ba3a93ff2d13aa379 Avoid dumpprivkey and watchonly behavior in rpc_signrawtransaction.py (Andrew Chow) 08067aebfd7e838e6ce6b030c31a69422260fc6f Add script equivalent of functions in address.py (Andrew Chow) 86968882a8a26312a7af29c572313c4aff488c11 Add descriptor wallet output to tool_wallet.py (Andrew Chow) 3457679870e8eff2a7d14fe59a479692738c48b6 Use separate watchonly wallet for multisig in feature_nulldummy.py (Andrew Chow) a42652ec10c733a5bf37e418e45d4841f54331b4 Move import and watchonly tests to be legacy wallet only in wallet_balance.py (Andrew Chow) 4b871909d6e4a51888e062d322bf53263deda15e Use importdescriptors for descriptor wallets in wallet_bumpfee.py (Andrew Chow) c2711e4230d9a423ead24f6609691fb338b1d26b Avoid dumpprivkey in wallet_listsinceblock.py (Andrew Chow) 553dbf9af4dea96e6a3e79bba9607003342029bd Make import tests in wallet_listtransactions.py legacy wallet only (Andrew Chow) dc81418fd01021070f3f66bab5fee1484456691a Use a separate watchonly wallet in rpc_fundrawtransaction.py (Andrew Chow) a357111047411f18c156cd34a002a38430f2901c Update wallet_importprunedfunds to avoid dumpprivkey (Andrew Chow) Pull request description: I went through all the tests and checked whether they passed with descriptor wallets. This partially informed some changes in #16528. Some tests needed changes to work with descriptor wallets. These were primarily due to import and watchonly behavior. There are some tests and test cases that only test legacy wallet behavior so those tests won't be run with descriptor wallets. This PR updates more tests to have to the `--descriptors` switch in `test_runner.py`. Additionally a mutually exclusive `--legacy-wallet` option has been added to force legacy wallets. This does nothing currently but will be useful in the future when descriptor wallets are the default. For the tests that rely on legacy wallet behavior, this option is being set so that we don't forget in the future. Those tests are `feature_segwit.py`, `wallet_watchonly.py`, `wallet_implicitsegwit.py`, `wallet_import_with_label.py`, and `wallet_import_with_label.py`. If you invert the `--descriptors`/`--legacy-wallet` default so that descriptor wallets are the default, all tests (besides the legacy wallet specific ones) will pass. ACKs for top commit: MarcoFalke: review ACK c7b7e0a69265946aecc885be911c7650911ba2e3 🎿 laanwj: ACK c7b7e0a69265946aecc885be911c7650911ba2e3 Tree-SHA512: 2f4e87815005d1d0a2543ea7947f7cd7593d8cf5312228ef85f8e096f19739b225769961943049cb44f6f07a35b8de988e2246ab9aca5bb5a0b2e62694d5637d
2020-11-02 16:54:06 +01:00
self.wallet_names = []
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
def skip_test_if_missing_module(self):
self.skip_if_no_wallet()
self.skip_if_no_sqlite()
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
def test_importdesc(self, req, success, error_code=None, error_message=None, warnings=None, wallet=None):
"""Run importdescriptors and assert success"""
if warnings is None:
warnings = []
wrpc = self.nodes[1].get_wallet_rpc('w1')
if wallet is not None:
wrpc = wallet
result = wrpc.importdescriptors([req])
observed_warnings = []
if 'warnings' in result[0]:
observed_warnings = result[0]['warnings']
assert_equal("\n".join(sorted(warnings)), "\n".join(sorted(observed_warnings)))
assert_equal(result[0]['success'], success)
if error_code is not None:
assert_equal(result[0]['error']['code'], error_code)
assert_equal(result[0]['error']['message'], error_message)
def run_test(self):
self.log.info('Setting up wallets')
Merge #18788: tests: Update more tests to work with descriptor wallets c7b7e0a69265946aecc885be911c7650911ba2e3 tests: Make only desc wallets for wallet_multwallet.py --descriptors (Andrew Chow) d4b67ad214ada7645c4ce2d5ec336fe5c3f7f7ca Avoid creating legacy wallets in wallet_importdescriptors.py (Andrew Chow) 6c9c12bf87f95066acc28ea2270a00196eb77703 Update feature_backwards_compatibility for descriptor wallets (Andrew Chow) 9a4c631e1c00eb1661c000978b133d7aa0226290 Update wallet_labels.py to not require descriptors=False (Andrew Chow) 242aed7cc1d003e8fed574bbebd19c7e54e23402 tests: Add a --legacy-wallet that is mutually exclusive with --descriptors (Andrew Chow) 388053e1722632c2e485c56a444bc75cf0152188 Disable some tests for tool_wallet when descriptors (Andrew Chow) 47d3243160fdec7e464cfb8f869be7f5d4ee25fe Make raw multisig tests legacy wallet only in rpc_rawtransaction.py (Andrew Chow) 59d3da5bce4ebd9c2291d8f201a53ee087938b21 Do addmultisigaddress tests in legacy wallet mode in wallet_address_types.py (Andrew Chow) 25bc5dccbfd52691adca6edd418dd54290300c28 Use importdescriptors when in descriptor wallet mode in wallet_createwallet.py (Andrew Chow) 0bd1860300b13b12a25d330ba3a93ff2d13aa379 Avoid dumpprivkey and watchonly behavior in rpc_signrawtransaction.py (Andrew Chow) 08067aebfd7e838e6ce6b030c31a69422260fc6f Add script equivalent of functions in address.py (Andrew Chow) 86968882a8a26312a7af29c572313c4aff488c11 Add descriptor wallet output to tool_wallet.py (Andrew Chow) 3457679870e8eff2a7d14fe59a479692738c48b6 Use separate watchonly wallet for multisig in feature_nulldummy.py (Andrew Chow) a42652ec10c733a5bf37e418e45d4841f54331b4 Move import and watchonly tests to be legacy wallet only in wallet_balance.py (Andrew Chow) 4b871909d6e4a51888e062d322bf53263deda15e Use importdescriptors for descriptor wallets in wallet_bumpfee.py (Andrew Chow) c2711e4230d9a423ead24f6609691fb338b1d26b Avoid dumpprivkey in wallet_listsinceblock.py (Andrew Chow) 553dbf9af4dea96e6a3e79bba9607003342029bd Make import tests in wallet_listtransactions.py legacy wallet only (Andrew Chow) dc81418fd01021070f3f66bab5fee1484456691a Use a separate watchonly wallet in rpc_fundrawtransaction.py (Andrew Chow) a357111047411f18c156cd34a002a38430f2901c Update wallet_importprunedfunds to avoid dumpprivkey (Andrew Chow) Pull request description: I went through all the tests and checked whether they passed with descriptor wallets. This partially informed some changes in #16528. Some tests needed changes to work with descriptor wallets. These were primarily due to import and watchonly behavior. There are some tests and test cases that only test legacy wallet behavior so those tests won't be run with descriptor wallets. This PR updates more tests to have to the `--descriptors` switch in `test_runner.py`. Additionally a mutually exclusive `--legacy-wallet` option has been added to force legacy wallets. This does nothing currently but will be useful in the future when descriptor wallets are the default. For the tests that rely on legacy wallet behavior, this option is being set so that we don't forget in the future. Those tests are `feature_segwit.py`, `wallet_watchonly.py`, `wallet_implicitsegwit.py`, `wallet_import_with_label.py`, and `wallet_import_with_label.py`. If you invert the `--descriptors`/`--legacy-wallet` default so that descriptor wallets are the default, all tests (besides the legacy wallet specific ones) will pass. ACKs for top commit: MarcoFalke: review ACK c7b7e0a69265946aecc885be911c7650911ba2e3 🎿 laanwj: ACK c7b7e0a69265946aecc885be911c7650911ba2e3 Tree-SHA512: 2f4e87815005d1d0a2543ea7947f7cd7593d8cf5312228ef85f8e096f19739b225769961943049cb44f6f07a35b8de988e2246ab9aca5bb5a0b2e62694d5637d
2020-11-02 16:54:06 +01:00
self.nodes[0].createwallet(wallet_name='w0', disable_private_keys=False, descriptors=True)
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
w0 = self.nodes[0].get_wallet_rpc('w0')
self.nodes[1].createwallet(wallet_name='w1', disable_private_keys=True, blank=True, descriptors=True)
w1 = self.nodes[1].get_wallet_rpc('w1')
assert_equal(w1.getwalletinfo()['keypoolsize'], 0)
self.nodes[1].createwallet(wallet_name="wpriv", disable_private_keys=False, blank=True, descriptors=True)
wpriv = self.nodes[1].get_wallet_rpc("wpriv")
assert_equal(wpriv.getwalletinfo()['keypoolsize'], 0)
self.log.info('Mining coins')
w0.generatetoaddress(101, w0.getnewaddress())
# RPC importdescriptors -----------------------------------------------
# # Test import fails if no descriptor present
key = get_generate_key()
self.log.info("Import should fail if a descriptor is not provided")
self.test_importdesc({"timestamp": "now"},
success=False,
error_code=-8,
error_message='Descriptor not found.')
# # Test importing of a P2PKH descriptor
key = get_generate_key()
self.log.info("Should import a p2pkh descriptor")
self.test_importdesc({"desc": descsum_create("pkh(" + key.pubkey + ")"),
"timestamp": "now",
"label": "Descriptor import test"},
success=True)
test_address(w1,
key.p2pkh_addr,
solvable=True,
ismine=True,
labels=["Descriptor import test"])
assert_equal(w1.getwalletinfo()['keypoolsize'], 0)
# Check persistence of data and that loading works correctly
w1.unloadwallet()
self.nodes[1].loadwallet('w1')
test_address(w1,
key.p2pkh_addr,
solvable=True,
ismine=True,
labels=["Descriptor import test"])
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
self.log.info("Internal addresses cannot have labels")
self.test_importdesc({"desc": descsum_create("pkh(" + key.pubkey + ")"),
"timestamp": "now",
"internal": True,
"label": "Descriptor import test"},
success=False,
error_code=-8,
error_message="Internal addresses should not have a label")
self.log.info("Internal addresses should be detected as such")
key = get_generate_key()
addr = key_to_p2pkh(key.pubkey)
self.test_importdesc({"desc": descsum_create("pkh(" + key.pubkey + ")"),
"timestamp": "now",
"internal": True},
success=True)
info = w1.getaddressinfo(addr)
assert_equal(info["ismine"], True)
assert_equal(info["ischange"], True)
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
# # Test importing of a P2SH-P2PKH descriptor
key = get_generate_key()
self.log.info("Should not import a p2sh-p2pkh descriptor without checksum")
self.test_importdesc({"desc": "sh(pkh(" + key.pubkey + "))",
"timestamp": "now"
},
success=False,
error_code=-5,
error_message="Missing checksum")
self.log.info("Should not import a p2sh-p2pkh descriptor that has range specified")
self.test_importdesc({"desc": descsum_create("sh(pkh(" + key.pubkey + "))"),
"timestamp": "now",
"range": 1,
},
success=False,
error_code=-8,
error_message="Range should not be specified for an un-ranged descriptor")
self.log.info("Should not import a p2sh-p2pkh descriptor and have it set to active")
self.test_importdesc({"desc": descsum_create("sh(pkh(" + key.pubkey + "))"),
"timestamp": "now",
"active": True,
},
success=False,
error_code=-8,
error_message="Active descriptors must be ranged")
self.log.info("Should import a (non-active) p2sh-p2pkh descriptor")
self.test_importdesc({"desc": descsum_create("sh(pkh(" + key.pubkey + "))"),
"timestamp": "now",
"active": False,
},
success=True)
assert_equal(w1.getwalletinfo()['keypoolsize'], 0)
# # Test importing of a multisig descriptor
key1 = get_generate_key()
key2 = get_generate_key()
self.log.info("Should import a 1-of-2 bare multisig from descriptor")
self.test_importdesc({"desc": descsum_create("multi(1," + key1.pubkey + "," + key2.pubkey + ")"),
"timestamp": "now"},
success=True)
self.log.info("Should not treat individual keys from the imported bare multisig as watchonly")
test_address(w1,
key1.p2pkh_addr,
ismine=False)
# # Test ranged descriptors
xpriv = "tprv8ZgxMBicQKsPeuVhWwi6wuMQGfPKi9Li5GtX35jVNknACgqe3CY4g5xgkfDDJcmtF7o1QnxWDRYw4H5P26PXq7sbcUkEqeR4fg3Kxp2tigg"
xpub = "tpubDAXcJ7s7ZwicqjprRaEWdPoHKrCS215qxGYxpusRLLmJuT69ZSicuGdSfyvyKpvUNYBW1s2U3NSrT6vrCYB9e6nZUEvrqnwXPF8ArTCRXMY"
addresses = ["yUxX4qnzWntXhEGrYB92v7ez4EZBnUjB1y", "yRhTPsPd2qYgYbFFCqY2nuPHJQBjTnMQxg"] # hdkeypath=m/0'/0'/0' and 1'
desc = "sh(pkh(" + xpub + "/0/0/*" + "))"
self.log.info("Ranged descriptors cannot have labels")
self.test_importdesc({"desc":descsum_create(desc),
"timestamp": "now",
"range": [0, 100],
"label": "test"},
success=False,
error_code=-8,
error_message='Ranged descriptors should not have a label')
self.log.info("Private keys required for private keys enabled wallet")
self.test_importdesc({"desc":descsum_create(desc),
"timestamp": "now",
"range": [0, 100]},
success=False,
error_code=-4,
error_message='Cannot import descriptor without private keys to a wallet with private keys enabled',
wallet=wpriv)
self.log.info("Ranged descriptor import should warn without a specified range")
self.test_importdesc({"desc": descsum_create(desc),
"timestamp": "now"},
success=True,
warnings=['Range not given, using default keypool range'])
assert_equal(w1.getwalletinfo()['keypoolsize'], 0)
# # Test importing of a ranged descriptor with xpriv
self.log.info("Should not import a ranged descriptor that includes xpriv into a watch-only wallet")
desc = "sh(pkh(" + xpriv + "/0'/0'/*'" + "))"
self.test_importdesc({"desc": descsum_create(desc),
"timestamp": "now",
"range": 1},
success=False,
error_code=-4,
error_message='Cannot import private keys to a wallet with private keys disabled')
self.log.info("Should not import a descriptor with hardened derivations when private keys are disabled")
self.test_importdesc({"desc": descsum_create("pkh(" + xpub + "/1h/*)"),
"timestamp": "now",
"range": 1},
success=False,
error_code=-4,
error_message='Cannot expand descriptor. Probably because of hardened derivations without private keys provided')
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
for address in addresses:
test_address(w1,
address,
ismine=False,
solvable=False)
self.test_importdesc({"desc": descsum_create(desc), "timestamp": "now", "range": -1},
success=False, error_code=-8, error_message='End of range is too high')
self.test_importdesc({"desc": descsum_create(desc), "timestamp": "now", "range": [-1, 10]},
success=False, error_code=-8, error_message='Range should be greater or equal than 0')
self.test_importdesc({"desc": descsum_create(desc), "timestamp": "now", "range": [(2 << 31 + 1) - 1000000, (2 << 31 + 1)]},
success=False, error_code=-8, error_message='End of range is too high')
self.test_importdesc({"desc": descsum_create(desc), "timestamp": "now", "range": [2, 1]},
success=False, error_code=-8, error_message='Range specified as [begin,end] must not have begin after end')
self.test_importdesc({"desc": descsum_create(desc), "timestamp": "now", "range": [0, 1000001]},
success=False, error_code=-8, error_message='Range is too large')
# Make sure ranged imports import keys in order
w1 = self.nodes[1].get_wallet_rpc('w1')
self.log.info('Key ranges should be imported in order')
xpub = "tpubDAXcJ7s7ZwicqjprRaEWdPoHKrCS215qxGYxpusRLLmJuT69ZSicuGdSfyvyKpvUNYBW1s2U3NSrT6vrCYB9e6nZUEvrqnwXPF8ArTCRXMY"
addresses = [
'yUxX4qnzWntXhEGrYB92v7ez4EZBnUjB1y', # m/0'/0'/0
'yRhTPsPd2qYgYbFFCqY2nuPHJQBjTnMQxg', # m/0'/0'/1
'yUyn3UV9rBdWfw6yJJ6eAoKuzDJ8RVLP1o', # m/0'/0'/2
'yi8GEkfLBgK85wGmBFsMFdSbEvPPNCSnVx', # m/0'/0'/3
'yYB4whdY8APWoCez6ryNdMBrrDjwzFbqMi', # m/0'/0'/4
]
self.test_importdesc({'desc': descsum_create('pkh([12345678/0h/0h]' + xpub + '/*)'),
'active': True,
'range' : [0, 2],
'timestamp': 'now'
},
success=True)
assert_equal(w1.getwalletinfo()['keypoolsize'], 5)
for i, expected_addr in enumerate(addresses):
received_addr = w1.getnewaddress('')
assert_raises_rpc_error(-4, 'This wallet has no available keys', w1.getrawchangeaddress)
assert_equal(received_addr, expected_addr)
pkh_addr = received_addr
pkh_addr_info = w1.getaddressinfo(pkh_addr)
assert_equal(pkh_addr_info['desc'][:22], 'pkh([12345678/0\'/0\'/{}]'.format(i))
assert_equal(w1.getwalletinfo()['keypoolsize'], 4) # After retrieving a key, we don't refill the keypool again, so it's one less for each address type
w1.keypoolrefill()
assert_equal(w1.getwalletinfo()['keypoolsize'], 5 )
# Check active=False default
self.log.info('Check imported descriptors are not active by default')
self.test_importdesc({'desc': descsum_create('pkh([12345678/0h/0h]' + xpub + '/*)'),
'range' : [0, 2],
'timestamp': 'now',
'internal': True
},
success=True)
assert_raises_rpc_error(-4, 'This wallet has no available keys', w1.getrawchangeaddress)
# # Test importing a descriptor containing a WIF private key
wif_priv = "cTT3BvHnd51YJf8fkdr2XvZTQRRUZruWhRvRyQY1raVFg5Lvam2A"
address = "ySWABbcNKyHUgBb1ffhpuETuis9jsdR3aq"
desc = "pkh(" + wif_priv + ")"
self.log.info("Should import a descriptor with a WIF private key as spendable")
self.test_importdesc({"desc": descsum_create(desc),
"timestamp": "now"},
success=True,
wallet=wpriv)
test_address(wpriv,
address,
solvable=True,
ismine=True)
txid = w0.sendtoaddress(address, 49.99995540)
w0.generatetoaddress(6, w0.getnewaddress())
self.sync_blocks()
tx = wpriv.createrawtransaction([{"txid": txid, "vout": 0}], {w0.getnewaddress(): 49.999})
signed_tx = wpriv.signrawtransactionwithwallet(tx)
w1.sendrawtransaction(signed_tx['hex'])
# Make sure that we can use import and use multisig as addresses
self.log.info('Test that multisigs can be imported, signed for, and getnewaddress\'d')
self.nodes[1].createwallet(wallet_name="wmulti_priv", disable_private_keys=False, blank=True, descriptors=True)
wmulti_priv = self.nodes[1].get_wallet_rpc("wmulti_priv")
assert_equal(wmulti_priv.getwalletinfo()['keypoolsize'], 0)
self.test_importdesc({"desc":"sh(multi(2,tprv8ZgxMBicQKsPevADjDCWsa6DfhkVXicu8NQUzfibwX2MexVwW4tCec5mXdCW8kJwkzBRRmAay1KZya4WsehVvjTGVW6JLqiqd8DdZ4xSg52/84h/0h/0h/*,tprv8ZgxMBicQKsPdSNWUhDiwTScDr6JfkZuLshTRwzvZGnMSnGikV6jxpmdDkC3YRc4T3GD6Nvg9uv6hQg73RVv1EiTXDZwxVbsLugVHU8B1aq/84h/0h/0h/*,tprv8ZgxMBicQKsPeonDt8Ka2mrQmHa61hQ5FQCsvWBTpSNzBFgM58cV2EuXNAHF14VawVpznnme3SuTbA62sGriwWyKifJmXntfNeK7zeqMCj1/84h/0h/0h/*))#f5nqn4ax",
"active": True,
"range": 1000,
"next_index": 0,
"timestamp": "now"},
success=True,
wallet=wmulti_priv)
self.test_importdesc({"desc":"sh(multi(2,tprv8ZgxMBicQKsPevADjDCWsa6DfhkVXicu8NQUzfibwX2MexVwW4tCec5mXdCW8kJwkzBRRmAay1KZya4WsehVvjTGVW6JLqiqd8DdZ4xSg52/84h/1h/0h/*,tprv8ZgxMBicQKsPdSNWUhDiwTScDr6JfkZuLshTRwzvZGnMSnGikV6jxpmdDkC3YRc4T3GD6Nvg9uv6hQg73RVv1EiTXDZwxVbsLugVHU8B1aq/84h/1h/0h/*,tprv8ZgxMBicQKsPeonDt8Ka2mrQmHa61hQ5FQCsvWBTpSNzBFgM58cV2EuXNAHF14VawVpznnme3SuTbA62sGriwWyKifJmXntfNeK7zeqMCj1/84h/1h/0h/*))#m4e4s5de",
"active": True,
"internal" : True,
"range": 1000,
"next_index": 0,
"timestamp": "now"},
success=True,
wallet=wmulti_priv)
assert_equal(wmulti_priv.getwalletinfo()['keypoolsize'], 1001) # Range end (1000) is inclusive, so 1001 addresses generated
addr = wmulti_priv.getnewaddress() # uses receive 0
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
assert_equal(addr, '8vEwYGKBMP3F2juEE36nNqh1uYpBv9QFyB') # Derived at m/84'/0'/0'/0
change_addr = wmulti_priv.getrawchangeaddress()
assert_equal(change_addr, '91WxMwg2NHD1PwHChhbAkeCN6nQ8ikdLEx')
assert_equal(wmulti_priv.getwalletinfo()['keypoolsize'], 1000)
txid = w0.sendtoaddress(addr, 10)
self.nodes[0].generate(6)
self.sync_all()
wmulti_priv.sendtoaddress(w0.getnewaddress(), 8) # uses change 1
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
self.nodes[0].generate(6)
self.sync_all()
self.nodes[1].createwallet(wallet_name="wmulti_pub", disable_private_keys=True, blank=True, descriptors=True)
wmulti_pub = self.nodes[1].get_wallet_rpc("wmulti_pub")
assert_equal(wmulti_pub.getwalletinfo()['keypoolsize'], 0)
self.test_importdesc({"desc":"sh(multi(2,[7b2d0242/84h/0h/0h]tpubDCJtdt5dgJpdhW4MtaVYDhG4T4tF6jcLR1PxL43q9pq1mxvXgMS9Mzw1HnXG15vxUGQJMMSqCQHMTy3F1eW5VkgVroWzchsPD5BUojrcWs8/*,[59b09cd6/84h/0h/0h]tpubDDBF2BTR6s8drwrfDei8WxtckGuSm1cyoKxYY1QaKSBFbHBYQArWhHPA6eJrzZej6nfHGLSURYSLHr7GuYch8aY5n61tGqgn8b4cXrMuoPH/*,[e81a0532/84h/0h/0h]tpubDCsWoW1kuQB9kG5MXewHqkbjPtqPueRnXju7uM2NK7y3JYb2ajAZ9EiuZXNNuE4661RAfriBWhL8UsnAPpk8zrKKnZw1Ug7X4oHgMdZiU4E/*))#x75vpsak",
"active": True,
"range": 1000,
"next_index": 0,
"timestamp": "now"},
success=True,
wallet=wmulti_pub)
self.test_importdesc({"desc":"sh(multi(2,[7b2d0242/84h/1h/0h]tpubDCXqdwWZcszwqYJSnZp8eARkxGJfHAk23KDxbztV4BbschfaTfYLTcSkSJ3TN64dRqwa1rnFUScsYormKkGqNbbPwkorQimVevXjxzUV9Gf/*,[59b09cd6/84h/1h/0h]tpubDCYfZY2ceyHzYzMMVPt9MNeiqtQ2T7Uyp9QSFwYXh8Vi9iJFYXcuphJaGXfF3jUQJi5Y3GMNXvM11gaL4txzZgNGK22BFAwMXynnzv4z2Jh/*,[e81a0532/84h/1h/0h]tpubDC6UGqnsQStngYuGD4MKsMy7eD1Yg9NTJfPdvjdG2JE5oZ7EsSL3WHg4Gsw2pR5K39ZwJ46M1wZayhedVdQtMGaUhq5S23PH6fnENK3V1sb/*))#v0t48ucu",
"active": True,
"internal" : True,
"range": 1000,
"next_index": 0,
"timestamp": "now"},
success=True,
wallet=wmulti_pub)
assert_equal(wmulti_pub.getwalletinfo()['keypoolsize'], 1000) # The first one was already consumed by previous import and is detected as used
addr = wmulti_pub.getnewaddress() # uses receive 1
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
assert_equal(addr, '91cA4fLGaDCr6b9W2c5j1ph9PDpq9WbEhk') # Derived at m/84'/0'/0'/1
change_addr = wmulti_pub.getrawchangeaddress() # uses receive 2
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
assert_equal(change_addr, '91WxMwg2NHD1PwHChhbAkeCN6nQ8ikdLEx')
assert_equal(wmulti_pub.getwalletinfo()['keypoolsize'], 999)
txid = w0.sendtoaddress(addr, 10)
vout = find_vout_for_address(self.nodes[0], txid, addr)
self.nodes[0].generate(6)
self.sync_all()
assert_equal(wmulti_pub.getbalance(), wmulti_priv.getbalance())
# Make sure that descriptor wallets containing multiple xpubs in a single descriptor load correctly
wmulti_pub.unloadwallet()
self.nodes[1].loadwallet('wmulti_pub')
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
self.log.info("Multisig with distributed keys")
self.nodes[1].createwallet(wallet_name="wmulti_priv1", descriptors=True)
wmulti_priv1 = self.nodes[1].get_wallet_rpc("wmulti_priv1")
res = wmulti_priv1.importdescriptors([
{
"desc": descsum_create("sh(multi(2,tprv8ZgxMBicQKsPevADjDCWsa6DfhkVXicu8NQUzfibwX2MexVwW4tCec5mXdCW8kJwkzBRRmAay1KZya4WsehVvjTGVW6JLqiqd8DdZ4xSg52/84h/0h/0h/*,[59b09cd6/84h/0h/0h]tpubDDBF2BTR6s8drwrfDei8WxtckGuSm1cyoKxYY1QaKSBFbHBYQArWhHPA6eJrzZej6nfHGLSURYSLHr7GuYch8aY5n61tGqgn8b4cXrMuoPH/*,[e81a0532/84h/0h/0h]tpubDCsWoW1kuQB9kG5MXewHqkbjPtqPueRnXju7uM2NK7y3JYb2ajAZ9EiuZXNNuE4661RAfriBWhL8UsnAPpk8zrKKnZw1Ug7X4oHgMdZiU4E/*))"),
"active": True,
"range": 1000,
"next_index": 0,
"timestamp": "now"
},
{
"desc": descsum_create("sh(multi(2,tprv8ZgxMBicQKsPevADjDCWsa6DfhkVXicu8NQUzfibwX2MexVwW4tCec5mXdCW8kJwkzBRRmAay1KZya4WsehVvjTGVW6JLqiqd8DdZ4xSg52/84h/1h/0h/*,[59b09cd6/84h/1h/0h]tpubDCYfZY2ceyHzYzMMVPt9MNeiqtQ2T7Uyp9QSFwYXh8Vi9iJFYXcuphJaGXfF3jUQJi5Y3GMNXvM11gaL4txzZgNGK22BFAwMXynnzv4z2Jh/*,[e81a0532/84h/1h/0h]tpubDC6UGqnsQStngYuGD4MKsMy7eD1Yg9NTJfPdvjdG2JE5oZ7EsSL3WHg4Gsw2pR5K39ZwJ46M1wZayhedVdQtMGaUhq5S23PH6fnENK3V1sb/*))"),
"active": True,
"internal" : True,
"range": 1000,
"next_index": 0,
"timestamp": "now"
}])
assert_equal(res[0]['success'], True)
assert_equal(res[0]['warnings'][0], 'Not all private keys provided. Some wallet functionality may return unexpected errors')
assert_equal(res[1]['success'], True)
assert_equal(res[1]['warnings'][0], 'Not all private keys provided. Some wallet functionality may return unexpected errors')
self.nodes[1].createwallet(wallet_name='wmulti_priv2', blank=True, descriptors=True)
wmulti_priv2 = self.nodes[1].get_wallet_rpc('wmulti_priv2')
res = wmulti_priv2.importdescriptors([
{
"desc": descsum_create("sh(multi(2,[7b2d0242/84h/0h/0h]tpubDCJtdt5dgJpdhW4MtaVYDhG4T4tF6jcLR1PxL43q9pq1mxvXgMS9Mzw1HnXG15vxUGQJMMSqCQHMTy3F1eW5VkgVroWzchsPD5BUojrcWs8/*,tprv8ZgxMBicQKsPdSNWUhDiwTScDr6JfkZuLshTRwzvZGnMSnGikV6jxpmdDkC3YRc4T3GD6Nvg9uv6hQg73RVv1EiTXDZwxVbsLugVHU8B1aq/84h/0h/0h/*,[e81a0532/84h/0h/0h]tpubDCsWoW1kuQB9kG5MXewHqkbjPtqPueRnXju7uM2NK7y3JYb2ajAZ9EiuZXNNuE4661RAfriBWhL8UsnAPpk8zrKKnZw1Ug7X4oHgMdZiU4E/*))"),
"active": True,
"range": 1000,
"next_index": 0,
"timestamp": "now"
},
{
"desc": descsum_create("sh(multi(2,[7b2d0242/84h/1h/0h]tpubDCXqdwWZcszwqYJSnZp8eARkxGJfHAk23KDxbztV4BbschfaTfYLTcSkSJ3TN64dRqwa1rnFUScsYormKkGqNbbPwkorQimVevXjxzUV9Gf/*,tprv8ZgxMBicQKsPdSNWUhDiwTScDr6JfkZuLshTRwzvZGnMSnGikV6jxpmdDkC3YRc4T3GD6Nvg9uv6hQg73RVv1EiTXDZwxVbsLugVHU8B1aq/84h/1h/0h/*,[e81a0532/84h/1h/0h]tpubDC6UGqnsQStngYuGD4MKsMy7eD1Yg9NTJfPdvjdG2JE5oZ7EsSL3WHg4Gsw2pR5K39ZwJ46M1wZayhedVdQtMGaUhq5S23PH6fnENK3V1sb/*))"),
"active": True,
"internal" : True,
"range": 1000,
"next_index": 0,
"timestamp": "now"
}])
assert_equal(res[0]['success'], True)
assert_equal(res[0]['warnings'][0], 'Not all private keys provided. Some wallet functionality may return unexpected errors')
assert_equal(res[1]['success'], True)
assert_equal(res[1]['warnings'][0], 'Not all private keys provided. Some wallet functionality may return unexpected errors')
rawtx = self.nodes[1].createrawtransaction([{'txid': txid, 'vout': vout}], {w0.getnewaddress(): 9.999})
tx_signed_1 = wmulti_priv1.signrawtransactionwithwallet(rawtx)
assert_equal(tx_signed_1['complete'], False)
tx_signed_2 = wmulti_priv2.signrawtransactionwithwallet(tx_signed_1['hex'])
assert_equal(tx_signed_2['complete'], True)
self.nodes[1].sendrawtransaction(tx_signed_2['hex'])
self.log.info("Combo descriptors cannot be active")
self.test_importdesc({"desc": descsum_create("combo(tpubDCJtdt5dgJpdhW4MtaVYDhG4T4tF6jcLR1PxL43q9pq1mxvXgMS9Mzw1HnXG15vxUGQJMMSqCQHMTy3F1eW5VkgVroWzchsPD5BUojrcWs8/*)"),
"active": True,
"range": 1,
"timestamp": "now"},
success=False,
error_code=-4,
error_message="Combo descriptors cannot be set to active")
# we have only one type of descriptors so we can't trigger this warning
self.log.info("Descriptors with no type cannot be active")
self.test_importdesc({"desc": descsum_create("pk(tpubDCJtdt5dgJpdhW4MtaVYDhG4T4tF6jcLR1PxL43q9pq1mxvXgMS9Mzw1HnXG15vxUGQJMMSqCQHMTy3F1eW5VkgVroWzchsPD5BUojrcWs8/*)"),
"active": True,
"range": 1,
"timestamp": "now"},
success=True,
#warnings=["Unknown output type, cannot set descriptor to active."]
)
if __name__ == '__main__':
ImportDescriptorsTest().main()