mirror of
https://github.com/dashpay/dash.git
synced 2024-12-25 12:02:48 +01:00
c172605cd7
c4a29d0a90b821c443c10891d9326c534d15cf97 Update wallet_multiwallet.py for descriptor and sqlite wallets (Russell Yanofsky)
310b0fde04639b7446efd5c1d2701caa4b991b86 Run dumpwallet for legacy wallets only in wallet_backup.py (Andrew Chow)
6c6639ac9f6e1677da066cf809f9e3fa4d2e7c32 Include sqlite3 in documentation (Andrew Chow)
f023b7cac0eb16d3c1bf40f1f7898b290de4cc73 wallet: Enforce sqlite serialized threading mode (Andrew Chow)
6173269866306058fcb1cc825b9eb681838678ca Set and check the sqlite user version (Andrew Chow)
9d3d2d263c331e3c77b8f0d01ecc9fea0407dd17 Use network magic as sqlite wallet application ID (Andrew Chow)
9af5de3798c49f86f27bb79396e075fb8c1b2381 Use SQLite for descriptor wallets (Andrew Chow)
9b78f3ce8ed1867c37f6b9fff98f74582d44b789 walletutil: Wallets can also be sqlite (Andrew Chow)
ac38a87225be0f1103ff9629d63980550d2f372b Determine wallet file type based on file magic (Andrew Chow)
6045f77003f167bee9a85e2d53f8fc6ff2e297d8 Implement SQLiteDatabase::MakeBatch (Andrew Chow)
727e6b2a4ee5abb7f2dcbc9f7778291908dc28ad Implement SQLiteDatabase::Verify (Andrew Chow)
b4df8fdb19fcded7e6d491ecf0b705cac0ec76a1 Implement SQLiteDatabase::Rewrite (Andrew Chow)
010e3659069e6f97dd7b24483f50ed71042b84b0 Implement SQLiteDatabase::TxnBegin, TxnCommit, and TxnAbort (Andrew Chow)
ac5c1617e7f4273daf24c24da1f6bc5ef5ab2d2b Implement SQLiteDatabase::Backup (Andrew Chow)
f6f9cd6a64842ef23777312f2465e826ca04b886 Implement SQLiteBatch::StartCursor, ReadAtCursor, and CloseCursor (Andrew Chow)
bf90e033f4fe86cfb90492c7e0962278ea3a146d Implement SQLiteBatch::ReadKey, WriteKey, EraseKey, and HasKey (Andrew Chow)
7aa45620e2f2178145a2eca58ccbab3cecff08fb Add SetupSQLStatements (Andrew Chow)
6636a2608a4e5906ee8092d5731595542261e0ad Implement SQLiteBatch::Close (Andrew Chow)
93825352a36456283bf87e39b5888363ee242f21 Implement SQLiteDatabase::Close (Andrew Chow)
a0de83372be83f59015cd3d61af2303b74fb64b5 Implement SQLiteDatabase::Open (Andrew Chow)
3bfa0fe1259280f8c32b41a798c9453b73f89b02 Initialize and Shutdown sqlite3 globals (Andrew Chow)
5a488b3d77326a0d957c1233493061da1b6ec207 Constructors, destructors, and relevant private fields for SQLiteDatabase/Batch (Andrew Chow)
ca8b7e04ab89f99075b093fa248919fd10acbdf7 Implement SQLiteDatabaseVersion (Andrew Chow)
7577b6e1c88a1a7b45ecf5c7f1735bae6f5a82bf Add SQLiteDatabase and SQLiteBatch dummy classes (Andrew Chow)
e87df8258090138d5c22ac46b8602b618620e8a1 Add sqlite to travis and depends (Andrew Chow)
54729f3f4e6765dfded590af5fb28c88331685f8 Add libsqlite3 (Andrew Chow)
Pull request description:
This PR adds a new class `SQLiteDatabase` which is a subclass of `WalletDatabase`. This provides access to a SQLite database that is used to store the wallet records. To keep compatibility with BDB and to complexity of the change down, we don't make use of many SQLite's features. We use it strictly as a key-value store. We create a table `main` which has two columns, `key` and `value` both with the type `blob`.
For new descriptor wallets, we will create a `SQLiteDatabase` instead of a `BerkeleyDatabase`. There is no requirement that all SQLite wallets are descriptor wallets, nor is there a requirement that all descriptor wallets be SQLite wallets. This allows for existing descriptor wallets to work as well as keeping open the option to migrate existing wallets to SQLite.
We keep the name `wallet.dat` for SQLite wallets. We are able to determine which database type to use by searching for specific magic bytes in the `wallet.dat` file. SQLite begins it's files with a null terminated string `SQLite format 3`. BDB has `0x00053162` at byte 12 (note that the byte order of this integer depends on the system endianness). So when we see that there is a `wallet.dat` file that we want to open, we check for the magic bytes to determine which database system to use.
I decided to keep the `wallet.dat` naming to keep things like backup script to continue to function as they won't need to be modified to look for a different file name. It also simplifies a couple of things in the implementation and the tests as `wallet.dat` is something that is specifically being looked for. If we don't want this behavior, then I do have another branch which creates `wallet.sqlite` files instead, but I find that this direction is easier.
ACKs for top commit:
Sjors:
re-utACK c4a29d0a90b821c443c10891d9326c534d15cf97
promag:
Tested ACK c4a29d0a90b821c443c10891d9326c534d15cf97.
fjahr:
reACK c4a29d0a90b821c443c10891d9326c534d15cf97
S3RK:
Re-review ACK c4a29d0a90b821c443c10891d9326c534d15cf97
meshcollider:
re-utACK c4a29d0a90b821c443c10891d9326c534d15cf97
hebasto:
re-ACK c4a29d0a90b821c443c10891d9326c534d15cf97, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/19077#pullrequestreview-507743699) review, verified with `git range-diff master d18892dcc c4a29d0a9`.
ryanofsky:
Code review ACK c4a29d0a90b821c443c10891d9326c534d15cf97. I am honestly confused about reasons for locking into `wallet.dat` again when it's so easy now to use a clean format. I assume I'm just very dense, or there's some unstated reason, because the only thing that's been brought up are unrealistic compatibility scenarios (all require actively creating a wallet with non-default descriptor+sqlite option, then trying to using the descriptor+sqlite wallets with old software or scripts and ignoring the results) that we didn't pay attention to with previous PRs like #11687, which did not require any active interfaction.
jonatack:
ACK c4a29d0a90b821c443c10891d9326c534d15cf97, debug builds and test runs after rebase to latest master @ c2c4dbaebd9, some manual testing creating, using, unloading and reloading a few different new sqlite descriptor wallets over several node restarts/shutdowns.
Tree-SHA512: 19145732e5001484947352d3175a660b5102bc6e833f227a55bd41b9b2f4d92737bbed7cead64b75b509decf9e1408cd81c185ab1fb4b90561aee427c4f9751c
232 lines
9.1 KiB
Python
Executable File
232 lines
9.1 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
# Copyright (c) 2014-2020 The Bitcoin Core developers
|
|
# Distributed under the MIT software license, see the accompanying
|
|
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
|
"""Test the wallet backup features.
|
|
|
|
Test case is:
|
|
4 nodes. 1 2 and 3 send transactions between each other,
|
|
fourth node is a miner.
|
|
1 2 3 each mine a block to start, then
|
|
Miner creates 100 blocks so 1 2 3 each have 500 mature
|
|
coins to spend.
|
|
Then 5 iterations of 1/2/3 sending coins amongst
|
|
themselves to get transactions in the wallets,
|
|
and the miner mining one block.
|
|
|
|
Wallets are backed up using dumpwallet/backupwallet.
|
|
Then 5 more iterations of transactions and mining a block.
|
|
|
|
Miner then generates 101 more blocks, so any
|
|
transaction fees paid mature.
|
|
|
|
Sanity check:
|
|
Sum(1,2,3,4 balances) == 114*500
|
|
|
|
1/2/3 are shutdown, and their wallets erased.
|
|
Then restore using wallet.dat backup. And
|
|
confirm 1/2/3/4 balances are same as before.
|
|
|
|
Shutdown again, restore using importwallet,
|
|
and confirm again balances are correct.
|
|
"""
|
|
from decimal import Decimal
|
|
import os
|
|
from random import randint
|
|
import shutil
|
|
|
|
from test_framework.blocktools import COINBASE_MATURITY
|
|
from test_framework.test_framework import BitcoinTestFramework
|
|
from test_framework.util import assert_equal, assert_raises_rpc_error
|
|
|
|
class WalletBackupTest(BitcoinTestFramework):
|
|
def set_test_params(self):
|
|
self.num_nodes = 4
|
|
self.setup_clean_chain = True
|
|
# nodes 1, 2,3 are spenders, let's give them a keypool=100
|
|
# whitelist all peers to speed up tx relay / mempool sync
|
|
self.extra_args = [
|
|
["-whitelist=noban@127.0.0.1", "-keypool=100"],
|
|
["-whitelist=noban@127.0.0.1", "-keypool=100"],
|
|
["-whitelist=noban@127.0.0.1", "-keypool=100"],
|
|
["-whitelist=noban@127.0.0.1"],
|
|
]
|
|
self.rpc_timeout = 120
|
|
|
|
def skip_test_if_missing_module(self):
|
|
self.skip_if_no_wallet()
|
|
|
|
def setup_network(self):
|
|
self.setup_nodes()
|
|
self.connect_nodes(0, 3)
|
|
self.connect_nodes(1, 3)
|
|
self.connect_nodes(2, 3)
|
|
self.connect_nodes(2, 0)
|
|
self.sync_all()
|
|
|
|
def one_send(self, from_node, to_address):
|
|
if (randint(1,2) == 1):
|
|
amount = Decimal(randint(1,10)) / Decimal(10)
|
|
self.nodes[from_node].sendtoaddress(to_address, amount)
|
|
|
|
def do_one_round(self):
|
|
a0 = self.nodes[0].getnewaddress()
|
|
a1 = self.nodes[1].getnewaddress()
|
|
a2 = self.nodes[2].getnewaddress()
|
|
|
|
self.one_send(0, a1)
|
|
self.one_send(0, a2)
|
|
self.one_send(1, a0)
|
|
self.one_send(1, a2)
|
|
self.one_send(2, a0)
|
|
self.one_send(2, a1)
|
|
|
|
# Have the miner (node3) mine a block.
|
|
# Must sync mempools before mining.
|
|
self.sync_mempools()
|
|
self.nodes[3].generate(1)
|
|
self.sync_blocks()
|
|
|
|
# As above, this mirrors the original bash test.
|
|
def start_three(self, args=()):
|
|
self.start_node(0, self.extra_args[0] + list(args))
|
|
self.start_node(1, self.extra_args[1] + list(args))
|
|
self.start_node(2, self.extra_args[2] + list(args))
|
|
self.connect_nodes(0, 3)
|
|
self.connect_nodes(1, 3)
|
|
self.connect_nodes(2, 3)
|
|
self.connect_nodes(2, 0)
|
|
|
|
def stop_three(self):
|
|
self.stop_node(0)
|
|
self.stop_node(1)
|
|
self.stop_node(2)
|
|
|
|
def erase_three(self):
|
|
os.remove(os.path.join(self.nodes[0].datadir, self.chain, 'wallets', self.default_wallet_name, self.wallet_data_filename))
|
|
os.remove(os.path.join(self.nodes[1].datadir, self.chain, 'wallets', self.default_wallet_name, self.wallet_data_filename))
|
|
os.remove(os.path.join(self.nodes[2].datadir, self.chain, 'wallets', self.default_wallet_name, self.wallet_data_filename))
|
|
|
|
def init_three(self):
|
|
self.init_wallet(0)
|
|
self.init_wallet(1)
|
|
self.init_wallet(2)
|
|
|
|
def run_test(self):
|
|
self.log.info("Generating initial blockchain")
|
|
self.nodes[0].generate(1)
|
|
self.sync_blocks()
|
|
self.nodes[1].generate(1)
|
|
self.sync_blocks()
|
|
self.nodes[2].generate(1)
|
|
self.sync_blocks()
|
|
self.nodes[3].generate(COINBASE_MATURITY)
|
|
self.sync_blocks()
|
|
|
|
assert_equal(self.nodes[0].getbalance(), 500)
|
|
assert_equal(self.nodes[1].getbalance(), 500)
|
|
assert_equal(self.nodes[2].getbalance(), 500)
|
|
assert_equal(self.nodes[3].getbalance(), 0)
|
|
|
|
self.log.info("Creating transactions")
|
|
# Five rounds of sending each other transactions.
|
|
for _ in range(5):
|
|
self.do_one_round()
|
|
|
|
self.log.info("Backing up")
|
|
|
|
self.nodes[0].backupwallet(os.path.join(self.nodes[0].datadir, 'wallet.bak'))
|
|
self.nodes[1].backupwallet(os.path.join(self.nodes[1].datadir, 'wallet.bak'))
|
|
self.nodes[2].backupwallet(os.path.join(self.nodes[2].datadir, 'wallet.bak'))
|
|
|
|
if not self.options.descriptors:
|
|
self.nodes[0].dumpwallet(os.path.join(self.nodes[0].datadir, 'wallet.dump'))
|
|
self.nodes[1].dumpwallet(os.path.join(self.nodes[1].datadir, 'wallet.dump'))
|
|
self.nodes[2].dumpwallet(os.path.join(self.nodes[2].datadir, 'wallet.dump'))
|
|
|
|
self.log.info("More transactions")
|
|
for _ in range(5):
|
|
self.do_one_round()
|
|
|
|
# Generate 101 more blocks, so any fees paid mature
|
|
self.nodes[3].generate(COINBASE_MATURITY + 1)
|
|
self.sync_all()
|
|
|
|
balance0 = self.nodes[0].getbalance()
|
|
balance1 = self.nodes[1].getbalance()
|
|
balance2 = self.nodes[2].getbalance()
|
|
balance3 = self.nodes[3].getbalance()
|
|
total = balance0 + balance1 + balance2 + balance3
|
|
|
|
# At this point, there are 214 blocks (103 for setup, then 10 rounds, then 101.)
|
|
# 114 are mature, so the sum of all wallets should be 114 * 500 = 57000.
|
|
assert_equal(total, 57000)
|
|
|
|
##
|
|
# Test restoring spender wallets from backups
|
|
##
|
|
self.log.info("Restoring using wallet.dat")
|
|
self.stop_three()
|
|
self.erase_three()
|
|
|
|
# Start node2 with no chain
|
|
shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'blocks'))
|
|
shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'chainstate'))
|
|
shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'evodb'))
|
|
shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'llmq'))
|
|
|
|
# Restore wallets from backup
|
|
shutil.copyfile(os.path.join(self.nodes[0].datadir, 'wallet.bak'), os.path.join(self.nodes[0].datadir, self.chain, 'wallets', self.default_wallet_name, self.wallet_data_filename))
|
|
shutil.copyfile(os.path.join(self.nodes[1].datadir, 'wallet.bak'), os.path.join(self.nodes[1].datadir, self.chain, 'wallets', self.default_wallet_name, self.wallet_data_filename))
|
|
shutil.copyfile(os.path.join(self.nodes[2].datadir, 'wallet.bak'), os.path.join(self.nodes[2].datadir, self.chain, 'wallets', self.default_wallet_name, self.wallet_data_filename))
|
|
|
|
self.log.info("Re-starting nodes")
|
|
self.start_three()
|
|
self.sync_blocks()
|
|
|
|
assert_equal(self.nodes[0].getbalance(), balance0)
|
|
assert_equal(self.nodes[1].getbalance(), balance1)
|
|
assert_equal(self.nodes[2].getbalance(), balance2)
|
|
|
|
if not self.options.descriptors:
|
|
self.log.info("Restoring using dumped wallet")
|
|
self.stop_three()
|
|
self.erase_three()
|
|
|
|
#start node2 with no chain
|
|
shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'blocks'))
|
|
shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'chainstate'))
|
|
shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'evodb'))
|
|
shutil.rmtree(os.path.join(self.nodes[2].datadir, self.chain, 'llmq'))
|
|
|
|
self.start_three(["-nowallet"])
|
|
self.init_three()
|
|
|
|
assert_equal(self.nodes[0].getbalance(), 0)
|
|
assert_equal(self.nodes[1].getbalance(), 0)
|
|
assert_equal(self.nodes[2].getbalance(), 0)
|
|
|
|
self.nodes[0].importwallet(os.path.join(self.nodes[0].datadir, 'wallet.dump'))
|
|
self.nodes[1].importwallet(os.path.join(self.nodes[1].datadir, 'wallet.dump'))
|
|
self.nodes[2].importwallet(os.path.join(self.nodes[2].datadir, 'wallet.dump'))
|
|
|
|
self.sync_blocks()
|
|
|
|
assert_equal(self.nodes[0].getbalance(), balance0)
|
|
assert_equal(self.nodes[1].getbalance(), balance1)
|
|
assert_equal(self.nodes[2].getbalance(), balance2)
|
|
|
|
# Backup to source wallet file must fail
|
|
sourcePaths = [
|
|
os.path.join(self.nodes[0].datadir, self.chain, 'wallets', self.default_wallet_name, self.wallet_data_filename),
|
|
os.path.join(self.nodes[0].datadir, self.chain, '.', 'wallets', self.default_wallet_name, self.wallet_data_filename),
|
|
os.path.join(self.nodes[0].datadir, self.chain, 'wallets', self.default_wallet_name),
|
|
os.path.join(self.nodes[0].datadir, self.chain, 'wallets')]
|
|
|
|
for sourcePath in sourcePaths:
|
|
assert_raises_rpc_error(-4, "backup failed", self.nodes[0].backupwallet, sourcePath)
|
|
|
|
|
|
if __name__ == '__main__':
|
|
WalletBackupTest().main()
|