dash/test/functional/test_runner.py

829 lines
33 KiB
Python
Raw Normal View History

#!/usr/bin/env python3
# Copyright (c) 2014-2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""Run regression test suite.
This module calls down into individual test cases via subprocess. It will
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
forward all unrecognized arguments onto the individual test scripts.
For a description of arguments recognized by test scripts, see
`test/functional/test_framework/test_framework.py:BitcoinTestFramework.main`.
"""
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
import argparse
2017-11-29 19:21:51 +01:00
from collections import deque
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
import configparser
import datetime
import os
import time
import shutil
import signal
import subprocess
import sys
import tempfile
import re
import logging
import unittest
# Formatting. Default colors to empty strings.
DEFAULT, BOLD, GREEN, RED = ("", ""), ("", ""), ("", ""), ("", "")
try:
# Make sure python thinks it can write unicode to its stdout
"\u2713".encode("utf_8").decode(sys.stdout.encoding)
TICK = ""
CROSS = ""
CIRCLE = ""
except UnicodeDecodeError:
TICK = "P "
CROSS = "x "
CIRCLE = "o "
if os.name != 'nt' or sys.getwindowsversion() >= (10, 0, 14393):
if os.name == 'nt':
import ctypes
Merge #18210: test: type hints in Python tests bd7e530f010d43816bb05d6f1590d1cd36cdaa2c This PR adds initial support for type hints checking in python scripts. (Kiminuo) Pull request description: This PR adds initial support for type hints checking in python scripts. Support for type hints was introduced in Python 3.5. Type hints make it easier to read and review code in my opinion. Also an IDE may discover a potential bug sooner. Yet, as PEP 484 says: "It should also be emphasized that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention." [Mypy](https://mypy.readthedocs.io/en/latest/index.html) is used in `lint-python.sh` to do the type checking. The package is standard so there is little chance that it will be abandoned. Mypy checks that type hints in source code are correct when they are not, it fails with an error. **Notes:** * [--ignore-missing-imports](https://mypy.readthedocs.io/en/latest/command_line.html#cmdoption-mypy-ignore-missing-imports) switch is passed on to `mypy` checker for now. The effect of this is that one does not need `# type: ignore` for `import zmq`. More information about import processing can be found [here](https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports). This can be changed in a follow-up PR, if it is deemed useful. * We are stuck with Python 3.5 until 04/2021 (see https://packages.ubuntu.com/xenial/python3). When Python version is bumped to 3.6+, one can change: ```python _opcode_instances = [] # type: List[CScriptOp] ``` to ```python _opcode_instances:List[CScriptOp] = [] ``` for type hints that are **not** function parameters and function return types. **Useful resources:** * https://docs.python.org/3.5/library/typing.html * https://www.python.org/dev/peps/pep-0484/ ACKs for top commit: fanquake: ACK bd7e530f010d43816bb05d6f1590d1cd36cdaa2c - the type checking is not the most robust (there are things it fails to detect), but I think this is worth adopting (in a limited capacity while we maintain 3.5 compat). MarcoFalke: ACK bd7e530f010d43816bb05d6f1590d1cd36cdaa2c fine with me Tree-SHA512: 21ef213915fb1dec6012f59ef17484e6c9e0abf542a316b63d5f21a7778ad5ebabf8961ef5fc8e5414726c2ee9c6ae07c7353fb4dd337f8fcef5791199c8987a
2020-06-03 16:01:29 +02:00
kernel32 = ctypes.windll.kernel32 # type: ignore
ENABLE_VIRTUAL_TERMINAL_PROCESSING = 4
STD_OUTPUT_HANDLE = -11
STD_ERROR_HANDLE = -12
# Enable ascii color control to stdout
stdout = kernel32.GetStdHandle(STD_OUTPUT_HANDLE)
stdout_mode = ctypes.c_int32()
kernel32.GetConsoleMode(stdout, ctypes.byref(stdout_mode))
kernel32.SetConsoleMode(stdout, stdout_mode.value | ENABLE_VIRTUAL_TERMINAL_PROCESSING)
# Enable ascii color control to stderr
stderr = kernel32.GetStdHandle(STD_ERROR_HANDLE)
stderr_mode = ctypes.c_int32()
kernel32.GetConsoleMode(stderr, ctypes.byref(stderr_mode))
kernel32.SetConsoleMode(stderr, stderr_mode.value | ENABLE_VIRTUAL_TERMINAL_PROCESSING)
# primitive formatting on supported
# terminal via ANSI escape sequences:
DEFAULT = ('\033[0m', '\033[0m')
BOLD = ('\033[0m', '\033[1m')
GREEN = ('\033[0m', '\033[0;32m')
RED = ('\033[0m', '\033[0;31m')
TEST_EXIT_PASSED = 0
TEST_EXIT_SKIPPED = 77
# List of framework modules containing unit tests. Should be kept in sync with
# the output of `git grep unittest.TestCase ./test/functional/test_framework`
TEST_FRAMEWORK_MODULES = [
"base58",
"crypto.bip324_cipher",
"blocktools",
"crypto.chacha20",
"crypto.ellswift",
"key",
"crypto.muhash",
"crypto.poly1305",
"crypto.ripemd160",
"script",
]
EXTENDED_SCRIPTS = [
# These tests are not run by default.
# Longest test should go first, to favor running tests in parallel
'feature_pruning.py', # NOTE: Prune mode is incompatible with -txindex, should work with governance validation disabled though.
'feature_dbcrash.py',
]
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
BASE_SCRIPTS = [
# Scripts that are run by default.
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
# Longest test should go first, to favor running tests in parallel
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_dip3_deterministicmns.py', # NOTE: needs dash_hash to pass
'feature_llmq_data_recovery.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_hd.py',
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
'wallet_hd.py --descriptors',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_backup.py',
Merge #19077: wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets c4a29d0a90b821c443c10891d9326c534d15cf97 Update wallet_multiwallet.py for descriptor and sqlite wallets (Russell Yanofsky) 310b0fde04639b7446efd5c1d2701caa4b991b86 Run dumpwallet for legacy wallets only in wallet_backup.py (Andrew Chow) 6c6639ac9f6e1677da066cf809f9e3fa4d2e7c32 Include sqlite3 in documentation (Andrew Chow) f023b7cac0eb16d3c1bf40f1f7898b290de4cc73 wallet: Enforce sqlite serialized threading mode (Andrew Chow) 6173269866306058fcb1cc825b9eb681838678ca Set and check the sqlite user version (Andrew Chow) 9d3d2d263c331e3c77b8f0d01ecc9fea0407dd17 Use network magic as sqlite wallet application ID (Andrew Chow) 9af5de3798c49f86f27bb79396e075fb8c1b2381 Use SQLite for descriptor wallets (Andrew Chow) 9b78f3ce8ed1867c37f6b9fff98f74582d44b789 walletutil: Wallets can also be sqlite (Andrew Chow) ac38a87225be0f1103ff9629d63980550d2f372b Determine wallet file type based on file magic (Andrew Chow) 6045f77003f167bee9a85e2d53f8fc6ff2e297d8 Implement SQLiteDatabase::MakeBatch (Andrew Chow) 727e6b2a4ee5abb7f2dcbc9f7778291908dc28ad Implement SQLiteDatabase::Verify (Andrew Chow) b4df8fdb19fcded7e6d491ecf0b705cac0ec76a1 Implement SQLiteDatabase::Rewrite (Andrew Chow) 010e3659069e6f97dd7b24483f50ed71042b84b0 Implement SQLiteDatabase::TxnBegin, TxnCommit, and TxnAbort (Andrew Chow) ac5c1617e7f4273daf24c24da1f6bc5ef5ab2d2b Implement SQLiteDatabase::Backup (Andrew Chow) f6f9cd6a64842ef23777312f2465e826ca04b886 Implement SQLiteBatch::StartCursor, ReadAtCursor, and CloseCursor (Andrew Chow) bf90e033f4fe86cfb90492c7e0962278ea3a146d Implement SQLiteBatch::ReadKey, WriteKey, EraseKey, and HasKey (Andrew Chow) 7aa45620e2f2178145a2eca58ccbab3cecff08fb Add SetupSQLStatements (Andrew Chow) 6636a2608a4e5906ee8092d5731595542261e0ad Implement SQLiteBatch::Close (Andrew Chow) 93825352a36456283bf87e39b5888363ee242f21 Implement SQLiteDatabase::Close (Andrew Chow) a0de83372be83f59015cd3d61af2303b74fb64b5 Implement SQLiteDatabase::Open (Andrew Chow) 3bfa0fe1259280f8c32b41a798c9453b73f89b02 Initialize and Shutdown sqlite3 globals (Andrew Chow) 5a488b3d77326a0d957c1233493061da1b6ec207 Constructors, destructors, and relevant private fields for SQLiteDatabase/Batch (Andrew Chow) ca8b7e04ab89f99075b093fa248919fd10acbdf7 Implement SQLiteDatabaseVersion (Andrew Chow) 7577b6e1c88a1a7b45ecf5c7f1735bae6f5a82bf Add SQLiteDatabase and SQLiteBatch dummy classes (Andrew Chow) e87df8258090138d5c22ac46b8602b618620e8a1 Add sqlite to travis and depends (Andrew Chow) 54729f3f4e6765dfded590af5fb28c88331685f8 Add libsqlite3 (Andrew Chow) Pull request description: This PR adds a new class `SQLiteDatabase` which is a subclass of `WalletDatabase`. This provides access to a SQLite database that is used to store the wallet records. To keep compatibility with BDB and to complexity of the change down, we don't make use of many SQLite's features. We use it strictly as a key-value store. We create a table `main` which has two columns, `key` and `value` both with the type `blob`. For new descriptor wallets, we will create a `SQLiteDatabase` instead of a `BerkeleyDatabase`. There is no requirement that all SQLite wallets are descriptor wallets, nor is there a requirement that all descriptor wallets be SQLite wallets. This allows for existing descriptor wallets to work as well as keeping open the option to migrate existing wallets to SQLite. We keep the name `wallet.dat` for SQLite wallets. We are able to determine which database type to use by searching for specific magic bytes in the `wallet.dat` file. SQLite begins it's files with a null terminated string `SQLite format 3`. BDB has `0x00053162` at byte 12 (note that the byte order of this integer depends on the system endianness). So when we see that there is a `wallet.dat` file that we want to open, we check for the magic bytes to determine which database system to use. I decided to keep the `wallet.dat` naming to keep things like backup script to continue to function as they won't need to be modified to look for a different file name. It also simplifies a couple of things in the implementation and the tests as `wallet.dat` is something that is specifically being looked for. If we don't want this behavior, then I do have another branch which creates `wallet.sqlite` files instead, but I find that this direction is easier. ACKs for top commit: Sjors: re-utACK c4a29d0a90b821c443c10891d9326c534d15cf97 promag: Tested ACK c4a29d0a90b821c443c10891d9326c534d15cf97. fjahr: reACK c4a29d0a90b821c443c10891d9326c534d15cf97 S3RK: Re-review ACK c4a29d0a90b821c443c10891d9326c534d15cf97 meshcollider: re-utACK c4a29d0a90b821c443c10891d9326c534d15cf97 hebasto: re-ACK c4a29d0a90b821c443c10891d9326c534d15cf97, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/19077#pullrequestreview-507743699) review, verified with `git range-diff master d18892dcc c4a29d0a9`. ryanofsky: Code review ACK c4a29d0a90b821c443c10891d9326c534d15cf97. I am honestly confused about reasons for locking into `wallet.dat` again when it's so easy now to use a clean format. I assume I'm just very dense, or there's some unstated reason, because the only thing that's been brought up are unrealistic compatibility scenarios (all require actively creating a wallet with non-default descriptor+sqlite option, then trying to using the descriptor+sqlite wallets with old software or scripts and ignoring the results) that we didn't pay attention to with previous PRs like #11687, which did not require any active interfaction. jonatack: ACK c4a29d0a90b821c443c10891d9326c534d15cf97, debug builds and test runs after rebase to latest master @ c2c4dbaebd9, some manual testing creating, using, unloading and reloading a few different new sqlite descriptor wallets over several node restarts/shutdowns. Tree-SHA512: 19145732e5001484947352d3175a660b5102bc6e833f227a55bd41b9b2f4d92737bbed7cead64b75b509decf9e1408cd81c185ab1fb4b90561aee427c4f9751c
2020-10-15 08:20:18 +02:00
'wallet_backup.py --descriptors',
# vv Tests less than 5m vv
'mining_getblocktemplate_longpoll.py', # FIXME: "socket.error: [Errno 54] Connection reset by peer" on my Mac, same as https://github.com/bitcoin/bitcoin/issues/6651
'feature_maxuploadtarget.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_block.py', # NOTE: needs dash_hash to pass
'rpc_fundrawtransaction.py',
'rpc_fundrawtransaction.py --nohd',
'wallet_multiwallet.py --usecli',
llmq|rpc|test|version: Implement P2P messages QGETDATA <-> QDATA (#3953) * version: Bump PROTOCOL_VERSION and MIN_MASTERNODE_PROTO_VERSION * version: Introduce LLMQ_DATA_MESSAGES_VERSION for QGETDATA/QDATA support * test: Bump MY_VERSION to 70219 (LLMQ_DATA_MESSAGES_VERSION) * llmq: Introduce CQuorumDataRequest as wrapper for QGETDATA requests * llmq: Implement CQuorum::{SetVerificationVector, SetSecretKeyShare} * llmq|net|protocol: Implement QGETDATA/QDATA P2P messages * llmq: Restrict processing QGETDATA/QDATA to masternodes only * llmq: Implement request limiting for QGETDATA/QDATA * llmq: Implement CQuorumManger::RequestQuorumData * rpc: Implement "quorum getdata" as wrapper around QGETDATA Allows to trigger sending QGETDATA messages to connected peers by RPC. * test: Handle QGETDATA/QDATA messages in mininode * test: Add data structures to support QGETDATA/QDATA * test: Add some helper in test_framework.py * test: Implement tests for QGETDATA/QDATA in p2p_quorum_data.py * test: Add p2p_quorum_data.py to BASE_SCRIPTS * llmq|test: Add QWATCH support for QGETDATA/QDATA * llmq: Store CQuorumPtr in cache, not CQuorumCPtr * llmq: Fix cache usage after recent changes * Use uacomment to create/find specific p2ps * No need to use network adjusted time here, GetTime should be enough * rpc: check proTxHash * minor tweaks * test: Adjustments after 4e27d6513e0073ed848ede262cfec82a9134abc0 * llmq: Rename and improve error lambda in CQuorumManager::ProcessMessage * llmq: Process QDATA if -watchquorums is enabled * test: Handle qwatch messages in mininode * test: Add test for -watchquorums support * test: Just some empty lines * test: Properly stop the p2p network thread at the end of the test * rpc: Adjust "quorum getdata" parameter descriptions Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com> * rpc: Fix optionality of proTxHash in "quorum getdata" command * test: Test optionality of proTxHash for "quorum getdata" command * test: Be more specific about imports in p2p_quorum_data.py * llmq|rpc: Add some comments about the request.GetDataMask checks * test: Some more empty lines * rpc: One more parameter description Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com> * test: Unify assert statements / drop parentheses for all of them * fix typo Signed-off-by: pasta <pasta@dashboost.org> * adjust some line wrapping to 80 chars Signed-off-by: pasta <pasta@dashboost.org> * tests: Seperate out into dif atomic methods, add logging Signed-off-by: pasta <pasta@dashboost.org> * test: Avoid restarting masternodes, just let available requests expire Just takes a lot time and isn't required imo. * test: Drop redundant code/tests after separation This was introduced in 9e224ec2f2ef4a58adaf0f9d4ffe110e379718ef * test: Merge three tests "test_mnauth_restriction", "test_invalid_messages" and "test_invalid_unexpected_qdata" with the resulting name "test_basics" because i don't feel like DKG recovery thing should be part of a test called "test_invalid_messages" and giving it an own test probably wouldn't make a lot sense because it would still depend on "test_invalid_messages". I also think there is no need for a separated "test_invalid_unexpected_qdata". * test: Rename test_ratelimiting_banscore -> test_request_limit * test: Apply python style * test: Wrap all at 120 characters Thats the default "draw annoying warnings" setting for PyCharm (and IMO a reasonable line length). * test: Move some variables * test: Optimize for speed * tests: use wait_until in get_mininode_id * test: Don't use `!=` to check for `None` Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com> Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com> Co-authored-by: pasta <pasta@dashboost.org>
2021-01-28 23:33:18 +01:00
'p2p_quorum_data.py',
# vv Tests less than 2m vv
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'p2p_instantsend.py',
'wallet_basic.py',
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
'wallet_basic.py --descriptors',
'wallet_labels.py',
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
'wallet_labels.py --descriptors',
'p2p_timeouts.py',
'feature_bip68_sequence.py',
'mempool_updatefromblock.py',
'p2p_tx_download.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_dump.py',
'wallet_listtransactions.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_multikeysporks.py',
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
'feature_dip3_v19.py',
'feature_llmq_signing.py', # NOTE: needs dash_hash to pass
'feature_llmq_signing.py --spork21', # NOTE: needs dash_hash to pass
'feature_llmq_chainlocks.py', # NOTE: needs dash_hash to pass
'feature_llmq_rotation.py', # NOTE: needs dash_hash to pass
'feature_llmq_connections.py', # NOTE: needs dash_hash to pass
'feature_llmq_evo.py', # NOTE: needs dash_hash to pass
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
'feature_llmq_simplepose.py', # NOTE: needs dash_hash to pass
'feature_llmq_is_cl_conflicts.py', # NOTE: needs dash_hash to pass
'feature_llmq_is_retroactive.py', # NOTE: needs dash_hash to pass
'feature_llmq_dkgerrors.py', # NOTE: needs dash_hash to pass
'feature_dip4_coinbasemerkleroots.py', # NOTE: needs dash_hash to pass
'feature_asset_locks.py', # NOTE: needs dash_hash to pass
'feature_mnehf.py', # NOTE: needs dash_hash to pass
# vv Tests less than 60s vv
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'p2p_sendheaders.py', # NOTE: needs dash_hash to pass
'p2p_sendheaders_compressed.py', # NOTE: needs dash_hash to pass
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_importmulti.py',
'mempool_limit.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'rpc_txoutproof.py',
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
'wallet_listreceivedby.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_abandonconflict.py',
'feature_csv_activation.py',
'rpc_rawtransaction.py',
'feature_reindex.py',
2019-07-25 02:30:18 +02:00
'feature_abortnode.py',
# vv Tests less than 30s vv
'rpc_quorum.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_keypool_topup.py',
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
'wallet_keypool_topup.py --descriptors',
'feature_fee_estimation.py',
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
'interface_zmq_dash.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'interface_zmq.py',
'rpc_invalid_address_message.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'interface_bitcoin_cli.py',
'mempool_resurrect.py',
'wallet_txn_doublespend.py --mineblock',
'tool_wallet.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_txn_clone.py',
'rpc_getchaintips.py',
'rpc_misc.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'interface_rest.py',
'mempool_spend_coinbase.py',
Merge #13756: wallet: "avoid_reuse" wallet flag for improved privacy 5ebc6b0eb267e0552c66fffc5e5afe7df8becf80 bitcoind: update -avoidpartialspends description to account for auto-enable for avoid_reuse wallets (Karl-Johan Alm) ada258f8c8f92d44d893cf9f22d15acdeca40b1a doc: release notes for avoid_reuse (Karl-Johan Alm) 27669551da52099e4a6a401acd7aa32b32832423 wallet: enable avoid_partial_spends by default if avoid_reuse is set (Karl-Johan Alm) 8f2e208f7c0468f9ba92bc789a698281b1c81284 test: add test for avoidreuse feature (Karl-Johan Alm) 0bdfbd34cf4015de87741ff549db35e5064f4e16 wallet/rpc: add 'avoid_reuse' option to RPC commands (Karl-Johan Alm) f904723e0d5883309cb0dd14b826bc45c5e776fb wallet/rpc: add setwalletflag RPC and MUTABLE_WALLET_FLAGS (Karl-Johan Alm) 8247a0da3a46d7c38943ee0304343ab7465305bd wallet: enable avoid_reuse feature (Karl-Johan Alm) eec15662fad917b169f5e3b8baaf4301dcf00a7b wallet: avoid reuse flags (Karl-Johan Alm) 58928098c299efdc7c5ddf2dc20716ca5272f21b wallet: make IsWalletFlagSet() const (Karl-Johan Alm) 129a5bafd9a3efa2fa16d780885048a06566d262 wallet: rename g_known_wallet_flags constant to KNOWN_WALLET_FLAGS (Karl-Johan Alm) Pull request description: Add a new wallet flag called `avoid_reuse` which, when enabled, will keep track of when a specific destination has been spent from, and will actively "blacklist" any new UTXOs which send to an already-spent-from destination. This improves privacy, as a payer could otherwise begin tracking a payee's wallet by regularly peppering a known UTXO with dust outputs, which would then be scooped up and used in payments by the payee, allowing the payer to map out (1) the inputs owned by the payee and (2) the destinations to which the payee is making payments. This replaces #10386 and together with the (now merged) #12257 it addresses #10065 in full. The concerns raised in https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-302361381 are also addressed due to #12257. ~~Note: this builds on top of #15780.~~ (merged) ACKs for commit 5ebc6b: jnewbery: ACK 5ebc6b0eb laanwj: Concept and code-review ACK 5ebc6b0eb267e0552c66fffc5e5afe7df8becf80 meshcollider: Code review ACK https://github.com/bitcoin/bitcoin/pull/13756/commits/5ebc6b0eb267e0552c66fffc5e5afe7df8becf80 achow101: ACK 5ebc6b0eb267e0552c66fffc5e5afe7df8becf80 modulo above nits Tree-SHA512: fdef45826af544cbbb45634ac367852cc467ec87081d86d08b53ca849e588617e9a0a255b7e7bb28692d15332de58d6c3d274ac003355220e4213d7d9070742e
2019-06-19 01:32:02 +02:00
'wallet_avoidreuse.py',
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
'wallet_avoidreuse.py --descriptors',
'mempool_reorg.py',
'mempool_persist.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_multiwallet.py',
Merge #19077: wallet: Add sqlite as an alternative wallet database and use it for new descriptor wallets c4a29d0a90b821c443c10891d9326c534d15cf97 Update wallet_multiwallet.py for descriptor and sqlite wallets (Russell Yanofsky) 310b0fde04639b7446efd5c1d2701caa4b991b86 Run dumpwallet for legacy wallets only in wallet_backup.py (Andrew Chow) 6c6639ac9f6e1677da066cf809f9e3fa4d2e7c32 Include sqlite3 in documentation (Andrew Chow) f023b7cac0eb16d3c1bf40f1f7898b290de4cc73 wallet: Enforce sqlite serialized threading mode (Andrew Chow) 6173269866306058fcb1cc825b9eb681838678ca Set and check the sqlite user version (Andrew Chow) 9d3d2d263c331e3c77b8f0d01ecc9fea0407dd17 Use network magic as sqlite wallet application ID (Andrew Chow) 9af5de3798c49f86f27bb79396e075fb8c1b2381 Use SQLite for descriptor wallets (Andrew Chow) 9b78f3ce8ed1867c37f6b9fff98f74582d44b789 walletutil: Wallets can also be sqlite (Andrew Chow) ac38a87225be0f1103ff9629d63980550d2f372b Determine wallet file type based on file magic (Andrew Chow) 6045f77003f167bee9a85e2d53f8fc6ff2e297d8 Implement SQLiteDatabase::MakeBatch (Andrew Chow) 727e6b2a4ee5abb7f2dcbc9f7778291908dc28ad Implement SQLiteDatabase::Verify (Andrew Chow) b4df8fdb19fcded7e6d491ecf0b705cac0ec76a1 Implement SQLiteDatabase::Rewrite (Andrew Chow) 010e3659069e6f97dd7b24483f50ed71042b84b0 Implement SQLiteDatabase::TxnBegin, TxnCommit, and TxnAbort (Andrew Chow) ac5c1617e7f4273daf24c24da1f6bc5ef5ab2d2b Implement SQLiteDatabase::Backup (Andrew Chow) f6f9cd6a64842ef23777312f2465e826ca04b886 Implement SQLiteBatch::StartCursor, ReadAtCursor, and CloseCursor (Andrew Chow) bf90e033f4fe86cfb90492c7e0962278ea3a146d Implement SQLiteBatch::ReadKey, WriteKey, EraseKey, and HasKey (Andrew Chow) 7aa45620e2f2178145a2eca58ccbab3cecff08fb Add SetupSQLStatements (Andrew Chow) 6636a2608a4e5906ee8092d5731595542261e0ad Implement SQLiteBatch::Close (Andrew Chow) 93825352a36456283bf87e39b5888363ee242f21 Implement SQLiteDatabase::Close (Andrew Chow) a0de83372be83f59015cd3d61af2303b74fb64b5 Implement SQLiteDatabase::Open (Andrew Chow) 3bfa0fe1259280f8c32b41a798c9453b73f89b02 Initialize and Shutdown sqlite3 globals (Andrew Chow) 5a488b3d77326a0d957c1233493061da1b6ec207 Constructors, destructors, and relevant private fields for SQLiteDatabase/Batch (Andrew Chow) ca8b7e04ab89f99075b093fa248919fd10acbdf7 Implement SQLiteDatabaseVersion (Andrew Chow) 7577b6e1c88a1a7b45ecf5c7f1735bae6f5a82bf Add SQLiteDatabase and SQLiteBatch dummy classes (Andrew Chow) e87df8258090138d5c22ac46b8602b618620e8a1 Add sqlite to travis and depends (Andrew Chow) 54729f3f4e6765dfded590af5fb28c88331685f8 Add libsqlite3 (Andrew Chow) Pull request description: This PR adds a new class `SQLiteDatabase` which is a subclass of `WalletDatabase`. This provides access to a SQLite database that is used to store the wallet records. To keep compatibility with BDB and to complexity of the change down, we don't make use of many SQLite's features. We use it strictly as a key-value store. We create a table `main` which has two columns, `key` and `value` both with the type `blob`. For new descriptor wallets, we will create a `SQLiteDatabase` instead of a `BerkeleyDatabase`. There is no requirement that all SQLite wallets are descriptor wallets, nor is there a requirement that all descriptor wallets be SQLite wallets. This allows for existing descriptor wallets to work as well as keeping open the option to migrate existing wallets to SQLite. We keep the name `wallet.dat` for SQLite wallets. We are able to determine which database type to use by searching for specific magic bytes in the `wallet.dat` file. SQLite begins it's files with a null terminated string `SQLite format 3`. BDB has `0x00053162` at byte 12 (note that the byte order of this integer depends on the system endianness). So when we see that there is a `wallet.dat` file that we want to open, we check for the magic bytes to determine which database system to use. I decided to keep the `wallet.dat` naming to keep things like backup script to continue to function as they won't need to be modified to look for a different file name. It also simplifies a couple of things in the implementation and the tests as `wallet.dat` is something that is specifically being looked for. If we don't want this behavior, then I do have another branch which creates `wallet.sqlite` files instead, but I find that this direction is easier. ACKs for top commit: Sjors: re-utACK c4a29d0a90b821c443c10891d9326c534d15cf97 promag: Tested ACK c4a29d0a90b821c443c10891d9326c534d15cf97. fjahr: reACK c4a29d0a90b821c443c10891d9326c534d15cf97 S3RK: Re-review ACK c4a29d0a90b821c443c10891d9326c534d15cf97 meshcollider: re-utACK c4a29d0a90b821c443c10891d9326c534d15cf97 hebasto: re-ACK c4a29d0a90b821c443c10891d9326c534d15cf97, only rebased since my [previous](https://github.com/bitcoin/bitcoin/pull/19077#pullrequestreview-507743699) review, verified with `git range-diff master d18892dcc c4a29d0a9`. ryanofsky: Code review ACK c4a29d0a90b821c443c10891d9326c534d15cf97. I am honestly confused about reasons for locking into `wallet.dat` again when it's so easy now to use a clean format. I assume I'm just very dense, or there's some unstated reason, because the only thing that's been brought up are unrealistic compatibility scenarios (all require actively creating a wallet with non-default descriptor+sqlite option, then trying to using the descriptor+sqlite wallets with old software or scripts and ignoring the results) that we didn't pay attention to with previous PRs like #11687, which did not require any active interfaction. jonatack: ACK c4a29d0a90b821c443c10891d9326c534d15cf97, debug builds and test runs after rebase to latest master @ c2c4dbaebd9, some manual testing creating, using, unloading and reloading a few different new sqlite descriptor wallets over several node restarts/shutdowns. Tree-SHA512: 19145732e5001484947352d3175a660b5102bc6e833f227a55bd41b9b2f4d92737bbed7cead64b75b509decf9e1408cd81c185ab1fb4b90561aee427c4f9751c
2020-10-15 08:20:18 +02:00
'wallet_multiwallet.py --descriptors',
'wallet_createwallet.py',
'wallet_createwallet.py --usecli',
'wallet_reorgsrestore.py',
Merge #16383: rpcwallet: default include_watchonly to true for watchonly wallets 72eaab073bc747425fe551777154b13a6c4c37c9 tests: functional watch-only wallet tests (William Casarin) 72ffbdc5799c1707ecad674d701b43fb80b031d0 doc: add release note for include_watchonly default changes (William Casarin) 003a3c73c0450aa18ac2ab2ca47def2b8c53a7df rpcwallet: document include_watchonly default for watchonly wallets (William Casarin) a50d9e6c0b8e8144d3deec58ec2e3449ba081151 rpcwallet: default include_watchonly to true for watchonly wallets (William Casarin) Pull request description: Right now it's a bit annoying to deal with watchonly wallets, many rpc commands have an `include_watchonly` argument that needs to be explicitly set. Wallets created with `createwallet` can have a `disable_private_keys` parameter, for those wallets we already know that they are watchonly, so there's no reason to have to explicitly ask for it for every command. Instead we check this wallet flag when the `include_watchonly` parameter isn't set. ACKs for top commit: achow101: Code review ACK 72eaab073bc747425fe551777154b13a6c4c37c9 Sjors: ACK 72eaab073bc747425fe551777154b13a6c4c37c9 promag: ACK 72eaab073bc747425fe551777154b13a6c4c37c9, code review only, didn't look closely to the test. kallewoof: ACK 72eaab073bc747425fe551777154b13a6c4c37c9 fanquake: ACK 72eaab073bc747425fe551777154b13a6c4c37c9 - I've looked over the changes, they make sense to me. Compiled and ran the tests etc. Tree-SHA512: d3646b55e97f386594d7efc994f0712f3888475c6a5dc7f131ac9f8c49bf5d4677182b88f42b34152abe1ad101ecadd152b4c20e9d3c1267190db36f77ab8bd7
2019-08-16 04:55:26 +02:00
'wallet_watchonly.py',
'wallet_watchonly.py --usecli',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'interface_http.py',
'interface_rpc.py',
'rpc_psbt.py',
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
'rpc_psbt.py --descriptors',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'rpc_users.py',
Merge #12763: Add RPC Whitelist Feature from #12248 2081442c421cc4376e5d7839f68fbe7630e89103 test: Add test for rpc_whitelist (Emil Engler) 7414d3820c833566b4f48c6c120a18bf53978c55 Add RPC Whitelist Feature from #12248 (Jeremy Rubin) Pull request description: Summary ==== This patch adds the RPC whitelisting feature requested in #12248. RPC Whitelists help enforce application policies for services being built on top of Bitcoin Core (e.g., your Lightning Node maybe shouldn't be adding new peers). The aim of this PR is not to make it advisable to connect your Bitcoin node to arbitrary services, but to reduce risk and prevent unintended access. Using RPC Whitelists ==== The way it works is you specify (in your bitcoin.conf) configurations such as ``` rpcauth=user1:4cc74397d6e9972e5ee7671fd241$11849357f26a5be7809c68a032bc2b16ab5dcf6348ef3ed1cf30dae47b8bcc71 rpcauth=user2:181b4a25317bff60f3749adee7d6bca0$d9c331474f1322975fa170a2ffbcb176ba11644211746b27c1d317f265dd4ada rpcauth=user3:a6c8a511b53b1edcf69c36984985e$13cfba0e626db19061c9d61fa58e712d0319c11db97ad845fa84517f454f6675 rpcwhitelist=user1:getnetworkinfo rpcwhitelist=user2:getnetworkinfo,getwalletinfo, getbestblockhash rpcwhitelistdefault=0 ``` Now user1 can only call getnetworkinfo, user2 can only call getnetworkinfo or getwalletinfo, while user3 can still call all RPCs. If any rpcwhitelist is set, act as if all users are subject to whitelists unless rpcwhitelistdefault is set to 0. If rpcwhitelistdefault is set to 1 and no rpcwhitelist is set, act as if all users are subject to whitelists. Review Request ===== In addition to normal review, would love specific review from someone working on LN (e.g., @ roasbeef) and someone working on an infrastructure team at an exchange (e.g., @ jimpo) to check that this works well with their system. Notes ===== The rpc list is spelling sensitive -- whitespace is stripped though. Spelling errors fail towards the RPC call being blocked, which is safer. It was unclear to me if HTTPReq_JSONRPC is the best function to patch this functionality into, or if it would be better to place it in exec or somewhere else. It was also unclear to me if it would be preferred to cache the whitelists on startup or parse them on every RPC as is done with multiUserAuthorized. I opted for the cached approach as I thought it was a bit cleaner. Future Work ===== In a future PR, I would like to add an inheritance scheme. This seemed more controversial so I didn't want to include that here. Inheritance semantics are tricky, but it would also make these whitelists easier to read. It also might be good to add a `getrpcwhitelist` command to facilitate permission discovery. Tests ===== Thanks to @ emilengler for adding tests for this feature. The tests cover all cases except for where `rpcwhitelistdefault=1` is used, given difficulties around testing with the current test framework. ACKs for top commit: laanwj: ACK 2081442c421cc4376e5d7839f68fbe7630e89103 Tree-SHA512: 0dc1ac6a6f2f4b0be9c9054d495dd17752fe7b3589aeab2c6ac4e1f91cf4e7e355deedcb5d76d707cbb5a949c2f989c871b74d6bf129351f429569a701adbcbf
2019-12-13 11:25:39 +01:00
'rpc_whitelist.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_proxy.py',
'rpc_signrawtransaction.py',
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
'p2p_addrv2_relay.py',
Merge #12257: [wallet] Use destination groups instead of coins in coin select 232f96f5c8a3920c09db92f4dbac2ad7d10ce8cf doc: Add release notes for -avoidpartialspends (Karl-Johan Alm) e00b4699cc6d2ee5697d38dd6607eb2631c9b77a clean-up: Remove no longer used ivars from CInputCoin (Karl-Johan Alm) 43e04d13b1ffc02b1082176e87f420198b40c7b1 wallet: Remove deprecated OutputEligibleForSpending (Karl-Johan Alm) 0128121101fb3ee82f3abd3973a967a4226ffe0e test: Add basic testing for wallet groups (Karl-Johan Alm) 59d6f7b4e2f847ec1f2ff46c84e6157655984f85 wallet: Switch to using output groups instead of coins in coin selection (Karl-Johan Alm) 87ebce25d66952f5ce565bb5130dcf5e24049872 wallet: Add output grouping (Karl-Johan Alm) bb629cb9dc567cc819724d9f4852652926e60cbf Add -avoidpartialspends and m_avoid_partial_spends (Karl-Johan Alm) 65b3eda458221644616d0fdd6ba0fe01bdbce893 wallet: Add input bytes to CInputCoin (Karl-Johan Alm) a443d7a0ca333b0bae63e04b5d476f9ad9c7aeac moveonly: CoinElegibilityFilter into coinselection.h (Karl-Johan Alm) 173e18a289088c6087ba6fac708e322aa63b7a94 utils: Add insert() convenience templates (Karl-Johan Alm) Pull request description: This PR adds an optional (off by default) `-avoidpartialspends` flag, which changes coin select to use output groups rather than outputs, where each output group corresponds to all outputs with the same destination. It is a privacy improvement, as each time you spend some output, any other output that is publicly associated with the destination (address) will also be spent at the same time, at the cost of fee increase for cases where coin select without group restriction would find a more optimal set of coins (see example below). For regular use without address reuse, this PR should have no effect on the user experience whatsoever; it only affects users who, for some reason, have multiple outputs with the same destination (i.e. address reuse). Nodes with this turned off will still try to avoid partial spending, if the fee of the resulting transaction is not greater than the fee of the original transaction. Example: a node has four outputs linked to two addresses `A` and `B`: * 1.0 btc to `A` * 0.5 btc to `A` * 1.0 btc to `B` * 0.5 btc to `B` The node sends 0.2 btc to `C`. Without `-avoidpartialspends`, the following coin selection will occur: * 0.5 btc to `A` or `B` is picked * 0.2 btc is output to `C` * 0.3 - fee is output to (unique change address) With `-avoidpartialspends`, the following will instead happen: * Both of (0.5, 1.0) btc to `A` or `B` is picked (one or the other pair) * 0.2 btc is output to `C` * 1.3 - fee is output to (unique change address) As noted, the pro here is that, assuming nobody sends to the address after you spend from it, you will only ever use one address once. The con is that the transaction becomes slightly larger in this case, because it is overpicking outputs to adhere to the no partial spending rule. This complements #10386, in particular it addresses @luke-jr and @gmaxwell's concerns in https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-300667926 and https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-302361381. Together with `-avoidreuse`, this fully addresses the concerns in #10065 I believe. Tree-SHA512: 24687a4490ba59cf4198ed90052944ff4996653a4257833bb52ed24d058b3e924800c9b3790aeb6be6385b653b49e304453e5d7ff960e64c682fc23bfc447621 # Conflicts: # src/Makefile.am # src/bench/coin_selection.cpp # src/wallet/coincontrol.h # src/wallet/coinselection.cpp # src/wallet/coinselection.h # src/wallet/init.cpp # src/wallet/test/coinselector_tests.cpp # src/wallet/wallet.cpp # src/wallet/wallet.h # test/functional/test_runner.py
2018-07-24 15:06:21 +02:00
'wallet_groups.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'p2p_disconnect_ban.py',
'feature_addressindex.py',
'feature_timestampindex.py',
'feature_spentindex.py',
'rpc_decodescript.py',
'rpc_blockchain.py',
'rpc_deprecated.py',
'wallet_disable.py',
'p2p_addr_relay.py',
'p2p_getaddr_caching.py',
'p2p_getdata.py',
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
'rpc_net.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_keypool.py',
'wallet_keypool_hd.py',
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
'wallet_keypool_hd.py --descriptors',
'wallet_descriptor.py --descriptors',
Merge #19083: test: msg_mempool, fRelay, and other bloomfilter tests (BACKPORT NOTICE: p2p_filter.py has also fixes for d5fbd4a92a3e7c1f8266d4cb4b639a0fe4c4c61f:test/functional/p2p_filter.py) + Merge #18726: test: check misbehavior more independently in p2p_filter.py --------------------- dca73941eb0f0a4c9b68efed3870b536f7dd6cfe scripted-diff: rename node to peer for mininodes (gzhao408) 0474ea25afc65546cbfe5f822c0212bf3e211023 [test] fix race conditions and test in p2p_filter (gzhao408) 4ef80f0827392a1310ca5a29cc1f8f5ca5d16f95 [test] sending invalid msgs to node with bloomfilters=0 causes disconnect (gzhao408) 497a619386008dfaec0db15ecaebcdfaf75f5011 [test] add BIP 37 test for node with fRelay=false (gzhao408) e8acc6015695c8439fc971a12709468995b96dcf [test] add mempool msg test for node with bloomfilter enabled (gzhao408) Pull request description: This PR adds a few tests that are bloomfilter-related, including behavior for when bloomfilters are turned _off_: 1. Tests p2p message `msg_mempool`: a node that has `peerbloomfilters` enabled should send its mempool (disabled behavior already tested [here](https://github.com/bitcoin/bitcoin/blob/master/test/functional/p2p_mempool.py)). 2. Tests that bloomfilter peers with [`fRelay=False`](https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki#extensions-to-existing-messages) in the `version` message should not receive any invs until they set the filter. The rest is the same as what’s already tested in `p2p_filter.py`. 3. Tests that peers get disconnected if they send `filterload` or `filteradd` p2p messages to a node with bloom filters disabled. 4. Refactor: renames p2p_mempool.py to p2p_nobloomfilter_messages.py. 5. Fixes race conditions in p2p_filter.py ACKs for top commit: MarcoFalke: ACK dca73941eb only changes is restoring accidentally deleted test 🍮 jonatack: ACK dca73941eb0f0a4c9b68efed3870b536f7dd6cfe modulo a few nits if you retouch, happy to re-ACK if you take any of them but don't feel obliged to. Tree-SHA512: 442aeab0755cb8b830251ea170d1d5e6da8ac9029b3276d407a20ee3d588cc61b77b8842368de18c244056316b8c63b911776d6e106bc7c023439ab915b27ad3
2020-06-11 20:34:39 +02:00
'p2p_nobloomfilter_messages.py',
'p2p_filter.py',
'p2p_blocksonly.py',
'rpc_setban.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'mining_prioritisetransaction.py',
'p2p_invalid_locator.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'p2p_invalid_block.py',
'p2p_invalid_messages.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'p2p_invalid_tx.py',
'feature_assumevalid.py',
'example_test.py',
'wallet_txn_doublespend.py',
Merge #12134: Build previous releases and run functional tests c456145b2c65f580683df03bf10cd39000cf24d5 [test] add 0.19 backwards compatibility tests (Sjors Provoost) b769cd142deda74fe46e231cc7b687a86514f2f1 [test] add v0.17.1 wallet upgrade test (Sjors Provoost) 9d9390dab716f07057c94e8e21f3c7dd06192f35 [tests] add wallet backwards compatility tests (Sjors Provoost) c7ca6308968b29a0e0edc485cd06e68e5edb7c7d [scripts] support release candidates of earlier releases (Sjors Provoost) 8b1460dbd1b732f06d4cebe1fa6844286c7a0056 [tests] check v0.17.1 and v0.18.1 backwards compatibility (Sjors Provoost) ae379cf7d12943fc192d58176673bcfe7d53da53 [scripts] build earlier releases (Sjors Provoost) Pull request description: This PR adds binaries for 0.17, 0.18 and 0.19 to Travis and runs a basic block propagation test. Includes test for upgrading v0.17.1 wallets and opening master wallets with older versions. Usage: ```sh contrib/devtools/previous_release.sh -f -b v0.19.0.1 v0.18.1 v0.17.1 test/functional/backwards_compatibility.py ``` Travis caches these earlier releases, so it should be able to run these tests with little performance impact. Additional scenarios where it might be useful to run tests against earlier releases: * creating a wallet with #11403's segwit implementation, copying it to an older node and making sure the user didn't lose any funds (although this PR doesn't support `v0.15.1`) * future consensus changes * P2P changes (e.g. to make sure we don't accidentally ban old nodes) ACKs for top commit: MarcoFalke: ACK c456145b2c65f580683df03bf10cd39000cf24d5 🔨 Tree-SHA512: 360bd870603f95b14dc0cd629532cc147344f632b808617c18e1b585dfb1f082b401e5d493a48196b719e0aeaee533ae0a773dfc9f217f704aae898576c19232
2020-02-12 15:19:50 +01:00
'feature_backwards_compatibility.py',
'wallet_txn_clone.py --mineblock',
'feature_notifications.py',
'rpc_getblockfilter.py',
'rpc_invalidateblock.py',
'feature_txindex.py',
'feature_utxo_set_hash.py',
'mempool_packages.py',
'mempool_package_onemore.py',
'rpc_createmultisig.py',
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
'rpc_createmultisig.py --descriptors',
'rpc_packages.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_versionbits_warning.py',
'rpc_preciousblock.py',
'wallet_importprunedfunds.py',
'p2p_leak_tx.py',
'p2p_eviction.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'rpc_signmessage.py',
'rpc_generateblock.py',
'wallet_balance.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_nulldummy.py',
'mempool_accept.py',
'mempool_expiry.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_import_rescan.py',
'wallet_import_with_label.py',
'wallet_upgradewallet.py',
'wallet_importdescriptors.py --descriptors',
'wallet_mnemonicbits.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'rpc_bind.py --ipv4',
'rpc_bind.py --ipv6',
'rpc_bind.py --nonloopback',
'mining_basic.py',
'rpc_named_arguments.py',
'wallet_listsinceblock.py',
'p2p_leak.py',
'p2p_compactblocks.py',
'p2p_connect_to_devnet.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_sporks.py',
'rpc_getblockstats.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'wallet_encryption.py',
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow) 869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow) cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow) 886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow) 3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow) 388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow) 1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow) f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen) ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow) 1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow) 82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow) b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow) 8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow) 72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow) 84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow) bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow) d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow) f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow) 586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow) f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow) a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow) bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow) 58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow) e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow) 46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow) 4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow) d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow) 953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow) 2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow) 46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow) ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow) 741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow) 2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow) db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow) 78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow) 834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow) d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow) 3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow) 6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow) aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow) 96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow) 6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow) 06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow) Pull request description: Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`. Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each. Descriptors can also be imported with a new `importdescriptors` RPC. Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things. A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`). ACKs for top commit: Sjors: utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed) jonatack: Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82. fjahr: re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 instagibbs: light re-ACK 223588b meshcollider: Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it Introduce DescriptorScriptPubKeyMan as a dummy class Add WALLET_FLAG_DESCRIPTORS Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet Create LegacyScriptPubKeyMan when not a descriptor wallet Introduce WalletDescriptor class WalletDescriptor is a Descriptor with other wallet metadata Add a lock cs_desc_man for DescriptorScriptPubKeyMan Store WalletDescriptor in DescriptorScriptPubKeyMan Implement SetType in DescriptorScriptPubKeyMan Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet Implement IsMine for DescriptorScriptPubKeyMan Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks. If the given script is in that set, it is considered ISMINE_SPENDABLE Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan Implement IsHDEnabled in DescriptorScriptPubKeyMan Implement GetID for DescriptorScriptPubKeyMan Load the descriptor cache from the wallet file Implement loading of keys for DescriptorScriptPubKeyMan Add IsSingleType to Descriptors IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys Implement several simple functions in DescriptorScriptPubKeyMan Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys, KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses, RewriteDB Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file Implement SetupGeneration for DescriptorScriptPubKeyMan Implement TopUp in DescriptorScriptPubKeyMan Implement GetNewDestination for DescriptorScriptPubKeyMan Implement Unlock and Encrypt in DescriptorScriptPubKeyMan Implement GetReservedDestination in DescriptorScriptPubKeyMan Implement ReturnDestination in DescriptorScriptPubKeyMan Implement GetKeypoolOldestTime and only display it if greater than 0 Implement GetSolvingProvider for DescriptorScriptPubKeyMan Internally, a GetSigningProvider function is introduced which allows for some private keys to be optionally included. This can be called with a script as the argument (i.e. a scriptPubKey from our wallet when we are signing) or with a pubkey. In order to know what index to expand the private keys for that pubkey, we need to also cache all of the pubkeys involved when we expand the descriptor. So SetCache and TopUp are updated to do this too. Implement SignTransaction in DescriptorScriptPubKeyMan Implement SignMessage for descriptor wallets Implement FillPSBT in DescriptorScriptPubKeyMan FillPSBT will add our own scripts to the PSBT if those inputs are ours. If an input also lists pubkeys that we happen to know the private keys for, we will sign those inputs too. Change GetMetadata to use unique_ptr<CKeyMetadata> Implement GetMetadata in DescriptorScriptPubKeyMan Be able to create new wallets with DescriptorScriptPubKeyMans as backing Generate new descriptors when encrypting Add IsLegacy to CWallet so that the GUI knows whether to show watchonly add importdescriptors RPC and tests for native descriptor wallets Co-authored-by: Andrew Chow <achow101-github@achow101.com> Functional tests for descriptor wallets Change wallet_encryption.py to use signmessage instead of dumpprivkey Return error when no ScriptPubKeyMan is available for specified type When a CWallet doesn't have a ScriptPubKeyMan for the requested type in GetNewDestination, give a meaningful error. Also handle this in Qt which did not do anything with errors. Implement CWallet::IsSpentKey for non-LegacySPKMans tests: Add RPCOverloadWrapper which overloads some disabled RPCs RPCOverloadWrapper overloads some deprecated or disabled RPCs with an implementation using other RPCs to avoid having a ton of code churn around replacing those RPCs. Add a --descriptors option to various tests Adds a --descriptors option globally to the test framework. This will make the test create and use descriptor wallets. However some tests may not work with this. Some tests are modified to work with --descriptors and run with that option in test_runer: * wallet_basic.py * wallet_encryption.py * wallet_keypool.py <---- wallet_keypool_hd.py actually * wallet_keypool_topup.py * wallet_labels.py * wallet_avoidreuse.py
2019-07-16 19:34:35 +02:00
'wallet_encryption.py --descriptors',
'wallet_upgradetohd.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_dersig.py',
'feature_cltv.py',
'feature_new_quorum_type_activation.py',
'feature_governance_objects.py',
'feature_governance.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'rpc_uptime.py',
'wallet_resendwallettransactions.py',
'wallet_fallbackfee.py',
'rpc_dumptxoutset.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_minchainwork.py',
'rpc_estimatefee.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'p2p_unrequested_blocks.py', # NOTE: needs dash_hash to pass
Merge #14670: http: Fix HTTP server shutdown 28479f926f21f2a91bec5a06671c60e5b0c55532 qa: Test bitcond shutdown (João Barbosa) 8d3f46ec3938e2ba17654fecacd1d2629f9915fd http: Remove timeout to exit event loop (João Barbosa) e98a9eede2fb48ff33a020acc888cbcd83e24bbf http: Remove unnecessary event_base_loopexit call (João Barbosa) 6b13580f4e3842c11abd9b8bee7255fb2472b6fe http: Unlisten sockets after all workers quit (João Barbosa) 18e968581697078c36a3c3818f8906cf134ccadd http: Send "Connection: close" header if shutdown is requested (João Barbosa) 02e1e4eff6cda0bfc24b455a7c1583394cbff6eb rpc: Add wait argument to stop (João Barbosa) Pull request description: Fixes #11777. Reverts #11006. Replaces #13501. With this change the HTTP server will exit gracefully, meaning that all requests will finish processing and sending the response, even if this means to wait more than 2 seconds (current time allowed to exit the event loop). Another small change is that connections are accepted even when the server is stopping, but HTTP requests are rejected. This can be improved later, especially if chunked replies are implemented. Briefly, before this PR, this is the order or events when a request arrives (RPC `stop`): 1. `bufferevent_disable(..., EV_READ)` 2. `StartShutdown()` 3. `evhttp_del_accept_socket(...)` 4. `ThreadHTTP` terminates (event loop exits) because there are no active or pending events thanks to 1. and 3. 5. client doesn't get the response thanks to 4. This can be verified by applying ```diff // Event loop will exit after current HTTP requests have been handled, so // this reply will get back to the client. StartShutdown(); + MilliSleep(2000); return "Bitcoin server stopping"; } ``` and checking the log output: ``` Received a POST request for / from 127.0.0.1:62443 ThreadRPCServer method=stop user=__cookie__ Interrupting HTTP server ** Exited http event loop Interrupting HTTP RPC server Interrupting RPC tor: Thread interrupt Shutdown: In progress... torcontrol thread exit Stopping HTTP RPC server addcon thread exit opencon thread exit Unregistering HTTP handler for / (exactmatch 1) Unregistering HTTP handler for /wallet/ (exactmatch 0) Stopping RPC RPC stopped. Stopping HTTP server Waiting for HTTP worker threads to exit msghand thread exit net thread exit ... sleep 2 seconds ... Waiting for HTTP event thread to exit Stopped HTTP server ``` For this reason point 3. is moved right after all HTTP workers quit. In that moment HTTP replies are queued in the event loop which keeps spinning util all connections are closed. In order to trigger the server side close with keep alive connections (implicit in HTTP/1.1) the header `Connection: close` is sent if shutdown was requested. This can be tested by ``` bitcoind -regtest nc localhost 18443 POST / HTTP/1.1 Authorization: Basic ... Content-Type: application/json Content-Length: 44 {"jsonrpc": "2.0","method":"stop","id":123} ``` Summing up, this PR: - removes explicit event loop exit — event loop exits once there are no active or pending events - changes the moment the listening sockets are removed — explained above - sends header `Connection: close` on active requests when shutdown was requested which is relevant when it's a persistent connection (default in HTTP 1.1) — libevent is aware of this header and closes the connection gracefully - removes event loop explicit break after 2 seconds timeout Tree-SHA512: 4dac1e86abe388697c1e2dedbf31fb36a394cfafe5e64eadbf6ed01d829542785a8c3b91d1ab680d3f03f912d14fc87176428041141441d25dcb6c98a1e069d8
2018-12-06 17:42:52 +01:00
'feature_shutdown.py',
'rpc_coinjoin.py',
'rpc_masternode.py',
'rpc_mnauth.py',
'rpc_verifyislock.py',
'rpc_verifychainlock.py',
'wallet_create_tx.py',
'wallet_send.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'p2p_fingerprint.py',
'rpc_platform_filter.py',
'rpc_wipewallettxes.py',
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
'feature_dip0020_activation.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_uacomment.py',
'wallet_coinbase_category.py',
'feature_filelock.py',
'feature_loadblock.py',
'p2p_blockfilters.py',
Merge #19509: Per-Peer Message Capture bff7c66e67aa2f18ef70139338643656a54444fe Add documentation to contrib folder (Troy Giorshev) 381f77be858d7417209b6de0b7cd23cb7eb99261 Add Message Capture Test (Troy Giorshev) e4f378a505922c0f544b4cfbfdb169e884e02be9 Add capture parser (Troy Giorshev) 4d1a582549bc982d55e24585b0ba06f92f21e9da Call CaptureMessage at appropriate locations (Troy Giorshev) f2a77ff97bec09dd5fcc043d8659d8ec5dfb87c2 Add CaptureMessage (Troy Giorshev) dbf779d5deb04f55c6e8493ce4e12ed4628638f3 Clean PushMessage and ProcessMessages (Troy Giorshev) Pull request description: This PR introduces per-peer message capture into Bitcoin Core. 📓 ## Purpose The purpose and scope of this feature is intentionally limited. It answers a question anyone new to Bitcoin's P2P protocol has had: "Can I see what messages my node is sending and receiving?". ## Functionality When a new debug-only command line argument `capturemessages` is set, any message that the node receives or sends is captured. The capture occurs in the MessageHandler thread. When receiving a message, it is captured as soon as the MessageHandler thread takes the message off of the vProcessMsg queue. When sending, the message is captured just before the message is pushed onto the vSendMsg queue. The message capture is as minimal as possible to reduce the performance impact on the node. Messages are captured to a new `message_capture` folder in the datadir. Each node has their own subfolder named with their IP address and port. Inside, received and sent messages are captured into two binary files, msgs_recv.dat and msgs_sent.dat, like so: ``` message_capture/203.0.113.7:56072/msgs_recv.dat message_capture/203.0.113.7:56072/msgs_sent.dat ``` Because the messages are raw binary dumps, included in this PR is a Python parsing tool to convert the binary files into human-readable JSON. This script has been placed on its own and out of the way in the new `contrib/message-capture` folder. Its usage is simple and easily discovered by the autogenerated `-h` option. ## Future Maintenance I sympathize greatly with anyone who says "the best code is no code". The future maintenance of this feature will be minimal. The logic to deserialize the payload of the p2p messages exists in our testing framework. As long as our testing framework works, so will this tool. Additionally, I hope that the simplicity of this tool will mean that it gets used frequently, so that problems will be discovered and solved when they are small. ## FAQ "Why not just use Wireshark" Yes, Wireshark has the ability to filter and decode Bitcoin messages. However, the purpose of the message capture added in this PR is to assist with debugging, primarily for new developers looking to improve their knowledge of the Bitcoin Protocol. This drives the design in a different direction than Wireshark, in two different ways. First, this tool must be convenient and simple to use. Using an external tool, like Wireshark, requires setup and interpretation of the results. To a new user who doesn't necessarily know what to expect, this is unnecessary difficulty. This tool, on the other hand, "just works". Turn on the command line flag, run your node, run the script, read the JSON. Second, because this tool is being used for debugging, we want it to be as close to the true behavior of the node as possible. A lot can happen in the SocketHandler thread that would be missed by Wireshark. Additionally, if we are to use Wireshark, we are at the mercy of whoever it maintaining the protocol in Wireshark, both as to it being accurate and recent. As can be seen by the **many** previous attempts to include Bitcoin in Wireshark (google "bitcoin dissector") this is easier said than done. Lastly, I truly believe that this tool will be used significantly more by being included in the codebase. It's just that much more discoverable. ACKs for top commit: MarcoFalke: re-ACK bff7c66e67aa2f18ef70139338643656a54444fe only some minor changes: 👚 jnewbery: utACK bff7c66e67aa2f18ef70139338643656a54444fe theStack: re-ACK bff7c66e67aa2f18ef70139338643656a54444fe Tree-SHA512: e59e3160422269221f70f98720b47842775781c247c064071d546c24fa7a35a0e5534e8baa4b4591a750d7eb16de6b4ecf54cbee6d193b261f4f104e28c15f47
2021-02-02 13:11:14 +01:00
'p2p_message_capture.py',
'feature_asmap.py',
'feature_includeconf.py',
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
'mempool_unbroadcast.py',
'mempool_compatibility.py',
'rpc_deriveaddresses.py',
'rpc_deriveaddresses.py --usecli',
'p2p_ping.py',
'rpc_scantxoutset.py',
'feature_logging.py',
2023-07-30 13:49:32 +02:00
'feature_coinstatsindex.py',
'wallet_orphanedreward.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'p2p_node_network_limited.py',
Merge #16248: Make whitebind/whitelist permissions more flexible c5b404e8f1973afe071a07c63ba1038eefe13f0f Add functional tests for flexible whitebind/list (nicolas.dorier) d541fa391844f658bd7035659b5b16695733dd56 Replace the use of fWhitelisted by permission checks (nicolas.dorier) ecd5cf7ea4c3644a30092100ffc399e30e193275 Do not disconnect peer for asking mempool if it has NO_BAN permission (nicolas.dorier) e5b26deaaa6842f7dd7c4537ede000f965ea0189 Make whitebind/whitelist permissions more flexible (nicolas.dorier) Pull request description: # Motivation In 0.19, bloom filter will be disabled by default. I tried to make [a PR](https://github.com/bitcoin/bitcoin/pull/16176) to enable bloom filter for whitelisted peers regardless of `-peerbloomfilters`. Bloom filter have non existent privacy and server can omit filter's matches. However, both problems are completely irrelevant when you connect to your own node. If you connect to your own node, bloom filters are the most bandwidth efficient way to synchronize your light client without the need of some middleware like Electrum. It is also a superior alternative to BIP157 as it does not require to maintain an additional index and it would work well on pruned nodes. When I attempted to allow bloom filters for whitelisted peer, my proposal has been NACKed in favor of [a more flexible approach](https://github.com/bitcoin/bitcoin/pull/16176#issuecomment-500762907) which should allow node operator to set fine grained permissions instead of a global `whitelisted` attribute. Doing so will also make follow up idea very easy to implement in a backward compatible way. # Implementation details The PR propose a new format for `--white{list,bind}`. I added a way to specify permissions granted to inbound connection matching `white{list,bind}`. The following permissions exists: * ForceRelay * Relay * NoBan * BloomFilter * Mempool Example: * `-whitelist=bloomfilter@127.0.0.1/32`. * `-whitebind=bloomfilter,relay,noban@127.0.0.1:10020`. If no permissions are specified, `NoBan | Mempool` is assumed. (making this PR backward compatible) When we receive an inbound connection, we calculate the effective permissions for this peer by fetching the permissions granted from `whitelist` and add to it the permissions granted from `whitebind`. To keep backward compatibility, if no permissions are specified in `white{list,bind}` (e.g. `--whitelist=127.0.0.1`) then parameters `-whitelistforcerelay` and `-whiterelay` will add the permissions `ForceRelay` and `Relay` to the inbound node. `-whitelistforcerelay` and `-whiterelay` are ignored if the permissions flags are explicitly set in `white{bind,list}`. # Follow up idea Based on this PR, other changes become quite easy to code in a trivially review-able, backward compatible way: * Changing `connect` at rpc and config file level to understand the permissions flags. * Changing the permissions of a peer at RPC level. ACKs for top commit: laanwj: re-ACK c5b404e8f1973afe071a07c63ba1038eefe13f0f Tree-SHA512: adfefb373d09e68cae401247c8fc64034e305694cdef104bdcdacb9f1704277bd53b18f52a2427a5cffdbc77bda410d221aed252bc2ece698ffbb9cf1b830577
2019-08-14 16:35:54 +02:00
'p2p_permissions.py',
'feature_blocksdir.py',
'wallet_startup.py',
'p2p_i2p_ports.py',
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
'feature_config_args.py',
'feature_settings.py',
'rpc_getdescriptorinfo.py',
Merge #17578: rpc: simplify getaddressinfo labels, deprecate previous behavior 8925df86c4df16b1070343fef8e4d238f3cc3bd1 doc: update release notes (Jon Atack) 8bb405bbadf11391ccba7b334b4cfe66dc85b390 test: getaddressinfo labels purpose deprecation test (Jon Atack) 60aba1f2f11529add115d963d05599130288ae28 rpc: simplify getaddressinfo labels, deprecate previous behavior (Jon Atack) 7851f14ccf2bcd1e9b2ad48e5e08881be06d9d21 rpc: incorporate review feedback from PR 17283 (Jon Atack) Pull request description: This PR builds on #17283 (now merged) and is followed by #17585. It modifies the value returned by rpc getaddressinfo `labels` to an array of label name strings and deprecates the previous behavior of returning an array of JSON hash structures containing label `name` and address `purpose` key/value pairs. before ``` "labels": [ { "name": "DOUBLE SPEND", "purpose": "receive" } ``` after ``` "labels": [ "DOUBLE SPEND" ] ``` The deprecated behavior can be re-enabled by starting bitcoind with `-deprecatedrpc=labelspurpose`. For context, see: - https://github.com/bitcoin/bitcoin/pull/17283#issuecomment-554458001 - http://www.erisian.com.au/bitcoin-core-dev/log-2019-12-13.html#l-425 (lines 425-427) - http://www.erisian.com.au/bitcoin-core-dev/log-2019-11-22.html#l-622 Reviewers: This PR may be tested manually by building, then running bitcoind with and without the `-deprecatedrpc=labelspurpose` flag while verifying the rpc getaddressinfo help text and `labels` output. Next steps: deprecate the rpc getaddressinfo `label` field (EDIT: done in #17585) and add support for multiple labels per address. This PR will unblock those. ACKs for top commit: jnewbery: reACK 8925df8 promag: Code review ACK 8925df86c4df16b1070343fef8e4d238f3cc3bd1. meshcollider: Code review ACK 8925df86c4df16b1070343fef8e4d238f3cc3bd1 Tree-SHA512: c2b717209996da32b6484de7bb8800e7048410f9ce6afdb3e02a6866bd4a8f2c730f905fca27b10b877b91cf407f546e69e8c4feb9cd934325a6c71c166bd438
2020-01-07 23:23:58 +01:00
'rpc_getaddressinfo_labels_purpose_deprecation.py',
2020-02-02 09:35:24 +01:00
'rpc_getaddressinfo_label_deprecation.py',
'rpc_help.py',
Merge #12843: [tests] Test starting bitcoind with -h and -version 63048ec73d [tests] Test starting bitcoind with -h and -version (John Newbery) Pull request description: Test that starting bitcoind/bitcoin-qt with `-h` and `-version` works as expected. Prompted by https://github.com/bitcoin/bitcoin/pull/10762#commitcomment-28345993, which is a nullpointer dereference triggered by starting bitcoin-qt with `-h`. On master, this test passes when run over bitcoind, but fails when running over bitcoin-qt. I used xvfb as a virtual frame buffer to test: ``` BITCOIND=/home/ubuntu/bitcoin/src/qt/bitcoin-qt xvfb-run ./feature_help.py --nocleanup 2018-03-30T17:09:37.767000Z TestFramework (INFO): Initializing test directory /tmp/user/1000/testdi4dre13 2018-03-30T17:09:37.767000Z TestFramework (INFO): Start bitcoin with -h for help text 2018-03-30T17:09:37.841000Z TestFramework (ERROR): Assertion failed Traceback (most recent call last): File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 126, in main self.run_test() File "./feature_help.py", line 25, in run_test assert_equal(ret_code, 0) File "/home/ubuntu/bitcoin/test/functional/test_framework/util.py", line 39, in assert_equal raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args)) AssertionError: not(-11 == 0) 2018-03-30T17:09:37.842000Z TestFramework (INFO): Stopping nodes Traceback (most recent call last): File "./feature_help.py", line 42, in <module> HelpTest().main() File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 149, in main self.stop_nodes() File "/home/ubuntu/bitcoin/test/functional/test_framework/test_framework.py", line 273, in stop_nodes node.stop_node() File "/home/ubuntu/bitcoin/test/functional/test_framework/test_node.py", line 141, in stop_node self.stop() File "/home/ubuntu/bitcoin/test/functional/test_framework/test_node.py", line 87, in __getattr__ assert self.rpc_connected and self.rpc is not None, "Error: no RPC connection" AssertionError: Error: no RPC connection ``` Passes for bitcoind and bitcoin-qt when run on #12836. Longer term, we should consider running functional tests over bitcoin-qt in one of the Travis jobs. Tree-SHA512: 0c2f40f3d5f0e78c3a1b07dbee8fd383eebab27ed0bf2a98a5b9cc66613dbd7b70e363c56163a37e02f68ae7ff7b3ae1769705d0e110ca68a00f8693315730a4
2018-04-01 21:27:44 +02:00
'feature_help.py',
'feature_blockfilterindex_prune.py'
# Don't append tests at the end to avoid merge conflicts
# Put them in a random line within the section that fits their approximate run-time
]
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
# Place EXTENDED_SCRIPTS first since it has the 3 longest running tests
ALL_SCRIPTS = EXTENDED_SCRIPTS + BASE_SCRIPTS
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
NON_SCRIPTS = [
# These are python files that live in the functional tests directory, but are not test scripts.
"combine_logs.py",
"create_cache.py",
"test_runner.py",
]
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
def main():
# Parse arguments and pass through unrecognised args
parser = argparse.ArgumentParser(add_help=False,
usage='%(prog)s [test_runner.py options] [script options] [scripts]',
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
description=__doc__,
epilog='''
Help text and arguments for individual test script:''',
formatter_class=argparse.RawTextHelpFormatter)
Merge #16561: tests: Use colors and dots in test_runner.py output only if standard output is a terminal 37f2784952cb6f598f82922f9ce71d40c9d74e26 tests: Use colors and dots in test_runner.py output only if standard output is a terminal -- allows for using the test runner output as input to other programs (practicalswift) Pull request description: Use colors and dots in `test_runner.py` output only if standard output is a terminal -- allows for using the test runner output as input to other programs. I found the need for this when parsing `test_runner.py` output while investigating intermittent functional test failures. Before: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074115 ESC[1mWARNING!ESC[0m There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] .......................................^M ^M1/1 - ESC[1mwallet_hd.pyESC[0m passed, Duration: 20 s ESC[1mTEST | STATUS | DURATION ESC[0mESC[0;32mwallet_hd.py | ✓ Passed | 20 s ESC[0mESC[1m ALL | ✓ Passed | 20 s (accumulated) ESC[0mRuntime: 20 s ``` After: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074244 1/1 - wallet_hd.py passed, Duration: 20 s WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] TEST | STATUS | DURATION wallet_hd.py | ✓ Passed | 20 s ALL | ✓ Passed | 20 s (accumulated) Runtime: 20 s ``` ACKs for top commit: laanwj: ACK 37f2784952cb6f598f82922f9ce71d40c9d74e26 Tree-SHA512: f15d95f9e07de2954c326d63d7a4bcd2971faeaa00386600dec2fb915ec89475aeef1dbc968b2c12aa5e988d4b3ed1974d6da0b6a3f1e1a105cfd90e8cb97cf6
2019-08-15 13:42:20 +02:00
parser.add_argument('--ansi', action='store_true', default=sys.stdout.isatty(), help="Use ANSI colors and dots in output (enabled by default when standard output is a TTY)")
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
parser.add_argument('--attempts', '-a', type=int, default=1, help='how many attempts should be allowed for the non-deterministic test suite. Default=1.')
parser.add_argument('--combinedlogslen', '-c', type=int, default=0, metavar='n', help='On failure, print a log (of length n lines) to the console, combined from the test framework and all test nodes.')
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
parser.add_argument('--coverage', action='store_true', help='generate a basic coverage report for the RPC interface')
parser.add_argument('--ci', action='store_true', help='Run checks and code that are usually only enabled in a continuous integration environment')
parser.add_argument('--exclude', '-x', help='specify a comma-separated-list of scripts to exclude.')
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
parser.add_argument('--extended', action='store_true', help='run the extended test suite in addition to the basic tests')
parser.add_argument('--help', '-h', '-?', action='store_true', help='print help text and exit')
parser.add_argument('--jobs', '-j', type=int, default=4, help='how many test scripts to run in parallel. Default=4.')
parser.add_argument('--keepcache', '-k', action='store_true', help='the default behavior is to flush the cache directory on startup. --keepcache retains the cache from the previous testrun.')
parser.add_argument('--quiet', '-q', action='store_true', help='only print dots, results summary and failure logs')
parser.add_argument('--tmpdirprefix', '-t', default=tempfile.gettempdir(), help="Root directory for datadirs")
parser.add_argument('--failfast', '-F', action='store_true', help='stop execution after the first test failure')
parser.add_argument('--filter', help='filter scripts to run by regular expression')
Merge #16561: tests: Use colors and dots in test_runner.py output only if standard output is a terminal 37f2784952cb6f598f82922f9ce71d40c9d74e26 tests: Use colors and dots in test_runner.py output only if standard output is a terminal -- allows for using the test runner output as input to other programs (practicalswift) Pull request description: Use colors and dots in `test_runner.py` output only if standard output is a terminal -- allows for using the test runner output as input to other programs. I found the need for this when parsing `test_runner.py` output while investigating intermittent functional test failures. Before: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074115 ESC[1mWARNING!ESC[0m There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] .......................................^M ^M1/1 - ESC[1mwallet_hd.pyESC[0m passed, Duration: 20 s ESC[1mTEST | STATUS | DURATION ESC[0mESC[0;32mwallet_hd.py | ✓ Passed | 20 s ESC[0mESC[1m ALL | ✓ Passed | 20 s (accumulated) ESC[0mRuntime: 20 s ``` After: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074244 1/1 - wallet_hd.py passed, Duration: 20 s WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] TEST | STATUS | DURATION wallet_hd.py | ✓ Passed | 20 s ALL | ✓ Passed | 20 s (accumulated) Runtime: 20 s ``` ACKs for top commit: laanwj: ACK 37f2784952cb6f598f82922f9ce71d40c9d74e26 Tree-SHA512: f15d95f9e07de2954c326d63d7a4bcd2971faeaa00386600dec2fb915ec89475aeef1dbc968b2c12aa5e988d4b3ed1974d6da0b6a3f1e1a105cfd90e8cb97cf6
2019-08-15 13:42:20 +02:00
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
args, unknown_args = parser.parse_known_args()
Merge #16561: tests: Use colors and dots in test_runner.py output only if standard output is a terminal 37f2784952cb6f598f82922f9ce71d40c9d74e26 tests: Use colors and dots in test_runner.py output only if standard output is a terminal -- allows for using the test runner output as input to other programs (practicalswift) Pull request description: Use colors and dots in `test_runner.py` output only if standard output is a terminal -- allows for using the test runner output as input to other programs. I found the need for this when parsing `test_runner.py` output while investigating intermittent functional test failures. Before: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074115 ESC[1mWARNING!ESC[0m There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] .......................................^M ^M1/1 - ESC[1mwallet_hd.pyESC[0m passed, Duration: 20 s ESC[1mTEST | STATUS | DURATION ESC[0mESC[0;32mwallet_hd.py | ✓ Passed | 20 s ESC[0mESC[1m ALL | ✓ Passed | 20 s (accumulated) ESC[0mRuntime: 20 s ``` After: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074244 1/1 - wallet_hd.py passed, Duration: 20 s WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] TEST | STATUS | DURATION wallet_hd.py | ✓ Passed | 20 s ALL | ✓ Passed | 20 s (accumulated) Runtime: 20 s ``` ACKs for top commit: laanwj: ACK 37f2784952cb6f598f82922f9ce71d40c9d74e26 Tree-SHA512: f15d95f9e07de2954c326d63d7a4bcd2971faeaa00386600dec2fb915ec89475aeef1dbc968b2c12aa5e988d4b3ed1974d6da0b6a3f1e1a105cfd90e8cb97cf6
2019-08-15 13:42:20 +02:00
if not args.ansi:
global DEFAULT, BOLD, GREEN, RED
DEFAULT = ("", "")
Merge #16561: tests: Use colors and dots in test_runner.py output only if standard output is a terminal 37f2784952cb6f598f82922f9ce71d40c9d74e26 tests: Use colors and dots in test_runner.py output only if standard output is a terminal -- allows for using the test runner output as input to other programs (practicalswift) Pull request description: Use colors and dots in `test_runner.py` output only if standard output is a terminal -- allows for using the test runner output as input to other programs. I found the need for this when parsing `test_runner.py` output while investigating intermittent functional test failures. Before: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074115 ESC[1mWARNING!ESC[0m There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] .......................................^M ^M1/1 - ESC[1mwallet_hd.pyESC[0m passed, Duration: 20 s ESC[1mTEST | STATUS | DURATION ESC[0mESC[0;32mwallet_hd.py | ✓ Passed | 20 s ESC[0mESC[1m ALL | ✓ Passed | 20 s (accumulated) ESC[0mRuntime: 20 s ``` After: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074244 1/1 - wallet_hd.py passed, Duration: 20 s WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] TEST | STATUS | DURATION wallet_hd.py | ✓ Passed | 20 s ALL | ✓ Passed | 20 s (accumulated) Runtime: 20 s ``` ACKs for top commit: laanwj: ACK 37f2784952cb6f598f82922f9ce71d40c9d74e26 Tree-SHA512: f15d95f9e07de2954c326d63d7a4bcd2971faeaa00386600dec2fb915ec89475aeef1dbc968b2c12aa5e988d4b3ed1974d6da0b6a3f1e1a105cfd90e8cb97cf6
2019-08-15 13:42:20 +02:00
BOLD = ("", "")
GREEN = ("", "")
RED = ("", "")
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
# args to be passed on always start with two dashes; tests are the remaining unknown args
tests = [arg for arg in unknown_args if arg[:2] != "--"]
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
passon_args = [arg for arg in unknown_args if arg[:2] == "--"]
# Read config generated by configure.
config = configparser.ConfigParser()
configfile = os.path.abspath(os.path.dirname(__file__)) + "/../config.ini"
config.read_file(open(configfile, encoding="utf8"))
passon_args.append("--configfile=%s" % configfile)
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
# Set up logging
logging_level = logging.INFO if args.quiet else logging.DEBUG
logging.basicConfig(format='%(message)s', level=logging_level)
# Create base test directory
tmpdir = "%s/test_runner_∋_🏃_%s" % (args.tmpdirprefix, datetime.datetime.now().strftime("%Y%m%d_%H%M%S"))
os.makedirs(tmpdir)
logging.debug("Temporary test directory at %s" % tmpdir)
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
enable_bitcoind = config["components"].getboolean("ENABLE_BITCOIND")
if not enable_bitcoind:
print("No functional tests to run.")
print("Rerun ./configure with --with-daemon and then make")
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
sys.exit(0)
# Build list of tests
test_list = []
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
if tests:
# Individual tests have been specified. Run specified tests that exist
Merge #16374: test: Enable passing wildcard test names to test runner from root e142ee03e7a139168aa1dbf5910c616f60d25042 doc: describe how to pass wildcard names to test runner (Jon Atack) 6a7a70b8cf05a82737c72020fd2b0eebc97cb5e4 test: enable passing wildcards with path to test runner (Jon Atack) Pull request description: Currently, passing wildcard testname args to the test runner from outside the test/functional/ directory does not work, even though developers expect it to. See these recent IRC discussions for more background: http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-10.html#l-262 (lines 262 to 323) and http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-11.html#l-134. 1. [BUGFIX] Enable passing wildcards with paths. Examples: - `test/functional/test_runner.py test/functional/wallet*` - `functional/test_runner.py functional/wallet*` - `test/functional/test_runner.py ./test/functional/tool* test/functional/mempool*` - A current limitation this PR does not change: 9 test files with arguments in their filename are not picked up by wildcard search. 2. [Docs] Describe how to pass wildcard names (multiple and with paths) to the test runner in test/README.md. ACKs for top commit: jnewbery: tested ACK e142ee03e7a139168aa1dbf5910c616f60d25042 jachiang: Tested ACK https://github.com/bitcoin/bitcoin/commit/e142ee03e7a139168aa1dbf5910c616f60d25042. Thanks a lot for this fix! MarcoFalke: ACK e142ee03e7a139168aa1dbf5910c616f60d25042, fine with me Tree-SHA512: cb3d994880cdc9b8918546b573a25faa5b4c7339826ac7cfe20f076aac6e731a34271609c0cf5a7ee5e4a2d5ae205298319d24bf36ef5b5d569a1a0c57883e54
2019-07-18 03:39:24 +02:00
# in the ALL_SCRIPTS list. Accept names with or without a .py extension.
# Specified tests can contain wildcards, but in that case the supplied
# paths should be coherent, e.g. the same path as that provided to call
# test_runner.py. Examples:
# `test/functional/test_runner.py test/functional/wallet*`
# `test/functional/test_runner.py ./test/functional/wallet*`
# `test_runner.py wallet*`
# but not:
# `test/functional/test_runner.py wallet*`
# Multiple wildcards can be passed:
# `test_runner.py tool* mempool*`
for test in tests:
Merge #16374: test: Enable passing wildcard test names to test runner from root e142ee03e7a139168aa1dbf5910c616f60d25042 doc: describe how to pass wildcard names to test runner (Jon Atack) 6a7a70b8cf05a82737c72020fd2b0eebc97cb5e4 test: enable passing wildcards with path to test runner (Jon Atack) Pull request description: Currently, passing wildcard testname args to the test runner from outside the test/functional/ directory does not work, even though developers expect it to. See these recent IRC discussions for more background: http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-10.html#l-262 (lines 262 to 323) and http://www.erisian.com.au/bitcoin-core-dev/log-2019-07-11.html#l-134. 1. [BUGFIX] Enable passing wildcards with paths. Examples: - `test/functional/test_runner.py test/functional/wallet*` - `functional/test_runner.py functional/wallet*` - `test/functional/test_runner.py ./test/functional/tool* test/functional/mempool*` - A current limitation this PR does not change: 9 test files with arguments in their filename are not picked up by wildcard search. 2. [Docs] Describe how to pass wildcard names (multiple and with paths) to the test runner in test/README.md. ACKs for top commit: jnewbery: tested ACK e142ee03e7a139168aa1dbf5910c616f60d25042 jachiang: Tested ACK https://github.com/bitcoin/bitcoin/commit/e142ee03e7a139168aa1dbf5910c616f60d25042. Thanks a lot for this fix! MarcoFalke: ACK e142ee03e7a139168aa1dbf5910c616f60d25042, fine with me Tree-SHA512: cb3d994880cdc9b8918546b573a25faa5b4c7339826ac7cfe20f076aac6e731a34271609c0cf5a7ee5e4a2d5ae205298319d24bf36ef5b5d569a1a0c57883e54
2019-07-18 03:39:24 +02:00
script = test.split("/")[-1]
script = script + ".py" if ".py" not in script else script
matching_scripts = [s for s in ALL_SCRIPTS if s.startswith(script)]
if matching_scripts:
test_list.extend(matching_scripts)
else:
print("{}WARNING!{} Test '{}' not found in full test list.".format(BOLD[1], BOLD[0], test))
elif args.extended:
# Include extended tests
test_list += ALL_SCRIPTS
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
else:
# Run base tests only
test_list += BASE_SCRIPTS
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
Backports 0.15 pr2 (#2597) * Merge #9815: Trivial: use EXIT_ codes instead of magic numbers a87d02a use EXIT_ codes instead of magic numbers (Marko Bencun) * Merge #9801: Removed redundant parameter from mempool.PrioritiseTransaction eaea2bb Removed redundant parameter from mempool.PrioritiseTransaction (gubatron) * remove extra parameter (see 3a3745bb) in dash specific code * Merge #9819: Remove harmless read of unusued priority estimates bc8fd12 Remove harmless read of unusued priority estimates (Alex Morcos) * Merge #9766: Add --exclude option to rpc-tests.py c578408 Add exclude option to rpc-tests.py (John Newbery) * Merge #9577: Fix docstrings in qa tests 3f95a80 Fix docstrings in qa tests (John Newbery) * Merge #9823: qa: Set correct path for binaries in rpc tests 3333ad0 qa: Set correct path for binaries in rpc tests (MarcoFalke) * Merge #9833: Trivial: fix comments referencing AppInit2 ef9f495 Trivial: fix comments referencing AppInit2 (Marko Bencun) * Merge #9612: [trivial] Rephrase the definition of difficulty. dc222f8 Trivial: Rephrase the definition of difficulty in the code. (Karl-Johan Alm) * Merge #9847: Extra test vector for BIP32 30aedcb BIP32 extra test vector (Pieter Wuille) * Merge #9839: [qa] Make import-rescan.py watchonly check reliable 864890a [qa] Make import-rescan.py watchonly check reliable (Russell Yanofsky) Tree-SHA512: ea0e2b1d4fc8f35174c3d575fb751b428daf6ad3aa944fad4e3ddcc9195e4f17051473acabc54203b1d27cca64cf911b737ab92e986c40ef384410652e2dbea1 * Change back file params
2019-01-07 10:55:35 +01:00
# Remove the test cases that the user has explicitly asked to exclude.
if args.exclude:
exclude_tests = [test.split('.py')[0] for test in args.exclude.split(',')]
for exclude_test in exclude_tests:
# Remove <test_name>.py and <test_name>.py --arg from the test list
exclude_list = [test for test in test_list if test.split('.py')[0] == exclude_test]
for exclude_item in exclude_list:
test_list.remove(exclude_item)
if not exclude_list:
print("{}WARNING!{} Test '{}' not found in current test list.".format(BOLD[1], BOLD[0], exclude_test))
Backports 0.15 pr2 (#2597) * Merge #9815: Trivial: use EXIT_ codes instead of magic numbers a87d02a use EXIT_ codes instead of magic numbers (Marko Bencun) * Merge #9801: Removed redundant parameter from mempool.PrioritiseTransaction eaea2bb Removed redundant parameter from mempool.PrioritiseTransaction (gubatron) * remove extra parameter (see 3a3745bb) in dash specific code * Merge #9819: Remove harmless read of unusued priority estimates bc8fd12 Remove harmless read of unusued priority estimates (Alex Morcos) * Merge #9766: Add --exclude option to rpc-tests.py c578408 Add exclude option to rpc-tests.py (John Newbery) * Merge #9577: Fix docstrings in qa tests 3f95a80 Fix docstrings in qa tests (John Newbery) * Merge #9823: qa: Set correct path for binaries in rpc tests 3333ad0 qa: Set correct path for binaries in rpc tests (MarcoFalke) * Merge #9833: Trivial: fix comments referencing AppInit2 ef9f495 Trivial: fix comments referencing AppInit2 (Marko Bencun) * Merge #9612: [trivial] Rephrase the definition of difficulty. dc222f8 Trivial: Rephrase the definition of difficulty in the code. (Karl-Johan Alm) * Merge #9847: Extra test vector for BIP32 30aedcb BIP32 extra test vector (Pieter Wuille) * Merge #9839: [qa] Make import-rescan.py watchonly check reliable 864890a [qa] Make import-rescan.py watchonly check reliable (Russell Yanofsky) Tree-SHA512: ea0e2b1d4fc8f35174c3d575fb751b428daf6ad3aa944fad4e3ddcc9195e4f17051473acabc54203b1d27cca64cf911b737ab92e986c40ef384410652e2dbea1 * Change back file params
2019-01-07 10:55:35 +01:00
if args.filter:
test_list = list(filter(re.compile(args.filter).search, test_list))
Backports 0.15 pr2 (#2597) * Merge #9815: Trivial: use EXIT_ codes instead of magic numbers a87d02a use EXIT_ codes instead of magic numbers (Marko Bencun) * Merge #9801: Removed redundant parameter from mempool.PrioritiseTransaction eaea2bb Removed redundant parameter from mempool.PrioritiseTransaction (gubatron) * remove extra parameter (see 3a3745bb) in dash specific code * Merge #9819: Remove harmless read of unusued priority estimates bc8fd12 Remove harmless read of unusued priority estimates (Alex Morcos) * Merge #9766: Add --exclude option to rpc-tests.py c578408 Add exclude option to rpc-tests.py (John Newbery) * Merge #9577: Fix docstrings in qa tests 3f95a80 Fix docstrings in qa tests (John Newbery) * Merge #9823: qa: Set correct path for binaries in rpc tests 3333ad0 qa: Set correct path for binaries in rpc tests (MarcoFalke) * Merge #9833: Trivial: fix comments referencing AppInit2 ef9f495 Trivial: fix comments referencing AppInit2 (Marko Bencun) * Merge #9612: [trivial] Rephrase the definition of difficulty. dc222f8 Trivial: Rephrase the definition of difficulty in the code. (Karl-Johan Alm) * Merge #9847: Extra test vector for BIP32 30aedcb BIP32 extra test vector (Pieter Wuille) * Merge #9839: [qa] Make import-rescan.py watchonly check reliable 864890a [qa] Make import-rescan.py watchonly check reliable (Russell Yanofsky) Tree-SHA512: ea0e2b1d4fc8f35174c3d575fb751b428daf6ad3aa944fad4e3ddcc9195e4f17051473acabc54203b1d27cca64cf911b737ab92e986c40ef384410652e2dbea1 * Change back file params
2019-01-07 10:55:35 +01:00
if not test_list:
print("No valid test scripts specified. Check that your test is in one "
"of the test lists in test_runner.py, or run test_runner.py with no arguments to run all tests")
Backports 0.15 pr2 (#2597) * Merge #9815: Trivial: use EXIT_ codes instead of magic numbers a87d02a use EXIT_ codes instead of magic numbers (Marko Bencun) * Merge #9801: Removed redundant parameter from mempool.PrioritiseTransaction eaea2bb Removed redundant parameter from mempool.PrioritiseTransaction (gubatron) * remove extra parameter (see 3a3745bb) in dash specific code * Merge #9819: Remove harmless read of unusued priority estimates bc8fd12 Remove harmless read of unusued priority estimates (Alex Morcos) * Merge #9766: Add --exclude option to rpc-tests.py c578408 Add exclude option to rpc-tests.py (John Newbery) * Merge #9577: Fix docstrings in qa tests 3f95a80 Fix docstrings in qa tests (John Newbery) * Merge #9823: qa: Set correct path for binaries in rpc tests 3333ad0 qa: Set correct path for binaries in rpc tests (MarcoFalke) * Merge #9833: Trivial: fix comments referencing AppInit2 ef9f495 Trivial: fix comments referencing AppInit2 (Marko Bencun) * Merge #9612: [trivial] Rephrase the definition of difficulty. dc222f8 Trivial: Rephrase the definition of difficulty in the code. (Karl-Johan Alm) * Merge #9847: Extra test vector for BIP32 30aedcb BIP32 extra test vector (Pieter Wuille) * Merge #9839: [qa] Make import-rescan.py watchonly check reliable 864890a [qa] Make import-rescan.py watchonly check reliable (Russell Yanofsky) Tree-SHA512: ea0e2b1d4fc8f35174c3d575fb751b428daf6ad3aa944fad4e3ddcc9195e4f17051473acabc54203b1d27cca64cf911b737ab92e986c40ef384410652e2dbea1 * Change back file params
2019-01-07 10:55:35 +01:00
sys.exit(0)
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
if args.help:
# Print help for test_runner.py, then print help of the first script (with args removed) and exit.
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
parser.print_help()
subprocess.check_call([sys.executable, os.path.join(config["environment"]["SRCDIR"], 'test', 'functional', test_list[0].split()[0]), '-h'])
sys.exit(0)
check_script_list(src_dir=config["environment"]["SRCDIR"], fail_on_warn=args.ci)
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
check_script_prefixes()
if not args.keepcache:
shutil.rmtree("%s/test/cache" % config["environment"]["BUILDDIR"], ignore_errors=True)
run_tests(
test_list=test_list,
src_dir=config["environment"]["SRCDIR"],
build_dir=config["environment"]["BUILDDIR"],
tmpdir=tmpdir,
jobs=args.jobs,
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
attempts=args.attempts,
enable_coverage=args.coverage,
args=passon_args,
combined_logs_len=args.combinedlogslen,
failfast=args.failfast,
Merge #16561: tests: Use colors and dots in test_runner.py output only if standard output is a terminal 37f2784952cb6f598f82922f9ce71d40c9d74e26 tests: Use colors and dots in test_runner.py output only if standard output is a terminal -- allows for using the test runner output as input to other programs (practicalswift) Pull request description: Use colors and dots in `test_runner.py` output only if standard output is a terminal -- allows for using the test runner output as input to other programs. I found the need for this when parsing `test_runner.py` output while investigating intermittent functional test failures. Before: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074115 ESC[1mWARNING!ESC[0m There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] .......................................^M ^M1/1 - ESC[1mwallet_hd.pyESC[0m passed, Duration: 20 s ESC[1mTEST | STATUS | DURATION ESC[0mESC[0;32mwallet_hd.py | ✓ Passed | 20 s ESC[0mESC[1m ALL | ✓ Passed | 20 s (accumulated) ESC[0mRuntime: 20 s ``` After: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074244 1/1 - wallet_hd.py passed, Duration: 20 s WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] TEST | STATUS | DURATION wallet_hd.py | ✓ Passed | 20 s ALL | ✓ Passed | 20 s (accumulated) Runtime: 20 s ``` ACKs for top commit: laanwj: ACK 37f2784952cb6f598f82922f9ce71d40c9d74e26 Tree-SHA512: f15d95f9e07de2954c326d63d7a4bcd2971faeaa00386600dec2fb915ec89475aeef1dbc968b2c12aa5e988d4b3ed1974d6da0b6a3f1e1a105cfd90e8cb97cf6
2019-08-15 13:42:20 +02:00
use_term_control=args.ansi,
)
def run_tests(*, test_list, src_dir, build_dir, tmpdir, jobs=1, attempts=1, enable_coverage=False, args=None, combined_logs_len=0,failfast=False, use_term_control):
args = args or []
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
# Warn if dashd is already running
try:
Merge #19368: test: improve functional tests compatibility with BSD/macOS 3a7e79478ab41af7c53ce14d9fca9815bffe1f73 test: retry when write to a socket fails on macOS (Ivan Metlushko) 8cf9d15b823d91d2a74fc83832fccca2219342c9 test: use pgrep for better compatibility (Ivan Metlushko) Pull request description: Rationale: a few minor changes to make experience of running tests on macOS a bit better 1.`pidof` is not available on BSD/macOS, while `pgrep` is present on BSD, Linux and macOS 2. Add retry as a workaround for a weird behavior when writing to a socket (https://bugs.python.org/issue33450). Stacktrace attached Man pages: https://www.freebsd.org/cgi/man.cgi?query=pgrep&apropos=0&sektion=1&manpath=FreeBSD+6.0-RELEASE&arch=default&format=html https://man7.org/linux/man-pages/man1/pgrep.1.html Related to #19281 Stacktrace example: ``` ... 33/161 - feature_abortnode.py failed, Duration: 63 s stdout: 2020-06-11T10:46:43.947000Z TestFramework (INFO): Initializing test directory /var/folders/2q/d5w9zh614r7g5c8r74ln3g400000gq/T/test_runner_₿_🏃_20200611_174102/feature_abortnode_128 2020-06-11T10:46:45.199000Z TestFramework (INFO): Waiting for crash 2020-06-11T10:47:15.921000Z TestFramework (INFO): Node crashed - now verifying restart fails 2020-06-11T10:47:47.068000Z TestFramework (INFO): Stopping nodes [node 1] Cleaning up leftover process stderr: Traceback (most recent call last): File "/Users/xxx/Projects/bitcoin/test/functional/feature_abortnode.py", line 50, in <module> AbortNodeTest().main() File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/test_framework.py", line 142, in main exit_code = self.shutdown() File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/test_framework.py", line 266, in shutdown self.stop_nodes() File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/test_framework.py", line 515, in stop_nodes node.stop_node(wait=wait) File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/test_node.py", line 318, in stop_node self.stop(wait=wait) File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/coverage.py", line 47, in __call__ return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs) File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/authproxy.py", line 142, in __call__ response, status = self._request('POST', self.__url.path, postdata.encode('utf-8')) File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/authproxy.py", line 107, in _request self.__conn.request(method, path, postdata, headers) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 1107, in request self._send_request(method, url, body, headers) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 1152, in _send_request self.endheaders(body) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 1103, in endheaders self._send_output(message_body) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 936, in _send_output self.send(message_body) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 908, in send self.sock.sendall(data) OSError: [Errno 41] Protocol wrong type for socket ``` ACKs for top commit: laanwj: ACK 3a7e79478ab41af7c53ce14d9fca9815bffe1f73 Tree-SHA512: fefbe40ce94ab29f18bbbed2a434194b1384ffa5279b1d04db7a3708e3dd422bd9e450f1db3f95a1a851fac5a626ab533c6ebcfd7ede96f8ccae9e6f3e9fff92
2020-07-01 15:06:29 +02:00
# pgrep exits with code zero when one or more matching processes found
if subprocess.run(["pgrep", "-x", "dashd"], stdout=subprocess.DEVNULL).returncode == 0:
print("%sWARNING!%s There is already a dashd process running on this system. Tests may fail unexpectedly due to resource contention!" % (BOLD[1], BOLD[0]))
Merge #19368: test: improve functional tests compatibility with BSD/macOS 3a7e79478ab41af7c53ce14d9fca9815bffe1f73 test: retry when write to a socket fails on macOS (Ivan Metlushko) 8cf9d15b823d91d2a74fc83832fccca2219342c9 test: use pgrep for better compatibility (Ivan Metlushko) Pull request description: Rationale: a few minor changes to make experience of running tests on macOS a bit better 1.`pidof` is not available on BSD/macOS, while `pgrep` is present on BSD, Linux and macOS 2. Add retry as a workaround for a weird behavior when writing to a socket (https://bugs.python.org/issue33450). Stacktrace attached Man pages: https://www.freebsd.org/cgi/man.cgi?query=pgrep&apropos=0&sektion=1&manpath=FreeBSD+6.0-RELEASE&arch=default&format=html https://man7.org/linux/man-pages/man1/pgrep.1.html Related to #19281 Stacktrace example: ``` ... 33/161 - feature_abortnode.py failed, Duration: 63 s stdout: 2020-06-11T10:46:43.947000Z TestFramework (INFO): Initializing test directory /var/folders/2q/d5w9zh614r7g5c8r74ln3g400000gq/T/test_runner_₿_🏃_20200611_174102/feature_abortnode_128 2020-06-11T10:46:45.199000Z TestFramework (INFO): Waiting for crash 2020-06-11T10:47:15.921000Z TestFramework (INFO): Node crashed - now verifying restart fails 2020-06-11T10:47:47.068000Z TestFramework (INFO): Stopping nodes [node 1] Cleaning up leftover process stderr: Traceback (most recent call last): File "/Users/xxx/Projects/bitcoin/test/functional/feature_abortnode.py", line 50, in <module> AbortNodeTest().main() File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/test_framework.py", line 142, in main exit_code = self.shutdown() File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/test_framework.py", line 266, in shutdown self.stop_nodes() File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/test_framework.py", line 515, in stop_nodes node.stop_node(wait=wait) File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/test_node.py", line 318, in stop_node self.stop(wait=wait) File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/coverage.py", line 47, in __call__ return_val = self.auth_service_proxy_instance.__call__(*args, **kwargs) File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/authproxy.py", line 142, in __call__ response, status = self._request('POST', self.__url.path, postdata.encode('utf-8')) File "/Users/xxx/Projects/bitcoin/test/functional/test_framework/authproxy.py", line 107, in _request self.__conn.request(method, path, postdata, headers) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 1107, in request self._send_request(method, url, body, headers) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 1152, in _send_request self.endheaders(body) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 1103, in endheaders self._send_output(message_body) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 936, in _send_output self.send(message_body) File "/Users/xxx/.pyenv/versions/3.5.6/lib/python3.5/http/client.py", line 908, in send self.sock.sendall(data) OSError: [Errno 41] Protocol wrong type for socket ``` ACKs for top commit: laanwj: ACK 3a7e79478ab41af7c53ce14d9fca9815bffe1f73 Tree-SHA512: fefbe40ce94ab29f18bbbed2a434194b1384ffa5279b1d04db7a3708e3dd422bd9e450f1db3f95a1a851fac5a626ab533c6ebcfd7ede96f8ccae9e6f3e9fff92
2020-07-01 15:06:29 +02:00
except OSError:
# pgrep not supported
pass
# Warn if there is a cache directory
cache_dir = "%s/test/cache" % build_dir
if os.path.isdir(cache_dir):
print("%sWARNING!%s There is a cache directory here: %s. If tests fail unexpectedly, try deleting the cache directory." % (BOLD[1], BOLD[0], cache_dir))
# Test Framework Tests
print("Running Unit Tests for Test Framework Modules")
test_framework_tests = unittest.TestSuite()
for module in TEST_FRAMEWORK_MODULES:
test_framework_tests.addTest(unittest.TestLoader().loadTestsFromName("test_framework.{}".format(module)))
result = unittest.TextTestRunner(verbosity=1, failfast=True).run(test_framework_tests)
if not result.wasSuccessful():
logging.debug("Early exiting after failure in TestFramework unit tests")
sys.exit(False)
tests_dir = src_dir + '/test/functional/'
flags = ['--cachedir={}'.format(cache_dir)] + args
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
if enable_coverage:
coverage = RPCCoverage()
flags.append(coverage.flag)
logging.debug("Initializing coverage directory at %s" % coverage.dir)
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
else:
coverage = None
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
if len(test_list) > 1 and jobs > 1:
# Populate cache
Merge #11466: Specify custom wallet directory with -walletdir param c1e5d40 Make debugging test crash easier (MeshCollider) 8263f6a Create walletdir if datadir doesn't exist and fix tests (MeshCollider) 9587a9c Default walletdir is wallets/ if it exists (MeshCollider) d987889 Add release notes for -walletdir and wallets/ dir (MeshCollider) 80c5cbc Add test for -walletdir (MeshCollider) 0530ba0 Add -walletdir parameter to specify custom wallet dir (MeshCollider) Pull request description: Closes #11348 Adds a `-walletdir` parameter which specifies a directory to use for wallets, allowing them to be stored separately from the 'main' data directory. Creates a new `wallets/` directory in datadir if this is the first time running, and defaults to using it if it exists. Includes tests and release notes. Things which might need to be considered more: - there is no 'lock' on the wallets directory, which might be needed? - because this uses a new wallets/ directory by default, downgrading to an earlier version won't see the wallets in that directory (not a big deal though, users can just copy them up to the main dir) - jnewbery suggested putting each wallet in its own directory, which is a good idea, but out of scope for this PR IMO. EDIT: this is being done in https://github.com/bitcoin/bitcoin/pull/11687 - doc/files.md needs updating (will do soon) I also considered including a cleanup by removing caching of data directory paths and instead just initialise them once on startup (c.f. #3073), but decided it wasn't super relevant here will just complicate review. Tree-SHA512: c8ac04bfe9a810c32055f2c8b8fa0d535e56125ceb8d96f12447dd3538bf3e5ee992b60b1cd2173bf5f3fa023a9feab12c9963593bf27ed419df929bb413398d
2017-11-18 14:32:50 +01:00
try:
subprocess.check_output([sys.executable, tests_dir + 'create_cache.py'] + flags + ["--tmpdir=%s/cache" % tmpdir])
except subprocess.CalledProcessError as e:
sys.stdout.buffer.write(e.output)
raise
#Run Tests
job_queue = TestHandler(
num_tests_parallel=jobs,
tests_dir=tests_dir,
tmpdir=tmpdir,
test_list=test_list,
flags=flags,
Merge #16561: tests: Use colors and dots in test_runner.py output only if standard output is a terminal 37f2784952cb6f598f82922f9ce71d40c9d74e26 tests: Use colors and dots in test_runner.py output only if standard output is a terminal -- allows for using the test runner output as input to other programs (practicalswift) Pull request description: Use colors and dots in `test_runner.py` output only if standard output is a terminal -- allows for using the test runner output as input to other programs. I found the need for this when parsing `test_runner.py` output while investigating intermittent functional test failures. Before: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074115 ESC[1mWARNING!ESC[0m There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] .......................................^M ^M1/1 - ESC[1mwallet_hd.pyESC[0m passed, Duration: 20 s ESC[1mTEST | STATUS | DURATION ESC[0mESC[0;32mwallet_hd.py | ✓ Passed | 20 s ESC[0mESC[1m ALL | ✓ Passed | 20 s (accumulated) ESC[0mRuntime: 20 s ``` After: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074244 1/1 - wallet_hd.py passed, Duration: 20 s WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] TEST | STATUS | DURATION wallet_hd.py | ✓ Passed | 20 s ALL | ✓ Passed | 20 s (accumulated) Runtime: 20 s ``` ACKs for top commit: laanwj: ACK 37f2784952cb6f598f82922f9ce71d40c9d74e26 Tree-SHA512: f15d95f9e07de2954c326d63d7a4bcd2971faeaa00386600dec2fb915ec89475aeef1dbc968b2c12aa5e988d4b3ed1974d6da0b6a3f1e1a105cfd90e8cb97cf6
2019-08-15 13:42:20 +02:00
use_term_control=use_term_control,
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
attempts=attempts,
)
start_time = time.time()
test_results = []
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
max_len_name = len(max(test_list, key=len))
test_count = len(test_list)
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
for i in range(test_count):
test_result, testdir, stdout, stderr = job_queue.get_next()
test_results.append(test_result)
done_str = "{}/{} - {}{}{}".format(i + 1, test_count, BOLD[1], test_result.name, BOLD[0])
if test_result.status == "Passed":
logging.debug("%s passed, Duration: %s s" % (done_str, test_result.time))
elif test_result.status == "Skipped":
logging.debug("%s skipped" % (done_str))
else:
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
print("%s failed, Duration: %s s\n" % (done_str, test_result.time))
print(BOLD[1] + 'stdout:\n' + BOLD[0] + stdout + '\n')
print(BOLD[1] + 'stderr:\n' + BOLD[0] + stderr + '\n')
if combined_logs_len and os.path.isdir(testdir):
# Print the final `combinedlogslen` lines of the combined logs
print('{}Combine the logs and print the last {} lines ...{}'.format(BOLD[1], combined_logs_len, BOLD[0]))
print('\n============')
print('{}Combined log for {}:{}'.format(BOLD[1], testdir, BOLD[0]))
print('============\n')
combined_logs_args = [sys.executable, os.path.join(tests_dir, 'combine_logs.py'), testdir]
if BOLD[0]:
combined_logs_args += ['--color']
combined_logs, _ = subprocess.Popen(combined_logs_args, universal_newlines=True, stdout=subprocess.PIPE).communicate()
print("\n".join(deque(combined_logs.splitlines(), combined_logs_len)))
if failfast:
logging.debug("Early exiting after test failure")
break
print_results(test_results, max_len_name, (int(time.time() - start_time)))
if coverage:
coverage_passed = coverage.report_rpc_coverage()
logging.debug("Cleaning up coverage data")
coverage.cleanup()
else:
coverage_passed = True
# Clear up the temp directory if all subdirectories are gone
if not os.listdir(tmpdir):
os.rmdir(tmpdir)
all_passed = all(map(lambda test_result: test_result.was_successful, test_results)) and coverage_passed
# Clean up dangling processes if any. This may only happen with --failfast option.
# Killing the process group will also terminate the current process but that is
# not an issue
if not os.getenv("CI_FAILFAST_TEST_LEAVE_DANGLING") and len(job_queue.jobs):
os.killpg(os.getpgid(0), signal.SIGKILL)
sys.exit(not all_passed)
def print_results(test_results, max_len_name, runtime):
results = "\n" + BOLD[1] + "%s | %s | %s\n\n" % ("TEST".ljust(max_len_name), "STATUS ", "DURATION") + BOLD[0]
test_results.sort(key=TestResult.sort_key)
all_passed = True
time_sum = 0
for test_result in test_results:
all_passed = all_passed and test_result.was_successful
time_sum += test_result.time
test_result.padding = max_len_name
results += str(test_result)
status = TICK + "Passed" if all_passed else CROSS + "Failed"
if not all_passed:
results += RED[1]
results += BOLD[1] + "\n%s | %s | %s s (accumulated) \n" % ("ALL".ljust(max_len_name), status.ljust(9), time_sum) + BOLD[0]
if not all_passed:
results += RED[0]
results += "Runtime: %s s\n" % (runtime)
print(results)
class TestHandler:
"""
Trigger the test scripts passed in via the list.
"""
def __init__(self, *, num_tests_parallel, tests_dir, tmpdir, test_list, flags, use_term_control, attempts):
assert num_tests_parallel >= 1
self.num_jobs = num_tests_parallel
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
self.tests_dir = tests_dir
self.tmpdir = tmpdir
self.test_list = test_list
self.flags = flags
self.num_running = 0
self.jobs = []
Merge #16561: tests: Use colors and dots in test_runner.py output only if standard output is a terminal 37f2784952cb6f598f82922f9ce71d40c9d74e26 tests: Use colors and dots in test_runner.py output only if standard output is a terminal -- allows for using the test runner output as input to other programs (practicalswift) Pull request description: Use colors and dots in `test_runner.py` output only if standard output is a terminal -- allows for using the test runner output as input to other programs. I found the need for this when parsing `test_runner.py` output while investigating intermittent functional test failures. Before: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074115 ESC[1mWARNING!ESC[0m There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] .......................................^M ^M1/1 - ESC[1mwallet_hd.pyESC[0m passed, Duration: 20 s ESC[1mTEST | STATUS | DURATION ESC[0mESC[0;32mwallet_hd.py | ✓ Passed | 20 s ESC[0mESC[1m ALL | ✓ Passed | 20 s (accumulated) ESC[0mRuntime: 20 s ``` After: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074244 1/1 - wallet_hd.py passed, Duration: 20 s WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] TEST | STATUS | DURATION wallet_hd.py | ✓ Passed | 20 s ALL | ✓ Passed | 20 s (accumulated) Runtime: 20 s ``` ACKs for top commit: laanwj: ACK 37f2784952cb6f598f82922f9ce71d40c9d74e26 Tree-SHA512: f15d95f9e07de2954c326d63d7a4bcd2971faeaa00386600dec2fb915ec89475aeef1dbc968b2c12aa5e988d4b3ed1974d6da0b6a3f1e1a105cfd90e8cb97cf6
2019-08-15 13:42:20 +02:00
self.use_term_control = use_term_control
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
self.attempts = attempts
def get_next(self):
while self.num_running < self.num_jobs and self.test_list:
# Add tests
self.num_running += 1
test = self.test_list.pop(0)
portseed = len(self.test_list)
portseed_arg = ["--portseed={}".format(portseed)]
log_stdout = tempfile.SpooledTemporaryFile(max_size=2**16)
log_stderr = tempfile.SpooledTemporaryFile(max_size=2**16)
test_argv = test.split()
2017-11-29 19:21:51 +01:00
testdir = "{}/{}_{}".format(self.tmpdir, re.sub(".py$", "", test_argv[0]), portseed)
tmpdir_arg = ["--tmpdir={}".format(testdir)]
self.jobs.append((test,
time.time(),
subprocess.Popen([sys.executable, self.tests_dir + test_argv[0]] + test_argv[1:] + self.flags + portseed_arg + tmpdir_arg,
universal_newlines=True,
stdout=log_stdout,
stderr=log_stderr),
2017-11-29 19:21:51 +01:00
testdir,
log_stdout,
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
log_stderr,
portseed,
1)) # attempt
if not self.jobs:
raise IndexError('pop from empty list')
# Print remaining running jobs when all jobs have been started.
if not self.test_list:
print("Remaining jobs: [{}]".format(", ".join(j[0] for j in self.jobs)))
dot_count = 0
while True:
# Return first proc that finishes
time.sleep(.5)
for job in self.jobs:
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
(name, start_time, proc, testdir, log_out, log_err, portseed, attempt) = job
if proc.poll() is not None:
log_out.seek(0), log_err.seek(0)
[stdout, stderr] = [log_file.read().decode('utf-8') for log_file in (log_out, log_err)]
log_out.close(), log_err.close()
if proc.returncode == TEST_EXIT_PASSED and stderr == "":
status = "Passed"
elif proc.returncode == TEST_EXIT_SKIPPED:
status = "Skipped"
fix(tests): Fix retries for the non-deterministic test suite (#5307) ## Issue being fixed or feature implemented the problem with retries implemented in #4793 is that they don't really do anything besides fetching results of a failed job multiple times 🙈 ## What was done? partially reverted changes done in #4793, implemented actual job restart. dropped `--sleep` and `--retries` and added `--attempts` instead. ## How Has This Been Tested? Pick any test, add some randomly failing expression somewhere and run it with some high number of retries. For example: ```diff diff --git a/test/functional/feature_dip0020_activation.py b/test/functional/feature_dip0020_activation.py index 471e4fdc66..b56a954b78 100755 --- a/test/functional/feature_dip0020_activation.py +++ b/test/functional/feature_dip0020_activation.py @@ -69,6 +69,7 @@ class DIP0020ActivationTest(BitcoinTestFramework): # Should be spendable now tx0id = self.node.sendrawtransaction(tx0_hex) assert tx0id in set(self.node.getrawmempool()) + assert int(tx0id[0], 16) < 4 if __name__ == '__main__': ``` On develop: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --retries=100 --sleep=0 ``` if this fails on the first run, it keeps "failing" (simply fetching the same results actually) till the end. With this patch: ``` ./test/functional/test_runner.py feature_dip0020_activation.py --attempts=100 ``` if this fails on the first run, it can actually succeed after a few attempts now, unless you are extremely unlucky ofc 😄 Also, check [ci results in my repo ](https://cdn.artifacts.gitlab-static.net/93/b4/93b4f8b17e5dcccab1afee165b4d74d90f05800caf65d6c48a83a1a78c979587/2023_04_08/4081291268/4478867166/job.log?response-content-type=text%2Fplain%3B%20charset%3Dutf-8&response-content-disposition=inline&Expires=1680945516&KeyName=gprd-artifacts-cdn&Signature=2d4mHCJBbgRaTDiSQ6kKIy1PdIM=). Note: ``` ... feature_dip3_v19.py failed at attempt 1/3, Duration: 159s ... 4/179 - feature_dip3_v19.py passed, Duration: 244 s ... feature_llmq_hpmn.py failed at attempt 1/3, Duration: 284s ... feature_llmq_hpmn.py failed at attempt 2/3, Duration: 296s ... 11/179 - feature_llmq_hpmn.py failed, Duration: 233 s ... ``` An example with 2 tests failing initially and then passing: https://gitlab.com/dashpay/dash/-/jobs/4089689970 ## Breaking Changes n/a ## Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have added or updated relevant unit/integration/functional/e2e tests - [ ] I have made corresponding changes to the documentation **For repository code-owners and collaborators only** - [x] I have assigned this pull request to a milestone
2023-04-15 06:13:47 +02:00
elif attempt < self.attempts:
# cleanup
if self.use_term_control:
clearline = '\r' + (' ' * dot_count) + '\r'
print(clearline, end='', flush=True)
dot_count = 0
shutil.rmtree(testdir, ignore_errors=True)
self.jobs.remove(job)
print(f"{name} failed at attempt {attempt}/{self.attempts}, Duration: {int(time.time() - start_time)} s")
# start over
portseed_arg = ["--portseed={}".format(portseed)]
log_stdout = tempfile.SpooledTemporaryFile(max_size=2**16)
log_stderr = tempfile.SpooledTemporaryFile(max_size=2**16)
test_argv = name.split()
tmpdir_arg = ["--tmpdir={}".format(testdir)]
self.jobs.append((name,
time.time(),
subprocess.Popen([sys.executable, self.tests_dir + test_argv[0]] + test_argv[1:] + self.flags + portseed_arg + tmpdir_arg,
universal_newlines=True,
stdout=log_stdout,
stderr=log_stderr),
testdir,
log_stdout,
log_stderr,
portseed,
attempt + 1)) # attempt
# no results for now, move to the next job
continue
else:
status = "Failed"
self.num_running -= 1
self.jobs.remove(job)
Merge #16561: tests: Use colors and dots in test_runner.py output only if standard output is a terminal 37f2784952cb6f598f82922f9ce71d40c9d74e26 tests: Use colors and dots in test_runner.py output only if standard output is a terminal -- allows for using the test runner output as input to other programs (practicalswift) Pull request description: Use colors and dots in `test_runner.py` output only if standard output is a terminal -- allows for using the test runner output as input to other programs. I found the need for this when parsing `test_runner.py` output while investigating intermittent functional test failures. Before: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074115 ESC[1mWARNING!ESC[0m There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] .......................................^M ^M1/1 - ESC[1mwallet_hd.pyESC[0m passed, Duration: 20 s ESC[1mTEST | STATUS | DURATION ESC[0mESC[0;32mwallet_hd.py | ✓ Passed | 20 s ESC[0mESC[1m ALL | ✓ Passed | 20 s (accumulated) ESC[0mRuntime: 20 s ``` After: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074244 1/1 - wallet_hd.py passed, Duration: 20 s WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] TEST | STATUS | DURATION wallet_hd.py | ✓ Passed | 20 s ALL | ✓ Passed | 20 s (accumulated) Runtime: 20 s ``` ACKs for top commit: laanwj: ACK 37f2784952cb6f598f82922f9ce71d40c9d74e26 Tree-SHA512: f15d95f9e07de2954c326d63d7a4bcd2971faeaa00386600dec2fb915ec89475aeef1dbc968b2c12aa5e988d4b3ed1974d6da0b6a3f1e1a105cfd90e8cb97cf6
2019-08-15 13:42:20 +02:00
if self.use_term_control:
clearline = '\r' + (' ' * dot_count) + '\r'
print(clearline, end='', flush=True)
dot_count = 0
return TestResult(name, status, int(time.time() - start_time)), testdir, stdout, stderr
Merge #16561: tests: Use colors and dots in test_runner.py output only if standard output is a terminal 37f2784952cb6f598f82922f9ce71d40c9d74e26 tests: Use colors and dots in test_runner.py output only if standard output is a terminal -- allows for using the test runner output as input to other programs (practicalswift) Pull request description: Use colors and dots in `test_runner.py` output only if standard output is a terminal -- allows for using the test runner output as input to other programs. I found the need for this when parsing `test_runner.py` output while investigating intermittent functional test failures. Before: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074115 ESC[1mWARNING!ESC[0m There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] .......................................^M ^M1/1 - ESC[1mwallet_hd.pyESC[0m passed, Duration: 20 s ESC[1mTEST | STATUS | DURATION ESC[0mESC[0;32mwallet_hd.py | ✓ Passed | 20 s ESC[0mESC[1m ALL | ✓ Passed | 20 s (accumulated) ESC[0mRuntime: 20 s ``` After: ``` $ test/functional/test_runner.py wallet_hd.py > output 2>&1 $ less output Temporary test directory at /tmp/test_runner_₿_🏃_20190807_074244 1/1 - wallet_hd.py passed, Duration: 20 s WARNING! There is already a bitcoind process running on this system. Tests may fail unexpectedly due to resource contention! Remaining jobs: [wallet_hd.py] TEST | STATUS | DURATION wallet_hd.py | ✓ Passed | 20 s ALL | ✓ Passed | 20 s (accumulated) Runtime: 20 s ``` ACKs for top commit: laanwj: ACK 37f2784952cb6f598f82922f9ce71d40c9d74e26 Tree-SHA512: f15d95f9e07de2954c326d63d7a4bcd2971faeaa00386600dec2fb915ec89475aeef1dbc968b2c12aa5e988d4b3ed1974d6da0b6a3f1e1a105cfd90e8cb97cf6
2019-08-15 13:42:20 +02:00
if self.use_term_control:
print('.', end='', flush=True)
dot_count += 1
class TestResult():
def __init__(self, name, status, time):
self.name = name
self.status = status
self.time = time
self.padding = 0
def sort_key(self):
if self.status == "Passed":
return 0, self.name.lower()
elif self.status == "Failed":
return 2, self.name.lower()
elif self.status == "Skipped":
return 1, self.name.lower()
def __repr__(self):
if self.status == "Passed":
color = GREEN
glyph = TICK
elif self.status == "Failed":
color = RED
glyph = CROSS
elif self.status == "Skipped":
color = DEFAULT
glyph = CIRCLE
return color[1] + "%s | %s%s | %s s\n" % (self.name.ljust(self.padding), glyph, self.status.ljust(7), self.time) + color[0]
@property
def was_successful(self):
return self.status != "Failed"
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
def check_script_prefixes():
"""Check that test scripts start with one of the allowed name prefixes."""
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
good_prefixes_re = re.compile("^(example|feature|interface|mempool|mining|p2p|rpc|wallet|tool)_")
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
bad_script_names = [script for script in ALL_SCRIPTS if good_prefixes_re.match(script) is None]
if bad_script_names:
print("%sERROR:%s %d tests not meeting naming conventions:" % (BOLD[1], BOLD[0], len(bad_script_names)))
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
print(" %s" % ("\n ".join(sorted(bad_script_names))))
raise AssertionError("Some tests are not following naming convention!")
Backport 11796 + 11774 (#3612) * Merge #11796: [tests] Functional test naming convention 5fecd84 [tests] Remove redundant import in blocktools.py test (Anthony Towns) 9b20bb4 [tests] Check tests conform to naming convention (Anthony Towns) 7250b4e [tests] README.md nit fixes (Anthony Towns) 82b2712 [tests] move witness util functions to blocktools.py (John Newbery) 1e10854 [tests] [docs] update README for new test naming scheme (John Newbery) Pull request description: Splitting #11774 into two parts -- this part updates the README with the proposed naming convention, and adds some checks to test_runner.py that the number of tests violating the naming convention doesn't increase too much. Idea is this part of the change should not introduce merge conflicts or require much rebasing, so reviews of the complicated bits won't become invalidated too often; while the second part will just be file renames, which will require regular rebasing and will introduce merge conflicts with pending PRs, but can be merged later, and should also be much easier to review, since it will only include relatively trivial changes. Tree-SHA512: b96557d41714addbbfe2aed62fb5a48639eaeb1eb3aba30ac1b3a86bb3cb8d796c6247f9c414c4695c4bf54c0ec9968ac88e2f88fb62483bc1a2f89368f7fc80 * update violation count Signed-off-by: pasta <pasta@dashboost.org> * Merge #11774: [tests] Rename functional tests 6f881cc880 [tests] Remove EXPECTED_VIOLATION_COUNT (Anthony Towns) 3150b3fea7 [tests] Rename misc functional tests. (Anthony Towns) 81b79f2c39 [tests] Rename rpc_* functional tests. (Anthony Towns) 61b8f7f273 [tests] Rename p2p_* functional tests. (Anthony Towns) 90600bc7db [tests] Rename wallet_* functional tests. (Anthony Towns) ca6523d0c8 [tests] Rename feature_* functional tests. (Anthony Towns) Pull request description: This PR changes the functional tests to have a consistent naming scheme: tests for individual RPC methods are named rpc_... tests for interfaces (REST, ZMQ, RPC features) are named interface_... tests that explicitly test the p2p interface are named p2p_... tests for wallet features are named wallet_... tests for mining features are named mining_... tests for mempool behaviour are named mempool_... tests for full features that aren't wallet/mining/mempool are named feature_... Rationale: it's sometimes difficult for new contributors to know what's already covered by existing tests and where new tests should be added. Naming in a consistent fashion makes it easier to see what's already covered at a glance. Tree-SHA512: 4246790552d42bbd95f6d5bdf67702b81b3b2c583ce7eaf1fe6d8e254721279b47315973c6e9ae82dad6e4c747f12188160764bf2624c0f8f3b4d39330ec8b16 * rename tests and edit associated strings to align test-suite with test name standards Signed-off-by: pasta <pasta@dashboost.org> * fix grammar in test/functional/test_runner.py Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> * ci: Fix excluded test names * rename feature_privatesend.py to rpc_privatesend.py Signed-off-by: pasta <pasta@dashboost.org> Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: MarcoFalke <falke.marco@gmail.com> Co-authored-by: dustinface <35775977+xdustinface@users.noreply.github.com> Co-authored-by: xdustinface <xdustinfacex@gmail.com>
2020-07-17 01:44:20 +02:00
def check_script_list(*, src_dir, fail_on_warn):
"""Check scripts directory.
Check that there are no scripts in the functional tests directory which are
not being run by pull-tester.py."""
script_dir = src_dir + '/test/functional/'
python_files = set([test_file for test_file in os.listdir(script_dir) if test_file.endswith(".py")])
missed_tests = list(python_files - set(map(lambda x: x.split()[0], ALL_SCRIPTS + NON_SCRIPTS)))
if len(missed_tests) != 0:
print("%sWARNING!%s The following scripts are not being run: %s. Check the test lists in test_runner.py." % (BOLD[1], BOLD[0], str(missed_tests)))
if fail_on_warn:
# On CI this warning is an error to prevent merging incomplete commits into master
sys.exit(1)
class RPCCoverage():
"""
Coverage reporting utilities for test_runner.
Coverage calculation works by having each test script subprocess write
coverage files into a particular directory. These files contain the RPC
commands invoked during testing, as well as a complete listing of RPC
commands per `dash-cli help` (`rpc_interface.txt`).
After all tests complete, the commands run are combined and diff'd against
the complete list to calculate uncovered RPC commands.
See also: test/functional/test_framework/coverage.py
"""
def __init__(self):
self.dir = tempfile.mkdtemp(prefix="coverage")
self.flag = '--coveragedir=%s' % self.dir
def report_rpc_coverage(self):
"""
Print out RPC commands that were unexercised by tests.
"""
uncovered = self._get_uncovered_rpc_commands()
if uncovered:
print("Uncovered RPC commands:")
print("".join((" - %s\n" % command) for command in sorted(uncovered)))
return False
else:
print("All RPC commands covered.")
return True
def cleanup(self):
return shutil.rmtree(self.dir)
def _get_uncovered_rpc_commands(self):
"""
Return a set of currently untested RPC commands.
"""
# This is shared from `test/functional/test-framework/coverage.py`
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
reference_filename = 'rpc_interface.txt'
coverage_file_prefix = 'coverage.'
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
coverage_ref_filename = os.path.join(self.dir, reference_filename)
coverage_filenames = set()
all_cmds = set()
covered_cmds = set()
if not os.path.isfile(coverage_ref_filename):
raise RuntimeError("No coverage reference found")
with open(coverage_ref_filename, 'r', encoding="utf8") as coverage_ref_file:
all_cmds.update([line.strip() for line in coverage_ref_file.readlines()])
for root, dirs, files in os.walk(self.dir):
for filename in files:
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
if filename.startswith(coverage_file_prefix):
coverage_filenames.add(os.path.join(root, filename))
for filename in coverage_filenames:
with open(filename, 'r', encoding="utf8") as coverage_file:
covered_cmds.update([line.strip() for line in coverage_file.readlines()])
return all_cmds - covered_cmds
if __name__ == '__main__':
Backports 0.15 pr1 (#2590) * Merge #9744: Remove unused module from rpc-tests a432aa0 Remove unused module from rpc-tests (Takashi Mitsuta) * Merge #9696: [trivial] Fix recently introduced typos in comments 0c9b9b7 [trivial] Fix recently introduced typos in comments (practicalswift) * Merge #9657: Improve rpc-tests.py a6a3e58 Various review markups for rpc-tests.py improvements (John Newbery) 3de3ccd Refactor rpc-tests.py (John Newbery) afd38e7 Improve rpc-tests.py arguments (John Newbery) 91bffff Use argparse in rpc_tests.py (John Newbery) 1581ecb Use configparser in rpc-tests.py (John Newbery) * Merge #9724: Qt/Intro: Add explanation of IBD process f6d18f5 Qt/Intro: Explain a bit more what will happen first time (Luke Dashjr) 50c5657 Qt/Intro: Storage shouldn't grow significantly with pruning enabled (Luke Dashjr) 9adb694 Qt/Intro: Move sizeWarningLabel text into C++ code (Luke Dashjr) * Merge #9794: Minor update to qrencode package builder 1bfe6b4 Use package name variable inside $(package)_file_name variable (Mitchell Cash) * Merge #9726: netbase: Do not print an error on connection timeouts through proxy 3ddfe29 netbase: Do not print an error on connection timeouts through proxy (Wladimir J. van der Laan) 13f6085 netbase: Make InterruptibleRecv return an error code instead of bool (Wladimir J. van der Laan) * Merge #9727: Remove fallbacks for boost_filesystem < v3 056aba2 Remove fallbacks for boost_filesystem < v3 (Wladimir J. van der Laan) * Merge #9485: ZMQ example using python3 and asyncio b471daf Adddress nits, use asyncio signal handling, create_task (Bob McElrath) 4bb7d1b Add python version checks and 3.4 example (Bob McElrath) 5406d51 Rewrite to not use Polling wrapper for asyncio, link to python2.7 example (Bob McElrath) 5ea5368 ZMQ example using python3 and asyncio (Bob McElrath) * Merge #9807: RPC doc fix-ups. 851f6a3 [qa][doc] Correct rpc test options in readme (fanquake) 41e7219 [trivial] Add tests_config.ini to .gitignore (fanquake) * Dashify Co-Authored-By: PastaPastaPasta <pasta@dashboost.org> * Change file permissions * update travis.yml -parallel -> --jobs
2019-01-03 10:18:47 +01:00
main()