Commit Graph

114 Commits

Author SHA1 Message Date
Kittywhiskers Van Gogh
cbff29a630
partial bitcoin#20524: Move MIN_VERSION_SUPPORTED to p2p.py
excludes:
- 9f21ed4037758f407b536c0dd129f8da83173c79
2024-09-04 16:28:19 +00:00
Konstantin Akimov
4d9837c21e
refactor: add a new flag disable_mocktime to set_test_params()
It's better than re-implement setup_nodes each time when you want just disable mocktime.
It seems more error prune and the code is shorter
2024-09-01 18:27:19 +07:00
Konstantin Akimov
a42e9df06f
fix: createwallet to require 'load_on_startup' for descriptor wallets
createwallet has changed list of arguments: createwallet "wallet_name" ( disable_private_keys blank "passphrase" avoid_reuse descriptors load_on_startup )
load_on_startup used to be an argument 5 but now has a number 6.
Both arguments 5 and 6 are boolean and it can confuse an user.

To prevent confusion if user is not aware about this breaking changes,
the RPC createwallet throws an exception if user trying to create descriptor wallet but has not mentioned load_on_startup.
This requirement can be removed when major amount of users updated to v21
2024-07-07 21:56:16 +07:00
fanquake
6d44f36afd
Merge bitcoin/bitcoin#22096: p2p: AddrFetch - don't disconnect on self-announcements
5730a43703f7e5a5ca26245ba3b55fbdd027d0b6 test: Add functional test for AddrFetch connections (Martin Zumsande)
c34ad3309f93979b274a37de013502b05d25fad8 net, rpc: Enable AddrFetch connections for functional testing (Martin Zumsande)
533500d9072b7d5a36a6491784bdeb9247e91fb0 p2p: Add timeout for AddrFetch peers (Martin Zumsande)
b6c5d1e450dde6a54bd785504c923adfb45c7060 p2p: AddrFetch - don't disconnect on self-announcements (Martin Zumsande)

Pull request description:

  AddrFetch connections (old name: oneshots) are intended to be short-lived connections on which we ask a peer for addresses via `getaddr` and disconnect after receiving them.

  This is done by disconnecting after receiving the first `addr`. However, it is no longer working as intended, because nowadays, the first `addr` a typical bitcoin core node sends is its self-announcement.
  So we'll disconnect before the peer gets a chance to answer our `getaddr`.

  I checked that this affects both `-seednode` peers specified manually, and DNS seeds when AddrFetch is used as a fallback if DNS doesn't work for us.

  The current behavior of getting peers via AddrFetch when starting with an empty addrman would be to connect to the peer, receive its self-announcement and add it to addrman, disconnect, reconnect to the same peer again as a full outbound (no other addresses in addrman) and then receive more `addr`. This is silly and not in line with AddrFetch peer being intended to be short-lived peers. 

  Fix this by only disconnecting after receiving an `addr` message of size > 1.

  [Edit] As per review discussion, this PR now also adds a timeout after which we disconnect if we haven't received any suitable `addr`, and a functional test.

ACKs for top commit:
  amitiuttarwar:
    reACK 5730a43703f7e5a5ca26245ba3b55fbdd027d0b6
  naumenkogs:
    ACK 5730a43703f7e5a5ca26245ba3b55fbdd027d0b6
  jnewbery:
    ACK 5730a43703

Tree-SHA512: 8a81234f37e827705138eb254223f7f3b3bf44a06cb02126fc7990b0d231b9bd8f07d38d185cc30d55bf35548a6fdc286b69602498d875b937e7c58332158bf9
2024-05-27 12:18:23 -05:00
MarcoFalke
b2d889380c
Merge bitcoin/bitcoin#21792: test: Fix intermittent issue in p2p_segwit.py
fad6269916dbf8adc14d757a18f19c74e95cf659 test: Assert that exit code indicates failure (MarcoFalke)
faecb72c3ca744f1adb77bd910c643cedec3b445 test: Fix intermittent issue in p2p_segwit.py (MarcoFalke)

Pull request description:

  Calling `start_node` might call `wait_for_rpc_connection`, which will fail.

  https://cirrus-ci.com/task/5669555591708672?logs=ci#L3504

  ```
    File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/p2p_segwit.py", line 1974, in test_upgrade_after_activation
      self.start_node(2, extra_args=["-reindex", f"-segwitheight={SEGWIT_HEIGHT}"])
    File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 508, in start_node
      node.wait_for_rpc_connection()
    File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_node.py", line 224, in wait_for_rpc_connection
      raise FailedToStartError(self._node_msg(
  test_framework.test_node.FailedToStartError: [node 2] bitcoind exited with status 1 during initialization

ACKs for top commit:
  jnewbery:
    ACK fad6269916dbf8adc14d757a18f19c74e95cf659
  dhruv:
    ACK fad6269

Tree-SHA512: 4c5e39ce25e135717ea433258518f93f09d1c528c4538a8627d3da13bc0c0ba4b45911703c26392ff0f5e0cb7831a6c7cc53e6e29102d3da9c8cfce7cef333cc
2024-04-23 22:41:07 +07:00
fanquake
b6dbd8bd7d
Merge bitcoin/bitcoin#22092: test: convert documentation into type annotations
68ace23fa3bc01baa734ddf2c0963acae1c75ed1 test: convert docs into type annotations in test_framework/test_node.py (fanquake)
8bfcba36db974326d258c610456bd55cf5818b1e test: convert docs into type annotations in test_framework/wallet.py (fanquake)
b043ca8e8b65199061ebe4bbed2200504dfc6ce9 test: convert docs into type annotations in test_framework/util.py (fanquake)

Pull request description:

  Rather than having function types exist as documentation, make them type annotations, which enables more `mypy` checking.

ACKs for top commit:
  instagibbs:
    utACK 68ace23fa3

Tree-SHA512: b705f26b48baabde07b9b2c0a8d547b4dcca291b16eaf5ac70462bb3a1f9e9c2783d93a9d4290889d0cbb3f7db3671446754411a1f498b265d336f6ff33478df
2024-04-23 09:15:20 -05:00
Wladimir J. van der Laan
4774e1e8f6
Merge #19809: log: Prefix log messages with function name and source code location if -logsourcelocations is set
b4511e2e2ed1a6077ae6826a9ee6b7a311293d08 log: Prefix log messages with function name if -logsourcelocations is set (practicalswift)

Pull request description:

  Prefix log messages with function name if `-logfunctionnames` is set.

  Yes, exactly like `-logthreadnames` but for function names instead of thread names :)

  This is a small developer ergonomics improvement: I've found this to be a cheap/simple way to correlate log output and originating function.

  For me it beats the ordinary cycle of 1.) try to figure out a regexp matching the static part of the dynamic log message, 2.) `git grep -E 'Using .* MiB out of .* requested for signature cache'`, 3.) `mcedit filename.cpp` (`openemacs filename.cpp` works too!) and 4.) search for log message and scroll up to find the function name :)

  Without any logging parameters:

  ```
  $ src/bitcoind -regtest
  2020-08-25T03:29:04Z Using RdRand as an additional entropy source
  2020-08-25T03:29:04Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
  2020-08-25T03:29:04Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
  2020-08-25T03:29:04Z Loaded best chain: hashBestChain=0fff88f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22ff height=0 date=2011-02-02T23:16:42Z progress=1.000000
  2020-08-25T03:29:04Z block tree size = 1
  2020-08-25T03:29:04Z nBestHeight = 0
  2020-08-25T03:29:04Z Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
  2020-08-25T03:29:04Z 0 addresses found from DNS seeds
  ```

  With `-logthreadnames` and `-logfunctionnames`:

  ```
  $ src/bitcoind -regtest -logthreadnames -logfunctionnames
  2020-08-25T03:29:04Z [init] [ReportHardwareRand] Using RdRand as an additional entropy source
  2020-08-25T03:29:04Z [init] [InitSignatureCache] Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
  2020-08-25T03:29:04Z [init] [InitScriptExecutionCache] Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
  2020-08-25T03:29:04Z [init] [LoadChainTip] Loaded best chain: hashBestChain=0fff88f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22ff height=0 date=2011-02-02T23:16:42Z progress=1.000000
  2020-08-25T03:29:04Z [init] [AppInitMain] block tree size = 1
  2020-08-25T03:29:04Z [init] [AppInitMain] nBestHeight = 0
  2020-08-25T03:29:04Z [loadblk] [LoadMempool] Imported mempool transactions from disk: 0 succeeded, 0 failed, 0 expired, 0 already there, 0 waiting for initial broadcast
  2020-08-25T03:29:04Z [dnsseed] [ThreadDNSAddressSeed] 0 addresses found from DNS seeds
  ```

ACKs for top commit:
  laanwj:
    Code review ACK b4511e2e2ed1a6077ae6826a9ee6b7a311293d08
  MarcoFalke:
    review ACK b4511e2e2ed1a6077ae6826a9ee6b7a311293d08 🌃

Tree-SHA512: d100f5364630c323f31d275259864c597f7725e462d5f4bdedcc7033ea616d7fc0d16ef1b2af557e692f4deea73c6773ccfc681589e7bf6ba970b9ec169040c7
2024-04-11 02:25:08 +07:00
Konstantin Akimov
43a94f0580
fix: adjust functional tests due to dash's support of thread name after v0.12 2024-04-11 02:25:08 +07:00
MarcoFalke
085120d9f9
Merge #20993: test: store subversion (user agent) as string in msg_version
de85af5cce727981383ac0fe81f635451b331f23 test: store subversion (user agent) as string in msg_version (Sebastian Falbesoner)

Pull request description:

  It seems more natural to treat the "subversion" field (=user agent string, see [BIP 14](https://github.com/bitcoin/bips/blob/master/bip-0014.mediawiki#Proposal)) of a node as pure string rather than a bytestring within the test framework. This is also suggested with the naming prefix in `msg_version.strSubVer`: one probably wouldn't expect a field starting with "str" to be a bytestring that needs further decoding to be useful. This PR moves the encoding/decoding parts to the serialization/deserialization routines so that the user doesn't have to bother with that anymore.

  Note that currently, in the master branch the `msg_version.strSubVer` is never read (only in `msg_version.__repr__`); However, one issue that is solved by this PR came up while testing #19509 (not merged yet): A decoding script for binary message capture files takes use of the functional test framework convert it into JSON format. Bytestrings will be convered to hexstrings, while pure strings will (surprise surprise) end up without modification in the file.

  So without this patch, we get:

  ```
  $ jq . out.json | grep -m5 strSubVer
        "strSubVer": "2f5361746f7368693a32312e39392e302f"
        "strSubVer": "2f5361746f7368693a302e32302e312f"
        "strSubVer": "2f5361746f7368693a32312e39392e302f"
        "strSubVer": "2f5361746f7368693a302e32302e312f"
        "strSubVer": "2f5361746f7368693a32312e39392e302f"
  ```

  After this patch:

  ```
  $ jq . out2.json | grep -m5 strSubVer
        "strSubVer": "/Satoshi:21.99.0/"
        "strSubVer": "/Satoshi:0.20.1/"
        "strSubVer": "/Satoshi:21.99.0/"
        "strSubVer": "/Satoshi:0.20.1/"
        "strSubVer": "/Satoshi:21.99.0/"
  ```

ACKs for top commit:
  jnewbery:
    utACK de85af5cce727981383ac0fe81f635451b331f23

Tree-SHA512: ff23642705c858e8387a625537dfec82e6b8a15da6d99b8d12152560e52d243ba17431b602b26f60996d897e00e3f37dcf8dc8a303ffb1d544df29a5937080f9
2024-04-11 02:25:08 +07:00
Kittywhiskers Van Gogh
1d4f10a378
merge bitcoin#19315: Allow outbound & block-relay-only connections in functional tests 2024-04-03 16:06:40 +00:00
MarcoFalke
14121ec5f3
Merge #18888: test: Remove RPCOverloadWrapper boilerplate
faa26d374425f52e03efff3a575c391b7862abe5 test: Remove RPCOverloadWrapper boilerplate (MarcoFalke)

Pull request description:

  There are too many wrappers in test_node already, so at least the code that implements the wrappers should be as minimal as possible.

ACKs for top commit:
  laanwj:
    code review ACK faa26d374425f52e03efff3a575c391b7862abe5

Tree-SHA512: 94e593907de22187524e2445afb3101e40b3b599d4b4015aa8c6ca902d7586ff9daf520828759029d199a3af79e61b96b490a822a5a193ac7bf946beacb11a24
2024-03-07 01:23:19 +07:00
Andrew Chow
f293c046f4
Merge #16528: Native Descriptor Wallets using DescriptorScriptPubKeyMan
223588b1bbc63dc57098bbd0baa48635e0cc0b82 Add a --descriptors option to various tests (Andrew Chow)
869f7ab30aeb4d7fbd563c535b55467a8a0430cf tests: Add RPCOverloadWrapper which overloads some disabled RPCs (Andrew Chow)
cf060628590fab87d73f278e744d70ef2d5d81db Correctly check for default wallet (Andrew Chow)
886e0d75f5fea2421190aa4812777d89f68962cc Implement CWallet::IsSpentKey for non-LegacySPKMans (Andrew Chow)
3c19fdd2a2fd5394fcfa75b2ba84ab2277cbdabf Return error when no ScriptPubKeyMan is available for specified type (Andrew Chow)
388ba94231f2f10a0be751c562cdd4650510a90a Change wallet_encryption.py to use signmessage instead of dumpprivkey (Andrew Chow)
1346e14831489f9c8f53a08f9dfed61d55d53c6f Functional tests for descriptor wallets (Andrew Chow)
f193ea889ddb53d9a5c47647966681d525e38368 add importdescriptors RPC and tests for native descriptor wallets (Hugo Nguyen)
ce24a944940019185efebcc5d85eac458ed26016 Add IsLegacy to CWallet so that the GUI knows whether to show watchonly (Andrew Chow)
1cb42b22b11c27e64462afc25a94b2fc50bfa113 Generate new descriptors when encrypting (Andrew Chow)
82ae02b1656819f4bd5023b8955447e1d4ea8692 Be able to create new wallets with DescriptorScriptPubKeyMans as backing (Andrew Chow)
b713baa75a62335ab9c0eed9ef76a95bfec30668 Implement GetMetadata in DescriptorScriptPubKeyMan (Andrew Chow)
8b9603bd0b443e2f7984eb72bf2e21cf02af0bcb Change GetMetadata to use unique_ptr<CKeyMetadata> (Andrew Chow)
72a9540df96ffdb94f039b9c14eaacdc7d961196 Implement FillPSBT in DescriptorScriptPubKeyMan (Andrew Chow)
84b4978c02102171775c77a45f6ec198930f0a88 Implement SignMessage for descriptor wallets (Andrew Chow)
bde7c9fa38775a81d53ac0484fa9c98076a0c7d1 Implement SignTransaction in DescriptorScriptPubKeyMan (Andrew Chow)
d50c8ddd4190f20bf0debd410348b73408ec3143 Implement GetSolvingProvider for DescriptorScriptPubKeyMan (Andrew Chow)
f1ca5feb4ad668a3e1ae543d0addd5f483f1a88f Implement GetKeypoolOldestTime and only display it if greater than 0 (Andrew Chow)
586b57a9a6b4b12a78f792785b63a5a1743bce0c Implement ReturnDestination in DescriptorScriptPubKeyMan (Andrew Chow)
f866957979c23cefd41efa9dae9e53b9177818dc Implement GetReservedDestination in DescriptorScriptPubKeyMan (Andrew Chow)
a775f7c7fd0b9094fcbeee6ba92206d5bbb19164 Implement Unlock and Encrypt in DescriptorScriptPubKeyMan (Andrew Chow)
bfdd0734869a22217c15858d7a76d0dacc2ebc86 Implement GetNewDestination for DescriptorScriptPubKeyMan (Andrew Chow)
58c7651821b0eeff0a99dc61d78d2e9e07986580 Implement TopUp in DescriptorScriptPubKeyMan (Andrew Chow)
e014886a342508f7c8d80323eee9a5f314eaf94c Implement SetupGeneration for DescriptorScriptPubKeyMan (Andrew Chow)
46dfb99768e7d03a3cf552812d5b41ceaebc06be Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file (Andrew Chow)
4cb9b69be031e1dc65d8964794781b347fd948f5 Implement several simple functions in DescriptorScriptPubKeyMan (Andrew Chow)
d1ec3e4f19487b4b100f80ad02eac063c571777d Add IsSingleType to Descriptors (Andrew Chow)
953feb3d2724f5398dd48990c4957a19313d2c8c Implement loading of keys for DescriptorScriptPubKeyMan (Andrew Chow)
2363e9fcaa41b68bf11153f591b95f2d41ff9a1a Load the descriptor cache from the wallet file (Andrew Chow)
46c46aebb7943e1e2e96755e94dc6c197920bf75 Implement GetID for DescriptorScriptPubKeyMan (Andrew Chow)
ec2f9e1178c8e38c0a5ca063fe81adac8f916348 Implement IsHDEnabled in DescriptorScriptPubKeyMan (Andrew Chow)
741122d4c1a62ced3e96d16d67f4eeb3a6522d99 Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan (Andrew Chow)
2db7ca765c8fb2c71dd6f7c4f29ad70e68ff1720 Implement IsMine for DescriptorScriptPubKeyMan (Andrew Chow)
db7177af8c159abbcc209f2caafcd45d54c181c5 Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet (Andrew Chow)
78f8a92910d34247fa5d04368338c598d9908267 Implement SetType in DescriptorScriptPubKeyMan (Andrew Chow)
834de0300cde57ca3f662fb7aa5b1bdaed68bc8f Store WalletDescriptor in DescriptorScriptPubKeyMan (Andrew Chow)
d8132669e10c1db9ae0c2ea0d3f822d7d2f01345 Add a lock cs_desc_man for DescriptorScriptPubKeyMan (Andrew Chow)
3194a7f88ac1a32997b390b4f188c4b6a4af04a5 Introduce WalletDescriptor class (Andrew Chow)
6b13cd3fa854dfaeb9e269bff3d67cacc0e5b5dc Create LegacyScriptPubKeyMan when not a descriptor wallet (Andrew Chow)
aeac157c9dc141546b45e06ba9c2e641ad86083f Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet (Andrew Chow)
96accc73f067c7c95946e9932645dd821ef67f63 Add WALLET_FLAG_DESCRIPTORS (Andrew Chow)
6b8119af53ee2fdb4c4b5b24b4e650c0dc3bd27c Introduce DescriptorScriptPubKeyMan as a dummy class (Andrew Chow)
06620302c713cae65ee8e4ff9302e4c88e2a1285 Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it (Andrew Chow)

Pull request description:

  Introducing the wallet of the glorious future (again): native descriptor wallets. With native descriptor wallets, addresses are generated from descriptors. Instead of generating keys and deriving addresses from keys, addresses come from the scriptPubKeys produced by a descriptor. Native descriptor wallets will be optional for now and can only be created by using `createwallet`.

  Descriptor wallets will store descriptors, master keys from the descriptor, and descriptor cache entries. Keys are derived from descriptors on the fly. In order to allow choosing different address types, 6 descriptors are needed for normal use. There is a pair of primary and change descriptors for each of the 3 address types. With the default keypool size of 1000, each descriptor has 1000 scriptPubKeys and descriptor cache entries pregenerated. This has a side effect of making wallets large since 6000 pubkeys are written to the wallet by default, instead of the current 2000. scriptPubKeys are kept only in memory and are generated every time a descriptor is loaded. By default, we use the standard BIP 44, 49, 84 derivation paths with an external and internal derivation chain for each.

  Descriptors can also be imported with a new `importdescriptors` RPC.

  Native descriptor wallets use the `ScriptPubKeyMan` interface introduced in #16341 to add a `DescriptorScriptPubKeyMan`. This defines a different IsMine which uses the simpler model of "does this scriptPubKey exist in this wallet". Furthermore, `DescriptorScriptPubKeyMan` does not have watchonly, so with native descriptor wallets, it is not possible to have a wallet with both watchonly and non-watchonly things. Rather a wallet with `disable_private_keys` needs to be used for watchonly things.

  A `--descriptor` option was added to some tests (`wallet_basic.py`, `wallet_encryption.py`, `wallet_keypool.py`, `wallet_keypool_topup.py`, and `wallet_labels.py`) to allow for these tests to use descriptor wallets. Additionally, several RPCs are disabled for descriptor wallets (`importprivkey`, `importpubkey`, `importaddress`, `importmulti`, `addmultisigaddress`, `dumpprivkey`, `dumpwallet`, `importwallet`, and `sethdseed`).

ACKs for top commit:
  Sjors:
    utACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82 (rebased, nits addressed)
  jonatack:
    Code review re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82.
  fjahr:
    re-ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82
  instagibbs:
    light re-ACK 223588b
  meshcollider:
    Code review ACK 223588b1bbc63dc57098bbd0baa48635e0cc0b82

Tree-SHA512: 59bc52aeddbb769ed5f420d5d240d8137847ac821b588eb616b34461253510c1717d6a70bab8765631738747336ae06f45ba39603ccd17f483843e5ed9a90986

Introduce SetType function to tell ScriptPubKeyMans the type and internal-ness of it

Introduce DescriptorScriptPubKeyMan as a dummy class

Add WALLET_FLAG_DESCRIPTORS

Return nullptr from GetLegacyScriptPubKeyMan if descriptor wallet

Create LegacyScriptPubKeyMan when not a descriptor wallet

Introduce WalletDescriptor class

WalletDescriptor is a Descriptor with other wallet metadata

Add a lock cs_desc_man for DescriptorScriptPubKeyMan

Store WalletDescriptor in DescriptorScriptPubKeyMan

Implement SetType in DescriptorScriptPubKeyMan

Add LoadDescriptorScriptPubKeyMan and SetActiveScriptPubKeyMan to CWallet

Implement IsMine for DescriptorScriptPubKeyMan

Adds a set of scriptPubKeys that DescriptorScriptPubKeyMan tracks.
If the given script is in that set, it is considered ISMINE_SPENDABLE

Implement MarkUnusedAddresses in DescriptorScriptPubKeyMan

Implement IsHDEnabled in DescriptorScriptPubKeyMan

Implement GetID for DescriptorScriptPubKeyMan

Load the descriptor cache from the wallet file

Implement loading of keys for DescriptorScriptPubKeyMan

Add IsSingleType to Descriptors

IsSingleType will return whether the descriptor will give one or multiple scriptPubKeys

Implement several simple functions in DescriptorScriptPubKeyMan

Implements a bunch of one liners: UpgradeKeyMetadata, IsFirstRun, HavePrivateKeys,
KeypoolCountExternalKeys, GetKeypoolSize, GetTimeFirstKey, CanGetAddresses,
RewriteDB

Implement writing descriptorkeys, descriptorckeys, and descriptors to wallet file

Implement SetupGeneration for DescriptorScriptPubKeyMan

Implement TopUp in DescriptorScriptPubKeyMan

Implement GetNewDestination for DescriptorScriptPubKeyMan

Implement Unlock and Encrypt in DescriptorScriptPubKeyMan

Implement GetReservedDestination in DescriptorScriptPubKeyMan

Implement ReturnDestination in DescriptorScriptPubKeyMan

Implement GetKeypoolOldestTime and only display it if greater than 0

Implement GetSolvingProvider for DescriptorScriptPubKeyMan

Internally, a GetSigningProvider function is introduced which allows for
some private keys to be optionally included. This can be called with a
script as the argument (i.e. a scriptPubKey from our wallet when we are
signing) or with a pubkey. In order to know what index to expand the
private keys for that pubkey, we need to also cache all of the pubkeys
involved when we expand the descriptor. So SetCache and TopUp are
updated to do this too.

Implement SignTransaction in DescriptorScriptPubKeyMan

Implement SignMessage for descriptor wallets

Implement FillPSBT in DescriptorScriptPubKeyMan

FillPSBT will add our own scripts to the PSBT if those inputs are ours.
If an input also lists pubkeys that we happen to know the private keys
for, we will sign those inputs too.

Change GetMetadata to use unique_ptr<CKeyMetadata>

Implement GetMetadata in DescriptorScriptPubKeyMan

Be able to create new wallets with DescriptorScriptPubKeyMans as backing

Generate new descriptors when encrypting

Add IsLegacy to CWallet so that the GUI knows whether to show watchonly

add importdescriptors RPC and tests for native descriptor wallets

Co-authored-by: Andrew Chow <achow101-github@achow101.com>

Functional tests for descriptor wallets

Change wallet_encryption.py to use signmessage instead of dumpprivkey

Return error when no ScriptPubKeyMan is available for specified type

When a CWallet doesn't have a ScriptPubKeyMan for the requested type
in GetNewDestination, give a meaningful error. Also handle this in
Qt which did not do anything with errors.

Implement CWallet::IsSpentKey for non-LegacySPKMans

tests: Add RPCOverloadWrapper which overloads some disabled RPCs

RPCOverloadWrapper overloads some deprecated or disabled RPCs with
an implementation using other RPCs to avoid having a ton of code churn
around replacing those RPCs.

Add a --descriptors option to various tests

Adds a --descriptors option globally to the test framework. This will
make the test create and use descriptor wallets. However some tests may
not work with this.

Some tests are modified to work with --descriptors and run with that
option in test_runer:
* wallet_basic.py
* wallet_encryption.py
* wallet_keypool.py <---- wallet_keypool_hd.py actually
* wallet_keypool_topup.py
* wallet_labels.py
* wallet_avoidreuse.py
2024-03-07 01:23:15 +07:00
fanquake
8368bd795e
Merge #19816: test: Rename wait until helper to wait_until_helper
fa1cd9e1ddc6918c3d600d36eadea71eebb242b6 test: Remove unused lock arg from BitcoinTestFramework.wait_until (MarcoFalke)
fad2794e93b4f5976e81793a4a63aa03a2c8c686 test: Rename wait until helper to wait_until_helper (MarcoFalke)
facb41bf1d1b7ee552c627f9829b4494b817ce28 test: Remove unused p2p_lock in VersionBitsWarningTest (MarcoFalke)

Pull request description:

  This avoids confusion with the `wait_until` member functions, which should be preferred because they take the appropriate locks and scale the timeout appropriately on their own.

ACKs for top commit:
  laanwj:
    Code review ACK fa1cd9e1ddc6918c3d600d36eadea71eebb242b6
  hebasto:
    ACK fa1cd9e1ddc6918c3d600d36eadea71eebb242b6, I have reviewed the code and it looks OK, I agree it can be merged.

Tree-SHA512: 319d400085606a4c738e314824037f72998e6657d8622b363726842aba968744f23c56d27275dfe506b8cbbb6e97fc39ca1d325db05d4d67df0e8b35f2244d5c
2024-03-06 02:00:39 +07:00
Wladimir J. van der Laan
a89bd562fd
Merge #19657: test: Wait until is_connected in add_p2p_connection
fa4dfd215f62e88d668311701735c332c264fa2a test: Wait until is_connected in add_p2p_connection (MarcoFalke)

Pull request description:

  Moving the wait_until from the individual test scripts to the test framework simplifies two tests

ACKs for top commit:
  jnewbery:
    Code review ACK fa4dfd215f62e88d668311701735c332c264fa2a
  theStack:
    ACK fa4dfd215f 

Tree-SHA512: 36eda7eb323614a4c4f9215f1d7b40b9f9c4036d1c08eb701ea705f3e2986fdabd2fc558965a6aadabeed861034aeaeef3c00f968ca17ed7a27e42e506cda87d
2024-03-06 02:00:38 +07:00
MarcoFalke
40420195be
Merge #19804: test/refactor: reference p2p objects explicitly and remove confusing Test_Node.p2p property
10d61505fe77880d6989115defa5e08417f3de2d [test] remove confusing p2p property (gzhao408)
549d30faf04612d9589c81edf9770c99e3221885 scripted-diff: replace p2p with p2ps[0] in p2p_invalid_tx (gzhao408)
7a0de46aeafb351cffa3410e1aae9809fd4698ad [doc] sample code for test framework p2p objects (gzhao408)
784f757994c1306bb6584b14c0c78617d6248432 [refactor] clarify tests by referencing p2p objects directly (gzhao408)

Pull request description:

  The `TestNode` has a `p2p` property which is an alias for `p2ps[0]`.

  I think this should be removed because it can be confusing and misleading (to both the test writer and reviewer), especially if a TestNode has multiple p2ps connected (which is the case for many tests).
  Another example is when a test has multiple subtests that connect 1 p2p and use the `p2p` property to reference it. If the subtests don't completely clean up after themselves, the subtests may affect one another.

  The best way to refer to a connected p2p is use the object returned by `add_p2p_connection` like this:
  ```py
  p2p_conn = node.add_p2p_connection(P2PInterface())
  ```
  A good example is [p2p_invalid_locator.py](https://github.com/bitcoin/bitcoin/blob/master/test/functional/p2p_invalid_locator.py), which cleans up after itself (waits in both `wait_for_disconnect` and in `disconnect_p2ps`) but wouldn't need so much complexity if it just referenced the connections directly.

  If there is only one connected, it's not really that tedious to just use `node.p2ps[0]` instead of `node.p2p` (and it can always be aliased inside the test itself).

ACKs for top commit:
  robot-dreams:
    utACK 10d61505fe77880d6989115defa5e08417f3de2d
  jnewbery:
    utACK 10d61505fe77880d6989115defa5e08417f3de2d
  guggero:
    Concept ACK 10d61505.

Tree-SHA512: 5965548929794ec660dae03467640cb2156d7d826cefd26d3a126472cbc2494b855c1d26bbb7b412281fbdc92b9798b9765a85c27bc1a97f7798f27f64db6f13
2024-01-27 22:55:29 -06:00
Konstantin Akimov
f34889dcf4
Merge #19760: test: Remove confusing mininode terminology
d5800da5199527a366024bc80cad7fcca17d5c4a [test] Remove final references to mininode (John Newbery)
5e8df3312e47a73e747ee892face55ed9ababeea test: resort imports (John Newbery)
85165d4332b0f72d30e0c584b476249b542338e6 scripted-diff: Rename mininode to p2p (John Newbery)
9e2897d020b114a10c860f90c5405be029afddba scripted-diff: Rename mininode_lock to p2p_lock (John Newbery)

Pull request description:

  New contributors are often confused by the terminology in the test framework, and what the difference between a _node_ and a _peer_ is. To summarize:

  - a 'node' is a bitcoind instance. This is the thing whose behavior is being tested. Each bitcoind node is managed by a python `TestNode` object which is used to start/stop the node, manage the node's data directory, read state about the node (eg process status, log file), and interact with the node over different interfaces.
  - one of the interfaces that we can use to interact with the node is the p2p interface. Each connection to a node using this interface is managed by a python `P2PInterface` or derived object (which is owned by the `TestNode` object). We can open zero, one or many p2p connections to each bitcoind node. The node sees these connections as 'peers'.

  For historic reasons, the word 'mininode' has been used to refer to those p2p interface objects that we use to connect to the bitcoind node (the code was originally taken from the 'mini-node' branch of https://github.com/jgarzik/pynode/tree/mini-node). However that name has proved to be confusing for new contributors, so rename the remaining references.

ACKs for top commit:
  amitiuttarwar:
    ACK d5800da519
  MarcoFalke:
    ACK d5800da5199527a366024bc80cad7fcca17d5c4a 🚞
Tree-SHA512: 2c46c2ac3c4278b6e3c647cfd8108428a41e80788fc4f0e386e5b0c47675bc687d94779496c09a3e5ea1319617295be10c422adeeff2d2bd68378e00e0eeb5de
2024-01-20 00:07:10 +07:00
MarcoFalke
0845e1956e Merge bitcoin/bitcoin#22139: test: add type annotations to util.get_rpc_proxy
fbeb8c43bc5bce131e15eb9e162ea457bfe2b83e test: add type annotations to util.get_rpc_proxy (fanquake)

Pull request description:

  Split out from #22092 while we address the functional test failure.

ACKs for top commit:
  instagibbs:
    ACK fbeb8c43bc

Tree-SHA512: 031ef8703202ae5271787719fc3fea8693574b2eb937ccf852875de95798d7fa3c39a8db7c91993d0c946b45d9b4d6de570bd1102e0344348784723bd84803a8
2023-12-03 20:13:09 -06:00
Konstantin Akimov
fab1031a70 fix: missing changes from Merge #18873: Fix intermittent sync_blocks failures 2023-09-19 08:54:12 -05:00
MarcoFalke
207b1c5877 Merge #12134: Build previous releases and run functional tests
c456145b2c65f580683df03bf10cd39000cf24d5 [test] add 0.19 backwards compatibility tests (Sjors Provoost)
b769cd142deda74fe46e231cc7b687a86514f2f1 [test] add v0.17.1 wallet upgrade test (Sjors Provoost)
9d9390dab716f07057c94e8e21f3c7dd06192f35 [tests] add wallet backwards compatility tests (Sjors Provoost)
c7ca6308968b29a0e0edc485cd06e68e5edb7c7d [scripts] support release candidates of earlier releases (Sjors Provoost)
8b1460dbd1b732f06d4cebe1fa6844286c7a0056 [tests] check v0.17.1 and v0.18.1 backwards compatibility (Sjors Provoost)
ae379cf7d12943fc192d58176673bcfe7d53da53 [scripts] build earlier releases (Sjors Provoost)

Pull request description:

  This PR adds binaries for 0.17, 0.18 and 0.19 to Travis and runs a basic block propagation test.

  Includes test for upgrading v0.17.1 wallets and opening master wallets with older versions.

  Usage:

  ```sh
  contrib/devtools/previous_release.sh -f -b v0.19.0.1 v0.18.1 v0.17.1
  test/functional/backwards_compatibility.py
  ```

  Travis caches these earlier releases, so it should be able to run these tests with little performance impact.

  Additional scenarios where it might be useful to run tests against earlier releases:

  * creating a wallet with #11403's segwit implementation, copying it to an older node and making sure the user didn't lose any funds (although this PR doesn't support `v0.15.1`)
  * future consensus changes
  * P2P changes (e.g. to make sure we don't accidentally ban old nodes)

ACKs for top commit:
  MarcoFalke:
    ACK c456145b2c65f580683df03bf10cd39000cf24d5 🔨

Tree-SHA512: 360bd870603f95b14dc0cd629532cc147344f632b808617c18e1b585dfb1f082b401e5d493a48196b719e0aeaee533ae0a773dfc9f217f704aae898576c19232
2023-09-19 08:54:12 -05:00
Konstantin Akimov
4aa197dbdb Merge #18673: scripted-diff: Sort test includes
fa4632c41714dfaa699bacc6a947d72668a4deef test: Move boost/stdlib includes last (MarcoFalke)
fa488f131fd4f5bab0d01376c5a5013306f1abcd scripted-diff: Bump copyright headers (MarcoFalke)
fac5c373006a9e4bcbb56843bb85f1aca4d87599 scripted-diff: Sort test includes (MarcoFalke)

Pull request description:

  When writing tests, often includes need to be added or removed. Currently the list of includes is not sorted, so developers that write tests and have `clang-format` installed will either have an unrelated change (sorting) included in their commit or they will have to manually undo the sort.

  This pull preempts both issues by just sorting all includes in one commit.

  Please be aware that this is **NOT** a change to policy to enforce clang-format or any other developer guideline or process. Developers are free to use whatever tool they want, see also #18651.

  Edit: Also includes a commit to bump the copyright headers, so that the touched files don't need to be touched again for that.

ACKs for top commit:
  practicalswift:
    ACK fa4632c41714dfaa699bacc6a947d72668a4deef
  jonatack:
    ACK fa4632c41714dfaa, light review and sanity checks with gcc build and clang fuzz build

Tree-SHA512: 130a8d073a379ba556b1e64104d37c46b671425c0aef0ed725fd60156a95e8dc83fb6f0b5330b2f8152cf5daaf3983b4aca5e75812598f2626c39fd12b88b180
2023-08-29 22:00:59 -05:00
Wladimir J. van der Laan
a365ed03c0 Merge #20683: test: Fix restart node race
fab46b34f4b13abbb0af276c3fb548f25ccc28bd test: Fix restart node race (MarcoFalke)

Pull request description:

  It is not allowed to start a node before it has been fully stopped. Otherwise it could lead to intermittent issues due to access issues (e.g. cookie file https://cirrus-ci.com/task/6409665024098304?command=ci#L4793)

  Fix that by waiting for the node to fully stop.

ACKs for top commit:
  laanwj:
    code review ACK fab46b34f4b13abbb0af276c3fb548f25ccc28bd

Tree-SHA512: 7605cac0573a7b04f05ff110d0131e8940d87f7baf6d698505ed16b363d4d15b1e552c5ffd1a187c8fe5639f7e265c3122734c85283275746e46bd789614fd21
2023-08-08 06:33:29 -05:00
MarcoFalke
9a3abd973c Merge #20613: test: Use Popen.wait instead of RPC in assert_start_raises_init_error
fa918dd537fea775c19a590e5f9161bf51a5839b test: Use Popen.wait instead of RPC in assert_start_raises_init_error (MarcoFalke)

Pull request description:

  Using RPC (`wait_for_rpc_connection`) has several issue:

  * It polls in a loop, which might be slow
  * It tries to read the RPC cookie file, which might not be present, thus leading to intermittent issues

  Fix both by using `Popen.wait`

ACKs for top commit:
  laanwj:
    Code review ACK ~~faf7b05be9c86ee61c39e5314511fe2410128a6b~~ fa918dd537fea775c19a590e5f9161bf51a5839b
  darosior:
    ACK fa918dd537fea775c19a590e5f9161bf51a5839b

Tree-SHA512: 5368ad0d0ea2deb0af9582a42667c9290efe8f2705f37a236afc2c7908b04265ab342e2dd356a57156e99389f4a27ab6da9fa7bf9161fb7568240aa005e693b9
2023-08-08 06:26:09 -05:00
MarcoFalke
5460866184 Merge #18561: test: Properly raise FailedToStartError when rpc shutdown before warmup finished
faede1b293354560317b67f0b4e6874dcac6ef41 test: Properly raise FailedToStartError when rpc shutdown before warmup finished (MarcoFalke)

Pull request description:

  Should fix issues such as https://travis-ci.org/github/bitcoin/bitcoin/jobs/671910152#L7034

Top commit has no ACKs.

Tree-SHA512: ac659f29c5ec91985c916b734e24911cbf4e2c5c4b1f1891a7e6c2d2511ec285167550fb03848eee4a7a3cbc9f8cdb0c766f4e881d9e44368c7415d007006368
2023-08-03 11:16:41 -05:00
Kittywhiskers Van Gogh
d40f28edb4 merge bitcoin#19762: Allow named and positional arguments to be used together 2023-07-28 00:18:27 -05:00
MarcoFalke
e9311d8098 Merge #18712: test: display command line options passed to send_cli() in debug log
8f5dc8800aeb524eee2fa2451cd22883b7b2bfec test: display command line options passed to send_cli() in debug log (Jon Atack)

Pull request description:

  as per https://github.com/bitcoin/bitcoin/pull/18691#discussion_r411382589, and revert two cli calls changed in #18691 from rpc commands back to command line options (these were the only occurrences).

ACKs for top commit:
  MarcoFalke:
    ACK 8f5dc8800aeb524eee2fa2451cd22883b7b2bfec

Tree-SHA512: fcb3eca00aa4099066028c90d5e50a02e074366e09a17f5f5b937d9f7562dd054ff65681aa0ad4c94f6de1e98b1e2b9ac4cd084ddc297010253989a80483b1b9
2023-07-24 11:42:34 -05:00
UdjinM6
3d77c539d2
test: Various test improvements (#5382)
## Issue being fixed or feature implemented
Speed thing up: 8075fc0c61
Unify things: ff1a390224 (and _probably_
fix issues like https://gitlab.com/dashpay/dash/-/jobs/4304343867),
876f5c3a9f,
ed58cdda13
Let tsan tests finish on smaller/slower machines:
ba1e3360f9

## What was done?
pls see individual commits


## How Has This Been Tested?
run tests locally and my in gitlab ci
https://gitlab.com/UdjinM6/dash/-/jobs/4319419014

## Breaking Changes
n/a

## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
2023-05-24 12:38:33 -05:00
Kittywhiskers Van Gogh
0d8932c979 merge bitcoin#18691: add wait_for_cookie_credentials() to framework for rpcwait tests 2023-04-17 08:30:49 +00:00
MarcoFalke
8746f52a51 Merge #20522: [test] Fix sync issue in disconnect_p2ps
3ebde2143aa98af213872b98b474d904e55056f7 [test] Fix wait condition in disconnect_p2ps (Amiti Uttarwar)

Pull request description:

  #19315 currently has a [test failure](https://cirrus-ci.com/task/4545582645641216) because of a race. `disconnect_p2ps` is intended to have a `wait_until` clause that prevents this race, but the conditional doesn't match since its comparing two different object types. `MY_SUBVERSION` is defined in messages.py as a byte string, but is compared to the value returned by the RPC. This PR simply converts types to ensure they match, which should prevent the race from occurring.

  HUGE PROPS TO jnewbery for discovering the issue 🔎

ACKs for top commit:
  jnewbery:
    ACK 3ebde2143aa98af213872b98b474d904e55056f7
  glozow:
    Code review ACK 3ebde2143a

Tree-SHA512: ca096b80a3e4d757a645f38846d6dc89d6a3d35c3435513a72d278e305faddd4aff9e75a767941b51b2abbf59c82679bac1e9a0140d6f285efe3053e51bcc2a8
2023-04-09 00:06:56 -05:00
MarcoFalke
8f0c22f93b Merge #19252: test: wait for disconnect in disconnect_p2ps + bloomfilter test followups
9a40cfc558b3f7fa4fff1270f969582af17479a5 [refactor] use waiting inside disconnect_p2ps (gzhao408)
aeb9fb414e2d000830287d9dd3fed7fc2eb570d2 [test] wait for disconnect_p2ps to be reflected in getpeerinfo (gzhao408)
e81942d2e1288367e8da94adb2b2a88be99e4751 [test] logging and style followups for bloomfilter tests (gzhao408)

Pull request description:

  Followup to #19083 which adds bloomfilter-related tests.

  1. Make test_node `disconnect_p2ps` wait until disconnection is complete to avoid race conditions (and not place the burden on tests) from MarcoFalke's [comment](https://github.com/bitcoin/bitcoin/pull/19083#discussion_r437383989). And clean up any redundant `wait_until`s in the functional tests.
  2. Clean up style + logging in p2p_filter.py and p2p_nobloomfilter_messages.py and jonatack's other [comments](https://github.com/bitcoin/bitcoin/pull/19083#pullrequestreview-428955784)

ACKs for top commit:
  jonatack:
    Code review ACK 9a40cfc from re-reviewing the diff and `git range-diff 5cafb46 8386ad5 9a40cfc`
  MarcoFalke:
    ACK 9a40cfc558b3f7fa4fff1270f969582af17479a5 🐂

Tree-SHA512: 2e14b1c12fc08a355bd5ccad7a2a734a4ccda4bc7dc7bac171cb57359819fc1599d764290729af74832fac3e2be258c5d406c701e78ab6d7262835859b9a7d87
2023-02-27 23:12:41 -06:00
Wladimir J. van der Laan
ae38bd6a52
Merge #19023: test: Fix intermittent ETIMEDOUT on FreeBSD
fab908f18a080912a0e833f12cccc27f27662e3b test: Fix intermittent ETIMEDOUT on FreeBSD (MarcoFalke)

Pull request description:

  Example backtrace: https://cirrus-ci.com/task/5307019047469056?command=functional_test#L1059

ACKs for top commit:
  laanwj:
    ACK fab908f18a080912a0e833f12cccc27f27662e3b

Tree-SHA512: 06e383e9993e083d6da4c546dde8811ace02559ddbb1c24e307938307763b3abe630699054e7067cf4aea993c82865e259b77b4bee518666a03d26e0f81c0656
2023-01-23 12:22:30 -06:00
MarcoFalke
db22cf0a7d Merge #18692: test: Bump timeout in wallet_import_rescan
fabfcad8764bb8f807b0ac5f3482b414278a4525 test: Bump timeout in wallet_import_rescan (MarcoFalke)

Pull request description:

  Avoid timeouts when starting the node, also make error message more verbose

ACKs for top commit:
  practicalswift:
    ACK fabfcad8764bb8f807b0ac5f3482b414278a4525 -- patch looks correct

Tree-SHA512: 8fd60a05380349f521d0e814d2f268702dfbe57c7567a4f6e94435498dfdd32909179d75fded44757ecb1a93a4045842bc6d00bfd6cd18ba751513461359c7b0
2023-01-11 21:42:32 -06:00
MarcoFalke
2e28e6eee3
Merge #18633: test: Properly raise FailedToStartError when rpc shutdown before warmup finished (take 2)
fa03713e133e3017112fdd5c278e0c8643054578 test: Properly raise FailedToStartError when rpc shutdown before warmup finished (take 2) (MarcoFalke)

Pull request description:

  actually (?) fix #18561

  See most recent traceback https://travis-ci.org/github/bitcoin/bitcoin/jobs/674668692#L7062

  I believe the reason the error is still there is that ConnectionResetError is derived from OSError:

  ConnectionResetError(ConnectionError(OSError))

  And IOError is an alias for OSError since python 3.3, see https://docs.python.org/3/library/exceptions.html#IOError

  So fix that by renaming IOError to the alias OSError and move the less specific catch clause down a few lines.

ACKs for top commit:
  jonatack:
    ACK fa03713e133e3017112fdd5c278e0c8643054578

Tree-SHA512: 6e5b214ed9101bf8ebe7472dcc1f9e9d128e2575c93ec00c8d0774ae1a9b52a8c2a653a45a0eab8d881570b08dd5ffeddf5aca88a10438c366e1f633253cb0b5
2022-09-16 19:22:12 +05:30
MarcoFalke
c0e3829631
Merge #18986: tests: Add capability to disable RPC timeout in functional tests
38c3dd9c706e7e84b2a4dbaf1424a3f1c3b694fc docs: Add notes on how to diasble rpc timeout in functional tests while attatching gdb. (codeShark149)
784ae096259955ea7a294114f5c021c9489d2717 test: Add capability to disable RPC timeout in functional tests. (codeShark149)

Pull request description:

  Many times, especially while debugging RPC callbacks to core using gdb, the test timeout kicks in before the response can get back. This can be annoying and requires restarting the functional test as well as gdb attachment.

  This PR adds a `--notimeout` flag into `test_framework` and sets the `rpc_timeout` accordingly if the flag is set.

  The same effect can be achieved with newly added `--factor` flag but keeping a separate flag that explicitly disables the timeout can be easier for new testers to find it out and separates its purpose from the `--factor` flag.

  Requesting review ryanofsky jnewbery as per the IRC discussion.

  Update: After initial round of review, the approach is modified to accommodate the functionality in already existing `--factor` flag. `--factor` is changed to `--timeout-factor` to express its intent better.

ACKs for top commit:
  MarcoFalke:
    ACK 38c3dd9c706e7e84b2a4dbaf1424a3f1c3b694fc and thanks for fixing up all my typos 😅
  jnewbery:
    ACK 38c3dd9c706e7e84b2a4dbaf1424a3f1c3b694fc.

Tree-SHA512: 9458dd1010288c62f8bb83f7a4893284fbbf938882dd65fc9e08810a910db07ef676e3100266028e5d4c8ce407b2267b3860595015da070c84a9d4a9816797db
2022-09-08 00:40:11 +03:00
MarcoFalke
60cfe097be
Merge #18247: test: Wait for both veracks in add_p2p_connection
faf1d047313e71658fb31f6b94fdd5d37705ab85 test: Remove redundant sync_with_ping after add_p2p_connection (MarcoFalke)
fa9064704524a0fd1fa9ea73eea45b07316ac3d1 test: Wait for both veracks in add_p2p_connection (MarcoFalke)

Pull request description:

  This fixes the race in p2p_blocksonly

  E.g. https://travis-ci.org/MarcoFalke/bitcoin-core/jobs/657038844#L4500

  ```
   ...
   test  2020-03-01T20:58:28.825000Z TestFramework.mininode (DEBUG): Closed connection to: 127.0.0.1:11828
   node0 2020-03-01T20:58:28.825642Z [net] disconnecting peer=0
   node0 2020-03-01T20:58:28.825826Z [net] Cleared nodestate for peer=0
   node0 2020-03-01T20:58:28.875835Z [http] Received a POST request for / from 127.0.0.1:53448
   node0 2020-03-01T20:58:28.876067Z [httpworker.0] ThreadRPCServer method=getmempoolinfo user=__cookie__
   test  2020-03-01T20:58:28.877000Z TestFramework.mininode (DEBUG): Connecting to Bitcoin Node: 127.0.0.1:11828
   test  2020-03-01T20:58:28.878000Z TestFramework.mininode (DEBUG): Connected & Listening: 127.0.0.1:11828
   test  2020-03-01T20:58:28.878000Z TestFramework.mininode (DEBUG): Send message to 127.0.0.1:11828: msg_version(nVersion=70014 nServices=9 nTime=Sun Mar  1 20:58:28 2020 addrTo=CAddress(nServices=1 ip=127.0.0.1 port=11828) addrFrom=CAddress(nServices=1 ip=0.0.0.0 port=0) nNonce=0x164D5DEB952A4A0B strSubVer=b'/python-mininode-tester:0.0.3/' nStartingHeight=-1 nRelay=1)
   node0 2020-03-01T20:58:28.883808Z [net] Added connection peer=1
   node0 2020-03-01T20:58:28.883950Z [net] connection from 127.0.0.1:33798 accepted
   node0 2020-03-01T20:58:28.884300Z [msghand] received: version (116 bytes) peer=1
   node0 2020-03-01T20:58:28.884483Z [msghand] sending version (114 bytes) peer=1
   node0 2020-03-01T20:58:28.884700Z [msghand] send version message: version 70015, blocks=200, us=[::]:0, peer=1
   node0 2020-03-01T20:58:28.884765Z [msghand] sending verack (0 bytes) peer=1
   test  2020-03-01T20:58:28.885000Z TestFramework.mininode (DEBUG): Received message from 127.0.0.1:11828: msg_version(nVersion=70015 nServices=1033 nTime=Sun Mar  1 20:58:28 2020 addrTo=CAddress(nServices=0 ip=0.0.0.0 port=0) addrFrom=CAddress(nServices=1033 ip=0.0.0.0 port=0) nNonce=0x4A0F2F4C549B3399 strSubVer=b'/Satoshi:0.19.99(testnode0)/' nStartingHeight=200 nRelay=0)
   test  2020-03-01T20:58:28.885000Z TestFramework.mininode (DEBUG): Send message to 127.0.0.1:11828: msg_verack()
   test  2020-03-01T20:58:28.885000Z TestFramework.mininode (DEBUG): Received message from 127.0.0.1:11828: msg_verack()
   node0 2020-03-01T20:58:28.885004Z [msghand] receive version message: /python-mininode-tester:0.0.3/: version 70014, blocks=-1, us=127.0.0.1:11828, peer=1
   test  2020-03-01T20:58:28.886000Z TestFramework (INFO): Check that txs from rpc are not rejected and relayed to other peers
   node0 2020-03-01T20:58:28.886556Z [http] Received a POST request for / from 127.0.0.1:53448
   node0 2020-03-01T20:58:28.886783Z [httpworker.1] ThreadRPCServer method=getpeerinfo user=__cookie__
   node0 2020-03-01T20:58:28.889032Z [http] Received a POST request for / from 127.0.0.1:53448
   node0 2020-03-01T20:58:28.889294Z [httpworker.2] ThreadRPCServer method=testmempoolaccept user=__cookie__
   node0 2020-03-01T20:58:28.891655Z [http] Received a POST request for / from 127.0.0.1:53448
   node0 2020-03-01T20:58:28.891963Z [httpworker.3] ThreadRPCServer method=sendrawtransaction user=__cookie__
   node0 2020-03-01T20:58:28.893115Z [httpworker.3] Enqueuing TransactionAddedToMempool: txid=af34fc5ff9ea8babbd4083fbb79ffd2ad5aff1d6def803c07ca5aeed880bd60f wtxid=af34fc5ff9ea8babbd4083fbb79ffd2ad5aff1d6def803c07ca5aeed880bd60f
   node0 2020-03-01T20:58:28.893443Z [scheduler] TransactionAddedToMempool: txid=af34fc5ff9ea8babbd4083fbb79ffd2ad5aff1d6def803c07ca5aeed880bd60f wtxid=af34fc5ff9ea8babbd4083fbb79ffd2ad5aff1d6def803c07ca5aeed880bd60f
   node0 2020-03-01T20:58:28.894814Z [msghand] received: verack (0 bytes) peer=1
   node0 2020-03-01T20:58:28.894937Z [msghand] sending sendheaders (0 bytes) peer=1
   node0 2020-03-01T20:58:28.895087Z [msghand] sending sendcmpct (9 bytes) peer=1
   node0 2020-03-01T20:58:28.895235Z [msghand] sending sendcmpct (9 bytes) peer=1
   node0 2020-03-01T20:58:28.895430Z [msghand] sending ping (8 bytes) peer=1
   node0 2020-03-01T20:58:28.895896Z [msghand] initial getheaders (199) to peer=1 (startheight:-1)
   test  2020-03-01T20:58:28.896000Z TestFramework.mininode (DEBUG): Received message from 127.0.0.1:11828: msg_sendheaders()
   node0 2020-03-01T20:58:28.896016Z [msghand] sending getheaders (645 bytes) peer=1
   node0 2020-03-01T20:58:28.896607Z [msghand] sending feefilter (8 bytes) peer=1
   test  2020-03-01T20:58:28.897000Z TestFramework.mininode (DEBUG): Received message from 127.0.0.1:11828: msg_sendcmpct(announce=False, version=2)
   test  2020-03-01T20:58:28.897000Z TestFramework.mininode (DEBUG): Received message from 127.0.0.1:11828: msg_sendcmpct(announce=False, version=1)
   test  2020-03-01T20:58:28.897000Z TestFramework.mininode (DEBUG): Received message from 127.0.0.1:11828: msg_ping(nonce=f735096062d217b5)
   test  2020-03-01T20:58:28.897000Z TestFramework.mininode (DEBUG): Send message to 127.0.0.1:11828: msg_pong(nonce=f735096062d217b5)
   test  2020-03-01T20:58:28.897000Z TestFramework.mininode (DEBUG): Received message from 127.0.0.1:11828: msg_getheaders(locator=CBlockLocator(nVersion=70014 vHave=[48924041037103782797700918670732352379567180837453042168545380831411841797392, 28010422273815860773972769588722664110955084223364219183119416607410792753789, 5954376895683677137597080246740451260829355661937599865380797589540815086241, 14500403275336359851183244421245184901482464358719551678581030092830439955257, 17853919108052771837249729512111680264864054213441538187113939176285784834878, 28843166929059356839755035875664073555480989477... (msg truncated)
   test  2020-03-01T20:58:28.897000Z TestFramework.mininode (DEBUG): Received message from 127.0.0.1:11828: msg_feefilter(feerate=000003e8)
   node0 2020-03-01T20:58:28.898144Z [msghand] received: pong (8 bytes) peer=1
   node0 2020-03-01T20:59:28.338539Z [scheduler] Feeding 13446 bytes of dynamic environment data into RNG
   test  2020-03-01T20:59:28.908000Z TestFramework.utils (ERROR): wait_until() failed. Predicate: ''''
                                             def test_function():
                                                 assert self.is_connected
                                                 if not self.last_message.get('tx'):
                                                     return False
                                                 return self.last_message['tx'].tx.rehash() == txid
                                     '''
   test  2020-03-01T20:59:28.908000Z TestFramework (ERROR): Assertion failed
                                     Traceback (most recent call last):
                                       File "/home/travis/build/MarcoFalke/bitcoin-core/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 112, in main
                                         self.run_test()
                                       File "/home/travis/build/MarcoFalke/bitcoin-core/build/bitcoin-x86_64-pc-linux-gnu/test/functional/p2p_blocksonly.py", line 57, in run_test
                                         self.nodes[0].p2p.wait_for_tx(txid)
                                       File "/home/travis/build/MarcoFalke/bitcoin-core/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/mininode.py", line 369, in wait_for_tx
                                         wait_until(test_function, timeout=timeout, lock=mininode_lock)
                                       File "/home/travis/build/MarcoFalke/bitcoin-core/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 234, in wait_until
                                         raise AssertionError("Predicate {} not true after {} seconds".format(predicate_source, timeout))
                                     AssertionError: Predicate ''''
                                             def test_function():
                                                 assert self.is_connected
                                                 if not self.last_message.get('tx'):
                                                     return False
                                                 return self.last_message['tx'].tx.rehash() == txid
                                     ''' not true after 60 seconds

ACKs for top commit:
  jonatack:
    ACK faf1d04

Tree-SHA512: 3b1a38a5c87d11c610eee0988f0c4af9bfcd978df9ac718ef611f663df2fd4a0eb04e077df5e940d15971bb2f22328fb6021cacccb6902f1e527f288ad2c4a2c
2022-09-08 00:02:35 +03:00
MarcoFalke
6bb3e54578
Merge #18617: test: add factor option to adjust test timeouts
2742c3428633b6ceaab6714635dc3adb74bf121b test: add factor option to adjust test timeouts (Harris)

Pull request description:

  This PR adds a new option **factor** that can be used to adjust timeouts in various functional tests.
  Several timeouts and functions from `authproxy`, `mininode`, `test_node` and `util` have been adapted to use this option. The factor-option definition is located in `test_framework.py`.

  Fixes https://github.com/bitcoin/bitcoin/issues/18266
  Also Fixes https://github.com/bitcoin/bitcoin/issues/18834

ACKs for top commit:
  MarcoFalke:
    Thanks! ACK 2742c3428633b6ceaab6714635dc3adb74bf121b

Tree-SHA512: 6d8421933ba2ac1b7db00b70cf2bc242d9842c48121c11aadc30b0985c4a174c86a127d6402d0cd73b993559d60d4f747872d21f9510cf4806e008349780d3ef
2022-09-08 00:02:35 +03:00
Wladimir J. van der Laan
86a11a83a2
Merge #18873: test: Fix intermittent sync_blocks failures
fa3f9a05660687bf4146e089050e944a1d6cbe3c test: Fix intermittent sync_blocks failures (MarcoFalke)

Pull request description:

  Fixes #18872
  Fixes #18737
  Fixes #18801

  See docstring for motivation and description

ACKs for top commit:
  laanwj:
    Code review ACK fa3f9a05660687bf4146e089050e944a1d6cbe3c

Tree-SHA512: acd52d386a6849f7ff1cb1a51a439dc3c76e0e3a4dd8d22030df0ebb09b44497c61c56331ff65724b695d82d86b0ebb24608f9e637008e5dacb7676b0c448889
2022-09-07 22:25:40 +03:00
MarcoFalke
9e41907692
Merge #16042: test: Bump MAX_NODES to 12
fa47330397 test: Speed up cache creation (MarcoFalke)
fa6ad7a5ec test: Bump MAX_NODES to 12 (MarcoFalke)

Pull request description:

  When testing a combination of settings that affect the datadir (e.g. prune, blockfilter, ...) we may need a lot of datadirs.
  Bump the maximum number of nodes proactively from 8 to 12, so that caches get populated with 12 node dirs, as opposed to 8.

  Also, add an assert that the list of deterministic keys is exactly the number of max nodes (and not more than that.

  Also, create the cache faster.

ACKs for commit fa4733:
  laanwj:
    utACK fa473303972b7dad600d949dc9b303d8136cb7e7

Tree-SHA512: 9803c765ed52d344102f5a3bce57b05d88a7429dcb05ed66ed6c881fda8d87c2834d02d21b95fe9f39c0efe3b8527e13cf94f006588cde22e8c2cd50b2d517a6
2022-09-07 20:15:52 +03:00
MarcoFalke
640616ff10 Merge #17469: test: Remove fragile assert_memory_usage_stable
fac942ca57dce6cfa5655a3ac8664d6a051bc01f test: Remove fragile assert_memory_usage_stable (MarcoFalke)

Pull request description:

  This test fails on arm64 and a fuzz tests seems inappropriate for the functional test suite anyway, so remove it.

  Example failures:

  * https://travis-ci.org/bitcoin/bitcoin/jobs/611497963#L14517
  * https://travis-ci.org/MarcoFalke/bitcoin-core/jobs/611029104#L3876

ACKs for top commit:
  jamesob:
    ACK fac942ca57

Tree-SHA512: 3577e7ce5891d221cb798454589ba796ed0c06621a26351bb919c23bc6bb46aafcd0b11cb02bbfde64b74d67cb2950da44959a7ecdc436491a34e8b045c1ccf4
2022-08-24 14:29:45 -04:00
PastaPastaPasta
b9efbdeab7 merge #16115: On bitcoind startup, write config args to debug.log
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2022-06-21 19:08:55 +05:30
fanquake
f08497c93e Merge #17705: test: re-enable CLI test support by using EncodeDecimal in json.dumps()
b6f9e3576a1ea18572e4803aeb3f39330f0cb759 test: re-enable CLI test support by using EncodeDecimal in json.dumps() (fanquake)

Pull request description:

  As mentioned in https://github.com/bitcoin/bitcoin/pull/17675#issuecomment-563188648.

ACKs for top commit:
  practicalswift:
    ACK b6f9e3576a1ea18572e4803aeb3f39330f0cb759 assuming Travis is happy too -- diff looks correct :)
  MarcoFalke:
    > ACK b6f9e35 assuming Travis is happy too -- diff looks correct :)

Tree-SHA512: 79fa535cc1756c8ee610a3d6a316a1c4f036797d6990a5620e44985393a2e52f78450f8e0021d0a148c08705fd1ba765508464a365f9030ae0d2cacbd7a93e19
2022-06-08 12:35:12 +07:00
MarcoFalke
0bbd8bfab6
Merge #14631: [tests] Move deterministic address import to setup_nodes
3fd7e76f6d [tests] Move deterministic address import to setup_nodes (John Newbery)

Pull request description:

  This requires a small changes to a few tests, but means that
  deterministic addresses will always be imported (unless setup_nodes
  behaviour is explicitly overridden).

  Tidies up the way we import deterministic addresses, requested in review comment here: https://github.com/bitcoin/bitcoin/pull/14468#discussion_r225594586.

Tree-SHA512: 2b32edf500e286c463398487ab1153116a1dc90f64a53614716373311abdc83d8a251fdd8f42d1146b56e308664deaf62952113f66e98bc37f23968096d1a961
2022-05-01 22:52:30 +03:00
MarcoFalke
e4f621fd57
merge #14468: [wallet] Deprecate generate RPC method
ab9aca2bdf [rpc] add 'getnewaddress' hint to 'generatetoaddress' help text. (John Newbery)
c9f02955b2 [wallet] Deprecate the generate RPC method (John Newbery)
aab81720de [tests] Add generate method to TestNode (John Newbery)
c269209336 [tests] Small fixups before deprecating generate (John Newbery)

Pull request description:

  Deprecates the `generate` RPC method.

  For concept discussion, see #14299.

  Fixes #14299.

Tree-SHA512: 16a3b8b742932e4f0476c06b23de07a34d9d215b41d9272c1c9d1e39966b0c2406f17c5ab3cc568947620c08171ebe5eb74fd7ed4b62151363e305ee2937cc80
2022-05-01 20:51:43 +03:00
MarcoFalke
77f106ebf5
Merge #17633: tests: Add option --valgrind to run the functional tests under Valgrind
5db506ba5943868cc2c845f717508739b7f05714 tests: Add option --valgrind to run nodes under valgrind in the functional tests (practicalswift)

Pull request description:

  What is better than fixing bugs? Fixing entire bug classes of course! :)

  Add option `--valgrind` to run the functional tests under Valgrind.

  Regular functional testing under Valgrind would have caught many of the uninitialized reads we've seen historically.

  Let's kill this bug class once and for all: let's never use an uninitialized value ever again. Or at least not one that would be triggered by running the functional tests! :)

  My hope is that this addition will make it super-easy to run the functional tests under Valgrind and thus increase the probability of people making use of it :)

  Hopefully `test/functional/test_runner.py --valgrind` will become a natural part of the pre-release QA process.

  **Usage:**

  ```
  $ test/functional/test_runner.py --help
  …
    --valgrind            run nodes under the valgrind memory error detector:
                          expect at least a ~10x slowdown, valgrind 3.14 or
                          later required
  ```

  **Live demo:**

  First, let's re-introduce a memory bug by reverting the recent P2P uninitialized read bug fix from PR #17624 ("net: Fix an uninitialized read in ProcessMessage(…, "tx", …) when receiving a transaction we already have").

  ```
  $ git diff
  diff --git a/src/consensus/validation.h b/src/consensus/validation.h
  index 3401eb64c..940adea33 100644
  --- a/src/consensus/validation.h
  +++ b/src/consensus/validation.h
  @@ -114,7 +114,7 @@ inline ValidationState::~ValidationState() {};

   class TxValidationState : public ValidationState {
   private:
  -    TxValidationResult m_result = TxValidationResult::TX_RESULT_UNSET;
  +    TxValidationResult m_result;
   public:
       bool Invalid(TxValidationResult result,
                    const std::string &reject_reason="",
  ```

  Second, let's test as normal without Valgrind:

  ```
  $ test/functional/p2p_segwit.py -l INFO
  2019-11-28T09:30:42.810000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test__fc8q3qo
  …
  2019-11-28T09:31:57.187000Z TestFramework (INFO): Subtest: test_non_standard_witness_blinding (Segwit active = True)
  …
  2019-11-28T09:32:08.265000Z TestFramework (INFO): Tests successful
  ```

  Third, let's test with `--valgrind` and see if the test fail (as we expect) when the unitialized value is used:

  ```
  $ test/functional/p2p_segwit.py -l INFO --valgrind
  2019-11-28T09:32:33.018000Z TestFramework (INFO): Initializing test directory /tmp/bitcoin_func_test_gtjecx2l
  …
  2019-11-28T09:40:36.702000Z TestFramework (INFO): Subtest: test_non_standard_witness_blinding (Segwit active = True)
  2019-11-28T09:40:37.813000Z TestFramework (ERROR): Assertion failed
  ConnectionRefusedError: [Errno 111] Connection refused
  ```

ACKs for top commit:
  MarcoFalke:
    ACK 5db506ba5943868cc2c845f717508739b7f05714
  jonatack:
    ACK 5db506ba5943868cc2c845f717508739b7f05714

Tree-SHA512: 2eaecacf4da166febad88b2a8ee6d7ac2bcd38d4c1892ca39516b6343e8f8c8814edf5eaf14c90f11a069a0389d24f0713076112ac284de987e72fc5f6cc3795
2022-04-03 08:35:46 +05:30
MarcoFalke
3834c10098 Merge #15415: [test] functional: allow custom cwd, use tmpdir as default
e3e1a5631e [test] functional: set cwd of nodes to tmpdir (Sjors Provoost)

Pull request description:

  Any process launched by bitcoind will have `self.datadir` as its `cwd`.

Tree-SHA512: 0b311643bb96c7dc2f693774620173243b3add40bf373284695af2f0071823b23485289fd2ffe152b7f63e0bfe989b16720077cfc2ce33905f9b8e7f2630f3c0
2022-02-27 13:33:26 -05:00
Wladimir J. van der Laan
7874e7852e
Merge #16804: test: Remove unused try-block in assert_debug_log
fae91a09c453a9a95c382df765bd71e54698d5b2 test: Remove incorrect and unused try-block in assert_debug_log (MarcoFalke)

Pull request description:

  This try block has accidentally been added by me in fa3e9f7627784ee00980590e5bf044a0e1249999.
  It was unused all the time, but commit 6011c9d72d1df5c2cd09de6f85c21eb4f7eb1ba8 added a `return` in the finally block, muting all exceptions.

  This can be tested by adding an `assert False` after any `with ...assert_debug_log...:` line.

ACKs for top commit:
  laanwj:
    ACK fae91a09c453a9a95c382df765bd71e54698d5b2
  ryanofsky:
    utACK fae91a09c453a9a95c382df765bd71e54698d5b2. I didn't know returning inside a `finally` block would cancel pending exceptions or return values, but I guess this makes sense and is a good thing to be aware of.

Tree-SHA512: 47ed0165062060e9af055a3e92f1a529cd41d00476bfad64e3cd141ae084d22f926a343bb1257717e164e15459a59ab66aed198c95d18bf780d8cb0b76aa3298
2021-11-25 06:38:16 +05:30
MarcoFalke
0841836441
Merge #16656: test: fix rpc_setban.py race
6011c9d72d1df5c2cd09de6f85c21eb4f7eb1ba8 QA: fix rpc_setban.py race (Jonas Schnelli)

Pull request description:

  The new `rpc_setban.py` test failes regularly on CIs due to a race between injecting the ban and testing the log "on the other side".

  The problem is, that the test immediately after the `addnode` command on node0 checks for the `dropped (banned)` entry on node1 (without giving some time).

  Adding a 2 seconds sleep seems to solve the race (I guess there is no better event-driven delay).

  Example of a failed test: https://bitcoinbuilds.org/index.php?ansilog=bf743910-103f-4b54-9a97-960c471061bd.log#l2906

Top commit has no ACKs.

Tree-SHA512: 680f8ea3e5ddb07e93f824f1aeff4a459e25e6c14715a39fc7670e50506d7cf25925348672c5c2d8ba3e1243ccf5effbc2456bcd094fb96868349f8d26e008f1
2021-11-25 06:38:15 +05:30
Kittywhiskers Van Gogh
f9d5542daa merge bitcoin#15927: log thread names by default in functional tests 2021-11-11 18:36:09 +05:30
Vijay Das Manikpuri
68de6f2973
Merge #15629: init: Throw error when network specific config is ignored 2021-10-20 05:05:10 +05:30
Wladimir J. van der Laan
f4cac953c3
Merge #14903: tests: Handle ImportError explicitly, improve comparisons against None
c9ba253f4f5d675d7736d24c1167229d0898ef1a Add E711 to flake8 check (Daniel Ingram)
17b55202dae8d6e21d2490de89b345c55f7694c0 Compare to None with is/is not (Daniel Ingram)
1b89074ae27ce123adbeed57343deaef13c14f81 Change '== None' to 'is None' (Daniel Ingram)
16d293772365d57cc1a279d5ad0fa6f44b12ed54 Handle exception as ImportError (Daniel Ingram)

Pull request description:

Tree-SHA512: aa5875ea3d9ac47ac898545ff023b511042cd377ea0c4594074daae119f3d4f3fc295721aad94a732a907086ecb32bce19a8eed38adf479f12663c4e8944f721
2021-10-08 19:13:52 +05:30
UdjinM6
b78ceed6ca
Merge pull request #4426 from Munkybooty/backports-0.18-pr17
Backports 0.18 pr17
2021-09-24 12:02:53 +03:00