Merge #6488: backport: merge bitcoin#27452, #29347, #29356, #29353, #29452, #29483, #30545, #31383 (BIP324 backports: part 5)

c6f23a718c merge bitcoin#31383: Add missing node.setmocktime(self.mocktime) to p2p_ibd_stalling.py (Kittywhiskers Van Gogh)
9072a10754 merge bitcoin#30545: fix intermittent failures in feature_proxy.py (Kittywhiskers Van Gogh)
7e2d435e35 merge bitcoin#29483: add --v1transport option, add --v2transport to a CI task (Kittywhiskers Van Gogh)
7e59a965f8 merge bitcoin#29452: document that BIP324 on by default (Kittywhiskers Van Gogh)
0f3b5e081e merge bitcoin#29353: adhere to typical VERSION message protocol flow (Kittywhiskers Van Gogh)
dfdddfd2ff rpc: enable `v2transport` for `masternode connect` when capable (Kittywhiskers Van Gogh)
3c1636174b merge bitcoin#29356: make v2transport arg in addconnection mandatory and few cleanups (Kittywhiskers Van Gogh)
c53cd93aee merge bitcoin#29347: enable v2transport by default (Kittywhiskers Van Gogh)
7dcf561306 merge bitcoin#27452: cover addrv2 anchors by adding TorV3 to CAddress in messages.py (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * When backporting [bitcoin#27452](https://github.com/bitcoin/bitcoin/pull/27452) in `feature_anchors.py`, `P2P_SERVICES` (`NODE_NETWORK | NODE_HEADERS_COMPRESSED`) has been replaced with `NODE_NETWORK` as the former evaluates to a value greater than `256` (specifically `2049`), which causes test failure. The replacement value is acceptable as `NODE_NETWORK` is the desired service flag expected by Dash Core ([source](779e4295ad/src/protocol.cpp (L249-L254))).

    <details>

    <summary>Test failure:</summary>

    ```
    dash@89afd55ae77e:/src/dash$ ./test/functional/feature_anchors.py
    2024-12-14T12:31:22.244000Z TestFramework (INFO): PRNG seed is: 8365703189892653614
    2024-12-14T12:31:22.244000Z TestFramework (INFO): Initializing test directory /tmp/dash_func_test_j0ya02yu
    2024-12-14T12:31:22.776000Z TestFramework (INFO): When node starts, check if anchors.dat doesn't exist
    2024-12-14T12:31:22.776000Z TestFramework (INFO): Add 2 block-relay-only connections to node
    2024-12-14T12:31:24.781000Z TestFramework (INFO): Add 5 inbound connections to node
    2024-12-14T12:31:29.843000Z TestFramework (INFO): Check node connections
    2024-12-14T12:31:30.848000Z TestFramework (INFO): Check the addresses in anchors.dat
    2024-12-14T12:31:30.848000Z TestFramework (INFO): Perturb anchors.dat to test it doesn't throw an error during initialization
    2024-12-14T12:31:31.356000Z TestFramework (INFO): When node starts, check if anchors.dat doesn't exist anymore
    2024-12-14T12:31:31.357000Z TestFramework (INFO): Ensure addrv2 support
    2024-12-14T12:31:32.364000Z TestFramework (INFO): Add 256-bit-address block-relay-only connections to node
    2024-12-14T12:31:33.368000Z TestFramework (INFO): Check for addrv2 addresses in anchors.dat
    2024-12-14T12:31:33.369000Z TestFramework (ERROR): Unexpected exception caught during testing
    Traceback (most recent call last):
      File "/src/dash/test/functional/test_framework/test_framework.py", line 161, in main
        self.run_test()
      File "/src/dash/./test/functional/feature_anchors.py", line 135, in run_test
        new_data[services_index] = P2P_SERVICES
    ValueError: byte must be in range(0, 256)
    2024-12-14T12:31:33.870000Z TestFramework (INFO): Stopping nodes
    2024-12-14T12:31:33.871000Z TestFramework (WARNING): Not cleaning up dir /tmp/dash_func_test_j0ya02yu
    2024-12-14T12:31:33.871000Z TestFramework (ERROR): Test failed. Test logging available at /tmp/dash_func_test_j0ya02yu/test_framework.log
    ```

    </details>

  ## Breaking Changes

  None expected.

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  UdjinM6:
    utACK c6f23a718c
  PastaPastaPasta:
    utACK c6f23a718c81b7c574ba3fecbe52f342eda35a46;

Tree-SHA512: ca134432d000d521827a20c75c03913421fe52a31fda1cbf632e0b207c31728406feb090469d592d8e2fd8d64298faa2752ff703de79f737a62c276c6a231097

ci/test/00_setup_env_native_multiprocess.sh

@@ -10,5 +10,6 @@ export CONTAINER_NAME=ci_native_multiprocess
 export PACKAGES="cmake python3 llvm clang"
 export DEP_OPTS="DEBUG=1 MULTIPROCESS=1"
 export GOAL="install"
+export TEST_RUNNER_EXTRA="--v2transport"
 export BITCOIN_CONFIG="--with-boost-process --enable-debug CC=clang CXX=clang++" # Use clang to avoid OOM
 export TEST_RUNNER_ENV="BITCOIND=dash-node"

doc/bips.md

@@ -41,5 +41,5 @@ Versions and PRs are relevant to Bitcoin's core if not mentioned other.
 * [`BIP 158`](https://github.com/bitcoin/bips/blob/master/bip-0158.mediawiki): Compact Block Filters for Light Clients can be indexed as of **Dash Core v18.0** ([PR dash#4314](https://github.com/dashpay/dash/pull/4314), [PR #14121](https://github.com/bitcoin/bitcoin/pull/14121)).
 * [`BIP 159`](https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki): The `NODE_NETWORK_LIMITED` service bit is signalled as of **v0.16.0** ([PR 11740](https://github.com/bitcoin/bitcoin/pull/11740)), and such nodes are connected to as of **v0.17.0** ([PR 10387](https://github.com/bitcoin/bitcoin/pull/10387)).
 * [`BIP 174`](https://github.com/bitcoin/bips/blob/master/bip-0174.mediawiki): RPCs to operate on Partially Signed Bitcoin Transactions (PSBT) are present as of **v18.0** ([PR 13557](https://github.com/bitcoin/bitcoin/pull/13557)).
-* [`BIP 324`](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki): The v2 transport protocol specified by BIP324 and the associated `NODE_P2P_V2` service bit are supported as of **v22.0**, but off by default ([PR 28331](https://github.com/bitcoin/bitcoin/pull/28331)).
+* [`BIP 324`](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki): The v2 transport protocol specified by BIP324 and the associated `NODE_P2P_V2` service bit are supported as of **v22.0**, but off by default ([PR 28331](https://github.com/bitcoin/bitcoin/pull/28331)). On by default as of **v22.1** ([PR 29347](https://github.com/bitcoin/bitcoin/pull/29347)).
 * [`BIP 339`](https://github.com/bitcoin/bips/blob/master/bip-0339.mediawiki): Relay of transactions by wtxid is supported as of **v0.21.0** ([PR 18044](https://github.com/bitcoin/bitcoin/pull/18044)).

src/net.h

@@ -117,7 +117,7 @@ static const bool DEFAULT_FIXEDSEEDS = true;
 static const size_t DEFAULT_MAXRECEIVEBUFFER = 5 * 1000;
 static const size_t DEFAULT_MAXSENDBUFFER    = 1 * 1000;
 
-static constexpr bool DEFAULT_V2_TRANSPORT{false};
+static constexpr bool DEFAULT_V2_TRANSPORT{true};
 
 #if defined USE_KQUEUE
 #define DEFAULT_SOCKETEVENTS "kqueue"

src/rpc/masternode.cpp

@@ -38,7 +38,7 @@ static RPCHelpMan masternode_connect()
         "Connect to given masternode\n",
         {
             {"address", RPCArg::Type::STR, RPCArg::Optional::NO, "The address of the masternode to connect"},
-            {"v2transport", RPCArg::Type::BOOL, RPCArg::Default{false}, "Attempt to connect using BIP324 v2 transport protocol"},
+            {"v2transport", RPCArg::Type::BOOL, RPCArg::DefaultHint{"set by -v2transport"}, "Attempt to connect using BIP324 v2 transport protocol"},
         },
         RPCResults{},
         RPCExamples{""},
@@ -51,12 +51,13 @@ static RPCHelpMan masternode_connect()
         throw JSONRPCError(RPC_INTERNAL_ERROR, strprintf("Incorrect masternode address %s", strAddress));
     }
 
-    bool use_v2transport = !request.params[1].isNull() && ParseBoolV(request.params[1], "v2transport");
-
     const NodeContext& node = EnsureAnyNodeContext(request.context);
     CConnman& connman = EnsureConnman(node);
 
-    if (use_v2transport && !(connman.GetLocalServices() & NODE_P2P_V2)) {
+    bool node_v2transport = connman.GetLocalServices() & NODE_P2P_V2;
+    bool use_v2transport = request.params[1].isNull() ? node_v2transport : ParseBoolV(request.params[1], "v2transport");
+
+    if (use_v2transport && !node_v2transport) {
         throw JSONRPCError(RPC_INVALID_PARAMETER, "Error: Adding v2transport connections requires -v2transport init flag to be set.");
     }
 

src/rpc/net.cpp

@@ -360,7 +360,7 @@ static RPCHelpMan addconnection()
         {
             {"address", RPCArg::Type::STR, RPCArg::Optional::NO, "The IP address and port to attempt connecting to."},
             {"connection_type", RPCArg::Type::STR, RPCArg::Optional::NO, "Type of connection to open (\"outbound-full-relay\", \"block-relay-only\", \"addr-fetch\" or \"feeler\")."},
-            {"v2transport", RPCArg::Type::BOOL, RPCArg::Default{false}, "Attempt to connect using BIP324 v2 transport protocol"},
+            {"v2transport", RPCArg::Type::BOOL, RPCArg::Optional::NO, "Attempt to connect using BIP324 v2 transport protocol"},
         },
         RPCResult{
             RPCResult::Type::OBJ, "", "",

test/functional/feature_anchors.py

@@ -7,11 +7,14 @@
 import os
 
 from test_framework.p2p import P2PInterface
+from test_framework.socks5 import Socks5Configuration, Socks5Server
+from test_framework.messages import CAddress, hash256, NODE_NETWORK
 from test_framework.test_framework import BitcoinTestFramework
-from test_framework.util import check_node_connections
+from test_framework.util import check_node_connections, assert_equal, p2p_port
 
 INBOUND_CONNECTIONS = 5
 BLOCK_RELAY_CONNECTIONS = 2
+ONION_ADDR = "pg6mmjiyjmcrsslvykfwnntlaru7p5svn6y2ymmju6nubxndf4pscryd.onion:8333"
 
 
 class AnchorsTest(BitcoinTestFramework):
@@ -55,7 +58,7 @@ class AnchorsTest(BitcoinTestFramework):
             else:
                 inbound_nodes_port.append(hex(int(addr_split[1]))[2:])
 
-        self.log.info("Stop node 0")
+        self.log.debug("Stop node")
         self.stop_node(0)
 
         # It should contain only the block-relay-only addresses
@@ -79,12 +82,64 @@ class AnchorsTest(BitcoinTestFramework):
                 tweaked_contents[20:20] = b'1'
                 out_file_handler.write(bytes(tweaked_contents))
 
-            self.log.info("Start node")
+            self.log.debug("Start node")
             self.start_node(0)
 
         self.log.info("When node starts, check if anchors.dat doesn't exist anymore")
         assert not os.path.exists(node_anchors_path)
 
+        self.log.info("Ensure addrv2 support")
+        # Use proxies to catch outbound connections to networks with 256-bit addresses
+        onion_conf = Socks5Configuration()
+        onion_conf.auth = True
+        onion_conf.unauth = True
+        onion_conf.addr = ('127.0.0.1', p2p_port(self.num_nodes))
+        onion_conf.keep_alive = True
+        onion_proxy = Socks5Server(onion_conf)
+        onion_proxy.start()
+        self.restart_node(0, extra_args=[f"-onion={onion_conf.addr[0]}:{onion_conf.addr[1]}"])
+
+        self.log.info("Add 256-bit-address block-relay-only connections to node")
+        self.nodes[0].addconnection(ONION_ADDR, 'block-relay-only', v2transport=False)
+
+        self.log.debug("Stop node")
+        with self.nodes[0].assert_debug_log([f"DumpAnchors: Flush 1 outbound block-relay-only peer addresses to anchors.dat"]):
+            self.stop_node(0)
+        # Manually close keep_alive proxy connection
+        onion_proxy.stop()
+
+        self.log.info("Check for addrv2 addresses in anchors.dat")
+        caddr = CAddress()
+        caddr.net = CAddress.NET_TORV3
+        caddr.ip, port_str = ONION_ADDR.split(":")
+        caddr.port = int(port_str)
+        # TorV3 addrv2 serialization:
+        # time(4) | services(1) | networkID(1) | address length(1) | address(32)
+        expected_pubkey = caddr.serialize_v2()[7:39].hex()
+
+        # position of services byte of first addr in anchors.dat
+        # network magic, vector length, version, nTime
+        services_index = 4 + 1 + 4 + 4
+        data = bytes()
+        with open(node_anchors_path, "rb") as file_handler:
+            data = file_handler.read()
+            assert_equal(data[services_index], 0x00)  # services == NONE
+            anchors2 = data.hex()
+            assert expected_pubkey in anchors2
+
+        with open(node_anchors_path, "wb") as file_handler:
+            # Modify service flags for this address even though we never connected to it.
+            # This is necessary because on restart we will not attempt an anchor connection
+            # to a host without our required services, even if its address is in the anchors.dat file
+            new_data = bytearray(data)[:-32]
+            new_data[services_index] = NODE_NETWORK
+            new_data_hash = hash256(new_data)
+            file_handler.write(new_data + new_data_hash)
+
+        self.log.info("Restarting node attempts to reconnect to anchors")
+        with self.nodes[0].assert_debug_log([f"Trying to make an anchor connection to {ONION_ADDR}"]):
+            self.start_node(0, extra_args=[f"-onion={onion_conf.addr[0]}:{onion_conf.addr[1]}"])
+
 
 if __name__ == "__main__":
     AnchorsTest().main()

test/functional/feature_proxy.py

@@ -125,7 +125,8 @@ class ProxyTest(BitcoinTestFramework):
         rv = []
         addr = "15.61.23.23:1234"
         self.log.debug(f"Test: outgoing IPv4 connection through node for address {addr}")
-        node.addnode(addr, "onetry")
+        # v2transport=False is used to avoid reconnections with v1 being scheduled. These could interfere with later checks.
+        node.addnode(addr, "onetry", v2transport=False)
         cmd = proxies[0].queue.get()
         assert isinstance(cmd, Socks5Command)
         # Note: dashd's SOCKS5 implementation only sends atyp DOMAINNAME, even if connecting directly to IPv4/IPv6
@@ -141,7 +142,7 @@ class ProxyTest(BitcoinTestFramework):
         if self.have_ipv6:
             addr = "[1233:3432:2434:2343:3234:2345:6546:4534]:5443"
             self.log.debug(f"Test: outgoing IPv6 connection through node for address {addr}")
-            node.addnode(addr, "onetry")
+            node.addnode(addr, "onetry", v2transport=False)
             cmd = proxies[1].queue.get()
             assert isinstance(cmd, Socks5Command)
             # Note: dashd's SOCKS5 implementation only sends atyp DOMAINNAME, even if connecting directly to IPv4/IPv6
@@ -157,7 +158,7 @@ class ProxyTest(BitcoinTestFramework):
         if test_onion:
             addr = "pg6mmjiyjmcrsslvykfwnntlaru7p5svn6y2ymmju6nubxndf4pscryd.onion:8333"
             self.log.debug(f"Test: outgoing onion connection through node for address {addr}")
-            node.addnode(addr, "onetry")
+            node.addnode(addr, "onetry", v2transport=False)
             cmd = proxies[2].queue.get()
             assert isinstance(cmd, Socks5Command)
             assert_equal(cmd.atyp, AddressType.DOMAINNAME)
@@ -172,7 +173,7 @@ class ProxyTest(BitcoinTestFramework):
         if test_cjdns:
             addr = "[fc00:1:2:3:4:5:6:7]:8888"
             self.log.debug(f"Test: outgoing CJDNS connection through node for address {addr}")
-            node.addnode(addr, "onetry")
+            node.addnode(addr, "onetry", v2transport=False)
             cmd = proxies[1].queue.get()
             assert isinstance(cmd, Socks5Command)
             assert_equal(cmd.atyp, AddressType.DOMAINNAME)
@@ -186,7 +187,7 @@ class ProxyTest(BitcoinTestFramework):
 
         addr = "node.noumenon:8333"
         self.log.debug(f"Test: outgoing DNS name connection through node for address {addr}")
-        node.addnode(addr, "onetry")
+        node.addnode(addr, "onetry", v2transport=False)
         cmd = proxies[3].queue.get()
         assert isinstance(cmd, Socks5Command)
         assert_equal(cmd.atyp, AddressType.DOMAINNAME)

test/functional/p2p_add_connections.py

@@ -17,7 +17,7 @@ class P2PFeelerReceiver(P2PInterface):
         # message is received from the test framework. Don't send any responses
         # to the node's version message since the connection will already be
         # closed.
-        pass
+        self.send_version()
 
 class P2PAddConnections(BitcoinTestFramework):
     def set_test_params(self):

test/functional/p2p_addr_relay.py

@@ -74,6 +74,7 @@ class AddrReceiver(P2PInterface):
         return self.num_ipv4_received != 0
 
     def on_version(self, message):
+        self.send_version()
         self.send_message(msg_verack())
         if (self.send_getaddr):
             self.send_message(msg_getaddr())

test/functional/p2p_addrv2_relay.py

@@ -18,6 +18,7 @@ from test_framework.test_framework import BitcoinTestFramework
 from test_framework.util import assert_equal
 
 I2P_ADDR = "c4gfnttsuwqomiygupdqqqyy5y5emnk5c73hrfvatri67prd7vyq.b32.i2p"
+ONION_ADDR = "pg6mmjiyjmcrsslvykfwnntlaru7p5svn6y2ymmju6nubxndf4pscryd.onion"
 
 ADDRS: List[CAddress] = []
 
@@ -37,6 +38,16 @@ class AddrReceiver(P2PInterface):
     def wait_for_addrv2(self):
         self.wait_until(lambda: "addrv2" in self.last_message)
 
+def calc_addrv2_msg_size(addrs):
+    size = 1  # vector length byte
+    for addr in addrs:
+        size += 4  # time
+        size += 1  # services, COMPACTSIZE(P2P_SERVICES)
+        size += 1  # network id
+        size += 1  # address length byte
+        size += addr.ADDRV2_ADDRESS_LENGTH[addr.net]  # address
+        size += 2  # port
+    return size
 
 class AddrTest(BitcoinTestFramework):
     def set_test_params(self):
@@ -48,14 +59,18 @@ class AddrTest(BitcoinTestFramework):
         for i in range(10):
             addr = CAddress()
             addr.time = int(self.mocktime) + i
+            addr.port = 8333 + i
             addr.nServices = NODE_NETWORK
-            # Add one I2P address at an arbitrary position.
+            # Add one I2P and one onion V3 address at an arbitrary position.
             if i == 5:
                 addr.net = addr.NET_I2P
                 addr.ip = I2P_ADDR
+                addr.port = 0
+            elif i == 8:
+                addr.net = addr.NET_TORV3
+                addr.ip = ONION_ADDR
             else:
                 addr.ip = f"123.123.123.{i % 256}"
-            addr.port = 8333 + i
             ADDRS.append(addr)
 
         self.log.info('Create connection that sends addrv2 messages')
@@ -73,14 +88,15 @@ class AddrTest(BitcoinTestFramework):
         addr_source = self.nodes[0].add_p2p_connection(P2PInterface())
         addr_receiver = self.nodes[0].add_p2p_connection(AddrReceiver())
         msg.addrs = ADDRS
+        msg_size = calc_addrv2_msg_size(ADDRS)
         with self.nodes[0].assert_debug_log([
-                'received: addrv2 (159 bytes) peer=1',
+                f'received: addrv2 ({msg_size} bytes) peer=1',
         ]):
             addr_source.send_and_ping(msg)
 
         # Wait until "Added ..." before bumping mocktime to make sure addv2 is (almost) fully processed
         with self.nodes[0].assert_debug_log([
-                'sending addrv2 (159 bytes) peer=2',
+                f'sending addrv2 ({msg_size} bytes) peer=2',
         ]):
             self.bump_mocktime(30 * 60)
             addr_receiver.wait_for_addrv2()

test/functional/p2p_ibd_stalling.py

@@ -77,6 +77,7 @@ class P2PIBDStallingTest(BitcoinTestFramework):
 
         self.log.info("Check that a staller does not get disconnected if the 1024 block lookahead buffer is filled")
         self.mocktime = int(time.time()) + 1
+        node.setmocktime(self.mocktime)
         for id in range(NUM_PEERS):
             peers.append(node.add_outbound_p2p_connection(P2PStaller(stall_block), services = NODE_NETWORK | NODE_BLOOM, p2p_idx=id, connection_type="outbound-full-relay"))
             peers[-1].block_store = block_dict

test/functional/p2p_sendtxrcncl.py

@@ -25,7 +25,7 @@ class PeerNoVerack(P2PInterface):
 
     def on_version(self, message):
         # Avoid sending verack in response to version.
-        pass
+        self.send_version()
 
 class SendTxrcnclReceiver(P2PInterface):
     def __init__(self):
@@ -38,7 +38,8 @@ class SendTxrcnclReceiver(P2PInterface):
 
 class P2PFeelerReceiver(SendTxrcnclReceiver):
     def on_version(self, message):
-        pass  # feeler connections can not send any message other than their own version
+        # feeler connections can not send any message other than their own version
+        self.send_version()
 
 
 class PeerTrackMsgOrder(P2PInterface):

test/functional/test_framework/messages.py

@@ -25,6 +25,7 @@ import random
 import socket
 import struct
 import time
+import unittest
 
 from test_framework.crypto.siphash import siphash256
 from test_framework.util import assert_equal
@@ -74,6 +75,9 @@ def sha256(s):
     return hashlib.sha256(s).digest()
 
 
+def sha3(s):
+    return hashlib.sha3_256(s).digest()
+
 
 def hash256(s):
     return sha256(sha256(s))
@@ -249,16 +253,25 @@ class CAddress:
 
     # see https://github.com/bitcoin/bips/blob/master/bip-0155.mediawiki
     NET_IPV4 = 1
+    NET_IPV6 = 2
+    NET_TORV3 = 4
     NET_I2P = 5
+    NET_CJDNS = 6
 
     ADDRV2_NET_NAME = {
         NET_IPV4: "IPv4",
-        NET_I2P: "I2P"
+        NET_IPV6: "IPv6",
+        NET_TORV3: "TorV3",
+        NET_I2P: "I2P",
+        NET_CJDNS: "CJDNS"
     }
 
     ADDRV2_ADDRESS_LENGTH = {
         NET_IPV4: 4,
-        NET_I2P: 32
+        NET_IPV6: 16,
+        NET_TORV3: 32,
+        NET_I2P: 32,
+        NET_CJDNS: 16
     }
 
     I2P_PAD = "===="
@@ -305,7 +318,7 @@ class CAddress:
         self.nServices = deser_compact_size(f)
 
         self.net = struct.unpack("B", f.read(1))[0]
-        assert self.net in (self.NET_IPV4, self.NET_I2P)
+        assert self.net in self.ADDRV2_NET_NAME
 
         address_length = deser_compact_size(f)
         assert address_length == self.ADDRV2_ADDRESS_LENGTH[self.net]
@@ -313,14 +326,25 @@ class CAddress:
         addr_bytes = f.read(address_length)
         if self.net == self.NET_IPV4:
             self.ip = socket.inet_ntoa(addr_bytes)
-        else:
+        elif self.net == self.NET_IPV6:
+            self.ip = socket.inet_ntop(socket.AF_INET6, addr_bytes)
+        elif self.net == self.NET_TORV3:
+            prefix = b".onion checksum"
+            version = bytes([3])
+            checksum = sha3(prefix + addr_bytes + version)[:2]
+            self.ip = b32encode(addr_bytes + checksum + version).decode("ascii").lower() + ".onion"
+        elif self.net == self.NET_I2P:
             self.ip = b32encode(addr_bytes)[0:-len(self.I2P_PAD)].decode("ascii").lower() + ".b32.i2p"
+        elif self.net == self.NET_CJDNS:
+            self.ip = socket.inet_ntop(socket.AF_INET6, addr_bytes)
+        else:
+            raise Exception(f"Address type not supported")
 
         self.port = struct.unpack(">H", f.read(2))[0]
 
     def serialize_v2(self):
         """Serialize in addrv2 format (BIP155)"""
-        assert self.net in (self.NET_IPV4, self.NET_I2P)
+        assert self.net in self.ADDRV2_NET_NAME
         r = b""
         r += struct.pack("<I", self.time)
         r += ser_compact_size(self.nServices)
@@ -328,10 +352,20 @@ class CAddress:
         r += ser_compact_size(self.ADDRV2_ADDRESS_LENGTH[self.net])
         if self.net == self.NET_IPV4:
             r += socket.inet_aton(self.ip)
-        else:
+        elif self.net == self.NET_IPV6:
+            r += socket.inet_pton(socket.AF_INET6, self.ip)
+        elif self.net == self.NET_TORV3:
+            sfx = ".onion"
+            assert self.ip.endswith(sfx)
+            r += b32decode(self.ip[0:-len(sfx)], True)[0:32]
+        elif self.net == self.NET_I2P:
             sfx = ".b32.i2p"
             assert self.ip.endswith(sfx)
             r += b32decode(self.ip[0:-len(sfx)] + self.I2P_PAD, True)
+        elif self.net == self.NET_CJDNS:
+            r += socket.inet_pton(socket.AF_INET6, self.ip)
+        else:
+            raise Exception(f"Address type not supported")
         r += struct.pack(">H", self.port)
         return r
 
@@ -2592,3 +2626,19 @@ class msg_sendtxrcncl:
     def __repr__(self):
         return "msg_sendtxrcncl(version=%lu, salt=%lu)" %\
             (self.version, self.salt)
+
+class TestFrameworkScript(unittest.TestCase):
+    def test_addrv2_encode_decode(self):
+        def check_addrv2(ip, net):
+            addr = CAddress()
+            addr.net, addr.ip = net, ip
+            ser = addr.serialize_v2()
+            actual = CAddress()
+            actual.deserialize_v2(BytesIO(ser))
+            self.assertEqual(actual, addr)
+
+        check_addrv2("1.65.195.98", CAddress.NET_IPV4)
+        check_addrv2("2001:41f0::62:6974:636f:696e", CAddress.NET_IPV6)
+        check_addrv2("2bqghnldu6mcug4pikzprwhtjjnsyederctvci6klcwzepnjd46ikjyd.onion", CAddress.NET_TORV3)
+        check_addrv2("255fhcp6ajvftnyo7bwz3an3t4a4brhopm3bamyh2iu5r3gnr2rq.b32.i2p", CAddress.NET_I2P)
+        check_addrv2("fc32:17ea:e415:c3bf:9808:149d:b5a2:c9aa", CAddress.NET_CJDNS)

test/functional/test_framework/p2p.py

@@ -253,11 +253,10 @@ class P2PConnection(asyncio.Protocol):
             send_handshake_bytes = self.v2_state.initiate_v2_handshake()
             logger.debug(f"sending {len(self.v2_state.sent_garbage)} bytes of garbage data")
             self.send_raw_message(send_handshake_bytes)
-        # if v2 connection, send `on_connection_send_msg` after initial v2 handshake.
+        # for v1 outbound connections, send version message immediately after opening
-        # if reconnection situation, send `on_connection_send_msg` after version message is received in `on_version()`.
+        # (for v2 outbound connections, send it after the initial v2 handshake)
-        if self.on_connection_send_msg and not self.supports_v2_p2p and not self.reconnect:
+        if self.p2p_connected_to_node and not self.supports_v2_p2p:
-            self.send_message(self.on_connection_send_msg)
+            self.send_version()
-            self.on_connection_send_msg = None  # Never used again
         self.on_open()
 
     def connection_lost(self, exc):
@@ -272,7 +271,7 @@ class P2PConnection(asyncio.Protocol):
         self.on_close()
 
     # v2 handshake method
-    def v2_handshake(self):
+    def _on_data_v2_handshake(self):
         """v2 handshake performed before P2P messages are exchanged (see BIP324). P2PConnection is the initiator
         (in inbound connections to TestNode) and the responder (in outbound connections from TestNode).
         Performed by:
@@ -314,9 +313,13 @@ class P2PConnection(asyncio.Protocol):
         if not is_mac_auth:
             raise ValueError("invalid v2 mac tag in handshake authentication")
         self.recvbuf = self.recvbuf[length:]
-        if self.v2_state.tried_v2_handshake and self.on_connection_send_msg:
+        if self.v2_state.tried_v2_handshake:
-            self.send_message(self.on_connection_send_msg)
+            # for v2 outbound connections, send version message immediately after v2 handshake
-            self.on_connection_send_msg = None
+            if self.p2p_connected_to_node:
+                self.send_version()
+            # process post-v2-handshake data immediately, if available
+            if len(self.recvbuf) > 0:
+                self._on_data()
 
     # Socket read methods
 
@@ -325,7 +328,7 @@ class P2PConnection(asyncio.Protocol):
         if len(t) > 0:
             self.recvbuf += t
             if self.supports_v2_p2p and not self.v2_state.tried_v2_handshake:
-                self.v2_handshake()
+                self._on_data_v2_handshake()
             else:
                 self._on_data()
 
@@ -598,11 +601,10 @@ class P2PInterface(P2PConnection):
 
     def on_version(self, message):
         assert message.nVersion >= MIN_P2P_VERSION_SUPPORTED, "Version {} received. Test framework only supports versions greater than {}".format(message.nVersion, MIN_P2P_VERSION_SUPPORTED)
-        # reconnection using v1 P2P has happened since version message can be processed, previously unsent version message is sent using v1 P2P here
+        # for inbound connections, reply to version with own version message
-        if self.reconnect:
+        # (could be due to v1 reconnect after a failed v2 handshake)
-            if self.on_connection_send_msg:
+        if not self.p2p_connected_to_node:
-                self.send_message(self.on_connection_send_msg)
+            self.send_version()
-                self.on_connection_send_msg = None
             self.reconnect = False
         if self.support_addrv2:
             self.send_message(msg_sendaddrv2())
@@ -631,9 +633,7 @@ class P2PInterface(P2PConnection):
 
     def wait_for_reconnect(self, timeout=60):
         def test_function():
-            if not (self.is_connected and self.last_message.get('version') and self.v2_state is None):
+            return self.is_connected and self.last_message.get('version') and not self.supports_v2_p2p
-                return False
-            return True
         self.wait_until(test_function, timeout=timeout, check_connected=False)
 
     # Message receiving helper methods
@@ -716,6 +716,11 @@ class P2PInterface(P2PConnection):
 
     # Message sending helper functions
 
+    def send_version(self):
+        if self.on_connection_send_msg:
+            self.send_message(self.on_connection_send_msg)
+            self.on_connection_send_msg = None  # Never used again
+
     def send_and_ping(self, message, timeout=60):
         self.send_message(message)
         self.sync_with_ping(timeout=timeout)

test/functional/test_framework/socks5.py

@@ -40,6 +40,7 @@ class Socks5Configuration():
         self.af = socket.AF_INET # Bind address family
         self.unauth = False  # Support unauthenticated
         self.auth = False  # Support authentication
+        self.keep_alive = False  # Do not automatically close connections
 
 class Socks5Command():
     """Information about an incoming socks5 command."""
@@ -115,12 +116,13 @@ class Socks5Connection():
 
             cmdin = Socks5Command(cmd, atyp, addr, port, username, password)
             self.serv.queue.put(cmdin)
-            logger.info('Proxy: %s', cmdin)
+            logger.debug('Proxy: %s', cmdin)
             # Fall through to disconnect
         except Exception as e:
             logger.exception("socks5 request handling failed.")
             self.serv.queue.put(e)
         finally:
+            if not self.serv.keep_alive:
                 self.conn.close()
 
 class Socks5Server():
@@ -133,6 +135,7 @@ class Socks5Server():
         self.running = False
         self.thread = None
         self.queue = queue.Queue() # report connections and exceptions to client
+        self.keep_alive = conf.keep_alive
 
     def run(self):
         while self.running:
@@ -157,4 +160,3 @@ class Socks5Server():
         s.connect(self.conf.addr)
         s.close()
         self.thread.join()
-

test/functional/test_framework/test_framework.py

@@ -225,6 +225,8 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
         parser.add_argument('--timeout-factor', dest="timeout_factor", type=float, default=1.0, help='adjust test timeouts by a factor. Setting it to 0 disables all timeouts')
         parser.add_argument("--v2transport", dest="v2transport", default=False, action="store_true",
                             help="use BIP324 v2 connections between all nodes by default")
+        parser.add_argument("--v1transport", dest="v1transport", default=False, action="store_true",
+                            help="Explicitly use v1 transport (can be used to overwrite global --v2transport option)")
 
         group = parser.add_mutually_exclusive_group()
         group.add_argument("--descriptors", action='store_const', const=True,
@@ -239,6 +241,8 @@ class BitcoinTestFramework(metaclass=BitcoinTestMetaClass):
         config = configparser.ConfigParser()
         config.read_file(open(self.options.configfile))
         self.config = config
+        if self.options.v1transport:
+            self.options.v2transport=False
 
         # Running TestShell in a Jupyter notebook causes an additional -f argument
         # To keep TestShell from failing with an "unrecognized argument" error, we add a dummy "-f" argument

test/functional/test_framework/test_node.py

@@ -131,8 +131,13 @@ class TestNode():
         # Default behavior from global -v2transport flag is added to args to persist it over restarts.
         # May be overwritten in individual tests, using extra_args.
         self.default_to_v2 = v2transport
-        if self.default_to_v2:
+        if self.version_is_at_least(22000000):
+            # 22.0 and later support v2transport
+            if v2transport:
                 self.args.append("-v2transport=1")
+            else:
+                self.args.append("-v2transport=0")
+        # if v2transport is requested via global flag but not supported for node version, ignore it
 
         self.cli = TestNodeCLI(bitcoin_cli, self.datadir)
         self.use_cli = use_cli

test/functional/test_framework/v2_p2p.py

@@ -259,6 +259,7 @@ class EncryptedP2PState:
             # decoy packets have contents = None. v2 handshake is complete only when version packet
             # (can be empty with contents = b"") with contents != None is received.
             if contents is not None:
+                assert contents == b""  # currently TestNode sends an empty version packet
                 self.tried_v2_handshake = True
                 return processed_length, True
             response = response[length:]

test/functional/test_runner.py

@@ -76,6 +76,7 @@ TEST_FRAMEWORK_MODULES = [
     "crypto.chacha20",
     "crypto.ellswift",
     "key",
+    "messages",
     "crypto.muhash",
     "crypto.poly1305",
     "crypto.ripemd160",
@@ -112,7 +113,7 @@ BASE_SCRIPTS = [
     'wallet_basic.py --descriptors',
     'wallet_labels.py --legacy-wallet',
     'wallet_labels.py --descriptors',
-    'p2p_timeouts.py',
+    'p2p_timeouts.py --v1transport',
     'p2p_timeouts.py --v2transport',
     'feature_bip68_sequence.py',
     'mempool_updatefromblock.py',
@@ -177,7 +178,7 @@ BASE_SCRIPTS = [
     'wallet_avoidreuse.py --descriptors',
     'mempool_reorg.py',
     'mempool_persist.py',
-    'p2p_block_sync.py',
+    'p2p_block_sync.py --v1transport',
     'p2p_block_sync.py --v2transport',
     'wallet_multiwallet.py --legacy-wallet',
     'wallet_multiwallet.py --descriptors',
@@ -207,15 +208,15 @@ BASE_SCRIPTS = [
     'p2p_addrv2_relay.py',
     'wallet_groups.py --legacy-wallet',
     'wallet_groups.py --descriptors',
-    'p2p_compactblocks_hb.py',
+    'p2p_compactblocks_hb.py --v1transport',
     'p2p_compactblocks_hb.py --v2transport',
-    'p2p_disconnect_ban.py',
+    'p2p_disconnect_ban.py --v1transport',
     'p2p_disconnect_ban.py --v2transport',
     'feature_addressindex.py',
     'feature_timestampindex.py',
     'feature_spentindex.py',
     'rpc_decodescript.py',
-    'rpc_blockchain.py',
+    'rpc_blockchain.py --v1transport',
     'rpc_blockchain.py --v2transport',
     'rpc_deprecated.py',
     'wallet_disable.py --legacy-wallet',
@@ -226,7 +227,7 @@ BASE_SCRIPTS = [
     'p2p_getaddr_caching.py',
     'p2p_getdata.py',
     'p2p_addrfetch.py',
-    'rpc_net.py',
+    'rpc_net.py --v1transport',
     'rpc_net.py --v2transport',
     'wallet_keypool.py --legacy-wallet',
     'wallet_keypool_hd.py --legacy-wallet',
@@ -235,14 +236,14 @@ BASE_SCRIPTS = [
     'p2p_nobloomfilter_messages.py',
     'p2p_filter.py',
     'p2p_blocksonly.py',
-    'rpc_setban.py',
+    'rpc_setban.py --v1transport',
     'rpc_setban.py --v2transport',
     'mining_prioritisetransaction.py',
     'p2p_invalid_locator.py',
-    'p2p_invalid_block.py',
+    'p2p_invalid_block.py --v1transport',
     'p2p_invalid_block.py --v2transport',
     'p2p_invalid_messages.py',
-    'p2p_invalid_tx.py',
+    'p2p_invalid_tx.py --v1transport',
     'p2p_invalid_tx.py --v2transport',
     'p2p_v2_transport.py',
     'p2p_v2_encrypted.py',
@@ -269,12 +270,12 @@ BASE_SCRIPTS = [
     'rpc_preciousblock.py',
     'wallet_importprunedfunds.py --legacy-wallet',
     'wallet_importprunedfunds.py --descriptors',
-    'p2p_leak_tx.py',
+    'p2p_leak_tx.py --v1transport',
     'p2p_leak_tx.py --v2transport',
     'p2p_eviction.py',
-    'p2p_ibd_stalling.py',
+    'p2p_ibd_stalling.py --v1transport',
     'p2p_ibd_stalling.py --v2transport',
-    'p2p_net_deadlock.py',
+    'p2p_net_deadlock.py --v1transport',
     'p2p_net_deadlock.py --v2transport',
     'rpc_signmessage.py',
     'rpc_generateblock.py',
@@ -366,7 +367,7 @@ BASE_SCRIPTS = [
     'feature_anchors.py',
     'feature_coinstatsindex.py',
     'wallet_orphanedreward.py',
-    'p2p_node_network_limited.py',
+    'p2p_node_network_limited.py --v1transport',
     'p2p_node_network_limited.py --v2transport',
     'p2p_permissions.py',
     'feature_blocksdir.py',