0e7c90eb37a687158c261ddd1ff9f1028a1e7012 test: speed up wallet_avoidreuse.py (Jon Atack)
6d50b2606ea9249627556051637080c3587b1b04 test: add logging to wallet_avoidreuse.py (Jon Atack)
Pull request description:
Inspired by PRs #17340 and #15881.
- add logging
- pass -whitelist in `set_test_params` to speed up transaction relay
`wallet_avoidreuse.py` is not intended to test P2P transaction relay/timing, so it should be fine to do this here. This reduces test run time variability and speeds up the test by 2-3 times on average.
Test run times in seconds:
- before: 20, 24, 22, 17, 27, 40, 30
- after: 10, 10, 8, 9, 10, 7, 8
ACKs for top commit:
MarcoFalke:
ACK 0e7c90eb37a687158c261ddd1ff9f1028a1e7012 🐊
fanquake:
ACK 0e7c90eb37a687158c261ddd1ff9f1028a1e7012
Tree-SHA512: 6d954a0aaf402c9594201626b59d29263479059e68fa5155bb44ed973cd0c3347729dd78b78b4d5a2275e45da365dc1afb4cc7e3293dea33fcc2e3e83a39faf5
5ebc6b0eb267e0552c66fffc5e5afe7df8becf80 bitcoind: update -avoidpartialspends description to account for auto-enable for avoid_reuse wallets (Karl-Johan Alm)
ada258f8c8f92d44d893cf9f22d15acdeca40b1a doc: release notes for avoid_reuse (Karl-Johan Alm)
27669551da52099e4a6a401acd7aa32b32832423 wallet: enable avoid_partial_spends by default if avoid_reuse is set (Karl-Johan Alm)
8f2e208f7c0468f9ba92bc789a698281b1c81284 test: add test for avoidreuse feature (Karl-Johan Alm)
0bdfbd34cf4015de87741ff549db35e5064f4e16 wallet/rpc: add 'avoid_reuse' option to RPC commands (Karl-Johan Alm)
f904723e0d5883309cb0dd14b826bc45c5e776fb wallet/rpc: add setwalletflag RPC and MUTABLE_WALLET_FLAGS (Karl-Johan Alm)
8247a0da3a46d7c38943ee0304343ab7465305bd wallet: enable avoid_reuse feature (Karl-Johan Alm)
eec15662fad917b169f5e3b8baaf4301dcf00a7b wallet: avoid reuse flags (Karl-Johan Alm)
58928098c299efdc7c5ddf2dc20716ca5272f21b wallet: make IsWalletFlagSet() const (Karl-Johan Alm)
129a5bafd9a3efa2fa16d780885048a06566d262 wallet: rename g_known_wallet_flags constant to KNOWN_WALLET_FLAGS (Karl-Johan Alm)
Pull request description:
Add a new wallet flag called `avoid_reuse` which, when enabled, will keep track of when a specific destination has been spent from, and will actively "blacklist" any new UTXOs which send to an already-spent-from destination.
This improves privacy, as a payer could otherwise begin tracking a payee's wallet by regularly peppering a known UTXO with dust outputs, which would then be scooped up and used in payments by the payee, allowing the payer to map out (1) the inputs owned by the payee and (2) the destinations to which the payee is making payments.
This replaces #10386 and together with the (now merged) #12257 it addresses #10065 in full. The concerns raised in https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-302361381 are also addressed due to #12257.
~~Note: this builds on top of #15780.~~ (merged)
ACKs for commit 5ebc6b:
jnewbery:
ACK 5ebc6b0eb
laanwj:
Concept and code-review ACK 5ebc6b0eb267e0552c66fffc5e5afe7df8becf80
meshcollider:
Code review ACK 5ebc6b0eb2
achow101:
ACK 5ebc6b0eb267e0552c66fffc5e5afe7df8becf80 modulo above nits
Tree-SHA512: fdef45826af544cbbb45634ac367852cc467ec87081d86d08b53ca849e588617e9a0a255b7e7bb28692d15332de58d6c3d274ac003355220e4213d7d9070742e
aa410c2b17 rpc: Validate maxfeerate with AmountFromValue (João Barbosa)
Pull request description:
With this change `maxfeerate` can also be set as a string, accordingly to the help test:
```
maxfeerate (numeric or string,
```
Beside, there are no tests for the removed errors.
ACKs for commit aa410c:
meshcollider:
utACK aa410c2b17
MarcoFalke:
utACK aa410c2b17 Good catch
Tree-SHA512: f3bfea91dc7daa943729e270585dbf333055aeda805fbd01eaab20a7e0e6147382647c11525334382d198df0d3d45da6102b541efda5a1361f96271c98d5d89d
90df92206cfce4e61eff9d584112643512f6b91c test: Change filemode of rpc_whitelist.py (Emil Engler)
Pull request description:
All python tests have the file mode `755`.
Probably due to a mistake `rpc_whitelist.py` is the only test with the permission `644`.
This PR makes it coherent with the other tests and updates it to `755` as well.
ACKs for top commit:
practicalswift:
ACK 90df92206cfce4e61eff9d584112643512f6b91c -- all tests should be executable
Tree-SHA512: b9e69cb5184a3bbee4c7b14ac35985145a9fd3403d0e449d79f15c18e9660cafec495d639f5f730e0c69dde5f4a3d7590b4e42d385e794cd02add1f4e3b785e7
2081442c421cc4376e5d7839f68fbe7630e89103 test: Add test for rpc_whitelist (Emil Engler)
7414d3820c833566b4f48c6c120a18bf53978c55 Add RPC Whitelist Feature from #12248 (Jeremy Rubin)
Pull request description:
Summary
====
This patch adds the RPC whitelisting feature requested in #12248. RPC Whitelists help enforce application policies for services being built on top of Bitcoin Core (e.g., your Lightning Node maybe shouldn't be adding new peers). The aim of this PR is not to make it advisable to connect your Bitcoin node to arbitrary services, but to reduce risk and prevent unintended access.
Using RPC Whitelists
====
The way it works is you specify (in your bitcoin.conf) configurations such as
```
rpcauth=user1:4cc74397d6e9972e5ee7671fd241$11849357f26a5be7809c68a032bc2b16ab5dcf6348ef3ed1cf30dae47b8bcc71
rpcauth=user2:181b4a25317bff60f3749adee7d6bca0$d9c331474f1322975fa170a2ffbcb176ba11644211746b27c1d317f265dd4ada
rpcauth=user3:a6c8a511b53b1edcf69c36984985e$13cfba0e626db19061c9d61fa58e712d0319c11db97ad845fa84517f454f6675
rpcwhitelist=user1:getnetworkinfo
rpcwhitelist=user2:getnetworkinfo,getwalletinfo, getbestblockhash
rpcwhitelistdefault=0
```
Now user1 can only call getnetworkinfo, user2 can only call getnetworkinfo or getwalletinfo, while user3 can still call all RPCs.
If any rpcwhitelist is set, act as if all users are subject to whitelists unless rpcwhitelistdefault is set to 0. If rpcwhitelistdefault is set to 1 and no rpcwhitelist is set, act as if all users are subject to whitelists.
Review Request
=====
In addition to normal review, would love specific review from someone working on LN (e.g., @ roasbeef) and someone working on an infrastructure team at an exchange (e.g., @ jimpo) to check that this works well with their system.
Notes
=====
The rpc list is spelling sensitive -- whitespace is stripped though. Spelling errors fail towards the RPC call being blocked, which is safer.
It was unclear to me if HTTPReq_JSONRPC is the best function to patch this functionality into, or if it would be better to place it in exec or somewhere else.
It was also unclear to me if it would be preferred to cache the whitelists on startup or parse them on every RPC as is done with multiUserAuthorized. I opted for the cached approach as I thought it was a bit cleaner.
Future Work
=====
In a future PR, I would like to add an inheritance scheme. This seemed more controversial so I didn't want to include that here. Inheritance semantics are tricky, but it would also make these whitelists easier to read.
It also might be good to add a `getrpcwhitelist` command to facilitate permission discovery.
Tests
=====
Thanks to @ emilengler for adding tests for this feature. The tests cover all cases except for where `rpcwhitelistdefault=1` is used, given difficulties around testing with the current test framework.
ACKs for top commit:
laanwj:
ACK 2081442c421cc4376e5d7839f68fbe7630e89103
Tree-SHA512: 0dc1ac6a6f2f4b0be9c9054d495dd17752fe7b3589aeab2c6ac4e1f91cf4e7e355deedcb5d76d707cbb5a949c2f989c871b74d6bf129351f429569a701adbcbf
b6f9e3576a1ea18572e4803aeb3f39330f0cb759 test: re-enable CLI test support by using EncodeDecimal in json.dumps() (fanquake)
Pull request description:
As mentioned in https://github.com/bitcoin/bitcoin/pull/17675#issuecomment-563188648.
ACKs for top commit:
practicalswift:
ACK b6f9e3576a1ea18572e4803aeb3f39330f0cb759 assuming Travis is happy too -- diff looks correct :)
MarcoFalke:
> ACK b6f9e35 assuming Travis is happy too -- diff looks correct :)
Tree-SHA512: 79fa535cc1756c8ee610a3d6a316a1c4f036797d6990a5620e44985393a2e52f78450f8e0021d0a148c08705fd1ba765508464a365f9030ae0d2cacbd7a93e19
1583498fb6781c01ca2f33c09319ed793964c574 Send and require SENDADDRV2 before VERACK (Pieter Wuille)
c5a89196602e43ebb1cdc9cd4f08d153419c13e1 Don't send 'sendaddrv2' to pre-70016 software (Pieter Wuille)
Pull request description:
BIP155 defines addrv2 and sendaddrv2 for all protocol versions, but some implementations reject messages they don't know. As a courtesy, don't send it to nodes with a version before 70016, as no software is known to support BIP155 that doesn't announce at least that protocol version number.
Also move the sending of sendaddrv2 earlier (before sending verack), as proposed in https://github.com/bitcoin/bips/pull/1043. This has the side effect that local address broadcast of torv3 will work (as it'll only trigger after we know whether or not the peer supports addrv2).
ACKs for top commit:
MarcoFalke:
ACK 1583498fb6781c01ca2f33c09319ed793964c574
jnewbery:
ACK 1583498fb6781c01ca2f33c09319ed793964c574
jonatack:
ACK 1583498fb6781c01ca2f33c09319ed793964c574
vasild:
ACK 1583498
Tree-SHA512: 3bd5833fa8c8567b6dedd99e4a9b6bb71c127aa66d5284b217503c86d597dc59aa7382c41f3a4bf561bb658b89db81d1a7703a700eef4ffc17cb916660e23a82
fa949b3c1325693ea7ecc5556b2de50d2a6c9ead test: Suppress epoll_ctl data race (MarcoFalke)
Pull request description:
Happens intermittently: https://cirrus-ci.com/task/5462892373868544?command=ci#L5385
ACKs for top commit:
hebasto:
ACK fa949b3c1325693ea7ecc5556b2de50d2a6c9ead, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: d5aa559fc105053da594531722f2a03d898eadeb4413c3a728fc5116cc4d1a2c16c49649a24c75ea810e4ec6bb9728b0bcd2ea991886bb9d206170218eddf6d2
a51d0ad2de89b9757d158df95ddeba2bfcb23935 rpc: Improve addnode remove command error message (Fabian Jahr)
Pull request description:
The `addnode` RPC with the `remove` command parameter is used to remove a node from the "added nodes". It did not have test coverage and in case of failure to remove the node it responded with the confusing message "Error: Node has not been added.".
This PR adds test coverage and introduces a new error code as well as changes the error message to something that makes sense.
ACKs for top commit:
laanwj:
Code review ACK a51d0ad2de89b9757d158df95ddeba2bfcb23935
theStack:
Tested ACK https://github.com/bitcoin/bitcoin/commit/a51d0ad2de
Tree-SHA512: 033ef5de0d4d49d58ef4df3759b838c9d19ee9dfb0aff9f814a3a63d124ca231a442c930efa7d343fe1f65727c4b59fc23dd5e26fe6ea69f9e84fda48b5c5cc2
ff22751417c6fbbd22f4eefd0e23431a83335c13 test: rm ascii art in rpc_fundrawtransaction (Jon Atack)
94fcc08541cf58bee864ab7c28a6c77e42472f17 test: add rpc_fundrawtransaction logging (Jon Atack)
Pull request description:
`test/functional/rpc_fundrawtransaction.py` is fairly slow to run and has no logging, so it can appear to be stalled.
This commit adds info logging at each test to provide feedback on the test run.
ACKs for top commit:
instagibbs:
utACK ff22751417
jnewbery:
tACK ff22751417c6fbbd22f4eefd0e23431a83335c13
Tree-SHA512: f4fabad8ef51c29981351bb4e66fb0c0e0517418a4a15892ef804df11d16b2d2ae1a1abc958d2b121819850278de90a2003b0edb8d7098d00360b89fa76e9062
f3f6dde56e Test coinbase category in wallet rpcs (andrewtoth)
e982f0b682 Add all category options to wallet rpc help (andrewtoth)
Pull request description:
The current helptext for `listtransactions`, `listsinceblock` and `gettransaction` only list two of the five possible options for `category`. This incorrectly implies that these are the only two options, and can cause problems if the other three options aren't accounted for. Also, some of the documentation is incorrect when specifying which options are returned for which categories.
This PR updates the helptext for these RPCs and adds a functional regression test for the cases when the other three categories are returned.
Tree-SHA512: 67dd7ff6269a3b0f17f5d1a61b0ae1fb1f3778f05e1c440bfbb9b3a005c9c6d740abcace20f3d597cf2bd6779c494448690f13fab0bd2340f206213bc7890b51
eaf4070e3a Add suppression for InterruptRPC (fRPCRunning) data race (practicalswift)
5e5138a721 travis: Use trap and set -e errtrace (Chun Kuan Lee)
069752b726 build: Enable functional tests in the ThreadSanitizer (TSan) build job (practicalswift)
Pull request description:
Enable functional tests in the ThreadSanitizer (TSan) build job.
This is a follow-up to @MarcoFalke's #14764 which added TSan but for unit tests only.
Tree-SHA512: dcc24d311fa124772c3036b16c2bf94732ece36c3e22b4bb8fe941772e52157ab2b1a90b1880b81079c2eef2d344ca7e1da58324b75dbf82d16204d591ad49fb
0dcac51049 wallet_keypool_topup.py: Test for all keypool address types (Gregory Sanders)
Pull request description:
To protect against regressions if key scanning is changed.
Tree-SHA512: d1c4bb033bafd97203a3f68fb262a501442be947907d67902f0391fbdec39c095196403c7675e602806cc68d7e2d1f552ab339a58346162379978d06dad1c4bb
* Fix build of qtbase in contrib for Gcc 11.x
It adds a patch with missing include <limits> in qtbase/src/tools/moc/generator.cpp
* Merge bitcoin/bitcoin#23716: test: replace hashlib.ripemd160 with an own implementation
5b559dc7ecf37ab1604b75ec8ffe8436377a5fb1 Swap out hashlib.ripemd160 for own implementation (Pieter Wuille)
ad3e9e1f214d739e098c6ebbd300da5df1026a44 Add pure Python RIPEMD-160 (Pieter Wuille)
Pull request description:
Closes#23710.
ACKs for top commit:
jamesob:
ACK 5b559dc7ec, pending CI
Tree-SHA512: dcd4ea2027eac572f7ab0da434b081b9a5d6b78675e559258a446b4d254b29d93c4d2cc12da4a28303543d6d99f5f2246fde4052e84af81d18e04399b137b39e
* Updates doc for Unix build: added missing dependency bison
Co-authored-by: MarcoFalke <falke.marco@gmail.com>
fa815255c70d32809aac640db4a8762c7d71e8db test: Add missing sync_all to wallet_balance test (MarcoFalke)
Pull request description:
A `syncwithvalidationinterfacequeue` should be sufficient.
Fixes #16020
ACKs for top commit:
promag:
ACK fa81525. This can be tested by adding sleep in `CWallet::BlockConnected` just before `LOCK(cs_wallet)` - master will always fail while this PR will succeed.
Tree-SHA512: 07e067c698627f90f0b9848f921b7067adc70c27105db3258e056384197e50dbee055c87839d238cc11bde11179d3f5879b39e1c8e15465f8f07558c694b677d
89339d14607434b33cfa343dc75877b62b1dfe0e tests: Add test for loadblock option (Fabian Jahr)
Pull request description:
Fixes#17019
Was initially part of #17044 but as the test got larger it made sense to split it into its own commit as suggested in #17019 .
This is testing the `-loadblock` option by using the scripts in `contrib/linearize` to generate a `bootstrap.dat` file and starting a disconnected node with it. So it is also testing the linearize scripts which were untested before and needed to be made available for the CI environment, hence they are added to `DIST_CONTRIB` in `Makefile.am`.
ACKs for top commit:
laanwj:
ACK 89339d14607434b33cfa343dc75877b62b1dfe0e
Tree-SHA512: aede0cd6e8b21194973f3633bc07fa2672d66a6f85dfe6a57cee2bb269a65d19ea49d5f9ed7914a173b3847c76e70257aa865f44bde170c1999d9655b4862d1c
0616138a0797cf68ad869906c36cf0767e20b313 tests: Remove no longer needed UBSan suppressions (issues fixed). Add documentation. (practicalswift)
Pull request description:
Remove no longer needed UBSan suppressions (issues fixed). Add documentation.
This PR is the CI-only subset of #17208 (which touches code).
From a fuzzing perspective it would be really nice to be able to run UBSan with as few suppressions as possible :)
Top commit has no ACKs.
Tree-SHA512: a926ab3e80e12a805af110fbff470cdc61ef4db536919a5b8896ea8b70f761114a52d9b1c0f48b11c1d48338351bf2e003e01ce60c613612f26ba298dcc29cd9
fa677d1801fb9153a95a1fc9855fd5f21fc440c0 ci: Remove redundant check for TRAVIS_OS_NAME (MarcoFalke)
fadccb263baf6b8694f750623add42f966e423a3 doc: Document that GNU tools are required for linters (MarcoFalke)
4444704ca9f66cdc24ab2d444941354db1dfed06 ci: Cleanup macOS runs (MarcoFalke)
Pull request description:
* Remove a commented out cleanup task in `before_cache`
* Remove the linter run on macOS, and document that GNU tools are required to run the linters
ACKs for top commit:
Sjors:
Code review ACK fa677d1801fb9153a95a1fc9855fd5f21fc440c0
laanwj:
ACK fa677d1801fb9153a95a1fc9855fd5f21fc440c0
ryanofsky:
Code review ACK fa677d1801fb9153a95a1fc9855fd5f21fc440c0 for new third commit replacing TRAVIS_OS_NAME check with NO_DEPENDS setting
Tree-SHA512: 9122a63bbe7887d9e379123152ea4ba44324cb18033b9e6b45bfdb1af665c10ea598564b9fcd57330d208a08e4696e41b4d6175f05f0843a3a76530da114f8c6
9576614d2d91ca946d164dfffca441f8bcbd6e2c doc: Describe log files + consistent paths in test READMEs (Martin Erlandsson)
Pull request description:
picks up #15830
I saw this was almost ready to merge but the test logging part was not 100% correct. I reworked that part, the rest is the same.
ACKs for top commit:
GChuf:
ACK 9576614d2d91ca946d164dfffca441f8bcbd6e2c
Tree-SHA512: 3de7f1b0a1b0419df6e7b55964d00e715b6cb7874b1849ad6f120597610d7df4182c4b61b9c9691ce04f4e392ed3caead4c623374be2066ac31319e702d45d09
32d665c2657793c8b2cc7248d26d80a940acfe20 test: fix "tx-size-small" errors after default address change (Sebastian Falbesoner)
Pull request description:
Addresses #17043, affects RBF and BIP68 functional tests.
The "tx-size-small" policy rule rejects transactions with a non-witness size of
smaller than 82 bytes (see `src/validation.cpp:MemPoolAccept::PreChecks(...)`),
which corresponds to a transaction with 1 segwit input and 1 P2WPKH output.
Through the default address change, the created test transactions have segwit
inputs now and sending to short scriptPubKeys might violate this rule. By
bumping the dummy scriptPubKey size to 22 bytes (= the size of a P2WPKH
scriptPubKey), on all occurences the problem is solved.
The dummy scriptPubKey has the format:
```21 <21-byte-long string of 'a' or 1s>```
ACKs for top commit:
instagibbs:
reACK 32d665c265 just s/Bytes/bytes/
MarcoFalke:
ACK 32d665c2657793c8b2cc7248d26d80a940acfe20
Tree-SHA512: 80e0386ff3c3f462901ba5c1e5ef2cbf095d9c0a40c8c3cfeacd4a3ab676afe744aa95b9eed77b4b3eec88bed930b33aa718117ed0977f6374e858a2f3bd5c57
