faa4043c66 qa: Run more tests with wallet disabled (MarcoFalke)
Pull request description:
Instead of skipping the whole test, only skip the wallet specific section of a test if the wallet is not compiled in. This is mostly an indentation change, so can be reviewed with `--ignore-all-space`.
Tree-SHA512: 5941a8b6b00dca5cf9438c5f6f010ba812115188a69e427d7ade4c1ab8cfe7a57c73daf52c66235dbb24b1cd9ab7c7a17c49bc23d931e041b605d79116a71f66
Merge #14324: qa: Run more tests with wallet disabled
faa4043c66 qa: Run more tests with wallet disabled (MarcoFalke)
Pull request description:
Instead of skipping the whole test, only skip the wallet specific section of a test if the wallet is not compiled in. This is mostly an indentation change, so can be reviewed with `--ignore-all-space`.
Tree-SHA512: 5941a8b6b00dca5cf9438c5f6f010ba812115188a69e427d7ade4c1ab8cfe7a57c73daf52c66235dbb24b1cd9ab7c7a17c49bc23d931e041b605d79116a71f66
168b6c317ca054c1287c36be532964e861f44266 add dummy file param to fix jupyter (Josiah Baker)
Pull request description:
this fixes argparse to use `parse_known_args`. previously, if an unknown argument was passed, argparse would fail with an `unrecognized arguments: %s` error.
## why
the documentation mentions being able to run `TestShell` in a REPL interpreter or a jupyter notebook. when i tried to run inside a jupyter notebook, i got the following error:
this was due to the notebook passing the filename of the notebook as an argument. this is a known problem with notebooks and argparse, documented here: https://stackoverflow.com/questions/48796169/how-to-fix-ipykernel-launcher-py-error-unrecognized-arguments-in-jupyter
## testing
to test, make sure you have jupyter notebooks installed. you can do this by running:
```
pip install notebook
```
or following instructions from [here](https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html).
once installed, start a notebook (`jupyter notebook`), launch a python3 kernel and run the following snippet:
```python
import sys
# make sure this is the path for your system
sys.path.insert(0, "/path/to/bitcoin/test/functional")
from test_framework.test_shell import TestShell
test = TestShell().setup(num_nodes=2, setup_clean_chain=True)
```
you should see the following output, without errors:
if you are unfamiliar with notebooks, here is a short guide on using them: https://jupyter.readthedocs.io/en/latest/running.html
ACKs for top commit:
MarcoFalke:
review ACK 168b6c317ca054c1287c36be532964e861f44266
jamesob:
crACK 168b6c317c
practicalswift:
cr ACK 168b6c317ca054c1287c36be532964e861f44266
Tree-SHA512: 4fee1563bf64a1cf9009934182412446cde03badf2f19553b78ad2cb3ceb0e5e085a5db41ed440473494ac047f04641311ecbba3948761c6553d0ca4b54937b4
3491bf358a81d41a386cd14581d15396354a6e6c test: Mention commit id in scripted diff error (Wladimir J. van der Laan)
Pull request description:
Add commit id to make spotting the issue easier.
ACKs for top commit:
robot-dreams:
ACK 3491bf358a81d41a386cd14581d15396354a6e6c
sipa:
utACK 3491bf358a81d41a386cd14581d15396354a6e6c
hebasto:
~ACK~ Concept ACK 3491bf358a81d41a386cd14581d15396354a6e6c, should help in situations like https://travis-ci.org/github/bitcoin/bitcoin/jobs/732481553
Tree-SHA512: 1ae66fa760f9e5d52e029bae71f6b5863f1efd7b95de3723ea09290944c9d7687f5ec6927aa115a3aebd6f2b993baa0c2433975c6ad5cd2858089013362eb599
e1fdd2963baab68bb6a77af2ad7a07fcacd4e73e Test batch rpc with params (Gregory Sanders)
Pull request description:
Useful as an example and test case.
ACKs for top commit:
laanwj:
ACK e1fdd2963baab68bb6a77af2ad7a07fcacd4e73e
theStack:
ACK e1fdd2963baab68bb6a77af2ad7a07fcacd4e73e
Tree-SHA512: 2d2ba8960916342b264a14624857d6dd10005be12efafb3e970b82656f721c8f3700ebc9b8809de1b2f887d482b772043504aeaeebc7f2e1c8203f076a451526
5f26855f109af53a336d5f98ed0ae584e7a31f84 test: Remove ubsan alignment suppressions (Wladimir J. van der Laan)
9d933ef9191417b4b7d29eaa3c3a571f814acc8e prevector: avoid misaligned member accesses (Anthony Towns)
Pull request description:
Ensure prevector data is appropriately aligned. Earlier discussion in #17530.
**Edit laanwj**: In contrast to #17530, it does this without increase in size of any of the coin cache data structures (x86_64, clang)
| Struct | (size,align) before | (size,align) after |
| ------------- | ------------- | ------- |
| Coin | 48, 8 | 48, 8 |
| CCoinsCacheEntry | 56, 8 | 56, 8 |
| CScript | 32, 1 | 32, 8 |
ACKs for top commit:
laanwj:
ACK 5f26855f109af53a336d5f98ed0ae584e7a31f84
practicalswift:
ACK 5f26855f109af53a336d5f98ed0ae584e7a31f84
jonatack:
ACK 5f26855f109af53a336d5f98ed0ae584e7a31f84
Tree-SHA512: 98d112d6856f683d5b212410b73f3071d2994f1efb046a2418a35890aa1cf1aa7c96a960fc2e963fa15241e861093c1ea41951cf5b4b5431f88345eb1dd0a98a
ff94da7887 tests: Make appveyor run with --usecli (practicalswift)
db01839361 test: Add missing call to skip_if_no_cli() (practicalswift)
Pull request description:
Add missing call to `skip_if_no_cli()` as suggested by @MarcoFalke in #14365.
Tree-SHA512: b0a2ddfad0f81cc9544f63c4e490fb983d833a47c23522549d1200ea6a8a132b2cd4bf0d66b862ef3a548d8471128b80aea3525fb5dec65221e23f32a8d46746
c7b3e487f2 tests: exclude all tests with difference parameters (Chun Kuan Lee)
Pull request description:
Fix broken exclusion list in functional tests. See https://github.com/bitcoin/bitcoin/pull/14007#pullrequestreview-158309105
Tree-SHA512: b6c2b86fef13e3c00c695adaeeb3e47ee9b48877c71bc605d24201ce931b2ef3ae9f5f199071fa1ec5de2d7aadc478410094c380cc297922e683e9b2569cda03
0ca4c8b3c6 Changed functional tests which do not require wallets to run without (sanket1729)
Pull request description:
Addresses #14216 . Changed Changed `get_deterministic_priv_key()` to return named tuple`(address, key)`
I have tried to be exhaustive as possible in maximum coverage for non-wallet mode without affecting any coverage for wallet mode.
However, I could not check the tests in wallet mode because of timeout issues. Hopefully, travis job checks those.
Tests `feature_block.py`, `feature_logging.py` and `feature_reindex.py` were skipping despite having no direct dependency on any wallet functions. So, I have also disabled the `skip_test_no_wallet()` for those files too.
Tree-SHA512: 8f84bd8400a732d4266c7518d5cbcf1eb761f623a64a74849e0470142c8ef22cb75364474ddae75d9213c3d16659a52917b5ed979a313695da6abd16c4fd7445
fac95398366f644911b58f1605e6bc37fb76782d qa: Run all tests even if wallet is not compiled (MarcoFalke)
faa669cbcd1fc799517b523b0f850e01b11bf40a qa: Premine to deterministic address with -disablewallet (MarcoFalke)
Pull request description:
Currently the test_runner would exit if the wallet was not compiled into the Bitcoin Core executable. However, a lot of the tests run without the wallet just fine and there is no need to globally require the wallet to run the tests.
Tree-SHA512: 63177260aa29126fd20f0be217a82b10b62288ab846f96f1cbcc3bd2c52702437703475d91eae3f8d821a3149fc62b725a4c5b2a7b3657b67ffcbc81532a03bb
661ac15a4a appveyor: Run functional tests on appveyor (Chun Kuan Lee)
2148c36b6e tests: Make it possible to run functional tests on Windows (Chun Kuan Lee)
Pull request description:
This PR do the following things:
- Make functional tests compatible with Windows
- Print color output in functional tests for Windows 10
- Run util and functional tests on appveyor
- Do not run symlink tests on Windows
Note:
- The wallet_multiwallet.py fail is unrelated to the test framework, it's a bug related to c++ code or maybe dependencies. `bitcoind` would exit with 0xC0000005(Access violation) during shutdown occasionally. Disable this for now.
- Not using `--failfast` because this is still in experimental. We should track if there is any other error.
- Disable ZMQ tests because the python zmq library could cause access violation sometimes.
- Disable `feature_notifications` because Bitcoin Core handles the command in different thread, whicha can cause a race condition.
Tree-SHA512: b76db137d264e62a5c130e1cbca7a2ca002a7a0f4153fa0b92c1ea6c9c09ef0533e11c49bdbd566c472d8ff59f245758feb5e5a6ec6cb6bb66a1c67bab5fa48a
82fc4017b774aaff8799c2b6e8ba5370d94dbf4d test: Catch decimal.InvalidOperation from TestNodeCLI#send_cli (Ben Woosley)
Pull request description:
`decimal.InvalidOperation` is a special case of a float parsing error, which
presumably should be handled in the same way as a general parsing error,
rather than blow up.
Alternatives include: logging the error, or re-raising with more information.
Example log output:
```
File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 603, in sync_all
self.sync_blocks(nodes)
File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 568, in sync_blocks
best_hash = [x.getbestblockhash() for x in rpc_connections]
File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 568, in <listcomp>
best_hash = [x.getbestblockhash() for x in rpc_connections]
File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_node.py", line 571, in __call__
return self.cli.send_cli(self.command, *args, **kwargs)
File "/home/travis/build/bitcoin/bitcoin/ci/scratch/build/bitcoin-i686-pc-linux-gnu/test/functional/test_framework/test_node.py", line 639, in send_cli
return json.loads(cli_stdout, parse_float=decimal.Decimal)
File "/usr/lib64/python3.6/json/__init__.py", line 367, in loads
return cls(**kw).decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>]
```
See: https://travis-ci.org/github/bitcoin/bitcoin/jobs/713502326
ACKs for top commit:
laanwj:
ACK 82fc4017b774aaff8799c2b6e8ba5370d94dbf4d
Tree-SHA512: 8c102b8bf831b05c5ca4b2e1feb5574dcbaed8cab0b2f22b013c5dfcb81788a38839a163dd1e2c6470ccbe5874214663b84485f45467738fd850ca38d539ae25
2c6a02e0248825e205e6deea4c38409044feb4ab Clean message_count and last_message (Troy Giorshev)
Pull request description:
From #19580
This PR changes comments to clarify the intended usage of `message_count` and `last_message`. Additionally it changes the only usage of `message_count` to use `last_message` instead, bringing the code into alignment with the intended usage.
Note: Now `message_count` is completely unused. However, it is ready to be used (i.e. the supporting code works) and likely will be used in some test in the future.
ACKs for top commit:
jnewbery:
utACK 2c6a02e0248825e205e6deea4c38409044feb4ab
Tree-SHA512: 07c7684c9586de4f845e10d7aac36c1aab9fb56b409949c1c70d5ca705bc3971ca7d5943245a0472def4efd7b4e1c5dad2f713db5ead8fca08404daf4891e98b
233a886b4221190a3e53128162d708266494576e test: check that getblockfilter RPC fails without block filter index (Sebastian Falbesoner)
Pull request description:
If a node was started without compact block filter index (parameter `--blockfilterindex=0`), the `getblockfilter` RPC call should fail.
ACKs for top commit:
MarcoFalke:
review ACK 233a886b4221190a3e53128162d708266494576e
Tree-SHA512: c8824373fad7d1de2dcb43c1d9541d736b478235be243080d2b7479c2588eac0e5722337ec1307394b331e0002fbcabb368e4955c2dc98dd5fce76d8c089e8a1
0d0bc3b5c1dad86cd5b2d7d90925d5722f2be6e8 build: Add locale fuzzer to FUZZERS_MISSING_CORPORA (practicalswift)
Pull request description:
Add `locale` fuzzer to `FUZZERS_MISSING_CORPORA`.
This is a follow-up to #18126 which broke Travis. Sorry about that :)
ACKs for top commit:
fanquake:
ACK 0d0bc3b5c1dad86cd5b2d7d90925d5722f2be6e8
Tree-SHA512: c0968dc798839f87c891d1dfccf5541883ac56b51a29f52244e78c221c9c087d2dea0a959612d907d53b29fca1f486b340227b17653227ecbf6ca5ab0e85b0d3
1b068c50dd1522990cc33e1aca444741c7e5a747 tests: Add --valgrind option to test/fuzz/test_runner.py for running fuzzing test cases under valgrind (practicalswift)
Pull request description:
Add `--valgrind` option to `test/fuzz/test_runner.py` for running fuzzing test cases under `valgrind`.
Test this PR using:
```
$ make distclean
$ ./autogen.sh
$ CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=fuzzer
$ make
$ git clone https://github.com/bitcoin-core/qa-assets
$ test/fuzz/test_runner.py --valgrind -l DEBUG qa-assets/fuzz_seed_corpus/
```
ACKs for top commit:
MarcoFalke:
ACK 1b068c50dd1522990cc33e1aca444741c7e5a747 🌒
Tree-SHA512: e6eb99af1bceaa6f36f49092a05de415848099ccc1497cc098a62e925954c978cb37a46410b44ed5eef2c6464ca4ecb06397b75b5d35701f5a8525436e47b9fd
9e165d0de4c3cd168137fc85b8f31b371bd4e851 test: Wait for 'cmpctblock' in p2p_compactblocks when it is expected (Ben Woosley)
Pull request description:
This is a more narrowly-construed wait which eliminates the possibility of the
wait being triggered by other messages.
Note `received_block_announcement` reflect three possible messages:
edec7f7c25/test/functional/p2p_compactblocks.py (L34-L53)
Prompted by looking into: #19449
ACKs for top commit:
laanwj:
Code review ACK 9e165d0de4c3cd168137fc85b8f31b371bd4e851
theStack:
ACK 9e165d0de4
Tree-SHA512: bc4a9c8bf031c8a7efb40d9625feaa3fd1f56f3b75da7034944af71ccea44328a6c708ab0c13fea85fb7cf4fd9043fe90eb94a25e95b2d42be44c2962b4904ce
d5766f223f627bf2eb731ce8552dfafa2b824378 tests: Add corpora suppression (FUZZERS_MISSING_CORPORA) for fuzzers missing in https://github.com/bitcoin-core/qa-assets/tree/master/fuzz_seed_corpus (practicalswift)
e75ecb91c730115290e1201371492c2cd334e9b4 tests: Add fuzzing harness for various CTxOut related functions (practicalswift)
ce935292c041162e160d95fc6afeda3dceded2cf tests: Add fuzzing harness for various CTxIn related functions (practicalswift)
Pull request description:
Add fuzzing harness for various `CTx{In,Out}` related functions.
**Testing this PR**
Run:
```
$ CC=clang CXX=clang++ ./configure --enable-fuzz --with-sanitizers=address,fuzzer,undefined
$ make
$ src/test/fuzz/tx_in
…
$ src/test/fuzz/tx_out
…
# And to to quickly verify that the relevant code regions are triggered, that the
# fuzzing throughput seems reasonable, etc.
$ contrib/devtools/test_fuzzing_harnesses.sh '^tx_'
```
`test_fuzzing_harnesses.sh` can be found in PR #17000.
Top commit has no ACKs.
Tree-SHA512: f1374307a2581ebc3968d012ea2438061bbb84ece068e584fae9750669a6cd003723dde14db88e77c9579281ecd4eaa2a7ff0614f253d8c075e6dd16dd2e68d5
fac3e22b18cd29053bc17065fd75db7b84ba6f40 qa: Read reject reasons from debug log, not p2p messages (MarcoFalke)
Pull request description:
For local testing we don't need to rely on p2p messages just to assert a reject reason.
Replace reading p2p messages with reading from the debug log file.
Tree-SHA512: fa59598ecf5e00cfb420ef1892d90aa415501fd882e1c608894dc577b0d00e93a442326d3a9167fef77d26aafbe345b730b49109982ccad68a5942384564a90b
bb08423d5ca866d4a139a3b57ff110d818d08b32 [doc] Add release notes for 'account' API removal (John Newbery)
1f4b865e57b4567270b1586bb1f348ab9106485d [wallet] Re-sort wallet RPC commands (John Newbery)
f0dc850bf698f7377797d7d68365d4fc79b0221c [wallet] Remove wallet account RPCs (John Newbery)
c410f415758913c933ad6c71cf50227cc85aa385 [tests] Remove wallet accounts test (John Newbery)
Pull request description:
This is the first part of #13825. It simply removes the RPC methods and tests.
#13825 touches lots of files and will require frequent rebasing.
Breaking it down for easier reviewing and fewer rebases.
Tree-SHA512: d29af8e7a035e4484e6b9bb56cb86592be0ec112d8ba4ce19c15d15366ff3086e89e99fca26b90c9d66f6d3e06894486d0f29948df0bb7dcb1e2c49c6887a85a
f13ad1cae0 modify test for memory locked in case locking pages failed at some point (Adam Jonas)
2fa85ebd1c add rpc_misc.py, mv test getmemoryinfo, add test mallocinfo (Adam Jonas)
Pull request description:
Creating the `rpc_misc.py` functional test file to add space for adding tests to a file that doesn't have a lot of coverage.
- Removing the `getmemoryinfo()` smoke test from wallet basic rather than moving it to keep the wallet decoupled. Feel like testing for reasonable memory allocation values should suffice.
- Adding coverage for `mallocinfo()`. Introduced standard lib XML parser since the function exports an XML string that describes the current state of the memory-allocation implementation in the caller.
Tree-SHA512: ced30115622916c88d1e729969ee331272ec9f2881eb36dee4bb7331bf633a6810a57fed63a0cfaf86de698edb5162e6a035efd07c89ece1df56b69d61288072
a2eb6f5405 [rpc] Add getnodeaddresses RPC command (chris-belcher)
Pull request description:
Implements issue https://github.com/bitcoin/bitcoin/issues/9463
New getnodeaddresses call gives access via RPC to the peers known by the node. It may be useful for bitcoin wallets to broadcast their transactions over tor for improved privacy without using the centralized DNS seeds. getnodeaddresses is very similar to the getaddr p2p method.
Please advise me on the best approach for writing an automated test. By my reading the getaddr p2p method also isn't really tested.
Tree-SHA512: ad03abf518847476495b76a2f5394b8030aa86654429167fa618e21460abb505c10ef9817ec1b80472320d41d0aff5dc94a8efce023aaaaf5e81386aa92b852b
Co-authored-by: MarcoFalke <falke.marco@gmail.com>
fa852f0e8d test: Bump timeout on tests that timeout on windows (MarcoFalke)
Pull request description:
Those tests build a ton of blocks and time out for me on Windows with:
```
test_framework.authproxy.JSONRPCException: 'generatetoaddress' RPC took longer than 60.000000 seconds. Consider using larger timeout for calls that take longer to return. (-344)
Tree-SHA512: a8fffeaddd02c051fbcc04bfac69f6ed826b8f16616e3b2e210a469d07c3e5706baab8121f1cd7ed265481de3a6197cf371513e2afbe506cf13b1dabfe3a0005
8e4b4f683a0b342cec24cd51b1e98433034ea2ea Address test todos by removing -txindex to nodes. Originally added when updating getrawtransaction to stop searching unspent utxos. (Amiti Uttarwar)
Pull request description:
Original todos added when removing getrawtransaction default behavior of searching unspent utxos.
Tree-SHA512: d080953c3b0d2e5dca2265a15966dc25985a614c9cc86271ecd6276178ce428c85e262c24df92501695c32fed7beec0339b989f03cce91b57fb2efba201b7809
fae8b8bb1a qa: Add tool-prefix to functional test readme (MarcoFalke)
faf3d22725 test_runner: Remove unused --force option (MarcoFalke)
Pull request description:
When someone calls the script they already have all intention to call it, no need to specify a redundant `--force`.
The functional tests are still disabled on the travis windows cross builds, where they'd run into issues when run under Wine.
Tree-SHA512: ada0dd9b3c0cd28c5832a12c5e04c029dc3bfe5ddf366fd0abc24fb7914d2e0f0a873fe756ade7ba780a561abe9bc731838c289accc421deda481269e08514cd
62d3f5057f2ed0c8646839f38dbe29adf4601502 qa: fix deprecated log.warn in feature_dbcrash test (Jon Atack)
Pull request description:
This clears up the following deprecation message when running test/functional/feature_dbcrash.py:
```
test/functional/feature_dbcrash.py:270:
DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
self.log.warn("Node %d never crashed during utxo flush!", i)
```
Git grepping indicates that this was the last remaining use of `log.warn` in the functional tests.
ACKs for top commit:
fanquake:
ACK 62d3f5057f2ed0c8646839f38dbe29adf4601502 - checked that there were no more occurrences.
Tree-SHA512: 2fe87400f82488e44391f4897876003a98736013e819a7dbc3b3e87a5ffbfba8d5ccab81cf2b7577f40135c95e4db96e93bb8cb24de396efb4ad814fbda09559
a407b6fdf3 [tests] Make random seed logged and settable (John Newbery)
Pull request description:
This allows tests which use randomness to be reproducibly run on failure.
ACKs for commit a407b6:
jonatack:
re-ACK a407b6fdf34f77eb347378674da9cf80394897de
jb55:
great! utACK a407b6fdf34f77eb347378674da9cf80394897de
Tree-SHA512: e1e89e6e76d11ddec71a8f0f077227e4b46303f80461b170900d3f95d4dcc4187b0d1decfd63562ea970aaaf530ef032a3e64ed1669aac29033d95161855fda3
fac174e2d1 lint: Check that all wallet args are hidden (MarcoFalke)
Pull request description:
Can be tested by calling `git revert 765d5890be` and then running the script
ACKs for commit fac174:
fanquake:
utACK fac174e
practicalswift:
tACK fac174e2d17e06b63aa5eb20b404af00d38db26e
Tree-SHA512: f7d40dc3d9f471c0cf77bc2746c1ef09b9df093b24508e72bfc50114c338e5dcb4a17741cf97566aeddc6d608f13e4eb1c986ae9935cebad1d589495ac16e0b2
fa6ab8ada1 rpc: Return more specific reject reason for submitblock (MarcoFalke)
Pull request description:
The second commit in #13439 made the `TODO` in the first commit impossible to solve.
The meaning of `fNewBlock` changed from "This is the first time we process this block" to "We are about to write the new *valid* block".
So whenever `fNewBlock` is true, the block was valid. And whenever the `fNewBlock` is false, the block is either valid or invalid. If it was valid and not new, we know it is a `"duplicate"`. In all other cases, the `BIP22ValidationResult()` will return the reason why it is invalid.
Tree-SHA512: 4b6edf7a912339c3acb0fccfabbdd6d812a0321fb1639c244c2714e58dc119aa2b8c6bf8f7d61ea609a1b861bbc23f920370fcf989c48452721e259a8ce93d24
Co-authored-by: MarcoFalke <falke.marco@gmail.com>
faee59103d test: Fix race in mempool_accept (MarcoFalke)
Pull request description:
If we happen to pick the same random coin to spend, there would be mempool conflicts in some runs of the test. Fix that by popping from a static list of coins to spend from.
Tree-SHA512: f6fd37e43d919371aa8bc3a2c93b569f9169961fe702f3641bb63180c3a88f12ca1857e9ed4d3723d5f04ca8ab5ef009a90e679580f36246a10b987620a55bee
8f5d9431a Add regtests for HTTP status codes. (Daniel Kraft)
Pull request description:
This adds explicit tests for the returned HTTP status codes to `interface_rpc.py` (for error cases) and the HTTP JSON-RPC client in general for success.
#15381 brought up discussion about the HTTP status codes in general, and the general opinion was that the current choice may not be ideal but should not be changed to preserve compatibility with existing JSON-RPC clients. Thus it makes sense to actually test the current status to ensure this desired compatibility is not broken accidentally.
ACKs for commit 8f5d94:
laanwj:
utACK 8f5d9431a36740aa12abc0acea64df48fe32d2a6
promag:
utACK 8f5d943.
jonasschnelli:
utACK 8f5d9431a36740aa12abc0acea64df48fe32d2a6
Tree-SHA512: 82503ccd134dd9145304e95cb6c61755f100bee27593d567cdd5c0c554d47e7b06d937456cab04107f46f4984930355db65d5e711008a0b05f2b8feec9f2950e
aaaa8eb1edba2a28916d5da6001d421c1b1b253b test: consensus: Check that final transactions are valid (MarcoFalke)
fae3617d79deee73dd375dc3ea5f4204a74420c5 test: Correctly deserialize without witness (MarcoFalke)
Pull request description:
There is no check that checks that final transactions are valid, i.e. the consensus rules could be changed (accidentally) with none of the tests failing.
Tree-SHA512: 48f4c24bfcc525ddbc1bfe8c37131953b464823428c1f7a278ba6d98b98827b6b84a8eb2b33396bfb5b8cc4012b7cc1cd771637f405ea20beddae001c22aa290
6440e61375 qa: Drop RPC connection if --usecli (João Barbosa)
Pull request description:
Drop the RPC connection used in `TestNode.wait_for_rpc_connection` if `--usecli` is set. If the connection is kept and not used the `Connection: close` header is never sent and so the connection only closes due to timeout (30 sec).
It might be sensible to revert e98a9eede2fb48ff33a020acc888cbcd83e24bbf in a follow up, however it changes the shutdown behavior.
Tree-SHA512: 2a8ee68b82ab612a556390aae521379e592c39ea0a7855a119282e6fe4cbf02ecafe7a5e2ee37d480f2c0600fa64791117a80fecc7bbe6bbb354107972b3b320
3fb09b9889665a24b34f25e9d1385a05058a28b7 Warn unrecognized sections in the config file (Akio Nakamura)
Pull request description:
This PR intends to resolve#14702.
In the config file, sections are specified by square bracket pair "[]"$,
or included in the option name itself which separated by a period"(.)".
Typicaly, [testnet] is not a correct section name and specified options
in that section are ignored but user cannot recognize what is happen.
So, add some log-warning messages if unrecognized section names are
present in the config file after checking section only args.
note: Currentry, followings are out of scope of this PR.
1) Empty section name or option name can describe.
e.g. [] , .a=b, =c
2) Multiple period characters can exist in the section name and option name.
e.g. [c.d.e], [..], f.g.h.i=j, ..=k
Tree-SHA512: 2cea02a0525feb40320613989a75cd7b7b1bd12158d5e6f3174ca77e6a25bb84425dd8812f62483df9fc482045c7b5402d69bc714430518b1847d055a2dc304b
bbbbb3f8850907d413db4715c10ef6df055234f6 qa: Add test to ensure node can generate all help texts at runtime (MarcoFalke)
Pull request description:
This might increase coverage, but more importantly this checks that the node doesn't crash when generating the help. (Right now the help is a static string, but in the future it might be generated at runtime)
Tree-SHA512: 0226e7c65f8a1a6fdc96c07dcf491d90559bc2355c92e9da9b1f174b09733fc349269e71da6d792f954de563a1e57c848471813eabae1a40b849a0d989520a0d
8931a95beca2b959c7ee73b154ce8a69acbe8599 Include util/strencodings.h which is required for IsSpace(...) (practicalswift)
7c9f7907615ff9c10a56ede5a8e47c91cb20fe3b Update KNOWN_VIOLATIONS: Remove fixed violations (practicalswift)
587924f0006d2eb9b8218b6abffe181bb9c27513 Use IsSpace(...) instead of boost::is_space (practicalswift)
c5fd143edb85d0c181e21a429f9e29d12a611831 Use ToLower(...) instead of std::tolower (practicalswift)
e70cc8983c570bbacee37a67df86b1bf959894df Use IsDigit(...) instead of std::isdigit (practicalswift)
Pull request description:
* Use `ToLower(...)` instead of `std::tolower`. `std::tolower` is locale dependent.
* Use `IsDigit(...)` instead of `std::isdigit`. Some implementations (e.g. Microsoft in 1252 codepage) may classify single-byte characters other than `[0-9]` as digits.
* Update `KNOWN_VIOLATIONS`: Remove fixed violations.
* ~~Replace use of locale dependent Boost trim (`boost::trim`) with locale independent `TrimString`.~~
* Use` IsSpace(...)` instead of `boost::is_space`
Tree-SHA512: defed016136b530b723fa185afdbd00410925a748856ba3afa4cee60f61a67617e30f304f2b9991a67b5fe075d9624f051e14342aee176f45fbc024d59e1aa82
d4d70eda33 Fix listreceivedbyaddress not taking address as a string (Eric Scrivner)
Pull request description:
Fixes#14173. Add the patch in #14173 and include a regression test.
Tree-SHA512: 5a9794e0c43e90d18c899841afbaf15eb9129d7d2f6570fccf0a1793697fe170d224c3c3995b1a35c536fac19819042823d9e3bd23b019d0f03434499243d2f5
380c843217 utils: Convert Windows args to utf-8 string (Chun Kuan Lee)
Pull request description:
Create a new class `WinCmdLineArgs` when building for Windows. It converts all command line arguments to utf8 string.
Tree-SHA512: f098520fd123a8a452bc84a55dc8c0b88f0c475410efe57f2ccc393f86c396eed59ea1575ddc1b920323792e390fdb092061d80cdcd9b682f0ac79a22a22ff82
* Merge #14060: ZMQ: add options to configure outbound message high water mark, aka SNDHWM
a4edb168b635b6f5c36324e44961cd42cf9bbbaa ZMQ: add options to configure outbound message high water mark, aka SNDHWM (mruddy)
Pull request description:
ZMQ: add options to configure outbound message high water mark, aka SNDHWM
This is my attempt at https://github.com/bitcoin/bitcoin/pull/13315
Tree-SHA512: a4cc3bcf179776899261a97c8c4f31f35d1d8950fd71a09a79c5c064879b38e600b26824c89c4091d941502ed5b0255390882f7d44baf9e6dc49d685a86e8edb
* High watermark settings for Dash-specific messages
Signed-off-by: Dzutte <dzutte.tomsk@gmail.com>
Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com>
4f4993fe2a Remove UBSan suppression (practicalswift)
958e1a307e streams: Remove unused seek(size_t) (practicalswift)
Pull request description:
Fix broken `streams_vector_reader` test. Remove unused `seek(size_t)`.
Before this change the test `streams_vector_reader` triggered an unintended unsigned integer wraparound. It tried so seek using a negative value in `reader.seek(-6)`.
Changes in this PR:
* Fix broken `VectorReader::seek(size_t)` test case
* Remove unused `seek(size_t)`
Tree-SHA512: 6c6affd680626363eef9e496748f2f86a522325abab9d6b13161f41125cdc29ceb36c2c1509c90b8ff108d606df7629e55e094cc2b6253b05a892b81ce176b71
3782075a5fd4ad0c15a6119e8cdaf136898f679e Move all PID file stuff to init.cpp (Hennadii Stepanov)
561e375c73a37934fe77a519762d81edf7a3325c Make PID file creating errors fatal (Hennadii Stepanov)
745a2ace18ce857bc712d7e66c8bad7c082c07e2 Improve PID file removing errors logging (Hennadii Stepanov)
Pull request description:
Digging into #15240 the lack of the proper logging has been discovered.
Fixed by this PR.
UPDATE (inspired by @laanwj's [comment](https://github.com/bitcoin/bitcoin/pull/15278#discussion_r252641810)):
Not being able to create the PID file is fatal now.
Output of `bitcoind`:
```
$ src/bitcoind -pid=/run/bitcoind/bitcoind.pid
2019-02-01T23:20:10Z Bitcoin Core version v0.17.99.0-561e375c7 (release build)
2019-02-01T23:20:10Z Assuming ancestors of block 0000000000000037a8cd3e06cd5edbfe9dd1dbcc5dacab279376ef7cfc2b4c75 have valid signatures.
2019-02-01T23:20:10Z Setting nMinimumChainWork=00000000000000000000000000000000000000000000007dbe94253893cbd463
2019-02-01T23:20:10Z Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
2019-02-01T23:20:10Z Using RdRand as an additional entropy source
2019-02-01T23:20:11Z Error: Unable to create the PID file '/run/bitcoind/bitcoind.pid': No such file or directory
Error: Unable to create the PID file '/run/bitcoind/bitcoind.pid': No such file or directory
2019-02-01T23:20:11Z Shutdown: In progress...
2019-02-01T23:20:11Z Shutdown: Unable to remove PID file: File does not exist
2019-02-01T23:20:11Z Shutdown: done
```
Output of `bitcoin-qt`:
**Notes for reviewers**
1. `CreatePidFile()` has been moved from `util/system.cpp` to `init.cpp` for the following reasons:
- to get the ability to use `InitError()`
- now `init.cpp` contains code of both creating PID file and removing it
2. Regarding 0.18 release process: this PR modifies 1 string and introduces 2 new ones.
Tree-SHA512: ac07d0f800e61ec759e427d0afc0ca43d67f232e977662253963afdd0a220d34b871050f58149fc9fabd427bfc8e0d3f6a6032f2a38f30ad366fc0d074b0f2b3
Merge #15278: Improve PID file error handling
c5ed6e73d Move CheckBlock() call to critical section (Hennadii Stepanov)
Pull request description:
This is an alternative to #14803.
Refs:
- #14058
- #14072
- https://github.com/bitcoin/bitcoin/pull/14803#issuecomment-442233211 by @gmaxwell
> It doesn't support multithreaded validation and there are lot of things that prevent that, which is why I was concerned. Why doesn't the lock on the block index or even cs main prevent concurrency here?
- https://github.com/bitcoin/bitcoin/pull/14803#issuecomment-442237566 by @MarcoFalke
Tree-SHA512: 2152e97106e11da5763b2748234ecd2982daadab13a0da04215f4db60af802a44ab5700f32249137d122eb13fc2a02e0f2d561d364607d727d8c6ab879339afb
ef362f2773 rpc/gui: Remove 'Unknown block versions being mined' warning (Wladimir J. van der Laan)
Pull request description:
Due to miners inserting garbage into the version numbers causing false positives, the current version signalling has become completely useless. This removes the "unknown block versions" warning which has the tendency to scare users unnecessarily (and might get them to "update" to something bad).
It preserves the warning in the logs. Whether this is desirable can be a point of discussion.
Tree-SHA512: 51407ccd24a571462465d9c7180f0f28307c50b82a03284abe783e181d8ab7e0638dbb710698d883f28de8a609db70763e39be2470d956e67c833da0768e43e9
f7e9e70468 [rpc] Remove deprecated sigrawtransaction rpc method. (John Newbery)
90c834089a [RPC] Remove warning about wallet addresses in createmultisig() (John Newbery)
df905e390e [rpc] Remove deprecated validateaddress usage. (John Newbery)
Pull request description:
The following rpc features were deprecated in V0.17:
- `validateaddress` returning wallet information about an address
- `signrawtransaction`
This PR fully removes those features. It can be merged once V0.17 has been branched from master.
Tree-SHA512: 28293d218cf7e348632081e362f8775f243d091f49aed54c354f017d4a12ae92b87b99f81ee592a1bbf4aebd5d8cd5119278141edde7a0399ff82917ed68b9f6
Co-authored-by: MarcoFalke <falke.marco@gmail.com>
* Backporting Merge #18584: test: Check that the version message does not leak the local address
fa404f1e4718e8155581f23826480086dfbcfaa6 test: Check that the version message does not leak the local address of the node (MarcoFalke)
Pull request description:
Add test for #8740
ACKs for top commit:
theStack:
ACK fa404f1e47
Tree-SHA512: 4d1c10d1c02fba4b51bd8b9eb3a0d9a682b6aac8c3f6924e295fdca3faefa5ecc3eaa87d347cfec5d2b2bc49963c10fe0a37c463f36088ed0304a2e3716b963b
* Merge #18584: test: Check that the version message does not leak the local address
Co-authored-by: MarcoFalke <falke.marco@gmail.com>
6f6514a08090b37b5e8c086015ee4881813ef867 Correct units for "-dbcache" and "-prune" (Hennadii Stepanov)
Pull request description:
Actually, all `dbcache`-related values in the code are measured in MiB (not in megabytes, MB) or in bytes (e.g., `nTotalCache`).
See: https://github.com/bitcoin/bitcoin/blob/master/src/txdb.hba8c8b2227/src/init.cpp (L1405-L1424)
Also, "-prune" is fixed:
1. The GUI values in GB are translated to the node values in MiB correctly.
2. The maximum of the "prune" `QSpinBox` is not limited by default value of 99 (GB).
Fix: #15106
Tree-SHA512: 151ec43b31b1074db8b345fedb1dcc10bde225899a5296bfc183f57e1553d13ac27db8db100226646769ad03c9fcab29d88763065a471757c6c41ac51108459d
948d8f4f10c31220ba4b6779cc862e2b6a0af5f6 lint: Enable python linters via an array (Ben Woosley)
Pull request description:
This assures consistent recording of the enabled linters.
This applies the same fix as #15170 to lint-python.sh
Tree-SHA512: 6d03f919e86e7c2465475c88b25dd84391282bcc11728078024daf0432a7dccddf9e4a2cdae35d6ef374971cb4e12f0fa21b58f757e25f2fe7c12ceb4f4b2c57
fa21568208 qa: Avoid race in p2p_invalid_block by waiting for the block request (MarcoFalke)
6c787d340c tests: Make feature_block pass on centos (MarcoFalke)
Pull request description:
This hopefully fixes#14661, which I believe is caused by a race in `send_blocks_and_test`. By setting `request_block=False` we only effectively check `node.getbestblockhash() != blocks[-1].hash` before returning and checking the debug.log. By setting `request_block=True` (the default) we make sure that we send the block, then sync with a ping before asserting on the debug.log.
Even if this patch doesn't fix the issue, it is good cleanup: There is no reason to not wait for the blocks to be requested, since in all these cases the header gives no indication that the block is consensus invalid. So this patch makes the test also a bit stricter and more useful.
Unrelated to this, I also include a fix that makes the tests pass on latest CentOS.
Tree-SHA512: c7abee3b7dc790a8af6c289159a7751bd962f6fa16c1537e7e21a0a0ef05b9596d1f4eb75319614603c05cb803e021314fa3596508ba443edd03046b25527e0f
9b4a36effcf642f3844c6696b757266686ece11a [qa] Test for duplicate inputs within a transaction (Suhas Daftuar)
b8f801964f59586508ea8da6cf3decd76bc0e571 Fix crash bug with duplicate inputs within a transaction (Suhas Daftuar)
Pull request description:
Tree-SHA512: 8c7ea34c7fa44188d86c04a690a7cbf8e9deda71ab1f7ca6d11de1f2abb3dd7222627071f86d0d39689a8b302ba9af142f0202466a67e30cd54aed3a08d4eb14
0385109444646561a718f34ae437b7e0e4d4d5bc Add test for rpcpassword hash error (MeshCollider)
13fe258e91e7a92326aedf151c571994166a06d4 Error if rpcpassword in conf contains a hash character (MeshCollider)
Pull request description:
Fixes#13143 now #13482 was merged
Tree-SHA512: e7d00c8df1657f6b8d0eee1e06b9ce2b1b0a2de487377699382c1b057836e1571dac313ca878b5877c862f0461ba789a50b239d2a9f34accd8a6321f126e3d2a
5a1f57646b qa: clean up assert_memory_usage_stable utility (James O'Beirne)
0cf1632f03 qa: fix p2p_invalid_messages on macOS (James O'Beirne)
Pull request description:
Infinite mea culpa for the number of problems with this test.
This change bumps the acceptable RSS increase threshold from 3% to 50% when spamming the test node with junk 4MB messages. On [@MarcoFalke's macOS test build](https://travis-ci.org/MarcoFalke/btc_nightly) we see RSS grow ~14% from ~71MB to 81MB, so a 50% increase threshold should be more than sufficient to avoid spurious failures.
Tree-SHA512: 150a7b88080fd883c7a5d0b9ffa470f61a97c4885fccc1a06fde6260aaef15640a7c1de7e89c581b245df7807d617ec3d86775330386ec5149ad567492fc5d31
3d305e3b89 Send fewer spam messages in p2p_invalid_messages (James O'Beirne)
Pull request description:
Builds on travis are failing because the test node isn't
able to drop all the bad messages sent within the given
timeout. Reduce the number of bad messages we're sending
and increase the timeout to avoid failures on travis.
Tree-SHA512: 11c389619d9590caf7eca74e0efe6d950469415d34220072770689024b350cc08a2d5ec90634237d87ff71ba8b638c1152b8a45ffbb2815a48bde6a88fbb8fc6
d20a9fa13d1c13f552e879798c0508be70190e71 tests: add tests for invalid P2P messages (James O'Beirne)
62f94d39f8de88a44bb0a8a2837d864f777aaacc tests: add P2PConnection.send_raw_message (James O'Beirne)
5aa31f6ef26f51ce461c917654dd1cfbbdd1409a tests: add utility to assert node memory usage hasn't increased (James O'Beirne)
Pull request description:
- Adds `p2p_invalid_messages.py`: tests based on behavior for dealing with invalid and malformed P2P messages. Includes a test verifying that we can't DoS a node by spamming it with large invalid messages.
- Adds `TestNode.assert_memory_usage_stable`: a context manager that allows us to ensure memory usage doesn't significantly increase on a node during some test.
- Adds `P2PConnection.send_raw_message`: which allows us to construct and send messages with tweaked headers.
Tree-SHA512: 720a4894c1e6d8f1551b2ae710e5b06c9e4f281524623957cb01599be9afea82671dc26d6152281de0acb87720f0c53b61e2b27d40434d30e525dd9e31fa671f
13782b8ba8 docs: add perf section to developer docs (James O'Beirne)
58180b5fd4 tests: add utility to easily profile node performance with perf (James O'Beirne)
Pull request description:
Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`.
While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code.
`perf` usage is detailed in the excellent (and sadly unmerged) https://github.com/bitcoin/bitcoin/pull/12649; all due props to @eklitzke.
### Example
```python
with node.profile_with_perf("large-msgs"):
for i in range(200):
node.p2p.send_message(some_large_msg)
node.p2p.sync_with_ping()
```
This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`).
Running `perf report` generates nice output about where the node spent most of its time while running that part of the test:
```bash
$ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \
| c++filt \
| less
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 135 of event 'cycles:pp'
# Event count (approx.): 1458205679493582
#
# Children Self Command Shared Object Symbol
# ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................
#
70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)
|
---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)
70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int)
|
---CNetMessage::readData(char const*, unsigned int)
CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)
35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&)
|
---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&)
CNetMessage::readData(char const*, unsigned int)
CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)
...
```
Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e
fa7d36b8e7 test: Move UBSAN suppressions to test/sanitizer_suppressions/ubsan (MarcoFalke)
fa36d4e456 travis: --disable-hardening for xenial thread sanitizer (MarcoFalke)
89bf196c88 travis: Run thread sanitizer (MarcoFalke)
Pull request description:
On unit tests only for now. Disabled for the gui unit tests and all functional tests.
Tree-SHA512: 56f7d3b44e7cb68c76a2dc5abd85658955b1c2188932e988667c5a1cbcdd6be995d37bb949d62c6eb08a4aebfc43ff0370b7da1719d4e4f322a3495c1941a5e0
* Merge #13743: refactor: Replace boost::bind with std::bind
cb53b825c2 scripted-diff: Replace boost::bind with std::bind (Chun Kuan Lee)
2196c51821 refactor: Use boost::scoped_connection in signal/slot, also prefer range-based loop instead of std::transform (Chun Kuan Lee)
Pull request description:
Replace boost::bind with std::bind
- In `src/rpc/server.cpp`, replace `std::transform` with simple loop.
- In `src/validation.cpp`, store the `boost::signals2::connection` object and use it to disconnect.
- In `src/validationinterface.cpp`, use 2 map to store the `boost::signals2::scoped_connection` object.
Tree-SHA512: 6653cbe00036fecfc495340618efcba6d7be0227c752b37b81a27184433330f817e8de9257774e9b35828026cb55f11ee7f17d6c388aebe22c4a3df13b5092f0
* Replace boost::bind with std::bind and remove Boost.Bind includes
Co-authored-by: MarcoFalke <falke.marco@gmail.com>
