c276df775914e4e42993c76e172ef159e3b830d4 zmq: enable tcp keepalive (mruddy)
Pull request description:
This addresses https://github.com/bitcoin/bitcoin/issues/12754.
These changes enable node operators to address the silent dropping (by network middle boxes) of long-lived low-activity ZMQ TCP connections via further operating system level TCP keepalive configuration. For example, ZMQ sockets that publish block hashes can be affected in this way due to the length of time it sometimes takes between finding blocks (e.g.- sometimes more than an hour).
Prior to this patch, operating system level TCP keepalive configurations would not take effect since the SO_KEEPALIVE option was not enabled on the underlying socket.
There are additional ZMQ socket options related to TCP keepalive that can be set. However, I decided not to implement those options in this changeset because doing so would require adding additional bitcoin node configuration options, and would not yield a better outcome. I preferred a small, easily reviewable patch that doesn't add a bunch of new config options, with the tradeoff that the fine tuning would have to be done via well-documented operating system specific configurations.
I tested this patch by running a node with:
`./src/qt/bitcoin-qt -regtest -txindex -datadir=/tmp/node -zmqpubhashblock=tcp://127.0.0.1:28332 &`
and connecting to it with:
`python3 ./contrib/zmq/zmq_sub.py`
Without these changes, `ss -panto | grep 28332 | grep ESTAB | grep bitcoin` will report no keepalive timer information. With these changes, the output from the prior command will show keepalive timer information consistent with the configuration at the time of connection establishment, e.g.-: `timer:(keepalive,119min,0)`.
I also tested with a non-TCP transport and did not witness any adverse effects:
`./src/qt/bitcoin-qt -regtest -txindex -datadir=/tmp/node -zmqpubhashblock=ipc:///tmp/bitcoin.block &`
ACKs for top commit:
adamjonas:
Just to summarize for those looking to review - as of c276df775914e4e42993c76e172ef159e3b830d4 there are 3 tACKs (n-thumann, Haaroon, and dlogemann), 1 "looks good to me" (laanwj) with no NACKs or any show-stopping concerns raised.
jonasschnelli:
utACK c276df775914e4e42993c76e172ef159e3b830d4
Tree-SHA512: b884c2c9814e97e666546a7188c48f9de9541499a11a934bd48dd16169a900c900fa519feb3b1cb7e9915fc7539aac2829c7806b5937b4e1409b4805f3ef6cd1
7117d7503f39f06b74c84777ec4db5d456a8086f Update 'Secure string handling' (Prayank)
Pull request description:
- Add information about possible path traversal attack
- [wallet_name](https://bitcoincore.org/en/doc/0.20.0/rpc/wallet/createwallet/) (string): _The name for the new wallet. If this is a 'path', the wallet will be created at the 'path' location._
Fixes https://github.com/bitcoin/bitcoin/issues/20128 (Not really fixing it but workaround)
This PR is an alternative to https://github.com/bitcoin/bitcoin/pull/20393
ACKs for top commit:
michaelfolkson:
ACK 7117d7503f39f06b74c84777ec4db5d456a8086f
RiccardoMasutti:
ACK 7117d7503f
benthecarman:
ACK 7117d7503f39f06b74c84777ec4db5d456a8086f
Tree-SHA512: 0d6c4f8db5feba848bbb583e87a99e6c4b655deaa2b566164e2632acc1aabf470d4626d2dc4b82c4997effc30d9b474d860d0e0d3e896648c5cc9bfdb623da6d
This commit does the following changes:
- [wallet] Remove 'account' argument from GetLegacyBalance()
- GetLegacyBalance() is never called with an account argument.
Remove the argument and helper functions.
- [wallet] Remove CWallet::ListAccountCreditDebit()
- Function no longer used.
- [wallet] Remove AccountMove()
- Function no longer used.
- [wallet] Remove AddAccountingEntry()
- Function no longer used.
- [wallet] Remove GetAccountCreditDebit()
- Function no longer used.
- [wallet] Don't rewrite accounting entries when reordering wallet transactions.
- Accounting entries are deprecated. Don't rewrite them to the wallet
database when re-ordering transactions.
- [wallet] Remove WriteAccountingEntry()
- Function no longer used.
- [wallet] Don't read acentry key-values from wallet on load.
- [wallet] Remove ListAccountCreditDebit()
- Function no longer used.
- [wallet] Remove CAccountingEntry class
- No longer used
- [wallet] Remove GetLabelDestination
- Function no longer used.
- [wallet] Delete unused account functions
- ReadAccount
- WriteAccount
- EraseAccount
- DeleteLabel
- [wallet] Remove fromAccount argument from CommitTransaction()
- [wallet] Remove strFromAccount.
- No longer used.
- [wallet] Remove strSentAccount from GetAmounts().
- No longer used.
- [wallet] Update zapwallettxes comment to remove accounts.
- [wallet] Remove CAccount
- No longer used
- [docs] fix typo in release notes for PR 14023
bb08423d5ca866d4a139a3b57ff110d818d08b32 [doc] Add release notes for 'account' API removal (John Newbery)
1f4b865e57b4567270b1586bb1f348ab9106485d [wallet] Re-sort wallet RPC commands (John Newbery)
f0dc850bf698f7377797d7d68365d4fc79b0221c [wallet] Remove wallet account RPCs (John Newbery)
c410f415758913c933ad6c71cf50227cc85aa385 [tests] Remove wallet accounts test (John Newbery)
Pull request description:
This is the first part of #13825. It simply removes the RPC methods and tests.
#13825 touches lots of files and will require frequent rebasing.
Breaking it down for easier reviewing and fewer rebases.
Tree-SHA512: d29af8e7a035e4484e6b9bb56cb86592be0ec112d8ba4ce19c15d15366ff3086e89e99fca26b90c9d66f6d3e06894486d0f29948df0bb7dcb1e2c49c6887a85a
a2eb6f5405 [rpc] Add getnodeaddresses RPC command (chris-belcher)
Pull request description:
Implements issue https://github.com/bitcoin/bitcoin/issues/9463
New getnodeaddresses call gives access via RPC to the peers known by the node. It may be useful for bitcoin wallets to broadcast their transactions over tor for improved privacy without using the centralized DNS seeds. getnodeaddresses is very similar to the getaddr p2p method.
Please advise me on the best approach for writing an automated test. By my reading the getaddr p2p method also isn't really tested.
Tree-SHA512: ad03abf518847476495b76a2f5394b8030aa86654429167fa618e21460abb505c10ef9817ec1b80472320d41d0aff5dc94a8efce023aaaaf5e81386aa92b852b
Co-authored-by: MarcoFalke <falke.marco@gmail.com>
e2c03c1156a1a8cb2c04c180f2ddbd3535126a46 doc: Add relase note for db→walletdb rename (Wladimir J. van der Laan)
4c1d263d93988ceed53e8f6b5decaf034b68137e scripted-diff: Change `BCLog::DB` to `BCLog::WALLETDB` (Wladimir J. van der Laan)
6b42b3ba9087225fddb91dd764c42c28d0c42d0f Rename `db` log category to `walletdb` (like `coindb`) (Wladimir J. van der Laan)
Pull request description:
Rename the `db` log category to `walletdb` (in the style of, and to distinguish from `coindb`). Deprecate (but still accept) '-debug=db'.
Second commit is a scripted commit that changes the enum item name.
ACKs for top commit:
hebasto:
ACK e2c03c1156a1a8cb2c04c180f2ddbd3535126a46, tested on Linux Mint 19.2:
Tree-SHA512: a044de6f9a70e735cbb1caa4ed6bf75bc2269b2d5bc3241a25b6a6d69c1fc1d83456e252b431388ae61f4821e4fc06ecc1b634816ceadbe9a3c0e494bee6c11e
1cf9b35c0dac5f685b7ae62ded16284803816570 doc: Add developer note on c_str() (Wladimir J. van der Laan)
Pull request description:
Add a note when to use and when not to use `c_str()`.
ACKs for top commit:
elichai:
ACK 1cf9b35c0dac5f685b7ae62ded16284803816570
MarcoFalke:
Looking nice ACK 1cf9b35c0dac5f685b7ae62ded16284803816570
Tree-SHA512: 38cb5e54695782c23a82d03db214a8999b5bb52553f4fbe5322281686f42616981a217ba987feb6d87f3e6b95919cadd8484efe69ecc364ba1731aaf173626c9
8e4b4f683a0b342cec24cd51b1e98433034ea2ea Address test todos by removing -txindex to nodes. Originally added when updating getrawtransaction to stop searching unspent utxos. (Amiti Uttarwar)
Pull request description:
Original todos added when removing getrawtransaction default behavior of searching unspent utxos.
Tree-SHA512: d080953c3b0d2e5dca2265a15966dc25985a614c9cc86271ecd6276178ce428c85e262c24df92501695c32fed7beec0339b989f03cce91b57fb2efba201b7809
faede747b3 doc: Explain how to pass in non-fundamental types into functions (MarcoFalke)
Pull request description:
There is a common misconception in C++ that one ampersand is better than no ampersand and two ampersands are better than one.
ACKs for commit faede7:
practicalswift:
ACK faede747b3493544f25d601b8e02c833e54c8751
jonasschnelli:
ACK faede747b3493544f25d601b8e02c833e54c8751
Tree-SHA512: be12c23287398e4525f16e13de30e51a42d9e38284644eed5b67fa23197b09436d75a3aa8db08555ee91a38a0f159d2722b8a9927ce0bc906e600d2a7976086b
5a5ea93e87c766dac1251f0369b2737d8e9dadde Doc: add information about security to the JSON-RPC doc (David A. Harding)
Pull request description:
This documents some information about using the RPC interface securely, as suggested in https://github.com/bitcoin-core/bitcoincore.org/pull/637 by @luke-jr and @TheBlueMatt. I think it should fit in well with #14458, but is not dependent on it (and shouldn't have any significant merge conflicts with it).
Tree-SHA512: e09d82c3029ed17a8bcf50722ea25a8c6c514731f3bce01908cbb6fe48bc96a3068a025beabebc602d18e1bc0dc3f2602848abc05dca1d3efe2a988ee50068c0
bad1716c6d30fdf4be6d5050a04e1211f920bbd6 init: Modify docs and add release note for 12255 (Carl Dong)
b0c7b54d0c2e116d61e686b1adfdea6a1f7f02fe init: Use systemd automatic directory creation (Carl Dong)
Pull request description:
- `-datadir` option specified.
- Ask systemd to create and set the right mode for PID directory, configuration directory, and data directory.
- Tell systemd our group so it will set the right owner for aforementioned directories.
More information: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
Tree-SHA512: a6fad1efa2be433c1fdd863df3ff232736ed709a9e281f51a003b40987d8c213dc64a52bc13a19c85bf85680e78f0be112ecaf32ac274b1ff93bac84a1208845
645e905c327411555073fa7964b36f652998059f doc: Add release notes for unloadwallet change to synchronous call (João Barbosa)
c37851de5752f107c16e19317f28038b6b7ca2dc rpc: Make unloadwallet wait for complete wallet unload (João Barbosa)
Pull request description:
Currently the `unloadwallet` RPC is asynchronous, it only signals the intent to unload the wallet and then returns the response to the client. The actual unload can happen later and the client has no way to be notified of that.
This PR makes the `unloadwallet` RPC synchronous, meaning that it blocks until the wallet is fully unloaded.
Replaces #14919, fixes#14917.
Tree-SHA512: ad88b980e2f3652809a58f904afbfe020299f3aa6a517f495ba943b8d54d4520f6e70074d6749be8f5967065c0f476e0faedcde64c8b4899e5f99c70f0fd6534
* Merge #14060: ZMQ: add options to configure outbound message high water mark, aka SNDHWM
a4edb168b635b6f5c36324e44961cd42cf9bbbaa ZMQ: add options to configure outbound message high water mark, aka SNDHWM (mruddy)
Pull request description:
ZMQ: add options to configure outbound message high water mark, aka SNDHWM
This is my attempt at https://github.com/bitcoin/bitcoin/pull/13315
Tree-SHA512: a4cc3bcf179776899261a97c8c4f31f35d1d8950fd71a09a79c5c064879b38e600b26824c89c4091d941502ed5b0255390882f7d44baf9e6dc49d685a86e8edb
* High watermark settings for Dash-specific messages
Signed-off-by: Dzutte <dzutte.tomsk@gmail.com>
Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com>
f1bd03eb013b96ff040a8f835e4137fbd2a38cda [depends, zmq, doc] upgrade zeromq to 4.2.5 and avoid deprecated zeromq api functions (mruddy)
Pull request description:
Upgrade the ZeroMQ dependency from version 4.2.3 to the latest stable version 4.2.5.
This PR Follows the lead of https://github.com/bitcoin/bitcoin/pull/11986.
I upgraded both patch files to correspond to the version `4.2.5` libzmq files.
I assume doing so is still necessary and correct.
Without updating the patch line numbers, things appear to work, but you get extra log messages while building `depends` because things don't exactly match, e.g.:
```
/bitcoin/depends> make zeromq
Extracting zeromq...
/bitcoin/depends/sources/zeromq-4.2.5.tar.gz: OK
Preprocessing zeromq...
patching file src/windows.hpp
Hunk #1 succeeded at 58 (offset 3 lines).
patching file src/thread.cpp
Hunk #1 succeeded at 307 with fuzz 2 (offset 87 lines).
Hunk #2 succeeded at 323 with fuzz 2 (offset 90 lines).
```
Updating the patches seemed cleaner, so I did it. Note that libzmq had some whitespace changes, so that's why the updated patches do too.
More info: https://github.com/zeromq/libzmq/releases/tag/v4.2.5
tags: libzmq, zmq, 0mq
Tree-SHA512: 78659dd276b5311e40634b1bbebb802ddd6b69662ba3c84995ef1e3795c49a78b1635112c7fd72a405ea36e2cc3bdeb84e6d00d4e491a349bba1dafff50e2fa5
Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com>
fa193dc8e6f3b96fa2dba2f1c1668f7720fed320 doc: Remove win32 from the release process (MarcoFalke)
faf666f8148eeb305a9c4f78459aff2c7268016b Remove Windows 32 bit build (MarcoFalke)
Pull request description:
The Windows 32 bit build has been removed from https://bitcoincore.org/en/download/, so unless there are complaints, we don't need to build it even
ACKs for commit fa193d:
fanquake:
utACK fa193dc8e6
Tree-SHA512: d6f2976a2e0c407698f720b00ac23ec4056626de4eff8621f4c5581120af0460afd1bdef72329cc0e7d92afca48d94ae5fce6777cb36bfabb60b8034ff08fd88
65bc38d1c1f666e2c2d773111921b115d4249563 [doc] add notes on release notes (John Newbery)
Pull request description:
Explains when and how release notes should be written.
Tree-SHA512: 94085d5a30499f41e6d1821b9f157aea40b3cff61a8ba606fed1b239e794ffe6769f985f53400715d712d12aadaa8db8cfca08dd1700a1fe17df86e0e554eac2
9d0e52834 implements different disk sizes for different networks on intro (marcoagner)
Pull request description:
Fixes https://github.com/bitcoin/bitcoin/issues/13213.
Mostly, I layed out the concept to open the PR for refinement and getting feedback if the approach is okay. Changes are expected.
Two points:
- The values for both new consts `TESTNET_BLOCK_CHAIN_SIZE` and `TESTNET_CHAIN_STATE_SIZE` is certainly not optimal; I just checked the size of my testnet3 related dirs and set them to little bit higher values. Which values should be used?
- Should we do something like this to regtest? Or these "niceties" do not matter when on regtest?
Thanks!
Tree-SHA512: 8ae87a29fa8356b899e7a823c76cde793d9126b4ee59554d7a2a8edb088fe42a19976b34c06c2fd4a98a727e1e4971dd983f42b6093ea6caa255b45004e22bb4
13782b8ba8 docs: add perf section to developer docs (James O'Beirne)
58180b5fd4 tests: add utility to easily profile node performance with perf (James O'Beirne)
Pull request description:
Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`.
While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code.
`perf` usage is detailed in the excellent (and sadly unmerged) https://github.com/bitcoin/bitcoin/pull/12649; all due props to @eklitzke.
### Example
```python
with node.profile_with_perf("large-msgs"):
for i in range(200):
node.p2p.send_message(some_large_msg)
node.p2p.sync_with_ping()
```
This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`).
Running `perf report` generates nice output about where the node spent most of its time while running that part of the test:
```bash
$ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \
| c++filt \
| less
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 135 of event 'cycles:pp'
# Event count (approx.): 1458205679493582
#
# Children Self Command Shared Object Symbol
# ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................
#
70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)
|
---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)
70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int)
|
---CNetMessage::readData(char const*, unsigned int)
CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)
35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&)
|
---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&)
CNetMessage::readData(char const*, unsigned int)
CNode::ReceiveMsgBytes(char const*, unsigned int, bool&)
...
```
Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e
688f665a5e526fda0fb797bf617412fe9cbe64fd Scripts and tools & Docs: Used #!/usr/bin/env bash instead of obsolete #!/bin/bash, added linting for .sh files shebang and updated the Developer Notes. (vim88)
Pull request description:
As it was discussed in [#13510](https://github.com/bitcoin/bitcoin/pull/13510), it is better to use `#!/usr/bin/env bash` instead of `#!/bin/bash`.
Tree-SHA512: 25f71eb9a6a0cdc91568b5c6863205c5fe095f77a69e633503a2ac7805bd9013af8538e538c0c666ce96a28e3f43ce7a8df5f08d4ff007723bb588d85674f2da
6c6ee8af8 Less confusing documentation for `torpassword` (Chakib Benziane)
Pull request description:
Rebased & squashed #14609.
> The current documentation leads the reader to think hash-password is an other option.
This change is less confusing and make it clear how to use this option.
Tree-SHA512: b94223715f1d28a8218e501117faa103d7cc791333e6d9a40e9253756070316d41de73e3b4593cd1fa9e1823fa8e8de4938cf0de89117a6e4649401d877e3477
f24ed6d39f963e7f1c0b4dbd9b784132ef975f2a Delete README_osx.md and move its contents into build-osx.md (Martin Erlandsson)
Pull request description:
With its current name, the file `doc/README_osx.md` looks like an entry point README for OSX users, but it only contains specific instructions on how to build a DMG.
This PR deletes the file and moves the contents of the file into `doc/build-osx.md`.
Tree-SHA512: 2636b9da967f2a4c0d68cb9a157fb3db137bdb8fbff5d9d004f28b5d816e9c27fddc5c403e6b0c363d1dc9ddc7cac8b295efa01fc691126c0e36e21bb9b3cbd3
75a4bf699fa6bdefa1b3d8cd405ea822d6ee01c0 Update release-process.md to include RC version bumping (Andrew Chow)
04b0bc7425e43de90856beeb1f33653db109fecd build: include rc number in version number (Andrew Chow)
895e6bbb2241e9175463734f3677398a9f38f0f8 build: if VERSION_BUILD is non-zero, include it in the package version (Andrew Chow)
Pull request description:
As noted on IRC, the filenames of the gitian build results do not contain the 4th digit of the version number if it has one, e.g. 0.17.0.1 produces files with the number 0.17.0. Furthermore, when RC's are built, the resulting filenames are of the release version and do not include `rc` in them. This occurs because `configure.ac` is written to create version numbers of the form `major.minor.rev` instead of `major.minor.rev.build` and without any rc version as it does not handle rc numbers.
This PR changes `configure.ac` to include the build number if it is greater than 0. It will also include the rc number if it is greater than 0. So the filenames of the gitian builds will now contain the full version number.
This behavior can be tested by setting `_CLIENT_VERSION_BUILD` and `_CLIENT_VERSION_RC` to non-zero values and then doing `make dist`. A tar file should be created with the correct versioning.
Tree-SHA512: b77990485f2c7770be897dc136737cd805306afff9882ebef7170741f363203587356ccf8bec83163268ace1bd77433fbd2ba8c213f993677bfb867d99a0bbe7
fa4da3c058 [doc] conf: Remove deprecated options from docs, Other cleanup (MarcoFalke)
Pull request description:
Some dumb fixes, like removing the mention that free transactions are still a thing or that rpcuser/pass should be used (as opposed to rpcauth or rpc cookie).
Combined with other fixes because I don't want to create 3 pull requests:
* conf: Remove deprecated options from docs
* Remove only mention of MIT/X11
* Link to developer notes in README.md
Tree-SHA512: 9e45dc6c63037e7618cf3c871d7d9e65b66f1a952f91a6e623d97d90171e29bc40299a06029c4dc21a0f579e68021e3663186bd3a65e3ab333aff711f7dcb2bf
fa8ced32a60dea37ac169241cf9a1f708ef46c4b doc: Mention blocksonly in reduce-traffic.md, unhide option (MarcoFalke)
fa320de79faaca2b088fcbe7f76701faa9bff236 test: Add test for p2p_blocksonly (MarcoFalke)
fa3872e7b4540857261aed948b94b6b2bfdbc3d1 test: Format predicate source as multiline on error (MarcoFalke)
fa1dce7329d3e74d46ab98b93772b1832a3f1819 net: Rename ::fRelayTxes to ::g_relay_txes (MarcoFalke)
Pull request description:
This is de-facto no longer hidden
ACKs for commit fa8ced:
jamesob:
utACK fa8ced32a6
Tree-SHA512: 474fbdee6cbd035ed9068a066b6056c1f909ec7520be0417820fcd1672ab3069b53f55c5147968978d9258fd3a3933fe1a9ef8e4f6e14fb6ebbd79701a0a1245
