dash/doc/developer-notes.md

1478 lines
57 KiB
Markdown
Raw Normal View History

Developer Notes
===============
2013-05-20 06:30:00 +02:00
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
<!-- markdown-toc start -->
**Table of Contents**
- [Developer Notes](#developer-notes)
- [Coding Style (General)](#coding-style-general)
- [Coding Style (C++)](#coding-style-c)
- [Coding Style (Python)](#coding-style-python)
- [Coding Style (Doxygen-compatible comments)](#coding-style-doxygen-compatible-comments)
- [Generating Documentation](#generating-documentation)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
- [Development tips and tricks](#development-tips-and-tricks)
- [Compiling for debugging](#compiling-for-debugging)
- [Show sources in debugging](#show-sources-in-debugging)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
- [Compiling for gprof profiling](#compiling-for-gprof-profiling)
- [`debug.log`](#debuglog)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
- [Testnet and Regtest modes](#testnet-and-regtest-modes)
- [DEBUG_LOCKORDER](#debug_lockorder)
- [Valgrind suppressions file](#valgrind-suppressions-file)
- [Compiling for test coverage](#compiling-for-test-coverage)
Merge #14519: tests: add utility to easily profile node performance with perf 13782b8ba8 docs: add perf section to developer docs (James O'Beirne) 58180b5fd4 tests: add utility to easily profile node performance with perf (James O'Beirne) Pull request description: Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`. While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code. `perf` usage is detailed in the excellent (and sadly unmerged) https://github.com/bitcoin/bitcoin/pull/12649; all due props to @eklitzke. ### Example ```python with node.profile_with_perf("large-msgs"): for i in range(200): node.p2p.send_message(some_large_msg) node.p2p.sync_with_ping() ``` This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`). Running `perf report` generates nice output about where the node spent most of its time while running that part of the test: ```bash $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \ | c++filt \ | less # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 135 of event 'cycles:pp' # Event count (approx.): 1458205679493582 # # Children Self Command Shared Object Symbol # ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................ # 70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) | ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int) | ---CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) | ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) ... ``` Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e
2019-02-05 23:40:11 +01:00
- [Performance profiling with perf](#performance-profiling-with-perf)
- [Sanitizers](#sanitizers)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
- [Locking/mutex usage notes](#lockingmutex-usage-notes)
- [Threads](#threads)
- [Ignoring IDE/editor files](#ignoring-ideeditor-files)
- [Development guidelines](#development-guidelines)
- [General Dash Core](#general-dash-core)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
- [Wallet](#wallet)
- [General C++](#general-c)
- [C++ data structures](#c-data-structures)
- [Strings and formatting](#strings-and-formatting)
- [Shadowing](#shadowing)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
- [Threads and synchronization](#threads-and-synchronization)
- [Scripts](#scripts)
- [Shebang](#shebang)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
- [Source code organization](#source-code-organization)
- [GUI](#gui)
- [Subtrees](#subtrees)
- [Upgrading LevelDB](#upgrading-leveldb)
- [File Descriptor Counts](#file-descriptor-counts)
- [Consensus Compatibility](#consensus-compatibility)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
- [Scripted diffs](#scripted-diffs)
- [Suggestions and examples](#suggestions-and-examples)
- [Release notes](#release-notes)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
- [RPC interface guidelines](#rpc-interface-guidelines)
Merge #18278: interfaces: Describe and follow some code conventions 3dc27a15242a22b5301904375e5880372e9b7f4d doc: Add internal interface conventions to developer notes (Russell Yanofsky) 1dca9dc4c772fa0a4ec52c4d88b7cd3d243aea7b refactor: Change createWallet, fillPSBT argument order (Russell Yanofsky) 96dfe5ced64979e51649d20555aa182defc80119 refactor: Change Chain::broadcastTransaction param order (Russell Yanofsky) 6ceb21909ce66b7b4762a855889acd46bb6b77f3 refactor: Rename Chain::Notifications methods to be consistent with other interfaces methods (Russell Yanofsky) 1c2ab1a6d29f2c6c065dae4f4a4e2ad1286311b3 refactor: Rename Node::disconnect methods (Russell Yanofsky) 77e4b0657298c715c835d8d2eb11e173852e6815 refactor: Get rid of Wallet::IsWalletFlagSet method (Russell Yanofsky) Pull request description: This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10). This PR doesn't change behavior at all, it just cleans up code in [`src/interfaces`](https://github.com/bitcoin/bitcoin/tree/master/src/interfaces) to simplify #10102, and [documents](https://github.com/ryanofsky/bitcoin/blob/pr/ipc-conv/doc/developer-notes.md#internal-interface-guidelines) coding conventions there better ACKs for top commit: hebasto: re-ACK 3dc27a15242a22b5301904375e5880372e9b7f4d, the only change since the [previous](https://github.com/bitcoin/bitcoin/pull/18278#pullrequestreview-372582146) review is rebasing. MarcoFalke: ACK 3dc27a15242a22b5301904375e5880372e9b7f4d 🕍 Tree-SHA512: 62e6a0f2488e3924e559d2074ed460b92e7a0a5d98eab492221cb20d59d04bbe32aef2a8aeba5e4ea9168cfa91acd5bc973dce6677be0180bd7a919354df53ed
2020-03-23 21:37:44 +01:00
- [Internal interface guidelines](#internal-interface-guidelines)
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
<!-- markdown-toc end -->
Coding Style (General)
----------------------
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
Various coding styles have been used during the history of the codebase,
and the result is not very consistent. However, we're now trying to converge to
a single style, which is specified below. When writing patches, favor the new
style over attempting to mimic the surrounding style, except for move-only
commits.
Do not submit patches solely to modify the style of existing code.
Coding Style (C++)
------------------
- **Indentation and whitespace rules** as specified in
[src/.clang-format](/src/.clang-format). You can use the provided
[clang-format-diff script](/contrib/devtools/README.md#clang-format-diffpy)
tool to clean up patches automatically before submission.
Merge #12982: Fix inconsistent namespace formatting guidelines cd0e1e91dd Fix inconsistent namespace formatting guidelines (Russell Yanofsky) Pull request description: Suggested formatting for namespaces in the developer guide is currently inconsistent. This commit updates the developer guide and clang-format configuration to consistently put "namespace" and opening/closing braces on the same line. Example: ```c++ namespace boost { namespace signals2 { class connection; } // namespace signals2 } // namespace boost ``` Currently the [Source code organization](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#source-code-organization) section has an example like the one above, but the [Coding style](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#coding-style) section example and description put a newline between the opening "namespace foo" and brace (but oddly no newline between closing namespace and brace). Avoiding newlines before namespace opening braces makes nested declarations less verbose and also avoids asymmetry with closing braces. It's also a common style used in our own and other codebases: * https://google.github.io/styleguide/cppguide.html#Namespaces * https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Coding_Style#Classes * https://llvm.org/docs/CodingStandards.html#namespace-indentation Tree-SHA512: 507259478e1a7f6f96db386dd04eb25aa04294f533503fdd82368cf809c3ceaa20204b2cb6ae65322eb27446e5934c1aa1ccb6240ead12aa06b314af76f68139
2018-04-17 19:22:13 +02:00
- Braces on new lines for classes, functions, methods.
- Braces on the same line for everything else.
- 4 space indentation (no tabs) for every block except namespaces.
- No indentation for `public`/`protected`/`private` or for `namespace`.
- No extra spaces inside parenthesis; don't do `( this )`.
- No space after function names; one space after `if`, `for` and `while`.
- If an `if` only has a single-statement `then`-clause, it can appear
on the same line as the `if`, without braces. In every other case,
braces are required, and the `then` and `else` clauses must appear
correctly indented on a new line.
Merge #20986: docs: update developer notes to discourage very long lines aa929abf8dc022e900755234c857541faeea8239 [docs] Update developer notes to discourage very long lines (John Newbery) Pull request description: Mandatory rules on line lengths are bad - there will always be cases where a longer line is more readable than the alternative. However, very long lines for no good reason _do_ hurt readability. For example, this declaration in validation.h is 274 chars: ```c++ bool ConnectTip(BlockValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexNew, const std::shared_ptr<const CBlock>& pblock, ConnectTrace& connectTrace, DisconnectedBlockTransactions& disconnectpool) EXCLUSIVE_LOCKS_REQUIRED(cs_main, m_mempool.cs); ``` That won't fit on one line without wrapping on my 27" monitor with a comfortable font size. Much easier to read is something like: ```c++ bool ConnectTip(BlockValidationState& state, const CChainParams& chainparams, CBlockIndex* pindexNew, const std::shared_ptr<const CBlock>& pblock, ConnectTrace& connectTrace, DisconnectedBlockTransactions& disconnectpool) EXCLUSIVE_LOCKS_REQUIRED(cs_main, m_mempool.cs); ``` Therefore, _discourage_ (don't forbid) line lengths greater than 100 characters in our developer style guide. 100 chars is somewhat arbitrary. The old standard was 80, but that seems very limiting with modern displays. ACKs for top commit: fanquake: ACK aa929abf8dc022e900755234c857541faeea8239 - this is basically just something to point too when a PR has unreasonably long lines for no particularly reason. practicalswift: ACK aa929abf8dc022e900755234c857541faeea8239 amitiuttarwar: ACK aa929abf8dc022e900755234c857541faeea8239 theStack: ACK aa929abf8dc022e900755234c857541faeea8239 glozow: ACK https://github.com/bitcoin/bitcoin/commit/aa929abf8dc022e900755234c857541faeea8239 Tree-SHA512: 17f1b11f811137497ede8851ede93fa612dc622922b5ad7ac8f065ea026d9a718db5b92325754b74d24012b4d45c4e2cd5cd439a6a8d34bbabf5da927d783970
2021-02-14 09:48:28 +01:00
- There's no hard limit on line width, but prefer to keep lines to <100
characters if doing so does not decrease readability. Break up long
function declarations over multiple lines using the Clang Format
[AlignAfterOpenBracket](https://clang.llvm.org/docs/ClangFormatStyleOptions.html)
style option.
- **Symbol naming conventions**. These are preferred in new code, but are not
required when doing so would need changes to significant pieces of existing
code.
- Variable (including function arguments) and namespace names are all lowercase and may use `_` to
separate words (snake_case).
- Class member variables have a `m_` prefix.
- Global variables have a `g_` prefix.
- Constant names are all uppercase, and use `_` to separate words.
- Enumerator constants may be `snake_case`, `PascalCase` or `ALL_CAPS`.
This is a more tolerant policy than the [C++ Core
Guidelines](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Renum-caps),
which recommend using `snake_case`. Please use what seems appropriate.
- Class names, function names, and method names are UpperCamelCase
(PascalCase). Do not prefix class names with `C`.
- Test suite naming convention: The Boost test suite in file
`src/test/foo_tests.cpp` should be named `foo_tests`. Test suite names
must be unique.
- **Miscellaneous**
- `++i` is preferred over `i++`.
- `nullptr` is preferred over `NULL` or `(void*)0`.
- `static_assert` is preferred over `assert` where possible. Generally; compile-time checking is preferred over run-time checking.
- Align pointers and references to the left i.e. use `type& var` and not `type &var`.
For function calls a namespace should be specified explicitly, unless such functions have been declared within it.
Otherwise, [argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl), also known as ADL, could be
triggered that makes code harder to maintain and reason about:
```c++
#include <filesystem>
namespace fs {
class path : public std::filesystem::path
{
};
// The intention is to disallow this function.
bool exists(const fs::path& p) = delete;
} // namespace fs
int main()
{
//fs::path p; // error
std::filesystem::path p; // compiled
exists(p); // ADL being used for unqualified name lookup
}
```
Block style example:
```c++
int g_count = 0;
Merge #12982: Fix inconsistent namespace formatting guidelines cd0e1e91dd Fix inconsistent namespace formatting guidelines (Russell Yanofsky) Pull request description: Suggested formatting for namespaces in the developer guide is currently inconsistent. This commit updates the developer guide and clang-format configuration to consistently put "namespace" and opening/closing braces on the same line. Example: ```c++ namespace boost { namespace signals2 { class connection; } // namespace signals2 } // namespace boost ``` Currently the [Source code organization](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#source-code-organization) section has an example like the one above, but the [Coding style](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#coding-style) section example and description put a newline between the opening "namespace foo" and brace (but oddly no newline between closing namespace and brace). Avoiding newlines before namespace opening braces makes nested declarations less verbose and also avoids asymmetry with closing braces. It's also a common style used in our own and other codebases: * https://google.github.io/styleguide/cppguide.html#Namespaces * https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Coding_Style#Classes * https://llvm.org/docs/CodingStandards.html#namespace-indentation Tree-SHA512: 507259478e1a7f6f96db386dd04eb25aa04294f533503fdd82368cf809c3ceaa20204b2cb6ae65322eb27446e5934c1aa1ccb6240ead12aa06b314af76f68139
2018-04-17 19:22:13 +02:00
namespace foo {
class Class
{
std::string m_name;
public:
bool Function(const std::string& s, int n)
{
// Comment summarising what this section of code does
for (int i = 0; i < n; ++i) {
int total_sum = 0;
// When something fails, return early
if (!Something()) return false;
...
if (SomethingElse(i)) {
total_sum += ComputeSomething(g_count);
} else {
DoSomething(m_name, total_sum);
}
}
2013-05-20 06:30:00 +02:00
// Success return is usually at the end
return true;
}
}
} // namespace foo
```
2013-05-20 06:30:00 +02:00
Coding Style (C++ named arguments)
------------------------------
When passing named arguments, use a format that clang-tidy understands. The
argument names can otherwise not be verified by clang-tidy.
For example:
```c++
void function(Addrman& addrman, bool clear);
int main()
{
function(g_addrman, /*clear=*/false);
}
```
### Running clang-tidy
To run clang-tidy on Ubuntu/Debian, install the dependencies:
```sh
apt install clang-tidy bear clang
```
Then, pass clang as compiler to configure, and use bear to produce the `compile_commands.json`:
```sh
./autogen.sh && ./configure CC=clang CXX=clang++
make clean && bear make -j $(nproc) # For bear 2.x
make clean && bear -- make -j $(nproc) # For bear 3.x
```
To run clang-tidy on all source files:
```sh
( cd ./src/ && run-clang-tidy -j $(nproc) )
```
To run clang-tidy on the changed source lines:
```sh
git diff | ( cd ./src/ && clang-tidy-diff -p2 -j $(nproc) )
```
Coding Style (Python)
---------------------
Refer to [/test/functional/README.md#style-guidelines](/test/functional/README.md#style-guidelines).
Coding Style (Doxygen-compatible comments)
------------------------------------------
Dash Core uses [Doxygen](https://www.doxygen.nl/) to generate its official documentation.
Use Doxygen-compatible comment blocks for functions, methods, and fields.
For example, to describe a function use:
```c++
/**
* ... Description ...
*
* @param[in] arg1 input description...
* @param[in] arg2 input description...
* @param[out] arg3 output description...
* @return Return cases...
* @throws Error type and cases...
* @pre Pre-condition for function...
* @post Post-condition for function...
*/
bool function(int arg1, const char *arg2, std::string& arg3)
```
A complete list of `@xxx` commands can be found at https://www.doxygen.nl/manual/commands.html.
As Doxygen recognizes the comments by the delimiters (`/**` and `*/` in this case), you don't
2015-04-28 16:48:28 +02:00
*need* to provide any commands for a comment to be valid; just a description text is fine.
To describe a class, use the same construct above the class definition:
```c++
/**
* Alerts are for notifying old versions if they become too obsolete and
* need to upgrade. The message is displayed in the status bar.
* @see GetWarnings()
*/
class CAlert
```
To describe a member or variable use:
```c++
//! Description before the member
int var;
```
or
```c++
int var; //!< Description after the member
```
Also OK:
```c++
///
/// ... Description ...
///
bool function2(int arg1, const char *arg2)
```
Not picked up by Doxygen:
```c++
//
// ... Description ...
//
```
Also not picked up by Doxygen:
```c++
/*
* ... Description ...
*/
```
A full list of comment syntaxes picked up by Doxygen can be found at https://www.doxygen.nl/manual/docblocks.html,
but the above styles are favored.
Recommendations:
- Avoiding duplicating type and input/output information in function
descriptions.
- Use backticks (&#96;&#96;) to refer to `argument` names in function and
parameter descriptions.
- Backticks aren't required when referring to functions Doxygen already knows
about; it will build hyperlinks for these automatically. See
https://www.doxygen.nl/manual/autolink.html for complete info.
- Avoid linking to external documentation; links can break.
- Javadoc and all valid Doxygen comments are stripped from Doxygen source code
previews (`STRIP_CODE_COMMENTS = YES` in [Doxyfile.in](doc/Doxyfile.in)). If
you want a comment to be preserved, it must instead use `//` or `/* */`.
### Generating Documentation
The documentation can be generated with `make docs` and cleaned up with `make
clean-docs`. The resulting files are located in `doc/doxygen/html`; open
`index.html` in that directory to view the homepage.
Before running `make docs`, you'll need to install these dependencies:
Linux: `sudo apt install doxygen graphviz`
MacOS: `brew install doxygen graphviz`
Development tips and tricks
---------------------------
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
### Compiling for debugging
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
Run configure with `--enable-debug` to add additional compiler flags that
produce better debugging builds.
### Show sources in debugging
If you have ccache enabled, absolute paths are stripped from debug information
with the -fdebug-prefix-map and -fmacro-prefix-map options (if supported by the
compiler). This might break source file detection in case you move binaries
after compilation, debug from the directory other than the project root or use
an IDE that only supports absolute paths for debugging.
There are a few possible fixes:
1. Configure source file mapping.
For `gdb` create or append to `.gdbinit` file:
```
set substitute-path ./src /path/to/project/root/src
```
For `lldb` create or append to `.lldbinit` file:
```
settings set target.source-map ./src /path/to/project/root/src
```
2. Add a symlink to the `./src` directory:
```
ln -s /path/to/project/root/src src
```
3. Use `debugedit` to modify debug information in the binary.
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
### Compiling for gprof profiling
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
Run configure with the `--enable-gprof` option, then make.
### `debug.log`
If the code is behaving strangely, take a look in the `debug.log` file in the data directory;
error and debugging messages are written there.
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
The `-debug=...` command-line option controls debugging; running with just `-debug` or `-debug=1` will turn
on all categories (and give you a very large `debug.log` file).
The Qt code routes `qDebug()` output to `debug.log` under category "qt": run with `-debug=qt`
to see it.
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
### Testnet and Regtest modes
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
Run with the `-testnet` option to run with "play coins" on the test network, if you
are testing multi-machine code that needs to operate across the internet.
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
If you are testing something that can run on one machine, run with the `-regtest` option.
In regression test mode, blocks can be created on-demand; see [test/functional/](/test/functional) for tests
that run in `-regtest` mode.
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
### DEBUG_LOCKORDER
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
Dash Core is a multi-threaded application, and deadlocks or other
multi-threading bugs can be very difficult to track down. The `--enable-debug`
configure option adds `-DDEBUG_LOCKORDER` to the compiler flags. This inserts
run-time checks to keep track of which locks are held and adds warnings to the
`debug.log` file if inconsistencies are detected.
### Assertions and Checks
The util file `src/util/check.h` offers helpers to protect against coding and
internal logic bugs. They must never be used to validate user, network or any
other input.
* `assert` or `Assert` should be used to document assumptions when any
violation would mean that it is not safe to continue program execution. The
code is always compiled with assertions enabled.
- For example, a nullptr dereference or any other logic bug in validation
code means the program code is faulty and must terminate immediately.
* `CHECK_NONFATAL` should be used for recoverable internal logic bugs. On
failure, it will throw an exception, which can be caught to recover from the
error.
- For example, a nullptr dereference or any other logic bug in RPC code
means that the RPC code is faulty and can not be executed. However, the
logic bug can be shown to the user and the program can continue to run.
* `Assume` should be used to document assumptions when program execution can
safely continue even if the assumption is violated. In debug builds it
behaves like `Assert`/`assert` to notify developers and testers about
nonfatal errors. In production it doesn't warn or log anything, though the
expression is always evaluated.
- For example it can be assumed that a variable is only initialized once,
but a failed assumption does not result in a fatal bug. A failed
assumption may or may not result in a slightly degraded user experience,
but it is safe to continue program execution.
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
### Valgrind suppressions file
Merge #11035: [contrib] Add Valgrind suppressions file 4a426d8 Add note about Valgrind suppressions file in developer-notes.md (practicalswift) 84e2462 contrib: Add Valgrind suppressions file (practicalswift) Pull request description: Includes known Valgrind warnings in our dependencies that cannot be fixed in-tree. Example use: ``` $ valgrind --suppressions=contrib/valgrind.supp src/test/test_bitcoin $ valgrind --suppressions=contrib/valgrind.supp --leak-check=full \ --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite ``` Running with the suppressions file under Ubuntu 16.04: ``` $ valgrind --suppressions=contrib/valgrind.supp --leak-check=full --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite --run_test=wallet_crypto … ==10769== LEAK SUMMARY: ==10769== definitely lost: 0 bytes in 0 blocks ==10769== indirectly lost: 0 bytes in 0 blocks ==10769== possibly lost: 0 bytes in 0 blocks ==10769== still reachable: 0 bytes in 0 blocks ==10769== suppressed: 72,704 bytes in 1 blocks ``` Running without the suppressions file under Ubuntu 16.04: ``` $ valgrind --leak-check=full --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite --run_test=wallet_crypto … ==10724== 72,704 bytes in 1 blocks are still reachable in loss record 1 of 1 ==10724== at 0x4C2DBF6: malloc (vg_replace_malloc.c:299) ==10724== by 0x6F74EFF: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21) ==10724== by 0x40106B9: call_init.part.0 (dl-init.c:72) ==10724== by 0x40107CA: call_init (dl-init.c:30) ==10724== by 0x40107CA: _dl_init (dl-init.c:120) ==10724== by 0x4000C69: ??? (in /lib/x86_64-linux-gnu/ld-2.23.so) ==10724== by 0x2: ??? ==10724== by 0x1FFF0006D2: ??? ==10724== by 0x1FFF0006E8: ??? ==10724== by 0x1FFF0006FF: ??? ==10724== ==10724== LEAK SUMMARY: ==10724== definitely lost: 0 bytes in 0 blocks ==10724== indirectly lost: 0 bytes in 0 blocks ==10724== possibly lost: 0 bytes in 0 blocks ==10724== still reachable: 72,704 bytes in 1 blocks ==10724== suppressed: 0 bytes in 0 blocks ``` Tree-SHA512: 9c92079fc61313ea678deb6aaa16a3a71c3154c757459793eb9ca0d90a9a74c6faebfb04c9135e1b398ca34224fb7f03bd9c488ea0e8debf6894f69f030a31d3
2017-11-13 14:59:29 +01:00
Valgrind is a programming tool for memory debugging, memory leak detection, and
profiling. The repo contains a Valgrind suppressions file
([`valgrind.supp`](https://github.com/dashpay/dash/blob/master/contrib/valgrind.supp))
Merge #11035: [contrib] Add Valgrind suppressions file 4a426d8 Add note about Valgrind suppressions file in developer-notes.md (practicalswift) 84e2462 contrib: Add Valgrind suppressions file (practicalswift) Pull request description: Includes known Valgrind warnings in our dependencies that cannot be fixed in-tree. Example use: ``` $ valgrind --suppressions=contrib/valgrind.supp src/test/test_bitcoin $ valgrind --suppressions=contrib/valgrind.supp --leak-check=full \ --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite ``` Running with the suppressions file under Ubuntu 16.04: ``` $ valgrind --suppressions=contrib/valgrind.supp --leak-check=full --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite --run_test=wallet_crypto … ==10769== LEAK SUMMARY: ==10769== definitely lost: 0 bytes in 0 blocks ==10769== indirectly lost: 0 bytes in 0 blocks ==10769== possibly lost: 0 bytes in 0 blocks ==10769== still reachable: 0 bytes in 0 blocks ==10769== suppressed: 72,704 bytes in 1 blocks ``` Running without the suppressions file under Ubuntu 16.04: ``` $ valgrind --leak-check=full --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite --run_test=wallet_crypto … ==10724== 72,704 bytes in 1 blocks are still reachable in loss record 1 of 1 ==10724== at 0x4C2DBF6: malloc (vg_replace_malloc.c:299) ==10724== by 0x6F74EFF: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21) ==10724== by 0x40106B9: call_init.part.0 (dl-init.c:72) ==10724== by 0x40107CA: call_init (dl-init.c:30) ==10724== by 0x40107CA: _dl_init (dl-init.c:120) ==10724== by 0x4000C69: ??? (in /lib/x86_64-linux-gnu/ld-2.23.so) ==10724== by 0x2: ??? ==10724== by 0x1FFF0006D2: ??? ==10724== by 0x1FFF0006E8: ??? ==10724== by 0x1FFF0006FF: ??? ==10724== ==10724== LEAK SUMMARY: ==10724== definitely lost: 0 bytes in 0 blocks ==10724== indirectly lost: 0 bytes in 0 blocks ==10724== possibly lost: 0 bytes in 0 blocks ==10724== still reachable: 72,704 bytes in 1 blocks ==10724== suppressed: 0 bytes in 0 blocks ``` Tree-SHA512: 9c92079fc61313ea678deb6aaa16a3a71c3154c757459793eb9ca0d90a9a74c6faebfb04c9135e1b398ca34224fb7f03bd9c488ea0e8debf6894f69f030a31d3
2017-11-13 14:59:29 +01:00
which includes known Valgrind warnings in our dependencies that cannot be fixed
in-tree. Example use:
```shell
$ valgrind --suppressions=contrib/valgrind.supp src/test/test_dash
Merge #11035: [contrib] Add Valgrind suppressions file 4a426d8 Add note about Valgrind suppressions file in developer-notes.md (practicalswift) 84e2462 contrib: Add Valgrind suppressions file (practicalswift) Pull request description: Includes known Valgrind warnings in our dependencies that cannot be fixed in-tree. Example use: ``` $ valgrind --suppressions=contrib/valgrind.supp src/test/test_bitcoin $ valgrind --suppressions=contrib/valgrind.supp --leak-check=full \ --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite ``` Running with the suppressions file under Ubuntu 16.04: ``` $ valgrind --suppressions=contrib/valgrind.supp --leak-check=full --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite --run_test=wallet_crypto … ==10769== LEAK SUMMARY: ==10769== definitely lost: 0 bytes in 0 blocks ==10769== indirectly lost: 0 bytes in 0 blocks ==10769== possibly lost: 0 bytes in 0 blocks ==10769== still reachable: 0 bytes in 0 blocks ==10769== suppressed: 72,704 bytes in 1 blocks ``` Running without the suppressions file under Ubuntu 16.04: ``` $ valgrind --leak-check=full --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite --run_test=wallet_crypto … ==10724== 72,704 bytes in 1 blocks are still reachable in loss record 1 of 1 ==10724== at 0x4C2DBF6: malloc (vg_replace_malloc.c:299) ==10724== by 0x6F74EFF: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21) ==10724== by 0x40106B9: call_init.part.0 (dl-init.c:72) ==10724== by 0x40107CA: call_init (dl-init.c:30) ==10724== by 0x40107CA: _dl_init (dl-init.c:120) ==10724== by 0x4000C69: ??? (in /lib/x86_64-linux-gnu/ld-2.23.so) ==10724== by 0x2: ??? ==10724== by 0x1FFF0006D2: ??? ==10724== by 0x1FFF0006E8: ??? ==10724== by 0x1FFF0006FF: ??? ==10724== ==10724== LEAK SUMMARY: ==10724== definitely lost: 0 bytes in 0 blocks ==10724== indirectly lost: 0 bytes in 0 blocks ==10724== possibly lost: 0 bytes in 0 blocks ==10724== still reachable: 72,704 bytes in 1 blocks ==10724== suppressed: 0 bytes in 0 blocks ``` Tree-SHA512: 9c92079fc61313ea678deb6aaa16a3a71c3154c757459793eb9ca0d90a9a74c6faebfb04c9135e1b398ca34224fb7f03bd9c488ea0e8debf6894f69f030a31d3
2017-11-13 14:59:29 +01:00
$ valgrind --suppressions=contrib/valgrind.supp --leak-check=full \
--show-leak-kinds=all src/test/test_dash --log_level=test_suite
$ valgrind -v --leak-check=full src/dashd -printtoconsole
$ ./test/functional/test_runner.py --valgrind
Merge #11035: [contrib] Add Valgrind suppressions file 4a426d8 Add note about Valgrind suppressions file in developer-notes.md (practicalswift) 84e2462 contrib: Add Valgrind suppressions file (practicalswift) Pull request description: Includes known Valgrind warnings in our dependencies that cannot be fixed in-tree. Example use: ``` $ valgrind --suppressions=contrib/valgrind.supp src/test/test_bitcoin $ valgrind --suppressions=contrib/valgrind.supp --leak-check=full \ --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite ``` Running with the suppressions file under Ubuntu 16.04: ``` $ valgrind --suppressions=contrib/valgrind.supp --leak-check=full --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite --run_test=wallet_crypto … ==10769== LEAK SUMMARY: ==10769== definitely lost: 0 bytes in 0 blocks ==10769== indirectly lost: 0 bytes in 0 blocks ==10769== possibly lost: 0 bytes in 0 blocks ==10769== still reachable: 0 bytes in 0 blocks ==10769== suppressed: 72,704 bytes in 1 blocks ``` Running without the suppressions file under Ubuntu 16.04: ``` $ valgrind --leak-check=full --show-leak-kinds=all src/test/test_bitcoin --log_level=test_suite --run_test=wallet_crypto … ==10724== 72,704 bytes in 1 blocks are still reachable in loss record 1 of 1 ==10724== at 0x4C2DBF6: malloc (vg_replace_malloc.c:299) ==10724== by 0x6F74EFF: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21) ==10724== by 0x40106B9: call_init.part.0 (dl-init.c:72) ==10724== by 0x40107CA: call_init (dl-init.c:30) ==10724== by 0x40107CA: _dl_init (dl-init.c:120) ==10724== by 0x4000C69: ??? (in /lib/x86_64-linux-gnu/ld-2.23.so) ==10724== by 0x2: ??? ==10724== by 0x1FFF0006D2: ??? ==10724== by 0x1FFF0006E8: ??? ==10724== by 0x1FFF0006FF: ??? ==10724== ==10724== LEAK SUMMARY: ==10724== definitely lost: 0 bytes in 0 blocks ==10724== indirectly lost: 0 bytes in 0 blocks ==10724== possibly lost: 0 bytes in 0 blocks ==10724== still reachable: 72,704 bytes in 1 blocks ==10724== suppressed: 0 bytes in 0 blocks ``` Tree-SHA512: 9c92079fc61313ea678deb6aaa16a3a71c3154c757459793eb9ca0d90a9a74c6faebfb04c9135e1b398ca34224fb7f03bd9c488ea0e8debf6894f69f030a31d3
2017-11-13 14:59:29 +01:00
```
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
### Compiling for test coverage
LCOV can be used to generate a test coverage report based upon `make check`
execution. LCOV must be installed on your system (e.g. the `lcov` package
on Debian/Ubuntu).
To enable LCOV report generation during test runs:
```shell
./configure --enable-lcov
make
make cov
# A coverage report will now be accessible at `./test_dash.coverage/index.html`.
```
Merge #14519: tests: add utility to easily profile node performance with perf 13782b8ba8 docs: add perf section to developer docs (James O'Beirne) 58180b5fd4 tests: add utility to easily profile node performance with perf (James O'Beirne) Pull request description: Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`. While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code. `perf` usage is detailed in the excellent (and sadly unmerged) https://github.com/bitcoin/bitcoin/pull/12649; all due props to @eklitzke. ### Example ```python with node.profile_with_perf("large-msgs"): for i in range(200): node.p2p.send_message(some_large_msg) node.p2p.sync_with_ping() ``` This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`). Running `perf report` generates nice output about where the node spent most of its time while running that part of the test: ```bash $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \ | c++filt \ | less # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 135 of event 'cycles:pp' # Event count (approx.): 1458205679493582 # # Children Self Command Shared Object Symbol # ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................ # 70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) | ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int) | ---CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) | ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) ... ``` Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e
2019-02-05 23:40:11 +01:00
### Performance profiling with perf
Profiling is a good way to get a precise idea of where time is being spent in
code. One tool for doing profiling on Linux platforms is called
[`perf`](https://www.brendangregg.com/perf.html), and has been integrated into
Merge #14519: tests: add utility to easily profile node performance with perf 13782b8ba8 docs: add perf section to developer docs (James O'Beirne) 58180b5fd4 tests: add utility to easily profile node performance with perf (James O'Beirne) Pull request description: Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`. While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code. `perf` usage is detailed in the excellent (and sadly unmerged) https://github.com/bitcoin/bitcoin/pull/12649; all due props to @eklitzke. ### Example ```python with node.profile_with_perf("large-msgs"): for i in range(200): node.p2p.send_message(some_large_msg) node.p2p.sync_with_ping() ``` This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`). Running `perf report` generates nice output about where the node spent most of its time while running that part of the test: ```bash $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \ | c++filt \ | less # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 135 of event 'cycles:pp' # Event count (approx.): 1458205679493582 # # Children Self Command Shared Object Symbol # ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................ # 70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) | ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int) | ---CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) | ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) ... ``` Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e
2019-02-05 23:40:11 +01:00
the functional test framework. Perf can observe a running process and sample
(at some frequency) where its execution is.
Perf installation is contingent on which kernel version you're running; see
[this thread](https://askubuntu.com/questions/50145/how-to-install-perf-monitoring-tool)
Merge #14519: tests: add utility to easily profile node performance with perf 13782b8ba8 docs: add perf section to developer docs (James O'Beirne) 58180b5fd4 tests: add utility to easily profile node performance with perf (James O'Beirne) Pull request description: Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`. While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code. `perf` usage is detailed in the excellent (and sadly unmerged) https://github.com/bitcoin/bitcoin/pull/12649; all due props to @eklitzke. ### Example ```python with node.profile_with_perf("large-msgs"): for i in range(200): node.p2p.send_message(some_large_msg) node.p2p.sync_with_ping() ``` This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`). Running `perf report` generates nice output about where the node spent most of its time while running that part of the test: ```bash $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \ | c++filt \ | less # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 135 of event 'cycles:pp' # Event count (approx.): 1458205679493582 # # Children Self Command Shared Object Symbol # ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................ # 70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) | ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int) | ---CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) | ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) ... ``` Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e
2019-02-05 23:40:11 +01:00
for specific instructions.
Certain kernel parameters may need to be set for perf to be able to inspect the
running process's stack.
Merge #14519: tests: add utility to easily profile node performance with perf 13782b8ba8 docs: add perf section to developer docs (James O'Beirne) 58180b5fd4 tests: add utility to easily profile node performance with perf (James O'Beirne) Pull request description: Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`. While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code. `perf` usage is detailed in the excellent (and sadly unmerged) https://github.com/bitcoin/bitcoin/pull/12649; all due props to @eklitzke. ### Example ```python with node.profile_with_perf("large-msgs"): for i in range(200): node.p2p.send_message(some_large_msg) node.p2p.sync_with_ping() ``` This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`). Running `perf report` generates nice output about where the node spent most of its time while running that part of the test: ```bash $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \ | c++filt \ | less # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 135 of event 'cycles:pp' # Event count (approx.): 1458205679493582 # # Children Self Command Shared Object Symbol # ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................ # 70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) | ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int) | ---CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) | ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) ... ``` Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e
2019-02-05 23:40:11 +01:00
```sh
$ sudo sysctl -w kernel.perf_event_paranoid=-1
$ sudo sysctl -w kernel.kptr_restrict=0
```
Make sure you [understand the security
trade-offs](https://lwn.net/Articles/420403/) of setting these kernel
parameters.
To profile a running dashd process for 60 seconds, you could use an
invocation of `perf record` like this:
```sh
$ perf record \
-g --call-graph dwarf --per-thread -F 140 \
-p `pgrep dashd` -- sleep 60
```
You could then analyze the results by running:
Merge #14519: tests: add utility to easily profile node performance with perf 13782b8ba8 docs: add perf section to developer docs (James O'Beirne) 58180b5fd4 tests: add utility to easily profile node performance with perf (James O'Beirne) Pull request description: Adds a context manager to easily (and selectively) profile node performance during functional test execution using `perf`. While writing some tests, I encountered some odd bitcoind slowness. I wrote up a utility (`TestNode.profile_with_perf`) that generates performance diagnostics for a node by running `perf` during the execution of a particular region of test code. `perf` usage is detailed in the excellent (and sadly unmerged) https://github.com/bitcoin/bitcoin/pull/12649; all due props to @eklitzke. ### Example ```python with node.profile_with_perf("large-msgs"): for i in range(200): node.p2p.send_message(some_large_msg) node.p2p.sync_with_ping() ``` This generates a perf data file in the test node's datadir (`/tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data`). Running `perf report` generates nice output about where the node spent most of its time while running that part of the test: ```bash $ perf report -i /tmp/testtxmpod0y/node0/node-0-TestName-large-msgs.perf.data --stdio \ | c++filt \ | less # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 135 of event 'cycles:pp' # Event count (approx.): 1458205679493582 # # Children Self Command Shared Object Symbol # ........ ........ ............... ................... ........................................................................................................................................................................................................................................................................ # 70.14% 0.00% bitcoin-net bitcoind [.] CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) | ---CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 70.14% 0.00% bitcoin-net bitcoind [.] CNetMessage::readData(char const*, unsigned int) | ---CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) 35.52% 0.00% bitcoin-net bitcoind [.] std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) | ---std::vector<char, zero_after_free_allocator<char> >::_M_fill_insert(__gnu_cxx::__normal_iterator<char*, std::vector<char, zero_after_free_allocator<char> > >, unsigned long, char const&) CNetMessage::readData(char const*, unsigned int) CNode::ReceiveMsgBytes(char const*, unsigned int, bool&) ... ``` Tree-SHA512: 9ac4ceaa88818d5eca00994e8e3c8ad42ae019550d6583972a0a4f7b0c4f61032e3d0c476b4ae58756bc5eb8f8015a19a7fc26c095bd588f31d49a37ed0c6b3e
2019-02-05 23:40:11 +01:00
```sh
perf report --stdio | c++filt | less
```
or using a graphical tool like [Hotspot](https://github.com/KDAB/hotspot).
See the functional test documentation for how to invoke perf within tests.
### Sanitizers
Dash Core can be compiled with various "sanitizers" enabled, which add
instrumentation for issues regarding things like memory safety, thread race
conditions, or undefined behavior. This is controlled with the
`--with-sanitizers` configure flag, which should be a comma separated list of
sanitizers to enable. The sanitizer list should correspond to supported
`-fsanitize=` options in your compiler. These sanitizers have runtime overhead,
so they are most useful when testing changes or producing debugging builds.
Some examples:
```bash
# Enable both the address sanitizer and the undefined behavior sanitizer
./configure --with-sanitizers=address,undefined
# Enable the thread sanitizer
./configure --with-sanitizers=thread
```
If you are compiling with GCC you will typically need to install corresponding
"san" libraries to actually compile with these flags, e.g. libasan for the
address sanitizer, libtsan for the thread sanitizer, and libubsan for the
undefined sanitizer. If you are missing required libraries, the configure script
will fail with a linker error when testing the sanitizer flags.
The test suite should pass cleanly with the `thread` and `undefined` sanitizers,
but there are a number of known problems when using the `address` sanitizer. The
address sanitizer is known to fail in
[sha256_sse4::Transform](/src/crypto/sha256_sse4.cpp) which makes it unusable
unless you also use `--disable-asm` when running configure. We would like to fix
sanitizer issues, so please send pull requests if you can fix any errors found
by the address sanitizer (or any other sanitizer).
Not all sanitizer options can be enabled at the same time, e.g. trying to build
with `--with-sanitizers=address,thread` will fail in the configure script as
these sanitizers are mutually incompatible. Refer to your compiler manual to
learn more about these options and which sanitizers are supported by your
compiler.
Additional resources:
* [AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html)
* [LeakSanitizer](https://clang.llvm.org/docs/LeakSanitizer.html)
* [MemorySanitizer](https://clang.llvm.org/docs/MemorySanitizer.html)
* [ThreadSanitizer](https://clang.llvm.org/docs/ThreadSanitizer.html)
* [UndefinedBehaviorSanitizer](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html)
* [GCC Instrumentation Options](https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html)
* [Google Sanitizers Wiki](https://github.com/google/sanitizers/wiki)
* [Issue #12691: Enable -fsanitize flags in Travis](https://github.com/bitcoin/bitcoin/issues/12691)
2013-05-20 06:30:00 +02:00
Locking/mutex usage notes
-------------------------
2013-05-20 06:30:00 +02:00
The code is multi-threaded and uses mutexes and the
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
`LOCK` and `TRY_LOCK` macros to protect data structures.
2013-05-20 06:30:00 +02:00
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
Deadlocks due to inconsistent lock ordering (thread 1 locks `cs_main` and then
`cs_wallet`, while thread 2 locks them in the opposite order: result, deadlock
as each waits for the other to release its lock) are a problem. Compile with
`-DDEBUG_LOCKORDER` (or use `--enable-debug`) to get lock order inconsistencies
reported in the `debug.log` file.
2013-05-20 06:30:00 +02:00
Re-architecting the core code so there are better-defined interfaces
between the various components is a goal, with any necessary locking
done by the components (e.g. see the self-contained `FillableSigningProvider` class
Merge #12759: [Docs] Improve formatting of developer notes 0bd2ec5 Improve formatting of developer notes (Evan Klitzke) Pull request description: The developer notes file has gotten pretty large and unwieldly. This reorganizes some content, most notably by adding a TOC to the page. Compare how the page looks in [my branch](https://github.com/eklitzke/bitcoin/blob/developer-notes/doc/developer-notes.md) vs [master](https://github.com/eklitzke/bitcoin/blob/master/doc/developer-notes.md). I know there's long-term interest in moving a lot of this content to the bitcoin-core/docs repo on GitHub, but this makes things better now. The TOC format here is a semi-standard extension to Markdown files that you may have seen on GitHub pages in other projects. The `<!-- markdown-toc -->` comments used by these tools to know where the TOC starts/ends. The following tools all understand this format: * [Sphinx](http://www.sphinx-doc.org/en/master/) * [Pandoc](https://pandoc.org/) * [markdown-toc.el](https://github.com/ardumont/markdown-toc) (what I used) * various plugins for [vim](https://github.com/mzlogin/vim-markdown-toc) and other editors I used this TOC extension at a previous job, and my observation is that it's fine if people just edit the TOC manually. It's just text and it's not a huge deal if it gets a little out of sync. It can also be regenerated at any time by anyone with any of these tools. Tree-SHA512: 298d1605ea5e8bfc0f75e70570c23ebd6891e4ffcdedd24fefadc23edd6e4b96509d8d102209868468a1b3ddbe2c3b8462698cdda8b9421348b5bc6f7b8d0cb8
2018-03-29 15:37:20 +02:00
and its `cs_KeyStore` lock for example).
2013-05-20 06:30:00 +02:00
Threads
-------
2013-05-20 06:30:00 +02:00
- [Main thread (`dashd`)](https://doxygen.bitcoincore.org/bitcoind_8cpp.html#a0ddf1224851353fc92bfbff6f499fa97)
: Started from `main()` in `bitcoind.cpp`. Responsible for starting up and
shutting down the application.
- [ThreadImport (`d-loadblk`)](https://doxygen.bitcoincore.org/init_8cpp.html#ae9e290a0e829ec0198518de2eda579d1)
: Loads blocks from `blk*.dat` files or `-loadblock=<file>` on startup.
- [ThreadScriptCheck (`d-scriptch.x`)](https://doxygen.bitcoincore.org/validation_8cpp.html#a925a33e7952a157922b0bbb8dab29a20)
: Parallel script validation threads for transactions in blocks.
- [ThreadHTTP (`d-http`)](https://doxygen.bitcoincore.org/httpserver_8cpp.html#abb9f6ea8819672bd9a62d3695070709c)
: Libevent thread to listen for RPC and REST connections.
2013-05-20 06:30:00 +02:00
- [HTTP worker threads(`d-httpworker.x`)](https://doxygen.bitcoincore.org/httpserver_8cpp.html#aa6a7bc27265043bc0193220c5ae3a55f)
: Threads to service RPC and REST requests.
- [Indexer threads (`d-txindex`, etc)](https://doxygen.bitcoincore.org/class_base_index.html#a96a7407421fbf877509248bbe64f8d87)
: One thread per indexer.
- [SchedulerThread (`d-scheduler`)](https://doxygen.bitcoincore.org/class_c_scheduler.html#a14d2800815da93577858ea078aed1fba)
: Does asynchronous background tasks like dumping wallet contents, dumping
addrman and running asynchronous validationinterface callbacks.
2013-05-20 06:30:00 +02:00
- [TorControlThread (`d-torcontrol`)](https://doxygen.bitcoincore.org/torcontrol_8cpp.html#a4faed3692d57a0d7bdbecf3b37f72de0)
: Libevent thread for tor connections.
- Net threads:
- [ThreadMessageHandler (`d-msghand`)](https://doxygen.bitcoincore.org/class_c_connman.html#aacdbb7148575a31bb33bc345e2bf22a9)
: Application level message handling (sending and receiving). Almost
all net_processing and validation logic runs on this thread.
2013-05-20 06:30:00 +02:00
- [ThreadDNSAddressSeed (`d-dnsseed`)](https://doxygen.bitcoincore.org/class_c_connman.html#aa7c6970ed98a4a7bafbc071d24897d13)
: Loads addresses of peers from the DNS.
- [ThreadMapPort (`d-upnp`)](https://doxygen.bitcoincore.org/net_8cpp.html#a63f82a71c4169290c2db1651a9bbe249)
: Universal plug-and-play startup/shutdown.
- [ThreadSocketHandler (`d-net`)](https://doxygen.bitcoincore.org/class_c_connman.html#a765597cbfe99c083d8fa3d61bb464e34)
: Sends/Receives data from peers on port 9999.
- [ThreadOpenAddedConnections (`d-addcon`)](https://doxygen.bitcoincore.org/class_c_connman.html#a0b787caf95e52a346a2b31a580d60a62)
: Opens network connections to added nodes.
- [ThreadOpenConnections (`d-opencon`)](https://doxygen.bitcoincore.org/class_c_connman.html#a55e9feafc3bab78e5c9d408c207faa45)
: Initiates new connections to peers.
- ThreadOpenMasternodeConnections (`d-mncon`)
: Opens network connections to masternodes.
- CSigSharesManager::WorkThreadMain (`d-sigshares`)
: Processes pending BLS signature shares.
- CInstantSendManager::WorkThreadMain (`d-isman`)
: Processes pending InstantSend locks.
Thread pools
------------
- CBLSWorker : A highly parallelized worker/helper for BLS/DKG calculations.
- CDKGSessionManager : A thread pool for processing LLMQ messages.
Ignoring IDE/editor files
--------------------------
In closed-source environments in which everyone uses the same IDE, it is common
to add temporary files it produces to the project-wide `.gitignore` file.
2016-03-06 16:26:01 +01:00
However, in open source software such as Dash Core, where everyone uses
their own editors/IDE/tools, it is less common. Only you know what files your
editor produces and this may change from version to version. The canonical way
to do this is thus to create your local gitignore. Add this to `~/.gitconfig`:
```
[core]
excludesfile = /home/.../.gitignore_global
```
(alternatively, type the command `git config --global core.excludesfile ~/.gitignore_global`
on a terminal)
Then put your favourite tool's temporary filenames in that file, e.g.
```
# NetBeans
nbproject/
```
Another option is to create a per-repository excludes file `.git/info/exclude`.
These are not committed but apply only to one repository.
If a set of tools is used by the build system or scripts the repository (for
example, lcov) it is perfectly acceptable to add its files to `.gitignore`
and commit them.
Development guidelines
============================
A few non-style-related recommendations for developers, as well as points to
2016-03-06 16:26:01 +01:00
pay attention to for reviewers of Dash Core code.
2016-03-06 16:26:01 +01:00
General Dash Core
----------------------
- New features should be exposed on RPC first, then can be made available in the GUI.
- *Rationale*: RPC allows for better automatic testing. The test suite for
the GUI is very limited.
- Make sure pull requests pass CI before merging.
- *Rationale*: Makes sure that they pass thorough testing, and that the tester will keep passing
on the master branch. Otherwise, all new pull requests will start failing the tests, resulting in
confusion and mayhem.
- *Explanation*: If the test suite is to be updated for a change, this has to
be done first.
Wallet
-------
2015-11-23 21:11:53 +01:00
- Make sure that no crashes happen with run-time option `-disablewallet`.
- Include `db_cxx.h` (BerkeleyDB header) only when `ENABLE_WALLET` is set.
- *Rationale*: Otherwise compilation of the disable-wallet build will fail in environments without BerkeleyDB.
General C++
-------------
For general C++ guidelines, you may refer to the [C++ Core
Guidelines](https://isocpp.github.io/CppCoreGuidelines/).
Common misconceptions are clarified in those sections:
- Passing (non-)fundamental types in the [C++ Core
Guideline](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Rf-conventional).
- Assertions should not have side-effects.
- *Rationale*: Even though the source code is set to refuse to compile
with assertions disabled, having side-effects in assertions is unexpected and
makes the code harder to understand.
- If you use the `.h`, you must link the `.cpp`.
2015-11-23 21:11:53 +01:00
- *Rationale*: Include files define the interface for the code in implementation files. Including one but
not linking the other is confusing. Please avoid that. Moving functions from
the `.h` to the `.cpp` should not result in build errors.
- Use the RAII (Resource Acquisition Is Initialization) paradigm where possible. For example, by using
`unique_ptr` for allocations in a function.
- *Rationale*: This avoids memory and resource leaks, and ensures exception safety.
C++ data structures
--------------------
- Never use the `std::map []` syntax when reading from a map, but instead use `.find()`.
2015-11-23 21:11:53 +01:00
- *Rationale*: `[]` does an insert (of the default element) if the item doesn't
exist in the map yet. This has resulted in memory leaks in the past, as well as
race conditions (expecting read-read behavior). Using `[]` is fine for *writing* to a map.
- Do not compare an iterator from one data structure with an iterator of
another data structure (even if of the same type).
- *Rationale*: Behavior is undefined. In C++ parlor this means "may reformat
the universe", in practice this has resulted in at least one hard-to-debug crash bug.
- Watch out for out-of-bounds vector access. `&vch[vch.size()]` is illegal,
including `&vch[0]` for an empty vector. Use `vch.data()` and `vch.data() +
vch.size()` instead.
- Vector bounds checking is only enabled in debug mode. Do not rely on it.
- Initialize all non-static class members where they are defined.
If this is skipped for a good reason (i.e., optimization on the critical
path), add an explicit comment about this.
- *Rationale*: Ensure determinism by avoiding accidental use of uninitialized
values. Also, static analyzers balk about this.
Initializing the members in the declaration makes it easy to
spot uninitialized ones.
```cpp
class A
{
uint32_t m_count{0};
}
```
- By default, declare constructors `explicit`.
- *Rationale*: This is a precaution to avoid unintended
[conversions](https://en.cppreference.com/w/cpp/language/converting_constructor).
- Use explicitly signed or unsigned `char`s, or even better `uint8_t` and
`int8_t`. Do not use bare `char` unless it is to pass to a third-party API.
This type can be signed or unsigned depending on the architecture, which can
lead to interoperability problems or dangerous conditions such as
out-of-bounds array accesses.
- Prefer explicit constructions over implicit ones that rely on 'magical' C++ behavior.
- *Rationale*: Easier to understand what is happening, thus easier to spot mistakes, even for those
that are not language lawyers.
- Prefer signed ints and do not mix signed and unsigned integers. If an unsigned int is used, it should have a good
reason. The fact a value will never be negative is not a good reason. The most common reason will be that mod two
arithmetic is needed, such as in cryptographic primitives. If you need to make sure that some value is always
a non-negative one, use an assertion or exception instead.
- *Rationale*: When signed ints are mixed with unsigned ints, the signed int is converted to a unsigned
int. If the signed int is some negative `N`, it'll become `INT_MAX - N` which might cause unexpected consequences.
- Use `Span` as function argument when it can operate on any range-like container.
- *Rationale*: Compared to `Foo(const vector<int>&)` this avoids the need for a (potentially expensive)
conversion to vector if the caller happens to have the input stored in another type of container.
However, be aware of the pitfalls documented in [span.h](../src/span.h).
```cpp
void Foo(Span<const int> data);
std::vector<int> vec{1,2,3};
Foo(vec);
```
- Prefer `enum class` (scoped enumerations) over `enum` (traditional enumerations) where possible.
- *Rationale*: Scoped enumerations avoid two potential pitfalls/problems with traditional C++ enumerations: implicit conversions to `int`, and name clashes due to enumerators being exported to the surrounding scope.
- `switch` statement on an enumeration example:
```cpp
enum class Tabs {
info,
console,
network_graph,
peers
};
int GetInt(Tabs tab)
{
switch (tab) {
case Tabs::info: return 0;
case Tabs::console: return 1;
case Tabs::network_graph: return 2;
case Tabs::peers: return 3;
} // no default case, so the compiler can warn about missing cases
assert(false);
}
```
*Rationale*: The comment documents skipping `default:` label, and it complies with `clang-format` rules. The assertion prevents firing of `-Wreturn-type` warning on some compilers.
Strings and formatting
------------------------
2015-11-23 21:11:53 +01:00
- Be careful of `LogPrint` versus `LogPrintf`. `LogPrint` takes a `category` argument, `LogPrintf` does not.
- *Rationale*: Confusion of these can result in runtime exceptions due to
formatting mismatch, and it is easy to get wrong because of subtly similar naming.
- Use `std::string`, avoid C string manipulation functions.
- *Rationale*: C++ string handling is marginally safer, less scope for
buffer overflows, and surprises with `\0` characters. Also, some C string manipulations
tend to act differently depending on platform, or even the user locale.
- Use `ParseInt32`, `ParseInt64`, `ParseUInt32`, `ParseUInt64`, `ParseDouble` from `utilstrencodings.h` for number parsing.
- *Rationale*: These functions do overflow checking and avoid pesky locale issues.
- Avoid using locale dependent functions if possible. You can use the provided
[`lint-locale-dependence.sh`](/test/lint/lint-locale-dependence.sh)
to check for accidental use of locale dependent functions.
- *Rationale*: Unnecessary locale dependence can cause bugs that are very tricky to isolate and fix.
- These functions are known to be locale dependent:
`alphasort`, `asctime`, `asprintf`, `atof`, `atoi`, `atol`, `atoll`, `atoq`,
`btowc`, `ctime`, `dprintf`, `fgetwc`, `fgetws`, `fprintf`, `fputwc`,
`fputws`, `fscanf`, `fwprintf`, `getdate`, `getwc`, `getwchar`, `isalnum`,
`isalpha`, `isblank`, `iscntrl`, `isdigit`, `isgraph`, `islower`, `isprint`,
`ispunct`, `isspace`, `isupper`, `iswalnum`, `iswalpha`, `iswblank`,
`iswcntrl`, `iswctype`, `iswdigit`, `iswgraph`, `iswlower`, `iswprint`,
`iswpunct`, `iswspace`, `iswupper`, `iswxdigit`, `isxdigit`, `mblen`,
`mbrlen`, `mbrtowc`, `mbsinit`, `mbsnrtowcs`, `mbsrtowcs`, `mbstowcs`,
`mbtowc`, `mktime`, `putwc`, `putwchar`, `scanf`, `snprintf`, `sprintf`,
`sscanf`, `stoi`, `stol`, `stoll`, `strcasecmp`, `strcasestr`, `strcoll`,
`strfmon`, `strftime`, `strncasecmp`, `strptime`, `strtod`, `strtof`,
`strtoimax`, `strtol`, `strtold`, `strtoll`, `strtoq`, `strtoul`,
`strtoull`, `strtoumax`, `strtouq`, `strxfrm`, `swprintf`, `tolower`,
`toupper`, `towctrans`, `towlower`, `towupper`, `ungetwc`, `vasprintf`,
`vdprintf`, `versionsort`, `vfprintf`, `vfscanf`, `vfwprintf`, `vprintf`,
`vscanf`, `vsnprintf`, `vsprintf`, `vsscanf`, `vswprintf`, `vwprintf`,
`wcrtomb`, `wcscasecmp`, `wcscoll`, `wcsftime`, `wcsncasecmp`, `wcsnrtombs`,
`wcsrtombs`, `wcstod`, `wcstof`, `wcstoimax`, `wcstol`, `wcstold`,
`wcstoll`, `wcstombs`, `wcstoul`, `wcstoull`, `wcstoumax`, `wcswidth`,
`wcsxfrm`, `wctob`, `wctomb`, `wctrans`, `wctype`, `wcwidth`, `wprintf`
- For `strprintf`, `LogPrint`, `LogPrintf` formatting characters don't need size specifiers.
- *Rationale*: Dash Core uses tinyformat, which is type safe. Leave them out to avoid confusion.
- Use `.c_str()` sparingly. Its only valid use is to pass C++ strings to C functions that take NULL-terminated
strings.
- Do not use it when passing a sized array (so along with `.size()`). Use `.data()` instead to get a pointer
to the raw data.
- *Rationale*: Although this is guaranteed to be safe starting with C++11, `.data()` communicates the intent better.
- Do not use it when passing strings to `tfm::format`, `strprintf`, `LogPrint[f]`.
- *Rationale*: This is redundant. Tinyformat handles strings.
- Do not use it to convert to `QString`. Use `QString::fromStdString()`.
- *Rationale*: Qt has built-in functionality for converting their string
type from/to C++. No need to roll your own.
- In cases where do you call `.c_str()`, you might want to additionally check that the string does not contain embedded '\0' characters, because
it will (necessarily) truncate the string. This might be used to hide parts of the string from logging or to circumvent
checks. If a use of strings is sensitive to this, take care to check the string for embedded NULL characters first
and reject it if there are any (see `ParsePrechecks` in `strencodings.cpp` for an example).
Shadowing
--------------
Although the shadowing warning (`-Wshadow`) is not enabled by default (it prevents issues arising
from using a different variable with the same name),
please name variables so that their names do not shadow variables defined in the source code.
When using nested cycles, do not name the inner cycle variable the same as in
the upper cycle, etc.
Threads and synchronization
----------------------------
Merge #19668: Do not hide compile-time thread safety warnings ea74e10acf17903e44c85e3678853414653dd4e1 doc: Add best practice for annotating/asserting locks (Hennadii Stepanov) 2ee7743fe723227f2ea1b031eddb14fc6863f4c8 sync.h: Make runtime lock checks require compile-time lock checks (Anthony Towns) 23d71d171e6e22ba5e4a909d597a54595b2a2c1f Do not hide compile-time thread safety warnings (Hennadii Stepanov) 3ddc150857178bfb1c854c05bf9b526777876f56 Add missed thread safety annotations (Hennadii Stepanov) af9ea55a72c94678b343f5dd98dc78f3a3ac58cb Use LockAssertion utility class instead of AssertLockHeld() (Hennadii Stepanov) Pull request description: On the way of transit from `RecursiveMutex` to `Mutex` (see #19303) it is crucial to have run-time `AssertLockHeld()` assertion that does _not_ hide compile-time Clang Thread Safety Analysis warnings. On master (65e4ecabd5b4252154640c7bac38c92a3f3a7018) using `AssertLockHeld()` could hide Clang Thread Safety Analysis warnings, e.g., with the following patch applied: ```diff --- a/src/txmempool.h +++ b/src/txmempool.h @@ -607,7 +607,7 @@ public: void addUnchecked(const CTxMemPoolEntry& entry, setEntries& setAncestors, bool validFeeEstimate = true) EXCLUSIVE_LOCKS_REQUIRED(cs, cs_main); void removeRecursive(const CTransaction& tx, MemPoolRemovalReason reason) EXCLUSIVE_LOCKS_REQUIRED(cs); - void removeForReorg(const CCoinsViewCache* pcoins, unsigned int nMemPoolHeight, int flags) EXCLUSIVE_LOCKS_REQUIRED(cs, cs_main); + void removeForReorg(const CCoinsViewCache* pcoins, unsigned int nMemPoolHeight, int flags) EXCLUSIVE_LOCKS_REQUIRED(cs_main); void removeConflicts(const CTransaction& tx) EXCLUSIVE_LOCKS_REQUIRED(cs); void removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int nBlockHeight) EXCLUSIVE_LOCKS_REQUIRED(cs); ``` Clang compiles the code without any thread safety warnings. See "Add missed thread safety annotations" commit for the actual thread safety warnings that are fixed in this PR. ACKs for top commit: MarcoFalke: ACK ea74e10acf 🎙 jnewbery: ACK ea74e10acf17903e44c85e3678853414653dd4e1 ajtowns: ACK ea74e10acf17903e44c85e3678853414653dd4e1 Tree-SHA512: 8cba996e526751a1cb0e613c0cc1b10f027a3e9945fbfb4bd30f6355fd36b9f9c2e1e95ed3183fc254b42df7c30223278e18e5bdb5e1ef85db7fef067595d447
2020-09-01 08:18:20 +02:00
- Prefer `Mutex` type to `RecursiveMutex` one
- Consistently use [Clang Thread Safety Analysis](https://clang.llvm.org/docs/ThreadSafetyAnalysis.html) annotations to
get compile-time warnings about potential race conditions in code. Combine annotations in function declarations with
run-time asserts in function definitions:
```C++
// txmempool.h
class CTxMemPool
{
public:
...
mutable RecursiveMutex cs;
...
void UpdateTransactionsFromBlock(...) EXCLUSIVE_LOCKS_REQUIRED(::cs_main, cs);
...
}
// txmempool.cpp
void CTxMemPool::UpdateTransactionsFromBlock(...)
{
AssertLockHeld(::cs_main);
AssertLockHeld(cs);
...
}
```
```C++
// validation.h
class ChainstateManager
{
public:
...
bool ProcessNewBlock(...) LOCKS_EXCLUDED(::cs_main);
Merge #19668: Do not hide compile-time thread safety warnings ea74e10acf17903e44c85e3678853414653dd4e1 doc: Add best practice for annotating/asserting locks (Hennadii Stepanov) 2ee7743fe723227f2ea1b031eddb14fc6863f4c8 sync.h: Make runtime lock checks require compile-time lock checks (Anthony Towns) 23d71d171e6e22ba5e4a909d597a54595b2a2c1f Do not hide compile-time thread safety warnings (Hennadii Stepanov) 3ddc150857178bfb1c854c05bf9b526777876f56 Add missed thread safety annotations (Hennadii Stepanov) af9ea55a72c94678b343f5dd98dc78f3a3ac58cb Use LockAssertion utility class instead of AssertLockHeld() (Hennadii Stepanov) Pull request description: On the way of transit from `RecursiveMutex` to `Mutex` (see #19303) it is crucial to have run-time `AssertLockHeld()` assertion that does _not_ hide compile-time Clang Thread Safety Analysis warnings. On master (65e4ecabd5b4252154640c7bac38c92a3f3a7018) using `AssertLockHeld()` could hide Clang Thread Safety Analysis warnings, e.g., with the following patch applied: ```diff --- a/src/txmempool.h +++ b/src/txmempool.h @@ -607,7 +607,7 @@ public: void addUnchecked(const CTxMemPoolEntry& entry, setEntries& setAncestors, bool validFeeEstimate = true) EXCLUSIVE_LOCKS_REQUIRED(cs, cs_main); void removeRecursive(const CTransaction& tx, MemPoolRemovalReason reason) EXCLUSIVE_LOCKS_REQUIRED(cs); - void removeForReorg(const CCoinsViewCache* pcoins, unsigned int nMemPoolHeight, int flags) EXCLUSIVE_LOCKS_REQUIRED(cs, cs_main); + void removeForReorg(const CCoinsViewCache* pcoins, unsigned int nMemPoolHeight, int flags) EXCLUSIVE_LOCKS_REQUIRED(cs_main); void removeConflicts(const CTransaction& tx) EXCLUSIVE_LOCKS_REQUIRED(cs); void removeForBlock(const std::vector<CTransactionRef>& vtx, unsigned int nBlockHeight) EXCLUSIVE_LOCKS_REQUIRED(cs); ``` Clang compiles the code without any thread safety warnings. See "Add missed thread safety annotations" commit for the actual thread safety warnings that are fixed in this PR. ACKs for top commit: MarcoFalke: ACK ea74e10acf 🎙 jnewbery: ACK ea74e10acf17903e44c85e3678853414653dd4e1 ajtowns: ACK ea74e10acf17903e44c85e3678853414653dd4e1 Tree-SHA512: 8cba996e526751a1cb0e613c0cc1b10f027a3e9945fbfb4bd30f6355fd36b9f9c2e1e95ed3183fc254b42df7c30223278e18e5bdb5e1ef85db7fef067595d447
2020-09-01 08:18:20 +02:00
...
}
// validation.cpp
bool ChainstateManager::ProcessNewBlock(...)
{
AssertLockNotHeld(::cs_main);
...
LOCK(::cs_main);
...
}
```
- Build and run tests with `-DDEBUG_LOCKORDER` to verify that no potential
deadlocks are introduced.
- When using `LOCK`/`TRY_LOCK` be aware that the lock exists in the context of
the current scope, so surround the statement and the code that needs the lock
with braces.
OK:
```c++
{
TRY_LOCK(cs_vNodes, lockNodes);
...
}
```
Wrong:
```c++
TRY_LOCK(cs_vNodes, lockNodes);
{
...
}
```
Scripts
--------------------------
### Shebang
- Use `#!/usr/bin/env bash` instead of obsolete `#!/bin/bash`.
- [*Rationale*](https://github.com/dylanaraps/pure-bash-bible#shebang):
`#!/bin/bash` assumes it is always installed to /bin/ which can cause issues;
`#!/usr/bin/env bash` searches the user's PATH to find the bash binary.
OK:
```bash
#!/usr/bin/env bash
```
Wrong:
```bash
#!/bin/bash
```
Source code organization
--------------------------
- Implementation code should go into the `.cpp` file and not the `.h`, unless necessary due to template usage or
when performance due to inlining is critical.
- *Rationale*: Shorter and simpler header files are easier to read and reduce compile time.
- Use only the lowercase alphanumerics (`a-z0-9`), underscore (`_`) and hyphen (`-`) in source code filenames.
- *Rationale*: `grep`:ing and auto-completing filenames is easier when using a consistent
naming pattern. Potential problems when building on case-insensitive filesystems are
avoided when using only lowercase characters in source code filenames.
- Every `.cpp` and `.h` file should `#include` every header file it directly uses classes, functions or other
definitions from, even if those headers are already included indirectly through other headers.
- *Rationale*: Excluding headers because they are already indirectly included results in compilation
failures when those indirect dependencies change. Furthermore, it obscures what the real code
dependencies are.
- Don't import anything into the global namespace (`using namespace ...`). Use
fully specified types such as `std::string`.
- *Rationale*: Avoids symbol conflicts.
- Terminate namespaces with a comment (`// namespace mynamespace`). The comment
should be placed on the same line as the brace closing the namespace, e.g.
```c++
namespace mynamespace {
Merge #12982: Fix inconsistent namespace formatting guidelines cd0e1e91dd Fix inconsistent namespace formatting guidelines (Russell Yanofsky) Pull request description: Suggested formatting for namespaces in the developer guide is currently inconsistent. This commit updates the developer guide and clang-format configuration to consistently put "namespace" and opening/closing braces on the same line. Example: ```c++ namespace boost { namespace signals2 { class connection; } // namespace signals2 } // namespace boost ``` Currently the [Source code organization](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#source-code-organization) section has an example like the one above, but the [Coding style](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#coding-style) section example and description put a newline between the opening "namespace foo" and brace (but oddly no newline between closing namespace and brace). Avoiding newlines before namespace opening braces makes nested declarations less verbose and also avoids asymmetry with closing braces. It's also a common style used in our own and other codebases: * https://google.github.io/styleguide/cppguide.html#Namespaces * https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Coding_Style#Classes * https://llvm.org/docs/CodingStandards.html#namespace-indentation Tree-SHA512: 507259478e1a7f6f96db386dd04eb25aa04294f533503fdd82368cf809c3ceaa20204b2cb6ae65322eb27446e5934c1aa1ccb6240ead12aa06b314af76f68139
2018-04-17 19:22:13 +02:00
...
} // namespace mynamespace
namespace {
Merge #12982: Fix inconsistent namespace formatting guidelines cd0e1e91dd Fix inconsistent namespace formatting guidelines (Russell Yanofsky) Pull request description: Suggested formatting for namespaces in the developer guide is currently inconsistent. This commit updates the developer guide and clang-format configuration to consistently put "namespace" and opening/closing braces on the same line. Example: ```c++ namespace boost { namespace signals2 { class connection; } // namespace signals2 } // namespace boost ``` Currently the [Source code organization](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#source-code-organization) section has an example like the one above, but the [Coding style](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#coding-style) section example and description put a newline between the opening "namespace foo" and brace (but oddly no newline between closing namespace and brace). Avoiding newlines before namespace opening braces makes nested declarations less verbose and also avoids asymmetry with closing braces. It's also a common style used in our own and other codebases: * https://google.github.io/styleguide/cppguide.html#Namespaces * https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Coding_Style#Classes * https://llvm.org/docs/CodingStandards.html#namespace-indentation Tree-SHA512: 507259478e1a7f6f96db386dd04eb25aa04294f533503fdd82368cf809c3ceaa20204b2cb6ae65322eb27446e5934c1aa1ccb6240ead12aa06b314af76f68139
2018-04-17 19:22:13 +02:00
...
} // namespace
```
- *Rationale*: Avoids confusion about the namespace context.
- Use `#include <primitives/transaction.h>` bracket syntax instead of
`#include "primitives/transactions.h"` quote syntax.
Backport 11651 (#3358) * scripted-diff: Replace #include "" with #include <> (ryanofsky) -BEGIN VERIFY SCRIPT- for f in \ src/*.cpp \ src/*.h \ src/bench/*.cpp \ src/bench/*.h \ src/compat/*.cpp \ src/compat/*.h \ src/consensus/*.cpp \ src/consensus/*.h \ src/crypto/*.cpp \ src/crypto/*.h \ src/crypto/ctaes/*.h \ src/policy/*.cpp \ src/policy/*.h \ src/primitives/*.cpp \ src/primitives/*.h \ src/qt/*.cpp \ src/qt/*.h \ src/qt/test/*.cpp \ src/qt/test/*.h \ src/rpc/*.cpp \ src/rpc/*.h \ src/script/*.cpp \ src/script/*.h \ src/support/*.cpp \ src/support/*.h \ src/support/allocators/*.h \ src/test/*.cpp \ src/test/*.h \ src/wallet/*.cpp \ src/wallet/*.h \ src/wallet/test/*.cpp \ src/wallet/test/*.h \ src/zmq/*.cpp \ src/zmq/*.h do base=${f%/*}/ relbase=${base#src/} sed -i "s:#include \"\(.*\)\"\(.*\):if test -e \$base'\\1'; then echo \"#include <\"\$relbase\"\\1>\\2\"; else echo \"#include <\\1>\\2\"; fi:e" $f done -END VERIFY SCRIPT- Signed-off-by: Pasta <pasta@dashboost.org> * scripted-diff: Replace #include "" with #include <> (Dash Specific) -BEGIN VERIFY SCRIPT- for f in \ src/bls/*.cpp \ src/bls/*.h \ src/evo/*.cpp \ src/evo/*.h \ src/governance/*.cpp \ src/governance/*.h \ src/llmq/*.cpp \ src/llmq/*.h \ src/masternode/*.cpp \ src/masternode/*.h \ src/privatesend/*.cpp \ src/privatesend/*.h do base=${f%/*}/ relbase=${base#src/} sed -i "s:#include \"\(.*\)\"\(.*\):if test -e \$base'\\1'; then echo \"#include <\"\$relbase\"\\1>\\2\"; else echo \"#include <\\1>\\2\"; fi:e" $f done -END VERIFY SCRIPT- Signed-off-by: Pasta <pasta@dashboost.org> * build: Remove -I for everything but project root Remove -I from build system for everything but the project root, and built-in dependencies. Signed-off-by: Pasta <pasta@dashboost.org> # Conflicts: # src/Makefile.test.include * qt: refactor: Use absolute include paths in .ui files * qt: refactor: Changes to make include paths absolute This makes all include paths in the GUI absolute. Many changes are involved as every single source file in src/qt/ assumes to be able to use relative includes. Signed-off-by: Pasta <pasta@dashboost.org> # Conflicts: # src/qt/dash.cpp # src/qt/optionsmodel.cpp # src/qt/test/rpcnestedtests.cpp * test: refactor: Use absolute include paths for test data files * Recommend #include<> syntax in developer notes * refactor: Include obj/build.h instead of build.h * END BACKPORT #11651 Remove trailing whitespace causing travis failure * fix backport 11651 Signed-off-by: Pasta <pasta@dashboost.org> * More of 11651 * fix blockchain.cpp Signed-off-by: pasta <pasta@dashboost.org> * Add missing "qt/" in includes * Add missing "test/" in includes * Fix trailing whitespaces Co-authored-by: Wladimir J. van der Laan <laanwj@gmail.com> Co-authored-by: Russell Yanofsky <russ@yanofsky.org> Co-authored-by: MeshCollider <dobsonsa68@gmail.com> Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2020-03-19 23:46:56 +01:00
- *Rationale*: Bracket syntax is less ambiguous because the preprocessor
searches a fixed list of include directories without taking location of the
source file into account. This allows quoted includes to stand out more when
the location of the source file actually is relevant.
- Use include guards to avoid the problem of double inclusion. The header file
`foo/bar.h` should use the include guard identifier `BITCOIN_FOO_BAR_H`, e.g.
```c++
#ifndef BITCOIN_FOO_BAR_H
#define BITCOIN_FOO_BAR_H
...
#endif // BITCOIN_FOO_BAR_H
```
GUI
-----
- Do not display or manipulate dialogs in model code (classes `*Model`).
- *Rationale*: Model classes pass through events and data from the core, they
should not interact with the user. That's where View classes come in. The converse also
holds: try to not directly access core data structures from Views.
- Avoid adding slow or blocking code in the GUI thread. In particular, do not
add new `interfaces::Node` and `interfaces::Wallet` method calls, even if they
may be fast now, in case they are changed to lock or communicate across
processes in the future.
Prefer to offload work from the GUI thread to worker threads (see
`RPCExecutor` in console code as an example) or take other steps (see
https://doc.qt.io/archives/qq/qq27-responsive-guis.html) to keep the GUI
responsive.
- *Rationale*: Blocking the GUI thread can increase latency, and lead to
hangs and deadlocks.
Subtrees
----------
Several parts of the repository are subtrees of software maintained elsewhere.
Some of these are maintained by active developers of Bitcoin Core, in which case changes should probably go
directly upstream without being PRed directly against the project. They will be merged back in the next
subtree merge.
Others are external projects without a tight relationship with our project. Changes to these should also
be sent upstream, but bugfixes may also be prudent to PR against Dash Core so that they can be integrated
quickly. Cosmetic changes should be purely taken upstream.
There is a tool in `test/lint/git-subtree-check.sh` ([instructions](../test/lint#git-subtree-checksh)) to check a subtree directory for consistency with
its upstream repository.
Current subtrees include:
- src/leveldb
Merge #12495: Increase LevelDB max_open_files ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke) Pull request description: Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed. When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible. If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid. The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory. The original concerns about file descriptor exhaustion are unwarranted on most systems because: * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files. * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there. This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files. Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:56:44 +02:00
- Upstream at https://github.com/google/leveldb ; Maintained by Google, but
open important PRs to Core to avoid delay.
- **Note**: Follow the instructions in [Upgrading LevelDB](#upgrading-leveldb) when
merging upstream changes to the LevelDB subtree.
- src/crc32c
- Used by leveldb for hardware acceleration of CRC32C checksums for data integrity.
- Upstream at https://github.com/google/crc32c ; Maintained by Google.
- src/secp256k1
- Upstream at https://github.com/bitcoin-core/secp256k1/ ; actively maintained by Core contributors.
- src/crypto/ctaes
- Upstream at https://github.com/bitcoin-core/ctaes ; actively maintained by Core contributors.
- src/univalue
- Upstream at https://github.com/bitcoin-core/univalue ; actively maintained by Core contributors, deviates from upstream https://github.com/jgarzik/univalue
Merge #12495: Increase LevelDB max_open_files ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke) Pull request description: Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed. When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible. If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid. The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory. The original concerns about file descriptor exhaustion are unwarranted on most systems because: * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files. * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there. This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files. Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:56:44 +02:00
Upgrading LevelDB
---------------------
Extra care must be taken when upgrading LevelDB. This section explains issues
you must be aware of.
### File Descriptor Counts
In most configurations, we use the default LevelDB value for `max_open_files`,
Merge #12495: Increase LevelDB max_open_files ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke) Pull request description: Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed. When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible. If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid. The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory. The original concerns about file descriptor exhaustion are unwarranted on most systems because: * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files. * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there. This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files. Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:56:44 +02:00
which is 1000 at the time of this writing. If LevelDB actually uses this many
file descriptors, it will cause problems with Bitcoin's `select()` loop, because
Merge #12495: Increase LevelDB max_open_files ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke) Pull request description: Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed. When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible. If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid. The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory. The original concerns about file descriptor exhaustion are unwarranted on most systems because: * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files. * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there. This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files. Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:56:44 +02:00
it may cause new sockets to be created where the fd value is >= 1024. For this
reason, on 64-bit Unix systems, we rely on an internal LevelDB optimization that
Merge #12495: Increase LevelDB max_open_files ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke) Pull request description: Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed. When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible. If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid. The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory. The original concerns about file descriptor exhaustion are unwarranted on most systems because: * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files. * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there. This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files. Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:56:44 +02:00
uses `mmap()` + `close()` to open table files without actually retaining
references to the table file descriptors. If you are upgrading LevelDB, you must
sanity check the changes to make sure that this assumption remains valid.
In addition to reviewing the upstream changes in `env_posix.cc`, you can use `lsof` to
check this. For example, on Linux this command will show open `.ldb` file counts:
```bash
$ lsof -p $(pidof dashd) |\
Merge #12495: Increase LevelDB max_open_files ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke) Pull request description: Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed. When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible. If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid. The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory. The original concerns about file descriptor exhaustion are unwarranted on most systems because: * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files. * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there. This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files. Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:56:44 +02:00
awk 'BEGIN { fd=0; mem=0; } /ldb$/ { if ($4 == "mem") mem++; else fd++ } END { printf "mem = %s, fd = %s\n", mem, fd}'
mem = 119, fd = 0
```
The `mem` value shows how many files are mmap'ed, and the `fd` value shows you
many file descriptors these files are using. You should check that `fd` is a
small number (usually 0 on 64-bit hosts).
See the notes in the `SetMaxOpenFiles()` function in `dbwrapper.cc` for more
details.
### Consensus Compatibility
It is possible for LevelDB changes to inadvertently change consensus
compatibility between nodes. This happened in Bitcoin 0.8 (when LevelDB was
first introduced). When upgrading LevelDB, you should review the upstream changes
Merge #12495: Increase LevelDB max_open_files ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke) Pull request description: Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed. When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible. If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid. The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory. The original concerns about file descriptor exhaustion are unwarranted on most systems because: * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files. * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there. This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files. Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:56:44 +02:00
to check for issues affecting consensus compatibility.
For example, if LevelDB had a bug that accidentally prevented a key from being
returned in an edge case, and that bug was fixed upstream, the bug "fix" would
be an incompatible consensus change. In this situation, the correct behavior
Merge #12495: Increase LevelDB max_open_files ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke) Pull request description: Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed. When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible. If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid. The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory. The original concerns about file descriptor exhaustion are unwarranted on most systems because: * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files. * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there. This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files. Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:56:44 +02:00
would be to revert the upstream fix before applying the updates to Bitcoin's
copy of LevelDB. In general, you should be wary of any upstream changes affecting
Merge #12495: Increase LevelDB max_open_files ccedbaf Increase LevelDB max_open_files unless on 32-bit Unix. (Evan Klitzke) Pull request description: Currently we set `max_open_files = 64` on all architectures due to concerns about file descriptor exhaustion. This is extremely expensive due to how LevelDB is designed. When a LevelDB file handle is opened, a bloom filter and block index are decoded, and some CRCs are checked. Bloom filters and block indexes in open table handles can be checked purely in memory. This means that when doing a key lookup, if a given table file may contain a given key, all of the lookup operations can happen completely in RAM until the block itself is fetched. In the common case fetching the block is one disk seek, because the block index stores its physical offset. This is the ideal case, and what we want to happen as often as possible. If a table file handle is not open in the table cache, then in addition to the regular system calls to open the file, the block index and bloom filter need to be decoded before they can be checked. This is expensive and is something we want to avoid. The current setting of 64 file handles means that on a synced node, only about 4% of key lookups can be satisifed by table file handles that are actually open and in memory. The original concerns about file descriptor exhaustion are unwarranted on most systems because: * On 64-bit POSIX hosts LevelDB will open up to 1000 file descriptors using `mmap()`, and it does not retain an open file descriptor for such files. * On Windows non-socket files do not interfere with the main network `select()` loop, so the same fd exhaustion issues do not apply there. This change keeps the default `max_open_files` value (which is 1000) on all systems except 32-bit POSIX hosts (which do not use `mmap()`). Open file handles use about 20 KB of memory (for the block index), so the extra file handles do not cause much memory overhead. At most 1000 will be open, and a fully synced node right now has about 1500 such files. Profile of `loadblk` thread before changes: https://monad.io/maxopenfiles-master.svg Profile of `loadblk` thread after changes: https://monad.io/maxopenfiles-increase.svg Tree-SHA512: de54f77d57e9f8999eaf8d12592aab5b02f5877be8fa727a1f42cf02da2693ce25846445eb19eb138ce4e5045d1c65e14054df72faf3ff32c7655c9cfadd27a9
2018-03-29 14:56:44 +02:00
what data is returned from LevelDB queries.
Scripted diffs
--------------
For reformatting and refactoring commits where the changes can be easily automated using a bash script, we use
scripted-diff commits. The bash script is included in the commit message and our CI job checks that
the result of the script is identical to the commit. This aids reviewers since they can verify that the script
does exactly what it is supposed to do. It is also helpful for rebasing (since the same script can just be re-run
on the new master commit).
To create a scripted-diff:
- start the commit message with `scripted-diff:` (and then a description of the diff on the same line)
- in the commit message include the bash script between lines containing just the following text:
- `-BEGIN VERIFY SCRIPT-`
- `-END VERIFY SCRIPT-`
The scripted-diff is verified by the tool `test/lint/commit-script-check.sh`. The tool's default behavior, when supplied
with a commit is to verify all scripted-diffs from the beginning of time up to said commit. Internally, the tool passes
the first supplied argument to `git rev-list --reverse` to determine which commits to verify script-diffs for, ignoring
commits that don't conform to the commit message format described above.
For development, it might be more convenient to verify all scripted-diffs in a range `A..B`, for example:
```bash
test/lint/commit-script-check.sh origin/master..HEAD
```
### Suggestions and examples
If you need to replace in multiple files, prefer `git ls-files` to `find` or globbing, and `git grep` to `grep`, to
avoid changing files that are not under version control.
For efficient replacement scripts, reduce the selection to the files that potentially need to be modified, so for
example, instead of a blanket `git ls-files src | xargs sed -i s/apple/orange/`, use
`git grep -l apple src | xargs sed -i s/apple/orange/`.
Also, it is good to keep the selection of files as specific as possible — for example, replace only in directories where
you expect replacements — because it reduces the risk that a rebase of your commit by re-running the script will
introduce accidental changes.
Some good examples of scripted-diff:
- [scripted-diff: Rename InitInterfaces to NodeContext](https://github.com/bitcoin/bitcoin/commit/301bd41a2e6765b185bd55f4c541f9e27aeea29d)
2021-07-19 12:39:22 +02:00
uses an elegant script to replace occurrences of multiple terms in all source files.
- [scripted-diff: Remove g_connman, g_banman globals](https://github.com/bitcoin/bitcoin/commit/8922d7f6b751a3e6b3b9f6fb7961c442877fb65a)
replaces specific terms in a list of specific source files.
- [scripted-diff: Replace fprintf with tfm::format](https://github.com/bitcoin/bitcoin/commit/fac03ec43a15ad547161e37e53ea82482cc508f9)
does a global replacement but excludes certain directories.
To find all previous uses of scripted diffs in the repository, do:
```
git log --grep="-BEGIN VERIFY SCRIPT-"
```
Release notes
-------------
Release notes should be written for any PR that:
- introduces a notable new feature
- fixes a significant bug
- changes an API or configuration model
- makes any other visible change to the end-user experience.
Release notes should be added to a PR-specific release note file at
`/doc/release-notes-<PR number>.md` to avoid conflicts between multiple PRs.
All `release-notes*` files are merged into a single
[/doc/release-notes.md](/doc/release-notes.md) file prior to the release.
Release notes
-------------
Release notes should be written for any PR that:
- introduces a notable new feature
- fixes a significant bug
- changes an API or configuration model
- makes any other visible change to the end-user experience.
Release notes should be added to a PR-specific release note file at
`/doc/release-notes-<PR number>.md` to avoid conflicts between multiple PRs.
All `release-notes*` files are merged into a single
[/doc/release-notes.md](/doc/release-notes.md) file prior to the release.
RPC interface guidelines
--------------------------
A few guidelines for introducing and reviewing new RPC interfaces:
- Method naming: use consecutive lower-case names such as `getrawtransaction` and `submitblock`.
- *Rationale*: Consistency with the existing interface.
- Argument and field naming: please consider whether there is already a naming
style or spelling convention in the API for the type of object in question
(`blockhash`, for example), and if so, try to use that. If not, use snake case
`fee_delta` (and not, e.g. `feedelta` or camel case `feeDelta`).
- *Rationale*: Consistency with the existing interface.
- Use the JSON parser for parsing, don't manually parse integers or strings from
arguments unless absolutely necessary.
- *Rationale*: Introduces hand-rolled string manipulation code at both the caller and callee sites,
which is error-prone, and it is easy to get things such as escaping wrong.
JSON already supports nested data structures, no need to re-invent the wheel.
- *Exception*: AmountFromValue can parse amounts as string. This was introduced because many JSON
parsers and formatters hard-code handling decimal numbers as floating-point
values, resulting in potential loss of precision. This is unacceptable for
monetary values. **Always** use `AmountFromValue` and `ValueFromAmount` when
inputting or outputting monetary values. The only exceptions to this are
`prioritisetransaction` and `getblocktemplate` because their interface
is specified as-is in BIP22.
- Missing arguments and 'null' should be treated the same: as default values. If there is no
Merge #11050: Avoid treating null RPC arguments different from missing arguments 745d2e3 Clean up getbalance RPC parameter handling (Russell Yanofsky) fd5d71e Update developer notes after params.size() cleanup (Russell Yanofsky) e067673 Avoid treating null RPC arguments different from missing arguments (Russell Yanofsky) e666efc Get rid of redundant RPC params.size() checks (Russell Yanofsky) Pull request description: This is a followup to #10783. - The first commit doesn't change behavior at all, just simplifies code. - The second commit just changes RPC methods to treat null arguments the same as missing arguments instead of throwing type errors. - The third commit updates developer notes after the cleanup. - The forth commit does some additional code cleanup in `getbalance`. Followup changes that should happen in future PRs: - [ ] Replace uses of `.isTrue()` with calls to `.get_bool()` so numbers, objects, and strings cause type errors instead of being interpreted as false. https://github.com/bitcoin/bitcoin/pull/11050#discussion_r133850525 - [ ] Add braces around if statements. https://github.com/bitcoin/bitcoin/pull/11050#discussion_r133851133 - [ ] Maybe improve UniValue type error exceptions and eliminate RPCTypeCheck and RPCTypeCheckArgument functions. https://github.com/bitcoin/bitcoin/pull/11050#discussion_r133829303 Tree-SHA512: e72f696011d20acc0778e996659e41f9426bffce387b29ff63bf59ad1163d5146761e4445b2b9b9e069a80596a57c7f4402b75a15d5d20f69f775ae558cf67e9 More of 11050 Fixes for bitcoin#11050 backport Fixes for bitcoin#11050 Fix masternode_winners RPC help check and revert few params[x].isNull() changes RPC: resolve index and '!' problems Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
2017-08-22 09:24:31 +02:00
default value, both cases should fail in the same way. The easiest way to follow this
guideline is to detect unspecified arguments with `params[x].isNull()` instead of
Merge #11050: Avoid treating null RPC arguments different from missing arguments 745d2e3 Clean up getbalance RPC parameter handling (Russell Yanofsky) fd5d71e Update developer notes after params.size() cleanup (Russell Yanofsky) e067673 Avoid treating null RPC arguments different from missing arguments (Russell Yanofsky) e666efc Get rid of redundant RPC params.size() checks (Russell Yanofsky) Pull request description: This is a followup to #10783. - The first commit doesn't change behavior at all, just simplifies code. - The second commit just changes RPC methods to treat null arguments the same as missing arguments instead of throwing type errors. - The third commit updates developer notes after the cleanup. - The forth commit does some additional code cleanup in `getbalance`. Followup changes that should happen in future PRs: - [ ] Replace uses of `.isTrue()` with calls to `.get_bool()` so numbers, objects, and strings cause type errors instead of being interpreted as false. https://github.com/bitcoin/bitcoin/pull/11050#discussion_r133850525 - [ ] Add braces around if statements. https://github.com/bitcoin/bitcoin/pull/11050#discussion_r133851133 - [ ] Maybe improve UniValue type error exceptions and eliminate RPCTypeCheck and RPCTypeCheckArgument functions. https://github.com/bitcoin/bitcoin/pull/11050#discussion_r133829303 Tree-SHA512: e72f696011d20acc0778e996659e41f9426bffce387b29ff63bf59ad1163d5146761e4445b2b9b9e069a80596a57c7f4402b75a15d5d20f69f775ae558cf67e9 More of 11050 Fixes for bitcoin#11050 backport Fixes for bitcoin#11050 Fix masternode_winners RPC help check and revert few params[x].isNull() changes RPC: resolve index and '!' problems Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
2017-08-22 09:24:31 +02:00
`params.size() <= x`. The former returns true if the argument is either null or missing,
while the latter returns true if is missing, and false if it is null.
- *Rationale*: Avoids surprises when switching to name-based arguments. Missing name-based arguments
are passed as 'null'.
- Try not to overload methods on argument type. E.g. don't make `getblock(true)` and `getblock("hash")`
do different things.
- *Rationale*: This is impossible to use with `dash-cli`, and can be surprising to users.
- *Exception*: Some RPC calls can take both an `int` and `bool`, most notably when a bool was switched
to a multi-value, or due to other historical reasons. **Always** have false map to 0 and
true to 1 in this case.
- Don't forget to fill in the argument names correctly in the RPC command table.
- *Rationale*: If not, the call can not be used with name-based arguments.
- Add every non-string RPC argument `(method, idx, name)` to the table `vRPCConvertParams` in `rpc/client.cpp`.
- *Rationale*: `dash-cli` and the GUI debug console use this table to determine how to
convert a plaintext command line to JSON. If the types don't match, the method can be unusable
from there.
- A RPC method must either be a wallet method or a non-wallet method. Do not
introduce new methods that differ in behavior based on the presence of a wallet.
- *Rationale*: As well as complicating the implementation and interfering
with the introduction of multi-wallet, wallet and non-wallet code should be
separated to avoid introducing circular dependencies between code units.
- Try to make the RPC response a JSON object.
- *Rationale*: If a RPC response is not a JSON object, then it is harder to avoid API breakage if
new data in the response is needed.
- Wallet RPCs call BlockUntilSyncedToCurrentChain to maintain consistency with
`getblockchaininfo`'s state immediately prior to the call's execution. Wallet
RPCs whose behavior does *not* depend on the current chainstate may omit this
call.
- *Rationale*: In previous versions of Dash Core, the wallet was always
in-sync with the chainstate (by virtue of them all being updated in the
same cs_main lock). In order to maintain the behavior that wallet RPCs
return results as of at least the highest best-known block an RPC
client may be aware of prior to entering a wallet RPC call, we must block
until the wallet is caught up to the chainstate as of the RPC call's entry.
This also makes the API much easier for RPC clients to reason about.
- Be aware of RPC method aliases and generally avoid registering the same
callback function pointer for different RPCs.
- *Rationale*: RPC methods registered with the same function pointer will be
considered aliases and only the first method name will show up in the
`help` RPC command list.
- *Exception*: Using RPC method aliases may be appropriate in cases where a
new RPC is replacing a deprecated RPC, to avoid both RPCs confusingly
showing up in the command list.
- Use *invalid* Dash addresses (e.g. in the constant array `EXAMPLE_ADDRESS`) for
`RPCExamples` help documentation.
- *Rationale*: Prevent accidental transactions by users.
- Use the `UNIX_EPOCH_TIME` constant when describing UNIX epoch time or
timestamps in the documentation.
- *Rationale*: User-facing consistency.
Merge #18278: interfaces: Describe and follow some code conventions 3dc27a15242a22b5301904375e5880372e9b7f4d doc: Add internal interface conventions to developer notes (Russell Yanofsky) 1dca9dc4c772fa0a4ec52c4d88b7cd3d243aea7b refactor: Change createWallet, fillPSBT argument order (Russell Yanofsky) 96dfe5ced64979e51649d20555aa182defc80119 refactor: Change Chain::broadcastTransaction param order (Russell Yanofsky) 6ceb21909ce66b7b4762a855889acd46bb6b77f3 refactor: Rename Chain::Notifications methods to be consistent with other interfaces methods (Russell Yanofsky) 1c2ab1a6d29f2c6c065dae4f4a4e2ad1286311b3 refactor: Rename Node::disconnect methods (Russell Yanofsky) 77e4b0657298c715c835d8d2eb11e173852e6815 refactor: Get rid of Wallet::IsWalletFlagSet method (Russell Yanofsky) Pull request description: This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10). This PR doesn't change behavior at all, it just cleans up code in [`src/interfaces`](https://github.com/bitcoin/bitcoin/tree/master/src/interfaces) to simplify #10102, and [documents](https://github.com/ryanofsky/bitcoin/blob/pr/ipc-conv/doc/developer-notes.md#internal-interface-guidelines) coding conventions there better ACKs for top commit: hebasto: re-ACK 3dc27a15242a22b5301904375e5880372e9b7f4d, the only change since the [previous](https://github.com/bitcoin/bitcoin/pull/18278#pullrequestreview-372582146) review is rebasing. MarcoFalke: ACK 3dc27a15242a22b5301904375e5880372e9b7f4d 🕍 Tree-SHA512: 62e6a0f2488e3924e559d2074ed460b92e7a0a5d98eab492221cb20d59d04bbe32aef2a8aeba5e4ea9168cfa91acd5bc973dce6677be0180bd7a919354df53ed
2020-03-23 21:37:44 +01:00
- Use `fs::path::u8string()` and `fs::u8path()` functions when converting path
to JSON strings, not `fs::PathToString` and `fs::PathFromString`
- *Rationale*: JSON strings are Unicode strings, not byte strings, and
RFC8259 requires JSON to be encoded as UTF-8.
Merge #18278: interfaces: Describe and follow some code conventions 3dc27a15242a22b5301904375e5880372e9b7f4d doc: Add internal interface conventions to developer notes (Russell Yanofsky) 1dca9dc4c772fa0a4ec52c4d88b7cd3d243aea7b refactor: Change createWallet, fillPSBT argument order (Russell Yanofsky) 96dfe5ced64979e51649d20555aa182defc80119 refactor: Change Chain::broadcastTransaction param order (Russell Yanofsky) 6ceb21909ce66b7b4762a855889acd46bb6b77f3 refactor: Rename Chain::Notifications methods to be consistent with other interfaces methods (Russell Yanofsky) 1c2ab1a6d29f2c6c065dae4f4a4e2ad1286311b3 refactor: Rename Node::disconnect methods (Russell Yanofsky) 77e4b0657298c715c835d8d2eb11e173852e6815 refactor: Get rid of Wallet::IsWalletFlagSet method (Russell Yanofsky) Pull request description: This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10). This PR doesn't change behavior at all, it just cleans up code in [`src/interfaces`](https://github.com/bitcoin/bitcoin/tree/master/src/interfaces) to simplify #10102, and [documents](https://github.com/ryanofsky/bitcoin/blob/pr/ipc-conv/doc/developer-notes.md#internal-interface-guidelines) coding conventions there better ACKs for top commit: hebasto: re-ACK 3dc27a15242a22b5301904375e5880372e9b7f4d, the only change since the [previous](https://github.com/bitcoin/bitcoin/pull/18278#pullrequestreview-372582146) review is rebasing. MarcoFalke: ACK 3dc27a15242a22b5301904375e5880372e9b7f4d 🕍 Tree-SHA512: 62e6a0f2488e3924e559d2074ed460b92e7a0a5d98eab492221cb20d59d04bbe32aef2a8aeba5e4ea9168cfa91acd5bc973dce6677be0180bd7a919354df53ed
2020-03-23 21:37:44 +01:00
Internal interface guidelines
-----------------------------
Internal interfaces between parts of the codebase that are meant to be
independent (node, wallet, GUI), are defined in
[`src/interfaces/`](../src/interfaces/). The main interface classes defined
there are [`interfaces::Chain`](../src/interfaces/chain.h), used by wallet to
access the node's latest chain state,
[`interfaces::Node`](../src/interfaces/node.h), used by the GUI to control the
node, and [`interfaces::Wallet`](../src/interfaces/wallet.h), used by the GUI
to control an individual wallet. There are also more specialized interface
types like [`interfaces::Handler`](../src/interfaces/handler.h)
[`interfaces::ChainClient`](../src/interfaces/chain.h) passed to and from
various interface methods.
Interface classes are written in a particular style so node, wallet, and GUI
code doesn't need to run in the same process, and so the class declarations
work more easily with tools and libraries supporting interprocess
communication:
- Interface classes should be abstract and have methods that are [pure
virtual](https://en.cppreference.com/w/cpp/language/abstract_class). This
allows multiple implementations to inherit from the same interface class,
particularly so one implementation can execute functionality in the local
process, and other implementations can forward calls to remote processes.
- Interface method definitions should wrap existing functionality instead of
implementing new functionality. Any substantial new node or wallet
functionality should be implemented in [`src/node/`](../src/node/) or
[`src/wallet/`](../src/wallet/) and just exposed in
[`src/interfaces/`](../src/interfaces/) instead of being implemented there,
so it can be more modular and accessible to unit tests.
- Interface method parameter and return types should either be serializable or
be other interface classes. Interface methods shouldn't pass references to
objects that can't be serialized or accessed from another process.
Examples:
```c++
// Good: takes string argument and returns interface class pointer
virtual unique_ptr<interfaces::Wallet> loadWallet(std::string filename) = 0;
// Bad: returns CWallet reference that can't be used from another process
virtual CWallet& loadWallet(std::string filename) = 0;
```
```c++
// Good: accepts and returns primitive types
virtual bool findBlock(const uint256& hash, int& out_height, int64_t& out_time) = 0;
// Bad: returns pointer to internal node in a linked list inaccessible to
// other processes
virtual const CBlockIndex* findBlock(const uint256& hash) = 0;
```
```c++
// Good: takes plain callback type and returns interface pointer
using TipChangedFn = std::function<void(int block_height, int64_t block_time)>;
virtual std::unique_ptr<interfaces::Handler> handleTipChanged(TipChangedFn fn) = 0;
// Bad: returns boost connection specific to local process
using TipChangedFn = std::function<void(int block_height, int64_t block_time)>;
virtual boost::signals2::scoped_connection connectTipChanged(TipChangedFn fn) = 0;
```
- For consistency and friendliness to code generation tools, interface method
input and inout parameters should be ordered first and output parameters
should come last.
Example:
```c++
// Good: error output param is last
virtual bool broadcastTransaction(const CTransactionRef& tx, CAmount max_fee, std::string& error) = 0;
// Bad: error output param is between input params
virtual bool broadcastTransaction(const CTransactionRef& tx, std::string& error, CAmount max_fee) = 0;
```
- For friendliness to code generation tools, interface methods should not be
overloaded:
Example:
```c++
// Good: method names are unique
virtual bool disconnectByAddress(const CNetAddr& net_addr) = 0;
virtual bool disconnectById(NodeId id) = 0;
// Bad: methods are overloaded by type
virtual bool disconnect(const CNetAddr& net_addr) = 0;
virtual bool disconnect(NodeId id) = 0;
```
- For consistency and friendliness to code generation tools, interface method
names should be `lowerCamelCase` and standalone function names should be
`UpperCamelCase`.
Examples:
```c++
// Good: lowerCamelCase method name
virtual void blockConnected(const CBlock& block, int height) = 0;
// Bad: uppercase class method
virtual void BlockConnected(const CBlock& block, int height) = 0;
```
```c++
// Good: UpperCamelCase standalone function name
std::unique_ptr<Node> MakeNode(LocalInit& init);
// Bad: lowercase standalone function
std::unique_ptr<Node> makeNode(LocalInit& init);
```
Note: This last convention isn't generally followed outside of
[`src/interfaces/`](../src/interfaces/), though it did come up for discussion
before in [#14635](https://github.com/bitcoin/bitcoin/pull/14635).