mirror of
https://github.com/dashpay/dash.git
synced 2024-12-25 12:02:48 +01:00
3411577473
84934bf70e11fe4cda1cfda60113a54895d4fdd5 multiprocess: Add echoipc RPC method and test (Russell Yanofsky) 7d76cf667eff512043a28d4407cc89f58796c42b multiprocess: Add comments and documentation (Russell Yanofsky) ddf7ecc8dfc64cf121099fb047e1ac871de94f4c multiprocess: Add bitcoin-node process spawning support (Russell Yanofsky) 10afdf0280fa93bfffb0a7665c60dc155cd84514 multiprocess: Add Ipc interface implementation (Russell Yanofsky) 745c9cebd50fea1664efef571dc1ee1bddc96102 multiprocess: Add Ipc and Init interface definitions (Russell Yanofsky) 5d62d7f6cd48bbc4e9f37ecc369f38d5e1e0036c Update libmultiprocess library (Russell Yanofsky) Pull request description: This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10). --- This PR adds basic process spawning and IPC method call support to `bitcoin-node` executables built with `--enable-multiprocess`[*]. These changes are used in https://github.com/bitcoin/bitcoin/pull/10102 to let node, gui, and wallet functionality run in different processes, and extended in https://github.com/bitcoin/bitcoin/pull/19460 and https://github.com/bitcoin/bitcoin/pull/19461 after that to allow gui and wallet processes to be started and stopped independently and connect to the node over a socket. These changes can also be used to implement new functionality outside the `bitcoin-node` process like external indexes or pluggable transports (https://github.com/bitcoin/bitcoin/pull/18988). The `Ipc::spawnProcess` and `Ipc::serveProcess` methods added here are entry points for spawning a child process and serving a parent process, and being able to make bidirectional, multithreaded method calls between the processes. A simple example of this is implemented in commit "Add echoipc RPC method and test." Changes in this PR aside from the echo test were originally part of #10102, but have been split and moved here for easier review, and so they can be used for other applications like external plugins. Additional notes about this PR can be found at https://bitcoincore.reviews/19160 [*] Note: the `--enable-multiprocess` feature is still experimental, and not enabled by default, and not yet supported on windows. More information can be found in [doc/multiprocess.md](https://github.com/bitcoin/bitcoin/blob/master/doc/multiprocess.md) ACKs for top commit: fjahr: re-ACK 84934bf70e11fe4cda1cfda60113a54895d4fdd5 ariard: ACK 84934bf. Changes since last ACK fixes the silent merge conflict about `EnsureAnyNodeContext()`. Rebuilt and checked again debug command `echoipc`. Tree-SHA512: 52a948b5e18a26d7d7a09b83003eaae9b1ed2981978c36c959fe9a55abf70ae6a627c4ff913a3428be17400a3dace30c58b5057fa75c319662c3be98f19810c6
73 lines
5.9 KiB
Markdown
73 lines
5.9 KiB
Markdown
# Multiprocess Dash
|
|
|
|
On unix systems, the `--enable-multiprocess` build option can be passed to `./configure` to build new `dash-node`, `dash-wallet`, and `dash-gui` executables alongside existing `dashd` and `dash-qt` executables.
|
|
|
|
`dash-node` is a drop-in replacement for `dashd`, and `dash-gui` is a drop-in replacement for `dash-qt`, and there are no differences in use or external behavior between the new and old executables. But internally (after backporting [bitcoin#10102](https://github.com/bitcoin/bitcoin/pull/10102)), `dash-gui` will spawn a `dash-node` process to run P2P and RPC code, communicating with it across a socket pair, and `dash-node` will spawn `dash-wallet` to run wallet code, also communicating over a socket pair. This will let node, wallet, and GUI code run in separate address spaces for better isolation, and allow future improvements like being able to start and stop components independently on different machines and environments.
|
|
|
|
## Next steps
|
|
|
|
Specific next steps after backporting [bitcoin#10102](https://github.com/bitcoin/bitcoin/pull/10102) will be:
|
|
|
|
- [ ] Adding `-ipcbind` and `-ipcconnect` options to `dash-node`, `dash-wallet`, and `dash-gui` executables so they can listen and connect to TCP ports and unix socket paths. This will allow separate processes to be started and stopped any time and connect to each other.
|
|
- [ ] Adding `-server` and `-rpcbind` options to the `dash-wallet` executable so wallet processes can handle RPC requests directly without going through the node.
|
|
- [ ] Supporting windows, not just unix systems. The existing socket code is already cross-platform, so the only windows-specific code that needs to be written is code spawning a process and passing a socket descriptor. This can be implemented with `CreateProcess` and `WSADuplicateSocket`. Example: https://memset.wordpress.com/2010/10/13/win32-api-passing-socket-with-ipc-method/.
|
|
- [ ] Adding sandbox features, restricting subprocess access to resources and data. See [https://eklitzke.org/multiprocess-bitcoin](https://eklitzke.org/multiprocess-bitcoin).
|
|
|
|
## Debugging
|
|
|
|
The `-debug=ipc` command line option can be used to see requests and responses between processes.
|
|
|
|
## Installation
|
|
|
|
The multiprocess feature requires [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) as dependencies. A simple way to get starting using it without installing these dependencies manually is to use the [depends system](../depends) with the `MULTIPROCESS=1` [dependency option](../depends#dependency-options) passed to make:
|
|
|
|
```
|
|
cd <DASH_SOURCE_DIRECTORY>
|
|
make -C depends NO_QT=1 MULTIPROCESS=1
|
|
CONFIG_SITE=$PWD/depends/x86_64-pc-linux-gnu/share/config.site ./configure
|
|
make
|
|
src/dash-node -regtest -printtoconsole -debug=ipc
|
|
DASHD=dash-node test/functional/test_runner.py
|
|
```
|
|
|
|
The configure script will pick up settings and library locations from the depends directory, so there is no need to pass `--enable-multiprocess` as a separate flag when using the depends system (it's controlled by the `MULTIPROCESS=1` option).
|
|
|
|
Alternately, you can install [Cap'n Proto](https://capnproto.org/) and [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) packages on your system, and just run `./configure --enable-multiprocess` without using the depends system. The configure script will be able to locate the installed packages via [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/). See [Installation](https://github.com/chaincodelabs/libmultiprocess#installation) section of the libmultiprocess readme for install steps. See [build-unix.md](build-unix.md) and [build-osx.md](build-osx.md) for information about installing dependencies in general.
|
|
|
|
## IPC implementation details
|
|
|
|
Cross process Node, Wallet, and Chain interfaces are defined in
|
|
[`src/interfaces/`](../src/interfaces/). These are C++ classes which follow
|
|
[conventions](developer-notes.md#internal-interface-guidelines), like passing
|
|
serializable arguments so they can be called from different processes, and
|
|
making methods pure virtual so they can have proxy implementations that forward
|
|
calls between processes.
|
|
|
|
When Wallet, Node, and Chain code is running in the same process, calling any
|
|
interface method invokes the implementation directly. When code is running in
|
|
different processes, calling an interface method invokes a proxy interface
|
|
implementation that communicates with a remote process and invokes the real
|
|
implementation in the remote process. The
|
|
[libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) code
|
|
generation tool internally generates proxy client classes and proxy server
|
|
classes for this purpose that are thin wrappers around Cap'n Proto
|
|
[client](https://capnproto.org/cxxrpc.html#clients) and
|
|
[server](https://capnproto.org/cxxrpc.html#servers) classes, which handle the
|
|
actual serialization and socket communication.
|
|
|
|
As much as possible, calls between processes are meant to work the same as
|
|
calls within a single process without adding limitations or requiring extra
|
|
implementation effort. Processes communicate with each other by calling regular
|
|
[C++ interface methods](../src/interfaces/README.md). Method arguments and
|
|
return values are automatically serialized and sent between processes. Object
|
|
references and `std::function` arguments are automatically tracked and mapped
|
|
to allow invoked code to call back into invoking code at any time, and there is
|
|
a 1:1 threading model where any thread invoking a method in another process has
|
|
a corresponding thread in the invoked process responsible for executing all
|
|
method calls from the source thread, without blocking I/O or holding up another
|
|
call, and using the same thread local variables, locks, and callbacks between
|
|
calls. The forwarding, tracking, and threading is implemented inside the
|
|
[libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) library
|
|
which has the design goal of making calls between processes look like calls in
|
|
the same process to the extent possible.
|