8794a4b3ae4d34a4cd21a7dee9f694eef7726a4f QA: add test for HKDF HMAC_SHA256 L32 (Jonas Schnelli)
551d489416339dae8f9d896013cd060a21406e2b Add HKDF HMAC_SHA256 L=32 implementations (Jonas Schnelli)
3b64f852e400c552f031697d6a86829dc6e74bd6 QA: add test for CKey::Negate() (Jonas Schnelli)
463921bb649d644f79f9d7f0f96f10aa0d165f76 CKey: add method to negate the key (Jonas Schnelli)
Pull request description:
This adds a limited implementation of `HKDF` (defined by rfc5869) that supports only HMAC-SHA256 and length output of 32 bytes (will be required for v2 transport protocol).
This PR also includes a method to negate a private key which is useful to enforce public keys starting with 0x02 (or 0x03) (a requirement for the v2 transport protocol). The new `CKey::Negate()` method is pretty much a wrapper around `secp256k1_ec_privkey_negate()`.
Including tests.
This is a subset of #14032 and a pre-requirement for the v2 transport protocol.
ACKs for commit 8794a4:
Tree-SHA512: 5341929dfa29f5da766ec3612784baec6a3ad69972f08b5a985a8aafdae4dae36f104a2b888d1f5d1f33561456bd111f960d7e32c2cc4fd18e48358468f26c1a
* build: detect the presence of libgmp before generating Makefile
* depends: add arximboldi/immer@v0.6.2 as a package and add detection
* depends: remove immer from source tree, build using package only
* Drop immer refs from tools
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
c5b404e8f1973afe071a07c63ba1038eefe13f0f Add functional tests for flexible whitebind/list (nicolas.dorier)
d541fa391844f658bd7035659b5b16695733dd56 Replace the use of fWhitelisted by permission checks (nicolas.dorier)
ecd5cf7ea4c3644a30092100ffc399e30e193275 Do not disconnect peer for asking mempool if it has NO_BAN permission (nicolas.dorier)
e5b26deaaa6842f7dd7c4537ede000f965ea0189 Make whitebind/whitelist permissions more flexible (nicolas.dorier)
Pull request description:
# Motivation
In 0.19, bloom filter will be disabled by default. I tried to make [a PR](https://github.com/bitcoin/bitcoin/pull/16176) to enable bloom filter for whitelisted peers regardless of `-peerbloomfilters`.
Bloom filter have non existent privacy and server can omit filter's matches. However, both problems are completely irrelevant when you connect to your own node. If you connect to your own node, bloom filters are the most bandwidth efficient way to synchronize your light client without the need of some middleware like Electrum.
It is also a superior alternative to BIP157 as it does not require to maintain an additional index and it would work well on pruned nodes.
When I attempted to allow bloom filters for whitelisted peer, my proposal has been NACKed in favor of [a more flexible approach](https://github.com/bitcoin/bitcoin/pull/16176#issuecomment-500762907) which should allow node operator to set fine grained permissions instead of a global `whitelisted` attribute.
Doing so will also make follow up idea very easy to implement in a backward compatible way.
# Implementation details
The PR propose a new format for `--white{list,bind}`. I added a way to specify permissions granted to inbound connection matching `white{list,bind}`.
The following permissions exists:
* ForceRelay
* Relay
* NoBan
* BloomFilter
* Mempool
Example:
* `-whitelist=bloomfilter@127.0.0.1/32`.
* `-whitebind=bloomfilter,relay,noban@127.0.0.1:10020`.
If no permissions are specified, `NoBan | Mempool` is assumed. (making this PR backward compatible)
When we receive an inbound connection, we calculate the effective permissions for this peer by fetching the permissions granted from `whitelist` and add to it the permissions granted from `whitebind`.
To keep backward compatibility, if no permissions are specified in `white{list,bind}` (e.g. `--whitelist=127.0.0.1`) then parameters `-whitelistforcerelay` and `-whiterelay` will add the permissions `ForceRelay` and `Relay` to the inbound node.
`-whitelistforcerelay` and `-whiterelay` are ignored if the permissions flags are explicitly set in `white{bind,list}`.
# Follow up idea
Based on this PR, other changes become quite easy to code in a trivially review-able, backward compatible way:
* Changing `connect` at rpc and config file level to understand the permissions flags.
* Changing the permissions of a peer at RPC level.
ACKs for top commit:
laanwj:
re-ACK c5b404e8f1973afe071a07c63ba1038eefe13f0f
Tree-SHA512: adfefb373d09e68cae401247c8fc64034e305694cdef104bdcdacb9f1704277bd53b18f52a2427a5cffdbc77bda410d221aed252bc2ece698ffbb9cf1b830577
18185b57c32d0a43afeca4c125b9352c692923e9 scripted-diff: batch-recase BanMan variables (Carl Dong)
c2e04d37f3841d109c1fe60693f9622e2836cc29 banman: Add, use CBanEntry ctor that takes ban reason (Carl Dong)
1ffa4ce27d4ea6c1067d8984455df97994c7713e banman: reformulate nBanUtil calculation (Carl Dong)
daae598feb034f2f56e0b00ecfb4854d693d3641 banman: add thread annotations and mark members const where possible (Cory Fields)
84fc3fbd0304a7d6e660bf783c84bed2dd415141 scripted-diff: batch-rename BanMan members (Cory Fields)
af3503d903b1a608cd212e2d74b274103199078c net: move BanMan to its own files (Cory Fields)
d0469b2e9386a7a4b268cb9725347e7517acace6 banman: pass in default ban time as a parameter (Cory Fields)
2e56702ecedd83c4b7cb8de9de5c437c8c08e645 banman: pass the banfile path in (Cory Fields)
4c0d961eb0d7825a1e6f8389d7f5545114ee18c6 banman: create and split out banman (Cory Fields)
83c1ea2e5e66b8a83072e3d5ad6a4ced406eb1ba net: split up addresses/ban dumps in preparation for moving them (Cory Fields)
136bd7926c72659dd277a7b795ea17f72e523338 tests: remove member connman/peerLogic in TestingSetup (Cory Fields)
7cc2b9f6786f9bc33853220551eed33ca6b7b7b2 net: Break disconnecting out of Ban() (Cory Fields)
Pull request description:
**Old English à la Beowulf**
```
Banman wæs bréme --blaéd wíde sprang--
Connmanes eafera Coreum in.
aéglaéca léodum forstandan
Swá bealdode bearn Connmanes
guma gúðum cúð gódum daédum·
dréah æfter dóme· nealles druncne slóg
```
**Modern English Translation**
```
Banman was famed --his renown spread wide--
Conman's hier, in Core-land.
against the evil creature defend the people
Thus he was bold, the son of Connman
man famed in war, for good deeds;
he led his life for glory, never, having drunk, slew
```
--
With @theuni's blessing, here is Banman, rebased. Original PR: https://github.com/bitcoin/bitcoin/pull/11457
--
Followup PRs:
1. Give `CNode` a `Disconnect` method ([source](https://github.com/bitcoin/bitcoin/pull/14605#discussion_r248065847))
2. Add a comment to `std::atomic_bool fDisconnect` in `net.h` that setting this to true will cause the node to be disconnected the next time `DisconnectNodes()` runs ([source](https://github.com/bitcoin/bitcoin/pull/14605#discussion_r248384309))
Tree-SHA512: 9c207edbf577415c22c9811113e393322d936a843d4ff265186728152a67c057779ac4d4f27b895de9729f7a53e870f828b9ebc8bcdab757520c2aebe1e9be35
* Update to leveldb upstream using subtree merge
* Import crc32c using subtree merge as as 'src/crc32c'
* build: Update build system for new leveldb
Upstream leveldb switched build systems, which means we need to define
a few different values.
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* doc: Add crc32c subtree to developer notes
* test: Add crc32c to subtree check linter
* test: Add crc32c exception to various linters and generation scripts
* build: Add LCOV exception for crc32c
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* build: CRC32C build system integration
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
Change ctpl implementation to use STL queue & mutex.
Use ctpl synchronized queue instead of boost lockfree queue in bls worker aggregator.
Use smart pointers for memory management of Aggregator and VectorAggregator. With 'delete this;' the objects are prone to data race on the delete operator.
Use smart pointers for memory management of ContributionVerifier.
Pass shared_ptr by value to other threads via worker pool.
f7696e6183 depends: qt: Don't hardcode pwd path (Carl Dong)
89bee1bdbf depends: tar: Always extract as yourself (Carl Dong)
340ef50772 depends: Defer to Python detected by autoconf (Carl Dong)
Pull request description:
Removes some implicit assumptions that the depends system has about its environment and, as a side-effect, makes it possible to build the depends tree under severely privilege-limited environments such as containers built by Guix.
Tree-SHA512: e8618f9310a0deae864b44f9b60baa29e6225ba16817973ff7830b55798ebd4343aa06da6c1f92682a7afb709d26f80d6ee794a139d4d44c27caf4f0c8fe95fc
e0664f7f54aa9cf99ba31ca4bd9de1db0a9310a4 build: Move interfaces/* to libbitcoin_server (Wladimir J. van der Laan)
Pull request description:
Move interfaces/* from libbitcoin_util to libbitcoin_server.
Usage of these is shared between `bitcoind` and `bitcoin-qt`. It is unnecessary for them to be linked against the other utilities. Also semantically they belong with the server/node, I think.
Tree-SHA512: f82f3a95d362051c0eb8092520715f77c2c75409d41f9c0fab9a15445ea9e79a2a36b5d00b1f5be09b266584051934a32a8b9b28f783f4d7be78885b4a29a383
1fabd59e7 Break circular dependency: init -> * -> init by extracting shutdown.h (Ben Woosley)
e62fdfeea Drop unused init.h includes (Ben Woosley)
Pull request description:
Most includers just wanted to react to pending shutdown.
This isolates access to `fRequestShutdown` and limits access to the shutdown api functions, including the new `CancelShutdown` for setting it to `false`.
Tree-SHA512: df42f75dfbba163576710e9a67cf1228531fd99d70a2f187bfba0bcc476d6749cf88180a97e66a81bb5b6c3c7f0917de7402d26039ba7b644cb7509b02f7e267
ec3073a274bf7affe1b8c87a10f75d126f5ac027 index: Move index DBs into index/ directory. (Jim Posen)
89eddcd365e9a2218648f5cc5b9f22b28023f50a index: Remove TxIndexDB from public interface of TxIndex. (Jim Posen)
2318affd27de436ddf9d866a4b82eed8ea2e738b MOVEONLY: Move BaseIndex to its own file. (Jim Posen)
f376a4924109af2496b5fd16a787299eb039f1c8 index: Generalize logged statements in BaseIndex. (Jim Posen)
61a1226d87d80234b2be123c5cad07534c318cfb index: Extract logic from TxIndex into reusable base class. (Jim Posen)
e5af5fc6fb4658599b940d1d50853129b31b8766 db: Make reusable base class for index databases. (Jim Posen)
9b0ec1a7f9ffae816fd5ca32ff7e7559640b6f6d db: Remove obsolete methods from CBlockTreeDB. (Jim Posen)
Pull request description:
This refactors most of the logic in TxIndex into a reusable base class for other indices. There are two commits moving code between files, which may be be more easily reviewed using `git diff --color-moved` (https://blog.github.com/2018-04-05-git-217-released/).
The motivation for this is to support BIP 157 by indexing block filters.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/bitcoin/bitcoin/13243)
<!-- Reviewable:end -->
Tree-SHA512: 0857f04df2aa920178dab2eb8e57984d8eb4d5010deca9971190358479e05b6672ccca2a08af0a7ac9fe02afb947be84cf35a3693204d0667263c6add2959cbf
7cee85807c4db679003c6659d247a2fe74c2464a Add compile time verification of assumptions we're currently making implicitly/tacitly (practicalswift)
Pull request description:
Add compile time verification of assumptions we're currently making implicitly/tacitly.
As suggested by @sipa in https://github.com/bitcoin/bitcoin/pull/14239#issuecomment-462508012 and @MarcoFalke in https://github.com/bitcoin/bitcoin/pull/14479#issuecomment-462534878.
Tree-SHA512: e68fe51164dbd3eeb76aa8a7e83dfcd3b4d5a66037c0f1822bbbd189bbe3c280e03b3b10af870880ecc09b612e62fb3d9bcd6cf1e16cb7ba818c257db0712ce4
232f96f5c8a3920c09db92f4dbac2ad7d10ce8cf doc: Add release notes for -avoidpartialspends (Karl-Johan Alm)
e00b4699cc6d2ee5697d38dd6607eb2631c9b77a clean-up: Remove no longer used ivars from CInputCoin (Karl-Johan Alm)
43e04d13b1ffc02b1082176e87f420198b40c7b1 wallet: Remove deprecated OutputEligibleForSpending (Karl-Johan Alm)
0128121101fb3ee82f3abd3973a967a4226ffe0e test: Add basic testing for wallet groups (Karl-Johan Alm)
59d6f7b4e2f847ec1f2ff46c84e6157655984f85 wallet: Switch to using output groups instead of coins in coin selection (Karl-Johan Alm)
87ebce25d66952f5ce565bb5130dcf5e24049872 wallet: Add output grouping (Karl-Johan Alm)
bb629cb9dc567cc819724d9f4852652926e60cbf Add -avoidpartialspends and m_avoid_partial_spends (Karl-Johan Alm)
65b3eda458221644616d0fdd6ba0fe01bdbce893 wallet: Add input bytes to CInputCoin (Karl-Johan Alm)
a443d7a0ca333b0bae63e04b5d476f9ad9c7aeac moveonly: CoinElegibilityFilter into coinselection.h (Karl-Johan Alm)
173e18a289088c6087ba6fac708e322aa63b7a94 utils: Add insert() convenience templates (Karl-Johan Alm)
Pull request description:
This PR adds an optional (off by default) `-avoidpartialspends` flag, which changes coin select to use output groups rather than outputs, where each output group corresponds to all outputs with the same destination.
It is a privacy improvement, as each time you spend some output, any other output that is publicly associated with the destination (address) will also be spent at the same time, at the cost of fee increase for cases where coin select without group restriction would find a more optimal set of coins (see example below).
For regular use without address reuse, this PR should have no effect on the user experience whatsoever; it only affects users who, for some reason, have multiple outputs with the same destination (i.e. address reuse).
Nodes with this turned off will still try to avoid partial spending, if the fee of the resulting transaction is not greater than the fee of the original transaction.
Example: a node has four outputs linked to two addresses `A` and `B`:
* 1.0 btc to `A`
* 0.5 btc to `A`
* 1.0 btc to `B`
* 0.5 btc to `B`
The node sends 0.2 btc to `C`. Without `-avoidpartialspends`, the following coin selection will occur:
* 0.5 btc to `A` or `B` is picked
* 0.2 btc is output to `C`
* 0.3 - fee is output to (unique change address)
With `-avoidpartialspends`, the following will instead happen:
* Both of (0.5, 1.0) btc to `A` or `B` is picked (one or the other pair)
* 0.2 btc is output to `C`
* 1.3 - fee is output to (unique change address)
As noted, the pro here is that, assuming nobody sends to the address after you spend from it, you will only ever use one address once. The con is that the transaction becomes slightly larger in this case, because it is overpicking outputs to adhere to the no partial spending rule.
This complements #10386, in particular it addresses @luke-jr and @gmaxwell's concerns in https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-300667926 and https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-302361381.
Together with `-avoidreuse`, this fully addresses the concerns in #10065 I believe.
Tree-SHA512: 24687a4490ba59cf4198ed90052944ff4996653a4257833bb52ed24d058b3e924800c9b3790aeb6be6385b653b49e304453e5d7ff960e64c682fc23bfc447621
# Conflicts:
# src/Makefile.am
# src/bench/coin_selection.cpp
# src/wallet/coincontrol.h
# src/wallet/coinselection.cpp
# src/wallet/coinselection.h
# src/wallet/init.cpp
# src/wallet/test/coinselector_tests.cpp
# src/wallet/wallet.cpp
# src/wallet/wallet.h
# test/functional/test_runner.py
9e49db2 Make --enable-debug to pick better options (Evan Klitzke)
Pull request description:
Cherry-picked (and rebased) 94189645e67f364c4445d62e2b00c282d885cbbf from the "up for grabs" PR: "[build] Make --enable-debug pick better options" (#12695).
See previous review in #12695.
Tree-SHA512: a93cdadcf13e2ef8519acb1ce4f41ce95057a388347bb0a86a5c164dc7d0b0d14d4bb2a466082d5a100b8d50de65c605c40abaed555e8ea77c99e28800a34439
9b2704777c [doc] Include txindex changes in the release notes. (Jim Posen)
ed77dd6b30 [test] Simple unit test for TxIndex. (Jim Posen)
6d772a3d44 [rpc] Public interfaces to GetTransaction block until synced. (Jim Posen)
a03f804f2a [index] Move disk IO logic from GetTransaction to TxIndex::FindTx. (Jim Posen)
e0a3b80033 [validation] Replace tx index code in validation code with TxIndex. (Jim Posen)
8181db88f6 [init] Initialize and start TxIndex in init code. (Jim Posen)
f90c3a62f5 [index] TxIndex method to wait until caught up. (Jim Posen)
70d510d93c [index] Allow TxIndex sync thread to be interrupted. (Jim Posen)
94b4f8bbb9 [index] TxIndex initial sync thread. (Jim Posen)
34d68bf3a3 [index] Create new TxIndex class. (Jim Posen)
c88bcec93f [db] Migration for txindex data to new, separate database. (Jim Posen)
0cb8303241 [db] Create separate database for txindex. (Jim Posen)
Pull request description:
I'm re-opening #11857 as a new pull request because the last one stopped loading for people
-------------------------------
This refactors the tx index code to be in it's own class and get built concurrently with validation code. The main benefit is decoupling and moving the txindex into a separate DB. The primary motivation is to lay the groundwork for other indexers that might be desired (such as the [compact filters](https://github.com/bitcoin/bips/pull/636)). The basic idea is that the TxIndex spins up its own thread, which first syncs the txindex to the current block index, then once in sync the BlockConnected ValidationInterface hook writes new blocks.
### DB changes
At the suggestion of some other developers, the txindex has been split out into a separate database. A data migration runs at startup on any nodes with a legacy txindex. Currently the migration blocks node initialization until complete.
### Open questions
- Should the migration of txindex data from the old DB to the new DB block in init or should it happen in a background thread? The downside to backgrounding it is that `getrawtransaction` would return an error message saying the txindex is syncing while the migration is running.
### Impact
In a sample size n=1 test where I synced nodes from scratch, the average time [Index writing](https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L1903) was 3.36ms in master and 1.72ms in this branch. The average time between `UpdateTip` log lines for sequential blocks between 400,000 and IBD end on mainnet was 0.297204s in master and 0.286134s in this branch. Most likely this is just variance in IBD times, but I can try with some more trials if people want.
Tree-SHA512: 451fd7d95df89dfafceaa723cdf0f7b137615b531cf5c5035cfb54e9ccc2026cec5ac85edbcf71b7f4e2f102e36e9202b8b3a667e1504a9e1a9976ab1f0079c4
* build: Add cmake as depends package (cmake.mk)
The bls-signatures library requires cmake 3.14
* depends: Update chia_bls to version 1.0.0 of dashpay/bls-signatures
* depends: Rename package chia_bls to bls-dash
* depends: Disable blspy/tests/benchmarks build for chia_bls
Note: Building with tests would require the following:
depends: Fix macOS build for versions < 10.12
Seems like older versions of macOS pretend to support c++17 std libs but do not have/have issues with uncaught_exceptions. "Catch", the testing framework used in the bls lib wants those by default but setting `DCATCH_CONFIG_NO_CPP17_UNCAUGHT_EXCEPTIONS` disables them which should be just fine to do in all cases here.
---
depends/packages/chia_bls.mk | 2 ++
1 file changed, 2 insertions(+)
diff --git a/depends/packages/chia_bls.mk b/depends/packages/chia_bls.mk
index e79a85844f..25749b74ab 100644
--- a/depends/packages/chia_bls.mk
+++ b/depends/packages/chia_bls.mk
@@ -28,6 +28,8 @@ define $(package)_set_vars
$(package)_config_opts_darwin+= -DCMAKE_AR="$(host_prefix)/native/bin/$($(package)_ar)"
$(package)_config_opts_darwin+= -DCMAKE_RANLIB="$(host_prefix)/native/bin/$($(package)_ranlib)"
endif
+
+ $(package)_cppflags+= -DCATCH_CONFIG_NO_CPP17_UNCAUGHT_EXCEPTIONS
endef
define $(package)_config_cmds
--
* depends: Drop obsolete variable
* bls: Integrate the upgraded version into the codebase
* depends: Pre-fetch relic to fix gitian
Can fetch apt packages only when building with lxc
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
* Merge #13815: Explicitly ignore the return value of DecodeBase58(...)
579497e77a
* Merge #13815: Default to DEFAULT_BLOCK_MIN_TX_FEE if unable to parse arg
7c5bc2a523
* Merge #13815: Add NODISCARD to all {Decode,Parse}[...](...) functions returning bool.
9cc0230cfc
92f1f8b31 Split off key_io_tests from base58_tests (Pieter Wuille)
119b0f85e Split key_io (address/key encodings) off from base58 (Pieter Wuille)
ebfe217b1 Stop using CBase58Data for ext keys (Pieter Wuille)
32e69fa0d Replace CBitcoinSecret with {Encode,Decode}Secret (Pieter Wuille)
Pull request description:
This PR contains some of the changes left as TODO in #11167 (and built on top of that PR). They are not intended for backporting.
This removes the `CBase58`, `CBitcoinSecret`, `CBitcoinExtKey`, and `CBitcoinExtPubKey` classes, in favor of simple `Encode`/`Decode` functions. Furthermore, all Bitcoin-specific logic (addresses, WIF, BIP32) is moved to `key_io.{h,cpp}`, leaving `base58.{h,cpp}` as a pure utility that implements the base58 encoding/decoding logic.
Tree-SHA512: a5962c0ed27ad53cbe00f22af432cf11aa530e3efc9798e25c004bc9ed1b5673db5df3956e398ee2c085e3a136ac8da69fe7a7d97a05fb2eb3be0b60d0479655
Make linter happy
Dashify
Implement {Encode,Decode}Destination without CBitcoinAddress
Import Bech32 C++ reference code & tests
This includes a reformatted version of the Bech32 reference code
(see https://github.com/sipa/bech32/tree/master/ref/c%2B%2B), with
extra documentation.
Convert base58_tests from type/payload to scriptPubKey comparison
Add regtest testing to base58_tests
Implement ConvertBits
A part of "Implement BIP173 addresses and tests"
Make linter happy
