Commit Graph

415 Commits

Author SHA1 Message Date
Kittywhiskers Van Gogh
fc8952aa19
build: add libgmp detection, make immer a package (#4311)
* build: detect the presence of libgmp before generating Makefile

* depends: add arximboldi/immer@v0.6.2 as a package and add detection

* depends: remove immer from source tree, build using package only

* Drop immer refs from tools

Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2021-08-10 22:35:55 +03:00
Kittywhiskers Van Gogh
522934703a merge #14978: Factor out PSBT utilities from RPCs for use in GUI code; related refactoring 2021-08-09 12:38:11 +05:30
Kittywhiskers Van Gogh
09ace18f03 merge #21052: Replace fs::unique_path with GetUniquePath(path) calls 2021-08-09 12:38:04 +05:30
Kittywhiskers Van Gogh
600c9f891b partial #14978: Factor BroadcastTransaction out of sendrawtransaction 2021-08-05 16:13:41 +05:30
PastaPastaPasta
edf0552c0c
Merge pull request #4187 from kittywhiskers/scantxout
merge #12196, #13697: Add scantxoutset RPC method
2021-07-25 19:10:11 -05:00
Kittywhiskers Van Gogh
2bdb8540b6 Merge #13697: Support output descriptors in scantxoutset
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2021-07-21 11:58:00 +05:30
Wladimir J. van der Laan
6c75d20277 Merge #16248: Make whitebind/whitelist permissions more flexible
c5b404e8f1973afe071a07c63ba1038eefe13f0f Add functional tests for flexible whitebind/list (nicolas.dorier)
d541fa391844f658bd7035659b5b16695733dd56 Replace the use of fWhitelisted by permission checks (nicolas.dorier)
ecd5cf7ea4c3644a30092100ffc399e30e193275 Do not disconnect peer for asking mempool if it has NO_BAN permission (nicolas.dorier)
e5b26deaaa6842f7dd7c4537ede000f965ea0189 Make whitebind/whitelist permissions more flexible (nicolas.dorier)

Pull request description:

  # Motivation

  In 0.19, bloom filter will be disabled by default. I tried to make [a PR](https://github.com/bitcoin/bitcoin/pull/16176) to enable bloom filter for whitelisted peers regardless of `-peerbloomfilters`.

  Bloom filter have non existent privacy and server can omit filter's matches. However, both problems are completely irrelevant when you connect to your own node. If you connect to your own node, bloom filters are the most bandwidth efficient way to synchronize your light client without the need of some middleware like Electrum.

  It is also a superior alternative to BIP157 as it does not require to maintain an additional index and it would work well on pruned nodes.

  When I attempted to allow bloom filters for whitelisted peer, my proposal has been NACKed in favor of [a more flexible approach](https://github.com/bitcoin/bitcoin/pull/16176#issuecomment-500762907) which should allow node operator to set fine grained permissions instead of a global `whitelisted` attribute.

  Doing so will also make follow up idea very easy to implement in a backward compatible way.

  # Implementation details

  The PR propose a new format for `--white{list,bind}`. I added a way to specify permissions granted to inbound connection matching `white{list,bind}`.

  The following permissions exists:
  * ForceRelay
  * Relay
  * NoBan
  * BloomFilter
  * Mempool

  Example:
  * `-whitelist=bloomfilter@127.0.0.1/32`.
  * `-whitebind=bloomfilter,relay,noban@127.0.0.1:10020`.

  If no permissions are specified, `NoBan | Mempool` is assumed. (making this PR backward compatible)

  When we receive an inbound connection, we calculate the effective permissions for this peer by fetching the permissions granted from `whitelist`  and add to it the permissions granted from `whitebind`.

  To keep backward compatibility, if no permissions are specified in `white{list,bind}` (e.g. `--whitelist=127.0.0.1`) then parameters `-whitelistforcerelay` and `-whiterelay` will add the permissions `ForceRelay` and `Relay` to the inbound node.

  `-whitelistforcerelay` and `-whiterelay` are ignored if the permissions flags are explicitly set in `white{bind,list}`.

  # Follow up idea

  Based on this PR, other changes become quite easy to code in a trivially review-able, backward compatible way:

  * Changing `connect` at rpc and config file level to understand the permissions flags.
  * Changing the permissions of a peer at RPC level.

ACKs for top commit:
  laanwj:
    re-ACK c5b404e8f1973afe071a07c63ba1038eefe13f0f

Tree-SHA512: adfefb373d09e68cae401247c8fc64034e305694cdef104bdcdacb9f1704277bd53b18f52a2427a5cffdbc77bda410d221aed252bc2ece698ffbb9cf1b830577
2021-07-19 19:00:04 -05:00
Wladimir J. van der Laan
38ee2a7a94 Merge #14605: Return of the Banman
18185b57c32d0a43afeca4c125b9352c692923e9 scripted-diff: batch-recase BanMan variables (Carl Dong)
c2e04d37f3841d109c1fe60693f9622e2836cc29 banman: Add, use CBanEntry ctor that takes ban reason (Carl Dong)
1ffa4ce27d4ea6c1067d8984455df97994c7713e banman: reformulate nBanUtil calculation (Carl Dong)
daae598feb034f2f56e0b00ecfb4854d693d3641 banman: add thread annotations and mark members const where possible (Cory Fields)
84fc3fbd0304a7d6e660bf783c84bed2dd415141 scripted-diff: batch-rename BanMan members (Cory Fields)
af3503d903b1a608cd212e2d74b274103199078c net: move BanMan to its own files (Cory Fields)
d0469b2e9386a7a4b268cb9725347e7517acace6 banman: pass in default ban time as a parameter (Cory Fields)
2e56702ecedd83c4b7cb8de9de5c437c8c08e645 banman: pass the banfile path in (Cory Fields)
4c0d961eb0d7825a1e6f8389d7f5545114ee18c6 banman: create and split out banman (Cory Fields)
83c1ea2e5e66b8a83072e3d5ad6a4ced406eb1ba net: split up addresses/ban dumps in preparation for moving them (Cory Fields)
136bd7926c72659dd277a7b795ea17f72e523338 tests: remove member connman/peerLogic in TestingSetup (Cory Fields)
7cc2b9f6786f9bc33853220551eed33ca6b7b7b2 net: Break disconnecting out of Ban() (Cory Fields)

Pull request description:

  **Old English à la Beowulf**
  ```
  Banman wæs bréme    --blaéd wíde sprang--
  Connmanes eafera    Coreum in.
  aéglaéca            léodum forstandan
  Swá bealdode        bearn Connmanes
  guma gúðum cúð      gódum daédum·
  dréah æfter dóme·   nealles druncne slóg
  ```

  **Modern English Translation**
  ```
  Banman was famed              --his renown spread wide--
  Conman's hier,                in Core-land.
  against the evil creature     defend the people
  Thus he was bold,             the son of Connman
  man famed in war,             for good deeds;
  he led his life for glory,    never, having drunk, slew
  ```

  --

  With @theuni's blessing, here is Banman, rebased. Original PR: https://github.com/bitcoin/bitcoin/pull/11457

  --

  Followup PRs:
  1. Give `CNode` a `Disconnect` method ([source](https://github.com/bitcoin/bitcoin/pull/14605#discussion_r248065847))
  2. Add a comment to `std::atomic_bool fDisconnect` in `net.h` that setting this to true will cause the node to be disconnected the next time `DisconnectNodes()` runs ([source](https://github.com/bitcoin/bitcoin/pull/14605#discussion_r248384309))

Tree-SHA512: 9c207edbf577415c22c9811113e393322d936a843d4ff265186728152a67c057779ac4d4f27b895de9729f7a53e870f828b9ebc8bcdab757520c2aebe1e9be35
2021-07-17 22:32:12 -05:00
Kittywhiskers Van Gogh
da33c9619c
partial merge #17398: Update leveldb to 1.22+ (#4230)
* Update to leveldb upstream using subtree merge

* Import crc32c using subtree merge as as 'src/crc32c'

* build: Update build system for new leveldb

Upstream leveldb switched build systems, which means we need to define
a few different values.

Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>

* doc: Add crc32c subtree to developer notes

* test: Add crc32c to subtree check linter

* test: Add crc32c exception to various linters and generation scripts

* build: Add LCOV exception for crc32c

Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>

* build: CRC32C build system integration

Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2021-07-15 15:42:55 -05:00
gabriel-bjg
ee6d2f5b0d
Fix data races in bls_worker and use ctpl_stl queue. (#4240)
Change ctpl implementation to use STL queue & mutex.

Use ctpl synchronized queue instead of boost lockfree queue in bls worker aggregator.

Use smart pointers for memory management of Aggregator and VectorAggregator. With 'delete this;' the objects are prone to data race on the delete operator.

Use smart pointers for memory management of ContributionVerifier.

Pass shared_ptr by value to other threads via worker pool.
2021-07-11 22:25:27 -05:00
MarcoFalke
4a543ac7ea Merge #15581: depends: Make less assumptions about build env
f7696e6183 depends: qt: Don't hardcode pwd path (Carl Dong)
89bee1bdbf depends: tar: Always extract as yourself (Carl Dong)
340ef50772 depends: Defer to Python detected by autoconf (Carl Dong)

Pull request description:

  Removes some implicit assumptions that the depends system has about its environment and, as a side-effect, makes it possible to build the depends tree under severely privilege-limited environments such as containers built by Guix.

Tree-SHA512: e8618f9310a0deae864b44f9b60baa29e6225ba16817973ff7830b55798ebd4343aa06da6c1f92682a7afb709d26f80d6ee794a139d4d44c27caf4f0c8fe95fc
2021-07-10 12:10:51 -05:00
UdjinM6
bc09637998
Merge pull request #4225 from PastaPastaPasta/backport-triv-pr5
backport: 'trivial' pr5
2021-07-04 00:07:27 +03:00
Wladimir J. van der Laan
2291ab7e59 Merge #14204: build: Move interfaces/* to libbitcoin_server
e0664f7f54aa9cf99ba31ca4bd9de1db0a9310a4 build: Move interfaces/* to libbitcoin_server (Wladimir J. van der Laan)

Pull request description:

  Move interfaces/* from libbitcoin_util to libbitcoin_server.

  Usage of these is shared between `bitcoind` and `bitcoin-qt`. It is unnecessary for them to be linked against the other utilities. Also semantically they belong with the server/node, I think.

Tree-SHA512: f82f3a95d362051c0eb8092520715f77c2c75409d41f9c0fab9a15445ea9e79a2a36b5d00b1f5be09b266584051934a32a8b9b28f783f4d7be78885b4a29a383
2021-07-01 16:54:18 -05:00
Wladimir J. van der Laan
4cd32e43e4
Merge #13235: Break circular dependency: init -> * -> init by extracting shutdown.h
1fabd59e7 Break circular dependency: init -> * -> init by extracting shutdown.h (Ben Woosley)
e62fdfeea Drop unused init.h includes (Ben Woosley)

Pull request description:

  Most includers just wanted to react to pending shutdown.

  This isolates access to `fRequestShutdown` and limits access to the shutdown api functions, including the new `CancelShutdown` for setting it to `false`.

Tree-SHA512: df42f75dfbba163576710e9a67cf1228531fd99d70a2f187bfba0bcc476d6749cf88180a97e66a81bb5b6c3c7f0917de7402d26039ba7b644cb7509b02f7e267
2021-07-02 00:42:18 +03:00
Wladimir J. van der Laan
22e4729496
Merge #13243: Make reusable base class for auxiliary indices
ec3073a274bf7affe1b8c87a10f75d126f5ac027 index: Move index DBs into index/ directory. (Jim Posen)
89eddcd365e9a2218648f5cc5b9f22b28023f50a index: Remove TxIndexDB from public interface of TxIndex. (Jim Posen)
2318affd27de436ddf9d866a4b82eed8ea2e738b MOVEONLY: Move BaseIndex to its own file. (Jim Posen)
f376a4924109af2496b5fd16a787299eb039f1c8 index: Generalize logged statements in BaseIndex. (Jim Posen)
61a1226d87d80234b2be123c5cad07534c318cfb index: Extract logic from TxIndex into reusable base class. (Jim Posen)
e5af5fc6fb4658599b940d1d50853129b31b8766 db: Make reusable base class for index databases. (Jim Posen)
9b0ec1a7f9ffae816fd5ca32ff7e7559640b6f6d db: Remove obsolete methods from CBlockTreeDB. (Jim Posen)

Pull request description:

  This refactors most of the logic in TxIndex into a reusable base class for other indices. There are two commits moving code between files, which may be be more easily reviewed using `git diff --color-moved` (https://blog.github.com/2018-04-05-git-217-released/).

  The motivation for this is to support BIP 157 by indexing block filters.

  <!-- Reviewable:start -->
  ---
  This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/bitcoin/bitcoin/13243)
  <!-- Reviewable:end -->

Tree-SHA512: 0857f04df2aa920178dab2eb8e57984d8eb4d5010deca9971190358479e05b6672ccca2a08af0a7ac9fe02afb947be84cf35a3693204d0667263c6add2959cbf
2021-07-02 00:42:14 +03:00
Wladimir J. van der Laan
9276c78497
Merge #15391: Add compile time verification of assumptions we're currently making implicitly/tacitly
7cee85807c4db679003c6659d247a2fe74c2464a Add compile time verification of assumptions we're currently making implicitly/tacitly (practicalswift)

Pull request description:

  Add compile time verification of assumptions we're currently making implicitly/tacitly.

  As suggested by @sipa in https://github.com/bitcoin/bitcoin/pull/14239#issuecomment-462508012 and @MarcoFalke in https://github.com/bitcoin/bitcoin/pull/14479#issuecomment-462534878.

Tree-SHA512: e68fe51164dbd3eeb76aa8a7e83dfcd3b4d5a66037c0f1822bbbd189bbe3c280e03b3b10af870880ecc09b612e62fb3d9bcd6cf1e16cb7ba818c257db0712ce4
2021-06-28 19:05:20 -05:00
Kittywhiskers Van Gogh
f946c68f83 partial bitcoin#21966: Add platform-independent float encoder/decoder 2021-06-27 12:03:15 +05:30
Kittywhiskers Van Gogh
4d6d3eb5a7 merge bitcoin#17073: PASTE macros to util/macros.h 2021-06-27 12:03:15 +05:30
Kittywhiskers Van Gogh
138997c8e6 partial bitcoin#15638: Add several util units 2021-06-27 12:03:15 +05:30
Kittywhiskers Van Gogh
680319643f merge bitcoin#14074: Use std::unordered_set instead of set in blockfilter interface 2021-06-27 12:03:15 +05:30
Kittywhiskers Van Gogh
46151acdb5 partial bitcoin#16889: Add some general std::vector utility functions 2021-06-27 12:03:14 +05:30
Kittywhiskers Van Gogh
ef4dfa8524 merge bitcoin#14555: Move util files to directory
(script modified to account for Dash backports, doesn't account for rebasing)

------------- BEGIN SCRIPT ---------------
mkdir -p src/util
git mv src/util.h src/util/system.h
git mv src/util.cpp src/util/system.cpp
git mv src/utilmemory.h src/util/memory.h
git mv src/utilmoneystr.h src/util/moneystr.h
git mv src/utilmoneystr.cpp src/util/moneystr.cpp
git mv src/utilstrencodings.h src/util/strencodings.h
git mv src/utilstrencodings.cpp src/util/strencodings.cpp
git mv src/utiltime.h src/util/time.h
git mv src/utiltime.cpp src/util/time.cpp
git mv src/utilasmap.h src/util/asmap.h
git mv src/utilasmap.cpp src/util/asmap.cpp
git mv src/utilstring.h src/util/string.h
git mv src/utilstring.cpp src/util/string.cpp

gsed -i 's/<util\.h>/<util\/system\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp')
gsed -i 's/<utilmemory\.h>/<util\/memory\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp')
gsed -i 's/<utilmoneystr\.h>/<util\/moneystr\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp')
gsed -i 's/<utilstrencodings\.h>/<util\/strencodings\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp')
gsed -i 's/<utiltime\.h>/<util\/time\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp')
gsed -i 's/<utilasmap\.h>/<util\/asmap\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp')
gsed -i 's/<utilstring\.h>/<util\/string\.h>/g' $(git ls-files 'src/*.h' 'src/*.cpp')

gsed -i 's/BITCOIN_UTIL_H/BITCOIN_UTIL_SYSTEM_H/g' src/util/system.h
gsed -i 's/BITCOIN_UTILMEMORY_H/BITCOIN_UTIL_MEMORY_H/g' src/util/memory.h
gsed -i 's/BITCOIN_UTILMONEYSTR_H/BITCOIN_UTIL_MONEYSTR_H/g' src/util/moneystr.h
gsed -i 's/BITCOIN_UTILSTRENCODINGS_H/BITCOIN_UTIL_STRENCODINGS_H/g' src/util/strencodings.h
gsed -i 's/BITCOIN_UTILTIME_H/BITCOIN_UTIL_TIME_H/g' src/util/time.h
gsed -i 's/BITCOIN_UTILASMAP_H/BITCOIN_UTIL_ASMAP_H/g' src/util/asmap.h
gsed -i 's/BITCOIN_UTILSTRING_H/BITCOIN_UTIL_STRING_H/g' src/util/string.h

gsed -i 's/ util\.\(h\|cpp\)/ util\/system\.\1/g' src/Makefile.am
gsed -i 's/utilmemory\.\(h\|cpp\)/util\/memory\.\1/g' src/Makefile.am
gsed -i 's/utilmoneystr\.\(h\|cpp\)/util\/moneystr\.\1/g' src/Makefile.am
gsed -i 's/utilstrencodings\.\(h\|cpp\)/util\/strencodings\.\1/g' src/Makefile.am
gsed -i 's/utiltime\.\(h\|cpp\)/util\/time\.\1/g' src/Makefile.am
gsed -i 's/utilasmap\.\(h\|cpp\)/util\/asmap\.\1/g' src/Makefile.am
gsed -i 's/utilstring\.\(h\|cpp\)/util\/string\.\1/g' src/Makefile.am

gsed -i 's/-> util ->/-> util\/system ->/' test/lint/lint-circular-dependencies.sh
gsed -i 's/src\/util\.cpp/src\/util\/system\.cpp/g' test/lint/lint-format-strings.py test/lint/lint-locale-dependence.sh
gsed -i 's/src\/utilmoneystr\.cpp/src\/util\/moneystr\.cpp/g' test/lint/lint-locale-dependence.sh
gsed -i 's/src\/utilstrencodings\.\(h\|cpp\)/src\/util\/strencodings\.\1/g' test/lint/lint-locale-dependence.sh
------------- END   SCRIPT ---------------
2021-06-27 12:03:13 +05:30
UdjinM6
11796986df
Merge pull request #4200 from Munkybooty/backports-0.18-pr1
Backports 0.18 pr1
2021-06-26 16:10:09 +03:00
UdjinM6
c5cc285d0e
Merge pull request #4191 from kittywhiskers/checkqueue
partial #15842, merge #15849, #17342, #18710: Add local thread pool to CCheckQueue
2021-06-26 16:08:02 +03:00
Kittywhiskers Van Gogh
3f8caf00bc merge #15849: Thread names in logs and deadlock debug tools
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2021-06-25 12:25:10 +05:30
Wladimir J. van der Laan
82659a4eb8 Merge #12257: [wallet] Use destination groups instead of coins in coin select
232f96f5c8a3920c09db92f4dbac2ad7d10ce8cf doc: Add release notes for -avoidpartialspends (Karl-Johan Alm)
e00b4699cc6d2ee5697d38dd6607eb2631c9b77a clean-up: Remove no longer used ivars from CInputCoin (Karl-Johan Alm)
43e04d13b1ffc02b1082176e87f420198b40c7b1 wallet: Remove deprecated OutputEligibleForSpending (Karl-Johan Alm)
0128121101fb3ee82f3abd3973a967a4226ffe0e test: Add basic testing for wallet groups (Karl-Johan Alm)
59d6f7b4e2f847ec1f2ff46c84e6157655984f85 wallet: Switch to using output groups instead of coins in coin selection (Karl-Johan Alm)
87ebce25d66952f5ce565bb5130dcf5e24049872 wallet: Add output grouping (Karl-Johan Alm)
bb629cb9dc567cc819724d9f4852652926e60cbf Add -avoidpartialspends and m_avoid_partial_spends (Karl-Johan Alm)
65b3eda458221644616d0fdd6ba0fe01bdbce893 wallet: Add input bytes to CInputCoin (Karl-Johan Alm)
a443d7a0ca333b0bae63e04b5d476f9ad9c7aeac moveonly: CoinElegibilityFilter into coinselection.h (Karl-Johan Alm)
173e18a289088c6087ba6fac708e322aa63b7a94 utils: Add insert() convenience templates (Karl-Johan Alm)

Pull request description:

  This PR adds an optional (off by default) `-avoidpartialspends` flag, which changes coin select to use output groups rather than outputs, where each output group corresponds to all outputs with the same destination.

  It is a privacy improvement, as each time you spend some output, any other output that is publicly associated with the destination (address) will also be spent at the same time, at the cost of fee increase for cases where coin select without group restriction would find a more optimal set of coins (see example below).

  For regular use without address reuse, this PR should have no effect on the user experience whatsoever; it only affects users who, for some reason, have multiple outputs with the same destination (i.e. address reuse).

  Nodes with this turned off will still try to avoid partial spending, if the fee of the resulting transaction is not greater than the fee of the original transaction.

  Example: a node has four outputs linked to two addresses `A` and `B`:

  * 1.0 btc to `A`
  * 0.5 btc to `A`
  * 1.0 btc to `B`
  * 0.5 btc to `B`

  The node sends 0.2 btc to `C`. Without `-avoidpartialspends`, the following coin selection will occur:
  * 0.5 btc to `A` or `B` is picked
  * 0.2 btc is output to `C`
  * 0.3 - fee is output to (unique change address)

  With `-avoidpartialspends`, the following will instead happen:
  * Both of (0.5, 1.0) btc to `A` or `B` is picked (one or the other pair)
  * 0.2 btc is output to `C`
  * 1.3 - fee is output to (unique change address)

  As noted, the pro here is that, assuming nobody sends to the address after you spend from it, you will only ever use one address once. The con is that the transaction becomes slightly larger in this case, because it is overpicking outputs to adhere to the no partial spending rule.

  This complements #10386, in particular it addresses @luke-jr and @gmaxwell's concerns in https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-300667926 and https://github.com/bitcoin/bitcoin/pull/10386#issuecomment-302361381.

  Together with `-avoidreuse`, this fully addresses the concerns in #10065 I believe.

Tree-SHA512: 24687a4490ba59cf4198ed90052944ff4996653a4257833bb52ed24d058b3e924800c9b3790aeb6be6385b653b49e304453e5d7ff960e64c682fc23bfc447621

# Conflicts:
#	src/Makefile.am
#	src/bench/coin_selection.cpp
#	src/wallet/coincontrol.h
#	src/wallet/coinselection.cpp
#	src/wallet/coinselection.h
#	src/wallet/init.cpp
#	src/wallet/test/coinselector_tests.cpp
#	src/wallet/wallet.cpp
#	src/wallet/wallet.h
#	test/functional/test_runner.py
2021-06-24 13:21:20 -04:00
UdjinM6
988fa6a235
Merge pull request #4196 from kittywhiskers/override
merge bitcoin#18914, #13306, #16424, #13899, #17486, #17880, #18145, #18843, #16710: split warnings out of CXXFLAGS, add more flags
2021-06-23 10:48:28 +03:00
Kittywhiskers Van Gogh
29288dc858 bitcoin#13306: split warnings out of CXXFLAGS 2021-06-17 19:50:41 +05:30
Kittywhiskers Van Gogh
0206ca2dcb bitcoin#16805: add timing information to FlushStateToDisk() 2021-06-17 10:57:37 +05:30
UdjinM6
bcc8b35194
Merge pull request #4178 from UdjinM6/backports-0.17-pr30
Merge #13033: Build txindex in parallel with validation
2021-06-05 22:08:47 +03:00
UdjinM6
b4b24d370c
Merge pull request #4180 from UdjinM6/pre_pr4025_serialization
Various serialization backports
2021-05-29 23:23:19 +03:00
Kittywhiskers Van Gogh
ac863d3955
partial merge #16670: Add Join helper to join a list of strings 2021-05-28 11:29:02 +03:00
Wladimir J. van der Laan
c646686fc0
Merge #13005: Make --enable-debug to pick better options
9e49db2 Make --enable-debug to pick better options (Evan Klitzke)

Pull request description:

  Cherry-picked (and rebased) 94189645e67f364c4445d62e2b00c282d885cbbf from the "up for grabs" PR: "[build] Make --enable-debug pick better options" (#12695).

  See previous review in #12695.

Tree-SHA512: a93cdadcf13e2ef8519acb1ce4f41ce95057a388347bb0a86a5c164dc7d0b0d14d4bb2a466082d5a100b8d50de65c605c40abaed555e8ea77c99e28800a34439
2021-05-25 14:09:36 +03:00
Pieter Wuille
7ff6515c88
Merge #13033: Build txindex in parallel with validation
9b2704777c [doc] Include txindex changes in the release notes. (Jim Posen)
ed77dd6b30 [test] Simple unit test for TxIndex. (Jim Posen)
6d772a3d44 [rpc] Public interfaces to GetTransaction block until synced. (Jim Posen)
a03f804f2a [index] Move disk IO logic from GetTransaction to TxIndex::FindTx. (Jim Posen)
e0a3b80033 [validation] Replace tx index code in validation code with TxIndex. (Jim Posen)
8181db88f6 [init] Initialize and start TxIndex in init code. (Jim Posen)
f90c3a62f5 [index] TxIndex method to wait until caught up. (Jim Posen)
70d510d93c [index] Allow TxIndex sync thread to be interrupted. (Jim Posen)
94b4f8bbb9 [index] TxIndex initial sync thread. (Jim Posen)
34d68bf3a3 [index] Create new TxIndex class. (Jim Posen)
c88bcec93f [db] Migration for txindex data to new, separate database. (Jim Posen)
0cb8303241 [db] Create separate database for txindex. (Jim Posen)

Pull request description:

  I'm re-opening #11857 as a new pull request because the last one stopped loading for people

  -------------------------------

  This refactors the tx index code to be in it's own class and get built concurrently with validation code. The main benefit is decoupling and moving the txindex into a separate DB. The primary motivation is to lay the groundwork for other indexers that might be desired (such as the [compact filters](https://github.com/bitcoin/bips/pull/636)). The basic idea is that the TxIndex spins up its own thread, which first syncs the txindex to the current block index, then once in sync the BlockConnected ValidationInterface hook writes new blocks.

  ### DB changes

  At the suggestion of some other developers, the txindex has been split out into a separate database. A data migration runs at startup on any nodes with a legacy txindex. Currently the migration blocks node initialization until complete.

  ### Open questions

  - Should the migration of txindex data from the old DB to the new DB block in init or should it happen in a background thread? The downside to backgrounding it is that `getrawtransaction` would return an error message saying the txindex is syncing while the migration is running.

  ### Impact

  In a sample size n=1 test where I synced nodes from scratch, the average time [Index writing](https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L1903) was 3.36ms in master and 1.72ms in this branch. The average time between `UpdateTip` log lines for sequential blocks between 400,000 and IBD end on mainnet was 0.297204s in master and 0.286134s in this branch. Most likely this is just variance in IBD times, but I can try with some more trials if people want.

Tree-SHA512: 451fd7d95df89dfafceaa723cdf0f7b137615b531cf5c5035cfb54e9ccc2026cec5ac85edbcf71b7f4e2f102e36e9202b8b3a667e1504a9e1a9976ab1f0079c4
2021-05-25 13:48:04 +03:00
PastaPastaPasta
b76e7fec1f
Merge pull request #4164 from kittywhiskers/auxprs
Merge #19660, #19373, #19841, #13862, #13866, #17280, #17682 and partial #19326, #14978: Auxiliary Backports
2021-05-21 14:25:27 -05:00
Kittywhiskers Van Gogh
41eaee235e Merge #19841: Implement Keccak and SHA3_256 2021-05-20 10:15:17 +05:30
Kittywhiskers Van Gogh
4b2b5f78d4 Merge #16702: supplying and using asmap to improve IP bucketing 2021-05-20 00:05:09 +05:30
PastaPastaPasta
5c485c5b0b
refactor: Move CCoinJoinClientOptions from coinjoin-client.* (#4066)
* create coinjoin-client-options.cpp/.h

Signed-off-by: pasta <pasta@dashboost.org>

* Drop redundant forward declarations

Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2021-03-27 04:11:33 +01:00
dustinface
9c64708269
bls|depends: Upgrade to bls-signatures version 1.0.0 (#4027)
* build: Add cmake as depends package (cmake.mk)

The bls-signatures library requires cmake 3.14

* depends: Update chia_bls to version 1.0.0 of dashpay/bls-signatures

* depends: Rename package chia_bls to bls-dash

* depends: Disable blspy/tests/benchmarks build for chia_bls

Note: Building with tests would require the following:

depends: Fix macOS build for versions < 10.12

Seems like older versions of macOS pretend to support c++17 std libs but do not have/have issues with uncaught_exceptions. "Catch", the testing framework used in the bls lib wants those by default but setting `DCATCH_CONFIG_NO_CPP17_UNCAUGHT_EXCEPTIONS` disables them which should be just fine to do in all cases here.
---
 depends/packages/chia_bls.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/depends/packages/chia_bls.mk b/depends/packages/chia_bls.mk
index e79a85844f..25749b74ab 100644
--- a/depends/packages/chia_bls.mk
+++ b/depends/packages/chia_bls.mk
@@ -28,6 +28,8 @@ define $(package)_set_vars
     $(package)_config_opts_darwin+= -DCMAKE_AR="$(host_prefix)/native/bin/$($(package)_ar)"
     $(package)_config_opts_darwin+= -DCMAKE_RANLIB="$(host_prefix)/native/bin/$($(package)_ranlib)"
   endif
+
+  $(package)_cppflags+= -DCATCH_CONFIG_NO_CPP17_UNCAUGHT_EXCEPTIONS
 endef

 define $(package)_config_cmds
--

* depends: Drop obsolete variable

* bls: Integrate the upgraded version into the codebase

* depends: Pre-fetch relic to fix gitian

Can fetch apt packages only when building with lxc

Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2021-03-26 15:32:31 +03:00
Kittywhiskers Van Gogh
e20dc9f374
partial merge #13815: Add [[nodiscard]] to all {Decode,Parse}[...](...) functions returning bool (#4035)
* Merge #13815: Explicitly ignore the return value of DecodeBase58(...)

579497e77a

* Merge #13815: Default to DEFAULT_BLOCK_MIN_TX_FEE if unable to parse arg

7c5bc2a523

* Merge #13815: Add NODISCARD to all {Decode,Parse}[...](...) functions returning bool.

9cc0230cfc
2021-03-22 13:10:27 -04:00
dustinface
ae506bae66
refactor: PrivateSend -> CoinJoin + Move the tab (#4038)
* qt|wallet|privatesend: Rename PrivateSend to CoinJoin in GUI strings

* qt: Move CoinJoin next to Transactions

* qt: Adjust status tip of privateSendCoinsMenuAction

Co-authored-by: thephez <thephez@users.noreply.github.com>

* rename: privateSend -> coinJoin

* rename: privatesend -> coinjoin

* rename: PrivateSend -> CoinJoin

* rename: use_ps -> use_cj

* rename: PRIVATESEND -> COINJOIN

* rename: privatesend -> coinjoin for files and folders

* refactor: Re-order coinjoin files in cmake/make files

* refactor: Re-order coinjoin includes where it makes sense

* test: Update lint-circular-dependencies.sh

* Few cleanups

* test: test/coinjoin_tests.cpp -> wallet/test/coinjoin_test.cpp

* s/AdvancedPSUI/AdvancedCJUI/g

* s/privateSentAmountChanged/coinJoinAmountChanged/g

* wallet: Rename "ps_salt" backwards compatible

* Minimal PrivateSend -> CoinJoin migration for settings and cmd-line

* wallet: Fix privatesendrounds -> coinjoinrounds migration

* qt: Migrate nPrivateSendAmount -> nCoinJoinAmount

* `-coinjoindenoms` never existed

* Migrate all PS options/settings

* rpc: Formatting only

* qt: Make Send/CoinJoin tabs a bit more distinguishable

Co-authored-by: thephez <thephez@users.noreply.github.com>
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2021-03-17 18:36:11 -04:00
Wladimir J. van der Laan
70c82570b2
Merge #11372: Address encoding cleanup
92f1f8b31 Split off key_io_tests from base58_tests (Pieter Wuille)
119b0f85e Split key_io (address/key encodings) off from base58 (Pieter Wuille)
ebfe217b1 Stop using CBase58Data for ext keys (Pieter Wuille)
32e69fa0d Replace CBitcoinSecret with {Encode,Decode}Secret (Pieter Wuille)

Pull request description:

  This PR contains some of the changes left as TODO in #11167 (and built on top of that PR). They are not intended for backporting.

  This removes the `CBase58`, `CBitcoinSecret`, `CBitcoinExtKey`, and `CBitcoinExtPubKey` classes, in favor of simple `Encode`/`Decode` functions. Furthermore, all Bitcoin-specific logic (addresses, WIF, BIP32) is moved to `key_io.{h,cpp}`, leaving `base58.{h,cpp}` as a pure utility that implements the base58 encoding/decoding logic.

Tree-SHA512: a5962c0ed27ad53cbe00f22af432cf11aa530e3efc9798e25c004bc9ed1b5673db5df3956e398ee2c085e3a136ac8da69fe7a7d97a05fb2eb3be0b60d0479655

Make linter happy

Dashify
2021-01-08 22:35:34 +03:00
Pieter Wuille
c9936a5115
Partial merge bitcoin#11167: Full BIP173 (Bech32) support
Implement {Encode,Decode}Destination without CBitcoinAddress

Import Bech32 C++ reference code & tests

This includes a reformatted version of the Bech32 reference code
(see https://github.com/sipa/bech32/tree/master/ref/c%2B%2B), with
extra documentation.

Convert base58_tests from type/payload to scriptPubKey comparison

Add regtest testing to base58_tests

Implement ConvertBits

A part of "Implement BIP173 addresses and tests"

Make linter happy
2021-01-08 22:35:13 +03:00
PastaPastaPasta
af25740477
Merge pull request #3884 from UdjinM6/bp9753
Backport some serialization PRs
2020-12-18 11:51:34 -06:00
UdjinM6
6ae4d32b4b
Backport bitcoin#10637 (partial) (#3878)
* Calculate and store the number of bytes required to spend an input

* Store effective value, fee, and long term fee in CInputCoin

Have CInputCOin store effective value information. This includes the effective
value itself, the fee, and the long term fee for the input

* Implement Branch and Bound coin selection in a new file

Create a new file for coin selection logic and implement the BnB algorithm in it.

* Move output eligibility to a separate function

* Use a struct for output eligibility

Instead of specifying 3 parameters, use a struct for those parameters
in order to reduce the number of arguments to SelectCoinsMinConf.

* Remove coinselection.h -> wallet.h circular dependency

Changes CInputCoin to coinselection and to use CTransactionRef in
order to avoid a circular dependency. Also moves other coin selection
specific variables out of wallet.h to coinselectoin.h

* Add tests for the Branch and Bound algorithm

* Move current coin selection algorithm to coinselection.{cpp,h}

Moves the current coin selection algorithm out of SelectCoinsMinConf
and puts it in coinselection.{cpp,h}. The new function, KnapsackSolver,
instead of taking a vector of COutputs, will take a vector of CInputCoins
that is prepared by SelectCoinsMinConf.

* Move original knapsack solver tests to coinselector_tests.cpp

* Add a GetMinimumFeeRate function which is wrapped by GetMinimumFee

* Have SelectCoinsMinConf and SelectCoins use BnB or Knapsack and use it (partial)

Allows SelectCoinsMinConf and SelectCoins be able to switch between
using BnB or Knapsack for choosing coins.

Has SelectCoinsMinConf do the preprocessing necessary to support either
BnB or Knapsack. This includes calculating the filtering the effective
values for each input.

Uses BnB in CreateTransaction to find an exact match for the output.
If BnB fails, it will fallback to the Knapsack solver.

Dash specific note: just always use Knapsack in CreateTransaction.

* Benchmark BnB in the worst case where it exhausts

* Add a test to make sure that negative effective values are filtered

* More of 12747: Fix typos

Co-authored-by: Andrew Chow <achow101-github@achow101.com>
2020-12-18 11:43:48 -06:00
dustinface
122078b9ec
backport: bitcoin#10583 - [RPC] Split part of validateaddress into getaddressinfo (#3880)
* [rpc] split wallet and non-wallet parts of DescribeAddressVisitor

* [rpc] Move DescribeAddressVisitor to rpc/util

* Create getaddressinfo RPC and deprecate parts of validateaddress

Moves the parts of validateaddress which require the wallet into getaddressinfo
which is part of the wallet RPCs. Mark those parts of validateaddress which
require the wallet as deprecated.

Validateaddress will  call getaddressinfo
for the data that both share for right now.

Moves IsMine functions to libbitcoin_common and then links libbitcoin_wallet
before libbitcoin_common in order to prevent linker errors since IsMine is no
longer used in libbitcoin_server.

* scripted-diff: validateaddress to getaddressinfo in tests

Change all instances of validateaddress to getaddressinfo since it seems that
no test actually uses validateaddress for actually validating addresses.

-BEGIN VERIFY SCRIPT-
find ./test/functional -path '*py' -not -path ./test/functional/wallet_disable.py -not -path ./test/functional/rpc_deprecated.py -not -path ./test/functional/wallet_address_types.py -exec sed -i'' -e 's/validateaddress/getaddressinfo/g' {} \;
-END VERIFY SCRIPT-

* wallet: Add missing description of "hdchainid"

* Update src/wallet/rpcwallet.cpp

Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>

Co-authored-by: John Newbery <john@johnnewbery.com>
Co-authored-by: Andrew Chow <achow101-github@achow101.com>
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
2020-12-17 13:46:20 +01:00
Wladimir J. van der Laan
4bb1132438
Merge #12886: Introduce Span type and use it instead of FLATDATA
9272d70 Support serializing Span<unsigned char> and use that instead of FLATDATA (Pieter Wuille)
833bc08 Add Slice: a (pointer, size) array view that acts like a container (Pieter Wuille)

Pull request description:

  Introduce a new data type `Span`, which is an encapsulated pointer + size (like C++20's `std::span` or LevelDB's `Slice`), and represents a view to a sequence of objects laid out continuously in memory.

  The immediate use case is replacing the remaining `FLATDATA` invocations. Instead of those, we support serializing/deserializing unsigned char `Span`s (treating them as arrays).

  A longer term goal for `Span`s is making the script execution operate on them rather than on `CScript` itself. This will allow separate storage mechanisms for scripts.

Tree-SHA512: 7b0da3c802e5df367f223275004d16b04262804c007b7c73fda927176f0a9c3b2ef3225fa842cb73500b0df73175ec1419f1f5239de2402e21dd9ae8e5d05233
2020-12-17 11:21:20 +03:00
Pieter Wuille
a2f107424b
Merge #10579: [RPC] Split signrawtransaction into wallet and non-wallet RPC command
d60234885b Add test for signrawtransaction (Andrew Chow)
eefff65a4b scripted-diff: change signrawtransaction to signrawtransactionwithwallet in tests (Andrew Chow)
1e79c055cd Split signrawtransaction into wallet and non-wallet (Andrew Chow)

Pull request description:

  This PR is part of #10570. It also builds on top of #10571.

  This PR splits `signrawtransaction` into two commands, `signrawtransactionwithkey` and `signrawtransactionwithwallet`. `signrawtransactionwithkey` requires private keys to be passed in and does not use the wallet for any signing. `signrawtransactionwithwallet` uses the wallet to sign a raw transaction and does not have any parameters to take private keys.

  The `signrawtransaction` RPC has been marked as deprecated and will call the appropriate RPC command based upon the parameters given. A test was added to check this behavior is still consistent with the original behavior.

  All tests that used `signrawtransaction` have been updated to use one of the two new RPCs. Most uses were changed to `signrawtransactionwithwallet`. These were changed via a scripted diff.

Tree-SHA512: d0adf5b4cd7077639c504ec07bee262a3b94658d34db0a5c86a263b6393f7aa62f45129eafe29a7c861aa58440dd19348ee0c8b685e8a62d6f4adae8ec8f8cb3
2020-12-15 11:06:25 -06:00
UdjinM6
b559a8f904
Backporting Statoshi and bitcoin#16728 (#2515)
* Backport Statoshi

This backports some of https://github.com/jlopp/statoshi.

Missing stuff: README.md and client name changes, segwit and fee estimation stats.

Fix RejectCodeToString

Fix copy-paste mistake s/InvalidBlockFound/InvalidChainFound/

* Merge #16728: move-only: move coins statistics utils out of RPC

8a3b2eb17572ca2131778d52cc25ec359470a90f move-only: move coins statistics utils out of RPC (James O'Beirne)

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11):

  Parent PR: #15606
  Issue: #15605
  Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal

  ---

  In the short-term, this move-only commit will help with fuzzing (https://github.com/bitcoin/bitcoin/pull/15606#issuecomment-524482297). Later, these procedures will be used to compute statistics (particularly a content hash) for UTXO sets coming in from snapshots.

  Most easily reviewed with `git ... --color-moved=dimmed_zebra`. A nice follow-up would be adding unittests, which I'll do if nobody else gets around to it.

ACKs for top commit:
  MarcoFalke:
    ACK 8a3b2eb17572ca2131778d52cc25ec359470a90f, checked --color-moved=dimmed-zebra

Tree-SHA512: a187d2f7590ad2450b8e8fa3d038c80a04fc3d903618c24222d7e3172250ce51badea35860c86101f2ba266eb4354e6efb8d7d508b353f29276e4665a1efdf74

* Fix 16728

* Modernize StatsdClient

- Reuse some functionality from netbase
- Switch from GetRand to FastRandomContext
- Drop `using namespace std` and add `// namespace statsd`

* Introduce PeriodicStats and make StatsdClient configurable via -stats<smth> (enabled/host/port/ns/period)

* Move/rename tip stats from CheckBlock to ConnectBlock

* Add new false positives to lint-format-strings.py

* Add snprintf in statsd_client to the list of known violations in lint-locale-dependence.sh

* Fix incorrect include guard

* Use bracket syntax includes

* Replace magic numbers with defaults

* Move connection stats calculation into its own function

And bail out early if stats are disabled

* assert in PeriodicStats

Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>

Co-authored-by: MarcoFalke <falke.marco@gmail.com>
Co-authored-by: PastaPastaPasta <6443210+PastaPastaPasta@users.noreply.github.com>
2020-12-15 10:22:23 -06:00
UdjinM6
548cdac180
Bring --enable-stacktraces configure option back (#3826)
Make it possible to disable stacktraces completely again. This is needed for OSes with no backtrace support e.g. Alpine Linux.
2020-12-01 04:18:46 +00:00