86b47fa741408b061ab0bda784b8678bfd7dfa88 speed up Unserialize_impl for prevector (Akio Nakamura)
Pull request description:
The unserializer for prevector uses `resize()` for reserve the area, but it's prefer to use `reserve()` because `resize()` have overhead to call its constructor many times.
However, `reserve()` does not change the value of `_size` (a private member of prevector).
This PR make the logic of read from stream to callback function, and prevector handles initilizing new values with that call-back and ajust the value of `_size`.
The changes are as follows:
1. prevector.h
Add a public member function named 'append'.
This function has 2 params, number of elemenst to append and call-back function that initilizing new appended values.
2. serialize.h
In the following two function:
- `Unserialize_impl(Stream& is, prevector<N, T>& v, const unsigned char&)`
- `Unserialize_impl(Stream& is, prevector<N, T>& v, const V&)`
Make a callback function from each original logic of reading values from stream, and call prevector's `append()`.
3. test/prevector_tests.cpp
Add a test for `append()`.
## A benchmark result is following:
[Machine]
MacBook Pro (macOS 10.13.3/i7 2.2GHz/mem 16GB/SSD)
[result]
DeserializeAndCheckBlockTest => 22% faster
DeserializeBlockTest => 29% faster
[before PR]
# Benchmark, evals, iterations, total, min, max, median
DeserializeAndCheckBlockTest, 60, 160, 94.4901, 0.0094644, 0.0104715, 0.0098339
DeserializeBlockTest, 60, 130, 65.0964, 0.00800362, 0.00895134, 0.00824187
[After PR]
# Benchmark, evals, iterations, total, min, max, median
DeserializeAndCheckBlockTest, 60, 160, 77.1597, 0.00767013, 0.00858959, 0.00805757
DeserializeBlockTest, 60, 130, 49.9443, 0.00613926, 0.00691187, 0.00635527
ACKs for top commit:
laanwj:
utACK 86b47fa741408b061ab0bda784b8678bfd7dfa88
Tree-SHA512: 62ea121ccd45a306fefc67485a1b03a853435af762607dae2426a87b15a3033d802c8556e1923727ddd1023a1837d0e5f6720c2c77b38196907e750e15fbb902
6262915 Add unit test for stale tip checking (Suhas Daftuar)
83df257 Add CConnmanTest to mutate g_connman in tests (João Barbosa)
ac7b37c Connect to an extra outbound peer if our tip is stale (Suhas Daftuar)
db32a65 Track tip update time and last new block announcement from each peer (Suhas Daftuar)
2d4327d net: Allow connecting to extra outbound peers (Suhas Daftuar)
Pull request description:
This is an alternative approach to #11534. Rather than disconnect an outbound peer when our tip looks stale, instead try to connect to an additional outbound peer.
Periodically, check to see if we have more outbound peers than we target (ie if any extra peers are in use), and if so, disconnect the one that least recently announced a new block (breaking ties by choosing the newest peer that we connected to).
Tree-SHA512: 8f19e910e0bb36867f81783e020af225f356451899adfc7ade1895d6d3bd5afe51c83759610dfd10c62090c4fe404efa0283b2f63fde0bd7da898a1aaa7fb281
e065249 Add unit test for outbound peer eviction (Suhas Daftuar)
5a6d00c Permit disconnection of outbound peers on bad/slow chains (Suhas Daftuar)
c60fd71 Disconnecting from bad outbound peers in IBD (Suhas Daftuar)
Pull request description:
The first commit will disconnect an outbound peer that serves us a headers chain with insufficient work while we're in IBD.
The second commit introduces a way to disconnect outbound peers whose chains fall out of sync with ours:
For a given outbound peer, we check whether their best known block (which is known from the blocks they announce to us) has at least as much work as our tip. If it doesn't, we set a 20 minute timeout, and if we still haven't heard about a block with as much work as our tip had when we set the timeout, then we send a single getheaders message, and wait 2 more minutes. If after two minutes their best known block has insufficient work, we disconnect that peer.
We protect 4 of our outbound peers (who provide some "good" headers chains, ie a chain with at least as much work as our tip at some point) from being subject to this logic, to prevent excessive network topology changes as a result of this algorithm, while still ensuring that we have a reasonable number of nodes not known to be on bogus chains.
We also don't require our peers to be on the same chain as us, to prevent accidental partitioning of the network in the event of a chain split. Note that if our peers are ever on a more work chain than our tip, then we will download and validate it, and then either reorg to it, or learn of a consensus incompatibility with that peer and disconnect. This PR is designed to protect against peers that are on a less work chain which we may never try to download and validate.
Tree-SHA512: 2e0169a1dd8a7fb95980573ac4a201924bffdd724c19afcab5efcef076fdbe1f2cec7dc5f5d7e0a6327216f56d3828884f73642e00c8534b56ec2bb4c854a656
2525b972a net: stop both net/net_processing before destroying them (Cory Fields)
80e2e9d0c net: drop unused connman param (Cory Fields)
8ad663c1f net: use an interface class rather than signals for message processing (Cory Fields)
28f11e940 net: pass CConnman via pointer rather than reference (Cory Fields)
Pull request description:
See individual commits.
Benefits:
- Allows us to begin moving stuff out of CNode and into CNodeState (after #10652 and follow-ups)
- Drops boost dependency and overhead
- Drops global signal registration
- Friendlier backtraces
Tree-SHA512: af2038c959dbec25f0c90c74c88dc6a630e6b9e984adf52aceadd6954aa463b6aadfccf979c2459a9f3354326b5077ee02048128eda2a649236fadb595b66ee3
eac64bb7a [qa] Test nMinimumChainWork (Suhas Daftuar)
0311836f6 Allow setting nMinimumChainWork on command line (Suhas Daftuar)
Pull request description:
As discussed briefly here: https://botbot.me/freenode/bitcoin-core-dev/2017-02-28/?msg=81712308&page=4
This adds a hidden command line option for setting `nMinimumChainWork`, which allows us to test this parameter in our functional tests, as well as allowing for niche use cases like syncing nodes that are otherwise disconnected from the network.
See also #10345, which proposes a new use of `nMinimumChainWork`.
Tree-SHA512: fe4d8f4f289697615c98d8760f1cc74c076110310ea0b5b875fcab78c127da9195b4eb84148aebacc7606c246e5773d3f13bd5d9559d0a8bffac20a3a28c62df
bb8376b Verify DBWrapper iterators are taking snapshots (Matt Corallo)
Pull request description:
The LevelDB docs seem to indicate that an iterator will not take
snapshots (even providing instructions on how to do so yourself).
In several of the places we use them, we assume snapshots to have
been taken.
In order to make sure LevelDB doesn't change out from under us
(and to prevent the next person who reads the docs from having the
same fright I did), verify that snapshots are taken in our tests.
Tree-SHA512: 54f24dabc294962e9c20882f61809604421a661208d1568bb107102248603e8e7c12e929ccb0812a73d4e4f23fea61f1b48e7cc24da5a7260f1d14d89ba88cd6
7a1e873 [script] Unit tests for IsMine (Jim Posen)
d7afe2d [script] Unit tests for script/standard functions (Jim Posen)
Pull request description:
Simply adding unit test coverage.
Tree-SHA512: aaf16b1b07b6d43c884a67f4fd5f83c31bf2c560f78798036d7aa37a3efe71a7ca3c82c4b3ba1f3119bcbe3b78013e64bb0020fe57ebc69aea1cb54943881959
d1138e362 Remove redundant testutil files (MeshCollider)
Pull request description:
The only function in testutil.cpp, `GetTempPath()` simply called `fs::temp_directory_path()` directly. This just tidies things up by removing that redundant function and the file containing it
I can understand wanting a general util file for tests to use, but if there's nothing in it, we might as well remove it, it can always be added back later when it's put to use.
Tree-SHA512: b923f99acf33328743755368a1aa90f5da4a7d5f61b163a4b0b894275c98db80a91edf8f051fbfb4893d970fda5a9078aae78a2672867ff521c4ca4b653c71c0
e7539f8 Fix some broken init-time prints/constants (Matt Corallo)
13ab353 Check for empty coinsview instead of just-reset coinsview in init (Matt Corallo)
fce3f4f Fix resume-of-reindex-after-restart (Matt Corallo)
efac91e Always wait for threadGroup to exit in bitcoind shutdown (Matt Corallo)
Pull request description:
This is a follow-on to #10758 to help move 10758 along. The first fixes a regression in master that was partially fixed in 10758, the second I'm not sure if its a regression or not, but its clearly a bug that should be fixed.
Tree-SHA512: aca7b97a97dca66e1a218a33cc6f4aa002292ff1bb0af64e35b81fbaa91b9504f2605375808b43e93a63fc73634ad079b30ef6c9f4ba338d3b5f72d816dfeaff
c0025d0 Fix segfault when shutting down before fully loading (Matt Corallo)
1385697 Order chainstate init more logically. (Matt Corallo)
ff3a219 Call RewindBlockIndex even if we're about to run -reindex-chainstate (Matt Corallo)
b0f3249 More user-friendly error message if UTXO DB runs ahead of block DB (Matt Corallo)
eda888e Fix some LoadChainTip-related init-order bugs. (Matt Corallo)
Pull request description:
This does a number of things to clean up chainstate init order,
fixing some issues as it goes:
* Order chainstate init more logically - first all of the
blocktree-related loading, then coinsdb, then
pcoinsTip/chainActive. Only create objects as needed.
* More clearly document exactly what is and isn't called in
-reindex and -reindex-chainstate both with comments noting
calls as no-ops and by adding if guards.
* Move the writing of fTxIndex to LoadBlockIndex - this fixes a
bug introduced in d6af06d68a where
InitBlockIndex was writing to fTxIndex which had not yet been
checked (because LoadChainTip hadn't yet initialized the
chainActive, which would otherwise have resulted in
InitBlockIndex being a NOP), allowing you to modify -txindex
without reindex, potentially corrupting your chainstate!
* Rename InitBlockIndex to LoadGenesisBlock, which is now a more
natural name for it. Also check mapBlockIndex instead of
chainActive, fixing a bug where we'd write the genesis block out
on every start.
* Move LoadGenesisBlock further down in init. This is a more logical
location for it, as it is after all of the blockindex-related
loading and checking, but before any of the UTXO-related loading
and checking.
* Give LoadChainTip a return value - allowing it to indicate that
the UTXO DB ran ahead of the block DB. This just provides a nicer
error message instead of the previous mysterious
assert(!setBlockIndexCandidates.empty()) error.
* Calls ActivateBestChain in case we just loaded the genesis
block in LoadChainTip, avoiding relying on the ActivateBestChain
in ThreadImport before continuing init process.
* Move all of the VerifyDB()-related stuff into a -reindex +
-reindex-chainstate if guard. It couldn't do anything useful
as chainActive.Tip() would be null at this point anyway.
Tree-SHA512: 3c96ee7ed44f4130bee3479a40c5cd99a619fda5e309c26d60b54feab9f6ec60fabab8cf47a049c9cf15e88999b2edb7f16cbe6819e97273560b201a89d90762
f4d00e6 Add a discard_rate (Alex Morcos)
b138585 Remove factor of 3 from definition of dust. (Alex Morcos)
Pull request description:
The definition of dust is redefined to remove the factor of 3.
Dust is redefined to be the value of an output such that it would
cost that value in fees to (create and) spend the output at the dust
relay rate. The previous definition was that it would cost 1/3 of the
value. The default dust relay rate is correspondingly increased to
3000 sat/kB so the actual default dust output value of 546 satoshis
for a non-segwit output remains unchanged. This commit is a refactor
only unless a dustrelayfee is passed on the commandline in which case
that number now needs to be increased by a factor of 3 to get the same
behavior. -dustrelayfee is a hidden command line option.
Note: It's not exactly a refactor due to edge case changes in rounding
as evidenced by the required change to the unit test.
A discard_rate is added which defaults to 10,000 sat/kB
Any change output which would be dust at the discard_rate you are
willing to discard completely and add to fee (as well as continuing to
pay the fee that would have been needed for creating the change)
This would be a nice addition for 0.15 and I think will remain useful for 0.16 with the new coin selection algorithms in discussion, but its not crucial.
It does add translation strings, but we could (should?) avoid that by hiding the option
Tree-SHA512: 5b6f655354d0ab6b8b6cac1e8d1fe3136d10beb15c6d948fb15bfb105155a9d03684c6240624039b3eed6428b7e60e54216cc8b2f90c4600701e39f646284a9b
Signed-off-by: Pasta <pasta@dashboost.org>
1f668b6 Expose if CScheduler is being serviced, assert its not in EmptyQueue (Matt Corallo)
3192975 Flush CValidationInterface callbacks prior to destruction (Matt Corallo)
08096bb Support more than one CScheduler thread for serial clients (Matt Corallo)
2fbf2db Add default arg to CScheduler to schedule() a callback now (Matt Corallo)
cda1429 Give CMainSignals a reference to the global scheduler (Matt Corallo)
3a19fed Make ValidationInterface signals-type-agnostic (Matt Corallo)
ff6a834 Use TestingSetup to DRY qt rpcnestedtests (Matt Corallo)
Tree-SHA512: fab91e34e30b080ed4d0a6d8c1214910e383c45440676e37be61d0bde6ae98d61e8903d22b846e95ba4e73a6ce788798350266feba246d8a2ab357e8523e4ac5
Contains RBF stuff to be removed in a later commit
11590d3 Properly bound check conf_target in wallet RPC calls (Alex Morcos)
fd29d3d Remove checking of mempool min fee from estimateSmartFee. (Alex Morcos)
2fffaa9 Make QT fee displays use GetMinimumFee instead of estimateSmartFee (Alex Morcos)
1983ca6 Use CoinControl to pass custom fee setting from QT. (Alex Morcos)
03ee701 Refactor to use CoinControl in GetMinimumFee and FeeBumper (Alex Morcos)
ecd81df Make CoinControl a required argument to CreateTransaction (Alex Morcos)
Pull request description:
This builds on #10589 (first 5 commits from that PR, last 5 commits are new)
The first couple commits refactor to use the CCoinControl class to pass fee calculation parameters around.
This allows for fixing the buggy interaction in QT between the global payTxFee which can be modified by the RPC call settxfee or temporarily modified by the QT custom fee settings. Before these changes the GUI could sometimes send a transaction with a recently set payTxFee and not respect the settings displayed in the GUI. After these changes, using the GUI does not involve the global transaction confirm target or payTxFee.
The prospective fee displays in the smart fee slider and the coin control dialog are changed to use the fee calculation from GetMinimumFee, this simplifies the code and makes them slightly more correct in edge cases.
Maxing the fee calculation with the mempool min fee is move from estimateSmartFee to GetMinimumFee.
This fixes a long standing bug, and should be tagged for 0.15 as it is holding up finalizing the estimatesmartfee RPC API before release.
Tree-SHA512: 4d36a1bd5934aa62f3806d380fcafbef73e9fe5bdf190fc5259a3e3a13349e5ce796e50e7068c46dc630ccf56d061bce5804f0bfe2e082bb01ca725b63efd4c1
693247b [test] Speed up fuzzing by ~200x when using afl-fuzz (practicalswift)
Tree-SHA512: 95922fc2616b8cb00dd531ed1140a52bbda4e04292dd8c1c60a8f49dbf6ccb797a18b61180b3fb68d695456b478a1f5ae7fda47e8ecee41dd65555487aef40a3
c5ebddd11 Tests: address placement should be deterministic by default (René Nyffenegger)
Pull request description:
Better version of wrong and closed pull request https://github.com/bitcoin/bitcoin/pull/10764
Tree-SHA512: dfda6ea4a9dd0f4c8b96212ad43a716ff1dddf115cd2712a2a7e42c97fc9494079c746906b39d880a9827c05d2b75c728afd4ca4519ce4d365f0dae0c4aec24c
ce07638 doc: Add comment to use ValueFromAmount/AmountFromValue for JSON, not utilmoneystr (Wladimir J. van der Laan)
ec05c50 rpc: Use ValueFromAmount instead of FormatMoney in TxToUniv (Wladimir J. van der Laan)
46347ad rpc: Move ValueFromAmount to core_write (Wladimir J. van der Laan)
dac3782 doc: Correct AmountFromValue/ValueFromAmount names (Wladimir J. van der Laan)
Pull request description:
With this, the amounts returned in `decoderawtransaction` will be padded to 8 digits like anywhere else in the API.
This is accomplished by using `ValueFromAmount` in `TxToUniv`, instead of `FormatMoney` which it currently (mistakingly) uses. The `FormatMoney` function is only for debugging/logging use!
To avoid dependency issues, `ValueFromAmount` is moved to `core_write.cpp`, where it also fits better. I don't move `AmountFromValue` to `core_read.cpp` at the same time, as this would have more impact due to the RPCError dependency there.
(n.b.: large number of changed files is solely due to the util_tests JSONs needing update)
Tree-SHA512: 10fc2d27d33a77dbcb57aa7eccd4f53110c05d38eb7df6d40f10f14c08fad4274472e93af75aa59fe68ad0720fdf0930f0108124abef518e0dd162b3d2b2b292
38bc1ec Make more json-like output from estimaterawfee (Alex Morcos)
2d2e170 Comments and improved documentation (Alex Morcos)
ef589f8 minor cleanup: remove unnecessary variable (Alex Morcos)
3ee76d6 Introduce a scale factor (Alex Morcos)
5f1f0c6 Historical block span (Alex Morcos)
aa19b8e Clean up fee estimate debug printing (Alex Morcos)
10f7cbd Track first recorded height (Alex Morcos)
3810e97 Rewrite estimateSmartFee (Alex Morcos)
c7447ec Track failures in fee estimation. (Alex Morcos)
4186d3f Expose estimaterawfee (Alex Morcos)
2681153 minor refactor: explicitly track start of new bucket range and don't update curNearBucket on final loop. (Alex Morcos)
1ba43cc Make EstimateMedianVal smarter about small failures. (Alex Morcos)
d3e30bc Refactor to update moving average on fly (Alex Morcos)
e5007ba Change parameters for fee estimation and estimates on all 3 time horizons. (Alex Morcos)
c0a273f Change file format for fee estimates. (Alex Morcos)
Tree-SHA512: 186e7508d86a1f351bb656edcd84ee9091f5f2706331eda9ee29da9c8eb5bf67b8c1f2abf6662835560e7f613b1377099054f20767f41ddcdbc89c4f9e78946d
6b8d872 Protect SSE4 code behind a compile-time flag (Pieter Wuille)
fa9be90 Add selftest for SHA256 transform (Pieter Wuille)
c1ccb15 Add SSE4 based SHA256 (Pieter Wuille)
2991c91 Add SHA256 dispatcher (Pieter Wuille)
4d50f38 Support multi-block SHA256 transforms (Pieter Wuille)
Pull request description:
This adds an SSE4 assembly version of the SHA256 transform by Intel, and uses it at run time if SSE4 instructions are available, and use a fallback C++ implementation otherwise. Nearly every x86_64 CPU supports SSE4. The feature is only enabled when compiled with `--enable-experimental-asm`.
In order to avoid build dependencies and other complications, the original Intel YASM code was translated to GCC extended asm syntax.
This gives around a 50% speedup on the SHA256 benchmark for me.
It is based on an earlier patch by @laanwj, though only includes a single assembly version (for now), and removes the YASM dependency.
Tree-SHA512: d31c50695ceb45264291537b93c0d7497670be38edf021ca5402eaa7d4e1e0e1ae492326e28d4e93979d066168129e62d1825e0384b1b906d36f85d93dfcb43c
d0413c670 Use range based for loop (René Nyffenegger)
Pull request description:
Instead of iterating over 0 .. 1 and then deciding on an actual desired
value, use a range based for loop for the desired value.
Tree-SHA512: 0a7a4a80516c9f16cf97fa7d257088b8386360e19b93c4deac3d745b6270ea452c513821686d7d14a159a235763e034f9b14eef222ca15f7eb71c37bd1c2c380
0aadc11fd Avoid dereference-of-casted-pointer (Pieter Wuille)
Pull request description:
And prefer a static_cast to the intended reference type.
Tree-SHA512: e83b20023a4dca6029b46f7040a8a6fd54e1b42112ec0c87c3c3b567ed641de97a9e2335b57a2efb075491f641e5b977bc226a474276bea0c3c3c71d8d6ac54d
bb326add9f38f2a8e5ce5ee29d98ce08038200d8 Add ChaCha20Poly1305@Bitcoin AEAD benchmark (Jonas Schnelli)
99aea045d688059caf89c0e485fa427bd28eddd8 Add ChaCha20Poly1305@Bitcoin tests (Jonas Schnelli)
af5d1b5f4a7b56628a76af21284c258d845894f0 Add ChaCha20Poly1305@Bitcoin AEAD implementation (Jonas Schnelli)
Pull request description:
This adds a new AEAD (authenticated encryption with additional data) construct optimised for small messages (like used in Bitcoins p2p network).
Includes: #15519, #15512 (please review those first).
The construct is specified here.
https://gist.github.com/jonasschnelli/c530ea8421b8d0e80c51486325587c52#ChaCha20Poly1305Bitcoin_Cipher_Suite
This aims for being used in v2 peer-to-peer messages.
ACKs for top commit:
laanwj:
code review ACK bb326add9f38f2a8e5ce5ee29d98ce08038200d8
Tree-SHA512: 15bcb86c510fce7abb7a73536ff2ae89893b24646bf108c6cf18f064d672dbbbea8b1dd0868849fdac0c6854e498f1345d01dab56d1c92031afd728302234686
Add new line
e9d5e975612e828ec44f9247b4c5c08f0268d360 Poly1305: tolerate the intentional unsigned wraparound in poly1305.cpp (Jonas Schnelli)
b34bf302f26c7cede47cc20b3bdfb613c51ab67e Add Poly1305 bench (Jonas Schnelli)
03be7f48fad10aa8da3291c28a185ed750193c7b Add Poly1305 implementation (Jonas Schnelli)
Pull request description:
This adds a currently unused Poly1305 implementation including test vectors from RFC7539.
Required for BIP151 (and related to #15512).
Tree-SHA512: f8c1ad2f686b980a7498ca50c517e2348ac7b1fe550565156f6c2b20faf764978e4fa6b5b1c3777a16e7a12e2eca3fb57a59be9c788b00d4358ee80f2959edb1
2dfe27517 Add ChaCha20 bench (Jonas Schnelli)
2bc2b8b49 Add ChaCha20 encryption option (XOR) (Jonas Schnelli)
Pull request description:
The current ChaCha20 implementation does not support message encryption (it can only output the keystream which is sufficient for the RNG).
This PR adds the actual XORing of the `plaintext` with the `keystream` in order to return the desired `ciphertext`.
Required for v2 message transport protocol.
ACKs for commit 2dfe27:
jnewbery:
Looks good. utACK 2dfe2751713c814aea53b5a7563eb74ad1baea00.
jnewbery:
utACK 2dfe2751713c814aea53b5a7563eb74ad1baea00
sipa:
utACK 2dfe2751713c814aea53b5a7563eb74ad1baea00
ryanofsky:
utACK 2dfe2751713c814aea53b5a7563eb74ad1baea00. Changes since last review are just renaming the Crypt method, adding comments, and simplifying the benchmark.
Tree-SHA512: 84bb234da2ca9fdc44bc29a786d9dd215520f81245270c1aef801ef66b6091b7793e2eb38ad6dbb084925245065c5dce9e5582f2d0fa220ab3e182d43412d5b5
5c643241e [utils] allow square brackets for ipv6 addresses in bitcoin-cli (John Newbery)
fe4fabaf1 [refactor] move SplitHostPort() into utilstrencodings (John Newbery)
Pull request description:
bitcoin-cli's `-rpcconnect` can accept ipv6 addresses (as long as the libevent version is new enough), but fails to parse ipv6 with square brackets. This PR makes `bitcoin-cli` parse ipv6 in square brackets correctly.
`bitcoin-cli -rpcconnect=[::1] <command>`
should now be equivalent to
`bitcoin-cli -rpcconnect=::1 <command>`
This is useful so the `bitcoin-cli` option can now be in the same format as the `bitcoind` option.
Doesn't include tests. I have a branch that fully tests `bitcoin-cli`, but that's queued behind several intermediate PRs.
- first commit moves `SplitHostPort()` from libbitcoin_common into libbitcoin_util
- second commit adds proper ipv6 parsing to bitcoin-cli
Tree-SHA512: 249d409f10360c989474283341f458cc97364a56a7d004ae6d5f13d8bffe3a51b5dc2484d42218848e2d42cd9c0b13a1b92e94ea19b209f7e91c875c208d8409
