Commit Graph

15703 Commits

Author SHA1 Message Date
Alexander Block
55a6182b10 Introduce QSIGSESANN/CSigSesAnn P2P message 2019-02-27 17:33:40 +01:00
Alexander Block
80375a0b4f Change CSigSharesInv and CBatchedSigShares to be sessionId based 2019-02-27 17:33:40 +01:00
Alexander Block
2249413a7c
Merge pull request #2725 from codablock/pr_llmq_hashmaps
Add more caching to CRecoveredSigsDb and use salted hashing for externally provided keys
2019-02-27 17:32:50 +01:00
Alexander Block
9b4285b1c8 Use salted hashing for keys for unordered maps/sets in LLMQ code
We must watch out to not blindly use externally provided keys in unordered
sets/maps, as attackers might find ways to cause unbalanced hash buckets
causing performance degradation.
2019-02-27 14:41:20 +01:00
Alexander Block
b5462f5246 Implement std::unordered_map/set compatible hasher classes for salted hashes
Allows convenient salted hashing with unordered maps and sets. Useful when
there is a risk of unbalanced hash buckets slowing things down, e.g. when
externally supplied hashes are used as keys into a map.
2019-02-27 14:40:52 +01:00
Alexander Block
c52e8402c0 Remove now obsolete TODO comment above CRecoveredSigsDb 2019-02-27 14:40:52 +01:00
Alexander Block
e83e32b956 Add in-memory cache for CRecoveredSigsDb::HasRecoveredSigForHash 2019-02-27 14:40:52 +01:00
Alexander Block
677c0040cd Add in-memory cache to CQuorumBlockProcessor::HasMinedCommitment 2019-02-27 14:40:52 +01:00
Alexander Block
f305cf77b6 Multiple fixes and optimizations for LLMQs and ChainLocks (#2724)
* Indicate success when signing was unnecessary

* Fix typo in name of LLMQ_400_60

* Move RemoveAskFor call for CLSIGs into ProcessNewChainLock

In case we got INV items for the same CLSIG that we recreated through
HandleNewRecoveredSig, (re-)requesting of the CLSIG from other peers
becomes unnecessary.

* Move Cleanup() call in CChainLocksHandler::UpdatedBlockTip up

We bail out early in a few situations from this method, so that Cleanup()
might not be called while its at the bottom.

* Bail out from CChainLocksHandler::UpdatedBlockTip if we already got the CLSIG

* Call RemoveAskFor when QFCOMMITMENT was received

Otherwise we might end up re-requesting it for a very long time when the
commitment INV was received shortly before it got mined.

* Call RemoveSigSharesForSession when a recovered sig is received

Otherwise we end up with session data in node states lingering around until
a fake "timeout" occurs (can be seen in the logs).

* Better handling of false-positive conflicts in CSigningManager

The old code was emitting a lot of messages in logs as it treated sigs
for exactly the same session as a conflict. This commit fixes this by
looking at the signHash before logging.

Also handle a corner-case where a recovered sig might be deleted between
the HasRecoveredSigForId and GetRecoveredSigById call.

* Don't run into session timeout when sig shares come in slow

Instead of just tracking when the first share was received, we now also
track when the last (non-duplicate) share was received. Sessios will now
timeout 5 minutes after the first share arrives, or 1 minute after the last
one arrived.
2019-02-27 16:10:12 +03:00
Pasta
c3eb0c7887 reset file perms 2019-02-26 16:50:25 -06:00
Wladimir J. van der Laan
c17356efcf Merge #9970: Improve readability of segwit.py, smartfees.py
1269b8a Fix logging bug and improve readability of smartfees.py (Suhas Daftuar)
b9f34e8 Improve readability of segwit.py (Suhas Daftuar)

Tree-SHA512: 2c8ff61678c6c407a95a6530e9bd650ae6bb7c9e52f6dd5f256e19253a1358dd1a7aa33a9639fcb07f443e3a21dae71b9f0865c5f1fcaacb2097a3c6766c7eef
2019-02-26 16:41:15 -06:00
Wladimir J. van der Laan
ee6e5654e0 Merge #9505: Prevector Quick Destruct
45a5aaf Only call clear on prevector if it isn't trivially destructible and don't loop in clear (Jeremy Rubin)
aaa02e7 Add prevector destructor benchmark (Jeremy Rubin)

Tree-SHA512: 52bc8163b65b71310252f2d578349d0ddc364a6c23795c5e06e101f5449f04c96cbdca41c0cffb1974b984b8e33006471137d92b8dd4a81a98e922610a94132a
2019-02-26 16:41:13 -06:00
Wladimir J. van der Laan
c4a3cd6a13 Merge #8665: Assert all the things!
4d51e9b Assert ConnectBlock block and pIndex are the same block (NicolasDorier)
972714c pow: GetNextWorkRequired never called with NULL pindexLast (Daniel Cousens)
cc44c8f ContextualCheckBlockHeader should never have pindexPrev to NULL (NicolasDorier)

Tree-SHA512: 7cc568bf9417267c335f21ec3d1505b26e56e5b3d5f4d3dbb555279489800aaa65a3bcd7bc376e274dd102912aec16ddbb18de2e2060b2667b41eb979cd9321e
2019-02-26 16:41:11 -06:00
MarcoFalke
b09e3e080d Merge #9977: QA: getblocktemplate_longpoll.py should always use >0 fee tx
655df06 QA: getblocktemplate_longpoll.py should always use >0 fee tx (Suhas Daftuar)

Tree-SHA512: 016647ba1408f8aaa3fc954835542354d54bf2391c4252c1c505101edbbe1cf8dd6a07060930109d0341f86360b9391dfef439999a93a5bae68fc901dace0b71
2019-02-26 16:41:09 -06:00
Wladimir J. van der Laan
e8df27f8ee Merge #9984: devtools: Make github-merge compute SHA512 from git, instead of worktree
a327e8e devtools: Make github-merge compute SHA512 from git, instead of worktree (Wladimir J. van der Laan)

Tree-SHA512: 22ec7712876be4ab361015a2dd75a09628ec59105ffe3260126f899d8f3ff8666351b65b9a4dfe83f78eb777730442cd0352b155d7f573424f7fc1c4dbc0ddd2
2019-02-26 16:41:07 -06:00
Wladimir J. van der Laan
c55e019bf7 Merge #9940: Fix verify-commits on OSX, update for new bad Tree-SHA512, point travis to different keyservers
df5bae2 Update trusted-sha512-root-commit for new bad tree hash (Matt Corallo)
efc06c2 If GNU sha512sum is missing, try perl shasum in verify-commits (Matt Corallo)
8ed849f Fix travis failing to fetch keys from the sks keyserver pool (Matt Corallo)
fd5e905 Make verify-commits.sh non-recursive (Matt Corallo)

Tree-SHA512: 457cc81d6e0a77ab32d030ecd058c59857f22cb998a1394593e115639081f3fdc74a6376035b77be0712ad5cb9143bc3f498b77e99eb66034492dbbb38c39bc6
2019-02-26 16:41:05 -06:00
Wladimir J. van der Laan
f9a2e4c4fc Merge #9514: release: Windows signing script
09fe2d9 release: update docs to show basic codesigning procedure (Cory Fields)
f642753 release: create a bundle for the new signing script (Cory Fields)
0068361 release: add win detached sig creator and our cert chain (Cory Fields)

Tree-SHA512: 032ad84697c70faaf857b9187f548282722cffca95d658e36413dc048ff02d9183253373254ffcc1158afb71140753f35abfc9fc8781ea5329c04d13c98759c0
2019-02-26 16:41:04 -06:00
Wladimir J. van der Laan
ee2048ae44 Merge #9830: Add safe flag to listunspent result
dcf2112 Add safe flag to listunspent result (NicolasDorier)
af61d9f Add COutput::fSafe member for safe handling of unconfirmed outputs (Russell Yanofsky)

Tree-SHA512: 311edb6fa8075b3ede5b24cb8c6e5d133ccd8ac9ecafea07b604ffa812ee4f071337e31695e662d8573590a0460af20aaaeb39d49c9ea87924449ea50bdfb0b3
2019-02-26 16:41:02 -06:00
MarcoFalke
914bd78770 Merge #9972: Fix extended rpc tests broken by #9768
d055bd6 Fix extended rpc tests broken by 8910b4717e (John Newbery)

Tree-SHA512: 30e019bd7e17013143e419777491ac0efd9760dddac7232d320c9afe80bc2fb383acd331e20cd5b3e21df1177c0e4a5221c99f831e90cf869d3affca206b9728
2019-02-26 16:41:00 -06:00
MarcoFalke
dad8c67d38 Merge #9768: [qa] Add logging to test_framework.py
64c0800 Use logging in individual tests (John Newbery)
38ad281 Use logging in test_framework/comptool.py (John Newbery)
ff19073 Use logging in test_framework/blockstore.py (John Newbery)
2a9c7c7 Use logging in test_framework/util.py (John Newbery)
b0dec4a Remove manual debug settings in qa tests. (John Newbery)
af1363c Always enable debug log and microsecond logging for test nodes. (John Newbery)
6d0e325 Use logging in mininode.py (John Newbery)
553a976 Add logging to p2p-segwit.py (John Newbery)
0e6d23d Add logging to test_framework.py (John Newbery)

Tree-SHA512: 42ee2acbf444ec32d796f930f9f6e272da03c75e93d974a126d4ea9b2dbaa77cc57ab5e63ce3fd33d609049d884eb8d9f65272c08922d10f8db69d4a60ad05a3
2019-02-26 15:54:32 -06:00
MarcoFalke
c75d7dc832 Merge #9962: [trivial] Fix typo in rpc/protocol.h
9ea2490 [trival] Fix typo introduced into rpc/protocol.h in commit 338bf06 (practicalswift)

Tree-SHA512: bfa60dc9f40db867b09e60dbe803db79c86ff939048c91e551c0794a91428bde3aa42c4aabf915c640cd15565005608da10dae051942e806fdf5d28e9704d765
2019-02-26 15:26:01 -06:00
Wladimir J. van der Laan
49b743e39a Merge #9538: [util] Remove redundant call to get() on smart pointer (thread_specific_ptr)
53a2ba3 [util] Remove redundant call to get() on smart pointer (thread_specific_ptr) (practicalswift)

Tree-SHA512: ec8ace4218ae0657d1aaf8ba7682b0cdd140fe0844173058d3cbf4575dee8d9ad5f241e5bacf1e9e4ad3f7dd282f56a8cb0f306de2093f2c0f6333daedd58261
2019-02-26 15:25:32 -06:00
Wladimir J. van der Laan
e5c4a67a20 Merge #9916: Fix msvc compiler error C4146 (minus operator applied to unsigned type)
8e0720b Fix msvc compiler error C4146 (unary minus operator applied to unsigned type) (kobake)
292112f Fix msvc compiler error C4146 (minus operator applied to unsigned type) (kobake)

Tree-SHA512: 25f408daf7bf9ffe4b8b4bd62f6f6d326219189a9faf8f8c0a135c5a0cb0511af765aa2b6087a091c8863c701289bda49a2379b00cd9b10854d316a5c3fc3f8e
2019-02-26 15:24:31 -06:00
Nathan Marley
fcd3b4fd49 Disallow new proposals using legacy serialization (#2722)
*  add flag to allow legacy proposal format

* add proposal validator ctor flag for legacy format

* add test for legacy proposal format disabled
2019-02-26 13:44:43 +03:00
UdjinM6
668b84b1e4 Fix stacktraces compilation issues (#2721)
* Drop linux/limits.h, we don't use it anyway

* Do not require backtrace lib when enable_stacktraces=no
2019-02-26 07:01:56 +01:00
UdjinM6
0fd1fb7d57
Don't build docker image when running Travis job on some another repo (#2718) 2019-02-21 21:44:47 +03:00
Alexander Block
48d92f116e Implement optional pretty printed stacktraces (#2420)
* Add libbacktrace to depends

This is currently only useful to extract symbols. It fails to gather
stacktraces when compiled with MinGW, so we can only use it to get symbol
information from a stack trace which we gathered outside of libbacktrace.

* Add -mbig-obj to CXXFLAGS for MinGW builds

* Implement stacktraces for C++ exceptions

This is a hack and should only be used for debugging. It works by wrapping
the C++ ABI __wrap___cxa_allocate_exception. The wrapper records a backtrace
and stores it in a global map. Later the stacktrace can be retrieved with
GetExceptionStacktraceStr.

This commit also adds handlers to pretty print uncaught exceptions and
signals.

* Use GetPrettyExceptionStr for all unhandled exceptions

* Use --enable-stacktraces in CI for linux32/linux64

* Register exception translators to pretty print exceptions in unit tests

* Catch and print python exceptions when stopping nodes

Otherwise the code at the bottom is never executed when nodes crash,
leading to no output of debug.log files on Travis.

* Remove now unneeded/unused TestCrash methods
2019-02-21 21:37:15 +03:00
UdjinM6
0b552be20b
Fix file permissions broken in 2682 (#2717) 2019-02-20 12:03:48 +03:00
Duke Leto
74bb23cacf Add link to bugcrowd in issue template (#2716) 2019-02-19 13:05:59 +03:00
UdjinM6
252ee89c34
Implement new algo for quorum connections (#2710)
Instead of trying to manually figure out params for different quorum/ring sizes, connect to nodes at indexes (i+2^k)%n where k: 0..floor(log2(n-1))-1, n: size of the quorum/ring
2019-02-19 13:05:39 +03:00
UdjinM6
a79b928532
Merge pull request #2682 from PastaPastaPasta/backports-0.15-pr6
Backports 0.15 pr6
2019-02-19 13:04:49 +03:00
Alexander Block
104c6e7767 Cleanup successful sessions before doing timeout check (#2712)
Otherwise we get some false-positive timeout messages in logs.
2019-02-18 14:07:57 +03:00
UdjinM6
26db020d17
Separate init/destroy and start/stop steps in LLMQ flow (#2709) 2019-02-17 14:39:43 +03:00
Alexander Block
9f58690322 Avoid using ordered maps in LLMQ signing code (#2708)
* Implement and use SigShareMap instead of ordered map with helper methods

The old implementation was relying on the maps being ordered, which allowed
us to grab all sig shares for the same signHash by doing range queries on
the map. This has the disadvantage of being unnecessarily slow when the
maps get larger. Using an unordered map would be the naive solution, but
then it's not possible to query by range anymore.

The solution now is to have a specialized map "SigShareMap" which is
indexed by "SigShareKey". It's internally just an unordered map, indexed by
the sign hash and another unordered map for the value, indexed by the
quorum member index.

* Only use unordered maps/sets in CSigSharesManager

These are faster when maps/sets get larger.

* Use unorderes sets/maps in CSigningManager
2019-02-17 14:38:56 +03:00
Pasta
bb90eb4bf0 backports-0.15-pr6 code review 2019-02-16 15:09:51 -06:00
Alexander Block
7a192e2e4c Optimize sleeping behavior in CSigSharesManager::WorkThreadMain (#2707)
* Don't sleep in WorkThreadMain when CPU intensive work was done

When the current iteration resulted in CPU intensive work, it's likely that
the next iteration will result in work as well. Do not sleep in that case,
as we're otherwise wasting (unused) CPU resources.

* No matter how fast we process sig shares, always force 100ms between sending

* Apply review suggestions
2019-02-16 19:59:48 +03:00
Alexander Block
feb4e0ac74
Merge pull request #2706 from codablock/pr_llmq_optimizations3
Implement caching in CRecoveredSigsDb
2019-02-16 15:51:24 +01:00
Alexander Block
dae1542a0e
Merge pull request #2705 from codablock/pr_llmq_optimizations2
Optimize LLMQs initialization and (de)serialization of BLS signatures
2019-02-16 15:50:23 +01:00
Alexander Block
01940616f0
Merge pull request #2704 from codablock/pr_llmq_optimizations1
Optimize LLMQs sending of sig shares
2019-02-16 15:49:19 +01:00
Alexander Block
d7bd0954f3 Use pipe() together with fcntl instead of pipe2()
pipe2 is not supported on MacOS
2019-02-15 16:30:42 +01:00
Alexander Block
742a258987 Implement caching in CRecoveredSigsDb
To avoid repeated calls into LevelDB
2019-02-15 15:52:51 +01:00
Alexander Block
500b9c89a7 Use CBLSLazySignature in CBatchedSigShares
This removes the burden on the message handler thread when many sig batches
arrive. The expensive part of deserialization is now performed in the sig
shares worker thread.

This also removes the need for the specialized deserialization of the sig
shares which tried to avoid the malleability check, as CBLSLazySignature does
not perform malleability checks at all.
2019-02-15 15:44:19 +01:00
Alexander Block
02b68885a0 Implement CBLSLazySignature for lazy serialization/deserialization
In some cases it takes too much time to perform full deserialization of
BLS signatures in the message handler thread. Better to just read the
buffer and do the actual deserialization when the signature is needed for
the first time (which is can be in another thread).
2019-02-15 15:44:02 +01:00
Alexander Block
6e8f50aa55 Faster default-initialization of BLS primitives by re-using the null-hash 2019-02-15 15:43:51 +01:00
Alexander Block
c03480d20f Disable optimistic sending when pushing sig share related messages
And instead let the network thread do the actual sending.
2019-02-15 15:20:15 +01:00
Alexander Block
acb87895f8 Implement WakeupSelect() to allow preliminary wakeup after message push
This adds the reading side of a pipe to the read-set when calling select().
Writing to the writing side of the pipe then causes select() to wake up
immediately. Otherwise it would wait for the timeout of 50ms, even if there
is data that could possibly be sent.

This is useful when many messages need are pushed with optimistic send being
disabled. After all messages have been pushed, WakeSelect() can then wakeup
the select() thread and force a re-check for pending data to send.

This is currently only implemented for POSIX compliant systems as we assume
that heavy-load daemons (like masternodes) are usually run on Linux.
2019-02-15 15:19:59 +01:00
Alexander Block
cf29320988 Allow to disable optimistic send in PushMessage()
Profiling has shown that optimistic send causes measurable slowdowns when
many messages are pushed, even if the sockets are non-blocking. Better to
allow disabling of optimistic sending in such cases and let the network
thread do the actual socket calls.
2019-02-15 15:19:36 +01:00
UdjinM6
bedfc262e2 Rework handling of CSigSharesManager worker thread (#2703) 2019-02-15 15:11:50 +01:00
UdjinM6
3e4286a584 Less cs_main locks in quorums (#2702)
* Drop cs_main from CQuorumManager::UpdatedBlockTip()

* CLLMQUtils::IsQuorumActive() shouldn't require cs_main to be held

* Revert comment deletion
2019-02-15 15:11:35 +01:00
Alexander Block
3bbc75fc48 Reintroduce spork15 so that it's relayed by 0.14 nodes (#2701) 2019-02-14 21:24:06 +03:00