Fixes issue #2178 : attacker could penny-flood with invalid-signature
transactions to deduce which addresses belonged to your node.
I'm committing this early for code review; I still need to write up
a test plan.
Executive summary of fix: check all transactions received from the network
for penny-flood rate-limiting before adding to the memory pool. But do NOT
ratelimit transactions added to the memory pool:
- because of blockchain reorgs
- stored in the wallet and added at startup
- sent from the GUI or one of the send* RPC commands (CWallet::CommitTransaction)
The limit-free-transactions code really should be a method on CNode, with
counters per-peer. But that is a bigger change for another day.
* During block verification (when parallelism is requested), script
check actions are stored instead of being executed immediately.
* After every processed transactions, its signature actions are
pushed to a CScriptCheckQueue, which maintains a queue and some
synchronization mechanism.
* Two or more threads (if enabled) start processing elements from
this queue,
* When the block connection code is finished processing transactions,
it joins the worker pool until the queue is empty.
As cs_main is held the entire time, and all verification must be
finished before the block continues processing, this does not reach
the best possible performance. It is a less drastic change than
some more advanced mechanisms (like doing verification out-of-band
entirely, and rolling back blocks when a failure is detected).
The -par=N flag controls the number of threads (1-16). 0 means auto,
and is the default.
-checklevel gets a new meaning:
0: verify blocks can be read from disk (like before)
1: verify (contextless) block validity (like before)
2: verify undo files can be read and have good checksums
3: verify coin database is consistent with the last few blocks
(close to level 6 before)
4: verify all validity rules of the last few blocks
Level 3 is the new default, as it's reasonably fast. As level 3 and
4 are implemented using an in-memory rollback of the database, they
are limited to as many blocks as possible without exceeding the
limits set by -dbcache. The default of -dbcache=25 allows for some
150-200 blocks to be rolled back.
In case an error is found, the application quits with a message
instructing the user to restart with -reindex. Better instructions,
and automatic recovery (when possible) or automatic reindexing are
left as future work.
When a transaction A is in the memory pool, while a transaction B
(which shares an input with A) gets accepted into a block, A was
kept forever in the memory pool.
This commit adds a CTxMemPool::removeConflicts method, which
removes transactions that conflict with a given transaction, and
all their children.
This results in less transactions in the memory pool, and faster
construction of new blocks.
These flags select features to be enabled/disabled during script
evaluation/checking, instead of several booleans passed along.
Currently these flags are defined:
* SCRIPT_VERIFY_P2SH: enable BIP16-style subscript evaluation
* SCRIPT_VERIFY_STRICTENC: enforce strict adherence to pubkey/sig encoding standards.
- remove an unwanted ";" at the end of the ~CCoinsView() destructor
- in FindBlockPos() and FindUndoPos() only call fclose(), is file is open
- fix an error string in the CBlockUndo class
Flushes the blktree/ and coins/ databases, and reindexes the
block chain files, as if their contents was loaded via -loadblock.
Based on earlier work by Jeff Garzik.
Split off CBlockTreeDB and CCoinsViewDB into txdb-*.{cpp,h} files,
implemented by either LevelDB or BDB.
Based on code from earlier commits by Mike Hearn in his leveldb
branch.
To prevent excessive copying of CCoins in and out of the CCoinsView
implementations, introduce a GetCoins() function in CCoinsViewCache
with returns a direct reference. The block validation and connection
logic is updated to require caching CCoinsViews, and exploits the
GetCoins() function heavily.
Use CBlock's vMerkleTree to cache transaction hashes, and pass them
along as argument in more function calls. During initial block download,
this results in every transaction's hash to be only computed once.
During the initial block download (or -loadblock), delay connection
of new blocks a bit, and perform them in a single action. This reduces
the load on the database engine, as subsequent blocks often update an
earlier block's transaction already.
This switches bitcoin's transaction/block verification logic to use a
"coin database", which contains all unredeemed transaction output scripts,
amounts and heights.
The name ultraprune comes from the fact that instead of a full transaction
index, we only (need to) keep an index with unspent outputs. For now, the
blocks themselves are kept as usual, although they are only necessary for
serving, rescanning and reorganizing.
The basic datastructures are CCoins (representing the coins of a single
transaction), and CCoinsView (representing a state of the coins database).
There are several implementations for CCoinsView. A dummy, one backed by
the coins database (coins.dat), one backed by the memory pool, and one
that adds a cache on top of it. FetchInputs, ConnectInputs, ConnectBlock,
DisconnectBlock, ... now operate on a generic CCoinsView.
The block switching logic now builds a single cached CCoinsView with
changes to be committed to the database before any changes are made.
This means no uncommitted changes are ever read from the database, and
should ease the transition to another database layer which does not
support transactions (but does support atomic writes), like LevelDB.
For the getrawtransaction() RPC call, access to a txid-to-disk index
would be preferable. As this index is not necessary or even useful
for any other part of the implementation, it is not provided. Instead,
getrawtransaction() uses the coin database to find the block height,
and then scans that block to find the requested transaction. This is
slow, but should suffice for debug purposes.
Introduce a AllocateFileRange() function in util, which wipes or
at least allocates a given range of a file. It can be overriden
by more efficient OS-dependent versions if necessary.
Block and undo files are now allocated in chunks of 16 and 1 MiB,
respectively.
Change the block storage layer again, this time with multiple files
per block, but tracked by txindex.dat database entries. The file
format is exactly the same as the earlier blk00001.dat, but with
smaller files (128 MiB for now).
The database entries track how many bytes each block file already
uses, how many blocks are in it, which range of heights is present
and which range of dates.
The CTxUndo class encapsulates data necessary to undo the effects of
a transaction on the txout set, namely the previous outputs consumed
by it (script + amount), and potentially transaction meta-data when
it is spent entirely.
The CCoins class represents a pruned set of transaction outputs from
a given transaction. It only retains information about its height in
the block chain, whether it was a coinbase transaction, and its
unspent outputs (script + amount).
It has a custom serializer that has very low redundancy.
Special serializer/deserializer for amount values. It is optimized for
values which have few non-zero digits in decimal representation. Most
amounts currently in the txout set take only 1 or 2 bytes to
represent.
Special serializers for script which detect common cases and encode
them much more efficiently. 3 special cases are defined:
* Pay to pubkey hash (encoded as 21 bytes)
* Pay to script hash (encoded as 21 bytes)
* Pay to pubkey starting with 0x02, 0x03 or 0x04 (encoded as 33 bytes)
Other scripts up to 121 bytes require 1 byte + script length. Above
that, scripts up to 16505 bytes require 2 bytes + script length.
These command are a leftover from send-to-IP transactions, which have been
removed a long time ago.
Also removes CNode::mapRequests and CNode::PushRequests, as these were
only used for the mentioned commands.
- I checked every occurance of strprintf() in the code and used %u, where
unsigned vars are used
- the change to GetByte() was made, as ip is an unsigned char
This fixes two alert system vulnerabilities found by
Sergio Lerner; you could send peers unlimited numbers
of invalid alert message to try to either fill up their
debug.log with messages and/or keep their CPU busy
checking signatures.
Fixed by disconnecting/banning peers if they send 10 or more
bad (invalid/expired/cancelled) alerts.
"Version 2" blocks are blocks that have nVersion=2 and
have the block height as the first item in their coinbase.
Block-height-in-the-coinbase is strictly enforced when
version=2 blocks are a supermajority in the block chain
(750 of the last 1,000 blocks on main net, 51 of 100 for
testnet). This does not affect old clients/miners at all,
which will continue producing nVersion=1 blocks, and
which will continue to be valid.
- If the height is in the first half, start at the genesis block and go up, rather than at the top
- Cache the last lookup and use it as a reference point if it's close to the next request, to make linear lookups always fast
Adds CBlock::CURRENT_VERSION and CTransaction::CURRENT_VERSION
constants, and makes non-CURRENT_VERSION transactions nonstandard.
This will help make future upgrades smoother.
Gets rid of `MainFrameRepaint` in favor of specific update functions that tell the UI exactly what changed.
This improves the efficiency of various handlers. Also fixes problems with mined transactions not showing up until restart.
The following notifications were added:
- `NotifyBlocksChanged`: Block chain changed
- `NotifyKeyStoreStatusChanged`: Wallet status (encrypted, locked) changed.
- `NotifyAddressBookChanged`: Address book entry changed.
- `NotifyTransactionChanged`: Wallet transaction added, removed or updated.
- `NotifyNumConnectionsChanged`: Number of connections changed.
- `NotifyAlertChanged`: New, updated or cancelled alert. As this finally makes it possible for the UI to know when a new alert arrived, it can be shown as OS notification.
These notifications could also be useful for RPC clients. However, currently, they are ignored in bitcoind (in noui.cpp).
Also brings back polling with timer for numBlocks in ClientModel. This value updates so frequently during initial download that the number of signals clogs the UI thread and causes heavy CPU usage. And after initial block download, the value changes so rarely that a delay of half a second until the UI updates is unnoticable.
Rather than storing ftell(3)'s return value -- a long -- in an
unsigned int, we store and check a properly typed temp. Then, assured a
non-negative value, we store in nBlockPosRet.
Works for wallet transactions, memory-pool transaction and block chain
transactions.
Available for all:
* txid
* version
* locktime
* size
* coinbase/inputs/outputs
* confirmations
Available only for wallet transactions:
* amount
* fee
* details
* blockindex
Available for wallet transactions and block chain transactions:
* blockhash
* time
This commit removes the dependency of serialize.h on PROTOCOL_VERSION,
and makes this parameter required instead of implicit. This is much saner,
as it makes the places where changing a version number can have an
influence obvious.
Open database once per "tx" message, rather than multiple times,
in the case of orphan transaction presence.
As a side effect, a now-unused CTransaction::AcceptToMemoryPool()
variant is removed.
All client version information is moved to version.cpp, which optionally
(-DHAVE_BUILD_INFO) includes build.h. build.h is automatically generated
on supporting platforms via contrib/genbuild.sh, using git describe.
The git export-subst attribute is used to put the commit id statically
in version.cpp inside generated archives, and this value is used if no
build.h is present.
The gitian descriptors are modified to use git archive instead of a
copy, to create the src/ directory in the output. This way,
src/src/version.cpp will contain the static commit id. To prevent
gitian builds from getting the "-dirty" marker in their git-describe
generated identifiers, no touching of files or running sed on the
makefile is performed anymore. This does not seem to influence
determinism.
Sometimes a new block arrives in a new chain that was already the
best valid one, but wasn't marked that way. This happens for example
when network rules change to recover after a fork.
In this case, it is not necessary to do the entire reorganisation
inside a single db commit. These can become huge, and exceed the
objects/lockers limits in bdb. This patch limits the blocks the
actual reorganisation is applied to, and adds the next blocks
afterwards in separate db transactions.
This also removes an un-needed sigops-per-byte check when accepting transactions to the memory pool (un-needed assuming only standard transactions are being accepted). And it only counts P2SH sigops after the switchover date.
so it takes a flag for how to interpret OP_EVAL.
Also increased IsStandard size of scriptSigs to 500 bytes, so
a 3-of-3 multisig transaction IsStandard.
OP_EVAL is a new opcode that evaluates an item on the stack as a script.
It enables a new type of bitcoin address that needs an arbitrarily
complex script to redeem.
Replaced all occurrences of #if* __WXMSW__ with WIN32,
and all occurrences of __WXMAC_OSX__ with MAC_OSX, and made
sure those are defined appropriately in the makefile and bitcoin-qt.pro.
getmemorypool [data]
If [data] is not specified, returns data needed to construct a block to work on:
"version" : block version
"previousblockhash" : hash of current highest block
"transactions" : contents of non-coinbase transactions that should be included in the next block
"coinbasevalue" : maximum allowable input to coinbase transaction, including the generation award and transaction fees
"time" : timestamp appropriate for next block
"bits" : compressed target of next block
If [data] is specified, tries to solve the block and returns true if it was successful.
Move CMessageHeader from net.h to protocol.[ch]pp, with the
implementation in the .cpp compilation unit (compiling once is enough).
This commit does *not* and should not modify *any* code, it only moves
it from net.h and splits it across protocol.cpp and protocol.hpp.
Indentation changes aside the closest thing to a modification of code is
the addition of the 'TODO' comment (the execution of which requires code
modifications and thus doesn't belong in this commit).
Signed-off-by: Giel van Schijndel <me@mortis.eu>
Explicitly make these global variables less-global to reduce the maximum
scope of this global state.
In my experience global variables tend to be a major source of bugs. As
such the less accessible they are the less likely they are to be the
source of a bug.
Signed-off-by: Giel van Schijndel <me@mortis.eu>
* A new class CKeyStore manages private keys, and script.cpp depends on access to CKeyStore.
* A new class CWallet extends CKeyStore, and contains all former wallet-specific globals; CWallet depends on script.cpp, not the other way around.
* Wallet-specific functions in CTransaction/CTxIn/CTxOut (GetDebit, GetCredit, GetChange, IsMine, IsFromMe), are moved to CWallet, taking their former 'this' argument as an explicit parameter
* CWalletTx objects know which CWallet they belong to, for convenience, so they have their own direct (and caching) GetDebit/... functions.
* Some code was moved from CWalletDB to CWallet, such as handling of reserve keys.
* Main.cpp keeps a set of all 'registered' wallets, which should be informed about updates to the block chain, and does not have any notion about any 'main' wallet. Function in main.cpp that require a wallet (such as GenerateCoins), take an explicit CWallet* argument.
* The actual CWallet instance used by the application is defined in init.cpp as "CWallet* pwalletMain". rpc.cpp and ui.cpp use this variable.
* Functions in main.cpp and db.cpp that are not used by other modules are marked static.
* The code for handling the 'submitorder' message is removed, as it not really compatible with the idea that a node is independent from the wallet(s) connected to it, and obsolete anyway.
This introduces two new source files, keystore.cpp and wallet.cpp with
corresponding headers. Code is moved from main and db, in a preparation
for a follow-up commit which introduces the classes CWallet and CKeyStore.
Use non-blocking connects, and a select() call to wait a predefined
time (5s by default, but configurable with -timeout) for either
success or failure. This allows much more connections to be tried
per time unit.
Based on a patch by phantomcircuit.
Transactions created with the new minimal fee policy would not be
relayed by the network. Therefore, we separate the minimal fee that
is necessary to relay and to create, leaving the creation one at
the old amount, for now.
When rescanning, if the scanned transaction is already in the wallet, it
is skipped. However, if someone sends a transaction, does not wait for
confirmation, switches wallets, waits for a block that contains his original
transaction, and switches wallets again, a rescan will leave his wallet
transaction (which has no merkle branch, so no confirmations) untouched.
there is no internal modification of any file in this commit
files are moved into directories according to established standards in
sourcecode distribution; these directories contain:
src - Files that are used in constructing the executable binaries,
but are not installed.
doc - Files in HTML and text format that document usage, quirks of
the implementation, and contributor checklists.
locale - Files that contain human language translation of strings
used in the program
contrib - Files contributed from distributions or other third party
implementing scripts and auxiliary programs
