sikkienl/dash
1
0
Fork 0
mirror of https://github.com/dashpay/dash.git synced 2024-12-29 13:59:06 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity
bccd5324ab
dash/src/test/serialize_tests.cpp

158 lines
4.4 KiB
C++
Raw Normal View History

Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
#include "serialize.h"
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
#include <stdint.h>
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
#include <boost/test/unit_test.hpp>
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
using namespace std;
BOOST_AUTO_TEST_SUITE(serialize_tests)
BOOST_AUTO_TEST_CASE(varints)
{
// encode
CDataStream ss(SER_DISK, 0);
CDataStream::size_type size = 0;
for (int i = 0; i < 100000; i++) {
ss << VARINT(i);
size += ::GetSerializeSize(VARINT(i), 0, 0);
BOOST_CHECK(size == ss.size());
}
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
for (uint64_t i = 0; i < 100000000000ULL; i += 999999937) {
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
ss << VARINT(i);
size += ::GetSerializeSize(VARINT(i), 0, 0);
BOOST_CHECK(size == ss.size());
}
// decode
for (int i = 0; i < 100000; i++) {
Fix signed/unsigned comparison warnings
2013-03-07 18:38:25 +01:00
int j = -1;
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
ss >> VARINT(j);
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
Cleanup code using forward declarations. Use misc methods of avoiding unnecesary header includes. Replace int typedefs with int##_t from stdint.h. Replace PRI64[xdu] with PRI[xdu]64 from inttypes.h. Normalize QT_VERSION ifs where possible. Resolve some indirect dependencies as direct ones. Remove extern declarations from .cpp files.
2013-04-13 07:13:08 +02:00
for (uint64_t i = 0; i < 100000000000ULL; i += 999999937) {
uint64_t j = -1;
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
ss >> VARINT(j);
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
}
BOOST_AUTO_TEST_CASE(compactsize)
{
CDataStream ss(SER_DISK, 0);
vector<char>::size_type i, j;
for (i = 1; i <= MAX_SIZE; i *= 2)
{
WriteCompactSize(ss, i-1);
WriteCompactSize(ss, i);
}
for (i = 1; i <= MAX_SIZE; i *= 2)
{
j = ReadCompactSize(ss);
BOOST_CHECK_MESSAGE((i-1) == j, "decoded:" << j << " expected:" << (i-1));
j = ReadCompactSize(ss);
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
}
static bool isCanonicalException(const std::ios_base::failure& ex)
{
Fix unit test error on OSX 10.9 using Apple LLVM v5.0. Before the fix, there were 6 errors such as : serialize_tests.cpp:77: error in "noncanonical": incorrect exception std::ios_base::failure is caught It turns out that ex.what() returns following string instead of "non-canonical ReadCompactSize()" "non-canonical ReadCompactSize(): unspecified iostream_category error" After the fix, unit test passed. The test ran using Apple LLVM v5.0 on OSX 10.9 and the unit test error happened because of different error messages by different compilers. g++ --version on my development environment. ``` Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Target: x86_64-apple-darwin13.0.0 Thread model: posix ```
2013-12-15 15:25:41 +01:00
std::string strExplanatoryString("non-canonical ReadCompactSize()");
return strExplanatoryString == ex.what() ||
// OSX Apple LLVM version 5.0 (OSX 10.9)
strExplanatoryString + ": unspecified iostream_category error" == ex.what();
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
}
Fix unit test error on OSX 10.9 using Apple LLVM v5.0. Before the fix, there were 6 errors such as : serialize_tests.cpp:77: error in "noncanonical": incorrect exception std::ios_base::failure is caught It turns out that ex.what() returns following string instead of "non-canonical ReadCompactSize()" "non-canonical ReadCompactSize(): unspecified iostream_category error" After the fix, unit test passed. The test ran using Apple LLVM v5.0 on OSX 10.9 and the unit test error happened because of different error messages by different compilers. g++ --version on my development environment. ``` Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Target: x86_64-apple-darwin13.0.0 Thread model: posix ```
2013-12-15 15:25:41 +01:00
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
BOOST_AUTO_TEST_CASE(noncanonical)
{
// Write some non-canonical CompactSize encodings, and
// make sure an exception is thrown when read back.
CDataStream ss(SER_DISK, 0);
vector<char>::size_type n;
// zero encoded with three bytes:
ss.write("\xfd\x00\x00", 3);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// 0xfc encoded with three bytes:
ss.write("\xfd\xfc\x00", 3);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// 0xfd encoded with three bytes is OK:
ss.write("\xfd\xfd\x00", 3);
n = ReadCompactSize(ss);
BOOST_CHECK(n == 0xfd);
// zero encoded with five bytes:
ss.write("\xfe\x00\x00\x00\x00", 5);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// 0xffff encoded with five bytes:
ss.write("\xfe\xff\xff\x00\x00", 5);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// zero encoded with nine bytes:
ss.write("\xff\x00\x00\x00\x00\x00\x00\x00\x00", 9);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
2013-08-07 04:21:34 +02:00
// 0x01ffffff encoded with nine bytes:
ss.write("\xff\xff\xff\xff\x01\x00\x00\x00\x00", 9);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
}
Bug fix: CDataStream::GetAndClear() when nReadPos > 0 Changed CDataStream::GetAndClear() to use the most obvious get get and clear instead of a tricky swap(). Added a unit test for CDataStream insert/erase/GetAndClear. Note: GetAndClear() is not performance critical, it is used only by the send-a-message-to-the-network code. Bug was not noticed before now because the send-a-message code never erased from the stream.
2013-10-29 02:16:27 +01:00
BOOST_AUTO_TEST_CASE(insert_delete)
{
// Test inserting/deleting bytes.
CDataStream ss(SER_DISK, 0);
BOOST_CHECK_EQUAL(ss.size(), 0);
ss.write("\x00\x01\x02\xff", 4);
BOOST_CHECK_EQUAL(ss.size(), 4);
char c = (char)11;
// Inserting at beginning/end/middle:
ss.insert(ss.begin(), c);
BOOST_CHECK_EQUAL(ss.size(), 5);
BOOST_CHECK_EQUAL(ss[0], c);
BOOST_CHECK_EQUAL(ss[1], 0);
ss.insert(ss.end(), c);
BOOST_CHECK_EQUAL(ss.size(), 6);
BOOST_CHECK_EQUAL(ss[4], (char)0xff);
BOOST_CHECK_EQUAL(ss[5], c);
ss.insert(ss.begin()+2, c);
BOOST_CHECK_EQUAL(ss.size(), 7);
BOOST_CHECK_EQUAL(ss[2], c);
// Delete at beginning/end/middle
ss.erase(ss.begin());
BOOST_CHECK_EQUAL(ss.size(), 6);
BOOST_CHECK_EQUAL(ss[0], 0);
ss.erase(ss.begin()+ss.size()-1);
BOOST_CHECK_EQUAL(ss.size(), 5);
BOOST_CHECK_EQUAL(ss[4], (char)0xff);
ss.erase(ss.begin()+1);
BOOST_CHECK_EQUAL(ss.size(), 4);
BOOST_CHECK_EQUAL(ss[0], 0);
BOOST_CHECK_EQUAL(ss[1], 1);
BOOST_CHECK_EQUAL(ss[2], 2);
BOOST_CHECK_EQUAL(ss[3], (char)0xff);
// Make sure GetAndClear does the right thing:
CSerializeData d;
ss.GetAndClear(d);
BOOST_CHECK_EQUAL(ss.size(), 0);
}
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
2012-06-15 14:19:11 +02:00
BOOST_AUTO_TEST_SUITE_END()
Reference in New Issue Copy Permalink