2 Commits

Author	SHA1	Message	Date
MarcoFalke	0de37c265a	Merge #16143: tests: Mark unit test blockfilter_index_initial_sync as non-deterministic ... c061be1e2f tests: Mark unit test blockfilter_index_initial_sync as non-deterministic (practicalswift) Pull request description: Mark unit test `blockfilter_index_tests/blockfilter_index_initial_sync` as non-deterministic. Before this PR: ``` $ contrib/devtools/test_deterministic_coverage.sh 500 [2019-06-04 09:58:57] Measuring coverage, run #1 of 500 [2019-06-04 10:00:33] Measuring coverage, run #2 of 500 [2019-06-04 10:02:19] Measuring coverage, run #3 of 500 The line coverage is non-deterministic between runs. Exiting. The test suite must be deterministic in the sense that the set of lines executed at least once must be identical between runs. This is a necessary condition for meaningful coverage measuring. --- gcovr.run-1.txt 2019-06-04 10:00:33.389059973 +0000 +++ gcovr.run-3.txt 2019-06-04 10:03:45.619491207 +0000 @@ -72,7 +72,7 @@ hash.h 54 33 61% 71,74-77,82,85-89,111,113,128,147-148,175,178-181 httprpc.cpp 120 3 2% 31,34-35,38-40,46,49,52,54,56,58,70,73-74,76,78-79,81,83-84,89,91,94-95,97,99-101,103,106-107,111-112,117-119,121-122,125,128,130,132,134-136,138-139,142,145,148,151-153,156-160,163-166,171,173-175,180-182,185,187,189-190,192,195,198-199,201,203-204,212,215,217,219-222,224,227-228,230,232,237,239-240,243-245,247-251,254,256,259,261-264,266-267 [* 205-206,208-209] httpserver.cpp 312 6 1% 46,49-50,53,55,80-81,90,92-93,96-98,101,104,106-109,111-112,114,118,120-122,126,128-129,153,155,157-158,164,166-178,180,182,184-188,192,194-196,198-199,201-202,204-205,207-208,213,216-221,225,228-232,236-239,243-244,247-254,256-258,264-267,270-271,274,279,281-282,286,288-290,292-293,297,299-300,303-307,309-310,312-317,322-328,330,332,335,339,341-342,346,352-353,355,358,360,364,368-369,375,378,381-384,388-391,393-394,398-400,402,404-406,409,411-412,414,416,426,428-431,433-434,438,440-441,443,445-446,449,451-455,457-459,463-464,466-469,471-473,475-477,479,482,484,487,490-493,496-497,499-500,502,504,506,508-509,511,513-514,517,519,521-522,527,529-533,535,538,540-543,550-555,558,560-562,570,572-574,577-582,585-590,594-597,600,602-604,606-609,611,614,616,619,621,625-626,628-629,631-632,634-635,640,642-643,646,648-651,653,655-656 -index/base.cpp 149 94 63% 20,22-25,28,66,98,102-103,117-118,140-141,145-146,155,163,175,177-178,181-182,184-185,200-201,203,212,214-215,219-221,228-229,234,236,240,243-244,247-249,258-260,262,270,292-294,308-309 [* 263] +index/base.cpp 149 97 65% 20,22-25,28,66,98,102-103,117-118,140-141,145-146,155,163,175,177-178,181-182,184-185,200-201,203,212,214-215,219-221,228-229,234,236,240,243-244,247-249,258-260,262,270,308-309 [* 263] index/base.h 3 2 66% 77 index/blockfilterindex.cpp 199 134 67% 70,79,81,84-88,91,122,139,142,179-181,184-185,188-189,193-194,201-202,207,233,258,262-263,265-266,268,271-272,274,277,279,284,286,288-289,294,301-302,304,322,329,332-333,350,371,373,438,440-441,444,446,449,455-456,459,461,464,466 [* 162-163] index/blockfilterindex.h 4 4 100% @@ -358,7 +358,7 @@ util/validation.cpp 5 1 20% 12,15-17 validation.cpp 2167 808 37% 291,293,297-300,302,330,332,340,348,355-357,359,362,364-365,368,371,380,382-383,385-386,388-389,396,398-402,406-413,415,417,419,422-425,439-440,442-443,446,449,455-458,461-464,467,469-470,472,474,476,492,494-495,502-503,505-507,511-513,515,517,523,526,528,533,535,540,542-544,550,552-556,558-560,564,574,578-583,586,590-591,594-596,601-602,607-608,611-612,616-617,619-621,635-636,638,640,647-648,651,657-658,660-662,665-667,673,675,677-678,682-683,690,693,700-701,703-705,709-710,713-714,716,719-720,724-727,733-735,737-739,741-743,747-748,751-752,754,757-764,771,773-774,776-779,785-788,793-794,796-800,815-816,818-822,825,827,830,835,838-839,841-843,846-848,850,853,859,864-867,875,877-879,884-885,887-891,895,899-900,904-906,908-909,911,930-931,933,936,942,944-950,952,959,962,965-968,972,978,982-984,990-991,994-996,999,1003-1004,1011,1013,1015-1019,1022-1023,1026-1032,1056,1065,1079,1091,1108,1112,1114-1118,1125,1127-1130,1133-1135,1138-1139,1147,1149,1151-1152,1155,1197,1199-1201,1206-1209,1211-1212,1226,1230,1232-1234,1236,1238-1241,1245-1246,1256,1258,1260-1262,1264-1266,1268,1278-1280,1282-1283,1286,1289,1291-1292,1294-1302,1305-1311,1319-1323,1330,1332-1333,1336-1339,1379,1383-1384,1395,1401,1405-1407,1411-1414,1423-1428,1438-1440,1451,1455,1458,1471,1480,1497,1503,1519,1525,1527-1530,1532-1533,1536,1538-1539,1549,1551,1553,1555,1559-1562,1571,1573,1578,1580,1582-1584,1588-1589,1594-1597,1601-1606,1613-1616,1619-1623,1630,1632,1635,1637,1639-1640,1642-1646,1658,1660,1675,1688,1711,1713-1715,1742,1755,1760,1765,1769,1811,1815,1817,1841-1845,1855,1942,1946-1947,1956,1984-1986,1991-1992,1994,1996-1999,2005-2007,2010-2012,2022-2023,2028-2031,2038-2039,2042,2044,2049,2058-2061,2064,2114-2115,2117-2118,2120-2124,2152-2153,2156,2159-2163,2165-2169,2171-2172,2176-2178,2187-2188,2191-2194,2199,2207-2211,2215-2220,2224,2227-2230,2235,2237-2238,2261-2263,2265,2274,2278,2286,2301,2303-2304,2306-2309,2311,2313-2318,2320,2322,2325,2327-2328,2330,2332-2334,2338,2340,2343-2344,2407-2410,2430,2445-2447,2507-2509,2511-2514,2518,2520-2521,2523-2524,2561,2564,2590,2592-2593,2595-2598,2603,2620,2626,2658,2719,2724,2773,2776-2777,2779,2781,2783,2785-2788,2791,2793-2795,2799,2801-2802,2805,2807-2809,2813,2816,2818-2821,2825-2826,2832-2834,2841-2845,2848,2854,2858-2859,2861,2865-2868,2872-2875,2880,2884-2885,2890-2891,2894-2895,2897,2900-2906,2908,2910,2912,2918-2922,2924,2928-2929,2940,3002-3005,3009-3010,3026-3028,3036-3037,3039-3040,3045,3053,3056,3077,3080,3090,3112,3118,3129,3133,3135-3136,3141-3142,3150,3190-3193,3259,3268,3273,3277,3282-3285,3303,3314,3321-3324,3338-3341,3345-3346,3348-3350,3360,3372,3392,3397,3403,3406,3408,3435-3441,3443,3468-3469,3485,3487-3488,3492-3493,3534-3536,3542,3547-3549,3552,3565-3566,3601-3602,3610,3628,3630,3632,3645,3647,3649-3651,3653,3657,3659,3661-3669,3675-3680,3686-3687,3691,3693-3697,3702,3704,3706-3708,3711-3718,3720,3724,3726-3729,3748,3750-3752,3754,3758-3759,3763,3765,3767,3772,3774,3777-3778,3780-3781,3783,3787-3788,3790,3792-3794,3798-3800,3823,3825,3828,3830,3832,3836-3838,3841-3843,3845,3848,3850,3854-3856,3858-3859,3861-3862,3864-3867,3870-3873,3875-3876,3879,3882-3883,3886-3893,3899,3901,3905-3909,3911-3915,3922-3924,3926-3928,3931,3933-3934,3940-3942,3945-3947,3952,3954-3955,3957,3960-3961,3964,3966,3968-3972,3975,3977,3980,3982,3985,3987-3988,3992-3996,3998-4006,4008-4009,4011-4012,4014,4016,4019,4021-4022,4024-4026,4028-4032,4037-4041,4043-4045,4047,4050,4053-4054,4057,4060-4064,4066-4067,4069-4075,4079-4080,4086,4089-4091,4094-4097,4101,4106,4108,4110,4112-4114,4116-4117,4119,4121,4123-4124,4126,4128-4130,4132-4134,4138-4142,4144-4147,4154,4158-4163,4166-4169,4172-4173,4177,4179-4180,4183,4185,4187-4189,4191-4193,4195,4197-4201,4207-4208,4212,4220-4223,4230,4232-4233,4237,4240,4243,4247,4249,4251,4253-4255,4265-4266,4277,4279,4282,4285-4287,4292-4293,4296,4298,4302,4305-4306,4310-4311,4315-4318,4360,4363-4367,4370,4377,4397,4412,4415-4416,4418,4421-4422,4424,4426-4429,4433-4437,4439-4441,4448-4452,4454-4456,4458,4460,4462-4467,4471-4475,4477,4480-4481,4486-4488,4493,4496-4503,4505,4507-4511,4513-4514,4517-4519,4529-4531,4546,4600,4638-4639,4647,4653,4662-4664,4696,4703-4704,4718,4720,4723,4725,4727,4730,4732-4733,4736,4738-4739,4742,4744-4745,4750,4752-4757,4761-4765,4769-4770,4774-4776,4779-4781,4783-4785,4787-4790,4793-4794,4800-4801,4803,4807,4809-4810,4812-4813,4815-4816,4823,4827,4829,4831-4832,4834-4835,4838-4840,4842,4845,4848-4849,4853,4855-4856,4858-4863,4866-4872,4877,4891,4907 [* 1085-1086,1140-1141,1513-1514,2201-2202,2428,3569-3570,4400-4401,4442,4453,4504,4522-4523,4526-4527,4818-4819,4873-4874] validation.h 19 5 26% 338,350-352,356-363,366,484 -validationinterface.cpp 81 50 61% 78-82,85-86,112-113,116,119-120,123-124,126-128,130,133-136,151-153,163-165,169-171 +validationinterface.cpp 83 60 72% 78-82,85-86,112-113,116,133-136,151-153,163-165,169-171 validationinterface.h 9 4 44% 94,105,112,118,135 versionbits.cpp 92 27 29% 33,35-36,38-39,48-50,52-54,56-57,61-62,67-71,73,75-76,80,82-83,91,98,100,102-103,105,109-110,113-118,121-122,124,127,129-130,134,137,141,149,151,153-155,159,177,179,184,194,196,199,201,204,206 [* 26] versionbits.h 1 1 100% @@ -400,5 +400,5 @@ zmq/zmqpublishnotifier.h 5 0 0% 12,31,37,43,49 zmq/zmqrpc.cpp 23 3 13% 16,18,20,23,33-35,37,40-47,51,62,64-65 ------------------------------------------------------------------------------ -TOTAL 52472 7784 14% +TOTAL 52474 7797 14% ------------------------------------------------------------------------------ $ ``` After this PR: ``` $ contrib/devtools/test_deterministic_coverage.sh 500 [2019-06-03 14:45:25] Measuring coverage, run #1 of 500 [2019-06-03 14:48:15] Measuring coverage, run #2 of 500 [2019-06-03 14:50:49] Measuring coverage, run #3 of 500 [2019-06-03 14:52:20] Measuring coverage, run #4 of 500 [2019-06-03 14:53:49] Measuring coverage, run #5 of 500 … [2019-06-04 09:04:58] Measuring coverage, run #496 of 500 [2019-06-04 09:07:42] Measuring coverage, run #497 of 500 [2019-06-04 09:10:32] Measuring coverage, run #498 of 500 [2019-06-04 09:13:26] Measuring coverage, run #499 of 500 [2019-06-04 09:16:32] Measuring coverage, run #500 of 500 Coverage test passed: Deterministic coverage across 500 runs. $ ``` ACKs for commit c061be: Tree-SHA512: 00cd55b4371290d8587ab667c64249bc31d26cc9dc3dd519677eb91ddb9dbc5333dfbdef5e90c7a0d74eecd24757113e7ec3eda836859ddc033b1de715df81b6	2021-07-12 17:11:12 -05:00
MarcoFalke	dd2f85c084	Merge #15296: tests: Add script checking for deterministic line coverage in unit tests ... 43206239a8 tests: Add script checking for deterministic line coverage (practicalswift) Pull request description: Add script checking for deterministic line coverage in unit tests. Context: #14343 ("coverage reports non-deterministic") When the coverage is deterministic this script can be invoked from Travis to guard against regressions, but left inactive for now. Output in case of determinism: ``` $ contrib/test_deterministic_coverage.sh 2 [2019-01-30 20:08:46] Measuring coverage, run #1 of 2 [2019-01-30 20:10:45] Measuring coverage, run #2 of 2 Coverage test passed: Deterministic coverage across 2 runs. ``` Output in case of non-determinism: ``` $ contrib/test_deterministic_coverage.sh 2 [2019-01-30 20:08:46] Measuring coverage, run #1 of 2 [2019-01-30 20:10:45] Measuring coverage, run #2 of 2 The line coverage is non-deterministic between runs. The test suite must be deterministic in the sense that the set of lines executed at least once must be identical between runs. This is a neccessary condition for meaningful coverage measuring. --- gcovr.run-1.txt 2019-01-30 23:14:07.419418694 +0100 +++ gcovr.run-2.txt 2019-01-30 23:15:57.998811282 +0100 @@ -471,7 +471,7 @@ test/crypto_tests.cpp 270 270 100% test/cuckoocache_tests.cpp 142 142 100% test/dbwrapper_tests.cpp 148 148 100% -test/denialofservice_tests.cpp 225 225 100% +test/denialofservice_tests.cpp 225 224 99% 363 test/descriptor_tests.cpp 116 116 100% test/fs_tests.cpp 24 3 12% 14,16-17,19-20,23,25-26,29,31-32,35-36,39,41-42,45-46,49,51-52 test/getarg_tests.cpp 111 111 100% @@ -585,5 +585,5 @@ zmq/zmqpublishnotifier.h 5 0 0% 12,31,37,43,49 zmq/zmqrpc.cpp 21 0 0% 16,18,20,22,33-35,38-45,49,52,56,60,62-63 ------------------------------------------------------------------------------ -TOTAL 61561 27606 44% +TOTAL 61561 27605 44% ------------------------------------------------------------------------------ ``` In this case line 363 of `test/denialofservice_tests.cpp` was executed only in the second run. Non-determinism detected! Tree-SHA512: 03f45590e70a87146f89aa7838beeff0925d7fd303697ff03e0e69f8a5861694be5f0dd10cb0020e3e3d40c9cf662f71dfcd838f6affb31bd5212314e0a4e3a9	2021-07-10 12:10:51 -05:00