faa8afe70b4992267176a97262f59413d1a1196b ci: Re-run wine tests once if they fail (MarcoFalke)
Pull request description:
Works around the intermittent wine issue: https://github.com/bitcoin/bitcoin/issues/21122#issuecomment-776517563
ACKs for top commit:
fanquake:
ACK faa8afe70b4992267176a97262f59413d1a1196b - thanks for following up with this.
Tree-SHA512: cb377a8e62c7fcf38fb5bdd6203af82c6c302c86f7fc219cb756eb7b6d3d4fe4ebf30a36ebba739199ff7814b26256a3701cf38dc5e681b18c2f249cdf6d852e
fa272dfdff711e324baefc9bdbb2bf4b52d99eb0 ci: Properly bump to focal for win cross build (MarcoFalke)
Pull request description:
Fixes#21122
ACKs for top commit:
fanquake:
ACK fa272dfdff711e324baefc9bdbb2bf4b52d99eb0 - Looks good to me. I can see the wine version output, `wine-5.0 (Ubuntu 5.0-3ubuntu1)`, in the [log](https://cirrus-ci.com/task/5743559502462976), and `make check` is running.
Tree-SHA512: cd37462afc5512e00cef5e9e7fd1bb5c43c600e833b30cdea2c1c443dc7b0e68f5f2cbaf9d7b655892059af0a226478211db1dc97c10fe4dcdfe666a784f3afc
Debug logs should not be printed to stdout, stderr exists for it.
Private keys should not be printed to console by activation very general
macros name "ENABLE_DASH_DEBUG".
This macros is used only for hdchain private keys, the location util/system.h
is too general for it.
Functional test "tool_wallet.py" is failed due to unexpected output for tsan.
It seems as easier to remove this logs due to too many issues with it
rather than address all of them.
a47635baad feat: drop symbol check from CI to unify with bitcoin, keep it for qt5 only (Konstantin Akimov)
0c38cc325e fix: drop -static-libstc++ from depends/hosts/{linux,mingw32}.mk which is clang-only (Konstantin Akimov)
14a67ee85e Merge #20182: ci: Build with --enable-werror by default, and document exceptions (MarcoFalke)
Pull request description:
## Issue being fixed or feature implemented
As discovered in #5957:
Jobs tsan/ubsan fails with backport bitcoin#20182
it fails, because both tsan and ubsan jobs use clang but all other jobs use gcc. Somehow, after configure there are set incorrect options:
```
clang++-16 -std=c++17 -c -pipe -static-libstdc++ -O1 -Werror -D_GLIBCXX_DEBUG -D_GLIBCXX_DEBUG_PEDANTIC -D_LIBCPP_DEBUG=1 -I/builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/ conftest.cpp
```
`clang` doesn't support `-static-llibstdc++` which is supposed to be gcc-only, it cause this failure:
```
clang: error: argument unused during compilation: '-static-libstdc++' [-Werror,-Wunused-command-line-argument]
```
This failure make `autoconf` to think that clang doesn't support `-Werror` and fails with this error. So, you can't activate this flag.
## What was done?
Backport bitcoin#20182 and related fixes to make it works
## How Has This Been Tested?
CI now succeed with bitcoin#20182. It means, that -Werror activated for clang; also there are not warnings such as:
```
clang: error: argument unused during compilation: '-static-libstdc++' [-Werror,-Wunused-command-line-argument]
```
https://gitlab.com/dashpay/dash/-/jobs/6494328698
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e tests
- [x] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
Top commit has no ACKs.
Tree-SHA512: f912824eaa1ec7513cda7278a3df9e067b0ab48a2d174b18654c8070aa6544bac33a52f494c1e35b4eab10392c1f26df4663e21d12a4dfff7c0a4a6a01ff9551
2f6fe4e4e9e9e35e713c0a20cf891b023592110a ci: Build with --enable-werror by default, and document exceptions (Hennadii Stepanov)
Pull request description:
This PR prevents introducing of new compiler warnings in the master branch, e.g., #19986, #20162.
ACKs for top commit:
practicalswift:
cr ACK 2f6fe4e4e9e9e35e713c0a20cf891b023592110a: patch looks correct
MarcoFalke:
re-ACK 2f6fe4e4e9 🏏
vasild:
ACK 2f6fe4e
Tree-SHA512: 23b5feb5bc472658c992d882ef61af23496f25adaa19f9c79bfaef5d2db273d44981aa93b1631a7d37cb58755283c1dacf3f2d68e501522d3fa8c965ab646d19
ac24af453d16afd1993fa1f292aa41ae6b16f138 ci: use Ubuntu Focal for macOS cross build (fanquake)
Pull request description:
I had assumed Cirrus was spinning up Docker containers to run the CI,
however we are actaully running on the Cirrus machines themselves. See
`DANGER_RUN_CI_ON_HOST` and in the logs:
```bash
Running on host system without docker wrapper
```
So with this change we will actually be using Focal for the macOS cross build.
Follow up to #21036.
This originally contained Windows changes, and an attempt to get Cirrus running without `DANGER_RUN_CI_ON_HOST`, however that seems non-trival, so Windows changes have been dropped from here for now.
ACKs for top commit:
MarcoFalke:
cr ACK ac24af453d16afd1993fa1f292aa41ae6b16f138
Tree-SHA512: 587ba5acf741bcefecf1bc262fa1177f565ebfa9de56125eca19ed3c7db7b9aabfb96866e9c140681b88cb7015a3ded2bc6b4b1b235543d6f6e9dfc6984d569f
f3ba916e8b5b5ee2a381cef38882671eadb231df lint: ignore gitian keys file for spelling linter (Sebastian Falbesoner)
da289a6c4a0a5e110e301f34f1db57b6d31bcdcc lint: update list of spelling linter false positives (Sebastian Falbesoner)
a0022f1cfbb3d8f1f8f3ff135f854be0cb89643f test: bump codespell linter version to 2.0.0 (Sebastian Falbesoner)
Pull request description:
This small patch updates the ignore list for the spelling linter script (which uses `codespell`), both removing false-positives that are not relevant anymore and adding new ones. As [suggested by jonatack](https://github.com/bitcoin/bitcoin/pull/20762#issuecomment-750889701)~~, whose last name is now also part of the list :)~~. Also changed the linter script to not check the gitian keys file, as [suggested by hebasto](https://github.com/bitcoin/bitcoin/pull/20817#discussion_r550763409). The codespell version used is bumped to most recent version 2.0.0, which is more aware of some terms that were previously needed in the ignorelist for v1.17.1, see https://github.com/bitcoin/bitcoin/pull/20817#issuecomment-753428669.
Running spelling linter on master branch (repeated findings in the same file are removed to keep the output short):
```
$ ./test/lint/lint-spelling.sh
contrib/gitian-keys/keys.txt:16: Atack ==> Attack
doc/developer-notes.md:1284: inout ==> input, in out
doc/psbt.md:122: Asend ==> Ascend, as end
src/bench/verify_script.cpp:27: Keypair ==> Key pair
src/blockencodings.h:30: Unser ==> Under, unset, unsure, user
src/compressor.h:65: Unser ==> Under, unset, unsure, user
src/core_read.cpp:131: presense ==> presence
src/index/disktxpos.h:21: blockIn ==> blocking
src/net_processing.h:67: anounce ==> announce
src/netaddress.h:486: compatiblity ==> compatibility
src/primitives/transaction.h:35: nIn ==> inn, min, bin, nine
src/qt/bitcoinunits.cpp:101: nIn ==> inn, min, bin, nine
src/rpc/blockchain.cpp:2150: nIn ==> inn, min, bin, nine
src/rpc/misc.cpp:198: nIn ==> inn, min, bin, nine
src/script/bitcoinconsensus.cpp:81: nIn ==> inn, min, bin, nine
src/script/bitcoinconsensus.h:63: nIn ==> inn, min, bin, nine
src/script/interpreter.cpp:1279: nIn ==> inn, min, bin, nine
src/script/interpreter.h:222: nIn ==> inn, min, bin, nine
src/script/sign.cpp:17: nIn ==> inn, min, bin, nine
src/script/sign.h:39: nIn ==> inn, min, bin, nine
src/serialize.h:181: Unser ==> Under, unset, unsure, user
src/signet.cpp:142: nIn ==> inn, min, bin, nine
src/test/base32_tests.cpp:17: fo ==> of, for
src/test/base64_tests.cpp:17: fo ==> of, for
src/test/script_tests.cpp:1509: nIn ==> inn, min, bin, nine
src/test/sighash_tests.cpp:27: nIn ==> inn, min, bin, nine
src/test/validation_tests.cpp:78: excercise ==> exercise
src/undo.h:36: Unser ==> Under, unset, unsure, user
src/validation.cpp:1403: nIn ==> inn, min, bin, nine
src/validation.h:255: nIn ==> inn, min, bin, nine
src/wallet/wallet.cpp:1532: nIn ==> inn, min, bin, nine
src/wallet/walletdb.cpp:429: Crypted ==> Encrypted
test/functional/feature_nulldummy.py:63: unnecssary ==> unnecessary
test/functional/wallet_encryption.py:81: crypted ==> encrypted
test/functional/wallet_upgradewallet.py:36: fpr ==> for, far, fps
^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in test/lint/lint-spelling.ignore-words.txt
```
Running spelling linter on PR branch:
```
$ ./test/lint/lint-spelling.sh
src/core_read.cpp:131: presense ==> presence
src/net_processing.h:67: anounce ==> announce
src/netaddress.h:486: compatiblity ==> compatibility
src/test/validation_tests.cpp:78: excercise ==> exercise
src/wallet/walletdb.cpp:429: Crypted ==> Encrypted
test/functional/feature_nulldummy.py:63: unnecssary ==> unnecessary
test/functional/wallet_encryption.py:81: crypted ==> encrypted
^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in test/lint/lint-spelling.ignore-words.txt
```
This list of remaining findings doesn't contain false positives anymore -- the typos are fixed in PR https://github.com/bitcoin/bitcoin/pull/20762.
Happy new year! 🍾
ACKs for top commit:
hebasto:
re-ACK f3ba916e8b5b5ee2a381cef38882671eadb231df, only suggested changes since my [previous](https://github.com/bitcoin/bitcoin/pull/20817#pullrequestreview-560632881) review.
jonatack:
ACK f3ba916e8b5b5ee2a381cef38882671eadb231df I don't know if there are any particular issues with bumping codespell to v2.0.0, but locally running the spelling linter and the cirrus job at https://cirrus-ci.com/task/5004066998714368 both LGTM. Thanks for also verifying and removing the unused words from the ignore list.
Tree-SHA512: e92ae6f16c01d4ff3d54f8c3a0ee95e12741f7bfe031d307a785f5cfd8a80525b16b34275f413b914c4a318f5166f9887399c21f2dad9cc7e9be41647042ef37
3c2478c38522c176e81befd4d991a259b09be063 ci: Print COMMIT_RANGE to the log as it was in Travis CI (Hennadii Stepanov)
c123892c2e47e3706f06820aba2454d494a39564 ci: Drop Travis-specific workaround for shellcheck (Hennadii Stepanov)
10af252d97532843b26505d215f6e975f4b21672 ci: Drop Travis-specific way to set COMMIT_RANGE variable (Hennadii Stepanov)
93504da3a932f33126545ebc9383f695a6efe51e ci: Fix COMMIT_RANGE variable value for PRs (Hennadii Stepanov)
Pull request description:
This PR:
- is a #20658 and #20682 followup
- set the `COMMIT_RANGE` variable correctly for PRs
- cleans up Travis-specific code
- prints COMMIT_RANGE value to the log for convenience as it was in Travis CI
ACKs for top commit:
MarcoFalke:
ACK 3c2478c38522c176e81befd4d991a259b09be063
Tree-SHA512: beb933352b10fd5eb3e66373ddb62439e4f3a03b50fb037ee89fa92c0706cec41d05f2d307f15bb18d1e634e6464f4e123b7e2f88703c8edfd145d8d6eff0b1a
faeb40bee3bca9477785659d89af53cafa2333b5 ci: Install missing lint packages (MarcoFalke)
Pull request description:
The cirrus container is vanilla ubuntu, so we need to install the needed packages
ACKs for top commit:
hebasto:
ACK faeb40bee3bca9477785659d89af53cafa2333b5, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: e56198108e26ea0ba2a344b1b74bc294652f34e9866cca053a25fb1b83bbd87ea40254c340e5e169fdfcbd4dcb39fdc2078b5157ca729a22a9a1792ec514a33e
95487b055328b590ba83f258de9637ab0f9a2f17 doc: Drop mentions of Travis CI as it is no longer used (Hennadii Stepanov)
09d105ef0f8b4b06bf248721a1209c9e16e9db75 ci: Drop travis_fold feature as Travis CI is no longer used (Hennadii Stepanov)
Pull request description:
As Travis CI is no longer used, this PR:
- drops `travis_fold` feature
- drops mentions of Travis CI in docs
ACKs for top commit:
MarcoFalke:
ACK 95487b055328b590ba83f258de9637ab0f9a2f17
Tree-SHA512: 2e259bb8b1e37bcefc1251737bb2716f06ddb57c490010b373825c4e70f42ca38efae69a2f63f21f577d7cee3725b94097bdddbd313f8ebf499281cf97c53cef
4045a6722c884be779e86016313061ac6ff80808 ci: Use cpu=1 for linter (Dhruv Mehta)
739d39022d2959c1114c14a0065daebf4fe812c1 ci: Move linter task to cirrus (Dhruv Mehta)
Pull request description:
Solves #20467: Move linter to Cirrus-CI as Travis-CI.org is shutting down
ACKs for top commit:
MarcoFalke:
ACK 4045a6722c884be779e86016313061ac6ff80808
Tree-SHA512: 9aa7487ac86c91fc68bb584d29134e304dbd46702514a5d47d1ef0de6b877d96d42b7589870fc67ad9a31f5d3a789728446da4418688f336111a9ba0f8de5feb
fa7a4385d08797876de9a05ae8a6fde18e36bd67 ci: Fix doc typos in .cirrus.yml (MarcoFalke)
fa73674738b0b5a271ae387e2ebd76d747e02d91 ci: Run i686 centos ci config on cirrus (MarcoFalke)
fa1f949a4dc2568456f3ba30bbd8ecbc78a423ca ci: Run nowallet ci config on cirrus (MarcoFalke)
Pull request description:
Travis CI Org is shutting down, so move the configs to cirrus ci
ACKs for top commit:
practicalswift:
ACK fa7a4385d08797876de9a05ae8a6fde18e36bd67: patch looks correct
Tree-SHA512: 1b7125c7f0d2288931fb8c5e90345891d5f7c1f00c4af136afbeb36bd2836f72920a1877dacc7be787e2c8d84de2a733c1ca71931e5c101b222c1fea588619b3
0cdf2a77ddfa1d53c6fbd830d557a3f20d7fc365 ci: add tsan debug symbols option (Russell Yanofsky)
9a2f12680b3f00a207f1cdd4e0c50a3c7613aefc ci: Add tsan suppression for race in DatabaseBatch (Hennadii Stepanov)
Pull request description:
Since #19325 was merged, the corresponding change in TSan suppression file gets required.
This PR is:
- an analogous to #19226 and #19450, and
- a temporary workaround for CI fail like https://cirrus-ci.com/task/5741795508224000?command=ci#L4993
ACKs for top commit:
MarcoFalke:
ACK 0cdf2a77ddfa1d53c6fbd830d557a3f20d7fc365
Tree-SHA512: 7832f143887c8a0df99dea03e00694621710378fbe923e3592185fcd3658546a590693b513abffc5ab96e9ef76c9c4bff3330eeee69a0c5dbe7574f34c417220
that's a result of:
contrib/devtools/copyright_header.py update ./
it is not scripted diff, because it works differentlly on my localhost and in CI:
CI doesn't want to use git commit date which is mocked to 30th Dec of 2023
057750c09d0a8331c33966d2cc2285ef82f08af8 ci: Upgrading pip version in macos environment (Tushar Singla)
Pull request description:
During each CI run, in macos native environment, python packages lief and zmq are rebuilt everytime which wastes a lot of resources and time and fixes#22206. The latest version of pip directly fetches pre-built binaries. Through this commit pip version is upgraded in macos environment before installation of these packages.
ACKs for top commit:
MarcoFalke:
cr ACK 057750c09d0a8331c33966d2cc2285ef82f08af8
Tree-SHA512: e61d02e46c8fe6a89119014d025a26aba090f9507d725315680893290f5bbc20a375ef408c71fa8db2f485b44ec91cfa0c140198ca44a9d3e0a57055b6bb9582
77777c8b5e3d0edbe1178d982a778b7cc9964b70 ci: Run windows ci config on cirrus (MarcoFalke)
3333d6942e8c5d823b97ea51ef88c8a0791742c6 ci: Run macos ci config on cirrus (MarcoFalke)
fa8b1114e60100f503673ccc7ef5618851c0c9e3 ci: Run arm ci config on cirrus (MarcoFalke)
fa0795f54dc5268bacc6acdfdbe23f9952a0e7d8 ci: Replace TRAVIS_OS_NAME with CI_OS_NAME (MarcoFalke)
fafce1a13a65cc3ae0cd5351648bb0290f9bfe8c ci: Move documentation to correct config file (MarcoFalke)
Pull request description:
ACKs for top commit:
hebasto:
ACK 77777c8b5e3d0edbe1178d982a778b7cc9964b70
Tree-SHA512: 581d3bdb2c3e3da20bd8492c6b23d90f3b4f7f9300ade384667e41de9fbbe42bbcef26dd99328839dc0b49be6e303c4c6d1b66ceb5d8009800c52c97685f9080
fa2c3c0d96b48bc1685d84186552f72e3609a331 ci: Set LC_ALL=C to allow running the s390x tests in qemu (MarcoFalke)
fac05178366a1c9100b1d19d66fb669d7266f306 travis: Remove s390x build (MarcoFalke)
Pull request description:
This has been discussed in the last meeting.
Refer to the commit body for more details.
Top commit has no ACKs.
Tree-SHA512: 8e0455286ce41c95ed2e5eb624ac534251bb4a321f13d26d14356497e0c39f841372e166373ffd4a0a9fa379636c2cfb535bd92534fff427cdcb827354e66b6c
BACKPORT NOTICE:
define/endif moved in original backport but some changes are not backported yet
---------
fa3967efdb07f1d22372f4ee2e602ea1fad04a57 test: Replace ARRAYLEN with C++11 ranged for loop (MarcoFalke)
fafc5290538fde76c3780976f4b2c11dc9f24d19 test: Run AssetTest even if built --with-libs=no (MarcoFalke)
faf58ab139949ca35b33217d010b350c9a59c61d ci: Add --with-libs=no to one ci config (MarcoFalke)
Pull request description:
`script_assets_test` doesn't call libbitcoinconsensus, so it seems confusing to require it
ACKs for top commit:
fanquake:
ACK fa3967efdb07f1d22372f4ee2e602ea1fad04a57 - looks ok to me.
Tree-SHA512: 8744fef64c5d7dc19a0ef4ae9b3df5b3f356253bf000f12723064794c5e50df071824d436059985f1112d94c1b530b65cbeb6b8435114a91b195480620eafc59
31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0 [util] add RunCommandParseJSON (Sjors Provoost)
c17f54ee535faaedf9033717403e1f775b5f1530 [ci] use boost::process (Sjors Provoost)
32128ba682033560d6eb2e4848a9f77a842016d2 [doc] include Doxygen comments for HAVE_BOOST_PROCESS (Sjors Provoost)
3c84d85f7d218fa27e9343c5cd1a55e519218980 [build] msvc: add boost::process (Sjors Provoost)
c47e4bbf0b44f2de1278f9538124ec98ee0815bb [build] make boost-process opt-in (Sjors Provoost)
929cda5470f98d1ef85c05b1cad4e2fb9227e3b0 configure: add ax_boost_process (Sjors Provoost)
8314c23d7b39fc36dde8b40b03b6efbe96f85698 [depends] boost: patch unused variable in boost_process (Sjors Provoost)
Pull request description:
Prerequisite for external signer support in #16546. Big picture overview in [this gist](https://gist.github.com/Sjors/29d06728c685e6182828c1ce9b74483d).
This adds a new dependency [boost process](https://github.com/boostorg/process/tree/boost-1.64.0). This is part of Boost since 1.64 which is part of `depends`. Because the minimum Boost version is 1.47, this functionality is skipped for older versions of Boost.
Use `./configure --with-boost-process` to opt in, which checks for the presence of Boost::Process.
We add `UniValue runCommandParseJSON(const std::string& strCommand)` to `system.{h,cpp}` which calls an arbitrary command and processes the JSON returned by it. This is currently only called by the test suite.
~For testing purposes this adds a new regtest-only RPC method `runcommand`, as well as `test/mocks/command.py` used by functional tests.~ (this is no longer the case)
TODO:
- [ ] review boost process in #15440
ACKs for top commit:
achow101:
ACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0
hebasto:
re-ACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0, only rebased (verified with `git range-diff`) and removed an unintentional tab character since the [previous](https://github.com/bitcoin/bitcoin/pull/15382#pullrequestreview-458371035) review.
meshcollider:
Very light utACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0, although I am not very confident with build stuff.
promag:
Code review ACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0, don't mind the nit.
ryanofsky:
Code review ACK 31cf68a3ad1f0a5537c8419e2912b55fbfb88fa0. I left some comments below that could be ignored or followed up later. The current change is clean and comprehensive.
Tree-SHA512: c506e747014b263606e1f538ed4624a8ad7bcf4e025cb700c12cc5739964e254dc04a2bbb848996b170e2ccec3fbfa4fe9e2b3976b191222cfb82fc3e6ab182d
BACKPORT NOTICE
There's some difference with original PR but that's not important because we do not actually use travis.
The variable TRAVIS_BRANCH would be removed anyway in bitcoin#20697 - let's just skip it for simplicity
--------------
a91ab86fae91d416d664d19d2f482a8d19c115a6 lint: Use TRAVIS_BRANCH in lint-git-commit-check.sh (Fabian Jahr)
c11dc995c98e908dfd9cea64d4b34329b1dbb5c6 lint: Don't use TRAVIS_COMMIT_RANGE in whitespace linter (Fabian Jahr)
1b41ce8f5f3debae03ca60e4acada14680df9185 lint: Don't use TRAVIS_COMMIT_RANGE for commit-script-check (Fabian Jahr)
Pull request description:
This is causing problems again, very similar to #19654.
UPDATE: This now removes all remaining usages of TRAVIS_COMMIT_RANGE and instead uses TRAVIS_BRANCH for the range, including `lint-git-commit-check` where TRAVIS_COMMIT_RANGE had already been removed. For builds triggered by a pull request, TRAVIS_BRANCH is the name of the branch targeted by the pull request. In the linters there is still a fallback that assumes master as the target branch.
ACKs for top commit:
sipa:
ACK a91ab86fae91d416d664d19d2f482a8d19c115a6. See test I tried in #20075.
Tree-SHA512: 1378bdebd5d8787a83fbda5d9999cc9447209423e7f0218fe5eb240e6a32dc1b51d1cd53b4f8cd1f71574d935ac5e22e203dfe09cce17e9976a48416038e1263
fa7166759789c1282609ff3ab2e80d4f70910a9f ci: Move travis workarounds to .travis.yml (MarcoFalke)
Pull request description:
It seems odd to have travis related workarounds in the general ci config files. Fix that oddity by moving the travis related workarounds to the travis yaml file.
For unexplained reasons, this should also work around and thus close#19171
ACKs for top commit:
hebasto:
ACK fa7166759789c1282609ff3ab2e80d4f70910a9f, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: b4419d38e2b41f6e4d6e6b7658f1d972c40c390a49fe78808f8640d28efd84cc6668ce292d45b7c539e65b9e2ecbad10e796cb8f9329a0f1e7d0132ce962d226
fa7e002d520d8390f3ff4b0383cfdfc14713355d ci: tsan with wallet (MarcoFalke)
Pull request description:
ACKs for top commit:
practicalswift:
ACK fa7e002d520d8390f3ff4b0383cfdfc14713355d -- patch looks correct and Travis is happy
hebasto:
ACK fa7e002d520d8390f3ff4b0383cfdfc14713355d, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: 1138459bbef72f402f32dae1e28d96f174901d4248d959b538b973747c8b06f221ecd81a386a1f915c0a2faaefb9fb8f11e3be39e6c5e2468bf9ae43d9f97757
BACKPORT NOTICE:
this PR doesn't actually swithc to libc++ due to multiple CI failures such as
linking errors or other
------------------
faf62e6ed0ca45db44c370844c3515eb5a8cda12 ci: Remove unused workaround (MarcoFalke)
fa7c8509153bfd2d5b4dcff86ad27dfd73e8788b ci: Install llvm to get llvm symbolizer (MarcoFalke)
fa563cef61e8a217c5e8ec059e174afae61087a5 test: Add more tsan suppressions (MarcoFalke)
fa0cc02c0a029133f080680ae9186002a144738f ci: Mute depends logs completely (MarcoFalke)
fa906bf2988c799765a04c484269f890964ec3ee test: Extend tsan suppressions for clang stdlib (MarcoFalke)
fa10d850790bbe52d948659bb1ebbb88fe718065 ci: Use libc++ instead of libstdc++ for tsan (MarcoFalke)
fa0d5ee1126a8cff9f30f863eb8f5c78bf57e168 ci: Set halt_on_error=1 for tsan (MarcoFalke)
fa2ffe87f794caa74f80c1c2d6e6067ee4849632 ci: Deduplicate DOCKER_EXEC (MarcoFalke)
fac2eeeb9d718bdb892eef9adf333ea61ba8f3d0 cirrus: Remove no longer needed install step (MarcoFalke)
Pull request description:
According to the [ThreadSanitizer docs](https://clang.llvm.org/docs/ThreadSanitizer.html#current-status):
> C++11 threading is supported with **llvm libc++**.
For example, the thread sanitizer build is currently not checking for double lock of mutexes.
Fixes (partially) https://github.com/bitcoin/bitcoin/issues/19038#issuecomment-632138003
ACKs for top commit:
practicalswift:
ACK faf62e6ed0ca45db44c370844c3515eb5a8cda12
fanquake:
ACK faf62e6ed0ca45db44c370844c3515eb5a8cda12
hebasto:
ACK faf62e6ed0ca45db44c370844c3515eb5a8cda12, maybe re-organize commits to modify suppressions in a single one?
Tree-SHA512: 98ce5154b4736dfb811ffdb6e6f63a7bc25fe50d3b73134404a8f3715ad53626c31f9c8132dbacf85de47b9409f1e17a4399e35f78b1da30b1577167ea2982ad
39d526bde48d98af4fa27906e85db0399b6aa8b1 test: Bump linter versions (Duncan Dean)
Pull request description:
As per #19346, `mypy==0.700` was incompatible with Python 3.8.
I've bumped the versions of all the linters to their latest stable versions.
Checked with both Python 3.7 and 3.8 and everything still seems to work fine.
ACKs for top commit:
hebasto:
ACK 39d526bde48d98af4fa27906e85db0399b6aa8b1, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: f3ee7fda8095aa25aa68685e863076d52a6b82649770d24b0064d652763c0ceb8ebcbf9024fc74fca45c754e67b2a831dd070b3af23bc099140e6d27e89a5319
60824b3c3a4221a31c3638008b05fecc040c3df2 ci: Fix configure options for macOS builds (Hennadii Stepanov)
687939e3d251125aea5f6f7202e00a12d60a899f ci: Drop Homebrew caching while using Homebrew addon on Travis (Hennadii Stepanov)
557d3f1cc065702ebbf929d39ac7c5561111ce4c ci: Do not activate Travis ccache caching strategy (Hennadii Stepanov)
2d747428e23c0b7e3e7d0ce0db4226372f5ec0dd ci: Disable functional tests on forked repos to avoid timeouts for macOS (Hennadii Stepanov)
Pull request description:
See: https://github.com/bitcoin-core/gui/issues/5#issuecomment-656819184
Additionally, this PR:
- updates macOS image to the recent 10.15.5 version
- drops Homebrew caching as the Travis Homebrew addon have been used since #18438
My forked repo build: https://travis-ci.org/github/hebasto/bitcoin/jobs/707200431
Top commit has no ACKs.
Tree-SHA512: 398e935f965a04babeb10e7b26d2341562f21a1ef671c2e7cc97c9ec79d5c31643f81ca18561ab7714b5c52e19df2e4bffe4223eadbab984daa9418ffbf8c2a8
fa74a54fad7abfe3b0c98c5a6e4780d63d35b13f ci: Increase test timeout for sanitizer configs (MarcoFalke)
Pull request description:
Hopefully fixes#19369
ACKs for top commit:
practicalswift:
ACK fa74a54fad7abfe3b0c98c5a6e4780d63d35b13f -- patch looks correct!
fanquake:
ACK fa74a54fad7abfe3b0c98c5a6e4780d63d35b13f - the test failure here is a different issue, and the problem referenced by this PR hasn't occurred, so I think this can be merged. It's also fixing the use of `--factor` which was replaced in #18986.
Tree-SHA512: bec44fff454f20b7c5f8a461560d2496765dea61186027cc0cdce5ac55be0488b6f7f172fec49b89fe59a75b455501e2b4ae91a98c4a17d5c1a722846d2b3b60
fa05f44893d228f672f39436d0cb6b3376f81ac2 ci: Upgrade most ci configs to focal (MarcoFalke)
fad67208914e5a74b64f4cc018368902ef3a2e9b doc: move doc to ci readme (MarcoFalke)
fa880773b425fcd292ed7669d237ee3151a15bc6 ci: Have one config run in xenial to test against python3.5 (MarcoFalke)
fa6ddb2fa167df52f59cb9eaabed48315ddcdb2e travis: Always run multiprocess build (MarcoFalke)
Pull request description:
Generally developers compile with recent compilers, so bumping the ci configs to a recent OS should be uncontroversial. Older OSes (especially with compiler sanitizers) need workarounds that can be dropped by running on a more recent OS.
This pull changes the asan sanitizer and the experimental multiprocess build to use focal.
Also, it runs the no_wallet config on xenial to test against python 3.5, according to `doc/dependencies.md`.
Finally, all configs that mimic gitian (win and mac) will stay at bionic.
ACKs for top commit:
Sjors:
ACK fa05f44893d228f672f39436d0cb6b3376f81ac2, assuming Travis passes
hebasto:
ACK fa05f44893d228f672f39436d0cb6b3376f81ac2
Tree-SHA512: 55ec56c71ba2280d27c1a8856a1e6c310b1fbf469d5a8a1dde228063e3892e1dd1e51408ecff7a3d77ac2ae018daa9e9bbbb60598cdeaab8c32a146b11b3e7c4
fa330ec2fe5f5ba68a8d43fff0b19584c0b1ff39 test: Remove confusing and broken use of wait_until global (MarcoFalke)
fa6583c30bf7d82cf7ffdae995f8f16524ad2c0d ci: Set increased --timeout-factor by default (MarcoFalke)
Pull request description:
Assuming that tests don't have a logic error or race, setting a high timeout should not cause any issues. The tests will still pass just as fast in the fastest case, but it allows for some buffer in case of slow disks or otherwise starved ci machines.
Fixes#19729
ACKs for top commit:
hebasto:
ACK fa330ec2fe5f5ba68a8d43fff0b19584c0b1ff39, I have reviewed the code, and it looks OK, I agree it can be merged.
Tree-SHA512: 3da80ee008c7b08bab5fdaf7804d57c79d6fed49a7d37b9c54fc89756659fcb9981fd10afc4d07bd90d93c1699fd410a0110a3cd34d016873759d114ce3cd82d
fa7166759789c1282609ff3ab2e80d4f70910a9f ci: Move travis workarounds to .travis.yml (MarcoFalke)
Pull request description:
It seems odd to have travis related workarounds in the general ci config files. Fix that oddity by moving the travis related workarounds to the travis yaml file.
For unexplained reasons, this should also work around and thus close#19171
ACKs for top commit:
hebasto:
ACK fa7166759789c1282609ff3ab2e80d4f70910a9f, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: b4419d38e2b41f6e4d6e6b7658f1d972c40c390a49fe78808f8640d28efd84cc6668ce292d45b7c539e65b9e2ecbad10e796cb8f9329a0f1e7d0132ce962d226
e2bab2aa162ae38b2bf8195b577c982402fbee9d multiprocess: add multiprocess travis configuration (Russell Yanofsky)
603fd6a2e708c04ef6c9880f89d0a4cbaa6fc7c5 depends: add MULTIPROCESS depends option (Russell Yanofsky)
5d1377b52bfcd4edf8553aaf332bfeb92fc554cc build: multiprocess autotools changes (Russell Yanofsky)
Pull request description:
This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).
---
This PR consists of build changes only. It adds an `--enable-multiprocess` autoconf option (off by default and marked experimental), that builds new `bitcoin-node` and `bitcoin-gui` binaries. These currently function the same as existing `bitcoind` and `bitcoin-qt` binaries, but are extended in #10102 with IPC features to execute node, wallet, and gui functions in separate processes.
In addition to adding the `--enable-multiprocess` config flag, it also adds a depends package and autoconf rules to build with the [libmultiprocess](https://github.com/chaincodelabs/libmultiprocess) library, and it adds new travis configuration to exercise the build code and run functional tests with the new binaries.
The changes in this PR were originally part of #10102 but were moved into #16367 to be able to develop and review the multiprocess build changes independently of the code changes. #16367 was briefly merged and then reverted in #18588. Only change since #16367 has been dropping the `native_boost.mk` depends package which was pointed out to be no longer necessary in https://github.com/bitcoin/bitcoin/pull/16367#issuecomment-596484337 and https://github.com/bitcoin/bitcoin/pull/18588#pullrequestreview-391765649
ACKs for top commit:
practicalswift:
ACK e2bab2aa162ae38b2bf8195b577c982402fbee9d
Sjors:
tACK e2bab2aa162ae38b2bf8195b577c982402fbee9d on macOS 10.15.4
hebasto:
ACK e2bab2aa162ae38b2bf8195b577c982402fbee9d, tested on Linux Mint 19.3 (x86_64):
Tree-SHA512: b5a76eab5abf63d9d8b6d628cbdff4cc1888eef15cafa0a5d56369e2f9d02595fed623f4b74b2cf2830c42c05a774f0943e700f9c768a82d9d348cad199e135c
060a2a64d40d75fecb60b7d2b9946a67e46aa6fc ci: remove boost thread installation (fanquake)
06e1d7d81d5a56d136c6fc88f09a2b0654a164f9 build: don't build or use Boost Thread (fanquake)
7097add83c8596f81be9edd66971ffd2486357eb refactor: replace Boost shared_mutex with std shared_mutex in sigcache (fanquake)
8e55981ef834490c438436719f95cbaf888c4914 refactor: replace Boost shared_mutex with std shared_mutex in cuckoocache tests (fanquake)
Pull request description:
This replaces `boost::shared_mutex` and `boost::unique_lock` with [`std::shared_mutex`](https://en.cppreference.com/w/cpp/thread/shared_mutex) & [`std::unique_lock`](https://en.cppreference.com/w/cpp/thread/unique_lock).
Even though [some concerns were raised](https://github.com/bitcoin/bitcoin/issues/16684#issuecomment-726214696) in #16684 with regard to `std::shared_mutex` being unsafe to use across some glibc versions, I still think this change is an improvement. As I mentioned in #21022, I also think trying to restrict standard library feature usage based on bugs in glibc is not only hard to do, but it's not currently clear exactly how we do that in practice (does it also extend to patching out use in our dependencies, should we be implementing more runtime checks for features we are using, when do we consider an affected glibc "old enough" not to worry about? etc). If you take a look through the [glibc bug tracker](https://sourceware.org/bugzilla/describecomponents.cgi?product=glibc) you'll no doubt find plenty of (active) bug reports for standard library code we already using. Obviously not to say we shouldn't try and avoid buggy code where possible.
Two other points:
[Cory mentioned in #21022](https://github.com/bitcoin/bitcoin/pull/21022#issuecomment-769274179):
> It also seems reasonable to me to worry that boost hits the same underlying glibc bug, and we've just not happened to trigger the right conditions yet.
Moving away from Boost to the standard library also removes the potential for differences related to Boosts configuration. Boost has multiple versions of `shared_mutex`, and what you end up using, and what it's backed by depends on:
* The version of Boost.
* The platform you're building for.
* Which version of `BOOST_THREAD_VERSION` is defined: (2,3,4 or 5) default=2. (see [here](https://www.boost.org/doc/libs/1_70_0/doc/html/thread/build.html#thread.build.configuration) for some of the differences).
* Is `BOOST_THREAD_V2_SHARED_MUTEX` defined? (not by default). If so, you might get the ["less performant, but more robust"](https://github.com/boostorg/thread/issues/230#issuecomment-475937761) version of `shared_mutex`.
A lot of these factors are eliminated by our use of depends, but users will have varying configurations. It's also not inconceivable to think that a distro, or some package manager might start defining something like `BOOST_THREAD_VERSION=3`. Boost tried to change the default from 2 to 3 at one point.
With this change, we no longer use Boost Thread, so this PR also removes it from depends, the build system, CI etc.
Previous similar PRs were #19183 & #20922. The authors are included in the commits here.
Also related to #21022 - pthread sanity checking.
ACKs for top commit:
laanwj:
Code review ACK 060a2a64d40d75fecb60b7d2b9946a67e46aa6fc
vasild:
ACK 060a2a64d40d75fecb60b7d2b9946a67e46aa6fc
Tree-SHA512: 572d14d8c9de20bc434511f20d3f431836393ff915b2fe9de5a47a02dca76805ad5c3fc4cceecb4cd43f3ba939a0508178c4e60e62abdbaaa6b3e8db20b75b03
bd7e530f010d43816bb05d6f1590d1cd36cdaa2c This PR adds initial support for type hints checking in python scripts. (Kiminuo)
Pull request description:
This PR adds initial support for type hints checking in python scripts.
Support for type hints was introduced in Python 3.5. Type hints make it easier to read and review code in my opinion. Also an IDE may discover a potential bug sooner. Yet, as PEP 484 says: "It should also be emphasized that Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention."
[Mypy](https://mypy.readthedocs.io/en/latest/index.html) is used in `lint-python.sh` to do the type checking. The package is standard so there is little chance that it will be abandoned. Mypy checks that type hints in source code are correct when they are not, it fails with an error.
**Notes:**
* [--ignore-missing-imports](https://mypy.readthedocs.io/en/latest/command_line.html#cmdoption-mypy-ignore-missing-imports) switch is passed on to `mypy` checker for now. The effect of this is that one does not need `# type: ignore` for `import zmq`. More information about import processing can be found [here](https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports). This can be changed in a follow-up PR, if it is deemed useful.
* We are stuck with Python 3.5 until 04/2021 (see https://packages.ubuntu.com/xenial/python3). When Python version is bumped to 3.6+, one can change:
```python
_opcode_instances = [] # type: List[CScriptOp]
```
to
```python
_opcode_instances:List[CScriptOp] = []
```
for type hints that are **not** function parameters and function return types.
**Useful resources:**
* https://docs.python.org/3.5/library/typing.html
* https://www.python.org/dev/peps/pep-0484/
ACKs for top commit:
fanquake:
ACK bd7e530f010d43816bb05d6f1590d1cd36cdaa2c - the type checking is not the most robust (there are things it fails to detect), but I think this is worth adopting (in a limited capacity while we maintain 3.5 compat).
MarcoFalke:
ACK bd7e530f010d43816bb05d6f1590d1cd36cdaa2c fine with me
Tree-SHA512: 21ef213915fb1dec6012f59ef17484e6c9e0abf542a316b63d5f21a7778ad5ebabf8961ef5fc8e5414726c2ee9c6ae07c7353fb4dd337f8fcef5791199c8987a
faca73000fa8975c28f6be8be01957c1ae94ea62 ci: Install fixed version of clang-format for linters (MarcoFalke)
fa4695da4c69646b58a8fa0b6b30146bb234deb8 build: Sort Makefile.am after renaming file (MarcoFalke)
cccc2784a3bb10fa8e43be7e68207cafb12bd915 scripted-diff: Move ui_interface to the node lib (MarcoFalke)
fa72ca6a9d90d66012765b0043fd819698b94ba8 qt: Remove unused includes (MarcoFalke)
fac96e6450d595fe67168cb7afa7692da6cc9973 wallet: Do not include server symbols (MarcoFalke)
fa0f6c58c1c6d10f04c4e65a424cc51ebca50a8c Revert "Fix link error with --enable-debug" (MarcoFalke)
Pull request description:
This reverts a hacky workaround from commit b83cc0f, which only happens to work due to compiler optimizations. Then, it actually fixes the linker error.
The underlying problem is that the wallet includes symbols from the server (ui_interface), which usually results in linker failures. Though, in this specific case the linker failures have not been observed (unless `-O0`) because our compilers were smart enough to strip unused symbols.
Fix the underlying problem by creating a new header-only with the needed symbol and move ui_interface to node to clarify that this is part of libbitcoin_server.
ACKs for top commit:
Sjors:
ACK faca730
laanwj:
ACK faca73000fa8975c28f6be8be01957c1ae94ea62
hebasto:
re-ACK faca73000fa8975c28f6be8be01957c1ae94ea62, since the [previous](https://github.com/bitcoin/bitcoin/pull/19331#pullrequestreview-434420539) review:
Tree-SHA512: e9731f249425aaea50b6db5fc7622e10078cf006721bb87989cac190a2ff224412f6f8a7dd83efd018835302337611f5839e29e15bef366047ed591cef58dfb4
fae49f6e424f31e93c5620d5ff893fb517ef4a8b ci: Add and document BASE_BUILD_DIR (MarcoFalke)
Pull request description:
Also fixes#18768
ACKs for top commit:
hebasto:
re-ACK fae49f6e424f31e93c5620d5ff893fb517ef4a8b, which is essentially the same as the previously [reviewed changes](https://github.com/bitcoin/bitcoin/pull/18735#pullrequestreview-400581536).
Tree-SHA512: 216565a05ccd513dd9f114b2333d3c283fd71914d32f9b05f145cb7c70633b083ff8ef60798d6f22f4be6a4d652b03806551fd74b5b596c92968501a4d9726d2
It does not includes changes that actually enable werror by default.
But this backport is necessary to prevent compilation error on mac such as this one:
In file included from mapport.cpp:26:
In file included from /builds/dashpay/dash/depends/x86_64-apple-darwin/include/miniupnpc/miniupnpc.h:14:
/builds/dashpay/dash/depends/x86_64-apple-darwin/include/miniupnpc/upnpdev.h:27:14: error: zero size arrays are an extension [-Werror,-Wzero-length-array]
char buffer[0];
fa23fbb42fa710e6be405596e2abfc3a289d020d ci: Run all tests on native mac again (MarcoFalke)
Pull request description:
They should pass again after f6072e601af68f3eee307478ad22ff3960680656
ACKs for top commit:
practicalswift:
ACK fa23fbb42fa710e6be405596e2abfc3a289d020d -- Travis is happy and so am I
Tree-SHA512: 49c16b6056d4e67d12a202744e1c56fee2788830213fe4a195955ad44c6b8ecce768a591463ffa0048821959a75b6fad4178629a8866c4a26799c4c8c13e933d
fad798be76dd5e330463c837fda768477d536078 test: Default --previous-releases to false if dir is empty (MarcoFalke)
faf1c3cc58d14f86ba5364e6ee5c8ef29cac2e26 test: Replace TEST_PREVIOUS_RELEASES env var with test_framework option (MarcoFalke)
Pull request description:
The "auto-detection" feature is kept in place, but making it an option allows to properly document it. For example, on my machine I get:
```
$ ./test/functional/wallet_disable.py --help | grep previous-releases
--previous-releases Force test of previous releases (default: False)
ACKs for top commit:
Sjors:
re-tACK fad798b
Tree-SHA512: a7377d0d5378be0a50be278d76396cc403583617b5fc43467773eee706df698acf3f4e67651491183b9b43a8e1816b052e4c17b90272b7ec4b6ac134ad811400
faf552117efdd456fb17b1d9490896de3b7d67a4 ci: Set DEBIAN_FRONTEND=noninteractive (MarcoFalke)
fa006caa136f05a18428c792802523637d2234ca ci: tsan on clang-9 (MarcoFalke)
Pull request description:
Bump the compiler runtime library that includes the sanitizers from clang-8 to clang-9 to get a more recent version. Also, bump the system packages from xenial to bionic to test packages closer to what is commonly used in production.
The second commit is needed to install the `tzdata` package, which is missing on some operating systems. See https://travis-ci.org/github/MarcoFalke/bitcoin-core/jobs/688455828#L1727
ACKs for top commit:
hebasto:
ACK faf552117efdd456fb17b1d9490896de3b7d67a4
practicalswift:
ACK faf552117efdd456fb17b1d9490896de3b7d67a4 -- patch looks correct and Travis is happy
Tree-SHA512: aa38fdae5f716966a83a21d5f7c121675cf7d663148ab3baa065142c8b3850bcd4bf88526d7da0fa51f5e08f2c317b537f950fcc9eb1e69fdacb0eac8863e1c6
cbd661122e5852d543467090459d33cf8cb4a3c7 Set LD_LIBRARY_PATH consistently in travis tests (Russell Yanofsky)
fa35c34df781cf46bbd15522961f214f03b958bf Remove unused ci configs that have been moved elsewhere (MarcoFalke)
3333cb96994182bbdbb21174b691feb716858bc2 fuzz: Pass down MAKEJOBS to test_runner (MarcoFalke)
Pull request description:
Just how `MAKEJOBS` is passed down to the functional test `test_runner`, do the same for the fuzz `test_runner`.
Also includes a commit to remove unused config files, which have been moved elsewhere.
Top commit has no ACKs.
Tree-SHA512: 32557102c9e40599b432aeb004c8427e8fbb07cdf4048050cdc8241d1b029aaad306b1131007eeca8315a4f71c38a7efbb833310e056cd11b835676cd19b8902
fa72a751026d43e1d01ae30e26bcfd9b4cc0cf45 ci: Document why tests can not be run on mac (MarcoFalke)
Pull request description:
Fixes#18794
Top commit has no ACKs.
Tree-SHA512: 297652eda412aa8cf7255e20a6f294d22773dad8637a3d7b5204f3b638e911ce5b2e40e85f81395a34c1b5a5b497665944c2d6ea17c70c30c0c9e0ab553f956e
fac24dea00c54fe1e2dcc7625bb65ada8b97ac0c ci: Run functional tests on mac again (MarcoFalke)
Pull request description:
ACKs for top commit:
hebasto:
ACK fac24dea00c54fe1e2dcc7625bb65ada8b97ac0c, verified travis build log.
Tree-SHA512: 406282a7ac03e5c193830b727366c7b1350639f1850aff951bf7ddd4b0c3e3ffb396b950ccb3a64ddc59500fa2739766f3c34806b4d144bc4535bb2bd765b959
e4c0cada791135e2d0a36638541c03feff0bd6bc ci, gitian: Drop unneeded python3-dev package for macOS builds (Hennadii Stepanov)
Pull request description:
ACKs for top commit:
fanquake:
ACK e4c0cada791135e2d0a36638541c03feff0bd6bc - gitian builds match and I checked that this doesn't end up installed as a side-effect of another package.
Tree-SHA512: 520a3909b106a0e005b195c5395691edf62b76ee2df43b6971b7aa193648d68e6dac69cb4f1dc474f594b015a2fc2074061865e571d89365174beb5c1780356f
