e4c0cada791135e2d0a36638541c03feff0bd6bc ci, gitian: Drop unneeded python3-dev package for macOS builds (Hennadii Stepanov)
Pull request description:
ACKs for top commit:
fanquake:
ACK e4c0cada791135e2d0a36638541c03feff0bd6bc - gitian builds match and I checked that this doesn't end up installed as a side-effect of another package.
Tree-SHA512: 520a3909b106a0e005b195c5395691edf62b76ee2df43b6971b7aa193648d68e6dac69cb4f1dc474f594b015a2fc2074061865e571d89365174beb5c1780356f
34c80d9eee7d21755f2bb80f7c97fd30d2c7b656 test: Add option to git-subtree-check to do full check, add help (Wladimir J. van der Laan)
Pull request description:
This adds a brief help text to `git-subtree-check.sh` and adds an option to do a full remote check instead of having two different code paths with a successful exit status. Also make it explicit that the CI is not doing this.
ACKs for top commit:
fjahr:
tested ACK 34c80d9eee7d21755f2bb80f7c97fd30d2c7b656
Tree-SHA512: 20f672fd3b3c1d633eccf9998fdd738194cdd7d10cc206691f2dcc28bbbf8187b8d06b87814f875a06145b179f5ca1f4f4f9922972be72759cf5ac6e0c11abd1
aaaaad6ac95b402fe18d019d67897ced6b316ee0 scripted-diff: Bump copyright of files changed in 2019 (MarcoFalke)
Pull request description:
ACKs for top commit:
practicalswift:
ACK aaaaad6ac95b402fe18d019d67897ced6b316ee0
promag:
ACK aaaaad6ac95b402fe18d019d67897ced6b316ee0 🎉
fanquake:
ACK aaaaad6ac95b402fe18d019d67897ced6b316ee0 - going to merge this now because the year is over and conflicts are minimal.
Tree-SHA512: 58cb1f53bc4c1395b2766f36fabc7e2332e213780a802762fff0afd59468dad0c3265f553714d761c7a2c44ff90f7dc250f04458f4b2eb8eef8b94f8c9891321
fa80e10d94dbf86da84fc761b09fb631155a5b25 test: Add feature_taproot.py --previous_release (MarcoFalke)
85ccffa26686c6c9adbd18bdde37fc1747281bab test: move releases download incantation to README (Sjors Provoost)
29d6b1da2a862bfbb14e7821979c97416c5400e8 test: previous releases: add v0.20.1 (Sjors Provoost)
Pull request description:
Disabling the new consensus code at runtime is fine, but potentially fragile and incomplete. Fix that by giving the option to run with a version that has been compiled without any taproot code.
ACKs for top commit:
Sjors:
tACK fa80e10
NelsonGaldeman:
tACK fa80e10d94dbf86da84fc761b09fb631155a5b25
Tree-SHA512: 1a1feef823f08c05268759645a8974e1b2d39a024258f5e6acecbe25097aae3fa9302c27262978b40f1aa8e7b525b60c0047199010f2a5d6017dd6434b4066f0
0374e821bd9e9498ce9c03aa8e5435870019978b util: Hard code previous release tarball checksums (Hennadii Stepanov)
bd897ce79f72a44a2e609f95433e251a3fd9eb9c scripted-diff: Move previous_release.py to test/get_previous_releases.py (Hennadii Stepanov)
Pull request description:
#19205 introduced signature verifying for the downloaded `SHA256SUMS.asc`.
This approach is brittle and does not work in CI environment for many reasons:
- https://github.com/bitcoin/bitcoin/issues/19812#issuecomment-680760663
- https://github.com/bitcoin/bitcoin/pull/19013#discussion_r459590779
This PR:
- implements **Sjors**' [idea](https://github.com/bitcoin/bitcoin/pull/19205#pullrequestreview-426080048):
> Alternatively we might as well hard code the checksum for each `tar.gz` release in the source code, here.
- is an alternative to 5a2c31e528e6bd60635096f233252f3c717f366d (#19013)
- fixes#19812
- updates v0.17.1 to v0.17.2
ACKs for top commit:
MarcoFalke:
cr ACK 0374e821bd9e9498ce9c03aa8e5435870019978b
Sjors:
tACK 0374e821bd9e9498ce9c03aa8e5435870019978b
Tree-SHA512: cacdcf9f5209eae7da357abb3445585ad2f980920fd5bf75527ce89974d3f531a4cf8b5b35edfc116b23bfdfb45c0437cb14cbc416d76ed2dc5b9e6d33cdad71
d135c294764add81683ba47575f9a5dde7d7c07f [ci] make list of previous releases to download a setting (Sjors Provoost)
9c246b873c74834a121edba00fcaecf0cba6f9b4 [test] backwards compatibility: bump v0.19.0.1 to v0.19.1 (Sjors Provoost)
89a28e02fa46f3d5eb07ab02aa34aa95c6fcee11 [test] add v0.16.3 backwards compatibility test (Sjors Provoost)
Pull request description:
Thanks to #18774's `adjust_bitcoin_conf_for_pre_17` we can now test backwards compatibility for v0.16.3, both for sync and loading a recent wallet.
This PR bumps v0.19.0.1 to v0.19.1.
I also made the version list consistent for the `contrib/devtools/previous_release.sh` instruction, between both tests.
ACKs for top commit:
MarcoFalke:
ACK d135c294764add81683ba47575f9a5dde7d7c07f
Tree-SHA512: 5ff137a7a934237fa220f1c2807ce9abeeb75929266558bf3e4045bec7dfcd0a8747fa74d700065c568330b18badf58c60c308eb13d1eed444d4bbfe6decc48b
c456145b2c65f580683df03bf10cd39000cf24d5 [test] add 0.19 backwards compatibility tests (Sjors Provoost)
b769cd142deda74fe46e231cc7b687a86514f2f1 [test] add v0.17.1 wallet upgrade test (Sjors Provoost)
9d9390dab716f07057c94e8e21f3c7dd06192f35 [tests] add wallet backwards compatility tests (Sjors Provoost)
c7ca6308968b29a0e0edc485cd06e68e5edb7c7d [scripts] support release candidates of earlier releases (Sjors Provoost)
8b1460dbd1b732f06d4cebe1fa6844286c7a0056 [tests] check v0.17.1 and v0.18.1 backwards compatibility (Sjors Provoost)
ae379cf7d12943fc192d58176673bcfe7d53da53 [scripts] build earlier releases (Sjors Provoost)
Pull request description:
This PR adds binaries for 0.17, 0.18 and 0.19 to Travis and runs a basic block propagation test.
Includes test for upgrading v0.17.1 wallets and opening master wallets with older versions.
Usage:
```sh
contrib/devtools/previous_release.sh -f -b v0.19.0.1 v0.18.1 v0.17.1
test/functional/backwards_compatibility.py
```
Travis caches these earlier releases, so it should be able to run these tests with little performance impact.
Additional scenarios where it might be useful to run tests against earlier releases:
* creating a wallet with #11403's segwit implementation, copying it to an older node and making sure the user didn't lose any funds (although this PR doesn't support `v0.15.1`)
* future consensus changes
* P2P changes (e.g. to make sure we don't accidentally ban old nodes)
ACKs for top commit:
MarcoFalke:
ACK c456145b2c65f580683df03bf10cd39000cf24d5 🔨
Tree-SHA512: 360bd870603f95b14dc0cd629532cc147344f632b808617c18e1b585dfb1f082b401e5d493a48196b719e0aeaee533ae0a773dfc9f217f704aae898576c19232
0c6318788beaf1a31aeba5a21f3f8bb5c07cea6c ci: Limit cache size regardless of NO_DEPENDS (Hennadii Stepanov)
Pull request description:
Close#18666.
ACKs for top commit:
MarcoFalke:
ACK 0c6318788beaf1a31aeba5a21f3f8bb5c07cea6c . Depends has ccache disabled anyway and is cached regardless of whether ccache is there or not, see #17248
Tree-SHA512: b1bf98be0f844b4704abd177841b014f3900be8160496f0d12596310db607b4f544547e8c3cbfcf17c086a78afd251653363f3dd467b769ac0062bc19adc8144
fa4632c41714dfaa699bacc6a947d72668a4deef test: Move boost/stdlib includes last (MarcoFalke)
fa488f131fd4f5bab0d01376c5a5013306f1abcd scripted-diff: Bump copyright headers (MarcoFalke)
fac5c373006a9e4bcbb56843bb85f1aca4d87599 scripted-diff: Sort test includes (MarcoFalke)
Pull request description:
When writing tests, often includes need to be added or removed. Currently the list of includes is not sorted, so developers that write tests and have `clang-format` installed will either have an unrelated change (sorting) included in their commit or they will have to manually undo the sort.
This pull preempts both issues by just sorting all includes in one commit.
Please be aware that this is **NOT** a change to policy to enforce clang-format or any other developer guideline or process. Developers are free to use whatever tool they want, see also #18651.
Edit: Also includes a commit to bump the copyright headers, so that the touched files don't need to be touched again for that.
ACKs for top commit:
practicalswift:
ACK fa4632c41714dfaa699bacc6a947d72668a4deef
jonatack:
ACK fa4632c41714dfaa, light review and sanity checks with gcc build and clang fuzz build
Tree-SHA512: 130a8d073a379ba556b1e64104d37c46b671425c0aef0ed725fd60156a95e8dc83fb6f0b5330b2f8152cf5daaf3983b4aca5e75812598f2626c39fd12b88b180
## Issue being fixed or feature implemented
Make CI builds faster.
We have this
https://github.com/dashpay/dash/blob/develop/ci/dash/build_src.sh#L18
but by that time `CCACHE_SIZE` is set to 100M via
https://github.com/dashpay/dash/blob/develop/ci/test/00_setup_env.sh#L51
already.
## What was done?
set `CCACHE_SIZE` variable in `.gitlab-ci.yml`, drop confusing and
useless lines from `ci/dash/build_src.sh` (`CCACHE_SIZE` and
`CCACHE_COMPRESS` defaults are handled by `ci/test/00_setup_env.sh`)
## How Has This Been Tested?
I set `CCACHE_SIZE` to 400M yesterday via Gitlab UI to test it.
results:
before, trivial doc change, 100M:
https://gitlab.com/dashpay/dash/-/jobs/4931233566#L59 (~17 minutes)
after, trivial code change, 400M:
https://gitlab.com/dashpay/dash/-/jobs/4935764148#L80 (~12 minutes)
Removed it from Gitlab UI now but it should still say `Set cache size
limit to 400.0 MB` in logs for this PR.
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
## Issue being fixed or feature implemented
40% smaller CI job artifacts. Should help with issues like
https://gitlab.com/dashpay/dash/-/jobs/4759700026 in #5493.
## What was done?
## How Has This Been Tested?
https://gitlab.com/UdjinM6/dash/-/jobs/4773517599
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
Used to avoid the following error:
```
In file included from test/fuzz/addition_overflow.cpp:7:
In file included from ./test/fuzz/util.h:26:
In file included from ./test/util/setup_common.h:16:
In file included from ./txmempool.h:31:
In file included from /builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/optional.hpp:15:
In file included from /builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/optional/optional.hpp:47:
In file included from /builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/type_traits/is_nothrow_move_assignable.hpp:16:
/builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/type_traits/has_nothrow_assign.hpp:65:7: error: builtin __has_nothrow_assign is deprecated; use __is_nothrow_assignable instead [-Werror,-Wdeprecated-builtins]
BOOST_HAS_NOTHROW_ASSIGN(T)
^
/builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/type_traits/intrinsics.hpp:205:43: note: expanded from macro 'BOOST_HAS_NOTHROW_ASSIGN'
^
/builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/serialization/serialization.hpp:61:1: note: in instantiation of template class 'boost::has_nothrow_assign<unsigned int>' requested here
BOOST_STRONG_TYPEDEF(unsigned int, version_type)
^
/builds/dashpay/dash/depends/x86_64-pc-linux-gnu/include/boost/serialization/strong_typedef.hpp:42:57: note: expanded from macro 'BOOST_STRONG_TYPEDEF'
D& operator=(const D& rhs) BOOST_NOEXCEPT_IF(boost::has_nothrow_assign<T>::value) {t = rhs.t; return *this;} \
```
clang warnings are quite noisy and easily cause gitlab's logs
to spill over, preventing logging of dash-specific warnings and errors
from making it to the CI logs.
our dependencies mostly track upstream so regardless, we cannot
act upon those warnings, so it's better to just suppress them if they're
too noisy.
faceeae49ae85fb644996fedb970ad9d0e297ca5 ci: Disable valgrind functionl tests on forked repos to avoid timeouts (MarcoFalke)
Pull request description:
Allows people to fork our repo and run the tests again
Also print more cache stats
ACKs for top commit:
hebasto:
ACK faceeae49ae85fb644996fedb970ad9d0e297ca5, tested on my own repo: https://travis-ci.org/github/hebasto/bitcoin/jobs/676257500
Tree-SHA512: 50e44edf94fcb997438eeaf7308b2b58a0854141ecb1fbb0ba7bf5ed662f19b60899f966f579cca90ad5e789234d0e90122d8c2c854da70339058adc3a475fa6
6136a96cdf631f824fb89cf2720422787439acc9 ci: Rename RUN_CI_ON_HOST to DANGER_RUN_CI_ON_HOST (Hennadii Stepanov)
97ba77aa8e58c3c7ff767e6ca804d0e46370e5dc ci: Add native s390x (Hennadii Stepanov)
Pull request description:
Unlike the Docker wrapped solution (#17591) this PR suggests running on host system directly.
This approach makes builds quick and stable (see: #18106).
The excerpt from the Travis log:
```
...
Running on host system without docker wrapper
...
Byte Order: Big Endian
...
```
ACKs for top commit:
MarcoFalke:
ACK 6136a96cdf631f824fb89cf2720422787439acc9
Tree-SHA512: 1b591de13e38d10a35217e1de11cbd648a359d18d16eed166fac18ea5788b58cc9fc6d407086ed342b99e57e479efd951a0ea693710177e500eb116316b9a788
fa7af33b4cb12315de86315c163393bdb50fad75 ci: Run unit tests sequential once (MarcoFalke)
fa68a3e7640fc00c47f03fa565357679bdafb4b2 appveyor: Enable minimal unit test logging to aid debugging (MarcoFalke)
Pull request description:
Fixes#16976
Top commit has no ACKs.
Tree-SHA512: 1f1ee8776a67afa8c1c5a16ef170c9975b6a486c087c8eba12e97d23382befd1c2801622ec70ca8e4cd1fbedce1dec46be67677ceaf07f35f1d3f3bead0200f0
faa9491870bf29583a989c407694d01f49b65bec ci: Use Focal for fuzzers (MarcoFalke)
Pull request description:
This gives us access to clang-10, as well as a newer version of valgrind
ACKs for top commit:
fanquake:
ACK faa9491870bf29583a989c407694d01f49b65bec - [Clang 10](https://packages.ubuntu.com/focal/clang) and [valgrind 3.15](https://packages.ubuntu.com/focal/valgrind).
practicalswift:
ACK faa9491870bf29583a989c407694d01f49b65bec -- diff looks correct & contemporary clang is better than vintage clang
Tree-SHA512: 0e67232673434c0309db79c1054e3e981115083585945967e346f4d58792635832100f89911428aab928155e44e5f401207a023681ae008fdb5280cf02c4d427
25c8b73656c90f7b78a90f25c5bcc308f8b7c598 ci: Use Homebrew addon on native macOS (Hennadii Stepanov)
596c627a1eb2b32e2755ae5b9bf32235cf8ce72b ci: Fix brew in Travis (Hennadii Stepanov)
Pull request description:
Recently almost every macOS image update on Travis breaks our builds:
- #17848
- #18436
This PR:
- fixes the error caused by the recent [update](https://changelog.travis-ci.com/xcode-11-3-1-xcode-11-2-1-xcode-11-1-and-xcode11-images-updated-142286) from 10.14.4 (18E226) to 10.14.6 (18G3020) on March 25
- leverages [Homebrew addon](https://config.travis-ci.com/ref/job/addons/homebrew) to install packages
Homebrew is not told to install `automake` and `pkg-config` packages, as the [docs](https://docs.travis-ci.com/user/reference/osx/#compilers-and-build-toolchain) states that they are pre-installed:
> - automake 1.16.1
> - pkg-config 0.29.2
Top commit has no ACKs.
Tree-SHA512: 1a70c06468fbe162503081b03dcf54614d67abf8ff0ce07d118b5bb50bbb92c182a76f769bea586c691aa82b9281a29cdef88091acc16895817a2e7cddafec6e
fae1e992898aa6b36c402cec4085fbf6da9b33ad ci: Only clone bitcoin-core/qa-assets when fuzzing (MarcoFalke)
Pull request description:
Currently the only content of that repo are some seeds, so we can speed up some ci builds
ACKs for top commit:
laanwj:
ACK fae1e992898aa6b36c402cec4085fbf6da9b33ad (provided this passes travis)
Tree-SHA512: ed813738e7f24bb56a2f12aa3b398e414eb4f0ba98379836a33ff3e5602cbf42a28e89aad10e346468191ecddc03e60d5b236097112e27c07cb1c2293533ea58
fa92af5af39a08982f785542df5419d6d5a4706d ci: Run feature_block and feature_abortnode in valgrind (MarcoFalke)
fa01febeaf801bade77a613e64f18b556ae16d86 test: Remove ci timeout restriction in test_runner (MarcoFalke)
Pull request description:
Also revert commit 0a4912e46a, because some tests take too long for this to be useful anymore.
Top commit has no ACKs.
Tree-SHA512: 363f14766e1f4a5860ab668a516b41acebc6fbdf11d8defb3a95a772dbf82304ca1f5f14b1dbad97f2029503e03d92e8c69df0466a8872409c20665838f617ed
4444edc2e6671d3f73de3725447130f73ecf0375 ci: Enable all functional tests in valgrind (MarcoFalke)
Pull request description:
The travis timeout for our repo has been bumped to 2h, so we can run all tests in valgrind now
ACKs for top commit:
practicalswift:
ACK 4444edc2e6671d3f73de3725447130f73ecf0375 -- regarding the three disabled cases (`feature_abortnode`, `feature_block` and `rpc_bind`): not a big deal since MSan will take care of those once #18288 is merged. More is more :)
Tree-SHA512: ea2f798112911b6d1f3d88cfcdf0a7cdb698687248343703d6fe55da144542c961c15d888bffb41672c10aa76765615cb7c7ff93d468bfad3c51f962f24e7abb
TSan builds export all sorts of symbols that aren't part of the
allowlist and report sanitizer failures when enabled with glibc
compatibility enabled.
The result of `check-symbols` is valuable but Bitcoin does not
run them during their CI runs and for certain developer-oriented
builds, it's proving to be more of an obstruction. It will still
be enabled for all other builds and will prove to be important
in backporting Guix pull requests, a finger on the pulse, but that
doesn't apply to dev-oriented builds.
fa6a548f5485ad5be0106e4727812559aefc5a20 ci: Disable s390x gui tests for now (MarcoFalke)
Pull request description:
ACKs for top commit:
hebasto:
ACK fa6a548f5485ad5be0106e4727812559aefc5a20, tested locally with
Tree-SHA512: 077bcda0b4612dfbf538eb2999ce9f32ec3739c2ed5709f7c4c5d1a39738fbe6e549bd54b237152fd808736c15af1024e67c95dc87b4c3102738543522cb14d7
## Description
Pull request was inspired by the need to debug lock problems when
working on https://github.com/dashpay/dash/pull/5352.
As far as I'm aware, only macOS has `-Werror=thread-safety` as part of
its default `CXXFLAGS` despite the capability being present on Linux as
well. This PR introduces thread safety checks for that into our thread
sanitizer build.
Additionally, since we're using Clang, something that on first glimpse,
appears to be something that `stacktraces.cpp` isn't happy with, due to
`-Wl,-wrap` being available only on GCC, that no longer seems to be the
case, since the version of Clang with comes with `focal`, its `lld`
_does_ have support for `-wrap` (see [man page for `lld` on
`focal`](https://manpages.ubuntu.com/manpages/focal/en/man1/lld.1.html)).
The current `stable` version of Clang/LLVM is 15, at the time of this
pull request (see https://apt.llvm.org/) but `focal` ships with an older
version, requiring us to use the official LLVM APT repository. I feel we
should be testing with recent compilers alongside the ones shipped by
LTS distributions.
Certain bugs are only made apparent when testing on rolling release
distros or distros that have faster update cycles, like Fedora (see
https://github.com/dashpay/dash/pull/5295 for an illustration of that),
which ship with more recent compilers. Until we overhaul our CI systems
to test using those distros directly (our current infrastructure is
centered around using a "development image" with an LTS distro as the
base), this is the best we can do.
A similar pull request testing against the latest GCC stable will be
welcome as that is currently outside the scope of this PR as the changes
made were to make sure that builds were operating as expected on
Clang/LLVM 15.
---------
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
9e111db088e4137865ae068d206c769994ea0a29 test: set a name for CI Docker containers (fanquake)
Pull request description:
Addresses one part of #16664, by making it easier to identify CI containers that are running locally. By default Docker will generate random names, like `peaceful_rubin`, with this change, we explicitly set names for all containers.
ACKs for top commit:
MarcoFalke:
ACK 9e111db088e4137865ae068d206c769994ea0a29
Tree-SHA512: 0a29ada0d8cf6b0e9ae7a35f4f6df7a3dcc448523ceaed01371124360d6e3d1bf351172104a5fb629488eeaa57994ba04134dcb83c261eb1dfd2f0d73edf5f60
fa569e1a9c5ad1bf8bdf866235b21aff56112224 ci: Set LC_ALL=C when running in qemu-s390x (MarcoFalke)
fa3d77623ecb25dde6b5f4be11626ed16966eb29 ci: Use debian to avoid apt install 404 errors (MarcoFalke)
fabb946090be2f604da3d7d4b1bbe93b79baf23e ci: Install needed gcc and qemu-user iff cross-compiling (MarcoFalke)
faba4672b64fb5ba89e5cb6299479887494b571a ci: Fix QEMU_USER_CMD parse issues (MarcoFalke)
fa5d709fb266c97d4db15006bf855e887a6c123b ci: Move wrap-qemu into separate script (MarcoFalke)
Pull request description:
Now that the ci system no longer propagates files from the docker container back to the host, the wrap-qemu script is broken.
To fix it, every statement in the script needs to be executed in the docker (with `DOCKER_EXEC`). Instead of juggling with triple escape sequences like `\\\"`, just move the script to a separate file and call it with `DOCKER_EXEC`.
Also, fix a bunch of other bugs that prevent running the ci system in qemu
See the `ci/README.md` on how to test. TLDR: Can be tested with (replace `arm` with `s390x` to run the s390x build):
```
FILE_ENV="./ci/test/00_setup_env_arm.sh" MAKEJOBS="-j9" ./ci/test_run_all.sh
```
ACKs for top commit:
laanwj:
Code review ACK fa569e1a9c5ad1bf8bdf866235b21aff56112224
Tree-SHA512: 84ebc44a4f0261ee6c29605a6896a1833ff6c81d729e6d08dd111941f570ce73221422bd3303e1108a266ec5eab2148bd5ee1cf6bc01477d8cc9a6c5bf2b34c2
