46b025e00df40724175735eb5606ac73067cb3b8 test: add new python linter to check file names and permissions (windsok)
6f6bb3ebc7cb8e17a5dfc8ef55aa2d3f2dc6bdea test: fix file permissions on various scripts (windsok)
Pull request description:
Adds a new python linter test which tests for correct filenames and file permissions in the repository.
Replaces the existing tests in the `test/lint/lint-filenames.sh` and `test/lint/lint-shebang.sh` linter tests, as well as adding some new and increased testing. This increased coverage is intended to catch issues such as in #21728 and https://github.com/bitcoin/bitcoin/pull/16807/files#r345547050
Summary of tests:
* Checks every file in the repository against an allowed regexp to make sure only lowercase or uppercase alphanumerics (a-zA-Z0-9), underscores (_), hyphens (-), at (@) and dots (.) are used in repository filenames.
* Checks only source files (*.cpp, *.h, *.py, *.sh) against a stricter allowed regexp to make sure only lowercase alphanumerics (a-z0-9), underscores (_), hyphens (-) and dots (.) are used in source code filenames. Additionally there is an exception regexp for directories or files which are excepted from matching this regexp (This should replicate the existing `test/lint/lint-filenames.sh` test)
* Checks all files in the repository match an allowed executable or non-executable file permission octal. Additionally checks that for executable files, the file contains a shebang line.
* Checks that for executable `.py` and `.sh` files, the shebang line used matches an allowable list of shebangs (This should replicate the existing `test/lint/lint-shebang.sh` test)
* Checks every file that contains a shebang line to ensure it has an executable permission
Additionally updates the permissions on various files to comply with the new tests.
Fixes#21729
ACKs for top commit:
practicalswift:
cr re-ACK 46b025e00df40724175735eb5606ac73067cb3b8: patch still looks correct
kiminuo:
code review ACK 46b025e00df40724175735eb5606ac73067cb3b8 if `contrib/gitian-descriptors/assign_DISTNAME` permission change is deemed OK.
laanwj:
Code review ACK 46b025e00df40724175735eb5606ac73067cb3b8
Tree-SHA512: 1c8201a2cee0d9cbce15652b68cec9a6458a8b493fcd5392f98560aca0b1a12e668baab65a47100f116f626dadc3f591deb47f7368468c6a46c6c712c2533455
663f6cd9ddadeec30b27ec12f0f5ed49f3146cc9 contrib: Use -daemonwait in systemd init script (Wladimir J. van der Laan)
Pull request description:
Make systemd invoke dependencies only when ready by using `-daemonwait` in the service file instead of `-daemon`.
Closes#21322 by making bitcoind conform to behavior specified for `type=forking`.
This may need some tuning of timeouts.
ACKs for top commit:
darosior:
ACK 663f6cd
hebasto:
re-ACK 663f6cd9ddadeec30b27ec12f0f5ed49f3146cc9
Tree-SHA512: 890005852b632a202caa578e6c796ebdc9da0b2379a9157a4f56f7db9d193c0ffbb78d120bbf112ab2f273855f2a08c3da000b1f7a9fb5222a3b94dcdb16b878
5f2be6e71e6130b58ebfbf81aaf48ce90dd9d179 Remove no longer used contrib/bitcoin-qt.pro from the repo (Hennadii Stepanov)
Pull request description:
From [IRC](http://www.erisian.com.au/bitcoin-core-dev/log-2021-04-15.html#l-209):
> \<hebasto> wumpus: I cannot see any way how the `contrib/bitcoin-qt.pro` is used in the translation process, neither in the main repo nor in https://github.com/bitcoin-core/bitcoin-maintainer-tools. Besides it looks outdated and unmaintained. May I ask you to confirm/deny my assumption?
> \<wumpus> hebasto: it is not used for anything, it exists to be able to edit the qt forms in qt designer nothing more
> \<wumpus> i'm not sure if it is even *necessary* for that, but it is why it is there
> \<hebasto> wumpus: thanks, qt designer does not need *.pro file at all
> \<hebasto> maybe qt creator does
> \<wumpus> feel free to create a PR to remove it, best way to find out if someone wants to keep it, you are right it hasn't been updated in a long time
> \<hebasto> ok
> \<wumpus> fwiw, the only question i get about it ever is why it exists
> \<hebasto> it was in use with `qmake` years ago (what I found digging into the repo history)
> \<wumpus> yes, that was the original reason, but when we switched to automake it was kept around for use w/ qt's GUI tools
> \<hebasto> I've noticed it in https://github.com/bitcoin/bitcoin/blame/master/doc/translation_process.md#L25
> \<wumpus> what it says there is definitely not true anymore
ACKs for top commit:
laanwj:
ACK 5f2be6e71e6130b58ebfbf81aaf48ce90dd9d179
jarolrod:
ACK 5f2be6e71e6130b58ebfbf81aaf48ce90dd9d179
Tree-SHA512: 7c105612f28185097fee9e4108b162b4c8b07cc527f4438bdf5bcab08c65421ea301de8584d58770cd113fa871f6781daa8145bd6463278523449e28bfc49d06
005a6b104a fix: format string in llmq/commitment - mismatched arguments (Konstantin Akimov)
4774e1e8f6 Merge #19809: log: Prefix log messages with function name and source code location if -logsourcelocations is set (Wladimir J. van der Laan)
43a94f0580 fix: adjust functional tests due to dash's support of thread name after v0.12 (Konstantin Akimov)
085120d9f9 Merge #20993: test: store subversion (user agent) as string in msg_version (MarcoFalke)
e866b43160 Merge #21542: ci: Bump macOS VM image to the latest version (fanquake)
a3702534e5 Merge #21354: build, doc: Drop no longer required packages from macOS cross-compiling dependencies (fanquake)
6bcc86ad3b Merge #21221: [tools] Allow argument/parameter bin packing in clang-format (MarcoFalke)
318c7263d0 Merge #19522: build: fix building libconsensus with reduced exports for Darwin targets (Wladimir J. van der Laan)
88a45d4a9a Merge #21138: ci: Re-run wine tests once if they fail (fanquake)
4abb768456 Merge #21126: ci: Properly bump to focal for win cross build (fanquake)
f254f77d75 Merge #21075: doc: Fix markdown formatting (MarcoFalke)
Pull request description:
## Issue being fixed or feature implemented
Regular backports from bitcoin v22 and related fixes
## What was done?
Follow-up fixes for bitcoin#19809
Backports:
- bitcoin/bitcoin#21075
- bitcoin/bitcoin#21126
- bitcoin/bitcoin#21138
- bitcoin/bitcoin#19522
- bitcoin/bitcoin#21221
- bitcoin/bitcoin#21354
- bitcoin/bitcoin#21542
- bitcoin/bitcoin#19809
- bitcoin/bitcoin#20993
## How Has This Been Tested?
Run unit/functional tests
## Breaking Changes
N/A
Notice, that function name is included now by default to logs with `-logfunctionnames` and many logs have function name twice now such as:
```
node0 2024-04-06T20:13:56.564123Z (mocktime: 2014-12-04T17:15:38Z) [httpworker.3] [masternode/sync.cpp:331] [NotifyHeaderTip] CMasternodeSync::NotifyHeaderTip -- pindexNew->nHeight: 5 fInitialDownload=0
```
For further development need to take it in account and do not use more direct calls `__func__` from code as well as reduce usages in codebase.
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
ACKs for top commit:
PastaPastaPasta:
utACK 005a6b104a
Tree-SHA512: f98949c4605dda7d6dfe790554e1d31a8c8178b57520578fcd58178c68fe7af33c0d66a79865683c1357de9a73fa4f484eb828b28e11ca110b8e1915c1fcf9b3
f7f3829a68df173c54ab11ecfe162b840afb64e8 build, doc: Drop libbz2-dev from macOS cross-compiling dependencies (Hennadii Stepanov)
d8239362e283107657a5a6577890fdb198a1d334 build, doc: Drop libcap-dev from macOS cross-compiling dependencies (Hennadii Stepanov)
Pull request description:
The `libcap-dev` and `libbz2-dev` packages are no longer required when cross-compiling for macOS.
ACKs for top commit:
fanquake:
ACK f7f3829a68df173c54ab11ecfe162b840afb64e8
Tree-SHA512: 820cdc2724f3346c0942d4d4115fc7206f7bf02889d9fa6cbdbd1d9e3afa03a067c1c3fa64dff596aefdc74898178b7c7d64027a6501486e3b606f4760de04ae
fa0074e2d82928016a43ca408717154a1c70a4db scripted-diff: Bump copyright headers (MarcoFalke)
Pull request description:
Needs to be done because no one has removed the years yet
ACKs for top commit:
practicalswift:
ACK fa0074e2d82928016a43ca408717154a1c70a4db
Tree-SHA512: 210e92acd7d400b556cf8259c3ec9967797420cfd19f0c2a4fa54cb2b3d32ad9ae27e771269201e7d554c0f4cd73a8b1c1a42c9f65d8685ca4d52e5134b071a3
961f148cb1b4caab86b4354357999e03433b04b1 doc: update contrib/seeds/README dnspython installation info (Jon Atack)
dd7b5f46d85401254630abf6976f59b5b8eed181 script: fix deprecation warning in makeseeds.py (Jon Atack)
Pull request description:
Seen while reviewing #20237.
1. Fix a deprecation warning in `contrib/seeds/makeseeds.py`
```
makeseeds.py:139: DeprecationWarning: please use dns.resolver.resolve() instead
asn = int([x.to_text() for x in dns.resolver.query('.'.join(
```
- Per https://dnspython.readthedocs.io/en/latest/whatsnew.html, `dns.resolver.query()` was deprecated in `dnspython` version 2.0.0.
- See https://dnspython.readthedocs.io/en/latest/resolver-class.html for more info on the resolver class.
2. Update the `dnspython` dependency installation instructions in `contrib/seeds/README`
- The markdown rendering can be seen here: https://github.com/jonatack/bitcoin/tree/contrib-seeds-fixups/contrib/seeds
ACKs for top commit:
laanwj:
code review ACK 961f148cb1b4caab86b4354357999e03433b04b1
Tree-SHA512: f9c4f318a1a0d35b8de147d24b72c534a1f58eece31e7cfa00b4149a63b6a618d8ca0312f52fd8056f3c645cf2ee68574ca02319fddffdad919a70cd33395d33
1ef2138c0db3bd4f9332c777fa3fb2770dc1b08c lint: run mypy over contrib/devtools (fanquake)
Pull request description:
wumpus mentioned on IRC that we don't currently run `mypy` over the `contrib/devtools` directory, and that it would likely be worthwhile given #20434. This just adds that dir to the linter, as well as some missing annotations to fix existing errors. Note that now we require Python 3.6 we can make use of variable annotations.
master (patched to check contrib devtools):
```bash
test/lint/lint-python.sh
contrib/devtools/symbol-check.py:154: error: Incompatible types in assignment (expression has type "List[str]", variable has type "str")
contrib/devtools/circular-dependencies.py:35: error: Need type annotation for 'deps' (hint: "deps: Dict[<type>, <type>] = ...")
contrib/devtools/circular-dependencies.py:67: error: Need type annotation for 'closure' (hint: "closure: Dict[<type>, <type>] = ...")
Found 4 errors in 3 files (checked 187 source files)
```
I haven't quite gone as far as to add annotations like
```python
CHECKS: Dict[str, List[Tuple[str, Callable[[Any], bool]]]] = {...
```
to `symbol-check.py`.
ACKs for top commit:
laanwj:
ACK 1ef2138c0db3bd4f9332c777fa3fb2770dc1b08c
Tree-SHA512: a58c2ece588c640289dc1d35dad5b1b8732788272daa0965d6bf44ee8a7f7c8e8585f94d233ac41c84b9ffcfc97841a00fe2c9acba41f58fd164f01de4b6512b
a93de8690b refactor: s/governanceManager/govman/g (Kittywhiskers Van Gogh)
0aa08ba80d refactor: remove CGovernanceManager global, move to NodeContext (Kittywhiskers Van Gogh)
405b8c669a refactor: s/sporkManager/sporkman/g (Kittywhiskers Van Gogh)
60fd1aa774 refactor: remove CSporkManager global, move to NodeContext (Kittywhiskers Van Gogh)
24ba2f027c refactor: remove redundant condition check in `IsOldBudgetBlockValueValid` (Kittywhiskers Van Gogh)
e2405e67fb refactor: move MasternodePayments::* functions into helper class (Kittywhiskers Van Gogh)
Pull request description:
## Additional Information
* `PeerManager`'s initialization has been moved downwards (it used to be initialized _after_ `CGovernanceManager`) to now _after_ `CMasternodeSync`. This is to avoid having to pass `CSporkManager`'s `unique_ptr` container and instead pass the its dereferenced pointer.
* `CChainstateHelper` is just proxy for helper classes meant to hold references to managers that would be needed by functions that are called by `CChainState`. It's the alternative to passing every single manager into `CChainState` through `ChainstateManager`.
Instead, they're all bunched up via `CChainstateHelper` and is accessible to `CChainState` through passing it as an argument. For this reason, it should ideally initialized _after_ all relevant managers are setup but _before_ the chain is validated. We would want to avoid deferred dereferencing if we can help it.
* Internal/private functions have been marked as `[[nodiscard]]`.
## Breaking Changes
None. Changes are limited to refactoring, no logical changes have been made.
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
- [x] I have added or updated relevant unit/integration/functional/e2e tests
- [x] I have made corresponding changes to the documentation **(note: N/A)**
- [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_
ACKs for top commit:
PastaPastaPasta:
utACK a93de8690b
Tree-SHA512: b576ca61a94b86b8a6fa8909379d156ff902198b5824fcfb4665e5eb9d1d5e250db737f84a877de634a490146b82759c2350370903f430997423fd71142106d1
```
$ ./makeseeds.py protx.txt > nodes_main.txt
Traceback (most recent call last):
File "DASH/contrib/seeds/./makeseeds.py", line 183, in <module>
main()
File "DASH/contrib/seeds/./makeseeds.py", line 167, in main
for onion in onions:
^^^^^^
UnboundLocalError: cannot access local variable 'onions' where it is not associated with a value
```
644772b9efffda4dac01aff54042b3162079514d message-capture-parser: fix AssertionError on parsing `headers` message (Sebastian Falbesoner)
Pull request description:
If a test framework message's field name is in the list of `HASH_INT_VECTORS`, we currently assume that it _always_ has to contain a vector of integers and throw otherwise:
0ebd4db32b/contrib/message-capture/message-capture-parser.py (L82-L83)
(introduced in PR #25367, commit 42bbbba7c83d1e2baad18b4c6f05bad1358eb117).
However, that assumption is too strict. The (de)serialization field name "headers" is used in two different message types, one for `cfcheckpt` (where it is serialized as an integer vector), and another time for `headers` (where it is serialized as a vector of `CBlockHeader`s). Parsing the latter fails as it is not an integer vector and thus triggers the assert.
Fix this by adding the integer type check as additional condition to the `HASH_INT_VECTORS` check rather than asserting.
Fixes#25954.
ACKs for top commit:
glozow:
ACK 644772b9efffda4dac01aff54042b3162079514d
Tree-SHA512: c98a107f6703c6c1a81771907c25bcc171c631b57fd605fbebaedd93d651e2ef02fb5601853a9bc7d659ab531c5f47770181173a36ea2b37f584aa7a37b66505
7a0b129c41d9fefdbc20d6d04983dd87bb8379e7 guix: patch NSIS to remove .reloc sections from install stubs (fanquake)
Pull request description:
With the release of binutils/ld 2.36, ld swapped to much improved
default settings when producing windows binaries with mingw-w64. One of
these changes was to stop stripping the .reloc section from binaries,
which is required for working ASLR.
When we switched to using a newer Guix time-machine in #23778, we begun
using binutils 2.37 to produce releases. Since then, our windows
installer (produced with makensis) has not functioned correctly when run on
a Windows system with the "Force randomization for images (Mandatory ASLR)"
option enabled. Note that all of our other release binaries, which all
contain .reloc sections, function fine under the same option, so it
cannot be just the presence of a .reloc section that is the issue.
The root cause of the problem is that when we compile NSIS (makensis), a number
of exe installer stubs are produced at the same time, for use later when makensis
is actually run. Given the new linker defaults, the stubs will contain .reloc sections,
when previously they would not. It seems that, in combination with how makensis
mutates the stub when it actually builds the installer, causes the problem.
According to upstream, https://sourceforge.net/p/nsis/bugs/1131/#abb6:
> Looks like the problem is the very existance of the .reloc section.
> It's not supposed to be there, and makensis doesn't handle it.
The most recent .reloc related upstream activity is in
https://sourceforge.net/p/nsis/bugs/1283/, where the conclusion again seemed to
be that .relo sections are not wanted, but there hasn't been any further follow up.
For now, restore pre-binutils-2.36 behaviour, by passing `-Wl,--disable-reloc-section`
to the linker when building the installer stubs, which fixes the produced installer.
The underlying issue can be further investigated in future.
.reloc section stripping is something we've accounted for previously,
see #18702, and related upstream discussion is in this thread:
https://sourceware.org/bugzilla/show_bug.cgi?id=19011.
Fixes#25726.
Guix Build (x86_64):
```bash
7e0723388913ac1ec9f650b943c6b23351ba0cd921c0ec830abf16b16724d503 guix-build-7a0b129c41d9/output/dist-archive/bitcoin-7a0b129c41d9.tar.gz
c3bb9c68895ffafa2900b0d18c1268e299d012a7dc70593f20f9900cf116eb05 guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/SHA256SUMS.part
b57aa99c242b0aae64653c64ada38f6d3f0cbd902bbc096d3dc529fdcf87d681 guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/bitcoin-7a0b129c41d9-win64-debug.zip
341d99afc9961299883be6cd9666e8bc0f3f6296cff758719a32d27419acad36 guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/bitcoin-7a0b129c41d9-win64-setup-unsigned.exe
1d9ef48d3c9ed93a925962356b41cdaeb9d09fd758de193cd4d5f4d1ec6791eb guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/bitcoin-7a0b129c41d9-win64-unsigned.tar.gz
28c81d99a9a4bd6648449393f91db213369e958add579ba9e9a1721540d2c4f7 guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/bitcoin-7a0b129c41d9-win64.zip
```
Guix Build (arm64):
```bash
7e0723388913ac1ec9f650b943c6b23351ba0cd921c0ec830abf16b16724d503 guix-build-7a0b129c41d9/output/dist-archive/bitcoin-7a0b129c41d9.tar.gz
c3bb9c68895ffafa2900b0d18c1268e299d012a7dc70593f20f9900cf116eb05 guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/SHA256SUMS.part
b57aa99c242b0aae64653c64ada38f6d3f0cbd902bbc096d3dc529fdcf87d681 guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/bitcoin-7a0b129c41d9-win64-debug.zip
341d99afc9961299883be6cd9666e8bc0f3f6296cff758719a32d27419acad36 guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/bitcoin-7a0b129c41d9-win64-setup-unsigned.exe
1d9ef48d3c9ed93a925962356b41cdaeb9d09fd758de193cd4d5f4d1ec6791eb guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/bitcoin-7a0b129c41d9-win64-unsigned.tar.gz
28c81d99a9a4bd6648449393f91db213369e958add579ba9e9a1721540d2c4f7 guix-build-7a0b129c41d9/output/x86_64-w64-mingw32/bitcoin-7a0b129c41d9-win64.zip
```
ACKs for top commit:
achow101:
ACK 7a0b129c41d9fefdbc20d6d04983dd87bb8379e7
hebasto:
ACK 7a0b129c41d9fefdbc20d6d04983dd87bb8379e7
jarolrod:
ACK 7a0b129c41d9fefdbc20d6d04983dd87bb8379e7
Tree-SHA512: 9e14e98207d20236b833603319fc4bb335c878a7c179ab495b33d143e2a900c6926125536bbb7499ee4f0f676cd5ea45c8c86cd7e544ed9a76bb298f98db6197
this will allow us to rebase the PR before merging as we do now; and then right after merge it into develop and push
note we can not simply rebase locally before merging as we would then violate the "all changes must be done through PR rule"
When rebasing locally; we also check the range-diff (should be all >'s indicating a commit being pulled in from the rebase and ='s indicating the commits are the same as the base PR. This ensures that when we force push we will not invalidate previously created reviews.
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
42bbbba7c83d1e2baad18b4c6f05bad1358eb117 message-capture-parser: fix out of bounds error for empty vectors (Sebastian Falbesoner)
Pull request description:
The script [message-capture-parser.py](https://github.com/bitcoin/bitcoin/blob/master/contrib/message-capture/message-capture-parser.py) currently throws an "out of bounds" error if a message containing an empty integer vector element is tried to converted to JSON (e.g. by the BIP157 message `cfcheckpt` with empty `FilterHeaders` vector):
```
Traceback (most recent call last):
File "/home/honey/bitcoin/./contrib/message-capture/message-capture-parser.py", line 217, in <module>
main()
File "/home/honey/bitcoin/./contrib/message-capture/message-capture-parser.py", line 202, in main
process_file(str(capture), messages, "recv" in capture.stem, progress_bar)
File "/home/honey/bitcoin/./contrib/message-capture/message-capture-parser.py", line 162, in process_file
msg_dict["body"] = to_jsonable(msg)
File "/home/honey/bitcoin/./contrib/message-capture/message-capture-parser.py", line 85, in to_jsonable
elif slot in HASH_INT_VECTORS and isinstance(val[0], int):
IndexError: list index out of range
```
Fix this by using the `all(...)` predicate rather to access the first element `val[0]` (which in the error case doesn't exist).
ACKs for top commit:
laanwj:
Code review ACK 42bbbba7c83d1e2baad18b4c6f05bad1358eb117
Tree-SHA512: 139ec6b90304a69f26ec731e6f12b216fa10e554f777505b61adfa1e569f6861a4a849159dd1eae7a1aa0427e8598af226b6f0c4015020dcac8ab109fbc35dba
that's a result of:
contrib/devtools/copyright_header.py update ./
it is not scripted diff, because it works differentlly on my localhost and in CI:
CI doesn't want to use git commit date which is mocked to 30th Dec of 2023
## What was done?
Happy new year and happy new lunar year!
2024 is here and 20.1 is coming.
## Breaking Changes
N/A
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone
## Issue being fixed or feature implemented
We did not previously ship any onion seeds. This results in people
needing to use `addnode` in order to actually get connected
## What was done?
Modified seed creation process to handle a list of onion seeds.
## How Has This Been Tested?
Running with and without onlynet=onion and with dnsseed=0 and deleting
peers.dat
## Breaking Changes
None
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
bff7c66e67aa2f18ef70139338643656a54444fe Add documentation to contrib folder (Troy Giorshev)
381f77be858d7417209b6de0b7cd23cb7eb99261 Add Message Capture Test (Troy Giorshev)
e4f378a505922c0f544b4cfbfdb169e884e02be9 Add capture parser (Troy Giorshev)
4d1a582549bc982d55e24585b0ba06f92f21e9da Call CaptureMessage at appropriate locations (Troy Giorshev)
f2a77ff97bec09dd5fcc043d8659d8ec5dfb87c2 Add CaptureMessage (Troy Giorshev)
dbf779d5deb04f55c6e8493ce4e12ed4628638f3 Clean PushMessage and ProcessMessages (Troy Giorshev)
Pull request description:
This PR introduces per-peer message capture into Bitcoin Core. 📓
## Purpose
The purpose and scope of this feature is intentionally limited. It answers a question anyone new to Bitcoin's P2P protocol has had: "Can I see what messages my node is sending and receiving?".
## Functionality
When a new debug-only command line argument `capturemessages` is set, any message that the node receives or sends is captured. The capture occurs in the MessageHandler thread. When receiving a message, it is captured as soon as the MessageHandler thread takes the message off of the vProcessMsg queue. When sending, the message is captured just before the message is pushed onto the vSendMsg queue.
The message capture is as minimal as possible to reduce the performance impact on the node. Messages are captured to a new `message_capture` folder in the datadir. Each node has their own subfolder named with their IP address and port. Inside, received and sent messages are captured into two binary files, msgs_recv.dat and msgs_sent.dat, like so:
```
message_capture/203.0.113.7:56072/msgs_recv.dat
message_capture/203.0.113.7:56072/msgs_sent.dat
```
Because the messages are raw binary dumps, included in this PR is a Python parsing tool to convert the binary files into human-readable JSON. This script has been placed on its own and out of the way in the new `contrib/message-capture` folder. Its usage is simple and easily discovered by the autogenerated `-h` option.
## Future Maintenance
I sympathize greatly with anyone who says "the best code is no code".
The future maintenance of this feature will be minimal. The logic to deserialize the payload of the p2p messages exists in our testing framework. As long as our testing framework works, so will this tool.
Additionally, I hope that the simplicity of this tool will mean that it gets used frequently, so that problems will be discovered and solved when they are small.
## FAQ
"Why not just use Wireshark"
Yes, Wireshark has the ability to filter and decode Bitcoin messages. However, the purpose of the message capture added in this PR is to assist with debugging, primarily for new developers looking to improve their knowledge of the Bitcoin Protocol. This drives the design in a different direction than Wireshark, in two different ways. First, this tool must be convenient and simple to use. Using an external tool, like Wireshark, requires setup and interpretation of the results. To a new user who doesn't necessarily know what to expect, this is unnecessary difficulty. This tool, on the other hand, "just works". Turn on the command line flag, run your node, run the script, read the JSON. Second, because this tool is being used for debugging, we want it to be as close to the true behavior of the node as possible. A lot can happen in the SocketHandler thread that would be missed by Wireshark.
Additionally, if we are to use Wireshark, we are at the mercy of whoever it maintaining the protocol in Wireshark, both as to it being accurate and recent. As can be seen by the **many** previous attempts to include Bitcoin in Wireshark (google "bitcoin dissector") this is easier said than done.
Lastly, I truly believe that this tool will be used significantly more by being included in the codebase. It's just that much more discoverable.
ACKs for top commit:
MarcoFalke:
re-ACK bff7c66e67aa2f18ef70139338643656a54444fe only some minor changes: 👚
jnewbery:
utACK bff7c66e67aa2f18ef70139338643656a54444fe
theStack:
re-ACK bff7c66e67aa2f18ef70139338643656a54444fe
Tree-SHA512: e59e3160422269221f70f98720b47842775781c247c064071d546c24fa7a35a0e5534e8baa4b4591a750d7eb16de6b4ecf54cbee6d193b261f4f104e28c15f47
4133c8104f522c403c55d26bd03436a8149ff106 guix: use gcc tool wrappers (fanquake)
Pull request description:
This way, correct `--plugin` arguments are passed through.
This is a prerequisite for LTO (see #25391). Split out, to try move things along, as this change is isolated, and should be straight-forward.
ACKs for top commit:
TheCharlatan:
ACK [4133c81](4133c8104f)
hebasto:
ACK 4133c8104f522c403c55d26bd03436a8149ff106
Tree-SHA512: 4311a72a613cf027bd4490caa29604c985ed455589acd972285f13cbdf4806d2184a4dc6f20cb6f47c3fa751d58bfd0bacc257b87d4a804bf5ecf5b240e4a757
4becee396f3bda40832138dd1aaa90368ed31857 guix: combine and document enable_werror (fanquake)
Pull request description:
Combine into `hardened-glibc`.
Document why we don't use `--disable-werror` directly.
https://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html
> By default, the GNU C Library is built with -Werror. If you wish
> to build without this option (for example, if building with a
> newer version of GCC than this version of the GNU C Library was
> tested with, so new warnings cause the build with -Werror to fail),
> you can configure with --disable-werror.
ACKs for top commit:
hebasto:
ACK 4becee396f3bda40832138dd1aaa90368ed31857, the diff is correct.
TheCharlatan:
ACK 4becee396f3bda40832138dd1aaa90368ed31857
Tree-SHA512: 8724415f51b4d72d40c4e797faf52c93a81147fb629332b9388ffd7f113f2b16db3b7496bf3063dd978ac629fd5bde3ec7df4f1ff1ed714cb56f316a9334d119
127c637cf0a80e0ea68a7c5aaa088e5ccc9d3d13 guix: pass --enable-initfini-array to release GCC (fanquake)
Pull request description:
This returns us to pre-Guix behaviour, where the compilers we were using to build releases, were configured with this option.
> [--enable-initfini-array](https://gcc.gnu.org/install/configure.html)
> Force the use of sections .init_array and .fini_array (instead of .init and .fini) for constructors and destructors. Option --disable-initfini-array has the opposite effect. If neither option is specified, the configure script will try to guess whether the .init_array and .fini_array sections are supported and, if they are, use them.
ACKs for top commit:
TheCharlatan:
ACK 127c637cf0a80e0ea68a7c5aaa088e5ccc9d3d13
vincenzopalazzo:
utACK 127c637cf0
Tree-SHA512: fa61227054d52d4dfb4524af3888203a501f680661bdef00bb0970d4e8f7c96cf7f592686c4795be5a0debca267b8e564a4960859297c31f6b261c0729238382
10660c0c60f651a52ba9c86c7dba4fa232ed6583 doc: move Guix uninstall instructions to INSTALL.md (Sjors Provoost)
68fab72a8ca7cb8fb26a154a43efd998b7f78738 guix: OpenSSL test failure workaround (Sjors Provoost)
d612dca852db493531f4c3f51e6ea9987cd5db37 guix: reminder to migrate guix-daemon-original customization (Sjors Provoost)
8aa460cd02a6ab1229463c59e965203e52b34748 guix: add guile-gnutls and guile-json to install list (Sjors Provoost)
9b9991e02693c68061ccd4d6040641e20f934e6c guix: recommend mounting a tmpfs on /tmp (Sjors Provoost)
682283445e2cc815cf2786da83314fa8b8350511 guix: bump recommended hash for manual installation (Sjors Provoost)
Pull request description:
I'm manually installing Guix on a fresh Ubuntu machine. Will be pushing more documentation fixes to this PR as I run into things.
1. Bump minimum hash to match time-machine bump in #25099. It's not necessary for the root Guix version to match the time-machine version in our build, because `guix build` will automatically perform an upgrade for the user, but imo it's better to get any build issues (in Guix itself) over with while the user is going though `INSTALL.md`, rather than during their first Guix build (of Bitcoin Core).
2. Recommend mapping a tmpfs to /tmp upfront, rather than in the troubleshooting section
3. Add `guile-gnutls` and `guile-json` to the table of stuff to install (avoids having to find out in the `./configure` phase)
4. Improve systemd doc
5. Workaround OpenSSL v1.1.1l and v1.1.1n test failure (change machine time)
6. Move uninstallation instructions to INSTALL.md, drop unused footnote / links
ACKs for top commit:
jamesob:
ACK 10660c0c60
Tree-SHA512: ff1278b16f03ea9c63e23e97a852340ab824d5f6c64645cb70237dd828b9a439b4133b60cd2b89672573f6546e99419021d092e236f731908158a7aa6473b0ef
0cd7928133eb8a605979c6338bbcbcb116cfa669 guix: use git-minimal over git (fanquake)
Pull request description:
From the [git-minimal package definition](https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/version-control.scm?id=998eda3067c7d21e0d9bb3310d2f5a14b8f1c681#n597):
> The size of the closure of 'git-minimal' is two thirds that of 'git'.
> Its test suite runs slightly faster and most importantly it doesn't
> depend on packages that are expensive to build such as Subversion.
We don't need any git functionality above the basics, so switch to `git-minimal` and save CPU when building the package, while also pruning the greater dependency graph (see `dependencies:` below). Note that git-minimal also lists `riscv64-linux` as a supported system, where `git` does not.
```diff
-name: git
+name: git-minimal
version: 2.37.3
outputs:
-+ send-email: see Appendix H
-+ svn: see Appendix H
-+ credential-netrc: see Appendix H
-+ credential-libsecret: see Appendix H
-+ subtree: see Appendix H
-+ gui: see Appendix H
+ out: everything else
-systems: x86_64-linux mips64el-linux aarch64-linux powerpc64le-linux i686-linux armhf-linux powerpc-linux
-dependencies: asciidoc@9.1.0 bash-minimal@5.1.8 bash@5.1.8 curl@7.79.1 docbook-xsl@1.79.2 expat@2.4.1 gettext-minimal@0.21 glib@2.70.2 libsecret@0.20.4 openssl@1.1.1l pcre2@10.37perl-authen-sasl@2.16perl-cgi@4.52
-+ perl-io-socket-ssl@2.068perl-net-smtp-ssl@1.04perl-term-readkey@2.38 perl@5.34.0 pkg-config@0.29.2 python@3.9.9 subversion@1.14.1 tcl@8.6.11 tk@8.6.11.1 xmlto@0.0.28 zlib@1.2.11
-location: gnu/packages/version-control.scm:222:2
+systems: x86_64-linux mips64el-linux aarch64-linux powerpc64le-linux riscv64-linux i686-linux armhf-linux powerpc-linux
+dependencies: bash-minimal@5.1.8 bash@5.1.8 curl@7.79.1 expat@2.4.1 gettext-minimal@0.21 openssl@1.1.1l perl@5.34.0 zlib@1.2.11
+location: gnu/packages/version-control.scm:608:2
homepage: https://git-scm.com/
license: GPL 2
synopsis: Distributed version control system
```
Guix Build (x86_64):
```bash
da4adca0304f19833893867418c8827e0213c58a1b605753355340a5f270754a guix-build-0cd7928133eb/output/aarch64-linux-gnu/SHA256SUMS.part
38c2b5f8e560018911ed776660fcd2aa8b6061a59af26118f06e23c9a335e80c guix-build-0cd7928133eb/output/aarch64-linux-gnu/bitcoin-0cd7928133eb-aarch64-linux-gnu-debug.tar.gz
de117782318d6e0ed55efaae7b2f11d033fe05e7a72fbda3ef7bbcbc758add69 guix-build-0cd7928133eb/output/aarch64-linux-gnu/bitcoin-0cd7928133eb-aarch64-linux-gnu.tar.gz
6ae8ebfac28c43488b9aa386b9a87937789a57e54dc1d77a9c7b95323a417abc guix-build-0cd7928133eb/output/arm-linux-gnueabihf/SHA256SUMS.part
97f5d9d14eeb4b2926304c142fa6c46b7126524b8f836655704f5643b58b9436 guix-build-0cd7928133eb/output/arm-linux-gnueabihf/bitcoin-0cd7928133eb-arm-linux-gnueabihf-debug.tar.gz
37815ea73941cf0a870e5ac4aafe9249a63ed1eeaa37440de23c2d9bf2b77be8 guix-build-0cd7928133eb/output/arm-linux-gnueabihf/bitcoin-0cd7928133eb-arm-linux-gnueabihf.tar.gz
64cd484fa48968dc7063c4f501e1ff62d1ba46ae9975bfa060a3c88e2a98d232 guix-build-0cd7928133eb/output/arm64-apple-darwin/SHA256SUMS.part
4e7e0daaf0ac1b5ed5a7e5ee8085e5e6446c48e70161f78938acd0e916c55729 guix-build-0cd7928133eb/output/arm64-apple-darwin/bitcoin-0cd7928133eb-arm64-apple-darwin-unsigned.dmg
0f2b534d16482e536552c7b3de605bd71997b898755fe5a9ac39b36aea2698b6 guix-build-0cd7928133eb/output/arm64-apple-darwin/bitcoin-0cd7928133eb-arm64-apple-darwin-unsigned.tar.gz
03cd1f509c60919c2ad1503d2f98be444c9770b62c4d303cb4cbdc1100ce131d guix-build-0cd7928133eb/output/arm64-apple-darwin/bitcoin-0cd7928133eb-arm64-apple-darwin.tar.gz
1e28183c1c314921a8404b72283bb861dff28061310c18535618683b097e7e61 guix-build-0cd7928133eb/output/dist-archive/bitcoin-0cd7928133eb.tar.gz
0f6459568d0369528ad35622d5378feccdac319eed618418841c22cc137cbd05 guix-build-0cd7928133eb/output/powerpc64-linux-gnu/SHA256SUMS.part
1cf0c8a48add60082c381935630b59a0bd483a7eda97f04b72dcb05143135109 guix-build-0cd7928133eb/output/powerpc64-linux-gnu/bitcoin-0cd7928133eb-powerpc64-linux-gnu-debug.tar.gz
5332f148efa1579b077747c8c7d6c763d31804d4ac454abaf34a3e2374c9b6b2 guix-build-0cd7928133eb/output/powerpc64-linux-gnu/bitcoin-0cd7928133eb-powerpc64-linux-gnu.tar.gz
5fc03945c2ab86ba43395ccf32cf4b338dcceb446e106c0f6e660dac47224183 guix-build-0cd7928133eb/output/powerpc64le-linux-gnu/SHA256SUMS.part
5cfabdb27dc8fb7de402c558e5f962ac4fdaf2c344d201f27f7ed1370a550407 guix-build-0cd7928133eb/output/powerpc64le-linux-gnu/bitcoin-0cd7928133eb-powerpc64le-linux-gnu-debug.tar.gz
ba265df6803d472434ecb3ad44983965a5eca1ccd42fea64760309ff70d17ee5 guix-build-0cd7928133eb/output/powerpc64le-linux-gnu/bitcoin-0cd7928133eb-powerpc64le-linux-gnu.tar.gz
ff40a374f215eb3010291569b8ed1958054e408469fc8b2fe97a30cca0ad5451 guix-build-0cd7928133eb/output/riscv64-linux-gnu/SHA256SUMS.part
7b7b89ac1905d58f1e96a7840c018a556c472015a44442d0742bf758cb5f67ca guix-build-0cd7928133eb/output/riscv64-linux-gnu/bitcoin-0cd7928133eb-riscv64-linux-gnu-debug.tar.gz
10431bd8ffca82dd9c59f568272a1e7473cf474996f750d9bed4b576591fcff1 guix-build-0cd7928133eb/output/riscv64-linux-gnu/bitcoin-0cd7928133eb-riscv64-linux-gnu.tar.gz
4ef532d8dbe42900146a5b3e02de2a6a59d66b3c66a4b9d919d3aeb0e9637ab1 guix-build-0cd7928133eb/output/x86_64-apple-darwin/SHA256SUMS.part
77a1abe4139c19d227309216e29cf55dae06c4469412b457c9f0e8cf1eccc25c guix-build-0cd7928133eb/output/x86_64-apple-darwin/bitcoin-0cd7928133eb-x86_64-apple-darwin-unsigned.dmg
33028b640efab25648d0ec1abe9e91abc983706623ca9e2e7ac5fbfca0970909 guix-build-0cd7928133eb/output/x86_64-apple-darwin/bitcoin-0cd7928133eb-x86_64-apple-darwin-unsigned.tar.gz
e10d2d5617b8b1a33a622d5904d2bd8eaf57a5b3605e22ef916a57105db2311e guix-build-0cd7928133eb/output/x86_64-apple-darwin/bitcoin-0cd7928133eb-x86_64-apple-darwin.tar.gz
bf65d3574afed2e017c9625d38cc31e0f2cbb7f1e8a9ce346644ea3dbb938d13 guix-build-0cd7928133eb/output/x86_64-linux-gnu/SHA256SUMS.part
ce3810e70c97b2698822e4f46fa64dfa12353f7b54400e671b64868e3e4d3472 guix-build-0cd7928133eb/output/x86_64-linux-gnu/bitcoin-0cd7928133eb-x86_64-linux-gnu-debug.tar.gz
4055370c15b199d1efef47cc262d9c43a3652dcd237a9434197ca3be4931b1d2 guix-build-0cd7928133eb/output/x86_64-linux-gnu/bitcoin-0cd7928133eb-x86_64-linux-gnu.tar.gz
e59ed970d1db5d4839fa67957945628f6919ef5491f4a595f89ed3d8c81f1a76 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/SHA256SUMS.part
19c443fab5cb2fe75c9a5ad51fc022c97e31d7d69e049a889bd06f740f8daf78 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/bitcoin-0cd7928133eb-win64-debug.zip
88f6ca5d299080114532ec550c59eca4a3cdb759d9ea35cb14eba0b135e72436 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/bitcoin-0cd7928133eb-win64-setup-unsigned.exe
bcdb0b7467d3e47a694e51e9bfbaab9d5dc7162efe6c6bf4c303d368272c0cc6 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/bitcoin-0cd7928133eb-win64-unsigned.tar.gz
db1d4bbfab53405080d3abd09d1f05b2642ed513f6d8fcb5d92b9d0b32745293 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/bitcoin-0cd7928133eb-win64.zip
```
Guix Build (arm64):
```bash
da4adca0304f19833893867418c8827e0213c58a1b605753355340a5f270754a guix-build-0cd7928133eb/output/aarch64-linux-gnu/SHA256SUMS.part
38c2b5f8e560018911ed776660fcd2aa8b6061a59af26118f06e23c9a335e80c guix-build-0cd7928133eb/output/aarch64-linux-gnu/bitcoin-0cd7928133eb-aarch64-linux-gnu-debug.tar.gz
de117782318d6e0ed55efaae7b2f11d033fe05e7a72fbda3ef7bbcbc758add69 guix-build-0cd7928133eb/output/aarch64-linux-gnu/bitcoin-0cd7928133eb-aarch64-linux-gnu.tar.gz
6ae8ebfac28c43488b9aa386b9a87937789a57e54dc1d77a9c7b95323a417abc guix-build-0cd7928133eb/output/arm-linux-gnueabihf/SHA256SUMS.part
97f5d9d14eeb4b2926304c142fa6c46b7126524b8f836655704f5643b58b9436 guix-build-0cd7928133eb/output/arm-linux-gnueabihf/bitcoin-0cd7928133eb-arm-linux-gnueabihf-debug.tar.gz
37815ea73941cf0a870e5ac4aafe9249a63ed1eeaa37440de23c2d9bf2b77be8 guix-build-0cd7928133eb/output/arm-linux-gnueabihf/bitcoin-0cd7928133eb-arm-linux-gnueabihf.tar.gz
64cd484fa48968dc7063c4f501e1ff62d1ba46ae9975bfa060a3c88e2a98d232 guix-build-0cd7928133eb/output/arm64-apple-darwin/SHA256SUMS.part
4e7e0daaf0ac1b5ed5a7e5ee8085e5e6446c48e70161f78938acd0e916c55729 guix-build-0cd7928133eb/output/arm64-apple-darwin/bitcoin-0cd7928133eb-arm64-apple-darwin-unsigned.dmg
0f2b534d16482e536552c7b3de605bd71997b898755fe5a9ac39b36aea2698b6 guix-build-0cd7928133eb/output/arm64-apple-darwin/bitcoin-0cd7928133eb-arm64-apple-darwin-unsigned.tar.gz
03cd1f509c60919c2ad1503d2f98be444c9770b62c4d303cb4cbdc1100ce131d guix-build-0cd7928133eb/output/arm64-apple-darwin/bitcoin-0cd7928133eb-arm64-apple-darwin.tar.gz
1e28183c1c314921a8404b72283bb861dff28061310c18535618683b097e7e61 guix-build-0cd7928133eb/output/dist-archive/bitcoin-0cd7928133eb.tar.gz
0f6459568d0369528ad35622d5378feccdac319eed618418841c22cc137cbd05 guix-build-0cd7928133eb/output/powerpc64-linux-gnu/SHA256SUMS.part
1cf0c8a48add60082c381935630b59a0bd483a7eda97f04b72dcb05143135109 guix-build-0cd7928133eb/output/powerpc64-linux-gnu/bitcoin-0cd7928133eb-powerpc64-linux-gnu-debug.tar.gz
5332f148efa1579b077747c8c7d6c763d31804d4ac454abaf34a3e2374c9b6b2 guix-build-0cd7928133eb/output/powerpc64-linux-gnu/bitcoin-0cd7928133eb-powerpc64-linux-gnu.tar.gz
5fc03945c2ab86ba43395ccf32cf4b338dcceb446e106c0f6e660dac47224183 guix-build-0cd7928133eb/output/powerpc64le-linux-gnu/SHA256SUMS.part
5cfabdb27dc8fb7de402c558e5f962ac4fdaf2c344d201f27f7ed1370a550407 guix-build-0cd7928133eb/output/powerpc64le-linux-gnu/bitcoin-0cd7928133eb-powerpc64le-linux-gnu-debug.tar.gz
ba265df6803d472434ecb3ad44983965a5eca1ccd42fea64760309ff70d17ee5 guix-build-0cd7928133eb/output/powerpc64le-linux-gnu/bitcoin-0cd7928133eb-powerpc64le-linux-gnu.tar.gz
ff40a374f215eb3010291569b8ed1958054e408469fc8b2fe97a30cca0ad5451 guix-build-0cd7928133eb/output/riscv64-linux-gnu/SHA256SUMS.part
7b7b89ac1905d58f1e96a7840c018a556c472015a44442d0742bf758cb5f67ca guix-build-0cd7928133eb/output/riscv64-linux-gnu/bitcoin-0cd7928133eb-riscv64-linux-gnu-debug.tar.gz
10431bd8ffca82dd9c59f568272a1e7473cf474996f750d9bed4b576591fcff1 guix-build-0cd7928133eb/output/riscv64-linux-gnu/bitcoin-0cd7928133eb-riscv64-linux-gnu.tar.gz
4ef532d8dbe42900146a5b3e02de2a6a59d66b3c66a4b9d919d3aeb0e9637ab1 guix-build-0cd7928133eb/output/x86_64-apple-darwin/SHA256SUMS.part
77a1abe4139c19d227309216e29cf55dae06c4469412b457c9f0e8cf1eccc25c guix-build-0cd7928133eb/output/x86_64-apple-darwin/bitcoin-0cd7928133eb-x86_64-apple-darwin-unsigned.dmg
33028b640efab25648d0ec1abe9e91abc983706623ca9e2e7ac5fbfca0970909 guix-build-0cd7928133eb/output/x86_64-apple-darwin/bitcoin-0cd7928133eb-x86_64-apple-darwin-unsigned.tar.gz
e10d2d5617b8b1a33a622d5904d2bd8eaf57a5b3605e22ef916a57105db2311e guix-build-0cd7928133eb/output/x86_64-apple-darwin/bitcoin-0cd7928133eb-x86_64-apple-darwin.tar.gz
bf65d3574afed2e017c9625d38cc31e0f2cbb7f1e8a9ce346644ea3dbb938d13 guix-build-0cd7928133eb/output/x86_64-linux-gnu/SHA256SUMS.part
ce3810e70c97b2698822e4f46fa64dfa12353f7b54400e671b64868e3e4d3472 guix-build-0cd7928133eb/output/x86_64-linux-gnu/bitcoin-0cd7928133eb-x86_64-linux-gnu-debug.tar.gz
4055370c15b199d1efef47cc262d9c43a3652dcd237a9434197ca3be4931b1d2 guix-build-0cd7928133eb/output/x86_64-linux-gnu/bitcoin-0cd7928133eb-x86_64-linux-gnu.tar.gz
e59ed970d1db5d4839fa67957945628f6919ef5491f4a595f89ed3d8c81f1a76 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/SHA256SUMS.part
19c443fab5cb2fe75c9a5ad51fc022c97e31d7d69e049a889bd06f740f8daf78 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/bitcoin-0cd7928133eb-win64-debug.zip
88f6ca5d299080114532ec550c59eca4a3cdb759d9ea35cb14eba0b135e72436 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/bitcoin-0cd7928133eb-win64-setup-unsigned.exe
bcdb0b7467d3e47a694e51e9bfbaab9d5dc7162efe6c6bf4c303d368272c0cc6 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/bitcoin-0cd7928133eb-win64-unsigned.tar.gz
db1d4bbfab53405080d3abd09d1f05b2642ed513f6d8fcb5d92b9d0b32745293 guix-build-0cd7928133eb/output/x86_64-w64-mingw32/bitcoin-0cd7928133eb-win64.zip
```
ACKs for top commit:
hebasto:
ACK 0cd7928133eb8a605979c6338bbcbcb116cfa669, I have reviewed the code and it looks OK. I have also checked out the usage of the `git-minimal` in the `git-download` Guix module which is being used. Did not compare actual build dependences while building from scratch.
jarolrod:
ACK 0cd7928133eb8a605979c6338bbcbcb116cfa669
Tree-SHA512: f949c4d2f9560f98b8a418a981da38bbb9cfee5d0814bea6bb676b7193f3cbddafd23a92f852ee59c6a68c9c282095e6368cb65c5f2352b2ab54f9692575349c
98383d6d0dade5c2af2adcf10e274141fc7981aa doc: minor updates to guix README (Stacie)
Pull request description:
Two minor updates to the guix docs:
- `contrib/guix/README.md`: fix broken link
- `contrib/guix/INSTALL.md`: Change Ubuntu version in the section on distribution maintained packages from 21.04 (Hirsute Hippo) to 22.04 (Jammy Jellyfish). The previous link to the Ubuntu Guix package (https://packages.ubuntu.com/hirsute/guix) was for Hirsute. That link is now broken, likely because Hirsute reached EOL in January. I was unable to locate a general page for Ubuntu Guix packages so I replaced the broken link with the search results for all Ubuntu Guix packages. That page currently displays Guix packages for three different versions of Ubuntu. Happy to replace this link if there is a better option.
ACKs for top commit:
jarolrod:
ACK 98383d6d0dade5c2af2adcf10e274141fc7981aa
Tree-SHA512: 6980f5952862773e79ca317edb4aadf6ff7c71726a0e4cb873c08bf51360c64e0498aabf4f53780f13cb06838eda93c89ba10fe35c4c8ae2b23191ab961b98f8
9b313dfef18792fcc36e78ef3caa693fafcce04e guix: Ensure EPOCH_SOURCE_DATE does not include GPG information (Andrew Chow)
43225f0a2a517ccd79dc49279b979ffd2eca6b85 guix: Remove extra \r from all.SHA256SUMS line ending (Andrew Chow)
d080c27066449f76bc8709fc50e422757971d2cf guix, doc: Add a note that codesigners need to rebuild after tagging (Andrew Chow)
4a466388a0092fbdf5f8969c6bfb65bf8cc962e1 guix: Allow changing the base manifest in guix-verify (Andrew Chow)
33455c76964b9e27b33e970d9722cc47657b291b guix: Make all.SHA256SUMS rather than codesigned.SHA256SUMS (Andrew Chow)
Pull request description:
`guix-verify` expects `all.SHA256SUMS` but `guix-attest` produces `codesigned.SHA256SUMS`. Since `all.SHA256SUMS` makes more sense (as the file contains all the sha256sums, not just the codesigned ones), `guix-attest` has been changed to output a file of that name.
As a quality of life improvement, `guix-verify` can take `SIGNER` and use the signer's manifest as the base to compare against. This makes it easier to compare a single person's attestations with everyone else's and can make it more obvious when one builder is clearly mismatching with everyone else.
Lastly `release-process.md` is updated with a note about a gotcha that can cause a mismatch in the codesigned attestation.
ACKs for top commit:
fanquake:
ACK 9b313dfef18792fcc36e78ef3caa693fafcce04e
Tree-SHA512: 0d60627def38288dbd3059ad1e72cad224f9205da11b1a561c082ef28250a074df5cc5f2797c91a7be027bc486a3fda3319c2e496a8724e5b539337236c6f990
BACKPORT NOTICE:
this PR doesn't actually swithc to libc++ due to multiple CI failures such as
linking errors or other
------------------
faf62e6ed0ca45db44c370844c3515eb5a8cda12 ci: Remove unused workaround (MarcoFalke)
fa7c8509153bfd2d5b4dcff86ad27dfd73e8788b ci: Install llvm to get llvm symbolizer (MarcoFalke)
fa563cef61e8a217c5e8ec059e174afae61087a5 test: Add more tsan suppressions (MarcoFalke)
fa0cc02c0a029133f080680ae9186002a144738f ci: Mute depends logs completely (MarcoFalke)
fa906bf2988c799765a04c484269f890964ec3ee test: Extend tsan suppressions for clang stdlib (MarcoFalke)
fa10d850790bbe52d948659bb1ebbb88fe718065 ci: Use libc++ instead of libstdc++ for tsan (MarcoFalke)
fa0d5ee1126a8cff9f30f863eb8f5c78bf57e168 ci: Set halt_on_error=1 for tsan (MarcoFalke)
fa2ffe87f794caa74f80c1c2d6e6067ee4849632 ci: Deduplicate DOCKER_EXEC (MarcoFalke)
fac2eeeb9d718bdb892eef9adf333ea61ba8f3d0 cirrus: Remove no longer needed install step (MarcoFalke)
Pull request description:
According to the [ThreadSanitizer docs](https://clang.llvm.org/docs/ThreadSanitizer.html#current-status):
> C++11 threading is supported with **llvm libc++**.
For example, the thread sanitizer build is currently not checking for double lock of mutexes.
Fixes (partially) https://github.com/bitcoin/bitcoin/issues/19038#issuecomment-632138003
ACKs for top commit:
practicalswift:
ACK faf62e6ed0ca45db44c370844c3515eb5a8cda12
fanquake:
ACK faf62e6ed0ca45db44c370844c3515eb5a8cda12
hebasto:
ACK faf62e6ed0ca45db44c370844c3515eb5a8cda12, maybe re-organize commits to modify suppressions in a single one?
Tree-SHA512: 98ce5154b4736dfb811ffdb6e6f63a7bc25fe50d3b73134404a8f3715ad53626c31f9c8132dbacf85de47b9409f1e17a4399e35f78b1da30b1577167ea2982ad
39d526bde48d98af4fa27906e85db0399b6aa8b1 test: Bump linter versions (Duncan Dean)
Pull request description:
As per #19346, `mypy==0.700` was incompatible with Python 3.8.
I've bumped the versions of all the linters to their latest stable versions.
Checked with both Python 3.7 and 3.8 and everything still seems to work fine.
ACKs for top commit:
hebasto:
ACK 39d526bde48d98af4fa27906e85db0399b6aa8b1, I have reviewed the code and it looks OK, I agree it can be merged.
Tree-SHA512: f3ee7fda8095aa25aa68685e863076d52a6b82649770d24b0064d652763c0ceb8ebcbf9024fc74fca45c754e67b2a831dd070b3af23bc099140e6d27e89a5319
fa68755364473e48cf039e8cc2d08036fe58c1f6 contrib: Fix gen_key_io_test_vectors.py imports (MarcoFalke)
Pull request description:
The script currently fails with
```
Traceback (most recent call last):
File "./gen_key_io_test_vectors.py", line 18, in <module>
from segwit_addr import bech32_encode, decode, convertbits, CHARSET
ImportError: cannot import name 'decode' from 'segwit_addr'
```
Fix that.
Also, unrelated cleanup to use the `bytearray.hex()` method instead of importing a library. https://docs.python.org/3.5/library/stdtypes.html#bytes.hex
ACKs for top commit:
theStack:
tested ACK fa68755364473e48cf039e8cc2d08036fe58c1f6
Tree-SHA512: 45ff7d710de3d0ef5ac6d91543cff0edff6189d2cd00b0f8889f4361e66ef1825f12aea9e71d62038c14a7a531bfc95ffe9a1df83b85aa7f3dd666df07a6be81
80968cf scripted-diff: rename movie folder to animation (Peter Bushnell)
Pull request description:
Rename the movies directory and RES_MOVIES make variable to animation and RES_ANIMATION respectively. Movies is a bit of an unexpected term to be found.
ACKs for top commit:
MarcoFalke:
ACK 80968cf
hebasto:
ACK 80968cf, tested on Linux Mint 20 (Qt 5.12.8).
Tree-SHA512: 6bd31ce36e821f6a1bef8a7972086a2387d6258c48fc9df12d3ffdae07d0237036afbc2dec673384b78d9567b91d6e12eafa59fa2305aa79153dfd9b7c3a8655
## Issue being fixed or feature implemented
keybase.pub isn't a thing anymore; instead use thepasta.org; also
validate all .asc files
## What was done?
## How Has This Been Tested?
Validated 20.0.4, 20.0.3 and 20.0.2 with the script
## Breaking Changes
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that
apply._
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
---------
Co-authored-by: Wladimir J. van der Laan <laanwj@protonmail.com>
Co-authored-by: fanquake <fanquake@gmail.com>
Co-authored-by: Konstantin Akimov <knstqq@gmail.com>
9d026546778629472574b26fa73338efc63d02da doc: Fix systemd spelling and link to doc/init.md (Hennadii Stepanov)
601778c3107adbd8d96eb0bb5c16a9d0a4b81594 script: Add Documentation key to bitcoind.service (Hennadii Stepanov)
d9392b724cae53b7a16fa5f84ebe152eea496502 script: Improve robustness of bitcoind.service on startup (Hennadii Stepanov)
Pull request description:
If network interfaces are not properly up the following happens:
```
...
2021-01-08T10:17:11Z scheduler thread start
2021-01-08T10:17:11Z libevent: getaddrinfo: address family for nodename not supported
2021-01-08T10:17:11Z Binding RPC on address 127.0.0.1 port 8332 failed.
2021-01-08T10:17:11Z HTTP: creating work queue of depth 16
2021-01-08T10:17:11Z Using random cookie authentication.
2021-01-08T10:17:11Z Generated RPC authentication cookie /var/lib/bitcoind/.cookie
2021-01-08T10:17:11Z HTTP: starting 2 worker threads
2021-01-08T10:17:11Z init message: Loading banlist...
2021-01-08T10:17:11Z SetNetworkActive: true
2021-01-08T10:17:11Z Error: Cannot resolve -externalip address: <EDITED>
2021-01-08T10:17:11Z Shutdown: In progress...
2021-01-08T10:17:11Z scheduler thread exit
2021-01-08T10:17:11Z Shutdown: done
```
This PR improves robustness on startup in such cases in documented way:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
Also minor doc improvements are added.
ACKs for top commit:
Sjors:
ACK 9d02654
practicalswift:
ACK 9d026546778629472574b26fa73338efc63d02da: patch looks correct
darosior:
ACK 9d026546778629472574b26fa73338efc63d02da -- been using the first patch too
Tree-SHA512: 38294f5682c09e6ea9008de7d7459098c920cf1b98ad8ef8a5d2ca01f2f781c0fec5591dc40ef36eeb19d94991b0c7fb7cb38c4e716bc7219875c9bcd0a55e1b
## Issue being fixed or feature implemented
Client version string is inconsistent. Building `v20.0.0-beta.8` tag
locally produces binaries that report `v20.0.0-beta.8` version but
binaries built in guix would report
`v20.0.0rc1-g3e732a952226a20505f907e4fd9b3fdbb14ea5ee` instead. Building
any commit after `v20.0.0-beta.8` locally would result in versions like
`v20.0.0rc1-8c94153d2497` which is close but it's still yet another
format. And both versions with `rc1` in their names are confusing cause
you'd expect them to mention `beta.8` instead maybe (or is it just me?
:D ).
## What was done?
Change it so that the version string would look like this:
on tag: ~`v20.0.0-beta.8-dev` or `v20.0.0-beta.8-gitarc`~
`v20.0.0-beta.8`
post-tag: ~`v20.0.0-beta.8-1-gb837e08164-gitarc`~
`v20.0.0-beta.8-1-gb837e08164`
post-tag format is
`recent tag`-`commits since that tag`-`g+12 chars of commit hash`-`dirty
(optional)` ~-`dev or gitarc`~
~`dev`/`gitarc` suffixes should help avoiding confusion with the release
versions and they also indicate the way non-release binaries were
built.~
Note that release binaries do not use any of this, they still use
`PACKAGE_VERSION` from `configure` like before.
Also, `CLIENT_VERSION_RC` is no longer used in this setup so it was
removed.
Few things aren't clear to me yet:
1. Version bump in `configure.ac` no longer affects the reported version
(unless it's an actual release). Are there any downsides I might be
missing?
2. Which tag should we use on `develop` once we bump version in
configure? `v21.0.0-init`? `v21.0.0-alpha1`?
3. How is it going to behave once `merge master back into develop` kind
of PR is merged? E.g. say `develop` branch is on `v21.0.0-alpha1` tag
and we merge v20.1.0 from `master` back into it. Will this bring
`v20.1.0` release tag into `develop`? Will it become the one that will
be used from that moment? If so we will probably need another tag on
`develop` every time such PR is merged e.g. `v21.0.0-alpha2` (or
whatever the next number is).
Don't think these are blockers but would like to hear thoughts from
others.
## How Has This Been Tested?
Built binaries locally, built them using guix at a specific tag and at
some commit on top of it.
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
## Issue being fixed or feature implemented
Implement a new code-singing certificate for windows.
Previously we used a certificate issued by DigiCert, however that
certificate recently expired. A renewed certificate would cost roughly
$200/year at the cheapest CAs and $370/year with DigiCert. EV
certificates are relatively novel types of certificates that start out
with positive reputation, reducing smart screen popups for users. EV
certificates start at $270/year.
As a result we had (/have) 4 options:
1. Get a new code signing certificate from a trusted CA
- - Pro: Certificate gains reputation over time in smart screen and
binaries are signed
- - Pro: Shows "Verified Publisher" and "Dash Core Group Inc" on install
- - Con: Costs, feels manipulative to pay at least $600 simply for
someone to sign a certificate
2. Get a new EV code signing certificate
- - Pro: Certificate starts with good reputation and gains reputation
over time
- - Con: Even greater costs for a signature that says that we are from
Dash Core Group
3. Continue signing with the expired certificate
- - Con: This is, it has been discovered, a terrible idea and these
binaries are treated worse than unsigned binaries
4. Deliver unsigned windows binaries
- - Pro: Binary will gain reputation over time as users download it
- - Pro: Easy, is what it says on the tin
- - Con: Binaries are completely unsigned, could be tampering or
corruption issues that go undetected
- - Con: Will visibly state "Unknown Publisher"
5. Deliver self-signed windows binaries
- - Pro: Binary will gain reputation over time as users download it
- - Pro: *Possibility* that certificate will gain reputation over time
as users download binaries signed by it. It may also be that only
certificates issued by a CA will gain reputation over time.
- - Pro: Binaries are still signed
- - Pro: Users have the option to import certificate into keychain to
remove "Unknown Publisher"
- - Pro: In limited testing, install is sometimes is treated better than
unsigned, otherwise is treated the same
- - Con: may appear sketchy, as Root CA is not a trusted Root CA
- - Con: will display "Unknown Publisher" to most users
- - Con: greater potential uncertainty around future changes to
treatment of self signing systems
Based on the above discussion and testing, the best route currently is
option 5; that is what this PR implements. In the future it may make
sense to move towards a codesigning certificate issued by a trusted CA.
The root certificate authority has the following information
with a sha256 fingerprint of `46 84 FF 27 11 D7 C8 C5 BB FA D1 55 41 B3
F0 43 77 97 AC 67 4C 32 19 AE B4 E7 15 11 1F BB 42 A0`
The code signing certificate is issued by the root CA, has a common name
of "Dash Core Windows Signing" and a sha256 fingerprint of `1A 09 54 6E
D3 81 E9 FC AD 62 44 32 35 40 39 FF 5F A7 30 0E 5E 03 C4 E0 96 5A 62 AA
19 2B 79 EE`. This certificate is only authorized for the purpose of
code signing.
## What was done?
## How Has This Been Tested?
Multiple users installing binaries of type 1,3,4 and 5.
## Breaking Changes
This new windows signing certificate should be documented in the release
notes.
## Checklist:
_Go over all the following points, and put an `x` in all the boxes that
apply._
- - [x] I have performed a self-review of my own code
- - [ ] I have commented my code, particularly in hard-to-understand
areas
- - [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- - [ ] I have made corresponding changes to the documentation
- - [x] I have assigned this pull request to a milestone _(for
repository code-owners and collaborators only)_
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCAAdFiEEKVkDYuyHioH9PCArUlJ77avoeYQFAmWfAbUACgkQUlJ77avo
eYTSCBAAuDEoWABdonIMs/4RaYP+DGTULltRu9CHBAqYuksXrl/4iV0r17DPSWWW
L/5vLNAUTI47Tsa7R45ZPb0hR8VPMBkvxTQipKBYK7vZpwefcR4VOprEBJJ0Bl3g
ZHtAVjZbcANEIAW3SlaiOgWbxWGKfDyM7gN3aNfoidMFBefbcYKEttuAGCnktWRI
Y3eLMGPCpxOVB0O1nLU+pzwixAWXOeVChiK31ecFfQrF3JmUc12yiFUI+OJTogg4
0G2GMIQYHiVwclj8hSWT/yZfjcyxXdLYqkmH4Nr5mye39hRI2aUQEkmkYOy8pjcB
ykKLg8JpUg/zg6GSuS6mFJnd5NHq5iSBxSRHPfR8xij1xFpmdgAaNCw4/6j9PEXB
l8cfuJ7hgX3yX09L4p2E4t7MYpM8igaenAIWAK37hmKs1WADBmaj/nf6ThKhjvzI
2GR0FOzm6Is36KYvdUQJDE0g70g31SvGy+qjlcK49MtX6BvecYt+dg8AaNZ5FIn7
d1kFI4NXM6JX2WdiHMenz5d+oFYRS/P1sXjQ1wtl9HSkiZQQkEBbgiWXfh+EXjpW
fNc8cej2LLCNZlhVcpffF8UaINsMTZVQsEGWGInjSi5eCs/YNrqL8XDdC/8mmZCu
cNvp0QBtQ+4lpbUSdhFUdgic0MRCsdeHuYIBfvPJN9tl8McbknA=
=kL6E
-----END PGP SIGNATURE-----
5d77549d8b287eb773db695b88c165ebe3be1005 doc: Add mypy to test dependencies (Hennadii Stepanov)
7dda912e1c28b02723c9f24fa6c4e9003d928978 test: Do not swallow flake8 exit code (Hennadii Stepanov)
Pull request description:
After #18210 the `flake8` exit code in `test/lint/lint-python.sh` just not used that makes the linter broken.
This PR:
- combines exit codes of `flake8` and `mypy` into the `test/lint/lint-python.sh` exit code
- documents `mypy` as the test dependency
ACKs for top commit:
MarcoFalke:
Approach ACK 5d77549d8b287eb773db695b88c165ebe3be1005, fine with me
practicalswift:
ACK 5d77549d8b287eb773db695b88c165ebe3be1005
Tree-SHA512: e948ba04dc4d73393967ebf3c6a26c40d428d33766382a0310fc64746cb7972e027bd62e7ea76898b742a656cf7d0fcda2fdd61560a21bfd7be249cea27f3d41
## Issue being fixed or feature implemented
We had this in Gitian
https://github.com/dashpay/dash/blob/master/contrib/gitian-descriptors/gitian-win.yml#L38.
We also had it for macos
https://github.com/dashpay/dash/blob/master/contrib/gitian-descriptors/gitian-osx.yml#L42
but it looks like it's no longer an issue there (or at least I did not
see anyone complaining about it).
## What was done?
tweak `CONFIGFLAGS` for `mingw` host
## How Has This Been Tested?
n/a
## Breaking Changes
n/a
## Checklist:
- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
