797b3ed9090030f32fade81803b580562d4a90a3 script: remove gitian reference from symbol-check.py (fanquake)
15fc9a0299091bfeb3370f993ad95ff638f6ba8c guix: add additional documentation to patches (fanquake)
4516e5ec9223486fe2eba7f4320d786d074a58fd lint: exclude Guix patches from spell-checking (fanquake)
de6ca41a52d2646598daae5f4620bbe766757e21 guix: no-longer pass --enable-glibc-back-compat to Guix (fanquake)
84dd81fb5bf7308b8070b53520266854fb6efad3 build: remove glibc backcompat requirement for Linux symbol checks (fanquake)
Pull request description:
Now that our Guix toolchains are based on glibc 2.24 and 2.27 (RISCV), we don't need to use the `--enable-glibc-back-compat` option to produce binaries that don't use any symbols from glibc 2.17 and 2.27 or later.
This also adds additional documentation to some Guix patches (pointed out in #22365) and removes Guix patches from the spelling linter, because that isn't our spelling.
Symbol usage: https://gist.github.com/fanquake/d15604fc580718444c5aa4b3c3c75fdc.
Guix Builds:
```bash
bash-5.1# find guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
ed54e6a6cf4fab328557c0c72eb08c73f2a58c6c70959544cf4b1882e75ea69e guix-build-797b3ed90900/output/aarch64-linux-gnu/SHA256SUMS.part
83bd9dadc59f89f848d143fa4fc3964f16fe0b4bdf35e5093b577ff2c4bd1f43 guix-build-797b3ed90900/output/aarch64-linux-gnu/bitcoin-797b3ed90900-aarch64-linux-gnu-debug.tar.gz
94cb8c35281f12dec6ea5b390b66cad5e27ac8c45a30c42c8d38c438695d54c0 guix-build-797b3ed90900/output/aarch64-linux-gnu/bitcoin-797b3ed90900-aarch64-linux-gnu.tar.gz
7318b63d65c0aa52d2446de8e1f40658d2e47ab8fb0268820c3b7585d140fb23 guix-build-797b3ed90900/output/arm-linux-gnueabihf/SHA256SUMS.part
95e1ffb372964b73f539653ca703b70cf0c018801a9c4c0ffc46a0b63539253c guix-build-797b3ed90900/output/arm-linux-gnueabihf/bitcoin-797b3ed90900-arm-linux-gnueabihf-debug.tar.gz
039d3842e6499626cf955ae0a7590dd6b3d0935cdc217c98aaf9d156b0ebd3b4 guix-build-797b3ed90900/output/arm-linux-gnueabihf/bitcoin-797b3ed90900-arm-linux-gnueabihf.tar.gz
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 guix-build-797b3ed90900/output/dist-archive/SKIPATTEST.TAG
2c4e7b6e7aff63ba811e5bf59362d16866c3a358f8844fba8739a61192870622 guix-build-797b3ed90900/output/dist-archive/bitcoin-797b3ed90900.tar.gz
955029b949c368eabd517dd33040d2f01e2ac6a55e7b4f9107907a7c6e0c6060 guix-build-797b3ed90900/output/powerpc64-linux-gnu/SHA256SUMS.part
fd6d6b137f8efedf58a879d11205b1d4649e1f97d7f91e193239ef206fcc285d guix-build-797b3ed90900/output/powerpc64-linux-gnu/bitcoin-797b3ed90900-powerpc64-linux-gnu-debug.tar.gz
51736ac8e77737999f1b5bd4c381b0016f19a8d5e40e786fe941ff04e84c11c9 guix-build-797b3ed90900/output/powerpc64-linux-gnu/bitcoin-797b3ed90900-powerpc64-linux-gnu.tar.gz
8c244c16bfa46c1efdb120e1d91fdd14d3f14eefee8d7e1fbb0a9b4664a5c315 guix-build-797b3ed90900/output/powerpc64le-linux-gnu/SHA256SUMS.part
704ee593251a1b1c65a5bebeef93b23f266af4e8cbf8ae556150c3b2e8f06a6c guix-build-797b3ed90900/output/powerpc64le-linux-gnu/bitcoin-797b3ed90900-powerpc64le-linux-gnu-debug.tar.gz
0ec06ae7d344de20d61e3965d8b383747ef20b0e9d93a3165733ea23bdf2ead8 guix-build-797b3ed90900/output/powerpc64le-linux-gnu/bitcoin-797b3ed90900-powerpc64le-linux-gnu.tar.gz
2dd6c6ecc67b0ea40ca9c43f92efca81ccd054b8db8c197ad84ad9674d510a25 guix-build-797b3ed90900/output/riscv64-linux-gnu/SHA256SUMS.part
5ebb27a855a677f7a188d83995be6b2a3ea8606be152abb7fc7832713fb0677a guix-build-797b3ed90900/output/riscv64-linux-gnu/bitcoin-797b3ed90900-riscv64-linux-gnu-debug.tar.gz
bdaf1783f5e1861597afa37c1880364e118d9a7a7af8017302d82202791019f6 guix-build-797b3ed90900/output/riscv64-linux-gnu/bitcoin-797b3ed90900-riscv64-linux-gnu.tar.gz
726c9092b60ac2e7d7e14b2c24467fcf276a6f89170a871ddab9dce6ac230699 guix-build-797b3ed90900/output/x86_64-apple-darwin18/SHA256SUMS.part
2af4d709b44952654f3c08c86593bf2ccc9a44ed422783a1b95b8a199a894db2 guix-build-797b3ed90900/output/x86_64-apple-darwin18/bitcoin-797b3ed90900-osx-unsigned.dmg
fd49ba445aa6cf3d8c47019a05e9e5740cb0f53349344dd80671297127f49f1a guix-build-797b3ed90900/output/x86_64-apple-darwin18/bitcoin-797b3ed90900-osx-unsigned.tar.gz
3f51cbf8cf18420d4be70e656aa993675cf5e828a255c2030047ae2e059ed5b7 guix-build-797b3ed90900/output/x86_64-apple-darwin18/bitcoin-797b3ed90900-osx64.tar.gz
afd1edee1447bb88d81e972abfae4c4e065b5b1827769f033cff9472084c7c1b guix-build-797b3ed90900/output/x86_64-linux-gnu/SHA256SUMS.part
ec468ef886d25e685f4f7a18b4f7d497dedf757495e0d5beb72c23cc32ab69b5 guix-build-797b3ed90900/output/x86_64-linux-gnu/bitcoin-797b3ed90900-x86_64-linux-gnu-debug.tar.gz
1934d7294f0c9e083d38a3f68d4a61cd679defa79ce0a89f77386978692b9b18 guix-build-797b3ed90900/output/x86_64-linux-gnu/bitcoin-797b3ed90900-x86_64-linux-gnu.tar.gz
94c11c328a628052eb6f50e9816aa768f87ea7acfbbbafdab60f6928da766811 guix-build-797b3ed90900/output/x86_64-w64-mingw32/SHA256SUMS.part
fd371922ba93d81bd4a2b711d617af6756f9f0494db6d83aa0e5f491a24168ef guix-build-797b3ed90900/output/x86_64-w64-mingw32/bitcoin-797b3ed90900-win-unsigned.tar.gz
4e4ad976bc029bbbf9596ad8493accaaba8b0d5c598dd342f8da330609bbdf21 guix-build-797b3ed90900/output/x86_64-w64-mingw32/bitcoin-797b3ed90900-win64-debug.zip
3a89a16b9101e9a17d98efb9234b5bdd264c0bba2c6326511017730e1a08311f guix-build-797b3ed90900/output/x86_64-w64-mingw32/bitcoin-797b3ed90900-win64-setup-unsigned.exe
e285ab737e3c843fd3f1c26c2f053e421a3c39b33995747ce48281884d3f28d1 guix-build-797b3ed90900/output/x86_64-w64-mingw32/bitcoin-797b3ed90900-win64.zip
```
ACKs for top commit:
sipa:
utACK 797b3ed9090030f32fade81803b580562d4a90a3
hebasto:
ACK 797b3ed9090030f32fade81803b580562d4a90a3
Tree-SHA512: 3a569702d8832c155c5ce8d2f6d823f7f12603885576078bc5192bc9038a48261ecb541800f79d1e9bc86d71fa640265c5b8b89df9d8bb680b3bb05d9d78a666
c491368d8cfddf3a5b6d574f10ed67492fcecbed scripts: add MACHO dylib checking to symbol-check.py (fanquake)
76bf97213f4b153dd3ccf1314088a73c4804601d scripts: fix check-symbols & check-security argument passing (fanquake)
Pull request description:
Based on #17857.
This adds dynamic library checks for MACHO executables to symbol-check.py. The script has been modified to function more like `security-check.py`. The error output is now also slightly different. i.e:
```bash
# Linux x86
bitcoin-cli: symbol operator new[](unsigned long) from unsupported version GLIBCXX_3.4
bitcoin-cli: export of symbol vtable for std::basic_ios<char, std::char_traits<char> > not allowed
bitcoin-cli: NEEDED library libstdc++.so.6 is not allowed
bitcoin-cli: failed IMPORTED_SYMBOLS EXPORTED_SYMBOLS LIBRARY_DEPENDENCIES
# RISCV (skips exported symbols checks)
bitcoin-tx: symbol operator new[](unsigned long) from unsupported version GLIBCXX_3.4
bitcoin-tx: NEEDED library libstdc++.so.6 is not allowed
bitcoin-tx: failed IMPORTED_SYMBOLS LIBRARY_DEPENDENCIES
# macOS
Checking macOS dynamic libraries...
libboost_filesystem.dylib is not in ALLOWED_LIBRARIES!
bitcoind: failed DYNAMIC_LIBRARIES
```
Compared to `v0.19.0.1` the macOS allowed dylibs has been slimmed down somewhat:
```diff
src/qt/bitcoin-qt:
/usr/lib/libSystem.B.dylib
-/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
/System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
/System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
-/System/Library/Frameworks/Security.framework/Versions/A/Security
-/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
/System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
-/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
-/System/Library/Frameworks/AGL.framework/Versions/A/AGL
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
/usr/lib/libc++.1.dylib
-/System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
/System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
/System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
/usr/lib/libobjc.A.dylib
```
ACKs for top commit:
laanwj:
ACK c491368d8cfddf3a5b6d574f10ed67492fcecbed
Tree-SHA512: f8624e4964e80b3e0d34e8d3cc33f3107938f3ef7a01c07828f09b902b5ea31a53c50f9be03576e1896ed832cf2c399e03a7943a4f537a1e1c705f3804aed979
3d0a82cff8cbb809876e82dbe62d14d2adc07d94 devtools: Accomodate block-style copyright blocks (Ben Woosley)
0ef0e51fe4bb592e67255776b5a0ba04679fb8c4 lint: Bump flake8 to 3.7.8 (Ben Woosley)
838920704ad90a71cf288b700052503db8abb17e lint: Disable flake8 W504 warning (Ben Woosley)
b21680baf5391a602b295b9d7d0ef66553661cb9 test/contrib: Fix invalid escapes in regex strings (Ben Woosley)
Pull request description:
This is a second go at #15221, fixing new lints in:
W504 line break after binary operator
W605 invalid escape sequence
F841 local variable 'e' is assigned to but never used
This time around:
* One commit per rule, for easier review
* I went with the PEP-8 style of breaking before binary operators
* I looked into the raw regex newline issue, and found that raw strings with newlines embedded do work appropriately. E.g. run `re.match(r" \n ", " \n ")` to check this for yourself. `re.MULTILINE` exists to modify `^` and `$` in multiline scenarios, but all of these searches are per-line.
ACKs for top commit:
practicalswift:
ACK 3d0a82cff8cbb809876e82dbe62d14d2adc07d94 -- diff looks correct
Tree-SHA512: bea0c144cadd72e4adf2e9a4b4ee0535dd91a8e694206924cf8a389dc9253f364a717edfe9abda88108fbb67fda19b9e823f46822d7303c0aaa72e48909a6105
eafd2593674b6890418a50eadf69425071b824e4 build: Add NX workaround for RV64 (Wladimir J. van der Laan)
f6e42256fe8900ed90f9644906b77f97e1600952 build: Allow export of environ symbols (Wladimir J. van der Laan)
Pull request description:
This export was introduced in #17270 which added
```
//! Necessary on some platforms
extern char** environ;
```
This should (finally) make the gitian build pass again (fix issue #17525.).
Built on top of #17538 which should be merged first.
Top commit has no ACKs.
Tree-SHA512: 5c2054d52d0957aec3dc945b76d8e219187d22dc03889e7a88fb76049bf8e4a3e9f4da00dd1e9dd0351211f8e70d1a1b8ad7244f0348dab698e9d14b9d0c0bd4
c516c3a770e93dcd283dc4a9275cba88221d0e3a [contrib] Support ARM and RISC-V symbol check (Chun Kuan Lee)
Pull request description:
Solve the TODO in the gitian-descripter
Tree-SHA512: 8115e2958af3dde43d9d9d05f0b1b1b93b1c2aa513e771a3e4e1342a5d78af2b0e40c0bbb7e9a0d15954897317e6f5a0d80996239af3b376d5ddd527f73428ae
# Conflicts:
# contrib/devtools/symbol-check.py
# contrib/gitian-descriptors/gitian-linux.yml
0c55d8b58186ba69fffc147cd02b174450dac578 depends: qt: Patch to remove dep on libX11 (Carl Dong)
222e6cc52075cf6f866ce4a2e5900f0d6358f903 gitignore: Actually pay attention to depends patches (Carl Dong)
65f8da08df601ac98bad6f638392343fd564c0b9 symbol-check: Disallow libX11-*.so.* shared libraries (Carl Dong)
924569914e6079763fd1692bed372ded2c63d685 depends: libXext isn't needed by anyone (Carl Dong)
689d3b4a030675b6de87a55463a8e155843293fb build-aux: Remove check for x11-xcb (Carl Dong)
aa53cb7a2f04a59a4722c662e67b7a6ec04e32b5 depends: libX11: Make package headers-only (Carl Dong)
9a01ab04e1125b37618266d7fd1c3a6e3bb7e3f8 depends: qt: Explicitly stop using Xlib/libX11 (Carl Dong)
1ec30b8fbe4a162d6e2e6a9711d83f1bafd9b645 depends: xproto is only directly needed by libXau (Carl Dong)
Pull request description:
Related to: #16150
We noticed that we could build QT without using XLib/libX11 as a library. XLib/libX11's headers are still used, and a minimal `configure.ac` has been added to eliminate overly-enthusiastic configure-time dependencies that aren't actually required to obtain the headers.
This also means that we eliminate XLib/libX11 as required shared libraries at runtime, which is desirable.
See commit messages for more details.
---
Reviewers: I am least sure about the minimal `configure.ac`, as I'm not too familiar with the autoconf syntax. Any improvements w/re robustness would be welcome.
ACKs for top commit:
theuni:
ACK 0c55d8b58186ba69fffc147cd02b174450dac578
fanquake:
ACK 0c55d8b58186ba69fffc147cd02b174450dac578
Tree-SHA512: 41f653a0f91bc0e0faac49713c0c6dfd8cb605f9c4e34eb75a790dd808ebf3e5c160f1dd40bc8fbc911ee718ea319313b526d63733c98ff62d8dffecb58caa01
* Trivial Dashification
* Tweak getnetworkinfo and dumpwallet help text
We don't have RBF and Segwit
* CopyrightHolders should also check for missing "Dash Core" copyright
78214588d Use for-loop instead of list comprehension (practicalswift)
823979436 Use the variable name _ for unused return values (practicalswift)
2e6080bbf Remove unused variables and/or function calls (practicalswift)
9b94054b7 Avoid reference to undefined name: stderr does not exist, sys.stderr does (practicalswift)
51cb6b822 Use print(...) instead of undefined printf(...) (practicalswift)
25cd520fc Use sys.exit(...) instead of exit(...): exit(...) should not be used in programs (practicalswift)
Pull request description:
Python cleanups:
* Avoid reference to undefined name: `stderr` does not exist, `sys.stderr` does
* Use `print(...)` instead of undefined `printf(...)`
* Avoid redefinition of variable (`tx`) in list comprehension
* Remove unused variables and/or function calls
* Use `sys.exit(...)` instead of `exit(...)`: [`exit(...)` should not be used in programs](https://github.com/bitcoin/bitcoin/pull/10753#discussion_r125935027)
Tree-SHA512: 1238dfbc1d20f7edadea5e5406a589f293065638f6234809f0d5b6ba746dffe3d276bc5884c7af388a6c798c61a8759faaccf57f381225644754c0f61914eb4b
* Add stdin, stdout, stderr to ignored export list
* Merge #13171: Change gitian-descriptors to use bionic instead
c1afe3232fa10e290fb355cf37ea4c7bc1084065 Change gitian-descriptors to use bionic instead (Chun Kuan Lee)
Pull request description:
I have tested this with Ubuntu Bionic host with lxc 3.0.0, the gitian-build for Windows and MacOSX work fine, but there is an issue about it for Linux. Failed at check-symbol:
```
test/test_bitcoin: symbol __divmoddi4 from unsupported version GCC_7.0.0
test/test_bitcoin: symbol log2f from unsupported version GLIBC_2.27
qt/bitcoin-qt: symbol __divmoddi4 from unsupported version GCC_7.0.0
qt/bitcoin-qt: symbol log2f from unsupported version GLIBC_2.27
```
I think this should be fixed in `./configure --enable-glibc-back-compat`
Should not be merged before #13177devrandom/gitian-builder#178Close#12511
Tree-SHA512: 257d75d5b6864e105279f7a7b992fbbd7903cdbe3300b66dacec0a783d267707d9dbbfe0e64a36983ca1eca50a2a5e1cdb222b6d3745ccc3e5fc5636c88b581f
* Use IN6ADDR_ANY_INIT instead of in6addr_any
This is the same fix as fc6a9f2ab1
Couldn't backport the original commit as we are missing some refactorings.
* Merge #11881: Remove Python2 support
1874058 Make base58 python contrib code work with python3 (Evan Klitzke)
bc6fdf2 Change all python files to use Python3 (John Newbery)
Pull request description:
Following discussion here: https://github.com/bitcoin/bitcoin/pull/11843#issuecomment-351033742
It's easier for maintainers if all python tools/scripts support only a single version of Python. There are only a few scripts that aren't explicitly python3 at this point, so this PR changes those remaining scripts to explicitly require python3.
Tree-SHA512: 5d38eef6e0fc7d8515e23a1f4c75e8b4160fd0fe23cba52a1f41689b114e54a9e503e0724829e8b41982ef98f2d113df80d9e238213b74f09ceaed0344a19e24
* Merge #12829: Python3 fixup
f50975b [contrib] fixup symbol-check.py Python3 support (John Newbery)
5de2b18 [contrib] fixup security-check.py Python3 support (John Newbery)
Pull request description:
security-check.py and symbol-check.py were broken by #11881. Fix them.
Tree-SHA512: 86de3d6dc3292b1ae4cc04c2d7d7dbbf39c9270551d7b224b8d8b19e3184c30c897dbf823200403706d06bb405c0decad5cfd690cb2c0312992a235a4ffcf6bf
- fix parsing of BIND_NOW with older readelf
- add _IO_stdin_used to ignored exports
For details see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634261#109
- add check-symbols and check-security make targets
These are not added to the default checks because some of them depend on
release-build configs.
- always link librt for glibc back-compat builds
glibc absorbed clock_gettime in 2.17. librt (its previous location) is safe to
link in anyway for back-compat.
Fixes#7420
- add security/symbol checks to gitian
Github-Pull: #7424
Rebased-From: cd27bf51e0475813ba5bf3d3eaf78ea8ce872118a81c87fafc
Perform the following ELF security checks:
- PIE: Check for position independent executable (PIE), allowing for address space randomization
- NX: Check that no sections are writable and executable (including the stack)
- RELRO: Check for read-only relocations, binding at startup
- Canary: Check for use of stack canary
Also add a check to symbol-check.py that checks that only the subset of
allowed libraries is imported (to avoid incompatibilities).
Add a script to check that the (Linux) executables produced by gitian
only contain allowed gcc, glibc and libstdc++ version symbols. This
makes sure they are still compatible with the minimum supported Linux
distribution versions.
