sikkienl/dash
1
0
Fork 0
mirror of https://github.com/dashpay/dash.git synced 2024-12-24 11:32:46 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge #6428: backport: merge bitcoin#24778, #24766, #24849, #24844, #24853, #24895, #24802, #24932, partial bitcoin#23212, #23462 (lint backports)

Browse Source 
852f55e23c merge bitcoin#24932: Convert lint-locale-dependence.sh to Python (Kittywhiskers Van Gogh)
f745b7f7ef merge bitcoin#24802: convert format strings linter test to python (Kittywhiskers Van Gogh)
a0b051b4ef partial revert dash#4807: enable more multi-threading and caching in linters (Kittywhiskers Van Gogh)
7864c21645 merge bitcoin#24895: Convert lint-includes.sh to Python (Kittywhiskers Van Gogh)
f63bafe253 merge bitcoin#24853: Convert lint-git-commit-check.sh to Python (Kittywhiskers Van Gogh)
d463d7d397 fix: clone full history on GitHub Actions build job, track `develop` (Kittywhiskers Van Gogh)
d23b6614e8 merge bitcoin#24844: Convert lint-whitespace.sh to Python (Kittywhiskers Van Gogh)
fe165168ba merge bitcoin#24849: Convert lint-logs.sh to Python (Kittywhiskers Van Gogh)
dbfd0b04e1 merge bitcoin#24766: convert spellchecking lint test to python (Kittywhiskers Van Gogh)
5c23addddd merge bitcoin#24778: Convert Python dead code linter test to Python (Kittywhiskers Van Gogh)
829dcfb936 partial bitcoin#23462: Enable SC2046 and SC2086 shellcheck rules (Kittywhiskers Van Gogh)
df6be0e8c0 partial bitcoin#23212: enable mypy import checking (Kittywhiskers Van Gogh)
f77eca9256 fix: make sure that parallelized `lint-all` runs incl. python scripts (Kittywhiskers Van Gogh)
e220175d0f merge bitcoin#22861: Update test README and lint script (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * Dependency for https://github.com/dashpay/dash/pull/6429

  * Even though support for Python scripts was extended to `lint-all` in [bitcoin#24762](f226e8dc1f) ([dash#6023](https://github.com/dashpay/dash/pull/6023)), the changes needed were not extended to parallelized linting added in [dash#4637](https://github.com/dashpay/dash/pull/4637).

    This meant the match expression didn't include non-shell scripts and additionally, `parallel` interpreted all scripts as Bash scripts. This has been resolved in this PR.

  * [bitcoin#23212](https://github.com/bitcoin/bitcoin/pull/23212) is partial as `--ignore-missing-imports` has not been removed from `mypy` arguments as the version of `mypy` used in the PR (0.910) isn't syntax aware to `imports` like [here](f9d044d5ec/test/functional/test_framework/crypto/bip324_cipher.py (L16-L17)) and [here](f9d044d5ec/test/functional/test_framework/crypto/muhash.py (L9)).

    <details>

    <summary>Lint errors:</summary>

    ```
    dash@96b217d539f7:/src/dash$ ./test/lint/lint-python.sh
    Consider install flake8-cached for cached flake8 results.
    [...]
    test/functional/test_framework/crypto/muhash.py:9: error: Skipping analyzing ".chacha20": found module but no type hints or library stubs  [import]
    test/functional/test_framework/crypto/ellswift.py:15: error: Cannot find implementation or library stub for module named "test_framework.crypto.secp256k1"  [import]
    test/functional/test_framework/crypto/bip324_cipher.py:16: error: Skipping analyzing ".chacha20": found module but no type hints or library stubs  [import]
    test/functional/test_framework/crypto/bip324_cipher.py:17: error: Skipping analyzing ".poly1305": found module but no type hints or library stubs  [import]
    test/functional/test_framework/messages.py:29: error: Cannot find implementation or library stub for module named "test_framework.crypto.siphash"  [import]
    test/functional/test_framework/messages.py:32: error: Cannot find implementation or library stub for module named "dash_hash"  [import]
    test/functional/test_framework/script.py:20: error: Cannot find implementation or library stub for module named "test_framework.crypto.ripemd160"  [import]
    test/functional/test_framework/key.py:16: error: Cannot find implementation or library stub for module named "test_framework.crypto"  [import]
    test/functional/test_framework/v2_p2p.py:9: error: Cannot find implementation or library stub for module named "test_framework.crypto.bip324_cipher"  [import]
    test/functional/test_framework/v2_p2p.py:10: error: Cannot find implementation or library stub for module named "test_framework.crypto.chacha20"  [import]
    test/functional/test_framework/v2_p2p.py:11: error: Cannot find implementation or library stub for module named "test_framework.crypto.ellswift"  [import]
    test/functional/test_framework/v2_p2p.py:12: error: Cannot find implementation or library stub for module named "test_framework.crypto.hkdf"  [import]
    test/functional/p2p_v2_encrypted.py:22: error: Cannot find implementation or library stub for module named "test_framework.crypto.chacha20"  [import]
    test/functional/feature_utxo_set_hash.py:12: error: Cannot find implementation or library stub for module named "test_framework.crypto.muhash"  [import]
    Found 17 errors in 12 files (checked 275 source files)
    ```

    </details>

    And upgrading to the latest version of `mypy` used upstream (1.4.1, [source](f7144b24be/ci/lint/04_install.sh (L52))) brings syntax awareness but new errors.

    <details>

    <summary>Lint errors:</summary>

    ```
    dash@96b217d539f7:/src/dash$ ./test/lint/lint-python.sh
    Consider install flake8-cached for cached flake8 results.
    [...]
    test/functional/test_framework/coverage.py:23: error: Incompatible default for argument "coverage_logfile" (default has type "None", argument has type "str")  [assignment]
    test/functional/test_framework/coverage.py:23: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
    test/functional/test_framework/coverage.py:23: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
    test/functional/test_framework/util.py:322: error: Incompatible default for argument "timeout" (default has type "None", argument has type "int")  [assignment]
    test/functional/test_framework/util.py:322: note: PEP 484 prohibits implicit Optional. Accordingly, mypy has changed its default to no_implicit_optional=True
    test/functional/test_framework/util.py:322: note: Use https://github.com/hauntsaninja/no_implicit_optional to automatically upgrade your codebase
    test/functional/test_framework/util.py:322: error: Incompatible default for argument "coveragedir" (default has type "None", argument has type "str")  [assignment]
    test/functional/test_framework/messages.py:32: error: Cannot find implementation or library stub for module named "dash_hash"  [import]
    test/functional/test_framework/test_framework.py:122: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
    test/functional/test_framework/test_framework.py:123: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
    test/functional/test_framework/test_framework.py:124: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
    test/functional/test_framework/test_framework.py:125: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
    test/functional/p2p_message_capture.py:48: note: By default the bodies of untyped functions are not checked, consider using --check-untyped-defs  [annotation-unchecked]
    Found 6 errors in 5 files (checked 275 source files)
    ```

    </details>

  * [bitcoin#23462](https://github.com/bitcoin/bitcoin/pull/23462) is partial as neither `SC2046` nor `SC2086` have been enabled. This backport was done for the sole purpose of backporting changes to shell scripts that would be replaced with their Python counterparts in later backports.

    The necessary changes needed to enable `SC2046` and `SC2086` will be done in a later pull request.

  * `actions/checkout` does not do a full clone nor tracks the default branch by default. It is already documented that `git merge-base` (used by `lint-git-commit-check.py` and `lint-whitespace.py`, [source](4aaa314a57/test/lint/lint-git-commit-check.py (L45))) doesn't play nice with it (see [actions/checkout#423](https://github.com/actions/checkout/discussions/423)).

    No such issue exists on GitLab ([build](https://gitlab.com/dashpay/dash/-/jobs/8456260939#L113)), but it remains present on GitHub Actions ([build](https://github.com/dashpay/dash/actions/runs/11996049800/job/33440106874?pr=6428#step:7:103)). This PR does a full clone, switches to the `develop` branch, then switches back to the intended commit, as a workaround (see working build [here](https://github.com/dashpay/dash/pull/6428#issuecomment-2495980410))

  * Changes to `run-lint-format-strings.py` made in [dash#4807](https://github.com/dashpay/dash/pull/4807) were reverted as they were interfering with `lint-format-strings.py`

    <details>

    <summary>Lint error:</summary>

    ```
    dash@96b217d539f7:/src/dash$ ./test/lint/lint-format-strings.py
    Traceback (most recent call last):
      File "/src/dash/test/lint/run-lint-format-strings.py", line 308, in <module>
        main()
      File "/src/dash/test/lint/run-lint-format-strings.py", line 304, in main
        sys.exit(max(exit_codes))
    ValueError: max() arg is an empty sequence
    Traceback (most recent call last):
      File "/src/dash/test/lint/run-lint-format-strings.py", line 308, in <module>
        main()
      File "/src/dash/test/lint/run-lint-format-strings.py", line 304, in main
        sys.exit(max(exit_codes))
    ValueError: max() arg is an empty sequence
    ```

    </details>

  ## Breaking Changes

  None expected.

  ## Checklist

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

ACKs for top commit:
  knst:
    utACK 852f55e23c
  UdjinM6:
    utACK 852f55e23c

Tree-SHA512: e508311c00249e7e48f91ca23e6f62117c07497f9c4471d07659b233e43a9f4d09ee2bfe0dd76f0c9746209cdba5895cb002a708924daa19d7aa76869815a7d9
This commit is contained in:
pasta 2024-12-04 09:44:00 -06:00
commit eb88589ea7
No known key found for this signature in database
GPG Key ID: E2F3D7916E722D38
48 changed files with 981 additions and 733 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.github/workflows/build.yml vendored
View File

@ -140,6 +140,7 @@ jobs:
uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 0
- name: Restore depends cache
uses: actions/cache/restore@v4
@ -167,7 +168,11 @@ jobs:
- name: Build source and run tests
run: |
git config --global --add advice.detachedHead false
git config --global --add safe.directory "$PWD"
GIT_HEAD="$(git rev-parse HEAD)"
git checkout develop
git checkout ${GIT_HEAD}
CCACHE_SIZE="400M"
CACHE_DIR="/cache"
mkdir /output

2
Makefile.am
View File

@ -252,7 +252,7 @@ EXTRA_DIST += \
test/fuzz
EXTRA_DIST += \
test/util/bitcoin-util-test.py \
test/util/test_runner.py \
test/util/data/bitcoin-util-test.json \
test/util/data/blanktxv1.hex \
test/util/data/blanktxv1.json \

8
ci/lint/04_install.sh
View File

@ -8,14 +8,14 @@ export LC_ALL=C
${CI_RETRY_EXE} apt-get update
${CI_RETRY_EXE} apt-get install -y clang-format-9 python3-pip curl git gawk jq
update-alternatives --install /usr/bin/clang-format clang-format $(which clang-format-9 ) 100
update-alternatives --install /usr/bin/clang-format-diff clang-format-diff $(which clang-format-diff-9) 100
update-alternatives --install /usr/bin/clang-format clang-format "$(which clang-format-9 )" 100
update-alternatives --install /usr/bin/clang-format-diff clang-format-diff "$(which clang-format-diff-9)" 100
${CI_RETRY_EXE} pip3 install codespell==2.0.0
${CI_RETRY_EXE} pip3 install flake8==3.8.3
${CI_RETRY_EXE} pip3 install mypy==0.910
${CI_RETRY_EXE} pip3 install pyzmq==22.3.0
${CI_RETRY_EXE} pip3 install vulture==2.3
${CI_RETRY_EXE} pip3 install yq
${CI_RETRY_EXE} pip3 install mypy==0.781
SHELLCHECK_VERSION=v0.8.0
curl -sL "https://github.com/koalaman/shellcheck/releases/download/${SHELLCHECK_VERSION}/shellcheck-${SHELLCHECK_VERSION}.linux.x86_64.tar.xz" | tar --xz -xf - --directory /tmp/

3
ci/lint/06_script.sh
View File

@ -25,7 +25,8 @@ test/lint/lint-all.sh
if [ "$CIRRUS_REPO_FULL_NAME" = "dashpay/dash" ] && [ -n "$CIRRUS_CRON" ]; then
git log --merges --before="2 days ago" -1 --format='%H' > ./contrib/verify-commits/trusted-sha512-root-commit
${CI_RETRY_EXE} gpg --keyserver hkps://keys.openpgp.org --recv-keys $(<contrib/verify-commits/trusted-keys) &&
mapfile -t KEYS < contrib/verify-commits/trusted-keys
${CI_RETRY_EXE} gpg --keyserver hkps://keys.openpgp.org --recv-keys "${KEYS[@]}" &&
./contrib/verify-commits/verify-commits.py --clean-merge=2;
fi

20
ci/test/04_install.sh
View File

@ -12,6 +12,7 @@ fi
if [ "$CI_OS_NAME" == "macos" ]; then
sudo -H pip3 install --upgrade pip
# shellcheck disable=SC2086
IN_GETOPT_BIN="/usr/local/opt/gnu-getopt/bin/getopt" ${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES
fi
@ -49,7 +50,7 @@ else
fi
DOCKER_EXEC () {
$DOCKER_CI_CMD_PREFIX bash -c "export PATH=$BASE_SCRATCH_DIR/bins/:\$PATH && cd $P_CI_DIR && $*"
$DOCKER_CI_CMD_PREFIX bash -c "export PATH=$BASE_SCRATCH_DIR/bins/:\$PATH && cd \"$P_CI_DIR\" && $*"
}
export -f DOCKER_EXEC
@ -59,11 +60,12 @@ fi
if [[ $DOCKER_NAME_TAG == centos* ]]; then
${CI_RETRY_EXE} DOCKER_EXEC yum -y install epel-release
${CI_RETRY_EXE} DOCKER_EXEC yum -y install $DOCKER_PACKAGES $PACKAGES
${CI_RETRY_EXE} DOCKER_EXEC yum -y install "$DOCKER_PACKAGES" "$PACKAGES"
elif [ "$CI_USE_APT_INSTALL" != "no" ]; then
${CI_RETRY_EXE} DOCKER_EXEC apt-get update
${CI_RETRY_EXE} DOCKER_EXEC apt-get install --no-install-recommends --no-upgrade -y $PACKAGES $DOCKER_PACKAGES
${CI_RETRY_EXE} DOCKER_EXEC apt-get install --no-install-recommends --no-upgrade -y "$PACKAGES" "$DOCKER_PACKAGES"
if [ -n "$PIP_PACKAGES" ]; then
# shellcheck disable=SC2086
${CI_RETRY_EXE} pip3 install --user $PIP_PACKAGES
fi
fi
@ -74,14 +76,14 @@ if [ "$CI_OS_NAME" == "macos" ]; then
else
DOCKER_EXEC free -m -h
DOCKER_EXEC echo "Number of CPUs \(nproc\):" \$\(nproc\)
DOCKER_EXEC echo $(lscpu | grep Endian)
DOCKER_EXEC echo "$(lscpu | grep Endian)"
fi
DOCKER_EXEC echo "Free disk space:"
DOCKER_EXEC df -h
if [ "$RUN_FUZZ_TESTS" = "true" ] || [ "$RUN_UNIT_TESTS" = "true" ] || [ "$RUN_UNIT_TESTS_SEQUENTIAL" = "true" ]; then
if [ ! -d ${DIR_QA_ASSETS} ]; then
DOCKER_EXEC git clone --depth=1 https://github.com/bitcoin-core/qa-assets ${DIR_QA_ASSETS}
if [ ! -d "${DIR_QA_ASSETS}" ]; then
DOCKER_EXEC git clone --depth=1 https://github.com/bitcoin-core/qa-assets "${DIR_QA_ASSETS}"
fi
export DIR_FUZZ_IN=${DIR_QA_ASSETS}/fuzz_seed_corpus/
@ -91,17 +93,17 @@ DOCKER_EXEC mkdir -p "${BASE_SCRATCH_DIR}/sanitizer-output/"
if [ -z "$DANGER_RUN_CI_ON_HOST" ]; then
echo "Create $BASE_ROOT_DIR"
DOCKER_EXEC rsync -a /ro_base/ $BASE_ROOT_DIR
DOCKER_EXEC rsync -a /ro_base/ "$BASE_ROOT_DIR"
fi
if [ "$USE_BUSY_BOX" = "true" ]; then
echo "Setup to use BusyBox utils"
DOCKER_EXEC mkdir -p $BASE_SCRATCH_DIR/bins/
DOCKER_EXEC mkdir -p "${BASE_SCRATCH_DIR}/bins/"
# tar excluded for now because it requires passing in the exact archive type in ./depends (fixed in later BusyBox version)
# find excluded for now because it does not recognize the -delete option in ./depends (fixed in later BusyBox version)
# ar excluded for now because it does not recognize the -q option in ./depends (unknown if fixed)
# shellcheck disable=SC1010
DOCKER_EXEC for util in \$\(busybox --list \| grep -v "^ar$" \| grep -v "^tar$" \| grep -v "^find$"\)\; do ln -s \$\(command -v busybox\) $BASE_SCRATCH_DIR/bins/\$util\; done
DOCKER_EXEC for util in \$\(busybox --list \| grep -v "^ar$" \| grep -v "^tar$" \| grep -v "^find$"\)\; do ln -s \$\(command -v busybox\) "${BASE_SCRATCH_DIR}/bins/\$util"\; done
# Print BusyBox version
DOCKER_EXEC patch --help
fi

8
ci/test/05_before_script.sh
View File

@ -8,12 +8,12 @@ export LC_ALL=C.UTF-8
# Make sure default datadir does not exist and is never read by creating a dummy file
if [ "$CI_OS_NAME" == "macos" ]; then
echo > $HOME/Library/Application\ Support/DashCore
echo > "${HOME}/Library/Application Support/DashCore"
else
DOCKER_EXEC echo \> \$HOME/.dashcore
fi
DOCKER_EXEC mkdir -p ${DEPENDS_DIR}/SDKs ${DEPENDS_DIR}/sdk-sources
DOCKER_EXEC mkdir -p "${DEPENDS_DIR}/SDKs" "${DEPENDS_DIR}/sdk-sources"
if [ -n "$XCODE_VERSION" ] && [ ! -f "$OSX_SDK_PATH" ]; then
DOCKER_EXEC curl --location --fail "${SDK_URL}/${OSX_SDK_BASENAME}" -o "$OSX_SDK_PATH"
@ -22,7 +22,7 @@ if [ -n "$XCODE_VERSION" ] && [ -f "$OSX_SDK_PATH" ]; then
DOCKER_EXEC tar -C "${DEPENDS_DIR}/SDKs" -xf "$OSX_SDK_PATH"
fi
if [[ $HOST = *-mingw32 ]]; then
DOCKER_EXEC update-alternatives --set $HOST-g++ \$\(which $HOST-g++-posix\)
DOCKER_EXEC update-alternatives --set "${HOST}-g++" \$\(which "${HOST}-g++-posix"\)
fi
if [ -z "$NO_DEPENDS" ]; then
if [[ $DOCKER_NAME_TAG == centos* ]]; then
@ -33,7 +33,7 @@ if [ -z "$NO_DEPENDS" ]; then
else
SHELL_OPTS="CONFIG_SHELL="
fi
DOCKER_EXEC $SHELL_OPTS make $MAKEJOBS -C depends HOST=$HOST $DEP_OPTS LOG=1
DOCKER_EXEC "$SHELL_OPTS" make "$MAKEJOBS" -C depends HOST="$HOST" "$DEP_OPTS" LOG=1
fi
if [ -n "$PREVIOUS_RELEASES_TO_DOWNLOAD" ]; then
DOCKER_EXEC test/get_previous_releases.py -b -t "$PREVIOUS_RELEASES_DIR" "${PREVIOUS_RELEASES_TO_DOWNLOAD}"

2
ci/test/wrap-valgrind.sh
View File

@ -8,7 +8,7 @@ export LC_ALL=C.UTF-8
for b_name in "${BASE_OUTDIR}/bin"/*; do
# shellcheck disable=SC2044
for b in $(find "${BASE_ROOT_DIR}" -executable -type f -name $(basename $b_name)); do
for b in $(find "${BASE_ROOT_DIR}" -executable -type f -name "$(basename "$b_name")"); do
echo "Wrap $b ..."
mv "$b" "${b}_orig"
echo '#!/usr/bin/env bash' > "$b"

2
ci/test/wrap-wine.sh
View File

@ -8,7 +8,7 @@ export LC_ALL=C.UTF-8
for b_name in {"${BASE_OUTDIR}/bin"/*,src/secp256k1/*tests,src/minisketch/test{,-verify},src/univalue/{no_nul,test_json,unitester,object}}.exe; do
# shellcheck disable=SC2044
for b in $(find "${BASE_ROOT_DIR}" -executable -type f -name "$(basename $b_name)"); do
for b in $(find "${BASE_ROOT_DIR}" -executable -type f -name "$(basename "$b_name")"); do
if (file "$b" | grep "Windows"); then
echo "Wrap $b ..."
mv "$b" "${b}_orig"

2
configure.ac
View File

@ -1923,7 +1923,7 @@ AC_CONFIG_LINKS([contrib/filter-lcov.py:contrib/filter-lcov.py])
AC_CONFIG_LINKS([contrib/macdeploy/background.tiff:contrib/macdeploy/background.tiff])
AC_CONFIG_LINKS([test/functional/test_runner.py:test/functional/test_runner.py])
AC_CONFIG_LINKS([test/fuzz/test_runner.py:test/fuzz/test_runner.py])
AC_CONFIG_LINKS([test/util/bitcoin-util-test.py:test/util/bitcoin-util-test.py])
AC_CONFIG_LINKS([test/util/test_runner.py:test/util/test_runner.py])
AC_CONFIG_LINKS([test/util/rpcauth-test.py:test/util/rpcauth-test.py])
dnl boost's m4 checks do something really nasty: they export these vars. As a

9
contrib/containers/ci/Dockerfile
View File

@ -77,11 +77,10 @@ RUN pip3 install \
flake8==3.8.3 \
jinja2 \
lief==0.13.2 \
pyzmq \
vulture==2.3 \
mypy==0.781 \
yq \
multiprocess
multiprocess \
mypy==0.910 \
pyzmq==22.3.0 \
vulture==2.3
# dash_hash
ARG DASH_HASH_VERSION=1.4.0

8
contrib/devtools/gen-manpages.sh
View File

@ -16,7 +16,7 @@ BITCOINTX=${BITCOINTX:-$BINDIR/dash-tx}
WALLET_TOOL=${WALLET_TOOL:-$BINDIR/dash-wallet}
BITCOINQT=${BITCOINQT:-$BINDIR/qt/dash-qt}
[ ! -x $BITCOIND ] && echo "$BITCOIND not found or not executable." && exit 1
[ ! -x "$BITCOIND" ] && echo "$BITCOIND not found or not executable." && exit 1
# Don't allow man pages to be generated for binaries built from a dirty tree
DIRTY=""
@ -29,7 +29,7 @@ done
if [ -n "$DIRTY" ]
then
echo -e "WARNING: the following binaries were built from a dirty tree:\n"
echo -e $DIRTY
echo -e "$DIRTY"
echo "man pages generated from dirty binaries should NOT be committed."
echo "To properly generate man pages, please commit your changes to the above binaries, rebuild them, then run this script again."
fi
@ -45,8 +45,8 @@ $BITCOIND --version | sed -n '1!p' >> footer.h2m
for cmd in $BITCOIND $BITCOINCLI $BITCOINTX $WALLET_TOOL $BITCOINQT; do
cmdname="${cmd##*/}"
help2man -N --version-string=${BTCVER[0]} --include=footer.h2m -o ${MANDIR}/${cmdname}.1 ${cmd}
sed -i "s/\\\-${BTCVER[1]}//g" ${MANDIR}/${cmdname}.1
help2man -N --version-string="${BTCVER[0]}" --include=footer.h2m -o "${MANDIR}/${cmdname}.1" "${cmd}"
sed -i "s/\\\-${BTCVER[1]}//g" "${MANDIR}/${cmdname}.1"
done
rm -f footer.h2m

2
contrib/devtools/security-check.py
View File

@ -10,7 +10,7 @@ Otherwise the exit status will be 1 and it will log which executables failed whi
import sys
from typing import List
import lief
import lief #type:ignore
def check_ELF_RELRO(binary) -> bool:
'''

2
contrib/devtools/symbol-check.py
View File

@ -13,7 +13,7 @@ Example usage:
import sys
from typing import Dict, List
import lief
import lief #type:ignore
# Debian 11 (Bullseye) EOL: 2026. https://wiki.debian.org/LTS
#

2
contrib/install_db4.sh
View File

@ -20,7 +20,7 @@ expand_path() {
cd "${1}" && pwd -P
}
BDB_PREFIX="$(expand_path ${1})/db4"; shift;
BDB_PREFIX="$(expand_path "${1}")/db4"; shift;
BDB_VERSION='db-4.8.30.NC'
BDB_HASH='12edc0df75bf9abd7f82f821795bcee50f42cb2e5f76a6a281b85732798364ef'
BDB_URL="https://download.oracle.com/berkeley-db/${BDB_VERSION}.tar.gz"

4
contrib/verify-commits/pre-push-hook.sh
View File

@ -9,11 +9,11 @@ if ! [[ "$2" =~ ^(git@)?(www.)?github.com(:|/)dashpay/dash(.git)?$ ]]; then
fi
while read LINE; do
set -- A $LINE
set -- A "$LINE"
if [ "$4" != "refs/heads/master" ]; then
continue
fi
if ! ./contrib/verify-commits/verify-commits.py $3 > /dev/null 2>&1; then
if ! ./contrib/verify-commits/verify-commits.py "$3" > /dev/null 2>&1; then
echo "ERROR: A commit is not signed, can't push"
./contrib/verify-commits/verify-commits.py
exit 1

1
contrib/windeploy/detached-sig-create.sh
View File

@ -23,6 +23,7 @@ TIMESERVER=http://timestamp.comodoca.com
CERTFILE="win-codesign.cert"
mkdir -p "${OUTSUBDIR}"
# shellcheck disable=SC2046
basename -a $(ls -1 "${SRCDIR}"/*-unsigned.exe) | while read UNSIGNED; do
echo Signing "${UNSIGNED}"
"${OSSLSIGNCODE}" sign -certs "${CERTFILE}" -t "${TIMESERVER}" -h sha256 -in "${SRCDIR}/${UNSIGNED}" -out "${WORKDIR}/${UNSIGNED}" "$@"

3
doc/fuzzing.md
View File

@ -16,6 +16,9 @@ $ FUZZ=process_message src/test/fuzz/fuzz
# abort fuzzing using ctrl-c
```
There is also a runner script to execute all fuzz targets. Refer to
`./test/fuzz/test_runner.py --help` for more details.
## Fuzzing harnesses and output
[`process_message`](https://github.com/dashpay/dash/blob/develop/src/test/fuzz/process_message.cpp) is a fuzzing harness for the [`ProcessMessage(...)` function (`net_processing`)](https://github.com/dashpay/dash/blob/develop/src/net_processing.cpp). The available fuzzing harnesses are found in [`src/test/fuzz/`](https://github.com/dashpay/dash/tree/develop/src/test/fuzz).

4
src/Makefile.test.include
View File

@ -375,8 +375,8 @@ check-unit: $(BITCOIN_TESTS:.cpp=.cpp.test)
check-local: check-unit
if BUILD_BITCOIN_TX
@echo "Running test/util/bitcoin-util-test.py..."
$(PYTHON) $(top_builddir)/test/util/bitcoin-util-test.py
@echo "Running test/util/test_runner.py..."
$(PYTHON) $(top_builddir)/test/util/test_runner.py
endif
@echo "Running test/util/rpcauth-test.py..."
$(PYTHON) $(top_builddir)/test/util/rpcauth-test.py

6
src/qt/res/animation/makespinner.sh
View File

@ -5,10 +5,10 @@
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
FRAMEDIR=$(dirname $0)
FRAMEDIR=$(dirname "$0")
for i in {0..89}
do
frame=$(printf "%03d" $i)
frame=$(printf "%03d" "$i")
angle=$((i * 4))
convert $FRAMEDIR/../src/spinner.png -background "rgba(0,0,0,0.0)" -distort SRT $angle $FRAMEDIR/spinner-$frame.png
convert "${FRAMEDIR}/../src/spinner.png" -background "rgba(0,0,0,0.0)" -distort SRT $angle "${FRAMEDIR}/spinner-${frame}.png"
done

2
src/random.cpp
View File

@ -14,7 +14,7 @@
#include <compat.h> // for Windows API
#include <wincrypt.h>
#endif
#include <logging.h> // for LogPrintf()
#include <logging.h>
#include <randomenv.h>
#include <support/allocators/secure.h>
#include <span.h>

31
test/README.md
View File

@ -5,20 +5,24 @@ etc.
This directory contains the following sets of tests:
- [fuzz](/test/fuzz) A runner to execute all fuzz targets from
[/src/test/fuzz](/src/test/fuzz).
- [functional](/test/functional) which test the functionality of
dashd and dash-qt by interacting with them through the RPC and P2P
interfaces.
- [util](/test/util) which tests the dash utilities, currently only
dash-tx.
- [util](/test/util) which tests the utilities (dash-tx, ...).
- [lint](/test/lint/) which perform various static analysis checks.
The util tests are run as part of `make check` target. The functional
The util tests are run as part of `make check` target. The fuzz tests, functional
tests and lint scripts can be run as explained in the sections below.
# Running tests locally
Before tests can be run locally, Dash Core must be built. See the [building instructions](/doc#building) for help.
## Fuzz tests
See [/doc/fuzzing.md](/doc/fuzzing.md)
### Functional tests
@ -297,20 +301,23 @@ For ways to generate more granular profiles, see the README in
### Util tests
Util tests can be run locally by running `test/util/bitcoin-util-test.py`.
Util tests can be run locally by running `test/util/test_runner.py`.
Use the `-v` option for verbose output.
### Lint tests
#### Dependencies
| Lint test | Dependency | Version [used by CI](../ci/lint/04_install.sh) | Installation
|-----------|:----------:|:-------------------------------------------:|--------------
| [`lint-python.sh`](lint/lint-python.sh) | [flake8](https://gitlab.com/pycqa/flake8) | [3.8.3](https://github.com/bitcoin/bitcoin/pull/19348) | `pip3 install flake8==3.8.3`
| [`lint-python.sh`](lint/lint-python.sh) | [mypy](https://github.com/python/mypy) | [0.781](https://github.com/bitcoin/bitcoin/pull/19348) | `pip3 install mypy==0.781`
| [`lint-shell.sh`](lint/lint-shell.sh) | [ShellCheck](https://github.com/koalaman/shellcheck) | [0.7.2](https://github.com/bitcoin/bitcoin/pull/21749) | [details...](https://github.com/koalaman/shellcheck#installing)
| [`lint-shell.sh`](lint/lint-shell.sh) | [yq](https://github.com/kislyuk/yq) | default | `pip3 install yq`
| [`lint-spelling.sh`](lint/lint-spelling.sh) | [codespell](https://github.com/codespell-project/codespell) | [2.0.0](https://github.com/bitcoin/bitcoin/pull/20817) | `pip3 install codespell==2.0.0`
| Lint test | Dependency |
|-----------|:----------:|
| [`lint-python.sh`](lint/lint-python.sh) | [flake8](https://gitlab.com/pycqa/flake8)
| [`lint-python.sh`](lint/lint-python.sh) | [mypy](https://github.com/python/mypy)
| [`lint-python.sh`](lint/lint-python.sh) | [pyzmq](https://github.com/zeromq/pyzmq)
| [`lint-python-dead-code.py`](lint/lint-python-dead-code.py) | [vulture](https://github.com/jendrikseipp/vulture)
| [`lint-shell.sh`](lint/lint-shell.sh) | [ShellCheck](https://github.com/koalaman/shellcheck)
| [`lint-spelling.py`](lint/lint-spelling.py) | [codespell](https://github.com/codespell-project/codespell)
In use versions and install instructions are available in the [CI setup](../ci/lint/04_install.sh).
Please be aware that on Linux distributions all dependencies are usually available as packages, but could be outdated.
@ -319,7 +326,7 @@ Please be aware that on Linux distributions all dependencies are usually availab
Individual tests can be run by directly calling the test script, e.g.:
```
test/lint/lint-files.sh
test/lint/lint-files.py
```
You can run all the shell-based lint tests by running:

2
test/config.ini.in
View File

@ -3,7 +3,7 @@
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
# These environment variables are set by the build process and read by
# test/functional/test_runner.py and test/util/bitcoin-util-test.py
# test/*/test_runner.py and test/util/rpcauth-test.py
[environment]
PACKAGE_NAME=@PACKAGE_NAME@

2
test/functional/combine_logs.py
View File

@ -188,7 +188,7 @@ def print_logs_plain(log_events, colors):
def print_logs_html(log_events):
"""Renders the iterator of log events into html."""
try:
import jinja2
import jinja2 #type:ignore
except ImportError:
print("jinja2 not found. Try `pip install jinja2`")
sys.exit(1)

0
test/functional/data/__init__.py Normal file
View File

2
test/functional/test_runner.py
View File

@ -40,7 +40,7 @@ except UnicodeDecodeError:
CROSS = "x "
CIRCLE = "o "
if os.name != 'nt' or sys.getwindowsversion() >= (10, 0, 14393):
if os.name != 'nt' or sys.getwindowsversion() >= (10, 0, 14393): #type:ignore
if os.name == 'nt':
import ctypes
kernel32 = ctypes.windll.kernel32 # type: ignore

16
test/lint/commit-script-check.sh
View File

@ -12,7 +12,7 @@
# one. Any remaining diff signals an error.
export LC_ALL=C
if test -z $1; then
if test -z "$1"; then
echo "Usage: $0 <commit>..."
exit 1
fi
@ -30,10 +30,10 @@ fi
RET=0
PREV_BRANCH=$(git name-rev --name-only HEAD)
PREV_HEAD=$(git rev-parse HEAD)
for commit in $(git rev-list --reverse $1); do
if git rev-list -n 1 --pretty="%s" $commit | grep -q "^scripted-diff:"; then
git checkout --quiet $commit^ || exit
SCRIPT="$(git rev-list --format=%b -n1 $commit | sed '/^-BEGIN VERIFY SCRIPT-$/,/^-END VERIFY SCRIPT-$/{//!b};d')"
for commit in $(git rev-list --reverse "$1"); do
if git rev-list -n 1 --pretty="%s" "$commit" | grep -q "^scripted-diff:"; then
git checkout --quiet "$commit"^ || exit
SCRIPT="$(git rev-list --format=%b -n1 "$commit" | sed '/^-BEGIN VERIFY SCRIPT-$/,/^-END VERIFY SCRIPT-$/{//!b};d')"
if test -z "$SCRIPT"; then
echo "Error: missing script for: $commit"
echo "Failed"
@ -42,16 +42,16 @@ for commit in $(git rev-list --reverse $1); do
echo "Running script for: $commit"
echo "$SCRIPT"
(eval "$SCRIPT")
git --no-pager diff --exit-code $commit && echo "OK" || (echo "Failed"; false) || RET=1
git --no-pager diff --exit-code "$commit" && echo "OK" || (echo "Failed"; false) || RET=1
fi
git reset --quiet --hard HEAD
else
if git rev-list "--format=%b" -n1 $commit | grep -q '^-\(BEGIN\|END\)[ a-zA-Z]*-$'; then
if git rev-list "--format=%b" -n1 "$commit" | grep -q '^-\(BEGIN\|END\)[ a-zA-Z]*-$'; then
echo "Error: script block marker but no scripted-diff in title of commit $commit"
echo "Failed"
RET=1
fi
fi
done
git checkout --quiet $PREV_BRANCH 2>/dev/null || git checkout --quiet $PREV_HEAD
git checkout --quiet "$PREV_BRANCH" 2>/dev/null || git checkout --quiet "$PREV_HEAD"
exit $RET

10
test/lint/git-subtree-check.sh
View File

@ -82,6 +82,7 @@ if [ -z "$latest_squash" ]; then
echo "ERROR: $DIR is not a subtree" >&2
exit 2
fi
# shellcheck disable=SC2086
set $latest_squash
old=$1
rev=$2
@ -92,6 +93,7 @@ if [ -z "$tree_actual" ]; then
echo "FAIL: subtree directory $DIR not found in $COMMIT" >&2
exit 1
fi
# shellcheck disable=SC2086
set $tree_actual
tree_actual_type=$2
tree_actual_tree=$3
@ -102,23 +104,23 @@ if [ "d$tree_actual_type" != "dtree" ]; then
fi
# get the tree at the time of the last subtree update
tree_commit=$(git show -s --format="%T" $old)
tree_commit=$(git show -s --format="%T" "$old")
echo "$DIR in $COMMIT was last updated in commit $old (tree $tree_commit)"
# ... and compare the actual tree with it
if [ "$tree_actual_tree" != "$tree_commit" ]; then
git diff $tree_commit $tree_actual_tree >&2
git diff "$tree_commit" "$tree_actual_tree" >&2
echo "FAIL: subtree directory was touched without subtree merge" >&2
exit 1
fi
if [ "$check_remote" != "0" ]; then
# get the tree in the subtree commit referred to
if [ "d$(git cat-file -t $rev 2>/dev/null)" != dcommit ]; then
if [ "d$(git cat-file -t "$rev" 2>/dev/null)" != dcommit ]; then
echo "subtree commit $rev unavailable: cannot compare. Did you add and fetch the remote?" >&2
exit 1
fi
tree_subtree=$(git show -s --format="%T" $rev)
tree_subtree=$(git show -s --format="%T" "$rev")
echo "$DIR in $COMMIT was last updated to upstream commit $rev (tree $tree_subtree)"
# ... and compare the actual tree with it

4
test/lint/lint-all.sh
View File

@ -30,13 +30,13 @@ if ! command -v parallel > /dev/null; then
else
SCRIPTS=()
for f in "${SCRIPTDIR}"/lint-*.sh; do
for f in "${SCRIPTDIR}"/lint-*; do
if [ "$(basename "$f")" != "$LINTALL" ]; then
SCRIPTS+=("$f")
fi
done
if ! parallel --jobs 100% --will-cite --joblog parallel_out.log bash ::: "${SCRIPTS[@]}"; then
if ! parallel --jobs 100% --will-cite --joblog parallel_out.log ::: "${SCRIPTS[@]}"; then
echo "^---- failure generated"
EXIT_CODE=1
fi

99
test/lint/lint-format-strings.py Executable file
View File

@ -0,0 +1,99 @@
#!/usr/bin/env python3
#
# Copyright (c) 2018-2022 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
"""
Lint format strings: This program checks that the number of arguments passed
to a variadic format string function matches the number of format specifiers
in the format string.
"""
import subprocess
import re
import sys
FUNCTION_NAMES_AND_NUMBER_OF_LEADING_ARGUMENTS = [
'FatalError,0',
'fprintf,1',
'tfm::format,1', # Assuming tfm::::format(std::ostream&, ...
'LogConnectFailure,1',
'LogPrint,1',
'LogPrintf,0',
'LogPrintLevel,2',
'printf,0',
'snprintf,2',
'sprintf,1',
'strprintf,0',
'vfprintf,1',
'vprintf,1',
'vsnprintf,1',
'vsprintf,1',
'WalletLogPrintf,0',
]
RUN_LINT_FILE = 'test/lint/run-lint-format-strings.py'
def check_doctest():
command = [
'python3',
'-m',
'doctest',
RUN_LINT_FILE,
]
try:
subprocess.run(command, check = True)
except subprocess.CalledProcessError:
sys.exit(1)
def get_matching_files(function_name):
command = [
'git',
'grep',
'--full-name',
'-l',
function_name,
'--',
'*.c',
'*.cpp',
'*.h',
]
try:
return subprocess.check_output(command, stderr = subprocess.STDOUT).decode('utf-8').splitlines()
except subprocess.CalledProcessError as e:
if e.returncode > 1: # return code is 1 when match is empty
print(e.output.decode('utf-8'), end='')
sys.exit(1)
return []
def main():
exit_code = 0
check_doctest()
for s in FUNCTION_NAMES_AND_NUMBER_OF_LEADING_ARGUMENTS:
function_name, skip_arguments = s.split(',')
matching_files = get_matching_files(function_name)
matching_files_filtered = []
for matching_file in matching_files:
if not re.search('^src/(dashbls|leveldb|secp256k1|minisketch|tinyformat|univalue|test/fuzz/strprintf.cpp)', matching_file):
matching_files_filtered.append(matching_file)
matching_files_filtered.sort()
run_lint_args = [
RUN_LINT_FILE,
'--skip-arguments',
skip_arguments,
function_name,
]
run_lint_args.extend(matching_files_filtered)
try:
subprocess.run(run_lint_args, check = True)
except subprocess.CalledProcessError:
exit_code = 1
sys.exit(exit_code)
if __name__ == '__main__':
main()

45
test/lint/lint-format-strings.sh
View File

@ -1,45 +0,0 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Lint format strings: This program checks that the number of arguments passed
# to a variadic format string function matches the number of format specifiers
# in the format string.
export LC_ALL=C
FUNCTION_NAMES_AND_NUMBER_OF_LEADING_ARGUMENTS=(
"FatalError,0"
"fprintf,1"
"tfm::format,1" # Assuming tfm::::format(std::ostream&, ...
"LogConnectFailure,1"
"LogPrint,1"
"LogPrintf,0"
"LogPrintLevel,2"
"printf,0"
"snprintf,2"
"sprintf,1"
"strprintf,0"
"vfprintf,1"
"vprintf,1"
"vsnprintf,1"
"vsprintf,1"
"WalletLogPrintf,0"
)
EXIT_CODE=0
if ! python3 -m doctest "test/lint/run-lint-format-strings.py"; then
EXIT_CODE=1
fi
for S in "${FUNCTION_NAMES_AND_NUMBER_OF_LEADING_ARGUMENTS[@]}"; do
IFS="," read -r FUNCTION_NAME SKIP_ARGUMENTS <<< "${S}"
for MATCHING_FILE in $(git grep --full-name -l "${FUNCTION_NAME}" -- "*.c" "*.cpp" "*.h" | sort | grep -vE "^src/(dashbls|leveldb|secp256k1|minisketch|tinyformat|univalue|test/fuzz/strprintf.cpp)"); do
MATCHING_FILES+=("${MATCHING_FILE}")
done
if ! "test/lint/run-lint-format-strings.py" --skip-arguments "${SKIP_ARGUMENTS}" "${FUNCTION_NAME}" "${MATCHING_FILES[@]}"; then
EXIT_CODE=1
fi
done
exit ${EXIT_CODE}

63
test/lint/lint-git-commit-check.py Executable file
View File

@ -0,0 +1,63 @@
#!/usr/bin/env python3
#
# Copyright (c) 2020-2022 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Linter to check that commit messages have a new line before the body
# or no body at all
import argparse
import os
import sys
from subprocess import check_output
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="""
Linter to check that commit messages have a new line before
the body or no body at all.
""",
epilog=f"""
You can manually set the commit-range with the COMMIT_RANGE
environment variable (e.g. "COMMIT_RANGE='47ba2c3...ee50c9e'
{sys.argv[0]}"). Defaults to current merge base when neither
prev-commits nor the environment variable is set.
""")
parser.add_argument("--prev-commits", "-p", required=False, help="The previous n commits to check")
return parser.parse_args()
def main():
args = parse_args()
exit_code = 0
if not os.getenv("COMMIT_RANGE"):
if args.prev_commits:
commit_range = "HEAD~" + args.prev_commits + "...HEAD"
else:
# This assumes that the target branch of the pull request will be develop.
merge_base = check_output(["git", "merge-base", "HEAD", "develop"], universal_newlines=True, encoding="utf8").rstrip("\n")
commit_range = merge_base + "..HEAD"
else:
commit_range = os.getenv("COMMIT_RANGE")
commit_hashes = check_output(["git", "log", commit_range, "--format=%H"], universal_newlines=True, encoding="utf8").splitlines()
for hash in commit_hashes:
commit_info = check_output(["git", "log", "--format=%B", "-n", "1", hash], universal_newlines=True, encoding="utf8").splitlines()
if len(commit_info) >= 2:
if commit_info[1]:
print(f"The subject line of commit hash {hash} is followed by a non-empty line. Subject lines should always be followed by a blank line.")
exit_code = 1
sys.exit(exit_code)
if __name__ == "__main__":
main()

48
test/lint/lint-git-commit-check.sh
View File

@ -1,48 +0,0 @@
#!/usr/bin/env bash
# Copyright (c) 2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Linter to check that commit messages have a new line before the body
# or no body at all
export LC_ALL=C
EXIT_CODE=0
while getopts "?" opt; do
case $opt in
?)
echo "Usage: $0 [N]"
echo " COMMIT_RANGE='<commit range>' $0"
echo " $0 -?"
echo "Checks unmerged commits, the previous N commits, or a commit range."
echo "COMMIT_RANGE='47ba2c3...ee50c9e' $0"
exit ${EXIT_CODE}
;;
esac
done
if [ -z "${COMMIT_RANGE}" ]; then
if [ -n "$1" ]; then
COMMIT_RANGE="HEAD~$1...HEAD"
else
# This assumes that the target branch of the pull request will be develop
MERGE_BASE=$(git merge-base HEAD develop)
COMMIT_RANGE="$MERGE_BASE..HEAD"
fi
fi
while IFS= read -r commit_hash || [[ -n "$commit_hash" ]]; do
n_line=0
while IFS= read -r line || [[ -n "$line" ]]; do
n_line=$((n_line+1))
length=${#line}
if [ $n_line -eq 2 ] && [ $length -ne 0 ]; then
echo "The subject line of commit hash ${commit_hash} is followed by a non-empty line. Subject lines should always be followed by a blank line."
EXIT_CODE=1
fi
done < <(git log --format=%B -n 1 "$commit_hash")
done < <(git log "${COMMIT_RANGE}" --format=%H)
exit ${EXIT_CODE}

184
test/lint/lint-includes.py Executable file
View File

@ -0,0 +1,184 @@
#!/usr/bin/env python3
#
# Copyright (c) 2018-2022 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Check for duplicate includes.
# Guard against accidental introduction of new Boost dependencies.
# Check includes: Check for duplicate includes. Enforce bracket syntax includes.
import os
import re
import sys
from subprocess import check_output, CalledProcessError
EXCLUDED_DIRS = ["src/leveldb/",
"src/crc32c/",
"src/secp256k1/",
"src/minisketch/",
"src/univalue/",
"src/dashbls/",
"src/immer/",
"src/crypto/x11/"]
EXPECTED_BOOST_INCLUDES = ["boost/date_time/posix_time/posix_time.hpp",
"boost/multi_index/hashed_index.hpp",
"boost/multi_index/identity.hpp",
"boost/multi_index/indexed_by.hpp",
"boost/multi_index/ordered_index.hpp",
"boost/multi_index/sequenced_index.hpp",
"boost/multi_index/tag.hpp",
"boost/multi_index_container.hpp",
"boost/pool/pool_alloc.hpp",
"boost/process.hpp",
"boost/signals2/connection.hpp",
"boost/signals2/optional_last_value.hpp",
"boost/signals2/signal.hpp",
"boost/test/included/unit_test.hpp",
"boost/test/unit_test.hpp"]
EXCLUDED_BOOST_DIRS = ["src/immer/"]
def get_toplevel():
return check_output(["git", "rev-parse", "--show-toplevel"], universal_newlines=True, encoding="utf8").rstrip("\n")
def list_files_by_suffix(suffixes):
exclude_args = [":(exclude)" + dir for dir in EXCLUDED_DIRS]
files_list = check_output(["git", "ls-files", "src"] + exclude_args, universal_newlines=True, encoding="utf8").splitlines()
return [file for file in files_list if file.endswith(suffixes)]
def find_duplicate_includes(include_list):
tempset = set()
duplicates = set()
for inclusion in include_list:
if inclusion in tempset:
duplicates.add(inclusion)
else:
tempset.add(inclusion)
return duplicates
def find_included_cpps():
included_cpps = list()
try:
included_cpps = check_output(["git", "grep", "-E", r"^#include [<\"][^>\"]+\.cpp[>\"]", "--", "*.cpp", "*.h"], universal_newlines=True, encoding="utf8").splitlines()
except CalledProcessError as e:
if e.returncode > 1:
raise e
return included_cpps
def find_extra_boosts():
included_boosts = list()
filtered_included_boost_set = set()
exclusion_set = set()
exclude_args = [":(exclude)" + dir for dir in EXCLUDED_BOOST_DIRS]
try:
included_boosts = check_output(["git", "grep", "-E", r"^#include <boost/", "--", "*.cpp", "*.h"] + exclude_args, universal_newlines=True, encoding="utf8").splitlines()
except CalledProcessError as e:
if e.returncode > 1:
raise e
for boost in included_boosts:
filtered_included_boost_set.add(re.findall(r'(?<=\<).+?(?=\>)', boost)[0])
for expected_boost in EXPECTED_BOOST_INCLUDES:
for boost in filtered_included_boost_set:
if expected_boost in boost:
exclusion_set.add(boost)
extra_boosts = set(filtered_included_boost_set.difference(exclusion_set))
return extra_boosts
def find_quote_syntax_inclusions():
exclude_args = [":(exclude)" + dir for dir in EXCLUDED_DIRS]
quote_syntax_inclusions = list()
try:
quote_syntax_inclusions = check_output(["git", "grep", r"^#include \"", "--", "*.cpp", "*.h"] + exclude_args, universal_newlines=True, encoding="utf8").splitlines()
except CalledProcessError as e:
if e.returncode > 1:
raise e
return quote_syntax_inclusions
def main():
exit_code = 0
os.chdir(get_toplevel())
# Check for duplicate includes
for filename in list_files_by_suffix((".cpp", ".h")):
with open(filename, "r", encoding="utf8") as file:
include_list = [line.rstrip("\n") for line in file if re.match(r"^#include", line)]
duplicates = find_duplicate_includes(include_list)
if duplicates:
print(f"Duplicate include(s) in {filename}:")
for duplicate in duplicates:
print(duplicate)
print("")
exit_code = 1
# Check if code includes .cpp-files
included_cpps = find_included_cpps()
if included_cpps:
print("The following files #include .cpp files:")
for included_cpp in included_cpps:
print(included_cpp)
print("")
exit_code = 1
# Guard against accidental introduction of new Boost dependencies
extra_boosts = find_extra_boosts()
if extra_boosts:
for boost in extra_boosts:
print(f"A new Boost dependency in the form of \"{boost}\" appears to have been introduced:")
print(check_output(["git", "grep", boost, "--", "*.cpp", "*.h"], universal_newlines=True, encoding="utf8"))
exit_code = 1
# Check if Boost dependencies are no longer used
for expected_boost in EXPECTED_BOOST_INCLUDES:
try:
check_output(["git", "grep", "-q", r"^#include <%s>" % expected_boost, "--", "*.cpp", "*.h"], universal_newlines=True, encoding="utf8")
except CalledProcessError as e:
if e.returncode > 1:
raise e
else:
print(f"Good job! The Boost dependency \"{expected_boost}\" is no longer used. "
"Please remove it from EXPECTED_BOOST_INCLUDES in test/lint/lint-includes.py "
"to make sure this dependency is not accidentally reintroduced.\n")
exit_code = 1
# Enforce bracket syntax includes
quote_syntax_inclusions = find_quote_syntax_inclusions()
if quote_syntax_inclusions:
print("Please use bracket syntax includes (\"#include <foo.h>\") instead of quote syntax includes:")
for quote_syntax_inclusion in quote_syntax_inclusions:
print(quote_syntax_inclusion)
exit_code = 1
sys.exit(exit_code)
if __name__ == "__main__":
main()

103
test/lint/lint-includes.sh
View File

@ -1,103 +0,0 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Check for duplicate includes.
# Guard against accidental introduction of new Boost dependencies.
# Check includes: Check for duplicate includes. Enforce bracket syntax includes.
export LC_ALL=C
IGNORE_REGEXP="/(dashbls|immer|leveldb|secp256k1|minisketch|univalue|crc32c|crypto/x11)/"
# cd to root folder of git repo for git ls-files to work properly
cd "$(dirname $0)/../.." || exit 1
filter_suffix() {
git ls-files | grep -E "^src/.*\.${1}"'$' | grep -Ev "${IGNORE_REGEXP}"
}
EXIT_CODE=0
for HEADER_FILE in $(filter_suffix h); do
DUPLICATE_INCLUDES_IN_HEADER_FILE=$(grep -E "^#include " < "${HEADER_FILE}" | sort | uniq -d)
if [[ ${DUPLICATE_INCLUDES_IN_HEADER_FILE} != "" ]]; then
echo "Duplicate include(s) in ${HEADER_FILE}:"
echo "${DUPLICATE_INCLUDES_IN_HEADER_FILE}"
echo
EXIT_CODE=1
fi
done
for CPP_FILE in $(filter_suffix cpp); do
DUPLICATE_INCLUDES_IN_CPP_FILE=$(grep -E "^#include " < "${CPP_FILE}" | sort | uniq -d)
if [[ ${DUPLICATE_INCLUDES_IN_CPP_FILE} != "" ]]; then
echo "Duplicate include(s) in ${CPP_FILE}:"
echo "${DUPLICATE_INCLUDES_IN_CPP_FILE}"
echo
EXIT_CODE=1
fi
done
INCLUDED_CPP_FILES=$(git grep -E "^#include [<\"][^>\"]+\.cpp[>\"]" -- "*.cpp" "*.h")
if [[ ${INCLUDED_CPP_FILES} != "" ]]; then
echo "The following files #include .cpp files:"
echo "${INCLUDED_CPP_FILES}"
echo
EXIT_CODE=1
fi
EXPECTED_BOOST_INCLUDES=(
boost/date_time/posix_time/posix_time.hpp
boost/multi_index/hashed_index.hpp
boost/multi_index/identity.hpp
boost/multi_index/indexed_by.hpp
boost/multi_index/ordered_index.hpp
boost/multi_index/sequenced_index.hpp
boost/multi_index/tag.hpp
boost/multi_index_container.hpp
boost/pool/pool_alloc.hpp
boost/process.hpp
boost/signals2/connection.hpp
boost/signals2/optional_last_value.hpp
boost/signals2/signal.hpp
boost/test/included/unit_test.hpp
boost/test/unit_test.hpp
)
for BOOST_INCLUDE in $(git grep '^#include <boost/' -- "*.cpp" "*.h" | grep -vE "src/(immer)/" | cut -f2 -d: | cut -f2 -d'<' | cut -f1 -d'>' | sort -u); do
IS_EXPECTED_INCLUDE=0
for EXPECTED_BOOST_INCLUDE in "${EXPECTED_BOOST_INCLUDES[@]}"; do
if [[ "${BOOST_INCLUDE}" == "${EXPECTED_BOOST_INCLUDE}" ]]; then
IS_EXPECTED_INCLUDE=1
break
fi
done
if [[ ${IS_EXPECTED_INCLUDE} == 0 ]]; then
EXIT_CODE=1
echo "A new Boost dependency in the form of \"${BOOST_INCLUDE}\" appears to have been introduced:"
git grep "${BOOST_INCLUDE}" -- "*.cpp" "*.h"
echo
fi
done
for EXPECTED_BOOST_INCLUDE in "${EXPECTED_BOOST_INCLUDES[@]}"; do
if ! git grep -q "^#include <${EXPECTED_BOOST_INCLUDE}>" -- "*.cpp" "*.h"; then
echo "Good job! The Boost dependency \"${EXPECTED_BOOST_INCLUDE}\" is no longer used."
echo "Please remove it from EXPECTED_BOOST_INCLUDES in $0"
echo "to make sure this dependency is not accidentally reintroduced."
echo
EXIT_CODE=1
fi
done
QUOTE_SYNTAX_INCLUDES=$(git grep '^#include "' -- "*.cpp" "*.h" | grep -Ev "${IGNORE_REGEXP}")
if [[ ${QUOTE_SYNTAX_INCLUDES} != "" ]]; then
echo "Please use bracket syntax includes (\"#include <foo.h>\") instead of quote syntax includes:"
echo "${QUOTE_SYNTAX_INCLUDES}"
echo
EXIT_CODE=1
fi
exit ${EXIT_CODE}

264
test/lint/lint-locale-dependence.py Executable file
View File

@ -0,0 +1,264 @@
#!/usr/bin/env python3
# Copyright (c) 2018-2022 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Be aware that bitcoind and bitcoin-qt differ in terms of localization: Qt
# opts in to POSIX localization by running setlocale(LC_ALL, "") on startup,
# whereas no such call is made in bitcoind.
#
# Qt runs setlocale(LC_ALL, "") on initialization. This installs the locale
# specified by the user's LC_ALL (or LC_*) environment variable as the new
# C locale.
#
# In contrast, bitcoind does not opt in to localization -- no call to
# setlocale(LC_ALL, "") is made and the environment variables LC_* are
# thus ignored.
#
# This results in situations where bitcoind is guaranteed to be running
# with the classic locale ("C") whereas the locale of bitcoin-qt will vary
# depending on the user's environment variables.
#
# An example: Assuming the environment variable LC_ALL=de_DE then the
# call std::to_string(1.23) will return "1.230000" in bitcoind but
# "1,230000" in bitcoin-qt.
#
# From the Qt documentation:
# "On Unix/Linux Qt is configured to use the system locale settings by default.
# This can cause a conflict when using POSIX functions, for instance, when
# converting between data types such as floats and strings, since the notation
# may differ between locales. To get around this problem, call the POSIX function
# setlocale(LC_NUMERIC,"C") right after initializing QApplication, QGuiApplication
# or QCoreApplication to reset the locale that is used for number formatting to
# "C"-locale."
#
# See https://doc.qt.io/qt-5/qcoreapplication.html#locale-settings and
# https://stackoverflow.com/a/34878283 for more details.
#
# TODO: Reduce KNOWN_VIOLATIONS by replacing uses of locale dependent stoul/strtol with locale
# independent ToIntegral<T>(...).
# TODO: Reduce KNOWN_VIOLATIONS by replacing uses of locale dependent snprintf with strprintf.
import re
import sys
from subprocess import check_output, CalledProcessError
KNOWN_VIOLATIONS = [
"src/bitcoin-tx.cpp.*stoul",
"src/dbwrapper.cpp:.*vsnprintf",
"src/test/dbwrapper_tests.cpp:.*snprintf",
"src/test/fuzz/locale.cpp",
"src/test/fuzz/string.cpp",
"src/util/strencodings.cpp:.*strtoll",
"src/util/system.cpp:.*fprintf"
]
REGEXP_EXTERNAL_DEPENDENCIES_EXCLUSIONS = [
"src/crypto/ctaes/",
"src/leveldb/",
"src/secp256k1/",
"src/minisketch/",
"src/tinyformat.h",
"src/univalue/",
"src/dashbls/",
"src/immer/"
]
LOCALE_DEPENDENT_FUNCTIONS = [
"alphasort", # LC_COLLATE (via strcoll)
"asctime", # LC_TIME (directly)
"asprintf", # (via vasprintf)
"atof", # LC_NUMERIC (via strtod)
"atoi", # LC_NUMERIC (via strtol)
"atol", # LC_NUMERIC (via strtol)
"atoll", # (via strtoll)
"atoq",
"btowc", # LC_CTYPE (directly)
"ctime", # (via asctime or localtime)
"dprintf", # (via vdprintf)
"fgetwc",
"fgetws",
"fold_case", # boost::locale::fold_case
"fprintf", # (via vfprintf)
"fputwc",
"fputws",
"fscanf", # (via __vfscanf)
"fwprintf", # (via __vfwprintf)
"getdate", # via __getdate_r => isspace // __localtime_r
"getwc",
"getwchar",
"is_digit", # boost::algorithm::is_digit
"is_space", # boost::algorithm::is_space
"isalnum", # LC_CTYPE
"isalpha", # LC_CTYPE
"isblank", # LC_CTYPE
"iscntrl", # LC_CTYPE
"isctype", # LC_CTYPE
"isdigit", # LC_CTYPE
"isgraph", # LC_CTYPE
"islower", # LC_CTYPE
"isprint", # LC_CTYPE
"ispunct", # LC_CTYPE
"isspace", # LC_CTYPE
"isupper", # LC_CTYPE
"iswalnum", # LC_CTYPE
"iswalpha", # LC_CTYPE
"iswblank", # LC_CTYPE
"iswcntrl", # LC_CTYPE
"iswctype", # LC_CTYPE
"iswdigit", # LC_CTYPE
"iswgraph", # LC_CTYPE
"iswlower", # LC_CTYPE
"iswprint", # LC_CTYPE
"iswpunct", # LC_CTYPE
"iswspace", # LC_CTYPE
"iswupper", # LC_CTYPE
"iswxdigit", # LC_CTYPE
"isxdigit", # LC_CTYPE
"localeconv", # LC_NUMERIC + LC_MONETARY
"mblen", # LC_CTYPE
"mbrlen",
"mbrtowc",
"mbsinit",
"mbsnrtowcs",
"mbsrtowcs",
"mbstowcs", # LC_CTYPE
"mbtowc", # LC_CTYPE
"mktime",
"normalize", # boost::locale::normalize
"printf", # LC_NUMERIC
"putwc",
"putwchar",
"scanf", # LC_NUMERIC
"setlocale",
"snprintf",
"sprintf",
"sscanf",
"std::locale::global",
"std::to_string",
"stod",
"stof",
"stoi",
"stol",
"stold",
"stoll",
"stoul",
"stoull",
"strcasecmp",
"strcasestr",
"strcoll", # LC_COLLATE
#"strerror",
"strfmon",
"strftime", # LC_TIME
"strncasecmp",
"strptime",
"strtod", # LC_NUMERIC
"strtof",
"strtoimax",
"strtol", # LC_NUMERIC
"strtold",
"strtoll",
"strtoq",
"strtoul", # LC_NUMERIC
"strtoull",
"strtoumax",
"strtouq",
"strxfrm", # LC_COLLATE
"swprintf",
"to_lower", # boost::locale::to_lower
"to_title", # boost::locale::to_title
"to_upper", # boost::locale::to_upper
"tolower", # LC_CTYPE
"toupper", # LC_CTYPE
"towctrans",
"towlower", # LC_CTYPE
"towupper", # LC_CTYPE
"trim", # boost::algorithm::trim
"trim_left", # boost::algorithm::trim_left
"trim_right", # boost::algorithm::trim_right
"ungetwc",
"vasprintf",
"vdprintf",
"versionsort",
"vfprintf",
"vfscanf",
"vfwprintf",
"vprintf",
"vscanf",
"vsnprintf",
"vsprintf",
"vsscanf",
"vswprintf",
"vwprintf",
"wcrtomb",
"wcscasecmp",
"wcscoll", # LC_COLLATE
"wcsftime", # LC_TIME
"wcsncasecmp",
"wcsnrtombs",
"wcsrtombs",
"wcstod", # LC_NUMERIC
"wcstof",
"wcstoimax",
"wcstol", # LC_NUMERIC
"wcstold",
"wcstoll",
"wcstombs", # LC_CTYPE
"wcstoul", # LC_NUMERIC
"wcstoull",
"wcstoumax",
"wcswidth",
"wcsxfrm", # LC_COLLATE
"wctob",
"wctomb", # LC_CTYPE
"wctrans",
"wctype",
"wcwidth",
"wprintf"
]
def find_locale_dependent_function_uses():
regexp_locale_dependent_functions = "|".join(LOCALE_DEPENDENT_FUNCTIONS)
exclude_args = [":(exclude)" + excl for excl in REGEXP_EXTERNAL_DEPENDENCIES_EXCLUSIONS]
git_grep_command = ["git", "grep", "-E", "[^a-zA-Z0-9_\\`'\"<>](" + regexp_locale_dependent_functions + "(_r|_s)?)[^a-zA-Z0-9_\\`'\"<>]", "--", "*.cpp", "*.h"] + exclude_args
git_grep_output = list()
try:
git_grep_output = check_output(git_grep_command, universal_newlines=True, encoding="utf8").splitlines()
except CalledProcessError as e:
if e.returncode > 1:
raise e
return git_grep_output
def main():
exit_code = 0
regexp_ignore_known_violations = "|".join(KNOWN_VIOLATIONS)
git_grep_output = find_locale_dependent_function_uses()
for locale_dependent_function in LOCALE_DEPENDENT_FUNCTIONS:
matches = [line for line in git_grep_output
if re.search("[^a-zA-Z0-9_\\`'\"<>]" + locale_dependent_function + "(_r|_s)?[^a-zA-Z0-9_\\`'\"<>]", line)
and not re.search("\\.(c|cpp|h):\\s*(//|\\*|/\\*|\").*" + locale_dependent_function, line)
and not re.search(regexp_ignore_known_violations, line)]
if matches:
print(f"The locale dependent function {locale_dependent_function}(...) appears to be used:")
for match in matches:
print(match)
print("")
exit_code = 1
if exit_code == 1:
print("Unnecessary locale depedence can cause bugs that are very tricky to isolate and fix. Please avoid using locale dependent functions if possible.\n")
print(f"Advice not applicable in this specific case? Add an exception by updating the ignore list in {sys.argv[0]}")
sys.exit(exit_code)
if __name__ == "__main__":
main()

246
test/lint/lint-locale-dependence.sh
View File

@ -1,246 +0,0 @@
#!/usr/bin/env bash
# Copyright (c) 2018-2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
export LC_ALL=C
# TODO: Reduce KNOWN_VIOLATIONS by replacing uses of locale dependent stoul/strtol with locale
# independent ToIntegral<T>(...).
# TODO: Reduce KNOWN_VIOLATIONS by replacing uses of locale dependent snprintf with strprintf.
# Be aware that bitcoind and bitcoin-qt differ in terms of localization: Qt
# opts in to POSIX localization by running setlocale(LC_ALL, "") on startup,
# whereas no such call is made in bitcoind.
#
# Qt runs setlocale(LC_ALL, "") on initialization. This installs the locale
# specified by the user's LC_ALL (or LC_*) environment variable as the new
# C locale.
#
# In contrast, bitcoind does not opt in to localization -- no call to
# setlocale(LC_ALL, "") is made and the environment variables LC_* are
# thus ignored.
#
# This results in situations where bitcoind is guaranteed to be running
# with the classic locale ("C") whereas the locale of bitcoin-qt will vary
# depending on the user's environment variables.
#
# An example: Assuming the environment variable LC_ALL=de_DE then the
# call std::to_string(1.23) will return "1.230000" in bitcoind but
# "1,230000" in bitcoin-qt.
#
# From the Qt documentation:
# "On Unix/Linux Qt is configured to use the system locale settings by default.
# This can cause a conflict when using POSIX functions, for instance, when
# converting between data types such as floats and strings, since the notation
# may differ between locales. To get around this problem, call the POSIX function
# setlocale(LC_NUMERIC,"C") right after initializing QApplication, QGuiApplication
# or QCoreApplication to reset the locale that is used for number formatting to
# "C"-locale."
#
# See https://doc.qt.io/qt-5/qcoreapplication.html#locale-settings and
# https://stackoverflow.com/a/34878283 for more details.
KNOWN_VIOLATIONS=(
"src/bitcoin-tx.cpp.*stoul"
"src/dbwrapper.cpp:.*vsnprintf"
"src/test/dbwrapper_tests.cpp:.*snprintf"
"src/test/fuzz/locale.cpp"
"src/test/fuzz/string.cpp"
"src/util/strencodings.cpp:.*strtoll"
"src/util/system.cpp:.*fprintf"
)
REGEXP_IGNORE_EXTERNAL_DEPENDENCIES="^src/(dashbls/|immer/|crypto/ctaes/|leveldb/|secp256k1/|minisketch/|tinyformat.h|univalue/)"
LOCALE_DEPENDENT_FUNCTIONS=(
alphasort # LC_COLLATE (via strcoll)
asctime # LC_TIME (directly)
asprintf # (via vasprintf)
atof # LC_NUMERIC (via strtod)
atoi # LC_NUMERIC (via strtol)
atol # LC_NUMERIC (via strtol)
atoll # (via strtoll)
atoq
btowc # LC_CTYPE (directly)
ctime # (via asctime or localtime)
dprintf # (via vdprintf)
fgetwc
fgetws
fold_case # boost::locale::fold_case
fprintf # (via vfprintf)
fputwc
fputws
fscanf # (via __vfscanf)
fwprintf # (via __vfwprintf)
getdate # via __getdate_r => isspace // __localtime_r
getwc
getwchar
is_digit # boost::algorithm::is_digit
is_space # boost::algorithm::is_space
isalnum # LC_CTYPE
isalpha # LC_CTYPE
isblank # LC_CTYPE
iscntrl # LC_CTYPE
isctype # LC_CTYPE
isdigit # LC_CTYPE
isgraph # LC_CTYPE
islower # LC_CTYPE
isprint # LC_CTYPE
ispunct # LC_CTYPE
isspace # LC_CTYPE
isupper # LC_CTYPE
iswalnum # LC_CTYPE
iswalpha # LC_CTYPE
iswblank # LC_CTYPE
iswcntrl # LC_CTYPE
iswctype # LC_CTYPE
iswdigit # LC_CTYPE
iswgraph # LC_CTYPE
iswlower # LC_CTYPE
iswprint # LC_CTYPE
iswpunct # LC_CTYPE
iswspace # LC_CTYPE
iswupper # LC_CTYPE
iswxdigit # LC_CTYPE
isxdigit # LC_CTYPE
localeconv # LC_NUMERIC + LC_MONETARY
mblen # LC_CTYPE
mbrlen
mbrtowc
mbsinit
mbsnrtowcs
mbsrtowcs
mbstowcs # LC_CTYPE
mbtowc # LC_CTYPE
mktime
normalize # boost::locale::normalize
printf # LC_NUMERIC
putwc
putwchar
scanf # LC_NUMERIC
setlocale
snprintf
sprintf
sscanf
std::locale::global
std::to_string
stod
stof
stoi
stol
stold
stoll
stoul
stoull
strcasecmp
strcasestr
strcoll # LC_COLLATE
# strerror
strfmon
strftime # LC_TIME
strncasecmp
strptime
strtod # LC_NUMERIC
strtof
strtoimax
strtol # LC_NUMERIC
strtold
strtoll
strtoq
strtoul # LC_NUMERIC
strtoull
strtoumax
strtouq
strxfrm # LC_COLLATE
swprintf
to_lower # boost::locale::to_lower
to_title # boost::locale::to_title
to_upper # boost::locale::to_upper
tolower # LC_CTYPE
toupper # LC_CTYPE
towctrans
towlower # LC_CTYPE
towupper # LC_CTYPE
trim # boost::algorithm::trim
trim_left # boost::algorithm::trim_left
trim_right # boost::algorithm::trim_right
ungetwc
vasprintf
vdprintf
versionsort
vfprintf
vfscanf
vfwprintf
vprintf
vscanf
vsnprintf
vsprintf
vsscanf
vswprintf
vwprintf
wcrtomb
wcscasecmp
wcscoll # LC_COLLATE
wcsftime # LC_TIME
wcsncasecmp
wcsnrtombs
wcsrtombs
wcstod # LC_NUMERIC
wcstof
wcstoimax
wcstol # LC_NUMERIC
wcstold
wcstoll
wcstombs # LC_CTYPE
wcstoul # LC_NUMERIC
wcstoull
wcstoumax
wcswidth
wcsxfrm # LC_COLLATE
wctob
wctomb # LC_CTYPE
wctrans
wctype
wcwidth
wprintf
)
function join_array {
local IFS="$1"
shift
echo "$*"
}
REGEXP_IGNORE_KNOWN_VIOLATIONS=$(join_array "|" "${KNOWN_VIOLATIONS[@]}")
# Invoke "git grep" only once in order to minimize run-time
REGEXP_LOCALE_DEPENDENT_FUNCTIONS=$(join_array "|" "${LOCALE_DEPENDENT_FUNCTIONS[@]}")
GIT_GREP_OUTPUT=$(git grep -E "[^a-zA-Z0-9_\`'\"<>](${REGEXP_LOCALE_DEPENDENT_FUNCTIONS}(_r|_s)?)[^a-zA-Z0-9_\`'\"<>]" -- "*.cpp" "*.h")
EXIT_CODE=0
for LOCALE_DEPENDENT_FUNCTION in "${LOCALE_DEPENDENT_FUNCTIONS[@]}"; do
MATCHES=$(grep -E "[^a-zA-Z0-9_\`'\"<>]${LOCALE_DEPENDENT_FUNCTION}(_r|_s)?[^a-zA-Z0-9_\`'\"<>]" <<< "${GIT_GREP_OUTPUT}" | \
grep -vE "\.(c|cpp|h):\s*(//|\*|/\*|\").*${LOCALE_DEPENDENT_FUNCTION}")
if [[ ${REGEXP_IGNORE_EXTERNAL_DEPENDENCIES} != "" ]]; then
MATCHES=$(grep -vE "${REGEXP_IGNORE_EXTERNAL_DEPENDENCIES}" <<< "${MATCHES}")
fi
if [[ ${REGEXP_IGNORE_KNOWN_VIOLATIONS} != "" ]]; then
MATCHES=$(grep -vE "${REGEXP_IGNORE_KNOWN_VIOLATIONS}" <<< "${MATCHES}")
fi
if [[ ${MATCHES} != "" ]]; then
echo "The locale dependent function ${LOCALE_DEPENDENT_FUNCTION}(...) appears to be used:"
echo "${MATCHES}"
echo
EXIT_CODE=1
fi
done
if [[ ${EXIT_CODE} != 0 ]]; then
echo "Unnecessary locale dependence can cause bugs that are very"
echo "tricky to isolate and fix. Please avoid using locale dependent"
echo "functions if possible."
echo
echo "Advice not applicable in this specific case? Add an exception"
echo "by updating the ignore list in $0"
fi
exit ${EXIT_CODE}

34
test/lint/lint-logs.py Executable file
View File

@ -0,0 +1,34 @@
#!/usr/bin/env python3
#
# Copyright (c) 2018-2022 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Check that all logs are terminated with '\n'
#
# Some logs are continued over multiple lines. They should be explicitly
# commented with /* Continued */
import re
import sys
from subprocess import check_output
def main():
logs_list = check_output(["git", "grep", "--extended-regexp", r"(LogPrintLevel|LogPrintf?)\(", "--", "*.cpp"], universal_newlines=True, encoding="utf8").splitlines()
unterminated_logs = [line for line in logs_list if not re.search(r'(\\n"|/\* Continued \*/)', line)]
if unterminated_logs != []:
print("All calls to LogPrintf(), LogPrint(), LogPrintLevel(), and WalletLogPrintf() should be terminated with \"\\n\".")
print("")
for line in unterminated_logs:
print(line)
sys.exit(1)
if __name__ == "__main__":
main()

29
test/lint/lint-logs.sh
View File

@ -1,29 +0,0 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Check that all logs are terminated with '\n'
#
# Some logs are continued over multiple lines. They should be explicitly
# commented with /* Continued */
#
# There are some instances of LogPrintf() in comments. Those can be
# ignored
export LC_ALL=C
UNTERMINATED_LOGS=$(git grep --extended-regexp "(LogPrintLevel|LogPrintf?)\(" -- "*.cpp" | \
grep -v '\\n"' | \
grep -v '\.\.\.' | \
grep -v "/\* Continued \*/" | \
grep -v "LogPrint()" | \
grep -v "LogPrintf()")
if [[ ${UNTERMINATED_LOGS} != "" ]]; then
# shellcheck disable=SC2028
echo "All calls to LogPrintf(), LogPrint(), LogPrintLevel(), and WalletLogPrintf() should be terminated with \\n"
echo
echo "${UNTERMINATED_LOGS}"
exit 1
fi

41
test/lint/lint-python-dead-code.py Executable file
View File

@ -0,0 +1,41 @@
#!/usr/bin/env python3
#
# Copyright (c) 2022 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""
Find dead Python code.
"""
from subprocess import check_output, STDOUT, CalledProcessError
FILES_ARGS = ['git', 'ls-files', '--', '*.py']
def check_vulture_install():
try:
check_output(["vulture", "--version"])
except FileNotFoundError:
print("Skipping Python dead code linting since vulture is not installed. Install by running \"pip3 install vulture\"")
exit(0)
def main():
check_vulture_install()
files = check_output(FILES_ARGS).decode("utf-8").splitlines()
# --min-confidence 100 will only report code that is guaranteed to be unused within the analyzed files.
# Any value below 100 introduces the risk of false positives, which would create an unacceptable maintenance burden.
vulture_args = ['vulture', '--min-confidence=100'] + files
try:
check_output(vulture_args, stderr=STDOUT)
except CalledProcessError as e:
print(e.output.decode("utf-8"), end="")
print("Python dead code detection found some issues")
exit(1)
if __name__ == "__main__":
main()

23
test/lint/lint-python-dead-code.sh
View File

@ -1,23 +0,0 @@
#!/usr/bin/env bash
#
# Copyright (c) 2021 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Find dead Python code.
export LC_ALL=C
if ! command -v vulture > /dev/null; then
echo "Skipping Python dead code linting since vulture is not installed. Install by running \"pip3 install vulture\""
exit 0
fi
# --min-confidence 100 will only report code that is guaranteed to be unused within the analyzed files.
# Any value below 100 introduces the risk of false positives, which would create an unacceptable maintenance burden.
if ! vulture \
--min-confidence 100 \
$(git ls-files -- "*.py"); then
echo "Python dead code detection found some issues"
exit 1
fi

4
test/lint/lint-python.sh
View File

@ -101,6 +101,7 @@ fi
EXIT_CODE=0
# shellcheck disable=SC2046
if ! PYTHONWARNINGS="ignore" $FLAKECMD --ignore=B,C,E,F,I,N,W --select=$(IFS=","; echo "${enabled[*]}") $(
if [[ $# == 0 ]]; then
git ls-files "*.py" | grep -vE "src/(immer)/"
@ -111,7 +112,8 @@ if ! PYTHONWARNINGS="ignore" $FLAKECMD --ignore=B,C,E,F,I,N,W --select=$(IFS=","
EXIT_CODE=1
fi
if ! mypy --ignore-missing-imports --show-error-codes $(git ls-files "test/functional/*.py" "contrib/devtools/*.py" | grep -v contrib/devtools/github-merge.py) ; then
mapfile -t FILES < <(git ls-files "test/functional/*.py" "contrib/devtools/*.py" | grep -v contrib/devtools/github-merge.py)
if ! mypy --ignore-missing-imports --show-error-codes "${FILES[@]}"; then
EXIT_CODE=1
fi

6
test/lint/lint-shell.sh
View File

@ -29,8 +29,10 @@ fi
SHELLCHECK_CMD=(shellcheck --external-sources --check-sourced)
EXCLUDE="--exclude=$(IFS=','; echo "${disabled[*]}")"
SOURCED_FILES=$(git ls-files | xargs gawk '/^# shellcheck shell=/ {print FILENAME} {nextfile}') # Check shellcheck directive used for sourced files
if ! "${SHELLCHECK_CMD[@]}" "$EXCLUDE" $SOURCED_FILES $(git ls-files -- '*.sh' | grep -vE 'src/(dashbls|immer|leveldb|secp256k1|minisketch|univalue)/'); then
# Check shellcheck directive used for sourced files
mapfile -t SOURCED_FILES < <(git ls-files | xargs gawk '/^# shellcheck shell=/ {print FILENAME} {nextfile}')
mapfile -t FILES < <(git ls-files -- '*.sh' | grep -vE 'src/(dashbls|immer|leveldb|secp256k1|minisketch|univalue)/')
if ! "${SHELLCHECK_CMD[@]}" "$EXCLUDE" "${SOURCED_FILES[@]}" "${FILES[@]}"; then
EXIT_CODE=1
fi

40
test/lint/lint-spelling.py Executable file
View File

@ -0,0 +1,40 @@
#!/usr/bin/env python3
#
# Copyright (c) 2022 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
"""
Warn in case of spelling errors.
Note: Will exit successfully regardless of spelling errors.
"""
from subprocess import check_output, STDOUT, CalledProcessError
IGNORE_WORDS_FILE = 'test/lint/spelling.ignore-words.txt'
FILES_ARGS = ['git', 'ls-files', '--', ":(exclude)build-aux/m4/", ":(exclude)contrib/seeds/*.txt", ":(exclude)depends/", ":(exclude)doc/release-notes/", ":(exclude)src/bip39_english.h", ":(exclude)src/dashbls/", ":(exclude)src/crc32c/", ":(exclude)src/crypto/", ":(exclude)src/ctpl_stl.h", ":(exclude)src/cxxtimer.hpp", ":(exclude)src/util/expected.h", ":(exclude)src/immer/", ":(exclude)src/leveldb/", ":(exclude)src/qt/locale/", ":(exclude)src/qt/*.qrc", ":(exclude)src/secp256k1/", ":(exclude)src/minisketch/", ":(exclude)src/univalue/", ":(exclude)contrib/builder-keys/", ":(exclude)contrib/guix/patches"]
def check_codespell_install():
try:
check_output(["codespell", "--version"])
except FileNotFoundError:
print("Skipping spell check linting since codespell is not installed.")
exit(0)
def main():
check_codespell_install()
files = check_output(FILES_ARGS).decode("utf-8").splitlines()
codespell_args = ['codespell', '--check-filenames', '--disable-colors', '--quiet-level=7', '--ignore-words={}'.format(IGNORE_WORDS_FILE)] + files
try:
check_output(codespell_args, stderr=STDOUT)
except CalledProcessError as e:
print(e.output.decode("utf-8"), end="")
print('^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in {}'.format(IGNORE_WORDS_FILE))
if __name__ == "__main__":
main()

20
test/lint/lint-spelling.sh
View File

@ -1,20 +0,0 @@
#!/usr/bin/env bash
#
# Copyright (c) 2018-2019 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Warn in case of spelling errors.
# Note: Will exit successfully regardless of spelling errors.
export LC_ALL=C
if ! command -v codespell > /dev/null; then
echo "Skipping spell check linting since codespell is not installed."
exit 0
fi
IGNORE_WORDS_FILE="test/lint/spelling.ignore-words.txt"
if ! codespell --check-filenames --disable-colors --quiet-level=7 --ignore-words=${IGNORE_WORDS_FILE} $(git ls-files -- ":(exclude)build-aux/m4/" ":(exclude)contrib/seeds/*.txt" ":(exclude)depends/" ":(exclude)doc/release-notes/" ":(exclude)src/bip39_english.h" ":(exclude)src/dashbls/" ":(exclude)src/crc32c/" ":(exclude)src/crypto/" ":(exclude)src/ctpl_stl.h" ":(exclude)src/cxxtimer.hpp" ":(exclude)src/util/expected.h" ":(exclude)src/immer/" ":(exclude)src/leveldb/" ":(exclude)src/qt/locale/" ":(exclude)src/qt/*.qrc" ":(exclude)src/secp256k1/" ":(exclude)src/minisketch/" ":(exclude)src/univalue/" ":(exclude)contrib/builder-keys/" ":(exclude)contrib/guix/patches"); then
echo "^ Warning: codespell identified likely spelling errors. Any false positives? Add them to the list of ignored words in ${IGNORE_WORDS_FILE}"
fi

138
test/lint/lint-whitespace.py Executable file
View File

@ -0,0 +1,138 @@
#!/usr/bin/env python3
#
# Copyright (c) 2017-2022 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Check for new lines in diff that introduce trailing whitespace or
# tab characters instead of spaces.
# We can't run this check unless we know the commit range for the PR.
import argparse
import os
import re
import sys
from subprocess import check_output
EXCLUDED_DIRS = ["depends/patches/",
"contrib/guix/patches/",
"src/leveldb/",
"src/crc32c/",
"src/secp256k1/",
"src/minisketch/",
"src/univalue/",
"src/dashbls/",
"src/immer/",
"src/util/expected.h",
"doc/release-notes/",
"src/qt/locale"]
def parse_args():
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="""
Check for new lines in diff that introduce trailing whitespace
or tab characters instead of spaces in unstaged changes, the
previous n commits, or a commit-range.
""",
epilog=f"""
You can manually set the commit-range with the COMMIT_RANGE
environment variable (e.g. "COMMIT_RANGE='47ba2c3...ee50c9e'
{sys.argv[0]}"). Defaults to current merge base when neither
prev-commits nor the environment variable is set.
""")
parser.add_argument("--prev-commits", "-p", required=False, help="The previous n commits to check")
return parser.parse_args()
def report_diff(selection):
filename = ""
seen = False
seenln = False
print("The following changes were suspected:")
for line in selection:
if re.match(r"^diff", line):
filename = line
seen = False
elif re.match(r"^@@", line):
linenumber = line
seenln = False
else:
if not seen:
# The first time a file is seen with trailing whitespace or a tab character, we print the
# filename (preceded by a newline).
print("")
print(filename)
seen = True
if not seenln:
print(linenumber)
seenln = True
print(line)
def get_diff(commit_range, check_only_code):
exclude_args = [":(exclude)" + dir for dir in EXCLUDED_DIRS]
if check_only_code:
what_files = ["*.cpp", "*.h", "*.md", "*.py", "*.sh"]
else:
what_files = ["."]
diff = check_output(["git", "diff", "-U0", commit_range, "--"] + what_files + exclude_args, universal_newlines=True, encoding="utf8")
return diff
def main():
args = parse_args()
if not os.getenv("COMMIT_RANGE"):
if args.prev_commits:
commit_range = "HEAD~" + args.prev_commits + "...HEAD"
else:
# This assumes that the target branch of the pull request will be develop.
merge_base = check_output(["git", "merge-base", "HEAD", "develop"], universal_newlines=True, encoding="utf8").rstrip("\n")
commit_range = merge_base + "..HEAD"
else:
commit_range = os.getenv("COMMIT_RANGE")
whitespace_selection = []
tab_selection = []
# Check if trailing whitespace was found in the diff.
for line in get_diff(commit_range, check_only_code=False).splitlines():
if re.match(r"^(diff --git|\@@|^\+.*\s+$)", line):
whitespace_selection.append(line)
whitespace_additions = [i for i in whitespace_selection if i.startswith("+")]
# Check if tab characters were found in the diff.
for line in get_diff(commit_range, check_only_code=True).splitlines():
if re.match(r"^(diff --git|\@@|^\+.*\t)", line):
tab_selection.append(line)
tab_additions = [i for i in tab_selection if i.startswith("+")]
ret = 0
if len(whitespace_additions) > 0:
print("This diff appears to have added new lines with trailing whitespace.")
report_diff(whitespace_selection)
ret = 1
if len(tab_additions) > 0:
print("This diff appears to have added new lines with tab characters instead of spaces.")
report_diff(tab_selection)
ret = 1
sys.exit(ret)
if __name__ == "__main__":
main()

115
test/lint/lint-whitespace.sh
View File

@ -1,115 +0,0 @@
#!/usr/bin/env bash
#
# Copyright (c) 2017-2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# Check for new lines in diff that introduce trailing whitespace.
# We can't run this check unless we know the commit range for the PR.
export LC_ALL=C
while getopts "?" opt; do
case $opt in
?)
echo "Usage: $0 [N]"
echo " COMMIT_RANGE='<commit range>' $0"
echo " $0 -?"
echo "Checks unstaged changes, the previous N commits, or a commit range."
echo "COMMIT_RANGE='47ba2c3...ee50c9e' $0"
exit 0
;;
esac
done
if [ -z "${COMMIT_RANGE}" ]; then
if [ -n "$1" ]; then
COMMIT_RANGE="HEAD~$1...HEAD"
else
# This assumes that the target branch of the pull request will be develop.
MERGE_BASE=$(git merge-base HEAD develop)
COMMIT_RANGE="$MERGE_BASE..HEAD"
fi
fi
showdiff() {
if ! git diff -U0 "${COMMIT_RANGE}" -- "." ":(exclude)depends/patches/" ":(exclude)contrib/guix/patches/" ":(exclude)src/dashbls/" ":(exclude)src/util/expected.h" ":(exclude)src/immer/" ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/secp256k1/" ":(exclude)src/minisketch/" ":(exclude)src/univalue/" ":(exclude)doc/release-notes/"; then
echo "Failed to get a diff"
exit 1
fi
}
showcodediff() {
if ! git diff -U0 "${COMMIT_RANGE}" -- *.cpp *.h *.md *.py *.sh ":(exclude)src/dashbls/" ":(exclude)src/util/expected.h" ":(exclude)src/immer/" ":(exclude)src/leveldb/" ":(exclude)src/crc32c/" ":(exclude)src/secp256k1/" ":(exclude)src/minisketch/" ":(exclude)src/univalue/" ":(exclude)doc/release-notes/"; then
echo "Failed to get a diff"
exit 1
fi
}
RET=0
# Check if trailing whitespace was found in the diff.
if showdiff | grep -E -q '^\+.*\s+$'; then
echo "This diff appears to have added new lines with trailing whitespace."
echo "The following changes were suspected:"
FILENAME=""
SEEN=0
SEENLN=0
while read -r line; do
if [[ "$line" =~ ^diff ]]; then
FILENAME="$line"
SEEN=0
elif [[ "$line" =~ ^@@ ]]; then
LINENUMBER="$line"
SEENLN=0
else
if [ "$SEEN" -eq 0 ]; then
# The first time a file is seen with trailing whitespace, we print the
# filename (preceded by a newline).
echo
echo "$FILENAME"
SEEN=1
fi
if [ "$SEENLN" -eq 0 ]; then
echo "$LINENUMBER"
SEENLN=1
fi
echo "$line"
fi
done < <(showdiff | grep -E '^(diff --git |@@|\+.*\s+$)')
RET=1
fi
# Check if tab characters were found in the diff.
if showcodediff | perl -nle '$MATCH++ if m{^\+.*\t}; END{exit 1 unless $MATCH>0}' > /dev/null; then
echo "This diff appears to have added new lines with tab characters instead of spaces."
echo "The following changes were suspected:"
FILENAME=""
SEEN=0
SEENLN=0
while read -r line; do
if [[ "$line" =~ ^diff ]]; then
FILENAME="$line"
SEEN=0
elif [[ "$line" =~ ^@@ ]]; then
LINENUMBER="$line"
SEENLN=0
else
if [ "$SEEN" -eq 0 ]; then
# The first time a file is seen with a tab character, we print the
# filename (preceded by a newline).
echo
echo "$FILENAME"
SEEN=1
fi
if [ "$SEENLN" -eq 0 ]; then
echo "$LINENUMBER"
SEENLN=1
fi
echo "$line"
fi
done < <(showcodediff | perl -nle 'print if m{^(diff --git |@@|\+.*\t)}')
RET=1
fi
exit $RET

50
test/lint/run-lint-format-strings.py
View File

@ -11,8 +11,6 @@
import argparse
import re
import sys
from functools import partial
from multiprocessing import Pool
FALSE_POSITIVES = [
("src/batchedlogger.h", "strprintf(fmt, args...)"),
@ -265,28 +263,6 @@ def count_format_specifiers(format_string):
return n
def handle_filename(filename, args):
exit_code = 0
with open(filename, "r", encoding="utf-8") as f:
for function_call_str in parse_function_calls(args.function_name, f.read()):
parts = parse_function_call_and_arguments(args.function_name, function_call_str)
relevant_function_call_str = unescape("".join(parts))[:512]
if (f.name, relevant_function_call_str) in FALSE_POSITIVES:
continue
if len(parts) < 3 + args.skip_arguments:
exit_code = 1
print("{}: Could not parse function call string \"{}(...)\": {}".format(f.name, args.function_name, relevant_function_call_str))
continue
argument_count = len(parts) - 3 - args.skip_arguments
format_str = parse_string_content(parts[1 + args.skip_arguments])
format_specifier_count = count_format_specifiers(format_str)
if format_specifier_count != argument_count:
exit_code = 1
print("{}: Expected {} argument(s) after format string but found {} argument(s): {}".format(f.name, format_specifier_count, argument_count, relevant_function_call_str))
continue
return exit_code
def main():
parser = argparse.ArgumentParser(description="This program checks that the number of arguments passed "
"to a variadic format string function matches the number of format "
@ -296,12 +272,26 @@ def main():
parser.add_argument("function_name", help="function name (e.g. fprintf)", default=None)
parser.add_argument("file", nargs="*", help="C++ source code file (e.g. foo.cpp)")
args = parser.parse_args()
exit_codes = []
with Pool(8) as pool:
exit_codes = pool.map(partial(handle_filename, args=args), args.file)
sys.exit(max(exit_codes))
exit_code = 0
for filename in args.file:
with open(filename, "r", encoding="utf-8") as f:
for function_call_str in parse_function_calls(args.function_name, f.read()):
parts = parse_function_call_and_arguments(args.function_name, function_call_str)
relevant_function_call_str = unescape("".join(parts))[:512]
if (f.name, relevant_function_call_str) in FALSE_POSITIVES:
continue
if len(parts) < 3 + args.skip_arguments:
exit_code = 1
print("{}: Could not parse function call string \"{}(...)\": {}".format(f.name, args.function_name, relevant_function_call_str))
continue
argument_count = len(parts) - 3 - args.skip_arguments
format_str = parse_string_content(parts[1 + args.skip_arguments])
format_specifier_count = count_format_specifiers(format_str)
if format_specifier_count != argument_count:
exit_code = 1
print("{}: Expected {} argument(s) after format string but found {} argument(s): {}".format(f.name, format_specifier_count, argument_count, relevant_function_call_str))
continue
sys.exit(exit_code)
if __name__ == "__main__":

0
test/util/bitcoin-util-test.py → test/util/test_runner.py
View File