Merge #6171: feat: implement GitHub Actions based starter CI

acf1315270 Merge bitcoin/bitcoin#25091: test: Remove extended lint (cppcheck) (MacroFake)
4dbdecdd1e refactor: rename builder-image -> build-image and builder as image name to dashcore-ci-runner (pasta)
ed8ffa7fb4 feat: have cppcheck linter respect CACHE_DIR env variable (pasta)
d1addb27aa fix: change fallback download path to be an s3 link which includes a few packages (pasta)
35c76705d1 feat: implement basic Github Actions based CI, which reuses underlying logic from GitLab CI (pasta)

Pull request description:

  ## Issue being fixed or feature implemented
  We currently rely on GitLab for running CI, and while it has worked quite well, I am worried about having all of our eggs in one basket! As such, I've long wanted to explore implemeenting Github Actions based CI, but was too lazy! Well I finally spent some 60 commits trying to figure everything out, and this PR is the result of it.

  As a result, we will now have two semi-redunant CI systems, the primary one on GitLab, and this one on Github Actions. Currently the GA based CI will only do one host, and does linting. Be aware this GA based CI does not actually run the tests, it does build depends, src and run linters on 1 host.
  In the future, we should expand it from simply arm32 builds to having feature parity to GitLab.

  While it appears the GA default runners are a bit slower than what we have on GitLab, there's a big difference, the GA runners are free :D If we decide to make the GA based CI primary, we'll probably want to setup some custom runners to have improved build speeds. Even still, a heavily cached build doing all linters took around 5 minutes if I recall correctly. Without caches I think it took maybe an hour, so defenitely not bad.

  ## What was done?
  See the individual commits, they're pretty self explanatory

  ## How Has This Been Tested?
  Lots of CI runs on my prior branch :) CI should run on this PR, and we should see how long it'll take w/o cache :D

  ## Breaking Changes
  N/A - CI only

  ## Checklist:
    _Go over all the following points, and put an `x` in all the boxes that apply._
  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation
  - [  ] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

Top commit has no ACKs.

Tree-SHA512: fef41d1b73fdeac29e5096ffa7fa26660efc44ac274376d5880b5fabf6b0e760aeda4a4ae9c24656ee0e3adb760459f0d45b955cefffc9d0eeb5384afbc9d473

.github/workflows/build.yml

@@ -0,0 +1,176 @@
+name: CI
+
+on:
+  push:
+  pull_request_target:
+
+permissions:
+  contents: read
+  packages: write
+
+env:
+  DOCKER_DRIVER: overlay2
+  FAST_MODE: false
+
+jobs:
+  build-image:
+    name: Build Image
+    runs-on: ubuntu-20.04
+    outputs:
+      image-tag: ${{ steps.prepare.outputs.image-tag }}
+      repo-name: ${{ steps.prepare.outputs.repo-name }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Prepare
+        id: prepare
+        run: |
+          BRANCH_NAME=$(echo "${GITHUB_REF##*/}" | tr '[:upper:]' '[:lower:]')
+          REPO_NAME=$(echo "${{ github.repository }}" | tr '[:upper:]' '[:lower:]')
+          echo "::set-output name=image-tag::${BRANCH_NAME}"
+          echo "::set-output name=repo-name::${REPO_NAME}"
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v6
+        with:
+          context: ./contrib/containers/ci
+          file: ./contrib/containers/ci/Dockerfile
+          push: true
+          tags: |
+            ghcr.io/${{ steps.prepare.outputs.repo-name }}/dashcore-ci-runner:${{ steps.prepare.outputs.image-tag }}
+            ghcr.io/${{ steps.prepare.outputs.repo-name }}/dashcore-ci-runner:latest
+          cache-from: type=registry,ref=ghcr.io/${{ steps.prepare.outputs.repo-name }}/dashcore-ci-runner:latest
+          cache-to: type=inline
+
+  build-depends:
+    name: Build Dependencies
+    needs: build-image
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        include:
+        - build_target: arm-linux
+          host: arm-linux-gnueabihf
+    container:
+      image: ghcr.io/${{ needs.build-image.outputs.repo-name }}/dashcore-ci-runner:${{ needs.build-image.outputs.image-tag }}
+      options: --user root
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Cache dependencies
+        uses: actions/cache@v4
+        with:
+          path: |
+            depends/built
+            depends/${{ matrix.host }}
+            depends/sdk-sources
+          # We don't care about no specific key as depends system will handle that for us
+          key: ${{ runner.os }}-depends-${{ matrix.host }}-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-depends-${{ matrix.host }}-${{ github.sha }}
+            ${{ runner.os }}-depends-${{ matrix.host }}
+            ${{ runner.os }}-depends
+
+      - name: Build dependencies
+        run: make -j$(nproc) -C depends HOST=${{ matrix.host }}
+
+  build:
+    name: Build
+    needs: [build-image, build-depends]
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        include:
+          - build_target: arm-linux
+            host: arm-linux-gnueabihf
+            dep_opts: DEBUG=1
+    container:
+      image: ghcr.io/${{ needs.build-image.outputs.repo-name }}/dashcore-ci-runner:${{ needs.build-image.outputs.image-tag }}
+      options: --user root
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Restore Cache dependencies
+        uses: actions/cache/restore@v4
+        with:
+          path: |
+            depends/built
+            depends/${{ matrix.host }}
+            depends/sdk-sources
+          # We don't care about no specific key as depends system will handle that for us
+          key: ${{ runner.os }}-depends-${{ matrix.host }}-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-depends-${{ matrix.host }}-${{ github.sha }}
+            ${{ runner.os }}-depends-${{ matrix.host }}
+            ${{ runner.os }}-depends
+
+      - name: CCache
+        uses: actions/cache@v4
+        with:
+          path: |
+            /cache
+          key: ${{ runner.os }}-${{ matrix.host }}-${{ github.sha }}
+          restore-keys: |
+            ${{ runner.os }}-${{ matrix.host }}-${{ github.sha }}
+            ${{ runner.os }}-${{ matrix.host }}
+            ${{ runner.os }}
+
+      - name: Build source and run tests
+        run: |
+          git config --global --add safe.directory "$PWD"
+          CCACHE_SIZE="400M"
+          CACHE_DIR="/cache"
+          mkdir /output
+          BASE_OUTDIR="/output"
+          BUILD_TARGET="${{ matrix.build_target }}"
+          source ./ci/dash/matrix.sh
+          ./ci/dash/build_src.sh
+          ./ci/dash/test_unittests.sh
+        shell: bash
+
+      - name: Upload build artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: build-artifacts
+          path: |
+            /output
+
+
+# Come back to this later and implement tests :)
+#  test:
+#    name: Test
+#    needs: [build-image, build]
+#    runs-on: ubuntu-20.04
+#    container:
+#      image: ghcr.io/${{ needs.build-image.outputs.repo-name }}/dashcore-ci-runner:${{ needs.build-image.outputs.image-tag }}
+#      options: --user root
+#    steps:
+#      - name: Checkout code
+#        uses: actions/checkout@v4
+#
+#      - name: Download build artifacts
+#        uses: actions/download-artifact@v4
+#        with:
+#          name: build-artifacts
+#          path: src/
+#
+##      - name: Setup environment
+##        run: |
+##          echo "BUILD_TARGET=${{ needs.build.matrix.build_target }}"
+##          source ./ci/dash/matrix.sh
+#
+#      - name: Run integration tests
+#        run: ./ci/dash/test_integrationtests.sh --extended --exclude feature_pruning,feature_dbcrash

ci/dash/build_src.sh

@@ -26,7 +26,6 @@ if [ "$CHECK_DOC" = 1 ]; then
     #test/lint/check-doc.py
     # Run all linters
     test/lint/lint-all.sh
-    test/lint/extended-lint-all.sh
 fi
 
 ccache --zero-stats --max-size=$CCACHE_SIZE

depends/Makefile

@@ -41,7 +41,7 @@ NO_ZMQ ?=
 NO_UPNP ?=
 NO_NATPMP ?=
 MULTIPROCESS ?=
-FALLBACK_DOWNLOAD_PATH ?= https://bitcoincore.org/depends-sources
+FALLBACK_DOWNLOAD_PATH ?= http://dash-depends-sources.s3-website-us-west-2.amazonaws.com
 
 BUILD = $(shell ./config.guess)
 HOST ?= $(BUILD)

test/lint/extended-lint-all.sh

@@ -1,26 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2019-2020 The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-#
-# This script runs all contrib/devtools/extended-lint-*.sh files, and fails if
-# any exit with a non-zero status code.
-
-# This script is intentionally locale dependent by not setting "export LC_ALL=C"
-# in order to allow for the executed lint scripts to opt in or opt out of locale
-# dependence themselves.
-
-set -u
-
-SCRIPTDIR=$(dirname "${BASH_SOURCE[0]}")
-LINTALL=$(basename "${BASH_SOURCE[0]}")
-
-for f in "${SCRIPTDIR}"/extended-lint-*.sh; do
-  if [ "$(basename "$f")" != "$LINTALL" ]; then
-    if ! "$f"; then
-      echo "^---- failure generated from $f"
-      exit 1
-    fi
-  fi
-done

test/lint/extended-lint-cppcheck.sh

@@ -1,86 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2019-2020 The Bitcoin Core developers
-# Distributed under the MIT software license, see the accompanying
-# file COPYING or http://www.opensource.org/licenses/mit-license.php.
-#
-
-export LC_ALL=C
-
-ENABLED_CHECKS=(
-    "Class '.*' has a constructor with 1 argument that is not explicit."
-    "Struct '.*' has a constructor with 1 argument that is not explicit."
-)
-
-IGNORED_WARNINGS=(
-    "src/arith_uint256.h:.* Class 'arith_uint256' has a constructor with 1 argument that is not explicit."
-    "src/arith_uint256.h:.* Class 'base_uint < 256 >' has a constructor with 1 argument that is not explicit."
-    "src/arith_uint256.h:.* Class 'base_uint' has a constructor with 1 argument that is not explicit."
-    "src/coins.h:.* Class 'CCoinsViewBacked' has a constructor with 1 argument that is not explicit."
-    "src/coins.h:.* Class 'CCoinsViewCache' has a constructor with 1 argument that is not explicit."
-    "src/coins.h:.* Class 'CCoinsViewCursor' has a constructor with 1 argument that is not explicit."
-    "src/net.h:.* Class 'CNetMessage' has a constructor with 1 argument that is not explicit."
-    "src/policy/feerate.h:.* Class 'CFeeRate' has a constructor with 1 argument that is not explicit."
-    "src/prevector.h:.* Class 'const_iterator' has a constructor with 1 argument that is not explicit."
-    "src/prevector.h:.* Class 'const_reverse_iterator' has a constructor with 1 argument that is not explicit."
-    "src/prevector.h:.* Class 'iterator' has a constructor with 1 argument that is not explicit."
-    "src/prevector.h:.* Class 'reverse_iterator' has a constructor with 1 argument that is not explicit."
-    "src/primitives/block.h:.* Class 'CBlock' has a constructor with 1 argument that is not explicit."
-    "src/primitives/transaction.h:.* Class 'CTransaction' has a constructor with 1 argument that is not explicit."
-    "src/protocol.h:.* Class 'CMessageHeader' has a constructor with 1 argument that is not explicit."
-    "src/qt/guiutil.h:.* Class 'ItemDelegate' has a constructor with 1 argument that is not explicit."
-    "src/rpc/util.h:.* Struct 'RPCResults' has a constructor with 1 argument that is not explicit."
-    "src/rpc/util.h:.* Struct 'UniValueType' has a constructor with 1 argument that is not explicit."
-    "src/rpc/util.h:.* style: Struct 'UniValueType' has a constructor with 1 argument that is not explicit."
-    "src/script/descriptor.cpp:.* Class 'AddressDescriptor' has a constructor with 1 argument that is not explicit."
-    "src/script/descriptor.cpp:.* Class 'ComboDescriptor' has a constructor with 1 argument that is not explicit."
-    "src/script/descriptor.cpp:.* Class 'ConstPubkeyProvider' has a constructor with 1 argument that is not explicit."
-    "src/script/descriptor.cpp:.* Class 'PKDescriptor' has a constructor with 1 argument that is not explicit."
-    "src/script/descriptor.cpp:.* Class 'PKHDescriptor' has a constructor with 1 argument that is not explicit."
-    "src/script/descriptor.cpp:.* Class 'RawDescriptor' has a constructor with 1 argument that is not explicit."
-    "src/script/descriptor.cpp:.* Class 'SHDescriptor' has a constructor with 1 argument that is not explicit."
-    "src/script/script.h:.* Class 'CScript' has a constructor with 1 argument that is not explicit."
-    "src/script/standard.h:.* Class 'CScriptID' has a constructor with 1 argument that is not explicit."
-    "src/span.h:.* Class 'Span < const CRPCCommand >' has a constructor with 1 argument that is not explicit."
-    "src/span.h:.* Class 'Span < const char >' has a constructor with 1 argument that is not explicit."
-    "src/span.h:.* Class 'Span < const std :: vector <unsigned char > >' has a constructor with 1 argument that is not explicit."
-    "src/span.h:.* Class 'Span < const uint8_t >' has a constructor with 1 argument that is not explicit."
-    "src/span.h:.* Class 'Span' has a constructor with 1 argument that is not explicit."
-    "src/support/allocators/secure.h:.* Struct 'secure_allocator < char >' has a constructor with 1 argument that is not explicit."
-    "src/support/allocators/secure.h:.* Struct 'secure_allocator < RNGState >' has a constructor with 1 argument that is not explicit."
-    "src/support/allocators/secure.h:.* Struct 'secure_allocator < unsigned char >' has a constructor with 1 argument that is not explicit."
-    "src/support/allocators/zeroafterfree.h:.* Struct 'zero_after_free_allocator < char >' has a constructor with 1 argument that is not explicit."
-    "src/test/checkqueue_tests.cpp:.* Struct 'FailingCheck' has a constructor with 1 argument that is not explicit."
-    "src/test/checkqueue_tests.cpp:.* Struct 'MemoryCheck' has a constructor with 1 argument that is not explicit."
-    "src/test/checkqueue_tests.cpp:.* Struct 'UniqueCheck' has a constructor with 1 argument that is not explicit."
-    "src/test/fuzz/util.h:.* Class 'FuzzedFileProvider' has a constructor with 1 argument that is not explicit."
-    "src/test/fuzz/util.h:.* Class 'FuzzedAutoFileProvider' has a constructor with 1 argument that is not explicit."
-    "src/wallet/db.h:.* Class 'BerkeleyEnvironment' has a constructor with 1 argument that is not explicit."
-)
-
-if ! command -v cppcheck > /dev/null; then
-    echo "Skipping cppcheck linting since cppcheck is not installed. Install by running \"apt install cppcheck\""
-    exit 0
-fi
-
-function join_array {
-    local IFS="$1"
-    shift
-    echo "$*"
-}
-
-ENABLED_CHECKS_REGEXP=$(join_array "|" "${ENABLED_CHECKS[@]}")
-IGNORED_WARNINGS_REGEXP=$(join_array "|" "${IGNORED_WARNINGS[@]}")
-WARNINGS=$(git ls-files -- "*.cpp" "*.h" ":(exclude)src/dashbls/" ":(exclude)src/util/expected.h" ":(exclude)src/leveldb/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" | \
-    xargs cppcheck --enable=all -j "$(getconf _NPROCESSORS_ONLN)" --language=c++ --std=c++17 --template=gcc -D__cplusplus -DCLIENT_VERSION_BUILD -DCLIENT_VERSION_IS_RELEASE -DCLIENT_VERSION_MAJOR -DCLIENT_VERSION_MINOR -DCOPYRIGHT_YEAR -DDEBUG -I src/ -q 2>&1 | sort -u | \
-    grep -E "${ENABLED_CHECKS_REGEXP}" | \
-    grep -vE "${IGNORED_WARNINGS_REGEXP}")
-if [[ ${WARNINGS} != "" ]]; then
-    echo "${WARNINGS}"
-    echo
-    echo "Advice not applicable in this specific case? Add an exception by updating"
-    echo "IGNORED_WARNINGS in $0"
-    # Uncomment to enforce the developer note policy "By default, declare single-argument constructors `explicit`"
-    # exit 1
-fi
-exit 0

test/lint/lint-cppcheck-dash.sh

@@ -80,10 +80,15 @@ ENABLED_CHECKS_REGEXP=$(join_array "|" "${ENABLED_CHECKS[@]}")
 IGNORED_WARNINGS_REGEXP=$(join_array "|" "${IGNORED_WARNINGS[@]}")
 FILES_REGEXP=$(join_array "|" "${FILES[@]}")
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+# Check if CACHE_DIR is set and non-empty, otherwise use default .cppcheck/ directory
+if [[ -n "$CACHE_DIR" ]]; then
+    CPPCHECK_DIR=$CACHE_DIR/cppcheck/
+else
     CPPCHECK_DIR=$SCRIPT_DIR/.cppcheck/
+fi
 if [ ! -d $CPPCHECK_DIR ]
 then
-    mkdir $CPPCHECK_DIR
+    mkdir -p $CPPCHECK_DIR
 fi
 WARNINGS=$(echo "${FILES}" | \
     xargs cppcheck --enable=all --inline-suppr --suppress=missingIncludeSystem --cppcheck-build-dir=$CPPCHECK_DIR -j "$(getconf _NPROCESSORS_ONLN)" --language=c++ --std=c++17 --template=gcc -D__cplusplus -DENABLE_WALLET -DCLIENT_VERSION_BUILD -DCLIENT_VERSION_IS_RELEASE -DCLIENT_VERSION_MAJOR -DCLIENT_VERSION_MINOR -DCOPYRIGHT_YEAR -DDEBUG -DUSE_EPOLL -DCHAR_BIT=8 -I src/ -q 2>&1 | sort -u | \