Merge #6171: feat: implement GitHub Actions based starter CI

acf1315270 Merge bitcoin/bitcoin#25091: test: Remove extended lint (cppcheck) (MacroFake)
4dbdecdd1e refactor: rename builder-image -> build-image and builder as image name to dashcore-ci-runner (pasta)
ed8ffa7fb4 feat: have cppcheck linter respect CACHE_DIR env variable (pasta)
d1addb27aa fix: change fallback download path to be an s3 link which includes a few packages (pasta)
35c76705d1 feat: implement basic Github Actions based CI, which reuses underlying logic from GitLab CI (pasta)

Pull request description:

  ## Issue being fixed or feature implemented
  We currently rely on GitLab for running CI, and while it has worked quite well, I am worried about having all of our eggs in one basket! As such, I've long wanted to explore implemeenting Github Actions based CI, but was too lazy! Well I finally spent some 60 commits trying to figure everything out, and this PR is the result of it.

  As a result, we will now have two semi-redunant CI systems, the primary one on GitLab, and this one on Github Actions. Currently the GA based CI will only do one host, and does linting. Be aware this GA based CI does not actually run the tests, it does build depends, src and run linters on 1 host.
  In the future, we should expand it from simply arm32 builds to having feature parity to GitLab.

  While it appears the GA default runners are a bit slower than what we have on GitLab, there's a big difference, the GA runners are free :D If we decide to make the GA based CI primary, we'll probably want to setup some custom runners to have improved build speeds. Even still, a heavily cached build doing all linters took around 5 minutes if I recall correctly. Without caches I think it took maybe an hour, so defenitely not bad.

  ## What was done?
  See the individual commits, they're pretty self explanatory

  ## How Has This Been Tested?
  Lots of CI runs on my prior branch :) CI should run on this PR, and we should see how long it'll take w/o cache :D

  ## Breaking Changes
  N/A - CI only

  ## Checklist:
    _Go over all the following points, and put an `x` in all the boxes that apply._
  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [x] I have made corresponding changes to the documentation
  - [  ] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

Top commit has no ACKs.

Tree-SHA512: fef41d1b73fdeac29e5096ffa7fa26660efc44ac274376d5880b5fabf6b0e760aeda4a4ae9c24656ee0e3adb760459f0d45b955cefffc9d0eeb5384afbc9d473
This commit is contained in:
pasta 2024-08-11 16:25:09 +07:00
commit de4e7e16b4
No known key found for this signature in database
GPG Key ID: 52527BEDABE87984
6 changed files with 184 additions and 116 deletions

176
.github/workflows/build.yml vendored Normal file
View File

@ -0,0 +1,176 @@
name: CI
on:
push:
pull_request_target:
permissions:
contents: read
packages: write
env:
DOCKER_DRIVER: overlay2
FAST_MODE: false
jobs:
build-image:
name: Build Image
runs-on: ubuntu-20.04
outputs:
image-tag: ${{ steps.prepare.outputs.image-tag }}
repo-name: ${{ steps.prepare.outputs.repo-name }}
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Prepare
id: prepare
run: |
BRANCH_NAME=$(echo "${GITHUB_REF##*/}" | tr '[:upper:]' '[:lower:]')
REPO_NAME=$(echo "${{ github.repository }}" | tr '[:upper:]' '[:lower:]')
echo "::set-output name=image-tag::${BRANCH_NAME}"
echo "::set-output name=repo-name::${REPO_NAME}"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push Docker image
uses: docker/build-push-action@v6
with:
context: ./contrib/containers/ci
file: ./contrib/containers/ci/Dockerfile
push: true
tags: |
ghcr.io/${{ steps.prepare.outputs.repo-name }}/dashcore-ci-runner:${{ steps.prepare.outputs.image-tag }}
ghcr.io/${{ steps.prepare.outputs.repo-name }}/dashcore-ci-runner:latest
cache-from: type=registry,ref=ghcr.io/${{ steps.prepare.outputs.repo-name }}/dashcore-ci-runner:latest
cache-to: type=inline
build-depends:
name: Build Dependencies
needs: build-image
runs-on: ubuntu-20.04
strategy:
matrix:
include:
- build_target: arm-linux
host: arm-linux-gnueabihf
container:
image: ghcr.io/${{ needs.build-image.outputs.repo-name }}/dashcore-ci-runner:${{ needs.build-image.outputs.image-tag }}
options: --user root
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Cache dependencies
uses: actions/cache@v4
with:
path: |
depends/built
depends/${{ matrix.host }}
depends/sdk-sources
# We don't care about no specific key as depends system will handle that for us
key: ${{ runner.os }}-depends-${{ matrix.host }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-depends-${{ matrix.host }}-${{ github.sha }}
${{ runner.os }}-depends-${{ matrix.host }}
${{ runner.os }}-depends
- name: Build dependencies
run: make -j$(nproc) -C depends HOST=${{ matrix.host }}
build:
name: Build
needs: [build-image, build-depends]
runs-on: ubuntu-20.04
strategy:
matrix:
include:
- build_target: arm-linux
host: arm-linux-gnueabihf
dep_opts: DEBUG=1
container:
image: ghcr.io/${{ needs.build-image.outputs.repo-name }}/dashcore-ci-runner:${{ needs.build-image.outputs.image-tag }}
options: --user root
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Restore Cache dependencies
uses: actions/cache/restore@v4
with:
path: |
depends/built
depends/${{ matrix.host }}
depends/sdk-sources
# We don't care about no specific key as depends system will handle that for us
key: ${{ runner.os }}-depends-${{ matrix.host }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-depends-${{ matrix.host }}-${{ github.sha }}
${{ runner.os }}-depends-${{ matrix.host }}
${{ runner.os }}-depends
- name: CCache
uses: actions/cache@v4
with:
path: |
/cache
key: ${{ runner.os }}-${{ matrix.host }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-${{ matrix.host }}-${{ github.sha }}
${{ runner.os }}-${{ matrix.host }}
${{ runner.os }}
- name: Build source and run tests
run: |
git config --global --add safe.directory "$PWD"
CCACHE_SIZE="400M"
CACHE_DIR="/cache"
mkdir /output
BASE_OUTDIR="/output"
BUILD_TARGET="${{ matrix.build_target }}"
source ./ci/dash/matrix.sh
./ci/dash/build_src.sh
./ci/dash/test_unittests.sh
shell: bash
- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: build-artifacts
path: |
/output
# Come back to this later and implement tests :)
# test:
# name: Test
# needs: [build-image, build]
# runs-on: ubuntu-20.04
# container:
# image: ghcr.io/${{ needs.build-image.outputs.repo-name }}/dashcore-ci-runner:${{ needs.build-image.outputs.image-tag }}
# options: --user root
# steps:
# - name: Checkout code
# uses: actions/checkout@v4
#
# - name: Download build artifacts
# uses: actions/download-artifact@v4
# with:
# name: build-artifacts
# path: src/
#
## - name: Setup environment
## run: |
## echo "BUILD_TARGET=${{ needs.build.matrix.build_target }}"
## source ./ci/dash/matrix.sh
#
# - name: Run integration tests
# run: ./ci/dash/test_integrationtests.sh --extended --exclude feature_pruning,feature_dbcrash

View File

@ -26,7 +26,6 @@ if [ "$CHECK_DOC" = 1 ]; then
#test/lint/check-doc.py
# Run all linters
test/lint/lint-all.sh
test/lint/extended-lint-all.sh
fi
ccache --zero-stats --max-size=$CCACHE_SIZE

View File

@ -41,7 +41,7 @@ NO_ZMQ ?=
NO_UPNP ?=
NO_NATPMP ?=
MULTIPROCESS ?=
FALLBACK_DOWNLOAD_PATH ?= https://bitcoincore.org/depends-sources
FALLBACK_DOWNLOAD_PATH ?= http://dash-depends-sources.s3-website-us-west-2.amazonaws.com
BUILD = $(shell ./config.guess)
HOST ?= $(BUILD)

View File

@ -1,26 +0,0 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019-2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
# This script runs all contrib/devtools/extended-lint-*.sh files, and fails if
# any exit with a non-zero status code.
# This script is intentionally locale dependent by not setting "export LC_ALL=C"
# in order to allow for the executed lint scripts to opt in or opt out of locale
# dependence themselves.
set -u
SCRIPTDIR=$(dirname "${BASH_SOURCE[0]}")
LINTALL=$(basename "${BASH_SOURCE[0]}")
for f in "${SCRIPTDIR}"/extended-lint-*.sh; do
if [ "$(basename "$f")" != "$LINTALL" ]; then
if ! "$f"; then
echo "^---- failure generated from $f"
exit 1
fi
fi
done

View File

@ -1,86 +0,0 @@
#!/usr/bin/env bash
#
# Copyright (c) 2019-2020 The Bitcoin Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
#
export LC_ALL=C
ENABLED_CHECKS=(
"Class '.*' has a constructor with 1 argument that is not explicit."
"Struct '.*' has a constructor with 1 argument that is not explicit."
)
IGNORED_WARNINGS=(
"src/arith_uint256.h:.* Class 'arith_uint256' has a constructor with 1 argument that is not explicit."
"src/arith_uint256.h:.* Class 'base_uint < 256 >' has a constructor with 1 argument that is not explicit."
"src/arith_uint256.h:.* Class 'base_uint' has a constructor with 1 argument that is not explicit."
"src/coins.h:.* Class 'CCoinsViewBacked' has a constructor with 1 argument that is not explicit."
"src/coins.h:.* Class 'CCoinsViewCache' has a constructor with 1 argument that is not explicit."
"src/coins.h:.* Class 'CCoinsViewCursor' has a constructor with 1 argument that is not explicit."
"src/net.h:.* Class 'CNetMessage' has a constructor with 1 argument that is not explicit."
"src/policy/feerate.h:.* Class 'CFeeRate' has a constructor with 1 argument that is not explicit."
"src/prevector.h:.* Class 'const_iterator' has a constructor with 1 argument that is not explicit."
"src/prevector.h:.* Class 'const_reverse_iterator' has a constructor with 1 argument that is not explicit."
"src/prevector.h:.* Class 'iterator' has a constructor with 1 argument that is not explicit."
"src/prevector.h:.* Class 'reverse_iterator' has a constructor with 1 argument that is not explicit."
"src/primitives/block.h:.* Class 'CBlock' has a constructor with 1 argument that is not explicit."
"src/primitives/transaction.h:.* Class 'CTransaction' has a constructor with 1 argument that is not explicit."
"src/protocol.h:.* Class 'CMessageHeader' has a constructor with 1 argument that is not explicit."
"src/qt/guiutil.h:.* Class 'ItemDelegate' has a constructor with 1 argument that is not explicit."
"src/rpc/util.h:.* Struct 'RPCResults' has a constructor with 1 argument that is not explicit."
"src/rpc/util.h:.* Struct 'UniValueType' has a constructor with 1 argument that is not explicit."
"src/rpc/util.h:.* style: Struct 'UniValueType' has a constructor with 1 argument that is not explicit."
"src/script/descriptor.cpp:.* Class 'AddressDescriptor' has a constructor with 1 argument that is not explicit."
"src/script/descriptor.cpp:.* Class 'ComboDescriptor' has a constructor with 1 argument that is not explicit."
"src/script/descriptor.cpp:.* Class 'ConstPubkeyProvider' has a constructor with 1 argument that is not explicit."
"src/script/descriptor.cpp:.* Class 'PKDescriptor' has a constructor with 1 argument that is not explicit."
"src/script/descriptor.cpp:.* Class 'PKHDescriptor' has a constructor with 1 argument that is not explicit."
"src/script/descriptor.cpp:.* Class 'RawDescriptor' has a constructor with 1 argument that is not explicit."
"src/script/descriptor.cpp:.* Class 'SHDescriptor' has a constructor with 1 argument that is not explicit."
"src/script/script.h:.* Class 'CScript' has a constructor with 1 argument that is not explicit."
"src/script/standard.h:.* Class 'CScriptID' has a constructor with 1 argument that is not explicit."
"src/span.h:.* Class 'Span < const CRPCCommand >' has a constructor with 1 argument that is not explicit."
"src/span.h:.* Class 'Span < const char >' has a constructor with 1 argument that is not explicit."
"src/span.h:.* Class 'Span < const std :: vector <unsigned char > >' has a constructor with 1 argument that is not explicit."
"src/span.h:.* Class 'Span < const uint8_t >' has a constructor with 1 argument that is not explicit."
"src/span.h:.* Class 'Span' has a constructor with 1 argument that is not explicit."
"src/support/allocators/secure.h:.* Struct 'secure_allocator < char >' has a constructor with 1 argument that is not explicit."
"src/support/allocators/secure.h:.* Struct 'secure_allocator < RNGState >' has a constructor with 1 argument that is not explicit."
"src/support/allocators/secure.h:.* Struct 'secure_allocator < unsigned char >' has a constructor with 1 argument that is not explicit."
"src/support/allocators/zeroafterfree.h:.* Struct 'zero_after_free_allocator < char >' has a constructor with 1 argument that is not explicit."
"src/test/checkqueue_tests.cpp:.* Struct 'FailingCheck' has a constructor with 1 argument that is not explicit."
"src/test/checkqueue_tests.cpp:.* Struct 'MemoryCheck' has a constructor with 1 argument that is not explicit."
"src/test/checkqueue_tests.cpp:.* Struct 'UniqueCheck' has a constructor with 1 argument that is not explicit."
"src/test/fuzz/util.h:.* Class 'FuzzedFileProvider' has a constructor with 1 argument that is not explicit."
"src/test/fuzz/util.h:.* Class 'FuzzedAutoFileProvider' has a constructor with 1 argument that is not explicit."
"src/wallet/db.h:.* Class 'BerkeleyEnvironment' has a constructor with 1 argument that is not explicit."
)
if ! command -v cppcheck > /dev/null; then
echo "Skipping cppcheck linting since cppcheck is not installed. Install by running \"apt install cppcheck\""
exit 0
fi
function join_array {
local IFS="$1"
shift
echo "$*"
}
ENABLED_CHECKS_REGEXP=$(join_array "|" "${ENABLED_CHECKS[@]}")
IGNORED_WARNINGS_REGEXP=$(join_array "|" "${IGNORED_WARNINGS[@]}")
WARNINGS=$(git ls-files -- "*.cpp" "*.h" ":(exclude)src/dashbls/" ":(exclude)src/util/expected.h" ":(exclude)src/leveldb/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" | \
xargs cppcheck --enable=all -j "$(getconf _NPROCESSORS_ONLN)" --language=c++ --std=c++17 --template=gcc -D__cplusplus -DCLIENT_VERSION_BUILD -DCLIENT_VERSION_IS_RELEASE -DCLIENT_VERSION_MAJOR -DCLIENT_VERSION_MINOR -DCOPYRIGHT_YEAR -DDEBUG -I src/ -q 2>&1 | sort -u | \
grep -E "${ENABLED_CHECKS_REGEXP}" | \
grep -vE "${IGNORED_WARNINGS_REGEXP}")
if [[ ${WARNINGS} != "" ]]; then
echo "${WARNINGS}"
echo
echo "Advice not applicable in this specific case? Add an exception by updating"
echo "IGNORED_WARNINGS in $0"
# Uncomment to enforce the developer note policy "By default, declare single-argument constructors `explicit`"
# exit 1
fi
exit 0

View File

@ -80,10 +80,15 @@ ENABLED_CHECKS_REGEXP=$(join_array "|" "${ENABLED_CHECKS[@]}")
IGNORED_WARNINGS_REGEXP=$(join_array "|" "${IGNORED_WARNINGS[@]}")
FILES_REGEXP=$(join_array "|" "${FILES[@]}")
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
CPPCHECK_DIR=$SCRIPT_DIR/.cppcheck/
# Check if CACHE_DIR is set and non-empty, otherwise use default .cppcheck/ directory
if [[ -n "$CACHE_DIR" ]]; then
CPPCHECK_DIR=$CACHE_DIR/cppcheck/
else
CPPCHECK_DIR=$SCRIPT_DIR/.cppcheck/
fi
if [ ! -d $CPPCHECK_DIR ]
then
mkdir $CPPCHECK_DIR
mkdir -p $CPPCHECK_DIR
fi
WARNINGS=$(echo "${FILES}" | \
xargs cppcheck --enable=all --inline-suppr --suppress=missingIncludeSystem --cppcheck-build-dir=$CPPCHECK_DIR -j "$(getconf _NPROCESSORS_ONLN)" --language=c++ --std=c++17 --template=gcc -D__cplusplus -DENABLE_WALLET -DCLIENT_VERSION_BUILD -DCLIENT_VERSION_IS_RELEASE -DCLIENT_VERSION_MAJOR -DCLIENT_VERSION_MINOR -DCOPYRIGHT_YEAR -DDEBUG -DUSE_EPOLL -DCHAR_BIT=8 -I src/ -q 2>&1 | sort -u | \