dash/ci
MarcoFalke 2d4db48606 Merge #18581: ci: Print ccache stats, add pip cache, and cleanups
73f83036298e4932772704b2d5359c78d81bd7cc ci: Add pip cache (Hennadii Stepanov)
6d480667b24dc630e7c3fe7ff032cbe1a958ab77 ci: Print ccache statistics summary (Hennadii Stepanov)
a6eea715ff549febe6d4bd52f17acb50abd99120 ci: Drop unused WINEDEBUG variable (Hennadii Stepanov)

Pull request description:

  The Travis [pip cache](https://docs.travis-ci.com/user/caching/#pip-cache) is free and saves a dozen of seconds :)

  Here are some excerpts from the Travis logs with `ccache` statistics (I found useful):

  2) [Arm64](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507749)
  ```
  ccache version 3.6
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats updated                       Fri Apr 10 18:21:06 2020
  stats zeroed                        Fri Apr 10 18:03:04 2020
  cache hit (direct)                   156
  cache hit (preprocessed)             198
  cache miss                           143
  cache hit rate                     71.23 %
  called for link                        8
  cleanups performed                     9
  files in cache                      1255
  cache size                          80.6 MB
  max cache size                     100.0 MB
  ```
  3) [s390x](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507750)
  ```
  ccache version 3.4.1
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats zero time                     Fri Apr 10 17:15:35 2020
  cache hit (direct)                   115
  cache hit (preprocessed)             163
  cache miss                            94
  cache hit rate                     74.73 %
  called for link                        6
  cleanups performed                     0
  files in cache                       804
  cache size                         304.8 MB
  max cache size                       5.0 GB
  ```
  4) [Win64](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507751)
  ```
  ccache version 3.4.1
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats zero time                     Fri Apr 10 17:38:07 2020
  cache hit (direct)                   147
  cache hit (preprocessed)             199
  cache miss                           138
  cache hit rate                     71.49 %
  called for link                        7
  cleanups performed                     7
  files in cache                      1242
  cache size                          87.0 MB
  max cache size                     100.0 MB
  ```
  5) [CentOS 7](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507752)
  ```
  ccache version 3.7.7
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats updated                       Fri Apr 10 17:45:59 2020
  stats zeroed                        Fri Apr 10 17:34:27 2020
  cache hit (direct)                   152
  cache hit (preprocessed)             200
  cache miss                           144
  cache hit rate                     70.97 %
  called for link                        8
  cleanups performed                     4
  files in cache                      1352
  cache size                          86.1 MB
  max cache size                     100.0 MB
  ```
  6) [bionic](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507753)
  ```
  ccache version 3.4.1
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats zero time                     Fri Apr 10 17:21:10 2020
  cache hit (direct)                   136
  cache hit (preprocessed)              35
  cache miss                           330
  cache hit rate                     34.13 %
  called for link                        8
  cleanups performed                    18
  files in cache                      1302
  cache size                          83.8 MB
  max cache size                     100.0 MB
  ```
  7) [xenial](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507754)
  ```
  ccache version 3.2.4
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  cache hit (direct)                   161
  cache hit (preprocessed)             174
  cache miss                            95
  called for link                        7
  files in cache                      3411
  cache size                           1.1 GB
  max cache size                       5.0 GB
  ```
  10) [focal](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507757)
  ```
  ccache version 3.7.7
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats updated                       Fri Apr 10 17:35:57 2020
  stats zeroed                        Fri Apr 10 17:21:32 2020
  cache hit (direct)                    78
  cache hit (preprocessed)             125
  cache miss                           107
  cache hit rate                     65.48 %
  called for link                      120
  cleanups performed                     0
  files in cache                      6218
  cache size                           1.8 GB
  max cache size                       5.0 GB
  ```
  14) [macOS 10.12](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507761)
  ```
  ccache version 3.4.1
  cache directory                     /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /home/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /etc/ccache.conf
  stats zero time                     Fri Apr 10 17:47:19 2020
  cache hit (direct)                    28
  cache hit (preprocessed)               1
  cache miss                           469
  cache hit rate                      5.82 %
  called for link                        8
  cleanups performed                    17
  files in cache                      1946
  cache size                          82.6 MB
  max cache size                     100.0 MB
  ```
  15) [macOS 10.14 native](https://travis-ci.org/github/bitcoin/bitcoin/jobs/673507762)
  ```
  ccache version 3.7.8
  cache directory                     /Users/travis/build/bitcoin/bitcoin/ci/scratch/.ccache
  primary config                      /Users/travis/build/bitcoin/bitcoin/ci/scratch/.ccache/ccache.conf
  secondary config      (readonly)    /usr/local/Cellar/ccache/3.7.8/etc/ccache.conf
  stats updated                       Fri Apr 10 17:38:44 2020
  stats zeroed                        Fri Apr 10 17:22:36 2020
  cache hit (direct)                   213
  cache hit (preprocessed)             293
  cache miss                           144
  cache hit rate                     77.85 %
  called for link                       11
  called for preprocessing              56
  compile failed                        30
  preprocessor error                    56
  bad compiler arguments                14
  autoconf compile/link                 77
  no input file                         66
  cleanups performed                     0
  files in cache                      1564
  cache size                         284.6 MB
  max cache size                       5.0 GB
  ```

Top commit has no ACKs.

Tree-SHA512: f9f2c6d72680bafc7a982efc04dc37ab9bba77ffd2a4cbbc56bb0e86c5c3501d57aa2f0ebeac0113e9d8daac1ee4eac1eddf74c1172c267b37ee2d9324366c8a
2023-08-29 22:00:59 -05:00
..
dash Merge #18581: ci: Print ccache stats, add pip cache, and cleanups 2023-08-29 22:00:59 -05:00
lint Merge #15693: travis: Switch to ubuntu keyserver to avoid timeouts 2023-04-06 20:14:58 +03:00
retry Merge #17284: build: update retry to current version 2022-04-03 18:08:40 -05:00
test Merge #18581: ci: Print ccache stats, add pip cache, and cleanups 2023-08-29 22:00:59 -05:00
README.md Merge #17423: ci: Make ci system read-only on the git work tree 2022-06-16 01:34:30 -04:00
test_run_all.sh merge bitcoin#16623: Add environment files for all settings 2022-02-12 19:24:32 +05:30

ci scripts

This directory contains scripts for each build step in each build stage.

Currently three stages lint, extended_lint and test are defined. Each stage has its own lifecycle, similar to the Travis CI lifecycle. Every script in here is named and numbered according to which stage and lifecycle step it belongs to.

Running a stage locally

Be aware that the tests will be built and run in-place, so please run at your own risk. If the repository is not a fresh git clone, you might have to clean files from previous builds or test runs first.

The ci needs to perform various sysadmin tasks such as installing packages or writing to the user's home directory. While most of the actions are done inside a docker container, this is not possible for all. Thus, cache directories, such as the depends cache or ccache, are mounted as read-write into the docker container. While it should be fine to run the ci system locally on you development box, the ci scripts can generally be assumed to have received less review and testing compared to other parts of the codebase. If you want to keep the work tree clean, you might want to run the ci system in a virtual machine with a Linux operating system of your choice.

To allow for a wide range of tested environments, but also ensure reproducibility to some extent, the test stage requires docker to be installed. To install all requirements on Ubuntu, run

sudo apt install docker.io bash

To run the default test stage,

./ci/test_run_all.sh

To run the test stage with a specific configuration,

FILE_ENV="./ci/test/00_setup_env_arm.sh" ./ci/test_run_all.sh