...
What this means for developers
- Deferral of advanced CI/CD testing. As a cost reduction measure (CircleCI for Besu costs a little under $2K USD per month), long running tests are only executed after PR peer reviewexecuted in parallel on low-spec, free action runners. Details on how to run those tests locally are now included in the pull request template, and they can also be manually initiated on the CI/CD infrastructure.
- Releases are now much more fully automated. Any maintainer may create a release from
mainor arelease-*named branch (hereafter referred to as "releasable branches") using standard Github tools. That process is outlined in more detail below. - Speed. Breaking up the various workflows into smaller pieces using test splitting results in much greater parallelization, and hence shorter overall runtimes.
...
It takes about 15 minutes, end to end, but if we can be smarter about splitting the unit tests we could easily cut that in half.
Once the PR is approved by any Besu Maintainer, the workflow executes the At the same time, other workflows execute the more expensive test suites before allowing a merge to a releasable branch.
...
There are a couple of common patterns worth explaining in this pipeline. We regularly need to determine if a long-running test suite should run, how to split up tests, and how to consolidate results.
...
.
- test splitting is handled by a github action which intends to group them by the most even runtime based on past runs. In the case of the EVM reference tests, the sheer volume seemed to overwhelm it. Those tests are split up using a simple bash script.
- testing workflows that use a matrix strategy for parallelization, will also have a consolidating job that waits for them all to complete, before marking the workflow run as passed.
...