Page History

Besu's CI process is broken into reusable phases, and defers long-running and expensive phases till as late and infrequently as possible, while still ensuring maximum quality checks. CI/CD used to be implemented via CircleCI, but it has since been migrated to Github Actions in an effort to reduce cost as well as be consistent with the majority of Hyperledger projects.

What this means for users

Users should not notice anything different, other than new delivery urls for Besu releases. Github Packages and the Github Container Registry will be the source for all binaries, but all releases will still be communicated out to users via Github releases.

What this means for developers

• Deferral of advanced CI/CD testing. As a cost reduction measure (CircleCI for Besu costs a little under $2K USD per month), long running tests are only executed after PR peer review. Details on how to run those tests locally are now included in the pull request template, and they can also be manually initiated on the CI/CD infrastructure.
• Releases are now much more fully automated. Any maintainer may create a release from main  or a release-*  named branch (hereafter referred to as "releasable branches") using standard Github tools.  That process is outlined in more detail below.
• Speed. Breaking up the various workflows into smaller pieces using test splitting results in much greater parallelization, and hence shorter overall runtimes.

How does it work?

On new PR open, draft or otherwise, the pre-review workflow is run.

• check for repo compliance via repolinter
• check for source code formatting via spotless
• check gradle tooling validation.
• compile all code
  • then validate javadocs- done later because this depends on bytecode output.
• run unitTests
  • test suites are grouped by gradle subproject, and run in parallel
  • any test failures will be annotated on the PR.

Overall this looks like:

Image Modified

It takes about 15 minutes, end to end, but if we can be smarter about splitting the unit tests we could easily cut that in half. 

Once the PR is approved by any Besu Maintainer, the workflow executes the more expensive test suites before allowing a merge to a releasable branch.

• run integrationTests
• run acceptanceTests
• run referenceTests

Test results can then be

...

annotaed directly on to the PR, and any failures would prevent merging into main.

...

 You'll also find junit test

...

result artifacts attached to the workflow run, and a summary of how each test shard performed

...

. 

Common Patterns

There are a couple of common patterns worth explaining in this pipeline. We regularly need to determine if a long-running test suite should run, how to split up tests, and how to consolidate results.

• at this time Github Actions does not have an event for a PR being approved for the first time. That means we need to make sure we do not run tests for each approval a PR receives.  To do this, we directly interact with the Github REST API, and check if the comment placed on the PR is an approval, and that the tests have not already run successfully.  See the shouldRun job defined the acceptance-tests.yml as an example.
• test splitting is handled by a github action which intends to group them by the most even runtime based on past runs. In the case of the EVM reference tests, the sheer volume seemed to overwhelm it. Those tests are split up using a simple bash script.
• testing workflows that use a matrix strategy for parallelization, will also have a consolidating job that waits for them all to complete, before marking the workflow run as passed.

Merging to Main

We now can rely on a branch ruleset to collect all our rules about merging to a releasable branch.  Merging to main or any release-* named branch

...

is denied until:

• pre-review checks have passed on the merge result, and the PR has a 

...

• unittests-

...

• passed  status.
• PR has an approval from at least one project maintainer.
• acceptance test checks have passed on the merge result, and the PR has an 

...

• accepttests-

...

• passed status.
• integration test checks have passed on the merge result, and the PR has an integration-tests status. This one is named differently because it does not need a consolidating job, as described above.
• reference test checks have passed on the merge result, and the PR has a 

...

• reftests-

...

• passed status.

Once all these are confirmed, the PR may be merged, and all statuses should be transferred to the corresponding commit on the target branch. That commit would now be considered releasable. 

...

Release Process

...

...

This process supports (and encourages) releasing directly from main, but

...

also allows for the creation of release specific branches for hotfixes or interim releases which must not include what is currently on main.

When a github release is created (pre-release or otherwise) then the following artifacts are built, and attached to the release:

• tarball - release version embedded in the file name.  The release description has the sha256 sum for this file appended to it.
• zipfile - release version embedded in the file name.  The release description has the sha256 sum for this file appended to it.
• docker images created and tagged.
• if a non-pre-release is published, the latest  tag is moved to the release image.

All artifacts will have the version number specified in the created github release, no modification to source is necessary, and so none of the testing above needs to be re-executed.

Example release can be seen here.