NOTE: this is a rough draft of the proposal for our new CI/CD and test net system. Feel free to comment and/or add content.

CI/CD and Test Net Proposal

Summary

Hyperledger would switch to running an instance of Gitlab and allow project maintainers to set up and run their own CI/CD and test net clusters. Hyperledger would run a few "shared" runners to ensure that there is always a machine or two available to run jobs for projects that do not have any volunteer runners. This solution allows for projects to not only handle CI/CD but also test nets. Gitlab runners are capable of running just about anything. Project maintainers can create test net projects with a CI pipeline definition that tells the runners to download a docker image of the latest build of a project, and all of our blockchain platforms publish ready-to-run docker images, they can easily stand up a test net for a scaling or soak test. This proposal reduces the human overhead and the financial overhead of running a CI/CD pipeline as well as increasing the level of control each project will have over their CI/CD pipeline.


TL;DR

Requirements

  • Shall be able to clone and build all of the code repositories for all Hyperledger projects. This includes repositories hosted on Github as well as Gerrit.
  • Shall present a dashboard for maintainers that reflects the build status and performance of the CI/CD pipeline.
  • Shall be "self service" in a few key ways:
    • Maintainers of a project shall be able to exercise full control over what gets accomplished by the runners in their CI/CD cluster.
    • Contributors/volunteers shall be able to set up computers as runners to contribute resources to a CI/CD cluster.
    • Maintainers shall be able to restrict access to which computers contribute to the CD portion of the cluster.
  • Shall have integrations with external services such as Github, Gerrit, Confluence, and Jira.

The proposal is to use Gitlab for managing and running the new CI/CD pipelines for all of the Hyperledger projects. Gitlab meets all of these requirements without any special configuration and with a specific setup, we can accomplish all of the self-service goals as well.

External Repositories

Per the documentation, Gitlab supports mirroring external git repositories and triggering CI/CD builds from merges and commits in the external repository. It has full integration with Github but also has the ability to mirror any external repository that is publicly accessible. So our Gerrit hosted code repositories are supported.

Dashboards

Gitlab has a CI/CD "charts" page that shows the statistics from past CI/CD runs over time.

Self Service

Gitlab's real power lies in the self-service nature of setting up the CI/CD configuration by maintainers and the setting up of runners by developers that contribute resources to the CI/CD cluster for each project. Gitlab has four different roles that actions are assigned to; Guest, Developer, Maintainer, and Admin. Gitlab's security model perfectly fits what Hyperledger is looking for. The HL/LF staff will be Admins and will be responsible for site-wide configuration. Maintainers from each project are able to create and configure projects without assistance from HL/LF staff. This gives maximum freedom to each project team in setting up their CI/CD pipeline as well as setting up test net like clusters for soak, scaling and other tests. Developers are able to trigger jobs on the runners cluster.

 

ActionGuest, ReporterDeveloperMaintainerAdmin
See commits and jobs✓✓✓✓
Retry or cancel job ✓✓✓
Erase job artifacts and trace ✓✓✓
Remove project  ✓✓
Create project  ✓✓
Change project configuration  ✓✓
Add specific runners  ✓✓
Add shared runners   ✓
See events in the system   ✓
Admin interface   ✓


Integrations

Gitlab integrates with Jira and therefore also integrates with Confluence.

  • No labels

7 Comments

  1. Morgan Bauer

    why is this under security?

    1. David Huseby

      Part of security is software delivery. Part of software delivery is CI/CD.

  2. Alejandro (Sasha) Vicente Grabovetsky

    That is pretty awesome, looking forward to this.

  3. Kyle Den Hartog

    Can the "dev" role erase artifacts other than the ones they started?

    1. David Huseby

      as far as I can tell, no.

  4. Morgan Bauer

    I think I need a level-set of the context. Any pointers to previous discussion would be very much appreciated.

    Not to distract too much from the beautiful looking proposal, but why is this an issue at all?

    Is it the lack of understanding how to use the jenkins for a specific project? How to configure jenkins in general?

    A desire for the increased capabilities of gitlab ci? The ability to add external test runners sounds very cool. Is it not something capable for jenkins to do?

    Overall I wonder if this is a documentation problem or a capability problem or a usability problem? What's the goal?

    1. Silas Davis

      I think these are good questions for us to raise at the TSC.

      I am slightly out of date with Jenkins (by about two years), I am also using self-hosted gitlab-ci for Burrow, so I am biased and possible wrong. However the edge I perceive gitlab to have is around how it handles 'git ops' and it execution.

      Git ops:

      • Last time I looked the in-repo gitlab-ci.yml was better at centralising and isolating configuration for a reproducible project build, as opposed to Jenkins that has a lot of global/ambient config and plugins you tend to need to understand on the instance you are running against.
      • The linkage between git pushes and merge requests and pipeline triggers (which can be automatic or manual) is tighter in gitlab because it hosts git. Seems like this requires less config and tends to have less breakage (provided you are using the git repo - more on that later)

      Execution:

      • The fact you can add runners which themselves can be tagged with certain capabilities is quite flexible for different workloads
      • There is built-in support for deploying to Kubernetes clusters (configured in the operations section)
      • There is built-in support for tracking deployments via the environments

      I'm pretty sure Jenkins could be made to do all fo this, but I personally found less friction with Gitlab


      However, to get the most out of gitlab I think you need to be using the repos. I guess we can just push to these Gitlab CI instances when we want to run testnets etc, but are we missing some of the benefit if the authoritative project repo is on github?