User Tools

Site Tools


projects:fabric:fabric_ci_documentation

Getting Started with Fabric CI

Contributors

Name Email
Jason T Clark jtclark@linux.vnet.ibm.com
Jessica Wagantall jwagantall@linuxfoundation.org
Ramesh Thoomu rameshbabu.thoomu@gmail.com
Scott Zwierzynski scottz@us.ibm.com

Summary

The Hyperledger Fabric (and associated) projects utilize various tools and workflows for continuous project development. This documentation will assist you in getting started with using these tools and understanding the workflow(s) our contributors use while working with the Fabric CI infrastructure.

Finding Help on Hyperledger CI

Welcome to the Hyperledger Fabric community! We are excited that you want to contribute to the Continuous Integration/Release Engineering efforts. You're in the right place to get started. In the event that you need additional assistance, We encourage you to engage with our CI contributors via the following channels:

  1. #ci-pipeline channel on Rocket.Chat (For general continuous integration discussions)
  2. #fabric-ci channel on Rocket.Chat (For fabric continuous integration discussions)
  3. #infra-support channel on Rocket.Chat (For general infrastructure discussions)
  4. Send an email to the fabric@lists.hyperledger.org mailing list
  5. Contact helpdesk@hyperledger.org for any infrastructure support

Getting Started with Fabric CI

The Fabric CI environment is comprised of two parallel deployments of the Jenkins CI system:

Note: You will notice that both the Jenkins production instance and the Jenkins sandbox instance are both accessible from the same URL, though the sandbox instance is accessed at /sandbox.

For more information on how to use the sandbox, use the documentation in https://github.com/hyperledger/ci-management/blob/master/Sandbox_Setup.md

Hyperledger Fabric projects on CI

Jenkins Dashboard

On the Jenkins dashboard, a list of all Jenkins jobs required to build, test, verify and release each of the Hyperledger Fabric projects is organized and displayed in separate views. For example, the Fabric view displays each of the jobs related to the fabric project:

Here, all of the Jenkins jobs for testing the Fabric project are displayed. The Jenkins jobs are displayed in a table, and indicates:

  • Status - the status of the most recently executed job,
  • Name - the name of the job
  • Last Success - the last successful run of the job
  • Last Failure - the last failed run of the job
  • Last Duration - indicates the length of time the last job took to run

Jenkins Job Types

Continuing with the Fabric project example, the jobs list contains multiple jobs required to accurately build, test, and verify the Hyperledger Fabric project. The job types for the Fabric project are as follows

* Verify
  * End-to-End (performs e2e_cli, sdk node e2e, java sdk e2e tests)
  * Unit Tests (performs make linter, make unit-tests)
  * Documentation changes (Builds the documentation and ship it to nexus log server. 

You can check the RTD format of you patch set documentation build here https://logs.hyperledger.org/production/vex-yul-hyp-jenkins-3/fabric-rtd-verify-master/<build_number>/html

ex: https://logs.hyperledger.org/production/vex-yul-hyp-jenkins-3/fabric-rtd-verify-master/359/html

  • Merge
    • End-to-End
    • Unit-Tests

IMPORTANT NOTE: Although many of the job types listed above are common across all Hyperledger Fabric projects, the list indicated above is specific to the Hyperledger Fabric project. Different projects will have different job types based on their individual build/test/release requirements.

Common Job Types

There are several Jenkins job types that are common across most Hyperledger Fabric projects. In some cases, you may/may not see all of the common job types in every project. This is heavily dependent on the needs of the Hyperledger Fabric project. That said, let's take a look at each of the most common jobs:

Verify Jobs

Verify jobs are triggers when a patchset-created-event is triggered in gerrit. All verify jobs are depends on the patchset parent commit and patchset commit. That tells, verify jobs run on the parent commit on which developer submitted patch not on the latest master code.

Merge Jobs

Merge jobs triggers when a “change-merged-event” event is triggered in gerrit. In all the merge jobs, Jenkins clones the latest master code and perform the tests unlike verify jobs.

Release Jobs

Release jobs are triggered after a “ref-updated-event” is created in each of the Fabric repositories targeted for release. Release jobs are intended to create to publish docker images, binaries and publish npm modules. For more information regarding the release process, refer to this Release Process Document.

Support for varying Architectures

With most job types, the Jenkins build jobs are broken down further to build, test, and release the Hyperledger Fabric projects with support for varying CPU architectures. They include:

  • x86_x64
  • s390x

Support for varying test types

Additionally with most job types, you will notice the Jenkins jobs are further isolated to include a number of test types. Those include:

  • End-to-End
  • BYFN Checks (byfn, eyfn with default, custom, couchdb, node lang chaincode)
  • Unit Tests
  • Smoke/Functional/Performance/Release tests ( More functional tests from fabric-test repository)

For more information on what each of these test types are used for, refer to the Fabric CI Process documentation.

CI-Management Repo

ref: https://github.com/hyperledger/ci-management/blob/master/docs/source/fabric_ci_process.rst

The ci-management repo hosts the CI job definitions for all of the Hyperledger Fabric projects integrated with Jenkins. Each job configuration is written in YAML format, and built with the Jenkins Job Builder (JJB) tool.

Writing Jenkins Job Definitions

Much of the work inside of Fabric CI involves the creation or modification of Jenkins job definitions. To get a better understanding of how to write Jenkins job definitions, start with reading through the JJB Job Definitions documentation.

Jenkins Sandbox

ref: https://github.com/hyperledger/ci-management/blob/master/Sandbox_Setup.md

When creating or modifying Jenkins job definitions, it is important to know whether or not your job definitions work prior to adding them to the production Jenkins instance. This is why Hyperledger Fabric has provided a Jenkins Sandbox instance, allowing you to test your Jenkins job definitions. To use the Jenkins sandbox, read through the Sandbox Setup process, which explains how to install Jenkins Job Builder (JJB), as well as testing, updating, and triggering jobs on the Jenkins Sandbox.

Contributing to Fabric CI

Contributing to the Fabric CI process starts with identifying tasks to work on, and bugs to be fixed. The JIRA site provided by the Hyperledger Community is where to find these items. To narrow down tasks and bugs that are directly related to Fabric CI, use the following URLs:

Comment Phrases to trigger failed jobs

Trigger builds through comments to a commit/change:

Re-verification/trigger of builds is possible in Jenkins by entering any of the below comment phrases as a comment to the Gerrit patch set. To do so, follow the below process:

Step 1: Open the failed Gerrit patch set

Step 2: Click on Reply and type the below comment phrases based on the project and click Post

CI jobs listens on the below comment phrases, when you type a comment as reverify-x in fabric project, this triggers fabric-verify-x86_64 CI job.

fabric:

  • Run VerifyBuild/VerifyBuild - Triggers fabric-verify-build-checks-x86_64
  • Run SmokeTest – Triggers fabric-smoke-tests-x86_64.
  • Run UnitTest – Triggers fabric-verify-unit-tests-x86_64.
  • Run IntegrationTest - Triggers fabric-verify-integration-tests-x86_64
  • Run DocsBuild – Triggers fabric-docs-build-x86_64

fabric-ca:

  • reverify-x – triggers fabric-ca-verify-x86_64 job on x86_64
  • reverify-z – triggers fabric-ca-verify-s390x job on s390x
  • reverify-e2e – triggers fabric-ca-verify-end-2-end-x86_64 job on x86_64
  • recheck – Triggers fabric-rtd-verify-job
  • reverify – triggers all failed jobs.

fabric-sdk-node:

  • reverify-node8x – triggers fabric-sdk-node8-verify-x86_64 job on x86_64
  • reverify-node8z – triggers fabric-sdk-node8-verify-s390x job on s390x
  • reverify-node6x – triggers fabric-sdk-node6-verify-x86_64 job on x86_64
  • reverify-node6z – triggers fabric-sdk-node6-verify-s390x job on s390x
  • reverify – triggers all the failed fabric-sdk-node jobs

fabric-sdk-java:

  • reverify-x – triggers fabric-sdk-java-verify-x86_64 job on x86_64
  • reverify-1.0.0 – triggers fabric-sdk-java-verify-1.0.0-x86_64 job on x86_64
  • reverify – triggers all the failed fabric-sdk-java jobs

fabric-chaincode-node:

  • reverify-x – triggers fabric-chaincode-node-verify-x86_64 job on x86_64
  • reverify-z – triggers fabric-chaincode-node-verify-x86_64 job on s390x
  • reverify – triggers all the failed fabric-chaincode-node jobs

All other repositories use any of the below comment phrases

  • reverify-x or reverify – Triggers jobs on x86_64 build node

Troubleshooting - Failing CI Jobs

For an overview of CI Processes, refer to the FAQ, below.

If you are a developer having difficulty obtaining that green checkmark consistently from hyperledger-jobbuilder because your merge job or verify job fails, then *create a bug* in jira to start the process rolling to fix the problem for everyone in the community!

  1. Take a look at why your patchset is failing verification. If it seems unrelated to your patchset changes, then…
  2. Examine console logs for other patchsets build failures occurring in Fabric Project builds. Confirm if any others are seeing build failures with the same symptoms. If so, then…
  3. Find out if a bug exists already for your problem. The first place to look is the list of open CI_Failure bugs. Next, SEARCH Fabric Issues for your error strings symptoms. If no bug exists for your problem, then…
  4. CREATE a bug with severity = 'Highest' and label = 'ci_failure', and set the appropriate component for triage. Squads regularly review their components bug lists.
  5. Alert the community by reaching out to #fabric-ci channel (or #infra-support channel, if the issue pertains to network latency issues, such as FAB-7971), where you may find help and discussions of similar topics. For more info, see the CI FAQs, below.

Frequently Asked Questions

Below is a table that shows you where to find the most informative resources for the most frequently asked questions about Fabric CI.

Question Resource
Which Fabric projects are currently using CI? hyperledger_fabric_projects_on_ci
What is the process used to build, test, and release Fabric projects? https://github.com/hyperledger/ci-management/blob/master/docs/fabric_ci_process.md
Where is the Jenkins CI for Hyperledger Fabric? https://jenkins.hyperledger.org/
Where is the sandbox installation of Jenkins CI for Hyperledger Fabric? https://jenkins.hyperledger.org/sandbox
Where can I found out more about each of the Jenkins jobs used to build, test, and release Hyperledger Fabric projects? https://github.com/hyperledger/ci-management
How do I write Jenkins job definitions for use with Hyperledger Fabric projects? https://docs.openstack.org/infra/jenkins-job-builder/definition.html
How should I test my Jenkins job definitions for use by the Fabric CI process? https://github.com/hyperledger/ci-management/blob/master/Sandbox_Setup.md
I want to find Fabric CI related tasks to work on. Where should I go? https://jira.hyperledger.org/issues/?filter=11500
I have questions about Fabric CI. Where should I go for help? https://chat.hyperledger.org/channel/ci-pipeline
projects/fabric/fabric_ci_documentation.txt · Last modified: 2018/10/30 08:03 by Ry Jones