This document lists the complete set of badges that can be awarded to a Hyperledger project and displayed on its website (and GitHub repository). It also lists the promotion and demotion criteria for projects based on fulfilling or not maintaining the stated criteria for particular badges respectively.
The evaluations for these badges can be tied to the annual review cycle. A project can seek one or more of these badges during the review cycle and the TOC can (upon a manual inspection) determine whether or not to grant the request.
Mandatory Badges
These badges are required by at least one stage in the project lifecycle, to maintain/upgrade/downgrade projects. (See the table further below and the links within for criteria).
- Legal
- Decentralized (contribution and maintenance diversity: 2+ or 3+ orgs)
- Release
- Testing
- Documentation (this doesn't necessarily cover the subjective criteria of usability)
- OpenSSF Best Practices: Passing
- Security
- Production
- Level (Top-Level or Lab)
- Inbuilt CI/CD (test for existence)
- Structure (Hyperledger GitHub repository contains minimum expected files and modules)
Optional Badges
These badges indicate softer (or more subjective) aspects of a project that are good to have but harder to measure, and so will not be mandatory for any lifecycle decisions. Projects can still use these badges to indicate their maturity and usability.
- Reuse (unique contributions)
- CI/CD Best Practices (recommendations from Automated Pipelines task force and online survey)
- Automation & Usability (recommendations from Documentation task force)
- Conformity (recommendations from Documentation task force)
- Responsiveness & Engagement
Lifecycle Criteria
Discussion Notes
| Badge Type | Qualification Criteria | Monitoring Method | Monitoring Frequency | Requirement for Stages | Penalty for Non-Compliance | Notes |
|---|---|---|---|---|---|---|
| Legal | Project Badging Proposal | Declarations in repo and in files, GitHub Action to monitor | Continuous | All | (To be determined by LF legal staff) (TOC would have to judge this on a case-by-case basis, depending on the precise nature of non-compliance) (Ask maintainers' to take action. If they don't act within a given time period, say a month, move it to EOL. What do we do for a multi-repo project? Should we move all repos to EOL or just the ones not in compliance?) Non-compliance should result in a downgrade unless the maintainers can provide additional justification, or if the legal rules are murky. | Ask Ry (or various project maintainers) about using GitHub action to track this Peter: keep dependency pinned to last version of license (that meets legal requirements)? Jim: does badge apply to releases or latest project snapshot? Ry: reasonable to have badges tied to release as opposed to latest snapshot. (no need to add noise, i.e., multiple badges for releases). Tracy: hard to track whether a project is continuing to be on track or going downhill. We need a good way to track the project health. Bobbi: pizza pie metaphor for project health Tracy: prevent rather than penalize. Test for compliance in every PR. |
| Diversity/Decentralized | Project Badging Proposal | Look at (or parse programmatically) MAINTAINERS.md (possibly with additional org field), check commit activity and dashboard over a time period | Continuous | Labs and Incubation: soft criteria is more than one institution, hard criteria is more than one maintainer. Graduated: requires 3 or more independent institutions contributing maintainers. Deprecated/Dormant: can have fewer institutions as the reason for being in these stages is low project activity, not low diversity. | Graduated project may get downgraded to Incubated if it has 2 or fewer institutions maintaining it. | Need at least 3 independent (i.e., from different orgs) regular committers for the projct to be deemed healthy from the community's perspective. (This is specified in the Incubation exit criteria). Ensure that project is not dominated by one particular org, even if superficially the project has a diversity of maintainers. Check if there's a programmatic way to find out how diverse the contributions have been. Ry: can add a company affiliation in Maintainers.md. Tracy: determine xyz% maintainers from each company? This check can also subsume a check for the presence of a MAINTAINERS.md file. |
| Release | Project Badging Proposal | Scan the GitHub repo to measure frequency of new releases, check ROADMAP.md, decent amount of activity on public channels (Discord and mailing lists), existence of public RFCs | Mandatory for Graduated. Soft criteria for other stages: should be considered when a Lab wishes to Incubate. | Graduated project may get downgraded to Incubated or Dormant if it is not following Hyperledger release guidelines. If a project has high activity levels but is not releasing, then it should be downgraded to Incubated. If, on the other hand, it is not actively being worked on (which implies lack of periodic releases), then it should be downgraded to Dormant. | It's easy to evaluate what's in the repo, but hard to track how much the project's maintenance has occurred publicly with community visibility. Perhaps we should just mandate the first and rely on self-declaration for the second. | |
| Testing | Project Badging Proposal | Ensure that processes for unit and integration testing are in place and run periodically. Test coverage MD file; ask maintainers to self-report this. | Mandatory for Graduated. Soft criteria for a project to Incubate: perhaps also when a project is moving from Lab to Top-Level. | Downgrade to Incubation (indicated immaturity to some extent). | (Evaluate this badge vs "Production" badge) Exact criteria to use is hard to determine. Collect coverage stats from various GitHub actions, check report badges on README? It may be enough to have projects report their test coverage results in a well-known location, as this will inspire confidence in users (the exact criteria and numbers are less important). | |
| Documentation | Project Badging Proposal | Inspection, compliance with look-and-feel, fulfil feature set defined in the 'mkdocs' template (and whatever set of topis the Documentation Task Force comes up with) | Mandatory for Graduated. Soft criteria for a project to Incubate: perhaps also when a project is moving from Lab to Top-Level. | Downgrade to Incubation (indicated immaturity to some extent). | Can we automate this inspection using GitHub actions? How do we handle empty files in the docs (if it's just a skeleton website)? | |
| Project Badging Proposal | We probably don't need this one as these criteria are assumed to be fulfilled if a project proposal is accepted. | |||||
Accepted (as Labs or as Top-Level Project) Call this "Level" to indicate whether a project is a Lab or a Top-level project. | Project Badging Proposal | Mandatory for all stages by definition. | (Evaluate the need for this against "Responsiveness" and "Conformity" badges). Project should already have the various features set up. What we need to infer are activity and engagement levels. Any potential for automation here? (Add Discord link or mailing list link in the main README. Also add invite link to Hyperledger Discord server.) Note: Labs' Discord channels are optional. Does Accepted (Project) take precedence over Accepted (Labs)? Does Accepted go away with EOL? (We can go with "yes") | |||
OpenSSF | Project Badging Proposal | Needs to be "Passing" for Graduated stage. Mandatory for Graduated. | Downgrade to Incubation (indicated immaturity to some extent). | CII's new incarnation is OpenSSF. (Check what OpenSSF recommends.) (Check if this can ever be downgraded, after someone fills out a questionnaire.) Compare and contrast this list with the HL Best Practices guidelines. Borrow badge ideas from the latter. How do we monitor whether a project is actively maintaining the criteria for having an OpenSSF Badge? How do we "refresh" the badge status? | ||
| Security? | https://github.com/arsulegai/tsc/blob/security-gov/governing-documents/security.md (PR under review, but approved) | All stages for Top-Level project Softer criteria for Labs. | Incubated, Graduated → Deprecated Deprecated → EOL Lab → archived (EOL) |
(Track security GitHub issues to see if they have been resolved within a given time period, like 90 days. Otherwise, flag as out-of-compliance; badge can be revoked if maintainers don't take action within a prescribed review period.) Badges may help during annual review, gives people an opportunity to examine the projects and see if they deserve the badges they currently possess. | ||
(Maybe revisit this later, if we can define the criteria more specifically.) | Should reuse components from other Hyperledger projects as needed and not reinvent the wheel | Q. Would this be useful to the end user? What is the right level of looking at this uniqueness? This criterion is not concrete enough yet. | ||||
(Revisit this when the Automated Pipelines Task Force has made more progress) | Best Practices for Automated Pipelines | Question is to what extent does a project meet these criteria. | ||||
| Automation/Usability? | Can users easily launch and test the software using automated scripts or do they have to jump through hoops to get started? | |||||
| https://github.com/hyperledger/toc/blob/gh-pages/guidelines/project-best-practices.md | ||||||
| Conformity? | Does the look and feel of the repo structure and the documentation match that of other projects (according to Hyperledger's recommended practices)? This may overlap with "documentation". Need suggestions from documentation task force. | Use action to track whether the repo has all the required files (maintainer, security policy, security SPOC, rfcs, etc.; list TBD). | ||||
| Responsiveness/Engagement? | Are maintainers engaging with the community and responding reasonably promptly to questions and solicitations for involvement? Need suggestions from onboarding task force. The number of summer mentees may also factor into this. | |||||
| Production? | Is the software currently being used in at least one production-grade system outside of Hyperledger (publicly verifiable or certified by third party consumer)? | Evaluate adopters and see what stage of their projects they are in | Mandatory for Graduated: at least one example of production usage. | Lack of new production uses implies lack of maintenance? This could perhaps move a Graduated project to Dormant. | This may be a more useful criterion and badge than "Testing". Maybe recommend an ADOPTERS.md file (Gives people an idea of who are using it.) |