What are the criteria for a "First Major Release"? How does Dave's proposed Project Readiness fit in? FWIW, I created this set of criteria for the Fabric 1.0 release.
Approved Resolution 2 (TSC 09/05/2019):
Prior to bringing a project to the TSC for a vote on granting a First Major Release a project must fulfill the following criteria:
- Project has reached active status (the processes and community are mature)
- The release features substantially fulfill the project's charter
- The release satisfies criteria defined by the project itself. All projects should meet some internally defined non-functional release criteria (bug counts, performance, etc.)
- Test coverage (unit and functional/integration) is sufficient to have confidence that the release supports the non-functional criteria
- Maintainers have reviewed all remaining open bugs and agree on severity/priority
- Zero high or highest priority bugs remain open
- The preceding release candidate has been published for a minimum period of time of 2 weeks
- The release should not require users/operators to compile code to operate the base project on supported platforms
- The documentation is sufficient to ensure that users/operators have clear guidance on how to get started and how to configure and operate.
- The project has met all technical criteria for the release.
- The project has met all the criteria for CII Best Practices Badge Best Practices Badge,
- A security audit provided by Hyperledger was completed
- no bugs that affect security of the system remain open unless they have mitigating workarounds published in release notes
- crypto code included in the release has been audited for crypto export compliance
- A license scan was completed and all issues were resolved, including receiving an exception from the legal committee.
- Changelog of all commits is generated and published.
- Release notes generated and published.
19 Comments
Arnaud J Le Hors
Vipin Bharathan
Let us not a create a draft that is dependent on another draft. According to me that (project readiness proposal) needs some serious rework mostly for prolixity. Also for a lot of soft and possibly irrelevant criteria for readiness.
Arnaud J Le Hors
I agree with the sentiment Vipin. For expediency I merely put a reference when I did this yesterday but "we" should simply incorporate whatever is relevant into this very document.
Vipin Bharathan
Proposed/Draft Resolution 2:
Can we just point to a doc (that is not yet ready but draft) which will be a form of the project readiness document.
Arnaud J Le Hors
Vipin, are you proposing to do exactly what you said earlier we shouldn't do?
Vipin Bharathan
No, I am saying work on the project readiness document first. It will subsume the criteria for 1.0
Dan Middleton
The point of this event is that Hyperledger gets behind the release with marketing and brand. To qualify for that a project should
The first three are largely objective. #4 is clearly subjective but says that if a project is chartered to fill some role at Hyperledger a major release should show that functionality.
#5 nudges projects to establish their own criteria without dictating to them things that are project-specific.
This Project Readiness doc additionally mentions use of HL infrastructure (wiki, chat, etc.). We should make that part of active status requirements.
Shawn Amundson
If we follow semver (which we do, generally), the version number is primarily an indicator of a commitment to some level of backward compatibility support. This may fit under Dan's (#5) but may be worth calling out specifically so projects spend some time considering what it means.
Christopher Ferris
Adding to Dan's recommendation above, I would cite the Fabric release criteria we used for 1.0 Release Exit Criteria
Dan Middleton
Good idea. I did a best effort to fold those two together and set that as the resolution. We can iterate on this a bit.
Arnaud J Le Hors
Thanks for taking a crack at putting a proposal together Dan. It looks good to me but I think we need to specify what "a minimum period of time" for the candidate release is. I realize this may be a bit arbitrary but without any specification at all the requirement becomes meaningless. Fabric set it to 2 weeks. You don't think that's good enough?
Dan Middleton
I don't have a strong opinion on the duration. I think it's some combination of the time necessary for availability testing (sawtooth uses a 7 day burn-in for example) and user feedback. If say RC1 was up for a month and had a bug, I would think that RC2 would only need to be up for long enough to QA (1 week in ST case) because other user acceptance aspects would not have changed from RC1 to RC2.
Binh Nguyen
I agreed we should specify "a minimum period of 2 weeks" Two weeks would give most willing people time to arrange to take a dive into the new release. I would personally prefer 4 weeks, so that's why "minimum period"
Christopher Ferris
"d. the preceding release candidate will have been published for a minimum period of time" - should we stipulate a duration? Otherwise, this could be nanoseconds;-)
Arnaud J Le Hors
I added a minimun of 2 weeks.
Christopher Ferris
Unless we are building for every platform, this seems unreasonable:
"e. release will not require users/operators to compile code to operate the base project"
Maybe should not.
Arnaud J Le Hors
As a middle ground I changed this to: "should not ... on supported platforms."
Christopher Ferris
"documentation sufficient to ensure that users/operators have clear guidance on how to get started and how to configure and operate. The operational aspects need to be correct and independently user tested"
This is rather subjective. I think that yes, the documentation should be complete and current, but I am not sure how you have it independently user tested. That would be quite an undertaking. Documentation is almost always insufficient;-) Continuous improvement and cleaning up technical debt as the project evolves are critical. As long as there is a plan and process in place that should be sufficient.
Arnaud J Le Hors
I dropped the last sentence.
Christopher Ferris
If we really aren't issuing code to be compiled, it is unclear why we need a signed source release tarball unless Hyperledger is going to provide the tools to create that from the git repos.
Dan Middleton
Yeah we might want to split some of these into a different topic. The ability to produce signed audited releases might be a topic for the ci/cd committee.
Arnaud J Le Hors
I dropped that requirement.