Issue 3: What criteria/steps/policies must releases after the First Major Release adhere to?

Vipin Bharathan

Proposed/Draft Resolution 3:

Backward compatibility, adherence to SemVer. Clear upgrade directions. Justification for breaking changes. Release notes (this seems obvious, but what is obvious to me may not be obvious to all) 

Some of these seem too strong and contradictory.  Backwards compatibility and justification for breaking changes?  I'd rather not touch this issue.

Some kind of release notes and security analysis would be a nice requirement for major releases though.

Maybe we just have one "Major Release" criteria that is applied to all major releases from the first to the nth.

Are we updating the active/incubation status rules?

I like Dan's idea. We should simply use the same criteria for every major release. So, I added a proposed resolution stating just that. Until we resolve issue 2 which is about defining what the criteria for the first major release are this doesn't give us much though.

I think that minor releases must maintain backwards compatibility.

Major releases should not need to maintain backwards compatibility. A major release is an opportunity to deprecate legacy APIs, etc that are no longer needed or used.

I incorporated Chris's suggestions.

Arnaud J Le Hors and Dan Middleton I found what I was looking for regarding follow up security audits. I remember having this discussion with the TSC and Brian before. The policy we are currently operating under is listed on this page: Security Code Audits.

Re-auditing Policy

After our projects reach 1.0 status, the policy for when we do another outside audit of a project is based on a few factors. The primary factor is code "churn"–the amount of code that has changed since the last audit. The secondary factor is major architectural changes (e.g. changing cryptography library implementations). When enough code has changed and/or architectural rework has happened, Hyperledger will invest money into having a follow up audit done to once again establish a baseline for project security.

I suggest that we encode this thinking into the lifecycle questions somehow.

It sounds reasonable to redo a security audit when extensive changes have been made, however dynamic audits or automatic (built into CII) tools for checking security are needed to back up extensive and expensive security audits.  Fortunately these tools are getting better. Good practices like chaos testing, testing in different deployment footprints etc. are needed as part of this practice.

We have to start with good coding guidelines (for example here are the coding guidelines for libra which is written in Rust: https://github.com/libra/libra/blob/master/documentation/coding_guidelines.md) and build from there. Security is a result of constant vigilance and code hygiene, some of which will have to be automated with tools. 

Unfortunately just number of lines, architecture etc. cannot  be used as a proxy for redoing security checks. A small change can cause a gaping security hole  (the fixes can also be single lines) .

Of course I am preaching to the choir, since you all are security experts including the Security Maven himself.

Vipin Bharathan you are correct about what you say. There's no one factor that we can watch and when it goes over a threshold we do another audit. It is a more comprehensive kind of decision. Maybe we should require that projects do an "intent to release" notification to the TSC for major releases and then that triggers a series of questions:

1. Does this major release contain enough changes that it justifies spending the money to re-audit it?
2. What is the recommended migration plan for existing users to move to the new release? Is it documented?
3. What are the outstanding bug counts in the critical and high category?
4. What is the marketing plan? Blog posts? Webinar?
5. Is the release notes document written and published?

If we formalize the process, then we don't require hard and fast rules like "if X lines of code changed, we do another audit".

As for the continual security situation, I agree with you too. I still have a list of outstanding "security goals" that are currently blocked by not having a stable CI solution. Both Ry and I mentioned in the TSC call this morning that there are a number of operational/transactional decisions that we'd like the TSC to make. One of those happens to be: "here is the official recommendation for the CI solution, please advise and form a policy and vote on it." I am of the opinion that we should bless one CI solution so that we can pool our resources behind it. That includes money to cover costs, building community expertise in the CI, and the security team building up a body of automated security checks and procedures.

Vipin Bharathan you and I very much agree on this. I just haven't been very vocal about it for the past year. Both Ry and I plan to engage the TSC pretty hard to straighten out a lot of the unresolved pain points in the day-to-day running of the HL community.