Version after Version; How the Open Source Project Kubernetes Releases its Software - 29/05/2023

We look at the ways the Kubernetes project releases its software, from how the community works to detailed practices maintained in the community

In this article, we look at how the open source project Kubernetes manages its software releases. By exploring the established and evolved community structures, this discussion will delve into the collaborative dynamics that underpin the management of Kubernetes releases, shifting the focus away from Kubernetes itself.

Open Source is fascinating ✨

2023-05-cop-version-after-version-kubernetes-release-management-worldwide-community

Open source is undeniably fascinating. Just think about how big projects like Kubernetes are, where thousands of people around the world come together to build software that is changing entire industries. The contributors devote their free time and passion to work on this project, alongside their professional and personal commitments. Through initiatives like Kubernetes, the cloud-native field has emerged, revolutionizing how applications are developed and deployed.

Open source serves as a powerful driver of innovation. By embracing open source, companies can tap into a pool of collective knowledge and expertise, benefiting from advancements and fresh perspectives that emerge from collaborative efforts. Neglecting open source means potentially missing out on innovation and talent that could propel a company ahead of its competitors. Therefore, actively contributing to open source projects is not only a way to stay connected with the latest advancements, but also a means to actively participate in shaping the future of technology.

Open source challenges traditional hierarchies and structures by letting different voices and ideas come together in a democratic process. The democratization and inclusion of principles in software development may encourage companies to adopt similar approaches to their internal source projects. Establishing a culture of collaboration, transparency, and ownership within an organization can foster greater creativity and innovation within the organization.

The appeal of open source lies not only in its ability to drive innovation, but also in the opportunities it presents for personal and professional growth. Participating in open source projects lets people learn from top-notch professionals and connect with a global community of people who are passionate about pushing the boundaries of technology.

Challenges in Open Source Software Projects

It’s fun to work on open source software projects, but it’s important to know that they have their own problems. The chaotic nature of open source software development is one of the most intriguing aspects, and it’s also where some obstacles come from.

While there are certainly additional challenges in the realm of open source, such as licensing and intellectual property, the aforementioned list should provide an adequate foundation for proceeding. Addressing these challenges is essential to the successful management of releases within the Kubernetes project.

Kubernetes Community Structure: SIG-Town Analogy

2023-05-cop-version-after-version-kubernetes-release-management-kubernetes-sig-town

Inspired by Paris Pittman’s insightful analogy in a recent episode of the Kubernetes Podcast by Google, the Kubernetes Community can be compared to a city. This analogy resonated and led to the creation of the SIG-Town diagram, illustrating the organizational structures within the community.

The Kubernetes Community is vast and diverse. The CNCF Devstats data highlights its scale, with nearly 3,000 individual contributors and approximately 60,000 commits recorded in the project over the past 12 months. Managing such a large and active community, driving decisions, fostering inclusivity, and ensuring effective collaboration can be a significant challenge. It’s important to learn about the Kubernetes Community’s different SIGs, roles, and processes. The SIG-Town analogy shows how big, complicated, and patient the Kubernetes Community is.

Collaborate

To make open-source projects work well together, it is important to set rules and guidelines that we all agree on. These rules help everyone work together in a good way.

Concepts of Cross-Cutting Collaboration

Improving collaboration is key in open-source projects like Kubernetes. Here are some concepts that help people work together in the community:

  1. Everything is a file: every contribution is managed through the version control system, Git. This means that changes, discussions, and decision-making processes revolve around files within the project’s repository. Recognizing and honoring the contributions made by project contributors is essential. But it’s worth noting that in some discussions, it can be hard to keep everything in a markdown file or similar format. In such cases, opting for alternative platforms like public Google Docs files or GitHub issues proves more suitable and conducive to effective collaboration.
  2. Ownership and Maintainer Roles: without maintainers taking ownership, the community would not be able to agree on decisions and ensure the project matures. Ownership is defined in files OWNERS that list approvers and reviewers. Approvers and reviewers take ownership of specific areas and contribute their expertise to guide and review contributions. There are other roles next to code approver and reviewer roles like SIG Chairs, Release Team Roles, etc. Serving in these roles implies a commitment to the community for a specific duration.
  3. Meritocracy: recognition and influence are earned based on demonstrated skills, knowledge, and contributions. This approach emphasizes the importance of individual contributions rather than hierarchical positions or affiliations.
  4. Transparency and Openness: includes open communication, decision-making processes, and community-driven initiatives. Encouraging participation from others and giving them a platform to express their opinions and concerns is vital. It is important to avoid private direct messages (DMs) and ensure that all communication happens publicly.

Collaboration in open-source projects is a vast topic, and there is much more to explore. However, understanding these key concepts provides a solid foundation for comprehending the Kubernetes release processes and how collaboration operates within the community.

Kubernetes Enhancement Proposals (KEPs) - The “Feature Design Contract”

In the Kubernetes Project, the management of contributions is facilitated through the creation of Kubernetes Enhancement Proposals (KEPs). A KEP serves as a design document that outlines the desired feature to be implemented. It acts as a contractual agreement, allowing for evaluation, discussion, and assessment of proposed changes. The KEP process is mandatory for any modifications made to Kubernetes. Simply opening a pull request (PR) to introduce support for a new protocol or similar changes is not sufficient. To ensure proper documentation, the provided template below illustrates the required structure and information that needs to be included in a KEP.

    Summary
    Motivation
    - Goals
    - Non-Goals
    Proposal
    - User Stories (Optional)
    - Notes/Constraints/Caveats (Optional)
    - Risks and Mitigations
    Design Details
    - Test Plan
    	- Prerequisite testing updates
    	- Unit tests
    	- Integration tests
    	- e2e tests
    - Graduation Criteria
    - Upgrade / Downgrade Strategy
    - Version Skew Strategy
    Production Readiness Review Questionnaire
    - Feature Enablement and Rollback
    - Rollout, Upgrade and Rollback Planning
    - Monitoring Requirements
    - Dependencies
    - Scalability
    - Troubleshooting
    Implementation History
    Drawbacks
    Alternatives
    Infrastructure Needed (Optional)

KEPs follow the concept and principles of the Google Design Document. Just like Google Design Documents, KEPs serve as a structured framework for proposing and documenting feature enhancements in the Kubernetes Project. They provide a systematic approach for outlining the design, rationale, and implementation details of a proposed feature.

2023-05-cop-version-after-version-kubernetes-release-management-kep-mess

Each Kubernetes Special Interest Group (SIG) is responsible for managing and overseeing its respective Kubernetes Enhancement Proposals (KEPs). These KEPs can be in various stages of development within the SIG’s purview. They may be under discussion, being considered for inclusion in a release (opted-in), or have already been merged and categorized as alpha, beta, or stable, indicating different levels of maturity and readiness. KEPs can vary significantly in size, but generally, smaller KEPs that enable atomic changes are preferred.

SIG-Release

As mentioned earlier, managing and monitoring all Kubernetes Enhancement Proposals (KEPs) and related discussions is an incredibly challenging task. SIG-Release takes on this responsibility, as outlined in its charter. The primary objectives of SIG-Release are to ensure the delivery of high-quality Kubernetes releases, continuously enhance release and development processes, and foster collaboration with downstream communities that utilize Kubernetes releases to build their own artifacts.

Release Team Structure

Within sig-release, there are two teams: the release engineering team, responsible for developing release tooling, with a more stable team of contributors, and the release team, which changes with each cycle. The primary role of the release team is to facilitate the release process.

2023-05-cop-version-after-version-kubernetes-release-management-teams

Referencing the diagram above, the team consists of Leads and Shadows who oversee different areas of interest throughout the release process. The shadow program serves as an onboarding pathway to Sig-release, the Kubernetes community, and the wider open-source realm. You can contribute to a release cycle by assisting in tasks such as collecting enhancements, monitoring test pipelines, and more. You can find out more about the release team roles and team here.

Contributing to open-source projects extends beyond coding. You can make valuable contributions through documentation, community engagement, operations, governance, infrastructure work, and various other areas of work. The release team is a prime example.

Overview of the Release Cycle

A typical Kubernetes release cycle spans approximately three months and consists of several key deadlines throughout this period. You can find the release timeline for the v1.28 release cycle here. In past releases, the milestones and deadlines have not changed significantly, and the schedule is fairly consistent.

Every Kubernetes Release Cycle demands a significant amount of effort. It involves numerous individuals, multiple deadlines, and a multitude of tasks that must be completed. The release lead is responsible for maintaining a comprehensive tracking issue for the entire release cycle, which currently includes more than 100 checkboxes representing various “formal” tasks. Additionally, a considerable amount of extra work, especially in terms of communication and enhancing the release team’s performance from one cycle to another, is necessary.

During my tenure as the release team lead for version 1.26, I vividly recall receiving a notification from the Slack bot that I have sent over 600 messages in a single week, which clearly indicates the substantial workload involved. The team continuously strives to enhance the overall experience and automate tasks as much as possible, but such improvements require time and iterative efforts across multiple cycles.

Incorporating KEPs into the release process: How does it work?

2023-05-cop-version-after-version-kubernetes-release-management-kep-process

Ensuring the inclusion of KEPs (Kubernetes Enhancement Proposals) in a Kubernetes release involves two key aspects that deserve attention. Firstly, there are tasks that the KEP author must undertake, and secondly, there are deadlines and tasks that the release team is responsible for.

To begin with, it is crucial for the KEP author to engage in discussions about the KEP with their relevant SIG (Special Interest Group). The SIG leads plays a pivotal role in opting the KEP into the release. The KEP author must diligently complete all the necessary tasks to prepare the KEP for inclusion. This includes obtaining approval and merging from the SIG, opening pull requests (PRs) to update the code, ensuring code review and approval, and updating the documentation. Additionally, the KEP author may choose to write a blog post to provide further insights. The release team ensures that the KEP author completes these steps and closely monitors the process to prevent the SIG leads responsible for the KEP from missing anything.

Why are certain features not included in releases?

because-its-open-source

It is important to remember that most maintainers contribute to the project in their spare time. This is a volunteer work and they are free to spend as much time as they want on things of their choice.

If you notice a missing feature, you have the opportunity to contribute yourself. Contact your engineering manager at work and explain the significance of this feature for your company’s IT systems. Investing in the Kubernetes project not only benefits your company but also allows you to collaborate with talented engineers and potentially gain valuable knowledge in the process. Moreover, your contributions can benefit the wider cloud native community, as they may find your changes extremely useful.

That’s it! If you would like to contact me, the best way to do so is via Slack in the Kubernetes or CNCF workspaces.