Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Helm project proposal #114

Merged
merged 2 commits into from Jun 1, 2018
Merged

Add Helm project proposal #114

merged 2 commits into from Jun 1, 2018

Conversation

mattfarina
Copy link
Contributor

@mattfarina mattfarina commented May 10, 2018

In addition to the proposal I wanted to add some context:

  • The helm and charts maintainers have discussed and voted on proceeding with this proposal
  • The Kubernetes steering committee has also discussed and agreed to go this route

We have tried to do our due diligence ahead of time. I'm happy to come to a TOC meeting to discuss this.

cc @bgrant0607

@caniszczyk
Copy link
Contributor

RFC @cncf/toc, we need to schedule you for a presentation if there are no objections from the TOC

@bgrant0607
Copy link
Contributor

I can provide more background during the presentation.

@technosophos
Copy link

Thanks, @mattfarina, for putting together that excellent proposal. And to @bgrant0607 for both your sponsorship and mentorship! We're really excited about this.

@caniszczyk
Copy link
Contributor

@bgrant0607 I'd slot Helm for incubation but want to check with you

@jessfraz
Copy link

+1 this :)

@bgrant0607
Copy link
Contributor

@caniszczyk Yes, incubation

@bgrant0607
Copy link
Contributor

BTW, anyone interested in technical diligence should be aware that there is a major redesign underway:
https://github.com/kubernetes-helm/community/blob/master/helm-v3/000-helm-v3.md

@caniszczyk
Copy link
Contributor

final RFC @cncf/toc and wider community, I plan on calling the incubation vote early next week

@bgrant0607
Copy link
Contributor

Helm presentation:
https://docs.google.com/presentation/d/1KNSv70fyTfSqUerCnccV7eEC_ynhLsm9A_kjnlmU_t0/edit#slide=id.g397215b9f5_41_0

To repeat answers to some questions that came up:

  • Helm is technically out of scope for Kubernetes: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/. It was made part of the Kubernetes project originally for pragmatic reasons (e.g., there was no CNCF Sandbox). Not every Kubernetes-specific project can or should be part of the Kubernetes core project.
  • Helm has stayed true to its vision of an OS package manager, such as Homebrew.
  • There are, and will be, alternate solutions for the various functions that Helm provides. For example: kubepack.com, ksonnet.io. Not to mention Terraform, Ansible, FaaSes, PaaSes, CI/CD (from Spinnaker to gitkube.sh), Operators, Service Catalog, and so on. I view that as inevitable and healthy.

@bgrant0607
Copy link
Contributor

This explanation was also posted to the chat during the meeting:
http://codeengineered.com/blog/2018/kubernetes-helm-related-tools/

An early blog post about Helm, from 2015, which described the analogy of apt and homebrew:
https://deis.com/blog/2015/why-kubernetes-needs-helm/

@bgrant0607
Copy link
Contributor

More information was posted here:
https://lists.cncf.io/g/cncf-toc/message/2026

@ghost
Copy link

ghost commented May 19, 2018

I have personally been OOTO since this PR was created, and would appreciate some additional time to review the proposal and tech due diligence. I'm back at work next week.

What's the rush here? I would assume that I'm not the only ToC member who's not fully reviewed the proposal yet? 1.5 weeks between proposal and vote feels to short to me. What do others think?

@caniszczyk
Copy link
Contributor

caniszczyk commented May 19, 2018 via email

@mattfarina
Copy link
Contributor Author

@quinton-hoole If you have any questions please don't hesitate to ask.


A goal is to consolidate all repositories under the link:https://github.com/helm[helm] GitHub org.

link:https://github.com/kubernetes/community/blob/6c3b1a6f0c1152f5e35a53ea93e692ed501abf7a/governance.md#subprojects[Kubernetes, where Helm grew up, has the concept of sub-projects]. For Kubernetes these can be ways the core Kubernetes codebase is organized as well as separate codebases, some with their own release schedules, that support Kubernetes as a whole. Under Kuberentes, Helm and its supporting projects were organized as several sub-projects. This proposal groups those supporting projects of Helm, coming from Kubernetes, as sub-projects of Helm. Sub-projects may have their own maintainers and release schedules.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Spelling: kuberentes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching that, fixing


_Note, these companies and their logos are listed on the link:https://helm.sh[Helm website]._

*Adopters*:
Copy link

@ghost ghost May 21, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CNCF Graduation Requirements state:

"Document that it is being used successfully in production by at least three independent end users which, in the TOC’s judgement, are of adequate quality and scope."

Please clarify which of the listed adopters are using Helm successfully in production, and provide some details of these uses to enable the evaluation of the "adequate quality and scope" part of the requirement.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@quinton-hoole I updated the list to include a note about who at the conference shared how they were using Helm in production. Is this sufficient or would you like me to change up the list to be those using who've publicly stated they use it in production?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Matt. What we'd like to understand is some of the detail behind the actual production use cases, to help gauge the scope and quality. For those of us who were not present at the conference, are there some slides or presentation videos that might include that info? I looked around a bit, but couldn't find anything.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, using an extreme and totally made-up example, there's a difference between e.g. "Reddit deploys their main public web site comprising 100 microservices, twice a week using Helm" vs "Reddit has an internal system that 2 people use, and it gets updated annually using Helm". So we'd need enough detail to be able to discern between these two kinds of scenarios.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you asking whether Helm is being used in production, or just want some examples to be documented?

One of the main use cases for Helm is templating applications so that they can be deployed in multiple environments, such as for testing as well as in production. People also use it to deploy off-the-shelf applications in production, such as monitoring systems.

Examples aren't hard to find.

Here's the Helm summit Reddit presentation, and the playlist, with videos from WPEngine, Ubisoft, and others:
https://www.youtube.com/watch?v=7Qxuo9W5SlY&index=7&list=PLVt9l4b66d5EjjJ_VBe_5tEiJrAGLsDb-

It's been mentioned in a number of Kubecon presentations over the years. One Ticketmaster example:
https://kccncna17.sched.com/event/CU7m/one-chart-to-rule-them-all-continuous-deployment-with-helm-at-ticketmaster-michael-goodness-raphael-deem-ticketmaster

Google search "kubernetes helm production", and you'll find blog posts about it.
https://medium.com/tailor-tech/production-grade-kubernetes-on-aws-4-tools-that-made-our-lives-easier-29438e8b58ca
https://engineering.saltside.se/building-our-helm-chart-e10da063581c
https://blog.risingstack.com/packing-a-kubernetes-microservices-with-helm/
https://medium.com/stashaway-engineering/running-a-modern-infrastructure-on-kubernetes-286ad6dab002

Or search the same text on Twitter (or helmpack production), and you'll see plenty of tweets about it, such as:
https://twitter.com/JoergHenning/status/953565524839096320
https://twitter.com/Liran_Last/status/998558781888966658

I don't think a question was specifically asked about production usage, but the Kubernetes SIG Apps survey showed about 40% of Kubernetes users using Helm:
https://docs.google.com/presentation/d/1utT0K-u1nl2apXRo29GaBvRV1x7mFLeQSgpw8mI_nGM/edit#slide=id.g38d258f7ff_0_31

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@quinton-hoole There is a YouTube playlist with the videos from the Helm Summit.

Another point, from the Kubernetes Application Survey, is that 26.7% of Helm users are using the community charts in production. This was a question we asked. Others use it as templates, or for demos, or just don't use them. We didn't collect data on the size or details of these production use cases.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I'll dig through the links Brian sent. I was hoping for a summary, saying something like:

n major production use cases are as follows:

  • Company A

    • Brief application description
    • Brief description of multiple environments it's deployed in, including size, complexity and update frequency.
  • Company B
    ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@quinton-hoole I'm happy to go into more detail. Knowing the kind of thing you're looking for will help me make sure I hit the points you'd like to see. Thanks for providing that framework. I'm not sure I can hit on all your points for all of those using it but I'll try to do my best.

For these bullets I'll continue with the companies from the Helm Summit.

  • Reddit
    • reddit.com which is in the process of moving to a SOA. The applications they are using helm for are, last I checked, stateless. It was stated that they aren't quite comfortable putting stateful services in Kubernetes, yet. They have not shared the exact microservices that are being deployed via Helm
    • Their environments include local development and amazon via multiple accounts. The environments are local dev, staging, and production. They keep building separate from production and use varying amazon controls including multiple-accounts and Kubernetes clusters in these accounts
    • Their CI is Drone and they use the drone-helm plugin. Charts are stored in S3
    • For config management they are using a project called helmfile and the workflow is essentially gitops
    • While that have stated they use Helm in production they did not say which services, how high they scale, or how often they update. Helm is used at many stages of their workflow including local dev, staging, and production
  • Ubisoft
    • Helm and Kubernetes is used for some of their games (did not list which ones).
    • Multiple clusters per production (on-premise, GKE, etc)
    • One of their goals is sharing and Helm is central to that. To share microservices between games
    • They have 60 charts with over 1,000 chart versions in their repository because they are doing continuous delivery
    • They have built tools on Helm to help automate some of their tasks
    • Helm is used as part of CI in GitLab CI
    • For configuration management they are using Terraform which has a plugin to work with Helm
  • WP Engine
    • WP Engine provides hosted WordPress where they managing more than 300,000 wordpress sites for customers
    • As of the helm summit they had more than 7,000 releases of WordPress going through Helm. A release to wordpress site is not a 1:1 mapping
    • To manage all the configuration data about each of these sites they are using a project called lostromos. Lostromos knows how to work with Helm
    • Jenkins is used for CI

Note, each of these is using Helm for package management and something else for config management. Each is using a different tool for this. In addition, each of these are using Helm in a different CI tool.

In addition to these there are other ways to look at production and users. We have user profiles to describe the different roles. One of those is Application Distributor and Bitnami is an excellent example of this who was doing application distribution prior to Kubernetes and still does for other platforms. They distribute numerous charts for common applications (e.g., mariadb, wordpress, and drupal).

If there is interest in more detail or other examples please let me know. Also, I'm not intimately familiar with these systems. If someone has a correction or more detail that information would be appreciated.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mattfarina . That addresses my question well.

@monadic
Copy link
Contributor

monadic commented May 22, 2018

Hi, this may be a bit left-field. First, I am broadly in favour of Helm being CNCF, and agree with Brian Grant's rationale above, plus I'd add that having its own lifecycle & release model & governance team is all good.

But I worry about Helm becoming all things to all people, or over-reaching in that direction. Is there value in the TOC insisting on a roadmap that shows focus?

On a related note, is there value in separating out the "chart spec" from the "implementation"? Could that lead to a higher level of discipline in thinking about what Helm does? Apologies if this is a stupid question ;/

@bgrant0607
Copy link
Contributor

@monadic Could you clarify your "all things to all people" comment?

Please see this comment:
#114 (comment)

Yes, Helm does multiple things, as do virtually all tools. But I don't see it growing in scope. It has stuck to its OS package-manager inspiration.

I also think that would be a different standard than we've applied to other projects.

As for chart spec vs implementation, there are at least a couple related efforts in Helm 3:

  1. Schematized values:
    https://github.com/kubernetes-helm/community/blob/master/helm-v3/001-charts.md#schematized-values-files

That CRD format could potentially be adopted by other tools.

  1. Application metadata CRD is being developed to be used by Helm and other tools:
    https://github.com/kubernetes-sigs/application

@monadic
Copy link
Contributor

monadic commented May 22, 2018

package management, templates, dependencies, config management, app def. it's a slippery slope if you don't have boundaries.

in http://codeengineered.com/blog/2018/kubernetes-helm-related-tools/ it is mooted that helm is all-enclosing -- a whole layer of the stack, with everything above it also depending on it

i'm not sure that OS packages are even the right metaphor

@mattfarina
Copy link
Contributor Author

@monadic Helm fills the same layer in the stack that apt does on Debian and Debian based distros (e.g., Ubuntu). For example, apt will collect the root MySQL password when you install it plus put the right bits in the right places. Apt will also install dependencies or an application that has them. So do other dependency managers like homebrew.

Helm has a clear boundary of package management in the same vein of existing package managers (OS, programming language, etc).

Kubernetes is billed as a datacenter (or cluster) operating system. See the idea of the datacenter as a computer put out by Google and the way borg fits in. That's why we talk about the relationship with OS package managers. It's a similar concept.

Helm is not trying to be a config management tool. Some people will try to bend it that way. As the k8s ecosystem grows I hope we get more and more mature config management solutions. I'll admit, I've worked at places that tired to use apt for some config management. People do try to bend things a little too far at times. Even outside their intent.

Application definition is definitely not part of Helm. Kubernetes SIG Apps owns that work scope and it's a different codebase and project. Helm's plan is to leverage that when it's ready the appropriate way.

Does this help clarify things?

@monadic
Copy link
Contributor

monadic commented May 23, 2018

Thanks @mattfarina

OK - thumbs up on config management & app def being out of scope. Let's keep it that way!

To repeat from my post above, I am in favour of Helm being an incubated CNCF project. For me, this DD is an opportunity to challenge assumptions and expose hidden/implicit issues. BTW, Matt, please could you comment on the idea of separating helm charts from a chart spec? Bad idea?

With that said.....

IMNSHO, it is dangerously misleading to create an entire rationale, based on an analogy. Forgive my pedantry but see also - https://en.wikipedia.org/wiki/Argument_from_analogy

The idea that helm is "like" apt style package management is fine for introducing people to concepts, just as the idea that Kubernetes is a DC operating system may help people get started. But, in some ways k8s is also "like" an app server or "like" a PaaS. Should we say Helm is "like" .war packaging, or "like" buildpacks. What mistakes might such assumptions lead us to?

Some disanalogies & concerns

  • helm is essentially a k8s client like kubectl
    helm can execute code, ditto kubectl
    apt does not, it makes binaries available
  • apt works on the local machine
    helm can execute code to a cluster anywhere
    these are not the same
  • apt doesn't do templating of your package to be installed in the same sense as helm
    what's the deal with templates? this seems a potential minefield
  • last but not least... helm: atomic? deterministic? compositional?

Saying X is a package manager may be fine for intros, but I worry that it will ultimately confuse development unless greater precision and care is used.

But, your call. I'm a future +1 anyway.

@bgrant0607
Copy link
Contributor

I'm confused by the first "disanalogy". apt does execute code. There are command hooks, and also generally applications are launched when packages are installed.

@mattfarina
Copy link
Contributor Author

@monadic You've asked some good questions here. Let me dig in a little deeper. This is long because I'm trying to add context and clearly explain things...

the idea of separating helm charts from a chart spec

We have issues in the backlog to document and version the chart spec and the repo spec. This would be in a separate specs repo.

Others, outside of Helm, have been implementing the repo specification (e.g., jForg Artifactory) so it would be good to have specs clearly documented and versioned. We like the idea of people being able to work with them and having clearer documentation on them.

It's also neat to see that Helm 3, in part, started with looking at changes that are technically spec changes to solve some of the real world problems we've encountered. To be practical, a number of experiments have been written to test those changes and see if they have a positive impact on the problem they were designed for.

There is still the idea of separating the the idea of a chart spec, altogether, from helm charts. So that the spec could be used on other things. I'm not opposed to this but, I think, we would need at least one other system (hopefully 2 other systems) to prove it out on. Without separate cases we could end up with Web SQL Databases that was only ever implemented in sqlite. We're open to the conversation but want to be practical at the same time.

it is dangerously misleading to create an entire rationale, based on an analogy.

One of the best ways for people to communicate is with analogy, according to experts. For example, the book Talk like TED dives into this. In this case I'm not trying to build a rationale based on analogy but rather communicate using analogy so folks can understand. I'm sorry if it I came across as trying to create a rationale that way.

helm is essentially a k8s client like kubectl
helm can execute code, ditto kubectl
apt does not, it makes binaries available

There are some misconceptions about how apt works. Packages contain code and apt executes it. For example, I recently pulled down the debian package for MySQL 5.5.59. I ran cloc on it and this was the output...

--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Perl                              2           1781           2421           8042
PO File                          20            613           2521           2266
Bourne Again Shell                9            114            181            552
Bourne Shell                      4             45             48            200
make                              1             38             22            177
--------------------------------------------------------------------------------
SUM:                             36           2591           5193          11237
--------------------------------------------------------------------------------

A simple example is in the post install script that's a 290 line file which is a shell script.

Apt does more than make binaries available. Same with other package managers.

apt works on the local machine
helm can execute code to a cluster anywhere
these are not the same

Sorta. This has more to do with the way the API works on different platforms and how the UX works differntly. The GNU/Linux API is a local API while Kubernetes is a REST one. The way we are expected to interact with those different APIs is currently different. So, the packages managers work in different ways based around these core differences.

On implementation, Helm currently runs a local part that talks to the Kubernetes API. It's named Tiller. This has been a pain point in practice and will be going away in Helm 3. Helm, now and in the future, will be run in the cluster sometimes and locally other times. It's more up to the user on how they want to do it.

At the core this is more about the difference interaction models and having a package manager appropriate for the one we have in Kubernetes.

apt doesn't do templating of your package to be installed in the same sense as helm
what's the deal with templates? this seems a potential minefield

If we step back and look at the "what" rather than the "how" and look at package managers we can see that many have the ability to collect information and pass that along to the install setup. To continue the apt and MySQL example, when apt installs MySQL it will collect the root password from the user and make sure that is properly setup.

In apt this is accomplished because you're on the local system. A function from dbconf, named db_input, is used to ask for this information. The source needed for this and working with Perl in the same way is located in /usr/share/debconf. This is available because you're on the local system.

How can we do this with Helm in a way the properly interacts with Kubernetes? And, do it in a way that's portable across Mac, Linux, Windows, and other operating systems? We can't assume POSIX on the local machine or in the Kubernetes container runtimes (Kubernetes now supports Windows). We can't assume shell, perl, or any of that is available. And, how can we do this in an approachable UX? If the experience was difficult or had a bunch of steps we would never have seen uptake.

Kubernetes uses objects typically described in files. An easy way to do this is with those and templates of them. The object files are documented in detail by Kubernetes so we have documentation there. Making them into templates so details can be passed to them is something people understand because templating is common. This is a solution that works and is approachable. What other solutions work, are portable, are approachable, and aren't complicated for end users?

There are issues with template systems. For example, people love to debate them and have personal preferences. There is also room for Helm to improve around them.

It's worth noting that Helm allows for other template systems to be used. A prototype plugin to helm was created to support ksonnet. The plugin author is not a ksonnet user and no one else was interested in picking up the work. We've asked others to do the same but we don't have people taking us up on it. This isn't stopping us from trying to make it easier to do. For all the talk we've experienced very little interest.

last but not least... helm: atomic? deterministic? compositional?

I'm not sure of the best way to answer this so if someone else has more to add to this please do.

Helm can only build on what Kubernetes provides. A client can't make a non-atomic database become atomic. For all of this it relies on the underlying platform. Kubernetes accepting something at the API doesn't mean it will get scheduled or that something scheduled will be able to run. And, that can change if the cluster configuration changes. For this, I'll try to focus on the API and accepting data as that is the layer Helm works at.

To be honest, I'm not entirely sure if the Kubernetes API is atomic. It's not documented.

For Helm though, when you do something (e.g., an install of a chart) and one of the objects fails (e.g., you have a deployment and service with the service failing validation at the API) the install will be marked as failed and none of the objects will be in the cluster. I'm not an expert on this bit of code to know how it works.

Is Helm deterministic? Sorta sometimes. It depends on the char. Typically, for the same inputs (if you specify everything) you'll get the same output. But, there are cases where this may not be true. For example, there are helpers. An example of a helper is, if you don't supply a release name one will be auto-generated for you. If the name is not part of the inputs than two exact same releases will have different names. This is part of the UX to make it approachable. If the name passed in is the same it will be consistent.

There are cases where chart specifically want to make sure some elements are different. For example, a common case is in generating a random UUID to pass to the application. For these charts it may not be deterministic.

This will also depend on what you're doing. Is this an install or an upgrade. Different things happen depending on these different types of actions. Installing an application twice is different from upgrading it. If you upgrade with the same input you should have the same config.

Is Helm compositional? That depends on what you mean. I'll try to share some things and see if that answers your question.

Helm has elements of composition. One way to look at it is with compositing an application of services and microservices. A application can be composed of multiple other applications where each is defined in a chart. A simple example is the wordpress chart which is composed of both wordpress and mariadb.

A second way to look at composition is with libraries that can be imported to make writing charts simpler because they inherit from a parent. We have experimented with these charts and they do work. The community charts has one called the common chart that others can import and use. While this works it has been an informal convention with a few bumps in the road. Part of the Helm 3 roadmap is to make this a first class citizen.

If there are more questions or more detail on any of these points I'd be happy to dig in.

@monadic
Copy link
Contributor

monadic commented May 23, 2018

I think we may have to agree to disagree. As I implied, none of my comments are an attempt at blocking. I'm satisfied that Helm has enough juice to justify CNCF Incubation, and I agree with Brian's reasons.

I worry about Helm though. I want it to get better and simpler.

If apt is so great, why don't we just use that? To unpick this requires us to clarify all the ways in which Kubernetes is NOT Debian Linux. I recommend doing so.

This may be in breach of Godwin's law (first person to mention Nix, loses) but imagine if we could say "Since package management operations never overwrite packages in the Nix store but just add new versions in different paths, they are atomic. So during a package upgrade, there is no time window in which the package has some files from the old version and some files from the new version — which would be bad because a program might well crash if it’s started during that period."

a

PS, re "analogy" see eg https://plato.stanford.edu/entries/reasoning-analogy/

@caniszczyk
Copy link
Contributor

@monadic @quinton-hoole and the rest of the @cncf/toc, are you OK moving forward with a vote next week or need more time for due diligence?

@ghost
Copy link

ghost commented May 25, 2018

Thanks @caniszczyk . Yes, my questions have been addressed in the updates to this proposal and I'm happy to move forward.

@monadic
Copy link
Contributor

monadic commented May 25, 2018

I'd like others in the TOC to chip in, and ask questions or indicate a go-forward. For my part, I feel Helm has done enough to be an Incubation project. Any DD or consultation at this time is an opportunity to agree some assumptions in the hope of a better and more likely success plateau.

ie. Yes.

@bcantrill
Copy link

I am basically +1 on this, but as essentially an outsider to the Kubernetes community, my question remains: why is Helm not simply part of Kubernetes? Why does it merit a separate CNCF project? I suspect that the real answer is that there are some significant points of divergence between Helm and Kubernetes; is there any substance to this? If so, it would be helpful to understand what these points of divergence are -- not to cast judgement on them, but rather to help support the case that these should be, in fact, separate projects.

@duglin
Copy link
Contributor

duglin commented May 30, 2018

@mattfarina to your no. 4 (just my 2 cents):

Is the CNCF the kind of organization that will be scaling the number of projects or is the idea to stay on the smaller side?

I'm not sure I can articulate very well why I think this, its more of a gut feel, but between the role of the TOC (which doesn't just make sure a check-list is adhered to, they vet projects for certain criteria), and that over time I see projects fading away or merging, I think the overall list of projects in the CNCF will stay smaller. I view one of the missions of the CNCF as to help guide the community to make smart choices, and while the CNCF won't necessarily pick winners I do think there will be some implicit filtering that will happen that at least narrows customer's choices down to a handful rather than a laundry list.

@bgrant0607
Copy link
Contributor

(I'm still catching up on the thread.)

I want to thank @mattfarina for patiently and thoroughly explaining the situation.

I understand the desire for more clarity about what is and isn't in scope for the kubernetes project. I'll attempt to add more detail, but it's never going to be 100% cut and dry.

  • Should it just include the kubernetes/kubernetes repo? No, as mentioned, github doesn't scale, so that's infeasible.
  • Should it just include what's in kubernetes releases? No, that's not necessary, as we link in many other projects as dependencies, but also not sufficient to actually operate the project. For instance, we need test infrastructure. And tools we built to make github scale better, which could be split into independent projects, but that would be more work than we have bandwidth for. Similarly, while there isn't One True Way to provision, bootstrap, and manage Kubernetes clusters, we need reference implementations to exercise mechanisms we add to facilitate bootstrapping, to run end-to-end tests, to run conformance tests, to provide local development environments for contributors, and so on.
  • What about things critical to making Kubernetes usable, such as documentation, a basic reference client tool (kubectl), or reference client libraries?

I could explain why a particular decision was made for every single subproject and non-subproject, but don't think that would actually be that useful. Helm, however, doesn't meet any of the criteria above.

Helm is somewhat of a special case. At the time Helm v2 started, there wasn't another vendor-neutral home for the project. CNCF didn't fully/officially exist, and certainly didn't have a notion of inception or a sandbox. Kubernetes didn't even have its incubator.

The Kubernetes project is now so big and has grown so fast that we're struggling to manage it, attempting to impose more order and uniformity and accountability, and cutting scope where we can. We froze the Kubernetes incubator last year and have since officially deprecated it. Kubernetes will likely retire some subprojects that no longer make sense. For the most part, things that are already part of the Kubernetes release can't be cut without breaking compatibility, but I've been working to clarify the boundaries, as Matt illustrated above, and we've been working hard on extensibility all over the system in order to make future scope creep less necessary.

Let me reframe the question:

Should Kubernetes have its own foundation? This is a question I've seriously asked in the past. Kubernetes has dozens of repositories, hundreds of org members, thousands of contributors, tens of thousands of members of its slack team. It's the anchor project of the CNCF, so let's assume not.

Not everything related to Kubernetes will be part of the Kubernetes project, so where do we want the "user space" for Kubernetes to live? AIUI, "user space" javascript libraries for NodeJS go to the Javascript foundation (https://js.foundation/) rather than the Nodejs foundation (https://foundation.nodejs.org/). Do we want something similar for Kubernetes? Assuming not, then that is something that the Kubernetes project needs from the CNCF.

It's fair to ask how many projects the foundation wants to have, whether it has the right tiers to scale, and whether it has the right marketing approach (e.g., differentiating strategic projects from useful ones, or platforms from components).

And the CNCF needs to make an independent judgement about the merits of specific projects, of course. In this case, does the CNCF want Helm to remain part of the CNCF? The Helm maintainers, Kubernetes Steering Committee, and Kubernetes Architecture SIG have all already agreed that Helm should officially become an independent project, for reasons that have already been covered. From a contributor standpoint and from a user standpoint, Helm has always been a separate project.

It's not a perfect metric, but Helm has 5737 github stars, which is more than:

@bgrant0607
Copy link
Contributor

To expand on "things critical to making Kubernetes usable": I mean things without which users otherwise were reverse-engineering Kubernetes to figure out, and/or copying code out of Kubernetes itself to make work.

@bgrant0607
Copy link
Contributor

That article about Kubernetes complexity (and similar articles) is getting pretty off topic.

Even before those articles, complexity has been very much on our minds. But not in the same ways, necessarily, as those authors. It sounds like that author in particular is asking Kubernetes to do MORE rather than LESS.

I'm happy to hear that many operators and SREs were at Kubecon. They are important categories of users. As one example, a number of larger enterprises enforce strict role separation where developers cannot push to production. Others build custom platforms or at least CI/CD pipelines atop Kubernetes for their developers, who then never need to use kubectl or Helm.

Developer focus: Are there applications that shouldn't or at least don't need to be run on Kubernetes? Sure. That author apparently wants a PaaS, though, or at least push to deploy. Literally dozens of other projects and products can do that on top of Kubernetes. He might find one useful. That doesn't mean that should be part of the Kubernetes project.

Generators and templates: As pointed out, the community Helm charts partly address that, and has been the biggest effort to date along those lines. OTOH, we've had trouble finding people to maintain the project's original example suite for quite some time.

Kubernetes has always been about managing containerized workloads and services, from the beginning in 2013, prior to being open-sourced, when Kubernetes had just 4 APIs. We've added more functionality, more APIs, more repos, more extension points, more provider implementations, new cluster lifecycle tools, etc., but fundamentally it's still about that, and the frequently copied/paraphrased descriptions I wrote on kubernetes.io for the 1.0 launch are still accurate.

@mattfarina
Copy link
Contributor Author

After a great talk with @bcantrill I'd like to add some more context to this issue around the relationship with Helm and Kubernetes.

  • While Kubernetes has several sub-projects that don't fit the definition of core (e.g., Kompose), I can't think of a project beyond Helm that could be a self sustaining project in its own right. The other sub-projects rely on Kubernetes where Helm doesn't need to. I think Helm fits as a sub-project of the CNCF with codebases, community, culture, the ability to setup governance, and so forth while the other sub-projects aren't at a point or moving to a point to have all of those characteristics.
  • The people who develop Helm and core Kuberentes tend to operate a little differently. Kubernetes developers tend to be more about infrastructure and infrastructure operation. Those who develop and use Helm are application developers, distributors, and operators. The culture, ways of thinking, and ways of building things can be a little different if you're if you're an infrastructure developer or an app developer.
  • Core Kubernetes is a form of infrastructure. Helm is all about apps. It's a pretty clear difference.
  • When Kubernetes decided to focus on the fuzzy term we call core there was an effort to categorize the different types of sub-project Kubernetes had to see what was what. At that time the steering committee decided to grandfather projects in that didn't fit and this included Helm. Everyone wanted a positive path forward and we have all lived amicably under the same roof. You can see this in things like one of the Helm core maintainers also being on the steering committee. The Helm maintainers were the ones that kicked off this process because we know we can operate as an independent successful project which relieves some burden from Kubernetes and gives Helm some freedom to strike out on its own. That's why I've used the analogy to a kid growing up and moving out.

I hope this adds some clarification. If there are more questions I'm happy to continue to field them.

@bgrant0607
Copy link
Contributor

As example of a judgement call for every subproject, since Kompose was mentioned:

The Kubernetes project staffed an oncall rotation to answer user questions on slack, stackoverflow, the kubernetes-users mailing list, etc.

Back in 2015-2016, we were fielding a fair number of questions from docker users exploring Kubernetes.

We created a cheatsheet for users moving from the docker CLI to kubectl:
https://kubernetes.io/docs/reference/kubectl/docker-cli-to-kubectl/

But we also got questions from docker-compose users, and answers to those questions were necessarily more complicated: pods, replication controllers, services, volumes, ...

A couple simple tools were built within the community to translate from the compose format to Kubernetes APIs, such as compose2kube:
https://stackoverflow.com/questions/37845715/can-kubernetes-be-used-like-docker-compose/40420216

We concluded that would be a better approach to help onboard such users than a long migration guide, and I believe it was:
https://stackoverflow.com/questions/46019872/docker-compose-to-kubernetes-migration?rq=1

So we joined forces on one tool, kompose:
https://kubernetes.io/blog/2016/11/kompose-tool-go-from-docker-compose-to-kubernetes/

At this point, there's not much development on the project:
https://github.com/kubernetes/kompose/graphs/contributors

And docker has its own tool for this:
https://blog.docker.com/2018/02/docker-compose-kubernetes-docker-desktop/

It's worth re-evaluating whether kompose is still needed now.

Note that the Kubernetes-focused follow-on project is not part of the Kubernetes project:
http://kedgeproject.org/
https://github.com/kedgeproject/kedge

@bgrant0607
Copy link
Contributor

As another example: Originally (2014), Kubernetes didn't really have an ecosystem, and there weren't great ways to discover what others were building on/for K8s. So, we accepted a small number of interesting tools/add-ons people in the community built into a contrib directory in the kubernetes repository (back when it was in the GoogleCloudPlatform github org).

That was one of the first things we moved out of the kubernetes repository, to https://github.com/kubernetes/contrib, which still exists, but has been deprecated since 2016.

Today, we definitely wouldn't take any subprojects we didn't absolutely need.

@ghost
Copy link

ghost commented May 30, 2018 via email

@monadic
Copy link
Contributor

monadic commented May 30, 2018

Shall we make this a topic for a toc call? say, 30 mins allocated

@ghost
Copy link

ghost commented May 30, 2018 via email

@bgrant0607
Copy link
Contributor

@monadic Do we really need to? If so, would 30 minutes resolve this?

@quinton-hoole I know you're looking for a clear, simple technical line, but I don't think that's feasible. There are other considerations, as well, which have already been discussed in this thread.

  • Support for specific applications, such as Strimzi, is absolutely out of scope for the Kubernetes project.
  • Poseidon probably should be an associated project rather than a SIG-sponsored one.
  • Dashboard is a reference client, similar to kubectl. The previous 2 UIs Kubernetes had were unmaintainable. As other similar open-source projects had UIs, it was considered table stakes, the lack of a decent UI made it harder to onboard new users to K8s, some community implementation was needed for demos and local development, and it served as a vehicle for community-driven conventions (e.g., around resource terminology).

I'm not going to address most of the others. They aren't significant and/or don't really provide standalone functionality, and several are candidates for retirement rather than for becoming top-level CNCF projects.

The only subproject in that list with many users is kops. Kubernetes doesn't have a clear vision/roadmap for its reference cluster management tools at the moment. I've asked SIG Cluster Lifecycle to create one. Once we have that, we can decide what we're doing with kube-up, kubeadm, kops, bootkube, kubespray, kube-aws, kubernetes-anywhere, cluster API, etc. We should retire some and cleanly integrate and/or unify others.

In short, I don't see any other candidates for things that would become top-level CNCF projects.

@mattfarina
Copy link
Contributor Author

@quinton-hoole Thanks for steering Strimzi the SIG Apps way. I've not heard about them, yet, and look forward to learning more.

I would be curious to know if something like Strimzi, which is focused on one application, is the kind of things the CNCF would be interested in for a project. It would help me better understand the intent of the CNCF.

One thing in the guidance that might be worth pulling apart is where it says:

Has a committed and excited team that appears to understand the challenges ahead and wishes to meet them

Many of the people who contribute to Helm do not work on other parts of Kubernetes. We've discovered this due to the number of people who are not part of the Kubernetes org. Quite a few we've ended up sponsoring so they can, for example, do more with the community charts.

Do the other sub-projects have the same split or are they regular contributors to other Kubernetes core elements? I honestly don't know but it might inform things.

Kubebuilder might be another interested example to pick apart in trying to figure out the line. Building Kubernetes controllers is hard. The goal of this is to make that experience easier alongside CRDs that work with the controller. It's a project meant to enable the ecosystem rather than be apart of it.

If there is anything I can do to help inform this decision I'm happy to help. Please feel free to let me know.

Note, for those that don't know, I co-chair SIG Apps

@bgrant0607
Copy link
Contributor

@mattfarina kubebuilder isn't necessarily just for the ecosystem. We're trying to make it easier (and less error-prone and inconsistent and difficult to maintain) for anyone, including us kubernetes project contributors, to create Kubernetes APIs and their business logic.

@monadic
Copy link
Contributor

monadic commented May 31, 2018

strawman / retcon -

A CNCF Project may have affiliated "Subprojects" that inherit governance and users from the parent Project.

Given a Project, eg Kubernetes, there are two types of Subproject:

  1. Subproject is necessary to operate Project in almost all cases
  2. Subproject is not necessary but "recommended" by Project leads

Detail on 1:

  • It is not required that every dependency be a subproject (eg. etcd/k8s)

Detail on 2:

  • Subproject is designed for use with Project and tracks updates / releases to Project.
  • Subproject implements optional functionality and is not necessary to operate Project.
  • Subproject is the "default" option, or, Subproject is one of a few "recommended" options.

Examples:

  • Kubernetes Dashboard is a commonly recommended K8s UI for CRUD management
  • Kops is a commonly used K8s provisioning tool for certain infrastructures

Note that both kops and dashboard pretty much have to upgrade when k8s does.

The decision on Subproject status lies with the Project leads.

The TOC recommends that:

  1. Projects publish guidelines on any additional criteria they use. For example "a Subproject should be used in over 25% of deployments".
  2. All Subprojects maintain a similar velocity to their parent, or be pruned by the Project leads, unless there is a good reason to keep them.

The CNCF accepts independent Projects that fall outside the above patterns:

  1. Sandbox projects that are new, experimental or relatively immature. An unusual but possible path is that Sandbox projects mature into Subprojects via the Graduation process. This would be by agreement between the TOC and all involved Project leads.
  2. Projects that have standalone use cases independent of other CNCF Projects. For example, Prometheus is a standalone monitoring, alerting and time series tool. It is not "Kubernetes monitoring" even if that is a common use case.
  3. Projects that are optional add-ons for another Project, but which do not fit into the Subproject pattern. Such Projects should ideally have sufficient release momentum and community to justify independent status and governance.

Examples of 3:

  • a Subproject that "outgrows" its parent because the community has its own identity, or an independent roadmap & release train
  • a Subproject wishes to support more than one Project, ie. it no longer has one 'parent'

@bgrant0607
Copy link
Contributor

@monadic

Kubernetes has a notion of subprojects, which we're working to better document and implement in our project automation. More on this can be found here:
https://github.com/kubernetes/community/issues/1673

Currently identified subprojects are enumerated under their SIGs here:
https://github.com/kubernetes/community/blob/master/sigs.yaml

@bgrant0607
Copy link
Contributor

@monadic I think what you suggested makes sense and mostly matches the Kubernetes notion. Part of our problem was that we didn't have governance and structure in place from the beginning, so the main project and Helm independently evolved their own ways of doing things.

@monadic
Copy link
Contributor

monadic commented May 31, 2018

@bgrant0607 thank-you. Would it make sense to merge the notions, or make them 100% consistent? I was trying to define something generic eg for Prometheus subprojects too.

@mattfarina & others -- thanks for all the patient contributions to this thread.

@bgrant0607 @quinton-hoole @bcantrill and other TOC voters -- can we declare victory on the merits of Helm and move the principles-of-subprojects to be its own topic?

@barkerd427
Copy link

It was stated in these comments that Helm is independent of Kubernetes. I understood the context to mean that the culture/work/history were independent. However, is Helm not dependent technically on Kubernetes? I believe Rook and Telepresence fall into this pattern as well. There have been critics of the CNCF who have jokingly called it the Kubernetes Foundation, and including projects that can't operate outside of Kubernetes appears to enforce that perception. It might be beneficial to require projects in the CNCF to have multiple primary platforms. A spec is required to be implemented in more than one system, so why shouldn't a tool be required to work with more than one primary system. Perhaps adding Mesos support or Swarm. Otherwise, I believe the CNCF will unintentionally exclude non-k8s-centric projects that might see the CNCF as the Kubernetes Foundation.

@ghost
Copy link

ghost commented May 31, 2018 via email

@monadic
Copy link
Contributor

monadic commented May 31, 2018

@barkerd427 thanks. I think that can work for projects lower down the app stack, but not for app def tools so easily.

we may need a big new metaproject and org... "app def tools"

@quinton-hoole great! now just the two bri/yans to go?

@ghost
Copy link

ghost commented May 31, 2018 via email

@mattfarina
Copy link
Contributor Author

As I thought more about this problem space I started to think more about the Apache foundation. It has a lot of projects, some of which are platforms. Some other projects build upon platforms and are specific to them. Because there are so many projects they do provide some taxonomy to navigate them.

This leads me with questions around the number of projects, scope control, and direction the CNCF wants to go in. My 2 cents is to have more projects and be an enabler for cross vendor projects that target the CNCF definition of cloud native. That project leaders could even help to mentor other project leaders and they try to solve problems with their experience.

There is a need for a vendor neutral place for Kubernetes specific things, that are not part of Kuberentes, to collaborate. If the CNCF is not the right place for that it's important to communicate so a proper place be formed or found.

@bgrant0607
Copy link
Contributor

@monadic Yes, I'd like to decouple the Helm decision from broader discussions about subprojects (K8s is more complex than all other current and likely future CNCF projects due to scale and scope), additional project categorizations, general project portfolio strategy/principles, and other more general topics.

I agree with @quinton-hoole that emergence of Kubernetes-specific projects is inevitable at this point given the magnitude of user interest (e.g., 200k people signed up for meetups) and breadth of industry support.

@barkerd427
Copy link

@quinton-hoole, my point here was not what is written down, but rather what's interpreted as included. If I was in charge of a project that was Mesos-centric, then I probably wouldn't look to the CNCF if I hadn't implemented a k8s integration. I might look to Apache. This might be inadvertently causing the CNCF and k8s to miss out on projects that would otherwise be great contributions to the Cloud Native community. If a new issue is created to discuss further segmenting or decomposing projects, then I'm happy to continue the conversation there. I'm not sure what's appropriate protocol here.

@monadic
Copy link
Contributor

monadic commented May 31, 2018

@barkerd427 most of the current CNCF projects are just as useful for k8s, swarm, mesos. Why would you look somewhere else?

@ghost
Copy link

ghost commented May 31, 2018 via email

@bgrant0607
Copy link
Contributor

This is getting pretty off topic (Helm), but Apache is almost 20 years old. At this point, it has a fairly large ecosystem of projects (~200?, not including incubating, labs, retired, etc.) that is broader than just Hadoop and other data-processing projects, with multiple "anchor" projects. They have more categories of projects than we have projects so far.

Not that we wouldn't take Mesos-related projects, but since Mesos is in the Apache Foundation, it may well make sense for related projects to also be in the Apache Foundation, as that may make code ownership, licensing, and contribution simpler, for instance. Certainly that issue has come up for CNCF projects.

@dankohn
Copy link
Contributor

dankohn commented May 31, 2018 via email

@caniszczyk caniszczyk merged commit 1269441 into cncf:master Jun 1, 2018
@caniszczyk
Copy link
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet