New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Helm project proposal #114
Conversation
RFC @cncf/toc, we need to schedule you for a presentation if there are no objections from the TOC |
I can provide more background during the presentation. |
Thanks, @mattfarina, for putting together that excellent proposal. And to @bgrant0607 for both your sponsorship and mentorship! We're really excited about this. |
@bgrant0607 I'd slot Helm for incubation but want to check with you |
+1 this :) |
@caniszczyk Yes, incubation |
BTW, anyone interested in technical diligence should be aware that there is a major redesign underway: |
final RFC @cncf/toc and wider community, I plan on calling the incubation vote early next week |
Helm presentation: To repeat answers to some questions that came up:
|
This explanation was also posted to the chat during the meeting: An early blog post about Helm, from 2015, which described the analogy of apt and homebrew: |
More information was posted here: |
I have personally been OOTO since this PR was created, and would appreciate some additional time to review the proposal and tech due diligence. I'm back at work next week. What's the rush here? I would assume that I'm not the only ToC member who's not fully reviewed the proposal yet? 1.5 weeks between proposal and vote feels to short to me. What do others think? |
We average about a week or two before we call for a vote. If you want to wait another week we can do that too cc: @cncf/toc
|
@quinton-hoole If you have any questions please don't hesitate to ask. |
proposals/helm.adoc
Outdated
|
||
A goal is to consolidate all repositories under the link:https://github.com/helm[helm] GitHub org. | ||
|
||
link:https://github.com/kubernetes/community/blob/6c3b1a6f0c1152f5e35a53ea93e692ed501abf7a/governance.md#subprojects[Kubernetes, where Helm grew up, has the concept of sub-projects]. For Kubernetes these can be ways the core Kubernetes codebase is organized as well as separate codebases, some with their own release schedules, that support Kubernetes as a whole. Under Kuberentes, Helm and its supporting projects were organized as several sub-projects. This proposal groups those supporting projects of Helm, coming from Kubernetes, as sub-projects of Helm. Sub-projects may have their own maintainers and release schedules. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Spelling: kuberentes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for catching that, fixing
|
||
_Note, these companies and their logos are listed on the link:https://helm.sh[Helm website]._ | ||
|
||
*Adopters*: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CNCF Graduation Requirements state:
"Document that it is being used successfully in production by at least three independent end users which, in the TOC’s judgement, are of adequate quality and scope."
Please clarify which of the listed adopters are using Helm successfully in production, and provide some details of these uses to enable the evaluation of the "adequate quality and scope" part of the requirement.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@quinton-hoole I updated the list to include a note about who at the conference shared how they were using Helm in production. Is this sufficient or would you like me to change up the list to be those using who've publicly stated they use it in production?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks Matt. What we'd like to understand is some of the detail behind the actual production use cases, to help gauge the scope and quality. For those of us who were not present at the conference, are there some slides or presentation videos that might include that info? I looked around a bit, but couldn't find anything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be clear, using an extreme and totally made-up example, there's a difference between e.g. "Reddit deploys their main public web site comprising 100 microservices, twice a week using Helm" vs "Reddit has an internal system that 2 people use, and it gets updated annually using Helm". So we'd need enough detail to be able to discern between these two kinds of scenarios.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you asking whether Helm is being used in production, or just want some examples to be documented?
One of the main use cases for Helm is templating applications so that they can be deployed in multiple environments, such as for testing as well as in production. People also use it to deploy off-the-shelf applications in production, such as monitoring systems.
Examples aren't hard to find.
Here's the Helm summit Reddit presentation, and the playlist, with videos from WPEngine, Ubisoft, and others:
https://www.youtube.com/watch?v=7Qxuo9W5SlY&index=7&list=PLVt9l4b66d5EjjJ_VBe_5tEiJrAGLsDb-
It's been mentioned in a number of Kubecon presentations over the years. One Ticketmaster example:
https://kccncna17.sched.com/event/CU7m/one-chart-to-rule-them-all-continuous-deployment-with-helm-at-ticketmaster-michael-goodness-raphael-deem-ticketmaster
Google search "kubernetes helm production", and you'll find blog posts about it.
https://medium.com/tailor-tech/production-grade-kubernetes-on-aws-4-tools-that-made-our-lives-easier-29438e8b58ca
https://engineering.saltside.se/building-our-helm-chart-e10da063581c
https://blog.risingstack.com/packing-a-kubernetes-microservices-with-helm/
https://medium.com/stashaway-engineering/running-a-modern-infrastructure-on-kubernetes-286ad6dab002
Or search the same text on Twitter (or helmpack production), and you'll see plenty of tweets about it, such as:
https://twitter.com/JoergHenning/status/953565524839096320
https://twitter.com/Liran_Last/status/998558781888966658
I don't think a question was specifically asked about production usage, but the Kubernetes SIG Apps survey showed about 40% of Kubernetes users using Helm:
https://docs.google.com/presentation/d/1utT0K-u1nl2apXRo29GaBvRV1x7mFLeQSgpw8mI_nGM/edit#slide=id.g38d258f7ff_0_31
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@quinton-hoole There is a YouTube playlist with the videos from the Helm Summit.
Another point, from the Kubernetes Application Survey, is that 26.7% of Helm users are using the community charts in production. This was a question we asked. Others use it as templates, or for demos, or just don't use them. We didn't collect data on the size or details of these production use cases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, I'll dig through the links Brian sent. I was hoping for a summary, saying something like:
n major production use cases are as follows:
-
Company A
- Brief application description
- Brief description of multiple environments it's deployed in, including size, complexity and update frequency.
-
Company B
...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@quinton-hoole I'm happy to go into more detail. Knowing the kind of thing you're looking for will help me make sure I hit the points you'd like to see. Thanks for providing that framework. I'm not sure I can hit on all your points for all of those using it but I'll try to do my best.
For these bullets I'll continue with the companies from the Helm Summit.
- Reddit
- reddit.com which is in the process of moving to a SOA. The applications they are using helm for are, last I checked, stateless. It was stated that they aren't quite comfortable putting stateful services in Kubernetes, yet. They have not shared the exact microservices that are being deployed via Helm
- Their environments include local development and amazon via multiple accounts. The environments are local dev, staging, and production. They keep building separate from production and use varying amazon controls including multiple-accounts and Kubernetes clusters in these accounts
- Their CI is Drone and they use the drone-helm plugin. Charts are stored in S3
- For config management they are using a project called helmfile and the workflow is essentially gitops
- While that have stated they use Helm in production they did not say which services, how high they scale, or how often they update. Helm is used at many stages of their workflow including local dev, staging, and production
- Ubisoft
- Helm and Kubernetes is used for some of their games (did not list which ones).
- Multiple clusters per production (on-premise, GKE, etc)
- One of their goals is sharing and Helm is central to that. To share microservices between games
- They have 60 charts with over 1,000 chart versions in their repository because they are doing continuous delivery
- They have built tools on Helm to help automate some of their tasks
- Helm is used as part of CI in GitLab CI
- For configuration management they are using Terraform which has a plugin to work with Helm
- WP Engine
- WP Engine provides hosted WordPress where they managing more than 300,000 wordpress sites for customers
- As of the helm summit they had more than 7,000 releases of WordPress going through Helm. A release to wordpress site is not a 1:1 mapping
- To manage all the configuration data about each of these sites they are using a project called lostromos. Lostromos knows how to work with Helm
- Jenkins is used for CI
Note, each of these is using Helm for package management and something else for config management. Each is using a different tool for this. In addition, each of these are using Helm in a different CI tool.
In addition to these there are other ways to look at production and users. We have user profiles to describe the different roles. One of those is Application Distributor and Bitnami is an excellent example of this who was doing application distribution prior to Kubernetes and still does for other platforms. They distribute numerous charts for common applications (e.g., mariadb, wordpress, and drupal).
If there is interest in more detail or other examples please let me know. Also, I'm not intimately familiar with these systems. If someone has a correction or more detail that information would be appreciated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @mattfarina . That addresses my question well.
Hi, this may be a bit left-field. First, I am broadly in favour of Helm being CNCF, and agree with Brian Grant's rationale above, plus I'd add that having its own lifecycle & release model & governance team is all good. But I worry about Helm becoming all things to all people, or over-reaching in that direction. Is there value in the TOC insisting on a roadmap that shows focus? On a related note, is there value in separating out the "chart spec" from the "implementation"? Could that lead to a higher level of discipline in thinking about what Helm does? Apologies if this is a stupid question ;/ |
@monadic Could you clarify your "all things to all people" comment? Please see this comment: Yes, Helm does multiple things, as do virtually all tools. But I don't see it growing in scope. It has stuck to its OS package-manager inspiration. I also think that would be a different standard than we've applied to other projects. As for chart spec vs implementation, there are at least a couple related efforts in Helm 3:
That CRD format could potentially be adopted by other tools.
|
package management, templates, dependencies, config management, app def. it's a slippery slope if you don't have boundaries. in http://codeengineered.com/blog/2018/kubernetes-helm-related-tools/ it is mooted that helm is all-enclosing -- a whole layer of the stack, with everything above it also depending on it i'm not sure that OS packages are even the right metaphor |
@monadic Helm fills the same layer in the stack that apt does on Debian and Debian based distros (e.g., Ubuntu). For example, apt will collect the root MySQL password when you install it plus put the right bits in the right places. Apt will also install dependencies or an application that has them. So do other dependency managers like homebrew. Helm has a clear boundary of package management in the same vein of existing package managers (OS, programming language, etc). Kubernetes is billed as a datacenter (or cluster) operating system. See the idea of the datacenter as a computer put out by Google and the way borg fits in. That's why we talk about the relationship with OS package managers. It's a similar concept. Helm is not trying to be a config management tool. Some people will try to bend it that way. As the k8s ecosystem grows I hope we get more and more mature config management solutions. I'll admit, I've worked at places that tired to use apt for some config management. People do try to bend things a little too far at times. Even outside their intent. Application definition is definitely not part of Helm. Kubernetes SIG Apps owns that work scope and it's a different codebase and project. Helm's plan is to leverage that when it's ready the appropriate way. Does this help clarify things? |
Thanks @mattfarina OK - thumbs up on config management & app def being out of scope. Let's keep it that way! To repeat from my post above, I am in favour of Helm being an incubated CNCF project. For me, this DD is an opportunity to challenge assumptions and expose hidden/implicit issues. BTW, Matt, please could you comment on the idea of separating helm charts from a chart spec? Bad idea? With that said..... IMNSHO, it is dangerously misleading to create an entire rationale, based on an analogy. Forgive my pedantry but see also - https://en.wikipedia.org/wiki/Argument_from_analogy The idea that helm is "like" apt style package management is fine for introducing people to concepts, just as the idea that Kubernetes is a DC operating system may help people get started. But, in some ways k8s is also "like" an app server or "like" a PaaS. Should we say Helm is "like" .war packaging, or "like" buildpacks. What mistakes might such assumptions lead us to? Some disanalogies & concerns
Saying X is a package manager may be fine for intros, but I worry that it will ultimately confuse development unless greater precision and care is used. But, your call. I'm a future +1 anyway. |
I'm confused by the first "disanalogy". apt does execute code. There are command hooks, and also generally applications are launched when packages are installed. |
@monadic You've asked some good questions here. Let me dig in a little deeper. This is long because I'm trying to add context and clearly explain things...
We have issues in the backlog to document and version the chart spec and the repo spec. This would be in a separate specs repo. Others, outside of Helm, have been implementing the repo specification (e.g., jForg Artifactory) so it would be good to have specs clearly documented and versioned. We like the idea of people being able to work with them and having clearer documentation on them. It's also neat to see that Helm 3, in part, started with looking at changes that are technically spec changes to solve some of the real world problems we've encountered. To be practical, a number of experiments have been written to test those changes and see if they have a positive impact on the problem they were designed for. There is still the idea of separating the the idea of a chart spec, altogether, from helm charts. So that the spec could be used on other things. I'm not opposed to this but, I think, we would need at least one other system (hopefully 2 other systems) to prove it out on. Without separate cases we could end up with Web SQL Databases that was only ever implemented in sqlite. We're open to the conversation but want to be practical at the same time.
One of the best ways for people to communicate is with analogy, according to experts. For example, the book Talk like TED dives into this. In this case I'm not trying to build a rationale based on analogy but rather communicate using analogy so folks can understand. I'm sorry if it I came across as trying to create a rationale that way.
There are some misconceptions about how apt works. Packages contain code and apt executes it. For example, I recently pulled down the debian package for MySQL 5.5.59. I ran cloc on it and this was the output...
A simple example is in the post install script that's a 290 line file which is a shell script. Apt does more than make binaries available. Same with other package managers.
Sorta. This has more to do with the way the API works on different platforms and how the UX works differntly. The GNU/Linux API is a local API while Kubernetes is a REST one. The way we are expected to interact with those different APIs is currently different. So, the packages managers work in different ways based around these core differences. On implementation, Helm currently runs a local part that talks to the Kubernetes API. It's named Tiller. This has been a pain point in practice and will be going away in Helm 3. Helm, now and in the future, will be run in the cluster sometimes and locally other times. It's more up to the user on how they want to do it. At the core this is more about the difference interaction models and having a package manager appropriate for the one we have in Kubernetes.
If we step back and look at the "what" rather than the "how" and look at package managers we can see that many have the ability to collect information and pass that along to the install setup. To continue the apt and MySQL example, when apt installs MySQL it will collect the root password from the user and make sure that is properly setup. In apt this is accomplished because you're on the local system. A function from dbconf, named How can we do this with Helm in a way the properly interacts with Kubernetes? And, do it in a way that's portable across Mac, Linux, Windows, and other operating systems? We can't assume POSIX on the local machine or in the Kubernetes container runtimes (Kubernetes now supports Windows). We can't assume shell, perl, or any of that is available. And, how can we do this in an approachable UX? If the experience was difficult or had a bunch of steps we would never have seen uptake. Kubernetes uses objects typically described in files. An easy way to do this is with those and templates of them. The object files are documented in detail by Kubernetes so we have documentation there. Making them into templates so details can be passed to them is something people understand because templating is common. This is a solution that works and is approachable. What other solutions work, are portable, are approachable, and aren't complicated for end users? There are issues with template systems. For example, people love to debate them and have personal preferences. There is also room for Helm to improve around them. It's worth noting that Helm allows for other template systems to be used. A prototype plugin to helm was created to support ksonnet. The plugin author is not a ksonnet user and no one else was interested in picking up the work. We've asked others to do the same but we don't have people taking us up on it. This isn't stopping us from trying to make it easier to do. For all the talk we've experienced very little interest.
I'm not sure of the best way to answer this so if someone else has more to add to this please do. Helm can only build on what Kubernetes provides. A client can't make a non-atomic database become atomic. For all of this it relies on the underlying platform. Kubernetes accepting something at the API doesn't mean it will get scheduled or that something scheduled will be able to run. And, that can change if the cluster configuration changes. For this, I'll try to focus on the API and accepting data as that is the layer Helm works at. To be honest, I'm not entirely sure if the Kubernetes API is atomic. It's not documented. For Helm though, when you do something (e.g., an install of a chart) and one of the objects fails (e.g., you have a deployment and service with the service failing validation at the API) the install will be marked as failed and none of the objects will be in the cluster. I'm not an expert on this bit of code to know how it works. Is Helm deterministic? Sorta sometimes. It depends on the char. Typically, for the same inputs (if you specify everything) you'll get the same output. But, there are cases where this may not be true. For example, there are helpers. An example of a helper is, if you don't supply a release name one will be auto-generated for you. If the name is not part of the inputs than two exact same releases will have different names. This is part of the UX to make it approachable. If the name passed in is the same it will be consistent. There are cases where chart specifically want to make sure some elements are different. For example, a common case is in generating a random UUID to pass to the application. For these charts it may not be deterministic. This will also depend on what you're doing. Is this an install or an upgrade. Different things happen depending on these different types of actions. Installing an application twice is different from upgrading it. If you upgrade with the same input you should have the same config. Is Helm compositional? That depends on what you mean. I'll try to share some things and see if that answers your question. Helm has elements of composition. One way to look at it is with compositing an application of services and microservices. A application can be composed of multiple other applications where each is defined in a chart. A simple example is the wordpress chart which is composed of both wordpress and mariadb. A second way to look at composition is with libraries that can be imported to make writing charts simpler because they inherit from a parent. We have experimented with these charts and they do work. The community charts has one called the common chart that others can import and use. While this works it has been an informal convention with a few bumps in the road. Part of the Helm 3 roadmap is to make this a first class citizen. If there are more questions or more detail on any of these points I'd be happy to dig in. |
I think we may have to agree to disagree. As I implied, none of my comments are an attempt at blocking. I'm satisfied that Helm has enough juice to justify CNCF Incubation, and I agree with Brian's reasons. I worry about Helm though. I want it to get better and simpler. If apt is so great, why don't we just use that? To unpick this requires us to clarify all the ways in which Kubernetes is NOT Debian Linux. I recommend doing so. This may be in breach of Godwin's law (first person to mention Nix, loses) but imagine if we could say "Since package management operations never overwrite packages in the Nix store but just add new versions in different paths, they are atomic. So during a package upgrade, there is no time window in which the package has some files from the old version and some files from the new version — which would be bad because a program might well crash if it’s started during that period." a PS, re "analogy" see eg https://plato.stanford.edu/entries/reasoning-analogy/ |
@monadic @quinton-hoole and the rest of the @cncf/toc, are you OK moving forward with a vote next week or need more time for due diligence? |
Thanks @caniszczyk . Yes, my questions have been addressed in the updates to this proposal and I'm happy to move forward. |
I'd like others in the TOC to chip in, and ask questions or indicate a go-forward. For my part, I feel Helm has done enough to be an Incubation project. Any DD or consultation at this time is an opportunity to agree some assumptions in the hope of a better and more likely success plateau. ie. Yes. |
I am basically +1 on this, but as essentially an outsider to the Kubernetes community, my question remains: why is Helm not simply part of Kubernetes? Why does it merit a separate CNCF project? I suspect that the real answer is that there are some significant points of divergence between Helm and Kubernetes; is there any substance to this? If so, it would be helpful to understand what these points of divergence are -- not to cast judgement on them, but rather to help support the case that these should be, in fact, separate projects. |
@mattfarina to your no. 4 (just my 2 cents):
I'm not sure I can articulate very well why I think this, its more of a gut feel, but between the role of the TOC (which doesn't just make sure a check-list is adhered to, they vet projects for certain criteria), and that over time I see projects fading away or merging, I think the overall list of projects in the CNCF will stay smaller. I view one of the missions of the CNCF as to help guide the community to make smart choices, and while the CNCF won't necessarily pick winners I do think there will be some implicit filtering that will happen that at least narrows customer's choices down to a handful rather than a laundry list. |
(I'm still catching up on the thread.) I want to thank @mattfarina for patiently and thoroughly explaining the situation. I understand the desire for more clarity about what is and isn't in scope for the kubernetes project. I'll attempt to add more detail, but it's never going to be 100% cut and dry.
I could explain why a particular decision was made for every single subproject and non-subproject, but don't think that would actually be that useful. Helm, however, doesn't meet any of the criteria above. Helm is somewhat of a special case. At the time Helm v2 started, there wasn't another vendor-neutral home for the project. CNCF didn't fully/officially exist, and certainly didn't have a notion of inception or a sandbox. Kubernetes didn't even have its incubator. The Kubernetes project is now so big and has grown so fast that we're struggling to manage it, attempting to impose more order and uniformity and accountability, and cutting scope where we can. We froze the Kubernetes incubator last year and have since officially deprecated it. Kubernetes will likely retire some subprojects that no longer make sense. For the most part, things that are already part of the Kubernetes release can't be cut without breaking compatibility, but I've been working to clarify the boundaries, as Matt illustrated above, and we've been working hard on extensibility all over the system in order to make future scope creep less necessary. Let me reframe the question: Should Kubernetes have its own foundation? This is a question I've seriously asked in the past. Kubernetes has dozens of repositories, hundreds of org members, thousands of contributors, tens of thousands of members of its slack team. It's the anchor project of the CNCF, so let's assume not. Not everything related to Kubernetes will be part of the Kubernetes project, so where do we want the "user space" for Kubernetes to live? AIUI, "user space" javascript libraries for NodeJS go to the Javascript foundation (https://js.foundation/) rather than the Nodejs foundation (https://foundation.nodejs.org/). Do we want something similar for Kubernetes? Assuming not, then that is something that the Kubernetes project needs from the CNCF. It's fair to ask how many projects the foundation wants to have, whether it has the right tiers to scale, and whether it has the right marketing approach (e.g., differentiating strategic projects from useful ones, or platforms from components). And the CNCF needs to make an independent judgement about the merits of specific projects, of course. In this case, does the CNCF want Helm to remain part of the CNCF? The Helm maintainers, Kubernetes Steering Committee, and Kubernetes Architecture SIG have all already agreed that Helm should officially become an independent project, for reasons that have already been covered. From a contributor standpoint and from a user standpoint, Helm has always been a separate project. It's not a perfect metric, but Helm has 5737 github stars, which is more than:
|
To expand on "things critical to making Kubernetes usable": I mean things without which users otherwise were reverse-engineering Kubernetes to figure out, and/or copying code out of Kubernetes itself to make work. |
That article about Kubernetes complexity (and similar articles) is getting pretty off topic. Even before those articles, complexity has been very much on our minds. But not in the same ways, necessarily, as those authors. It sounds like that author in particular is asking Kubernetes to do MORE rather than LESS. I'm happy to hear that many operators and SREs were at Kubecon. They are important categories of users. As one example, a number of larger enterprises enforce strict role separation where developers cannot push to production. Others build custom platforms or at least CI/CD pipelines atop Kubernetes for their developers, who then never need to use kubectl or Helm. Developer focus: Are there applications that shouldn't or at least don't need to be run on Kubernetes? Sure. That author apparently wants a PaaS, though, or at least push to deploy. Literally dozens of other projects and products can do that on top of Kubernetes. He might find one useful. That doesn't mean that should be part of the Kubernetes project. Generators and templates: As pointed out, the community Helm charts partly address that, and has been the biggest effort to date along those lines. OTOH, we've had trouble finding people to maintain the project's original example suite for quite some time. Kubernetes has always been about managing containerized workloads and services, from the beginning in 2013, prior to being open-sourced, when Kubernetes had just 4 APIs. We've added more functionality, more APIs, more repos, more extension points, more provider implementations, new cluster lifecycle tools, etc., but fundamentally it's still about that, and the frequently copied/paraphrased descriptions I wrote on kubernetes.io for the 1.0 launch are still accurate. |
After a great talk with @bcantrill I'd like to add some more context to this issue around the relationship with Helm and Kubernetes.
I hope this adds some clarification. If there are more questions I'm happy to continue to field them. |
As example of a judgement call for every subproject, since Kompose was mentioned: The Kubernetes project staffed an oncall rotation to answer user questions on slack, stackoverflow, the kubernetes-users mailing list, etc. Back in 2015-2016, we were fielding a fair number of questions from docker users exploring Kubernetes. We created a cheatsheet for users moving from the docker CLI to kubectl: But we also got questions from docker-compose users, and answers to those questions were necessarily more complicated: pods, replication controllers, services, volumes, ... A couple simple tools were built within the community to translate from the compose format to Kubernetes APIs, such as compose2kube: We concluded that would be a better approach to help onboard such users than a long migration guide, and I believe it was: So we joined forces on one tool, kompose: At this point, there's not much development on the project: And docker has its own tool for this: It's worth re-evaluating whether kompose is still needed now. Note that the Kubernetes-focused follow-on project is not part of the Kubernetes project: |
As another example: Originally (2014), Kubernetes didn't really have an ecosystem, and there weren't great ways to discover what others were building on/for K8s. So, we accepted a small number of interesting tools/add-ons people in the community built into a That was one of the first things we moved out of the kubernetes repository, to https://github.com/kubernetes/contrib, which still exists, but has been deprecated since 2016. Today, we definitely wouldn't take any subprojects we didn't absolutely need. |
On Tue, May 29, 2018 at 7:47 PM, Brian Grant ***@***.***> wrote:
@quinton-hoole <https://github.com/quinton-hoole> "100s" of subprojects
is off by an order of magnitude.
You're right, 100's was a exaggeration. Dozens is probably more accurate.
But that doesn't significantly change the problem.
Dozens, sure, though most of those are "core" (which, to be honest, is an
ambiguous, imprecise, virtually meaningless term).
Which other subprojects would you propose would meet the bar as top-level
CNCF projects?
The point I was trying to make is that "the bar" for top-level CNCF
projects is fairly inclusive. Specifically, it's this:
https://github.com/cncf/toc/blob/master/PRINCIPLES.md#what-were-looking-for
Off the top of my head, the following Kubernetes sub-projects might
arguably meet that bar (and to be clear, I'm not at all recommending that
all of them be moved from Kubernetes to the CNCF, but rather observing that
we don't currently seem to have a succinct argument for why they should
not):
Cluster Registry
<https://github.com/kubernetes/cluster-registry/blob/master/README.md> - "A
lightweight tool for maintaining a list of clusters and associated metadata"
Cluster Federation
<https://github.com/kubernetes/federation/blob/master/README.md> - "Enables
users to federate multiple Kubernetes clusters together"
Kops <https://github.com/kubernetes/kops/blob/master/README.md> - "The
easiest way to get a production grade Kubernetes cluster up and running."
Kube State Metrics
<https://github.com/kubernetes/kube-state-metrics/blob/master/README.md> -
"a service that listens to the Kubernetes API server and generates metrics
about the state of the objects"
Kubernetres Dashboard
<https://github.com/kubernetes/dashboard/blob/master/README.md>- "a general
purpose, web-based UI for Kubernetes clusters"
Kompose <https://github.com/kubernetes/kompose/blob/master/README.md> - "a
tool to help users who are familiar with docker-compose move to Kubernetes"
- you've discussed this one already.
Node Problem Detector
<https://github.com/kubernetes/node-problem-detector/blob/master/README.md>
- "make various node problems visible to the upstream layers in cluster
management stack".
Frakti <https://github.com/kubernetes/frakti> - "hypervisor-based container
runtime for Kubernetes. Lets Kubernetes run pods and containers directly
inside hypervisors via runV."
Kubernetes Anywhere - "{concise,reliable,cross-platform} turnup of
Kubernetes clusters".
Cluster API
<https://github.com/kubernetes-sigs/cluster-api/blob/master/README.md> -
"bring declarative, Kubernetes-style APIs to cluster creation,
configuration, and management"
Kubebuilder
<https://github.com/kubernetes-sigs/kubebuilder/blob/master/README.md> - "a
framework for building Kubernetes APIs using custom resource definitions".
Posseidon <https://github.com/kubernetes-sigs/poseidon> - "scheduler that
augments the current Kubernetes scheduling capabilities by incorporating a
new novel flow network graph based scheduling"
Kustomize <https://github.com/kubernetes-sigs/kustomize> - "Customize raw,
template-free YAML files. Kustomize targets kubernetes; it understands and
can patch kubernetes style API objects".
Strimzi <http://strimzi.io/> - "run an Apache Kafka cluster on Kubernetes
in various deployment configurations" - they approached me recently for
advice on becoming a top level CNCF project, and I advised them to first
speak to Kubernetes SIG-Apps.
Essentially everything outside of the Nucleus and Application Layers in
this diagram:
https://github.com/kubernetes/community/blob/master/contributors/devel/
architectural-roadmap.md#system-layers
I'm sure we could have a long and passionate debate about each one of the
above, but I think it would be more productive to put a set of clear
guidelines together as to where the dividing line lies between top-level
CNCF projects,and Kubernetes sub-projects, so that we only need to debate
those that lie very close to that line, or represent exceptions.
Q
—
… You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#114 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AJ6NAXq6zsZkxzI0qlNtAfs6TNSdkd7Vks5t3ghdgaJpZM4T6iWm>
.
--
Quinton Hoole
quinton@hoole.biz
|
Shall we make this a topic for a toc call? say, 30 mins allocated |
SGTM
…On Wed, May 30, 2018 at 2:20 PM, alexis richardson ***@***.*** > wrote:
Shall we make this a topic for a toc call? say, 30 mins allocated
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#114 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AJ6NAX1jrzXmccg26VpEjJzS-U8bVChcks5t3w0ggaJpZM4T6iWm>
.
--
Quinton Hoole
quinton@hoole.biz
|
@monadic Do we really need to? If so, would 30 minutes resolve this? @quinton-hoole I know you're looking for a clear, simple technical line, but I don't think that's feasible. There are other considerations, as well, which have already been discussed in this thread.
I'm not going to address most of the others. They aren't significant and/or don't really provide standalone functionality, and several are candidates for retirement rather than for becoming top-level CNCF projects. The only subproject in that list with many users is kops. Kubernetes doesn't have a clear vision/roadmap for its reference cluster management tools at the moment. I've asked SIG Cluster Lifecycle to create one. Once we have that, we can decide what we're doing with kube-up, kubeadm, kops, bootkube, kubespray, kube-aws, kubernetes-anywhere, cluster API, etc. We should retire some and cleanly integrate and/or unify others. In short, I don't see any other candidates for things that would become top-level CNCF projects. |
@quinton-hoole Thanks for steering Strimzi the SIG Apps way. I've not heard about them, yet, and look forward to learning more. I would be curious to know if something like Strimzi, which is focused on one application, is the kind of things the CNCF would be interested in for a project. It would help me better understand the intent of the CNCF. One thing in the guidance that might be worth pulling apart is where it says:
Many of the people who contribute to Helm do not work on other parts of Kubernetes. We've discovered this due to the number of people who are not part of the Kubernetes org. Quite a few we've ended up sponsoring so they can, for example, do more with the community charts. Do the other sub-projects have the same split or are they regular contributors to other Kubernetes core elements? I honestly don't know but it might inform things. Kubebuilder might be another interested example to pick apart in trying to figure out the line. Building Kubernetes controllers is hard. The goal of this is to make that experience easier alongside CRDs that work with the controller. It's a project meant to enable the ecosystem rather than be apart of it. If there is anything I can do to help inform this decision I'm happy to help. Please feel free to let me know. Note, for those that don't know, I co-chair SIG Apps |
@mattfarina kubebuilder isn't necessarily just for the ecosystem. We're trying to make it easier (and less error-prone and inconsistent and difficult to maintain) for anyone, including us kubernetes project contributors, to create Kubernetes APIs and their business logic. |
strawman / retcon - A CNCF Project may have affiliated "Subprojects" that inherit governance and users from the parent Project. Given a Project, eg Kubernetes, there are two types of Subproject:
Detail on 1:
Detail on 2:
Examples:
Note that both kops and dashboard pretty much have to upgrade when k8s does. The decision on Subproject status lies with the Project leads. The TOC recommends that:
The CNCF accepts independent Projects that fall outside the above patterns:
Examples of 3:
|
Kubernetes has a notion of subprojects, which we're working to better document and implement in our project automation. More on this can be found here: Currently identified subprojects are enumerated under their SIGs here: |
@monadic I think what you suggested makes sense and mostly matches the Kubernetes notion. Part of our problem was that we didn't have governance and structure in place from the beginning, so the main project and Helm independently evolved their own ways of doing things. |
@bgrant0607 thank-you. Would it make sense to merge the notions, or make them 100% consistent? I was trying to define something generic eg for Prometheus subprojects too. @mattfarina & others -- thanks for all the patient contributions to this thread. @bgrant0607 @quinton-hoole @bcantrill and other TOC voters -- can we declare victory on the merits of Helm and move the principles-of-subprojects to be its own topic? |
It was stated in these comments that Helm is independent of Kubernetes. I understood the context to mean that the culture/work/history were independent. However, is Helm not dependent technically on Kubernetes? I believe Rook and Telepresence fall into this pattern as well. There have been critics of the CNCF who have jokingly called it the Kubernetes Foundation, and including projects that can't operate outside of Kubernetes appears to enforce that perception. It might be beneficial to require projects in the CNCF to have multiple primary platforms. A spec is required to be implemented in more than one system, so why shouldn't a tool be required to work with more than one primary system. Perhaps adding Mesos support or Swarm. Otherwise, I believe the CNCF will unintentionally exclude non-k8s-centric projects that might see the CNCF as the Kubernetes Foundation. |
Yes, I'm in broad agreement with your strawman proposal Alexis. Thanks for
taking the time to write it down.
And assuming that most of the ToC is also in agreement with that general
direction, I'm fine with moving that discussion to it's own topic, and
proceeding with the Helm vote independently of that.
The crux of the whole combined discussion, IMO, is to make sure that we
don't end up with projects that we would like to have as part of the CNCF
family, but which can't find a home as a subproject (due to the policies of
the relevant top-level CNCF project), but also can't find a home as a
top-level CNCF project (e.g. because we want to keep the number of such
top-level projects small, and their size relatively large).
I think that point number 3 in your strawman ("Projects that are optional
add-ons") addresses that adequately, although I want to call out that I
think that invoking that item 3 will probably significantly increase the
number of top-level CNCF projects, perhaps beyond what some members of the
ToC might consider to be reasonable. That seems somewhat inevitable to me,
and perhaps the best way forward is to figure out how to manage that
situation appropriately. Our current tiering system
(sandbox->incubation->graduated) focusses mainly on maturity, and is aimed
at a relatively smaller number (say 10's) of relatively large top-level
CNCF projects. It seems we will, in the near future, need a way to handle
a larger number of smaller top-level CNCF projects, that don't fit as
subprojects.
Q
…On Thu, May 31, 2018 at 8:54 AM, alexis richardson ***@***.*** > wrote:
@bgrant0607 <https://github.com/bgrant0607> thank-you. Would it make
sense to merge the notions, or make them 100% consistent? I was trying to
define something generic eg for Prometheus subprojects too.
@mattfarina <https://github.com/mattfarina> & others -- thanks for all
the patient contributions to this thread.
@bgrant0607 <https://github.com/bgrant0607> @quinton-hoole
<https://github.com/quinton-hoole> @bcantrill
<https://github.com/bcantrill> and other TOC voters -- can we declare
victory on the merits of Helm and move the principles-of-subprojects to be
its own topic?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#114 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AJ6NAWEerbybu2-eoiP7gj-aQkB193Xvks5t4BI_gaJpZM4T6iWm>
.
--
Quinton Hoole
quinton@hoole.biz
|
@barkerd427 thanks. I think that can work for projects lower down the app stack, but not for app def tools so easily. we may need a big new metaproject and org... "app def tools" @quinton-hoole great! now just the two bri/yans to go? |
@daniel Barker: The CNCF welcomes projects that operate without
Kubernetes. The fact that many of the CNCF projects primarily target
Kubernetes is more a reflection of the popularity of Kubernetes, than any
active filtering. I don't think that making "the ability to operate with
something other than Kubernetes" a requirement for CNCF projects would
serve our community well.
Q
…On Thu, May 31, 2018 at 9:53 AM, Daniel Barker ***@***.***> wrote:
It was stated in these comments that Helm is independent of Kubernetes. I
understood the context to mean that the culture/work/history were
independent. However, is Helm not dependent technically on Kubernetes? I
believe Rook and Telepresence fall into this pattern as well. There have
been critics of the CNCF who have jokingly called it the Kubernetes
Foundation, and including projects that can't operate outside of Kubernetes
appears to enforce that perception. It might be beneficial to require
projects in the CNCF to have multiple primary platforms. A spec is required
to be implemented in more than one system, so why shouldn't a tool be
required to work with more than one primary system. Perhaps adding Mesos
support or Swarm. Otherwise, I believe the CNCF will unintentionally
exclude non-k8s-centric projects that might see the CNCF as the Kubernetes
Foundation.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#114 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AJ6NAZt3h7S4-Ds7egdviz3UfJbMSDn0ks5t4B_xgaJpZM4T6iWm>
.
--
Quinton Hoole
quinton@hoole.biz
|
As I thought more about this problem space I started to think more about the Apache foundation. It has a lot of projects, some of which are platforms. Some other projects build upon platforms and are specific to them. Because there are so many projects they do provide some taxonomy to navigate them. This leads me with questions around the number of projects, scope control, and direction the CNCF wants to go in. My 2 cents is to have more projects and be an enabler for cross vendor projects that target the CNCF definition of cloud native. That project leaders could even help to mentor other project leaders and they try to solve problems with their experience. There is a need for a vendor neutral place for Kubernetes specific things, that are not part of Kuberentes, to collaborate. If the CNCF is not the right place for that it's important to communicate so a proper place be formed or found. |
@monadic Yes, I'd like to decouple the Helm decision from broader discussions about subprojects (K8s is more complex than all other current and likely future CNCF projects due to scale and scope), additional project categorizations, general project portfolio strategy/principles, and other more general topics. I agree with @quinton-hoole that emergence of Kubernetes-specific projects is inevitable at this point given the magnitude of user interest (e.g., 200k people signed up for meetups) and breadth of industry support. |
@quinton-hoole, my point here was not what is written down, but rather what's interpreted as included. If I was in charge of a project that was Mesos-centric, then I probably wouldn't look to the CNCF if I hadn't implemented a k8s integration. I might look to Apache. This might be inadvertently causing the CNCF and k8s to miss out on projects that would otherwise be great contributions to the Cloud Native community. If a new issue is created to discuss further segmenting or decomposing projects, then I'm happy to continue the conversation there. I'm not sure what's appropriate protocol here. |
@barkerd427 most of the current CNCF projects are just as useful for k8s, swarm, mesos. Why would you look somewhere else? |
I understand Daniel's point to be that what you say is not necessarily the
general perception. There may be an (incorrect) perception that the CNCF
is for "Kubernetes focussed projects". I think that we can adjust our
communication to help address that misperception.
Q
…On Thu, May 31, 2018 at 1:17 PM, alexis richardson ***@***.*** > wrote:
@barkerd427 <https://github.com/barkerd427> most of the current CNCF
projects are just as useful for k8s, swarm, mesos. Why would you look
somewhere else?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#114 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AJ6NAT24bopDHIzUX3Sl6x7i0Ko-5ZO4ks5t4E_xgaJpZM4T6iWm>
.
--
Quinton Hoole
quinton@hoole.biz
|
This is getting pretty off topic (Helm), but Apache is almost 20 years old. At this point, it has a fairly large ecosystem of projects (~200?, not including incubating, labs, retired, etc.) that is broader than just Hadoop and other data-processing projects, with multiple "anchor" projects. They have more categories of projects than we have projects so far. Not that we wouldn't take Mesos-related projects, but since Mesos is in the Apache Foundation, it may well make sense for related projects to also be in the Apache Foundation, as that may make code ownership, licensing, and contribution simpler, for instance. Certainly that issue has come up for CNCF projects. |
I would point out that not only does the (pending) cloud native definition
avoid mention of the word Kubernetes, it doesn't even mention the term
orchestration (although it's certainly implied by several of the concepts).
https://github.com/cncf/toc/blob/cloud-native-definition/DEFINITION.md
In addition, numerous CNCF projects like Prometheus, Linkerd, and Envoy
work equally well with Mesos as Kubernetes. However, I think it would be a
mistake for such support to be a requirement.
--
Dan Kohn <dan@linuxfoundation.org>
Executive Director, Cloud Native Computing Foundation https://www.cncf.io
+1-415-233-1000 https://www.dankohn.com
On Thu, May 31, 2018 at 4:51 PM, Quinton Hoole <notifications@github.com>
wrote:
… I understand Daniel's point to be that what you say is not necessarily the
general perception. There may be an (incorrect) perception that the CNCF
is for "Kubernetes focussed projects". I think that we can adjust our
communication to help address that misperception.
Q
On Thu, May 31, 2018 at 1:17 PM, alexis richardson <
***@***.***
> wrote:
> @barkerd427 <https://github.com/barkerd427> most of the current CNCF
> projects are just as useful for k8s, swarm, mesos. Why would you look
> somewhere else?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#114 (comment)>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/
AJ6NAT24bopDHIzUX3Sl6x7i0Ko-5ZO4ks5t4E_xgaJpZM4T6iWm>
> .
>
--
Quinton Hoole
***@***.***
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#114 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AC8MBoW7U5ZzppilFr_itom9kTYnJ609ks5t4FfjgaJpZM4T6iWm>
.
|
Helm is accepted as an incubating project: +1 TOC binding votes (7 / 9): Alexis Richardson: https://lists.cncf.io/g/cncf-toc/message/2073 |
In addition to the proposal I wanted to add some context:
We have tried to do our due diligence ahead of time. I'm happy to come to a TOC meeting to discuss this.
cc @bgrant0607