New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Request to present the Operator Framework to the TOC as an incubating project #303
Conversation
Hi Eryn! Some quick questions on stuff that is missing from the proposal.
Thanks! |
I believe that this falls within the scope of SIG App Delivery and should be channelled through there. |
Hi @jbeda, Can you clarify the ask here a little?
Are you looking for a list of users you can verify experiences with? I'm a little confused on this one. |
Also, for the wider group, the intent is 100% to move operatorhub.io into CNCF. But, how does the infrastructure handoff work? Build pipelines, storage, etc. all have to be maintained somewhere. Is there a model to follow here? We understand this will take some time to hand off but, we can start working that piece now if it helps. |
@chris-short I would recommend getting in touch with the Helm folks for more info on that. Helm Hub strikes me as a directly analogous model to follow on that. There is definitely precedent for it and hasn’t presented any significant institutional hurdles (to my knowledge). |
@chris-short each project does it their own way, depending on the size, if it's more complex something like helm hub or https://github.com/kubernetes/community/tree/master/wg-k8s-infra are good models to follow |
Wrt users -- I'm thinking looking at operators that use the SDK and are production ready. Seeing wide usage across projects/companies would be an indication, IMO, of "production ready". Not sure how other folks on the TOC would see this. |
A quick way of tracking some (ie. public on GitHub) usage Currently 354 instances of any version of Note tracking this over time might be more instructive than the absolute number. |
The link I have used in the past is below. I know that Go module handling has changed recently that throws this off. "Showing 2,788 available code results" but that does double or triple count some projects. |
github search doesn't account for enterprises building with operator-sdk in private repos for internal only use |
This signature captures Go-based Operator-SDK projects: https://github.com/search?q=github.com%2Foperator-framework%2Foperator-sdk+filename%3Ago.mod+filename%3AGopkg.toml+filename%3Aglide.yaml&type=Code - it does not depend on Go modules, which the SDK only moved to in one of the recent 0.9.x releases To find additional Helm-based Operator-SDK projects: https://github.com/search?q=filename%3ADockerfile+helm-operator&type=Code And additional Ansible-based Operator-SDK projects: https://github.com/search?q=filename%3ADockerfile+ansible-operator&type=Code |
We're building a number of internal operators on Ansible that are not yet out in the open. Certainly appreciate the operator-sdk tool for that. It has allowed us to go from general concept to working prototype rapidly and this has been a great way to build excitement as people actually see what can be done. We're also deploying an internal version of OperatorHub.io with just our operators. That was surprisingly simple to do, allows us to be sure our Operators or moving toward a place of public consumption and eases internal documentation desires. We're planning to look into what rebranding can be done on that soon so it isn't too confusing, but still provides a familiar feel. |
Do you have rough idea on the costs of hosting OperatorHub currently... maybe RPS and other useful data for estimating the infrastructure needed for CNCF |
OF team - There was discussion on the TOC meeting today that I believe
should be documented in this PR for longevity:
Please provide a response around:
1) Will the Hub host non-operator framework operators?
2) How does the OLM work with non-framework operators?
3) What is the current community accepted definition of an operator?
…On Tue, Nov 5, 2019 at 9:36 AM Chris Aniszczyk ***@***.***> wrote:
Do you have rough idea on the costs of hosting OperatorHub currently...
maybe RPS and other useful data for estimating the infrastructure needed
for CNCF
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#303?email_source=notifications&email_token=AKMZZWJ2YARPT6CMD2FJR43QSGOI7A5CNFSM4I4RUZE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDDLFKI#issuecomment-549892777>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AKMZZWOMUC6LQADL7VVDWTDQSGOI7ANCNFSM4I4RUZEQ>
.
--
Erin A. Boyd
Senior Principal Software Engineer, OCTO
Red Hat <https://www.redhat.com>
eboyd@redhat.com
<https://red.ht/sig>
|
Right now it requires the packaging format for correct display and easy installation. Once we have a discussion and consensus in upstream (e.g. sig-app-delivery) around a common metadata format for Operators we should relax that - maybe at the expense of the quick install method.
OLM currently requires the CSV and externally shipped CRDs for management. How the Operator is created, e.g. which programming language, is not important. We have ideas around adding support for other packaging formats (CNAB, helm, plain manifests) and are putting code in place that allows us to be flexible in the near future.
A custom controller with its own CRDs. |
We believe that out of the four things that constitute Operator framework - Operator SDK, operatorhub.io, OLM, Operator Metering - SDK and hub have definite value to the community. We had done analysis of open source Operators a while back in which Operator SDK was seen to be used quite a lot (https://medium.com/@cloudark/analysis-of-open-source-kubernetes-operators-f6be898f2340). Similarly, operatorhub provides a good central place with searching capabilities for discovering 3rd-party Operators. We are not so sure about OLM, since Operator packaging and installation are already covered by tools like Helm. The CSV CRD defined by OLM mandates a specific format for Operator installation definition. It is not clear why that is the best way of defining Operator packaging. It is also questionable whether there is really a need for new format for Operator installation definition, given Operator is just a pattern leveraging Kubernetes extensibility. Kubernetes YAMLs along with tools like Helm already cover the requirements for Operator packaging and installation. This also aligns with our observations in the field - Operator writers are not defining CSVs for their Operators to the extent that they naturally do Helm charts. Given this observation a thought/suggestion is - may be include only Operator SDK as an incubating project and not the entire Operator framework. Note Operator Hub currently is tied to OLM. |
I like that idea, at least as a stepping stone that might get things moving.
I do think OperatorHub could be a useful project as well, but I'd love to
see someone from one of the other operator projects like KUDO say they
would be involved. I think even intent would be enough for sandbox.
Given KUDO also have a submission in for sandbox:
#261
Do we see it as more useful to have:
* Operator Framework (including SDK, Hub and OLM)
* KUDO
or
* Operator SDK
* KUDO
* OperatorHub (including if needed OLM as a detail)
The later factoring feels more useful to me. I worry in the former case
there is pressure for every project to come with it's own Hub. Conversely
SIG Security is looking at SecurityHub (https://securityhub.dev/). Today
this is just Falco rules, but conversations have started about other types
of security content under one banner.
Hearing the KUDO submission on the SIG sooner rather than later would
probably also help the conversation about Operators in general.
Gareth
…On Fri, 8 Nov 2019 at 16:58, Dev ***@***.***> wrote:
We believe that out of the four things that constitute Operator framework
- Operator SDK, operatorhub.io, OLM, Operator Metering - SDK and hub have
definite value to the community. We had done analysis of open source
Operators a while back in which Operator SDK was seen to be used quite a
lot (
***@***.***/analysis-of-open-source-kubernetes-operators-f6be898f2340).
Similarly, operatorhub provides a good central place with searching
capabilities for discovering 3rd-party Operators.
We are not so sure about OLM, since Operator packaging and installation
are already covered by tools like Helm. The CSV CRD defined by OLM mandates
a specific format for Operator installation definition. It is not clear why
that is the best way of defining Operator packaging. It is also
questionable whether there is really a need for new format for Operator
installation definition, given Operator is just a pattern leveraging
Kubernetes extensibility. Kubernetes YAMLs along with tools like Helm
already cover the requirements for Operator packaging and installation.
This also aligns with our observations in the field - Operator writers are
not defining CSVs for their Operators to the extent that they naturally do
Helm charts.
Given this observation a thought/suggestion is - may be include only
Operator SDK as an incubating project and not the entire Operator
framework. Note Operator Hub currently is tied to OLM.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#303?email_source=notifications&email_token=AAAAP3OY3NXHFAZNHLS6FHTQSWLFHA5CNFSM4I4RUZE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDSXB2Y#issuecomment-551907563>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAP3J4IRFRF6CHENLZC5DQSWLFHANCNFSM4I4RUZEQ>
.
--
Gareth Rushgrove
@garethr
garethr.dev
devopsweekly.com
|
Thanks for this analysis and consideration. OLM does come with a packaging method today which is currently revamped and made open to other formats and tools, like Helm. Packaging is however just one of the responsibilities OLM fulfills. A main use case is to manage the lifecycle of potentially many Operators on shared clusters, in order to allow tenants to use them safely. Since Operators are cluster-wide extensions there is a need for safeguards that fences these extensions of each other and make them discoverable to end users. At the same time cluster admins want to rely on over-the-air updates to Operators being long-running workloads, which is where collision prevention is needed on ownership of APIs and Webhooks - all of which are core parts of every Operator. OLM has been born out of the need for a component that does this on-cluster and scales to dozens of Operators installed, updated and used there. |
Thanks for clarifying the additional responsibilities that OLM performs. While the points about orchestrating the set of Operators in a cluster is valid (since CRDs/Operators are cluster-wide control points), the key question is - is it necessary to apply the reconciliation loop based approach for managing the lifecycle of Operators themselves? While it is appealing - an Operator to manage other Operators - CRDs are not the only objects in Kubernetes that are cluster scoped and whose creation and management needs to be carefully handled by cluster admins. Today, cluster admins are predominantly using Helm for installing all types of objects |
I've worked on both Helm and OLM.
A large problem that OLM is attempting to solve is dependency resolution. Because CRDs (and Operator deployments) can be cluster-scoped, it is possible to have incompatible CRDs installed (CRD versions used to only be cosmetic). Managing OLM catalogs is closer in practice to repository mirrors for Linux distributions where every version presented in a catalog is compatible with each other. This is in contrast to Helm, which is moving towards a global "namespace" where you can obtain charts from any registry similar to container images. OLM's CSV format actually inspired the upstream Application Definition and is extremely similar barring the data used for dependency resolution. A goal of the OLM team has been to eventually converge with the community on this. Catalogs can largely be considered orthogonal to the core controller that manages lifecycle events of other controllers. It is optional behavior and used to be encapsulated in a completely separate binary.
If you refer to the design discussion around Helm v3 and CRD management, you'll find that the lifecycle related work for CRDs/Operators is explicitly out of scope until the community has experimented enough that Helm feels confident in moving forwards with something. From the Helm perspective, OLM is a testbed for CRD lifecycle management. From the OLM perspective, Helm is a future for packaging content.
It is clearly an idea worth pursuing from the perspective of Red Hat the portion of Operator authors that have packaged their software with OLM. Even the aforementioned App Definition, which was designed not to use a controller, now does. I think it could be a notable goal that something like OLM wouldn't need to be a controller in the future because Kubernetes will eventually be able to provide the level of resource validation. For now, I'd consider this a compromise for practicality's sake. |
Thanks @garethr! I had an initial response, but want to read and understand this whole thread and discuss with the KUDO team - I’ll link the issue once we open it. |
Capturing some of the questions for longer answers from the TOC call on Dec 3:
|
@brendandburns @kgamanji polite nudge for status, please. |
@erinaboyd and others First, many apologies for the length of time that it has take to get this far, this has ended up twistier and thornier than I think we imagined it would be, and also there have been other distractions, but I know this has been frustrating for folks involved. The ToC has decided, given the different discussions and our general concern about how we think about various different hubs that we would prefer to have operator framework join CNCF as a Sandbox project rather than an Incubating project. We have likewise recommended that whatever work is done on CNCF Hub that it also be done as a open sandbox project. Placing these projects in Sandbox provides the opportunities for them to be involved with the CNCF while the ToC figures out how it wants to approach the technical, community and design requirements that are needed for *Hub type projects. Please let me know if there are any questions about this. |
@brendandburns Thanks for getting back. Given the concerns are mostly around OperatorHub, what do you think about the following scenario: OperatorHub becomes a sandbox project and as you propose, the community will take it from there with the ToCs efforts around coordinating the various hub type projects. Meanwhile, OLM and SDK as projects are well isolated from OperatorHub, so they could admitted as incubators as proposed? |
@brendandburns thank you for your feedback! I am deferring to the operator community (@dmesser ) above as a request on how to move forward. I think it's a good compromise. Please let us know if that is possible. |
@dmesser if you are interested in splitting OperatorSDK and Operator Lifecycle Manager out into separate projects we would be willing to consider them for incubation. They would need to go through the standard due-diligence process with the SIGs (I think SIG-App-Delivery would probably be the right SIG to review and due-diligence) |
@brendandburns Understood. SDK and OLM have already been reviewed by sig-app-delivery and have undergone the due diligence as part of the submission of OF as a whole. sig-app-delivery also sponsored and voted for the inclusion of OF. Would we need to repeat this process, despite none of the proposal aspects have changed, other than OperatorHub aiming for sandbox whereas the rest looks at incubating? |
@brendandburns bubbling this back up, tia! |
ok, given that, we can probably take this forward to the ToC. Is there a ToC sponsor (sorry, I don't remember the details here) |
@brendandburns I think @kgamanji volunteered to be the sponsor. |
@brendandburns @kgamanji did you guys have any more questions? Can you please provide the team an update on this when you have a chance? Thanks :) |
@brendandburns @kgamanji @lizrice cane someone please provide an update on this? |
Apologies for the delay in response. I have stepped forward to start the DD for Operator Framework sub-projects. Just to confirm we have the same birds-eye view of the current state:
At this stage, it is unclear to me if SDK and OML are to be considered as separate projects or as one project with 2 sub-components? Also, is it possible to update the DD document to reflect the current submission? e.g. some of the links to the user groups usign OF in production are 404ing |
@kgamanji @brendandburns @lizrice Beyond that, the DD document is updated to reflect the proposals for OLM/SDK and OperatorHub.io. All 3 components really belong to the same top-level project, Operator Framework. The 404 links were due to a migration of a content platform and should be fixed now, thanks for pointing out. |
Here's the comment on Helm Hub where I've tried to explain the TOC position |
Thank you for sharing that @lizrice At this stage, I am proceeding with the DD for OLM and SDK as one submission for incubation. Also, I have reached out to end used for feedback on using OF. Will update once I have more details. |
I have completed the review of the DD doc and Operator Framework sub-projects. Thank you for all your patience throughout! Here are my remarks: SDK and OLM for incubation:
Would like to have more outline on the following (if possible/applicable):
Once, the above have been outlined I am happy to move the SDK and OML to a TOC vote. OperatorHub for sandbox:
|
There are two projects related to cluster-addons:
These are still under discussion/review as the goal is to solve shared problems, not operator-framework problems.
The bundle KEP linked above addresses the packaging relationship with OCI artifacts. OCI artifacts seem like a good choice, but there are roadblocks to using them today. They are on the radar and an area of active interest (and would've been the choice if not for the issues outlined in the KEP). |
@kgamanji Thanks for getting back. OperatorHub will adopt the CNCF Code of Conduct as part of the rest of Operator Framework. This https://github.com/operator-framework/community/pull/11/files should be merged soon. |
Thank you for the updates @ecordell @dmesser! I have now opened the public comment period: https://lists.cncf.io/g/cncf-toc/message/4643 |
Operator Framework SDK and OLM sub-projects have applied to join as incubating projects +1 Binding: note: Quorum is 10 as Jeff Brewer has been away |
@amye @kgamanji After much consideration we will not be moving forward with contributing the Operatorhub.io as as sandbox project. Let's close out this PR in favor of what was already voted. Thanks. cc: @dmesser @robszumski @dmueller2001 |
Please let us know if there is a preferred SIG to present to first.
Thanks in advance!