Can one of the admins verify this patch?

Thank you so much for writing this. This PR could potentially make minikube work much better in China, as well as environments with limited internet connectivity. I'll give this a closer look tomorrow, but at least initially this seems to be doing the right things.

Do you have a mirror or two to recommend that I could test this PR against?

One potential concern to eventually address is cache-poisoning, which will require minikube addressing images with hashes instead of versions -- but I won't let "perfect be the enemy of good".

A potential eventual enhancement would be to have an automated fall-back mechanism between a set of known-good image repositories (similar to "apt" or other systems), so that minikube in users in China would not have to manually specify a --image-repository URL unless they wish to.

Thanks again!

@minikube-bot OK to test

You may start with registry.cn-hangzhou.aliyuncs.com/google_containers.
This is one of the well known mirrors for gcr.io maintained by Aliyun guys.
I verified with this mirror so it should work as expected (v1.13.3).

We may use --image-repository as an ultimate workaround
if none of mirrors provided is accessible in user's environment.
I agree that maintaining a mirror list as like the APT / pacman world also helps,
or even trying to guess a reliable mirror by IP address during the setup.
I can work on that later.

Let me know if you have any other concerns.

How does this new --image-repository flag interact with the existing --registry-mirror flag?

--registry-mirror - Registry mirrors to pass to the Docker daemon

I'll admit, I didn't know about this flag up until today. I haven't looked into it's implementation yet to see if it's compatible with non-docker container runtimes, which your flag seems to allow for.

I believe --registry-mirror only impacts those images with no repository prefix - images that come from the Docker official registry.
It won't work on images from private registries, which is the case like gcr.io/kube-proxy. For private images, docker will still go to the private registry and fetch them.

If you set up minikube with --registry-mirror, it could work for pods/deployments that use ubuntu/18.04, but not for gcr.io/ - you will need to wipe the "gcr.io/" out from the references to make it look like an image from the official registry. For the latter case, it could be achieved using --registry-mirror https://private_server --image-registry [private_server/]google_containers

For instance crio has a --registryflag, but that only works for images without prefix ("unqualified"):

   --registry value              registry to be prepended when pulling unqualified images, can be specified multiple times

The same goes for the global /etc/containers/registries.conf, which lists the default registries:

[registries.search]
registries = ['docker.io']

[registries.insecure]
registries = []

And /etc/crio/crio.conf with CRI-O's:

# insecure_registries is used to skip TLS verification when pulling images.
insecure_registries = [
]

# registries is used to specify a comma separated list of registries to be used
# when pulling an unqualified image (e.g. fedora:rawhide).
registries = [
        "docker.io"
]

As stated above, this would not help for the k8s.gcr.io images.

/approve

Thank you for the clarification and excellent work here. I've added a few nitpicks, but it's good enough as is. This is really going to help minikube users around the world.

We're going to want to add an integration test for this, but until #3767 lands it's rather painful to add start arguments. We can do that in a follow-up PR.

@minikube-bot OK to test

Have some other work based on this PR.
Any chance to have it merged sooner?
Let me know if you have any concerns.

@minikube-bot OK to test

I think it may have been my attempt at merging to master, but it appears that "minikube dashboard" is broken with this PR at the moment.

Do you mind remerging this PR? I'd like to also confirm that minikube dashboard still works.

I'm checking the case you describe and looking into the PR.

Thanks for making the changes!

/lgtm /approve

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: laozc, tstromberg

The full list of commands accepted by this bot can be found here.

The pull request process is described here

@laozc got it, thank you for the clarification!