Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid certificates (Let's Encrypt related) #8555

Closed
tfardet opened this issue Oct 4, 2021 · 16 comments · Fixed by #8930
Closed

Invalid certificates (Let's Encrypt related) #8555

tfardet opened this issue Oct 4, 2021 · 16 comments · Fixed by #8930
Assignees
@humitos
Labels
Accepted Accepted issue on our roadmap Bug A bug

Comments

@tfardet
Copy link

tfardet commented Oct 4, 2021

Probably due to the fact that the Let's Encrypt root certificate expired on 30 September 2021, trying to pull from many non-Github repositories leads to server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none.
Would it be possible to update the certificates to resolve this?
This is for instance an issue when using submodules that are not hosted on GitHub.
@humitos
Copy link
Member

humitos commented Oct 4, 2021

Would it be possible to update the certificates to resolve this?

What are the required steps that Read the Docs should follow to do this?

@humitos
Copy link
Member

humitos commented Oct 4, 2021

trying to pull from many non-Github repositories leads to server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none.

Do you have an example of a build URL failing with this error?

@tfardet
Copy link
Author

tfardet commented Oct 4, 2021 •
edited

Do you have an example of a build URL failing with this error?

https://readthedocs.org/api/v2/build/14898416.txt

What are the required steps that Read the Docs should follow to do this?

I don't know exactly how you manage your VMs but you need to patch the system certificate of your Linux (Ubuntu LTS?) to use the new Let'sEncrypt certificate (most probably Ubuntu release an update for OpenSSL or something like that, I don't use this distro so I don't know exactly)

@humitos humitos added the Bug A bug label Oct 4, 2021
@humitos
Copy link
Member

humitos commented Oct 4, 2021

Thanks for your reply.

I found what's required: apt-get update and apt-get install ca-certificates. After doing that, I'm able to clone the repository and its submodules from inside the VMs.

Now the problems 😞 . To update ca-certificates in our build images, we have to rebuild them all and that will bring known issues that we can't cleanly solve (basically, re-building an image doesn't make it exactly the same with the same software versions and people's builds start failing because of that). So, I'm not sure we will be able to re-build all the Docker images (stable, latest, testing).

However, we can easily rebuild our latest one (readthedocs/build:ubuntu-20.04) with the latest ca-certificates version (20210119~20.04.2) installed and tell users to define this image on their config file.

@humitos humitos added the Accepted Accepted issue on our roadmap label Oct 4, 2021
@humitos
Copy link
Member

humitos commented Oct 4, 2021

Another workaround would be to execute apt-get install ca-certificates as part of the build process from inside Read the Docs application itself.

@agjohnson
Copy link
Contributor

I think we should be able to rebuild all the images, not just our beta image. Ubuntu releases are stable enough that I would trust things to work. Package versions do not drift heavily once in an LTS

@tfardet
Copy link
Author

tfardet commented Oct 4, 2021 •
edited

However, we can easily rebuild our latest one (readthedocs/build:ubuntu-20.04) with the latest ca-certificates version (20210119~20.04.2) installed and tell users to define this image on their config file.

That's fine with me 👍 can you ping me once that's done?

I don't think a workaround is a good idea, the first option would probably save you some work when others start having that same issue

@rjw57
Copy link

rjw57 commented Oct 4, 2021

However, we can easily rebuild our latest one (readthedocs/build:ubuntu-20.04) with the latest ca-certificates version (20210119~20.04.2) installed and tell users to define this image on their config file.

Wouldn't that introduce a chicken and egg problem whereby the repo needs to be able to be cloned so that the configuration required to clone the repo can be read?

@tfardet tfardet changed the title Invalid certificates (probably Let's Encrypt related) Invalid certificates (Let's Encrypt related) Oct 4, 2021
@ExtremOPS
Copy link

I am experiencing the same problem as op described. My code repository is hosted on a GitLab server.

If there is something with which I (a noob) could help, please let me know.

@humitos
Copy link
Member

humitos commented Oct 5, 2021

@rjw57

Wouldn't that introduce a chicken and egg problem whereby the repo needs to be able to be cloned so that the configuration required to clone the repo can be read?

Yes. I may express myself wrong here.

We use a default Docker image to run git commands inside it (currently we use readthedocs/build:latest) and I think we could use the new image readthedocs/build:ubuntu-20.04 with a newer version of ca-certificates to do the git commands.

However, even if the git commands succeed, there may be other commands that may fail in the build process due to the old CA certificates. In that case, users could define readthedocs/build:ubuntu-20.04 as the build image to fix that problem.

humitos added a commit that referenced this issue Oct 5, 2021
@humitos
Build: update ca-certificates before cloning
fae0101 
This is a temporal solution while we decide how to fix the real problem. For
now, we are installing a newer version of `ca-certificates` before starting to
clone the repository.

Reference #8555
@humitos humitos mentioned this issue Oct 5, 2021
Merged
@astrojuanlu
Copy link
Contributor

xref https://stackoverflow.com/q/69439345/554319

@tfardet
Copy link
Author

tfardet commented Oct 5, 2021

@humitos can I approve the PR or does it have to be someone from RtD?

@stsewd
Copy link
Member

stsewd commented Oct 5, 2021

@humitos can I approve the PR or does it have to be someone from RtD?

that's a core team decision, but if you have any feedback there, feel free to comment!

@tfardet
Copy link
Author

tfardet commented Oct 5, 2021

that's a core team decision, but if you have any feedback there, feel free to comment!

thought so, so no, nothing to change on my side; I think this is the right server-side fix for now

@stsewd stsewd mentioned this issue Oct 5, 2021
Merged
@humitos
Copy link
Member

humitos commented Oct 5, 2021

We deployed a quick and temporal fix for now. Please, let us know if you still have issues with the certificates. Thanks!

@guigomcha
Copy link

I can confirm that the hotfix fixed our project build. Thanks a lot everyone!

@humitos humitos self-assigned this Feb 16, 2022
humitos added a commit to readthedocs/readthedocs-docker-images that referenced this issue Feb 16, 2022
@humitos
Release ubuntu-20.04-2022.02.16
ee768b8 
This Docker image does not contain any change in the Dockerfile. However, as it
re-builds completely new versions of the same packages will be installed.

We are forcing this because we've hit issues with the old `ca-certificates`
version installed in our current production image. See
readthedocs/readthedocs.org#8555

Note that we are currently executing `apt-get install ca-certificates` on each
build to upgrade this package and workaround this problem. However, this adds
extra time to _all builds_ and we want to avoid that.

Also note that this will be still required for `readthedocs/build:latest` image,
but we will hopefully use `readthedocs/build:ubuntu-20.04` as default image for
the "Clonning" step sooner than later.
@humitos humitos mentioned this issue Feb 16, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@humitos humitos

Labels
Accepted Accepted issue on our roadmap Bug A bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Build: use ubuntu-20.04 image for setup VCS step readthedocs/readthedocs.org
8 participants
@rjw57 @humitos @astrojuanlu @agjohnson @stsewd @ExtremOPS @guigomcha @tfardet