Skip to main content

Troubleshooting - Back-off pulling image registry.thalesdigital.io

Context​

The Trustnest Software Factory has recently changed the URL of the gitlab registry during the migration of the 28/29 august. This procedure helps you to patch your configuration.

The issue​

Previously, the url of the gitlab registry was: gitlab.thalesdigital.io:5005. So during the first deployment, most of you deployed a dockerconfigjson that point to this previous url.

And now, you should see in the kubernetes events:

$ kubectl get events -n prod -w
Back-off pulling image "registry.thalesdigital.io/platform-team-canada/k8saas-private/k8saas-website:925002-60d8a15f203392c77e9fd3889c2481fc4601c50f"

As you can see, the cluster is not able to get the docker images from the registry, because the url has changed.

  • Previously: gitlab.thalesdigital.io:5005
  • Now: registry.thalesdigital.io

Confirm the issue​

First, look at the secret, and fine some dockerconfigjson

$ kubectl get secrets -A|grep dockerconfigjson
prod                esaas-internal-documentation-docker-pull-secret                   kubernetes.io/dockerconfigjson        1      167d
prod                esaas-public-documentation-docker-pull-secret                     kubernetes.io/dockerconfigjson        1      167d
prod                internal-documentation-docker-pull-secret                         kubernetes.io/dockerconfigjson        1      203d
prod                public-documentation-docker-pull-secret                           kubernetes.io/dockerconfigjson        1      203d
prod                website-docker-pull-secret                                        kubernetes.io/dockerconfigjson        1      138d

You can download a secret and check the url in the hash using the following command:

$ kubectl get secrets $SECRET_NAME -n $NAMESPACE -o=json |jq -r '.data | to_entries | .[].value'|base64 -d

You should the previous url

HOWTO fix it ?​

K8Saas team provides a small script that allows you to patch easily your dockerconfigjson.

The script is available here as innersource

To patch all dockerconfigjson:

  • close the project
  • list your secrets with a dockerconfigjson format and its namespace

Then:

$ ./k8saas-update-gitlab-registry.sh $NAMESPACE $SECRET_NAME

Once Done, you probably have to delete all the ImagePullBackOff pods