Step 12. Verify Docker Image Configuration and Access

Kubernetes configuration and environment issues

New remote sessions in Kubernetes appear to start, but are stuck in PENDING status.

Symptoms

New remote sessions in Kubernetes appear to start, but are stuck in PENDING status

Workbench Home Page - New Session Pending Status

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page - Session Info Dialog Box

Cluster Kubernetes
Back-off pulling image "rstudio/r-session-complete:jammy-<workbench version>"

/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 05:24:43 [rstudio-kubernetes-launcher] Received pod update with ID 39
04 May 2020 05:24:47 [rstudio-kubernetes-launcher] Queueing response: {"messageType":0,"requestId":0,"responseId":0}
04 May 2020 05:24:47 [rstudio-kubernetes-launcher] Received getJobState request for rstudio: jobID: * startTime: none endTime: none statuses: none
04 May 2020 05:24:47 [rstudio-kubernetes-launcher] Queueing response: {"messageType":2,"requestId":21,"responseId":44,"jobs":[{"id":"session-4d4356ec67e7b7f57b7e3-rstudio---rstudio-session-6ctxm","name":"Session 4d4356ec67e7b7f57b7e3 (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-<workbench version>"},"status":"Pending","statusMessage":"Back-off pulling image \"rstudio/r-session-complete:<workbench version>\"","lastUpdateTime":"2020-05-04T05:23:07Z","submissionTime":"2020-05-04T05:21:22.815492Z","tags":["4d4356ec67e7b7f57b7e3","4d4356ec67e7b","7f57b7e3","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:4d4356ec67e7b7f57b7e3"]}]}

File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received pod update with ID 53
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 chunk: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","uid":"957f54e6-b165-41b6-908e-71337c6e0866","resourceVersion":"12902","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"BackOff","message":"Back-off pulling image \"rstudio/r-session-complete:jammy-<workbench version>\"","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}|||
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 object: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766.160bbbeb961d67e9","uid":"957f54e6-b165-41b6-908e-71337c6e0866","resourceVersion":"12902","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"BackOff","message":"Back-off pulling image \"rstudio/r-session-complete:jammy-<workbench version>\"","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Normal","eventTime":null,"reportingComponent":"","reportingInstance":""}}
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received event with ID 54
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 chunk: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","uid":"29e7a5e6-af53-42b1-8556-8bf25f7d1ea1","resourceVersion":"12904","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"Failed","message":"Error: ImagePullBackOff","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}|||
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 object: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766.160bbbeb961d8e63","uid":"29e7a5e6-af53-42b1-8556-8bf25f7d1ea1","resourceVersion":"12904","creationTimestamp":"2020-05-04T05:25:14Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sg86766","uid":"07ef102a-6ab0-42ba-82d7-0ccf9626c6d3","apiVersion":"v1","resourceVersion":"12885","fieldPath":"spec.containers{3269596f7a7a2b4d357644334e6e436e4d676b357a673d3d}"},"reason":"Failed","message":"Error: ImagePullBackOff","source":{"component":"kubelet","host":"ip-172-31-10-178.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T05:25:14Z","lastTimestamp":"2020-05-04T05:25:15Z","count":2,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Received event with ID 55
04 May 2020 05:25:15 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":63,"sequences":[{"seqId":37,"requestId":7}],"id":"session-4d4356ec67e7b7f48129e-rstudio---rstudio-session-sgr7g","status":"Pending","statusMessage":"Error: ImagePullBackOff","name":"Session 4d4356ec67e7b7f48129e (rstudio) - RStudio Session"}

Possible cause

For Workbench, Launcher, and Kubernetes to function properly, the Kubernetes worker nodes should be able to pull the Docker images that are used for Launcher sessions from Docker Hub or the Docker registry that is configured in your Kubernetes profile configuration.

The following troubleshooting steps will help you determine if the Docker images are configured correctly in your Kubernetes profile configuration and that the Kubernetes worker nodes have the required network access to pull Docker images from Docker Hub or your Docker registry.

Troubleshooting steps

You should test the functionality of Launcher and Kubernetes using the rstudio/r-session-complete Docker image that is referenced in the documentation for integrating Workbench with Kubernetes. Even if you are planning to use custom Docker images with Launcher, you should initially test the Launcher functionality with the rstudio/r-session-complete Docker images.

Verify that you are using the correct Docker images in your configuration. You should have lines similar to the following in your Kubernetes profile configuration file:

File: /etc/rstudio/launcher.kubernetes.profiles.conf

container-images=rstudio/r-session-complete:jammy-<workbench version>
default-container-image=rstudio/r-session-complete:jammy-<workbench version>

Note

Ensure that the version of the Docker images in your Kubernetes profile configuration file matches the version of Workbench that you have installed.

Verify that you are able to pull the default Docker images from Docker Hub by running the following command from a Kubernetes worker node or a server that is in the same network group as the Kubernetes worker nodes (replace <workbench version> with your desired version of Workbench, including the build number):

$ docker pull rstudio/r-session-complete:jammy-<workbench version>

Note

Using our default images from Docker Hub requires that your Kubernetes worker nodes have access to pull images from Docker Hub for testing.

If that is not possible in your restricted network environment, then you will need to follow the steps outlined in the Using Docker images in an offline environment” section of the support article for Using Docker images with Workbench, Launcher, and Kubernetes.

Restart services and test

After verifying that you are using the correct Docker images in your configuration and checking connectivity to Docker Hub or your Docker registry, restart the Workbench and Launcher services:

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the Workbench home page.

If you have verified that the Kubernetes worker is able to pull images for Docker Hub or made the images available in your own Docker registry and you are still experiencing errors when starting a new session, then proceed to
Step 13 - Verify Kubernetes Worker Nodes.