Step 13. Verify Kubernetes Worker Nodes

Kubernetes configuration and environment issues

New remote sessions in Kubernetes appear to start, but are stuck in PENDING status.

Symptoms

New remote sessions in Kubernetes appear to start, but are stuck in PENDING status

Workbench Home Page - New Session Pending Status

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page - Session Info Dialog Box

Cluster Kubernetes
Status  no nodes available to schedule pods

/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 05:52:23 [rstudio-kubernetes-launcher] Queueing response: {"messageType":0,"requestId":0,"responseId":0}
04 May 2020 05:52:24 [rstudio-kubernetes-launcher] Received getJobState request for rstudio: jobID: * startTime: none endTime: none statuses: none
04 May 2020 05:52:24 [rstudio-kubernetes-launcher] Queueing response: {"messageType":2,"requestId":23,"responseId":29,"jobs":[{"id":"session-4d4356ec67e7b147cd67b-rstudio---rstudio-session-przws","name":"Session 4d4356ec67e7b147cd67b (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio:5000/r-session-complete:jammy-<workbench version>"},"status":"Canceled","lastUpdateTime":"2020-05-04T05:51:22Z","submissionTime":"2020-05-04T05:28:29Z","tags":["4d4356ec67e7b147cd67b","4d4356ec67e7b","147cd67b","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:4d4356ec67e7b147cd67b"]},{"id":"session-4d4356ec67e7bc10879e6-rstudio---rstudio-session-9kc4d","name":"Session 4d4356ec67e7bc10879e6 (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-<workbench version>"},"status":"Pending","statusMessage":"no nodes available to schedule pods","lastUpdateTime":"2020-05-04T05:51:52Z","submissionTime":"2020-05-04T05:50:52.411649Z","tags":["4d4356ec67e7bc10879e6","4d4356ec67e7b","c10879e6","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:4d4356ec67e7bc10879e6"]}]}

Possible cause

For Workbench, Launcher, and Kubernetes to function properly, you should have at least one worker node in the Kubernetes cluster that is available to run pods/containers, and the node should have an adequate amount of CPU and RAM to run Launcher sessions in containers on the Kubernetes cluster.

When starting a new remote session in Kubernetes, if you experience errors in Workbench and Launcher related to insufficient CPU/memory, no worker nodes, or no resources available to schedule pods, then this might be due to a lack of adequate CPU and RAM resources on the Kubernetes worker nodes.

The following troubleshooting steps will help you confirm the existence of worker nodes on your Kubernetes cluster.

Troubleshooting steps

Run the following command to verify that you have worker nodes present on your Kubernetes cluster:

$ kubectl get nodes

which should return output similar to the following if there are worker nodes in your Kubernetes cluster:

NAME                STATUS   ROLES    AGE    VERSION
compute1.internal   Ready    <none>   4h2m   v1.15.10
compute2.internal   Ready    <none>   4h2m   v1.15.10

If there are no worker nodes present on your Kubernetes cluster, then you will see output similar to the following:

No resources found.

If your Kubernetes cluster does not contain any worker nodes, then you should refer to the documentation for your Kubernetes cluster and add one or more worker nodes to the cluster.

For example, when you create a Kubernetes cluster in Amazon EKS, this will only provision the Kubernetes control plane with no worker nodes. You need to perform additional steps to provision worker nodes on your Kubernetes cluster, which is typically done by adding one or more node groups to your EKS cluster.

Test and verify

After adding worker nodes to the Kubernetes cluster, try to start a new session from the Workbench home page.

If you are still experiencing errors when starting a new session, then proceed to
Step 14 - Verify Kubernetes Worker Nodes.