Step 14. Verify Kubernetes Worker Node Resources

Kubernetes configuration and environment issues

New remote sessions in Kubernetes appear to start, but are stuck in PENDING status.

Symptoms

New remote sessions in Kubernetes appear to start, but are stuck in PENDING status

Workbench Home Page - New Session Pending Status

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page - Session Info Dialog Box

Cluster Kubernetes
Status  0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory, 2 Insufficient pods.

/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 06:09:22 [rstudio-kubernetes-launcher] Queueing response: {"messageType":2,"requestId":81,"responseId":213,"jobs":[{"id":"session-4d4356ec67e7b147cd67b-rstudio---rstudio-session-przws","name":"Session 4d4356ec67e7b147cd67b (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio:5000/r-session-complete:jammy-<workbench version>"},"status":"Canceled","lastUpdateTime":"2020-05-04T06:05:12Z","submissionTime":"2020-05-04T05:28:29Z","tags":["4d4356ec67e7b147cd67b","4d4356ec67e7b","147cd67b","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:4d4356ec67e7b147cd67b"]},{"id":"session-4d4356ec67e7be10ec09f-rstudio---rstudio-session-kkq6h","name":"Session 4d4356ec67e7be10ec09f (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-<workbench version>"},"status":"Running","lastUpdateTime":"2020-05-04T06:07:03Z","submissionTime":"2020-05-04T06:00:16.347857Z","tags":["4d4356ec67e7be10ec09f","4d4356ec67e7b","be10ec09f","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:4d4356ec67e7be10ec09f"]},{"id":"session-4d4356ec67e7b586e809c-rstudio---rstudio-session-shsl7","name":"Session 4d4356ec67e7b586e809c (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-<workbench version>"},"status":"Pending","statusMessage":"0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory, 2 Insufficient pods.","lastUpdateTime":"2020-05-04T06:08:22Z","submissionTime":"2020-05-04T06:05:54.394732Z","tags":["4d4356ec67e7b586e809c","4d4356ec67e7b","586e809c","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:4d4356ec67e7b586e809c"]},{"id":"session-4d4356ec67e7bc10879e6-rstudio---rstudio-session-9kc4d","name":"Session 4d4356ec67e7bc10879e6 (rstudio) - RStudio Session","workingDirectory":"","container":{"image":"rstudio/r-session-complete:jammy-<workbench version>"},"status":"Canceled","lastUpdateTime":"2020-05-04T06:01:43Z","submissionTime":"2020-05-04T05:50:52.411649Z","tags":["4d4356ec67e7bc10879e6","4d4356ec67e7b","c10879e6","rstudio-r-session","rstudio-r-session-name:RStudio Session","rstudio-r-session-id:4d4356ec67e7bc10879e6"]}]}
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Received getJobStatus request for rstudio: jobID: * cancel: false
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Adding request id 82 to job status stream for job *
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":214,"sequences":[{"seqId":1,"requestId":82}],"id":"session-4d4356ec67e7b147cd67b-rstudio---rstudio-session-przws","status":"Canceled","name":"Session 4d4356ec67e7b147cd67b (rstudio) - RStudio Session"}
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":215,"sequences":[{"seqId":2,"requestId":82}],"id":"session-4d4356ec67e7be10ec09f-rstudio---rstudio-session-kkq6h","status":"Running","name":"Session 4d4356ec67e7be10ec09f (rstudio) - RStudio Session"}
04 May 2020 06:09:23 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":216,"sequences":[{"seqId":3,"requestId":82}],"id":"session-4d4356ec67e7b586e809c-rstudio---rstudio-session-shsl7","status":"Pending","statusMessage":"0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory, 2 Insufficient pods.","name":"Session 4d4356ec67e7b586e809c (rstudio) - RStudio Session"}

Possible cause

For Workbench, Launcher, and Kubernetes to function properly, you should have at least one worker node in the Kubernetes cluster that is available to run pods/containers, and the node should have an adequate amount of CPU and RAM to run Launcher sessions in containers on the Kubernetes cluster.

When starting a new remote session in Kubernetes, if you experience errors in Workbench and Launcher related to insufficient CPU/memory, no worker nodes, or no resources available to schedule pods, then this might be due to a lack of adequate CPU and RAM resources on the Kubernetes worker nodes.

The following troubleshooting steps will help you determine if the worker nodes on your Kubernetes cluster have an adequate amount of CPU and RAM to run sessions.

Troubleshooting steps

Run the following command to verify the amount of CPU and RAM on the worker nodes in your Kubernetes cluster:

$ kubectl describe nodes

which should return output similar to the following for each node in the Kubernetes cluster (only the relevant CPU and memory information is included here):

Capacity:
 cpu:                         4
 memory:                      15950184Ki
Allocatable:
 cpu:                         3920m
 memory:                      13204840Ki

If you are using the default CPU and RAM values for new sessions from the documentation on integrating Workbench with Kubernetes, then each session will require 1 CPU and 512 MB RAM. The default CPU and RAM values are specified using the default-cpus and default-mem-mb configuration in the Kubernetes profile configuration file.

If you do not have an adequate amount of CPU and RAM on the nodes in your Kubernetes cluster, then you should refer to the documentation for your Kubernetes cluster and add additional worker nodes to the cluster that have an adequate amount of CPU and RAM to be able to run sessions.

Test and verify

After adding additional worker nodes that have an adequate amount of CPU and RAM to run sessions, try to start a new session from the Workbench home page.

If you are still experiencing errors when starting a new session, then proceed to
Step 15 - Verify NFS Server Configuration.