Step 16. Verify NFS Server Connectivity

Kubernetes configuration and environment issues

New remote sessions in Kubernetes appear to start, but are stuck in PENDING status.

Symptoms

New remote sessions in Kubernetes appear to start, but are stuck in PENDING status

Workbench Home Page - New Session Pending Status

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page - Session Info Dialog Box

Cluster Kubernetes
Status  Unable to mount volumes for pod "session-4d4356ec67e7b20322db1-rstudio---rstudio-session-9rrpvcf_rstudio(648baae4-83c1-46a2-a893-e3afa007b799)": timeout expired waiting for volumes to attach or mount for pod "rstudio"/"session-4d4356ec67e7b20322db1-rstudio---rstudio-session-9rrpvcf". list of unmounted volumes=[mount0]. list of unattached volumes=[mount0 default-token-cxqwp]

File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 06:50:17 [rstudio-kubernetes-launcher] Queueing response: {"messageType":3,"requestId":0,"responseId":72,"sequences":[{"seqId":1,"requestId":29}],"id":"session-4d4356ec67e7b493f16b0-rstudio---rstudio-session-t28hc","status":"Pending","statusMessage":"Unable to mount volumes for pod \"session-4d4356ec67e7b493f16b0-rstudio---rstudio-session-t2jfr4j_rstudio(8eddb2ab-3cdc-44bc-8d23-4d490c76b195)\": timeout expired waiting for volumes to attach or mount for pod \"rstudio\"/\"session-4d4356ec67e7b493f16b0-rstudio---rstudio-session-t2jfr4j\". list of unmounted volumes=[mount0]. list of unattached volumes=[mount0 default-token-cxqwp]","name":"Session 4d4356ec67e7b493f16b0 (rstudio) - RStudio Session"}
04 May 2020 06:50:34 [rstudio-kubernetes-launcher] Received watch api/v1/namespaces/rstudio/events?watch=true&timeoutSeconds=300 object: {"type":"MODIFIED","object":{"kind":"Event","apiVersion":"v1","metadata":{"name":"session-4d4356ec67e7b493f16b0-rstudio---rstudio-session-t2jfr4j.160bc074549c6fad","namespace":"rstudio","selfLink":"/api/v1/namespaces/rstudio/events/session-4d4356ec67e7b493f16b0-rstudio---rstudio-session-t2jfr4j.160bc074549c6fad","uid":"0e36fb14-9010-4a16-8216-aa8a34b0eed5","resourceVersion":"22592","creationTimestamp":"2020-05-04T06:48:19Z"},"involvedObject":{"kind":"Pod","namespace":"rstudio","name":"session-4d4356ec67e7b493f16b0-rstudio---rstudio-session-t2jfr4j","uid":"8eddb2ab-3cdc-44bc-8d23-4d490c76b195","apiVersion":"v1","resourceVersion":"22162"},"reason":"FailedMount","message":"Unable to mount volumes for pod \"session-4d4356ec67e7b493f16b0-rstudio---rstudio-session-t2jfr4j_rstudio(8eddb2ab-3cdc-44bc-8d23-4d490c76b195)\": timeout expired waiting for volumes to attach or mount for pod \"rstudio\"/\"session-4d4356ec67e7b493f16b0-rstudio---rstudio-session-t2jfr4j\". list of unmounted volumes=[mount0]. list of unattached volumes=[mount0 default-token-cxqwp]","source":{"component":"kubelet","host":"ip-172-31-18-187.us-east-2.compute.internal"},"firstTimestamp":"2020-05-04T06:48:19Z","lastTimestamp":"2020-05-04T06:50:34Z","count":2,"type":"Warning","eventTime":null,"reportingComponent":"","reportingInstance":""}}

Possible cause

For Workbench, Launcher, and Kubernetes to function properly, both Workbench and the Kubernetes worker nodes should have network connectivity to the NFS server configured in the Launcher mounts configuration.

Error messages in Launcher sessions related to timeouts when mounting volumes are typically due to a lack of network connectivity from Workbench or the Kubernetes worker nodes to the NFS server.

The following troubleshooting steps will help you test and confirm that the NFS server is accessible over the network from other machines, including the Kubernetes worker nodes.

Troubleshooting steps

Check network and firewall rules

Check to see if you are using a private or public IP address in your Launcher mounts configuration file and the associated network and verify the firewall behavior for each case.

Ensure that inbound/ingress network traffic (typically via port 2049 for NFS) is allowed to the NFS server from both Workbench and the Kubernetes worker nodes.

Verify NFS server connectivity from another server

Verify that the NFS server is configured correctly and can be mounted from another server.

Before troubleshooting this connectivity from within the Kubernetes cluster, it is often easier to verify connectivity to the NFS server from another server that is in the same network group as Workbench.

Install the NFS client on a separate server, attempt to mount the /home directory export from the NFS server, then verify that you can access the mounted home directories from that server.

Verify NFS server connectivity from the Kubernetes worker nodes

Verify that the NFS server is configured correctly and can be mounted from the Kubernetes worker nodes.

Ensure that inbound/ingress network traffic (typically via port 2049 for NFS) is allowed to the NFS server from the Kubernetes worker nodes.

You can check the connectivity to the NFS server from a Kubernetes worker node or within a container running with a Kubernetes worker node by installing the NFS client, attempting to mount the /home directory export from the NFS server, then verifying that you can access the mounted home directories from that server.

Test and verify

After verifying connectivity to your NFS server, try to start a new session from the Workbench home page.

If you are still getting errors when starting a new session, then contact our Support team with all of the relevant logs for Launcher and Kubernetes.