Troubleshooting Kubernetes Sessions in Posit Workbench: stdin errors within pod

Abstract

How to resolve “stdin” errors created by Kubernetes sessions on Workbench

Description

When attempting to access Kubernetes sessions from a Posit Workbench deployment hosted within a VM, sessions may get stuck in a “starting” state. Although the sessions are shown to be “running” in the details view, they fail to fully initialize.

The following error may appear in logs:

Terminal

ERROR Could not write stdin for pod: session-<sessionID>-<User>---rstudio-pro-<SessionType>; LOGGED FROM: rstudio::job_launcher::impls::kubernetes::KubernetesApi::writeStdinToPod

The root cause of this issue is that the Kubernetes Launcher plugin cannot establish websocket connections to the Kubernetes API to write STDIN.

Solution

1. Verify Launcher Configuration

Confirm /etc/rstudio/launcher.kubernetes.conf exists and is configured according to the Workbench documentation.

2. Check Service Account Permissions

The Launcher’s service account needs permissions to exec into pods. Verify RBAC is correctly configured:

Terminal

kubectl auth can-i create pods/exec --as=system:serviceaccount:<namespace>:rstudio-launcher-rbac

If this returns no, RBAC is misconfigured. The official Posit Workbench Helm chart automatically configures the required RBAC. If you’re not using the official chart, we recommend migrating to it to ensure correct configuration.

The key permissions required for stdin/exec functionality are:

role.yaml

- apiGroups: [""]
  resources: ["pods", "pods/attach", "pods/exec"]
  verbs: ["get", "create", "update", "patch", "watch", "list", "delete"]

3. Test Websocket Connectivity

From the Workbench server, test that websocket connections to the Kubernetes API work:

Terminal

kubectl exec -it <any-running-pod> -- /bin/sh

If this hangs or fails, the issue is between your Workbench node and the Kubernetes API.

4. Verify Callback Address Configuration

The Launcher sessions require bidirectional communication. Verify that launcher-sessions-callback-address in /etc/rstudio/rserver.conf is reachable from pods:

Terminal

# From inside a session pod, test connectivity back to Workbench
kubectl exec -it <session-pod> -- curl -v http://<SERVER-ADDRESS>:8787

If this fails, pods cannot communicate back to Workbench, which may cause session initialization issues.

5. Check Network Policies

List network policies that might block traffic:

Terminal

kubectl get networkpolicies -n <namespace>

Ensure policies allow:

• Egress from the Workbench node to the Kubernetes API server (typically port 6443)
• Ingress from session pods back to the Workbench server (typically port 8787)

6. Review Proxy/Load Balancer Configuration

If traffic to the Kubernetes API passes through a proxy, load balancer, or ingress controller, verify it supports websocket upgrades. Common issues:

• HTTP proxies must support the Connection: Upgrade header
• Load balancers need websocket or TCP passthrough mode enabled
• Timeouts may need to be increased for long-lived websocket connections

7. Examine API Server Logs

Enable audit logging on your Kubernetes cluster to capture requests from the Launcher:

Terminal

# For managed Kubernetes (EKS/AKS/GKE), enable control plane logging in your cloud console
# For self-managed clusters, check kube-apiserver logs:
kubectl logs -n kube-system -l component=kube-apiserver

8. Capture Traffic for Analysis

If the issue persists, capture traffic between the Launcher and the Kubernetes API:

Terminal

tcpdump -i any -w launcher-traffic.pcap port 6443

Share this capture with Posit Support for further analysis.