Kubernetes Installation

Workbench | Advanced

These instructions describe how to install Workbench on a Kubernetes cluster.

Important

The following sections of this guide assume that the reader has a working knowledge of Kubernetes and Helm.

For alternative installation instructions, see our recommended installation paths.

This page includes instructions for downloading Posit professional products. Download and/or use of these products is governed under the terms of the Posit End User License Agreement. By downloading, you agree to the terms posted there. The same instructions apply if you still use a legacy RStudio Server Pro license configuration (no launcher enabled).

Posit Workbench can run in a Kubernetes cluster where Workbench runs in a pod (or multiple in the case of replicas), and each IDE session/job runs in its own separate pod. This architecture is distinctly different than installing Posit Workbench on a server where IDE sessions/jobs run on the same machine as Workbench. The choice to leverage a container-based Kubernetes infrastructure can ease the management of resource constraints, maximize process isolation, improve reproducibility, and simplify the maintenance of R and Python environments and system dependencies.

Posit recommends using the Workbench Helm chart to install and run Workbench in a Kubernetes cluster.

For help with any other topology or deployment options, please contact your Posit Customer Success Representative and request an architecture review session with Posit Solutions Engineering.

Feature requirements

  • A supported version of Posit Workbench
  • A valid Posit Workbench license
  • Kubernetes:
  • PostgreSQL database
  • StorageClass backed by POSIX-compliant PersistentVolume storage that supports symlinks and ReadWriteMany access
    • PersistentVolume storage backed by AWS EFS must be statically provisioned.

Planning

Several planning and preparation steps should be performed ahead of the deployment sequence.

Pull the Helm charts

To pull the Helm charts used in this guide:

# Posit Workbench
helm repo add rstudio https://helm.rstudio.com

# Pull the latest versions of the Helm charts
helm repo update

Network connectivity verification

  • Verify that the Kubernetes cluster has connectivity to your shared storage.
  • Verify that the Kubernetes cluster has connectivity to your PostgreSQL database.

Shared storage capacity verification

When running Workbench in Kubernetes, shared storage is required to persist data. Your shared storage must be POSIX-compliant and be accessible to your cluster. Before beginning the installation, ensure you have enough space in your shared storage. This is used for user home directories and server-shared-storage-path (shared storage directory for Workbench). We recommend 100+ GB of combined storage in our Recommended system requirements.

Posit Workbench license validation

A valid Posit Workbench license is required to run Posit Workbench in Kubernetes. If you use an existing Posit Workbench license, you must also confirm that you have enough license activations available. Check your license status using the steps outlined in the Licensing Management guide. If you have questions, reach out to your Posit Customer Success representative before proceeding.

To request a separate evaluation license, you can email Customer Success or sales@posit.co and specify that you are trialing this feature.

Kubernetes cluster preparation

Create a namespace for Posit Workbench

A Kubernetes namespace is required for Posit Workbench. We recommend creating a new one called posit-workbench or having a cluster administrator create one on your behalf.

This can be accomplished with the following commands:

# Create the new namespace
kubectl create namespace posit-workbench

# Switch to the new namespace in your current context
kubectl config set-context --current --namespace=posit-workbench

Create a StorageClass with ReadWriteMany access

Your cluster must have a StorageClass backed by POSIX-compliant PersistentVolume (PV) storage that supports symlinks and ReadWriteMany access. This storage class is used by PersistentVolumeClaim (PVC) objects to either dynamically provision PV objects or use static PV objects for user home directory storage, and Workbench shared storage.

Additionally, PersistentVolume storage backed by AWS EFS must be statically provisioned. For more information, review the aws-efs-csi-driver: Static Provisioning documentation.

Create a Secret containing a license file

We recommend storing a license file as a Secret and setting the license.file.secret and license.file.secretKey values accordingly as shown in Configure Helm chart values.

Create the Secret declaratively with YAML or imperatively using the following command:

kubectl create secret generic rstudio-workbench-license --from-file=licenses/rstudio-workbench.lic

Create a Secret containing a PostgreSQL database password

We recommend storing the PostgreSQL database password as a Secret and making it available to the container as an environment variable, as shown in Configure Helm chart values.

Create the secret declaratively with YAML or imperatively using the following command (replace with your own password):

kubectl create secret generic rstudio-workbench-database --from-literal=password=YOURPASSWORDHERE

Configure Helm chart values

Posit maintains a Helm chart that is recommended for deploying Posit Workbench on Kubernetes. It is highly configurable and supports multiple deployment options to meet your organization’s requirements.

The values.yaml file is used to override defaults specified within the Helm chart. The steps below will help you set the values for the initial deployment.

The config section of your values.yaml allows for setting application configuration options in one section which are converted to the correct format and mounted to the right location. To see a full list of how certain configuration values are translated into specific files and mounts, see the Configuration files section of the Helm chart README.

Create your initial values.yaml file

Create a file called values.yaml with the following contents:

# Controls how many instances of Posit Workbench are created.
replicas: 1

# Mounts the license file appropriately from the Secret
license:
  file:
    # Replace with the name of your license file secret and key
    secret: rstudio-workbench-license
    secretKey: rstudio-workbench.lic

# Configures user home directory shared storage
homeStorage:
  create: true
  mount: true

  # The name of the PVC created for Workbench's user home directory shared storage.
  name: workbench-user-pvc

  # The storageClass to use for Workbench's user home directory. Must support RWX.
  # Replace with your storage class name.
  storageClassName: nfs-rwx
  requests:
    storage: 100G

# Configures Workbench shared storage
sharedStorage:
  create: true
  mount: true

  # The name of the PVC created for Workbench's shared storage directory.
  name: workbench-shared-pvc

  # The storageClass to use for Workbench's shared storage directory. Must support RWX.
  # Replace with your storage class name.
  storageClassName: nfs-rwx
  requests:
    storage: 1G

# Adds an environment variable containing the PostgreSQL password from a Secret
pod:
  env:
    - name: WORKBENCH_POSTGRES_PASSWORD
      valueFrom:
        secretKeyRef:
          # Replace with the name of your database password secret and key
          name: rstudio-workbench-database
          key: password

# Creates a test user to verify installation
userCreate: true

# The config section is converted into the correct configuration files and are
# mounted to the server/session pods as needed.
config:
  secret:
    database.conf:
      provider: "postgresql"
      connection-uri: "postgres://<USERNAME>@<HOST>:<PORT>/<DATABASE>?sslmode=allow"
      # While it is possible to set a Postgres password here in the values file,
      # we recommend adding it from a Secret as an environment variable as shown in pod.env
  session:
    rsession.conf:
      # These settings apply to RStudio Pro IDE sessions
      session-timeout-minutes: 60
      session-timeout-suspend: 1
      session-quit-child-processes-on-exit: 1
    repos.conf:
      # This will set the Posit Public Package Manager (P3M) as the default R repository
      # for Workbench users. This is recommended as P3M provides linux binaries for many
      # R packages which will decrease package installation time. If you have your own
      # Posit Package Manager server then replace these URLs with the URLs of your server.
      CRAN: https://packagemanager.posit.co/cran/__linux__/jammy/latest
  server:
    jupyter.conf:
      # These settings apply to Jupyter Notebook and JupyterLab IDE sessions
      session-cull-minutes: 60
      session-shutdown-minutes: 5
  profiles:
    launcher.kubernetes.profiles.conf:
      "*":
        # These settings are applied for all users and can be changed to suit
        # your particular user needs. See the following resources for more information:
        # https://github.com/rstudio/helm/tree/main/charts/rstudio-workbench#etcrstudiolauncherkubernetesprofilesconf
        # https://docs.posit.co/ide/server-pro/job_launcher/kubernetes_plugin.html#kube-profiles
        default-cpus: "1.0"
        default-mem-mb: "2048"
        max-cpus: "4.0"
        max-mem-mb: "8192"
Caution

For initial deployment and testing the sample values.yaml file includes the setting userCreate: true. This creates a test user rstudio with a password of rstudio. This should only be used for initial install verification and not used in production. See the user provisioning section for more information about user provisioning setup considerations. When setting up user provisioning, userCreate should then be set to false.

Replace the sample values

You should modify the initial values.yaml file to match the needs of your environment. For example, you will need to fill in the placeholder values for the PostgreSQL connection string, specify the name of your storage class that supports ReadWriteMany access (replace nfs-rwx with your specific storage class name), specify the name of the Secret containing your license file and specify the name of the Secret containing your PostgreSQL database password.

If you would like to view the chart’s entire set of default values, use the command:

helm show values rstudio/rstudio-workbench

Kubernetes deployment

Installing Posit Workbench within Kubernetes

To complete your installation of Posit Workbench within Kubernetes, run the following commands:

helm upgrade --install rstudio-workbench-prod \
    rstudio/rstudio-workbench \
    --values values.yaml

To ensure a stable production deployment please:

  • Ensure you “pin” the version of the Helm chart that you are using. You can do this using the helm dependency command and the associated “Chart.lock” files or the --version flag.

    Important

    Pinning the version protects you from breaking changes. For example, to pin the release to version 0.6.2 add the --version=0.6.2 flag to your helm upgrade --install command.

  • Use helm diff upgrade before upgrading, to avoid breaking changes. This requires the helm-diff plugin.

  • Pay close attention to the Helm chart NEWS.md for updates on changes.

Use the following command to check the status of Workbench:

kubectl get pod -l app.kubernetes.io/name=rstudio-workbench

You should see output like the following:

NAME                                      READY   STATUS    RESTARTS   AGE
rstudio-workbench-prod-5d45d7b9bc-5wrcn   2/2     Running   0          9m50s
Note

The Workbench pod indicates that 2/2 containers are running. In addition to the Workbench container, the pod includes a Graphite exporter container which allows Prometheus to consume metrics about the pod’s health. When prometheusExporter.enabled is true (the default), prometheus.io/ annotations are added to the pod so that metrics can be easily consumed.

If your Workbench pod is failing to start, see the Debugging workbench in Kubernetes section for details on how to diagnose and fix problems with your deployment.

Access Workbench and validate your installation

Now, Posit Workbench should be running. To confirm the successful completion of this phase, log into the application and validate the correct functionality as described below.

Warning

Manual port-forwarding is appropriate for local testing and validation of the installation. Once you have validated the installation, configure an Ingress and DNS records for Workbench. See the configure external access section for more details.

To interact with your new Workbench installation, temporarily enable port-forwarding. For example, to use local port 8787:

kubectl port-forward svc/rstudio-workbench-prod 8787:80

From your browser, navigate to http://localhost:8787/ and log into Workbench using the test user created with the userCreate: true setting in the values.yaml (username: rstudio, password: rstudio).

To verify the correct functionality of your Workbench server:

  • Log in successfully
  • Start and join an RStudio Pro session
  • Start and join a JupyterLab session
  • Start and join a Jupyter Notebook session
  • Start and join a VS Code session
Note

Initially, sessions may be slow to start as container images must be pulled and certain extensions installed.

Post-deployment considerations

This section guides you through the post-deployment steps for your Kubernetes installation of Posit Workbench.

Updating and changing the deployment

If you have made changes to your values file and wish to update an existing installation, you can edit your values.yaml and run the same helm upgrade command again.

Custom container image preparation

Some organizations want control over the Docker images used rather than using the public images which Posit makes available. There are two different images to consider:

  • rstudio-workbench: which is used for the actual Workbench pods.
  • r-session-complete: which is used for the IDE sessions and jobs. Both can be extended and used in the Helm chart.

rstudio-workbench images

By default, the Helm chart uses ubuntu2204 images from the rstudio/rstudio-workbench repository for Workbench containers, which are controlled by the image. settings in the values.yaml file.

Docker Hub: https://hub.docker.com/r/rstudio/rstudio-workbench
GitHub: https://github.com/rstudio/rstudio-docker-products/tree/dev/workbench

r-session-complete images

By default, the Helm chart uses ubuntu2204 images from the rstudio/r-session-complete repository for session containers, which are controlled by the session.image. settings in the values.yaml file.

Docker Hub: https://hub.docker.com/r/rstudio/r-session-complete
GitHub: https://github.com/rstudio/rstudio-docker-products/tree/dev/r-session-complete

Using custom images

To change the image used to create the main Workbench pods, set the following values:

image:
  repository: "yourprivateregistry.com/rstudio-workbench"
  tag: "ubuntu2204-2023.03.1-custom"

To change the image used to create Workbench session pods, set the following values:

session:
  image:
      repository: "yourprivateregistry.com/r-session-complete"
      tag: "ubuntu2204-2023.03.1-custom"

As shown above, Setting the session image in the values.yaml sets the default image for all users when they go to launch a session. The selection of images presented to each user and group can be customized further by setting default-container-image and container-images in launcher.kubernetes.profiles.conf.

See the setup user and group profiles section for more details on how settings apply globally, per group and user. For example to setup three custom images for all users to choose from:

config:
  profiles:
    launcher.kubernetes.profiles.conf:
      "*":
        default-container-image: "yourprivateregistry.com/r-session-complete:ubuntu2204-2023.03.1"
        container-images:
          - "yourprivateregistry.com/r-session-complete:ubuntu2204-2023.03.1"
          - "yourprivateregistry.com/r-session-complete:ubuntu2204-2023.03.1-old-versions"
          - "yourprivateregistry.com/r-session-complete:ubuntu2204-2023.03.1-ml-dep"

Implement load balancing

With the Helm Chart values.yaml file created earlier, the deployment of Workbench was configured with one replica so that traffic for a single connection is always routed to the same Workbench pod.

To implement multiple replicas of the Workbench pod, update the replicas count in your values.yaml file and then run helm upgrade. For example, the following change would enable three running replicas:

# Controls how many instances of Workbench are created.
replicas: 3

Setup user provisioning

Workbench supports a variety of authentication methods including Local Accounts, LDAP/Active Directory, and Single Sign-On (SSO). For a full list review the Authenticating users page. Regardless of the method used for authentication, users must be created in the Workbench containers.

The most common way to provision users is via sssd. The latest Posit Workbench container has sssd included and running by default.

The sssd configuration can be set in the values.yaml file under config.userProvisioning. For example:

config:
  userProvisioning:
    mysssd.conf:
      sssd:
        config_file_version: 2
        services: nss, pam
        domains: rstudio.com
      domain/rstudio.com:
        id_provider: ldap
        auth_provider: ldap
Caution

For initial deployment and testing the sample values.yaml file included the setting userCreate: true. This creates a test user rstudio with a password of rstudio. This should only be used for initial install verification and not used in production. When setting up user provisioning with sssd, userCreate should then be set to false.

Setup user and group profiles

User and group profiles can be used to customize various aspects of the sessions created by each user. These settings include default CPU/memory, max CPU/memory, default image, available images, placement constraints, and more. For a full list of options, see the User and group profiles.

These options can be set globally across all users, per group or per user in the values.yaml under config.profiles.launcher.kubernetes.profiles.conf. See the /etc/rstudio/launcher.kubernetes.profiles.conf section of the Helm chart README for more details on the required format.

In the example values.yaml provided in an earlier section, there are some default values set which can be customized to suit your user and infrastructure needs.

config:
  profiles:
    launcher.kubernetes.profiles.conf:
      "*":
        default-cpus: "1.0"
        default-mem-mb: "2048"
        max-cpus: "4.0"
        max-mem-mb: "8192"

Setup resource profiles

Resource profiles can be set up to simplify the task of assigning CPU, memory, or GPU resources. Instead of setting resources by group or user, you create resource profiles with user-friendly names with certain resources and users choose from a list of profiles available to them.

Here is an example of resource profiles that can be specified in your values.yaml:

config:
  server:
    launcher.kubernetes.resources.conf:
      "small":
        cpus: "1.0"
        mem-mb: "512"
      "medium":
        cpus: "2.0"
        mem-mb: "4096"
      "large":
        cpus: "3.0"
        mem-mb: "8192"

By default, all profiles are available to all users, however the resource-profiles option in /etc/rstudio/launcher.kubernetes.profiles.conf can be used to control which resource profiles groups, and users have access to.

For a full list of options see Resource profiles.

Configure external access

For users to access your installation of Posit Workbench running in Kubernetes, you need to configure an Ingress. There are many different ways to accomplish this, and the steps may vary depending on the requirements of your organization.

In this guide, we use the Traefik v2 Ingress Controller to configure external access to our Posit Workbench instance using locally managed TLS certificates. We use the value workbench.posit.co as our public domain name in this example, but you should modify this everywhere it occurs to use your own domain.

It is also possible to use external certificate management tools (like cert-manager, Amazon ACM, etc.) if you prefer not to manage local certificates, but the configurations for these varies depending on which Ingress Controller and certificate manager is used.

Step 1: Install the Traefik Ingress Controller

Although the Traefik documentation provides detailed installation instructions, the simplest installation steps are:

helm repo add traefik https://helm.traefik.io/traefik
helm repo update
helm install traefik traefik/traefik

Step 2: Create TLS Secrets

Replace workbench.crt and workbench.key with the local path to your TLS certificate files.

kubectl create secret tls workbench-tls \
    --cert workbench.crt \
    --key workbench.key

Step 3: Configure the Ingress in your Helm chart values

service:
  # For High Availability installations of Posit Workbench, where
  # multiple `replicas` of the Workbench pod are in play, it is
  # necessary to enable "sticky sessions" so that traffic for a
  # single connection is always routed to the same Workbench pod.
  annotations:
    traefik.ingress.kubernetes.io/service.sticky.cookie: "true"
    traefik.ingress.kubernetes.io/service.sticky.cookie.name: WORKBENCH-SESSION-COOKIE
    traefik.ingress.kubernetes.io/service.sticky.cookie.secure: "true"
    traefik.ingress.kubernetes.io/service.sticky.cookie.samesite: "none"
    traefik.ingress.kubernetes.io/service.sticky.cookie.httponly: "true"

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: traefik

  hosts:
    - host: workbench.posit.co
      paths:
        - /

  # Tell the ingress controller to use your TLS secret
  tls:
    - secretName: workbench-tls
      hosts:
        - workbench.posit.co

Step 4: Apply the changes to your installation

See the updating and changing the deployment section to see how to apply these changes to an existing installation.

Step 5: Create DNS records for your installation

In order for you to access your Posit Workbench installation via an Ingress, you must create a DNS record. There are many different DNS service providers to choose from, or you can host your own DNS servers. Creating the DNS records is out of scope for this guide as the process most likely varies for each organization.

Note

A common way to do this in Kubernetes is by automating the provisioning of DNS records by using a tool like external-dns.

For this guide, the EXTERNAL-IP of the Traefik Ingress Controller Service must resolve to workbench.posit.co. To obtain the EXTERNAL-IP of the Ingress Controller, inspect the Service that was created by the Traefik Helm chart:

kubectl get svc traefik

You should see output like the following:

NAME      TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                      AGE
traefik   LoadBalancer   10.110.77.164   <xx.xx.xx.xx>     80:31869/TCP,443:31047/TCP   20s

Once your DNS records are in place, you can use netcat to verify that your new DNS records resolve to the correct host. In the example below, update your host path for workbench.posit.co:

nc -vz workbench.posit.co 443

Output:

Connection to workbench.posit.co port 443 [tcp/https] succeeded!

Step 6: Connect to the Workbench homepage

You should now be able to visit Workbench’s homepage through your web browser.

Debugging Workbench in Kubernetes

Startup failure

If your Posit Workbench pod is failing to start, use the kubectl describe command to get its diagnostic information:

kubectl describe pod -l app.kubernetes.io/name=rstudio-workbench

It’s possible for the pod’s Events to indicate an error. In this case, we can see that the Posit Workbench pod is failing to start and Kubernetes is repeatedly attempting to restart it.

Name:         rstudio-workbench-prod-57b9fcd98f-7zbx2
...
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  27s                default-scheduler  Successfully assigned posit-workbench/rstudio-workbench-prod-57b9fcd98f-7zbx2 to gke-workbench-default-pool-2dbe9201-vlv8
  Normal   Pulled     22s                kubelet            Container image "prom/graphite-exporter:v0.9.0" already present on machine
  Normal   Created    22s                kubelet            Created container exporter
  Normal   Started    22s                kubelet            Started container exporter
  Normal   Pulled     14s (x2 over 22s)  kubelet            Container image "rstudio/rstudio-workbench:ubuntu2204-2023.06.0" already present on machine
  Normal   Created    14s (x2 over 22s)  kubelet            Created container rstudio
  Normal   Started    14s (x2 over 22s)  kubelet            Started container rstudio
  Warning  Unhealthy  10s (x3 over 19s)  kubelet            Readiness probe failed: Get "http://10.96.2.16:8787/health-check": dial tcp 10.96.2.16:8787: connect: connection refused
  Warning  BackOff    8s                 kubelet            Back-off restarting failed container

If the container fails, you can check the logs of the failed container. This can be done with the following command:

kubectl logs -f --tail 50 \
    -l app.kubernetes.io/name=rstudio-workbench \
    --container rstudio

In Workbench’s logs, we can see that the container is failing to start because Workbench cannot reach our Postgres database:

2023-06-28T21:03:24.048603Z [rserver] INFO Reading database configuration from '/mnt/secret-configmap/rstudio/database.conf'
2023-06-28T21:03:24.048907Z [rserver] INFO Connecting to Postgres database: postgres://workbench@workbench.posit.co:5432/workbench?sslmode=disable
2023-06-28T21:03:24.049003Z [rserver] INFO Creating database connection pool of size 2 (source: logical CPU count)
2023-06-28T21:03:24.049414Z [rserver] WARNING A plain text value is potentially being used for the PostgreSQL password, or an encrypted password could not be decrypted. The RStudio Server documentation for PostgreSQL shows how to encrypt this value.; LOGGED FROM: rstudio::core::Error rstudio::core::database::ConnectVisitor::getPassword(const rstudio::core::database::PostgresqlConnectionOptions&, std::string&) const src/cpp/core/Database.cpp:426
2023-06-28T21:03:24.088341Z [rserver] ERROR database error 7 (Cannot establish connection to the database.
could not translate host name "workbench.posit.co" to address: Name or service not known
); OCCURRED AT rstudio::core::Error rstudio::core::database::ConnectVisitor::operator()(const rstudio::core::database::PostgresqlConnectionOptions&) const src/cpp/core/Database.cpp:235; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:769
+ deactivate
+ echo '== Exiting =='
+ rstudio-server stop
== Exiting ==
+ echo 'Deactivating license ...'

Opening a support ticket

If you have a support plan with Posit and would like to file a request for help with your Posit Workbench deployment on Kubernetes, please execute the following script and attach the resulting diagnostics to your ticket.

Using this script requires that the deployment was created with the helm upgrade --install command as outlined in the Installing Posit Workbench within Kubernetes section.

#!/usr/bin/env bash
# Copyright (C) 2023 by Posit Software, PBC.

set -euxo pipefail

NAMESPACE="${1:-posit-workbench}"
RELEASE_NAME=$(helm list -n $NAMESPACE -o yaml | grep "chart: rstudio-workbench-[0-9]" -A1 | grep name | awk '{print $2}')

echo "### Kubernetes version ###"
kubectl version
echo

echo "### Helm version ###"
helm version
echo

echo "### Helm releases (namespace: $NAMESPACE) ###"
helm list -n $NAMESPACE
echo

echo "### values.yaml (release: $RELEASE_NAME) ###"
helm get values -n $NAMESPACE $RELEASE_NAME | grep -v "password:"
echo

echo "### Posit Workbench Pod describe ###"
kubectl describe pod -n $NAMESPACE -l app.kubernetes.io/name=rstudio-workbench
echo

echo "### Posit Workbench server logs ###"
kubectl logs -n $NAMESPACE $(kubectl get pod -n $NAMESPACE -l app.kubernetes.io/name=rstudio-workbench -o=jsonpath='{.items[0].metadata.name}') -c rstudio

To produce a diagnostic file for your support ticket:

  1. Save the above script to a file called posit-workbench-run-diagnostics-k8s.sh

  2. Make the script executable:

    chmod 750 ./posit-workbench-run-diagnostics-k8s.sh

  3. Invoke the script and save the output to a file:

    ./posit-workbench-run-diagnostics-k8s.sh > posit-workbench-diagnostic-info-k8s.txt

  4. Attach the output file posit-workbench-diagnostic-info-k8s.txt to your support ticket.

    Note

    posit-workbench-run-diagnostics-k8s.sh accepts an optional argument that can be used to provide a namespace other than posit-workbench. This allows you to invoke the script with a non-default namespace:

    ./posit-workbench-run-diagnostics-k8s.sh my-custom-namespace > posit-workbench-diagnostic-info-k8s.txt
Back to top