3 Kubernetes Plugin

The Kubernetes Job Launcher Plugin provides the capability to launch executables on a Kubernetes cluster.

3.1 Configuration

It is recommended not to change any of the default values and only configure required fields as outlined below.

/etc/rstudio/launcher.kubernetes.conf

Config Option	Description	Required (Y/N)	Default Value
server-user	Service user. The plugin should be started as root, and will lower its privilege to this user for normal execution. It is recommended not to change the default value, as this is populated by the Launcher service itself.	N	rstudio-server
thread-pool-size	Size of the thread pool used by the plugin. It is recommended not to change the default value, as this is populated by the Launcher service itself.	N	Number of CPUs * 2
enable-debug-logging	Enables/disables verbose debug logging. Can be 1 (enabled) or 0 (disabled).	N	0
scratch-path	Scratch directory where the plugin writes temporary state.	N	/var/lib/rstudio-launcher/{name of plugin}
logging-dir	Specifies the path where debug logs should be written.	N	/var/log/rstudio/launcher
job-expiry-hours	Number of hours before completed jobs are removed from the system.	N	24
profile-config	Path to the user and group profiles configuration file (explained in more detail below).	N	/etc/rstudio/launcher.kubernetes.profiles.conf
api-url	The Kubernetes API base URL. This can be an HTTP or HTTPS URL. The URL should be up to, but not including the /api endpoint.	Y	Example: https://192.168.99.100:8443
auth-token-path	The path to a file that contains the auth token for the job-launcher service account. This is used to authenticate with the Kubernetes API. See below for more information. Required unless auth-token is set.	Y
auth-token	The auth token for the job-launcher service account. This is used to authenticate with the Kubernetes API. See below for more information. Required if auth-token-path is not set.	N
kubernetes-namespace	The Kubernetes namespace in which to create jobs. Note that the account specified by the auth-token setting must have full API privileges within this namespace. See Kubernetes Cluster Requirements below for more information.	N	rstudio
shared-process-namespace	Use a shared process namespace. This improves behavior when sending termination signals to processes.	N	true
verify-ssl-certs	Whether or not to verify SSL certificates when connecting to api-url. Only applicable if connecting over HTTPS. Do not disable this option in production use.	N	1
certificate-authority	Certificate authority to use when connecting to Kubernetes over SSL and when verifying SSL certificates. This must be the Base64-encoded PEM certificate reported by Kubernetes as the certificate authority in use. Leave this blank to use the system root CA store.	N
watch-timeout-seconds	Number of seconds before the watch calls to Kubernetes stop. It is strongly recommended to not change this value unless instructed by RStudio support.	N	180
fetch-limit	The maximum amount of objects to request per API call from the Kubernetes Service for GET collection requests. It is recommended you only change the default if you run into size issues with the returned payloads.	N	500
use-templating	Enables the new Kubernetes object templating feature (see Kubernetes Object Templating below). When enabled, any configured job-json-overrides are ignored.	N	0

In order to generate the contents for the file pointed to by auth-token-path (or the value for auth-token), run the following commands. Note that the account must first be created and given appropriate permissions (see Kubernetes Cluster Requirements below). The file pointed to by auth-token-path must be owned by the account configured as your server-user (usually rstudio-server).

KUBERNETES_AUTH_SECRET=$(kubectl get serviceaccount job-launcher --namespace=rstudio -o jsonpath='{.secrets[0].name}')

# Write token to file. This file must be owned by the `server-user` (usually `rstudio-server`).
kubectl get secret $KUBERNETES_AUTH_SECRET --namespace=rstudio -o jsonpath='{.data.token}' | base64 -d > /etc/rstudio/kubernetes.launcher.token
chmod 0600 /etc/rstudio/kubernetes.launcher.token
sudo chown rstudio-server /etc/rstudio/kubernetes.launcher.token

# Print token for copy/paste.
kubectl get secret $KUBERNETES_AUTH_SECRET --namespace=rstudio -o jsonpath='{.data.token}' | base64 -d

3.1.1 Kubernetes Container Auto Configuration

If you are running the Launcher within a Kubernetes container, a few configuration variables can be inferred automatically by using Kubernetes-injected environment variables and files. These values are automatically added by Kubernetes when a container is launched. Therefore, it is not required to configure these options when running the Launcher within Kubernetes.

Config Option	Obtained From
api-url	https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}
auth-token	/var/run/secrets/kubernetes.io/serviceaccount/token
certificate-authority	Base64-encoded value of /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

3.1.2 User and Group Profiles

The Kubernetes plugin also allows you to specify user and group configuration profiles, similar to RStudio Workbench’s profiles, in the configuration file /etc/rstudio/launcher.kubernetes.profiles.conf (or any arbitrary file as specified in profile-config within the main configuration file; see above). These are entirely optional.

Profiles are divided into sections of three different types:

Global ([*])

Per-group ([@groupname])

Per-user ([username])

Here’s an example profiles file that illustrates each of these types:

/etc/rstudio/launcher.kubernetes.profiles.conf

[*]
placement-constraints=node,region:us,region:eu
default-cpus=1
default-mem-mb=512
max-cpus=2
max-mem-mb=1024
container-images=r-session:3.4.2,r-session:3.5.0
allow-unknown-images=0

[@rstudio-power-users]
default-cpus=4
default-mem-mb=4096
default-nvidia-gpus=0
default-amd-gpus=0
max-nvidia-gpus=2
max-amd-gpus=3
max-cpus=20
max-mem-mb=20480
container-images=r-session:3.4.2,r-session:3.5.0,r-session:preview
allow-unknown-images=1

[jsmith]
max-cpus=3

This configuration specifies that by default users will be allowed to launch jobs with a maximum of 1024 MB of memory, and use only two different R containers. It also specifies that members of the rstudio-power-users group will be allowed to use much more resources, including GPUs, and the ability to see the r-session:preview image, in addition to being able to run any image they specify.

Note that the profiles file is processed from top to bottom (i.e. settings matching the current user that occur later in the file always override ones that appeared prior). The settings available in the file are described in more depth in the table below.

/etc/rstudio/launcher.kubernetes.profiles.conf

Config Option	Description	Required (Y/N)	Default Value
container-images	Comma-separated string of allowed images that users may see and run.	N
default-container-image	The default container image to use for the Job if none is specified.	N
allow-unknown-images	Whether or not to allow users to run any image they want within their job containers, or if they have to use the ones specified in container-images	N	1
placement-constraints	Comma-separated string of available placement constraints in the form of key1:value1,key2:value2,... where the :value part is optional to indicate free-form fields. See next section for more details	N
default-cpus	Number of CPUs available to a job by default if not specified by the job.	N	0.0 (infinite - managed by Kubernetes)
default-mem-mb	Number of MB of RAM available to a job by default if not specified by the job.	N	0.0 (infinite - managed by Kubernetes)
max-cpus	Maximum number of CPUs available to a job.	N	0.0 (infinite - managed by Kubernetes)
max-mem-mb	Maximum number of MB of RAM available to a job.	N	0.0 (infinite - managed by Kubernetes)
job-json-overrides	JSON path overrides of the generated Kubernetes Job JSON. See Modifying Jobs.	N
cpu-request-ratio	Ratio within the range (0.0, 1.0] representing the Kubernetes container resource request to set for the CPU. This will be the ratio of the limit amount specified by the user when creating the job.	N	1.0
memory-request-ratio	Ratio within the range (0.0, 1.0] representing the Kubernetes container resource request to set for the memory. This will be the ratio of the limit amount specified by the user when creating the job.	N	1.0
default-nvidia-gpus	Number of NVIDIA GPUs available to a job by default if not specified by the job. See below for more information.	N	0
default-amd-gpus	Number of AMD GPUs available to a job by default if not specified by the job. See below for more information.	N	0
max-nvidia-gpus	Maximum number of NVIDIA GPUs available to a job. See below for more information.	N	0
max-amd-gpus	Maximum number of AMD GPUs available to a job. See below for more information.	N	0

Note that resource limits correspond to the Kubernetes container resource limits, which represent hard caps for the resources a job can use. Kubernetes allows jobs to request less resources and occasionally burst up to the limit amount, and this can be controlled by setting the cpu-request-ratio and memory-request-ratio settings as detailed above. Note that resource management in Kubernetes is a complex topic, and in general you should simply leave these to the default value of 1.0 unless you understand the implications of using both requests and limits. See here for more information.

In order to provide GPUs as a schedulable resource, you must first enable the feature in Kubernetes by installing the necessary GPU drivers and device plugins supplied by your desired vendor (AMD or NVIDIA). Once available in Kubernetes, simply set the desired default and max values for the GPU type you intend to use. If not using GPUs, no GPU configuration in the profiles is necessary. For information on adding support for GPUs in Kubernetes, see the Kubernetes documentation.

3.1.3 Modifying Jobs

Note: job-json-overrides are deprecated. We recommend you use the new Kubernetes Object Templating feature instead.

Whenever a job is submitted to the Kubernetes Launcher plugin, a JSON job object is generated and sent to Kubernetes. In some cases, it may be desirable to add or modify fields within this automatically generated JSON blob.

In order to do that, you may specify job-json-overrides within the profiles file. The form of the value should be "{json path}":"{path to json value file}","{json path 2}":"{path to json value file 2}",....

The JSON path should be a valid JSON path pointer as specified in the JSON Pointer RFC.

The JSON value path specified must be a file readable by the service user, and must contain valid JSON. For example, to add Host Aliases to all submitted jobs:

/etc/rstudio/launcher.kubernetes.profiles.conf

job-json-overrides="/spec/template/spec/hostAliases":"/etc/rstudio/kube-host-aliases"

/etc/rstudio/kube-host-aliases

[
  {
    "ip": "10.2.141.12",
    "hostnames": ["db01"]
  },
  {
    "ip": "10.2.141.13",
    "hostnames": ["db02"]
  }
]

Because the pod itself is nested within the Kubernetes Job object, it is located at the path /spec/template/spec. In the example above, we simply add a JSON object representing the HostAlias array as defined by the Kubernetes API. See the Kubernetes API Documentation for an exhaustive list of fields that can be set.

Any job-json-overrides-specified fields will overwrite already existing fields in the auto-generated job spec. Note that the Kubernetes Launcher plugin requires certain fields to be set in order to properly parse saved job data. It is strongly recommended you use the job-json-overrides feature sparingly, and only use it to add additional fields to the automatically generated job object when necessary.

3.1.4 Kubernetes Object Templating

The new preferred method for modifying objects (jobs and services) submitted to Kubernetes is to use the new templating feature. This allows templating of the entire YAML payload that is submitted to the Kubernetes API using a syntax similar to Helm charts. This new method is easier to use than the previous job-json-overrides functionality, and allows for the use of conditional logic to make job modification dynamic, as opposed to the static transformations offered by job-json-overrides.

To use the templating feature, enable it in /etc/rstudio/launcher.kubernetes.conf:

/etc/rstudio/launcher.kubernetes.conf

use-templating=1

3.1.4.1 Generating Templates

After restarting the Launcher, job and service templates will automatically be written to the Kubernetes Launcher scratch-path (/var/lib/rstudio-launcher/Kubernetes by default), and these templates will be used to to create the jobs and services that are submitted to Kubernetes when starting Launcher jobs. These templates will only be created if they do not already exist to ensure that any changes made are not overwritten.

You can also generate the templates via a command instead of having to first run the Launcher to create them. This can be done with the --generate-templates command:

sudo /path/to/rstudio-kubernetes-launcher --generate-templates

Note that the templates will be created in the scratch-path as mentioned above. Only the templates found in the scratch-path will be used for templating. The scratch-path is determined by parsing the launcher.conf and launcher.kubernetes.conf files. If these files do not yet exist, you can specify the scratch-path by adding --scratch-path <path> to the --generate-templates command.

3.1.4.2 Modifying Templates

Once the templates have been generated, you can modify them as necessary for your Kubernetes cluster. For example, you can add additional annotations to jobs, add Linux capabilities to the container, or even run a side-car container.

To modify the templates, edit the job.tpl and service.tpl files that were previously generated within the scratch-path. You can add additional fields to the job, but it is strongly recommended that you do not delete fields from the template. Doing so may cause your jobs to fail to launch properly, or cause unintended subtle issues with Job Launcher functionality.

Changes to the template require either a restart of the Launcher or a SIGHUP signal. The SIGHUP signal can be sent to the process to cause the templates to be reloaded during run-time. Note that if the new changes are not valid, the original templates will continue to be used until valid changes are reloaded.

sudo kill -s SIGHUP $(pidof rstudio-kubernetes-launcher)

The object templates are modeled after Helm charts and have identical syntax. Like Helm charts, conditional logic, loops, and various functions are supported via the Sprig library. In contrast to Helm, there are no Values files and no directory structure for templates - all templates are simply stored in the scratch-path. The only values available for use in the template are available on the .Job object which represents the job being submitted via the Launcher.

All templates must start with a comment indicating the version of the template in use, which must match the version required by the Launcher. This is to ensure that your templates are up-to-date with what is needed by the Launcher. Currently, this version must be set to 1 for both the job and service templates (see Examples for details). The expected version comment is generated when invoking the --generate-templates command.

The following fields are available on the .Job object:

Field	Type	Description
id	string	The unique ID for the job. Only set for service objects.
cluster	string	The name of the Launcher cluster.
generateName	string	Used for generating unique object names within Kubernetes to ensure object names do not collide.
name	string	The name of the Job.
user	string	The submitting user of the Job.
workingDirectory	string	The working directory for the job command being executed.
container	map	The container that will run the job command.
container.image	string	The Docker image of the container.
container.runAsUser	int	The UID of the container. May be nil if the default of 0 (root) is used.
container.runAsGroup	int	The GID of the container. May be nil if the default of 0 (root) is used.
container.supplementalGroupIds	int array	An array of supplemental UIDs for the container user. May be empty.
host	string	The desired host to run the Job on. Usually empty.
command	string	The Job command to execute. If empty, exe will be specified.
exe	string	The executable to execute. If empty, command will be specified.
stdin	string	The Job’s stdin.
args	string array	The arguments for the command/executable to be run.
placementConstraints	map array	The placement constraints to be used for deciding where the job should be run.
placementConstraint.name	string	The name of the Placement Constraint. Example: availability-zone, region, etc.
placementConstraint.value	string	The value of the Placement Constraint. Example: us-east-2, m3xlarge, etc.
exposedPorts	map array	The ports that should be exposed/opened for the job.
exposedPort.protocol	string	The protocol for the port (TCP or UDP).
exposedPort.targetPort	string	The port within the container to expose/open.
mounts	map array	The requested mounts for the job. This is generally complicated to work with to transform into Kubernetes objects, so volumes and volumeMounts are provided.
mount.mountPath	string	The path where the mount should be mounted to.
mount.readOnly	bool	Whether or not the mount should be read-only.
mount.mountSource	map	The description of the mount itself.
mount.mountSource.type	string	The type of the mount (e.g. host, nfs, etc.).
mount.mountSource.source	map	The underyling description of the type of the mount. Varies by mount type.
config	map array	Job-specific config unique to the Kubernetes Launcher. Currently used to specify secret env vars.
config.name	string	The name of the Job config.
config.value	string	The value of the Job config.
resourceLimits	map array	The resource limits to be used for the Job.
resourceLimit.type	string	The type of the resource limit (memory, cpuCount, NVIDIA GPUs, or AMD GPUs).
resourceLimit.value	string	The value of the resource limit.
volumes	map array	The Kubernetes volumes that should be mounted. This is constructed from the requested Job mounts.
volume.name	string	The unique name of the volume.
volume.?	map	The sub-object describing the volume. This varies based on the type of the volume being mounted.
volumeMounts	map array	The Kubernetes volume mounts describing how volumes should be mounted. This is constructed from the requested Job mounts.
volumeMount.name	string	The name of the volume to be mounted. Must match a volume.name.
volumeMount.mountPath	string	The path where the mount should be mounted to.
volumeMount.readOnly	bool	Whether or not the mount should be read-only.
tags	string array	The tags for the job.
servicePortsJson	string	Used by the Launcher to ensure that services are created properly after a restart.
shareProcessNamespace	bool	Reflects the current sharedProcessNamespace configuration setting to control if a container should use shared process namespacing.
memoryRequestRatio	decimal	Reflects the memoryRequestRatio as specified in the User/Group profiles.
cpuRequestRatio	decimal	Reflects the cpuRequestRatio as specified in the User/Group profiles.

The following template functions are available for use in addition to the previously mentioned Sprig template functions:

Name	Description	Example
include	Renders the specified template file (other than job.tpl and service.tpl) with the specified values and returns the result.	{{ include “custom.tpl” . }}
toYaml	Renders the specified object as YAML.	{{ toYaml .Job.volumes }}
exec	Executes the specified command or executable with the given arguments, returning an ExecResult type, which if rendered directly returns the stdout for the process. See Examples below.	{{ exec “echo” “Hello, world” }}
groups	Returns a list of groups for the specified user by shelling out to the groups Linux command.	{{ groups .Job.user }}

To see what has been modified in the template files, you can use the --diff-templates command. This will show the output of the diff Linux command comparing the changes that you’ve made to the original templates generated with the --generate-templates command. Note that the diff command must be present on the PATH in order to use this functionality.

sudo /path/to/rstudio-kubernetes-launcher --diff-templates

3.1.4.3 Examples

** Adding Host Aliases to the Job **

Host Aliases are defined on spec.template.spec.

job.tpl

# Version: 1
apiVersion: batch/v1
kind: Job
(...omitted for brevity...)
spec:
  backoffLimit: 0
  template:
    (...omitted for brevity...)
    spec:
      hostAliases:
        - ip: "10.2.141.12",
          hostnames: ["db01"]
        - ip: "10.2.141.13",
          hostnames: ["db02"]

** Adding Linux Capabilities to the Job Container **

Capabilities are defined on spec.template.spec.securityContext. The securityContext is conditionally constructed and is not present for all jobs, so you need to modify the template so that it is always constructed with your desired capabilities.

job.tpl

# Version: 1
apiVersion: batch/v1
kind: Job
(...omitted for brevity...)
spec:
  backoffLimit: 0
  template:
    (...omitted for brevity...)
    spec:
      (...omitted for brevity...)
      securityContext:
        {{- if $securityContext }}
        {{- range $key, $val := $securityContext }}
        {{ $key }}: {{ $val }}
        {{- end }}
        capabilities:
          add: ["NET_ADMIN", "SYS_PTRACE"]

** Adding an ImagePullSecret **

imagePullSecrets are defined on spec.template.spec.

job.tpl

# Version: 1
apiVersion: batch/v1
kind: Job
(...omitted for brevity...)
spec:
  backoffLimit: 0
  template:
    (...omitted for brevity...)
    spec:
      imagePullSecrets:
        - name: mysecret

** Custom Annotation with Dynamic Execution **

It can be useful to annotate jobs with special annotations depending on certain business logic. This business logic can be encapsulated in a command or executable that you run that decides the value of the annotation. The following example shows what it might look like to fetch the organizational cost center for a user given their user name and a custom application maintained by the business.

job.tpl

# Version: 1
apiVersion: batch/v1
kind: Job
metadata:
  generateName: {{ toYaml .Job.generateName }}
spec:
  backoffLimit: 0
  template:
    metadata:
      annotations:
        my.org/costCenter: {{ exec "get-user-details" "--cost-center" ".Job.user" }}

The hypothetical get-user-details command would then write the user’s cost center to stdout when invoked, which would cause it to be stamped as an annotation on the job as my.org/costCenter which could be used with various Kubernetes reporting tools.

Using the exec function, it is also possible to obtain more information about the result of the invoked process from the ExecResult return value:

job.tpl

# Version: 1
apiVersion: batch/v1
kind: Job
metadata:
  generateName: {{ toYaml .Job.generateName }}
spec:
  backoffLimit: 0
  template:
    metadata:
      annotations:
        {{- $res := exec "get-user-details" "--cost-center" ".Job.user" }}
        {{- if eq $res.ExitCode 1 }}
        my.org/costCenter: "UNKNOWN: {{ $res.Err }}"
        {{- else }}
        my.org/costCenter: {{ $res.Stdout }}
        {{- end }} 

The following fields are available on an ExecResult:

Field	Type	Description
ExitCode	int	The exit code of the process. If the process encountered an error before running or exited due to signal, this is set to -1.
Err	error	The error that occurred while running the process, if any. May be nil.
Stdout	string	The stdout of the process.
Stderr	string	The stderr of the process.

** Add Annotation For Group Members **

In this example, we use the groups template function to determine if the job user belongs to the admin-grp group, adding an annotation if so.

job.tpl

# Version: 1
apiVersion: batch/v1
kind: Job
metadata:
  generateName: {{ toYaml .Job.generateName }}
spec:
  backoffLimit: 0
  template:
    metadata:
      annotations:
        {{- $grp := groups .Job.user }}
        {{- if has "admin-grp" $grp }}
        my.org/admin: "true"
        {{- end }}

** IAM Permissions with kiam **

You can extend the above example to include IAM roles per group using kiam. The following example will set the users in the group admin-grp to an existing IAM role (rs-admin-role). All other users are given a different IAM role (rs-user-role).

# Version: 1
apiVersion: batch/v1
kind: Job
metadata:
  generateName: {{ toYaml .Job.generateName }}
spec:
  backoffLimit: 0
  template:
    metadata:
      annotations:
        {{- $grp := groups .Job.user }}
        {{- if has "admin-grp" $grp }}
        my.org/admin: "true"
        iam.amazonaws.com/role: "rs-admin-role"
        {{- else }}
        iam.amazonaws.com/role: "rs-user-role"
        {{- end }}

3.1.4.4 Validating Templates

Once changes have been made to templates, it is recommended run the --validate-templates command to ensure that the changes are valid YAML and that no important Job Launcher pieces have been tampered with inadvertently. This command generates a test Job payload and renders the templates in the scratch-path.

sudo /path/to/rstudio-kubernetes-launcher --validate-templates

If any errors are found during validation, they will be reported so that the templates can be modified to fix them. The validator tests for the following issues: * Is the template itself valid? Do all functions exist and is all syntax correct? * Is the rendered template YAML valid? * Is there any extra white space on the YAML object? * Are all pieces of the object needed by the Job Launcher available and of the correct type? * Are all required hard-coded values set correctly?

Note that errors produced during validation are strongly indicative of issues that should be addressed, but the Launcher does not require validation to complete without errors before using the templates.

It is also sometimes desirable to inspect the actual YAML that is generated after templating. This can be done by adding --verbose to the --validate-templates command detailed above.

3.2 Kubernetes Cluster Requirements

In order for the Kubernetes plugin to run correctly, the following assumptions about the Kubernetes cluster must be true:

• The Kubernetes API must be enabled and reachable from the machine running the Job Launcher
• There must be a namespace to create jobs in, which can be specified via the kubernetes-namespace configuration mentioned above (this defaults to rstudio)
• There must be a service account that has full API access for all endpoints and API groups underneath the aforementioned namespace, and the account’s auth token must be supplied to the plugin via the auth-token setting
• The service account must have access to view the nodes list via the API (optional, but will restrict IP addresses returned for a job to the internal IP if not properly configured, as /nodes is needed to fetch a node’s external IP address)
• The cluster must have the metrics-server addon running and working properly to provide job resource utilization streaming

In order to use placement constraints, you must attach labels to the node that match the given configured placement constraints. For example, if you have a node with the label az=us-east and have a placement constraint defined az:us-east, incoming jobs specified with the az:us-east placement constraint will be routed to the desired node. For more information on Kubernete’s placement constraints, see here.

The following sample script can be run to create a job-launcher service account and rstudio namespace, granting the service account (and thus, the launcher) full API access to manage RStudio jobs:

kubectl create namespace rstudio
kubectl create serviceaccount job-launcher --namespace rstudio
cat > job-launcher-role.yaml <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: job-launcher-role
  namespace: rstudio
rules:
  - apiGroups:
      - ""
    resources:
      - "pods"
      - "pods/log"
      - "pods/attach"
      - "pods/exec"
    verbs:
      - "get"
      - "create"
      - "update"
      - "patch"
      - "watch"
      - "list"
      - "delete"
  - apiGroups:
      - ""
    resources:
      - "events"
    verbs:
      - "watch"
  - apiGroups:
      - ""
    resources:
      - "services"
    verbs:
      - "create"
      - "get"
      - "watch"
      - "list"
      - "delete"
  - apiGroups:
      - "batch"
    resources:
      - "jobs"
    verbs:
      - "create"
      - "update"
      - "patch"
      - "get"
      - "watch"
      - "list"
      - "delete"
  - apiGroups:
      - "metrics.k8s.io"
    resources:
      - "pods"
    verbs:
      - "get"
EOF
kubectl create -f job-launcher-role.yaml; rm job-launcher-role.yaml
kubectl create rolebinding job-launcher-role-binding --namespace rstudio \
   --role=job-launcher-role \
   --serviceaccount=rstudio:job-launcher
kubectl create clusterrole job-launcher-clusters \
   --verb=get,watch,list \
   --resource=nodes
kubectl create clusterrolebinding job-launcher-list-clusters \
  --clusterrole=job-launcher-clusters \
  --group=system:serviceaccounts:rstudio

It should be noted that the ClusterRole created above is only used to get information about the nodes in the cluster that can run Launcher jobs. This is sometimes necessary to ensure that the Launcher can determine all of the IP addresses that belong to the node to ensure that they are reported properly to upstream clients of the Launcher. This ensures that external clients can connect to their Launcher jobs as required. If you ensure that all clients are connecting to the Launcher internally within the same network segment (meaning that clients can connect to the internal IP address of the Kubernetes node), you can forego the ClusterRole and ClusterRoleBinding. If you run into problems where clients cannot connect to their Launcher jobs, you may need the external IP address of the node(s) in your cluster, which will require the ClusterRole to be given to the Launcher service account.