This site is archived

Posit Connect support for off-host content execution has entered General Availability (GA).
Documentation about off-host execution is now included in the Connect Admin Guide.
The instructions presented here were associated with the off-host execution beta and are now obsolete.

Appendix: Custom Content Images

Overview

Posit Connect has experimental support for container-based execution within Kubernetes. This appendix explains how images are used by Posit Connect and how to configure and customize images to run your content.

Capabilities

This section discusses how Posit Connect uses your configured images to run content in Kubernetes.

When first deployed, Posit Connect uses the selected image to create a build container (environment restore) to install R and Python packages required by your content. Subsequently, each time your content needs to run, that same image is used to create a container for content execution.

Using an approach very similar to process sandboxing, Posit Connect instructs Kubernetes to mount storage locations into the containers that run your content. R and Python package locations, the source code for your content, and locations for rendering output are all mounted into the container. This approach lets you use smaller images and share images across multiple content items.

You will need to choose images to use with your content and then configure Posit Connect so it knows about those images and their capabilities. Posit Connect does not automatically detect images nor does it probe those images for their capabilities.

Image selection

When your content is deployed, Posit Connect may automatically determine which image is most appropriate for your content. Given the R and Python version demands of your content, Posit Connect uses the configured R and Python matching algorithms to discard images that do not have compatible R and Python interpreters. From the remaining images, the “best” candidate is chosen by performing a pairwise comparison of the R and Python version components.

The Server.RVersionMatching setting controls which R installations are considered compatible with your content. We recommend a setting of major-minor rather than the default nearest configuration. The Posit Connect Admin Guide contains more information about R version matching.

The Python.VersionMatching setting controls which Python installations are considered compatible with your content. The default of major-minor is appropriate for most installations. The Posit Connect Admin Guide contains additional details about Python version matching.

Once an image and appropriate R and Python interpreters are selected, Posit Connect uses that environment to install R packages and Python packages before using that same image to run your content.

Alternatively, you may select an image for your content build/execution by setting the environment.image field in the content’s manifest.json. The Python client package, rsconnect-python (1.8.0+), supports a --image parameter when deploying content. Similarly, the rsconnect (0.8.26+) R package supports an image argument so that users can specify an image when deploying content to Posit Connect.

Note

Note that the image selected must also be listed in the configuration defined by the Launcher.ClusterDefinition setting, and the R and Python version demands of your content must be satisfied using the same version matching criteria listed above.

Excluding images from selection

In some cases, it is useful to exclude an image from automatic selection. For example, a highly customized content image that is only suitable for running a very specific content item might not be generally suitable for selection to run most content. In this instance, users can specify the matching field in the runtimes.yaml configuration to control if/how an image is chosen for content execution.

name: Kubernetes
images:
  - name: ghcr.io/rstudio/content-base:r3.6.3-py3.8.8-bionic
    matching: exact
    python:
      installations:
        - path: /opt/python/3.8.8/bin/python3.8
          version: 3.8.8
    r:
      installations:
        - path: /opt/R/3.6.3/bin/R
          version: 3.6.3

There are 3 possible values accepted by the matching field: none, any, and exact. A value of none indicates that the image is effectively disabled; it will never be considered for selection. A value of exact indicates that the image can only be used when it is explicitly targeted by the bundle’s manifest.json; it will not be selected automatically by Connect, even if it is compatible with the content. A value of any (the default) means that the image can be selected automatically by Connect or it can be explicitly targeted by the bundle’s manifest.json.

Workflow

  1. Choose and create images
  2. Describe image capabilities
  3. Enable off-host execution with Kubernetes and configure Posit Connect to use your chosen images

Once you have chosen your target images and have YAML files describing those images, you are ready to configure Posit Connect to use those images with Kubernetes.

The following configuration snippet turns on both the Launcher.Enabled and Launcher.Kubernetes settings and specifies a YAML with image definitions. Both Launcher.Enabled and Launcher.Kubernetes are required in order to enable Kubernetes support. Without these settings, the Posit Connect server uses local processes.

; /etc/rstudio-connect/rstudio-connect.gcfg
[Launcher]
Enabled = true
Kubernetes = true
ClusterDefinition = "/etc/rstudio-connect/runtimes.yaml"

The Launcher.ClusterDefinition setting names the YAML files describing the available images.

Restart Posit Connect after applying these configuration changes.

Images

Posit Connect asks Kubernetes to create a container from an image with appropriate R and Python installations. Your content is run within that container. You will need to configure at least one image in order for content to run within Kubernetes.

You can start by using the basic images created by Posit, or by using images already available within your organization.

Multiple content items will use the same image. Posit Connect manages the installation of R and Python packages needed by your content.

You will probably want a selection of images that reflect the versions of R and Python used by your organization. We recommend considering images which:

  • Cover each MAJOR.MINOR version of R. For recently created content, this probably means having images with R-3.6, R-4.0, and R-4.1.

  • Cover each MAJOR.MINOR version of Python. For recently created content, this probably means images with Python-3.8, and Python-3.9.

Basic images

Posit has created a set of basic images that are available to use in your environment. Each image uses the Ubuntu 18.04 LTS (Bionic Beaver) release and includes a single version of R, a single version of Python, and system libraries commonly used by R and Python packages.

  • The R installation is available at /opt/R/<VERSION>.
  • The Python installation is available at /opt/python/<VERSION> and has virtualenv installed.

These are the currently available images. New combinations may be added as new R and Python releases occur. If these images do not suit your organization, you can create and use custom images.

Image Name Python R
rstudio/content-base:r3.1.3-py2.7.18-bionic 2.7.18 3.1.3
rstudio/content-base:r3.2.5-py2.7.18-bionic 2.7.18 3.2.5
rstudio/content-base:r3.3.3-py3.6.13-bionic 3.6.13 3.3.3
rstudio/content-base:r3.4.4-py3.6.13-bionic 3.6.13 3.4.4
rstudio/content-base:r3.5.3-py3.7.10-bionic 3.7.10 3.5.3
rstudio/content-base:r3.6.3-py3.8.8-bionic 3.8.8 3.6.3
rstudio/content-base:r4.0.5-py3.8.8-bionic 3.8.8 4.0.5
rstudio/content-base:r4.0.5-py3.9.2-bionic 3.9.2 4.0.5
rstudio/content-base:r4.1.0-py3.8.8-bionic 3.8.8 4.1.0
rstudio/content-base:r4.1.0-py3.9.2-bionic 3.9.2 4.1.0

These images are available from GitHub Container Registry: https://github.com/rstudio/rstudio-docker-products/pkgs/container/content-base

The source that builds these images is available from GitHub: https://github.com/rstudio/rstudio-docker-products/tree/main/content

Once you have selected your target images, you’ll want to describe them so Posit Connect understands the capabilities available within each image.

Custom images

You are welcome to create your own images by extending the rstudio/content-base images or by creating new images from scratch.

Hint

The Server API Cookbook contains the recipe Content Runtimes - R and Python versions in use which will assist you in understanding the needs of your existing content.

Some general guidance when creating your own images:

  1. Prefer smaller images where possible. Large images will lead to increased launch times, as images may need to be downloaded from their hosting repository.

  2. An image should contain at least one Python or R installation. Images without R or Python will not be compatible with the runtime requirements of your content and will not be used.

  3. Images should include the system dependencies of the R and Python packages you use.

  4. Avoid “kitchen sink” images that contain many different Python and R interpreters along with the system requirements needed across all those versions. This approach will produce a large image.

    You may want to have Posit Connect use different, smaller images than you use during development.

  5. You do not need to include both R and Python in every image; choose combinations that mirror what you use during development.

  6. You do not need to produce an image for every patch release of R and Python; the most recent patch for each MAJOR.MINOR release is usually sufficient.

Once you have built your images, you’ll want to describe them so Posit Connect understands the capabilities available within each image.

For further information, see Creating Custom Images for Content Execution.

Runtime Components

You may notice that these images do not contain any runtime components of Posit Connect. When the job is launched, the container is augmented with a handful of required runtime components using a separate, auxiliary Posit Connect init-container. The use of an init-container to prepare the runtime components means that the content image does not need to be rebuilt for a particular version of Posit Connect. For more details about the init-container, see the Runtime Init Container section.

Describing images

Posit Connect needs to understand the capabilities of each image. The Launcher.ClusterDefinition configuration setting provides the path to YAML files that describe the permitted images.

You can create the YAML files yourself or generate them.

Here is a YAML file template that can be modified to describe an image containing one version of Python and one version of R.

# /etc/rstudio-connect/example-runtime.yaml
name: "Kubernetes"
images:
  -
    # This is the location (name) of your image.
    name: "organization/content-image:image-version"
    # An (optional) Python installation. Replace "X.Y.Z" with the
    # available Python version.
    python:
      installations:
        -
          path: "/opt/python/X.Y.Z/bin/python"
          version: "X.Y.Z"
    # An (optional) R installation. Replace "A.B.C" with the
    # available R version.
    r:
      installations:
        -
          path: "/opt/R/A.B.C/bin/R"
          version: "A.B.C"

Here is a sample Posit Connect configuration that uses this YAML definition.

; /etc/rstudio-connect/rstudio-connect.gcfg
[Launcher]
ClusterDefinition = "/etc/rstudio-connect/example-runtime.yaml"

Generating YAML

The build-image-yaml.sh and examine-image.sh scripts can help generate YAML for your images. These scripts are available from the content/scripts subdirectory of the rstudio/rstudio-docker-products repository.

These scripts:

  • Use docker run to inspect a named image. Images not already in the local Docker cache will be downloaded. Images present in the local cache are not re-pulled.

  • Look for R installations at /opt/R/<VERSION> and /opt/local/R/<VERSION>. If none are found, they look for R using the PATH.

  • Look for Python installations at /opt/python/<VERSION> and /opt/local/python/<VERSION>. If none are found, they look for python, python2, and python3 using the PATH.

The following command will produce YAML describing three images.

./scripts/build-image-yaml.sh \
    rstudio/content-base:r3.6.3-py3.8.8-bionic \
    rstudio/content-base:r4.0.5-py3.8.8-bionic \
    rstudio/content-base:r4.1.0-py3.9.2-bionic \
    > runtimes.yaml

Review the generated YAML to confirm that the expected R and Python installations are included. Remove any detected installations that you do not want in use.

Per-Image Supervisors

Under the traditional local execution model, it is possible to configure Posit Connect to invoke a program supervisor before building or executing content. The supervisor is a script or program that must be available and executable on the Posit Connect server.

Under the remote execution (Kubernetes) model, it may make more sense to construct image definitions to configure the environment correctly, rather than use a supervisor to modify the environment at runtime. If desired, you may still use a supervisor, but it is necessary to configure it per-image rather than globally (as the same script or program might not be installed on every image.)

Please see the Program Supervisors section of the Posit Connect Admin Guide to read more about the uses and requirements of supervisors.

Note

It is still possible to configure a global supervisor in the Posit Connect configuration file when using remote execution; it will only be used when invoking processes local to the Posit Connect server (like git processes). A warning will be printed at startup to remind you that the supervisor will not be invoked when building or executing content.

Note

Unlike the global supervisor, per-image supervisors cannot be validated at startup. Please ensure that any supervisor you configure is available and executable in the relevant image and that it conforms to the rules described in the Admin Guide.

The configured supervisor script or program must exist and be executable within the image you are describing. One way to accomplish this would be to add the supervisor script to the image during the Docker build and bake it into the image.

See YAML file with supervisor for an example of describing an image with a program supervisor.

Image examples

This section includes some working examples of describing image capabilities in YAML and the Posit Connect configuration needed to use that YAML.

Single YAML file

Here is an example configuration that uses a single YAML file to describe all of the permitted images.

The Posit Connect configuration declares that one YAML file describes the available images.

; /etc/rstudio-connect/rstudio-connect.gcfg
[Launcher]
ClusterDefinition = "/etc/rstudio-connect/runtimes.yaml"

This YAML file describes two images: one having R-3.6 and Python-3.8.8 and another with R-4.0 and Python-3.9.2.

# /etc/rstudio-connect/runtimes.yaml
name: "Kubernetes"
images:
  -
    name: "rstudio/content-base:r3.6.3-py3.8.8-bionic"
    r:
      installations:
        -
          path: "/opt/R/3.6.3/bin/R"
          version: "3.6.3"
    python:
      installations:
        -
          path: "/opt/python/3.8.8/bin/python3.8"
          version: "3.8.8"
  -
    name: "rstudio/content-base:r4.0.5-py3.9.2-bionic"
    python:
      installations:
        -
          path: "/opt/python/3.9.2/bin/python3.9"
          version: "3.9.2"
    r:
      installations:
        -
          path: "/opt/R/4.0.5/bin/R"
          version: "4.0.5"

Multiple YAML files

This example configuration has multiple YAML files, each describing a single, permitted image.

The Posit Connect configuration declares that two separate YAML files describe the available images.

; /etc/rstudio-connect/rstudio-connect.gcfg
[Launcher]
ClusterDefinition = "/etc/rstudio-connect/content-base-r3.6-py3.8.yaml"
ClusterDefinition = "/etc/rstudio-connect/content-base-r4.0-py3.9.yaml"

This YAML file defines a single image that contains R-3.6 and Python-3.8.8.

# /etc/rstudio-connect/content-base-r3.6-py3.8.yaml
name: "Kubernetes"
images:
  -
    name: "rstudio/content-base:r3.6.3-py3.8.8-bionic"
    r:
      installations:
        -
          path: "/opt/R/3.6.3/bin/R"
          version: "3.6.3"
    python:
      installations:
        -
          path: "/opt/python/3.8.8/bin/python3.8"
          version: "3.8.8"

This YAML file defines a single image that contains R-4.0 and Python-3.9.2.

# /etc/rstudio-connect/content-base-r4.0-py3.9.yaml
name: "Kubernetes"
images:
  -
    name: "rstudio/content-base:r4.0.5-py3.9.2-bionic"
    python:
      installations:
        -
          path: "/opt/python/3.9.2/bin/python3.9"
          version: "3.9.2"
    r:
      installations:
        -
          path: "/opt/R/4.0.5/bin/R"
          version: "4.0.5"

YAML file with supervisor

Here is an example YAML file that describes an image that contains R-3.6 and Python-3.8.8. Additionally, when content is run using this image, it will be invoked with the specified supervisor. Note that the supervisor must exist and be executable in the named image.

# /etc/rstudio-connect/content-base-r3.6-py3.8.yaml
name: "Kubernetes"
images:
  -
    name: "rstudio/content-base:r3.6.3-py3.8.8-bionic"
    r:
      installations:
        -
          path: "/opt/R/3.6.3/bin/R"
          version: "3.6.3"
    python:
      installations:
        -
          path: "/opt/python/3.8.8/bin/python3.8"
          version: "3.8.8"
    # The full supervisor command to invoke. Instead of a program like `nice`,
    # this could also be a custom script, e.g. "/usr/local/bin/supervisor.sh",
    # providing that script existed in the image.
    supervisor: "/usr/bin/nice -n 2"

YAML Definition

This section describes all the fields allowed in the Launcher.ClusterDefinition YAML file. We recommend that you manage these YAML files alongside your rstudio-connect.gcfg Posit Connect configuration file.

name

A string that provides the name of the target cluster. This is a required setting. Always “Kubernetes”, today.

name: "Kubernetes"

images

An array of image definitions, describing the images permitted in this cluster. This is a required setting.

images:
  -
    name: "rstudio/content-base:r3.6.3-py3.8.8-bionic"
    # ...
  -
    name: "rstudio/content-base:r4.0.5-py3.9.2-bionic"
    # ...

Images must have at least Python or R capabilities. Images may support both Python and R. Images without either Python or R capabilities cannot be used to execute content.

images.name

A string that provides the location of the target image. Each image is required to have a name.

images:
  -
    name: "rstudio/content-base:r3.6.3-py3.8.8-bionic"

images.python

Describes the (optional) Python capabilities available in this image.

images.python.installations

A list of Python installations available within this image. Each installation must specify a path to the Python interpreter and the version of that interpreter.

images:
  -
    python:
      installations:
        -
          path: "/opt/python/3.9.2/bin/python3.9"
          version: "3.9.2"

images.r

Describes the (optional) R capabilities available in this image.

images.r.installations

A list of R installations available within this image. Each installation must specify a path to the R interpreter and the version of that interpreter.

images:
  -
    r:
      installations:
        -
          path: "/opt/R/4.0.5/bin/R"
          version: "4.0.5"

images.supervisor

Specifies the (optional) program supervisor command to invoke when executing content using this image.

images:
  -
    supervisor: "/usr/bin/nice -n 2"