Python Package Management

Package Installation

Posit Connect installs the Python package dependencies of Python-based content when that content is deployed. This includes FastAPI and Flask APIs, Jupyter notebooks, apps built using Dash, Bokeh, or Streamlit, and Quarto or R projects that use Python.

Package dependencies are captured in one of two ways:

  1. If a requirements.txt file exists in the directory containing the project being deployed, then the contents of that file specify the dependencies. See the pip documentation for details. If you provide a requirements.txt file, you must ensure that the listed dependencies are correct for the content you are deploying.

  2. Otherwise, the pip freeze command is used to produce a full specification of the current Python environment including all installed packages and their version numbers. In a Jupyter notebook, pip freeze will be run with the version of Python being used by the active Jupyter notebook kernel. In the RStudio IDE, the environment variable RETICULATE_PYTHON will be used to determine which Python environment to inspect.

The resulting package list is included in the bundle archive file, which is uploaded to Posit Connect. Posit Connect receives the bundle archive file, unpacks it, and uses venv and pip to install the identified package dependencies. In addition to the specified dependencies, Posit Connect also installs packages that it uses to deploy and render your content.

The execution environment created by Posit Connect contains the same package versions you are using in your development environment. The use of venv isolates environments from one another to avoid package version conflicts.

Administrators and content owners can view a list of the package versions installed for each content item in the Content Settings Runtime pane or by requesting the list from the Connect Server API. This list is a comprehensive description of the environment to include packages requested by the content, packages required by the Connect runtime environment, any external packages, and all package dependencies.

Requirements Files

A requirements.txt file lists the packages that are required by a piece of content and (optionally) their versions. The pip package manager allows additional options in requirements files, giving authors more flexibility and control over package installation. See the pip documentation for details.

Environment Caching

Posit Connect maintains a cache of Python environments to enable faster deployments. New environments are created as needed, based on the list of package dependencies received in the bundle and the python version in use.

Subsequent deployments that have the same list of dependencies will reuse the previously-built environment. If any dependencies are different, a new environment will be created. This enables published content to make use of different versions of dependent packages without conflict.

Providing a requirements.txt file which is the same across multiple projects is one way to facilitate environment reuse and enable faster deployments. A similar benefit is achieved in the automatic (pip freeze) case if the Python environment on the publishing computer remains the same between deployments.

While environment caching is the default behavior in Posit Connect, there are alternative approaches that can be used to share packages across content items. These alternatives are useful in situations where the administrator needs more control over the packages used by the content at runtime, for example when packages are centrally managed (e.g., internal company packages).

The first option is to disable environment management entirely. By disabling environment management, the administrator declares that Connect should not perform any package installation. Only packages that are installed system-wide are available to the content at runtime. Alternatively, administrators can override specific packages by declaring the package as External in the Posit Connect configuration. This ensures that the system-wide version of that specific package is used when the content runs. Any packages not defined as external are still installed.

Posit Connect will periodically delete Python virtual environments that are no longer in use by any deployed content. The setting Application.PythonEnvironmentReapFrequency can be used to control how often this occurs.

Warning

In rare situations, systems maintenance requires you to clear the environment caches. See the Runtime Caches section for details and instructions.

Environment management

By default, Posit Connect is responsible for managing the Python environment that is required by the bundle at runtime. Connect manages the environment during content deployment by using venv to create a virtual environment and installing the required packages with pip. The Python environment is then made available at runtime during content execution. The Python environment management behavior can be controlled by either an administrator or the content owner. When environment management is disabled, it is the responsibility of the administrator to ensure that the packages required by the content bundle are available in the runtime environment.

With local execution, the Python packages must be installed in the system Python library for the version of Python that is requested by the content. When off-host execution is enabled, the Python packages must be installed in the Python library in the image that is used to execute the content. The selected Python installation (and image name) can be found on the dashboard Info tab in the Application Settings pane.

There are 3 ways to configure the Python environment management strategy. From lowest to highest precedence:

  1. Server administrators can configure a global environment management strategy at the server-level with the Python.EnvironmentManagement setting (default true).

  2. Administrators and content owners can configure an application-level Python environment management strategy. If set, this option is used, and the global server setting is ignored. Users can set an application-level default with the Server API or by selecting the Python environment management strategy in the Runtime tab on the Content Settings panel in the Connect dashboard.

  3. Publishers can configure a bundle-level environment management strategy. If set, this option is always used, and the application-level default and the global server-level default settings are ignored. To set a bundle-level environment management strategy, users can set the environment.environment_management.python field in the bundle manifest.json. The rsconnect-python (1.20.0+) client package supports a --disable-env-management-py flag so that users can specify a Python environment management strategy when deploying content.

External package installation

Warning

Adding external packages decreases the reproducibility and isolation of content on Posit Connect, and should only be done as a last resort.

Configuring external packages is an advanced configuration option. If possible, users should prefer using the Python environment management settings for their content.

You can indicate that a system-wide installation of a package should be used instead of one fetched by pip. Use the Python.External setting to enumerate system-provided Python packages that should not be managed by Connect.

For example, to make numpy and scipy external packages:

Install these packages in every Python installation that Posit Connect will be using, e.g.:

/opt/python/3.8.1/bin/python -m pip install numpy scipy

Then configure Posit Connect to treat those packages as external:

; /etc/rstudio-connect/rstudio-connect.gcfg
[Python]
External = numpy
External = scipy

Posit Connect will report an error during startup if some of the external Python packages are missing from any configured Python installation. If you are not able to install an external Python package into all Python installations (perhaps because that package is not compatible with some versions of Python), use the Python.ExternalsCheckIsFatal setting to prevent this check.

; /etc/rstudio-connect/rstudio-connect.gcfg
[Python]
ExternalsCheckIsFatal = false

External Package Version Matching

By default, Posit Connect attempts to match external packages by name. In the example above, if an uploaded bundle requests numpy==1.15 and the Python installation in Posit Connect has numpy 1.18.1 installed, the external version will be used even though the version number does not match. This is similar to how external R packages are handled. Effectively, the version number specified in the incoming requirements.txt file is ignored for external packages.

To require strict version matching, honoring exactly what is specified in the bundle’s requirements.txt file, you can set Python.ExternalVersionMatching to true. In this case, the version requested in the bundle’s requirements.txt file will be installed if needed, and the external version will only be used if it’s the correct version.

Excluding Python Packages

To exclude a certain Python package from being utilized, set Python.ProhibitedPackage to the name of the package. Specify this property once for each Python package that is to be excluded.

This may be used, for example, when deploying Python-enabled content that utilizes OS-specific packages which are unavailable on the OS that Posit Connect runs on.

Posit Connect excludes certain Python packages by default. For a list of these packages please see the Python configuration appendix.

Configuring pip

Since Posit Connect uses pip to install Python packages, you can set package installation options by creating or modifying the pip.conf file.

The global pip configuration file is /etc/pip.conf. Alternatively, since all Python environment restore processes are run under the user account specified in the Applications.RunAs configuration, you can configure options in the default RunAs user’s pip.conf file. For example, if the default RunAs user is rstudio-connect, the configuration file might be at /home/rstudio-connect/.config/pip/pip.conf.

For more information about configuring pip, refer to the pip user guide.

Specifying a Package Repository

If you have a Python package repository for your own Python packages, or have a PyPI mirror inside your firewall, you can configure Posit Connect to use that package repository when installing packages.

For example, to configure a private package repository with a timeout of 60 seconds, add the following to pip.conf:

[global]
timeout = 60
index-url = https://my-python-package-repo.internal.com

Note that setting index-url replaces pip’s default repository (PyPI). To add a new repository, use the extra-index-url setting.