TensorFlow Serving

TensorFlow Serving is a serving system for machine learning models.

Saved models can be published to Posit Connect. Connect hosts your model using TensorFlow Serving.

Installing TensorFlow Serving

TensorFlow Serving must be installed on your system before Connect can host TensorFlow content. Follow the TensorFlow Serving installation instructions.

Note

View the discovered TensorFlow Serving versions in the Connect UI at System > Info. The TensorFlow Versions row lists each version that Connect discovers. The row does not report the runtime state or the executable path.

If Connect discovers no TensorFlow Serving installation, the row shows (none). The row shows the same value when TensorFlow is disabled and when TensorFlow is enabled with no configured installation. To find which case applies, examine the TensorFlow.Enabled and TensorFlow.Executable settings in your configuration file.

Restart Connect after you install TensorFlow Serving so Connect can discover the installation.

Enabling TensorFlow support

If you installed the TensorFlow Serving binary using APT, that binary is available at /usr/bin/tensorflow_model_server. Configure Connect to use this default location:

/etc/rstudio-connect/rstudio-connect.gcfg

[TensorFlow]
Enabled = true
Executable = "/usr/bin/tensorflow_model_server"

TensorFlow support is only available when the TensorFlow.Enabled is enabled and the TensorFlow.Executable setting references a valid TensorFlow Serving installation. Posit Connect warns at startup if TensorFlow is enabled without a configured installation. Connect fails to start if TensorFlow is enabled but references an invalid installation.

The TensorFlow Serving binary is used to host saved models deployed to Connect.

Your Connect server can only use one TensorFlow Serving binary at this time. Should you provide multiple TensorFlow.Executable entries, only the binary with the highest version is used.