Step 9. Verify Kubernetes Certificate Configuration

Kubernetes plugin configuration issues

You’re unable to start new remote sessions in Kubernetes.

Symptoms

Unable to start new remote sessions in Kubernetes

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page

Job information unavailable. Sessions are read-only.
Contact your administrator for help.

Error occurred while executing method (100)
Connection refused

void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197

/var/lib/rstudio-server/monitor/log/rstudio-server.log

04 May 2020 04:22:35 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678

/var/lib/rstudio-launcher/rstudio-launcher.log

04 May 2020 04:22:17 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 04:22:17 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 04:22:17 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 04:22:17 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 04:22:17 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240

/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 04:22:17 [rstudio-kubernetes-launcher] ERROR asio.netdb error 1 (Host not found (authoritative)); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleResolve(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:150; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 04:25:06 [rstudio-kubernetes-launcher] ERROR system error 22 (Invalid argument) [description=Invalid Kubernetes URL]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::impls::kubernetes::KubernetesApi::parseUrl() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:55; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 04:28:47 [rstudio-kubernetes-launcher] ERROR system error 110 (Connection timed out); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

Possible cause

For Workbench, Launcher, and Kubernetes to function properly, Launcher needs to be able to communicate to the Kubernetes API.

The following troubleshooting steps will help you verify that you have correctly configured the Kubernetes API URL in the Kubernetes plugin configuration file.

Troubleshooting steps

Verify that you are using a valid Kubernetes API URL in the Kubernetes plugin configuration:

/etc/rstudio/launcher.kubernetes.conf

api-url=<KUBERNETES-API-ENDPOINT>

where <KUBERNETES-API-ENDPOINT> is the URL for the Kubernetes API such as https://kubernetes.yourcompany.com.

Ensure that the protocol is included at the beginning of the Kubernetes API URL, such as https://.

You should also include the port at the end of the Kubernetes API URL if you are using a different port, such as https://kubernetes.yourcompany.com:6443.

Verify that your Kubernetes API URL appears similar to the above example, that it was copied to the configuration file in its entirety, and that it is associated with the correct Kubernetes cluster.

Restart services and test

After you’ve verified that you have configured the correct Kubernetes api-url in the Kubernetes plugin configuration file, restart the Workbench and Launcher services:

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the Workbench home page.

If the services are still not starting or you are still experiencing errors when starting a new session, then proceed to
Step 10 - Verify Kubernetes Token Configuration.