Step 7. Skip Certificate Verification

Kubernetes plugin configuration issues

You’re unable to start new remote sessions in Kubernetes.

Symptoms

Unable to start new remote sessions in Kubernetes

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page

Job information unavailable. Sessions are read-only.
Contact your administrator for help.

Error occurred while executing method (100)
Connection refused

void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197

/var/lib/rstudio-server/monitor/log/rstudio-server.log

04 May 2020 03:36:03 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678

/var/lib/rstudio-launcher/rstudio-launcher.log

04 May 2020 03:34:35 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 03:34:35 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 03:34:35 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 03:34:35 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 03:34:35 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240

/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 03:34:35 [rstudio-kubernetes-launcher] ERROR system error 84 (Invalid or incomplete multibyte or wide character) [reason=string length 1367 is not a multiple of 4]; OCCURRED AT: rstudio::core::Error rstudio::core::base64::{anonymous}::Decoder::operator()(const Byte*, std::size_t, std::string*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/Base64.cpp:231; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134
04 May 2020 03:34:35 [rstudio-kubernetes-launcher] ERROR Could not base64 decode certificate-authority; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::impls::kubernetes::KubernetesApi::decodeCertificateAuthority() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:75

/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 03:54:57 [rstudio-kubernetes-launcher] ERROR asio.ssl error 218529960 (wrong tag); OCCURRED AT: rstudio::core::http::TcpIpAsyncClientSsl::TcpIpAsyncClientSsl(rstudio_boost::asio::io_service&, const string&, const string&, bool, const string&, const rstudio_boost::posix_time::time_duration&, const string&) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:67; LOGGED FROM: rstudio::core::http::TcpIpAsyncClientSsl::TcpIpAsyncClientSsl(rstudio_boost::asio::io_service&, const string&, const string&, bool, const string&, const rstudio_boost::posix_time::time_duration&, const string&) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:67
04 May 2020 03:54:57 [rstudio-kubernetes-launcher] ERROR asio.ssl error 336134278 (certificate verify failed); OCCURRED AT: void rstudio::core::http::TcpIpAsyncClientSsl::handleHandshake(const rstudio_boost::system::error_code&) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:142; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

Possible cause

For Workbench, Launcher, and Kubernetes to function properly, the Launcher service needs to be able to communicate to the Kubernetes API over HTTPS.

This requires that the certificate authority (CA) for the Kubernetes API is configured correctly as shown in the following Kubernetes plugin configuration file:

File: /etc/rstudio/launcher.kubernetes.conf

certificate-authority=<BASE-64-ENCODED-CA-CERTIFICATE>

where <BASE-64-ENCODED-CA-CERTIFICATE> is the Base64 encoded CA certificate for the Kubernetes API.

The following troubleshooting steps will help you determine if your certificate authority is misconfigured by skipping SSL certificate verification.

Troubleshooting steps

For troubleshooting purposes, you can bypass SSL certificate verification that is used when Launcher communicates with the Kubernetes API.

Warning

For production use, you should always have SSL verification enabled with verify-ssl-certs=1 (which is the default when this option is not present) so that the communication between Launcher and the Kubernetes API is secure. SSL verification should only be disabled for troubleshooting purposes as described in this section.

To skip SSL certificate verification when communicating with the Kubernetes API, add the following line to your Kubernetes plugin configuration:

/etc/rstudio/launcher.kubernetes.conf

verify-ssl-certs=0

Restart services and test

Restart the Workbench and Launcher services:

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the Workbench home page.

If you are able to start a new session after skipping SSL certificate verification, then you should refer to the troubleshooting page on verifying certificate configuration to confirm that you are using the correct certificate authority in the correct format.

Note

After you fix any issues with your certificate authority configuration, you should re-enable certificate authority verification by removing the above option for verify-ssl-certs=0 that was used for troubleshooting purposes.

If the services are still not starting or you are still experiencing errors when starting a new session, then proceed to
Step 8 - Verify Certificate Configuration.