Step 8. Verify Certificate Configuration

Kubernetes plugin configuration issues

You’re unable to start new remote sessions in Kubernetes.

Symptoms

Unable to start new remote sessions in Kubernetes

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page

Job information unavailable. Sessions are read-only.
Contact your administrator for help.

Error occurred while executing method (100)
Connection refused

void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197

/var/lib/rstudio-server/monitor/log/rstudio-server.log

04 May 2020 03:36:03 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678

/var/lib/rstudio-launcher/rstudio-launcher.log

04 May 2020 03:34:35 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 03:34:35 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 03:34:35 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 03:34:35 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 03:34:35 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240

File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 03:34:35 [rstudio-kubernetes-launcher] ERROR system error 84 (Invalid or incomplete multibyte or wide character) [reason=string length 1367 is not a multiple of 4]; OCCURRED AT: rstudio::core::Error rstudio::core::base64::{anonymous}::Decoder::operator()(const Byte*, std::size_t, std::string*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/Base64.cpp:231; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134
04 May 2020 03:34:35 [rstudio-kubernetes-launcher] ERROR Could not base64 decode certificate-authority; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::impls::kubernetes::KubernetesApi::decodeCertificateAuthority() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:75

/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 03:54:57 [rstudio-kubernetes-launcher] ERROR asio.ssl error 218529960 (wrong tag); OCCURRED AT: rstudio::core::http::TcpIpAsyncClientSsl::TcpIpAsyncClientSsl(rstudio_boost::asio::io_service&, const string&, const string&, bool, const string&, const rstudio_boost::posix_time::time_duration&, const string&) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:67; LOGGED FROM: rstudio::core::http::TcpIpAsyncClientSsl::TcpIpAsyncClientSsl(rstudio_boost::asio::io_service&, const string&, const string&, bool, const string&, const rstudio_boost::posix_time::time_duration&, const string&) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:67
04 May 2020 03:54:57 [rstudio-kubernetes-launcher] ERROR asio.ssl error 336134278 (certificate verify failed); OCCURRED AT: void rstudio::core::http::TcpIpAsyncClientSsl::handleHandshake(const rstudio_boost::system::error_code&) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncClientSsl.hpp:142; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

Possible cause

For Workbench, Launcher, and Kubernetes to function properly, the Launcher service needs to be able to communicate to the Kubernetes API over HTTPS.

This requires that the certificate authority (CA) for the Kubernetes API is configured correctly in the Kubernetes plugin configuration file.

The following troubleshooting steps will help you verify that you have correctly configured the Kubernetes certificate authority in the Kubernetes plugin configuration file.

Troubleshooting steps

Verify that you have configured the correct certificate-authority in the Kubernetes plugin configuration file:

File: /etc/rstudio/launcher.kubernetes.conf

certificate-authority=<BASE-64-ENCODED-CA-CERTIFICATE>

where <BASE-64-ENCODED-CA-CERTIFICATE> is the Base64 encoded CA certificate for the Kubernetes API.

You can usually obtain the Kubernetes CA certificate from your Kubernetes cluster console or dashboard, or from your kubectl configuration in ~/.kube/config, and it will often be provided in a Base64-encoded format already.

You can tell if your CA certificate is Base64 encoded if it appears similar to the following:

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EVXdNVEU0TURRME4xb1hEVE13TURReU9URTRNRFEwTjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS3NMClgzemZYNDVaYlhpRHllSGxhZWRZSm1UQ2lzVFYvc284aVlYaThpaHRaMnRnNkNmUnQzR1NtWDBMeDk5OC9rbFQKa0I5UVJOdHQ5VUJWZmQ2VW1DVXhFNU56UUV2YnI3MmNNaFh5a0wvcFdZdEdGTTJyWDUyTzBVZ3J2L3AwMURiZApTeGF6OWV3OXcvRjRRbHlxekZPNTJsaXZPeXpIbDRJL3hJRUd3L0IyN0I1ZVR3Z1pURVYyUDU4eTlmVE91bjBDCjlmZ0VBYUUvR1I1S3NrQWVaa2ZseWZaYk52WjQ2MG90R0tXaWlXaEllT0NNSUdOM0FnLzNOa1llOW1sZmVUb2MKRzY1THhVSTgxaTNjeXlvOVVGa1UvZ3N6TVpFYkM3VUhSSkdORG9MaS9aVnFucHFhMEJKM3JXeHl5WW93V3kyZwpPR2JGeFBHM0Y0ekloclFCeDNVQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLSjN0eWtyRHZKUkFDVGc1dE50MGFia3dOVk4KL0JLVjUxb0Z5dnVkTGp6NjBOUGVrb0FwRHUxZFRsMzhiMmI4WFVVVTZ6Q2FmYTdtc2RZNDJZbmU5blpyMnI4KwpHZTJ4c1F2enRHSmljOGZjVGdma0xZczVkblF2S3hiWWdEVlFwVmQ0Y2JoRkRiZGJGaHI2M2VVU3JCZWptSjMvCnpCUDltb0VaekVDd0dudVY5WXVvV2UrZW5ZdDYvaVY5OTlDTWtrZW9kdTFoRlA5R1pwcGdtQWVWN0h4R2ZvaHcKWGhneWtDbGI2T0d2dWsvT05wdTc3S1JYSGhTYm00aFhVck1MNnJlenZpaDZyNWQxM1dBMzZCUE1XQUFjZUV5RwpOVGFGczN0dytrWHpXOENkSWdZa0pCTFFoMG1SYzlCMitqdUF0L1YwZlkrTElzUmdYeDZRZnFzUjZWYz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

Whereas a CA certificate that is Base64 decoded appears similar to the following:

Ensure that you are using the Base64-encoded version of the CA certificate in the Kubernetes plugin configuration file.

If you have correctly configured the Kubernetes certificate authority in Launcher, then you should have a line similar to the following in your Kubernetes plugin configuration file:

File: /etc/rstudio/launcher.kubernetes.conf

certificate-authority=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EVXdNVEU0TURRME4xb1hEVE13TURReU9URTRNRFEwTjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS3NMClgzemZYNDVaYlhpRHllSGxhZWRZSm1UQ2lzVFYvc284aVlYaThpaHRaMnRnNkNmUnQzR1NtWDBMeDk5OC9rbFQKa0I5UVJOdHQ5VUJWZmQ2VW1DVXhFNU56UUV2YnI3MmNNaFh5a0wvcFdZdEdGTTJyWDUyTzBVZ3J2L3AwMURiZApTeGF6OWV3OXcvRjRRbHlxekZPNTJsaXZPeXpIbDRJL3hJRUd3L0IyN0I1ZVR3Z1pURVYyUDU4eTlmVE91bjBDCjlmZ0VBYUUvR1I1S3NrQWVaa2ZseWZaYk52WjQ2MG90R0tXaWlXaEllT0NNSUdOM0FnLzNOa1llOW1sZmVUb2MKRzY1THhVSTgxaTNjeXlvOVVGa1UvZ3N6TVpFYkM3VUhSSkdORG9MaS9aVnFucHFhMEJKM3JXeHl5WW93V3kyZwpPR2JGeFBHM0Y0ekloclFCeDNVQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLSjN0eWtyRHZKUkFDVGc1dE50MGFia3dOVk4KL0JLVjUxb0Z5dnVkTGp6NjBOUGVrb0FwRHUxZFRsMzhiMmI4WFVVVTZ6Q2FmYTdtc2RZNDJZbmU5blpyMnI4KwpHZTJ4c1F2enRHSmljOGZjVGdma0xZczVkblF2S3hiWWdEVlFwVmQ0Y2JoRkRiZGJGaHI2M2VVU3JCZWptSjMvCnpCUDltb0VaekVDd0dudVY5WXVvV2UrZW5ZdDYvaVY5OTlDTWtrZW9kdTFoRlA5R1pwcGdtQWVWN0h4R2ZvaHcKWGhneWtDbGI2T0d2dWsvT05wdTc3S1JYSGhTYm00aFhVck1MNnJlenZpaDZyNWQxM1dBMzZCUE1XQUFjZUV5RwpOVGFGczN0dytrWHpXOENkSWdZa0pCTFFoMG1SYzlCMitqdUF0L1YwZlkrTElzUmdYeDZRZnFzUjZWYz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

Verify that your certificate authority appears similar to the above example, that it was copied to the configuration file in its entirety, and that it is associated with the correct Kubernetes cluster.

Restart services and test

After you’ve verified that you have configured the correct certificate-authority in the Kubernetes plugin configuration file, restart the Workbench and Launcher services:

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the Workbench home page.

If the services are still not starting or you are still experiencing errors when starting a new session, then proceed to
Step 9 - Verify Kubernetes Certificate Configuration.