Step 10. Verify Kubernetes Token Configuration

Kubernetes plugin configuration issues

You’re unable to start new remote sessions in Kubernetes.

Workbench

Return to Troubleshooting index

Symptoms

  • Unable to start new remote sessions in Kubernetes

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page
Job information unavailable. Sessions are read-only.
Contact your administrator for help.

Error occurred while executing method (100)
Connection refused

void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197
/var/lib/rstudio-server/monitor/log/rstudio-server.log
04 May 2020 04:30:45 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678
/var/lib/rstudio-launcher/rstudio-launcher.log
04 May 2020 04:30:45 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 04:30:45 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 04:30:45 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 04:30:45 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 04:30:45 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240
/var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] Received get collection response from /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] ERROR system error 71 (Protocol error) [description=Error retrieving collection at /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||]; OCCURRED AT: rstudio::job_launcher::impls::kubernetes::KubernetesApi::getCollectionImpl(rstudio_boost::asio::io_service&, const string&, const GetCollectionHandler&, const ErrorHandler&, const rstudio_boost::shared_ptr<rstudio::core::rapid_json::Value::Array>&, const rstudio_boost::shared_ptr<rstudio_boost::asio::io_service::work>&, const string&)::__lambda3 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:251; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

Possible cause

For Workbench, Launcher, and Kubernetes to function properly, Launcher uses a service account that has access to the rstudio namespace in Kubernetes. All of this communication occurs from Launcher to the Kubernetes API.

You should have created the service account and service account token as part of the steps in the integrating Workbench with Kubernetes documentation.

The following troubleshooting steps will help you verify that you have correctly configured the Kubernetes service account token in the Kubernetes plugin configuration file.

Troubleshooting steps

You should have created a service account token in Kubernetes as part of the steps in the integrating Workbench with Kubernetes documentation, which should appear similar to the following line in your Kubernetes plugin configuration:

/etc/rstudio/launcher.kubernetes.conf
auth-token=<KUBERNETES-CLUSTER-TOKEN>

An example of a correctly configured service account token in the Kubernetes plugin configuration will appear similar to:

/etc/rstudio/launcher.kubernetes.conf
auth-token=6cVZmttolrEMJCe7NQv3jP5YdnF9Ybcf.fpoeKmbwWn17nYJZPRBoRTjBhCoqa8L7siXhvJMuOUQzInrssf2GsJpmAtTdzkDJVe1zlsrgZXSktGJ6Ulnp77MqibnlseOZGvnnf0ZgjvDDsBzoFg3UgMNQM0sEk75Ievdc3KSACLodI5ZP1l0EV7NUo72BFdnqIL3mDSis5yPaa7JnqN4S4GKDlRd4Va16fVNRCvzhr4cR223RCoMu4mp5r95ePjPnB2YLJzRu2RrM6aXFW3rQGYs9ApI71wn0A3Gfi1PU8K0weaKTvKt8QQyce1vlbif98aUdZM7EvKYtQUMbFGOIWjDO8rpzOyEttX5U7nLYwLq21RTVblLa6PPawmWMyev0fftc3zBzQJLdYo1LKHp4KSRO9lcswdYYlRRLo8wrwHINDNaUZ3igs1qs8XOknGWWoqnbXNcn3JTpVeESIY5meX7TsGm7MWgJ8TzwigcjSXg5XMwL0TRnE1pLfn3Q6f2qNQHezL.Xh6syJclMvxPyzV_OwUcPZxuaMPXakBSliH9xTCf6DQSogaEE4yN64I8orrWvYmydoRtkUfKa0JzQehFIUqDBE1J6Jcy6E9h2w-_2vl-ZlnG-jse6181Y9tqW3Qv7f7X42MXDGAYm4cl2RfkRMCtJlqBG3O0MSuURyPfh7tZkmuMe8Svv7GSzX17SXla_IVOtIGjOLkwirxORrlzsOvymF8I4haQh4v2RqgKCiVE7uEKNwERDEc_Zilzr_H3mRuUSoeyeJB28VQ0io3IPtYPjUJ8FAQIrasJJxz1S64Mc4aUeHddAKlMnBUy7atijp3v5xgtLq2jQ8u22iZf2GY9Ur

Verify that your service account token appears similar to the above example, that it was copied to the configuration file in its entirety, and that it is associated with the correct Kubernetes cluster.

Restart services and test

After you’ve verified that you have configured the correct Kubernetes auth-token in the Kubernetes plugin configuration file, restart the RStudio Server Pro and Launcher services:

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the Workbench home page.

If the services are still not starting or you are still experiencing errors when starting a new session, then proceed to Step 11 - Verify Kubernetes API Access, which will verify that the service account token has access to the rstudio namespace on the Kubernetes cluster.

Back to top