Step 11. Verify Kubernetes API Access

Kubernetes configuration and environment issues

You’re unable to start new remote sessions in Kubernetes.

Workbench

Return to Troubleshooting index

Symptoms

  • Unable to start new remote sessions in Kubernetes

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page
Job information unavailable. Sessions are read-only.
Contact your administrator for help.

Error occurred while executing method (100)
Connection refused

void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197
/var/lib/rstudio-server/monitor/log/rstudio-server.log
04 May 2020 04:30:45 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678
File: /var/lib/rstudio-launcher/rstudio-launcher.log
04 May 2020 04:30:45 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 04:30:45 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 04:30:45 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 04:30:45 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 04:30:45 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240
File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] Received get collection response from /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] ERROR system error 71 (Protocol error) [description=Error retrieving collection at /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||]; OCCURRED AT: rstudio::job_launcher::impls::kubernetes::KubernetesApi::getCollectionImpl(rstudio_boost::asio::io_service&, const string&, const GetCollectionHandler&, const ErrorHandler&, const rstudio_boost::shared_ptr<rstudio::core::rapid_json::Value::Array>&, const rstudio_boost::shared_ptr<rstudio_boost::asio::io_service::work>&, const string&)::__lambda3 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:251; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

Possible cause

For Workbench, Launcher, and Kubernetes to function properly, Launcher uses a service account that has access to the rstudio namespace in Kubernetes. All of this communication occurs from Launcher to the Kubernetes API.

You should have created a service account and service account token as part of the steps in the integrating Workbench with Kubernetes documentation.

The following troubleshooting steps ensure that you can connect to the Kubernetes API, that your service account token is valid, and that the service account has access to the rstudio namespace.

Troubleshooting steps

Verify that you can connect to the Kubernetes API and that the service account token that you generated has access to the rstudio namespace:

$ export TOKEN=<KUBERNETES-CLUSTER-TOKEN>
$ curl -k -H "Authorization: Bearer $TOKEN" https://<KUBERNETES-API-ENDPOINT>/api/v1/namespaces/rstudio/​

where <KUBERNETES-CLUSTER-TOKEN> is the Kubernetes service account token that you generated from the documentation for Integrating Workbench with Kubernetes, and <KUBERNETES-API-ENDPOINT> is the URL for the Kubernetes API.

If the Kubernetes cluster token is valid and the associated service account is authorized to access the rstudio namespace, then you should see a response from the Kubernetes API such as:

{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "rstudio",
    "selfLink": "/api/v1/namespaces/rstudio",
    "uid": "d802450e-2c5b-4520-a7d5-6ef2ffa5c969",
    "resourceVersion": "769",
    "creationTimestamp": "2020-04-09T17:20:45Z"
  },

  "spec": {
    "finalizers": [
      "kubernetes"
    ]
  },
  "status": {
    "phase": "Active"
  }
}

If the Kubernetes cluster token is not valid or the associated service account is not authorized to access the rstudio namespace, then you will see a response from the Kubernetes API such as:

{
 "kind": "Status",
 "apiVersion": "v1",
 "metadata": {

},
 "status": "Failure",
 "message": "Unauthorized",
 "reason": "Unauthorized",
 "code": 401
}

If you are receiving an "message": "Unauthorized" as shown above, then you can attempt to recreate the service account and generate a service account token as part of the steps in the integrating Workbench with Kubernetes documentation.

After you’ve generated a new service account token, you can repeat these troubleshooting steps to ensure that your service account token is valid and has access to the rstudio namespace.

Once you’ve verified that the Kubernetes service account token is valid, ensure that you’ve configured it correctly in the Kubernetes plugin configuration:

/etc/rstudio/launcher.kubernetes.conf
auth-token=<KUBERNETES-CLUSTER-TOKEN>

Restart services and test

After you’ve verified that you have configured the correct Kubernetes auth token and that it is authorized to access the rstudio namespace on the Kubernetes cluster, restart the Workbench and Launcher services:

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the Workbench home page.

If the services are still not starting or you are still experiencing errors when starting a new session, then proceed to
Step 12 - Verify Docker Image Configuration and Access.

Back to top