Troubleshooting Launcher and Kubernetes:
Verify Kubernetes API Access#

Symptoms#

Unable to start new remote sessions in Kubernetes

Error messages#

When inspecting the log files for Posit Workbench, (formerly RStudio Workbench), Launcher, and Kubernetes, you might see errors similar to the following:

Workbench Home Page

Job information unavailable. Sessions are read-only.
Contact your administrator for help.

Error occurred while executing method (100)
Connection refused

void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197

File: /var/lib/rstudio-server/monitor/log/rstudio-server.log

04 May 2020 04:30:45 [rserver] ERROR system error 111 (Connection refused); OCCURRED AT: void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code&, rstudio_boost::asio::ip::basic_resolver<rstudio_boost::asio::ip::tcp>::iterator) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/core/include/core/http/TcpIpAsyncConnector.hpp:197; LOGGED FROM: rstudio::server::job_launcher::{anonymous}::ensureServerUserIsLauncherAdmin()::__lambda7::__lambda10 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/server/ServerJobLauncher.cpp:678

File: /var/lib/rstudio-launcher/rstudio-launcher.log

04 May 2020 04:30:45 [rstudio-launcher] Bootstrapping plugin Kubernetes
04 May 2020 04:30:45 [rstudio-launcher] Sending message to plugin Kubernetes: {"messageType":1,"requestId":0}
04 May 2020 04:30:45 [rstudio-launcher] Plugin Kubernetes exited with code 1
04 May 2020 04:30:45 [rstudio-launcher] ERROR Could not start plugin Kubernetes; LOGGED FROM: rstudio::core::Error rstudio::job_launcher::plugins::PluginManager::start() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/PluginManager.cpp:148
04 May 2020 04:30:45 [rstudio-launcher] ERROR system error 71 (Protocol error) [description=Failed to properly bootstrap plugin Kubernetes]; OCCURRED AT: rstudio::core::Error rstudio::job_launcher::plugins::Plugin::bootstrap() /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/plugins/Plugin.cpp:189; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/LauncherMain.cpp:240

File: /var/lib/rstudio-launcher/Kubernetes/rstudio-kubernetes-launcher.log

04 May 2020 04:30:44 [rstudio-kubernetes-launcher] Received get collection response from /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||
04 May 2020 04:30:44 [rstudio-kubernetes-launcher] ERROR system error 71 (Protocol error) [description=Error retrieving collection at /apis/batch/v1/namespaces/rstudio/jobs?limit=500: Unauthorized - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}|||]; OCCURRED AT: rstudio::job_launcher::impls::kubernetes::KubernetesApi::getCollectionImpl(rstudio_boost::asio::io_service&, const string&, const GetCollectionHandler&, const ErrorHandler&, const rstudio_boost::shared_ptr<rstudio::core::rapid_json::Value::Array>&, const rstudio_boost::shared_ptr<rstudio_boost::asio::io_service::work>&, const string&)::__lambda3 /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/kubernetes/KubernetesApi.cpp:251; LOGGED FROM: int rstudio::job_launcher::impls::entry_point::run(rstudio_boost::shared_ptr<rstudio::job_launcher::impls::ApiBase>, rstudio_boost::shared_ptr<rstudio::job_launcher::impls::FrameworkCommunicator>) /var/lib/jenkins/workspace/IDE/pro-pipeline/v1.2-patch/src/cpp/job_launcher/impls/EntryPoint.cpp:134

Possible cause#

For Workbench, Launcher, and Kubernetes to function properly, Launcher uses a service account that has access to the rstudio namespace in Kubernetes. All of this communication occurs from Launcher to the Kubernetes API.

You should have created a service account and service account token as part of the steps in the integrating Workbench with Kubernetes documentation.

The following troubleshooting steps ensure that you can connect to the Kubernetes API, that your service account token is valid, and that the service account has access to the rstudio namespace.

Troubleshooting steps#

Verify that you can connect to the Kubernetes API and that the service account token that you generated has access to the rstudio namespace:

Terminal

$ export TOKEN=<KUBERNETES-CLUSTER-TOKEN>
$ curl -k -H "Authorization: Bearer $TOKEN" https://<KUBERNETES-API-ENDPOINT>/api/v1/namespaces/rstudio/

where <KUBERNETES-CLUSTER-TOKEN> is the Kubernetes service account token that you generated from the documentation for Integrating Workbench with Kubernetes, and <KUBERNETES-API-ENDPOINT> is the URL for the Kubernetes API.

If the Kubernetes cluster token is valid and the associated service account is authorized to access the rstudio namespace, then you should see a response from the Kubernetes API such as:

{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "rstudio",
    "selfLink": "/api/v1/namespaces/rstudio",
    "uid": "d802450e-2c5b-4520-a7d5-6ef2ffa5c969",
    "resourceVersion": "769",
    "creationTimestamp": "2020-04-09T17:20:45Z"
  },

  "spec": {
    "finalizers": [
      "kubernetes"
    ]
  },
  "status": {
    "phase": "Active"
  }
}

If the Kubernetes cluster token is not valid or the associated service account is not authorized to access the rstudio namespace, then you will see a response from the Kubernetes API such as:

{
 "kind": "Status",
 "apiVersion": "v1",
 "metadata": {

},
 "status": "Failure",
 "message": "Unauthorized",
 "reason": "Unauthorized",
 "code": 401
}

If you are receiving an "Unauthorized" message as shown above, then you can attempt to recreate the service account and generate a service account token as part of the steps in the integrating Workbench with Kubernetes documentation.

After you've generated a new service account token, you can repeat these troubleshooting steps to ensure that your service account token is valid and has access to the rstudio namespace.

Once you've verified that the Kubernetes service account token is valid, ensure that you've configured it correctly in the Kubernetes plugin configuration:

File: /etc/rstudio/launcher.kubernetes.conf

auth-token=<KUBERNETES-CLUSTER-TOKEN>

Restart services and test#

After you've verified that you have configured the correct Kubernetes auth token and that it is authorized to access the rstudio namespace on the Kubernetes cluster, restart the Workbench and Launcher services: