Step 5. Verify that clocks between the rserver node and Kubernetes cluster’s clocks are synced

Launcher configuration issues

You’re unable to start new remote sessions in Kubernetes.

Symptoms

Unable to run new Kubernetes sessions

Error messages

When inspecting the log files for Posit Workbench, Launcher, and Kubernetes, you might see errors similar to the following:

/var/lib/rstudio-server/monitor/log/rstudio-server.log

19 Feb 2021 04:24:33 [rsession-test] ERROR system error 13 (Permission denied) [description: Received stale message with date Fri, 19 Feb 2021 03:25:51 GMT]; OCCURRED AT rstudio::core::Error rstudio::server_core::sessions::verifyRequestSignature(const string&, const string&, const rstudio::core::http::Request&, bool) src/cpp/server_core/sessions/SessionSignature.cpp:139; LOGGED FROM: bool rstudio::session::{anonymous}::verifyRequestSignature(const rstudio::core::http::Request&) src/cpp/session/SessionHttpMethods.cpp:366

/var/lib/rstudio-server/monitor/log/rstudio-server.log

19 Feb 2021 04:24:33 [rsession-test] ERROR Invalid signature for request URI /rpc/client_init; LOGGED FROM: bool rstudio::session::http_methods::waitForMethod(const string&, const rstudio_boost::function<void()>&, const rstudio_boost::function<bool()>&, rstudio::core::json::JsonRpcRequest*) src/cpp/session/SessionHttpMethods.cpp:504

Possible cause

A 401 error combined with log messages stating invalid signature (similar to the ones shown above) points to a clock skew issue.

Messages like:

Received stale message with date Fri, 19 Feb 2021 03:25:51 GMT

indicate that the clocks between the rserver node and the newly added Kubernetes node are not in sync.

Whenever a browser request is made to the session, the rserver process creates a signature for the session, and the Date header is used to notate the time that the request was forwarded to the session (running as a job in Kubernetes). Once the session receives the request, it compares the signatures to validate that they match. The HTTP date on the request is compared to the system’s local time, and if the delta is more than 60 seconds, then the request is denied - resulting in a 401 error.

Troubleshooting steps

In the /var/lib/rstudio-server/monitor/log/rstudio-server.log file, compare the HTTP date on the request to the system’s local time. If the delta is more than 60 seconds, then the clock on one of the nodes is not correct.

Note

Additionally, if the server’s clock is sufficiently incorrect, licensing verification will fail.

Synchronize all node clocks via NTP, or some other mechanism. Beware that depending on configuration, some clock skew resolution patterns can take a while to resolve (i.e., marginally speeding up the clock, etc.)

Restart Services and test

After running the verification process then identifying and correcting any issues, restart the Workbench and Launcher services:

$ sudo rstudio-server restart
$ sudo rstudio-launcher restart

Verify that the services are running and try to start a new session from the Workbench home page.

If the services are still not starting or you are still experiencing errors when starting a new session, then proceed to
Step 6 - Verify Configuration Files.