Connection Refused Error Messages
Resolving Connection Refused error messaging
Description
There may be times when you see error messages that read something like this:
rserver-monitor[23661]: ERROR system error 111 (Connection refused); OCCURRED AT void rstudio::core::http::TcpIpAsyncConnector::handleConnect(const rstudio_boost::system::error_code
Generally, this type of error message indicates a connection timeout, which more often than not, is the result of a firewall blocking access somewhere along the transmission path.
Solution
Whilst there may be any number of causes to this issue, there is a checklist of questions that can be asked which will help narrow down the issue. This can be broken down into three subsections - simple server setups, high availability setups, and connections to resource managers like Kubernetes or Slurm.
Single server
A single server runs Posit Workbench and may communicate with the internet or a Posit Connect server. If you see connection refused errors, we would suggest checking:
- Does your Posit Workbench server have internet access? You can test against the Google public DNS from the terminal by running
ping 8.8.8.8. - Does your Posit Workbench server have the appropriate firewall rules to allow internet traffic to be passed back to Workbench?
- If unable to publish to Posit Connect:
- does the firewall on your Posit Connect server permit your Posit Workbench instance to access it via its inbound firewall rules?
- does the Posit Connect server firewall have inbound access allowed for the Posit Workbench server?
- does your Posit Workbench server permit your Posit Connect instance to access it via its inbound firewall rules?
Load-balanced cluster
This architecture uses a load balancer with multiple Posit Workbench instances and an external database and filesystem. If you see connection refused errors, in addition to the questions above, investigate:
- Does your Posit Workbench instance have access to the external database via the database server’s inbound firewall rules?
- Does your external database have access to your Posit Workbench instance via its inbound firewall rules?
- Do each of the Workbench instances that sit behind the load balancer have the appropriate access via their firewall rules to pass traffic through the load balancer?
- Does your load balancer configuration permit traffic from the internet to be passed through it, and sent to your Workbench instances?
- Do your Workbench instances have access to an NFS share and can they read and write to the NFS server?
- Does your NFS share have the appropriate firewall rules to access your Workbench instances?
External resource manager
This type of server setup uses a resource manager (such as Kubernetes or Slurm) to launch containers that run your Posit Workbench instance. Resource managers can be part of a high availability setup, or they can exist on their own. This section assumes that your resource manager work standalone. If you use a resource manager with high availability, we would suggest reviewing high availability setups before this section. Ask the following questions:
- Does your Posit Workbench instance have access to your resource manager cluster via its inbound firewall rules?
- Does your resource manager cluster have access to Posit Workbench via its inbound firewall rules?
- Does your resource manager cluster have access to your NFS share?
- Does your NFS share have access to your resource manager cluster?
- Are you able to launch new pods into your resource manager cluster via the command line in your Posit Workbench instance?