Troubleshooting

Debugging

You can debug the behavior or Workbench in two different ways. Both of these methods are only available to the WORKBENCH_ADMIN application role.

  • The Administer Workbench Streamlit app (recommended)
  • Stored procedures included in the Native App

Administer Workbench

The Administer Workbench Streamlit app includes functionality:

  1. Visit the application page, navigating through Data Products > Apps.

  2. Click Administer Workbench in the left of the top navigation bar. If this option is not available, your current role does not have the WORKBENCH_ADMIN application role, or the Workbench Native App is in a PENDING / DONE state and you need to use stored procedures for diagnosing your application.

Administer Workbench using the Streamlit App

Debug page

Access the Debug from the left navigation pane.

Debug page in the Administration App

Application state

The Workbench Native App can be in one of the following states:

State Meaning
READY Workbench is up and running.
PENDING Workbench is starting up. This should take less than five minutes to complete.
DONE Workbench failed to start. Check the Workbench Logs section and click View log lines. Lines with an ERROR should help identify the reason that startup failed.

Workbench logs

Workbench logs

Executing SQL statements

Warning

Executing arbitrary SQL can be destructive and could make your Workbench Native App unusable.

Only execute SHOW, SELECT, and DESCRIBE commands.

We do not recommend executing any statements which alter, replace, or drop resources.

A Debug Native Application section is provided at the bottom of the page. This functionality allows you to execute SQL statements as the application itself.

We recommend using this only as a last resort.

Running SQL statements as the Native App

Stored Procedures

Several stored procedures are included as part of the Native App.

The examples below assume that your application name is posit_workbench

Workbench State

CALL posit_workbench.app.workbench_state();

If "workbench_status": "DONE" it typically means that the application has errored.

Full Workbench State
{
  "application_name": "POSIT_WORKBENCH",
  "current_account": "...",
  "current_account_name": "...",
  "current_organization_name": "...",
  "privileges_set": {
    "CREATE_COMPUTE_POOL": true
  },
  "references_set": {
    "EGRESS": true,
    "POSIT_LICENSE": true,
    "SNOWFLAKE_OAUTH": true
  },
  "workbench_status": "READY"
}

Workbench Logs

CALL posit_workbench.app.workbench_logs();
...
2024-06-25T15:16:02.229952Z [rserver] INFO Start request: /health-check
2024-06-25T15:16:02.336270Z [rserver] INFO Result for /health-check cpu: 2.5% mem: 24.2% swap: 0.0% load: 0.1 sessions: 0 idle: 23623s in: 0.0s
2024-06-25T15:16:07.229626Z [rserver] INFO Start request: /health-check
...

Procedure list

App Schema
--- object representation of the Workbench state
posit_workbench.APP.WORKBENCH_STATE()
--- string service manifest for the given Workbench state
posit_workbench.APP.WORKBENCH_SPEC(snapshot_name STRING DEFAULT '')

--- start or resume the Workbench service (does not modify the service manifest)
posit_workbench.APP.WORKBENCH_START(snapshot_name STRING DEFAULT '', wait BOOLEAN DEFAULT false)
--- start or upgrade the Workbench service (ensures service is up to date with the service manifest)
posit_workbench.APP.WORKBENCH_UPGRADE(snapshot_name STRING DEFAULT '', wait BOOLEAN DEFAULT false)
--- create a snapshot of the current /home volume within Workbench
posit_workbench.APP.WORKBENCH_BACKUP(snapshot_name STRING)
--- restore the /home volume within Workbench to a snapshot
posit_workbench.APP.WORKBENCH_RESTORE(snapshot_name STRING, wait BOOLEAN DEFAULT false)
--- stop Workbench, which will put the service in a suspended state (call workbench_start/workbench_upgrade to resume)
posit_workbench.APP.WORKBENCH_STOP()

--- retrieve the endpoint for the Workbench service
posit_workbench.APP.WORKBENCH_ENDPOINT()
--- retrieve logs from the Workbench container
posit_workbench.APP.WORKBENCH_LOGS()
Callback Schema
--- internal callbacks used by the Snowsight UI, and should not be called unless you really know 
--- what you are doing. This can be used to configure custom references for the EGRESS
posit_workbench.CALLBACK.CONFIGURE_REFERENCE(ref_name STRING)
posit_workbench.CALLBACK.REGISTER_REFERENCE(ref_name STRING, operation STRING, ref_alias STRING)
posit_workbench.CALLBACK.GRANT_PRIVILEGES(privileges ARRAY)
posit_workbench.CALLBACK.INITIALIZE_VERSION()

Known Issues

These issues are ordered from most to least frequently seen.

Activating Posit Workbench for over 10 minutes

If your deployment hangs on the Activating Posit Workbench screen for over 15 minutes you should check on the state of Workbench. It is likely in the DONE state, indicating that Workbench failed to start. A current limitation of Workbench while in this state is that the Streamlit Administer Workbench app is not accessible in the UI and the stored procedures must be used.

Waiting for the Native App to activate

Run the stored procedure to investigate the state of Workbench. If the workbench_state is:

  • PENDING: wait a few more minutes for Workbench to finalize provisioning. This should never take over 20 minutes.
  • READY: at times this Snowflake page will not refresh. Please refresh the activating Posit Workbench page.
  • DONE: this is a clear indication that the Workbench service failed to start and the logs will indicate the failure.

If the state is DONE check the Workbench logs. If the known issues listed below do not provide a solution, open a support ticket.

Entering the license

The Posit License file has the following file format.

-----BEGIN RSTUDIO LICENSE-----
...
...
...
-----END RSTUDIO LICENSE-----

When configuring the Posit license, you must copy the entire contents of the file and paste the license key into the secret string value. This must include the lines with -----. Upon changing the license file, the Workbench service takes 2-3 minutes to restart. If it takes longer than this, please check the Workbench State and Workbench Logs.

Add a license to the Native App

Custom Egress (Network Policies and Network Rules) Policies

If you have custom network policies, or network rules, and your deployment is failing, or OAuth is not completing, we recommend deactivating those network rules and network policies. Once you have a working deployment, then try to re-enable those policies. Workbench requires OAuth Snowflake endpoints, Snowflake OCSP caching endpoints, and Snowflake ODBC query endpoints, which can be tricky to get correct.

Custom egress policies can lead to hard-to-debug failures of the Workbench Native App. See changing the egress policy for detailed information on how to restrict egress. Consult the Workbench logs or the workbench admin endpoint and view logs if you encounter issues with egress. The most common signs of issues with your egress policy are request timeouts, OAuth errors, HTTP 403 errors when connecting to, or using Workbench, and CORS issues. The Snowflake egress policy network rules drop packets which results in requests hanging from the user’s perspective. It is strongly suggested to modify the egress rules after a successful installation, only changing one rule at a time, and to allow 2-3 minutes for each egress rule to take effect.

32 character limit on Snowflake usernames

In the Workbench logs, this error looks like the following.

{"time":"2024-06-24T15:24:10.27987497Z","level":"ERROR","msg":"error syncing snowflake users with system","error":"useradd failed: exit status 3, stderr: useradd: invalid user name '........'}

Currently there is a 32 character limit on Workbench usernames. These usernames were populated in the installation of users. The next Workbench release in September 2024 (2024.09) will remove this restriction. For now, we recommend removing usernames in the workbench_users table which exceed this length.

DELETE FROM posit_workbench.app.workbench_users WHERE LENGTH(username) > 32

No outbound SSH access

Snowflake egress does not allow a large number ports under 1024 for egress. Port 22 (ssh) is one of those that is entirely blocked. This affects git clone and other operations, but usually there is a workaround by using port 443. Github has documented ssh over https as a workaround.

Ingress URL has changed

Existing OAuth integrations break if the endpoint URL for Workbench changes. You will see an error message like the following:

Error occurred in authorization

There is a mismatch in the given redirect URI with the one in the registered OAuth client integration.

You can fix this by running by executing the following SQL in a worksheet.

In this example the application name is posit_workbench and the integration name is posit_workbench_oauth. If you are using different names you must update the query accordingly.

DECLARE
    endpoint_url STRING;
    stmt STRING;
BEGIN
    CALL posit_workbench.app.workbench_endpoint() INTO endpoint_url;
    stmt := 'ALTER INTEGRATION posit_workbench_oauth SET OAUTH_REDIRECT_URI = ''' || :endpoint_url || '/oauth_redirect_callback''';
    EXECUTE IMMEDIATE :stmt;
    RETURN stmt;
END;

Authenticating with multiple roles other than your default role

Currently the Workbench Native App only allows one authenticated role that is their default role. The next Workbench release in September 2024 (2024.09) will remove this restriction and allow multiple authenticated roles.

Error: Unauthorized user.

Error: Unauthorized user.

This error will disappear in the next Workbench release in September 2024 (2024.09). It indicates that the user has not been initiazed in Workbench. Make sure to complete Step 4 of the Installation documentation, Adding Workbench users. You can view your currently initialized users by executing the following SQL.

SELECT username, display_name FROM posit_workbench.data.workbench_users;

Adding Workbench users takes 5 minutes to sync with the Workbench Native App.

User unable to see Workbench Native App in Data Products -> Apps

If you are unable to see the Workbench Native App within the Snowsight UI, it indicates that your current role does not have access to the application. In order to see the application, the given role must have the Workbench Native App application role WORKBENCH_USER or WORKBENCH_ADMIN. Either switch to a role which has access to the application, or contact your IT team to grant the application role to your role.

Back to top