Troubleshooting

Debugging

The Posit Workbench Native App can be debugged using the following two methods and requires the WORKBENCH_ADMINapplication role. Refer to the user-access section of the install guide for more information.

  1. Administer Workbench user interface (Recommended)

Provides an interactive interface for debugging and administrative tasks such as backup and restore. Learn more in the interacting with your app documention.

  1. Stored SQL procedures included in the Native App

Allows debugging directly via SQL commands included with the Native App. Learn more in the write your own Native App SQL documentation.

Administer Workbench user interface

  1. Visit the application page, navigating through Data Products > Apps.

  2. Click Administer Workbench in the left of the top navigation bar. If this option is not available, your current role does not have the WORKBENCH_ADMIN application role, or the Workbench Native App is in a PENDING / DONE state and you need to use stored procedures for diagnosing your application.

Administer Workbench UI

Administer Workbench UI

Debug page

Access the Debug from the left navigation pane.

Debug page in the Administration App

Debug page in the Administration App

Application state

The Workbench Native App can be in one of the following states:

State Meaning
READY Workbench is up and running.
PENDING Workbench is starting up. This should take less than five minutes to complete.
DONE Workbench failed to start. Check the Workbench Logs section and click View log lines. Lines with an ERROR should help identify the reason that startup failed.

Workbench logs

Workbench logs

Workbench logs

Executing SQL statements

Warning

Executing arbitrary SQL can be destructive and could make your Workbench Native App unusable.

Only execute SHOW, SELECT, and DESCRIBE commands.

We do not recommend executing any statements which alter, replace, or drop resources.

A debug section is provided at the bottom of the Administer Workbench page. This functionality allows you to execute SQL statements as the application itself. Note that this is different from running SQL queries in a Snowsight worksheet as a WORKBENCH_ADMIN or ACCOUNTADMIN role.

Running SQL statements as the Native App

Running SQL statements as the Native App

Stored procedures

Several stored procedures are included as part of the Native App. These can also be run in a Snowsight worksheet. Refer to the Snowflake user guide Snowsight worksheets documentation for more information.

The examples below assume that your application name is posit_workbench

Workbench state

CALL posit_workbench.app.workbench_state();

If "workbench_status": "DONE" it typically means that the application has errored.

Full Workbench State
{
  "application_name": "POSIT_WORKBENCH",
  "current_account": "...",
  "current_account_name": "...",
  "current_organization_name": "...",
  "privileges_set": {
    "CREATE_COMPUTE_POOL": true
  },
  "references_set": {
    "EGRESS": true,
    "POSIT_LICENSE": true,
    "SNOWFLAKE_OAUTH": true
  },
  "workbench_status": "READY",
  "workbench_message": "Running."
}

Workbench logs

CALL posit_workbench.app.workbench_logs();
...
2024-06-25T15:16:02.229952Z [rserver] INFO Start request: /health-check
2024-06-25T15:16:02.336270Z [rserver] INFO Result for /health-check cpu: 2.5% mem: 24.2% swap: 0.0% load: 0.1 sessions: 0 idle: 23623s in: 0.0s
2024-06-25T15:16:07.229626Z [rserver] INFO Start request: /health-check
...

Procedure list

The Workbench Native App includes several stored procedures for managing its state, upgrading, backups, and more. Below is a categorized list with brief descriptions of each procedure.

Instructions:

  1. Use the ACCOUNTADMIN or WORKBENCH_ADMIN roles

  2. Run the procedure in a Snowsight worksheet or in the debug section of the Administer Workbench user interface debug section. Note that these run using different roles by default.

  3. The procedure syntax is CALL posit_workench.APP.STORED_PROCEDURE() where posit_workbench is the name of your Native App, APP is the stored procedure schema, and STORED_PROCEDURE() is one of the below functions.

Application state management

  • WORKBENCH_STATE()

    Retrieves an object representation of the current Workbench state.

  • WORKBENCH_SPEC(snapshot_name STRING DEFAULT '')

    Returns a string service manifest for the given Workbench state.

Starting, upgrading, and stopping Workbench

  • WORKBENCH_START(snapshot_name STRING DEFAULT '', wait BOOLEAN DEFAULT false)

    Starts or resumes the Workbench service without modifying the service manifest.

  • WORKBENCH_UPGRADE(snapshot_name STRING DEFAULT '', wait BOOLEAN DEFAULT false)

    Starts or upgrades the Workbench service to ensure it is up to date with the service manifest.

  • WORKBENCH_STOP()

    Stops the Workbench service, putting it in a suspended state. Use WORKBENCH_START or WORKBENCH_UPGRADE to resume the service.

Backup and restore

  • WORKBENCH_BACKUP(snapshot_name STRING, wait BOOLEAN DEFAULT false)

    Creates a snapshot of the /home volume within Workbench.

  • WORKBENCH_RESTORE(snapshot_name STRING, wait BOOLEAN DEFAULT false)

    Restores the /home volume within Workbench to a specific snapshot.

Compute Pool Management

  • WORKBENCH_RESIZE_COMPUTE(instance_name STRING DEFAULT 'CPU_X64_S', wait BOOLEAN DEFAULT false)

    Resizes the compute pool for a running Workbench instance.

Logging and endpoint management

  • WORKBENCH_ENDPOINT()

    Retrieves the endpoint URL for the Workbench service.

  • WORKBENCH_LOGS()

    Fetches logs from the Workbench container.

Advanced configuration (use with caution)

  • CALLBACK.CONFIGURE_REFERENCE(ref_name STRING)

    Configures custom references for the egress network.

  • CALLBACK.REGISTER_REFERENCE(ref_name STRING, operation STRING, ref_alias STRING)

    Registers a reference for operations or aliases.

  • CALLBACK.GRANT_PRIVILEGES(privileges ARRAY)

    Grants specified privileges to the Workbench service.

  • CALLBACK.INITIALIZE_VERSION()

    Initializes the version for internal callbacks used by the Snowflake UI.

Commonly used procedures

Examples of the procedures you are most likely to use:

--- Check the current state of the Workbench.
CALL posit_workbench.APP.WORKBENCH_STATE();

--- View logs for troubleshooting.
CALL posit_workbench.APP.WORKBENCH_LOGS();

--- Manage Workbench snapshots.
CALL posit_workbench.APP.WORKBENCH_BACKUP(); 
CALL posit_workbench.APP.WORKBENCH_RESTORE();

Known issues

These issues are ordered from most to least frequently seen.

Activating Posit Workbench for over 10 minutes

If your deployment hangs on the Activating Posit Workbench screen for over 15 minutes you should check on the state of Workbench. It is likely in the DONE state, indicating that Workbench failed to start.

A current limitation of Workbench while in this state is that the Administer Workbench user interface is not accessible in the UI and the stored procedures must be used.

Waiting for the Native App to activate

Waiting for the Native App to activate

Run the stored procedure to investigate the state of Workbench. If the workbench_state is:

  • PENDING: wait a few more minutes for Workbench to finalize provisioning. This should never take over 20 minutes.
  • READY: at times this Snowflake page will not refresh. Please refresh the activating Posit Workbench page.
  • DONE: this is a clear indication that the Workbench service failed to start and the logs will indicate the failure.

If the state is DONE check the Workbench logs. If the known issues listed below do not provide a solution, open a support ticket.

License entered incorrectly

The Posit License file has the following file format.

-----BEGIN RSTUDIO LICENSE-----
...
...
...
-----END RSTUDIO LICENSE-----

When configuring the Posit license, you must copy the entire contents of the file and paste the license key into the secret string value. This must include the lines with -----. Upon changing the license file, the Workbench service takes 2-3 minutes to restart. If it takes longer than this, please check the Workbench state and Workbench logs.

To update your license if it has expired, use the instructions detailed in the Updating your Posit license documentation.

Add a license to the Native App

Add a license to the Native App

Custom egress errors

Custom egress policies can sometimes lead to failures in the Workbench Native App that are hard to debug. Below are guidelines for managing egress policies effectively and troubleshooting related issues.

Minimum requirements

At a minimum, Workbench requires the following for proper functionality:

Best practices for custom egress policies

  1. Deploy Workbench without custom egress first

Ensure a successful deployment with default egress policies before making custom changes. Then gradually tighten rules while monitoring functionality. This helps confirm that Workbench is functioning correctly.

  1. Modify egress policies gradually

Change only one rule at a time. Allow five seconds for each change to take effect. Always test the system after each change.

Ensure that key endpoints are accessible, such as those for Snowflake OAuth and OCSP, are always accessible.

Common egress policy issues

  • Request timeouts: Requests to Workbench seem to hang indefinitely.

  • OAuth errors: Authentication fails with errors such as HTTP 403.

  • CORS issues: Cross-Origin Resource Sharing (CORS) errors when using the Workbench UI.

  • General failures: Network requests originating from the Workbench Native App are dropped by the Snowflake network policy.

Troubleshooting egress

  1. Check Workbench logs

Use the Workbench logs in the Administer Workbench app or stored procedures to identify errors.

  1. Enable initial network check

Workbench performs a network check on startup. Errors in this check can often indicate egress policy issues.

  1. Rollback custom egress

Undo recent changes to egress policies to determine whether they caused the issue.

For more details on configuring egress policies, refer to the change the egress policy documentation.

Account Level Network Policies

If your Snowflake account has an active network policy, you may encounter issues with the Workbench Native App during the OAuth flow or while running Snowflake SQL queries from within Workbench sessions. For more details on activating a network policy for your account, refer to Snowflake’s network policies documentation.

Common Errors

  1. Oauth Flow Error

Error message:

Error 100: Error occurred while executing method.

Typically occurs during the Workbench managed credential OAuth flow.

  1. SQL Query Error:

Error message:

Incoming request with IP/Token aaa.bbb.ccc.ddd is not allowed to access Snowflake. Contact your account administrator.

Occurs when executing SQL queries via R, Python, or a CLI in the Native App.

These errors occur because Snowflake does not trust the public IPs originating from Snowflake SPCS services (e.g., Workbench Native App). To resolve this, you need to whitelist the public IPs associated with your Workbench deployment.

Resolution Steps

Step 1: Obtain the Public IPs of the Workbench Native App

You can identify the public IPs of your Workbench deployment by running the following Python script. This example uses ipinfo.io, a trusted service, to fetch the IP addresses. Ensure that your egress policy allows traffic to ipinfo.io:443.

import requests
ips = {(requests.get('https://ipinfo.io/json').json())['ip'] for _ in range(20)}
print('\n'.join(ips))

Alternatively, execute the following command in your shell to obtain the IPs:

python -c "import requests; print('\n'.join({(requests.get('https://ipinfo.io/json').json())['ip'] for _ in range(20)}))"

This process typically returns three IPs, but repeat the commands if necessary to ensure all IPs are captured.

Step 2: Update the Network Policy

Once you have the public IPs, add them to the ALLOWED_NETWORK_RULE_LIST of your Snowflake account’s active network policy. This change will allow the Workbench Native App to authenticate and execute queries successfully.

Example SQL to update the network policy:

ALTER NETWORK POLICY your_network_policy_name
SET ALLOWED_IP_LIST = ('existing_ip_1', 'existing_ip_2', 'new_workbench_ip_1', 'new_workbench_ip_2', 'new_workbench_ip_3')
BLOCKED_IP_LIST=('blocked_ip_1');

DESC NETWORK POLICY your_network_policy_name;

Changes to the network policy take effect within five seconds. You should see OAuth flows and SQL queries resolve immediately.

Additional Notes

  • Public IP Changes:

Snowflake may occasionally update the public IPs associated with SPCS services. While changes are infrequent, it’s a good practice to periodically verify and update the IP list to prevent disruptions. We are working with Snowflake to provide an official mechanism for retrieving these IPs.

  • Custom Egress Rules:

If you are using custom egress policies, ensure that ipinfo.io:443 or an equivalent IP identification service is allowed.

  • Troubleshooting Tip:

If errors persist, verify the updated network policy and ensure all relevant public IPs are included. You can also consult the Workbench logs for additional details.

No outbound SSH access

Port 22 (ssh) is blocked by SPCS. This affects git clone and other operations, but usually there is a workaround by using port 443. Github has documented ssh over https as a workaround.

Ingress URL has changed

Existing OAuth integrations break if the endpoint URL for Workbench changes. The endpoint URL changes anytime the service is deleted and recreated. This happens on compute pool resizes and Native App upgrades. You will see an error message like the following:

Error occurred in authorization

There is a mismatch in the given redirect URI with the one in the registered OAuth client integration.

You can fix this by running the following SQL in a worksheet with ACCOUNTADMIN credentials.

DECLARE
    stmt STRING;
BEGIN
    CALL posit_workbench.app.workbench_oauth_update() INTO :stmt;
    EXECUTE IMMEDIATE :stmt;
END;

User unable to see Workbench Native App in Snowsight

If you are unable to see the Workbench Native App within the Snowsight user interface, it indicates that your current role does not have access to the application. In order to see the application, the given role must have the Workbench Native App application role WORKBENCH_USER or WORKBENCH_ADMIN. Either switch to a role which has access to the application, or contact your IT team to grant the application role to your role.

User credentials not persisting

User credentials not persisting in an Rstudio IDE or VS Code session usually means that the OAuth credential flow was not fully completed. Please review the Snowflake Managed credentials guide for details.

A common error is not fully completing the Session Credentials flow to pass in your role credentials.

The below shows an unclicked blue circle which will result in missing user credentials:

unsuccessful flow

unsuccessful flow

The below shows a clicked blue circle which will successfully pass in the user credentials specified:

successful flow

successful flow

After a successful flow the user will be able to see their role in the key icon on the right hand side of the session screen.

credential-passthrough

credential-passthrough

Users are also able to switch roles in the Edit Credentials dialog. See the Signing into Snowflake Roles and Accounts for more information.

Back to top