Search Engine Indexing (robots.txt)

Web crawlers, such as those operated by search engines and AI tools, use a file named robots.txt to learn which parts of a site they are allowed to visit. Connect Cloud manages robots.txt for every address it serves so that your content is not indexed by search engines unless you opt in.

Default behavior

By default, Connect Cloud responds to a request for /robots.txt on every content address — both the default sharing URL (https://[content-id].share.connect.posit.cloud) and any Custom Domain you have configured — with a policy that asks all crawlers to stay away:

User-agent: *
Disallow: /

This means your published content is kept out of search engine results by default. No action is required, and this applies whether your content is public or private.

Note

robots.txt is a request, not an enforcement mechanism. Well-behaved crawlers (major search engines, most AI crawlers) honor it, but it does not block access to your content. Use sharing controls — such as disabling Public Access — to actually restrict who can view your content.

Serving your own robots.txt

If you want search engines to crawl and index your content, you can provide your own robots.txt. When your content supplies one, Connect Cloud serves it instead of the default — but only when your content is publicly accessible. How you provide it depends on the type of content you deploy:

  • Static files (static sites and other files served as-is): include a robots.txt file alongside your content, at the root so it is reachable at /robots.txt.
  • Rendered content (Quarto, R Markdown, and other formats that are rendered before publishing): Connect Cloud serves the rendered output, not your source files, so a robots.txt in your project is not enough on its own — you must ensure it ends up in the rendered output directory. For example, in Quarto you can list it under resources in _quarto.yml so it is copied into the output.
  • Applications (Shiny, Dash, Streamlit, and other running apps): make your app respond to a request for /robots.txt with your crawl policy. When your content is deployed, Connect Cloud probes the running app with a request for /robots.txt. Only an HTTP 200 response counts as “the app supplies its own robots.txt”; in that case Connect Cloud serves your app’s response to crawlers. Any other outcome — a 404, a 500, a redirect, or no response at all — is treated as “no custom robots.txt,” and the default disallow-all policy is used. Because this check runs at deploy time, redeploy your content after adding or changing how your app handles /robots.txt.

To publish your own crawl policy:

  1. Add a robots.txt to your content using the method above for your content type.
  2. Make sure the content’s Public Access is enabled.
  3. Redeploy the content if it was already published.

A request for /robots.txt on that content’s address will then return your file. For example, to allow all crawlers:

User-agent: *
Allow: /

Or to allow most of your site while keeping one path private from crawlers:

User-agent: *
Disallow: /drafts/
Important

Your own robots.txt is only served when your content is publicly accessible — that is, anyone can view it without signing in. If Public Access is disabled so that the content requires a login or is restricted to named individuals or an organization, Connect Cloud always serves the default disallow-all policy. Because crawlers are anonymous visitors, they could not read a non-public content’s own robots.txt in any case.

Keeping a public page out of search results

Because robots.txt controls crawling rather than indexing, disallowing a page does not guarantee it will never appear in search results — a search engine can still list a URL it has discovered elsewhere. To keep a specific public page out of search results entirely, add a noindex directive to the page itself, for example with an HTML meta tag:

<meta name="robots" content="noindex">

The default Connect Cloud policy (disallow all) is sufficient for the common goal of keeping content from being crawled.