To edit a Shopify robots.txt, add templates/robots.txt.liquid to your theme, loop over robots.default_groups and put your rule inside the loop. Then compare the output with your live file, because that template replaces Shopify’s generated robots.txt and, as of November 2026, the rules it exposes can be older than the ones served by default.
Most stores should not add it.
When should you leave your Shopify robots.txt alone?
Shopify’s template reference says the generated file works for most shops, and no theme ships with this template. The help centre page adds two warnings: Support will not help with edits, and “incorrect use of the feature can result in loss of all traffic”.
If the goal is keeping a page out of Google, this is the wrong file, because robots.txt controls crawling and not indexing. Google’s noindex documentation says the page must not be blocked by robots.txt, because a crawler that cannot fetch it never sees the noindex, and the URL can still appear in results if other pages link to it.
Shopify gives you three better tools for hiding a page: a noindex meta tag in theme.liquid, the seo.hidden metafield set to 1, and Unlisted status for products.
That leaves a short list of reasons to edit. You might have a crawl trap specific to your store, such as an app’s query parameter or the sort and filter URLs from our Liquid collection filters guide. Otherwise it is a named crawler you want gone, an extra sitemap, or per-market rules using request.host.
What does robots.txt.liquid actually replace?
All of it. The customisation guide says the template “replaces Shopify’s generated robots.txt output”, and that rendered output “doesn’t always match Shopify’s generated file exactly”. It does not say how far apart they are.
A bug report on Shopify’s developer forum from 6 October 2026 does. Dropping the unmodified docs example into a theme produced 63 rules under User-agent: * where the generated file had 45. These are the reporter’s counts, on their store.
The older set brings back Disallow: /search, Disallow: /policies/ and separate groups for Nutch, AhrefsBot, AhrefsSiteAudit, MJ12bot and Pinterest. It loses Allow: / and its 20 companion Allow rules, plus Disallow rules for /sf_*, /cart.js and the locale-prefixed checkout and account paths. So a template that changes nothing can make /search uncrawlable. A Shopify staff member replied that they were looking into updating the Liquid drop and the docs. We found no announced fix as of 4 November 2026.
We also fetched /robots.txt from 25 well-known Shopify stores that day. Ten served the generated header, the comment block with the agent instructions. Seven had no header and carried the older shape, with a Nutch group, and six of those block /search. Eight had no header and carried the current rules. From outside you cannot tell a template from a pasted file, so this does not prove what default_groups returns today. It does show that customised stores end up on either rule set, and only a comparison tells you which one you got.
How do you add a rule without fusing the lines?
Keep the loop and write your rule as literal text, flush left, on its own line:
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.user_agent.value == '*' %}
Disallow: /*?q=*
{% endif -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}This departs from Shopify’s example, which outputs {{ 'Disallow: /*?q=*' }} with whitespace stripped on both sides. Stripped that way, the line break has to come from the objects’ own string output, and the docs do not describe what that output contains.
We have not rendered either version on a real store. We rendered both in liquidjs 10.30.1 with mock objects, once assuming each object ends in a newline and once assuming each starts with one. Shopify’s example fused the custom rule to a neighbour both times. The version above kept every line separate both times, with one blank line at worst, and Shopify’s generated file already has blank lines inside its * group. A mock cannot settle what Shopify’s objects do, so treat this as a safer default and read the rendered file.
Fused lines do reach production, though we cannot see which template produced this one. A large store’s live file contained this single line on 4 November 2026:
Disallow: /*?*preview_script_id=*Disallow: /collections/*-loopCrawlers read that as one long pattern. Run through the matcher below, /x?preview_script_id=1 and /collections/x-loop both come back allowed, so neither rule does its job, and nothing in the file or the admin reports an error.
How do you remove a default rule?
Skip it inside the loop:
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{%- unless rule.directive == 'Disallow' and rule.value contains 'sort_by' -%}
{{ rule }}
{%- endunless -%}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}contains catches /collections/*sort_by* and any locale-prefixed copy in one condition. The rule object exposes directive and value, and the template only supports robots, group, rule, user_agent, sitemap and request.
The sort rule is a fair example because of what blocking did for the forum reporter. They say Google held 258 of their sorted collection URLs as “Indexed, though blocked by robots.txt”, unable to crawl them and read the canonicals, and that three weeks after they removed the rule the count was 13. That is one store’s report. Shopify’s help page calls the same Search Console message normal and usually not a concern, so look at whether those URLs are actually taking traffic before you open a crawl path.
What happens when you add a crawler group?
Custom groups go after the closing {% endfor %}, as plain text:
User-agent: discobot
Disallow: /
Sitemap: https://example.com/extra-sitemap.xmlThat is safe because Disallow: / is complete on its own. A group for a search crawler is not. Google’s robots.txt specification says each crawler follows only the group with the most specific matching user agent, and that specific groups and the * group are not combined. Add User-agent: Googlebot with one rule and Googlebot stops reading every * rule, the checkout and cart blocks included. Repeat whatever it still needs.
How do you check the rendered output?
Save your live /robots.txt before touching anything, as Shopify’s guide says. The forum reporter previewed the template with shopify theme dev and fetched /robots.txt from there.
That copy goes stale. After the template is live you can no longer see your own generated file, so later changes from Shopify are invisible. Compare against a store with no template instead, which you can recognise by the agent instructions comment at the top of its file. This script reports fused lines, rule differences in the * group, and whether specific paths are crawlable:
// Usage: node robots-check.mjs <store-url> <reference-url> [path ...]
const [store, reference, ...paths] = process.argv.slice(2);
if (!store || !reference) {
console.error('Usage: node robots-check.mjs <store-url> <reference-url> [path ...]');
process.exit(2);
}
async function fetchRobots(origin) {
const res = await fetch(new URL('/robots.txt', origin), {
headers: { 'user-agent': 'robots-check' },
signal: AbortSignal.timeout(15_000),
});
if (!res.ok) throw new Error(`${origin}: HTTP ${res.status}`);
return res.text();
}
// Collects the Allow and Disallow rules that apply to the "User-agent: *" group.
function starRules(text) {
const rules = [];
let inStar = false;
let previousWasAgent = false;
for (const raw of text.split(/\r\n|\n|\r/)) {
const match = /^([A-Za-z-]+):\s*(.*)$/.exec(raw.split('#')[0].trim());
if (!match) continue;
const field = match[1].toLowerCase();
const value = match[2];
if (field === 'user-agent') {
inStar = (previousWasAgent && inStar) || value === '*';
previousWasAgent = true;
continue;
}
previousWasAgent = false;
if (inStar && value && (field === 'allow' || field === 'disallow')) {
rules.push({ allow: field === 'allow', value });
}
}
return rules;
}
// Shop IDs differ between stores, so they are masked before comparing.
const label = (r) =>
`${r.allow ? 'Allow' : 'Disallow'}: ${r.value.replace(/^\/\d{6,}$/, '/<shop-id>')}`;
function matches(pattern, path) {
const source = pattern
.replace(/[.+?^{}()|[\]\\]/g, '\\$&')
.replace(/\*/g, '.*');
return new RegExp(`^${source}`).test(path);
}
// Google's precedence: the longest matching rule wins, and Allow wins a tie.
function isAllowed(rules, path) {
let best = null;
for (const rule of rules) {
if (!matches(rule.value, path)) continue;
const longer = !best || rule.value.length > best.value.length;
const tieToAllow = best && rule.value.length === best.value.length && rule.allow;
if (longer || tieToAllow) best = rule;
}
return best ? best.allow : true;
}
try {
const [mine, theirs] = await Promise.all([fetchRobots(store), fetchRobots(reference)]);
const fused = mine.split(/\r\n|\n|\r/).filter((l) => /\S(Allow|Disallow|Sitemap|User-agent):/.test(l));
for (const line of fused) console.log(`FUSED ${line}`);
const mineRules = starRules(mine);
const mineSet = new Set(mineRules.map(label));
const theirSet = new Set(starRules(theirs).map(label));
for (const r of mineSet) if (!theirSet.has(r)) console.log(`ONLY MINE ${r}`);
for (const r of theirSet) if (!mineSet.has(r)) console.log(`ONLY REFERENCE ${r}`);
for (const path of paths) {
console.log(`${isAllowed(mineRules, path) ? 'ALLOWED' : 'BLOCKED'} ${path}`);
}
process.exitCode = fused.length ? 1 : 0;
} catch (err) {
console.error(err.message);
process.exitCode = 2;
}node robots-check.mjs https://your-store.example https://reference-store.example \
/products/some-handle /search '/collections/all?sort_by=price'It exits 1 on a fused line, so it can sit in the same pipeline as your theme deployment workflow. It only reads the * group, and a reference store may have extras of its own, so read the differences rather than failing on them.
We ran it on Node 24.12 against live store files. Its allow and block decisions matched robots-parser 3.0.1 on 33 storefront paths across two real files, and on a small fixture covering $ and tied rules. It follows Google’s precedence, where the longest match wins and Allow wins a tie. Other crawlers may differ.
Google generally caches robots.txt for up to 24 hours, so a bad block outlives your revert. The file also lives in the theme, so expect a newly published theme without it to fall back to the generated output. That last one is our inference from the docs, so check it after any theme switch.
What we would do
If noindex, seo.hidden or Unlisted solves the problem, stop there and leave robots.txt alone. If you have a crawl trap you can name, use the loop, diff the rendered result against a template-free store, and accept the template only if every difference is one you can live with. If you want /search or /policies/ crawled, pass those paths to the script before you publish.
If the diff is unacceptable, pasting the generated file as plain text is the remaining option, and the forum reporter took it. Shopify’s guide strongly recommends the Liquid objects instead, and your rules then stay frozen at the paste date, so run the script on a schedule and re-paste when Shopify’s file moves. Either way, deleting robots.txt.liquid restores the default.
If you would rather have someone else make and check this change, it sits within our Shopify development work.