17.5% of new .com websites serve an llms.txt file, and most of those files are Shopify, GoDaddy and Wix defaults
We crawled 4,472 live sites on .com domains registered 9 to 22 September 2026 and 1,076 established sites: new sites serve llms.txt more often (17.5% against 12.5%), but without platform defaults their rate is 5.6%, half the established 11.2%.
Summary
Of 4,472 live websites on .com domains registered between 9 and 22 September 2026, 17.5% served an llms.txt file when we crawled them on 23 September 2026, against 12.5% of 1,076 established sites sampled from the Tranco top 100,000. The gap comes from site platforms: 68.2% of the new sites' 782 llms.txt files are defaults written by Shopify, GoDaddy Website Builder, Wix, Hostinger or Base44. Leaving those defaults out, 5.6% of new sites publish llms.txt (n = 4,472), half the established rate of 11.2% (n = 1,076). Other files for AI agents are rare in both groups: a Markdown copy of the home page is on 0.5% of new and 0.7% of established sites, and /.well-known/api-catalog on 0.2% and 0.3%. New sites also say little to AI crawlers in robots.txt: 1.8% block at least one by name, against 10.1% of established sites.
Findings
- 17.5% of 4,472 live new .com sites served an llms.txt file on 23 September 2026, against 12.5% of 1,076 established sites from the Tranco top 100,000.
- 68.2% of the 782 llms.txt files on new sites are defaults written by the site platform: Shopify wrote 203, GoDaddy Website Builder 199, Wix 87, and Hostinger or Base44 44.
- Without platform defaults, 5.6% of new sites publish llms.txt (n = 4,472), half the established rate of 11.2% (n = 1,076).
- Other files for AI agents are rare: llms-full.txt is on 5.2% of new sites (n = 4,472), mostly Shopify stores, a Markdown copy of the home page on 0.5%, and /.well-known/api-catalog on 0.2%.
- 1.8% of new sites block at least one AI crawler by name in robots.txt (n = 4,472), against 10.1% of established sites (n = 1,076).
- 44.3% of new domains that served an HTML home page were parked, for sale or placeholders (3,635 of 8,199), and GoDaddy served an llms.txt on 368 of the excluded pages.
New sites serve llms.txt more often
Source: Strandline Research crawl of 23 September 2026. New sites: 4,472 live sites on .com domains registered 9 to 22 September 2026. Established sites: 1,076 live sites from Tranco list Y8YYG, top 100,000. Platform defaults: files written by Shopify, GoDaddy Website Builder, Wix, Hostinger or Base44.
We fetched six fixed URLs from each site on 23 September 2026. A valid llms.txt came back from 782 of 4,472 new .com sites (17.5%) and from 135 of 1,076 established sites (12.5%). The 95% intervals do not overlap: 16.4% to 18.6% for new sites and 10.7% to 14.7% for established sites.
The established sample includes every domain ending found in the Tranco top 100,000, while the new sample is .com only. Among the 508 established .com sites, 14.8% serve llms.txt, with an interval of 11.9% to 18.1%. The new-site figure sits inside that range, so on a like-for-like .com basis the two groups are close.
The files differ more than the rates. A new site's llms.txt has a median size of 2,857 bytes and 4 links (n = 782). An established site's has 7,872 bytes and 34 links (n = 135). 34.4% of new-site files (269 of 782) link to one page or none, against 9.6% of established files (13 of 135).
Platform defaults write most of the new files
Source: Strandline Research. Base: the 782 llms.txt files served by 4,472 new .com sites. Attribution rules are in the method section.
Three platforms account for 62.5% of the new sites' llms.txt files (489 of 782).
- Shopify serves an "Agent Instructions" llms.txt that tells shopping agents how to browse and buy from the store, with a section on the Universal Commerce Protocol. 205 of the 231 Shopify stores among the new sites have one (88.7%).
- GoDaddy Website Builder writes a short file with the site name, tagline and a list of pages. 199 of 215 GoDaddy builder sites have one (92.6%).
- Wix publishes a file that describes AI agent access to the site through the Model Context Protocol. 85 of 125 Wix sites have one (68.0%).
Files from Hostinger's builder and Base44 add 44 more (5.6% of 782). The file is on 21 of 55 Hostinger builder sites and 25 of 66 Base44 sites. Together, platform defaults are 533 of the 782 files (68.2%).
Sites without such a default rarely add the file. It is on 6.0% of 602 WordPress sites, 8.6% of 452 Next.js sites, 1 of 65 Squarespace sites and none of 55 sites built with Lovable.
Established sites get their files another way. Platform defaults are 14 of their 135 files (10.4%). WordPress SEO plugins (Rank Math, Yoast and All in One SEO) wrote 31 (23.0%), and 90 (66.7%) are hand-written or come from other tools. FamilySearch, Picsart and Awwwards are among the established sites in the sample that serve one.
With platform defaults set aside, the order flips. 5.6% of new sites publish llms.txt (249 of 4,472), against 11.2% of established sites (121 of 1,076) and 13.4% of established .com sites (68 of 508).
Other files for agents are rare
llms-full.txt is on 5.2% of new sites (232 of 4,472) and 2.6% of established sites (28 of 1,076). Shopify serves it next to its default llms.txt: 204 of the 232 new-site files are on Shopify stores, as are 14 of the 28 established ones.
Markdown copies of the home page are close to absent. 10 new sites (0.2% of 4,472) and 3 established sites (0.3% of 1,076) serve /index.md. 15 new sites (0.3%) and 6 established sites (0.6%) point to a Markdown version from the home page with a rel="alternate" link. Either signal appears on 0.5% of new sites (21) and 0.7% of established sites (8).
/.well-known/api-catalog, the RFC 9727 file that lists a site's APIs, is on 7 new sites (0.2% of 4,472) and 3 established sites (0.3% of 1,076). 5 of the 7 new-site catalogs point to a Universal Commerce Protocol endpoint.
Counting any of these files, 17.6% of new sites (786 of 4,472) and 12.8% of established sites (138 of 1,076) publish at least one. llms.txt accounts for nearly all of it.
New sites rarely address AI crawlers in robots.txt
Source: Strandline Research crawl of 23 September 2026. Base: 4,472 new .com sites and 1,076 established sites. A block is a robots.txt group that names the crawler and disallows the whole site. Blocks that come only from a rule for all crawlers are in robots-ai-rules.csv.
65.5% of new sites serve a valid robots.txt (2,929 of 4,472), against 81.4% of established sites (876 of 1,076). 5.5% of new sites name at least one AI crawler in it (244), against 15.1% of established sites (163).
The two groups use those rules in opposite ways. Established sites that name AI crawlers mostly block them: 109 of 163 block at least one. New sites mostly let them in: 163 of 244 name AI crawlers without blocking any. All 65 Squarespace sites in the new sample name AI crawlers, because Squarespace's default robots.txt lists them in the same group as every other crawler, and 64 of them allow them.
By crawler, GPTBot is blocked by name on 1.5% of new sites (65) and 7.0% of established sites (75). CCBot is at 1.2% (53) and 6.2% (67), ClaudeBot at 1.2% (52) and 4.6% (50), and OAI-SearchBot, which fetches pages for ChatGPT search, at 0.2% (7) and 1.9% (20).
Cloudflare's managed robots.txt, which disallows AI training, search and agent crawlers, appears on 12 new sites (0.3% of 4,472) and 4 established sites (0.4% of 1,076). Content-Signal lines, which state whether content may be used for search, AI input or AI training, are on 0.7% of new sites (33) and 1.5% of established sites (16). A blanket Disallow: / for all crawlers is on 2.7% of new sites (122) and 1.7% of established sites (18).
Almost half of new domains with a page are parked or placeholders
| Step | New .com domains | Established (Tranco top 100k) |
|---|---|---|
| Drawn at random | 15,000 | 2,000 |
| Answered the home page request | 10,777 | 1,500 |
| Served an HTML home page with status 200 on their own domain | 8,199 | 1,112 |
| Parked, for sale or placeholder | 3,635 | 28 |
| Bot-check or redirect page | 92 | 8 |
| Analyzed | 4,472 | 1,076 |
Of the 8,199 new domains that served an HTML home page, 2,366 were parked, 142 were for sale and 1,127 were placeholders such as coming-soon pages, server default pages and fresh CMS installs (44.3% in total). exclusions.csv lists how many pages each rule caught.
Parking and placeholder pages carry agent files too. 376 of the 3,727 excluded pages (10.1%) served a valid llms.txt. 210 were GoDaddy for-sale landers, whose llms.txt says the domain is listed on GoDaddy's aftermarket, and 158 were GoDaddy "Launching Soon" pages. A crawl that skips the parking filter would count these files as adoption.
Method
New-site sample. WhoisDS free daily files of newly registered domains, dated 9 to 22 September 2026 (14 files). Each free file lists at most 70,000 domains across all extensions, 980,000 lines in all, and WhoisDS does not say how the capped list is chosen. The 11 September file held no .com domains and the 19 September file about half the usual number (nrd-files.csv). We kept every .com domain, 309,444 unique names, and drew 15,000 at random.
Established sample. 2,000 domains drawn at random from the top 100,000 of Tranco list Y8YYG, created 22 September 2026 from data for 24 August to 22 September 2026. 944 of the 2,000 are .com.
Crawl. 23 September 2026, 12:13 to 14:31 US Eastern, both samples shuffled together. The crawler identified itself as StrandlineResearch/1.0 (+https://strandlineresearch.com/methods) and requested at most six URLs per domain, one at a time: /, /robots.txt, /llms.txt, /llms-full.txt, /index.md and /.well-known/api-catalog. It tried https:// first, then https://www. and http:// only when no HTTP response came back. The five file paths were requested only when the home page returned 200 on the same domain. At most 60 domains were in progress at once, with an 8-second connect and read timeout, a 10-second limit per request and up to 5 redirects. No JavaScript was run.
Live HTML site. The home page returned 200 with HTML, after redirects, on the domain itself or a subdomain such as www. Domains that redirected to another domain (988 new, 146 established) or answered with another status (1,548 new, 231 established) are outside the base.
Parked, for-sale and placeholder pages. Rules applied to the home page HTML, in order. Parked: a redirect to a ww-numbered subdomain, a Google AdSense for Domains script, a parking lander script, a parking provider's host or script, or wording such as "this domain has been registered", "parked free" or "related searches" on a short page. For sale: wording such as "buy this domain", "make an offer" or "is for sale" in the title or on a short page, or a link to a domain marketplace on a short page. Placeholder: coming-soon or construction wording in the title or on a short page (several languages), server and control-panel default pages, error pages served with status 200, fresh WordPress installs, and blank pages. Bot-check pages and pages that only redirect to another domain by script were also excluded. We then grouped the remaining new-site pages by identical text, read every group of three or more, and added a rule for each parking, holding or placeholder template found. Pages built with JavaScript that show little text without it stay in the base. method.txt lists every rule with its word limits.
What counts as present. Status 200 after redirects, on the same domain, not HTML, not empty, and not the same bytes the site returned for robots.txt or api-catalog. llms.txt and llms-full.txt must start with # or contain a Markdown link, and must not be code, JSON, a robots.txt file or a not-found message. /index.md must have a Markdown content type or be clearly Markdown (a leading heading or front matter, or headings and links). A Markdown alternate link is a <link rel="alternate" type="text/markdown"> tag or Link header on the home page. /.well-known/api-catalog must be JSON with a linkset array. robots.txt must contain at least one User-agent, Allow, Disallow or Sitemap line.
robots.txt. Parsed by the RFC 9309 rules: the longest matching rule wins and Allow wins a tie. A crawler is blocked when a group that names it disallows /. We checked 17 AI crawler names, listed in method.txt. Rules for all crawlers (User-agent: *) are reported separately.
Platforms and sources. Platforms are read from home page markers such as generator tags and asset hosts. Shopify and Wix files are identified by their own text, GoDaddy, Hostinger and Base44 files by the builder markers on the site that serves them, and plugin files by the plugin name in the file.
Method note. The frame is one 14-day window of .com registrations from the WhoisDS free files, crawled once. Only /index.md was requested.
Data
The CSV files below hold every number in this report. Per-domain results are not published. The files are free to reuse with credit to Strandline Research.
- Samples and exclusions One row per sample: source list, date window, domains drawn, domains that answered, home pages that were not 200, redirected to another domain or were not HTML, live HTML sites, sites excluded as parked, for sale or placeholder (with the split), other exclusions, and sites analyzed.
- Newly registered domain files used One row per WhoisDS daily file: file date, domains listed, .com domains, and domains drawn into the new-site sample.
- Files for AI agents by sample One row per file type and sample: sites analyzed, sites where the file is present, share, and Wilson 95% interval. Covers llms.txt, llms.txt without platform defaults, llms-full.txt, index.md, Markdown alternate links, /.well-known/api-catalog and robots.txt.
- robots.txt rules for AI crawlers One row per AI crawler and sample: sites analyzed, sites whose robots.txt blocks the crawler by name, sites that name it and allow the root, sites that block it only through a rule for all crawlers, and shares.
- robots.txt summary by sample Sites with a valid robots.txt, sites that name any AI crawler, sites that block one by name, Cloudflare managed robots.txt files, Content-Signal and Content-Usage lines, and blanket blocks of all crawlers.
- Where llms.txt files come from The llms.txt files in each sample split by source: Shopify, GoDaddy Website Builder and Wix defaults, other site builders, WordPress SEO plugins, and other or not attributable.
- llms.txt with and without platform defaults Per sample: sites analyzed, sites with llms.txt, how many of those files are platform defaults, and the share of sites with an llms.txt that is not a platform default.
- Size and links of llms.txt files Per sample: number of llms.txt files, median size in bytes, median number of Markdown links, files with one link or none, and files that start with a heading.
- Site platforms and their files Per sample and platform (detected from home page markers): sites on the platform, share of the sample, sites with llms.txt and llms-full.txt, and sites whose robots.txt names or blocks an AI crawler.
- Exclusion rules and counts Every parked, for-sale, placeholder, redirect and bot-check rule, with the number of live HTML sites it excluded in each sample.
- Excluded pages that still serve llms.txt Per sample: excluded sites, how many of them serve a valid llms.txt, and how many of those are GoDaddy for-sale landers or GoDaddy launching-soon pages.
- API catalog files Per sample: sites serving a valid /.well-known/api-catalog linkset, and how many of those link to a Universal Commerce Protocol endpoint.
- Chart data: files for AI agents The shares plotted in Exhibit 1, taken from ai-files.csv.
- Chart data: llms.txt sources on new sites The shares plotted in Exhibit 2, taken from llms-txt-sources.csv.
- Chart data: robots.txt blocks by crawler The shares plotted in Exhibit 3, taken from robots-ai-rules.csv.
How to cite: Strandline Research. "17.5% of new .com websites serve an llms.txt file, and most of those files are Shopify, GoDaddy and Wix defaults." 23 September 2026. https://strandlineresearch.com/reports/llms-txt-new-sites