Free GEO tool

robots.txt Generator with AI Bots

Choose which search and AI crawlers may visit, and get a ready robots.txt.

robots.txt

# robots.txt — generated by THB SEO
User-agent: *
Disallow: /admin/
Disallow: /cart/

# Allowed search & AI crawlers
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://www.yourbusiness.com/sitemap.xml

Upload as robots.txt in your site root (public_html). If you use Cloudflare, also check its AI crawler setting.

Short answer

What does the robots.txt Generator with AI Bots do?

The robots.txt Generator lets you choose, bot by bot, which search engines and AI crawlers may access your site, then outputs a ready robots.txt file. It separates answer bots that cite you from training bots that only collect data.

How do I use it?

Enter your domain.
Switch each crawler on or off.
Add folders to block, if any.
Upload the result as /robots.txt at your site root.

Frequently asked questions

Does robots.txt stop a page from appearing in Google?

Not reliably. It stops crawling, not indexing. Use a noindex tag to keep a page out of results.

Is robots.txt enough to allow AI bots?

Not always. Firewalls and CDNs such as Cloudflare can block AI crawlers before robots.txt is read — check those settings too.

Analyze your website — free

Enter your domain for an instant report on what’s holding it back. Want us to fix it? Call or message us any day.

THB — The House of Business, Level 10, Shyamoli Square, Dhaka 1207, Bangladesh

Open daily 9:00 AM – 7:00 PM

Mobile: +8801805-564599

Telephone: +8809639-321355

WhatsApp (direct): +8801805-564599

WhatsApp (information): +8801550-721721