LLM Crawler for WordPress

Make Your WordPress Content AI-Ready

Download Plugin
LLM Crawler is a free WordPress plugin that creates structured, AI-friendly endpoints for your website content. As AI tools increasingly need to understand and interact with web content, this plugin ensures your WordPress site can be efficiently and accurately crawled by Large Language Models (LLMs) and AI services.

🤖 AI-Optimized Content Delivery

Transform your WordPress content into clean, structured data that AI systems can easily understand. The plugin provides multiple REST API endpoints specifically designed for LLM consumption, including manifest, delta updates, and content endpoints.

Download Plugin
A screenshot of the dashboard of the plugin, with options like "require token", "shared token", and "IP allowlist"
The "general" tab of the plugin, where users can edit how SEO no index is respected, how posts are included, and other useful features

🔒 Complete Control Over Your Content

  • Respect existing SEO settings - Automatically honors Yoast and RankMath noindex settings
  • Per-post exclusion - Choose exactly which content to share
  • Token-based access control - Optionally require authentication for crawling
  • IP allowlisting - Restrict access to specific services
  • Rate limiting - Prevent abuse with configurable request limits
Download Plugin

Universal Discovery

Spec-compliant /llms.txt + /llms-full.txt (llmstxt.org format)
MCP server + agents.json + ai-crawl.json
AI crawler detection with analytics dashboard
Schema.org JSON-LD + robots.txt Allow rules for AI bots

A list of endpoints that can be set and edited within the plugin

📊 Advanced Features

  • MCP server with 6 built-in tools for AI agent integration
  • agents.json machine-readable API description
  • AI crawler analytics dashboard with per-bot visit tracking
  • Schema.org JSON-LD and content optimization meta tags
  • Ed25519 signed updates with SHA-384 hash verification
  • JSONL bulk export + webhook notifications
  • Smart caching, rate limiting, ACF support, and more
Download Plugin
The Export and Webhooks tab of the plugin, where JSON export can be provided and webhook url set

🚀 Easy Setup

The plugin comes with sensible defaults that work out of the box:

  • All recommended settings enabled by default
  • One-click "Reset to Defaults" option
  • Clean admin interface with organized settings tabs
  • Copy-ready endpoint URLs for easy sharing

How It Works

Once installed, LLM Crawler automatically creates structured endpoints that AI services can use to understand your content:

  1. Manifest Endpoint - Lists all available content with metadata
  2. Content Endpoint - Delivers clean, structured content by ID
  3. Delta Endpoint - Shows only content that's changed since a specific date
  4. Export Endpoint - Provides bulk JSONL export for data processing

Privacy & Control

Your content remains under your complete control. The plugin respects all existing privacy settings and adds additional layers of protection:

  • Content marked as noindex is automatically excluded
  • Password-protected posts are never exposed
  • Draft and private content stays private
  • Optional token authentication adds security
  • Telemetry is fully opt-in (disabled by default)

Perfect For

  • Publishers wanting to make content available to AI search and chat services
  • Businesses looking to integrate with AI-powered tools
  • Educational sites sharing knowledge with LLM training systems
  • Documentation sites providing structured technical content
  • News organizations distributing content to AI aggregators

Always Free

LLM Crawler is completely free and always will be. There are no premium tiers, no locked features, and no hidden costs. We believe in making the web more accessible to AI while keeping control in the hands of content creators.

Technical Specifications

  • Requires: WordPress 5.9 or higher
  • PHP: 7.4 or higher
  • License: Free/Open Source
  • Support: Community-driven
  • Updates: Automatic updates available

Get Started

Simply install the plugin from your WordPress admin panel or download it directly. The plugin activates with optimal settings by default, but everything is customizable through an intuitive settings interface.

LLM Crawler is developed and maintained by Explore Digital https://www.exploredigital.com. We're committed to helping WordPress sites participate in the AI revolution while maintaining full control over their content.

Download Plugin