---
title: "Home"
url: "https://llm-crawler.exploredigital.network/"
post_type: "page"
date_published: "2025-10-02T22:06:22+00:00"
date_modified: "2026-10-24T15:55:43+00:00"
---

# Home

LLM Crawler for WordPress

Make Your WordPress Content AI-Ready

Download Plugin

LLM Crawler is a free WordPress plugin that creates structured, AI-friendly endpoints for your website content. As AI tools increasingly need to understand and interact with web content, this plugin ensures your WordPress site can be efficiently and accurately crawled by Large Language Models (LLMs) and AI services.

🤖 AI-Optimized Content Delivery

Transform your WordPress content into clean, structured data that AI systems can easily understand. The plugin provides multiple REST API endpoints specifically designed for LLM consumption, including manifest, delta updates, and content endpoints.

Download Plugin

🔒 Complete Control Over Your Content

Respect existing SEO settings - Automatically honors Yoast and RankMath noindex settings

Per-post exclusion - Choose exactly which content to share

Token-based access control - Optionally require authentication for crawling (Authorization: Bearer)

IP allowlisting - Restrict access to specific services, Cloudflare-aware (the real visitor address is used behind Cloudflare)

Rate limiting - Prevent abuse with configurable request limits

Download Plugin

Universal Discovery

Spec-compliant /llms.txt + /llms-full.txt (llmstxt.org format) + per-page .md twins
MCP server + Server Card + agents.json + ai-crawl.json + RFC 9727 /.well-known/api-catalog
AI crawler detection with per-crawler analytics + CSV/PDF reports
Schema.org JSON-LD + WooCommerce agent shopping (product feed, MCP tools, /.well-known/ucp)

📊 Advanced Features

Per-page Markdown twins (.md + Accept negotiation) so agents read clean content

Modern MCP server (2026-07-28, with 2025-11-25 still supported) with an MCP Server Card, structured output and read-only tool annotations

WooCommerce mode - read-only product feed, product MCP tools and /.well-known/ucp

Per-crawler analytics with CSV and printable PDF reports

agents.md agent-instructions + agents.json API description + RFC 9727 API catalog

IndexNow pings on publish, update and removal + WordPress Abilities (WordPress 6.9+) for core-native agent access

Ed25519 signed updates with SHA-384 hash verification

JSONL bulk export, webhooks, smart caching, rate limiting and ACF support

Download Plugin

🚀 Easy Setup

The plugin comes with sensible defaults that work out of the box:

All recommended settings enabled by default

One-click "Reset to Defaults" option

Clean admin interface with organized settings tabs

Copy-ready endpoint URLs for easy sharing

How It Works

Once installed, LLM Crawler automatically creates structured endpoints that AI services can use to understand your content:

Manifest Endpoint - Lists all available content with metadata

Content Endpoint - Delivers clean, structured content by ID

Delta Endpoint - Shows only content that's changed since a specific date

Export Endpoint - Provides bulk JSONL export for data processing

MCP Server Card and API Catalog - Let agents discover the MCP server and every endpoint (/.well-known/api-catalog)

Privacy & Control

Your content remains under your complete control. The plugin respects all existing privacy settings and adds additional layers of protection:

Content marked as noindex is automatically excluded

Pages marked nosnippet, noarchive or max-snippet:0 are listed with title and URL only; a site set to discourage search engines is hidden from agents entirely

Password-protected posts are never exposed

Draft and private content stays private

Optional token authentication adds security

Telemetry is fully opt-in (disabled by default)

Perfect For

Publishers wanting to make content available to AI search and chat services

Businesses looking to integrate with AI-powered tools

Educational sites sharing knowledge with LLM training systems

Documentation sites providing structured technical content

News organizations distributing content to AI aggregators

Always Free

LLM Crawler is completely free and always will be. There are no premium tiers, no locked features, and no hidden costs. We believe in making the web more accessible to AI while keeping control in the hands of content creators.

Technical Specifications

Requires: WordPress 5.9 or higher

PHP: 8.2 or higher

License: Free/Open Source

Support: Community-driven

Updates: Automatic updates available

Get Started

Simply install the plugin from your WordPress admin panel or download it directly. The plugin activates with optimal settings by default, but everything is customizable through an intuitive settings interface.

LLM Crawler is developed and maintained by Explore Digital https://www.exploredigital.com. We're committed to helping WordPress sites participate in the AI revolution while maintaining full control over their content.

Download Plugin