# LLM Crawler Plugin > Giving the power of AI the ability to crawl your website ## Home URL: https://llm-crawler.exploredigital.network/ LLM Crawler for WordPress Make Your WordPress Content AI-Ready Download Plugin LLM Crawler is a free WordPress plugin that creates structured, AI-friendly endpoints for your website content. As AI tools increasingly need to understand and interact with web content, this plugin ensures your WordPress site can be efficiently and accurately crawled by Large Language Models (LLMs) and AI services. ๐Ÿค– AI-Optimized Content Delivery Transform your WordPress content into clean, structured data that AI systems can easily understand. The plugin provides multiple REST API endpoints specifically designed for LLM consumption, including manifest, delta updates, and content endpoints. Download Plugin ๐Ÿ”’ Complete Control Over Your Content Respect existing SEO settings - Automatically honors Yoast and RankMath noindex settings Per-post exclusion - Choose exactly which content to share Token-based access control - Optionally require authentication for crawling IP allowlisting - Restrict access to specific services Rate limiting - Prevent abuse with configurable request limits Download Plugin Universal Discovery Spec-compliant /llms.txt + /llms-full.txt (llmstxt.org format) + per-page .md twinsMCP server + agents.json + ai-crawl.jsonAI crawler detection with per-crawler analytics + CSV/PDF reportsSchema.org JSON-LD + WooCommerce agent shopping (product feed, MCP tools, /.well-known/ucp) ๐Ÿ“Š Advanced Features Per-page Markdown twins (.md + Accept negotiation) so agents read clean content Modern MCP server (2025-11-25) with structured output and read-only tool annotations WooCommerce mode - read-only product feed, product MCP tools and /.well-known/ucp Per-crawler analytics with CSV and printable PDF reports agents.md agent-instructions + agents.json API description Ed25519 signed updates with SHA-384 hash verification JSONL bulk export, webhooks, smart caching, rate limiting and ACF support Download Plugin ๐Ÿš€ Easy Setup The plugin comes with sensible defaults that work out of the box: All recommended settings enabled by default One-click "Reset to Defaults" option Clean admin interface with organized settings tabs Copy-ready endpoint URLs for easy sharing How It Works Once installed, LLM Crawler automatically creates structured endpoints that AI services can use to understand your content: Manifest Endpoint - Lists all available content with metadata Content Endpoint - Delivers clean, structured content by ID Delta Endpoint - Shows only content that's changed since a specific date Export Endpoint - Provides bulk JSONL export for data processing Privacy & Control Your content remains under your complete control. The plugin respects all existing privacy settings and adds additional layers of protection: Content marked as noindex is automatically excluded Password-protected posts are never exposed Draft and private content stays private Optional token authentication adds security Telemetry is fully opt-in (disabled by default) Perfect For Publishers wanting to make content available to AI search and chat services Businesses looking to integrate with AI-powered tools Educational sites sharing knowledge with LLM training systems Documentation sites providing structured technical content News organizations distributing content to AI aggregators Always Free LLM Crawler is completely free and always will be. There are no premium tiers, no locked features, and no hidden costs. We believe in making the web more accessible to AI while keeping control in the hands of content creators. Technical Specifications Requires: WordPress 5.9 or higher PHP: 7.4 or higher License: Free/Open Source Support: Community-driven Updates: Automatic updates available Get Started Simply install the plugin from your WordPress admin panel or download it directly. The plugin activates with optimal settings by default, but everything is customizable through an intuitive settings interface. LLM Crawler is developed and maintained by Explore Digital https://www.exploredigital.com. We're committed to helping WordPress sites participate in the AI revolution while maintaining full control over their content. Download Plugin --- ## Change Log URL: https://llm-crawler.exploredigital.network/changelog/ LLM Crawler for WordPress Changelog & Version History Version 0.8.1 Latest Patch June 2026 Stability Fix Fixed a critical issue where saving the plugin settings on 0.8.0 could exhaust PHP memory and return an HTTP 500 error - settings now save reliably The internal version marker now round trips correctly through settings sanitization, preventing a repeated re-save loop Hardened discovery route registration so it never runs before WordPress has initialized its rewrite system Version 0.8.0 Major Release June 2026 Agent-Ready Content (Markdown twins) Per-page .md twins - append .md to any URL, or send Accept: text/markdown, for clean Markdown with YAML frontmatter New /agents.md agent-instructions file llms.txt link lists now point straight at the Markdown twins Modernized MCP Server Protocol-version negotiation (speaks up to 2025-11-25) Structured tool output (structuredContent + output schemas) and read-only tool annotations Streamable-HTTP conformance, stateless, with Origin checks, token gate and rate limiting WooCommerce Commerce Mode Read-only product feed at /llm-products.json (price, availability, SKU, GTIN, variants) search_products and get_product MCP tools /.well-known/ucp capability profile + enriched Product schema Strictly read-only - agents never complete a purchase without explicit buyer consent Analytics and Reports Crawler hits now tracked per page and per crawler "See more" to expand beyond the top 10 pages Expand any page to see its hit rate broken down by crawler Export the data as CSV or a printable PDF report Crawler Detection Refreshed, filterable AI user-agent list (Claude-User, Meta-ExternalAgent, Amazonbot, DuckAssistBot, Perplexity-User, and more) RFC 9421 Signature-Agent detection Expanded robots.txt Allow rules for current AI crawlers Version 0.7.0 Major Release February 2026 New Standards Support Spec-compliant /llms.txt following llmstxt.org format (Markdown with H1/H2/link lists) New /llms-full.txt endpoint with inline full content New /.well-known/agents.json for machine-readable API descriptions MCP (Model Context Protocol) server at /wp-json/llm/v1/mcp with 6 tools AI-specific XML sitemap provider for WordPress 5.5+ sitemaps AI Crawler Intelligence Automatic detection of 12+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) X-LLM-Endpoints header sent to detected AI bots Enhanced robots.txt with per-bot Allow rules for all AI crawlers New Analytics dashboard tracking bot visits, top pages, and last-seen timestamps Content Optimization Schema.org JSON-LD output on singular posts (Article/WebPage) LLM-specific meta tags: llm-content-type, llm-language, llm-last-modified Markdown alternate link for each post via rel="alternate" AIOSEO compatibility added (noindex detection) Security & Privacy Telemetry changed to opt-in only (GDPR compliance) No data sent during activation โ€” requires explicit user consent SHA-384 package hash verification on updates Ed25519 package signing infrastructure (WordPress 5.2+ native) Site URL hashed (SHA-256) in update checks for privacy Explicit SSL verification on all remote requests Admin UI New Discovery tab with toggles for llms.txt, llms-full.txt, agents.json, and MCP New Analytics tab with bot visit charts and top crawled pages Improved telemetry description listing exactly what data is sent Configurable per-type limit for llms.txt entries Version 0.6.8 Minor Release October 2025 New Features Added "Reset to Defaults" button for quick configuration Confirmation dialog lists all settings that will be changed Preserves important data like Client ID when resetting Improvements Better handling of settings persistence Improved user experience with clear reset functionality Version 0.6.7 Patch October 2025 Bug Fixes Fixed default checkboxes not showing as checked on General tab Forced critical defaults on plugin upgrade Completely removed Update Server field from UI UI Improvements Discovery URLs now display on individual lines Added copy button next to each endpoint URL Improved Discovery Endpoints section layout JavaScript copy functionality with visual feedback System Updates Update server URL hardcoded to https://llm-crawler.exploredigital.network Improved llms.txt file writing with better error handling Dynamic serving of llms.txt if file creation fails Version 0.6.6 Patch October 2025 Settings Updates Updated default settings for optimal LLM crawling Cache TTL default changed from 5 to 60 minutes Auto-update preference enabled by default All discovery options enabled by default Improvements Better activation process for new and existing installs Simplified file writing logic for llms.txt Improved error messages Version 0.6.5 Minor Release October 2025 New Features Added Settings link on plugins page for quick access Added "About this plugin" link pointing to update server Plugin action links for better navigation Default Settings Respect SEO noindex checked by default Public discovery enabled by default All discovery methods activated by default Telemetry and auto-updates enabled by default Version 0.6.4 Major Fix October 2025 Critical Fixes Fixed duplicate Client ID systems causing multiple registrations Consolidated Client ID management to single system Fixed Client ID disappearing when toggling auto-update Removed duplicate settings storage Technical Improvements Update client now uses main plugin's Client ID via dependency injection Improved settings persistence Better integration between plugin components Added public get_client_id() method Versions 0.6.0 - 0.6.3 Initial Releases September - October 2025 Core Features Initial release of LLM Crawler plugin REST API endpoints for manifest, content, and delta updates Token-based authentication system IP allowlisting and rate limiting ACF field support with configurable allowlists JSONL export functionality Webhook notifications for content updates Universal discovery via /.well-known and robots.txt Self-update mechanism from dedicated update server Per-post exclusion capabilities SEO plugin integration (Yoast, RankMath) Smart caching system --- ## Contact URL: https://llm-crawler.exploredigital.network/contact/ Contact Us If you are seeing an issue, wish to report a problem or have a feature request please reach out to us. ๏ƒ  General Inquiries contact@llm-crawler.exploredigital.network ๏กน Support support@llm-crawler.exploredigital.network Get In Touch Name Email Address Message Website URL Submit Contact Us If you are seeing an issue, wish to report a problem or have a feature request please reach out to us. Download Plugin --- ## Terms and Conditions URL: https://llm-crawler.exploredigital.network/terms-and-conditions/ Terms and Conditions LLM Crawler WordPress Plugin Effective Date: February 2026 1. Acceptance of Terms By downloading, installing, or using the LLM Crawler WordPress Plugin ("Plugin"), you agree to be bound by these Terms and Conditions. If you do not agree to these terms, do not use the Plugin. The Plugin is provided by Explore Digital ("we," "us," or "our") as a free service to the WordPress community. 2. License Grant The LLM Crawler Plugin is provided as free, open-source software. You are granted a non-exclusive, worldwide, royalty-free license to: Use the Plugin on unlimited WordPress websites Modify the Plugin for your own use Distribute the Plugin in accordance with its open-source license This license is subject to your compliance with these Terms and Conditions. 3. No Warranty Disclaimer THE PLUGIN IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED. We specifically disclaim all warranties, including but not limited to: Implied warranties of merchantability Fitness for a particular purpose Non-infringement Compatibility with your WordPress installation Uninterrupted or error-free operation We do not warrant that the Plugin will meet your requirements or that its operation will be uninterrupted, secure, or error-free. Any use of the Plugin is at your own risk. 4. Automatic Updates and Version Checking 4.1 Update Mechanism The Plugin includes an automatic update mechanism that periodically checks our update server at https://llm-crawler.exploredigital.network for new versions. This ensures you have access to: Security patches and bug fixes New features and improvements Compatibility updates for new WordPress versions 4.2 Data Transmitted During Updates When checking for updates, the Plugin transmits the following information to our update server: Current plugin version WordPress version PHP version Site URL A unique, anonymous Client ID 4.3 Telemetry and Analytics Telemetry is disabled by default and must be explicitly enabled by the site administrator. No data is sent during plugin activation. If you choose to enable telemetry, the Plugin sends periodic heartbeat signals containing: Anonymous Client ID (UUID) Plugin version, WordPress version, PHP version Hashed site URL (SHA-256, not plaintext) This data helps us understand plugin usage and improve the service. You can disable telemetry at any time through the plugin settings. 4.4 Your Control You maintain full control over updates and telemetry: Automatic updates can be enabled or disabled in plugin settings Update checks can be completely disabled Telemetry is opt-in (disabled by default) and can be disabled at any time 4.5 Package Integrity Starting with version 0.7.0, all update packages are signed using Ed25519 digital signatures and verified with SHA-384 hashes before installation. This ensures that updates have not been tampered with during transit. 5. Limitation of Liability IN NO EVENT SHALL EXPLORE DIGITAL BE LIABLE FOR ANY DAMAGES arising from the use or inability to use the Plugin, including but not limited to: Direct, indirect, incidental, or consequential damages Loss of profits, data, or business opportunities Service interruptions or website downtime Security breaches or data exposure This limitation applies even if we have been advised of the possibility of such damages. 6. User Responsibilities By using the Plugin, you agree to: Maintain Backups: Keep regular backups of your WordPress site and database Test Updates: Test plugin updates in a staging environment before deploying to production Security: Properly configure access controls and authentication if exposing content to AI services Compliance: Ensure your use complies with applicable laws and regulations Content Rights: Only expose content you have the right to share 7. Privacy and Data Collection 7.1 What We Collect We collect minimal, anonymous data necessary for: Providing plugin updates Understanding usage patterns (if telemetry is enabled) Improving the plugin service 7.2 What We Don't Collect We do NOT collect: Your website content or posts User personal information Passwords or authentication tokens Financial or payment information 7.3 Data Storage Any collected data is stored securely and is not sold or shared with third parties. 8. Indemnification You agree to indemnify, defend, and hold harmless Explore Digital and its affiliates from any claims, damages, losses, liabilities, and expenses (including legal fees) arising from: Your use of the Plugin Your violation of these Terms Your violation of any rights of a third party Content you expose through the Plugin 9. Modifications to Terms We reserve the right to modify these Terms and Conditions at any time. Changes will be effective immediately upon posting to our website. Your continued use of the Plugin after any modifications indicates your acceptance of the updated terms. We recommend checking these terms periodically for updates. 10. Termination These Terms remain in effect until terminated. You may terminate by uninstalling and deleting the Plugin. We reserve the right to terminate or suspend access to our update services at any time without notice for any reason. Upon termination, all provisions which by their nature should survive will continue in effect. 11. Governing Law These Terms and Conditions shall be governed by and construed in accordance with the laws of the jurisdiction in which Explore Digital operates, without regard to conflict of law principles. 12. Severability If any provision of these Terms is found to be invalid or unenforceable, the remaining provisions shall continue in full force and effect. Contact Information For questions about these Terms and Conditions or the LLM Crawler Plugin: Explore Digital Email: support@exploredigital.network Website: https://llm-crawler.exploredigital.network By using the LLM Crawler Plugin, you acknowledge that you have read, understood, and agree to be bound by these Terms and Conditions. ---