LLM Crawler for WordPress

Changelog & Version History

Version 0.8.1 Latest Patch
June 2026
Stability Fix
  • Fixed a critical issue where saving the plugin settings on 0.8.0 could exhaust PHP memory and return an HTTP 500 error - settings now save reliably
  • The internal version marker now round trips correctly through settings sanitization, preventing a repeated re-save loop
  • Hardened discovery route registration so it never runs before WordPress has initialized its rewrite system
Version 0.8.0 Major Release
June 2026
Agent-Ready Content (Markdown twins)
  • Per-page .md twins - append .md to any URL, or send Accept: text/markdown, for clean Markdown with YAML frontmatter
  • New /agents.md agent-instructions file
  • llms.txt link lists now point straight at the Markdown twins
Modernized MCP Server
  • Protocol-version negotiation (speaks up to 2025-11-25)
  • Structured tool output (structuredContent + output schemas) and read-only tool annotations
  • Streamable-HTTP conformance, stateless, with Origin checks, token gate and rate limiting
WooCommerce Commerce Mode
  • Read-only product feed at /llm-products.json (price, availability, SKU, GTIN, variants)
  • search_products and get_product MCP tools
  • /.well-known/ucp capability profile + enriched Product schema
  • Strictly read-only - agents never complete a purchase without explicit buyer consent
Analytics and Reports
  • Crawler hits now tracked per page and per crawler
  • "See more" to expand beyond the top 10 pages
  • Expand any page to see its hit rate broken down by crawler
  • Export the data as CSV or a printable PDF report
Crawler Detection
  • Refreshed, filterable AI user-agent list (Claude-User, Meta-ExternalAgent, Amazonbot, DuckAssistBot, Perplexity-User, and more)
  • RFC 9421 Signature-Agent detection
  • Expanded robots.txt Allow rules for current AI crawlers
Version 0.7.0 Major Release
February 2026
New Standards Support
  • Spec-compliant /llms.txt following llmstxt.org format (Markdown with H1/H2/link lists)
  • New /llms-full.txt endpoint with inline full content
  • New /.well-known/agents.json for machine-readable API descriptions
  • MCP (Model Context Protocol) server at /wp-json/llm/v1/mcp with 6 tools
  • AI-specific XML sitemap provider for WordPress 5.5+ sitemaps
AI Crawler Intelligence
  • Automatic detection of 12+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.)
  • X-LLM-Endpoints header sent to detected AI bots
  • Enhanced robots.txt with per-bot Allow rules for all AI crawlers
  • New Analytics dashboard tracking bot visits, top pages, and last-seen timestamps
Content Optimization
  • Schema.org JSON-LD output on singular posts (Article/WebPage)
  • LLM-specific meta tags: llm-content-type, llm-language, llm-last-modified
  • Markdown alternate link for each post via rel="alternate"
  • AIOSEO compatibility added (noindex detection)
Security & Privacy
  • Telemetry changed to opt-in only (GDPR compliance)
  • No data sent during activation — requires explicit user consent
  • SHA-384 package hash verification on updates
  • Ed25519 package signing infrastructure (WordPress 5.2+ native)
  • Site URL hashed (SHA-256) in update checks for privacy
  • Explicit SSL verification on all remote requests
Admin UI
  • New Discovery tab with toggles for llms.txt, llms-full.txt, agents.json, and MCP
  • New Analytics tab with bot visit charts and top crawled pages
  • Improved telemetry description listing exactly what data is sent
  • Configurable per-type limit for llms.txt entries
Version 0.6.8 Minor Release
October 2025
New Features
  • Added "Reset to Defaults" button for quick configuration
  • Confirmation dialog lists all settings that will be changed
  • Preserves important data like Client ID when resetting
Improvements
  • Better handling of settings persistence
  • Improved user experience with clear reset functionality
Version 0.6.7 Patch
October 2025
Bug Fixes
  • Fixed default checkboxes not showing as checked on General tab
  • Forced critical defaults on plugin upgrade
  • Completely removed Update Server field from UI
UI Improvements
  • Discovery URLs now display on individual lines
  • Added copy button next to each endpoint URL
  • Improved Discovery Endpoints section layout
  • JavaScript copy functionality with visual feedback
System Updates
  • Update server URL hardcoded to https://llm-crawler.exploredigital.network
  • Improved llms.txt file writing with better error handling
  • Dynamic serving of llms.txt if file creation fails
Version 0.6.6 Patch
October 2025
Settings Updates
  • Updated default settings for optimal LLM crawling
  • Cache TTL default changed from 5 to 60 minutes
  • Auto-update preference enabled by default
  • All discovery options enabled by default
Improvements
  • Better activation process for new and existing installs
  • Simplified file writing logic for llms.txt
  • Improved error messages
Version 0.6.5 Minor Release
October 2025
New Features
  • Added Settings link on plugins page for quick access
  • Added "About this plugin" link pointing to update server
  • Plugin action links for better navigation
Default Settings
  • Respect SEO noindex checked by default
  • Public discovery enabled by default
  • All discovery methods activated by default
  • Telemetry and auto-updates enabled by default
Version 0.6.4 Major Fix
October 2025
Critical Fixes
  • Fixed duplicate Client ID systems causing multiple registrations
  • Consolidated Client ID management to single system
  • Fixed Client ID disappearing when toggling auto-update
  • Removed duplicate settings storage
Technical Improvements
  • Update client now uses main plugin's Client ID via dependency injection
  • Improved settings persistence
  • Better integration between plugin components
  • Added public get_client_id() method
Versions 0.6.0 - 0.6.3 Initial Releases
September - October 2025
Core Features
  • Initial release of LLM Crawler plugin
  • REST API endpoints for manifest, content, and delta updates
  • Token-based authentication system
  • IP allowlisting and rate limiting
  • ACF field support with configurable allowlists
  • JSONL export functionality
  • Webhook notifications for content updates
  • Universal discovery via /.well-known and robots.txt
  • Self-update mechanism from dedicated update server
  • Per-post exclusion capabilities
  • SEO plugin integration (Yoast, RankMath)
  • Smart caching system