LLM Crawler for WordPress

Changelog & Version History

Version 0.9.3 Latest Patch
October 24, 2026
Reliability
  • A theme or plugin shortcode or block that fails when it runs outside a normal page view (while the plugin activates, while it refreshes its files on schedule, or while it answers an agent) no longer stops the plugin: activation completes, the files are refreshed and the agent gets its answer
  • Only the one affected page is touched: it is listed with its plain text (builder tags removed) instead of the fully rendered version, and the rest of the site keeps working exactly as before
  • This fixes a real activation failure on a site running the Flatsome theme, where an accordion shortcode stopped the plugin from activating during the 0.9.2 rollout
What you will see
  • One line in the PHP error log per affected page, naming the page and the error, so the site owner or developer can find the shortcode or block at fault
  • A new warning on the settings page lists the affected pages, so the site owner can see which pages are served as plain text
  • Pages whose shortcodes depend on the current post (for example podcast players or theme-builder dynamic content) may now include that content in llms-full.txt and the Markdown twins where before they showed nothing
Compatibility
  • Nothing else changed: every setting and every text surface for pages that render normally is identical to 0.9.2
  • Requires PHP 8.2 or newer, as before; updating from 0.9.2 is a single step and changes nothing an administrator chose
Version 0.9.2 Minor
October 2026
Agent discovery
  • MCP server speaks the current MCP specification (2026-07-28) alongside the previous one (2025-11-25), and publishes an MCP Server Card at /wp-json/llm/v1/mcp/server-card so agents can find and verify the server before connecting
  • New RFC 9727 API catalog at /.well-known/api-catalog (with a Link header on the front page) lists every endpoint the plugin offers
  • WordPress Abilities (WordPress 6.9 and newer): search, list and read content through the core Abilities API, with the same rate limits as the REST and MCP endpoints
  • IndexNow: when a page is published, updated or removed, Bing, Yandex and the other IndexNow search engines are told right away; the key file is created for you at your site root (/{key}.txt)
  • Edge check button on the settings page shows what Cloudflare or your server cache actually serve for llms.txt, ai-crawl.json, the API catalog and a Markdown twin, plus new conflict diagnostics (another plugin writing llms.txt, a physical robots.txt, other MCP plugins)
Access and security
  • Authorization: Bearer tokens are accepted everywhere a token is; ?token= still works but is marked deprecated in the response
  • IP allowlist is Cloudflare-aware: behind Cloudflare the real visitor address is used, and a Cloudflare address is only believed when the request really came from a Cloudflare edge (a list that only contained Cloudflare ranges was open to every visitor and is now correctly closed)
  • A site set to discourage search engines (Settings, Reading) is hidden from every agent surface: llms.txt, Markdown twins, REST, MCP, Server Card, API catalog and Abilities answer 404 until it is switched back
  • Every 429 (rate limit) answer carries a Retry-After header so well-behaved agents know when to come back
  • Requires PHP 8.2 or newer (8.0 and 8.1 reached end of life)
Content rules
  • Your SEO plugin's AI-input exclusions are honoured on every text surface: pages marked nosnippet, max-snippet:0 or noarchive are listed with title and URL only, max-snippet:N caps the text, data-nosnippet regions are removed (Yoast, Rank Math, All in One SEO, SEOPress)
  • Pages with no prose at all (galleries, forms, portfolio layouts) now use the SEO meta description from Yoast, Rank Math, All in One SEO or SEOPress as their summary instead of an empty line; pages that already had text are unchanged
  • Words no longer run together at headings, list items and line breaks in summaries (a space is inserted at every block boundary)
  • Optional Content-Signal line (off by default) in robots.txt and as a response header to state your search, AI input and AI training preferences; robots.txt gains a YouBot group
Performance
  • llms.txt, llms-full.txt, Markdown twins, manifest and delta send Last-Modified and answer 304 Not Modified to conditional requests, so agents re-download only what changed; llms.txt is capped at 1.5 MB with a pointer to the full manifest
  • WooCommerce sites get an OpenAI ACP product feed at /wp-json/llm/v1/acp-feed next to the existing product tools
  • Updating from 0.8.9 is a single step: one settings write, one IndexNow key file and one scheduled refresh of the static files; nothing an administrator chose is changed
Version 0.8.9 Patch
October 2026
Content quality
  • Summaries for Divi 5 and other block-builder pages: when the stored content keeps its words inside block data, the plugin now renders the page and summarises the result instead of producing an empty description
  • Line breaks are kept between headings and paragraphs in Markdown twins and summaries so words no longer run together
Version 0.8.8 Patch
October 2026
Content quality
  • llms.txt summaries, schema.org descriptions and MCP search results no longer contain raw page-builder shortcodes (Divi, WPBakery, Fusion) - the words inside the modules are kept
  • HTML entities (&, ’,  ) are decoded in every agent-facing text: titles, summaries, Markdown twins, llms-full.txt, agents.md
Content rules
  • The plugin now honours your SEO plugin's post-type default (Yoast "Show in search results", Rank Math robots) in addition to per-page settings; a per-page "index" override still wins
  • The sitemap provider applies the same inclusion rules as every other endpoint (excluded, noindex, password-protected and taxonomy-filtered content is left out)
API hardening
  • /manifest and /delta cap per_page at 200 (was unbounded)
  • /delta normalises the since parameter to UTC so equivalent timestamps share one cache entry and the comparison against modification dates is timezone-correct; unparsable values return 400
Version 0.8.7 Patch
October 2026
Fixes
  • The front page's Markdown twin (/index.md, the first link in most sites' llms.txt) now serves the static front page instead of returning 404
  • Sites with a static front page now advertise it with a rel="alternate" text/markdown link to /index.md
Version 0.8.6 Patch
October 2026
Fleet fixes
  • llms.txt: the plugin recognises its own pre-0.8.4 files again and regenerates them - sites that had been serving a stale early-2026 llms.txt get the current index, Markdown-twin links and the ownership footer back
  • Physical /.well-known/ai-crawl.json and agents.json are now written to disk, so discovery works on hosts whose web server never passes .well-known requests to WordPress
  • Discovery files refresh twice daily and whenever published content is deleted; .md twins, agents.md and llms-full.txt no longer redirect before serving
Security
  • Update packages are now signature-verified before install: the Ed25519 signature shipped with every release is checked against the plugin's trusted key, and a missing or invalid signature aborts the update
Agent protocols
  • UCP business profile and catalog tools rebuilt to the formal v2026-08-25 specification (search_catalog, lookup_catalog, get_product) - read-only, no checkout
  • WooCommerce MCP hint updated for WooCommerce 11.3; the WordPress MCP Adapter is advertised only when installed
  • New crawler tokens: OAI-AdsBot, Amzn-SearchBot, Amzn-User, meta-webindexer, meta-externalads, MistralAI-Training, Kimi, Reflectionbot; Web Bot Auth Signature-Agent parsing updated for the IETF draft
  • llms.txt v2: rel="describedby" link and header on every page
Version 0.8.5 Patch
August 2026
Analytics API
  • New token-protected analytics endpoint lets your own AI agent or dashboard pull your AI-crawler visit data - per-bot totals, top crawled pages, a 30-day trend and last-seen timestamps
  • Off by default and always requires your private access token - analytics are never exposed publicly
Version 0.8.4 Patch
August 2026
AI Crawler Coverage
  • Now detects Google-Agent, the Gemini Notebook research assistant (formerly NotebookLM) and the Mistral AI crawlers, so visits from these assistants show up in your analytics
  • robots.txt welcome rules extended to Google-Agent and the Mistral crawlers
Bug Fixes
  • The bulk JSONL export now streams clean newline-delimited JSON - previously the whole export arrived wrapped in an extra layer of JSON encoding that tripped strict parsers
  • Turning off llms-full.txt now removes the generated file from the web root instead of leaving a stale copy behind
  • Removed a redundant legacy update-verification hook - package integrity is enforced by WordPress core signature verification
Plays Well With Others
  • Detects when another SEO plugin such as Yoast, All in One SEO or Rank Math takes over the physical llms.txt file and shows a clear admin warning instead of silently fighting over the file
  • llms.txt now carries a generated-by footer so ownership of the file is always clear
Version 0.8.2 Patch
June 2026
Update Reliability
  • Automatic updates no longer risk silently deactivating the plugin - it now re-activates itself the instant it finishes updating to a new version
  • Discovery endpoints such as llms.txt, the MCP server and the per-page Markdown twins keep serving without interruption after an unattended background update
  • The self-reactivation safeguard only ever moves forward to a newer version and never rolls the plugin back to an older one
Version 0.8.1 Patch
June 2026
Stability Fix
  • Fixed a critical issue where saving the plugin settings on 0.8.0 could exhaust PHP memory and return an HTTP 500 error - settings now save reliably
  • The internal version marker now round trips correctly through settings sanitization, preventing a repeated re-save loop
  • Hardened discovery route registration so it never runs before WordPress has initialized its rewrite system
Version 0.8.0 Major Release
June 2026
Agent-Ready Content (Markdown twins)
  • Per-page .md twins - append .md to any URL, or send Accept: text/markdown, for clean Markdown with YAML frontmatter
  • New /agents.md agent-instructions file
  • llms.txt link lists now point straight at the Markdown twins
Modernized MCP Server
  • Protocol-version negotiation (speaks up to 2025-11-25)
  • Structured tool output (structuredContent + output schemas) and read-only tool annotations
  • Streamable-HTTP conformance, stateless, with Origin checks, token gate and rate limiting
WooCommerce Commerce Mode
  • Read-only product feed at /llm-products.json (price, availability, SKU, GTIN, variants)
  • search_products and get_product MCP tools
  • /.well-known/ucp capability profile + enriched Product schema
  • Strictly read-only - agents never complete a purchase without explicit buyer consent
Analytics and Reports
  • Crawler hits now tracked per page and per crawler
  • "See more" to expand beyond the top 10 pages
  • Expand any page to see its hit rate broken down by crawler
  • Export the data as CSV or a printable PDF report
Crawler Detection
  • Refreshed, filterable AI user-agent list (Claude-User, Meta-ExternalAgent, Amazonbot, DuckAssistBot, Perplexity-User, and more)
  • RFC 9421 Signature-Agent detection
  • Expanded robots.txt Allow rules for current AI crawlers
Version 0.7.0 Major Release
February 2026
New Standards Support
  • Spec-compliant /llms.txt following llmstxt.org format (Markdown with H1/H2/link lists)
  • New /llms-full.txt endpoint with inline full content
  • New /.well-known/agents.json for machine-readable API descriptions
  • MCP (Model Context Protocol) server at /wp-json/llm/v1/mcp with 6 tools
  • AI-specific XML sitemap provider for WordPress 5.5+ sitemaps
AI Crawler Intelligence
  • Automatic detection of 12+ AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.)
  • X-LLM-Endpoints header sent to detected AI bots
  • Enhanced robots.txt with per-bot Allow rules for all AI crawlers
  • New Analytics dashboard tracking bot visits, top pages, and last-seen timestamps
Content Optimization
  • Schema.org JSON-LD output on singular posts (Article/WebPage)
  • LLM-specific meta tags: llm-content-type, llm-language, llm-last-modified
  • Markdown alternate link for each post via rel="alternate"
  • AIOSEO compatibility added (noindex detection)
Security & Privacy
  • Telemetry changed to opt-in only (GDPR compliance)
  • No data sent during activation — requires explicit user consent
  • SHA-384 package hash verification on updates
  • Ed25519 package signing infrastructure (WordPress 5.2+ native)
  • Site URL hashed (SHA-256) in update checks for privacy
  • Explicit SSL verification on all remote requests
Admin UI
  • New Discovery tab with toggles for llms.txt, llms-full.txt, agents.json, and MCP
  • New Analytics tab with bot visit charts and top crawled pages
  • Improved telemetry description listing exactly what data is sent
  • Configurable per-type limit for llms.txt entries
Version 0.6.8 Minor Release
October 2025
New Features
  • Added "Reset to Defaults" button for quick configuration
  • Confirmation dialog lists all settings that will be changed
  • Preserves important data like Client ID when resetting
Improvements
  • Better handling of settings persistence
  • Improved user experience with clear reset functionality
Version 0.6.7 Patch
October 2025
Bug Fixes
  • Fixed default checkboxes not showing as checked on General tab
  • Forced critical defaults on plugin upgrade
  • Completely removed Update Server field from UI
UI Improvements
  • Discovery URLs now display on individual lines
  • Added copy button next to each endpoint URL
  • Improved Discovery Endpoints section layout
  • JavaScript copy functionality with visual feedback
System Updates
  • Update server URL hardcoded to https://llm-crawler.exploredigital.network
  • Improved llms.txt file writing with better error handling
  • Dynamic serving of llms.txt if file creation fails
Version 0.6.6 Patch
October 2025
Settings Updates
  • Updated default settings for optimal LLM crawling
  • Cache TTL default changed from 5 to 60 minutes
  • Auto-update preference enabled by default
  • All discovery options enabled by default
Improvements
  • Better activation process for new and existing installs
  • Simplified file writing logic for llms.txt
  • Improved error messages
Version 0.6.5 Minor Release
October 2025
New Features
  • Added Settings link on plugins page for quick access
  • Added "About this plugin" link pointing to update server
  • Plugin action links for better navigation
Default Settings
  • Respect SEO noindex checked by default
  • Public discovery enabled by default
  • All discovery methods activated by default
  • Telemetry and auto-updates enabled by default
Version 0.6.4 Major Fix
October 2025
Critical Fixes
  • Fixed duplicate Client ID systems causing multiple registrations
  • Consolidated Client ID management to single system
  • Fixed Client ID disappearing when toggling auto-update
  • Removed duplicate settings storage
Technical Improvements
  • Update client now uses main plugin's Client ID via dependency injection
  • Improved settings persistence
  • Better integration between plugin components
  • Added public get_client_id() method
Versions 0.6.0 - 0.6.3 Initial Releases
September - October 2025
Core Features
  • Initial release of LLM Crawler plugin
  • REST API endpoints for manifest, content, and delta updates
  • Token-based authentication system
  • IP allowlisting and rate limiting
  • ACF field support with configurable allowlists
  • JSONL export functionality
  • Webhook notifications for content updates
  • Universal discovery via /.well-known and robots.txt
  • Self-update mechanism from dedicated update server
  • Per-post exclusion capabilities
  • SEO plugin integration (Yoast, RankMath)
  • Smart caching system