{"id":43,"date":"2025-10-02T22:06:22","date_gmt":"2025-10-02T22:06:22","guid":{"rendered":"https:\/\/llm-crawler.exploredigital.network\/?page_id=43"},"modified":"2026-03-10T23:06:21","modified_gmt":"2026-03-10T23:06:21","slug":"home-2","status":"publish","type":"page","link":"https:\/\/llm-crawler.exploredigital.network\/","title":{"rendered":"Home"},"content":{"rendered":"<div class=\"et_pb_section_0 et_pb_section et_section_regular et_flex_section\" id=\"hero\">\n<div class=\"et_pb_row_0 et_pb_row et_flex_row preset--module--divi-row--default\">\n<div class=\"et_pb_column_0 et_pb_column et_flex_column et_pb_css_mix_blend_mode_passthrough et_flex_column_12_24 et_flex_column_24_24_phone\">\n<div class=\"et_pb_text_0 et_pb_text et_pb_bg_layout_light et_pb_module et_flex_module preset--module--divi-text--default\"><div class=\"et_pb_text_inner\"><h1>LLM Crawler for WordPress<\/h1>\n<p>Make Your WordPress Content AI-Ready<\/p>\n<\/div><\/div>\n\n<div class=\"et_pb_module et_pb_button_module_wrapper et_pb_button_0_wrapper preset--module--divi-button--default_wrapper\"><a class=\"et_pb_button_0 et_pb_button et_pb_bg_layout_light et_pb_module et_block_module preset--module--divi-button--default\" href=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2026\/02\/llm-crawler-0.7.0.zip\" id=\"hero-button\">Download Plugin<\/a><\/div>\n<\/div>\n\n<div class=\"et_pb_column_1 et_pb_column et-last-child et_flex_column et_pb_css_mix_blend_mode_passthrough et_flex_column_12_24 et_flex_column_24_24_phone\">\n<div class=\"et_pb_image_0 et_pb_image et_pb_module et_flex_module\" alt=\"A screenshot of the dashboard for the plugin\"><span class=\"et_pb_image_wrap\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2026\/03\/The-Bottom-Line.webp\" title=\"The Bottom Line\" width=\"642\" height=\"381\" srcset=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2026\/03\/The-Bottom-Line.webp 642w, https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2026\/03\/The-Bottom-Line-480x285.webp 480w\" sizes=\"(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 642px, 100vw\" class=\"wp-image-246\" \/><\/span><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n<div class=\"et_pb_section_1 et_pb_section et_section_regular et_block_section\">\n<div class=\"et_pb_row_1 et_pb_row et_block_row preset--module--divi-row--default\">\n<div class=\"et_pb_column_2 et_pb_column et_pb_column_4_4 et-last-child et_block_column et_pb_css_mix_blend_mode_passthrough\">\n<div class=\"et_pb_heading_0 et_pb_heading et_pb_module et_block_module preset--module--divi-heading--default\"><div class=\"et_pb_heading_container\"><h5 class=\"et_pb_module_header\">LLM Crawler is a free WordPress plugin that creates structured, AI-friendly endpoints for your website content. As AI tools increasingly need to understand and interact with web content, this plugin ensures your WordPress site can be efficiently and accurately crawled by Large Language Models (LLMs) and AI services.<\/h5><\/div><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n<div class=\"et_pb_section_2 et_pb_section et_section_regular et_block_section\">\n<div class=\"et_pb_row_2 et_pb_row et_block_row preset--module--divi-row--default\">\n<div class=\"et_pb_column_3 et_pb_column et_pb_column_1_2 et_block_column et_pb_css_mix_blend_mode_passthrough\">\n<div class=\"et_pb_heading_1 et_pb_heading et_pb_module et_block_module preset--module--divi-heading--default\"><div class=\"et_pb_heading_container\"><h2 class=\"et_pb_module_header\">\ud83e\udd16 AI-Optimized Content Delivery<\/h2><\/div><\/div>\n\n<div class=\"et_pb_text_1 et_pb_text et_pb_bg_layout_light et_pb_module et_block_module preset--module--divi-text--default\"><div class=\"et_pb_text_inner\"><p>Transform your WordPress content into clean, structured data that AI systems can easily understand. The plugin provides multiple REST API endpoints specifically designed for LLM consumption, including manifest, delta updates, and content endpoints.<\/p>\n<\/div><\/div>\n\n<div class=\"et_pb_module et_pb_button_module_wrapper et_pb_button_1_wrapper preset--module--divi-button--default_wrapper\"><a class=\"et_pb_button_1 et_pb_button et_pb_bg_layout_light et_pb_module et_block_module preset--module--divi-button--default\" href=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2026\/02\/llm-crawler-0.7.0.zip\" id=\"hero-button\">Download Plugin<\/a><\/div>\n<\/div>\n\n<div class=\"et_pb_column_4 et_pb_column et_pb_column_1_2 et-last-child et_block_column et_pb_css_mix_blend_mode_passthrough preset--module--divi-column--85a2fe7d-7d65-431e-aa1f-cf5cb89dee4d\">\n<div class=\"et_pb_image_1 et_pb_image et_pb_module et_block_module\"><span class=\"et_pb_image_wrap\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152035.png\" width=\"768\" height=\"533\" srcset=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152035.png 768w, https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152035-480x333.png 480w\" sizes=\"(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 768px, 100vw\" class=\"wp-image-88\" alt=\"A screenshot of the dashboard of the plugin, with options like &amp;amp;quot;require token&amp;amp;quot;, &amp;amp;quot;shared token&amp;amp;quot;, and &amp;amp;quot;IP allowlist&amp;amp;quot;\" \/><\/span><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n<div class=\"et_pb_section_3 et_pb_section et_section_regular et_block_section\">\n<div class=\"et_pb_row_3 et_pb_row et_block_row preset--module--divi-row--default\">\n<div class=\"et_pb_column_5 et_pb_column et_pb_column_1_2 et_block_column et_pb_css_mix_blend_mode_passthrough preset--module--divi-column--983ca724-762b-4ae3-9563-049d6ed31e4a\">\n<div class=\"et_pb_image_2 et_pb_image et_pb_module et_block_module preset--module--divi-image--4f4e52d9-d08a-4dd1-bbb6-a6ba5ad53f6a\"><span class=\"et_pb_image_wrap\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-151835.png\" width=\"655\" height=\"922\" srcset=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-151835.png 655w, https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-151835-480x676.png 480w\" sizes=\"(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 655px, 100vw\" class=\"wp-image-87\" alt=\"The &amp;amp;amp;quot;general&amp;amp;amp;quot; tab of the plugin, where users can edit how SEO no index is respected, how posts are included, and other useful features\" \/><\/span><\/div>\n<\/div>\n\n<div class=\"et_pb_column_6 et_pb_column et_pb_column_1_2 et-last-child et_block_column et_pb_css_mix_blend_mode_passthrough\">\n<div class=\"et_pb_heading_2 et_pb_heading et_pb_module et_block_module preset--module--divi-heading--default\"><div class=\"et_pb_heading_container\"><h2 class=\"et_pb_module_header\">\ud83d\udd12 Complete Control Over Your Content<\/h2><\/div><\/div>\n\n<div class=\"et_pb_text_2 et_pb_text et_pb_bg_layout_light et_pb_module et_block_module preset--module--divi-text--default\"><div class=\"et_pb_text_inner\"><ul>\n<li><strong>Respect existing SEO settings<\/strong> - Automatically honors Yoast and RankMath noindex settings<\/li>\n<li><strong>Per-post exclusion<\/strong> - Choose exactly which content to share<\/li>\n<li><strong>Token-based access control<\/strong> - Optionally require authentication for crawling<\/li>\n<li><strong>IP allowlisting<\/strong> - Restrict access to specific services<\/li>\n<li><strong>Rate limiting<\/strong> - Prevent abuse with configurable request limits<\/li>\n<\/ul>\n<\/div><\/div>\n\n<div class=\"et_pb_module et_pb_button_module_wrapper et_pb_button_2_wrapper preset--module--divi-button--default_wrapper\"><a class=\"et_pb_button_2 et_pb_button et_pb_bg_layout_light et_pb_module et_block_module preset--module--divi-button--default\" href=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2026\/02\/llm-crawler-0.7.0.zip\" id=\"hero-button\">Download Plugin<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n<div class=\"et_pb_section_4 et_pb_section et_section_regular et_block_section\">\n<div class=\"et_pb_row_4 et_pb_row et_block_row preset--module--divi-row--default\">\n<div class=\"et_pb_column_7 et_pb_column et_pb_column_4_4 et-last-child et_block_column et_pb_css_mix_blend_mode_passthrough\">\n<div class=\"et_pb_heading_3 et_pb_heading et_pb_module et_block_module preset--module--divi-heading--default\"><div class=\"et_pb_heading_container\"><h2 class=\"et_pb_module_header\">Universal Discovery<\/h2><\/div><\/div>\n\n<div class=\"et_pb_text_3 et_pb_text et_pb_bg_layout_light et_pb_module et_block_module preset--module--divi-text--default\"><div class=\"et_pb_text_inner\"><p>Spec-compliant <code>\/llms.txt<\/code> + <code>\/llms-full.txt<\/code> (llmstxt.org format)<br \/>MCP server + <code>agents.json<\/code> + <code>ai-crawl.json<\/code><br \/>AI crawler detection with analytics dashboard<br \/>Schema.org JSON-LD + robots.txt Allow rules for AI bots<\/p>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n<div class=\"et_pb_row_5 et_pb_row et_block_row preset--module--divi-row--default\">\n<div class=\"et_pb_column_8 et_pb_column et_pb_column_4_4 et-last-child et_block_column et_pb_css_mix_blend_mode_passthrough\">\n<div class=\"et_pb_image_3 et_pb_image et_pb_module et_block_module\"><span class=\"et_pb_image_wrap\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152159.png\" width=\"1012\" height=\"334\" srcset=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152159.png 1012w, https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152159-980x323.png 980w, https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152159-480x158.png 480w\" sizes=\"(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) and (max-width: 980px) 980px, (min-width: 981px) 1012px, 100vw\" class=\"wp-image-89\" alt=\"A list of endpoints that can be set and edited within the plugin\" \/><\/span><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n<div class=\"et_pb_section_5 et_pb_section et_section_regular et_block_section\">\n<div class=\"et_pb_row_6 et_pb_row et_pb_equal_columns et_pb_gutters1 et_block_row preset--module--divi-row--9a4720ce-5cc6-4446-b32c-1e590556f88c\">\n<div class=\"et_pb_column_9 et_pb_column et_pb_column_1_2 et_block_column et_pb_css_mix_blend_mode_passthrough\">\n<div class=\"et_pb_cta_0 et_pb_promo et_pb_bg_layout_dark et_pb_module et_block_module preset--module--divi-cta--64ff4712-5cb8-4ee2-ab92-dd4039d199cc\"><div class=\"et_pb_promo_description\"><h2 class=\"et_pb_module_header\">\ud83d\udcca Advanced Features<\/h2><div class=\"et_pb_promo_content\"><ul>\n<li><strong>MCP server<\/strong> with 6 built-in tools for AI agent integration<\/li>\n<li><strong>agents.json<\/strong> machine-readable API description<\/li>\n<li><strong>AI crawler analytics<\/strong> dashboard with per-bot visit tracking<\/li>\n<li><strong>Schema.org JSON-LD<\/strong> and content optimization meta tags<\/li>\n<li><strong>Ed25519 signed updates<\/strong> with SHA-384 hash verification<\/li>\n<li><strong>JSONL bulk export<\/strong> + webhook notifications<\/li>\n<li><strong>Smart caching<\/strong>, rate limiting, ACF support, and more<\/li>\n<\/ul>\n<\/div><\/div><\/div>\n\n<div class=\"et_pb_module et_pb_button_module_wrapper et_pb_button_3_wrapper preset--module--divi-button--default_wrapper\"><a class=\"et_pb_button_3 et_pb_button et_pb_bg_layout_light et_pb_module et_block_module preset--module--divi-button--default\" href=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2026\/02\/llm-crawler-0.7.0.zip\" id=\"hero-button\">Download Plugin<\/a><\/div>\n<\/div>\n\n<div class=\"et_pb_column_10 et_pb_column et_pb_column_1_2 et-last-child et_block_column et_pb_css_mix_blend_mode_passthrough\">\n<div class=\"et_pb_image_4 et_pb_image et_pb_module et_block_module\"><span class=\"et_pb_image_wrap\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152333.png\" width=\"731\" height=\"349\" srcset=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152333.png 731w, https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2025\/10\/Screenshot-2025-10-02-152333-480x229.png 480w\" sizes=\"(min-width: 0px) and (max-width: 480px) 480px, (min-width: 481px) 731px, 100vw\" class=\"wp-image-90\" alt=\"The Export and Webhooks tab of the plugin, where JSON export can be provided and webhook url set\" \/><\/span><\/div>\n<\/div>\n<\/div>\n<\/div>\n\n<div class=\"et_pb_section_6 et_pb_section et_section_regular et_flex_section\">\n<div class=\"et_pb_row_7 et_pb_row et_block_row preset--module--divi-row--default\">\n<div class=\"et_pb_column_11 et_pb_column et-last-child et_flex_column et_pb_css_mix_blend_mode_passthrough et_flex_column_24_24 et_flex_column_24_24_tablet et_flex_column_24_24_phone\">\n<div class=\"et_pb_text_4 et_pb_text et_pb_bg_layout_light et_clickable et_pb_module et_block_module preset--module--divi-text--default\"><div class=\"et_pb_text_inner\"><h3 class=\"text-lg font-bold text-text-100 mt-1 -mb-1.5\">\ud83d\ude80 <strong>Easy Setup<\/strong><\/h3>\n<p>The plugin comes with sensible defaults that work out of the box:<\/p>\n<ul class=\"[&amp;:not(:last-child)_ul]:pb-1 [&amp;:not(:last-child)_ol]:pb-1 list-disc space-y-1.5 pl-7\">\n<li>All recommended settings enabled by default<\/li>\n<li>One-click \"Reset to Defaults\" option<\/li>\n<li>Clean admin interface with organized settings tabs<\/li>\n<li>Copy-ready endpoint URLs for easy sharing<\/li>\n<\/ul>\n<h3>How It Works<\/h3>\n<p>Once installed, LLM Crawler automatically creates structured endpoints that AI services can use to understand your content:<\/p>\n<ol class=\"[&amp;:not(:last-child)_ul]:pb-1 [&amp;:not(:last-child)_ol]:pb-1 list-decimal space-y-1.5 pl-7\">\n<li><strong>Manifest Endpoint<\/strong> - Lists all available content with metadata<\/li>\n<li><strong>Content Endpoint<\/strong> - Delivers clean, structured content by ID<\/li>\n<li><strong>Delta Endpoint<\/strong> - Shows only content that's changed since a specific date<\/li>\n<li><strong>Export Endpoint<\/strong> - Provides bulk JSONL export for data processing<\/li>\n<\/ol>\n<h3 class=\"text-xl font-bold text-text-100 mt-1 -mb-0.5\">Privacy &amp; Control<\/h3>\n<p>Your content remains under your complete control. The plugin respects all existing privacy settings and adds additional layers of protection:<\/p>\n<ul class=\"[&amp;:not(:last-child)_ul]:pb-1 [&amp;:not(:last-child)_ol]:pb-1 list-disc space-y-1.5 pl-7\">\n<li>Content marked as noindex is automatically excluded<\/li>\n<li>Password-protected posts are never exposed<\/li>\n<li>Draft and private content stays private<\/li>\n<li>Optional token authentication adds security<\/li>\n<li>Telemetry is fully opt-in (disabled by default)<\/li>\n<\/ul>\n<h3 class=\"text-xl font-bold text-text-100 mt-1 -mb-0.5\">Perfect For<\/h3>\n<ul class=\"[&amp;:not(:last-child)_ul]:pb-1 [&amp;:not(:last-child)_ol]:pb-1 list-disc space-y-1.5 pl-7\">\n<li><strong>Publishers<\/strong> wanting to make content available to AI search and chat services<\/li>\n<li><strong>Businesses<\/strong> looking to integrate with AI-powered tools<\/li>\n<li><strong>Educational sites<\/strong> sharing knowledge with LLM training systems<\/li>\n<li><strong>Documentation sites<\/strong> providing structured technical content<\/li>\n<li><strong>News organizations<\/strong> distributing content to AI aggregators<\/li>\n<\/ul>\n<h3 class=\"text-xl font-bold text-text-100 mt-1 -mb-0.5\">Always Free<\/h3>\n<p>LLM Crawler is completely free and always will be. There are no premium tiers, no locked features, and no hidden costs. We believe in making the web more accessible to AI while keeping control in the hands of content creators.<\/p>\n<h3 class=\"text-xl font-bold text-text-100 mt-1 -mb-0.5\">Technical Specifications<\/h3>\n<ul class=\"[&amp;:not(:last-child)_ul]:pb-1 [&amp;:not(:last-child)_ol]:pb-1 list-disc space-y-1.5 pl-7\">\n<li><strong>Requires:<\/strong> WordPress 5.9 or higher<\/li>\n<li><strong>PHP:<\/strong> 7.4 or higher<\/li>\n<li><strong>License:<\/strong> Free\/Open Source<\/li>\n<li><strong>Support:<\/strong> Community-driven<\/li>\n<li><strong>Updates:<\/strong> Automatic updates available<\/li>\n<\/ul>\n<h3 class=\"text-xl font-bold text-text-100 mt-1 -mb-0.5\">Get Started<\/h3>\n<p>Simply install the plugin from your WordPress admin panel or download it directly. The plugin activates with optimal settings by default, but everything is customizable through an intuitive settings interface.<\/p>\n<hr class=\"border-border-300 my-2\" \/>\n<p><em>LLM Crawler is developed and maintained by Explore Digital https:\/\/www.exploredigital.com. We're committed to helping WordPress sites participate in the AI revolution while maintaining full control over their content.<\/em><\/p>\n<\/div><\/div>\n\n<div class=\"et_pb_module et_pb_button_module_wrapper et_pb_button_4_wrapper preset--module--divi-button--default_wrapper\"><a class=\"et_pb_button_4 et_pb_button et_pb_bg_layout_light et_pb_module et_block_module preset--module--divi-button--default\" href=\"https:\/\/llm-crawler.exploredigital.network\/wp-content\/uploads\/2026\/02\/llm-crawler-0.7.0.zip\" id=\"hero-button\">Download Plugin<\/a><\/div>\n<\/div>\n<\/div>\n<\/div>","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-43","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/llm-crawler.exploredigital.network\/index.php\/wp-json\/wp\/v2\/pages\/43","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/llm-crawler.exploredigital.network\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/llm-crawler.exploredigital.network\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/llm-crawler.exploredigital.network\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/llm-crawler.exploredigital.network\/index.php\/wp-json\/wp\/v2\/comments?post=43"}],"version-history":[{"count":28,"href":"https:\/\/llm-crawler.exploredigital.network\/index.php\/wp-json\/wp\/v2\/pages\/43\/revisions"}],"predecessor-version":[{"id":272,"href":"https:\/\/llm-crawler.exploredigital.network\/index.php\/wp-json\/wp\/v2\/pages\/43\/revisions\/272"}],"wp:attachment":[{"href":"https:\/\/llm-crawler.exploredigital.network\/index.php\/wp-json\/wp\/v2\/media?parent=43"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}