Wordpress LScache Plugin: Crawler Doesnt Seem to Crawl Product Pages
Last Updated on: Wed, 15 Apr 2026 00:00:02 Hi, I have a pretty large site ? a woocommerce store with over 6000 products. I recently moved my site to an openlitespeed server on Google Cloud with Litespeed Cache. I feel it would be useful to have the crawler cache the site so that users dont hit uncached pages, which is very often since we have a large site. Ive tried setting up the crawler by following the instructions here: https://docs.litespeedtech.com/lscache/lscwp/crawler/ When I refresh the Map, it only detects about 285 pages and completely skips all custom taxonomies despite the Exclude Custom Taxonomies being empty. The crawler also seems to classify almost all the pages that it does crawl as Cache Miss or Blacklisted despite all those pages working properly when I visit them manually. Im attaching a screenshot of the same. To fix the sitemap, I added our sitemap URL /sitemap_index.xml in the Sitemap Settings. When I tried to refresh crawl map, it just went to 0 pages. I removed that sitemap url and it went back to 285 odd pages still ignoring the product pages and custom taxonomies. The watch crawler status is almost always completely empty and only shows dots progressing. Could you please help me out? Report Details: Report number: MMKBVNMM Report date: 06/11/2020 08:21:29 Screenshot: https://imgur.com/pJIjqK2.jpg Regards, Arjun This topic was modified 2 years, 5 months ago by arjunmurali1993. Reason: Missed Screenshot The page I need help with: https://swagswami.com Hi, Could you please go to admin area , left-side bar ?> LiteSpeed Cache ?> Toolbox ?> [6] Debug Settings in this page: set Debug Log to Admin IP Only add your IP into Admin IPs (you will see your IP below.) set Debug Level to Advanced save changes. Now ,open a new window, go to sitemap page , dont refresh it yet. then click button [D] Clear Log in previous window, keep this window open, dont close it nor navigate away from it. now refresh sitemap, after it completed, go back to Log View window and refresh this page, you should see the debug log is showing up. Then please copy all the text , paste it into https://paste.ubuntu.com/ then post the link it generated for you. Best regards, Hi, Thanks for the quick response. Heres the link to the generated log: https://paste.ubuntu.com/p/hQmZP3K98b/ Regards, Arjun Hi, Is this with custom sitemap URL ? Best regards, Hi, My site has sitemap generated by Yoast plugin. However, I havent entered anything into the sitemap section on Litespeed settings. Ive left the sitemap field blank. Regards, Arjun Hi, Please set your custom sitemap , then refresh it with debug log on , then post the log Best regards, Hi, Heres the log with the custom sitemap: https://paste.ubuntu.com/p/gGxdGPYF6D/ Regards, Arjun Hi, Please create a ticket here Best regards,
LiteCache Rush: Speed comes from using less, not from doing it faster
Reference