Wordpress LScache Plugin: Crawler is adding all urls to the blacklist
Last Updated on: Wed, 15 Apr 2026 00:00:02 Hello! I generated a report to you with this code: ZZRJVBNX All the urls that scan the crawler are added to the blacklist in first time, so stops to continue crawling the website. As I can read in the description, this list is generated in basis to all urls that have the tag no-cache. I dont setup nothing related with that and I cant found anything regarding that. Where I can setup what urls should have or not the no-chache tag? Thank you! Hello, From Litespeed cache -> Settings -> Crawler see if HTTP/2 Crawl is enable and if so, please disable it and re-run the crawler. Hi Tishu! Thank you so much for your very fast reply! Seems that is working now, I think is not adding more urls to the blacklist. I say seems due for any reason now stops the crawling: Start watching? 18 Apr 2018 11:04:47 Size: 419 Crawler: #1 Position: 370 Threads: 3 Status: crawling, updated position ??. Is like is running, but all time is fixed in that. We have 32 core server and litespeed license for 4 cores. So I think 3 threads should be fine. Checking server resources while is crawling the CPU usage the top change from 3 to 6 of the 32 available cores, and from 5% to 10% of all the available CPU. Is not a problem with the RAM neither due we have 192Gb. Maybe is due to any of my settings limitation? One thing to have in consideration, I added 3 additional ID user roles to crawler, but stay on first one in 370 of 419 url. Thanks! Hello again, You can check the options in Litespeed cache -> Settings -> Crawler and edit them depend on the server load. Thanks! Yes I do it. But I am not sure about the good values to put here. I will play with all the options. Thanks again! According to this issue, we set that HTTP/2 Crawl default to OFF instead of ON for new installations.
LiteCache Rush: Speed comes from using less, not from doing it faster
Reference