Wordpress LScache Plugin: Crawling the site from within WordPress
Last Updated on: Wed, 15 Apr 2026 00:00:02 This plugin uses the power of the LiteSpeed server, and this brings many benefits, but on shared servers, it also disables crawling as an option, unless I can convince my hosting provider to enable it. Unfortunately, I cant. I think its a strange choice, but they wont do it. Other caching plugins use the WordPress Cron to preload their caches, and this is their main advantage for me right now, so Im looking for a way to preload/crawl the site from within WordPress, without having to ask for permission. Personally, Im happy to create a cPanel Cron job, but for others, using WP Cron will make life a lot easier. Please advise how to do this (or add it as a feature). Hi, if you have shell access, or your cron can run shell script , here is a quick idea as workaround use shell script to parse URL from sitemap then use curl to mimic Chrome user agent and loop through the URLs parsed in sitemap. PS: use at your own risk and make sure it doesnt break AUP Best regards, This might work, but I was hoping more for a program that takes into consideration the timing settings in the plugins configuration and is provided by the people who write the plugin. Being on a shared server has its advantages (lower cost, includes system administrations and support), but comes with limitations, like no crawler. I think many plugin users can benefit from having a way to crawl the site from WordPress. PLEASE consider this. BTW, if you have any stats on server load with and without crawling enabled, so that I (and others) can convince the hosting provider(s) that allowing crawling is good, perhaps you could share those. Also, do you have any stats about the difference crawling makes to website performance? Reason: Added a question Hi, yeah well ? this is one of my VPS , 4 cores ,16GB RAM I started 3 sites crawler simultaneously and you can clearly see how CPU usage and load increased , like tenfold my VPS only got few test sites and doesnt have any real traffic. Its something provider decides how/what to do as if they want to give crawler or not. and even somehow , you can bypass the limit by use the way I mentioned before with shell script , it may still breaks AUP and if server has CloudLinux LVE that limits resources (which many providers do) , when crawler is running and real user accesses, due to strict resources limit (on CPU/RAM/Disk IO ?etc) it will make compete on resources between real user and crawler and could make things worse. performance-wise , no really , the only thing crawler does is mimic user access before real user does , so to pre-cache the page , so next user comes will get cached page directly instead of waiting for cache generation. Best regards, I appreciate you running and recording this experiment, but its difficult to gauge the results without regular visitor traffic and without knowing the timing setup. I believe that timings can be tweaked to accommodate the limits of every account and use only the leftover resources from visitor traffic. This may take time, but its doable. To me, the main advantage of pre-loading is shorter loading time in the first visit to a page after its expired or cleared. On very active sites, the effect is minimal. On very large sites, especially blogs, where most visits are to the freshest content, pre-loading involves a lot of unnecessary processing. However, for small business sites, improving the user experience on the initial pages load (which is always the longest anyway, because the browser needs to download static resources), this can be the difference between getting a sale (or lead) or not. Long story short, if you can add my request for a pre-loader that runs within WordPress, thatll be excellent. Hi, but its difficult to gauge the results without regular visitor traffic I think it doesnt matter, and as matter of fact, without real traffic it even eases the server load. and without knowing the timing setup this doesnt matter either I think , lets just pretend its 3 users accessed 3 different sites on my server, and on a real shared hosting server , they could have couple dozens if not hundreds real user at any given moment I believe that timings can be tweaked to accommodate the limits of every account and use only the leftover resources from visitor traffic. This may take time, but its doable. yes , it is doable , but that bring another question , imagine a provider that hosts couple hundreds sites , there is no way for them to properly set up each time to stagger the load across different timing technically , the crawler is within wordpress , it just needs to have permission from server to run it. I will pass your opinion as crawler improvement to our developers , they will evaluate and assess it. for the time being , Id suggest to use shell script as workaround. Best regards,
LiteCache Rush: Speed comes from using less, not from doing it faster
Reference