The Missing Link to complete LiteSpeed PrestaShop Cache Plugin
Crawls up to 200,000 URLs within 1 hour, even on Shared Hosting
Dynamic Load Control
Cache Vary Support for Guest Mode, Webp Replacement and Mobile View
GUI Version and Version for Command Line Interface (Cron)
Tuneup Settings for Power Users
True webp Image Support
Multi-Purge-Feature
Anti-Block Protection
High Quality "Made in Germany"
Note: Kitt Cache Crawler is not a PrestaShop Module. Kitt Cache Crawler is a standalone Application that runs parallel to PrestaShop, so don't try to install it as an Module in PrestaShop.This will fail! Download the .zip File, unzip it locally and read the install notes carefully.
Kitt Cache Crawler - The missing Link for PrestaShop LScache Plugin
A page cache like that of LiteSpeed LScache speeds up any PrestaShop many times over. The typical problem of any cache is that you can only use its advantages when the source to be cached is in the cache. If the items in a PrestaShop change too often, the cache must be purged with each update. As a result, you can only enjoy the advantages of a page cache to a very limited extent. In order to counteract this well-known fact, one is forced to warm up the cache. However, further problems arise from this.
As is well known, the LiteSpeed LScache offers so-called Cache Varies. Cache Varies enable the caching of an unlimited number of copies of the cache if the content changes under the same URL. A cache crawler that warms up the cache must therefore be able to mimic the different conditions for the cache varieties. This feature excludes any universal cache crawler that does not support LiteSpeed LScache.
A cache crawler is optionally available for the LScache plugin for PrestaShop, which allows crawling, but only to a very limited extent, as it does not support all the required cache varies. In addition, this is comparatively slow and the execution can be blocked by the Internet provider.
The Kitt Cache Crawler is free from any restrictions. Not only is it lightning fast, it supports all required cache varies and has dynamic load control, so that crawling never overloads the frontend. Also, Kitt cannot be blocked. In short, Kitt is a cache warmup specialist for the PrestaShop LiteSpeed LScache plugin.
Requirements to run Kitt Cache Crawler for PrestaShop LScache Plugin
The Kitt Cache Crawler is a self-sufficient application that works parallel to PrestaShop and independently of this and the LScache plugin, but was specially developed for it. The Kitt Cache Crawler version for the LiteSpeed LScache plugin can therefore only be used for this.
Kitt Cache Crawler is a standalone Application that has to be installed in parallel to your PrestaShop Installation. Installation is simple and can be done by everyone. If you were able to install PrestaShop, then you can install Kitt as well.
Requirements:
PrestaShop version >= 8.x
LiteSpeed web server (LSWS)
LiteSpeed LScache Plugin for PrestaShop
PHP Version >= PHP 8.1 & <= 8.3
cURL
ionCube Loader Version > 11
/proc/stat must be readable
/proc/cpuinfo must be readable
PHP Function sys_getloadavg must exist
DirectAdmin Path for document_root has to be modified in DA httpd configuration.
All requirements are mandatory! If any of the listed requirements are not met, Kitt will not run.
If you do not know whether all requirements are available on your server, ask your hosting provider. All listed Requirements are normally Standard, even on Shared Hosting Server.
The fastest and safest method to check the requirements is the Kitt Tester Script. Download the tester script, upload it to the root directory of the PrestaShop installation and request the tester script in the browser. If all requirements are met, you should see this test result: Test Result
Many Settings - Less to configure
The Kitt Cache Crawler allows you to adapt the crawling to individual requirements with numerous settings. However, Kitt is already preconfigured during installation, so that in most cases it is sufficient to simply accept these default settings. You don't have to be an expert or spend a lot of time learning how to use it.
Crawler Settings
Delay
Kitt is a so-called multi-thread crawler. This means that Kitt does not make the URL requests for crawling one at a time, but makes several requests at the same time. This multi-thread mode enables much faster crawling compared to the LiteSpeed Cache Crawler Script. For trouble-free operation, the MultiThread mode requires a short delay between the individual requests, which is defined in microseconds. 1 second = 1,000,000 microseconds.
Threads
The thread setting largely determines the crawling speed and defines the number of requests to be executed at the same time. While there is no limit to this setting, Kitt is designed not to attempt a new request until all previous requests have passed without errors. A value that is set too high does not necessarily lead to faster crawling. It is therefore advisable to leave the default value even when using a dedicated server.
Max. Server Load Management
The automated crawling of a page sometimes creates a high server load. Conventional crawlers and the LiteSpeed Crawler Script do not take this load into account. This often leads to the front end and thus the shop operation being significantly impaired by the crawling. In order not to affect the shop operation during the main visiting hours, PrestaShop admins are forced to move the warm-up process to the night hours.
With the adaptive server load management, Kitt continuously checks the current server load during loading. If this reaches a limit, Kitt automatically reduces the crawling speed and increases it again when the threshold is undershot. With putty, you no longer have to worry about crawling overloading your server.
Network Timeout
It is typical of the Internet that network disruptions can occur at one point or another. Depending on the server settings, this leads to unnecessarily long waiting times and can dramatically increase the total crawl time. To prevent this, the network timeout function can be used to set the maximum waiting time before a request is aborted.
Limit
The Limit and Offset settings are used to limit the amount of URLs to be crawled. Offset allows you to set the start position. Limit and offset are based on the usage of MySQL.
Offset
See Limit.
Logging
With activated logging you get detailed information about the current cache status via the cache inventory, but not only about each individual URL, but also about the used cache vary.
cURL Debugging
If you want to debug the individual requests, cURL debugging enables verbose information.
Crawler Mode
If the crawler mode setting is enabled, the entire cache is purged before crawling.
Cache Mode
Cache Mode provides another tuning option, but with different cache behavior. We call this the walker or runner mode. The recommended Walker Mode updates the cache of a page by the set TTL value if the cache of the respective page has not yet expired. The fast runner mode is only for cache warmup, but does not update the cache if a page's URL is still cached. The Cache Mode feature is also used to serve the different requirements when using a proxy or the LiteSpeed Web ADC.
Server IP Address
To warm up the cache, Kitt simulates conventional user requests. Therefore, Kitt is subject to the same network conditions as normal user requests. However, Kitt requires direct access to the host, bypassing a CDN node. It is therefore necessary to enter the public IP address of the origin host.
Seperate Mobile View
Kitt Cache Crawler supports the Cache Varies used by the LScache Plugin for PrestaShop. If the function for separate mobile view or for the guest mode has been activated in the plugin settings, Kitt detects this setting automatically.
Guestmode
See Seperate Mobile View
webp Replacement
If you have installed a plugin that allows alternative use of webp images, Kitt also supports this feature.
Follow Redirection
Normally, there should be no forwarding or redirection on a website, because this is bad for SEO reasons alone. If a redirect is necessary, Kitt can be configured so that Kitt follows a forwarding address and in this way this URL is crawled and thus cached. However, following a redirect should only be activated in exceptional cases, because this can dramatically decrease the crawl time.
Compression
Kitt can compress the cached static files if needed, reducing disk space significantly. However, this compression not only serves to reduce the disk space, but also corresponds to the server-side content encoding. When the data is delivered to the client, it does not have to be compressed beforehand because it is already compressed, which improves the loading time.
URLs Most Wanted
The URLs Most Wanted (UMW) feature is a unique feature that is only available with the Kitt Cache Crawler. The UMW function is an alternative to the sitemap, which usually provides the basis for the URLs to be crawled. As the name of the UMW function suggests, this function records the URLs requested by natural users. The unique advantage in this feature is that it also records the URLs that are not normally included in the sitemap and therefore cannot warm up the cache for these URLs. The further advantage is that up to 70% of the total existing URLs do not have to be crawled, because up to 70% of all URLs in the site size are either never or only very rarely requested. So why waste resources crawling URLs if no one gets any use out of it?
Sitemap
Kitt Cache Crawler uses URLs from Sitemap for Crawling.
Cache Inventory
The cache inventory provides you with detailed information about the cache status of each URL. Depending on the cache plugin settings, different cache vary are used for each URL. The cache inventory therefore also shows you the cache status for each URL and the cache vary used. If necessary, you can use the cache inventory to purge the cache per URL or exclude certain URLs from the warmup.
URLs Most Wanted Inventory
The URLs Most Wanted (UMW) function is one of the unique features of Kitt. In the UMW inventory you will find a real-time overview of all URLs currently visited by users, which are used for the cache warmup. The optional UMW function is a very efficient method to warm up the cache of only the URLs that are actually requested. This reduces the effort required for cache warmup by up to 70%. This not only dramatically reduces the time required for the cache warmup, but also requires significantly fewer server resources and therefore a significantly lower server load.
Unique Features
Crawls up to 200,000 URLs within 1 hourDepending on current Cache Status Kitt is able to warmup up to 200,000 URLs within 1 hour without heavy Load, even on Shared Hosting. This is up to 10x faster as built-in Crawler.
Dynamic Load ControlDon't care about the Load of your Server while Crawling. Kitt has a intelligent Load Control that dynamically reduces the Crawling Speed if the Load goes too high and encreases the Speed if the Load goes under the Limit. In Comparison to built-in Crawler this Load Control works and not only cancels Crawling if the Load goes too high.
Auto-Adjust ConfigurationKitt Cache Crawler comes with pre-configured Settings, that fits in most cases. So you don't have to learn which settings are the best. Each setting is also provided with recommendations that give you even more security when making any changes.
Cache Vary SupportKitt Cache Crawler supports LSCWP Cache Varies. So if you have activated the Mobile View, webp Replacement or the Guest Mode, Kitt warms up the different Caches. However, Kitt does not use up to 8 different Crawlers for this, but does this Job at the same Time with only 1 Crawler.
Control Panel Version and Version for Command Line Interface (CLI)Kitt contains 2 Versions. You can run Kitt either via the Control Panel or scheduled and regularly via a Cron Job.
Tune-up SettingsKitt Cache Crawler is extremely flexible, so you can make numerous Fine-Tuning and Tuning Settings depending on the System Resources.
Cache InventoryThe Cache Inventory gives you detailed Information about the current Status of the Cache, which is automatically generated after each Crawling Session.
Crawling without TimeoutKitt is able to remove the known Limitations on the maximum Execution Time of Scripts without having to make any Changes to the Server Configuration. This also applies to Shared Hosting! If you use Control Panel Version, this Feature ensures that Kitt can complete his Job successfully.
URLs Most WantedURLs Most Wanted reduces cache warmup time and resources by up to 70% and dramatically offloads the cache warmup server.
Kitt Cache Crawler vs LiteSpeed Crawler Script
Features
Built-in Crawler
Kitt Cache Crawler
Request Method
Serial
MultiThread
Crawling Speed
<10,000/24h
<200,000/h
Delay
Threads
1
unlimited
Dynamic Load Control
Cache Inventory
Network Timeout
fixed
variable
Limit
Offset
Logging
Curl Debug
Purge Method
Redirection
on/off
Cache Compression
identity, gzip, deflate, br
Cache Vary
buggy
extended
True Webp Support
Block Protection
Unblockable Cache Crawler
Crawling a page causes load. If a crawler does not take into account the current load on the server, it can quickly become overloaded. This is especially true in the case of the LiteSpeed Crawler Script. This crawler is linked to the server settings so that Internet providers, and especially with shared hosting, have the opportunity to limit the load. This also means that the execution of the LiteSpeed Crawler Script can be prevented by the provider. However, this does not apply to Kitt Cache Crawler!
Kitt Cache Crawler is free, free from every limitation, because Kitt neither can't be detected nor overloads your server. You can run Kitt at every daytime and he will never influence your frontend.
Still Questions?
We try very hard to give you as much information as possible. If you can't find the information you need or if you don't understand something, don't hesitate to ask us.
