Cache Warmer Replacement for successful eCommerce Results
Crawls up to 200,000 URLs within 1 hour, even on Shared Hosting
Dynamic Load Control
Cache Vary Support
Cache Inventory
New LScache Cache Varies
Multi-Purge-Feature
Multi-Language Support
Multi-Currency Support
Control Panel Version and Version for Command Line Interface (Cron)
Tune-up Settings for Power Users
True webp Image Support
Journal 3 Support
High Quality "Made in Germany"
Note: Kitt Cache Crawler is not a OpenCart Extension. Kitt Cache Crawler is a standalone Application that runs parallel to OpenCart, so don't try to install it as an Extension in OpenCart.This will fail! Download the .zip File, unzip it locally and read the install notes carefully.
If you are already using the LiteSpeed Cache Extension for OpenCart, then you should know why you need a cache crawler, also known as a cache warmer. However, every cache crawler is not always the same and there are huge differences. This is especially true when comparing the built-in crawler with the Kitt Cache crawler, with the built-in crawler doing very poorly. The built-in one is sufficient if you only want to warmup the cache very rarely or if there are only a few products. The built-in crawler is completely useless for professional use.
With the Kitt Cache Crawler you get everything that many OpenCart users have long wished for. Kitt is not only lightning fast and crawls up to 200,000 URLs within 1 hour even on shared hosting, but also treats your server with care. For example, a dynamic and adaptive load control prevents your server from being overloaded by crawling. You can therefore run Kitt at any time of the day and regardless of how busy the shop is with visitors.
Kitt is not a universal crawler, but is specially adapted to OpenCart and the LiteSpeed LScache. As a specialist for this requirement, Kitt knows no limits. No matter how many products you have, how long it takes to warm up the cache in no time, or you use different languages or currencies, Kitt can do it all!
Your server must meet these requirements
Kitt Cache Crawler is a standalone Application that has to be installed in parallel to your OpenCart Installation. Installation is simple and can be done by everyone. If you were able to install OPenCart, then you can install Kitt as well.
Requirements:
OpenCart 3.x
LiteSpeed Enterprise (LSWS) or OpenLiteSpeed (OLS)
LiteSpeed LScache Extension for OpenCart (LSCOC)
PHP Version > PHP 7.3
cURL
ionCube Loader
/proc/stat must be readable
/proc/cpuinfo must be readable
PHP Function sys_getloadavg must exist
DirectAdmin Path for document_root has to be modified in DA httpd configuration.
All requirements are mandatory! If any of the listed requirements are not met, Kitt will not run.
If you do not know whether all requirements are available on your server, ask your hosting provider. All listed Requirements are normally Standard, even on Shared Hosting Server.
The fastest and safest method to check the requirements is the Kitt Tester Script. Download the tester script, upload it to the root directory of the OpenCart installation and request the tester script in the browser. If all requirements are met, you should see this test result: Test Result
Many Settings but less to configure
Kitt Cache Crawler comes with a long list of settings, but don't worry, Kitt already made most of the settings for you during the installation. In most cases it is sufficient to accept the default settings. There are only a few settings that should interest you. This affects the settings for the load to a large extent, but Kitt also supports you here. All other settings are field-tested settings that you should adopt regardless of the type and performance of your server.
Crawler Settings
Delay
Kitt Cache Crawler is a so-called multi-thread crawler. This means that he does not request the URLs one after the other, but several at the same time. Nevertheless, there is only a short delay between the individual requests, which is specified in microseconds. The default value of 500 us has proven itself and should only be changed in exceptional cases.
Threads
Threads relates directly to how Kitt works. The Threads setting defines the number of simultaneous requests. Again, leave the default value of 3. A higher value does not automatically increase the speed, but also increases the risk of URLs not being cached properly, even with dedicated servers. Kitt automatically adjusts the crawling speed if the web server allows it.
Max Server Load
The Max Server Load setting is probably the most important setting. Contrary to what many users think, the load on a Linux system is not determined solely by the CPU load. To determine the load and what influences it, there are several factors that you do not need to know in detail. Kitt regulates the crawling speed automatically, but it needs a limit that should not be exceeded. This limit is very variable and different from server to server. On a VPS or dedicated server, this limit is usually 1. On shared hosting, there are completely different guide values. For better orientation, you should use the percentage utilization as a guide. A limit of max. 50% should not be exceeded for crawling.
Network Timeout
No matter which Internet call is made, connection problems can always occur. Even Kitt is not immune to this. To avoid waiting for a long connection, the Network Limits setting can be used to determine when Kitt should terminate the connection.
Limit
The Limit setting is used to limit the maximum number of URLs that should be crawled. The limit value and its handling are based directly on MySQL queries.
Offset
Offset setting works in combination with Limit setting and works as known in MySQL.
Logging
If logging is enabled this is needed for the cache inventory. Cache Inventory gives you verbose information about the status cache of each requested URL and each cache vary.
cURL Debugging
cURL debugging is for Experts only. It generates a log file about each request with verbose data about this request.
Crawler Mode
The crawler mode determines whether the cache should be purged before each crawling session. This setting must be disabled when setting an Offset Wet.
Cache Mode
Cache Mode provides another tuning option, but with different cache behavior. We call this the walker or runner mode. The recommended Walker Mode updates the cache of a page by the set TTL value if the cache of the respective page has not yet expired. The fast runner mode is only for cache warmup, but does not update the cache if a page's URL is still cached. The Cache Mode feature is also used to serve the different requirements when using a proxy or the LiteSpeed Web ADC.
Follow Redirection
Kitt is able to follow redirects, although defining redirects is not recommended for SEO reasons. If this is nevertheless necessary, Kitt follows the target address and caches this URL. However, it is not advisable to do this, as this dramatically reduces the crawling speed.
Compression
When enabled, Kitt compresses the content of the cached URL. Activation is recommended for several reasons. On the one hand, this saves considerable storage space. On the other hand, the activation eliminates the need for renewed compression through content encoding or accept encoding, because Kitt already carries out the necessary type of compression. Compression will always speedup any Request!
Mobile Device Cache Vary New Function
Since version 3.0.6, the Kitt Cache Crawler for the LiteSpeed LScache Extension is fully Journal 3.x ready. Since the cache plugin's device detection is inadequate and does not properly support Journal 3, the device detection in the Kitt Cache Crawler has been completely reworked, which also makes the cache plugin's inadequate device detection obsolete. Kitt now also supports device detection for desktop, mobile devices and tablets.
Webp Image Cache Vary New Function
The incorrect and inadequate "Separate View for Safari" feature in the LiteSpeed Cache Extension necessitated a complete overhaul of this feature in the Kitt Cache Crawler. With version 3.0.6 of Kitt, not only has the spectrum of webp browser support detection been expanded and it is no longer limited to the iPhone. In addition, webp detection has been synchronized with Journal 3, so Kitt is already compatible with Journal version 3.2.
Sitemap
Kitt fetches Sitemap to get URLs to crawl.
MultiLanguage/MultiCurrency Selection
Kitt Cache Crawler is the only crawler for LiteSpeed LScache Extension for OpenCart that allows optional language and currency selection. If you use multiple languages and currencies, different content is generated for each selection, which requires a different cache copy for each.
Journal Product Preview
Journal 3 offers a preview function in the product listings, for which a separate URL is generated for each preview. To speed up the display of the preview, the cache can also be warmed up for these URLs. But only if you use Journal 3.
Server IP Address
To warm up the cache, Kitt simulates conventional user requests. Therefore, Kitt is subject to the same network conditions as normal user requests. However, Kitt requires direct access to the host, bypassing a CDN node. It is therefore necessary to enter the public IP address of the origin host.
URLs Most Wanted
The Most Wanted Title of this Feature stands for URLs Most Wanted and is a unique Feature unique to the Kitt Cache Crawler. If this Function is activated, Kitt does not use the Sitemap as the Basis for the URLs to be crawled, but rather the most requested URLs. This reduces the Number of URLs to be crawled by up to 70%, which inevitably leads to a significant Reduction in Server Load.
In addition, the Most Wanted Function also records the URLs that can never be included in the Sitemap. For the first time, the Pagination Pages, Search Result Pages and Filter URLs are also crawled and the cache is warmed up for them.
Crawl Output New Function
When Kitt is run manually in the CLI, Crawl Output can be enabled if necessary. This output shows the currently crawled URL, the current server load and the TTFB in real time.
Cache Inventory
The cache inventory provides you with detailed information about the cache status of each URL. Depending on the cache plugin settings, different cache vary are used for each URL. The cache inventory therefore also shows you the cache status for each URL and the cache vary used. If necessary, you can use the cache inventory to purge the cache per URL or exclude certain URLs from the warmup.
URLs Most Wanted Inventory
The URLs Most Wanted (UMW) function is one of the unique features of Kitt. In the UMW inventory you will find a real-time overview of all URLs currently visited by users, which are used for the cache warmup. The optional UMW function is a very efficient method to warm up the cache of only the URLs that are actually requested. This reduces the effort required for cache warmup by up to 70%. This not only dramatically reduces the time required for the cache warmup, but also requires significantly fewer server resources and therefore a significantly lower server load.
Unique Features for LiteSpeed LScache Extension for OpenCart
Crawling up to 200,000 URLs within 1 hourKitt Cache Crawler can crawl up to 200,000 URLs within 1 hour, even on Shared Hosting. This is up to 20x faster as built-in Crawler.
Dynamic Load ControlKitt Cache Crawler cares about the Load of your Server. A Dynamic Load Control checks the current Load while Crawling. When the load gets too high, Kitt automatically reduces the speed and the number of concurrent requests until a predefined limit is reached. This dynamic load control guarantees that frontend operation is not adversely affected by crawling.
Language and Currency SelectionIn contrast to the built-in crawler, Kitt allows a high degree of flexibility. If more than 1 language or currency is used, these can be optionally selected.
Auto-Adjust ConfigurationKitt Cache Crawler comes with pre-configured Settings, that fits in most cases. So you don't have to learn which settings are the best. Each setting is also provided with recommendations that give you even more security when making any changes.
Cache Vary SupportThe flexible application possibilities of OpenCart usually require more than just 1 cache copy per URL. To meet this requirement, the Kitt Cache Crawler supports the LiteSpeed Cache Vary.
Control Panel Version and Version for Command Line Interface (CLI) Each Kitt Cache Crawler version can be run in at least 2 ways. Kitt can be run from within the Control Panel, but also for use in the Command Line Interface. The CLI version is therefore also suitable for use as a cron job.
Tune-up SettingsA wealth of tuning settings are available for experts to make Kitt even faster.
Cache InventoryThe cache inventory optionally gives a detailed report on the cache status of each URL. This report also includes information about Cache Vary's cache status.
Crawling without TimeoutThe LiteSpeed web server allows scripts to be run without time limits, so the control panel version of Kitt is not prematurely terminated by PHP limits. This no limit feature is also available on shared hosting.
Journal 3 ReadyJournal 3 has several different features from OpenCart. In order for these features to be considered for caching, a specially adapted crawler is needed, which is not only adapted for LiteSpeed LScache, but also supports the additional features of Journal 3.
Kitt Cache Crawler vs Built-in Crawler
Features
Built-in Crawler
Kitt Cache Crawler
Request Method
Serial
MultiThread
Crawling Speed
<10,000/24h
<200,000/h
Delay
Threads
1
unlimited
Dynamic Load Control
Cache Inventory
Network Timeout
Limit
Offset
Logging
Curl Debug
Purge Method
fixed
variable
Redirection
Always Follow
on/off
Cache Compression
Fixed
identity, gzip, deflate, br
Cache Vary
buggy
extended
Webp Support
MultiLanguage Support
fixed
adjustable
MultiCurrency Support
fixed
adjustable
Journal 3 Product Preview
adjustable
Unblockable Cache Crawler
Crawling a page causes load. If a crawler does not take into account the current load on the server, it can quickly become overloaded. This is especially true in the case of the built-in crawler. The built-in crawler is linked to the server settings so that Internet providers, and especially with shared hosting, have the opportunity to limit the load. This also means that the execution of the built-in crawler can be prevented by the provider. However, this does not apply to Kitt Cache Crawler!
Kitt Cache Crawler is free, free from every limitation, because Kitt neither can't be detected nor overloads your server. You can run Kitt at every daytime and he will never influence your frontend.
Still Questions?
We try very hard to give you as much information as possible. If you can't find the information you need or if you don't understand something, don't hesitate to ask us.
