Kitt Cache Warmup Crawler for Shopware 6 (LCSW)

With the release of version 6 of Shopware, a lot has changed in terms of cache. With version 5 you could still set the cache behavior reasonably flexibly, in version 6 there is only the option of either caching everything or nothing. This also applies to the warmup of the cache. There is no way to warm up the cache specifically for a specific URL or URL group.

The Kitt Cache Crawler offers a way out of this dilemma. Kitt is a standalone application that is installed in parallel to Shopware 6 and works completely autark from it, but Kitt is developed especially for the needs of Shopware 6.

Kitt has numerous features that make the warmup of the Shopware cache much faster and has dynamic server load management. The latter ensures that the server is not overloaded by the crawling and the shop operation is not affected by this. The Kitt Cache Crawler can therefore be run independently of the respective visitor volume.

The Kitt Cache Crawler also enables flexible crawling, which can be used, for example, to select which route names should be crawled or which products and categories should be crawled from a specific release or update date.

All in all, the Kitt Cache Crawler has numerous setting options to flexibly adapt the crawl behavior to individual needs.

Kitt Cache Crawler is a standalone Application that has to be installed in parallel to your Shopware Installation. Installation is simple and can be done by everyone. If you were able to install Shopware, then you can install Kitt as well.

Requirements:

• Shopware 6
• Apache Web Server (All other web server are unsupported)
• PHP Version >= PHP 7.4 & < 8.2
• cURL
• ionCube Loader Version > 11
• /proc/stat must be readable
• /proc/cpuinfo must be readable
• PHP Function sys_getloadavg must exist
• DirectAdmin is unsupported!!!

Kitt Cache Crawler has numerous setting options that allow you to adapt the cache warmup to your personal needs, but also to the available server resources. However, KITT is already preconfigured during installation, so it is usually sufficient to accept these default settings. The default settings are suitable for both shared hosting and dedicated servers, so you don't have to spend time trying to figure out which setting is best.

Each Kitt Cache crawler version contains 2 different crawler versions that work the same way. Kitt can be run as a GUI version within the control panel, but can also be set up as a cron job. Both versions use the same configuration.

You should pay special attention to the server load control. Pretty much every crawler available on the market for the cache warmup crawls URLs regardless of the resulting server load, which can sometimes be significant. As a result, this increased load can affect shop operations. In order to mitigate this circumstance and not to affect shop operations too much during the warmup, administrators usually run the cache warmup at night. This restriction does not exist with the Kitt Cache Crawler, since it not only has a load control, but also automatically adapts the crawl behavior to the current load status. It can therefore never come to an overload.

• Multithread Process Mode The multithreading process mode allows multiple requests to be executed simultaneously and makes crawling lightning fast without overloading the server.
• Crawling up to 200,000 URLs within 1 hour Kitt Cache Crawler can crawl up to 200,000 URLs within 1 hour, even on Shared Hosting. This is up to 20x faster as built-in Crawler.
• Dynamic Load Control Kitt Cache Crawler cares about the Load of your Server. A Dynamic Load Control checks the current Load while Crawling. When the load gets too high, Kitt automatically reduces the speed and the number of concurrent requests until a predefined limit is reached. This dynamic load control guarantees that frontend operation is not adversely affected by crawling.
• Auto-Adjust Configuration Kitt Cache Crawler comes with pre-configured Settings, that fits in most cases. So you don't have to learn which settings are the best. Each setting is also provided with recommendations that give you even more security when making any changes.
• Control Panel Version and Version for Command Line Interface (CLI) Each Kitt Cache Crawler version can be run in at least 2 ways. Kitt can be run from within the Control Panel, but also for use in the Command Line Interface. The CLI version is therefore also suitable for use as a cron job.
• Tune-up Settings A wealth of tuning settings are available for experts to make Kitt even faster.
• Cache Inventory The cache inventory optionally gives a detailed report on the cache status of each URL. This report also includes information about Cache Vary's cache status.
• Route Name Selection Route Name Selection allows to filter and to limit which kind of Routes should be crawled. This also includes a Filter to limit the to be crawled URLs by Days.

The Most Wanted Title of this Feature stands for URLs Most Wanted and is a unique Feature unique to the Kitt Cache Crawler. If this Function is activated, Kitt does not use the Sitemap as the Basis for the URLs to be crawled, but rather the most requested URLs. This reduces the Number of URLs to be crawled by up to 70%, which inevitably leads to a significant Reduction in Server Load.

In addition, the Most Wanted Function also records the URLs that can never be included in the Sitemap. For the first time, the Pagination Pages, Search Result Pages and Filter URLs are also crawled and the cache is warmed up for them.