Kitt Cache Warmup Crawler for Wordpress W3 Total Cache Plugin (W3TC) "plaid" Edition

Ask A Question

Start » Download » Cache Warmup Crawler Wordpress W3 Total Cache Plugin
Kitt Cache Warmup Crawler for Wordpress W3 Total Cache Plugin (W3TC) plaid Edition

Cache Warmer Replacement for Popular Cache Plugin


  • Up to 50-times faster as built-in Crawler
  • Dynamic Load Control
  • Post Type Selection (no WooCommerce Support)
  • Available as Desktop Version and Version for Command Line Interface (Cron)
  • Tuneup Settings for Power Users
  • Cache Inventory
  • Kitt Cache Crawler for W3 Total Cache is available for Download, but further Development has been discontinued
Note: Kitt Cache Crawler is not a WordPress Plugin. Kitt Cache Crawler is a standalone Application that runs parallel to WordPress, so don't try to install it as a Plugin in WordPress. This will fail! Download the .zip File, unzip it locally and read the install notes carefully.

HighSpeed Cache Crawler for W3 Total Cache

The W3 Total Cache Plugin for WordPress is one of the most popular cache plugins. It speeds up page views tremendously. As with any other page cache, you only benefit from a page cache if each URL has been requested at least once beforehand. Depending on the number of posts, this process can take a long time if you leave the requests to the users. To speed up this process, the W3 Total Cache Plugin enables the so-called warmup of the cache. Even if this warmup does its job properly, it is very slow. If you change or add new posts in a short time, this built-in crawler can hardly keep up with the crawling. Especially since this built-in crawler allows only little flexibility. As good as the W3 Total Cache plugin is, it severely limits the quality and benefits of this plugin.

With Kitt Cache Crawler, the W3 Total Cache Plugin becomes a fully-fledged Page Cache Plugin for Wordpress. Although the exorbitant speed is in the foreground with Kitt, Kitt also impresses with its adaptive, flexible and dynamic properties, which are particularly important when it comes to load control. When the built-in crawler of the W3 Total Cache and every other 3rd party crawler works regardless of the resulting load, Kitt handles it very carefully and is also up to 50 times faster. With Kitt CAche Crawler, congestion can never occur, so the cache can be warmed up regardless of the time of day and the amount of views by visitors.

Requirements to run Kitt Cache Crawler for W3 Total Cache

Kitt Cache Crawler is a standalone Application that has to be installed in parallel to your WordPress Installation. Installation is simple and can be done by everyone. If you were able to install WordPress, then you can install Kitt as well.

Requirements:

  • WordPress
  • Apache web server (nginx and LiteSpeed (LSWS/OLS) are unsupported)
  • W3 Total Cache Plugin
  • YOAST Seo Plugin for WordPress (optional)
    If YOAST Seo Plugin isn't installed URLs to be crawled cannot be filtered by Post Types.
  • PHP Version >= PHP 7.4 & < 8.2
  • cURL
  • ionCube Loader (Version >11)
  • /proc/stat must be readable
  • /proc/cpuinfo must be readable
  • PHP Function sys_getloadavg must exist
  • DirectAdmin is unsupported!!!

All requirements are mandatory! If any of the listed requirements are not met, Kitt will not run.
If you do not know whether all requirements are available on your server, ask your hosting provider. All listed Requirements are normally Standard, even on Shared Hosting Server.

The fastest and safest method to check the requirements is the Kitt Tester Script. Download the tester script, upload it to the root directory of the WordPress installation and request the tester script in the browser.

Get Maxx Cache Power with Kitt Cache Crawler

Kitt is unique in every respect. This uniqueness turns the W3 Total Cache Plugin into a fully-fledged cache plugin. You can adapt Kitt to your personal needs with a variety of settings. The Kitt Control Panel supports you and provides you with many presettings during the installation, so you don't necessarily have to be an expert.


Crawler Settings


Delay
Kitt Cache Crawler - Delay
Kitt is a so-called multi-thread crawler. This means that it does not request URLs in sequence like the built-in crawler, but several at the same time, depending on the setting. However, it needs a short delay in microseconds. 1second = 1,000,000 microseconds
Threads
Kitt Cache Crawler - Threads
Threads is directly related to Delay and has a significant impact on crawling speed. The more simultaneous threads, the more URLs are requested at the same time. Threads have a direct impact on the server load. Therefore, the default value should also be used for dedicated servers and VPS, since the higher the threads, the higher the failure rate.
Max Server Load
Kitt Cache Crawler - Max Server Load
The Max. Server Load Control feature is a core element of Kitt Cache Crawler. This feature not only serves to display the current load, but also controls the load during crawling and adjusts the speed accordingly.
Network Timeout
Kitt Cache Crawler - Network Timeout
With all Internet requests, network disruptions can always occur, which sometimes lead to severe delays. In order to prevent an excessively long waiting time and not unnecessarily delay the crawling, the maximum waiting time can be determined with Network Timeout before Kitt aborts the request.
Limit
Kitt Cache Crawler - Limit
LIMIT sets the maximum number of URLs to be crawled. LIMIT corresponds to the limitation that can also be applied to MySQL queries.
Offset
Kitt Cache Crawler - Offset
With Offset you can start crawling at specific row.
Logging
Kitt Cache Crawler - Logging
If Logging is enabled Kitt writes the result of each request to the cache inventory where you can check the cache status of each URL and its cache vary.
cURL Debugging
Kitt Cache Crawler - cURL Debugging
cURL debugging is for experts only and writes verbose information about each cURL request to a log file.
Crawler Mode
Kitt Cache Crawler - Crawler Mode
With Crawler Mode you can enable to purge the cache before Kitt starts crawling.
Follow Redirection
Kitt Cache Crawler - Follow Redirection
Kitt can be configured to follow redirections so that the redirected URL will be cached. Forwarding should only be activated in exceptional cases, as this dramatically increases the crawl time.
Server IP Address
Kitt Cache Crawler - Follow Redirection
To warm up the cache, Kitt simulates conventional user requests. Therefore, Kitt is subject to the same network conditions as normal user requests. However, Kitt requires direct access to the host, bypassing a CDN node. It is therefore necessary to enter the public IP address of the origin host.
Crawl Output
Kitt Cache Crawler - Follow Redirection
If Kitt is run manually in the console, the output for it can be activated if necessary. This will display the currently requested URLs, the transfer time and the current server load for each URL. However, activating the output function should not generally be activated as this can have a negative impact on the crawl speed.
Post Type Selection
Kitt Cache Crawler - Post Type Selection
With the support of the YOAST Seo plugin, Kitt enables the selection of certain post types, so that only these post types are crawled and thus the cache of these pages is warmed up.
URLs Most Wanted
Kitt Cache Crawler - Server IP Address
The URLs Most Wanted (UMW) feature is a unique feature that is only available with the Kitt Cache Crawler. The UMW function is an alternative to the sitemap, which usually provides the basis for the URLs to be crawled. As the name of the UMW function suggests, this function records the URLs requested by natural users. The unique advantage in this feature is that it also records the URLs that are not normally included in the sitemap and therefore cannot warm up the cache for these URLs. The further advantage is that up to 70% of the total existing URLs do not have to be crawled, because up to 70% of all URLs in the site size are either never or only very rarely requested. So why waste resources crawling URLs if no one gets any use out of it?
Sitemaps
Kitt Cache Crawler - Post Type Selection
If the YOAST plugin is not installed, then at least 1 sitemap file must be specified. Kitt supports both traditional sitemaps and index sitemaps.
Cache Inventory
Kitt Cache Crawler - Cache Inventory
The cache inventory provides you with detailed information about the cache status of each URL. Depending on the cache plugin settings, different cache vary are used for each URL. The cache inventory therefore also shows you the cache status for each URL and the cache vary used. If necessary, you can use the cache inventory to purge the cache per URL or exclude certain URLs from the warmup.
URLs Most Wanted Inventory
Kitt Cache Crawler - URLs Most Wanted Inventory
The URLs Most Wanted (UMW) function is one of the unique features of Kitt. In the UMW inventory you will find a real-time overview of all URLs currently visited by users, which are used for the cache warmup. The optional UMW function is a very efficient method to warm up the cache of only the URLs that are actually requested. This reduces the effort required for cache warmup by up to 70%. This not only dramatically reduces the time required for the cache warmup, but also requires significantly fewer server resources and therefore a significantly lower server load.

Unique Features for W3 Total Cache Plugin

  • Crawling up to 200,000 URLs within 1 hour Kitt Cache Crawler can crawl up to 200,000 URLs within 1 hour, even on Shared Hosting. This is up to 20x faster as built-in Crawler.
  • Dynamic Load Control Kitt Cache Crawler cares about the Load of your Server. A Dynamic Load Control checks the current Load while Crawling. When the load gets too high, Kitt automatically reduces the speed and the number of concurrent requests until a predefined limit is reached. This dynamic load control guarantees that frontend operation is not adversely affected by crawling.
  • Auto-Adjust Configuration Kitt Cache Crawler comes with pre-configured Settings, that fits in most cases. So you don't have to learn which settings are the best. Each setting is also provided with recommendations that give you even more security when making any changes.
  • Control Panel Version and Version for Command Line Interface (CLI) Each Kitt Cache Crawler version can be run in at least 2 ways. Kitt can be run from within the Control Panel, but also for use in the Command Line Interface. The CLI version is therefore also suitable for use as a cron job.
  • Tune-up Settings A wealth of tuning settings are available for experts to make Kitt even faster.
  • Cache Inventory The cache inventory optionally gives a detailed report on the cache status of each URL. This report also includes information about Cache Vary's cache status.
  • URLs Most Wanted URLs Most Wanted reduces cache warmup time and resources by up to 70% and dramatically offloads the cache warmup server.

Kitt Cache Crawler vs Built-in Crawler

Features
Built-in Crawler
Kitt Cache Crawler
Request Method
Serial
MultiThread
Crawling Speed
<5,000/24h
<200,000/h
Delay
Threads
1
unlimited
Dynamic Load Control
Cache Inventory
Network Timeout
Limit
Offset
Logging
Curl Debug
Redirection
Always Follow
on/off

Still Questions?

We try very hard to give you as much information as possible. If you can't find the information you need or if you don't understand something, don't hesitate to ask us.
Ask A Question

More Kitt Cache Crawler Versions

Kitt Cache Warmup Crawler for Shopware 6 (LCSW) Kitt Cache Warmup Crawler for Shopware 6 (LCSW)
Kitt Cache Warmup Crawler for WP WooCommerce & LiteSpeed LScache Plugin (LSCWPWC) "plaid" Edition Kitt Cache Warmup Crawler for WP WooCommerce & LiteSpeed LScache Plugin (LSCWPWC) plaid Edition
Kitt Cache Warmup Crawler for LiteSpeed LScache PrestaShop Plugin (LSCPS) Kitt Cache Warmup Crawler for LiteSpeed LScache PrestaShop Plugin (LSCPS)
Kitt Cache Warmup Crawler for LiteSpeed LScache OpenCart Extension (LSCOC) Kitt Cache Warmup Crawler for LiteSpeed LScache OpenCart Extension (LSCOC)
Kitt Cache Warmup Crawler for LiteSpeed LScache Wordpress Plugin (LSCWP) "plaid" Edition Kitt Cache Warmup Crawler for LiteSpeed LScache Wordpress Plugin (LSCWP) plaid Edition
Kitt Cache Warmup Crawler for WooCommerce W3 Total Cache Plugin (W3TCWC) "plaid" Edition Kitt Cache Warmup Crawler for WooCommerce W3 Total Cache Plugin (W3TCWC) plaid Edition
LiteCache & Kitt Cache Crawler for Magento 2 LiteCache & Kitt Cache Crawler for Magento 2