Frequently Asked Question about Kitt

Kitt is an application that automatically requests all URLs on a website. This process is also known as crawling. However, Kitt does not index any content like a search engine crawler, but leaves it with a request. This crawling serves the sole purpose of warming up the page cache. Thanks to the multithreading mode, Kitt is able to request up to 200,000 URLs within 1 hour without overloading the server. However, Kitt is not a universal crawler like almost every other external crawler or crawler service. On the one hand, Kitt imitates a natural user, which is particularly important, but not only important for LiteSpeed LScache. On the other hand, each Kitt Cache Crawler version is individually adapted to the respective page cache of an application, because not all page caches are the same.

Kitt is neither a plugin nor an extension. Kitt is an independently functioning application that is installed parallel to the respective main application.

No, you can only use Kitt to crawl the URLs on your own host (domain). Each domain requires its own Kitt Cache Crawler installation.

The Kitt Cache Crawler is available for various applications and is not limited to a specific page cache or application.

The answer is simple. Without warming up the Cache with a Crawler each URL must be requested first by User. Depending on number of daily users it can take a lot of time until all URLs are cached.

Yes and No. This depends on Application, Cache Plugin or how the Cache works.

Yes, it has, but for this feature an Application must be installed on a Server where LiteSpeed Web Server is running. However, the well-known timeout limit is limited to PHP scripts that are executed outside of the command line interface. Even if you don't run a LiteSpeed web server, there is no risk of a timeout when using an Apache web server, since you can also run the Kitt Cache Crawler in the CLI. In the Command Line Interface there is usually no limit to the script execution time.

There are 2 main reasons for this. In the case of external crawlers, which are mostly only universal crawlers, they do not receive any feedback from the server about the respective load status. In order not to overload the server, the crawl interval must be very low. This is why external crawlers are always very slow.

The second and even more important reason is that Kitt has an adaptive and dynamic load control that checks the load state in real time while crawling. If this exceeds a limit, Kitt automatically reduces the crawl speed and increases it again if the limit is undershot. As a result, there can never be an overload. The presumably decisive advantage is that Kitt can be run at any time of the day because the frontend operation is not affected by the automatic load management.

The Requirements depend on which Version for which Application is used, because each Crawler Version is different. That's why the requirements are different. The respective requirements are listed in the description of each Kitt Cache Crawler version. There you will also find a test program that you can download with which you can find out in a matter of seconds whether the server meets the necessary requirements.

No, but we don't want to generally rule out the possibility that there may also be crawler versions in the future that are multisite ready.