What is Max Server Load Feature?
In addition to the high crawl speed of the Kitt Cache crawler, the Max Server Load feature is the most important function. The automated crawling of a website,regardless of the purpose, is generally not uncritical, since without appropriate control the server can quickly become overloaded and through crawling can affectthe performance of the front end. In order to prevent this, a load control is needed that either stops crawling or, better still, reduces the crawl speed if a guidelinevalue is exceeded. In any case, it must be ensured that there is no overload and that front-end operation is not impaired by the crawling.
Compared to the built-in crawler of the Cache Plugin for Wordpress, this crawler works very similar to the Kitt Cache Crawler. The built-in crawler of this cache pluginalso checks the maximum load based on the number of CPU cores, but with a serious limitation. The build-in crawler only knows the value 1 regardless of how many cores theCPU has, which makes identifying the CPU and its number of cores meaningless. This can be explained by the lack of this crawler, that it completely ignores the type of hosting,because the value 1 is only a guideline for dedicated servers and some VPS. Completely different values result with shared hosting, so that the attempt to control the loadwith shared hosting always fails, or the number of simultaneous requests (threads) is reduced to the lowest value of 1.
However, this misbehavior only shows up with the built-in crawler of the cache plugin for Wordpress. All other LiteSpeed crawlers have no load control function at all.
The load control of the Kitt Cache Crawler is unique in the way it works. This way of working is dynamic and adaptive. This means that while crawling, Kitt measures thecurrent load status in almost real time and, if necessary, reduces the crawl speed if the limit value has been exceeded and dynamically increases the crawl speed again,but does not allow the limit value to be exceeded.
The information requested by the operating system is primarily used to determine the limit value, but this information is always relative and depends significantly, but notonly, on the performance of the CPU. However, a lot of other data flows into the load state, which makes it impossible, especially for the layman, to find the criticallimit value for his server. In order to simplify this, the Kitt Cache Crawler also provides a percentage load evaluation. Although this is less accurate than the datareturned by the OS, it allows the guide value to be determined to a sufficient extent.