What is Delay Feature?
The Kitt Cache Crawler is one of the few and is probably the only multithread cache warmup crawler that requests URLs in parallel rather than serially. The resulting differences are striking.If the serial method is used to request the URLs in sequence, i.e. one after the other, this happens in parallel in multithread mode, so that theoretically an infinite number of simultaneousrequests can be made. This inevitably results in enormously faster crawling compared to the serial method. In practice, the number of simultaneous requests is limited by the available serverresources. Another system-related limitation is that new requests are only made when the previous ones have been successfully completed. This limitation thus prevents too many requests thathave not yet been completed from being created, which would result in an overflow of the required resources. However, this form of limitation does not represent exclusive overload protection,since the actual server load is not taken into account.
Irrespective of this, a delay between the individual parallel requests must be defined for multithread crawling. This delay time is specified in microseconds. 1 second is therefore equal to1,000,000 microseconds or 0.000001 seconds. The standard value of 500 microseconds has proven to be the best value after many tests and can be used unconditionally for any type of hosting fromshared hosting to dedicated servers.
We recommend that you keep this default value!