E.1 Crawler Sits a Long Time On a URL, Then Fails After 9 Minutes

If QuickFinder encounters URLs which it cannot connect to (server is down, DNS servers are down, network is down, TCP/IP is down, etc.) it may hang for up to 3 minutes, then try again two more times, for a total of 9 minutes per failed URL. This situation is especially bad if customers include 1 or 2 levels of “Off-Site” URLs because Web administrators rarely know where all of the off-site links point.

The tcp_syn_retries setting allows admins to control the timeout value when making a connection to a remote site. This sets the maximum number of times initial SYNs for an active TCP connection attempt will be retransmitted. The value should not be higher than 255. The default value is 5, which corresponds to approximately 180 seconds.

Did you catch that, 180 seconds! That’s 3 minutes per attempt. Because QuickFinder automatically retries failed URLs on its own, it is best to set this value to 1, which corresponds to about 15 seconds per attempt.