Timeout when site is crawled


Hi There,

Before you read this, we're currently using the latest version of IIRF 2.1

We have been using IIRF (version one) for a while now, and recently changed the URL structure of our website. The old URL structure was redirected via regex commands and this worked well, however we also had to add about 100 manual redirection URLS to the isapirewrite4.ini file.

Our SEO then noticed that when he tried to crawl our site using XENU, the crawler picked up "timeout" on many URLs that were working. Through trial and error we isolated the problem to the IIRF rewriter. Removing the entries we added from the ini file leaving just the regex stuff (reduced the ini file from 32k to 6k) then made the IIRF work again.

Thinking this was a bug in IIRF we upgraded to 2.1 but it's still doing the same thing. Provided the IIRF.ini file remains small it works, if the IIRF.ini file grows (in this case to 32k) it causes crawler timeouts again.

Are you able to help us resolve this?



NickChristou wrote Dec 13, 2013 at 5:51 PM

Think I found the issue,
RedirectRule ^/loldpage.asp /newlocation/oldpage.asp [R=301,U]

The [R=301,U] seems to be causing the issue, it may have been a typo our end in the list, but just [R=301] resolved the issue, I think!

If you don't hear from me again, it has been fixed, but you may want to trap this at the code level in case others make the same typo, just ignore the additional parameter.

You're welcome! =)

NickChristou wrote Dec 13, 2013 at 5:52 PM

No, that wasn't the problem :( Can you help me!!!?