[Rivet] [Fastjet] FastJet "robot" downloads blocked

[Gavin Salam]

Hi Andy,

This is weird -- I know nothing about any robot blocking and we don't 
have a robots.txt file at all. Do you have a small example script to 
illustrate the problem (or is the easiest option to try rivet's 
bootstrap script)? I just checked things with wget and that worked fine. 
But our web servers did get updated (downgraded really) lately, so 
something might have changed relative to a few weeks ago.

Cheers,
Gavin


On 4/19/13 12:42 PM, Andy Buckley wrote:
> Hi Gregory, Gavin, et al,
>
> We've noticed recently that the FastJet website blocks "robot" downloads
> of the FastJet tarball, e.g. http://fastjet.fr/repo/fastjet-3.0.3.tar.gz
>
> Unfortunately this means that the Rivet bootstrap script can fail if it
> tries to download and build FastJet, rather than using the LCG installed
> copy from AFS. We're using Python's urllib2 to do the fetching... is
> there anything weor you can do to not fall foul of this blocking? (I'm
> not sure if urllib2 automatically respects robots.txt files, but if you
> want to give us a special unblocked User-Agent name to use, I'm sure we
> can manage to update our script accordingly)
>
> Thanks!
> Andy & co
>