[Rivet] [Fastjet] FastJet "robot" downloads blocked

[Frank Siegert]

Hi Gavin, all,

Thanks for looking into it. I have also not noticed any problems with
wget, only with the Python library used in Rivet's bootstrap script
(urllib2). One gets a 403 error as reply -- I thought this might
happen to deny access to (Python) bots. I have tried to work around
this by specifying the User-Agent header in the urllib2 request, but I
still got a 403.

I'm preparing a minimal script to reproduce this, will send it to you
in a few minutes. The problem I had was that the rejection seemed to
be dynamic, it worked one day, but not the other. Since we had several
users report the same issue in the last weeks we thought it probably
makes sense to contact you.

Matteo: If you are around for the afternoon tutorials at MC4BSM we
could also have a quick look at this together.

Cheers,
Frank


On 19 April 2013 14:03, Gavin Salam <gavin.salam at cern.ch> wrote:
> Hi Andy,
>
> This is weird -- I know nothing about any robot blocking and we don't have a
> robots.txt file at all. Do you have a small example script to illustrate the
> problem (or is the easiest option to try rivet's bootstrap script)? I just
> checked things with wget and that worked fine. But our web servers did get
> updated (downgraded really) lately, so something might have changed relative
> to a few weeks ago.
>
> Cheers,
> Gavin
>
>
>
> On 4/19/13 12:42 PM, Andy Buckley wrote:
>>
>> Hi Gregory, Gavin, et al,
>>
>> We've noticed recently that the FastJet website blocks "robot" downloads
>> of the FastJet tarball, e.g. http://fastjet.fr/repo/fastjet-3.0.3.tar.gz
>>
>> Unfortunately this means that the Rivet bootstrap script can fail if it
>> tries to download and build FastJet, rather than using the LCG installed
>> copy from AFS. We're using Python's urllib2 to do the fetching... is
>> there anything weor you can do to not fall foul of this blocking? (I'm
>> not sure if urllib2 automatically respects robots.txt files, but if you
>> want to give us a special unblocked User-Agent name to use, I'm sure we
>> can manage to update our script accordingly)
>>
>> Thanks!
>> Andy & co
>>
>
>
> _______________________________________________
> Rivet mailing list
> Rivet at projects.hepforge.org
> http://www.hepforge.org/lists/listinfo/rivet