[Rivet] installation

[Andy Buckley]

On 27/01/14 00:51, Marek Schoenherr wrote:
> Hi all,
> 
>  
> 
> since everyone is adding something here, I feel obliged to give two
> remarks myself:
> 
>  
> 
> a) regarding its dependencies: they do need to be installed from scratch
> as on almost no system the packaged versions of cython, boost or even
> autotools (!!!) are new enough.

But that's for developer builds, right? I only make those on my laptop
for the Cython and autotools requirements, which are not at all required
on the often outdated "production" system where the build is made from a
tarball. The requirement of a very new Cython was consciously made on
the basis that "users won't even see it, developers can deal with it,
and the workarounds needed otherwise are totally not worth the effort".

I'm surprised that Boost isn't new enough... this is maybe due to YODA
declaring a requirement of version 1.47, which might not be really
required -- I'm not sure what was the motivator for that. If someone
could try editing the generated configure script (very easy) to replace
1.47 -> 1.40 or so and then try building on some old system, we'd know
if an older Boost version can actually be used, and most likely there
won't be any need to install a custom version. (Although at least for
Rivet's use of Boost, just copying some headers into place is enough:
dealing with the compiled Boost libraries is a *real* nightmare.)

> b) that being said, the old bootstrap script, though horribly
> convoluted, managed to get me both rivet-1 and rivet-2 installed on all
> systems I tried pretty much straight out of the box (mostly because I
> instructed it to install everything, I think).
> 
>  
> 
> I can see why any sensible developer would refuse to support the latter,
> however the current one is hardly helping anywhere except for lxplus. I
> therefore very much welcome FrankS' initiative.

Agreed! I only had lxplus to test it on, and that's what was being
demanded, hence that's where it works. It was intended to be a template
to be hacked as required for other systems, but maybe that wasn't clear
enough.

> In the end, I hardly ever used Rivet-2 so far (as much as would have
> liked to), because analyses written for Rivet-2 are not backwards
> compatible to Rivet-1 (and vice versa) (the changes are trivial, I know)
> and hardly anyone of the people I am working with managed to get Rivet-2
> installed. So here I am manipulating aidafiles with book-keeping helper
> analyses etc. Not sure if any of the above is helpful ...

Sort of ;-)  We do need to get people on to version 2 and if (despite
the lack of any complaints/requests until now) lots of people are
blocked from moving because of difficulty to build the new version, then
we need to pay attention to that and smooth the migration. I still find
this a bit mysterious since really not that much changed in the build
system, other than the YODA package, but ok...

Andy


> On Sunday 26 Jan 2014 15:34:49 Stefan Hoeche wrote:
> 
>>Hi All,
> 
>>
> 
>>just adding something to the discussion from the nerdy user point of
> 
>>view: I've managed to install Rivet on most of the clusters accessible
> 
>>to me, even on OSG, Cray XK7 and an IBM BG/Q. So I'd say I bring quite a
> 
>>bit of experience into the discussion.
> 
>>
> 
>>That being said, it was the worst nightmare I ever had installing a HEP
> 
>>library! I agree with both Franks that, in the current status, Rivet and
> 
>>its bootstarp scripts are next to unusable for any un-nerdy person.
> 
>>
> 
>>Admittedly, even I refuse to upgrade my old versions on lxplus and on
> 
>>the SLAC cluster because it is such a pain to meet all the dependencies.
> 
>>I'd very much appreciate a bootstrap script which by default installs
> 
>>all the dependencies, no matter whether they are found on the system or
> 
>>not. This would help in particular those users with admins who refuse to
> 
>>upgrade to newer system-wide libraries (SLAC!).
> 
>>
> 
>>All that being said, once Rivet _is_ installed, it works like a charm ;)
> 
>>Thanks for the great tool!
> 
>>
> 
>>Cheers
> 
>>Stefan
> 
>>
> 
>>On 01/26/2014 03:18 PM, Frank Siegert wrote:
> 
>>> Hi Andy,
> 
>>>
> 
>>> I'm not sure that many experiment users necessarily install their own
> 
>>> copy of Rivet, in particular not if they have access to LCG. So even
> 
>>> with limited manpower, and to make any effort on Rivet have the impact
> 
>>> we want it to have, we should avoid locking a large part of our
> 
>>> userbase out. We didn't do so badly for that before Rivet2, i.e. with
> 
>>> easier dependencies and the old bootstrap script, except that it being
> 
>>> in Python wasn't very smart for maintainability. So back to
> 
>>> constructiveness:
> 
>>>
> 
>>> I'm attaching a proposal for a rivet-2-bootstrap script. It's not
> 
>>> trying to be smart about what is already installed, but simply
> 
>>> installs everything by default, unless the user specifies that
> 
>>> something already exists (and if necessary, in which directory). For
> 
>>>
> 
>>> example:
> 
>>> INSTALL_BOOST=0 BOOSTPATH=/usr/local ./rivet-2-bootstrap
> 
>>>
> 
>>> I'm still running a test for the RIVETDEV installation option, so
> 
>>> don't consider it completely working and final yet. But if the basic
> 
>>> idea looks Ok to you, I'd put this into the repository as
> 
>>> rivet-2-bootstrap, and move the current one to
> 
>>> rivet-2-bootstrap-lcg(?). Or would you rather have both in one with
> 
>>> automatic switching if /afs is available or not?
> 
>>>
> 
>>> Cheers,
> 
>>> Frank
> 
>>>
> 
>>> On 26 January 2014 23:59, Andy Buckley <andy.buckley at cern.ch> wrote:
> 
>>>> Hi Franks,
> 
>>>>
> 
>>>> Well, it's usable in proportion to the amount of developer time that we
> 
>>>> have had to spend on making it nice... the main users what we have had
> 
>>>> to win over are the experiment ones who access it through experiment
> 
>>>> frameworks or from the Genser-built copy. So certainly that's who I've
> 
>>>> been aiming my limited build system dev time at, and to be honest I
> 
>>>> still stand by that choice although I'd rather not have *anyone* annoyed
> 
>>>> by our code.
> 
>>>>
> 
>>>> For a tarball build I think the dependencies are as follows:
> 
>>>>
> 
>>>> YODA requires:
> 
>>>> * Boost (headers only, but currently asks for quite a new version...
> 
>>>>
> 
>>>> can maybe be relaxed: any testers?)
> 
>>>>
> 
>>>> Rivet requires:
> 
>>>> * HepMC, FastJet (unavoidable, I think)
> 
>>>> * YODA (yes, could have been built-in, but it has wider utility)
> 
>>>> * yaml-cpp (for 2.1.0 I'd like to use a built-in copy, cf. the latest
> 
>>>>
> 
>>>> LHAPDF6)
> 
>>>>
> 
>>>> * GNU Scientific Library (volunteers to eliminate that dependency? But
> 
>>>>
> 
>>>> is typically an easily-met requirement, I think)
> 
>>>>
> 
>>>> * Boost (headers only, fairly easy version requirement, I think)
> 
>>>>
> 
>>>> If we eliminate yaml-cpp, how much is there to object to? Anyway, yes
> 
>>>> Frank I'd be very happy for you to make a new rivet2-bootstrap script
> 
>>>> which builds everything from scratch... and/or attempts to work out what
> 
>>>> is already on the system? At some level users should make their own
> 
>>>> minds up about what to install from packaging systems and what by
> 
>>>> hand... but by saying that I probably give away what side of Frank's
> 
>>>> nerd fence I'm sitting on ;-)
> 
>>>>
> 
>>>> Anyway, in short I don't see so much room for cutting things down, but
> 
>>>> always think that smoothing the entry paths (with scripts or otherwise)
> 
>>>> is a good thing as long as it's kept manageable for the developers, too.
> 
>>>> Since that's translated as "next to no manpower" for the last... two
> 
>>>> years?, I'm afraid the situation isn't so surprising. But we'll do what
> 
>>>> we can -- certainly I want to get rid of the yaml-cpp explicity
> 
>>>> requirement, and hopefully we can get a couple of new part-time
> 
>>>> developers (re)activated in the coming months.
> 
>>>>
> 
>>>> Andy
> 
>>>>
> 
>>>> On 26/01/14 19:47, Frank Siegert wrote:
> 
>>>>> I hate to admit that I agree with Frank's rant here, and I have had
> 
>>>>> multiple users recently who made the same point, though admittedly
> 
>>>>> more politely. The bootstrap script currently is close to useless to
> 
>>>>> everybody who isn't installing on lxplus. I have it on my agenda to
> 
>>>>> replace it with a working version, but haven't got around to it yet.
> 
>>>>> And to be honest, all the changes and external dependencies that came
> 
>>>>> with Rivet2 have left me as a second-class developer (if even),
> 
>>>>> because there is a lot of stuff in the codebase which is beyond my
> 
>>>>> capabilities to fix -- I'm normally quite annoyed myself with even
> 
>>>>> installing Rivet on a new cluster, where
> 
>>>>> cython/yaml/yoda/boost/cmake/hg/fastjet/hepmc... isn't available. Some
> 
>>>>> of those are only necessary for the development version, but it's
> 
>>>>> still rather annoying.
> 
>>>>> This goes far beyond the switch to YODA, which I agree with as
> 
>>>>> necessary, though tbh I originally thought would be Rivet-in-house as
> 
>>>>> well.
> 
>>>>>
> 
>>>>> Anyway, as I'm clearly not the majority of Rivet developers, and
> 
>>>>> mostly everybody else on the team seems to be happy about this usage
> 
>>>>> of fancy new technologies, separation and dependencies, there is no
> 
>>>>> point in discussing that in itself. But we definitely need a usable
> 
>>>>> bootstrap script much more urgently than we needed it for Rivet1.
> 
>>>>>
> 
>>>>> Andy, what about renaming the current rivet2 bootstrap script to
> 
>>>>> rivet-2-bootstrap-lcg, and (me) creating a separate one which actually
> 
>>>>> does all the installations on its own (but without support for Rivet1
> 
>>>>> of course, to avoid it becoming too bloated as the old one did)?
> 
>>>>>
> 
>>>>> Cheers,
> 
>>>>> Frank
> 
>>>>>
> 
>>>>> On 26 January 2014 17:13, Frank Krauss <frank.krauss at durham.ac.uk>
> wrote:
> 
>>>>>> Dear Rivet Authors,
> 
>>>>>> I am just towards the end of my second hour running trying to install
> 
>>>>>> Rivet 2.0. Admittedly, I am not half as geeky and nerdy as you are, so
> 
>>>>>> I accepted/anticipated that this would take me an hour. But, please,
> 
>>>>>> can you explain the rationale why you use cython, yaml, yoda, boost,
> 
>>>>>> fastjet & hepmc (I am sure i missed some!) in a non-trivial way with
> 
>>>>>> non-trivial version specifications without providing a meaningful
> 
>>>>>> installation guide? You certainly do not want the common people to use
> 
>>>>>> your code - is that it? If you think this helps the user base you are
> 
>>>>>> mightily mistaken. If you think I will continue to advertise your
> 
>>>>>> code, you're delusional - this is next to uninstallable as it stands
> 
>>>>>> now, with a pretty superficial installation help. Best wishes
> 
>>>>>>
> 
>>>>>> Frank
> 
> -- 
> 
> Marek Schoenherr
> 
> Institute for Particle Physics Phenomenology
> 
> Durham University
> 
> Tel.: +44 191 334 3770
> 
> http://www.ippp.dur.ac.uk/~mschoenherr
> 


-- 
Dr Andy Buckley, Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow / PH Dept, CERN