[Rivet] installation

[Frank Krauss]

Look, you know that I supported Rivet and advertised it over many 
years.  I never found it particularly straightforward to install, and 
therefore appreciated the bootstrap script.  With it gone, version 2 
became useless for me - the old bootstrap definitely didn't work and 
your install notes were pretty useless.  (Btw: Typically we try to 
ensure that Sherpa at least builds on our trunk at any given time - if 
it doesn't which is typically my fault - this gets fixed very quickly.  
In so far there is a difference between Sherpa and rivet, David.)  When 
despite investing a serious amount of time I did not manage to install 
the ** tarball ** due to some dependencies I got really annoyed.  And 
telling me about the dependencies ** after ** I had figured them out the 
hard way, still failing with the installation, did definitely not help.

And of course I, like most likely any other MC author,  would try to 
take a look at some recent analyses to make sure that we do not fuck up 
to badly - it's part of our service to the experimental community.  In 
my case it was the check if the hadronisation I muddled with over Xmas 
does not screw things up completely.

So, sorry, but I still think the situation is far from being superb or 
even good.  Blaming it on limited developer time also does not really 
help there.  Right now, all people I talked to (Jeppe, Marek, Frank S, 
Steffen - should I make this a larger poll?) are just ** not ** moving 
to 2.0 because of the pain involved in it - if this is your plan to make 
your developer time useful, I guess I see a flaw.

But, anyway, for me the case is closed now - the bootstrap worked and I 
will not likely move to any other version soon ... .



On 27/01/14 14:15, Andy Buckley wrote:
> On 27/01/14 14:53, James Monk wrote:
>> Hi everyone,
>>
>> I have some sympathy with Frank K’s complaint (though perhaps not the
>> tone!)  - I tend to work infrequently on Rivet development, but when
>> I do there tends to be some overhead to getting the bleeding edge dev
>> version working again.  I think it is right that yaml-cpp and its
>> need for cmake that is the biggest annoyance, although hopefully the
>> yaml-cpp version is not going to change again (for a long while,
>> anyway).  If HepMC 3 uses cmake, would there be a case for Rivet to
>> also move to cmake?  In that case, we could forget about
>> old/incompatible autotools versions being a problem at least.
> I thought autotools versions had to be pretty damned old to cause
> trouble, hence it's SL5 clusters and similar crap where difficulties
> appear? To be clear, this is not an issue except for *framework*
> developers: users of releases for either existing analyses or developing
> new ones should not require autotools, etc. at all. If they do, that's a
> bug and we should fix it pronto.
>
>> Once you’ve got all the correct dependencies installed, I find no
>> problems, it’s just working your way through the list of everything
>> (often having old or incompatible versions already installed) is
>> sometimes a bit of a faff (depending on how long it was since you
>> last installed).  A less changeable list of dependencies now that the
>> 1.9->2.0 transition is over will help, I hope.
> Hopefully, although reducing dependencies is (even before this
> discussion!) on my >= 2.2 TODO list, so there may be some more changes
> to come, but I hope they will just be removals rather than changes or,
> god forbid, additions.
>
>> Additionally, how hard/annoying would it be to include YODA within
>> Rivet?    Would it be wrong to have YODA as a separately build-able
>> package within the Rivet tree, i.e. if you checkout and build Rivet
>> you always get YODA, but it is also possible to obtain YODA on its
>> own, for those that need it.  I guess this would mean keeping the
>> Rivet and YODA releases in sync to some extent (a new Rivet release
>> would have to imply a concurrent YODA release, though not necessarily
>> vice-versa), but would that be an issue?
> I *really* want to keep them separate: YODA is intended to be much more
> general than just a part of Rivet, and I think that has been a valuable
> separation to have in evolving the design.
>
> Merging them into the same code tree would create umpteen other
> problems, and is it really such an issue to build both? The header
> dependencies and so-on which can force big rebuilds are annoying but
> would be the same if YODA's code were inside the Rivet tree, and I
> highly recommend ccache for making accidental no-real-change rebuilds
> super-fast. Bootstrap-based "normal" users shouldn't care at all, since
> both packages always got installed automatically, even with the
> LCG-based script version before Frank's latest contribution.
>
> Andy
>
>
>> On 27 Jan 2014, at 13:57, Andy Buckley <andy.buckley at cern.ch> wrote:
>>
>>> Turning off the PDF manual build by default is possible, and
>>> perhaps would be welcome: what do others think?
>>>
>>> The build system / scripts in the doc directory try to bootstrap
>>> their environment to only use the uninstalled libraries from inside
>>> the Rivet build dir. There were some bugs in that which I fixed
>>> "recently" and greatly tidied things up, but I forget whether
>>> they're in version 2.0.0 or just the dev head. So I'm surprised
>>> that it didn't work, but if disabling the default manual build
>>> would avoid a whole bunch of common problems then I think we should
>>> do it: for released versions it will be obtainable via the arXiv
>>> anyway.
>>>
>>> Cheers, Andy
>>>
>>>
>>> On 27/01/14 13:47, Frank Siegert wrote:
>>>> Ah, and one thing that I forgot: probably not a conceptual
>>>> problem, but a bug, which basically stopped Frank's Rivet
>>>> installation efforts in the end yesterday:
>>>>
>>>> When we do configure; make; make install; then the make step will
>>>> also go into doc/ and try to use rivet-mkanalyses-html. As far as
>>>> I understand the Makefile and also Frank's error from yesterday,
>>>> this assumes that libRivet.so is already installed into --prefix,
>>>> which clearly isn't the case in the make step. Am I
>>>> misunderstanding this? Could we move the complete doc/ building
>>>> into the --enable-pdfmanual conditional (and disable it by
>>>> default), to avoid such simple compile failures?
>>>>
>>>> Cheers, Frank
>>>>
>>>>
>>>>
>>>> On 27 January 2014 13:43, Frank Siegert <frank.siegert at cern.ch>
>>>> wrote:
>>>>> Hi Andy,
>>>>>
>>>>> I think there is one simple issue explaining the difference in
>>>>> perception between you and Frank et al.: One actually needs to
>>>>> install *Rivet-dev* to get current analyses.
>>>>>
>>>>> And installing Rivet-dev is the annoying bit, in particular for
>>>>> Rivet2 vs. Rivet1 due to:
>>>>>
>>>>> 1. Cython It's simply not possible to install cython properly
>>>>> into the install prefix automatically in a bootstrap script,
>>>>> which is why we have worked around by setting the PATH and
>>>>> PYTHONPATH to find it during the bootstrap. But that "solution"
>>>>> fails in case you want to actually update your Rivet later (new
>>>>> analyses!) and compile again, since those variables aren't set
>>>>> anymore. Swig was much easier in that respect.
>>>>>
>>>>> 2. hg While SVN was basically available on all systems that I
>>>>> have worked on, I often had to install hg myself.
>>>>>
>>>>> 3. YODA from hg (release sometimes isn't new enough for
>>>>> Rivet-dev) + its own dependencies.
>>>>>
>>>>> 4. autotools I have seen two independent clusters where the
>>>>> autotools from SL6 (or SL5, not 100% sure) lead to very hard to
>>>>> understand linker problems, which went away with recent
>>>>> autotools.
>>>>>
>>>>> 5. yaml_cpp This needs cmake for installation, i.e. yet another
>>>>> dependency. So if that's bundled in the future, it will already
>>>>> make quite a difference.
>>>>>
>>>>>
>>>>> Bottom line: no, it's not just about the Boost version in
>>>>> YODA.
>>>>>
>>>>> Cheers, Frank
>>>>>
>>>>>
>>>>> On 27 January 2014 13:03, Andy Buckley <andy.buckley at cern.ch>
>>>>> wrote:
>>>>>> On 27/01/14 11:17, Frank Krauss wrote:
>>>>>>> Hi Andy, well, that's great.  I found out about the
>>>>>>> dependencies the hard way, but I still did not manage to
>>>>>>> produce an installation.  And since you guys appear to
>>>>>>> claim zero time of developer time it has about zero
>>>>>>> usability at the moment for anyone who cannot rely on it
>>>>>>> being already installed on lxplus.  I guess however that
>>>>>>> Rivet was also meant for the wider MC community, including
>>>>>>> theorists not sitting at CERN.  For those it is pretty much
>>>>>>> useless, and, the way I see it, because the developers
>>>>>>> mainly aimed at proving how smart they are rather than at
>>>>>>> having something useful.  So, up to now, you just showed me
>>>>>>> that you guys do not give a toss about this part of the
>>>>>>> community.  Good to know.  I sincerely hope this attitude
>>>>>>> of producing monumental software for the nerds is not the
>>>>>>> new trend in HEP computing.
>>>>>> I'll try to reply nicely... but not exactly the style of
>>>>>> feedback that makes me want to be Mr Helpful, Frank.
>>>>>>
>>>>>> First off, of course we care about MC/pheno theorists. But as
>>>>>> far as I was aware the part of the community that Rivet was
>>>>>> started for (i.e. you guys) were more-or-less happily using
>>>>>> it without significant trouble. So the recent work (shall I
>>>>>> moan about manpower again?) has been aimed at getting the
>>>>>> experiment uptake to the same level as in the generator
>>>>>> community. Understandable, no? So it looks like time for the
>>>>>> balance to swing a bit back toward the non-LCG users again.
>>>>>>
>>>>>> I _really_ hope we have better ways to look (be!) clever than
>>>>>> introduce new pain-in-the-ass library dependencies. It's more
>>>>>> the other way around: when there's a better implementation
>>>>>> somewhere else, which we have no chance/time to reimplement
>>>>>> internally, we had to be stupid and acquire the dependency.
>>>>>> Most of this happened > 5 years ago. Where there have been
>>>>>> changes they have been to *reduce* build system problems
>>>>>> (cf. the previous way of bundling yaml-cpp). I didn't think
>>>>>> that much had changed with the release of version 2, but
>>>>>> clearly some changes have had painful impact: it would be
>>>>>> good to know exactly what these are. As I mentioned in
>>>>>> another email, I suspect that YODA's Boost version
>>>>>> requirement is too high -- is that right?
>>>>>>
>>>>>> The dependencies that we have were chosen (I thought fairly
>>>>>> carefully) for compatibility with the "canonical" HEP system,
>>>>>> i.e. SLC6 and the LCG libraries. Frank's new script should
>>>>>> handle tarball builds on systems where LCG isn't present,
>>>>>> which is great. Installation on personal laptops _should_ be
>>>>>> far easier because so many dependencies can come direct from
>>>>>> system packages... but you're right that that's assuming a
>>>>>> degree of nerdity in that you're already administrating your
>>>>>> own machine so can presumably choose what to install from
>>>>>> where. If not, Frank's new "install everything" script should
>>>>>> help... but be aware that if you build one library with a
>>>>>> system Boost/GSL/whatever and Rivet with this all-in-one
>>>>>> bootstrap then the fact that it fails to run isn't really a
>>>>>> Rivet problem! Finally, building an unreleased development
>>>>>> version is of course a bit harder: heck, *I* don't try to
>>>>>> build the developer version on lxplus direct from the
>>>>>> repository: I make a tarball on my laptop, copy it over, and
>>>>>> run the standard bootstrap.
>>>>>>
>>>>>> Ok, enough. The feedback and the more general bootstrap
>>>>>> script are really good to have, and I wish we'd known the
>>>>>> strength of feeling about Rivet much earlier... but if no-one
>>>>>> reports it then we don't know.  Let us know (nicely :-P) if
>>>>>> there are more problems -- particularly, concrete help like
>>>>>> Frank S' is hugely appreciated.
>>>>>>
>>>>>> Andy
>>>>>>
>>>>>> PS. Current manpower, since there was incredulity: I would
>>>>>> guess < 1 day/month on average for me, plus something similar
>>>>>> or less for David G, and occasional (very welcome!)
>>>>>> contributions from others like Frank S. Much of that time
>>>>>> goes on analysis integration and (slow) feature development.
>>>>>> Does it look like more from outside?
>>>>>>
>>>>>> PPS. When yaml-cpp is bundled back inside Rivet, the tarball
>>>>>> dependencies will be exactly the same as Rivet 1, except that
>>>>>> the histogramming now has to be built separately. So I think
>>>>>> the 1 vs. 2 dichotomy is a false one, and the issue is more
>>>>>> to with the change to bootstrap scripts. You'll probably
>>>>>> still need to have cmake installed, unless we bundle that as
>>>>>> well... and why not an entire operating system? But while I
>>>>>> dislike that cmake requirement, it's the same for a *lot* of
>>>>>> other HEP packages now since LCG standardised on it. There
>>>>>> will be a lot of pressure for HepMC3 to go that way, I
>>>>>> think.
>>>>>>
>>>>>>
>>>>>>> On 26/01/14 22:59, Andy Buckley wrote:
>>>>>>>> Hi Franks,
>>>>>>>>
>>>>>>>> Well, it's usable in proportion to the amount of
>>>>>>>> developer time that we have had to spend on making it
>>>>>>>> nice... the main users what we have had to win over are
>>>>>>>> the experiment ones who access it through experiment
>>>>>>>> frameworks or from the Genser-built copy. So certainly
>>>>>>>> that's who I've been aiming my limited build system dev
>>>>>>>> time at, and to be honest I still stand by that choice
>>>>>>>> although I'd rather not have *anyone* annoyed by our
>>>>>>>> code.
>>>>>>>>
>>>>>>>> For a tarball build I think the dependencies are as
>>>>>>>> follows:
>>>>>>>>
>>>>>>>> YODA requires: * Boost (headers only, but currently asks
>>>>>>>> for quite a new version... can maybe be relaxed: any
>>>>>>>> testers?)
>>>>>>>>
>>>>>>>> Rivet requires: * HepMC, FastJet (unavoidable, I think) *
>>>>>>>> YODA (yes, could have been built-in, but it has wider
>>>>>>>> utility) * yaml-cpp (for 2.1.0 I'd like to use a built-in
>>>>>>>> copy, cf. the latest LHAPDF6) * GNU Scientific Library
>>>>>>>> (volunteers to eliminate that dependency? But is
>>>>>>>> typically an easily-met requirement, I think) * Boost
>>>>>>>> (headers only, fairly easy version requirement, I think)
>>>>>>>>
>>>>>>>> If we eliminate yaml-cpp, how much is there to object to?
>>>>>>>> Anyway, yes Frank I'd be very happy for you to make a new
>>>>>>>> rivet2-bootstrap script which builds everything from
>>>>>>>> scratch... and/or attempts to work out what is already on
>>>>>>>> the system? At some level users should make their own
>>>>>>>> minds up about what to install from packaging systems and
>>>>>>>> what by hand... but by saying that I probably give away
>>>>>>>> what side of Frank's nerd fence I'm sitting on ;-)
>>>>>>>>
>>>>>>>> Anyway, in short I don't see so much room for cutting
>>>>>>>> things down, but always think that smoothing the entry
>>>>>>>> paths (with scripts or otherwise) is a good thing as long
>>>>>>>> as it's kept manageable for the developers, too. Since
>>>>>>>> that's translated as "next to no manpower" for the
>>>>>>>> last... two years?, I'm afraid the situation isn't so
>>>>>>>> surprising. But we'll do what we can -- certainly I want
>>>>>>>> to get rid of the yaml-cpp explicity requirement, and
>>>>>>>> hopefully we can get a couple of new part-time developers
>>>>>>>> (re)activated in the coming months.
>>>>>>>>
>>>>>>>> Andy
>>>>>>>>
>>>>>>>>
>>>>>>>> On 26/01/14 19:47, Frank Siegert wrote:
>>>>>>>>> I hate to admit that I agree with Frank's rant here,
>>>>>>>>> and I have had multiple users recently who made the
>>>>>>>>> same point, though admittedly more politely. The
>>>>>>>>> bootstrap script currently is close to useless to
>>>>>>>>> everybody who isn't installing on lxplus. I have it on
>>>>>>>>> my agenda to replace it with a working version, but
>>>>>>>>> haven't got around to it yet. And to be honest, all the
>>>>>>>>> changes and external dependencies that came with Rivet2
>>>>>>>>> have left me as a second-class developer (if even),
>>>>>>>>> because there is a lot of stuff in the codebase which
>>>>>>>>> is beyond my capabilities to fix -- I'm normally quite
>>>>>>>>> annoyed myself with even installing Rivet on a new
>>>>>>>>> cluster, where
>>>>>>>>> cython/yaml/yoda/boost/cmake/hg/fastjet/hepmc... isn't
>>>>>>>>> available. Some of those are only necessary for the
>>>>>>>>> development version, but it's still rather annoying.
>>>>>>>>> This goes far beyond the switch to YODA, which I agree
>>>>>>>>> with as necessary, though tbh I originally thought
>>>>>>>>> would be Rivet-in-house as well.
>>>>>>>>>
>>>>>>>>> Anyway, as I'm clearly not the majority of Rivet
>>>>>>>>> developers, and mostly everybody else on the team seems
>>>>>>>>> to be happy about this usage of fancy new technologies,
>>>>>>>>> separation and dependencies, there is no point in
>>>>>>>>> discussing that in itself. But we definitely need a
>>>>>>>>> usable bootstrap script much more urgently than we
>>>>>>>>> needed it for Rivet1.
>>>>>>>>>
>>>>>>>>> Andy, what about renaming the current rivet2 bootstrap
>>>>>>>>> script to rivet-2-bootstrap-lcg, and (me) creating a
>>>>>>>>> separate one which actually does all the installations
>>>>>>>>> on its own (but without support for Rivet1 of course,
>>>>>>>>> to avoid it becoming too bloated as the old one did)?
>>>>>>>>>
>>>>>>>>> Cheers, Frank
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 26 January 2014 17:13, Frank Krauss
>>>>>>>>> <frank.krauss at durham.ac.uk> wrote:
>>>>>>>>>> Dear Rivet Authors, I am just towards the end of my
>>>>>>>>>> second hour running trying to install Rivet 2.0.
>>>>>>>>>> Admittedly, I am not half as geeky and nerdy as you
>>>>>>>>>> are, so I accepted/anticipated that this would take
>>>>>>>>>> me an hour. But, please, can you explain the
>>>>>>>>>> rationale why you use cython, yaml, yoda, boost,
>>>>>>>>>> fastjet & hepmc (I am sure i missed some!) in a
>>>>>>>>>> non-trivial way with non-trivial version
>>>>>>>>>> specifications without providing a meaningful
>>>>>>>>>> installation guide?  You certainly do not want the
>>>>>>>>>> common people to use your code - is that it?  If you
>>>>>>>>>> think this helps the user base you are mightily
>>>>>>>>>> mistaken.  If you think I will continue to advertise
>>>>>>>>>> your code, you're delusional - this is next to
>>>>>>>>>> uninstallable as it stands now, with a pretty
>>>>>>>>>> superficial installation help. Best wishes Frank
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________ Rivet
>>>>>>>>>> mailing list Rivet at projects.hepforge.org
>>>>>>>>>> http://www.hepforge.org/lists/listinfo/rivet
>>>>>>>>> _______________________________________________ Rivet
>>>>>>>>> mailing list Rivet at projects.hepforge.org
>>>>>>>>> http://www.hepforge.org/lists/listinfo/rivet
>>>>>>>>>
>>>>>>
>>>>>> -- Dr Andy Buckley, Royal Society University Research Fellow
>>>>>> Particle Physics Expt Group, University of Glasgow / PH Dept,
>>>>>> CERN
>>>
>>> -- Dr Andy Buckley, Royal Society University Research Fellow
>>> Particle Physics Expt Group, University of Glasgow / PH Dept, CERN
>>> _______________________________________________ Rivet mailing list
>>> Rivet at projects.hepforge.org
>>> http://www.hepforge.org/lists/listinfo/rivet
>