[Rivet] GENSER bootstrap script

[Oleg ZENIN]

Hello, Andy,

Thank you for the detailed comments. 
I am sorry for having delayed the reply.
(I needed several days to fix several issues mentioned in your letter.)

> Can I specify particular compilers like ccache, gfortran, ifort etc.
> when I run the script (e.g. by use of CC/FC/CXXFLAGS etc. variables, a
> la configure?) 

Just export CC, CFLAGS, etc. in the usual manner and they'll be used 
to build the content of external/ and external/MCGenerators.

Though, note that for safety reasons user's compiler settings are 
temporarily disabled for the initial part of `bootstrapping', i.e. 
when the script builds the pkgsrc service routines. 
I guess, at this stage we should rely upon pkgsrc native machinery 
as it proved to be reasonable in a variety of environments.
Having compiled service routines the script restores user's CC, CFLAGS, etc. 
settings and  then can proceed to building external/* and/or external/MCGenerators/* 
with any combination of compiler & flags specified. 
At least it would try to.
(For example, CLHEP complains about CC like 'ccache cc' at the configuration stage
 and finally fails due to an apparent error in the Makefile triggered by such 
 an exotic CC. This should be addressed by CLHEP authors.)

>---
>The `--external' key specifies the path to the /external/ area
>containing the MCGenerators directory where MC generators will be installed.
>---
>
>Could this be changed to something more self-explanatory, such as
>"--install"? Similarly, "--prefix" could be "--build": 
>...

Changed to --external-prefix and --build-prefix, respectively.


> ---
>         [ --build-external                   ]   \
>              [ --clhep-version=clhep_version ] \
>              [ --hepmc-version=hepmc_version ] \
>              [ --root-version=root_version ] \
>              [ --fastjet-version=fastjet_version ] \
>         [ --mcgenerators=lhapdf[,pythia6[,...]] \
>         [ --build-genser ]
> ---
> CLHEP, HepMC, ROOT and fastjet packages can be optionally installed into
> /external/ directory (see `--build-external').
> GENSER packages are compiled if and only if the `--build-genser' key is
> specified.
> The content of `external' is also built if one specifies the
> `--build-external'
> key.
> ---
> 
> I'm confused by this: what do you mean by "Genser packages"? 
> I thought
> that everything built by this script was a Genser package? 

On a machine without an access to /afs/cern.ch/sw/lcg/external/ the script 
can optionally compile clhep, hepmc, ROOT and FastJet 
(formally, this stuff is not provided by GENSER) in order to resolve 
the dependencies of some MC generators. 

>Do you mean that unless --build-external is used, the install area will be 
> assumed to contain copies of CLHEP, HepMC, ROOT and FastJet? 

No, CLHEP, etc. can be installed anywhere or be absent at all.

> What do I do if
> a) I don't want to install e.g. ROOT or CLHEP?

Just don't specify `--build-external' option. 
In this case only skeleton directories are created for clhep, hepmc, root, 
fastjet ( <build_prefix>/pkgsrc/external/{clhep,HepMC,root,fastjet}/ ) 
so that you can optionally get there later and run bmake && bmake install 
to compile a particular package.

> b) I want to use existing system versions of these programs that aren't
> in the install area?

In this case HEPMC_INSTALL_DIR, CLHEP_INSTALL_DIR, ROOTSYS and FASTJET_INSTALL_DIR
environment variables must be set before compiling any generator however it is 
actually built: either by "genser-bootstrap.sh --build-genser ..."  or via 
"cd <build_prefix>/pkgsrc/MCGenerators/generator_name/ && bmake && bmake install"
after the bootstrapping.

> I think more documentation/help text is needed to make this generally
> usable: *I'm* wary about running the script until I have some idea of
> what it will install to where!

Certainly, I'll eventually write a kind of man-page to explain all the details.
For the time being, I can say that the script doesn't install anything 
beyond directories specified by `--build-prefix=...' and `--external-prefix=...'.
As long as the script is normally launched w/o root privileges it's quite safe.

> Another point is that it's a bit odd to treat LHAPDF as a generator.
> Like HepMC, it's a utility package where having more than one version is
> not useful: the latest version supercedes all previous ones. Is there a
> reason to not treat it like the other utility libraries?

Yes, the reason is that sometimes we need to exactly reproduce a result 
obtained with a particular version of a generator linked with a particular 
version of LHAPDF. Having several LHAPDF versions installed simultaneously 
is quite useful in such a case.

>---
>At CERN site,  MC generators can be linked against libraries already
>installed in /afs/cern.ch/sw/lcg/external/ area, so that
>'--build-external'  is  not neccessary.
>---
>
>Very minor: speling mistak here ^^^ ;)

Thx! Fixed. (This doesn't necessarily mean that I'll finally embed a spell 
checker in my `vi' editor.)

> Finally, it would be really useful if the script could print out a lit
> of valid generators to be passed to the --mcgenerators flag (and the default 
> list), 

It already lists them implicitly if started w/o any parameters:

% ./genser-bootstrap.sh

Usage: ./genser-bootstrap.sh \
          --build-prefix=/genser/build/directory      \
        [ --external-prefix=/path/to/local/external ]  \ 
        [ --build-external                   ]   \
             [ --clhep-version=clhep_version ] \
             [ --hepmc-version=hepmc_version ] \
             [ --root-version=root_version ] \
             [ --fastjet-version=fastjet_version ] \
        [ --mcgenerators=lhapdf[,pythia6[,...]] \
        [ --build-genser ]                      \
        [ --lhapdf-version=lhapdf_version ] \
        [ --mctester-version=mctester_version[,mctester_version_1[,...]] ] \
        [ --mcatnlo-version=mcatnlo_version[,mcatnlo_version_1[,...]] ] \
        [ --pythia6-version=pythia6_version[,pythia6_version_1[,...]] ] \
        [ --sherpa-version=sherpa_version[,sherpa_version_1[,...]] ]

<skip>

Here `--<name>-version=...' arguments are printed out only for installable 
generators.

However, I've added the `--list-available-generators|-L' option to the 'TODO' list.

> and similarly if the valid version numbers for LHAPDF,
> MCtester, MC at NLO etc. were listed. 

Thanks, added to TODO's too.

> And is there a way to specify the Pythia 8 version?

Unfortunately, we haven't yet created pkgsrc Makefile's for Pythia 8 
and many others (these short Makefile's, one per generator, are used only to steer
fetching and extraction of the sources, resolve the dependencies, guess compiler 
settings and pass the control to the generator's native
build & installation system). 
As soon as such a Makefile for Pythia 8 gets into GENSER CVS one will be able 
to specify the version in a standard way.

> If you can let me know what the recommended options are for a system where
> I already have HepMC and FastJet (not in the install area) and don't
> want to install ROOT and CLHEP, 

The following should work (yet untested!):

 export HEPMC_INSTALL_DIR=/path/to/hepmc ; 
 export FASTJET_INSTALL_DIR=/path/to/fastjet ;
 ./genser-bootstrap --build-prefix=/path/to/the/build/directory   \
                    --external-prefix=/path/to/the/local/external \ 
                    --build-genser 
		    
But please remember that only Pythia 6, Sherpa, LHAPDF, MC at NLO and MC-TESTER 
can be compiled by the script (more generally, in the framework created 
by the script) so far.
Making other generators available is just a matter of time. 
Currently we're planning to gradually move them under the `genser-bootstrap' 
framework as long as their new versions are released.

> ... (this will be really useful for testing AGILe's compatibility with
> Jimmy, AlpGen and Charybdis, which couldn't be installed before on
> non-AFS systems)

The problem is that we haven't created pkgsrc Makefile's for them yet.
This can be a matter of several days though. 
So, if you urgently need a particular MC generator to be installable using 
this machinery I can check it in.

Looking forward to further comments and/or bug reports.

With best regards,
Oleg.

P.S. I guess, continuing the discussion in https://savannah.cern.ch/bugs/?34786
     would make tracking the things somewhat easier.


On Mon, 7 Jul 2008, Andy Buckley wrote:

> Oleg ZENIN wrote:
> > Dear colleagues, 
> > 
> > Currently, we have a preliminary version of a script to check out, 
> > build & install a subset of generators provided by GENSER.
> > HepMC, CLHEP, ROOT and FastJet packages can be optionally compiled as well.
> > 
> > Please see a brief description at http://lcgapp.cern.ch/project/simu/generator/genser-bootstrap.html
> 
> Hi Oleg,
> 
> Sorry for the delay: I've been on holiday! This is a great development
> and I'm very keen to test it out.
> 
> > Although the procedure has not been extensively tested yet it is believed
> > to work on any recent GNU/Linux system satisfying minimum requirements
> > to compile GENSER software: coreutils, cvs, bash, make, gcc, gcc-g77/gfortran, gcc-c++. 
> 
> Can I specify particular compilers like ccache, gfortran, ifort etc.
> when I run the script (e.g. by use of CC/FC/CXXFLAGS etc. variables, a
> la configure?) [Maybe this is covered in the pkgsrc documentation, but
> I'm not sure how that interacts with your wrapper script.]
> 
> > Besides several tests on  SLC4, SLC5, Fedora 7,9 and Ubuntu 8.04 we already 
> > used the new framework to install new versions of Pythia 6, MC at NLO, 
> > MC-TESTER and LHAPDF to /afs/cern.ch/sw/lcg/external/MCGenerators area.
> > 
> > The bootstrap script urgently needs wider testing than we could 
> > have performed so far.
> 
> I'm happy to get going on that!
> 
> > Looking forward to your comments, suggestions and bug reports.
> 
> Okay, here goes! I've got quite a few comments, so keep in mind that I'm
> overall positive about the whole thing, but need a bit of help to
> understand the behaviour ;)
> 
> ---
> The `--external' key specifies the path to the /external/ area
> containing the MCGenerators directory where MC generators will be installed.
> ---
> 
> Could this be changed to something more self-explanatory, such as
> "--install"? Similarly, "--prefix" could be "--build": the current usage
> of "prefix" to mean the place where the build occurs, rather than where
> the built files are installed, is confusingly opposite to the meaning in
> autotools!
> 
> ---
>         [ --build-external                   ]   \
>              [ --clhep-version=clhep_version ] \
>              [ --hepmc-version=hepmc_version ] \
>              [ --root-version=root_version ] \
>              [ --fastjet-version=fastjet_version ] \
>         [ --mcgenerators=lhapdf[,pythia6[,...]] \
>         [ --build-genser ]
> ---
> CLHEP, HepMC, ROOT and fastjet packages can be optionally installed into
> /external/ directory (see `--build-external').
> GENSER packages are compiled if and only if the `--build-genser' key is
> specified.
> The content of `external' is also built if one specifies the
> `--build-external'
> key.
> ---
> 
> I'm confused by this: what do you mean by "Genser packages"? I thought
> that everything built by this script was a Genser package? Do you mean
> that unless --build-external is used, the install area will be assumed
> to contain copies of CLHEP, HepMC, ROOT and FastJet? What do I do if
> 
> a) I don't want to install e.g. ROOT or CLHEP?
> b) I want to use existing system versions of these programs that aren't
> in the install area?
> 
> I think more documentation/help text is needed to make this generally
> usable: *I'm* wary about running the script until I have some idea of
> what it will install to where!
> 
> Another point is that it's a bit odd to treat LHAPDF as a generator.
> Like HepMC, it's a utility package where having more than one version is
> not useful: the latest version supercedes all previous ones. Is there a
> reason to not treat it like the other utility libraries?
> 
> ---
> At CERN site,  MC generators can be linked against libraries already
> installed in /afs/cern.ch/sw/lcg/external/ area, so that
> '--build-external'  is  not neccessary.
> ---
> 
> Very minor: speling mistak here ^^^ ;)
> 
> Finally, it would be really useful if the script could print out a lit
> of valid generators to be passed to the --mcgenerators flag (and the
> default list), and similarly if the valid version numbers for LHAPDF,
> MCtester, MC at NLO etc. were listed. And is there a way to specify the
> Pythia 8 version?
> 
> Sorry for so many comments --- this is a big step forward already! If
> you can let me know what the recommended options are for a system where
> I already have HepMC and FastJet (not in the install area) and don't
> want to install ROOT and CLHEP, then I'll do a proper test of the script
> here (this will be really useful for testing AGILe's compatibility with
> Jimmy, AlpGen and Charybdis, which couldn't be installed before on
> non-AFS systems)
> 
> Thanks,
> Andy
> 
>