[Rivet] yodamerge and normalisation

Andy Buckley andy.buckley at cern.ch
Thu Oct 15 10:24:22 BST 2015
Previous message: [Rivet] yodamerge and normalisation
Next message: [Rivet] scaling 2d histograms
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Hi Hannes,

The problem is that combining normalised histograms requires inversion 
of the normalisation. When that inversion doesn't exist because the norm 
is 0 (or close as dammit), the combination can't work.

What you want here is really to get your best-stats estimates of the 
Z+jet pT shape and the DY+jet normalisation independently, then combine 
them. That would be a very special-case behaviour for yodamerge, hence 
it doesn't work by default. But as you see, there's a fairly simple bit 
of post-processing that will do what you want.*

Not that I think it's a good argument for the natural order of things, 
but note that if you did the same analysis in ROOT you would a) lose the 
information to invert the normalisation, and b) certainly have to write 
a script!

So that's how it is for now. Our long-term plan is that you'll be able 
to feed the merged YODA file back into Rivet, and re-run the finalize 
operation. This will effectively ignore the per-run normalized 
histograms and recompute them from the pre-finalize inputs, so no 
special post-processing will be needed other than re-running Rivet with 
the same analysis as before. It's technically clear how to do that, but 
<insert usual manpower moan, yadda yadda> ;-)

Andy




On 15/10/15 06:56, Hannes Jung wrote:
> Hi Andy
>
> thanks a lot for your mail and the suggestion.
>
> Why do you say, it would be better outside Rivet?
>
> Maybe it is not clear to me, how the "un-normalising" in yodamerge is done.
> Is perhaps the problem the large (or small) number which I took to
> replace infty or zero ?
>
> The point is, that we have the measurements in such a way, and it would
> be just
> fine, to have a Rivet plugin, which would fill the histos such that it
> can be
> plotted easily with rivet-mkhtml.
>
> Thanks again
>
> Best
> Hannes
>
>> On 14.10.2015, at 23:39, Andy Buckley <andy.buckley at cern.ch
>> <mailto:andy.buckley at cern.ch>> wrote:
>>
>> Hi Hannes,
>>
>> I think this sort of case is better dealt with by *not* normalizing
>> the DY+jet histos, or normalizing them to an area of 1. Then use
>> yodamerge to combine both the Z+jet and DY+jet pt plots and write a
>> (very) little Python script to do the last bit:
>>
>> import yoda
>> hs = yoda.read("merged.yoda")
>> norm = hs["/MYANA/DYpt"].integral()
>> hs["/MYANA/DYpt"].normalize(norm)
>> yoda.write(hs, "rescaled.yoda")
>>
>> Not tested, but that's the basic idea. Let us know if that works for you.
>>
>> Andy
>>
>>
>> On 14/10/15 18:23, Hannes Jung wrote:
>>> Hi Andy et al
>>>
>>> the new yodamerge works much better now, this is really great.
>>>
>>> But I have now another problem (which I tried to solve with the hack
>>> from before):
>>>
>>> I fill histograms and normalize them to some xsection.
>>> Now it can happen, that the histo is filled, but the normalization is
>>> zero,
>>> for example when I plot the pt of DY+jet normalized to Z+jet, then it
>>> can happen that
>>> I have no Z+jet events, while the histo for DY+jet is filled, if the DY
>>> mass is small.
>>>
>>> Now, I want to run many jobs in parallel, so that I have enough
>>> statistics, and then
>>> I want to merge all histos.
>>> In the case above, I just artificially put a very small xsection, so
>>> that I still can
>>> normalize the histo, and I was hoping, that by merging this would work.
>>> But apparently, yodamerge does not really like value of 1E99=1/xsect
>>> where
>>> xsect is put to a small number, but not zero.
>>>
>>> At the moment I am a bit stuck with this, just don't know, how I can
>>> tell the
>>> merging at the end to take into account a very small normalization
>>> factor for some runs.
>>>
>>> Any idea on this would be very welcome
>>>
>>> thanks a lot
>>> Cheers
>>> Hannes
>>>
>>>
>>>
>>>
>>>
>>>> On 13.10.2015, at 17:38, Andy Buckley <andy.buckley at cern.ch
>>>> <mailto:andy.buckley at cern.ch>
>>>> <mailto:andy.buckley at cern.ch>> wrote:
>>>>
>>>> Oops, I didn't realise I'd taken the discussion with Hannes off-list.
>>>>
>>>> Here's the status: looks like he was experiencing a side-effect of a
>>>> hack around merge failures for completely empty histos. We fixed that
>>>> in the latest YODA, so hopefully the hack can also be removed and all
>>>> will be well...
>>>>
>>>> Andy
>>>>
>>>>
>>>> -------- Forwarded Message --------
>>>> Subject:Re: [Rivet] Problem with yodamerge
>>>> Date:Tue, 13 Oct 2015 15:42:57 +0200
>>>> From:Hannes Jung <hannes.jung at desy.de
>>>> <mailto:hannes.jung at desy.de><mailto:hannes.jung at desy.de>>
>>>> To:andy.buckley at cern.ch <http://cern.ch/><mailto:andy.buckley at cern.ch>
>>>>
>>>>
>>>>
>>>> Hi Andy
>>>>
>>>> ahhh.... I put by hand a normalization factor very small (so that
>>>> 1/factor is very large),
>>>> so that when the histo is normalized with this factor it becomes zero...
>>>> but perhaps
>>>> this screws up when unpacking the normalized histos in yodamerge.
>>>>
>>>> The problem occurred in previous versions when I had a histo with no
>>>> entires,
>>>> then yodamerge had problems with the normalization... this is why I
>>>> introduced this hack....  perhaps I should check again whether it
>>>> works now
>>>> if I have sometimes a histo without entires and without normalization
>>>> factor...
>>>>
>>>> Cheers
>>>> Hannes
>>>>
>>>>
>>>>
>>>>> On 13.10.2015, at 15:33, Andy Buckley <andy.buckley at cern.ch
>>>>> <mailto:andy.buckley at cern.ch>
>>>>> <mailto:andy.buckley at cern.ch>
>>>>> <mailto:andy.buckley at cern.ch>> wrote:
>>>>>
>>>>> Hi Hannes,
>>>>>
>>>>> Yes, those ScaledBy values are quite extreme. They are created by the
>>>>> calls to scale() or normalize() in Rivet, and the value just
>>>>> corresponds to the scale/normalization factor. How are you managing to
>>>>> generate factors of nearly 10^100?!
>>>>>
>>>>> In the latest version of YODA, the --assume-normalized flag for
>>>>> yodamerge doesn't do anything. We changed the default logic since it
>>>>> seemed that lots of people were using that flag "blindly" to work
>>>>> around a class of problem for which it wasn't really the right answer.
>>>>> But since it's now a do-nothing flag it shouldn't be related to your
>>>>> ScaledBy problems.
>>>>>
>>>>> Andy
>>>>>
>>>>>
>>>>> On 13/10/15 11:05, Hannes Jung wrote:
>>>>>> Hi Andy, Chris
>>>>>>
>>>>>> thanks  a lot for your reply.
>>>>>> Yes, I am trying to merge normalised Histograms.
>>>>>> ahh, and yes, I have some strange Scaledby values:
>>>>>> 5.1279402177630292e+97
>>>>>> I guess this is what causes the problem.
>>>>>> Where are those set, is this an issue of statistics ?
>>>>>>
>>>>>> The cpmmand I use is
>>>>>>
>>>>>> yodamerge --assume-normalized yoda-files
>>>>>>
>>>>>> Thanks again
>>>>>> Cheers
>>>>>> Hannes
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 13.10.2015, at 11:23, Andy Buckley <andy.buckley at cern.ch
>>>>>>> <mailto:andy.buckley at cern.ch>
>>>>>>> <mailto:andy.buckley at cern.ch>
>>>>>>> <mailto:andy.buckley at cern.ch>
>>>>>>> <mailto:andy.buckley at cern.ch>> wrote:
>>>>>>>
>>>>>>> Hi Hannes,
>>>>>>>
>>>>>>> What sort of data object are you merging here? From the y-axis it
>>>>>>> looks like a normalized histogram rather than {unnormalized histo,
>>>>>>> profile, ratio}.
>>>>>>>
>>>>>>> If that's right, yodamerge will look for the ScaledBy attribute of
>>>>>>> that plot to reverse the normalisation, add all the unnormalised
>>>>>>> histos together, then rescale by the weighted mean ScaledBy. It
>>>>>>> *shouldn't* be sensitive to one run having much smaller stats
>>>>>>> than the
>>>>>>> others, certainly not in a way that would show up like this.
>>>>>>>
>>>>>>> If you can provide a bit more detail of what this object is, and how
>>>>>>> you're trying to merge the runs, we can try to help. You can also
>>>>>>> plot
>>>>>>> or take a look inside the .yoda files to see if the individual runs
>>>>>>> look sane, i.e. have reasonable fill counts and ScaledBy values.
>>>>>>>
>>>>>>> Andy
>>>>>>>
>>>>>>>
>>>>>>> On 13/10/15 08:22, Hannes Jung wrote:
>>>>>>>> Dear Rivet experts
>>>>>>>>
>>>>>>>> I am using yodamerge to merge several yoda files (actually it is
>>>>>>>> 1000 or
>>>>>>>> so),
>>>>>>>> to get reasonable statistics in a reasonable computing time.
>>>>>>>> Sometimes I obtain strange results, like in the plot below,
>>>>>>>> where the normalization is completely off.
>>>>>>>> I am using Rivet 2.4.0, but a similar problem I had with Rivet
>>>>>>>> 2.2.0.
>>>>>>>>
>>>>>>>> Any idea what this could be ?
>>>>>>>>
>>>>>>>> Since I am merging 1000 yoda files, it's hard to tell whether it
>>>>>>>> comes
>>>>>>>> from one
>>>>>>>> specific file.... however, all jobs finished without error...
>>>>>>>>
>>>>>>>> Thanks a lot
>>>>>>>> Cheers
>>>>>>>> Hannes
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ***********************************************************************
>>>>>>>> Hannes Jung
>>>>>>>> Email:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
>>>>>>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>>>>>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>>>>>>> mobile :+49 40 8998 93741
>>>>>>>> http://www.desy.de/~jung
>>>>>>>> Tel: +49 (0) 40 8998 3741
>>>>>>>> Fax: +49 (0) 40 8994 3741
>>>>>>>> DESY, CMS 01B/02.213
>>>>>>>> Notkestr.85, 22603 Hamburg, FRG
>>>>>>>> ***********************************************************************
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Rivet mailing list
>>>>>>>> Rivet at projects.hepforge.org
>>>>>>>> <mailto:Rivet at projects.hepforge.org><mailto:Rivet at projects.hepforge.org>
>>>>>>>> <mailto:Rivet at projects.hepforge.org><mailto:Rivet at projects.hepforge.org>
>>>>>>>> https://www.hepforge.org/lists/listinfo/rivet
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>>>>>> Particle Physics Expt Group, University of Glasgow
>>>>>>
>>>>>> ***********************************************************************
>>>>>> Hannes Jung
>>>>>> Email:Hannes.Jung at desy.de
>>>>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>>>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>>>>> mobile :+49 40 8998 93741
>>>>>> http://www.desy.de/~jung
>>>>>> Tel: +49 (0) 40 8998 3741
>>>>>> Fax: +49 (0) 40 8994 3741
>>>>>> DESY, CMS 01B/02.213
>>>>>> Notkestr.85, 22603 Hamburg, FRG
>>>>>> ***********************************************************************
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>>>> Particle Physics Expt Group, University of Glasgow
>>>>
>>>> ***********************************************************************
>>>> Hannes Jung
>>>> Email:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
>>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>>> mobile :+49 40 8998 93741
>>>> http://www.desy.de/~jung
>>>> Tel: +49 (0) 40 8998 3741
>>>> Fax: +49 (0) 40 8994 3741
>>>> DESY, CMS 01B/02.213
>>>> Notkestr.85, 22603 Hamburg, FRG
>>>> ***********************************************************************
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Rivet mailing list
>>>> Rivet at projects.hepforge.org
>>>> <mailto:Rivet at projects.hepforge.org><mailto:Rivet at projects.hepforge.org>
>>>> https://www.hepforge.org/lists/listinfo/rivet
>>>
>>> ***********************************************************************
>>> Hannes Jung
>>> Email:Hannes.Jung at desy.de
>>> <mailto:Hannes.Jung at desy.de><mailto:Hannes.Jung at desy.de>
>>> mobile :+49 40 8998 93741
>>> http://www.desy.de/~jung
>>> Tel: +49 (0) 40 8998 3741
>>> Fax: +49 (0) 40 8994 3741
>>> DESY, CMS 01B/02.213
>>> Notkestr.85, 22603 Hamburg, FRG
>>> ***********************************************************************
>>>
>>>
>>>
>>
>>
>> --
>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>> Particle Physics Expt Group, University of Glasgow
>
> ***********************************************************************
> Hannes Jung
> Email: Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
> mobile :+49 40 8998 93741
> http://www.desy.de/~jung
> Tel: +49 (0) 40 8998 3741
> Fax: +49 (0) 40 8994 3741
> DESY, CMS 01B/02.213
> Notkestr.85, 22603 Hamburg, FRG
> ***********************************************************************
>
>
>


-- 
Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow
Previous message: [Rivet] yodamerge and normalisation
Next message: [Rivet] scaling 2d histograms
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
More information about the Rivet mailing list