[Rivet] yodamerge and normalisation

[Hannes Jung]

Hi Andy et al

the new yodamerge works much better now, this is really great.

But I have now another problem (which I tried to solve with the hack from before):

I fill histograms and normalize them to some xsection.
Now it can happen, that the histo is filled, but the normalization is zero,
for example when I plot the pt of DY+jet normalized to Z+jet, then it can happen that 
I have no Z+jet events, while the histo for DY+jet is filled, if the DY mass is small.

Now, I want to run many jobs in parallel, so that I have enough statistics, and then
I want to merge all histos.
In the case above, I just artificially put a very small xsection, so that I still can
normalize the histo, and I was hoping, that by merging this would work.
But apparently, yodamerge does not really like value of 1E99=1/xsect where 
xsect is put to a small number, but not zero.

At the moment I am a bit stuck with this, just don't know, how I can tell the
merging at the end to take into account a very small normalization factor for some runs.

Any idea on this would be very welcome

thanks a lot
Cheers
Hannes





> On 13.10.2015, at 17:38, Andy Buckley <andy.buckley at cern.ch> wrote:
> 
> Oops, I didn't realise I'd taken the discussion with Hannes off-list.
> 
> Here's the status: looks like he was experiencing a side-effect of a hack around merge failures for completely empty histos. We fixed that in the latest YODA, so hopefully the hack can also be removed and all will be well...
> 
> Andy
> 
> 
> -------- Forwarded Message --------
> Subject: 	Re: [Rivet] Problem with yodamerge
> Date: 	Tue, 13 Oct 2015 15:42:57 +0200
> From: 	Hannes Jung <hannes.jung at desy.de <mailto:hannes.jung at desy.de>>
> To: 	andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>
> 
> 
> 
> Hi Andy
> 
> ahhh.... I put by hand a normalization factor very small (so that
> 1/factor is very large),
> so that when the histo is normalized with this factor it becomes zero...
> but perhaps
> this screws up when unpacking the normalized histos in yodamerge.
> 
> The problem occurred in previous versions when I had a histo with no
> entires,
> then yodamerge had problems with the normalization... this is why I
> introduced this hack....  perhaps I should check again whether it works now
> if I have sometimes a histo without entires and without normalization
> factor...
> 
> Cheers
> Hannes
> 
> 
> 
>> On 13.10.2015, at 15:33, Andy Buckley <andy.buckley at cern.ch
>> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>> wrote:
>> 
>> Hi Hannes,
>> 
>> Yes, those ScaledBy values are quite extreme. They are created by the
>> calls to scale() or normalize() in Rivet, and the value just
>> corresponds to the scale/normalization factor. How are you managing to
>> generate factors of nearly 10^100?!
>> 
>> In the latest version of YODA, the --assume-normalized flag for
>> yodamerge doesn't do anything. We changed the default logic since it
>> seemed that lots of people were using that flag "blindly" to work
>> around a class of problem for which it wasn't really the right answer.
>> But since it's now a do-nothing flag it shouldn't be related to your
>> ScaledBy problems.
>> 
>> Andy
>> 
>> 
>> On 13/10/15 11:05, Hannes Jung wrote:
>>> Hi Andy, Chris
>>> 
>>> thanks  a lot for your reply.
>>> Yes, I am trying to merge normalised Histograms.
>>> ahh, and yes, I have some strange Scaledby values: 5.1279402177630292e+97
>>> I guess this is what causes the problem.
>>> Where are those set, is this an issue of statistics ?
>>> 
>>> The cpmmand I use is
>>> 
>>> yodamerge --assume-normalized yoda-files
>>> 
>>> Thanks again
>>> Cheers
>>> Hannes
>>> 
>>> 
>>> 
>>>> On 13.10.2015, at 11:23, Andy Buckley <andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>
>>>> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>
>>>> <mailto:andy.buckley at cern.ch <mailto:andy.buckley at cern.ch>>> wrote:
>>>> 
>>>> Hi Hannes,
>>>> 
>>>> What sort of data object are you merging here? From the y-axis it
>>>> looks like a normalized histogram rather than {unnormalized histo,
>>>> profile, ratio}.
>>>> 
>>>> If that's right, yodamerge will look for the ScaledBy attribute of
>>>> that plot to reverse the normalisation, add all the unnormalised
>>>> histos together, then rescale by the weighted mean ScaledBy. It
>>>> *shouldn't* be sensitive to one run having much smaller stats than the
>>>> others, certainly not in a way that would show up like this.
>>>> 
>>>> If you can provide a bit more detail of what this object is, and how
>>>> you're trying to merge the runs, we can try to help. You can also plot
>>>> or take a look inside the .yoda files to see if the individual runs
>>>> look sane, i.e. have reasonable fill counts and ScaledBy values.
>>>> 
>>>> Andy
>>>> 
>>>> 
>>>> On 13/10/15 08:22, Hannes Jung wrote:
>>>>> Dear Rivet experts
>>>>> 
>>>>> I am using yodamerge to merge several yoda files (actually it is
>>>>> 1000 or
>>>>> so),
>>>>> to get reasonable statistics in a reasonable computing time.
>>>>> Sometimes I obtain strange results, like in the plot below,
>>>>> where the normalization is completely off.
>>>>> I am using Rivet 2.4.0, but a similar problem I had with Rivet 2.2.0.
>>>>> 
>>>>> Any idea what this could be ?
>>>>> 
>>>>> Since I am merging 1000 yoda files, it's hard to tell whether it comes
>>>>> from one
>>>>> specific file.... however, all jobs finished without error...
>>>>> 
>>>>> Thanks a lot
>>>>> Cheers
>>>>> Hannes
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> ***********************************************************************
>>>>> Hannes Jung
>>>>> Email:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
>>>>> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>><mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
>>>>> mobile :+49 40 8998 93741
>>>>> http://www.desy.de/~jung <http://www.desy.de/~jung>
>>>>> Tel: +49 (0) 40 8998 3741
>>>>> Fax: +49 (0) 40 8994 3741
>>>>> DESY, CMS 01B/02.213
>>>>> Notkestr.85, 22603 Hamburg, FRG
>>>>> ***********************************************************************
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Rivet mailing list
>>>>> Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>
>>>>> <mailto:Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>><mailto:Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>>
>>>>> https://www.hepforge.org/lists/listinfo/rivet <https://www.hepforge.org/lists/listinfo/rivet>
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>>>> Particle Physics Expt Group, University of Glasgow
>>> 
>>> ***********************************************************************
>>> Hannes Jung
>>> Email:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>
>>> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>><mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
>>> mobile :+49 40 8998 93741
>>> http://www.desy.de/~jung <http://www.desy.de/~jung>
>>> Tel: +49 (0) 40 8998 3741
>>> Fax: +49 (0) 40 8994 3741
>>> DESY, CMS 01B/02.213
>>> Notkestr.85, 22603 Hamburg, FRG
>>> ***********************************************************************
>>> 
>>> 
>>> 
>> 
>> 
>> --
>> Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
>> Particle Physics Expt Group, University of Glasgow
> 
> ***********************************************************************
> Hannes Jung
> Email: Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de> <mailto:Hannes.Jung at desy.de <mailto:Hannes.Jung at desy.de>>
> mobile :+49 40 8998 93741
> http://www.desy.de/~jung <http://www.desy.de/~jung>
> Tel: +49 (0) 40 8998 3741
> Fax: +49 (0) 40 8994 3741
> DESY, CMS 01B/02.213
> Notkestr.85, 22603 Hamburg, FRG
> ***********************************************************************
> 
> 
> 
> 
> 
> _______________________________________________
> Rivet mailing list
> Rivet at projects.hepforge.org <mailto:Rivet at projects.hepforge.org>
> https://www.hepforge.org/lists/listinfo/rivet <https://www.hepforge.org/lists/listinfo/rivet>
***********************************************************************
Hannes Jung 
Email: Hannes.Jung at desy.de 
mobile :+49 40 8998 93741
http://www.desy.de/~jung                                  
Tel: +49 (0) 40 8998 3741         
Fax: +49 (0) 40 8994 3741
DESY, CMS 01B/02.213
Notkestr.85, 22603 Hamburg, FRG   
***********************************************************************



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.hepforge.org/lists-archive/rivet/attachments/20151014/f6a7130f/attachment.html>