<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Andy<div class=""> </div><div class="">thanks a lot for your mail and the suggestion.</div><div class=""> </div><div class="">Why do you say, it would be better outside Rivet?</div><div class=""> </div><div class="">Maybe it is not clear to me, how the "un-normalising" in yodamerge is done.</div><div class="">Is perhaps the problem the large (or small) number which I took to replace infty or zero ?</div><div class=""> </div><div class="">The point is, that we have the measurements in such a way, and it would be just</div><div class="">fine, to have a Rivet plugin, which would fill the histos such that it can be</div><div class="">plotted easily with rivet-mkhtml.</div><div class=""> </div><div class="">Thanks again</div><div class=""> </div><div class="">Best</div><div class="">Hannes</div><div class=""> <div><blockquote type="cite" class=""><div class="">On 14.10.2015, at 23:39, Andy Buckley <<a href="mailto:andy.buckley@cern.ch" class="">andy.buckley@cern.ch</a>> wrote:</div> <div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Hi Hannes,<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I think this sort of case is better dealt with by *not* normalizing the DY+jet histos, or normalizing them to an area of 1. Then use yodamerge to combine both the Z+jet and DY+jet pt plots and write a (very) little Python script to do the last bit:<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">import yoda<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">hs = yoda.read("merged.yoda")<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">norm = hs["/MYANA/DYpt"].integral()<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">hs["/MYANA/DYpt"].normalize(norm)<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">yoda.write(hs, "rescaled.yoda")<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Not tested, but that's the basic idea. Let us know if that works for you.<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Andy<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On 14/10/15 18:23, Hannes Jung wrote:<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Hi Andy et al the new yodamerge works much better now, this is really great. But I have now another problem (which I tried to solve with the hack from before): I fill histograms and normalize them to some xsection. Now it can happen, that the histo is filled, but the normalization is zero, for example when I plot the pt of DY+jet normalized to Z+jet, then it can happen that I have no Z+jet events, while the histo for DY+jet is filled, if the DY mass is small. Now, I want to run many jobs in parallel, so that I have enough statistics, and then I want to merge all histos. In the case above, I just artificially put a very small xsection, so that I still can normalize the histo, and I was hoping, that by merging this would work. But apparently, yodamerge does not really like value of 1E99=1/xsect where xsect is put to a small number, but not zero. At the moment I am a bit stuck with this, just don't know, how I can tell the merging at the end to take into account a very small normalization factor for some runs. Any idea on this would be very welcome thanks a lot Cheers Hannes <blockquote type="cite" class="">On 13.10.2015, at 17:38, Andy Buckley <<a href="mailto:andy.buckley@cern.ch" class="">andy.buckley@cern.ch</a> <<a href="mailto:andy.buckley@cern.ch" class="">mailto:andy.buckley@cern.ch</a>>> wrote: Oops, I didn't realise I'd taken the discussion with Hannes off-list. Here's the status: looks like he was experiencing a side-effect of a hack around merge failures for completely empty histos. We fixed that in the latest YODA, so hopefully the hack can also be removed and all will be well... Andy -------- Forwarded Message -------- Subject:Re: [Rivet] Problem with yodamerge Date:Tue, 13 Oct 2015 15:42:57 +0200 From:Hannes Jung <<a href="mailto:hannes.jung@desy.de" class="">hannes.jung@desy.de</a> <<a href="mailto:hannes.jung@desy.de" class="">mailto:hannes.jung@desy.de</a>>> To:andy.buckley@<a href="http://cern.ch/" class="">cern.ch</a> <<a href="mailto:andy.buckley@cern.ch" class="">mailto:andy.buckley@cern.ch</a>> Hi Andy ahhh.... I put by hand a normalization factor very small (so that 1/factor is very large), so that when the histo is normalized with this factor it becomes zero... but perhaps this screws up when unpacking the normalized histos in yodamerge. The problem occurred in previous versions when I had a histo with no entires, then yodamerge had problems with the normalization... this is why I introduced this hack.... perhaps I should check again whether it works now if I have sometimes a histo without entires and without normalization factor... Cheers Hannes <blockquote type="cite" class="">On 13.10.2015, at 15:33, Andy Buckley <<a href="mailto:andy.buckley@cern.ch" class="">andy.buckley@cern.ch</a> <<a href="mailto:andy.buckley@cern.ch" class="">mailto:andy.buckley@cern.ch</a>> <<a href="mailto:andy.buckley@cern.ch" class="">mailto:andy.buckley@cern.ch</a>>> wrote: Hi Hannes, Yes, those ScaledBy values are quite extreme. They are created by the calls to scale() or normalize() in Rivet, and the value just corresponds to the scale/normalization factor. How are you managing to generate factors of nearly 10^100?! In the latest version of YODA, the --assume-normalized flag for yodamerge doesn't do anything. We changed the default logic since it seemed that lots of people were using that flag "blindly" to work around a class of problem for which it wasn't really the right answer. But since it's now a do-nothing flag it shouldn't be related to your ScaledBy problems. Andy On 13/10/15 11:05, Hannes Jung wrote: <blockquote type="cite" class="">Hi Andy, Chris thanks a lot for your reply. Yes, I am trying to merge normalised Histograms. ahh, and yes, I have some strange Scaledby values: 5.1279402177630292e+97 I guess this is what causes the problem. Where are those set, is this an issue of statistics ? The cpmmand I use is yodamerge --assume-normalized yoda-files Thanks again Cheers Hannes <blockquote type="cite" class="">On 13.10.2015, at 11:23, Andy Buckley <<a href="mailto:andy.buckley@cern.ch" class="">andy.buckley@cern.ch</a> <<a href="mailto:andy.buckley@cern.ch" class="">mailto:andy.buckley@cern.ch</a>> <<a href="mailto:andy.buckley@cern.ch" class="">mailto:andy.buckley@cern.ch</a>> <<a href="mailto:andy.buckley@cern.ch" class="">mailto:andy.buckley@cern.ch</a>>> wrote: Hi Hannes, What sort of data object are you merging here? From the y-axis it looks like a normalized histogram rather than {unnormalized histo, profile, ratio}. If that's right, yodamerge will look for the ScaledBy attribute of that plot to reverse the normalisation, add all the unnormalised histos together, then rescale by the weighted mean ScaledBy. It *shouldn't* be sensitive to one run having much smaller stats than the others, certainly not in a way that would show up like this. If you can provide a bit more detail of what this object is, and how you're trying to merge the runs, we can try to help. You can also plot or take a look inside the .yoda files to see if the individual runs look sane, i.e. have reasonable fill counts and ScaledBy values. Andy On 13/10/15 08:22, Hannes Jung wrote: <blockquote type="cite" class="">Dear Rivet experts I am using yodamerge to merge several yoda files (actually it is 1000 or so), to get reasonable statistics in a reasonable computing time. Sometimes I obtain strange results, like in the plot below, where the normalization is completely off. I am using Rivet 2.4.0, but a similar problem I had with Rivet 2.2.0. Any idea what this could be ? Since I am merging 1000 yoda files, it's hard to tell whether it comes from one specific file.... however, all jobs finished without error... Thanks a lot Cheers Hannes *********************************************************************** Hannes Jung Email:<a href="mailto:Hannes.Jung@desy.de" class="">Hannes.Jung@desy.de</a> <<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>><<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>> <<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>><<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>> mobile :+49 40 8998 93741 <a href="http://www.desy.de/~jung" class="">http://www.desy.de/~jung</a> Tel: +49 (0) 40 8998 3741 Fax: +49 (0) 40 8994 3741 DESY, CMS 01B/02.213 Notkestr.85, 22603 Hamburg, FRG *********************************************************************** _______________________________________________ Rivet mailing list <a href="mailto:Rivet@projects.hepforge.org" class="">Rivet@projects.hepforge.org</a> <<a href="mailto:Rivet@projects.hepforge.org" class="">mailto:Rivet@projects.hepforge.org</a>> <<a href="mailto:Rivet@projects.hepforge.org" class="">mailto:Rivet@projects.hepforge.org</a>><<a href="mailto:Rivet@projects.hepforge.org" class="">mailto:Rivet@projects.hepforge.org</a>> <a href="https://www.hepforge.org/lists/listinfo/rivet" class="">https://www.hepforge.org/lists/listinfo/rivet</a> </blockquote> -- Dr Andy Buckley, Lecturer / Royal Society University Research Fellow Particle Physics Expt Group, University of Glasgow </blockquote> *********************************************************************** Hannes Jung Email:<a href="mailto:Hannes.Jung@desy.de" class="">Hannes.Jung@desy.de</a> <<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>> <<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>><<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>> mobile :+49 40 8998 93741 <a href="http://www.desy.de/~jung" class="">http://www.desy.de/~jung</a> Tel: +49 (0) 40 8998 3741 Fax: +49 (0) 40 8994 3741 DESY, CMS 01B/02.213 Notkestr.85, 22603 Hamburg, FRG *********************************************************************** </blockquote> -- Dr Andy Buckley, Lecturer / Royal Society University Research Fellow Particle Physics Expt Group, University of Glasgow </blockquote> *********************************************************************** Hannes Jung Email:<a href="mailto:Hannes.Jung@desy.de" class="">Hannes.Jung@desy.de</a> <<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>><<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>> mobile :+49 40 8998 93741 <a href="http://www.desy.de/~jung" class="">http://www.desy.de/~jung</a> Tel: +49 (0) 40 8998 3741 Fax: +49 (0) 40 8994 3741 DESY, CMS 01B/02.213 Notkestr.85, 22603 Hamburg, FRG *********************************************************************** _______________________________________________ Rivet mailing list <a href="mailto:Rivet@projects.hepforge.org" class="">Rivet@projects.hepforge.org</a> <<a href="mailto:Rivet@projects.hepforge.org" class="">mailto:Rivet@projects.hepforge.org</a>> <a href="https://www.hepforge.org/lists/listinfo/rivet" class="">https://www.hepforge.org/lists/listinfo/rivet</a> </blockquote> *********************************************************************** Hannes Jung Email: <a href="mailto:Hannes.Jung@desy.de" class="">Hannes.Jung@desy.de</a> <<a href="mailto:Hannes.Jung@desy.de" class="">mailto:Hannes.Jung@desy.de</a>> mobile :+49 40 8998 93741 <a href="http://www.desy.de/~jung" class="">http://www.desy.de/~jung</a> Tel: +49 (0) 40 8998 3741 Fax: +49 (0) 40 8994 3741 DESY, CMS 01B/02.213 Notkestr.85, 22603 Hamburg, FRG *********************************************************************** </blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">-- <br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Dr Andy Buckley, Lecturer / Royal Society University Research Fellow<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Particle Physics Expt Group, University of Glasgow</div></blockquote></div> <div apple-content-edited="true" class=""> *********************************************************************** Hannes Jung Email: <a href="mailto:Hannes.Jung@desy.de" class="">Hannes.Jung@desy.de</a> mobile :+49 40 8998 93741 <a href="http://www.desy.de/~jung" class="">http://www.desy.de/~jung</a> Tel: +49 (0) 40 8998 3741 Fax: +49 (0) 40 8994 3741 DESY, CMS 01B/02.213 Notkestr.85, 22603 Hamburg, FRG *********************************************************************** </div> </div></body></html>