|
[Rivet] Compatibility of YODA reference data from Rivet with HEPDataGraeme Watt Graeme.Watt at durham.ac.ukWed Aug 15 12:32:02 BST 2018
Dear Rivet developers, As a follow-up to my mail from Monday, here is the output of my rivet-diffhepdata-all script for Rivet 2.6.1: http://ippp.dur.ac.uk/~watt/RivetDiffHEPData/Rivet-2.6.1/ "Of 378 Rivet analyses in ../Rivet-2.6.1/analyses, 73 (19.3%) were compatible and 305 (80.7%) were incompatible." Compared to Rivet 2.6.0, you added 6 compatible analyses and 13 incompatible analyses, so clearly you're not yet enforcing HEPData compatibility for new analyses. At least the percentage of compatible analyses has improved slightly (18.7% to 19.3%). I modified http://ippp.dur.ac.uk/~watt/RivetDiffHEPData/release-2-6-x/yodadiff to print numbers to five significant figures where there are differences. Here is a summary of the HEPData compatibility for each of the new analyses listed at http://rivet.hepforge.org/anadiffs.txt : ALICE_2017_I1620477 : NOT compatible (different 'y') ATLAS_2012_I1094061 : compatible ATLAS_2014_I1310835 : compatible ATLAS_2016_I1502620_W : multiple Rivet analyses for one INSPIRE ID ATLAS_2016_I1502620_W_EL : multiple Rivet analyses for one INSPIRE ID ATLAS_2016_I1502620_W_MU : multiple Rivet analyses for one INSPIRE ID ATLAS_2016_I1502620_Z : multiple Rivet analyses for one INSPIRE ID ATLAS_2016_I1502620_Z_EL : multiple Rivet analyses for one INSPIRE ID ATLAS_2016_I1502620_Z_MU : multiple Rivet analyses for one INSPIRE ID ATLAS_2017_I1604029 : no HEPData record exists ATLAS_2017_I1614149 : HEPData record has extra tables ATLAS_2017_I1624693 : compatible ATLAS_2017_I1625109 : HEPData record has extra tables ATLAS_2017_I1626105 : compatible ATLAS_2017_I1644367 : NOT compatible (different 'y') ATLAS_2017_I1645627 : no HEPData record exists ATLAS_2018_I1646686 : NOT compatible CMS_2012_I1111014 : missing from analyses.json CMS_2016_I1487277 : compatible CMS_2016_I1491950 : missing from analyses.json CMS_2017_I1499471 : missing from analyses.json CMS_2017_I1605749 : missing from analyses.json CMS_2017_I1610623 : no HEPData record exists CMS_2017_I1635889 : no HEPData record exists CMS_2018_I1662081 : no HEPData record exists CMS_2018_I1663452 : no HEPData record exists CMS_2018_I1663958 : missing from analyses.json LHCF_2015_I1351909 : compatible LHCF_2016_I1385877 : NOT compatible The idea of having multiple Rivet analyses (and therefore multiple .yoda files) for a single INSPIRE ID is really not compatible with a HEPData download. Could the multiple .yoda files be concatenated into one .yoda file per INSPIRE ID? Also, 5 of the new CMS analyses appear in analyses.html but not in analyses.json and hence are not processed by my script, so something must have gone wrong in producing the latest analyses.json file. Best regards, Graeme On 13/08/18 18:24, Graeme Watt wrote: > Dear Andy, > > Thanks for the comments. The new HEPData site already supports 1D > data types exported to YODA as a Scatter1D. Looking at the SVN > history of AidaFormatter.java in the old HepData software, I see that > you made a commit on 13th November 2009 with a message "Adding edge > guessing to AIDA formatter, for Rivet use". These artificial bin > widths are introduced not only for single-bin measurements, e.g. there > is a comment: > > // If there are several bins, make the internal > // edges match, and the external edges duplicate > // their +/- partners > > It would be better if these artificial bin widths were added in the > Rivet analysis code rather than in the YODA reference data, for the > cases where only the bin centres are given in HEPData without the > corresponding bin widths. > > I modified the latest "yodadiff" script from the hg repository to give > information on whether the 'x' or 'y' (or both) axes differ between > two YODA objects. I also added an option to compare annotations, > removed an unimplemented option to write output to a given file, and > added a check to the "eq(a, b)" function to catch the case of "a == > -b" mentioned in my last message. Can you please include these > changes in the next YODA release? > > http://ippp.dur.ac.uk/~watt/RivetDiffHEPData/release-2-6-x/yodadiff > > I upgraded the YODA version used in the HEPData export to 1.7.0 and > fixed the bug with the table index beginning at 0 or 2 instead of 1 > (as mentioned in my original message). Running my > "rivet-diffhepdata-all" script again over all analyses listed in > http://rivet.hepforge.org/analyses.json and taking the YODA files from > the Rivet hg repository gives output: > > http://ippp.dur.ac.uk/~watt/RivetDiffHEPData/release-2-6-x/ > > The main result has only improved by one analysis since last time: > > "Of 359 Rivet analyses in ../rivet/analyses, 67 (18.7%) were > compatible and 292 (81.3%) were incompatible." > > Again, 31 of these 292 incompatible analyses are missing a HEPData > record. Grepping the .txt files in the YodaDiffOutput/ directory for > information on which axes are affected, there are 8010 data points > (across 88 analyses) with only the 'x' axis affected, 7930 data points > (across 100 analyses) with only the 'y' axis affected, and 5791 data > points (across 50 analyses) with both 'x' and 'y' axes affected. Can > you please include the "rivet-diffhepdata" and "rivet-diffhepdata-all" > scripts in the bin/ directory of the next Rivet release? > > https://github.com/HEPData/miscellaneous/blob/master/scripts/rivet-diffhepdata > https://github.com/HEPData/miscellaneous/blob/master/scripts/rivet-diffhepdata-all > > Is the "rivet-diffhepdata" script already being used as part of the > validation procedure for analyses added to new Rivet releases? If > not, it would be good to build it into your validation machinery. Can > you please also modify the "rivet-mkanalysis" script to pass the Rivet > analysis name as an argument to the HEPData download URL as I > requested below? > > hdurl = "http://www.hepdata.net/record/ins%s?format=yoda&rivet=%s" % > (ANAINSPIREID, ANANAME) > > Best regards, > Graeme > > P.S. Have you seen https://github.com/hdembinski/histogram (maybe an > alternative to YODA if it becomes part of Boost)? > > > On 18/06/18 14:31, Andy Buckley wrote: >> Hi Graeme, >> >> Sorry to be coming late to this. >> >> I absolutely agree that we should get rid of the fake "unit bin >> width" for unbinned quantities -- that was an ad hoc solution from >> deep in prehistory to HepData's fundamental data type being binned, >> and now that's not the case we should move away from it. Writing out >> HepData 1D data types to YODA as a Scatter1D would be best (and then, >> as Jon as noted before, we just need to make sure that those get >> plotted in a reasonable way). >> >> The level of disagreement between HepData and Rivet records is >> staggering, and I'm sorry that we've not had the manpower to deal >> with that. Holger was previously working on it, and I think found >> similar numbers, but then moved institute (to FNAL) before he could >> act on it; maybe there is still some possibility to follow it up, Holger? >> >> Since you have diagnostics scripts operational at the moment, Graeme, >> could you produce a list/summary of how many records only differ by >> the /binning/ of observables, and maybe another with a /loose/ >> consistency check on bin contents? That would give us some insight >> into how many differ because the Rivet files "just" need to be synced >> with HepData, and how many have completely inconsistent dataset >> naming. The latter are the ones that will need some analysis code >> updates. >> >> I appreciate that this task maybe falls into a gap between purely >> clerical and something requiring physics insight. However, if we can >> manage to isolate a purely clerical component then help from Joanna >> would be really valuable: as you're aware we have no dedicated >> manpower whatsoever on the Rivet side, and tasks of this sort are the >> hardest to motivate postdocs to work on. I am also pessimistic that >> we can get help from the LHC experiments (who supplied us with the >> inconsistent Rivet/HD records in the first place...), but we can try: >> breakdown of the numbers by experiment would also be very useful. >> >> Thanks, >> Andy >> >> *Dr Andy Buckley, Lecturer / Royal Society University Research Fellow* >> Particle Physics Experiment Group, University of Glasgow >> >> >> >> On Jun 15 2018, at 2:20 pm, Graeme Watt <Graeme.Watt at durham.ac.uk> wrote: >> >> >> Dear Peter (and Rivet developers), >> >> No, I don't think this is something that Joanne (or Keith) could >> help with. >> >> https://github.com/HEPData/miscellaneous/blob/master/scripts/rivet-diffhepdata-all >> >> I wrote another script "rivet-diffhepdata-all" that loops over >> all Rivet analyses listed in >> http://rivet.hepforge.org/analyses.json and compares each Rivet >> .yoda file with the HEPData download. It calls functions from >> the previous "rivet-diffhepdata" script which in turn calls >> "yodadiff". >> >> I added the URL option to pass the Rivet analysis name to HEPData >> when requesting a YODA conversion, e.g. >> >> https://hepdata.net/record/ins319520?format=yoda&rivet=ALEPH_1991_S2435284 >> >> This is necessary, for example, for Rivet analysis names >> containing the SPIRES ID rather than the INSPIRE ID. The >> "rivet-mkanalysis" script could be modified accordingly: >> >> https://rivet.hepforge.org/trac/browser/bin/rivet-mkanalysis#L139 >> hdurl = *MailScanner has detected a possible fraud attempt from >> "www.hepdata.net" claiming to be* >> <http://www.hepdata.net/record/ins%s?format=yoda&rivet=%s>*MailScanner >> has detected a possible fraud attempt from "www.hepdata.net" >> claiming to be* >> "http://www.hepdata.net/record/ins%s?format=yoda&rivet=%s" >> <http://www.hepdata.net/record/ins%s?format=yoda&rivet=%s> % >> (ANAINSPIREID, ANANAME) >> >> The web directory >> http://ippp.dur.ac.uk/~watt/RivetDiffHEPData/Rivet-2.6.0/ >> <http://ippp.dur.ac.uk/%7Ewatt/RivetDiffHEPData/Rivet-2.6.0/> >> contains the output of running: >> >> rivet-diffhepdata-all -r ../Rivet-2.6.0/analyses -d HEPDataYoda >> -o YodaDiffOutput > rivet-diffhepdata-all.txt >> >> The summary line doesn't look too promising: >> >> "Of 359 Rivet analyses in ../Rivet-2.6.0/analyses, 66 (18.4%) >> were compatible and 293 (81.6%) were incompatible." >> >> Here, compatibility is defined as a zero exit status returned by >> "yodadiff". Only 31 of these 293 incompatible analyses are >> missing a HEPData record. >> >> Note that the "yodadiff" script gives a ZeroDivisionError in the >> function "eq(a, b)" when "a" and "b" have opposite sign due to >> the return value: >> >> return abs(float(a) - float(b))/(float(a) + float(b)) < opts.TOL >> >> For example, d01-x01-y01 of ATLAS_2013_I1190187.yoda distributed >> with Rivet has yerr- = 6.8 and yerr+ = -6.8. The HEPData table ( >> http://www.hepdata.net/record/ins1190187?version=1&table=Table1 ) >> gives both yerr- and yerr+ as 1.212930e+01, so it seems that the >> Rivet .yoda file contains only the statistical error (with a >> wrong sign for yerr+). The Rivet .yoda file also assigns an >> artificial bin width of 1 for sqrt(s), whereas the HEPData table >> does not assign a bin width for sqrt(s). Looking at the YODA >> export from the old HepData site: >> >> http://hepdata.cedar.ac.uk/view/ins1190187/d1/yoda >> >> again there are zero xerr- and xerr+ values, but the AIDA export: >> >> http://hepdata.cedar.ac.uk/view/ins1190187/d1/aida >> >> has a unit bin width written by "AidaFormatter.java" with a >> comment (by Andy?): >> >> // If there's only one bin and it has no width, give >> // it unit width so that it can be filled and the height >> // doesn't go mad when Rivet tries to use it. >> >> I would argue that any construction of artificial bin widths >> would be better handled on the Rivet side rather than in the >> HEPData export of the data. Removing artificial bin widths from >> the Scatter objects in the Rivet .yoda files would resolve some >> of the incompatibilities between HEPData and Rivet, but many more >> would remain. I don't propose a universal solution for now, but >> just want to start by identifying where the differences lie. It >> will be interesting to monitor whether the degree of >> incompatibility, now easily quantified by my new >> "rivet-diffhepdata-all" script, improves for subsequent Rivet >> releases. >> >> Best regards, >> Graeme Watt (HEPData) >> >> >> On 25/05/18 23:17, Peter Skands wrote: >> >> >> Hi All, >> >> Agree it sounds like it could make sense as part of Rivet, >> and could make a real difference by making the step to check >> consistency almost trivial; easy to run when developing / >> releasing analyses. In the latter case, feedback could go >> back to the people submitting a new analysis if there is a >> discrepancy, which would then put the burden of ensuring >> consistency mainly on the people who contribute the analyses, >> without adding significantly to the Rivet / HepData authors. >> Regarding the existing / old analyses, Graeme, any chance you >> think of the person in Durham having a go at the backlog of >> old analyses, or would that be too challenging for her? It >> might be worth preparing an example, show it to Keith, and >> see what he thinks? >> >> Cheers, >> Peter >> >> — >> *PETER SKANDS* >> Associate Professor >> >> *School of Physics and Astronomy* >> Monash University >> 10 College Walk, Clayton Campus >> Melbourne, VIC 3800 >> Australia >> >> T: +61 3 990 53692 <tel://+61%203%20990%2053692> >> E: peter.skands at monash.edu <mailto:peter.skands at monash.edu> >> W: skands.physics.monash.edu <http://skands.physics.monash.edu/> >> >> On 26 May 2018 at 2:52:23 am, Graeme Watt >> (graeme.watt at durham.ac.uk <mailto:graeme.watt at durham.ac.uk>) >> wrote: >> >> Dear David, >> >> OK, here's another much simpler Python script that >> downloads the YODA >> file from HEPData and then calls yodadiff: >> >> https://github.com/HEPData/miscellaneous/blob/master/scripts/rivet-diffhepdata >> >> I'm not sure if this even warrants a script, given that the >> functionality of: >> >> rivet-diffhepdata ATLAS_2017_I1614149.yoda -i 1614149 >> >> could be obtained with two commands, e.g. >> >> curl -L >> https://hepdata.net/record/ins1614149?format=yoda | tar zx >> yodadiff ATLAS_2017_I1614149.yoda >> HEPData-ins1614149-v2-yoda.yoda >> >> The yodadiff script gives more detailed output of >> differences than my >> previous script, but it does not compare annotations >> (maybe this >> functionality could be added to yodadiff as an option?). >> Also, the >> yodadiff script flags additional analysis objects (like >> covariance >> matrices) that are present in the HEPData YODA file but >> not in the Rivet >> YODA file (as is the case for ATLAS_2017_I1614149), >> whereas my previous >> script was specifically written to ignore these >> differences. I suppose >> this is OK if the user just ignores the warnings from >> yodadiff about >> additional analysis objects in the HEPData YODA file. >> >> Best regards, >> Graeme >> >> >> On 25/05/18 11:01, David Grellscheid wrote: >> > Hi Graeme, >> > >> > Yes, I did mean yodadiff, sorry! There's no way it will >> start to include >> > a Hepdata download option, it is meant to do one job of >> comparing two >> > yoda files, which may have nothing at all to do with >> HEP or Rivet. >> > >> > The download option could be provided (in rivet/bin, >> not yoda/bin!) by a >> > thin layer over the top of yoadiff, though. If you'd >> like to adapt your >> > script in that way, we'd be happy to include it in the >> rivet distribution. >> > >> > See you, >> > >> > David >> > >> > >> > On 22/05/2018 12:20, Graeme Watt wrote: >> >> Dear David, >> >> >> >> Thanks, I think you mean yodadiff (not yodacmp). >> You're right, it looks >> >> like yodadiff does much the same as my script, other >> than the HEPData >> >> download and optional comparison of annotations. >> Maybe these features >> >> could be added to yodadiff? In any case, it would be >> good to make such >> >> comparisons part of the validation and release >> procedure of each new >> >> Rivet analysis, which is apparently not happening at >> the moment (other >> >> than possibly for ATLAS analyses). >> >> >> >> Best regards, >> >> Graeme >> >> >> >> >> >> On 22/05/18 09:54, David Grellscheid wrote: >> >>> Hi Graeme, >> >>> >> >>> thanks for your email, we have started discussing it. >> Just one technical >> >>> point, YODA comes with a yodacmp script already, >> which addresses many of >> >>> your technical issues with comparisons. Maybe you can >> use that >> >>> internally in your script? >> >>> >> >>> See you, >> >>> >> >>> David >> >>> >> >>> >> >>> On 21/05/2018 20:16, Graeme Watt wrote: >> >>>> Dear Rivet developers, >> >>>> >> >>>> I wrote a Python script to compare a YODA reference >> data file, intended >> >>>> for inclusion in Rivet, with the corresponding YODA >> file downloaded from >> >>>> HEPData: >> >>>> >> >>>> >> https://github.com/HEPData/miscellaneous/blob/master/scripts/yoda_compare_hepdata.py >> >>>> >> >>>> >> >>>> Example usage: ./yoda_compare_hepdata.py >> ATLAS_2017_I1614149.yoda -i >> >>>> 1614149 -a >> >>>> >> >>>> This means: compare a local YODA file >> "ATLAS_2017_I1614149.yoda" with a >> >>>> YODA file downloaded from the HEPData record with >> INSPIRE ID "1614149" >> >>>> and also compare YODA annotations "-a". Since the >> HEPData YODA file >> >>>> might contain additional analysis objects compared >> to the Rivet YODA >> >>>> file, and since there might be inconsequential >> rounding errors or >> >>>> differences in number formats, comparison using a >> simple "diff" of .yoda >> >>>> files is not always adequate. >> >>>> >> >>>> I had a few problems with the YODA 1.7.0 software >> when writing the >> >>>> Python script, which could perhaps be improved in >> future: >> >>>> >> >>>> * Calling dump() on Scatter objects does not output >> the same format as >> >>>> in the input .yoda files, e.g. dumping a Scatter2D >> gives "HISTO1D" in >> >>>> the output and the central value of the x bin is not >> output. This might >> >>>> be due to deficiencies in >> >>>> >> https://yoda.hepforge.org/trac/browser/src/WriterFLAT.cc . >> >>>> * I expected to be able to check (fuzzy) equality of >> two Scatter objects >> >>>> using "s == s1", which seems to be implemented in >> C++ but not in Python. >> >>>> * Checking (fuzzy) equality of two Point2D objects >> using "p == p1" is >> >>>> implemented in Python, but it only compares the x >> axes and not the y >> >>>> axes. Similarly, for Point3D objects, the (fuzzy) >> equality operator >> >>>> "==" only compares the x and y axes, but not the z axes. >> >>>> * In the end, I just copied the definition of >> "fuzzyEquals" from the C++ >> >>>> code into my script and did my own comparisons, >> without relying on the >> >>>> "==" operator for Point or Scatter objects. >> >>>> >> >>>> Recall that Holger Schulz made some similar >> comparisons in 2016 between >> >>>> YODA reference data files from Rivet and from the >> old HepData, where he >> >>>> found significant inconsistencies: >> >>>> >> >>>> >> https://www.hepforge.org/lists-archive/rivet/2016-October/007318.html >> >>>> >> https://rivet.hepforge.org/trac/browser/contrib/devscripts/HepDataConsistency >> >>>> >> >>>> >> >>>> >> >>>> While fixing all Rivet/HEPData inconsistencies is >> probably unrealistic >> >>>> for now, we could at least start by ensuring that >> analyses added to new >> >>>> Rivet releases include a YODA file that's compatible >> with the YODA >> >>>> download given by HEPData. My new script should be >> useful for these >> >>>> purposes. (This work was prompted by a conversation >> with Peter Skands >> >>>> and Jon Butterworth last month in Durham.) You're >> welcome to modify my >> >>>> script as you need and include it in a future Rivet >> release. The script >> >>>> could be run by the experiment contact persons >> before they upload new >> >>>> analyses to the Rivet "contrib" area. It could also >> be run (perhaps in >> >>>> a more automated way) by the Rivet developers when >> moving analyses from >> >>>> the "contrib" area to a new Rivet release. I just >> ran the script for >> >>>> all the new analyses in the current Rivet "contrib" >> area ( >> >>>> https://www.hepforge.org/archive/rivet/contrib/ ) >> and it already turned >> >>>> up some useful information: >> >>>> >> >>>> * ATLAS_2014_I1310835, ATLAS_2017_I1614149, and >> ATLAS_2017_I1624693, are >> >>>> all compatible with HEPData. >> >>>> * ATLAS_2016_I1502620 has multiple .yoda files which >> can't be handled by >> >>>> my script. >> >>>> * ATLAS_2017_I1625109 showed some apparent >> inconsistencies but the >> >>>> problem looks to be on the HEPData side. The >> dataset index starts at 0 >> >>>> (instead of 1) due to a bug in the hepdata-converter >> package (which I'll >> >>>> fix) and the year in the analysis name is 2018 >> (instead of 2017) taken >> >>>> from the journal publication. >> >>>> * ALICE_2017_I1620477 is incompatible with HEPData. >> >>>> * CMS_2016_I1487277 is compatible with HEPData. >> >>>> * CMS_2012_I1111014 and CMS_2016_I1491950 are >> incompatible with HEPData. >> >>>> * CMS_2017_I1499471, CMS_2017_I1635889, >> CMS_2018_I1662081 and >> >>>> CMS_2018_I1663958 are all missing from HEPData. >> >>>> * CMS_2017_I1605749 has a HEPData record, but the >> YODA conversion fails >> >>>> due to a problem with the original HEPData submission. >> >>>> * LHCF_2015_I1351909 is compatible with HEPData, but >> the annotations >> >>>> differ. >> >>>> * LHCF_2016_I1385877 is incompatible with HEPData. >> >>>> >> >>>> I hope this gives you something to discuss at this >> week's Rivet >> >>>> workshop! :-) >> >>>> >> >>>> Best regards, >> >>>> Graeme Watt (HEPData) >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> _______________________________________________ >> >>>> Rivet mailing list >> >>>> Rivet at projects.hepforge.org >> <mailto:Rivet at projects.hepforge.org> >> >>>> https://www.hepforge.org/lists/listinfo/rivet >> >>>> >> >> >> _______________________________________________ >> Rivet mailing list >> Rivet at projects.hepforge.org >> https://www.hepforge.org/lists/listinfo/rivet >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://www.hepforge.org/lists-archive/rivet/attachments/20180815/b0082861/attachment.html>
More information about the Rivet mailing list |