[Rivet] Compatibility of YODA reference data from Rivet with HEPData

[Graeme Watt]

Dear David,

Thanks, I think you mean yodadiff (not yodacmp).  You're right, it looks 
like yodadiff does much the same as my script, other than the HEPData 
download and optional comparison of annotations.  Maybe these features 
could be added to yodadiff?  In any case, it would be good to make such 
comparisons part of the validation and release procedure of each new 
Rivet analysis, which is apparently not happening at the moment (other 
than possibly for ATLAS analyses).

Best regards,
Graeme


On 22/05/18 09:54, David Grellscheid wrote:
> Hi Graeme,
>
> thanks for your email, we have started discussing it. Just one technical
> point, YODA comes with a yodacmp script already, which addresses many of
> your technical issues with comparisons. Maybe you can use that
> internally in your script?
>
> See you,
>
>   David
>
>
> On 21/05/2018 20:16, Graeme Watt wrote:
>> Dear Rivet developers,
>>
>> I wrote a Python script to compare a YODA reference data file, intended
>> for inclusion in Rivet, with the corresponding YODA file downloaded from
>> HEPData:
>>
>> https://github.com/HEPData/miscellaneous/blob/master/scripts/yoda_compare_hepdata.py
>>
>>    Example usage:  ./yoda_compare_hepdata.py ATLAS_2017_I1614149.yoda -i
>> 1614149 -a
>>
>> This means: compare a local YODA file "ATLAS_2017_I1614149.yoda" with a
>> YODA file downloaded from the HEPData record with INSPIRE ID "1614149"
>> and also compare YODA annotations "-a".  Since the HEPData YODA file
>> might contain additional analysis objects compared to the Rivet YODA
>> file, and since there might be inconsequential rounding errors or
>> differences in number formats, comparison using a simple "diff" of .yoda
>> files is not always adequate.
>>
>> I had a few problems with the YODA 1.7.0 software when writing the
>> Python script, which could perhaps be improved in future:
>>
>> * Calling dump() on Scatter objects does not output the same format as
>> in the input .yoda files, e.g. dumping a Scatter2D gives "HISTO1D" in
>> the output and the central value of the x bin is not output.  This might
>> be due to deficiencies in
>> https://yoda.hepforge.org/trac/browser/src/WriterFLAT.cc .
>> * I expected to be able to check (fuzzy) equality of two Scatter objects
>> using "s == s1", which seems to be implemented in C++ but not in Python.
>> * Checking (fuzzy) equality of two Point2D objects using "p == p1" is
>> implemented in Python, but it only compares the x axes and not the y
>> axes.  Similarly, for Point3D objects, the (fuzzy) equality operator
>> "==" only compares the x and y axes, but not the z axes.
>> * In the end, I just copied the definition of "fuzzyEquals" from the C++
>> code into my script and did my own comparisons, without relying on the
>> "==" operator for Point or Scatter objects.
>>
>> Recall that Holger Schulz made some similar comparisons in 2016 between
>> YODA reference data files from Rivet and from the old HepData, where he
>> found significant inconsistencies:
>>
>> https://www.hepforge.org/lists-archive/rivet/2016-October/007318.html
>> https://rivet.hepforge.org/trac/browser/contrib/devscripts/HepDataConsistency
>>
>>
>> While fixing all Rivet/HEPData inconsistencies is probably unrealistic
>> for now, we could at least start by ensuring that analyses added to new
>> Rivet releases include a YODA file that's compatible with the YODA
>> download given by HEPData.  My new script should be useful for these
>> purposes.  (This work was prompted by a conversation with Peter Skands
>> and Jon Butterworth last month in Durham.)  You're welcome to modify my
>> script as you need and include it in a future Rivet release.  The script
>> could be run by the experiment contact persons before they upload new
>> analyses to the Rivet "contrib" area.  It could also be run (perhaps in
>> a more automated way) by the Rivet developers when moving analyses from
>> the "contrib" area to a new Rivet release.  I just ran the script for
>> all the new analyses in the current Rivet "contrib" area (
>> https://www.hepforge.org/archive/rivet/contrib/ ) and it already turned
>> up some useful information:
>>
>> * ATLAS_2014_I1310835, ATLAS_2017_I1614149, and ATLAS_2017_I1624693, are
>> all compatible with HEPData.
>> * ATLAS_2016_I1502620 has multiple .yoda files which can't be handled by
>> my script.
>> * ATLAS_2017_I1625109 showed some apparent inconsistencies but the
>> problem looks to be on the HEPData side.  The dataset index starts at 0
>> (instead of 1) due to a bug in the hepdata-converter package (which I'll
>> fix) and the year in the analysis name is 2018 (instead of 2017) taken
>> from the journal publication.
>> * ALICE_2017_I1620477 is incompatible with HEPData.
>> * CMS_2016_I1487277 is compatible with HEPData.
>> * CMS_2012_I1111014 and CMS_2016_I1491950 are incompatible with HEPData.
>> * CMS_2017_I1499471, CMS_2017_I1635889, CMS_2018_I1662081 and
>> CMS_2018_I1663958 are all missing from HEPData.
>> * CMS_2017_I1605749 has a HEPData record, but the YODA conversion fails
>> due to a problem with the original HEPData submission.
>> * LHCF_2015_I1351909 is compatible with HEPData, but the annotations
>> differ.
>> * LHCF_2016_I1385877 is incompatible with HEPData.
>>
>> I hope this gives you something to discuss at this week's Rivet
>> workshop! :-)
>>
>> Best regards,
>> Graeme Watt (HEPData)
>>
>>
>>
>>
>> _______________________________________________
>> Rivet mailing list
>> Rivet at projects.hepforge.org
>> https://www.hepforge.org/lists/listinfo/rivet
>>