[Rivet] Patch for Rivet to allow ordering of analyses and more fleshed out rivet-nopy

[Andy Buckley]

Hi,

Can you explain what you mean by analysis-ordering? The analyses run 
independently of each other -- there should be no meaning to "ordering". 
I am wary of adding more run-options to the library unless there is a 
very compelling use-case.

rivet-nopy is intentionally very minimal, both because we don't want to 
support any substantial features beyond basic analysis-running, and 
because it's a simple example of how a steering code can run Rivet. What 
have you "fleshed out"? It might be that it's better for you to just 
keep this extension for personal use.

It would also be good to keep these two distinct patches separate from 
each other, so we can apply them "atomically" -- or do they depend on 
each other somehow?

Regarding parallelisation, we are in the lucky position of event streams 
being "embarrassingly parallel"... as you've shown. You can already 
handle most situations using rivet and yodamerge (or scripted uses of 
their APIs), and developments in the pipeline for v3 will make the 
"re-finalize" step possible as part of the core Rivet behaviour without 
needing to specify special run-modes. It's taken a lot longer than hoped 
to get this working, though: a lot of potential pitfalls, and a lack of 
time! (Which is why your last patch hasn't been applied yet -- I thought 
David had done so, but am actually quite glad he hasn't since I had some 
queries about that, too!)

Andy



On 01/05/17 13:26, cholm wrote:
> Hi Developers,
>
> I've made a small patch for Rivet that
>
> - allow ordering of analyses
> - fleshes out the program rivet-nopy (useful for debugging)
>
> I attach the patch and a text file that explains the changes in some
> more detail.
>
> I sent you a patch some time ago that makes configure figure out which
> FastJet plugins are available on the target system, but I cannot see
> that it has been applied yet.
>
> Also, another thing I was thinking about: Parallelization of event
> processing.
>
> I don't think it makes sense to parallelise AnalysisHandler it self -
> that is, there's no real need for calling Analysis::analyze in separate
> threads - I just don't think one will gain much from that, and the extra
> complexity in terms of synchronization could be too much and a killer.
>
> So instead, I think one could use another paradigme - to parallelise on
> the events.  Suppose we had N inputs (files, fifo's, what not), we could
> then start N identical processes - one on each of the inputs.  E.g.,
> with the input "files"
>
>   input1
>   input2
>   input3
>
> on which we want to run the analyses ana1, ana2, ana3, and ana4, we
> could do
>
>   rivet -m process -a ana1,ana2,ana3,ana4 -o output1 input1
>   rivet -m process -a ana1,ana2,ana3,ana4 -o output2 input2
>   rivet -m process -a ana1,ana2,ana3,ana4 -o output3 input3
>
> However, we want to collect the outputs into one merged output and then
> run Analysis::finalize for each of ana1, ana2, ana3, and ana4 on the
> merged output.  To do that, we'd call (another) application which
>
> - reads the inputs
> - merges what can be merged (e.g., YODA::Histo1D objects with the same
> path)
> - sets up the same analyses objects
> - sets the appropriate analysis objects on each analyses
> - calls finalize on each analyses object
>
> e.g.,
>
>   rivet -m finalize -a ana1,ana2,ana3,ana4 -o final output1 output2 output3
>
> This would require that all classes derived from Analysis retrieves the
> needed analysis objects from the internal _analysisObjects container in
> Analysis::finalize - other than that, the user code could stay as is.
> PeRhaps the best option would be to add an interface for that - e.g.,
> Analysis::initFinalize except to bring internal variables in order and
> called before Analysis::finalize.
>
> Yours,
>
>
>
> _______________________________________________
> Rivet mailing list
> Rivet at projects.hepforge.org
> https://www.hepforge.org/lists/listinfo/rivet
>


-- 
Dr Andy Buckley, Lecturer / Royal Society University Research Fellow
Particle Physics Expt Group, University of Glasgow