"Massi File" Specifications
| |
"Massi File" Specifications |
|
Drafted by
Colin.Barschel@cern.ch
Massimiliano.Ferro-Luzzi@cern.ch
updated by
Christoph.Schwick@cern.ch
Jamie.Boyd@cern.ch
Summary files: the experiments provide data for each physics fills which had at least some time spent in physics, i.e. in the beam mode 'stable beams' (or 'unstable beams', if ever used in 2011). The data are requested only for the time spent in physics, but data during non-physics are also welcome.
The data will be put on experiment-owned AFS directories as quickly as possible at the end of a fill (e.g. within 24h).
The data files can be updated, if required (e.g. if better calibrations constants become available, if a more precise analysis technique leads to better results or to cope with occasional problems with DAQ, fill numbers, etc.).
ALICE: alicedir = /afs/cern.ch/alice/lhc/<year>/<lumi|lumiregion|beamgas>
ATLAS: atlasdir = /afs/cern.ch/atlas/www/GROUPS/DATAPREPARATION/DataSummary/lumifiles/<year>/<lumi|lumiregion|beamgas>
CMS: cmsdir = /afs/cern.ch/cms/CAF/CMSCOMM/COMM_GLOBAL/LHCFILES/<year>/<lumi|lumiregion|beamgas>
LHCb: lhcbdir = /afs/cern.ch/lhcb/online/TFC/LPCfiles/<year>/<lumi|lumiregion|beamgas>
where
<year> is the 4 digit year in which the data is produced, followed by one of the subdirectories <lumi|lumiregion|beamgas> which contain the data for various "topics" covered by the massi files (see below).
Data in each of these subdirectories is provided in one tgz or zip file per fill. The name of the file is the fill number. The extension is either tgz or zip. When the archive file is expanded, one directory carrying the fill-number as a name is created and all data files are contained in this directory.
Example: ATLAS lumi data for fill 1234 will be provided in the directory
/afs/cern.ch/atlas/www/GROUPS/DATAPREPARATION/DataSummary/lumifiles/2017/lumi/1234.tgz
Expanding the tgz file will result in one subdirectory called 1234/ in which all data files are contained.
The LPC is copying these data-files into an EOS area which is readable for everybody at CERN and from where users are supposed to get these files. This area is under the following path:
/eos/project/l/lpc/public/MassiFiles/<year>/measurements/<experiment>/<lumi|lumiregion|beamgas>
This section defines the names of the files contained in the archives described above. In the following
<fillnr> denotes the fill-number (4 digits or more in future)
<expt> denotes the experiments name: ALICE ATLAS CMS or LHCb
<rfbktbeam1> and <rfbktbeam2> denote the RF bucket of a specific bunch in beam 1 or 2. The valid range it 1 ... 35631 (i.e. bucket 1 denotes the first bucket after the abort gap where protons might be contained.)
Luminosity data:
Files listed in this section are contained in the archive under the .../<year>/lumi/ subdirectory.
Luminosity-region data:
Files listed in this section are contained in the archive under the .../<year>/lumiregion/ subdirectory.
Beam gas data for beam profile measurements:
Files listed in this section are contained in the archive under the .../<year>/beamgas/ subdirectory. In the past these files have only been provided by LHCb, however other experiments may re-evaluate if they can also provide this data.
Meta data:
Data in the Massi files should reflect the best possible data for a given fill in any moment. This implies that experiments will updated the data e.g. when more precise offline analysis results become available, or if the calibration for an ongoing (or past) year had been updated. In order to let users know about these updates so that they can judge the quality of the data for their analysis in each compressed archive, files with meta information is provided:
Definitions and rules for the following sections:
Coordinates: Please use the following coordinate frame:
Meta data files:
The file version.txt
This file contains a version number for the data in the relevant archive. Version numbers have the following format:
a[.b[.c[.d...]]][._u[.v...]]
where the letters denote integer numbers and everything in [ ] is optional. Numbers without a leading underscore (in the following named "plain" version numbers) have fixed meanings over the year with an experiment. For example an experiment could use for 'a' either 1 or 2 for "raw online values" or "values from offline processing". 'b' could be used to denote the calibration which might change some times over the year e.g. 1 for initial calibration, 2 for the calibration available 2 weeks after the first VdM scan and 3 for a new calibration available in September.
Version numbers with a leading '_' denote special features for a fill or a series of fills. For example if a lumi detector has a technical problem for some fills in a year this could be marked with a version _1. For a given fill an underscore integer may be defined only once, i.e. a version number like 1.2._1._2 is ok whereas 1.2._1._1 is illegal. The reason is that underscore numbers may be combined in the version number in arbitrary sequence.
The file version_annotation.txt
This file contains a textual description of the meaning of the version numbers. The file contains multiple lines formatted in one of the following ways:
n : description of the meaning of the nth integer in the version number
n u : description of the meaning for the nth integer having the value u
_k : meaning of the _k version number
An example could look like this:
1 : denotes the algorithm used to derive the luminosity data
2 : denotes the calibration used to obtain the luminosity data
1 1 : raw online luminometer data; available some hours after the fill
1 2 : offline processing of the silicon tracker data; highest precision; available after approx. 2 weeks
_1 : the main luminometer was down for 50% of the time. Online data is derived from calorimeters as a backup.
Additional Rules:
Annotations for version numbers with a leading '_' should only be given in the files of fills where they are used. On the contrary for the plain version numbers all annotations used up to that point in the year should be listed. This means for the plain version numbers it should be sufficient to look into the annotation file of the last fill of the year to get a full description of all plain numbers used.
If during the year new values for a plain version numberrs become necessary it is allowed to add these values from the first fill onwards which uses the new value. It is only allowed to add information to the annotation file, not to change or to delete information from it (if the latter becomes necessary for some reason, the LPC should be informed since all Massi files need to be re-scanned)
If during the ongoing year it turns out that an experiments needs an additional plain version number it is allowed to add this number from the fill onwards where it is needed without updating all the previous fills. It would be good to add in the description of the plain version number a comment which explains what the situation before the introduction of the version number was.
Example:
In June experiment X re-aligns all lumi detectors in a technical stop and wants to communicate this fact to users of the Massi files. The first physics fill with the new alignment is 8765. They introduce a new plain version number (e.g. the third since they use already 2 numbers for other things). Fills with the newly aligned lumi detectors will get the version number 1 for this new number. In the file version_annotation.txt two new lines are added. The first one describing the new version number and the second line describing the meaning of this version number to be 1, i.e.
3 : This number describes changes in the alignment of the lumi detectors. Fills with this numbers missing (before Fill 8765) have the initial alignment of the year.
3 1 : A new alignment of the lumi detector was introduced during TS2. The first fill with this alignment is fill 8765.
Further comments on this design can be read here .
The additional workload to maintain these version files should be kept small and the procedure is simple: The annotation for the plain version numbers only requires to add lines to the file when a new version number is used (or a new version number is introduced). The system allows for a very simple versioning but also allows for more detailed information to be communicated to the users of the files.
The system should allow users to write simple filters with scripts in order to process only data of their choice. This should be particularly simple when filtering on plain versions. The cases annotated with the version numbers preceded by an underscore need to be handled case by case.
It should be relatively easy to generate an overview table for all versions on a web page which will be automatically updated when new Massi files or updated Massi files are coming in. This will be implemented by the LPC and the page will be accessible from the LPC homepage.
Each lumi file contains:
time stab l dl sl dsl
where
time = UNIX time, i.e. in seconds since UTC Jan 1, 1970, 00:00:00, not counting leap seconds.
stab = stable beam flag: a float value between 0 and 1 which corresponds to the fraction of time spent in stable beams for this time bin.
l = luminosity in Hz/ub
dl = point-to-point error on luminosity in Hz/ub
sl = specific luminosity in Hz/ub (see below)
dsl = point-to-point error on specific luminosity in Hz/ub
Each lumireg file contains:
time stab x dx y dy z dz xsu dxsu ysu dysu zsu dzsu ax dax ay day
where
time = UNIX time, i.e. in seconds since UTC Jan 1, 1970, 00:00:00, not counting leap seconds.
stab = stable beam flag: a float value between 0 and 1 which corresponds to the fraction of time spent in stable beams for this time bin.
x, y, z = x-mean, y-mean, z-mean in mm
dx, dy, dz = "one sigma" errors on x-mean, y-mean, z-mean in mm
xsu, ysu, zsu = x-sigma, y-sigma, z-sigma, after unfolding the vtx detector resolution in mm
dxsu, dysu, dzsu = "one sigma" errors on xsu, ysu, zsu in mm
ax, ay = angle in x and y when projecting the lumi region on the xz and yz planes in rad
dax, day = "one sigma" errors on ax, ay in rad
Please remember that the agreed coordinate system is that of the LHC beams. Please convert from your local system into the LHC system.
Beam gas based profile measurement files for beam 1 and beam 2 contain:
time stab x dx y dy ax dax ay day xsu dxsu ysu dysu
where
time = UNIX time, i.e. in seconds since UTC Jan 1, 1970, 00:00:00, not counting leap seconds.
stab = stable beam flag: a float value between 0 and 1 which corresponds to the fraction of time spent in stable beams for this time bin.
x, y = x-mean, y-mean at z=0 in mm
dx, dy = "one sigma" errors on x-mean, y-mean in mm
ax, ay = angle in x and y when projecting the beam axis on the xz and yz planes in rad
dax, day = "one sigma" errors on ax, ay in rad
xsu, ysu = x-sigma, y-sigma, after unfolding the vtx detector resolution in mm
dxsu, dysu = "one sigma" errors on xsu, ysu in mm
The summary file contains:
a number of lines, one per "segment" of physics periods (normally, there is one segment per fill, but we have to cope with the possibility of stable beam interruptions during the fill) Each line contains:
timestart, timestop, plu, lui
where
timestart = start of stable beams of this segment in seconds UNIX time.
timestop = stop of stable beams of this segment in seconds UNIX time.
plu = peak lumi in Hz/ub in this segment
lui = integrated lumi in 1/ub in this segment
You can add comments, headers, footers ... to all your files. Please use the number sign '#' to signify this! Any extra information is to be put behind a '#' and finish with an end of line. This could be used for example to put the date/time/version of production of your file, your internal run numbers, or anything you want.