Offline Checklist

Before you continue make sure Java and JavaScript are supported and enabled.

This page contains detailed instructions and examples for the SLD Offline checklist.

The goal is to monitor the data quality promptly as the production processing wends it way from the online tape to full reconstruction.

It is the responsibility of the person on the day "non-expert" shift to run through this checklist once a day.

Data problems are to be reported ASAP to the processing overseer, (currently Erez Etzion) or one of the production processing crew (Richard Dubois, Joe Perl, Gary Bower, and Karen Heidenreich) as appropriate.

That follows may seem to be a checklist of numbers to compare to standards and make a simple binary decision on data quality. Rather you should use the detailed questions to get yourself oriented and warmed up to use your physics skills to think about the run output and look for problems. If data quality was just a matter of some binary decisions based on some numbers we would just have the computer do it and we wouldn't need you!

You are responsible for any stages of run processing that finish during your shift perio. The easiest way to determine this is to use the links from the checklist bellow. If some of the runs are 'catch-up' processing on older missed runs, concentrate on the runs from the day/previous day that are most time critical to report on.

Readme file

See the OFFSITE README file for the latest temporary kludges to the monitoring. It will be updated regularly as problems come and go.

Processing Stages

Once the end-run button is pressed in the online, the acquisition (ACQ) data tape is released for further processing in the SLACVX cluster.

The SLDPM server is notified that the tape is ready for processing. It copies the tape to disk, followed by running a PASS1 filter, followed by a full reconstruction and PASS2 filter.

Relevant Pages from the STATUS DISPLAY System

Page 12: Offline: gives details on the 3 stages of processing for the last run processed. Things to watch are error messages, the number of truncations, reasonable agreement between the SLC estimation and the SLD number of Zs, the level of polarization and it error (to be around 78-80 +/- 0.5%), the distribution of the reconstructed events and and the hadrons parameters similiar to that in the following ( example. )
Page 14: Offline Zs: includes plot of the number of SLD Reconstructed Zs. ( example. )
page 15: Processing: gives list offline jobs currently queued or running in the three prcessing stages. In green the runs that succesfuly finished one stage, In yellow runs that are currently been processed, where what you mainly need to worry about are the red runs (i.e. Failed in that stage). ( example. )

Tape copy (ACQCOPY)

This stage performs two functions. It copies the input tape to disk for effective access in subsequent steps. The second function is to monitor the trigger records for inconsistencies. The run monitor is notified if any are found.

ACQCOPY Duties

The only duty here is to determine whether trigger errors ( example ) have been flagged, and notify the run coordinator if there have.

For each run there is a file, called ASrun# SDATCHK ( example ). Look at this file to see if any trigger errors have been flagged. These are the ones to notify the run coordinator of.

Also examine ASrun#.STATS ( example ) to see if any subsystem data is missing.

Do it (Check tape copy/trigger)

Filter (Pass 1) duties

Check the Filter stat. files. ( example )

Compare SLD Z's with Luminosity 'precise' bhabhas. They should be about 1:2. Look up the online SLC Zs they should agree approximately with the SLD Zs.
Polarization these days should be between 70% and 80%.
If running conditions are good the bad polarization fraction should be essentially zero.
If running conditions are good the CDC should be on basically 100%..
All diagonal elements of the filter and trigger correlation matrices should be non-zero (for a reasonable number of input events).
Precise/total bhabhas ratio should be ???.

Check Pass 1 Histograms. Most of these histograms are for diagnostics in the filter job itself. Of interest for monitoring are the polarization and CDC HV-on plots. We would like a nice gaussian for POL and a spike at 10 layers for the CDC.

The ID=QPOL polarization histogram should be gaussian with few low P measurements.
The ID=DCHV CDC HV layers plot should show 10 layers on most of the time.

Do it (Check Filter, pass1 reconstruction)

Recon (pass 2) duties

Examine Recon stat. files ( example )

The z at IP should be small. We are looking for RF shifts in SLC that cause a 5 cm shift (units here are cm).
Pol errors should be zero.
L/R asymmetries should be flagged if more than 4 sigma from zero.
The number of Z passing the 'Energy' filter should also be 1:2 with the LUM bhabhas (from the Filter STATS file).
All diagonal elements of the filter correlation matrix should be non-zero (for a reason number of input events).

Examine recon histograms

Most of the RECON histograms are the same ones used for the RTH online monitoring system, and undergo the same statistical comparison to the 'expected' distributions. Histograms that fail this comparison are flagged for the monitor's further investigation. (Not implemented yet).

Plot ID=DCHD should show a flattish shape in the barrel with maybe a 20-30% rise to the endcaps.
Plot ID=DTFD should show a flat phi.
Plot ID=TZND should show a sharp spike near zero.
D should show a smooth efficiency for putting hits on tracks vs CDC layer number. A dip in layers 0 and 1 are expected.

Check the Event Displays

Both raw and reconstructed data from the KAL and DC systems are displayed for a sample of hadron candidates. The sample is selected from events that pass either hadron filter; it shoots to randomly select about 5 events; if there are less, all are displayed.

Each event gets three views: the first is a 3-view with KAL cluster hits and CDC tracks shown; the 2nd is a blowup of the CDC with tracks and vectored hits; the 3rd shows a blowup of the VXD and CCD hits on tracks.

These are just there for visual verification that the detector is OK; no event classification is called for!

Look for lots of extra vectored hits in the tracking or missing layers. Make sure CCD hits are being associated to CDC tracks.

Do it (Check pass 2 Reconstruction )

ZCHKDAY

This stage runs vertexing code (ZXFIND) on all the PASS2 output events for a given day. It looks at the primary vertex position, gamma conversions and K-shorts, and track muliplicities and VXD linking efficiency. (Default parameters are used in ZXFIND.)

Output diagnostics from this stage include statistics files ( example ) :

IP positions should be within errors of previous days' values.
Ratio of gamma conversions at VXD vs CDC inner wall should be around 2.
Should be about 0.15 gamma conversions/Z
Should be about 0.20 V/Z
The V mass should be with errors of 0.498 and width around 6 MeV.
The total track multiplicity should be around 18. VXD inefficiency around 4%.

And Histograms

Plots XPRI, YPRI and ZPRI should be single peaked gaussians. Plots XVSN, YVSN and ZVSN should show no time dependence.
Plots VEP, VECT and VEPH should show no areas of inefficiency.
Peruse the other plots to see if they match your physics intuition.

Links to ZCHKDAY Output

creates a table of links from which you can browse statistics and histogram files from the previous day's ZCHKDAY job.

Unanalyzed Runs

shows information on runs never seen by the offline, runs currently being analyzed and runs with analysis in suspended state. The SLC estimate for the Z count is given.

Offline Shift Report

An example.
Quantities that need to be verified are:

polarization magnitude & error rate
SLC luminosity estimates agree with SLD LUM & Z rates
Beam spot z centered
Event Displays show no diseases
RTH histograms check out or have been reported
CDC HV-on fraction is high

The last three items, plus an explicit overall evaluation, are asked for in the report. Other problems should be reported in the comments section.

Generate an Offline Shift Report: prompts you for some basic information then generates the report form.
Old Offline Shift Reports