ATLAS FullChainTest

Author: Seth Zenz (Email: Seth dot Zenz at cern dot ch)

Welcome to the results page for the automated ATLAS Full Chain Test. The purpose of the test is to rapidly find errors in the full production chain that occur every few events, so that the production system doesn't have to. It runs on the RTT queues at CERN for each completed nightly, and is started via a cron job that checks for new releases once per hour.

Scroll down for details on the tests, see the links at right, or see these instructions for how to get the FullChainTest results.

Status (Last update: 20 June)

Ran on 12.0.7 (14 June): ERROR checkFile Stats
- all ok
pcache settings updated, only limited set of jobs run
Web-accessible list of currently-running batch jobs (see link at right)

Links

ERROR output: 12.0.X 13.0.X: bugfix val pcache

checkFile.py Output: 12.0.X 13.0.X: bugfix val pcache

Job CPU/memory statistics: 12.0.X 13.0.X: bugfix val pcache

Last good root file from each stage: 12.0.X 12.0.6.5

Directories where transforms were run: 12.0.X (long) 13.0.X: bugfix (long) val (long) pcache (long)
From here, navigate to individual run directories labelled by date, release and job type, and original log files can be found

Currently running jobs: 12.0.X 13.0.X: bugfix val pcache

Space use log (scroll to bottom)
AFS access

/afs/cern.ch/atlas/offline/external/FullChainTest

Links to last good root file - /{branch}/last_good_root

Run directories - /{branch}/full_output

Scripts in CVS

Talks

Details

The test does the following:

Automatically checks stamp link once per hour to see if it has changed, and starts jobs if so.
- Release 12 bugfix - /afs/cern.ch/atlas/software/builds/nightlies/12.0.X/AtlasProduction/latest_copied_releaseBF32BS3ProdOpt
- Release 13 bugfix - /afs/cern.ch/atlas/software/builds/nightlies/bugfix/AtlasProduction/latest_copied_releaseBF32BS4ProdOpt
- Release 13 val - /afs/cern.ch/atlas/software/builds/nightlies/val/AtlasProduction/latest_copied_releaseVAL32BS4ProdOpt
- Release 13 pcache - /afs/cern.ch/atlas/software/builds/nightlies/pcache/AtlasProduction/latest_copied_release13010Y32BS3ProdOpt
Detailed list of jobs as of 9 May: Excel format
Summary of jobs:
- DC3.005200.T1_McAtNlo_Jimmy - 5000 events generated, 50 events simulated/digitized
- DC3.005300.PythiaH130zz4l - 5000 events generated, 50 events simulated/digitized
- DC3.006384.PythiaH120gamgam - 5000 events generated, 50 events simulated/digitized
- DC3.005188.A3_Ztautau_filter - 5000 events generated, 50 events simulated/digitized
- Atlfast on 5000 ttbar events
- RecoAll - basic reconstruction of the output from all of the above in one job (200 events)
  - ESD, AOD, TAG
- BackRecoAll - same as RecoAll, but reconstructs using older release (currently 12.0.6.5)
- BackTagAll - same as BackRecoAll, but for tag building
- LongRecoAll - same as RecoAll, but adds an additional 800 ttbar events from 12.0.31 RDO's
  - Total of 1000 events
  - To be replaced by equivalent RTT job and removed
- RecoNoTruth - Runs on RDO from streaming test, 15 events
- PileupDigi - Digitize, with pileup, hits from DC3.005200.T1_McAtNlo_Jimmy (50 events)
  - Uses luminosity of 1*10^33, cavern background 5x nominal (since 10 May, 10x before that)
  - Input files for cavern background and min bias
- PileupReco using pileup digi output
  - ESD, AOD, TAG
- Tag Merge - runs TAG building on two AOD's to make sure the merging works
If any step in production fails, use the last good version of the output for the next step.
Make the "last good version" for each step publicly available.
Post-processing
- ERROR file
  - First lists job transform summary, including all un-ignored ERRORs
  - Next lists all ERRORs in context
- Output from checkFile.py available for all ESD and AOD files produced
- CPU/memory statistics
Key results are accessible from this website, and the entire directory can be browed on afs:
- /afs/cern.ch/atlas/offline/external/FullChainTest

In progress

Documentation

To be added

List selected warnings
More systematic status reporting