ATLAS FullChainTest Author: Seth Zenz (Email: Seth dot Zenz at cern dot ch) Welcome to the results page for the automated ATLAS Full Chain Test. The purpose of the test is to rapidly find errors in the full production chain that occur every few events, so that the production system doesn't have to. It runs on the RTT queues at CERN for each completed nightly, and is started via a cron job that checks for new releases once per hour. Status (Last update: March 21, 13h00 CET) Running automatically on 12.0.6 bugfix nightlies (reopened) rel_0: ran out of disk; all ok in quick test rerun rel_1: all ok except TAG failed as above. Problem now reproducable, under investigation "Oracle error ORA-02391: exceeded simultaneous SESSIONS_PER_USER limit" in one job; under investigation rel_2: all ok (including pileup) except TAG as above rel_3: pileup accidentally killed; all ok otherwise except tag ERROR Unable to set property PrepareAll of TriggerPrepLooper.L2TrigPrep_L2JetPreparator rel_4: pileup still running, all non pileup ok! Now also running automatically on release 13 dev nightlies All jobs have failed so far, due to unignored ERRORs Pileup digitization and reconstruction tested starting in rel_6 Running only two events as of rel_1 due to large CPU time (~11 min/event)

Links ERROR output: bugfix dev checkFile.py Output: bugfix dev Last good root file from each stage: bugfix dev 12.0.6.1 Directories where transforms were run: bugfix dev From here, navigate to individual run directories labelled by date, release and job type, and original log files can be found AFS access /afs/cern.ch/atlas/offline/external/FullChainTest Links to last good root file - /{branch}/last_good_root Run directories - /{branch}/full_output

• 27 February 2007
• 13 March 2007

The test does the following:

• Automatically checks stamp link once per hour to see if it has changed, and starts jobs if so.
  • bugfix - /afs/cern.ch/atlas/software/builds/nightlies/12.0.X/AtlasProduction/latest_copied_releaseBF32BS3ProdOpt
  • dev - /afs/cern.ch/atlas/software/builds/nightlies/dev/AtlasProduction/latest_copied_releaseDev32BS4ProdOpt
• Current jobs:
  • DC3.005200.T1_McAtNlo_Jimmy - 5000 events generated, 50 events simulated/digitized
  • DC3.005300.PythiaH130zz4l - 5000 events generated, 50 events simulated/digitized
  • DC3.006384.PythiaH120gamgam - 5000 events generated, 50 events simulated/digitized
  • DC3.005188.A3_Ztautau_filter - 5000 events generated, 50 events simulated/digitized
  • RecoAll - basic reconstruction of the output from all of the above in one job (200 events)
    • ESD, AOD, TAG
  • BackRecoAll - same as RecoAll, but reconstructs using older release (currently 12.0.6.1)
  • PileupDigi - Digitize, with pileup, hits from DC3.005200.T1_McAtNlo_Jimmy (50 events)
    • Uses luminosity of 2*10^33, cavern background 10x nominal
    • Input files for cavern background and min bias
  • Pileup reconstruction using pileup digi output
    • ESD, AOD, TAG
• If any step in production fails, use the last good version of the output for the next step.
• Make the "last good version" for each step publicly available.
• Post-processing
  • ERROR file
    • First lists job transform summary, including all un-ignored ERRORs
    • Next lists all ERRORs in context
  • Output from checkFile.py available for all ESD and AOD files produced
• Key results are accessible from this website, and the entire directory can be browed on afs:
  • /afs/cern.ch/atlas/offline/external/FullChainTest

In progress

• (See status.)

• List selected warnings
• Documentation

• Week of March 5
  • rel_2: run on AOD and TAG fails
  • rel_3: TAG fails, everything else (including AOD) ok
  • Rerun due to intermittent environment problem - now fixed
  • rel_4: Build late; reco not run due to setup error (see rel_5 instead)
  • rel_5: all ok
• Week of March 12
  • rel_6: all ok except pileup AOD (pileup TAG untested)
  • rel_0: all ok except pileup AOD (pileup TAG untested)
  • rel_1: all ok except pileup AOD
  • rel_2: all ok (including pileup AOD), except...
    • pileup digi and non-pileup TAG not run due to script typos
  • rel_2 rerun: non-pileup TAG ok... pileup digi exceeded queue time limit
    • Killed after 13134 CPU sec (queue = atlasrttmedium)
    • Pileup digi will use long queue starting in rel_4
  • rel_3: all ok, except pileup digi exceeded time limit again
  • rel_4: non-pileup all ok except TAG, pileup ok
    • TAG OK in rerun — problem not reproducable
  • rel_5: bad run, old packages in local test area
  • rel_6: all ok except TAG
    • "not reproducable" problem is back
    • under further investigation