more precise timing info for beam on/beam off using the CDC occupancy
There was another instance where hydra was indicating that the cdc occupancy image was from a beam off period but the image itself was clearly from beam on. Cross referencing RSAI's logs with keeper's logs, the CDC occupancy ring histograms are filled about 3 minutes prior to hydra receiving them. If we want to avoid this repeated confusion, we should attempt to use timing information from RSAI (or at least guess closer to the actual time). I've pasted logs and images below. Note RSAI starts filling the histogram for chunk 36 after resetting the occupancies from chunk 35. The wave screenshot has beam current information, and I also attached a screenshot of the run page for reference.
FROM KEEPER:
2025-08-22 09:49:27.528010 [INFO]: HydraReport {"MetaData": {"plotType_ID": 9, "modelID": 258, "inDATA": "/gluonraid5/data1/monitoring/AI/hydra_in_converted//RunPeriod-2025-01/Run133466/CDC_occupancy_0036.png", "runNumber": 133466, "runPeriod": "RunPeriod-2025-01", "chunkNum": 36, "outDir": "DELETE", "datetime": "2025-08-22 09:49:26.984532", "FeederTime": "0.137433", "FeederStatus_ID": "159", "BalancerTime": 0.003, "BalancerStatus_ID": 164, "PredictTime": 88.407, "PredictStatus_ID": 197, "gradCAMheatmap": ""}, "Analysis": {"Label
2025-08-22 09:49:27.528026 [INFO]: Message received from predict!
2025-08-22 09:49:27.528030 [INFO]: updating message count
2025-08-22 09:49:27.528979 [INFO]: Report generated in 0 ms
-------------------------------------------------
2025-08-22 09:49:27.528991 [INFO]: HydraReport
MAX VALUE: 0.9998747110366821
analysis: {'Labels': {0: 'Bad', 1: 'Good'}, 'Confidences': [0.0001253199443453923, 0.9998747110366821]}
2025-08-22 09:49:27.528995 [INFO]: /gluonraid5/data1/monitoring/AI/hydra_in_converted//RunPeriod-2025-01/Run133466/CDC_occupancy_0036.png Good @ 0.9998747110366821
2025-08-22 09:49:27.529012 [INFO]: Output directory: DELETE
2025-08-22 09:49:27.570546 [INFO]: I removed /gluonraid5/data1/monitoring/AI/hydra_in_converted//RunPeriod-2025-01/Run133466/CDC_occupancy_0036.png
2025-08-22 09:49:27.570571 [INFO]: chunk num?: 0036
2025-08-22 09:49:27.570595 [INFO]: GETTING keep percent for CDC_occupancy
2025-08-22 09:49:27.570604 [INFO]: SELECT CollectPercent,ID FROM Plot_Types where Name="CDC_occupancy" && FileType="png" && IsChunked=1
2025-08-22 09:49:27.570989 [INFO]: got keep percent 9=0.01 for CDC_occupancy
2025-08-22 09:49:27.571003 [main INFO]: Run Period: RunPeriod-2025-01 Run Number: 133466
2025-08-22 09:49:27.571515 [main INFO]: has EPICS? True. reading current from IBCAD00CRCUR6
2025-08-22 09:49:27.571524 [main INFO]:myStats -b '2025-08-22 09:48:56' -e '2025-08-22 09:49:26' -l IBCAD00CRCUR6 -u
2025-08-22 09:49:27.598362 [main INFO]: obtained beam current: 0.55
MAX VALUE: 0.9998747110366821
analysis: {'Labels': {0: 'Bad', 1: 'Good'}, 'Confidences': [0.0001253199443453923, 0.9998747110366821]}
2025-08-22 09:49:27.598409 [main INFO]: plot Time=2025-08-22 09:49:26.984532
2025-08-22 09:49:27.598416 [main INFO]: BEAM CURRENT IS: 0.55
2025-08-22 09:49:27.598428 [INFO]: Checking 9 against 0.01
2025-08-22 09:49:27.598437 [INFO]: don't keep
2025-08-22 09:49:27.598454 [main INFO]: Analyzing report with beam current: 0.55
2025-08-22 09:49:27.598539 [INFO]: Analyzing report
MAX VALUE: 0.9998747110366821
analysis: {'Labels': {0: 'Bad', 1: 'Good'}, 'Confidences': [0.0001253199443453923, 0.9998747110366821]}
2025-08-22 09:49:27.598554 [INFO]: Verdict: Good Confirmed
2025-08-22 09:49:27.598563 [INFO]: EPICS: True
REPORT META DATA: {'plotType_ID': 9, 'modelID': 258, 'inDATA': '/tmp/keeper_tmpout//CDC_occupancy_0036.png', 'runNumber': 133466, 'runPeriod': 'RunPeriod-2025-01', 'chunkNum': 36, 'outDir': 'DELETE', 'datetime': '2025-08-22 09:49:26.984532', 'FeederTime': '0.137433', 'FeederStatus_ID': '159', 'BalancerTime': 0.003, 'BalancerStatus_ID': 164, 'PredictTime': 88.407, 'PredictStatus_ID': 197, 'gradCAMheatmap': ''}
2025-08-22 09:49:27.598594 [Analyze Report INFO]: Epics value: 0.9997493910923367
2025-08-22 09:49:27.598606 [Analyze Repor tINFO]: HD:monitoring:ai:CDC_occupancy=0.9997493910923367
2025-08-22 09:49:27.598767 [INFO]: Confirmation? Confirmed
2025-08-22 09:49:27.598775 [INFO]: Confirmed Verdict
2025-08-22 09:49:27.598779 [INFO]: ALARM OFF
2025-08-22 09:49:27.598883 [INFO]: Hippocampus insert: INSERT INTO Hippocampus (HydraHostName,ModelID,PlotDateTime,InsertDateTime,RunPeriod,RunNumber,ChunkNumber,PlotType_ID,IMG,gradCAM,Output,ModelThresholds,VerdictLabel,VerdictConfidence,IsConfirmed,IsU...
2025-08-22 09:49:27.610313 [INFO]: AnalyzeReport done in 12 ms
2025-08-22 09:49:27.610332 [INFO]: Removing /tmp/keeper_tmpout//CDC_occupancy_0036.png
FROM RSAI:
[Fri Aug 22 09:46:37 2025] ------- TODO: Run hydra on the following files:
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/CDC_occupancy_0035.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/ECAL_occupancy_0141.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/FCAL_occupancy_0140.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/FDC_occupancy_0140.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/PS_occupancy_0140.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/RF_TOF_selftiming_0140.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/RF_TPOL_occupancy-02_0140.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/RF_TPOL_occupancy-RF_0140.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/ST_occupancy_0140.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/TAGGER_occupancy_0140.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/TOF_occupancy_0140.png
------- /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/fa125_itrig_0140.png
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_axes
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_num_events
Resetting histo: /occupancy/cdc_occ_ring_01
Resetting histo: /occupancy/cdc_occ_ring_02
Resetting histo: /occupancy/cdc_occ_ring_03
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_04
Resetting histo: /occupancy/cdc_occ_ring_05
Resetting histo: /occupancy/cdc_occ_ring_06
Resetting histo: /occupancy/cdc_occ_ring_07
Resetting histo: /occupancy/cdc_occ_ring_08
Resetting histo: /occupancy/cdc_occ_ring_09
Resetting histo: /occupancy/cdc_occ_ring_10
Resetting histo: /occupancy/cdc_occ_ring_11
Resetting histo: /occupancy/cdc_occ_ring_12
Resetting histo: /occupancy/cdc_occ_ring_13
Resetting histo: /occupancy/cdc_occ_ring_14
Resetting histo: /occupancy/cdc_occ_ring_15
Resetting histo: /occupancy/cdc_occ_ring_16
Resetting histo: /occupancy/cdc_occ_ring_17
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_18
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_19
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_20
Resetting histo: /occupancy/cdc_occ_ring_21
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_22
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_23
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_24
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_25
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_26
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_27
[Fri Aug 22 09:46:37 2025] Resetting histo: /occupancy/cdc_occ_ring_28
…… 3 minutes later ……..
[Fri Aug 22 09:49:26 2025] CDC Flagging AI check after 548647 events (>=500000)
[Fri Aug 22 09:49:26 2025] Scheduled reset of 30 histograms for macro: //CDC_occupancy
[Fri Aug 22 09:49:26 2025] Info in <TCanvas::Print>: png file /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/tmp_canvas_531475.png has been created
[Fri Aug 22 09:49:26 2025] Warning in <TASImage::Crop>: input size larger than image
- Writing file /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/CDC_occupancy_0036.png
[Fri Aug 22 09:49:26 2025] - Creating symlink /gluonraid5/data1/monitoring/AI/RunPeriod-2025-01/Run133466/CDC_occupancy_0036.png <- /gluonraid5/data1/monitoring/AI/hydra_in/RunPeriod-2025-01/Run133466/CDC_occupancy_0036.png
