Notebooks for Circuit Analysis of HWC Tests and Events during Operation
Although, as the project name indicates, our primary goal is the development of signal monitoring applications, we realized that the analysis modules developed so far can be pieced together into HWC test and operation analysis notebooks.
The monitoring applications are organized system by system. Each analysis has been developed in a general way to account for all circuits in which the system was present. Thus, by taking a perpendicular view of the analysis table, a circuit analysis for this stance was possible.
In particular, those notebooks are suited for HWC tests and operational analysis (FPA):
- can be adjusted on-the-fly for new requirements while performing a test;
- can immediately generate a report for storage and distribution among a team of domain experts;
- provide a sequential way of testing each system in a given order.
The execution of notebooks is carried out with SWAN service (http://swan.cern.ch). In order to start using the notebooks you need to follow these four steps:
- Getting NXCALS Access (once only)
- Logging to SWAN
- Setting up an appropriate environment script (done at each login)
- Running an appropriate notebook
1. NXCALS Access
The analysis notebooks query certain signals with NXCALS. The NXCALS database requires an assignment of dedicated access rights for a user. Please follow a procedure below to request the NXCALS access.
- Go to http://nxcals-docs.web.cern.ch/current/user-guide/data-access/nxcals-access-request/ for most recent procedure
Send an e-mail to mailto:email@example.com with the following pieces of information:
your NICE username
- system: WinCCOA, CMW
- NXCALS environment: PRO
Optionally please can mention that the NXCALS database will be accessed through SWAN. Once the access is granted, you can use NXCALS with SWAN.
2. Logging to SWAN
The following steps should be followed in order to log-in to SWAN
- Go to http://swan.cern.ch
- Login with your NICE account
- SWAN is tightly integrated with CERNBox service (in fact, files created in SWAN are accessible in CERNBox). In case you have not yet used CERNBox, the following error message will be displayed indicating that your CERNBox account has not been activated yet. In order to activate your CERNBox account, please login on the website: http://cernbox.cern.ch. Afterwards, please login to SWAN service again. In case the error persists, please contact the SWAN support (see section Help).
3. Setting an Environment Script
In order to execute the HWC notebooks, one requires
lhc-sm-api package and HWC notebooks. To this end, we created a dedicated environment script to prepare the SWAN project space.
The script sets a path to a virtual environment with the necessary packages (for more details, cf. API User Guide) as well as makes a copy of HWC notebooks to
Note that in order to ensure compatibility between package and notebook versions, the
hwc folder is deleted each time the script is executed.
You need to contact the Signal Monitoring team (firstname.lastname@example.org) in order to get read access to the EOS folder with pre-installed packages and HWC analysis notebooks.
Once the access is granted, at every log-in to SWAN, please provide the following environment script:
Note the following settings while configuring environment:
- Software stack:
CentOS 7 (gcc7)- default
- Environment script:
- Number of cores:
- Spark cluster:
BE NXCALS (NXCals)
4. Running Notebook
4.1. Open notebook
To do so simply open
hwc folder and then select a circuit. Afterwards click name of a notebook to open a new page. The top of the notebook is presented in Figure below.
4.2. Connect to the NXCALS Spark Cluster
Once a notebook is opened, please click a star button as shown in Figure below in order to open the Spark cluster configuration in a panel on the right side of an active notebook.
Figure below shows a three-step procedure of Spark cluster connection. The first step involves providing the NICE account password. The second step allows setting additional settings for the connection. In order to connect with NXCALS please make sure to enable the following options:
- Include NXCALS options - to connect to the cluster
- Include SparkMetrics options - to enable metrics used for analysing NXCALS queries
The last step is a confirmation of a successful connection to the cluster.
4.3. Analysis Notebook Execution
A notebook is composed by cells. A cell contains either a markdown text with description or python code to execute. Cells with markdown text have white background and can contain text, tables, figures, and hyperlinks. Cells with code have gray background and are executed by clicking a run icon in the top bar highlighted in Figure below. Alternatively, one can put a cursor in a cell with code an press a keyboard shortcut Ctrl+Enter.
A state of a cell is indicated by square brackets located on the left to a cell. Execution of a cell is indicated by a star in the square brackets. Once cell execution is completed the star changes into a number representing the order of cell execution. A cell can execute for too long due to connection problems, issues with a database query, kernel problems. In this case, two actions are recommended:
- Select from the top menu:
Interruptand execute the problematic cell again (either a run button (cf. Figure above) or Ctrl+Enter).
- In case the first option does not help, select from the top menu
Restart & Clear Output. Then all cells prior to the problematic one have to be executed again (multiple cell selection is possible by clicking on the left of a cell to select it and afterwards selecting others with pressed Shift button). After this operation one needs to reconnect to the NXCALS Spark cluster.
4.4. Analysis Assumptions
- We consider standard analysis scenarios, i.e., all signals can be queried. Depending on what signal is missing, an analysis can raise a warning and continue or an error and abort the analysis.
- It is recommended to execute each cell one after another. However, since the signals are queried prior to an analysis, any order of execution is allowed. In case an analysis cell is aborted, the following ones may not be executed (e.g. I_MEAS not present).
4.5. FPA Notebooks
An FPA analysis workflow consists of four steps: (i) finding an FGC Post Mortem timestamp; (ii) executing analysis cells on the cluster (iii); (iv) storing output files on EOS; see Figure below.
The RB FPA Analysis notebook is organized into 10 chapters (Note that for the remaining circuits, some analyses may not be present.):
- Initialise the working environment Loads external packages as well as lhcsmapi classes required to perform analysis and plot results.
- Select FGC Post Mortem Entry After executing this cell, a FGC Post Mortem GUI with default settings is displayed.
The GUI consists of 8 widgets described in Table below.
|Circuit name||Circuit name|
|Start||Start date and time|
|End||End date and time|
|Analysis||Automatic (each cell executed without user input); Manual (some analysis steps take expert comment)|
|Done by||NICE login of a person executing the analysis|
|Find FGC PM entries||Button triggering a search of FGC PM entries|
|Query progress bar||Displays progress of querying days in between indicated datesFGC PM EntriesList of FGC PM timestamps|
Please note that in order to execute any of the following cells, there should be at least one entry in the FGC PM Entries list. The list is populated after clicking [Find FGC PM entries button].
Figure below shows the GUI after clicking button [Find FGC PM entries] with the default settings. Note that the list only contains FGC PM timestamps surrounded by QPS timestamps (1 minute before and 5 minutes after an FGC PM timestamp).
- Query All Signals Prior to Analysis
In order to avoid delays between analyses, the necessary signals are queried prior to performing the analysis.
Table of timestamps main systems representing the sequence of events for a given analysis.
Interactive schematic of the RB circuit composed of: power converter, two energy extraction systems, current leads, magnets, and nQPS crates. Hovering a mouse over a center of a box representing a system provides additional pieces of information. Location of quenched magnets is highlighted. Slider below the schematic enables its scrolling.
Check of PIC timestamps
- Power Converter
Analysis of the main power converter as well as earth currents.
- Energy Extraction Analysis of the energy extraction voltage and temperature
- Quench Protection System Analysis of the quench detection system, quench heaters, diode, voltage feelers, diode leads resistance, current leads voltage (resistive and HTS).
- Plot of Energy Extraction after 3 h from an FPA
- Final Report
Saving of the CSV results table and HTML report to EOS folder. The RQ analysis notebook follows the same structure except for the lack of schematic. Typically, there is only a single main quadrupole magnet quenching and the schematic does not provide more information as compared to the timestamps table in point 3. For the remaining circuits the analysis cells reflect the presence of particular hardware.
The notebook creates two output files in the folder (path with Windows convention)
- HTML report file with the snapshot of the entire notebook - [circuit-name]_FPA-[fgc-timestamp]-[analysis-execution-date].html;
- CSV file with MP3 results table with a subset analysis results - [circuit-name]_FPA-[fgc-timestamp]-[analysis-execution-date].csv;
4.6. HWC Notebooks
A HWC analysis workflow consists of four steps: (i) finding of start and end time of an HWC test (ii) executing analysis cells on the cluster (iii); (iv) storing output files on EOS; see Figure below.
Each notebook is composed of initial part with a circuit schematic, test current profile, and table summarising test criteria. This part is followed by package import instructions, display and the browser of historical HWC tests (until and including HWC 2018); see Figure below.
There is a manual integration with AccTesting that requires a copy&paste of a text parameterising a notebook
The remainder of each notebook depends on the particular test to be performed. At the end of an HWC notebook there are instructions for saving the output files.
The notebook creates an output in the folder (path with Windows convention)
- HTML report file with the snapshot of the entire notebook - [circuit-name]_[test_name]-[test-start]-[analysis-execution-date].html;
4.7. Terminating an Analysis
After completing an analysis with Spark you need to make sure to disconnect. Please note that closing a notebook does not close the Spark connection. Leaving open sessions may lead to holding resources and blocking you from performing an analysis.
Thus, in case you do not plan to use the notebook, please click the Spark connection icon and press
Restart Spark session button.
The notebook will maintain the results of your analysis.
In case you want to erase the notebook content and disconnect from Spark at the same time, please select from the top menu
Kernel and afterwards
Restart and Clear Output. Then, when you return to that notebook, you won't see the output from previous analysis.
The last option is to close the notebook from notebook menu , File -> Close and Halt. This will disconnect from Spark connection, and safely terminate the notebook execution.
4.8. Access to Reports
Execution of each notebook ends with generation of an HTML report. In addition, for FPA analysis (and f-type tests, i.e., test with heater provoked quenches) there is also a csv file generated to be copied into the MP3 Excel Quench database.
Both the HTML reports and csv tables are stored on EOS drive (
\eos\project\m\mp3\). They are accessible through either a local file explorer or an active SWAN session:
A file explorer on local machine (below an example on Windows)
An active session in SWAN
CERNBoxin the top bar
mp3folder (you may see more folders as it is your personal CERNBox view)
- Then, inside you should see the same view as in the file explorer
Note that if you open an HTML report from SWAN, this will render a report directly in your browser.
4.9. Update of the MP3 Quench Database
Update quench file in repository with the new .csv file in case of quenches and QH firing with significant current (> 3 kA for RB, RQ, IT; > 1 kA for IPD, IPQ; > 70 A for 600A; > 10 A for 60, 80, 120 A circuits).
Open MP3 Quench repository at https://cern.ch/MP3-onedrive/QuenchData
Open the Excel (in this example IT for Inner Triplets)
Edit file locally by clicking
Import the .csv file with Excel
- Select a file from EOS circuit folder
- The file uses tab as column separator and has a header
Check that the extra imported header matches the one in the Excel file
Remove extra header and any blank rows in-between existing and new data
Sort by FGC date and time
Do not forget to save!