TIBTIDNotes/TIBTIDIntNote/IntegrationTests.tex

\section{Integration Tests}
\label{sec:Tests}

The CMS tracker has been designed to survive at ten years of LHC operation, with little or no chance 
for maintenance. Its various components have been tested during the production
to meet stringent quality requirements. Few important problems have been spotted and
solved.\\
During the TIB integration all the operations have been monitored step by step by a chain of tests
aimed at a final control of the components just after the installation and at a verification of the
shell overall quality and functionality. The step by step tests are of particular importance 
because in most cases it is very difficult and in some cases even dangerous 
to replace a single faulty component when it is embedded in a fully equipped shell.

\subsection{Test Setup}
\label{sec:TestSetup}
The integration activities started well before the tracker 
final data acquisition hardware and software
were available to the Collaboration, and thus had to rely on prototype peripherals
and a developing software freezing at the integration start up time.
Several further upgrades were actually implemented in time, but they all relied on the 
same version of the underlying framework.

\subsubsection{Hardware}
Here a brief account of the hardware used in DAQ for the integration 
is given (see Fig.~\ref{fig:integration daq}).
\begin{figure}
\centering
\includegraphics[width=\textwidth]{Figs/integrationDAQ.pdf}
\caption{Hardware used for the integration DAQ. 
\textbf{On the left:} 
\textbf{1:} The analog opto-electrical converter.
\textbf{2:} Optical lines from the AOHs inside the (yellow) ribbon. 
\textbf{3:} The signal is converted to electric.
\textbf{On the right:} 
\textbf{1:} The signal converted to electric goes to FEDs through 2-poles LEMOs. 
\textbf{2:} TSC provides trigger and clock to FEC through an (orange) optical fibre and 
\textbf{3:} to FEDs though 4-poles LEMOs. 
\textbf{4:} FEC also implements the 2-way communication towards CCUs and modules through a
(yellow) fibre ribbon.}
\label{fig:integration daq}
\end{figure}

The A/D conversion is managed by FEDs~\cite{bib:fedpci}, 
with electrical differential analog input, mounted on 
PCI carrier boards and installed in an industrial PC.
The opto-electrical conversion of the analog signals is done externally by a 24-channel unit. 
A setup containing 3 FEDs, with the electro-opical converter, is able
to readout 48 APV; this is equivalent to 12 single sided modules (4 complete strings) 
or 4 double sided modules (1 string plus 1 module). This is more than what is needed to
test the strings during the module installation.\\
The trigger and clock signals
were provided by the Trigger Sequencer Card (or TSC,~\cite{bib:specs:tsc}). It has up to four
electrical clock/trigger outputs, which were enough to drive the FEDs, and an optical clock/trigger
output for the FEC.
This card is used to generate a 40~MHz clock and provide it to the system and also to generate
triggers, either internally via software or by accepting external inputs.
The TSC may also generate the RESET and CALIBRATION signal, by coding them properly on the 
clock/trigger line.\\
The FEC mezzanine used during the integration
was laid on a PCI carrier, supporting trigger/clock optical input as fed by the TSC. Its
output was also optical, and could be directly connected to DOHs on the DOHM. \\
The peculiarity of a PCI FED with respect to a final VME FED, other than having electrical inputs
instead of optical ones, is its timing system: a final FED is able to recognise which
conversion count
corresponds to a frame header coming from a module on each input channel separately,
while a PCI FED has this capability (\textit{header finding}) implemented only on the first channel
out of the eight available and assumes that the signals coming from the modules are synchronised.
This makes the PLL-based time alignment procedure even more important in a setup with PCI FEDs.
Furthermore, PCI FEDs are not able to insert a programmable delay on their inputs and thus
it is important that the clock/trigger connections from the TSC to FEDs have all the same delay.
Last, PCI FEDs cannot perform an online pedestal subtraction and zero suppression (they cannot
run in Processed Raw data nor in Zero Suppression mode).

\subsubsection{Software}
The software used to carry out integration tests was essentially the CMS
tracker implementation of a more general software, named xDAQ, which is the official CMS
DAQ framework. This implementation is known as TrackerOnline.\\
The present version of TrackerOnline makes use of a set of
xml files which store the parameters needed by all the devices present on the structure
to be tested. In the final implementation of the software the use of a database is foreseen.

\begin{figure}[bth!]
\centering
\includegraphics[width=\textwidth]{Figs/fecmodulexml_2.pdf}
\caption{An example of data contained in fec.xml and module.xml files for one module.
Part of the data is not shown for simplicity.}
\label{fig:fecmodulexml}
\end{figure}
The information on a given setup can be divided in two sections: one describing the
readout hardware (and corresponding software) setup and another describing which part of 
the tracker is going to be connected to the Control System and to FEDs.\\
The hardware/software setup is written in a single xml file, which was prepared once
and for all at the start of the integration.
The information regarding the readout tracker section
is stored in two more files, commonly named fec.xml and module.xml. The first
contains all data which the FEC will need to download to modules before starting the data
taking, that is all the values to be written in devices' $I^2C$ registers. 
The second contains all other
information needed by the DAQ setup to rearrange data coming from the FEDs: from an input channel
based indexing to a module based one (see Fig.~\ref{fig:fecmodulexml}).\\
The module.xml file contains two tables: the first joins FEDs input channel indexes
with the respective modules' ring, CCU and $I^2C$ address (i.e.\ readout coordinates 
with Control System coordinates). Each row corresponds to an APV
pair, an AOH laser, an optical fibre and a FED input.\\
The second table of module.xml reassembles the information on a module basis. 
Here all the active modules are listed, with a row for every module. 
The Control System coordinates are repeated both in module.xml and fec.xml, 
so that they can be used as a pivot between the 3 tables.\\
It is clear that these files have to be archived along with raw data taken during a run, and
to do it automatically is the first task of an integration validation software.
Another required task is an automatic run logging; a fast data analysis is also
desirable.
Last, but not least, this software should easily allow the user to perform all preliminary 
(commissioning) runs, adjusting modules' parameters accordingly.

\begin{figure}
\centering
\includegraphics[width=.9\textwidth]{Figs/integration_package.pdf}
\caption{Scheme of the integration software. 
\textit{Arrows}: a relation.
\textit{Gears}: an application. 
\textit{Mouse:} an interactive application. 
\textit{Sheet}: a file.}
\label{fig:integration_package}
\end{figure}

Figure~\ref{fig:integration_package} shows a scheme of the relations between all the software
components that will be described, the local files and the remote database.
\paragraph*{FecTool}
FecTool is a front-end Graphics User Interface (GUI) aimed to ease the creation of the device 
description, i.e.\ fec.xml. This program is used to launch
two standalone applications deployed along with TrackerOnline: ProgramTest and FecProfiler. 
The first application can test the ring functionality, the connection to all devices reporting 
a list of detected hardware. Also the second application is able to retrieve a 
list of hardware connected to CCUs, but its output is the fec.xml file needed by TrackerOnline.\\
By accessing the output of these two programs, the FecTool GUI enables the user to
test the functionality of a string, or of a whole ring. FecTool also checks that the found hardware
corresponds to what one expects to find in tracker's modules: for every $I^2C$ address 
there should be either 4 or 6 APVs, one PLL, one AOH, and so on.\\
The user can also input the GeoId(s) of tested string(s) before starting the test. In this case
FecTool also checks that the DCU Hardware Id read from each module matches the one declared 
in the construction database performing an important consistency check between what is
registered on the integration database and what is really present on the structure spotting
possible module registration error.\\
Only if this last test is passed, the user is allowed to create the fec.xml description file
needed to go forth with integration tests. Hence tests proceed with the data readout from
modules, which rely on TrackerOnline.
\paragraph*{The integration package}
The xDAQ version installed on the integration setups is very user-unfriendly, and requires
an expert user: many run-specific parameters are set manually and there is no
input validation.

\begin{figure}
\centering
\includegraphics[width=0.35\textwidth]{Figs/FedGuiMain.png}
\caption{Main GUI window.}
\label{fig:fedgui}
\end{figure}

A package interacting with TrackerOnline, capable
of automatically setting all relevant parameters and performing all data collection at the end 
of a run has been written. This is a 
finite state machine which cycle through the needed states setting run-specific parameters 
in the xDAQ software.

The state machine cycles through the following states:
\begin{enumerate}
\item TrackerOnline initialisation (only once)
\item Ask for the desired run
\item Launch run in TrackerOnline
\item Wait for run completion (polling the number of acquired events)
\item Stop data taking
\item Launch fast data analysis package
\item Ask user for data validation
\item If acknowledged, pack all the data and log the run with proper data quality flag. If the run
was a commissioning run, update fec.xml and module.xml
\end{enumerate}

For every required run, the integration package shows the proper TrackerOnline output
through some GUIs, so that the user can acknowledge the outcome of the run and give the approval 
for archiving the run data or, in case of a commissioning run, for updating the parameter set.\\
The TrackerOnline software computes the new parameter set for each commissioning run 
(except for the ``find connection'' run, as we'll see below) and writes
it locally as a fec.xml file, which is retrieved by the integration package at need.\\
A main integration database, for centralized archiving purposes, 
was also installed. Software implementing data analysis runs automatically on the
archived files producing validation outputs, accessible througth a web interface,
for every tested module.

\subsection{Test Description}
\label{ref:test-description}
\paragraph*{Find connections:} 
This commissioning run is used to know which module is connected to which FED input channel.
This procedure consists of switching on all the lasers of all AOH one by one, while checking the 
signal on all the FED inputs. If the difference of the signal seen on a FED channel is above
a given threshold, the connection between that laser and input channel is tagged and registered
in the module.xml as a connection table. 

\begin{figure}
\centering
\includegraphics[width=0.27\textwidth]{Figs/FedGuiChannels.png}
\caption{Connections displayed during the run}
\label{fig:fedguichan}
\end{figure}

At this point of the commissioning procedures both the descriptions for FECs 
(i.e. tracker hardware connected to the FECs) and for FEDs are present.

\paragraph*{Time alignment:}
This step is used to compensate the different delays in the control and readout chain,
i.e. the connections between FECs and FEEs.
This also makes the APVs' sampling time synchronous provided that AOH
fibres and ribbons are the same length. The latter is not important during integration
quality checks, as no external signal is ever measured, but it becomes so when
one tries to detect ionising particles.\\
This run type is relevant because, if FEDs are to sample properly
the APV signal, the clock must go to the modules synchronously, with a skew of the order of a few ns.
Also the clock coming to the three FEDs must be synchronous but this is guaranteed
by using cables of the same length between the clock-generating
board (TSC) and the FEDs themselves.\\
The time allignment run
makes use of the periodic tick mark signal sent by the APVs when it is clocked: 
after these devices receive a reset, they produce a tick mark signal every 70 clock cycles. 
During this run the FEDs continously sample the signals at the full clock frequency. 
This means that for every DAQ cycle the output of all APVs is measured as with a 40~MSample scope. 
After every cycle is completed all the PLLs' delays are icreased by (25/24)~ns 
(the minimum delay step), and the signal readout is performed again. After 24 such cycles the full
tick mark signal is measured as with a 960~MSample scope.

\begin{figure}
\centering
\includegraphics[width=.6\textwidth]{Figs/tickmark.pdf}
\caption{A tick mark sampled during a time alignment. The raising edge and the sampling point
are marked. In the picture are reported only those samplings around the tick mark, while
during the time alignment an interval of $1\,\mu\mathrm{s}$ is scanned.}
\label{fig:tick}
\end{figure}
The DAQ takes the time delay between tick marks as a measurement of the difference in delays in the
FEC-FEE connections and computes what delay must be set on each PLL in order to compensate this.
The tick mark raising edge $t_R$ time is measured by taking the time corresponding to the highest
signal derivative (see Fig.~\ref{fig:tick}).
The best sampling point is considered $t_R+15\,\mathrm{ns}$, to avoid
the possible overshoot. This is important also later, when measuring the analogue data frame, as
it allows measuring the signal coming from each strip after transient effects due to the signal
switching between strips are over.\\
At the end of this procedure, the user is shown all proposed adjustments to PLL delay values.
Then he can accept the outcome of the first time alignment run and, possibly,
repeat it. If the time allignment has been done correctly maximum variation of two or less
nanoseconds will be found. The delays are written in the correspoding xml file.

\paragraph*{Laser Scan:} 
This run makes a scan of all AOH bias values for the four possible gain settings
to determine the optimal gain and the corresponding optimal bias (see Fig.~\ref{fig:laserscan}).
In this run the APV generated tick marks are sampled (a correctly done
time allignment has to be done before) for all gain and bias values.

\begin{figure}[t!]
\begin{center}
\subfigure[A pictorial representation of a tick mark as produced by the APVs (dotted) 
and as transmitted by the lasers (solid) when the laser driver's bias is too 
low (left), correct (centre) or too high (right), with the subsequent signal saturation.]
{
        \label{fig:laserscan}
        \includegraphics[width=.9\textwidth]{Figs/laserscan.pdf}
}
\\
\subfigure[The sampled tick mark top and baseline as a function of the laser driver's bias.]
{
        \label{fig:gainscan_basetop}
        \includegraphics[width=.45\textwidth]{Figs/gainscan_basetop.pdf}
}
\hspace{5mm}
\subfigure[The corresponding tick mark height for a given gain.]
{
        \label{fig:gainscan_range}
        \includegraphics[width=.45\textwidth]{Figs/gainscan_range.pdf} 
}
\caption{Plots computed during a gain scan run.}
\end{center}
\end{figure}
For each trigger sent to FEDs, the tick mark's top is sampled twice and all other samplings fall
on the baseline. The highest samples are used to estimate the higher limit of the signal
for a given gain/bias pair and the lower values provide an estimate of the lower limit.
For each gain value three plots are computed: in the first two the high and low edges of 
the tick mark are represented as a function of bias (Fig.~\ref{fig:gainscan_basetop}) 
and the third, being the difference between the former, represent the dynamic range 
as a function of bias (Fig.~\ref{fig:gainscan_range}).\\
For each laser these plots are created 4 times: one for each possible gain value. The best laser
gain is computed as that providing an overall gain as close as possible to a given optimal value,
the overall gain being estimated as the slope of the first two curves in their central section.
The best bias value is taken as the one maximising the tick mark height keeping 
the maximum and minimum not saturated.\\
After the run is completed, values proposed by TrackerOnline are shown to the user, who
intervenes if there is any abnormal proposed gain, which may indicate a problem either on the
AOH or on the fibre.

\paragraph*{VPSP Scan:} 
This is the first run with FEDs with the header finding function active. In this run the 
trigger is dispatched also to modules, which send their data frames.
During this run a scan of VPSP parameter is performed on the modules, and for each value
their frames are acquired several times. As there is no physics
signal on the detectors, the sampled signal is a measurement of the pedestal of the analog channels.
The average strip pedestal is computed for every APV as a function of the VPSP parameter and 
at the end of the run the best VPSP pedestal is computed as that which moves the baseline 
to 1/3 of the dynamic range. This choice avoids setting a baseline too near the
lower saturation value leaving anyway enough range for possible signals from particles 
($\sim 6$ MIP equivalent).
At the end of the run computed values are proposed to the user for approval and written 
to the xml file.

\begin{figure}
\begin{center}
\subfigure[Strip pedestals of a module in ADC counts vs.\ strip number.]
{
        \label{fig:saturationpedestal}
        \includegraphics[width=.45\textwidth]{Figs/saturation_pedestal.pdf}
}
\hspace{5mm}
\subfigure[Strip noise of a module in ADC counts vs.\ strip number.]
{
        \label{fig:saturationnoise}
        \includegraphics[width=.45\textwidth]{Figs/saturation_noise.pdf} 
}
\caption{The pedestal of strips after \#{}640 is low, approaching to the bottom of the
dynamic range, and their noise is therefore lower.}
\label{fig:saturation}
\end{center}
\end{figure}
This run is not normally performed during the integration, as a default value was set on
all the APVs after a measurement on a sample. Anyway it is sometimes needed because the optimal
VPSP value may change from APV to APV and is also strongly dependent on temperature.
An example of this is shown in Fig.~\ref{fig:saturation}, where
only a few readout channels suffer from a signal saturation, while most of the strips of a module
are placed correctly inside the dynamic range.
This usually happens because the pedestal values of the channels of an APV are different one
from another and may have a dependency on the strip index like that shown in 
Fig.~\ref{fig:saturationpedestal}.
In these cases when the pedestal of a strip approaches to the edge of dynamic range     the
APV-AOH output is no more linear and the channel's RMS is lower, (see Fig.~\ref{fig:saturationnoise}).
This is one of the most frequent problems with the pedestals and it is solved
by a VPSP scan.

\paragraph*{Pedestal and Noise:} 
This is the main run for qualifying the detector performances during
the integration. The 400V bias are applied to the module under test which are 
also checked for any possible overcurrent or breakdown.\\
Triggers are sent to the modules and FEDs are placed in header recognition mode.
All the analogue frames from the modules are acquired and for each channel both the average value
and the RMS are computed.
Two analyses are performed on these data: one is done online by the TrackerOnline software, and
another one is performed offline through procedures taken from the ORCA package (ORCA was the
official reconstruction package of CMS at the time, now substituted by CMSSW) and run
on the files containing the raw data as acquired by FEDs in just the same way as it would
do with data coming to a Filter Farm.\\
The average value of the signal read on each strip is an estimate of
its pedestal, while the RMS is a good estimate of its noise, provided that the noise itself
is Gaussian, which is true to a first approximation. This value is often referred to as
\textit{raw noise}, as opposed to the \textit{common-mode subtracted noise} (or CMN). The latter
can be computed after pedestals are measured, which happens after the first hundreds of frames
are acquired.
The common mode noise subtraction performed by ORCA and TrackerOnline is similar to that
performed by the final FEDs.
This subtraction
eliminates the Common Mode Noise contribution to a specific event. After the Common
Mode Noise subtraction the noise can be computed again as the RMS of the remaining signal on strips
and this new noise measurement is called Common Mode Subtracted Noise (or just CMN).\\
Because of the difference in gain between the various optical links to compare the noise
on different APV pairs a renormalization is needed.
To implement this a gain measuring procedure to correct
noise measurements has been done. When FEDs acquire analogue frames, they store all acquired
raw data, comprising the samplings on the digital header. As the digital header has
the same amplitude on every APV, its measurement in terms of FED ADC counts
was used as an estimate of the relative gain of optical links.
This method allows a contextual measurement of noise and gain, and it is accurate,
provided that there is no signal saturation both on low and high values.\\
The normalisation factor was chosen so as to bring the normalised header height to 220 ADC counts,
which was the value of header's height as it was read out in the module test setup
of the module production line. This allowed the scaled measurements to be easily
compared with those done during module production tests. Both normalised CMN noise and raw noise
are shown to the user, while only uncalibrated CMN noise is plotted as a reference.
Also the distribution of normalised CMN and raw strip noise is computed and shown to
complete the information. 

\begin{figure}[t!]
\centering
\includegraphics[width=0.6\textwidth]{Figs/noiseprofile.pdf}
\caption{Noise profile of a TIB Layer 3 module. X axis represent the strip index
and y axis represent the noise in ADC counts (or ADC count equivalent for the
normalised noise).}
\label{fig:noiseprofile}
\end{figure}
Figure~\ref{fig:noiseprofile} shows an example of the output 
at the end of a noise measurement run: the normalised raw noise and CMN and the uncalibrated CMN
for each strip are plotted against the strip index. The first 256 strips belong to the
first APV pair and are multiplexed to a single optical line and the strips from 257 to 512
belong to the second APV pair. It can be noted here that the
raw noise without normalisation suffers from a different gain of optical links
and that after the normalisation procedures the noise level is the same
for the two APV pairs.\\
At the end of this run, once the outcomes are showed to the user, he can decide whether to
store these data or to cancel the procedure. In the first case data are packed along with
possible comments and sent to the central archive system, where they are processed 
again and made available on a web page. The system also automatically recognises where
the modules where mounted by checking their DCU Hardware Id (which is written in the fec.xml file)
in the construction database, this information is stored in the test table of
the integration database and allows to build a geographical table of mounted modules with
a link to a page containing all the tests performed for each module.\\

%%%%%%% aggiunta C.G.
\subsubsection{Single Module}
After a module of a string was mounted, its basic functionalities were tested. 
A fast test on the I$^2$C communication permitted to spot possible electrical problems
in the module front-end electronics in the AOH or in the mother cable. Since  the safe removal of a MC
from the shell requires the dismounting of al the modules of a string, it is very important
to perform this test of as soon as the first module of a string is integrated. After the 
I$^2$C test FecTool checks the identity of the components against the construction database. 
The results of the tests can then be monitored through a web page.

\subsubsection{String}
When the third and last module of a string is mounted the commissioning runs described in section~\ref{ref:test-description} are performed after the I$^2$C communication tests. The first run, ``Find connections'' permitted to check the full functionality of the AOH. Since the AOHs can be tested only after the modules are mounted this is the first test which can spot possibly broken fibres. It was necessary to perform this test after the integration of each string, because the subtitution of an AOH implies the dismounting of all the modules of the string mounted between the AOH and the front flange.\\
After this test a ``Time Alignement'' run, a  ``Laser scan'' run and finally a ``Pedestal and Noise'' run with HV on were performed. The ``Laser scan'' run was done limiting the laser gain to the lower value which was found to be optimal for all the AOHs in the integration setup. 

\subsubsection{Control Ring and Redundancy}
The control electronics can be fully tested only after complete assembly of a shell. This last test forsees a check of the correct operation of the ring and of the redundancy circuitry. A failure on a CCU or a control cable can be immediatly spotted as it causes the interruption of the ring; the communication with the other components is then checked using ProgramTest.\\
Finally the test of redundancy circuitry is performed by-passing each CCU one by one and verifying the correct response of the ring. The test is successful only if both the primary and secondary circuits of the DOHM are working correctly and if the CCUs are connected in the right order to the DOHM ports.
%%%%%%%% fine aggiunta C.G.

\section{Safety of operations}
The integration procedure posed many possible problems in the safety of operations,
from the routing of optical fibres to the handling of modules and each of these items
were addressed to minimise the risk of an accident. \\
For example
as the integration setup could not make use of the shell cooling system,
temperature measurements on the most sensible devices were done 
to be sure not to damage any components when the system is powered up.\\
The component with the most stringent requirement on temperature is the AOH
and in particular the optical coupling of the pig-tail fibres to lasers.

\begin{figure}[t!]
\centering
\includegraphics[width=.5\textwidth]{Figs/ir.pdf}
\caption{IR picture of and AOH seen from below (laser side)
taken after a laser scan when bias and gain are maximum. The lasers reach a temperature
of $40^\circ \mathrm{C}$, while the highest recorded temperature $48^\circ \mathrm{C}$,
on the hybrid.}
\label{fig:ir}
\end{figure}

An infrared camera has been used to monitor the temperature during the integration
tests. Figure~\ref{fig:ir} show a shot taken just
after a laser scan performed on one string of double-sided modules.
This run proved to be the most thermal
stressing among the commissioning run performed, because at its end
all laser drivers have the maximum gain and the highest bias possible with a complete
signal saturation.\\
For this test a double-sided layer has been used. In this case the modules have 
two back-to-back hybrids
and thus a higher power dissipation density with respect to the single-sided ones; 
furthermore the AOH are equipped with three lasers.\\
Even in the most stressing condition and after a few minutes of settling,
the highest measured temperature on the lasers was $\sim 40^\circ \mathrm{C}$
(while it was $48^\circ \mathrm{C}$ on the hybrid).
These are safe temperatures 
thus the integration tests could be performed without problems.
Revision:	1.1
Committed:	Tue Jan 20 11:13:35 2009 UTC (16 years, 3 months ago) by sguazz
Content type:	application/x-tex
Branch:	MAIN
Log Message:	First commit of TIB TID Integration Note