TIBTIDNotes/TIBTIDIntNote/IntegrationTests.tex

\section{Integration Tests}
\label{sec:Tests}

The CMS tracker has been designed to survive at ten years of LHC operation, with little or no chance 
for maintenance. Its various components have been tested during the production
to meet stringent quality requirements. Few important problems have been spotted and
solved.\\
During the TIB/TID integration all the operations have been monitored step by step by a chain of tests
aimed at a final control of the components just after the installation described here and at a verification of the
shell overall quality and functionality (~\ref{burnin}). The step by step tests are of particular importance 
because in most cases it is very difficult and in some cases even dangerous 
to replace a single faulty component when it is embedded in a fully equipped shell.

\subsection{Test Setup}
\label{sec:TestSetup}
The integration started well before the tracker 
final data acquisition hardware and software
were available to the Collaboration. Integration tests thus had to
rely on prototype DAQ hardware and peripherals and software versions
that has been frozen to ensure consistent conditions during the
integration activities time-span, except few minor upgrades and bug
fixes. 

\subsubsection{Hardware}
Here a brief account of the hardware used in DAQ for the integration 
is given (see Fig.~\ref{fig:integration daq}).
\begin{figure}
\centering
\includegraphics[width=\textwidth]{Figs/integrationDAQ.pdf}
\caption{Hardware used for the integration DAQ. 
\textbf{On the left:} 
\textbf{1:} The analog opto-electrical converter.
\textbf{2:} Optical lines from the AOHs inside the (yellow) ribbon. 
\textbf{3:} The signal is converted to electric.
\textbf{On the right:} 
\textbf{1:} The signal converted to electric goes to FEDs through 2-poles LEMOs. 
\textbf{2:} TSC provides trigger and clock to FEC through an (orange) optical fibre and 
\textbf{3:} to FEDs though 4-poles LEMOs. 
\textbf{4:} FEC also implements the 2-way communication towards CCUs and modules through a
(yellow) fibre ribbon.}
\label{fig:integration daq}
\end{figure}

\begin{description}
\item[TSC] The Trigger Sequencer Card or TSC~\cite{bib:specs:tsc}  generate the
40~MHz clock for the entire system and triggers as well, either
internally via software or by accepting external inputs. It has up to four
electrical clock/trigger outputs, enough to drive the FEDs, and an optical clock/trigger
output for the FEC.
The TSC may also generate the RESET and CALIBRATION signal that are
also encoded on the clock/trigger line.\\
\item[FED] The analog-to-digital conversion is done by special PCI FEDs~\cite{bib:fedpci}, 
with electrical differential analog input, mounted on 
PCI carrier boards and installed in an industrial PC.
The opto-electrical conversion of the analog signals is done externally by a 24-channel unit. 
A setup containing 3 FEDs, with the electro-opical converter, is able
to readout 48 APV; this is equivalent to 12 single sided modules
%(four complete strings) 
or four double sided modules assemblies.
%(one string plus one module). 
These figures are
pefectly suited for the tests during the integration.\\
For obvious reason the readout of the data from the APVs is not
synchronous with the L1 trigger. A crucial capability of the FED is the
\textit{header finding}, i.e. the automatical tagging of the
analog data stream from APV pairs with respect to the idle
signals at its inputs. This is possible since the APVs embeds the
analogue data stream within a {\em digital frame} made up of a leading 
digital header and a trailing tick-mark. The peculiarity of the PCI FED
with respect to the VME FED that will be used in the experiment, other than having electrical inputs 
instead of optical ones, is the timing system: the VME FED is able to
perform the header finding on each input channel independently; the PCI FED has
this capability only on the first channel of the eight available
and assumes that the all input signals are synchronised. 
This makes the PLL-based time alignment procedure of crucial
importance in a setup with PCI FEDs. Furthermore, PCI FEDs are not
able to insert a programmable delay on their inputs and thus 
it is important that the clock/trigger connections from the TSC to
FEDs have all the same delay. Last, PCI FEDs cannot perform an online
pedestal subtraction and zero suppression (they cannot run in
Processed Raw data nor in Zero Suppression mode).
\item[FEC] The special FEC used during the integration is the {\em FEC
  mezzanine} also installed into a PC on a PCI carrier. It supports
  the optical trigger/clock provided by the TSC and features an
  optical output directly connected to DOHs on the DOHM. 
\end{description}


\subsubsection{Software}
The software used to carry out integration tests is {\em TrackerOnline}, the CMS
tracker implementation of a more general software, named
xDAQ~\cite{ref:xdaq}, which is the official CMS DAQ framework.
In place of a database as in the
experiment version, the integration version of TrackerOnline uses a set of
xml files for all the configurations needed to perform a test run.
%, i.e. all the parameters needed by the devices and the
%software involved in the test run,
%\begin{figure}[bth!]
%\centering
%\includegraphics[width=\textwidth]{Figs/fecmodulexml_2.pdf}
%\caption{An example of data contained in fec.xml and module.xml files for one module.
%Part of the data is not shown for simplicity.}
%\label{fig:fecmodulexml}
%\end{figure}
\begin{description}
\item[Configuration of the DAQ hardware and software, daq.xml] The hardware
  and software configuration of the DAQ is written in a single xml file, which
  reflect the setup used for the test and has to be rarely changed
  during the integration procedures.
\item[Configuration of the control system, fec.xml] The most important
  part of this configuration section is the settings the FEC must
  upload to the modules and AOHs and in general any configurable
  $I^2C$ device, before starting the data
taking. Altough these settings are not specifically
  related to the control system, it is duty of the control system to
  write them to the devices' $I^2C$ registers and read them back for verification.
\item[Configuration of the readout, module.xml] All other information needed by the DAQ to
rearrange data coming from the FEDs is in module.xml that allows each
FEDs input channel to be mapped to an APV pair of a specific module as
identified by ring, CCU and $I^2C$ addresses (i.e. the correspondance
between readout coordinates and Control System coordinates);
%Each row corresponds to an APV
%pair, an AOH laser, an optical fibre and a FED input.\\
%The second table of module.xml reassembles the information on a module basis. 
%Here all the active modules are listed, with a row for every module. 
%The Control System coordinates are repeated both in module.xml and fec.xml, 
%so that they can be used as a pivot between the 3 tables.\\
\end{description}

The tasks performed by the integration software are the following:
execution of the commissioning runs needed to optimal adjust of the
module parameters (preparation of fec.xml and module.xml); execution
of the test runs with complete and automated logging; fast analysis
for immediate feedback; archival of xml configuration files to log the
test conditions; archival of the raw data.


%\begin{figure}
%\centering
%\includegraphics[width=.9\textwidth]{Figs/integration_package.pdf}
%\caption{Scheme of the integration software. 
%\textit{Arrows}: a relation.
%\textit{Gears}: an application. 
%\textit{Mouse:} an interactive application. 
%\textit{Sheet}: a file.}
%\label{fig:integration_package}
%\end{figure}

%Figure~\ref{fig:integration_package} shows a scheme of the relations between all the software
%components that will be described, the local files and the remote database.

This is achieved by using a specific set of components of TrackerOnline.
\begin{description}
\item[FecTool.]
FecTool is GUI based front-end to two standalone
applications deployed with TrackerOnline, FecProfiler 
and ProgramTest, aimed to ease the creation of the device 
description. 
FecProfiler is able to detect
the devices connected to the CCUs and builds the fec.xml file needed by
TrackerOnline. FecTool takes care of checking that thedetected devices
corresponds to expected ones, i.e., per module, 4 or 6 APVs, one
PLL, one AOH, and so on.  ProgramTest allows the ring functionalities,
i.e. the redundancy, to be deeply tested.

The geographical identity of the strings under test must be
entered to allows for verification from FecTool of the matching
between the DCU Hardware Id read from each
module and the one declared in the module database. This
consistency check is crucial to spot possible errors in recording the
location where a module is mounted during the assembly. If the check
is passed, the fec.xml description file needed to go forth with
integration tests can be created. 
%Hence tests proceed with the data
%readout from modules, which rely on . 
\item[The Integration Package.]
TrackerOnline as any xDAQ implementation requires an expert user as
many run-specific parameters must be set and there is no  input
validation. 
%\begin{figure}
%\centering
%\includegraphics[width=0.35\textwidth]{Figs/FedGuiMain.png}
%\caption{Main GUI window.}
%\label{fig:fedgui}
%\end{figure}
The integration setup is mode more user-friendly by a special {\em
  Integration Package}, a GUI based front end that interacts with
  TrackerOnline to automatically set all relevant parameters and to harvest all data at the end 
of a run. The package is organized as a finite state machine by which
  the user can cycle between the various states, i.e. the following integration test steps:
\begin{enumerate}
\item TrackerOnline initialisation (only once);
\item choice of the the desired run;
\item execution of the run via TrackerOnline;
\item on run completion (i.e. after a given number of events), stop
  data taking and execution of the fast data analysis;
\item presentation of the run outcome on summary GUIs;
\item on positive validation from the user, data are stored together
  with run logs and data quality flags;
\item in case of commissioning runs, on positive validation from the
  user, fec.xml and module.xml are updated accordingly to be used from
  now on.
\end{enumerate}

%For every required run, the integration package shows the TrackerOnline output
%through some GUIs, so that the user can acknowledge the outcome of the run and give the approval 
%for archiving the data or, in case of a commissioning run, for updating the parameter set.\\
%The TrackerOnline software computes the new parameter set for each commissioning run 
%(except for the ``find connection'' run, as we'll see below) and writes
%it locally as a fec.xml file, which is retrieved by the integration package at need.\\
A main integration database has been setup for centralized archiving purposes, 
was also installed. For later reference, if needed, the data analysis
can also be run on the archived files with validation outputs made
available through a web interface. 
\end{description}

\subsection{Test Description}
\label{ref:test-description}

Each run type that can be choosen by the user corresponds to a
commissioning run or to a test run, as described below.

\begin{description}
\item[Find connections.] This commissioning run is used associate each
  FED input channel to a module. The procedure, repeated in sequence
  for all AOHs laser drivers, consists of switching on only a given
  AOH laser driver while checking the signal on all the FED inputs. If
  the difference of the signal seen on a FED channel is above 
a given threshold, the connection between that laser and input channel is tagged and stored
in module.xml. 
\item[Time alignment.]
This commissioning run measures the appropriate delays to be later set
in the PLLs delay registers. 
Doing so the different delays in the control and
readout chain are compensated, the clock arrives to the modules
synchronously, with a skew of the order of a few ns, and the APVs
signal are properly sampled by the FEDs. This requires also the clock
to all the FEDs to be synchronous, but this is guaranteed
by using cables equal in length between the TSC and the FEDs.\\
The time alignment run uses of the periodic tick mark signal issued by
the idle APVs every 70 clock cycles. The APV signals are sampled by FEDs in
scope mode, i.e. without waiting for an header but continously,
sampling the inputs at the full clock frequency as with a 40~MSample
scope. The measurement is repeated after all the PLL delays are
icreased by the minimum delay step, 25/24~ns. After 24 such cycles the
idle APV output and thus the tick mark signal also are measured with
an effective 960~MSample scope. 
\begin{figure}
\centering
\includegraphics[width=.6\textwidth]{Figs/tickmark.pdf}
\caption{A tick mark sampled during a time alignment. The raising edge and the sampling point
are marked. In the picture are reported only those samplings around the tick mark, while
during the time alignment an interval of $1\,\mu\mathrm{s}$ is scanned.}
\label{fig:tick}
\end{figure}
The time differences between the variuos APV tick marks are a
measurement of the relative delays introduced by the connections and
can be used to compute the optimal delay to be set on each PLL for compensation.
The tick mark raising edge $t_R$ time is measured by taking the time corresponding to the highest
signal derivative (see Fig.~\ref{fig:tick}).
The best sampling point is considered $t_R+15\,\mathrm{ns}$, to avoid
the possible overshoot.
%[???This is important also later, when measuring the analogue data frame, as
%it allows measuring the signal coming from each strip after transient effects due to the signal
%switching between strips are over.???]\\
At the end of the procedure, all proposed adjustments to PLL delay
values are proposed to the user. If accepted the delays are written in
fec.xml. If the setup is correctly aligned in time, a further time
alignment procedure should not propose delay corrections greater than
$\sim 2$ns.
It is worth noticing that by the time alignment procedure all APVs are
made sampling synchronously, since in the integration setup AOH fibres
and ribbons are all equal in length. This is not important during
integration quality checks, as no external signal is ever measured,
but it would be so in trying to detect ionising particles.
\item[Laser Scan.] By this commissioning run a scan of any bias and
  gain value pair is done to determine the optimal working point for
  each AOH. 
The procedure requires a sucessfull time alignment since again tick marks are
sampled but in this case changing gain and bias values.
%
\begin{figure}[t!]
\begin{center}
\subfigure[The sampled tick mark top and baseline as a function of the laser driver's bias.]
{
        \label{fig:gainscan_basetop}
        \includegraphics[width=.45\textwidth]{Figs/gainscan_basetop.pdf}
}
\hspace{5mm}
\subfigure[The corresponding tick mark height for a given gain.]
{
        \label{fig:gainscan_range}
        \includegraphics[width=.45\textwidth]{Figs/gainscan_range.pdf} 
}
\subfigure[A pictorial representation of a tick mark as produced by the APVs (dotted) 
and as transmitted by the lasers (solid) when the laser driver's bias is too 
low (left), correct (centre) or too high (right), with the subsequent signal saturation.]
{
        \label{fig:laserscan}
        \includegraphics[width=.9\textwidth]{Figs/laserscan.pdf}
}
\caption{Plots computed during a gain scan run.}
\end{center}
\end{figure}
%
For each trigger sent to FEDs, the is sampled twice and all other samplings fall
on the baseline. The tick mark's top samples are used to estimate the higher limit of the signal
for a given gain/bias setting pair and the baseline samples provide an estimate of the lower limit.
For each gain setting the tick mark top samples and the baseline samples are measured as a
function of the bias, as shown in Fig.~\ref{fig:gainscan_basetop}. For
each bias setting their difference is the tick mark height, shown in Fig.~\ref{fig:gainscan_range},
that represents the dynamic range as a function of the bias.\\
The best bias setting for a given gain setting is taken as the one maximising the tick mark height keeping 
the maximum and minimum not saturated, as pictorially represented in Fig.~\ref{fig:laserscan}.
The same measurement is done for each possible gain setting.
The best laser gain setting is the one providing an overall gain of
the optical chain as close as possible to the design one, 0.8~\cite{ref:gain}.
The overall gain is estimated from the slope of the curves of
Fig.~\ref{fig:gainscan_basetop} in their central section.
After the run is completed a set of values are proposed to the
user. Abnormal gain values may indicate a problem either on the AOH or
on the fibre and are investigated.
\item[VPSP Scan.] 
This commissioning run is devoted to optimise the pedestal of
the APV, i.e. the average output level in absence of any signal, with
respect to the dynamical range of the FEDs. This level is managed by a
specific APV register, know as {\em VPSP}, which controls a voltage
setting within the deconvolution circuitry. The procedure consists of
a scan of VPSP values while acquiring data frames from modules in the
standard way, i.e. trigger sent to the modules and FEDs in
``header finding'' mode.
The optimal VPSP setting correspond to a pedestal baseline placed
around 1/3 of the available dynamic range, a good compromise to keep
the baseline not too close the lower saturation value while leaving a good
range for particle signals, corresponding to $\sim 6$ MIP.
At the end of the run a set of values are proposed to the user for
approval andin case written in the relevant xml file.
\begin{figure}
\begin{center}
\subfigure[Strip pedestals of a module in ADC counts vs.\ strip number.]
{
        \label{fig:saturationpedestal}
        \includegraphics[width=.45\textwidth]{Figs/saturation_pedestal.pdf}
}
\hspace{5mm}
\subfigure[Strip noise of a module in ADC counts vs.\ strip number.]
{
        \label{fig:saturationnoise}
        \includegraphics[width=.45\textwidth]{Figs/saturation_noise.pdf} 
}
\caption{The pedestal of strips after \#{}640 is low, approaching to the bottom of the
dynamic range, and their noise is therefore lower.}
\label{fig:saturation}
\end{center}
\end{figure}
The VPSP scan is not sistematically performed during the integration,
since the default VPSP setting is adequate in most of the
cases. Nevertheless, VPSP optimal values change considerably within
the APV population and are strongly temperature dependent and is
tather common to have a stuation in which the pedestal of few readout
channels approaches to the lower edge of dynamic range
(Fig.~\ref{fig:saturationpedestal}) resulting in a lower RMS (see
Fig.~\ref{fig:saturationnoise}). The VPSP scan allows for this issue to be
fixed.
\item[Pedestal and Noise Run.] 
This is the main run type for qualifying the performances during
the integration. Tipically a bias of 400V is applied to the modules
under test to check for any possible overcurrent or breakdown.\\
Triggers are sent to the modules and FEDs work in ``header finding'' mode.
All the analogue frames from the modules are collected two analyses
are performed on these data: online, by the TrackerOnline 
software; offline, in a way very similar to the final experiment by
using algorythms of the ORCA package~\cite{bib:orca}, the CMS
reconstruction package at that time, now replaced by CMSSW.
The average value of the signal read on each strip is an estimate of
its pedestal, while the RMS is a good estimate of its noise, provided that the noise itself
is Gaussian, which is true to a first approximation. This value is often referred to as
\textit{raw noise}, as opposed to the \textit{common-mode subtracted
  noise} (or CMN). The latter is the RMS computed after having
subtracted the {\em common noise}, i.e. the correlated noise-like fluctuation
common to a given group of channels (tipically an entire APV).
The common mode noise subtraction method implemented in ORCA and
TrackerOnline is similar to that performed by the final FEDs.\\
Noise comparison between different APV pairs requires a
renormalization, because of the difference in gain between the various
optical links. The normalization procedure relies on the digital
headers whose amplitude, being the same on each APV, is used to
estimate of the relative gain of optical links so to apply an
appropriate correction. In such a way noise and gain are
simultaneously measured provided that the signal is not saturation both on low and high values.
The normalisation factor is chosen so as to bring the normalised
header height to 220 ADC counts, as measured in quality controls
during the module production. This allowed the scaled measurements to be easily
compared with those done during module production tests.\\
At the end of the run, pedestal ando noise profiles and distributions
are shown to the user.
\begin{figure}[t!]
\centering
\includegraphics[width=0.6\textwidth]{Figs/noiseprofile.pdf}
\caption{Noise profile of a TIB Layer 3 module. X axis represent the strip index
and y axis represent the noise in ADC counts (or ADC count equivalent for the
normalised noise).}
\label{fig:noiseprofile}
\end{figure}
Figure~\ref{fig:noiseprofile} shows an example of the noise output:
the normalised raw noise and CMN and the uncalibrated CMN 
for each strip are plotted against the strip index. The first 256 strips belong to the
first APV pair and are multiplexed to a single optical line and the strips from 257 to 512
belong to the second APV pair. It can be noted here that the
raw noise without normalisation suffers from a different gain of
optical links, corrected by the normalisation procedure.\\
If validated by the user, data are packed along with
possible comments and sent to the central archive system, where they are processed 
again and made available on a web page. 
%The system also automatically recognises where
%the modules where mounted by checking their DCU Hardware Id (which is written in the fec.xml file)
%in the construction database, this information is stored in the test table of
%the integration database and allows to build a geographical table of mounted modules with
%a link to a page containing all the tests performed for each module.\\
\end{description}

%%%%%%% aggiunta C.G.
The basic run types and tests described above are appropriately
combined according to the devices and/or the group of devices under test. 
\begin{description}
\item[Single Module.] After a module is mounted, its basic
  functionalities can be tested. In particular, a fast test on the
  I$^2$C communication permits possible 
electrical problems to be spotted in the module front-end electronics,
in the AOH or in the mother cable. For mother cable and AOH this is of
  particular importance: an AOHs can be practically tested only
  after the corresponding modules is mounted and this is the first
  test which can spot possibly broken fibres; similarly for the MC, it
  is very important to perform the test of as soon as possible. In
  fact, a safe removal and replacement of either an AOH or the MC is a
  difficult intervention possibly requiring the dismounting of many
  modules already put in place. 
  safe removal of a MC from the shell \\ Since  After the 
I$^2$C test FecTool checks the identity of the components with respect
to the construction database. The results of the tests can then be
monitored through a web page. 
\item[String]
When the third and last module or double sided assembly is mounted on
a string, all the commissioning runs described above are performed
just after the I$^2$C communication tests. The ``Find
connections'' run allows for the full functionality to be checked. 
Afterwards a ``Time Alignement'' run, a  ``Laser scan'' run and
finally a ``Pedestal and Noise'' run with HV bias are performed. The
``Laser scan'' run is done limiting the laser gain to the lower value
which is found to be enough for the integration setup needs.
\item[Control Ring and Redundancy.]
The control electronics can be fully tested only after complete
assembly of a shell. This last test forsees a check of the correct
operation of the ring and of the redundancy circuitry. A failure on a
CCU or a control cable can be immediatly spotted as it causes the
interruption of the ring; the communication with the other components
is then checked using ProgramTest. Finally the test of redundancy
circuitry is performed by-passing each CCU one by one and verifying
the correct response of the ring. The test is successful only if both
the primary and secondary circuits of the DOHM are working correctly
and if the CCUs are connected in the right order to the DOHM ports.
%%%%%%%% fine aggiunta C.G.
\end{description}

\section{Safety of operations}
The integration procedure posed many possible problems in the safety of operations,
from the routing of optical fibres to the handling of modules and each of these items
were addressed to minimise the risk of an accident. \\
For example
as the integration setup could not make use of the shell cooling system,
temperature measurements on the most sensible devices were done 
to be sure not to damage any components when the system is powered up.\\
The component with the most stringent requirement on temperature is the AOH
and in particular the optical coupling of the pig-tail fibres to lasers.

\begin{figure}[t!]
\centering
\includegraphics[width=.5\textwidth]{Figs/ir.pdf}
\caption{IR picture of and AOH seen from below (laser side)
taken after a laser scan when bias and gain are maximum. The lasers reach a temperature
of $40^\circ \mathrm{C}$, while the highest recorded temperature $48^\circ \mathrm{C}$,
on the hybrid.}
\label{fig:ir}
\end{figure}

An infrared camera has been used to monitor the temperature during the integration
tests. Figure~\ref{fig:ir} show a shot taken just
after a laser scan performed on one string of double-sided modules.
This run proved to be the most thermal
stressing among the commissioning run performed, because at its end
all laser drivers have the maximum gain and the highest bias possible with a complete
signal saturation.\\
For this test a double-sided layer has been used. In this case the modules have 
two back-to-back hybrids
and thus a higher power dissipation density with respect to the single-sided ones; 
furthermore the AOH are equipped with three lasers.\\
Even in the most stressing condition and after a few minutes of settling,
the highest measured temperature on the lasers was $\sim 40^\circ \mathrm{C}$
(while it was $48^\circ \mathrm{C}$ on the hybrid).
These are safe temperatures 
thus the integration tests could be performed without problems.
Revision:	1.2
Committed:	Mon Mar 9 15:41:05 2009 UTC (16 years, 1 month ago) by sguazz
Content type:	application/x-tex
Branch:	MAIN
Changes since 1.1:	+352 -315 lines
Log Message:	Text revision, some figures added, some removed