claudioc/OSNote2010/datadriven.tex

\section{Data Driven Background Estimation Methods}
\label{sec:datadriven}
We have developed two data-driven methods to 
estimate the background in the signal region.
The first one explouts the fact that 
\met and \met$/\sqrt{\rm SumJetPt}$ are nearly 
uncorrelated for the $t\bar{t}$ background 
(Section~\ref{sec:abcd});  the second one 
is based on the fact that in $t\bar{t}$ the
$P_T$ of the dilepton pair is on average 
nearly the same as the $P_T$ of the pair of neutrinos
from $W$-decays, which is reconstructed as \met in the
detector.

in 30 pb$^{-1}$ we expect $\approx$ 1 SM event in 
the signal region.  The expectations from the LMO
and LM1 SUSY benchmark points are {\color{red} XX} and
{\color{red} XX} events respectively.


\subsection{ABCD method}
\label{sec:abcd}

We find that in $t\bar{t}$ events \met and 
\met$/\sqrt{\rm SumJetPt}$ are nearly uncorrelated.
This is demonstrated in Figure~\ref{fig:uncor}.
Thus, we can use an ABCD method in the \met$/\sqrt{\rm SumJetPt}$ vs
sumJetPt plane to estimate the background in a data driven way.

\begin{figure}[tb]
\begin{center}
\includegraphics[width=0.75\linewidth]{uncorrelated.pdf}
\caption{\label{fig:uncor}\protect Distributions of SumJetPt 
in MC $t\bar{t}$ events for different intervals of 
MET$/\sqrt{\rm SumJetPt}$.}
\end{center}
\end{figure}

\begin{figure}[bt]
\begin{center}
\includegraphics[width=0.75\linewidth]{abcdMC.jpg}
\caption{\label{fig:abcdMC}\protect Distributions of SumJetPt 
vs. MET$/\sqrt{\rm SumJetPt}$ for SM Monte Carlo.  Here we also
show our choice of ABCD regions. {\color{red} We need a better
picture with the letters A-B-C-D and with the numerical values
of the boundaries clearly indicated.}}
\end{center}
\end{figure}


Our choice of ABCD regions is shown in Figure~\ref{fig:abcdMC}.
The signal region is region D.  The expected number of events 
in the four regions for the SM Monte Carlo, as well as the BG 
prediction AC/B are given in Table~\ref{tab:abcdMC} for an integrated
luminosity of 30 pb$^{-1}$.  The ABCD method is accurate 
to about 10\%.

\begin{table}[htb]
\begin{center}
\caption{\label{tab:abcdMC} Expected SM Monte Carlo yields for 
30 pb$^{-1}$ in the ABCD regions.}
\begin{tabular}{|l|c|c|c|c||c|}
\hline
Sample   & A   & B    & C   & D   & AC/D \\ \hline
ttdil    & 6.4 & 28.4 & 4.2 & 1.0 & 0.9  \\
Zjets    & 0.0 & 1.3  & 0.2 & 0.0 & 0.0  \\
Other SM & 0.6 & 2.1  & 0.2 & 0.1 & 0.0  \\ \hline
total MC & 7.0 & 31.8 & 4.5 & 1.1 & 1.0 \\ \hline
\end{tabular}
\end{center}
\end{table}

\subsection{Dilepton $P_T$ method}
\label{sec:victory}
This method is based on a suggestion by V. Pavlunin\cite{ref:victory},
and was investigated by our group in 2009\cite{ref:ourvictory}.
The idea is that in dilepton $t\bar{t}$ events the lepton and neutrinos
from $W$ decays have the same $P_T$ spectrum (modulo $W$ polarization 
effects).  One can then use the observed 
$P_T(\ell\ell)$ distribution to model the sum of neutrino $P_T$'s which 
is identified with the \met.

Then, in order to predict the $t\bar{t} \to$ dilepton contribution to a 
selection with \met$+$X, one applies a cut on $P_T(\ell\ell)+$X instead.
In practice one has to rescale the result of the $P_T(\ell\ell)+$X selection
to account for the fact that any dilepton selection must include a 
moderate \met cut in order to reduce Drell Yan backgrounds.  This 
is discussed in Section 5.3 of Reference~\cite{ref:ourvictory}; for a \met
cut of 50 GeV, the rescaling factor is obtained from the data as

\newcommand{\ptll} {\ensuremath{P_T(\ell\ell)}}
\begin{center}
$ K = \frac{\int_0^{\infty} {\cal N}(\ptll)~~d\ptll~}{\int_{50}^{\infty} {\cal N}(\ptll)~~d\ptll~}$
\end{center}


Monte Carlo studies give values of $K$ that are typically between 1.5 and 1.6,
depending on selection details.

There are several effects that spoil the correspondance between \met and
$P_T(\ell\ell)$: 
\begin{itemize}
\item $Ws$ in top events are polarized.  Neutrinos are emitted preferentially
forward in the $W$ rest frame, thus the $P_T(\nu\nu)$ distribution is harder
than the $P_T(\ell\ell)$ distribution for top dilepton events.
\item The lepton selections results in $P_T$ and $\eta$ cuts on the individual
leptons that have no simple correspondance to the neutrino requirements.
\item Similarly, the \met$>$50 GeV cut introduces an asymmetry between leptons and
neutrinos which is only partially compensated by the $K$ factor above.
\item The \met resolution is much worse than the dilepton $P_T$ resolution.
When convoluted with a falling spectrum in the tails of \met, this result
in a harder spectrum for \met than the original $P_T(\nu\nu)$.
\item The \met response in CMS is not exactly 1.  This causes a distortion
in the \met distribution that is not present in the $P_T(\ell\ell)$ distribution.
\item The $t\bar{t} \to$ dilepton signal includes contributions from 
$W \to \tau \to \ell$.  For these events the arguments about the equivalence 
of $P_T(\ell\ell)$ and $P_T(\nu\nu)$ do not apply.
\item A dilepton selection will include SM events from non $t\bar{t}$ 
sources.  These events can affect the background prediction.  Particularly 
dangerous are high $P_T$ Drell Yan events that barely pass the \met$>$ 50 
GeV selection.  They will tend to push the data-driven background prediction up.
\end{itemize}

We have studied these effects in SM Monte Carlo, using a mixture of generator and
reconstruction level studies, putting the various effects in one at a time.
For each configuration, we apply the data-driven method and report as figure
of merit the ratio of observed and predicted events in the signal region.
The results are summarized in Table~\ref{tab:victorybad}.

\begin{table}[htb]
\begin{center}
\caption{\label{tab:victorybad} Test of the data driven method in Monte Carlo 
under different assumptions.  See text for details.}
\begin{tabular}{|l|c|c|c|c|c|c|c|}
\hline
& True $t\bar{t}$ dilepton & $t\to W\to\tau$& other SM & GEN or  & Lepton $P_T$    & \met $>$ 50& obs/pred \\
& included                 & included  & included & RECOSIM & and $\eta$ cuts &      &     \\ \hline
1&Y                        &     N     &   N      &  GEN    &   N             &   N  &       \\
2&Y                        &     N     &   N      &  GEN    &   Y             &   N  &   \\
3&Y                        &     N     &   N      &  GEN    &   Y             &   Y  &   \\
4&Y                        &     N     &   N      & RECOSIM &   Y             &   Y  &   \\
5&Y                        &     Y     &   N      & RECOSIM &   Y             &   Y  &   \\
6&Y                        &     Y     &   Y      & RECOSIM &   Y             &   Y  &   \\
\hline
\end{tabular}
\end{center}
\end{table}


The largest discrepancy between prediction and observation occurs on the first 
line of Table~\ref{tab:victorybad}, {\em i.e.}, at the generator level with no
cuts.  We have verified that this effect is due to the polarization of
the $W$ (we remove the polarization by reweighting the events and we get
good agreement between prediction and observation).  The kinematical 
requirements (lines 2 and 3) do not have a significant additional effect. 
Going from GEN to RECOSIM there is a significant change in observed/predicted.  
We have tracked this down to the fact that tcMET underestimates the true \met
by $\approx 4\%$\footnote{We find that observed/predicted changes by roughly 0.1
for each 1.5\% change in \met response.}.  Finally, contamination from non $t\bar{t}$
events can have a significant impact on the BG prediction.  The changes between 
lines 5 and 6 of Table~\ref{tab:victorybad} is driven by only {\color{red} 3}
Drell Yan events that pass the \met selection.

An additional source of concern is that the CMS Madgraph $t\bar{t}$ MC does
not include effects of spin correlations between the two top quarks.  
We have studied this effect at the generator level using Alpgen.  We find
that the bias is a the few percent level.

Based on the results of Table~\ref{tab:victorybad}, we conclude that the 
naive data driven background estimate based on $P_T{\ell\ell)}$ needs to 
be corrected by a factor of {\color{red} $1.4 \pm 0.3$  (We need to 
decide what this number should be)}.  The quoted 
uncertainty is based on the stability of the Monte Carlo tests under 
variations of event selections, choices of \met algorithm, etc.


\subsection{Signal Contamination}
\label{sec:sigcont}

All data-driven methods are principle subject to signal contaminations
in the control regions, and the methods described in 
Sections~\ref{sec:abcd} and~\ref{sec:victory} are not exceptions.
Signal contamination tends to dilute the significance of a signal
present in the data by inflating the background prediction.

It is hard to quantify how important these effects are because we 
do not know what signal may be hiding in the data.  Having two
independent methods (in addition to Monte Carlo ``dead-reckoning'')
adds redundancy because signal contamination can have different effects
in the different control regions for the two methods.
For example, in the extreme case of a
new physics signal 
with $P_T(\ell \ell) = \met$, an excess of ev ents would be seen 
in the ABCD method but not in the $P_T(\ell \ell)$ method.

The LM points are benchmarks for SUSY analyses at CMS.  The effects
of signal contaminations for a couple such points are summarized
in Table~\ref{tab:sigcontABCD} and~\ref{tab:sigcontPT}. 
Signal contamination is definitely an important
effect for these two LM points, but it does not totally hide the
presence of the signal.


\begin{table}[htb]
\begin{center}
\caption{\label{tab:sigcontABCD} Effects of signal contamination 
for the background predictions of the ABCD method including LM0 or 
LM1.  Results
are normalized to 30 pb$^{-1}$.}
\begin{tabular}{|c||c|c||c|c|}
\hline
SM         & LM0         & BG Prediction & LM1          & BG Prediction \\
Background & Contribution& Including LM0 & Contribution & Including LM1  \\ \hline
x          & x           & x             & x            & x \\ 
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[htb]
\begin{center}
\caption{\label{tab:sigcontPT} Effects of signal contamination 
for the background predictions of the $P_T(\ell\ell)$ method including LM0 or 
LM1.  Results
are normalized to 30 pb$^{-1}$.}
\begin{tabular}{|c||c|c||c|c|}
\hline
SM         & LM0         & BG Prediction & LM1          & BG Prediction \\
Background & Contribution& Including LM0 & Contribution & Including LM1  \\ \hline
x          & x           & x             & x            & x \\ 
\hline
\end{tabular}
\end{center}
\end{table}

Revision:	1.2
Committed:	Fri Oct 29 02:29:40 2010 UTC (14 years, 6 months ago) by claudioc
Content type:	application/x-tex
Branch:	MAIN
Changes since 1.1:	+163 -3 lines
Log Message:	blah
#	User	Rev	Content
1	claudioc	1.1	\section{Data Driven Background Estimation Methods}
2			\label{sec:datadriven}
3			We have developed two data-driven methods to
4			estimate the background in the signal region.
5			The first one explouts the fact that
6			\met and \met$/\sqrt{\rm SumJetPt}$ are nearly
7			uncorrelated for the $t\bar{t}$ background
8			(Section~\ref{sec:abcd}); the second one
9			is based on the fact that in $t\bar{t}$ the
10			$P_T$ of the dilepton pair is on average
11			nearly the same as the $P_T$ of the pair of neutrinos
12			from $W$-decays, which is reconstructed as \met in the
13			detector.
14
15			in 30 pb$^{-1}$ we expect $\approx$ 1 SM event in
16			the signal region. The expectations from the LMO
17			and LM1 SUSY benchmark points are {\color{red} XX} and
18			{\color{red} XX} events respectively.
19
20
21			\subsection{ABCD method}
22			\label{sec:abcd}
23
24			We find that in $t\bar{t}$ events \met and
25			\met$/\sqrt{\rm SumJetPt}$ are nearly uncorrelated.
26			This is demonstrated in Figure~\ref{fig:uncor}.
27			Thus, we can use an ABCD method in the \met$/\sqrt{\rm SumJetPt}$ vs
28			sumJetPt plane to estimate the background in a data driven way.
29
30	claudioc	1.2	\begin{figure}[tb]
31	claudioc	1.1	\begin{center}
32			\includegraphics[width=0.75\linewidth]{uncorrelated.pdf}
33			\caption{\label{fig:uncor}\protect Distributions of SumJetPt
34			in MC $t\bar{t}$ events for different intervals of
35			MET$/\sqrt{\rm SumJetPt}$.}
36			\end{center}
37			\end{figure}
38
39	claudioc	1.2	\begin{figure}[bt]
40	claudioc	1.1	\begin{center}
41			\includegraphics[width=0.75\linewidth]{abcdMC.jpg}
42			\caption{\label{fig:abcdMC}\protect Distributions of SumJetPt
43			vs. MET$/\sqrt{\rm SumJetPt}$ for SM Monte Carlo. Here we also
44			show our choice of ABCD regions. {\color{red} We need a better
45			picture with the letters A-B-C-D and with the numerical values
46			of the boundaries clearly indicated.}}
47			\end{center}
48			\end{figure}
49
50
51			Our choice of ABCD regions is shown in Figure~\ref{fig:abcdMC}.
52			The signal region is region D. The expected number of events
53			in the four regions for the SM Monte Carlo, as well as the BG
54	claudioc	1.2	prediction AC/B are given in Table~\ref{tab:abcdMC} for an integrated
55	claudioc	1.1	luminosity of 30 pb$^{-1}$. The ABCD method is accurate
56			to about 10\%.
57
58			\begin{table}[htb]
59			\begin{center}
60			\caption{\label{tab:abcdMC} Expected SM Monte Carlo yields for
61			30 pb$^{-1}$ in the ABCD regions.}
62			\begin{tabular}{\|l\|c\|c\|c\|c\|\|c\|}
63			\hline
64			Sample & A & B & C & D & AC/D \\ \hline
65			ttdil & 6.4 & 28.4 & 4.2 & 1.0 & 0.9 \\
66			Zjets & 0.0 & 1.3 & 0.2 & 0.0 & 0.0 \\
67			Other SM & 0.6 & 2.1 & 0.2 & 0.1 & 0.0 \\ \hline
68			total MC & 7.0 & 31.8 & 4.5 & 1.1 & 1.0 \\ \hline
69			\end{tabular}
70			\end{center}
71			\end{table}
72
73	claudioc	1.2	\subsection{Dilepton $P_T$ method}
74			\label{sec:victory}
75			This method is based on a suggestion by V. Pavlunin\cite{ref:victory},
76			and was investigated by our group in 2009\cite{ref:ourvictory}.
77			The idea is that in dilepton $t\bar{t}$ events the lepton and neutrinos
78			from $W$ decays have the same $P_T$ spectrum (modulo $W$ polarization
79			effects). One can then use the observed
80			$P_T(\ell\ell)$ distribution to model the sum of neutrino $P_T$'s which
81			is identified with the \met.
82
83			Then, in order to predict the $t\bar{t} \to$ dilepton contribution to a
84			selection with \met$+$X, one applies a cut on $P_T(\ell\ell)+$X instead.
85			In practice one has to rescale the result of the $P_T(\ell\ell)+$X selection
86			to account for the fact that any dilepton selection must include a
87			moderate \met cut in order to reduce Drell Yan backgrounds. This
88			is discussed in Section 5.3 of Reference~\cite{ref:ourvictory}; for a \met
89			cut of 50 GeV, the rescaling factor is obtained from the data as
90
91			\newcommand{\ptll} {\ensuremath{P_T(\ell\ell)}}
92			\begin{center}
93			$ K = \frac{\int_0^{\infty} {\cal N}(\ptll)~~d\ptll~}{\int_{50}^{\infty} {\cal N}(\ptll)~~d\ptll~}$
94			\end{center}
95
96
97			Monte Carlo studies give values of $K$ that are typically between 1.5 and 1.6,
98			depending on selection details.
99
100			There are several effects that spoil the correspondance between \met and
101			$P_T(\ell\ell)$:
102			\begin{itemize}
103			\item $Ws$ in top events are polarized. Neutrinos are emitted preferentially
104			forward in the $W$ rest frame, thus the $P_T(\nu\nu)$ distribution is harder
105			than the $P_T(\ell\ell)$ distribution for top dilepton events.
106			\item The lepton selections results in $P_T$ and $\eta$ cuts on the individual
107			leptons that have no simple correspondance to the neutrino requirements.
108			\item Similarly, the \met$>$50 GeV cut introduces an asymmetry between leptons and
109			neutrinos which is only partially compensated by the $K$ factor above.
110			\item The \met resolution is much worse than the dilepton $P_T$ resolution.
111			When convoluted with a falling spectrum in the tails of \met, this result
112			in a harder spectrum for \met than the original $P_T(\nu\nu)$.
113			\item The \met response in CMS is not exactly 1. This causes a distortion
114			in the \met distribution that is not present in the $P_T(\ell\ell)$ distribution.
115			\item The $t\bar{t} \to$ dilepton signal includes contributions from
116			$W \to \tau \to \ell$. For these events the arguments about the equivalence
117			of $P_T(\ell\ell)$ and $P_T(\nu\nu)$ do not apply.
118			\item A dilepton selection will include SM events from non $t\bar{t}$
119			sources. These events can affect the background prediction. Particularly
120			dangerous are high $P_T$ Drell Yan events that barely pass the \met$>$ 50
121			GeV selection. They will tend to push the data-driven background prediction up.
122			\end{itemize}
123
124			We have studied these effects in SM Monte Carlo, using a mixture of generator and
125			reconstruction level studies, putting the various effects in one at a time.
126			For each configuration, we apply the data-driven method and report as figure
127			of merit the ratio of observed and predicted events in the signal region.
128			The results are summarized in Table~\ref{tab:victorybad}.
129
130			\begin{table}[htb]
131			\begin{center}
132			\caption{\label{tab:victorybad} Test of the data driven method in Monte Carlo
133			under different assumptions. See text for details.}
134			\begin{tabular}{\|l\|c\|c\|c\|c\|c\|c\|c\|}
135			\hline
136			& True $t\bar{t}$ dilepton & $t\to W\to\tau$& other SM & GEN or & Lepton $P_T$ & \met $>$ 50& obs/pred \\
137			& included & included & included & RECOSIM & and $\eta$ cuts & & \\ \hline
138			1&Y & N & N & GEN & N & N & \\
139			2&Y & N & N & GEN & Y & N & \\
140			3&Y & N & N & GEN & Y & Y & \\
141			4&Y & N & N & RECOSIM & Y & Y & \\
142			5&Y & Y & N & RECOSIM & Y & Y & \\
143			6&Y & Y & Y & RECOSIM & Y & Y & \\
144			\hline
145			\end{tabular}
146			\end{center}
147			\end{table}
148
149
150			The largest discrepancy between prediction and observation occurs on the first
151			line of Table~\ref{tab:victorybad}, {\em i.e.}, at the generator level with no
152			cuts. We have verified that this effect is due to the polarization of
153			the $W$ (we remove the polarization by reweighting the events and we get
154			good agreement between prediction and observation). The kinematical
155			requirements (lines 2 and 3) do not have a significant additional effect.
156			Going from GEN to RECOSIM there is a significant change in observed/predicted.
157			We have tracked this down to the fact that tcMET underestimates the true \met
158			by $\approx 4\%$\footnote{We find that observed/predicted changes by roughly 0.1
159			for each 1.5\% change in \met response.}. Finally, contamination from non $t\bar{t}$
160			events can have a significant impact on the BG prediction. The changes between
161			lines 5 and 6 of Table~\ref{tab:victorybad} is driven by only {\color{red} 3}
162			Drell Yan events that pass the \met selection.
163
164			An additional source of concern is that the CMS Madgraph $t\bar{t}$ MC does
165			not include effects of spin correlations between the two top quarks.
166			We have studied this effect at the generator level using Alpgen. We find
167			that the bias is a the few percent level.
168
169			Based on the results of Table~\ref{tab:victorybad}, we conclude that the
170			naive data driven background estimate based on $P_T{\ell\ell)}$ needs to
171			be corrected by a factor of {\color{red} $1.4 \pm 0.3$ (We need to
172			decide what this number should be)}. The quoted
173			uncertainty is based on the stability of the Monte Carlo tests under
174			variations of event selections, choices of \met algorithm, etc.
175
176
177			\subsection{Signal Contamination}
178			\label{sec:sigcont}
179
180			All data-driven methods are principle subject to signal contaminations
181			in the control regions, and the methods described in
182			Sections~\ref{sec:abcd} and~\ref{sec:victory} are not exceptions.
183			Signal contamination tends to dilute the significance of a signal
184			present in the data by inflating the background prediction.
185
186			It is hard to quantify how important these effects are because we
187			do not know what signal may be hiding in the data. Having two
188			independent methods (in addition to Monte Carlo ``dead-reckoning'')
189			adds redundancy because signal contamination can have different effects
190			in the different control regions for the two methods.
191			For example, in the extreme case of a
192			new physics signal
193			with $P_T(\ell \ell) = \met$, an excess of ev ents would be seen
194			in the ABCD method but not in the $P_T(\ell \ell)$ method.
195
196			The LM points are benchmarks for SUSY analyses at CMS. The effects
197			of signal contaminations for a couple such points are summarized
198			in Table~\ref{tab:sigcontABCD} and~\ref{tab:sigcontPT}.
199			Signal contamination is definitely an important
200			effect for these two LM points, but it does not totally hide the
201			presence of the signal.
202	claudioc	1.1
203
204	claudioc	1.2	\begin{table}[htb]
205			\begin{center}
206			\caption{\label{tab:sigcontABCD} Effects of signal contamination
207			for the background predictions of the ABCD method including LM0 or
208			LM1. Results
209			are normalized to 30 pb$^{-1}$.}
210			\begin{tabular}{\|c\|\|c\|c\|\|c\|c\|}
211			\hline
212			SM & LM0 & BG Prediction & LM1 & BG Prediction \\
213			Background & Contribution& Including LM0 & Contribution & Including LM1 \\ \hline
214			x & x & x & x & x \\
215			\hline
216			\end{tabular}
217			\end{center}
218			\end{table}
219
220			\begin{table}[htb]
221			\begin{center}
222			\caption{\label{tab:sigcontPT} Effects of signal contamination
223			for the background predictions of the $P_T(\ell\ell)$ method including LM0 or
224			LM1. Results
225			are normalized to 30 pb$^{-1}$.}
226			\begin{tabular}{\|c\|\|c\|c\|\|c\|c\|}
227			\hline
228			SM & LM0 & BG Prediction & LM1 & BG Prediction \\
229			Background & Contribution& Including LM0 & Contribution & Including LM1 \\ \hline
230			x & x & x & x & x \\
231			\hline
232			\end{tabular}
233			\end{center}
234			\end{table}
235	claudioc	1.1