COMP/CSA06DOC/datamanagement.tex

\section{Data Management}
\subsection{Dataset Bookkeeping System}
The Dataset Bookkeeping System (DBS) for CSA06 included the functionality needed for cataloging Monte Carlo data and tracking some of the processing history. Included were data-related concepts of Dataset, File, File Block and Data Tier. The processing related concepts of Application and Application Configuration were provided to track the actual operations that were performed to produce the data. In addition, data parentage relationships were provided. A client level API enabled the creation of each of the entities described above. File information  including size, number of events, status and Logical File Name (LFN) are included as attributes of each file. A discovery service was developed that enabled users to find data of interest for further processing and analysis. 

The architecture of the DBS service included a middle tier server running a CG script under Apache. All client access to the server was through an HTTP API. The CG script was written in PERL and access the database via the PERL DBI module. All activity for CSA06 was established on the CMSR production Oracle database server, with one Global DBS account and a half dozen so-called Local DBS accounts. The procedure was to produce Monte Carlo data under the control of four Prod Agents, each with access to its own Local DBS instance. When the data was appropriately merged and validated its catalog entries were migrated to the global catalog for use by CMOS at large. This migration task used allowed block-by-block transfer of Datasets to be done through a simple API. 

 There were two servers provided for CSA06, a ``test'' and ``production'' machine. The production servers were both dual Pentium  2.8 Hz processor with  2GB memory. The test server was heavily used by remote sites, including the initial data production, CMOS Robot submissions, and final skimming operations. The production server was used by the Tier-0 reconstruction farm,and some skimming operations near the end of CSA06. There were also ongoing CMOS activities included in the loads for the test server that are not related to CSA06. Access statistics for the service were obtained by mining the Apache access log files for each of the servers. The activity for the production server is shown during the month of November in Fig.~\ref{fig:dbs-prod-stats-chart} and Fig.~\ref{fig:dbs-prod-stats-table}. The important features to observe in this data are the number of pages served (Pages), and the total amount of data transfered (Bandwidth).As an example of a particularly busy day, November 27 showed 220k pages (query requests) and over 10 GB of data. This is a request rate of over 2.5 Hz and the server CPU was around 50% loaded.  Demand on the Test server was heavy in October and the first part of November with peak rates of  around 3 Hz in Mid October and again in early November. 
\begin{figure}[hbtp]
  \begin{center}
    \resizebox{15cm}{!}{\includegraphics{figs/dbs-prod-server-stats-nov-chart}}
    \caption{DBS production server November daily statistics. The bars on the chart for each day represent the number/amount of ``Visits'', ``Pages'', ``Hits'', and ``Bandwidth'' served, relative to the maximum day.  The scale and legend for the bar chart can be determined from the data in the table in Fig.~\ref{fig:dbs-prod-stats-table}.}
    \label{fig:dbs-prod-stats-chart}
  \end{center}
\end{figure} 

\begin{figure}[hbtp]
  \begin{center}
    \resizebox{18cm}{!}{\includegraphics{figs/dbs-prod-server-stats-nov-table}}
    \caption{DBS production server November daily statistics. }
    \label{fig:dbs-prod-stats-table}
  \end{center}
\end{figure} 
 
The overall operation and performance of the system was very good throughout the course of the CSA06 exercise. The limited functionality provided by the schema and API was sufficient for the test although many additional features are needed for the ultimate system. Clients occasionally reported slow response during peak periods, but the servers held up well. The service can be easily scaled by adding additional machines and a load balancing mechanism, such as round robin DNS, and this will be examined for the final system. The loads of the CSA06 operation were artificially inflated because many Local DBS instances were being managed centrally at CERN, in addition to the Global instance. In the final system the local instances will not be operated at CERN.  There were two incidents which resulted in service interruption, both caused by problems with the central CMSR database system. Ultimately, the DBS  service rates needed will be reduced by the fact that only Global-instance traffic will go through the central CERN service. 
\subsection{Data Location Services}
\subsection{Data File Catalogs} 
\subsection{Data Transfer Mechanism}
\subsection{Data Access}
Revision:	1.3
Committed:	Thu Nov 30 10:50:42 2006 UTC (18 years, 5 months ago) by malgeri
Content type:	application/x-tex
Branch:	MAIN
CVS Tags:	pdflatex_v4, pdflatex_v3, pdflatex_v2, pdflatex
Changes since 1.2:	+2 -2 lines
Log Message:	added calibration part and updated for pdflatex
#	User	Rev	Content
1	fisk	1.1	\section{Data Management}
2	lueking	1.2	\subsection{Dataset Bookkeeping System}
3			The Dataset Bookkeeping System (DBS) for CSA06 included the functionality needed for cataloging Monte Carlo data and tracking some of the processing history. Included were data-related concepts of Dataset, File, File Block and Data Tier. The processing related concepts of Application and Application Configuration were provided to track the actual operations that were performed to produce the data. In addition, data parentage relationships were provided. A client level API enabled the creation of each of the entities described above. File information including size, number of events, status and Logical File Name (LFN) are included as attributes of each file. A discovery service was developed that enabled users to find data of interest for further processing and analysis.
4
5			The architecture of the DBS service included a middle tier server running a CG script under Apache. All client access to the server was through an HTTP API. The CG script was written in PERL and access the database via the PERL DBI module. All activity for CSA06 was established on the CMSR production Oracle database server, with one Global DBS account and a half dozen so-called Local DBS accounts. The procedure was to produce Monte Carlo data under the control of four Prod Agents, each with access to its own Local DBS instance. When the data was appropriately merged and validated its catalog entries were migrated to the global catalog for use by CMOS at large. This migration task used allowed block-by-block transfer of Datasets to be done through a simple API.
6
7			There were two servers provided for CSA06, a ``test'' and ``production'' machine. The production servers were both dual Pentium 2.8 Hz processor with 2GB memory. The test server was heavily used by remote sites, including the initial data production, CMOS Robot submissions, and final skimming operations. The production server was used by the Tier-0 reconstruction farm,and some skimming operations near the end of CSA06. There were also ongoing CMOS activities included in the loads for the test server that are not related to CSA06. Access statistics for the service were obtained by mining the Apache access log files for each of the servers. The activity for the production server is shown during the month of November in Fig.~\ref{fig:dbs-prod-stats-chart} and Fig.~\ref{fig:dbs-prod-stats-table}. The important features to observe in this data are the number of pages served (Pages), and the total amount of data transfered (Bandwidth).As an example of a particularly busy day, November 27 showed 220k pages (query requests) and over 10 GB of data. This is a request rate of over 2.5 Hz and the server CPU was around 50% loaded. Demand on the Test server was heavy in October and the first part of November with peak rates of around 3 Hz in Mid October and again in early November.
8			\begin{figure}[hbtp]
9			\begin{center}
10	malgeri	1.3	\resizebox{15cm}{!}{\includegraphics{figs/dbs-prod-server-stats-nov-chart}}
11	lueking	1.2	\caption{DBS production server November daily statistics. The bars on the chart for each day represent the number/amount of ``Visits'', ``Pages'', ``Hits'', and ``Bandwidth'' served, relative to the maximum day. The scale and legend for the bar chart can be determined from the data in the table in Fig.~\ref{fig:dbs-prod-stats-table}.}
12			\label{fig:dbs-prod-stats-chart}
13			\end{center}
14			\end{figure}
15
16			\begin{figure}[hbtp]
17			\begin{center}
18	malgeri	1.3	\resizebox{18cm}{!}{\includegraphics{figs/dbs-prod-server-stats-nov-table}}
19	lueking	1.2	\caption{DBS production server November daily statistics. }
20			\label{fig:dbs-prod-stats-table}
21			\end{center}
22			\end{figure}
23
24			The overall operation and performance of the system was very good throughout the course of the CSA06 exercise. The limited functionality provided by the schema and API was sufficient for the test although many additional features are needed for the ultimate system. Clients occasionally reported slow response during peak periods, but the servers held up well. The service can be easily scaled by adding additional machines and a load balancing mechanism, such as round robin DNS, and this will be examined for the final system. The loads of the CSA06 operation were artificially inflated because many Local DBS instances were being managed centrally at CERN, in addition to the Global instance. In the final system the local instances will not be operated at CERN. There were two incidents which resulted in service interruption, both caused by problems with the central CMSR database system. Ultimately, the DBS service rates needed will be reduced by the fact that only Global-instance traffic will go through the central CERN service.
25			\subsection{Data Location Services}
26			\subsection{Data File Catalogs}
27			\subsection{Data Transfer Mechanism}
28			\subsection{Data Access}