1 |
sashby |
1.1 |
%%____________________________________________________________________
|
2 |
|
|
%% File: BuildSystem.tex
|
3 |
|
|
%%____________________________________________________________________
|
4 |
|
|
%%
|
5 |
|
|
%% Author: Shaun ASHBY <Shaun.Ashby@cern.ch>
|
6 |
|
|
%% Update: 2005-11-02 17:07:01+0100
|
7 |
sashby |
1.4 |
%% Revision: $Id: BuildSystem.tex,v 1.3 2005/11/09 12:46:11 sashby Exp $
|
8 |
sashby |
1.1 |
%%
|
9 |
|
|
%% Copyright: 2005 (C) Shaun ASHBY
|
10 |
|
|
%%
|
11 |
|
|
%%--------------------------------------------------------------------
|
12 |
|
|
\chapter{The SCRAM Build System}\label{ch:buildsystem}
|
13 |
|
|
|
14 |
|
|
The primary function of the \scram\ build system is to allow efficient,
|
15 |
|
|
ordered compilation of source code of different types (\eg \texttt{C},
|
16 |
|
|
\texttt{C++}, \texttt{FORTRAN}) into libraries, binary executables or
|
17 |
|
|
other build products and to make it possible to easily extend the
|
18 |
|
|
supported product types, as dictated by changing project requirements,
|
19 |
|
|
and manage external and internal dependencies between software units
|
20 |
|
|
in a transparent and uniform way.
|
21 |
|
|
|
22 |
|
|
\ni In previous versions of \scram, much of the build system functionality
|
23 |
|
|
was hard-coded in parts of the \scram\ source tree making it difficult
|
24 |
|
|
to make changes to adapt to changing software development patterns.
|
25 |
|
|
Any such change often meant a new release of \scram\ and the
|
26 |
|
|
subsequent delays entailed by a release sequence. In addition, the
|
27 |
|
|
actual algorithm for determining what build actions should be
|
28 |
|
|
activated for each software unit was not optimum with many
|
29 |
|
|
configuration documents needlessly being parsed multiple times,
|
30 |
|
|
greatly adding to project build times. This fundamental problem
|
31 |
|
|
stemmed from the fact there was never a clear separation between
|
32 |
|
|
parsing of configuration documents and the creation of a
|
33 |
|
|
\texttt{Makefile}- the two were run on the fly. Supporting the
|
34 |
|
|
building a project for a new architecture, particularly where the
|
35 |
|
|
syntax of the \texttt{Makefile} is very different (\eg
|
36 |
|
|
\texttt{WIN32}), was also practically impossible within the old
|
37 |
|
|
design.
|
38 |
|
|
|
39 |
|
|
\ni In \scram~\scramvx, a new approach is taken by separating the
|
40 |
|
|
collection of build metadata from the generation of the
|
41 |
|
|
\texttt{Makefile} and compilation of the software units. Metadata in
|
42 |
|
|
this context refers to all information needed to build all products.
|
43 |
|
|
This includes package-level dependency requirements and all
|
44 |
|
|
information needed to use external tools (\eg libraries to link
|
45 |
|
|
against, compilation flags, pre-processor macros, library directories,
|
46 |
|
|
header file locations and runtime requirements).
|
47 |
|
|
Having parsed all necessary project documents, and the metadata
|
48 |
|
|
obtained and processed as required, data elements are substituted
|
49 |
|
|
into text templates to create the project \texttt{Makefile}. The data
|
50 |
|
|
elements include directory names, library names and names of
|
51 |
|
|
build targets all of which must be generated by \scram.
|
52 |
|
|
Of course, the text templates can be written so that the output is in
|
53 |
|
|
some other format than that for a \texttt{Makefile}.
|
54 |
|
|
|
55 |
|
|
\section{BuildFiles and Build System Control}
|
56 |
|
|
|
57 |
|
|
The \buildfile\ is the configuration document for the build system. For
|
58 |
|
|
each software unit, a \buildfile\ defines the external dependencies, the
|
59 |
|
|
internal dependencies (\ie the other software units that are used by
|
60 |
|
|
this one) and the interface. This interface supplies the product
|
61 |
|
|
provided by this software unit and the external/internal dependencies
|
62 |
|
|
required by it to any other software unit that requires it.
|
63 |
|
|
A software unit can be thought of as a product entity which is self-contained
|
64 |
|
|
and can be used either in linking to satisfy the dependencies of
|
65 |
|
|
a build product, or can be used to participate in some way on the
|
66 |
|
|
build process (for example, an executable used to process a source
|
67 |
|
|
file template to generate soucre files in a particular package).
|
68 |
|
|
Most of the time, a software unit is synonymous with a package (and
|
69 |
|
|
hence a shared library).
|
70 |
|
|
|
71 |
|
|
\subsection{The Markup Syntax}
|
72 |
|
|
\index{\buildfile}\index{SCRAM!build files}
|
73 |
|
|
|
74 |
|
|
Differing from other configuration documents in which the document
|
75 |
|
|
class defines the parse type, \scram\ knows about build files only
|
76 |
|
|
from their name, which must be \buildfile\footnote{\textit{note the uppercase letters!}.}.
|
77 |
|
|
A \buildfile\ only has a significance in a project area. The
|
78 |
|
|
valid tags that can be used in a \buildfile\ are listed below.
|
79 |
|
|
\index{\buildfile!valid basic markup tags}
|
80 |
|
|
|
81 |
|
|
\begin{description}
|
82 |
|
|
|
83 |
|
|
\item[\lbkt\texttt{use} name=\textit{unit}\rbkt]\mbox{}\\
|
84 |
|
|
Specify that there is a dependency on an external tool or a
|
85 |
|
|
local/external package \texttt{unit} (\eg a library, or everything the package
|
86 |
|
|
exports). External tools and packages are treated in the same way.
|
87 |
|
|
The interface to \textit{unit} must be defined via the
|
88 |
|
|
\lbkt\texttt{export}\rbkt\tagend{export} tag.
|
89 |
|
|
|
90 |
|
|
\item[\lbkt\texttt{export}\rbkt\tagend{export}]\mbox{}\\
|
91 |
|
|
A software unit can be declared and exported to be used inside the
|
92 |
|
|
project or externally by another \scram\ project using the current
|
93 |
|
|
project as an external tool. The interface is defined by listing
|
94 |
|
|
the libraries (this can include those corresponding to a subset of a
|
95 |
|
|
subsystem) that are to be exported when the package (or subsystem)
|
96 |
|
|
is used. Typically, a package will export its' library, all
|
97 |
|
|
dependencies and specific compiler flags.
|
98 |
|
|
|
99 |
|
|
\item[\lbkt\texttt{lib} name=\textit{libname} {[}\texttt{position}=\textit{first}{]}\rbkt]\mbox{}\\
|
100 |
|
|
Add a library to the list of libraries passed to the linker. The
|
101 |
|
|
optional \texttt{position} argument will move the library to the
|
102 |
|
|
front of the list of libraries as received by the linker (this
|
103 |
|
|
bypasses the automatic library ordering as determined by depth-first
|
104 |
|
|
sorting of the package metadata).
|
105 |
|
|
|
106 |
|
|
\item[\lbkt\texttt{group} name=\textit{groupname}\rbkt]\mbox{}\\
|
107 |
|
|
Specify that all dependencies defined within the group
|
108 |
|
|
\textit{groupname} should be used by the current software unit.
|
109 |
|
|
|
110 |
|
|
\item[\lbkt\texttt{include\_path} path=\textit{path}\rbkt]\mbox{}\\
|
111 |
|
|
Add the path \textit{path} to the global \texttt{INCLUDE} path
|
112 |
|
|
passed to the compiler. The include paths of individual
|
113 |
|
|
tools/packages will be ordered according to their dependency order
|
114 |
|
|
when added to the project or package \texttt{INCLUDE}.
|
115 |
|
|
|
116 |
|
|
\item[\lbkt\texttt{libtype} type=\textit{type}\rbkt]\mbox{}\\
|
117 |
|
|
Specify the type of library that should be built.
|
118 |
|
|
This option overrides the project defaults but is currently unused
|
119 |
|
|
since libraries are \texttt{shared} only (\textit{support for debug libs to follow....}).
|
120 |
|
|
|
121 |
|
|
\item[\lbkt\texttt{flags} \texttt{NAME}='definition'\rbkt]\mbox{}\\
|
122 |
|
|
Extra compiler flags can be added either in a tool description or in
|
123 |
|
|
a package. These flags will be added to the global compiler flags
|
124 |
|
|
and propageted to the build product defined in the \buildfile\ where
|
125 |
|
|
the declaration was made.
|
126 |
|
|
\end{description}
|
127 |
|
|
|
128 |
|
|
\ni There are also tags for the supported build products:
|
129 |
|
|
\index{\buildfile!valid product markup tags}
|
130 |
|
|
\begin{description}
|
131 |
|
|
|
132 |
|
|
\item[\lbkt\texttt{bin} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{bin}]\mbox{}\\
|
133 |
|
|
Specify an executable to build. The name of the executable can be
|
134 |
|
|
changed using the optional \textit{name} argument, otherwise the
|
135 |
|
|
name will be the same as \textit{filename}, less the file ending
|
136 |
|
|
(typically `.cpp'). More than one file can be compiled and linked to
|
137 |
|
|
make the executable. The file list can be specified like "main.cpp,
|
138 |
|
|
*.cc", "*.cpp" or "main.cpp, file.cc" (\ie file globs are
|
139 |
|
|
supported in a minimal way).
|
140 |
|
|
|
141 |
|
|
All dependencies are contained between the opening and closing tags:
|
142 |
|
|
these dependencies are specific \textit{only} to this binary
|
143 |
|
|
executable. Dependencies or other metadata listed outside the
|
144 |
|
|
individual product tag will be passed to all products defined in the
|
145 |
|
|
\buildfile.
|
146 |
|
|
|
147 |
|
|
\item[\lbkt\texttt{module} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{module}]\mbox{}\\
|
148 |
|
|
Specify a plug-in module to build. The name of the module can be
|
149 |
|
|
changed using the optional \textit{name} argument. How the plug-in
|
150 |
|
|
is defined can be customised within the project. By default (\ie
|
151 |
|
|
when using initial templates provided with \scram, copied to a
|
152 |
|
|
project \texttt{config} directory), a plug-in module is just a shared
|
153 |
|
|
library with all dependencies fully resolved when it is loaded (in
|
154 |
|
|
fact, there is not much difference between a plug-in module and a
|
155 |
|
|
shared library built by default in package \texttt{src} directories).
|
156 |
|
|
|
157 |
|
|
\item[\lbkt\texttt{library} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{library}]\mbox{}\\
|
158 |
|
|
Define an additional library to build (in addition to the library
|
159 |
|
|
built automatically from the contents of \texttt{src} in the parent
|
160 |
|
|
package directory). This is usually used to build libraries for unit tests.
|
161 |
|
|
|
162 |
|
|
\item[\lbkt\texttt{application} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{application}]\mbox{}\\
|
163 |
|
|
Define an application. How one defines an application is
|
164 |
|
|
project-specific: it could be that an application differs from a
|
165 |
|
|
binary executable in only the compiler flags or the storage
|
166 |
|
|
location when released. This product type could also be used to
|
167 |
|
|
inject custom rules into the build system.
|
168 |
|
|
|
169 |
|
|
% Not yet advertised: <<FIXME
|
170 |
|
|
%\item[\lbkt\texttt{skip}\rbkt\tagend{skip}]\mbox{}\\
|
171 |
|
|
% Indicate that a directory should be skipped. Comments can be entered
|
172 |
|
|
% between the tags which will be printed to STDOUT during building.
|
173 |
|
|
% \textit{Work in progress!!}.
|
174 |
|
|
%
|
175 |
|
|
\end{description}
|
176 |
|
|
|
177 |
|
|
|
178 |
|
|
|
179 |
|
|
\subsection{The Project BuildFile}\label{sec:projectbuildfile}
|
180 |
|
|
\index{\buildfile!project}
|
181 |
|
|
|
182 |
|
|
Global behaviour is controlled by the project \buildfile\ which is
|
183 |
|
|
located in the \texttt{config} directory in all \scram-managed
|
184 |
|
|
projects. This file is parsed first, before all others. The
|
185 |
|
|
storage areas for build products are defined in this \buildfile. More
|
186 |
|
|
importantly, instructions on what actions to perform throughout the
|
187 |
|
|
source code tree occur here using \texttt{ClassPath} directives.
|
188 |
|
|
The directives indicate which templates to apply at each directory
|
189 |
|
|
location based on matching the directory structure to the class path.
|
190 |
|
|
There are three keywords which refer to certain levels in a directory
|
191 |
|
|
tree and the definitions of these are fixed. The keywords are
|
192 |
|
|
\texttt{Project}, which refers to the top-level source directory
|
193 |
|
|
(usually \texttt{src}), \texttt{SubSystem}, which refers to the
|
194 |
|
|
directory one level up, and \texttt{Package}, which refers to a
|
195 |
|
|
subdirectory of a \texttt{SubSystem}. A project is not bound to a
|
196 |
|
|
structure in which there are always subsystems: in fact, it is
|
197 |
|
|
possible to have only a package level under the project \texttt{src}
|
198 |
|
|
directory. The advantage to having subsystems is that it becomes
|
199 |
|
|
possible to group packages together according to some common task (for
|
200 |
|
|
example): groups can then be defined which allow several independent
|
201 |
|
|
packages to become a single dependency unit which can be used by other
|
202 |
|
|
packages or executables at link-time.
|
203 |
|
|
|
204 |
|
|
\ni Each directory which is an exact match to a keyword location will be
|
205 |
|
|
flagged as such and \textit{structure templates}\index{structure templates}
|
206 |
|
|
will be used to create the rules for compiling in these locations.
|
207 |
|
|
Of course, it is possible to redefine what actions to apply at the
|
208 |
|
|
subsystem, package or even project level, just by supplying a different
|
209 |
|
|
template or overriding an action in the project class path.
|
210 |
|
|
For libraries and executables (in general, build products), which are
|
211 |
|
|
associated to a subdirectory inside a package, \textit{product
|
212 |
|
|
templates} define the build targets.
|
213 |
|
|
Generated build rules for directories that do not fully match any
|
214 |
|
|
\texttt{ClassPath} will simply print a message that no action is required
|
215 |
|
|
at that location (\eg for an \texttt{include} or \texttt{interface} directory).
|
216 |
|
|
Complex build operations can be activated by modifying the
|
217 |
|
|
structure templates to do specific things for specific directories.
|
218 |
|
|
This could be for a \texttt{Documentation} subsystem, for example,
|
219 |
|
|
where the source code happens to be \texttt{html} or \LaTeX\
|
220 |
|
|
sources, or for single packages where source code must be generated
|
221 |
|
|
first before normal package build actions.
|
222 |
|
|
|
223 |
|
|
\ni The syntax of the \texttt{ClassPath} tag is \index{\texttt{ClassPath}
|
224 |
|
|
tag}%
|
225 |
|
|
\begin{tagprint}
|
226 |
|
|
\lbkt\texttt{ClassPath}
|
227 |
|
|
path={[}\textit{pattern\_match}{]}+\textit{template\_type}/\ldots/\rbkt
|
228 |
|
|
\end{tagprint}
|
229 |
|
|
|
230 |
|
|
\ni In the above case, the \texttt{template\_type} could be
|
231 |
|
|
\texttt{library}, \texttt{binary} \texttt{module} \etc. (the build
|
232 |
|
|
products). The template file name corresponding to the
|
233 |
|
|
\texttt{library} template type would be
|
234 |
|
|
\texttt{library\_template.tmpl} and this would be located in the
|
235 |
|
|
\texttt{config} directory of the project. Note that where there are
|
236 |
|
|
multiple \lbkt\texttt{ClassPath}\rbkt tags defined, it is always the
|
237 |
|
|
last tag that matches the current location that will be used.
|
238 |
|
|
See examples in Chapter~\ref{ch:examples}.
|
239 |
|
|
|
240 |
|
|
\subsubsection{Product Storage Locations}
|
241 |
|
|
\index{\texttt{ProductStore} tag}%
|
242 |
|
|
|
243 |
|
|
The storage locations are defined using
|
244 |
|
|
\begin{tagprint}
|
245 |
|
|
\lbkt\texttt{ProductStore} name=\textit{name} {[}type=\textit{type}
|
246 |
sashby |
1.4 |
swap=\textit{t}{]} {[}path=\textit{path}{]}\rbkt
|
247 |
sashby |
1.1 |
\end{tagprint}
|
248 |
|
|
|
249 |
|
|
\ni The name \textit{name} is the name of the directory to be created
|
250 |
|
|
and the option \textit{type} can be set to \texttt{arch} so that an
|
251 |
|
|
architecture-dependent subdirectory for platform-specific products
|
252 |
|
|
will be added in the project area with the product directory
|
253 |
|
|
\textit{name} underneath. Without this type option the directory
|
254 |
|
|
\textit{name} will be created in the project area. Using the
|
255 |
|
|
\textit{swap} option reverses the order of the architecture-dependent
|
256 |
|
|
sub-directory and the name of the storage directory. That is, setting
|
257 |
|
|
swap to \textit{true} or \textit{t} will create the product directory
|
258 |
|
|
first with an architecture-dependent sub-directory
|
259 |
|
|
underneath.\footnote{This is the fixed behaviour for all SCRAM
|
260 |
|
|
releases prior to \scramvx.}
|
261 |
|
|
|
262 |
sashby |
1.4 |
\ni The path \texttt{path} can be specified as the location where all
|
263 |
|
|
products corresponding to \textit{name} will be installed: a symbolic
|
264 |
|
|
link will be created in the project area which points to this
|
265 |
|
|
directory.
|
266 |
|
|
|
267 |
|
|
\ni When \scram\ finds a \tagstart{ProductStore} tag,
|
268 |
|
|
a \texttt{Makefile} variable is set which can be used anywhere in a project makefile
|
269 |
sashby |
1.1 |
when writing custom rules:
|
270 |
|
|
\begin{description}
|
271 |
|
|
\item[\texttt{SCRAMSTORENAME\_name}]\mbox{}\\
|
272 |
|
|
The path to the storage location \texttt{name} from the project area
|
273 |
|
|
directory (\eg \texttt{lib} or \texttt{lib/slc3\_ia32\_gcc323}).
|
274 |
|
|
\end{description}
|
275 |
|
|
|
276 |
|
|
\ni This could be used, for example, where there may be a default rule for
|
277 |
|
|
copying all \texttt{include} files to a single location after building
|
278 |
|
|
all libraries. By default, all build products are copied to their
|
279 |
|
|
storage areas once built.
|
280 |
|
|
|
281 |
|
|
\subsection{Local Metadata}\index{local metadata}\index{config/self}
|
282 |
|
|
|
283 |
|
|
In order to propagate metadata from the local area to the source tree
|
284 |
|
|
at compile time, the local settings like \texttt{INCLUDE} or library paths in the
|
285 |
|
|
project templates plus the runtime environment, are defined in a local
|
286 |
|
|
file called \texttt{Self} found in the \texttt{config} directory\footnote{Note that this is unlike \scram\ V0 where
|
287 |
|
|
such settings were hard-coded in the top-level \buildfile.}.
|
288 |
|
|
This file behaves like a tool description file and is set up when the project
|
289 |
|
|
area is bootstrapped: all of the information defined within it is
|
290 |
|
|
automatically propagated to the whole tree.
|
291 |
|
|
In addition, all \texttt{scram} commands used to query tools
|
292 |
|
|
can also be used to inspect the local settings and changes can be made
|
293 |
|
|
at will with the tool set up by hand to propagate new settings.
|
294 |
|
|
|
295 |
|
|
\subsection{Defining Groups}\label{sec:defininggroups}
|
296 |
|
|
\index{software units!defining a group}
|
297 |
|
|
A group can be used by adding a a \lbkt\texttt{group}\dots\rbkt statement
|
298 |
|
|
giving the name of the group to be included.
|
299 |
|
|
|
300 |
|
|
\ni A group can be defined like this
|
301 |
|
|
|
302 |
|
|
\small{
|
303 |
|
|
\begin{verbatim}
|
304 |
|
|
<define_group name=GA>
|
305 |
|
|
<use name=D>
|
306 |
|
|
<use name=zlib>
|
307 |
|
|
<Group name=XY>
|
308 |
|
|
<Flags CPPFLAGS="-DGROUP_GA">
|
309 |
|
|
</define_group>
|
310 |
|
|
\end{verbatim}
|
311 |
|
|
}\normalsize
|
312 |
|
|
|
313 |
|
|
\ni in a subsystem \buildfile.
|
314 |
|
|
|
315 |
|
|
\ni Note that it is not necessary to declare where the group can be
|
316 |
|
|
found (for example in another \scram\ project included as an
|
317 |
|
|
external product in the local configuration environment) since \scram\
|
318 |
|
|
determines this automatically. Duplicated/overridden groups will raise
|
319 |
|
|
a warning when \scram\ parses the \buildfile.
|
320 |
|
|
|
321 |
|
|
\section{Configuring a New Package}
|
322 |
|
|
\label{sec:exportingsoftwareunits}
|
323 |
|
|
\index{software units}
|
324 |
|
|
\index{software units!defining the interface to}
|
325 |
|
|
|
326 |
|
|
When a new package is added to a project, it is important that the
|
327 |
|
|
directory contains a \buildfile\ (note that it must be the package
|
328 |
|
|
directory and \textit{not} the \texttt{src} directory where the
|
329 |
|
|
sources are located that contains the \buildfile).
|
330 |
|
|
This \buildfile\ should have the following components:
|
331 |
|
|
\begin{description}
|
332 |
|
|
\item[\textbf{Declarations for all compile-time/link-time dependencies}]\mbox{}\\
|
333 |
|
|
The dependencies should be deduced from the \texttt{include}
|
334 |
|
|
statements in the package sources and a \lbkt\texttt{use}
|
335 |
|
|
name=\textit{unit}\rbkt should be added for each required unit
|
336 |
|
|
(external/internal package or external software product).
|
337 |
|
|
\item[\textbf{Export of the dependencies and package product}]\mbox{}\\
|
338 |
|
|
Every package providing a shared library should permit client software
|
339 |
|
|
units to use it at compile or link time. The
|
340 |
|
|
\lbkt\texttt{export}\rbkt\tagend{export} tag is used to define the
|
341 |
|
|
interface to the package and contains the full list of
|
342 |
|
|
\lbkt\texttt{use} name=\textit{unit}\rbkt statements (as required
|
343 |
|
|
by the package) and a \lbkt\texttt{lib} name=\textit{PackageName}\rbkt
|
344 |
|
|
statement with the name of the library provided by the package.
|
345 |
|
|
Of course, the library name should be omitted if the package does
|
346 |
|
|
not provide a shared library, having no \texttt{src} directory or
|
347 |
|
|
source files. This could be the case for abstract packages providing
|
348 |
|
|
only template \texttt{include} files to derived packages.
|
349 |
|
|
\end{description}
|
350 |
|
|
|
351 |
|
|
\ni A typical package \buildfile\ will look like this:
|
352 |
|
|
\index{example of using \texttt{export} tag}%
|
353 |
|
|
\index{software units!exporting}%
|
354 |
|
|
\small{
|
355 |
|
|
\begin{verbatim}
|
356 |
|
|
<export>
|
357 |
|
|
<lib name=PX>
|
358 |
|
|
<use name=S/A>
|
359 |
|
|
<use name=B>
|
360 |
|
|
</export>
|
361 |
|
|
|
362 |
|
|
<use name=S/A>
|
363 |
|
|
<use name=B>
|
364 |
|
|
|
365 |
|
|
\end{verbatim}
|
366 |
|
|
}\normalsize
|
367 |
|
|
\index{defining a set of libraries for a package}
|
368 |
|
|
\ni Any other package or executable requiring this package would have
|
369 |
|
|
a statement \lbkt\texttt{use} name=\texttt{PX}\rbkt
|
370 |
|
|
in the \buildfile\ of the package/executable, where
|
371 |
|
|
\texttt{PX} is a path to the package in the
|
372 |
|
|
project (and hence could actually look like \lbkt\texttt{use}
|
373 |
|
|
name=\texttt{Subsystem/PX}\rbkt if the package is located in
|
374 |
|
|
a subsystem).
|
375 |
|
|
|
376 |
|
|
\ni External projects and external tools can be used in the same way
|
377 |
|
|
using the \texttt{use} statement, \ie it is not necessary to qualify
|
378 |
|
|
the statement with a project name as with the older \scram\ syntax.
|
379 |
|
|
|
380 |
|
|
\section{Build System Caches}
|
381 |
sashby |
1.3 |
\label{sec:bscaches}
|
382 |
|
|
\index{Build system!caches}
|
383 |
sashby |
1.1 |
|
384 |
|
|
Tracking changes to files in the source tree is achieved by caching
|
385 |
|
|
file timestamps and this is especially important for files that are
|
386 |
|
|
part of the project configuration. Caching is also extensively used
|
387 |
|
|
to store metadata as it is read from the \buildfile s
|
388 |
|
|
and tool description documents and to minimise subsequent re-parsing of
|
389 |
|
|
build data when the dependencies of a target change.
|
390 |
|
|
|
391 |
|
|
\subsection{File Timestamp Cache}
|
392 |
sashby |
1.3 |
\label{sec:bstscache}
|
393 |
|
|
\index{File Timestamp Cache}
|
394 |
sashby |
1.1 |
|
395 |
|
|
Build tools like \texttt{Make} keep track of relationships between
|
396 |
|
|
targets and the source files or headers needed to build those targets.
|
397 |
|
|
When a timestamp on a source file or header changes, \texttt{Make}
|
398 |
|
|
knows how to rebuild the target having determined that the target is
|
399 |
|
|
out of date with respect to those modified files. A limitation arises
|
400 |
|
|
when code is retrieved from a code repository such as one based on
|
401 |
|
|
\texttt{CVS}. Often, the timestamps on checked-out files are in the
|
402 |
|
|
past (usually the time of the commit) which implies that a target will
|
403 |
|
|
not be rebuilt if these files are added to the source code tree since
|
404 |
|
|
they will be older than the target.
|
405 |
|
|
|
406 |
|
|
\ni To cope with situations like this, \scram\ employs a cache mechanism
|
407 |
|
|
which stores the timestamps of the \buildfile s and the directories in
|
408 |
|
|
the source code tree. Note that it is not necessary to store the
|
409 |
|
|
timestamp of every source file since it is the change of timestamp of
|
410 |
|
|
the parent directory that will change when files are added or removed.
|
411 |
|
|
Once populated (when \scram\ first performs a build), the existing
|
412 |
|
|
cache will be used to determine when the status of a directory or a
|
413 |
|
|
file has changed. Such a change to the status can be found by using
|
414 |
|
|
the \texttt{stat} command and comparing information like access mode
|
415 |
|
|
in addition to the timestamp. Changes in timestamp provoke a rebuild
|
416 |
|
|
of the updated package. If a \buildfile\ changes then the package is
|
417 |
|
|
rebuilt automatically, otherwise \scram\ will trigger a rebuild by
|
418 |
|
|
modifying the timestamp of the \buildfile\ of the package where the
|
419 |
|
|
files were added or removed first.
|
420 |
|
|
|
421 |
|
|
\subsection{Metadata Cache}
|
422 |
sashby |
1.3 |
\label{sec:bsmdcache}
|
423 |
|
|
\index{Metadata Cache}
|
424 |
sashby |
1.1 |
|
425 |
|
|
The actions to be taken by the build system when compiling and the
|
426 |
|
|
relationships between project software units are described in
|
427 |
|
|
\buildfile s located in the package directories in the source
|
428 |
|
|
code tree and the project configuration directory. Since it is
|
429 |
|
|
important to discover when metadata has changed, the timestamps of all
|
430 |
|
|
templates and the tool cache are also monitored. Since a change in a
|
431 |
|
|
tool setting or a template could affect the whole project, these
|
432 |
|
|
changes force a reparse of all \buildfile s and a complete rebuild of
|
433 |
|
|
the project.
|
434 |
|
|
|
435 |
|
|
\section{Understanding the Build Templates}
|
436 |
sashby |
1.3 |
\label{sec:bstemplates}
|
437 |
|
|
\index{Build system!buildtemplates}
|
438 |
sashby |
1.1 |
|
439 |
|
|
Many open source and commercial software projects use \texttt{make} to manage rebuilds
|
440 |
|
|
of large programs or collections of programs. Each build action is
|
441 |
|
|
defined using a rule which describes the steps that should be executed
|
442 |
|
|
to build the program. The rules are written in a file called a
|
443 |
|
|
\texttt{Makefile}. In very large projects, a \texttt{Makefile} can typically be
|
444 |
|
|
thousands of lines long and is very difficult to write
|
445 |
|
|
from scratch and manage manually; it is preferable to generate it
|
446 |
|
|
automatically using a more abstract process. Since \scram\ projects can
|
447 |
|
|
contain many occurrences of the same class of build object (\eg shared
|
448 |
|
|
libraries, executables), a complete \texttt{Makefile} can be
|
449 |
|
|
generated from templates which implement the rules for these different classes.
|
450 |
|
|
Thus, \scram\ can generate a \texttt{Makefile} for a complex project structure easily.
|
451 |
|
|
|
452 |
sashby |
1.2 |
\subsection{What is a Template?}
|
453 |
sashby |
1.1 |
|
454 |
|
|
A template is a file which is written in the same way as a
|
455 |
|
|
\texttt{Makefile} except that some parts of the rules or environment
|
456 |
sashby |
1.4 |
variables are inserted at the time of generation.
|
457 |
|
|
|
458 |
|
|
%%\begin{center}
|
459 |
|
|
%% \textit{More to be added here over time. Check snapshots pages!}
|
460 |
|
|
%%\end{center}
|
461 |
sashby |
1.2 |
|
462 |
sashby |
1.1 |
% An Example Rule: <<FIXME
|
463 |
|
|
|
464 |
|
|
|
465 |
|
|
|
466 |
|
|
|
467 |
|
|
%% Build System
|
468 |
|
|
%% ------------
|
469 |
|
|
|
470 |
|
|
%% The SCRAM version 1.0 build system consists of two stages. Firstly,
|
471 |
|
|
%% all external requirements and build actions are determined and cached
|
472 |
|
|
%% for every location in the project. Secondly, the processed metadata
|
473 |
|
|
%% (lists of libraries used, INCLUDE/LIBDIR paths and compiler flags) are
|
474 |
|
|
%% used by the template engine to produce Makefile fragments used by
|
475 |
|
|
%% gmake in the usual way. Templates can easily be adapted to support
|
476 |
|
|
%% both different kinds of compilations (e.g. using java) or different
|
477 |
|
|
%% architectures.
|
478 |
|
|
|
479 |
|
|
%% The project data and the state of the project area are made persistent
|
480 |
|
|
%% so that, by checking the current state relative to the last cached
|
481 |
|
|
%% state, any new actions (e.g. a rebuild after a modification to a BuildFile)
|
482 |
|
|
%% can be affected very quickly without resorting to a traversal of the
|
483 |
|
|
%% entire directory tree.
|
484 |
|
|
|
485 |
|
|
|
486 |
|
|
%% - The Directory Cache
|
487 |
|
|
|
488 |
|
|
%% In an empty project area, the first step is to populate the directory
|
489 |
|
|
%% cache with the timestamp and content information for the project
|
490 |
|
|
%% directory tree. All timestamps and file modes are stored for parent
|
491 |
|
|
%% and subdirectories in the ``src'' tree. Parameters for important files
|
492 |
|
|
%% used in the build system, such as templates and other files in
|
493 |
|
|
%% ``config'' are also recorded. The timestamps of other directories in
|
494 |
|
|
%% the project area (tmp and build product stores for example), are not
|
495 |
|
|
%% stored.
|
496 |
|
|
%% Once the timestamp information is stored, the cache (simply a Perl
|
497 |
|
|
%% object of type ``Cache::Cache'') is converted to a data structure
|
498 |
|
|
%% using the CPAN module ``Data::Dumper'' and written to a file
|
499 |
|
|
%% called ``.SCRAM/DirCache.db''. This is the directory cache.
|
500 |
|
|
%% Removing this file will result in a re-read of the directory tree and
|
501 |
|
|
%% repopulation of the cache on a subsequent ``scram build''. This can
|
502 |
|
|
%% happen intentionally via a ``scram build distclean''.
|
503 |
|
|
|
504 |
|
|
|
505 |
|
|
|
506 |
|
|
%% - The Build Metadata Cache
|
507 |
|
|
|
508 |
|
|
%% The next step is to collect the build metadata and store it in a cache
|
509 |
|
|
%% that is separate to the directory cache. This cache is also a Perl
|
510 |
|
|
%% object (the same methods are used for reading and writing caches in
|
511 |
|
|
%% all cache handling operations in SCRAM version 1.0) which is written
|
512 |
|
|
%% to a file called ``.SCRAM/ProjectCache.db''. The object is of type
|
513 |
|
|
%% ``BuildSystem::BuildDataStorage''.
|
514 |
|
|
|
515 |
|
|
%% After checking that there is a project BuildFile in the config
|
516 |
|
|
%% directory, which is essential for obtaining the ClassPath data which
|
517 |
|
|
%% directs all build operations, each directory known to the directory
|
518 |
|
|
%% cache is scanned. Every path is assigned a ``BuildSystem::TreeItem''
|
519 |
|
|
%% object which is used to store all metadata required by the template
|
520 |
|
|
%% engine which will be used to generate a Makefile.
|
521 |
|
|
|
522 |
|
|
%% Two main actions are performed for each path:
|
523 |
|
|
%% ---------------------------------------------
|
524 |
|
|
|
525 |
|
|
%% i. if a BuildFile exists under ``path/Buildfile'', it is parsed and
|
526 |
|
|
%% this raw data is stored in the TreeItem object. Any groups defined in
|
527 |
|
|
%% the BuildFile are recorded in the build cache as a KNOWNGROUPS hash
|
528 |
|
|
%% with the name of the group as the key and the path to the buildFile
|
529 |
|
|
%% defining it as the value. This is required so that later the
|
530 |
|
|
%% KNOWNGROUPS hash can be used as a lookup table when resolving
|
531 |
|
|
%% <group name=X> type tags.
|
532 |
|
|
%% The raw metadata is itself stored as an object of type
|
533 |
|
|
%% ``BuildSystem::BuildFile'' in the TreeItem.
|
534 |
|
|
|
535 |
|
|
%% ii. the path is compared to each known ClassPath (as obtained from
|
536 |
|
|
%% config/BuildFile) to establish what build actions should be applied
|
537 |
|
|
%% there. The directory is assigned the following based on a best match to a
|
538 |
|
|
%% ClassPath:
|
539 |
|
|
|
540 |
|
|
%% - a class, i.e. a type of template to apply (Project, SubSystem,
|
541 |
|
|
%% Package, Library, Binary, etc..)
|
542 |
|
|
|
543 |
|
|
%% - a classdir, the part of the path that matched the ClassPath
|
544 |
|
|
|
545 |
|
|
%% - a suffix, the part of the path that dir *not* match
|
546 |
|
|
|
547 |
|
|
%% Because package dependencies are in general stated in <use name=X>
|
548 |
|
|
%% tags as ``subsystem/package''and not ``src/subsystem/package'', the
|
549 |
|
|
%% path is converted to a DATAPATH by removing the ``src'', or rather,
|
550 |
|
|
%% by removing the start of the path that matches \$ENV{SCRAM\_SOURCEDIR}.
|
551 |
|
|
%% This data is stored in the TreeItem which is itself stored in the
|
552 |
|
|
%% build cache, accessed using the DATAPATH as the key. This provides an
|
553 |
|
|
%% efficient lookup table for accessing package-level metadata when
|
554 |
|
|
%% resolving build requirements.
|
555 |
|
|
|
556 |
|
|
%% The parent directory and subdirectories (children) are recorded in
|
557 |
|
|
%% each TreeItem: this allows traversal from parent to children (a
|
558 |
|
|
%% directory ``branch'') and when coupled with the class information, can
|
559 |
|
|
%% permit simple determination of proper locations for build actions
|
560 |
|
|
%% (i.e. the actual path where the build of a certain class should be
|
561 |
|
|
%% applied) and greatly facilitates updating of build data when a
|
562 |
|
|
%% BuildFile somewhere in the branch has been modified. Thus it is not
|
563 |
|
|
%% necessary to reparse all BuildFiles after an modification to only one
|
564 |
|
|
%% of them.
|
565 |
|
|
|
566 |
|
|
|
567 |
|
|
%% Updating Metadata
|
568 |
|
|
%% -----------------
|
569 |
|
|
|
570 |
|
|
%% At the start of a build, the directory cache is scanned and files that
|
571 |
|
|
%% are new or have newer timestamps are updated. Any BuildFiles that have
|
572 |
|
|
%% been updated are passed to the update routines of the
|
573 |
|
|
%% ``BuildSystem::BuildDataStorage'' object which is responsible for
|
574 |
|
|
%% collecting and managing the metadata.
|
575 |
|
|
%% The BuildFiles are re-parsed and the TreeItem for the path and any
|
576 |
|
|
%% subdirectories is updated. Only the BuildFiles that have been
|
577 |
|
|
%% modified are re-processed in this way and thus, only the relevant
|
578 |
|
|
%% makefile fragments, are remade. After updating, the directory cache is
|
579 |
|
|
%% marked as up-to-date and the build proceeds as normal.
|
580 |
|
|
|
581 |
|
|
|
582 |
|
|
%% From Metadata to Makefile
|
583 |
|
|
%% -------------------------
|
584 |
|
|
|
585 |
|
|
%% Once all metadata is available, the template engine takes over. The
|
586 |
|
|
%% TemplateInterface object, which is a global, handles the interfacing
|
587 |
|
|
%% to the templates and their plugin modules.
|
588 |
|
|
%% The templates know how to obtain the metadata from the collected data
|
589 |
|
|
%% via the methods in the plugins. The TreeItem object provides all
|
590 |
|
|
%% needed metadata for a particular path: once the template engine has a
|
591 |
|
|
%% TreeItem, it uses methods in the PluginCore object to obtain
|
592 |
|
|
|
593 |
|
|
%% - lists of libraries in link order;
|
594 |
|
|
%% - LIBDIR and INCLUDEDIR lists in link order;
|
595 |
|
|
%% - compiler flags and makefile text, correctly formatted;
|
596 |
|
|
%% - package dependencies
|
597 |
|
|
|
598 |
|
|
|
599 |
|
|
|
600 |
|
|
|
601 |
|
|
%% Using Private Plugins
|
602 |
|
|
%% ---------------------
|
603 |
|
|
|
604 |
|
|
|
605 |
|
|
%% Customizations can be provided by way of project Template plugin
|
606 |
|
|
%% modules. These modules are standard Perl modules, inheriting from the
|
607 |
|
|
%% SCRAM base plugin module class.
|
608 |
|
|
|
609 |
|
|
%% For example, to override the main PluginCore module, one could do the
|
610 |
|
|
%% following:
|
611 |
|
|
|
612 |
|
|
%% - Write a module called SCRAM::Plugins::MyPluginCore inheriting from
|
613 |
|
|
%% BuildSystem::Template::Plugins::PluginCore;
|
614 |
|
|
|
615 |
|
|
%% - Add
|
616 |
|
|
%% [% USE MyPluginCore %]
|
617 |
|
|
|
618 |
|
|
%% to the project templates. The build system will then have access to
|
619 |
|
|
%% customized build metadata.
|
620 |
|
|
|
621 |
|
|
%% NB: Custom base class MUST be SCRAM::Plugins....
|
622 |
|
|
|
623 |
|
|
|
624 |
|
|
|
625 |
|
|
|
626 |
|
|
|
627 |
|
|
|
628 |
|
|
%% \section{Brief Description of the Build Process}\label{sec:buildprocess}
|
629 |
|
|
%% \index{description of the SCRAM build process}
|
630 |
|
|
|
631 |
|
|
%% Assuming that the build area is clean (no builds have already been
|
632 |
|
|
%% performed) and that all tools are correctly set up, the actions taken
|
633 |
|
|
%% by \scram are as follows:
|
634 |
|
|
|
635 |
|
|
%% \begin{itemize}
|
636 |
|
|
%% \item Initially, \scram checks the current location to test if it is a
|
637 |
|
|
%% project area. If it isn't, an error message like
|
638 |
|
|
|
639 |
|
|
%% %% \small{\begin{verbatim}
|
640 |
|
|
%% %% SCRAM error: Unable to locate the top of local release. Exitting.
|
641 |
|
|
%% %% \end{verbatim}}\normalsize
|
642 |
|
|
|
643 |
|
|
%% \ni will appear on \texttt{STDERR}.
|
644 |
|
|
|
645 |
|
|
|
646 |
|
|
%% \item Tool settings are read and a makefile stub is created which
|
647 |
|
|
%% contains all relevant path information for all tools in makefile
|
648 |
|
|
%% syntax. The makefile stub is called \texttt{clientmakefile} and
|
649 |
|
|
%% is located in the directory \texttt{tmp} in the project area.
|
650 |
|
|
|
651 |
|
|
%% \item \scram parses the project \texttt{BuildFile}, creating a
|
652 |
|
|
%% corresponding makefile called \texttt{BuildFile.mk} located
|
653 |
|
|
%% under \texttt{tmp/config}. The \texttt{ClassPath} settings are
|
654 |
|
|
%% parsed to determine the appropriate build actions.
|
655 |
|
|
|
656 |
|
|
%% \item A makefile is created according to the current directory.
|
657 |
|
|
%% The makefile is located in a directory under \texttt{tmp/src}
|
658 |
|
|
%% which has the same name as the current directory. This makefile
|
659 |
|
|
%% will be merged with appropriate makefiles determined from the
|
660 |
|
|
%% \texttt{ClassPath} and will include the makefile generated from
|
661 |
|
|
%% the project \texttt{BuildFile}. A list of subdirectories in
|
662 |
|
|
%% which build actions should occur is stored in a make variable
|
663 |
|
|
%% \texttt{\$(SUBDIRS)}-- the build order of various packages is
|
664 |
|
|
%% the same as the order in which the package directories appear in
|
665 |
|
|
%% this variable. To view the order, type \texttt{scram b echo\_SUBDIRS}.
|
666 |
|
|
|
667 |
|
|
%% \item Before running \texttt{gmake}, the generated makefiles are
|
668 |
|
|
%% merged with \texttt{basics.mk} from the \scram sources (which
|
669 |
|
|
%% also includes \texttt{toolrules.mk} which contains the rules
|
670 |
|
|
%% for compiling different file types and how to create libraries,
|
671 |
|
|
%% binaries or modules).
|
672 |
|
|
|
673 |
|
|
%% \item The build runs: any extra makefile statements or compiler
|
674 |
|
|
%% flags are passed directly to \texttt{gmake}. The working directory for
|
675 |
|
|
%% compilation is \texttt{/tmp/\$(SCRAM\_ARCH)}. Errors and warnings
|
676 |
|
|
%% are reported directly to standard output.
|
677 |
|
|
|
678 |
|
|
%% \item The build products are moved from the working directory to
|
679 |
|
|
%% the product storage areas.
|
680 |
|
|
|
681 |
|
|
%% \end{itemize}
|
682 |
|
|
|
683 |
|
|
|
684 |
|
|
%%% Local Variables:
|
685 |
|
|
%%% mode: latex
|
686 |
|
|
%%% TeX-master: "SCRAM-manual"
|
687 |
|
|
%%% End:
|
688 |
|
|
|
689 |
|
|
%%____________________________________________________________________
|
690 |
|
|
%% End of BuildSystem.tex
|
691 |
|
|
%%____________________________________________________________________
|
692 |
|
|
%%
|