ViewVC Help
View File | Revision Log | Show Annotations | Root Listing
root/cvsroot/COMP/SCRAM/doc/tex/manual/BuildSystem.tex
Revision: 1.4.2.1
Committed: Thu Nov 2 15:01:02 2006 UTC (18 years, 6 months ago) by sashby
Content type: application/x-tex
Branch: v103_with_xml
CVS Tags: forV1_1_0, v103_xml_071106, V110p2, V110p1
Changes since 1.4: +46 -47 lines
Log Message:
update manual for V1_1_0 XML

File Contents

# User Rev Content
1 sashby 1.1 %%____________________________________________________________________
2     %% File: BuildSystem.tex
3     %%____________________________________________________________________
4     %%
5     %% Author: Shaun ASHBY <Shaun.Ashby@cern.ch>
6     %% Update: 2005-11-02 17:07:01+0100
7 sashby 1.4.2.1 %% Revision: $Id: BuildSystem.tex,v 1.4 2006/09/11 13:37:36 sashby Exp $
8 sashby 1.1 %%
9     %% Copyright: 2005 (C) Shaun ASHBY
10     %%
11     %%--------------------------------------------------------------------
12     \chapter{The SCRAM Build System}\label{ch:buildsystem}
13    
14     The primary function of the \scram\ build system is to allow efficient,
15     ordered compilation of source code of different types (\eg \texttt{C},
16     \texttt{C++}, \texttt{FORTRAN}) into libraries, binary executables or
17     other build products and to make it possible to easily extend the
18     supported product types, as dictated by changing project requirements,
19     and manage external and internal dependencies between software units
20     in a transparent and uniform way.
21    
22     \ni In previous versions of \scram, much of the build system functionality
23     was hard-coded in parts of the \scram\ source tree making it difficult
24     to make changes to adapt to changing software development patterns.
25     Any such change often meant a new release of \scram\ and the
26     subsequent delays entailed by a release sequence. In addition, the
27     actual algorithm for determining what build actions should be
28     activated for each software unit was not optimum with many
29     configuration documents needlessly being parsed multiple times,
30     greatly adding to project build times. This fundamental problem
31     stemmed from the fact there was never a clear separation between
32     parsing of configuration documents and the creation of a
33     \texttt{Makefile}- the two were run on the fly. Supporting the
34     building a project for a new architecture, particularly where the
35     syntax of the \texttt{Makefile} is very different (\eg
36     \texttt{WIN32}), was also practically impossible within the old
37     design.
38    
39     \ni In \scram~\scramvx, a new approach is taken by separating the
40     collection of build metadata from the generation of the
41     \texttt{Makefile} and compilation of the software units. Metadata in
42     this context refers to all information needed to build all products.
43     This includes package-level dependency requirements and all
44     information needed to use external tools (\eg libraries to link
45     against, compilation flags, pre-processor macros, library directories,
46     header file locations and runtime requirements).
47     Having parsed all necessary project documents, and the metadata
48     obtained and processed as required, data elements are substituted
49     into text templates to create the project \texttt{Makefile}. The data
50     elements include directory names, library names and names of
51     build targets all of which must be generated by \scram.
52     Of course, the text templates can be written so that the output is in
53     some other format than that for a \texttt{Makefile}.
54    
55     \section{BuildFiles and Build System Control}
56    
57     The \buildfile\ is the configuration document for the build system. For
58     each software unit, a \buildfile\ defines the external dependencies, the
59     internal dependencies (\ie the other software units that are used by
60     this one) and the interface. This interface supplies the product
61     provided by this software unit and the external/internal dependencies
62     required by it to any other software unit that requires it.
63     A software unit can be thought of as a product entity which is self-contained
64     and can be used either in linking to satisfy the dependencies of
65     a build product, or can be used to participate in some way on the
66     build process (for example, an executable used to process a source
67     file template to generate soucre files in a particular package).
68     Most of the time, a software unit is synonymous with a package (and
69     hence a shared library).
70    
71     \subsection{The Markup Syntax}
72     \index{\buildfile}\index{SCRAM!build files}
73    
74     Differing from other configuration documents in which the document
75     class defines the parse type, \scram\ knows about build files only
76     from their name, which must be \buildfile\footnote{\textit{note the uppercase letters!}.}.
77     A \buildfile\ only has a significance in a project area. The
78     valid tags that can be used in a \buildfile\ are listed below.
79     \index{\buildfile!valid basic markup tags}
80    
81     \begin{description}
82    
83 sashby 1.4.2.1 \item[\lbkt\texttt{use} name=\textit{"unit"}$/$\rbkt]\mbox{}\\
84 sashby 1.1 Specify that there is a dependency on an external tool or a
85     local/external package \texttt{unit} (\eg a library, or everything the package
86     exports). External tools and packages are treated in the same way.
87     The interface to \textit{unit} must be defined via the
88     \lbkt\texttt{export}\rbkt\tagend{export} tag.
89    
90     \item[\lbkt\texttt{export}\rbkt\tagend{export}]\mbox{}\\
91     A software unit can be declared and exported to be used inside the
92     project or externally by another \scram\ project using the current
93     project as an external tool. The interface is defined by listing
94     the libraries (this can include those corresponding to a subset of a
95     subsystem) that are to be exported when the package (or subsystem)
96     is used. Typically, a package will export its' library, all
97     dependencies and specific compiler flags.
98    
99 sashby 1.4.2.1 \item[\lbkt\texttt{lib} name=\textit{"libname"} {[}\texttt{position}=\textit{"first"}{]}$/$\rbkt]\mbox{}\\
100 sashby 1.1 Add a library to the list of libraries passed to the linker. The
101     optional \texttt{position} argument will move the library to the
102     front of the list of libraries as received by the linker (this
103     bypasses the automatic library ordering as determined by depth-first
104     sorting of the package metadata).
105    
106 sashby 1.4.2.1 \item[\lbkt\texttt{group} name=\textit{"groupname"}$/$\rbkt]\mbox{}\\
107 sashby 1.1 Specify that all dependencies defined within the group
108     \textit{groupname} should be used by the current software unit.
109    
110 sashby 1.4.2.1 \item[\lbkt\texttt{include\_path} path=\textit{"path"}$/$\rbkt]\mbox{}\\
111 sashby 1.1 Add the path \textit{path} to the global \texttt{INCLUDE} path
112     passed to the compiler. The include paths of individual
113     tools/packages will be ordered according to their dependency order
114     when added to the project or package \texttt{INCLUDE}.
115    
116 sashby 1.4.2.1 \item[\lbkt\texttt{libtype} type=\textit{"type"}$/$\rbkt]\mbox{}\\
117 sashby 1.1 Specify the type of library that should be built.
118     This option overrides the project defaults but is currently unused
119 sashby 1.4.2.1 since libraries are \texttt{shared} only.
120 sashby 1.1
121 sashby 1.4.2.1 \item[\lbkt\texttt{flags} \texttt{NAME}="definition"$/$\rbkt]\mbox{}\\
122 sashby 1.1 Extra compiler flags can be added either in a tool description or in
123     a package. These flags will be added to the global compiler flags
124     and propageted to the build product defined in the \buildfile\ where
125     the declaration was made.
126     \end{description}
127    
128     \ni There are also tags for the supported build products:
129     \index{\buildfile!valid product markup tags}
130     \begin{description}
131    
132 sashby 1.4.2.1 \item[\lbkt\texttt{bin} file=\textit{"filename"} {[}name=\textit{"name"}{]}\rbkt\tagend{bin}]\mbox{}\\
133 sashby 1.1 Specify an executable to build. The name of the executable can be
134     changed using the optional \textit{name} argument, otherwise the
135     name will be the same as \textit{filename}, less the file ending
136     (typically `.cpp'). More than one file can be compiled and linked to
137     make the executable. The file list can be specified like "main.cpp,
138     *.cc", "*.cpp" or "main.cpp, file.cc" (\ie file globs are
139     supported in a minimal way).
140    
141     All dependencies are contained between the opening and closing tags:
142     these dependencies are specific \textit{only} to this binary
143     executable. Dependencies or other metadata listed outside the
144     individual product tag will be passed to all products defined in the
145     \buildfile.
146    
147 sashby 1.4.2.1 \item[\lbkt\texttt{module} file=\textit{"filename"} {[}name=\textit{"name"}{]}\rbkt\tagend{module}]\mbox{}\\
148 sashby 1.1 Specify a plug-in module to build. The name of the module can be
149     changed using the optional \textit{name} argument. How the plug-in
150     is defined can be customised within the project. By default (\ie
151     when using initial templates provided with \scram, copied to a
152     project \texttt{config} directory), a plug-in module is just a shared
153     library with all dependencies fully resolved when it is loaded (in
154     fact, there is not much difference between a plug-in module and a
155     shared library built by default in package \texttt{src} directories).
156    
157 sashby 1.4.2.1 \item[\lbkt\texttt{library} file=\textit{"filename"} {[}name=\textit{"name"}{]}\rbkt\tagend{library}]\mbox{}\\
158 sashby 1.1 Define an additional library to build (in addition to the library
159     built automatically from the contents of \texttt{src} in the parent
160     package directory). This is usually used to build libraries for unit tests.
161    
162 sashby 1.4.2.1 \item[\lbkt\texttt{application} file=\textit{"filename"} {[}name=\textit{"name"}{]}\rbkt\tagend{application}]\mbox{}\\
163 sashby 1.1 Define an application. How one defines an application is
164     project-specific: it could be that an application differs from a
165     binary executable in only the compiler flags or the storage
166     location when released. This product type could also be used to
167     inject custom rules into the build system.
168    
169     % Not yet advertised: <<FIXME
170     %\item[\lbkt\texttt{skip}\rbkt\tagend{skip}]\mbox{}\\
171     % Indicate that a directory should be skipped. Comments can be entered
172     % between the tags which will be printed to STDOUT during building.
173     % \textit{Work in progress!!}.
174     %
175     \end{description}
176    
177    
178    
179     \subsection{The Project BuildFile}\label{sec:projectbuildfile}
180     \index{\buildfile!project}
181    
182     Global behaviour is controlled by the project \buildfile\ which is
183     located in the \texttt{config} directory in all \scram-managed
184     projects. This file is parsed first, before all others. The
185     storage areas for build products are defined in this \buildfile. More
186     importantly, instructions on what actions to perform throughout the
187 sashby 1.4.2.1 source code tree occur here using \texttt{classpath} directives.
188 sashby 1.1 The directives indicate which templates to apply at each directory
189     location based on matching the directory structure to the class path.
190     There are three keywords which refer to certain levels in a directory
191     tree and the definitions of these are fixed. The keywords are
192     \texttt{Project}, which refers to the top-level source directory
193     (usually \texttt{src}), \texttt{SubSystem}, which refers to the
194     directory one level up, and \texttt{Package}, which refers to a
195     subdirectory of a \texttt{SubSystem}. A project is not bound to a
196     structure in which there are always subsystems: in fact, it is
197     possible to have only a package level under the project \texttt{src}
198     directory. The advantage to having subsystems is that it becomes
199     possible to group packages together according to some common task (for
200     example): groups can then be defined which allow several independent
201     packages to become a single dependency unit which can be used by other
202     packages or executables at link-time.
203    
204     \ni Each directory which is an exact match to a keyword location will be
205     flagged as such and \textit{structure templates}\index{structure templates}
206     will be used to create the rules for compiling in these locations.
207     Of course, it is possible to redefine what actions to apply at the
208     subsystem, package or even project level, just by supplying a different
209     template or overriding an action in the project class path.
210     For libraries and executables (in general, build products), which are
211     associated to a subdirectory inside a package, \textit{product
212     templates} define the build targets.
213     Generated build rules for directories that do not fully match any
214 sashby 1.4.2.1 \texttt{classpath} will simply print a message that no action is required
215 sashby 1.1 at that location (\eg for an \texttt{include} or \texttt{interface} directory).
216     Complex build operations can be activated by modifying the
217     structure templates to do specific things for specific directories.
218     This could be for a \texttt{Documentation} subsystem, for example,
219     where the source code happens to be \texttt{html} or \LaTeX\
220     sources, or for single packages where source code must be generated
221     first before normal package build actions.
222    
223 sashby 1.4.2.1 \ni The syntax of the \texttt{classpath} tag is \index{\texttt{classpath}
224 sashby 1.1 tag}%
225     \begin{tagprint}
226 sashby 1.4.2.1 \lbkt\texttt{classpath}
227     path="{[}\textit{pattern\_match}{]}+\textit{template\_type}/\ldots"$/$\rbkt
228 sashby 1.1 \end{tagprint}
229    
230     \ni In the above case, the \texttt{template\_type} could be
231     \texttt{library}, \texttt{binary} \texttt{module} \etc. (the build
232     products). The template file name corresponding to the
233     \texttt{library} template type would be
234     \texttt{library\_template.tmpl} and this would be located in the
235     \texttt{config} directory of the project. Note that where there are
236 sashby 1.4.2.1 multiple \lbkt\texttt{classpath}\rbkt tags defined, it is always the
237 sashby 1.1 last tag that matches the current location that will be used.
238     See examples in Chapter~\ref{ch:examples}.
239    
240     \subsubsection{Product Storage Locations}
241 sashby 1.4.2.1 \index{\texttt{productstore} tag}%
242 sashby 1.1
243     The storage locations are defined using
244     \begin{tagprint}
245 sashby 1.4.2.1 \lbkt\texttt{productstore} name=\textit{"name"} {[}type=\textit{"type"}
246     swap=\textit{"t"}{]} {[}path=\textit{"path"}{]}$/$\rbkt
247 sashby 1.1 \end{tagprint}
248    
249     \ni The name \textit{name} is the name of the directory to be created
250     and the option \textit{type} can be set to \texttt{arch} so that an
251     architecture-dependent subdirectory for platform-specific products
252     will be added in the project area with the product directory
253     \textit{name} underneath. Without this type option the directory
254     \textit{name} will be created in the project area. Using the
255     \textit{swap} option reverses the order of the architecture-dependent
256     sub-directory and the name of the storage directory. That is, setting
257     swap to \textit{true} or \textit{t} will create the product directory
258     first with an architecture-dependent sub-directory
259     underneath.\footnote{This is the fixed behaviour for all SCRAM
260     releases prior to \scramvx.}
261    
262 sashby 1.4 \ni The path \texttt{path} can be specified as the location where all
263     products corresponding to \textit{name} will be installed: a symbolic
264     link will be created in the project area which points to this
265     directory.
266    
267 sashby 1.4.2.1 \ni When \scram\ finds a \tagstart{productstore} tag,
268 sashby 1.4 a \texttt{Makefile} variable is set which can be used anywhere in a project makefile
269 sashby 1.1 when writing custom rules:
270     \begin{description}
271     \item[\texttt{SCRAMSTORENAME\_name}]\mbox{}\\
272     The path to the storage location \texttt{name} from the project area
273     directory (\eg \texttt{lib} or \texttt{lib/slc3\_ia32\_gcc323}).
274     \end{description}
275    
276     \ni This could be used, for example, where there may be a default rule for
277     copying all \texttt{include} files to a single location after building
278     all libraries. By default, all build products are copied to their
279     storage areas once built.
280    
281     \subsection{Local Metadata}\index{local metadata}\index{config/self}
282    
283     In order to propagate metadata from the local area to the source tree
284     at compile time, the local settings like \texttt{INCLUDE} or library paths in the
285     project templates plus the runtime environment, are defined in a local
286     file called \texttt{Self} found in the \texttt{config} directory\footnote{Note that this is unlike \scram\ V0 where
287     such settings were hard-coded in the top-level \buildfile.}.
288     This file behaves like a tool description file and is set up when the project
289     area is bootstrapped: all of the information defined within it is
290     automatically propagated to the whole tree.
291     In addition, all \texttt{scram} commands used to query tools
292     can also be used to inspect the local settings and changes can be made
293     at will with the tool set up by hand to propagate new settings.
294    
295     \subsection{Defining Groups}\label{sec:defininggroups}
296     \index{software units!defining a group}
297     A group can be used by adding a a \lbkt\texttt{group}\dots\rbkt statement
298     giving the name of the group to be included.
299    
300     \ni A group can be defined like this
301    
302     \small{
303     \begin{verbatim}
304 sashby 1.4.2.1 <define_group name="GA">
305     <use name="D"/>
306     <use name="zlib"/>
307     <group name="XY"/>
308     <Flags CPPFLAGS="-DGROUP_GA"/>
309 sashby 1.1 </define_group>
310     \end{verbatim}
311     }\normalsize
312    
313     \ni in a subsystem \buildfile.
314    
315     \ni Note that it is not necessary to declare where the group can be
316     found (for example in another \scram\ project included as an
317     external product in the local configuration environment) since \scram\
318     determines this automatically. Duplicated/overridden groups will raise
319     a warning when \scram\ parses the \buildfile.
320    
321     \section{Configuring a New Package}
322     \label{sec:exportingsoftwareunits}
323     \index{software units}
324     \index{software units!defining the interface to}
325    
326     When a new package is added to a project, it is important that the
327     directory contains a \buildfile\ (note that it must be the package
328     directory and \textit{not} the \texttt{src} directory where the
329     sources are located that contains the \buildfile).
330     This \buildfile\ should have the following components:
331     \begin{description}
332     \item[\textbf{Declarations for all compile-time/link-time dependencies}]\mbox{}\\
333     The dependencies should be deduced from the \texttt{include}
334     statements in the package sources and a \lbkt\texttt{use}
335 sashby 1.4.2.1 name=\textit{"unit"}$/$\rbkt should be added for each required unit
336     (external/internal package or external software product) which
337     provides a library needed at link-time.
338 sashby 1.1 \item[\textbf{Export of the dependencies and package product}]\mbox{}\\
339     Every package providing a shared library should permit client software
340     units to use it at compile or link time. The
341     \lbkt\texttt{export}\rbkt\tagend{export} tag is used to define the
342     interface to the package and contains the full list of
343 sashby 1.4.2.1 \lbkt\texttt{use} name=\textit{"unit"}$/$\rbkt statements (as required
344     by the package) and a \lbkt\texttt{lib} name=\textit{"PackageName"}$/$\rbkt
345 sashby 1.1 statement with the name of the library provided by the package.
346     \end{description}
347    
348     \ni A typical package \buildfile\ will look like this:
349     \index{example of using \texttt{export} tag}%
350     \index{software units!exporting}%
351     \small{
352     \begin{verbatim}
353 sashby 1.4.2.1 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
354     <doc type="BuildSystem::BuildFile" version="1.0">
355     <export>
356     <lib name="PX"/>
357     <use name="S/A"/>
358     <use name="B"/>
359     </export>
360    
361     <use name="S/A"/>
362     <use name="B"/>
363     </doc>
364 sashby 1.1 \end{verbatim}
365     }\normalsize
366     \index{defining a set of libraries for a package}
367     \ni Any other package or executable requiring this package would have
368 sashby 1.4.2.1 a statement \lbkt\texttt{use} name=\texttt{"PX"}$/$\rbkt
369 sashby 1.1 in the \buildfile\ of the package/executable, where
370 sashby 1.4.2.1 \texttt{"PX"} is a path to the package in the
371 sashby 1.1 project (and hence could actually look like \lbkt\texttt{use}
372 sashby 1.4.2.1 name=\texttt{"Subsystem/PX"}$/$\rbkt if the package is located in
373 sashby 1.1 a subsystem).
374    
375     \ni External projects and external tools can be used in the same way
376     using the \texttt{use} statement, \ie it is not necessary to qualify
377     the statement with a project name as with the older \scram\ syntax.
378    
379     \section{Build System Caches}
380 sashby 1.3 \label{sec:bscaches}
381     \index{Build system!caches}
382 sashby 1.1
383     Tracking changes to files in the source tree is achieved by caching
384     file timestamps and this is especially important for files that are
385     part of the project configuration. Caching is also extensively used
386     to store metadata as it is read from the \buildfile s
387     and tool description documents and to minimise subsequent re-parsing of
388     build data when the dependencies of a target change.
389    
390     \subsection{File Timestamp Cache}
391 sashby 1.3 \label{sec:bstscache}
392     \index{File Timestamp Cache}
393 sashby 1.1
394     Build tools like \texttt{Make} keep track of relationships between
395     targets and the source files or headers needed to build those targets.
396     When a timestamp on a source file or header changes, \texttt{Make}
397     knows how to rebuild the target having determined that the target is
398     out of date with respect to those modified files. A limitation arises
399     when code is retrieved from a code repository such as one based on
400     \texttt{CVS}. Often, the timestamps on checked-out files are in the
401     past (usually the time of the commit) which implies that a target will
402     not be rebuilt if these files are added to the source code tree since
403     they will be older than the target.
404    
405     \ni To cope with situations like this, \scram\ employs a cache mechanism
406     which stores the timestamps of the \buildfile s and the directories in
407     the source code tree. Note that it is not necessary to store the
408     timestamp of every source file since it is the change of timestamp of
409     the parent directory that will change when files are added or removed.
410     Once populated (when \scram\ first performs a build), the existing
411     cache will be used to determine when the status of a directory or a
412     file has changed. Such a change to the status can be found by using
413     the \texttt{stat} command and comparing information like access mode
414     in addition to the timestamp. Changes in timestamp provoke a rebuild
415     of the updated package. If a \buildfile\ changes then the package is
416     rebuilt automatically, otherwise \scram\ will trigger a rebuild by
417     modifying the timestamp of the \buildfile\ of the package where the
418     files were added or removed first.
419    
420     \subsection{Metadata Cache}
421 sashby 1.3 \label{sec:bsmdcache}
422     \index{Metadata Cache}
423 sashby 1.1
424     The actions to be taken by the build system when compiling and the
425     relationships between project software units are described in
426     \buildfile s located in the package directories in the source
427     code tree and the project configuration directory. Since it is
428     important to discover when metadata has changed, the timestamps of all
429     templates and the tool cache are also monitored. Since a change in a
430     tool setting or a template could affect the whole project, these
431     changes force a reparse of all \buildfile s and a complete rebuild of
432     the project.
433    
434     \section{Understanding the Build Templates}
435 sashby 1.3 \label{sec:bstemplates}
436     \index{Build system!buildtemplates}
437 sashby 1.1
438     Many open source and commercial software projects use \texttt{make} to manage rebuilds
439     of large programs or collections of programs. Each build action is
440     defined using a rule which describes the steps that should be executed
441     to build the program. The rules are written in a file called a
442     \texttt{Makefile}. In very large projects, a \texttt{Makefile} can typically be
443     thousands of lines long and is very difficult to write
444     from scratch and manage manually; it is preferable to generate it
445     automatically using a more abstract process. Since \scram\ projects can
446     contain many occurrences of the same class of build object (\eg shared
447     libraries, executables), a complete \texttt{Makefile} can be
448     generated from templates which implement the rules for these different classes.
449     Thus, \scram\ can generate a \texttt{Makefile} for a complex project structure easily.
450    
451 sashby 1.2 \subsection{What is a Template?}
452 sashby 1.1
453     A template is a file which is written in the same way as a
454     \texttt{Makefile} except that some parts of the rules or environment
455 sashby 1.4 variables are inserted at the time of generation.
456    
457     %%\begin{center}
458     %% \textit{More to be added here over time. Check snapshots pages!}
459     %%\end{center}
460 sashby 1.2
461 sashby 1.1 % An Example Rule: <<FIXME
462    
463    
464    
465    
466     %% Build System
467     %% ------------
468    
469     %% The SCRAM version 1.0 build system consists of two stages. Firstly,
470     %% all external requirements and build actions are determined and cached
471     %% for every location in the project. Secondly, the processed metadata
472     %% (lists of libraries used, INCLUDE/LIBDIR paths and compiler flags) are
473     %% used by the template engine to produce Makefile fragments used by
474     %% gmake in the usual way. Templates can easily be adapted to support
475     %% both different kinds of compilations (e.g. using java) or different
476     %% architectures.
477    
478     %% The project data and the state of the project area are made persistent
479     %% so that, by checking the current state relative to the last cached
480     %% state, any new actions (e.g. a rebuild after a modification to a BuildFile)
481     %% can be affected very quickly without resorting to a traversal of the
482     %% entire directory tree.
483    
484    
485     %% - The Directory Cache
486    
487     %% In an empty project area, the first step is to populate the directory
488     %% cache with the timestamp and content information for the project
489     %% directory tree. All timestamps and file modes are stored for parent
490     %% and subdirectories in the ``src'' tree. Parameters for important files
491     %% used in the build system, such as templates and other files in
492     %% ``config'' are also recorded. The timestamps of other directories in
493     %% the project area (tmp and build product stores for example), are not
494     %% stored.
495     %% Once the timestamp information is stored, the cache (simply a Perl
496     %% object of type ``Cache::Cache'') is converted to a data structure
497     %% using the CPAN module ``Data::Dumper'' and written to a file
498     %% called ``.SCRAM/DirCache.db''. This is the directory cache.
499     %% Removing this file will result in a re-read of the directory tree and
500     %% repopulation of the cache on a subsequent ``scram build''. This can
501     %% happen intentionally via a ``scram build distclean''.
502    
503    
504    
505     %% - The Build Metadata Cache
506    
507     %% The next step is to collect the build metadata and store it in a cache
508     %% that is separate to the directory cache. This cache is also a Perl
509     %% object (the same methods are used for reading and writing caches in
510     %% all cache handling operations in SCRAM version 1.0) which is written
511     %% to a file called ``.SCRAM/ProjectCache.db''. The object is of type
512     %% ``BuildSystem::BuildDataStorage''.
513    
514     %% After checking that there is a project BuildFile in the config
515     %% directory, which is essential for obtaining the ClassPath data which
516     %% directs all build operations, each directory known to the directory
517     %% cache is scanned. Every path is assigned a ``BuildSystem::TreeItem''
518     %% object which is used to store all metadata required by the template
519     %% engine which will be used to generate a Makefile.
520    
521     %% Two main actions are performed for each path:
522     %% ---------------------------------------------
523    
524     %% i. if a BuildFile exists under ``path/Buildfile'', it is parsed and
525     %% this raw data is stored in the TreeItem object. Any groups defined in
526     %% the BuildFile are recorded in the build cache as a KNOWNGROUPS hash
527     %% with the name of the group as the key and the path to the buildFile
528     %% defining it as the value. This is required so that later the
529     %% KNOWNGROUPS hash can be used as a lookup table when resolving
530     %% <group name=X> type tags.
531     %% The raw metadata is itself stored as an object of type
532     %% ``BuildSystem::BuildFile'' in the TreeItem.
533    
534     %% ii. the path is compared to each known ClassPath (as obtained from
535     %% config/BuildFile) to establish what build actions should be applied
536     %% there. The directory is assigned the following based on a best match to a
537     %% ClassPath:
538    
539     %% - a class, i.e. a type of template to apply (Project, SubSystem,
540     %% Package, Library, Binary, etc..)
541    
542     %% - a classdir, the part of the path that matched the ClassPath
543    
544     %% - a suffix, the part of the path that dir *not* match
545    
546     %% Because package dependencies are in general stated in <use name=X>
547     %% tags as ``subsystem/package''and not ``src/subsystem/package'', the
548     %% path is converted to a DATAPATH by removing the ``src'', or rather,
549     %% by removing the start of the path that matches \$ENV{SCRAM\_SOURCEDIR}.
550     %% This data is stored in the TreeItem which is itself stored in the
551     %% build cache, accessed using the DATAPATH as the key. This provides an
552     %% efficient lookup table for accessing package-level metadata when
553     %% resolving build requirements.
554    
555     %% The parent directory and subdirectories (children) are recorded in
556     %% each TreeItem: this allows traversal from parent to children (a
557     %% directory ``branch'') and when coupled with the class information, can
558     %% permit simple determination of proper locations for build actions
559     %% (i.e. the actual path where the build of a certain class should be
560     %% applied) and greatly facilitates updating of build data when a
561     %% BuildFile somewhere in the branch has been modified. Thus it is not
562     %% necessary to reparse all BuildFiles after an modification to only one
563     %% of them.
564    
565    
566     %% Updating Metadata
567     %% -----------------
568    
569     %% At the start of a build, the directory cache is scanned and files that
570     %% are new or have newer timestamps are updated. Any BuildFiles that have
571     %% been updated are passed to the update routines of the
572     %% ``BuildSystem::BuildDataStorage'' object which is responsible for
573     %% collecting and managing the metadata.
574     %% The BuildFiles are re-parsed and the TreeItem for the path and any
575     %% subdirectories is updated. Only the BuildFiles that have been
576     %% modified are re-processed in this way and thus, only the relevant
577     %% makefile fragments, are remade. After updating, the directory cache is
578     %% marked as up-to-date and the build proceeds as normal.
579    
580    
581     %% From Metadata to Makefile
582     %% -------------------------
583    
584     %% Once all metadata is available, the template engine takes over. The
585     %% TemplateInterface object, which is a global, handles the interfacing
586     %% to the templates and their plugin modules.
587     %% The templates know how to obtain the metadata from the collected data
588     %% via the methods in the plugins. The TreeItem object provides all
589     %% needed metadata for a particular path: once the template engine has a
590     %% TreeItem, it uses methods in the PluginCore object to obtain
591    
592     %% - lists of libraries in link order;
593     %% - LIBDIR and INCLUDEDIR lists in link order;
594     %% - compiler flags and makefile text, correctly formatted;
595     %% - package dependencies
596    
597    
598    
599    
600     %% Using Private Plugins
601     %% ---------------------
602    
603    
604     %% Customizations can be provided by way of project Template plugin
605     %% modules. These modules are standard Perl modules, inheriting from the
606     %% SCRAM base plugin module class.
607    
608     %% For example, to override the main PluginCore module, one could do the
609     %% following:
610    
611     %% - Write a module called SCRAM::Plugins::MyPluginCore inheriting from
612     %% BuildSystem::Template::Plugins::PluginCore;
613    
614     %% - Add
615     %% [% USE MyPluginCore %]
616    
617     %% to the project templates. The build system will then have access to
618     %% customized build metadata.
619    
620     %% NB: Custom base class MUST be SCRAM::Plugins....
621    
622    
623    
624    
625    
626    
627     %% \section{Brief Description of the Build Process}\label{sec:buildprocess}
628     %% \index{description of the SCRAM build process}
629    
630     %% Assuming that the build area is clean (no builds have already been
631     %% performed) and that all tools are correctly set up, the actions taken
632     %% by \scram are as follows:
633    
634     %% \begin{itemize}
635     %% \item Initially, \scram checks the current location to test if it is a
636     %% project area. If it isn't, an error message like
637    
638     %% %% \small{\begin{verbatim}
639     %% %% SCRAM error: Unable to locate the top of local release. Exitting.
640     %% %% \end{verbatim}}\normalsize
641    
642     %% \ni will appear on \texttt{STDERR}.
643    
644    
645     %% \item Tool settings are read and a makefile stub is created which
646     %% contains all relevant path information for all tools in makefile
647     %% syntax. The makefile stub is called \texttt{clientmakefile} and
648     %% is located in the directory \texttt{tmp} in the project area.
649    
650     %% \item \scram parses the project \texttt{BuildFile}, creating a
651     %% corresponding makefile called \texttt{BuildFile.mk} located
652     %% under \texttt{tmp/config}. The \texttt{ClassPath} settings are
653     %% parsed to determine the appropriate build actions.
654    
655     %% \item A makefile is created according to the current directory.
656     %% The makefile is located in a directory under \texttt{tmp/src}
657     %% which has the same name as the current directory. This makefile
658     %% will be merged with appropriate makefiles determined from the
659     %% \texttt{ClassPath} and will include the makefile generated from
660     %% the project \texttt{BuildFile}. A list of subdirectories in
661     %% which build actions should occur is stored in a make variable
662     %% \texttt{\$(SUBDIRS)}-- the build order of various packages is
663     %% the same as the order in which the package directories appear in
664     %% this variable. To view the order, type \texttt{scram b echo\_SUBDIRS}.
665    
666     %% \item Before running \texttt{gmake}, the generated makefiles are
667     %% merged with \texttt{basics.mk} from the \scram sources (which
668     %% also includes \texttt{toolrules.mk} which contains the rules
669     %% for compiling different file types and how to create libraries,
670     %% binaries or modules).
671    
672     %% \item The build runs: any extra makefile statements or compiler
673     %% flags are passed directly to \texttt{gmake}. The working directory for
674     %% compilation is \texttt{/tmp/\$(SCRAM\_ARCH)}. Errors and warnings
675     %% are reported directly to standard output.
676    
677     %% \item The build products are moved from the working directory to
678     %% the product storage areas.
679    
680     %% \end{itemize}
681    
682    
683     %%% Local Variables:
684     %%% mode: latex
685     %%% TeX-master: "SCRAM-manual"
686     %%% End:
687    
688     %%____________________________________________________________________
689     %% End of BuildSystem.tex
690     %%____________________________________________________________________
691     %%