ViewVC Help
View File | Revision Log | Show Annotations | Root Listing
root/cvsroot/COMP/SCRAM/doc/tex/manual/BuildSystem.tex
Revision: 1.4
Committed: Mon Sep 11 13:37:36 2006 UTC (18 years, 8 months ago) by sashby
Content type: application/x-tex
Branch: MAIN
CVS Tags: V1_0_3-p4, V1_0_3-p3, V1_0_3-p2, V1_0_3-p1, V1_0_3
Branch point for: v103_with_xml
Changes since 1.3: +14 -11 lines
Log Message:
Merged from v103_branch

File Contents

# User Rev Content
1 sashby 1.1 %%____________________________________________________________________
2     %% File: BuildSystem.tex
3     %%____________________________________________________________________
4     %%
5     %% Author: Shaun ASHBY <Shaun.Ashby@cern.ch>
6     %% Update: 2005-11-02 17:07:01+0100
7 sashby 1.4 %% Revision: $Id: BuildSystem.tex,v 1.3 2005/11/09 12:46:11 sashby Exp $
8 sashby 1.1 %%
9     %% Copyright: 2005 (C) Shaun ASHBY
10     %%
11     %%--------------------------------------------------------------------
12     \chapter{The SCRAM Build System}\label{ch:buildsystem}
13    
14     The primary function of the \scram\ build system is to allow efficient,
15     ordered compilation of source code of different types (\eg \texttt{C},
16     \texttt{C++}, \texttt{FORTRAN}) into libraries, binary executables or
17     other build products and to make it possible to easily extend the
18     supported product types, as dictated by changing project requirements,
19     and manage external and internal dependencies between software units
20     in a transparent and uniform way.
21    
22     \ni In previous versions of \scram, much of the build system functionality
23     was hard-coded in parts of the \scram\ source tree making it difficult
24     to make changes to adapt to changing software development patterns.
25     Any such change often meant a new release of \scram\ and the
26     subsequent delays entailed by a release sequence. In addition, the
27     actual algorithm for determining what build actions should be
28     activated for each software unit was not optimum with many
29     configuration documents needlessly being parsed multiple times,
30     greatly adding to project build times. This fundamental problem
31     stemmed from the fact there was never a clear separation between
32     parsing of configuration documents and the creation of a
33     \texttt{Makefile}- the two were run on the fly. Supporting the
34     building a project for a new architecture, particularly where the
35     syntax of the \texttt{Makefile} is very different (\eg
36     \texttt{WIN32}), was also practically impossible within the old
37     design.
38    
39     \ni In \scram~\scramvx, a new approach is taken by separating the
40     collection of build metadata from the generation of the
41     \texttt{Makefile} and compilation of the software units. Metadata in
42     this context refers to all information needed to build all products.
43     This includes package-level dependency requirements and all
44     information needed to use external tools (\eg libraries to link
45     against, compilation flags, pre-processor macros, library directories,
46     header file locations and runtime requirements).
47     Having parsed all necessary project documents, and the metadata
48     obtained and processed as required, data elements are substituted
49     into text templates to create the project \texttt{Makefile}. The data
50     elements include directory names, library names and names of
51     build targets all of which must be generated by \scram.
52     Of course, the text templates can be written so that the output is in
53     some other format than that for a \texttt{Makefile}.
54    
55     \section{BuildFiles and Build System Control}
56    
57     The \buildfile\ is the configuration document for the build system. For
58     each software unit, a \buildfile\ defines the external dependencies, the
59     internal dependencies (\ie the other software units that are used by
60     this one) and the interface. This interface supplies the product
61     provided by this software unit and the external/internal dependencies
62     required by it to any other software unit that requires it.
63     A software unit can be thought of as a product entity which is self-contained
64     and can be used either in linking to satisfy the dependencies of
65     a build product, or can be used to participate in some way on the
66     build process (for example, an executable used to process a source
67     file template to generate soucre files in a particular package).
68     Most of the time, a software unit is synonymous with a package (and
69     hence a shared library).
70    
71     \subsection{The Markup Syntax}
72     \index{\buildfile}\index{SCRAM!build files}
73    
74     Differing from other configuration documents in which the document
75     class defines the parse type, \scram\ knows about build files only
76     from their name, which must be \buildfile\footnote{\textit{note the uppercase letters!}.}.
77     A \buildfile\ only has a significance in a project area. The
78     valid tags that can be used in a \buildfile\ are listed below.
79     \index{\buildfile!valid basic markup tags}
80    
81     \begin{description}
82    
83     \item[\lbkt\texttt{use} name=\textit{unit}\rbkt]\mbox{}\\
84     Specify that there is a dependency on an external tool or a
85     local/external package \texttt{unit} (\eg a library, or everything the package
86     exports). External tools and packages are treated in the same way.
87     The interface to \textit{unit} must be defined via the
88     \lbkt\texttt{export}\rbkt\tagend{export} tag.
89    
90     \item[\lbkt\texttt{export}\rbkt\tagend{export}]\mbox{}\\
91     A software unit can be declared and exported to be used inside the
92     project or externally by another \scram\ project using the current
93     project as an external tool. The interface is defined by listing
94     the libraries (this can include those corresponding to a subset of a
95     subsystem) that are to be exported when the package (or subsystem)
96     is used. Typically, a package will export its' library, all
97     dependencies and specific compiler flags.
98    
99     \item[\lbkt\texttt{lib} name=\textit{libname} {[}\texttt{position}=\textit{first}{]}\rbkt]\mbox{}\\
100     Add a library to the list of libraries passed to the linker. The
101     optional \texttt{position} argument will move the library to the
102     front of the list of libraries as received by the linker (this
103     bypasses the automatic library ordering as determined by depth-first
104     sorting of the package metadata).
105    
106     \item[\lbkt\texttt{group} name=\textit{groupname}\rbkt]\mbox{}\\
107     Specify that all dependencies defined within the group
108     \textit{groupname} should be used by the current software unit.
109    
110     \item[\lbkt\texttt{include\_path} path=\textit{path}\rbkt]\mbox{}\\
111     Add the path \textit{path} to the global \texttt{INCLUDE} path
112     passed to the compiler. The include paths of individual
113     tools/packages will be ordered according to their dependency order
114     when added to the project or package \texttt{INCLUDE}.
115    
116     \item[\lbkt\texttt{libtype} type=\textit{type}\rbkt]\mbox{}\\
117     Specify the type of library that should be built.
118     This option overrides the project defaults but is currently unused
119     since libraries are \texttt{shared} only (\textit{support for debug libs to follow....}).
120    
121     \item[\lbkt\texttt{flags} \texttt{NAME}='definition'\rbkt]\mbox{}\\
122     Extra compiler flags can be added either in a tool description or in
123     a package. These flags will be added to the global compiler flags
124     and propageted to the build product defined in the \buildfile\ where
125     the declaration was made.
126     \end{description}
127    
128     \ni There are also tags for the supported build products:
129     \index{\buildfile!valid product markup tags}
130     \begin{description}
131    
132     \item[\lbkt\texttt{bin} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{bin}]\mbox{}\\
133     Specify an executable to build. The name of the executable can be
134     changed using the optional \textit{name} argument, otherwise the
135     name will be the same as \textit{filename}, less the file ending
136     (typically `.cpp'). More than one file can be compiled and linked to
137     make the executable. The file list can be specified like "main.cpp,
138     *.cc", "*.cpp" or "main.cpp, file.cc" (\ie file globs are
139     supported in a minimal way).
140    
141     All dependencies are contained between the opening and closing tags:
142     these dependencies are specific \textit{only} to this binary
143     executable. Dependencies or other metadata listed outside the
144     individual product tag will be passed to all products defined in the
145     \buildfile.
146    
147     \item[\lbkt\texttt{module} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{module}]\mbox{}\\
148     Specify a plug-in module to build. The name of the module can be
149     changed using the optional \textit{name} argument. How the plug-in
150     is defined can be customised within the project. By default (\ie
151     when using initial templates provided with \scram, copied to a
152     project \texttt{config} directory), a plug-in module is just a shared
153     library with all dependencies fully resolved when it is loaded (in
154     fact, there is not much difference between a plug-in module and a
155     shared library built by default in package \texttt{src} directories).
156    
157     \item[\lbkt\texttt{library} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{library}]\mbox{}\\
158     Define an additional library to build (in addition to the library
159     built automatically from the contents of \texttt{src} in the parent
160     package directory). This is usually used to build libraries for unit tests.
161    
162     \item[\lbkt\texttt{application} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{application}]\mbox{}\\
163     Define an application. How one defines an application is
164     project-specific: it could be that an application differs from a
165     binary executable in only the compiler flags or the storage
166     location when released. This product type could also be used to
167     inject custom rules into the build system.
168    
169     % Not yet advertised: <<FIXME
170     %\item[\lbkt\texttt{skip}\rbkt\tagend{skip}]\mbox{}\\
171     % Indicate that a directory should be skipped. Comments can be entered
172     % between the tags which will be printed to STDOUT during building.
173     % \textit{Work in progress!!}.
174     %
175     \end{description}
176    
177    
178    
179     \subsection{The Project BuildFile}\label{sec:projectbuildfile}
180     \index{\buildfile!project}
181    
182     Global behaviour is controlled by the project \buildfile\ which is
183     located in the \texttt{config} directory in all \scram-managed
184     projects. This file is parsed first, before all others. The
185     storage areas for build products are defined in this \buildfile. More
186     importantly, instructions on what actions to perform throughout the
187     source code tree occur here using \texttt{ClassPath} directives.
188     The directives indicate which templates to apply at each directory
189     location based on matching the directory structure to the class path.
190     There are three keywords which refer to certain levels in a directory
191     tree and the definitions of these are fixed. The keywords are
192     \texttt{Project}, which refers to the top-level source directory
193     (usually \texttt{src}), \texttt{SubSystem}, which refers to the
194     directory one level up, and \texttt{Package}, which refers to a
195     subdirectory of a \texttt{SubSystem}. A project is not bound to a
196     structure in which there are always subsystems: in fact, it is
197     possible to have only a package level under the project \texttt{src}
198     directory. The advantage to having subsystems is that it becomes
199     possible to group packages together according to some common task (for
200     example): groups can then be defined which allow several independent
201     packages to become a single dependency unit which can be used by other
202     packages or executables at link-time.
203    
204     \ni Each directory which is an exact match to a keyword location will be
205     flagged as such and \textit{structure templates}\index{structure templates}
206     will be used to create the rules for compiling in these locations.
207     Of course, it is possible to redefine what actions to apply at the
208     subsystem, package or even project level, just by supplying a different
209     template or overriding an action in the project class path.
210     For libraries and executables (in general, build products), which are
211     associated to a subdirectory inside a package, \textit{product
212     templates} define the build targets.
213     Generated build rules for directories that do not fully match any
214     \texttt{ClassPath} will simply print a message that no action is required
215     at that location (\eg for an \texttt{include} or \texttt{interface} directory).
216     Complex build operations can be activated by modifying the
217     structure templates to do specific things for specific directories.
218     This could be for a \texttt{Documentation} subsystem, for example,
219     where the source code happens to be \texttt{html} or \LaTeX\
220     sources, or for single packages where source code must be generated
221     first before normal package build actions.
222    
223     \ni The syntax of the \texttt{ClassPath} tag is \index{\texttt{ClassPath}
224     tag}%
225     \begin{tagprint}
226     \lbkt\texttt{ClassPath}
227     path={[}\textit{pattern\_match}{]}+\textit{template\_type}/\ldots/\rbkt
228     \end{tagprint}
229    
230     \ni In the above case, the \texttt{template\_type} could be
231     \texttt{library}, \texttt{binary} \texttt{module} \etc. (the build
232     products). The template file name corresponding to the
233     \texttt{library} template type would be
234     \texttt{library\_template.tmpl} and this would be located in the
235     \texttt{config} directory of the project. Note that where there are
236     multiple \lbkt\texttt{ClassPath}\rbkt tags defined, it is always the
237     last tag that matches the current location that will be used.
238     See examples in Chapter~\ref{ch:examples}.
239    
240     \subsubsection{Product Storage Locations}
241     \index{\texttt{ProductStore} tag}%
242    
243     The storage locations are defined using
244     \begin{tagprint}
245     \lbkt\texttt{ProductStore} name=\textit{name} {[}type=\textit{type}
246 sashby 1.4 swap=\textit{t}{]} {[}path=\textit{path}{]}\rbkt
247 sashby 1.1 \end{tagprint}
248    
249     \ni The name \textit{name} is the name of the directory to be created
250     and the option \textit{type} can be set to \texttt{arch} so that an
251     architecture-dependent subdirectory for platform-specific products
252     will be added in the project area with the product directory
253     \textit{name} underneath. Without this type option the directory
254     \textit{name} will be created in the project area. Using the
255     \textit{swap} option reverses the order of the architecture-dependent
256     sub-directory and the name of the storage directory. That is, setting
257     swap to \textit{true} or \textit{t} will create the product directory
258     first with an architecture-dependent sub-directory
259     underneath.\footnote{This is the fixed behaviour for all SCRAM
260     releases prior to \scramvx.}
261    
262 sashby 1.4 \ni The path \texttt{path} can be specified as the location where all
263     products corresponding to \textit{name} will be installed: a symbolic
264     link will be created in the project area which points to this
265     directory.
266    
267     \ni When \scram\ finds a \tagstart{ProductStore} tag,
268     a \texttt{Makefile} variable is set which can be used anywhere in a project makefile
269 sashby 1.1 when writing custom rules:
270     \begin{description}
271     \item[\texttt{SCRAMSTORENAME\_name}]\mbox{}\\
272     The path to the storage location \texttt{name} from the project area
273     directory (\eg \texttt{lib} or \texttt{lib/slc3\_ia32\_gcc323}).
274     \end{description}
275    
276     \ni This could be used, for example, where there may be a default rule for
277     copying all \texttt{include} files to a single location after building
278     all libraries. By default, all build products are copied to their
279     storage areas once built.
280    
281     \subsection{Local Metadata}\index{local metadata}\index{config/self}
282    
283     In order to propagate metadata from the local area to the source tree
284     at compile time, the local settings like \texttt{INCLUDE} or library paths in the
285     project templates plus the runtime environment, are defined in a local
286     file called \texttt{Self} found in the \texttt{config} directory\footnote{Note that this is unlike \scram\ V0 where
287     such settings were hard-coded in the top-level \buildfile.}.
288     This file behaves like a tool description file and is set up when the project
289     area is bootstrapped: all of the information defined within it is
290     automatically propagated to the whole tree.
291     In addition, all \texttt{scram} commands used to query tools
292     can also be used to inspect the local settings and changes can be made
293     at will with the tool set up by hand to propagate new settings.
294    
295     \subsection{Defining Groups}\label{sec:defininggroups}
296     \index{software units!defining a group}
297     A group can be used by adding a a \lbkt\texttt{group}\dots\rbkt statement
298     giving the name of the group to be included.
299    
300     \ni A group can be defined like this
301    
302     \small{
303     \begin{verbatim}
304     <define_group name=GA>
305     <use name=D>
306     <use name=zlib>
307     <Group name=XY>
308     <Flags CPPFLAGS="-DGROUP_GA">
309     </define_group>
310     \end{verbatim}
311     }\normalsize
312    
313     \ni in a subsystem \buildfile.
314    
315     \ni Note that it is not necessary to declare where the group can be
316     found (for example in another \scram\ project included as an
317     external product in the local configuration environment) since \scram\
318     determines this automatically. Duplicated/overridden groups will raise
319     a warning when \scram\ parses the \buildfile.
320    
321     \section{Configuring a New Package}
322     \label{sec:exportingsoftwareunits}
323     \index{software units}
324     \index{software units!defining the interface to}
325    
326     When a new package is added to a project, it is important that the
327     directory contains a \buildfile\ (note that it must be the package
328     directory and \textit{not} the \texttt{src} directory where the
329     sources are located that contains the \buildfile).
330     This \buildfile\ should have the following components:
331     \begin{description}
332     \item[\textbf{Declarations for all compile-time/link-time dependencies}]\mbox{}\\
333     The dependencies should be deduced from the \texttt{include}
334     statements in the package sources and a \lbkt\texttt{use}
335     name=\textit{unit}\rbkt should be added for each required unit
336     (external/internal package or external software product).
337     \item[\textbf{Export of the dependencies and package product}]\mbox{}\\
338     Every package providing a shared library should permit client software
339     units to use it at compile or link time. The
340     \lbkt\texttt{export}\rbkt\tagend{export} tag is used to define the
341     interface to the package and contains the full list of
342     \lbkt\texttt{use} name=\textit{unit}\rbkt statements (as required
343     by the package) and a \lbkt\texttt{lib} name=\textit{PackageName}\rbkt
344     statement with the name of the library provided by the package.
345     Of course, the library name should be omitted if the package does
346     not provide a shared library, having no \texttt{src} directory or
347     source files. This could be the case for abstract packages providing
348     only template \texttt{include} files to derived packages.
349     \end{description}
350    
351     \ni A typical package \buildfile\ will look like this:
352     \index{example of using \texttt{export} tag}%
353     \index{software units!exporting}%
354     \small{
355     \begin{verbatim}
356     <export>
357     <lib name=PX>
358     <use name=S/A>
359     <use name=B>
360     </export>
361    
362     <use name=S/A>
363     <use name=B>
364    
365     \end{verbatim}
366     }\normalsize
367     \index{defining a set of libraries for a package}
368     \ni Any other package or executable requiring this package would have
369     a statement \lbkt\texttt{use} name=\texttt{PX}\rbkt
370     in the \buildfile\ of the package/executable, where
371     \texttt{PX} is a path to the package in the
372     project (and hence could actually look like \lbkt\texttt{use}
373     name=\texttt{Subsystem/PX}\rbkt if the package is located in
374     a subsystem).
375    
376     \ni External projects and external tools can be used in the same way
377     using the \texttt{use} statement, \ie it is not necessary to qualify
378     the statement with a project name as with the older \scram\ syntax.
379    
380     \section{Build System Caches}
381 sashby 1.3 \label{sec:bscaches}
382     \index{Build system!caches}
383 sashby 1.1
384     Tracking changes to files in the source tree is achieved by caching
385     file timestamps and this is especially important for files that are
386     part of the project configuration. Caching is also extensively used
387     to store metadata as it is read from the \buildfile s
388     and tool description documents and to minimise subsequent re-parsing of
389     build data when the dependencies of a target change.
390    
391     \subsection{File Timestamp Cache}
392 sashby 1.3 \label{sec:bstscache}
393     \index{File Timestamp Cache}
394 sashby 1.1
395     Build tools like \texttt{Make} keep track of relationships between
396     targets and the source files or headers needed to build those targets.
397     When a timestamp on a source file or header changes, \texttt{Make}
398     knows how to rebuild the target having determined that the target is
399     out of date with respect to those modified files. A limitation arises
400     when code is retrieved from a code repository such as one based on
401     \texttt{CVS}. Often, the timestamps on checked-out files are in the
402     past (usually the time of the commit) which implies that a target will
403     not be rebuilt if these files are added to the source code tree since
404     they will be older than the target.
405    
406     \ni To cope with situations like this, \scram\ employs a cache mechanism
407     which stores the timestamps of the \buildfile s and the directories in
408     the source code tree. Note that it is not necessary to store the
409     timestamp of every source file since it is the change of timestamp of
410     the parent directory that will change when files are added or removed.
411     Once populated (when \scram\ first performs a build), the existing
412     cache will be used to determine when the status of a directory or a
413     file has changed. Such a change to the status can be found by using
414     the \texttt{stat} command and comparing information like access mode
415     in addition to the timestamp. Changes in timestamp provoke a rebuild
416     of the updated package. If a \buildfile\ changes then the package is
417     rebuilt automatically, otherwise \scram\ will trigger a rebuild by
418     modifying the timestamp of the \buildfile\ of the package where the
419     files were added or removed first.
420    
421     \subsection{Metadata Cache}
422 sashby 1.3 \label{sec:bsmdcache}
423     \index{Metadata Cache}
424 sashby 1.1
425     The actions to be taken by the build system when compiling and the
426     relationships between project software units are described in
427     \buildfile s located in the package directories in the source
428     code tree and the project configuration directory. Since it is
429     important to discover when metadata has changed, the timestamps of all
430     templates and the tool cache are also monitored. Since a change in a
431     tool setting or a template could affect the whole project, these
432     changes force a reparse of all \buildfile s and a complete rebuild of
433     the project.
434    
435     \section{Understanding the Build Templates}
436 sashby 1.3 \label{sec:bstemplates}
437     \index{Build system!buildtemplates}
438 sashby 1.1
439     Many open source and commercial software projects use \texttt{make} to manage rebuilds
440     of large programs or collections of programs. Each build action is
441     defined using a rule which describes the steps that should be executed
442     to build the program. The rules are written in a file called a
443     \texttt{Makefile}. In very large projects, a \texttt{Makefile} can typically be
444     thousands of lines long and is very difficult to write
445     from scratch and manage manually; it is preferable to generate it
446     automatically using a more abstract process. Since \scram\ projects can
447     contain many occurrences of the same class of build object (\eg shared
448     libraries, executables), a complete \texttt{Makefile} can be
449     generated from templates which implement the rules for these different classes.
450     Thus, \scram\ can generate a \texttt{Makefile} for a complex project structure easily.
451    
452 sashby 1.2 \subsection{What is a Template?}
453 sashby 1.1
454     A template is a file which is written in the same way as a
455     \texttt{Makefile} except that some parts of the rules or environment
456 sashby 1.4 variables are inserted at the time of generation.
457    
458     %%\begin{center}
459     %% \textit{More to be added here over time. Check snapshots pages!}
460     %%\end{center}
461 sashby 1.2
462 sashby 1.1 % An Example Rule: <<FIXME
463    
464    
465    
466    
467     %% Build System
468     %% ------------
469    
470     %% The SCRAM version 1.0 build system consists of two stages. Firstly,
471     %% all external requirements and build actions are determined and cached
472     %% for every location in the project. Secondly, the processed metadata
473     %% (lists of libraries used, INCLUDE/LIBDIR paths and compiler flags) are
474     %% used by the template engine to produce Makefile fragments used by
475     %% gmake in the usual way. Templates can easily be adapted to support
476     %% both different kinds of compilations (e.g. using java) or different
477     %% architectures.
478    
479     %% The project data and the state of the project area are made persistent
480     %% so that, by checking the current state relative to the last cached
481     %% state, any new actions (e.g. a rebuild after a modification to a BuildFile)
482     %% can be affected very quickly without resorting to a traversal of the
483     %% entire directory tree.
484    
485    
486     %% - The Directory Cache
487    
488     %% In an empty project area, the first step is to populate the directory
489     %% cache with the timestamp and content information for the project
490     %% directory tree. All timestamps and file modes are stored for parent
491     %% and subdirectories in the ``src'' tree. Parameters for important files
492     %% used in the build system, such as templates and other files in
493     %% ``config'' are also recorded. The timestamps of other directories in
494     %% the project area (tmp and build product stores for example), are not
495     %% stored.
496     %% Once the timestamp information is stored, the cache (simply a Perl
497     %% object of type ``Cache::Cache'') is converted to a data structure
498     %% using the CPAN module ``Data::Dumper'' and written to a file
499     %% called ``.SCRAM/DirCache.db''. This is the directory cache.
500     %% Removing this file will result in a re-read of the directory tree and
501     %% repopulation of the cache on a subsequent ``scram build''. This can
502     %% happen intentionally via a ``scram build distclean''.
503    
504    
505    
506     %% - The Build Metadata Cache
507    
508     %% The next step is to collect the build metadata and store it in a cache
509     %% that is separate to the directory cache. This cache is also a Perl
510     %% object (the same methods are used for reading and writing caches in
511     %% all cache handling operations in SCRAM version 1.0) which is written
512     %% to a file called ``.SCRAM/ProjectCache.db''. The object is of type
513     %% ``BuildSystem::BuildDataStorage''.
514    
515     %% After checking that there is a project BuildFile in the config
516     %% directory, which is essential for obtaining the ClassPath data which
517     %% directs all build operations, each directory known to the directory
518     %% cache is scanned. Every path is assigned a ``BuildSystem::TreeItem''
519     %% object which is used to store all metadata required by the template
520     %% engine which will be used to generate a Makefile.
521    
522     %% Two main actions are performed for each path:
523     %% ---------------------------------------------
524    
525     %% i. if a BuildFile exists under ``path/Buildfile'', it is parsed and
526     %% this raw data is stored in the TreeItem object. Any groups defined in
527     %% the BuildFile are recorded in the build cache as a KNOWNGROUPS hash
528     %% with the name of the group as the key and the path to the buildFile
529     %% defining it as the value. This is required so that later the
530     %% KNOWNGROUPS hash can be used as a lookup table when resolving
531     %% <group name=X> type tags.
532     %% The raw metadata is itself stored as an object of type
533     %% ``BuildSystem::BuildFile'' in the TreeItem.
534    
535     %% ii. the path is compared to each known ClassPath (as obtained from
536     %% config/BuildFile) to establish what build actions should be applied
537     %% there. The directory is assigned the following based on a best match to a
538     %% ClassPath:
539    
540     %% - a class, i.e. a type of template to apply (Project, SubSystem,
541     %% Package, Library, Binary, etc..)
542    
543     %% - a classdir, the part of the path that matched the ClassPath
544    
545     %% - a suffix, the part of the path that dir *not* match
546    
547     %% Because package dependencies are in general stated in <use name=X>
548     %% tags as ``subsystem/package''and not ``src/subsystem/package'', the
549     %% path is converted to a DATAPATH by removing the ``src'', or rather,
550     %% by removing the start of the path that matches \$ENV{SCRAM\_SOURCEDIR}.
551     %% This data is stored in the TreeItem which is itself stored in the
552     %% build cache, accessed using the DATAPATH as the key. This provides an
553     %% efficient lookup table for accessing package-level metadata when
554     %% resolving build requirements.
555    
556     %% The parent directory and subdirectories (children) are recorded in
557     %% each TreeItem: this allows traversal from parent to children (a
558     %% directory ``branch'') and when coupled with the class information, can
559     %% permit simple determination of proper locations for build actions
560     %% (i.e. the actual path where the build of a certain class should be
561     %% applied) and greatly facilitates updating of build data when a
562     %% BuildFile somewhere in the branch has been modified. Thus it is not
563     %% necessary to reparse all BuildFiles after an modification to only one
564     %% of them.
565    
566    
567     %% Updating Metadata
568     %% -----------------
569    
570     %% At the start of a build, the directory cache is scanned and files that
571     %% are new or have newer timestamps are updated. Any BuildFiles that have
572     %% been updated are passed to the update routines of the
573     %% ``BuildSystem::BuildDataStorage'' object which is responsible for
574     %% collecting and managing the metadata.
575     %% The BuildFiles are re-parsed and the TreeItem for the path and any
576     %% subdirectories is updated. Only the BuildFiles that have been
577     %% modified are re-processed in this way and thus, only the relevant
578     %% makefile fragments, are remade. After updating, the directory cache is
579     %% marked as up-to-date and the build proceeds as normal.
580    
581    
582     %% From Metadata to Makefile
583     %% -------------------------
584    
585     %% Once all metadata is available, the template engine takes over. The
586     %% TemplateInterface object, which is a global, handles the interfacing
587     %% to the templates and their plugin modules.
588     %% The templates know how to obtain the metadata from the collected data
589     %% via the methods in the plugins. The TreeItem object provides all
590     %% needed metadata for a particular path: once the template engine has a
591     %% TreeItem, it uses methods in the PluginCore object to obtain
592    
593     %% - lists of libraries in link order;
594     %% - LIBDIR and INCLUDEDIR lists in link order;
595     %% - compiler flags and makefile text, correctly formatted;
596     %% - package dependencies
597    
598    
599    
600    
601     %% Using Private Plugins
602     %% ---------------------
603    
604    
605     %% Customizations can be provided by way of project Template plugin
606     %% modules. These modules are standard Perl modules, inheriting from the
607     %% SCRAM base plugin module class.
608    
609     %% For example, to override the main PluginCore module, one could do the
610     %% following:
611    
612     %% - Write a module called SCRAM::Plugins::MyPluginCore inheriting from
613     %% BuildSystem::Template::Plugins::PluginCore;
614    
615     %% - Add
616     %% [% USE MyPluginCore %]
617    
618     %% to the project templates. The build system will then have access to
619     %% customized build metadata.
620    
621     %% NB: Custom base class MUST be SCRAM::Plugins....
622    
623    
624    
625    
626    
627    
628     %% \section{Brief Description of the Build Process}\label{sec:buildprocess}
629     %% \index{description of the SCRAM build process}
630    
631     %% Assuming that the build area is clean (no builds have already been
632     %% performed) and that all tools are correctly set up, the actions taken
633     %% by \scram are as follows:
634    
635     %% \begin{itemize}
636     %% \item Initially, \scram checks the current location to test if it is a
637     %% project area. If it isn't, an error message like
638    
639     %% %% \small{\begin{verbatim}
640     %% %% SCRAM error: Unable to locate the top of local release. Exitting.
641     %% %% \end{verbatim}}\normalsize
642    
643     %% \ni will appear on \texttt{STDERR}.
644    
645    
646     %% \item Tool settings are read and a makefile stub is created which
647     %% contains all relevant path information for all tools in makefile
648     %% syntax. The makefile stub is called \texttt{clientmakefile} and
649     %% is located in the directory \texttt{tmp} in the project area.
650    
651     %% \item \scram parses the project \texttt{BuildFile}, creating a
652     %% corresponding makefile called \texttt{BuildFile.mk} located
653     %% under \texttt{tmp/config}. The \texttt{ClassPath} settings are
654     %% parsed to determine the appropriate build actions.
655    
656     %% \item A makefile is created according to the current directory.
657     %% The makefile is located in a directory under \texttt{tmp/src}
658     %% which has the same name as the current directory. This makefile
659     %% will be merged with appropriate makefiles determined from the
660     %% \texttt{ClassPath} and will include the makefile generated from
661     %% the project \texttt{BuildFile}. A list of subdirectories in
662     %% which build actions should occur is stored in a make variable
663     %% \texttt{\$(SUBDIRS)}-- the build order of various packages is
664     %% the same as the order in which the package directories appear in
665     %% this variable. To view the order, type \texttt{scram b echo\_SUBDIRS}.
666    
667     %% \item Before running \texttt{gmake}, the generated makefiles are
668     %% merged with \texttt{basics.mk} from the \scram sources (which
669     %% also includes \texttt{toolrules.mk} which contains the rules
670     %% for compiling different file types and how to create libraries,
671     %% binaries or modules).
672    
673     %% \item The build runs: any extra makefile statements or compiler
674     %% flags are passed directly to \texttt{gmake}. The working directory for
675     %% compilation is \texttt{/tmp/\$(SCRAM\_ARCH)}. Errors and warnings
676     %% are reported directly to standard output.
677    
678     %% \item The build products are moved from the working directory to
679     %% the product storage areas.
680    
681     %% \end{itemize}
682    
683    
684     %%% Local Variables:
685     %%% mode: latex
686     %%% TeX-master: "SCRAM-manual"
687     %%% End:
688    
689     %%____________________________________________________________________
690     %% End of BuildSystem.tex
691     %%____________________________________________________________________
692     %%