ViewVC Help
View File | Revision Log | Show Annotations | Root Listing
root/cvsroot/COMP/SCRAM/doc/tex/manual/BuildSystem.tex
Revision: 1.2
Committed: Tue Nov 8 13:28:28 2005 UTC (19 years, 6 months ago) by sashby
Content type: application/x-tex
Branch: MAIN
CVS Tags: V1_0_2
Branch point for: v103_branch
Changes since 1.1: +6 -2 lines
Log Message:
More doc updates, add ReleaseNotes

File Contents

# User Rev Content
1 sashby 1.1 %%____________________________________________________________________
2     %% File: BuildSystem.tex
3     %%____________________________________________________________________
4     %%
5     %% Author: Shaun ASHBY <Shaun.Ashby@cern.ch>
6     %% Update: 2005-11-02 17:07:01+0100
7 sashby 1.2 %% Revision: $Id: BuildSystem.tex,v 1.1 2005/11/02 16:24:18 sashby Exp $
8 sashby 1.1 %%
9     %% Copyright: 2005 (C) Shaun ASHBY
10     %%
11     %%--------------------------------------------------------------------
12     \chapter{The SCRAM Build System}\label{ch:buildsystem}
13    
14     The primary function of the \scram\ build system is to allow efficient,
15     ordered compilation of source code of different types (\eg \texttt{C},
16     \texttt{C++}, \texttt{FORTRAN}) into libraries, binary executables or
17     other build products and to make it possible to easily extend the
18     supported product types, as dictated by changing project requirements,
19     and manage external and internal dependencies between software units
20     in a transparent and uniform way.
21    
22     \ni In previous versions of \scram, much of the build system functionality
23     was hard-coded in parts of the \scram\ source tree making it difficult
24     to make changes to adapt to changing software development patterns.
25     Any such change often meant a new release of \scram\ and the
26     subsequent delays entailed by a release sequence. In addition, the
27     actual algorithm for determining what build actions should be
28     activated for each software unit was not optimum with many
29     configuration documents needlessly being parsed multiple times,
30     greatly adding to project build times. This fundamental problem
31     stemmed from the fact there was never a clear separation between
32     parsing of configuration documents and the creation of a
33     \texttt{Makefile}- the two were run on the fly. Supporting the
34     building a project for a new architecture, particularly where the
35     syntax of the \texttt{Makefile} is very different (\eg
36     \texttt{WIN32}), was also practically impossible within the old
37     design.
38    
39     \ni In \scram~\scramvx, a new approach is taken by separating the
40     collection of build metadata from the generation of the
41     \texttt{Makefile} and compilation of the software units. Metadata in
42     this context refers to all information needed to build all products.
43     This includes package-level dependency requirements and all
44     information needed to use external tools (\eg libraries to link
45     against, compilation flags, pre-processor macros, library directories,
46     header file locations and runtime requirements).
47     Having parsed all necessary project documents, and the metadata
48     obtained and processed as required, data elements are substituted
49     into text templates to create the project \texttt{Makefile}. The data
50     elements include directory names, library names and names of
51     build targets all of which must be generated by \scram.
52     Of course, the text templates can be written so that the output is in
53     some other format than that for a \texttt{Makefile}.
54    
55     \section{BuildFiles and Build System Control}
56    
57     The \buildfile\ is the configuration document for the build system. For
58     each software unit, a \buildfile\ defines the external dependencies, the
59     internal dependencies (\ie the other software units that are used by
60     this one) and the interface. This interface supplies the product
61     provided by this software unit and the external/internal dependencies
62     required by it to any other software unit that requires it.
63     A software unit can be thought of as a product entity which is self-contained
64     and can be used either in linking to satisfy the dependencies of
65     a build product, or can be used to participate in some way on the
66     build process (for example, an executable used to process a source
67     file template to generate soucre files in a particular package).
68     Most of the time, a software unit is synonymous with a package (and
69     hence a shared library).
70    
71     \subsection{The Markup Syntax}
72     \index{\buildfile}\index{SCRAM!build files}
73    
74     Differing from other configuration documents in which the document
75     class defines the parse type, \scram\ knows about build files only
76     from their name, which must be \buildfile\footnote{\textit{note the uppercase letters!}.}.
77     A \buildfile\ only has a significance in a project area. The
78     valid tags that can be used in a \buildfile\ are listed below.
79     \index{\buildfile!valid basic markup tags}
80    
81     \begin{description}
82    
83     \item[\lbkt\texttt{use} name=\textit{unit}\rbkt]\mbox{}\\
84     Specify that there is a dependency on an external tool or a
85     local/external package \texttt{unit} (\eg a library, or everything the package
86     exports). External tools and packages are treated in the same way.
87     The interface to \textit{unit} must be defined via the
88     \lbkt\texttt{export}\rbkt\tagend{export} tag.
89    
90     \item[\lbkt\texttt{export}\rbkt\tagend{export}]\mbox{}\\
91     A software unit can be declared and exported to be used inside the
92     project or externally by another \scram\ project using the current
93     project as an external tool. The interface is defined by listing
94     the libraries (this can include those corresponding to a subset of a
95     subsystem) that are to be exported when the package (or subsystem)
96     is used. Typically, a package will export its' library, all
97     dependencies and specific compiler flags.
98    
99     \item[\lbkt\texttt{lib} name=\textit{libname} {[}\texttt{position}=\textit{first}{]}\rbkt]\mbox{}\\
100     Add a library to the list of libraries passed to the linker. The
101     optional \texttt{position} argument will move the library to the
102     front of the list of libraries as received by the linker (this
103     bypasses the automatic library ordering as determined by depth-first
104     sorting of the package metadata).
105    
106     \item[\lbkt\texttt{group} name=\textit{groupname}\rbkt]\mbox{}\\
107     Specify that all dependencies defined within the group
108     \textit{groupname} should be used by the current software unit.
109    
110     \item[\lbkt\texttt{include\_path} path=\textit{path}\rbkt]\mbox{}\\
111     Add the path \textit{path} to the global \texttt{INCLUDE} path
112     passed to the compiler. The include paths of individual
113     tools/packages will be ordered according to their dependency order
114     when added to the project or package \texttt{INCLUDE}.
115    
116     \item[\lbkt\texttt{libtype} type=\textit{type}\rbkt]\mbox{}\\
117     Specify the type of library that should be built.
118     This option overrides the project defaults but is currently unused
119     since libraries are \texttt{shared} only (\textit{support for debug libs to follow....}).
120    
121     \item[\lbkt\texttt{flags} \texttt{NAME}='definition'\rbkt]\mbox{}\\
122     Extra compiler flags can be added either in a tool description or in
123     a package. These flags will be added to the global compiler flags
124     and propageted to the build product defined in the \buildfile\ where
125     the declaration was made.
126     \end{description}
127    
128     \ni There are also tags for the supported build products:
129     \index{\buildfile!valid product markup tags}
130     \begin{description}
131    
132     \item[\lbkt\texttt{bin} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{bin}]\mbox{}\\
133     Specify an executable to build. The name of the executable can be
134     changed using the optional \textit{name} argument, otherwise the
135     name will be the same as \textit{filename}, less the file ending
136     (typically `.cpp'). More than one file can be compiled and linked to
137     make the executable. The file list can be specified like "main.cpp,
138     *.cc", "*.cpp" or "main.cpp, file.cc" (\ie file globs are
139     supported in a minimal way).
140    
141     All dependencies are contained between the opening and closing tags:
142     these dependencies are specific \textit{only} to this binary
143     executable. Dependencies or other metadata listed outside the
144     individual product tag will be passed to all products defined in the
145     \buildfile.
146    
147     \item[\lbkt\texttt{module} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{module}]\mbox{}\\
148     Specify a plug-in module to build. The name of the module can be
149     changed using the optional \textit{name} argument. How the plug-in
150     is defined can be customised within the project. By default (\ie
151     when using initial templates provided with \scram, copied to a
152     project \texttt{config} directory), a plug-in module is just a shared
153     library with all dependencies fully resolved when it is loaded (in
154     fact, there is not much difference between a plug-in module and a
155     shared library built by default in package \texttt{src} directories).
156    
157     \item[\lbkt\texttt{library} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{library}]\mbox{}\\
158     Define an additional library to build (in addition to the library
159     built automatically from the contents of \texttt{src} in the parent
160     package directory). This is usually used to build libraries for unit tests.
161    
162     \item[\lbkt\texttt{application} file=\textit{filename} {[}name=\textit{name}{]}\rbkt\tagend{application}]\mbox{}\\
163     Define an application. How one defines an application is
164     project-specific: it could be that an application differs from a
165     binary executable in only the compiler flags or the storage
166     location when released. This product type could also be used to
167     inject custom rules into the build system.
168    
169     % Not yet advertised: <<FIXME
170     %\item[\lbkt\texttt{skip}\rbkt\tagend{skip}]\mbox{}\\
171     % Indicate that a directory should be skipped. Comments can be entered
172     % between the tags which will be printed to STDOUT during building.
173     % \textit{Work in progress!!}.
174     %
175     \end{description}
176    
177    
178    
179     \subsection{The Project BuildFile}\label{sec:projectbuildfile}
180     \index{\buildfile!project}
181    
182     Global behaviour is controlled by the project \buildfile\ which is
183     located in the \texttt{config} directory in all \scram-managed
184     projects. This file is parsed first, before all others. The
185     storage areas for build products are defined in this \buildfile. More
186     importantly, instructions on what actions to perform throughout the
187     source code tree occur here using \texttt{ClassPath} directives.
188     The directives indicate which templates to apply at each directory
189     location based on matching the directory structure to the class path.
190     There are three keywords which refer to certain levels in a directory
191     tree and the definitions of these are fixed. The keywords are
192     \texttt{Project}, which refers to the top-level source directory
193     (usually \texttt{src}), \texttt{SubSystem}, which refers to the
194     directory one level up, and \texttt{Package}, which refers to a
195     subdirectory of a \texttt{SubSystem}. A project is not bound to a
196     structure in which there are always subsystems: in fact, it is
197     possible to have only a package level under the project \texttt{src}
198     directory. The advantage to having subsystems is that it becomes
199     possible to group packages together according to some common task (for
200     example): groups can then be defined which allow several independent
201     packages to become a single dependency unit which can be used by other
202     packages or executables at link-time.
203    
204     \ni Each directory which is an exact match to a keyword location will be
205     flagged as such and \textit{structure templates}\index{structure templates}
206     will be used to create the rules for compiling in these locations.
207     Of course, it is possible to redefine what actions to apply at the
208     subsystem, package or even project level, just by supplying a different
209     template or overriding an action in the project class path.
210     For libraries and executables (in general, build products), which are
211     associated to a subdirectory inside a package, \textit{product
212     templates} define the build targets.
213     Generated build rules for directories that do not fully match any
214     \texttt{ClassPath} will simply print a message that no action is required
215     at that location (\eg for an \texttt{include} or \texttt{interface} directory).
216     Complex build operations can be activated by modifying the
217     structure templates to do specific things for specific directories.
218     This could be for a \texttt{Documentation} subsystem, for example,
219     where the source code happens to be \texttt{html} or \LaTeX\
220     sources, or for single packages where source code must be generated
221     first before normal package build actions.
222    
223     \ni The syntax of the \texttt{ClassPath} tag is \index{\texttt{ClassPath}
224     tag}%
225     \begin{tagprint}
226     \lbkt\texttt{ClassPath}
227     path={[}\textit{pattern\_match}{]}+\textit{template\_type}/\ldots/\rbkt
228     \end{tagprint}
229    
230     \ni In the above case, the \texttt{template\_type} could be
231     \texttt{library}, \texttt{binary} \texttt{module} \etc. (the build
232     products). The template file name corresponding to the
233     \texttt{library} template type would be
234     \texttt{library\_template.tmpl} and this would be located in the
235     \texttt{config} directory of the project. Note that where there are
236     multiple \lbkt\texttt{ClassPath}\rbkt tags defined, it is always the
237     last tag that matches the current location that will be used.
238     See examples in Chapter~\ref{ch:examples}.
239    
240     \subsubsection{Product Storage Locations}
241     \index{\texttt{ProductStore} tag}%
242    
243     The storage locations are defined using
244     \begin{tagprint}
245     \lbkt\texttt{ProductStore} name=\textit{name} {[}type=\textit{type}
246     swap=\textit{t}{]}\rbkt
247     \end{tagprint}
248    
249     \ni The name \textit{name} is the name of the directory to be created
250     and the option \textit{type} can be set to \texttt{arch} so that an
251     architecture-dependent subdirectory for platform-specific products
252     will be added in the project area with the product directory
253     \textit{name} underneath. Without this type option the directory
254     \textit{name} will be created in the project area. Using the
255     \textit{swap} option reverses the order of the architecture-dependent
256     sub-directory and the name of the storage directory. That is, setting
257     swap to \textit{true} or \textit{t} will create the product directory
258     first with an architecture-dependent sub-directory
259     underneath.\footnote{This is the fixed behaviour for all SCRAM
260     releases prior to \scramvx.}
261    
262     \ni When \scram\ finds a \tagstart{ProductStore} tag, \texttt{Makefile}
263     a variable is set which can be used anywhere in a project makefile
264     when writing custom rules:
265     \begin{description}
266     \item[\texttt{SCRAMSTORENAME\_name}]\mbox{}\\
267     The path to the storage location \texttt{name} from the project area
268     directory (\eg \texttt{lib} or \texttt{lib/slc3\_ia32\_gcc323}).
269     \end{description}
270    
271     \ni This could be used, for example, where there may be a default rule for
272     copying all \texttt{include} files to a single location after building
273     all libraries. By default, all build products are copied to their
274     storage areas once built.
275    
276     \subsection{Local Metadata}\index{local metadata}\index{config/self}
277    
278     In order to propagate metadata from the local area to the source tree
279     at compile time, the local settings like \texttt{INCLUDE} or library paths in the
280     project templates plus the runtime environment, are defined in a local
281     file called \texttt{Self} found in the \texttt{config} directory\footnote{Note that this is unlike \scram\ V0 where
282     such settings were hard-coded in the top-level \buildfile.}.
283     This file behaves like a tool description file and is set up when the project
284     area is bootstrapped: all of the information defined within it is
285     automatically propagated to the whole tree.
286     In addition, all \texttt{scram} commands used to query tools
287     can also be used to inspect the local settings and changes can be made
288     at will with the tool set up by hand to propagate new settings.
289    
290     \subsection{Defining Groups}\label{sec:defininggroups}
291     \index{software units!defining a group}
292     A group can be used by adding a a \lbkt\texttt{group}\dots\rbkt statement
293     giving the name of the group to be included.
294    
295     \ni A group can be defined like this
296    
297     \small{
298     \begin{verbatim}
299     <define_group name=GA>
300     <use name=D>
301     <use name=zlib>
302     <Group name=XY>
303     <Flags CPPFLAGS="-DGROUP_GA">
304     </define_group>
305     \end{verbatim}
306     }\normalsize
307    
308     \ni in a subsystem \buildfile.
309    
310     \ni Note that it is not necessary to declare where the group can be
311     found (for example in another \scram\ project included as an
312     external product in the local configuration environment) since \scram\
313     determines this automatically. Duplicated/overridden groups will raise
314     a warning when \scram\ parses the \buildfile.
315    
316     \section{Configuring a New Package}
317     \label{sec:exportingsoftwareunits}
318     \index{software units}
319     \index{software units!defining the interface to}
320    
321     When a new package is added to a project, it is important that the
322     directory contains a \buildfile\ (note that it must be the package
323     directory and \textit{not} the \texttt{src} directory where the
324     sources are located that contains the \buildfile).
325     This \buildfile\ should have the following components:
326     \begin{description}
327     \item[\textbf{Declarations for all compile-time/link-time dependencies}]\mbox{}\\
328     The dependencies should be deduced from the \texttt{include}
329     statements in the package sources and a \lbkt\texttt{use}
330     name=\textit{unit}\rbkt should be added for each required unit
331     (external/internal package or external software product).
332     \item[\textbf{Export of the dependencies and package product}]\mbox{}\\
333     Every package providing a shared library should permit client software
334     units to use it at compile or link time. The
335     \lbkt\texttt{export}\rbkt\tagend{export} tag is used to define the
336     interface to the package and contains the full list of
337     \lbkt\texttt{use} name=\textit{unit}\rbkt statements (as required
338     by the package) and a \lbkt\texttt{lib} name=\textit{PackageName}\rbkt
339     statement with the name of the library provided by the package.
340     Of course, the library name should be omitted if the package does
341     not provide a shared library, having no \texttt{src} directory or
342     source files. This could be the case for abstract packages providing
343     only template \texttt{include} files to derived packages.
344     \end{description}
345    
346     \ni A typical package \buildfile\ will look like this:
347     \index{example of using \texttt{export} tag}%
348     \index{software units!exporting}%
349     \small{
350     \begin{verbatim}
351     <export>
352     <lib name=PX>
353     <use name=S/A>
354     <use name=B>
355     </export>
356    
357     <use name=S/A>
358     <use name=B>
359    
360     \end{verbatim}
361     }\normalsize
362     \index{defining a set of libraries for a package}
363     \ni Any other package or executable requiring this package would have
364     a statement \lbkt\texttt{use} name=\texttt{PX}\rbkt
365     in the \buildfile\ of the package/executable, where
366     \texttt{PX} is a path to the package in the
367     project (and hence could actually look like \lbkt\texttt{use}
368     name=\texttt{Subsystem/PX}\rbkt if the package is located in
369     a subsystem).
370    
371     \ni External projects and external tools can be used in the same way
372     using the \texttt{use} statement, \ie it is not necessary to qualify
373     the statement with a project name as with the older \scram\ syntax.
374    
375     \section{Build System Caches}
376    
377     Tracking changes to files in the source tree is achieved by caching
378     file timestamps and this is especially important for files that are
379     part of the project configuration. Caching is also extensively used
380     to store metadata as it is read from the \buildfile s
381     and tool description documents and to minimise subsequent re-parsing of
382     build data when the dependencies of a target change.
383    
384     \subsection{File Timestamp Cache}
385    
386     Build tools like \texttt{Make} keep track of relationships between
387     targets and the source files or headers needed to build those targets.
388     When a timestamp on a source file or header changes, \texttt{Make}
389     knows how to rebuild the target having determined that the target is
390     out of date with respect to those modified files. A limitation arises
391     when code is retrieved from a code repository such as one based on
392     \texttt{CVS}. Often, the timestamps on checked-out files are in the
393     past (usually the time of the commit) which implies that a target will
394     not be rebuilt if these files are added to the source code tree since
395     they will be older than the target.
396    
397     \ni To cope with situations like this, \scram\ employs a cache mechanism
398     which stores the timestamps of the \buildfile s and the directories in
399     the source code tree. Note that it is not necessary to store the
400     timestamp of every source file since it is the change of timestamp of
401     the parent directory that will change when files are added or removed.
402     Once populated (when \scram\ first performs a build), the existing
403     cache will be used to determine when the status of a directory or a
404     file has changed. Such a change to the status can be found by using
405     the \texttt{stat} command and comparing information like access mode
406     in addition to the timestamp. Changes in timestamp provoke a rebuild
407     of the updated package. If a \buildfile\ changes then the package is
408     rebuilt automatically, otherwise \scram\ will trigger a rebuild by
409     modifying the timestamp of the \buildfile\ of the package where the
410     files were added or removed first.
411    
412     \subsection{Metadata Cache}
413    
414     The actions to be taken by the build system when compiling and the
415     relationships between project software units are described in
416     \buildfile s located in the package directories in the source
417     code tree and the project configuration directory. Since it is
418     important to discover when metadata has changed, the timestamps of all
419     templates and the tool cache are also monitored. Since a change in a
420     tool setting or a template could affect the whole project, these
421     changes force a reparse of all \buildfile s and a complete rebuild of
422     the project.
423    
424     \section{Understanding the Build Templates}
425    
426     Many open source and commercial software projects use \texttt{make} to manage rebuilds
427     of large programs or collections of programs. Each build action is
428     defined using a rule which describes the steps that should be executed
429     to build the program. The rules are written in a file called a
430     \texttt{Makefile}. In very large projects, a \texttt{Makefile} can typically be
431     thousands of lines long and is very difficult to write
432     from scratch and manage manually; it is preferable to generate it
433     automatically using a more abstract process. Since \scram\ projects can
434     contain many occurrences of the same class of build object (\eg shared
435     libraries, executables), a complete \texttt{Makefile} can be
436     generated from templates which implement the rules for these different classes.
437     Thus, \scram\ can generate a \texttt{Makefile} for a complex project structure easily.
438    
439 sashby 1.2 \subsection{What is a Template?}
440 sashby 1.1
441     A template is a file which is written in the same way as a
442     \texttt{Makefile} except that some parts of the rules or environment
443     variables are inserted at the time of generation. A Perl toolkit is
444     included in the \scram\ sources (installed with \scram) which
445     facilitates the parsing of these templates.
446    
447 sashby 1.2 \begin{center}
448     \textit{More to be added here over time. Check snapshots pages!}
449     \end{center}
450    
451 sashby 1.1 % An Example Rule: <<FIXME
452    
453    
454    
455    
456     %% Build System
457     %% ------------
458    
459     %% The SCRAM version 1.0 build system consists of two stages. Firstly,
460     %% all external requirements and build actions are determined and cached
461     %% for every location in the project. Secondly, the processed metadata
462     %% (lists of libraries used, INCLUDE/LIBDIR paths and compiler flags) are
463     %% used by the template engine to produce Makefile fragments used by
464     %% gmake in the usual way. Templates can easily be adapted to support
465     %% both different kinds of compilations (e.g. using java) or different
466     %% architectures.
467    
468     %% The project data and the state of the project area are made persistent
469     %% so that, by checking the current state relative to the last cached
470     %% state, any new actions (e.g. a rebuild after a modification to a BuildFile)
471     %% can be affected very quickly without resorting to a traversal of the
472     %% entire directory tree.
473    
474    
475     %% - The Directory Cache
476    
477     %% In an empty project area, the first step is to populate the directory
478     %% cache with the timestamp and content information for the project
479     %% directory tree. All timestamps and file modes are stored for parent
480     %% and subdirectories in the ``src'' tree. Parameters for important files
481     %% used in the build system, such as templates and other files in
482     %% ``config'' are also recorded. The timestamps of other directories in
483     %% the project area (tmp and build product stores for example), are not
484     %% stored.
485     %% Once the timestamp information is stored, the cache (simply a Perl
486     %% object of type ``Cache::Cache'') is converted to a data structure
487     %% using the CPAN module ``Data::Dumper'' and written to a file
488     %% called ``.SCRAM/DirCache.db''. This is the directory cache.
489     %% Removing this file will result in a re-read of the directory tree and
490     %% repopulation of the cache on a subsequent ``scram build''. This can
491     %% happen intentionally via a ``scram build distclean''.
492    
493    
494    
495     %% - The Build Metadata Cache
496    
497     %% The next step is to collect the build metadata and store it in a cache
498     %% that is separate to the directory cache. This cache is also a Perl
499     %% object (the same methods are used for reading and writing caches in
500     %% all cache handling operations in SCRAM version 1.0) which is written
501     %% to a file called ``.SCRAM/ProjectCache.db''. The object is of type
502     %% ``BuildSystem::BuildDataStorage''.
503    
504     %% After checking that there is a project BuildFile in the config
505     %% directory, which is essential for obtaining the ClassPath data which
506     %% directs all build operations, each directory known to the directory
507     %% cache is scanned. Every path is assigned a ``BuildSystem::TreeItem''
508     %% object which is used to store all metadata required by the template
509     %% engine which will be used to generate a Makefile.
510    
511     %% Two main actions are performed for each path:
512     %% ---------------------------------------------
513    
514     %% i. if a BuildFile exists under ``path/Buildfile'', it is parsed and
515     %% this raw data is stored in the TreeItem object. Any groups defined in
516     %% the BuildFile are recorded in the build cache as a KNOWNGROUPS hash
517     %% with the name of the group as the key and the path to the buildFile
518     %% defining it as the value. This is required so that later the
519     %% KNOWNGROUPS hash can be used as a lookup table when resolving
520     %% <group name=X> type tags.
521     %% The raw metadata is itself stored as an object of type
522     %% ``BuildSystem::BuildFile'' in the TreeItem.
523    
524     %% ii. the path is compared to each known ClassPath (as obtained from
525     %% config/BuildFile) to establish what build actions should be applied
526     %% there. The directory is assigned the following based on a best match to a
527     %% ClassPath:
528    
529     %% - a class, i.e. a type of template to apply (Project, SubSystem,
530     %% Package, Library, Binary, etc..)
531    
532     %% - a classdir, the part of the path that matched the ClassPath
533    
534     %% - a suffix, the part of the path that dir *not* match
535    
536     %% Because package dependencies are in general stated in <use name=X>
537     %% tags as ``subsystem/package''and not ``src/subsystem/package'', the
538     %% path is converted to a DATAPATH by removing the ``src'', or rather,
539     %% by removing the start of the path that matches \$ENV{SCRAM\_SOURCEDIR}.
540     %% This data is stored in the TreeItem which is itself stored in the
541     %% build cache, accessed using the DATAPATH as the key. This provides an
542     %% efficient lookup table for accessing package-level metadata when
543     %% resolving build requirements.
544    
545     %% The parent directory and subdirectories (children) are recorded in
546     %% each TreeItem: this allows traversal from parent to children (a
547     %% directory ``branch'') and when coupled with the class information, can
548     %% permit simple determination of proper locations for build actions
549     %% (i.e. the actual path where the build of a certain class should be
550     %% applied) and greatly facilitates updating of build data when a
551     %% BuildFile somewhere in the branch has been modified. Thus it is not
552     %% necessary to reparse all BuildFiles after an modification to only one
553     %% of them.
554    
555    
556     %% Updating Metadata
557     %% -----------------
558    
559     %% At the start of a build, the directory cache is scanned and files that
560     %% are new or have newer timestamps are updated. Any BuildFiles that have
561     %% been updated are passed to the update routines of the
562     %% ``BuildSystem::BuildDataStorage'' object which is responsible for
563     %% collecting and managing the metadata.
564     %% The BuildFiles are re-parsed and the TreeItem for the path and any
565     %% subdirectories is updated. Only the BuildFiles that have been
566     %% modified are re-processed in this way and thus, only the relevant
567     %% makefile fragments, are remade. After updating, the directory cache is
568     %% marked as up-to-date and the build proceeds as normal.
569    
570    
571     %% From Metadata to Makefile
572     %% -------------------------
573    
574     %% Once all metadata is available, the template engine takes over. The
575     %% TemplateInterface object, which is a global, handles the interfacing
576     %% to the templates and their plugin modules.
577     %% The templates know how to obtain the metadata from the collected data
578     %% via the methods in the plugins. The TreeItem object provides all
579     %% needed metadata for a particular path: once the template engine has a
580     %% TreeItem, it uses methods in the PluginCore object to obtain
581    
582     %% - lists of libraries in link order;
583     %% - LIBDIR and INCLUDEDIR lists in link order;
584     %% - compiler flags and makefile text, correctly formatted;
585     %% - package dependencies
586    
587    
588    
589    
590     %% Using Private Plugins
591     %% ---------------------
592    
593    
594     %% Customizations can be provided by way of project Template plugin
595     %% modules. These modules are standard Perl modules, inheriting from the
596     %% SCRAM base plugin module class.
597    
598     %% For example, to override the main PluginCore module, one could do the
599     %% following:
600    
601     %% - Write a module called SCRAM::Plugins::MyPluginCore inheriting from
602     %% BuildSystem::Template::Plugins::PluginCore;
603    
604     %% - Add
605     %% [% USE MyPluginCore %]
606    
607     %% to the project templates. The build system will then have access to
608     %% customized build metadata.
609    
610     %% NB: Custom base class MUST be SCRAM::Plugins....
611    
612    
613    
614    
615    
616    
617     %% \section{Brief Description of the Build Process}\label{sec:buildprocess}
618     %% \index{description of the SCRAM build process}
619    
620     %% Assuming that the build area is clean (no builds have already been
621     %% performed) and that all tools are correctly set up, the actions taken
622     %% by \scram are as follows:
623    
624     %% \begin{itemize}
625     %% \item Initially, \scram checks the current location to test if it is a
626     %% project area. If it isn't, an error message like
627    
628     %% %% \small{\begin{verbatim}
629     %% %% SCRAM error: Unable to locate the top of local release. Exitting.
630     %% %% \end{verbatim}}\normalsize
631    
632     %% \ni will appear on \texttt{STDERR}.
633    
634    
635     %% \item Tool settings are read and a makefile stub is created which
636     %% contains all relevant path information for all tools in makefile
637     %% syntax. The makefile stub is called \texttt{clientmakefile} and
638     %% is located in the directory \texttt{tmp} in the project area.
639    
640     %% \item \scram parses the project \texttt{BuildFile}, creating a
641     %% corresponding makefile called \texttt{BuildFile.mk} located
642     %% under \texttt{tmp/config}. The \texttt{ClassPath} settings are
643     %% parsed to determine the appropriate build actions.
644    
645     %% \item A makefile is created according to the current directory.
646     %% The makefile is located in a directory under \texttt{tmp/src}
647     %% which has the same name as the current directory. This makefile
648     %% will be merged with appropriate makefiles determined from the
649     %% \texttt{ClassPath} and will include the makefile generated from
650     %% the project \texttt{BuildFile}. A list of subdirectories in
651     %% which build actions should occur is stored in a make variable
652     %% \texttt{\$(SUBDIRS)}-- the build order of various packages is
653     %% the same as the order in which the package directories appear in
654     %% this variable. To view the order, type \texttt{scram b echo\_SUBDIRS}.
655    
656     %% \item Before running \texttt{gmake}, the generated makefiles are
657     %% merged with \texttt{basics.mk} from the \scram sources (which
658     %% also includes \texttt{toolrules.mk} which contains the rules
659     %% for compiling different file types and how to create libraries,
660     %% binaries or modules).
661    
662     %% \item The build runs: any extra makefile statements or compiler
663     %% flags are passed directly to \texttt{gmake}. The working directory for
664     %% compilation is \texttt{/tmp/\$(SCRAM\_ARCH)}. Errors and warnings
665     %% are reported directly to standard output.
666    
667     %% \item The build products are moved from the working directory to
668     %% the product storage areas.
669    
670     %% \end{itemize}
671    
672    
673     %%% Local Variables:
674     %%% mode: latex
675     %%% TeX-master: "SCRAM-manual"
676     %%% End:
677    
678     %%____________________________________________________________________
679     %% End of BuildSystem.tex
680     %%____________________________________________________________________
681     %%