ViewVC Help
View File | Revision Log | Show Annotations | Root Listing
root/cvsroot/COMP/CRAB/python/crab_help.py
Revision: 1.47
Committed: Tue Feb 26 22:23:03 2008 UTC (17 years, 2 months ago) by ewv
Content type: text/x-python
Branch: MAIN
CVS Tags: CRAB_2_1_2_pre1, CRAB_2_1_1, CRAB_2_1_1_pre3, CRAB_2_2_0_pre1, CRAB_2_1_1_pre1
Branch point for: CRAB_2_1_2_br, CRAB_2_1_1_pre2
Changes since 1.46: +19 -4 lines
Log Message:
Python config file, remove reliance on sed

File Contents

# User Rev Content
1 nsmirnov 1.1
2     ###########################################################################
3     #
4     # H E L P F U N C T I O N S
5     #
6     ###########################################################################
7    
8     import common
9    
10     import sys, os, string
11 spiga 1.34
12 nsmirnov 1.1 import tempfile
13    
14     ###########################################################################
15     def usage():
16 slacapra 1.43 print 'in usage()'
17 nsmirnov 1.1 usa_string = common.prog_name + """ [options]
18 slacapra 1.3
19     The most useful general options (use '-h' to get complete help):
20    
21 slacapra 1.26 -create -- Create all the jobs.
22     -submit n -- Submit the first n available jobs. Default is all.
23 slacapra 1.46 -status [range] -- check status of all jobs.
24 slacapra 1.20 -getoutput|-get [range] -- get back the output of all jobs: if range is defined, only of selected jobs
25 ewv 1.44 -publish [dbs_url] -- after the getouput, publish the data user in a local DBS instance
26 slacapra 1.4 -kill [range] -- kill submitted jobs
27     -clean -- gracefully cleanup the idrectory of a task
28     -testJdl [range] -- check if resources exist which are compatible with jdl
29 slacapra 1.27 -list [range] -- show technical job details
30 slacapra 1.4 -postMortem [range] -- provide a file with information useful for post-mortem analysis of the jobs
31 slacapra 1.20 -printId [range] -- print the job SID
32     -continue|-c [dir] -- Apply command to task stored in [dir].
33 slacapra 1.33 -h [format] -- Detailed help. Formats: man (default), tex, html, txt.
34 nsmirnov 1.1 -cfg fname -- Configuration file name. Default is 'crab.cfg'.
35 slacapra 1.4 -debug N -- set the verbosity level to N
36 nsmirnov 1.1 -v -- Print version and exit.
37    
38 slacapra 1.4 "range" has syntax "n,m,l-p" which correspond to [n,m,l,l+1,...,p-1,p] and all possible combination
39    
40 nsmirnov 1.1 Example:
41 slacapra 1.26 crab -create -submit 1
42 nsmirnov 1.1 """
43 slacapra 1.43 print usa_string
44 nsmirnov 1.1 sys.exit(2)
45    
46     ###########################################################################
47     def help(option='man'):
48     help_string = """
49     =pod
50    
51     =head1 NAME
52    
53     B<CRAB>: B<C>ms B<R>emote B<A>nalysis B<B>uilder
54    
55 slacapra 1.3 """+common.prog_name+""" version: """+common.prog_version_str+"""
56 nsmirnov 1.1
57 slacapra 1.19 This tool B<must> be used from an User Interface and the user is supposed to
58 fanzago 1.37 have a valid Grid certificate.
59 nsmirnov 1.1
60     =head1 SYNOPSIS
61    
62 slacapra 1.13 B<"""+common.prog_name+"""> [I<options>] [I<command>]
63 nsmirnov 1.1
64     =head1 DESCRIPTION
65    
66 fanzago 1.37 CRAB is a Python program intended to simplify the process of creation and submission into Grid environment of CMS analysis jobs.
67 nsmirnov 1.1
68 slacapra 1.3 Parameters for CRAB usage and configuration are provided by the user changing the configuration file B<crab.cfg>.
69 nsmirnov 1.1
70 fanzago 1.37 CRAB generates scripts and additional data files for each job. The produced scripts are submitted directly to the Grid. CRAB makes use of BOSS to interface to the Grid scheduler, as well as for logging and bookkeeping and eventually real time monitoring.
71 nsmirnov 1.1
72 slacapra 1.26 CRAB support any CMSSW based executable, with any modules/libraries, including the user provided one, and deals with the output produced by the executable. Up to version 1_2_1, also ORCA (and FAMOS) based executable were supported. CRAB provides an interface with CMS data discovery services (DBS and DLS), which are completely hidden to the final user. It also splits a task (such as analyzing a whole dataset) into smaller jobs, according with user requirements.
73 nsmirnov 1.1
74 slacapra 1.46 CRAB can be used in two ways: StandAlone and with a Server.
75     The StandAlone mode is suited for small task, of the order of O(100) jobs: it submits the jobs directly to the scheduler, and these jobs are under user responsibility.
76     Instead, in the Server mode, suited for larger task, the jobs are prepared locally and then passed to a dedicated CRAB server, which then interact with the scheduler on behalf of the user, including additional services, such as automatic resubmission, status caching, output retrival, and more.
77     The CRAB commands are exactly the same in both cases.
78    
79 slacapra 1.13 CRAB web page is available at
80    
81     I<http://cmsdoc.cern.ch/cms/ccs/wm/www/Crab/>
82 slacapra 1.6
83 slacapra 1.19 =head1 HOW TO RUN CRAB FOR THE IMPATIENT USER
84    
85     Please, read all anyway!
86    
87     Source B<crab.(c)sh> from the CRAB installation area, which have been setup either by you or by someone else for you.
88    
89     Modify the CRAB configuration file B<crab.cfg> according to your need: see below for a complete list: in particular set your jobtype (orca or famos) and fill the corresponding section. A template and commented B<crab.cfg> can be found on B<$CRABDIR/python/crab.cfg>
90    
91 ewv 1.44 ~>crab -create
92 slacapra 1.19 create all jobs (no submission!)
93    
94 spiga 1.25 ~>crab -submit 2 -continue [ui_working_dir]
95 slacapra 1.19 submit 2 jobs, the ones already created (-continue)
96    
97 slacapra 1.26 ~>crab -create -submit 2
98 slacapra 1.19 create _and_ submit 2 jobs
99    
100 spiga 1.25 ~>crab -status
101 slacapra 1.19 check the status of all jobs
102    
103 spiga 1.25 ~>crab -getoutput
104 slacapra 1.19 get back the output of all jobs
105    
106 ewv 1.44 ~>crab -publish
107     publish all user outputs in the DBS specified in the crab.cfg (dbs_url_for_publication) or written as argument of this option
108 fanzago 1.42
109 slacapra 1.20 =head1 RUNNING CMSSW WITH CRAB
110 nsmirnov 1.1
111 slacapra 1.3 =over 4
112    
113     =item B<A)>
114    
115 slacapra 1.26 Develop your code in your CMSSW working area. Do anything which is needed to run interactively your executable, including the setup of run time environment (I<eval `scramv1 runtime -sh|csh`>), a suitable I<ParameterSet>, etc. It seems silly, but B<be extra sure that you actaully did compile your code> I<scramv1 b>.
116 slacapra 1.3
117 ewv 1.44 =item B<B)>
118 slacapra 1.3
119 slacapra 1.20 Source B<crab.(c)sh> from the CRAB installation area, which have been setup either by you or by someone else for you. Modify the CRAB configuration file B<crab.cfg> according to your need: see below for a complete list.
120    
121     The most important parameters are the following (see below for complete description of each parameter):
122    
123     =item B<Mandatory!>
124    
125     =over 6
126    
127     =item B<[CMSSW]> section: datasetpath, pset, splitting parameters, output_file
128    
129     =item B<[USER]> section: output handling parameters, such as return_data, copy_data etc...
130    
131     =back
132    
133     =item B<Run it!>
134    
135 fanzago 1.37 You must have a valid voms-enabled Grid proxy. See CRAB web page for details.
136 slacapra 1.20
137     =back
138    
139 slacapra 1.19 =head1 HOW TO RUN ON CONDOR-G
140    
141     The B<Condor-G> mode for B<CRAB> is a special submission mode next to the standard Resource Broker submission. It is designed to submit jobs directly to a site and not using the Resource Broker.
142    
143 ewv 1.44 Due to the nature of this submission possibility, the B<Condor-G> mode is restricted to OSG sites within the CMS Grid, currently the 7 US T2: Florida(ufl.edu), Nebraska(unl.edu), San Diego(ucsd.edu), Purdue(purdue.edu), Wisconsin(wisc.edu), Caltech(ultralight.org), MIT(mit.edu).
144 slacapra 1.19
145     =head2 B<Requirements:>
146    
147     =over 2
148    
149     =item installed and running local Condor scheduler
150    
151     (either installed by the local Sysadmin or self-installed using the VDT user interface: http://www.uscms.org/SoftwareComputing/UserComputing/Tutorials/vdt.html)
152    
153     =item locally available LCG or OSG UI installation
154    
155 ewv 1.44 for authentication via Grid certificate proxies ("voms-proxy-init -voms cms" should result in valid proxy)
156 slacapra 1.19
157 ewv 1.44 =item set of the environment variable EDG_WL_LOCATION to the edg directory of the local LCG or OSG UI installation
158 slacapra 1.19
159     =back
160    
161     =head2 B<What the Condor-G mode can do:>
162    
163     =over 2
164    
165     =item submission directly to a single OSG site,
166    
167 ewv 1.44 the requested dataset has to be published correctly by the site in the local and global services
168 slacapra 1.19
169     =back
170    
171     =head2 B<What the Condor-G mode cannot do:>
172    
173     =over 2
174    
175     =item submit jobs if no condor scheduler is running on the submission machine
176    
177     =item submit jobs if the local condor installation does not provide Condor-G capabilities
178    
179     =item submit jobs to more than one site in parallel
180    
181     =item submit jobs to a LCG site
182    
183 fanzago 1.37 =item support Grid certificate proxy renewal via the myproxy service
184 slacapra 1.19
185     =back
186    
187     =head2 B<CRAB configuration for Condor-G mode:>
188    
189     The CRAB configuration for the Condor-G mode only requires changes in crab.cfg:
190 nsmirnov 1.1
191 slacapra 1.19 =over 2
192 slacapra 1.3
193 slacapra 1.19 =item select condor_g Scheduler:
194 slacapra 1.4
195 slacapra 1.19 scheduler = condor_g
196 slacapra 1.4
197 ewv 1.44 =item select the domain for a single OSG site:
198 nsmirnov 1.1
199 slacapra 1.19 CE_white_list = "one of unl.edu,ufl.edu,ucsd.edu,wisc.edu,purdue.edu,ultralight.org,mit.edu"
200 slacapra 1.4
201 slacapra 1.19 =back
202 slacapra 1.4
203     =head1 COMMAND
204    
205     =over 4
206    
207 slacapra 1.26 =item B<-create>
208 slacapra 1.4
209 slacapra 1.26 Create the jobs: from version 1_3_0 it is only possible to create all jobs.
210     The maximum number of jobs depens on dataset and splittig directives. This set of identical jobs accessing the same dataset are defined as a task.
211 slacapra 1.4 This command create a directory with default name is I<crab_0_date_time> (can be changed via ui_working_dir parameter, see below). Inside this directory it is placed whatever is needed to submit your jobs. Also the output of your jobs (once finished) will be place there (see after). Do not cancel by hand this directory: rather use -clean (see).
212     See also I<-continue>.
213    
214 slacapra 1.46 =item B<-submit [range]>
215 slacapra 1.4
216 slacapra 1.46 Submit n jobs: 'n' is either a positive integer or 'all' or a [range]. Default is all.
217     If 'n' is passed as argument, the first 'n' suitable jobs will be submitted. Please note that this is behaviour is different from other commenads, where -command N means act the command to the job N, and not to the first N jobs. If a [range] is passed, the selected jobs will be submitted.
218     This option must be used in conjunction with -create (to create and submit immediately) or with -continue (which is assumed by default), to submit previously created jobs. Failure to do so will stop CRAB and generate an error message. See also I<-continue>.
219 slacapra 1.4
220     =item B<-continue [dir] | -c [dir]>
221    
222     Apply the action on the task stored on directory [dir]. If the task directory is the standard one (crab_0_date_time), the more recent in time is taken. Any other directory must be specified.
223 slacapra 1.46 Basically all commands (but -create) need -continue, so it is automatically assumed. Of course, the standard task directory is used in this case.
224 slacapra 1.4
225 slacapra 1.26 =item B<-status>
226 nsmirnov 1.1
227 slacapra 1.26 Check the status of the jobs, in all states. If BOSS real time monitor is enabled, also some real time information are available, otherwise all the info will be available only after the output retrieval.
228 nsmirnov 1.1
229 slacapra 1.20 =item B<-getoutput|-get [range]>
230 nsmirnov 1.1
231 slacapra 1.20 Retrieve the output declared by the user via the output sandbox. By default the output will be put in task working dir under I<res> subdirectory. This can be changed via config parameters. B<Be extra sure that you have enough free space>. See I<range> below for syntax.
232 nsmirnov 1.1
233 fanzago 1.42 =item B<-publish [dbs_url]>
234    
235     Publish user output in a local DBS instance after retrieving of output. By default the publish uses the dbs_url_for_publication specified in the crab.cfg file, otherwise you can write it as argument of this option.
236    
237 slacapra 1.4 =item B<-resubmit [range]>
238 nsmirnov 1.1
239 fanzago 1.37 Resubmit jobs which have been previously submitted and have been either I<killed> or are I<aborted>. See I<range> below for syntax.
240     The resubmit option can be used only with CRAB without server. For the server this option will be implemented as soon as possible
241 nsmirnov 1.1
242 slacapra 1.4 =item B<-kill [range]>
243 nsmirnov 1.1
244 slacapra 1.4 Kill (cancel) jobs which have been submitted to the scheduler. A range B<must> be used in all cases, no default value is set.
245 nsmirnov 1.1
246 slacapra 1.4 =item B<-testJdl [range]>
247 nsmirnov 1.1
248 slacapra 1.4 Check if the job can find compatible resources. It's equivalent of doing I<edg-job-list-match> on edg.
249 nsmirnov 1.1
250 slacapra 1.20 =item B<-printId [range]>
251    
252 slacapra 1.46 Just print the job identifier, which can be the SID (Grid job identifier) of the job(s) or the taskId if you are using CRAB with the server or local scheduler Id.
253 slacapra 1.20
254 slacapra 1.4 =item B<-postMortem [range]>
255 nsmirnov 1.1
256 slacapra 1.46 Try to collect more information of the job from the scheduler point of view.
257 nsmirnov 1.1
258 slacapra 1.13 =item B<-list [range]>
259    
260     Dump technical informations about jobs: for developers only.
261    
262 slacapra 1.4 =item B<-clean [dir]>
263 nsmirnov 1.1
264 slacapra 1.26 Clean up (i.e. erase) the task working directory after a check whether there are still running jobs. In case, you are notified and asked to kill them or retrieve their output. B<Warning> this will possibly delete also the output produced by the task (if any)!
265 nsmirnov 1.1
266 slacapra 1.4 =item B<-help [format] | -h [format]>
267 nsmirnov 1.1
268 slacapra 1.4 This help. It can be produced in three different I<format>: I<man> (default), I<tex> and I<html>.
269 nsmirnov 1.1
270 slacapra 1.4 =item B<-v>
271 nsmirnov 1.1
272 slacapra 1.4 Print the version and exit.
273 nsmirnov 1.1
274 slacapra 1.4 =item B<range>
275 nsmirnov 1.1
276 slacapra 1.13 The range to be used in many of the above commands has the following syntax. It is a comma separated list of jobs ranges, each of which may be a job number, or a job range of the form first-last.
277 slacapra 1.4 Example: 1,3-5,8 = {1,3,4,5,8}
278 nsmirnov 1.1
279 ewv 1.44 =back
280 slacapra 1.6
281 slacapra 1.4 =head1 OPTION
282 nsmirnov 1.1
283 slacapra 1.6 =over 4
284    
285 slacapra 1.4 =item B<-cfg [file]>
286 nsmirnov 1.1
287 slacapra 1.4 Configuration file name. Default is B<crab.cfg>.
288 nsmirnov 1.1
289 slacapra 1.4 =item B<-debug [level]>
290 nsmirnov 1.1
291 slacapra 1.13 Set the debug level: high number for high verbosity.
292 nsmirnov 1.1
293 ewv 1.44 =back
294 slacapra 1.6
295 slacapra 1.5 =head1 CONFIGURATION PARAMETERS
296    
297 spiga 1.25 All the parameter describe in this section can be defined in the CRAB configuration file. The configuration file has different sections: [CRAB], [USER], etc. Each parameter must be defined in its proper section. An alternative way to pass a config parameter to CRAB is via command line interface; the syntax is: crab -SECTION.key value . For example I<crab -USER.outputdir MyDirWithFullPath> .
298 slacapra 1.5 The parameters passed to CRAB at the creation step are stored, so they cannot be changed by changing the original crab.cfg . On the other hand the task is protected from any accidental change. If you want to change any parameters, this require the creation of a new task.
299 slacapra 1.6 Mandatory parameters are flagged with a *.
300 slacapra 1.5
301     B<[CRAB]>
302 slacapra 1.6
303 slacapra 1.13 =over 4
304 slacapra 1.5
305 slacapra 1.6 =item B<jobtype *>
306 slacapra 1.5
307 slacapra 1.26 The type of the job to be executed: I<cmssw> jobtypes are supported
308 slacapra 1.6
309     =item B<scheduler *>
310    
311 slacapra 1.46 The scheduler to be used: I<glitecoll> is the more efficient grid scheduler and should be used. Other choice are I<glite>, same as I<glitecoll> but without bulk sumbission (and so slower) or I<condor_g> (see specific paragraph) or I<edg> which is the former Grid scheduler, which will be dismissed in some future
312     From version 210, also local scheduler are supported, for the tim being only at CERN. I<LSF> is the standard CERN local scheduler or I<CAF> which is LSF dedicated to CERN Analysis Facilities.
313 slacapra 1.5
314 mcinquil 1.35 =item B<server_mode>
315    
316     To use the CRAB-server mode put 1 in this field. If the server_mode key is equal to 0 crab works, as usual, in standalone way.
317    
318     =item B<server_name>
319    
320 slacapra 1.46 The name of the server that you want to use plus the path of the server storage (eg: hostname/data/cms/). The server available to users can be found from CRAB web page.
321 mcinquil 1.35
322 slacapra 1.5 =back
323    
324 slacapra 1.20 B<[CMSSW]>
325    
326     =over 4
327    
328 slacapra 1.22 =item B<datasetpath *>
329 slacapra 1.20
330 slacapra 1.22 the path of processed dataset as defined on the DBS. It comes with the format I</PrimaryDataset/DataTier/Process> . In case no input is needed I<None> must be specified.
331 slacapra 1.20
332 slacapra 1.22 =item B<pset *>
333 slacapra 1.20
334 slacapra 1.22 the ParameterSet to be used
335 slacapra 1.20
336 slacapra 1.26 =item I<Of the following three parameter exactly two must be used, otherwise CRAB will complain.>
337 slacapra 1.20
338 slacapra 1.22 =item B<total_number_of_events *>
339    
340 slacapra 1.26 the number of events to be processed. To access all available events, use I<-1>. Of course, the latter option is not viable in case of no input. In this case, the total number of events will be used to split the task in jobs, together with I<event_per_job>.
341 slacapra 1.22
342 slacapra 1.26 =item B<events_per_job*>
343 slacapra 1.22
344 slacapra 1.26 number of events to be accessed by each job. Since a job cannot cross the boundary of a fileblock it might be that the actual number of events per job is not exactly what you asked for. It can be used also with No input.
345 slacapra 1.22
346     =item B<number_of_jobs *>
347    
348     Define the number of job to be run for the task. The number of event for each job is computed taking into account the total number of events required as well as the granularity of EventCollections. Can be used also with No input.
349    
350     =item B<output_file *>
351    
352     the output files produced by your application (comma separated list).
353 slacapra 1.20
354 ewv 1.47 =item B<increment_seeds>
355    
356     Specifies a comma separated list of seeds to increment from job to job. The initial value is taken
357     from the CMSSW config file. I<increment_seeds=sourceSeed,g4SimHits> will set sourceSeed=11,12,13 and g4SimHits=21,22,23 on
358     subsequent jobs if the values of the two seeds are 10 and 20 in the CMSSW config file.
359    
360     See also I<preserve_seeds>. Seeds not listed in I<increment_seeds> or I<preserve_seeds> are randomly set for each job.
361    
362     =item B<preserve_seeds>
363    
364     Specifies a comma separated list of seeds to which CRAB will not change from their values in the user's
365     CMSSW config file. I<preserve_seeds=sourceSeed,g4SimHits> will leave the Pythia and GEANT seeds the same for every job.
366    
367     See also I<increment_seeds>. Seeds not listed in I<increment_seeds> or I<preserve_seeds> are randomly set for each job.
368    
369 slacapra 1.23 =item B<pythia_seed>
370    
371 ewv 1.47 This parameter is deprecated. See the documentation for I<increment_seeds> and I<preserve_seeds>.
372 slacapra 1.23
373 slacapra 1.24 =item B<vtx_seed>
374    
375 ewv 1.47 This parameter is deprecated. See the documentation for I<increment_seeds> and I<preserve_seeds>.
376 slacapra 1.24
377 slacapra 1.30 =item B<g4_seed>
378    
379 ewv 1.47 This parameter is deprecated. See the documentation for I<increment_seeds> and I<preserve_seeds>.
380 slacapra 1.30
381     =item B<mix_seed>
382    
383 ewv 1.47 This parameter is deprecated. See the documentation for I<increment_seeds> and I<preserve_seeds>.
384 slacapra 1.30
385     =item B<first_run>
386    
387     First run to be generated in a generation jobs. Relevant only for no-input workflow.
388    
389 slacapra 1.31 =item B<executable>
390 slacapra 1.30
391 slacapra 1.31 The name of the executable to be run on remote WN. The default is cmsrun. The executable is either to be found on the release area of the WN, or has been built on user working area on the UI and is (automatically) shipped to WN. If you want to run a script (which might internally call I<cmsrun>, use B<USER.script_exe> instead.
392 slacapra 1.30
393     =item I<DBS and DLS parameters:>
394    
395 slacapra 1.26 =item B<dbs_url>
396 slacapra 1.6
397 slacapra 1.40 The URL of the DBS query page. For expert only.
398 slacapra 1.13
399     =back
400    
401     B<[USER]>
402    
403     =over 4
404    
405 slacapra 1.6 =item B<additional_input_files>
406    
407 mcinquil 1.35 Any additional input file you want to ship to WN: comma separated list. These are the files which might be needed by your executable: they will be placed in the WN working dir. You don\'t need to specify the I<ParameterSet> you are using, which will be included automatically. Wildcards are allowed.
408 slacapra 1.6
409 slacapra 1.31 =item B<script_exe>
410    
411 ewv 1.44 A user script that will be run on WN (instead of default cmsrun). It\'s up to the user to setup properly the script itself to run on WN enviroment. CRAB guarantees that the CMSSW environment is setup (eg scram is in the path) and that the modified pset.cfg will be placed in the working directory, with name CMSSW.cfg . The user must ensure that a job report named crab_fjr.xml will be written. This can be guaranteed by passing the arguments "-j crab_fjr.xml" to cmsRun in the script. The script itself will be added automatically to the input sandbox.
412 slacapra 1.31
413 slacapra 1.6 =item B<ui_working_dir>
414    
415     Name of the working directory for the current task. By default, a name I<crab_0_(date)_(time)> will be used. If this card is set, any CRAB command which require I<-continue> need to specify also the name of the working directory. A special syntax is also possible, to reuse the name of the dataset provided before: I<ui_working_dir : %(dataset)s> . In this case, if eg the dataset is SingleMuon, the ui_working_dir will be set to SingleMuon as well.
416    
417 mcinquil 1.35 =item B<thresholdLevel>
418    
419     This has to be a value between 0 and 100, that indicates the percentage of task completeness (jobs in a ended state are complete, even if failed). The server will notify the user by e-mail (look at the field: B<eMail>) when the task will reach the specified threshold. Works just with the server_mode = 1.
420    
421     =item B<eMail>
422    
423 ewv 1.44 The server will notify the specified e-mail when the task will reaches the specified B<thresholdLevel>. A notification is also sended when the task will reach the 100\% of completeness. This field can also be a list of e-mail: "B<eMail = user1@cern.ch, user2@cern.ch>". Works just with the server_mode = 1.
424 mcinquil 1.35
425 slacapra 1.6 =item B<return_data *>
426    
427 mcinquil 1.41 The output produced by the executable on WN is returned (via output sandbox) to the UI, by issuing the I<-getoutput> command. B<Warning>: this option should be used only for I<small> output, say less than 10MB, since the sandbox cannot accomodate big files. Depending on Resource Broker used, a size limit on output sandbox can be applied: bigger files will be truncated. To be used in alternative to I<copy_data>.
428 slacapra 1.6
429     =item B<outputdir>
430    
431 mcinquil 1.41 To be used together with I<return_data>. Directory on user interface where to store the output. Full path is mandatory, "~/" is not allowed: the defaul location of returned output is ui_working_dir/res .
432 slacapra 1.6
433     =item B<logdir>
434    
435 mcinquil 1.41 To be used together with I<return_data>. Directory on user interface where to store the standard output and error. Full path is mandatory, "~/" is not allowed: the defaul location of returned output is ui_working_dir/res .
436 slacapra 1.6
437     =item B<copy_data *>
438    
439 slacapra 1.43 The output (only that produced by the executable, not the std-out and err) is copied to a Storage Element of your choice (see below). To be used as an alternative to I<return_data> and recomended in case of large output.
440 slacapra 1.6
441     =item B<storage_element>
442    
443     To be used together with I<copy_data>. Storage Element name.
444    
445     =item B<storage_path>
446    
447     To be used together with I<copy_data>. Path where to put output files on Storage Element. Full path is needed, and the directory must be writeable by all.
448    
449 slacapra 1.46 =item B<copyCommand>
450    
451     Only for LSF scheduer: allow to define the command to be used to copy the output to final location. Default is rfcp
452    
453 slacapra 1.6 =item B<register_data>
454    
455 fanzago 1.37 Not more supported.
456 slacapra 1.6
457     =item B<use_central_bossDB>
458    
459     Use central BOSS DB instead of one for each task: the DB must be already been setup. See installation istruction for more details.
460    
461     =item B<use_boss_rt>
462    
463     Use BOSS real time monitoring.
464    
465     =back
466    
467 slacapra 1.5 B<[EDG]>
468 nsmirnov 1.1
469 slacapra 1.13 =over 4
470 slacapra 1.6
471 slacapra 1.13 =item B<RB>
472 slacapra 1.6
473 slacapra 1.46 Which RB you want to use instead of the default one, as defined in the configuration of your UI. The ones available for CMS are I<CERN> and I<CNAF>. They are actaully identical, being a colelction of all RB/WMS available for CMS: the configuration files needed to change the broker will be automatically downloaded from CRAB web page and used.
474 slacapra 1.29 You can use any other RB which is available, if you provide the proper configuration files. Eg, for RB XYZ, you should provide I<edg_wl_ui.conf.CMS_XYZ> and I<edg_wl_ui_cmd_var.conf.CMS_XYZ> for EDG RB, or I<glite.conf.CMS_XYZ> for glite WMS. These files are searched for in the current working directory, and, if not found, on crab web page. So, if you put your private configuration files in the working directory, they will be used, even if they are not available on crab web page.
475     Please get in contact with crab team if you wish to provide your RB or WMS as a service to the CMS community.
476 slacapra 1.6
477 slacapra 1.14 =item B<proxy_server>
478    
479     The proxy server to which you delegate the responsibility to renew your proxy once expired. The default is I<myproxy.cern.ch> : change only if you B<really> know what you are doing.
480    
481 slacapra 1.26 =item B<role>
482    
483     The role to be set in the VOMS. See VOMS documentation for more info.
484    
485 slacapra 1.27 =item B<group>
486    
487     The group to be set in the VOMS, See VOMS documentation for more info.
488    
489 slacapra 1.28 =item B<dont_check_proxy>
490    
491     If you do not want CRAB to check your proxy. The creation of the proxy (with proper lenght), its delegation to a myproxyserver is your responsability.
492    
493 slacapra 1.6 =item B<requirements>
494    
495     Any other requirements to be add to JDL. Must be written in compliance with JDL syntax (see LCG user manual for further info). No requirement on Computing element must be set.
496    
497 slacapra 1.27 =item B<additional_jdl_parameters:>
498    
499     Any other parameters you want to add to jdl file: comma separated list, each
500 ewv 1.44 item B<must> be complete, including the closing ";".
501 mcinquil 1.35 With this field it\'s also possible to specify which WMS you want to use, adding the parameter "additional_jdl_parameters = WMProxyEndpoints ={"https://hostname:port/pathcode"};" where "hostname" is WMS\' name, the "port" generally is 7443 and the "pathcode" should be something like "glite_wms_wmproxy_server".
502 slacapra 1.27
503 slacapra 1.6 =item B<max_cpu_time>
504    
505     Maximum CPU time needed to finish one job. It will be used to select a suitable queue on the CE. Time in minutes.
506    
507     =item B<max_wall_clock_time>
508    
509     Same as previous, but with real time, and not CPU one.
510    
511     =item B<CE_black_list>
512    
513 slacapra 1.27 All the CE (Computing Element) whose name contains the following strings (comma separated list) will not be considered for submission. Use the dns domain (eg fnal, cern, ifae, fzk, cnaf, lnl,....)
514 slacapra 1.6
515     =item B<CE_white_list>
516    
517 slacapra 1.45 Only the CE (Computing Element) whose name contains the following strings (comma separated list) will be considered for submission. Use the dns domain (eg fnal, cern, ifae, fzk, cnaf, lnl,....). Please note that if the selected CE(s) does not contain the data you want to access, no submission can take place.
518 slacapra 1.27
519     =item B<SE_black_list>
520    
521 fanzago 1.37 All the SE (Storage Element) whose name contains the following strings (comma separated list) will not be considered for submission.It works only if a datasetpath is specified.
522 slacapra 1.27
523     =item B<SE_white_list>
524    
525 slacapra 1.45 Only the SE (Storage Element) whose name contains the following strings (comma separated list) will be considered for submission.It works only if a datasetpath is specified. Please note that if the selected CE(s) does not contain the data you want to access, no submission can take place.
526 slacapra 1.6
527     =item B<virtual_organization>
528    
529 mcinquil 1.35 You don\'t want to change this: it\'s cms!
530 slacapra 1.6
531     =item B<retry_count>
532    
533 fanzago 1.37 Number of time the Grid will try to resubmit your job in case of Grid related problem.
534 slacapra 1.6
535 slacapra 1.27 =item B<shallow_retry_count>
536    
537 fanzago 1.37 Number of time shallow resubmission the Grid will try: resubmissions are tried B<only> if the job aborted B<before> start. So you are guaranteed that your jobs run strictly once.
538 slacapra 1.27
539 slacapra 1.30 =item B<maxtarballsize>
540    
541     Maximum size of tar-ball in Mb. If bigger, an error will be generated. The actual limit is that on the RB input sandbox. Default is 9.5 Mb (sandbox limit is 10 Mb)
542    
543 slacapra 1.6 =back
544    
545 slacapra 1.46 B<[LSF]>
546    
547     =over 4
548    
549     =item B<queue>
550    
551     The LSF queue you want to use: if none, the default one will be used. For CAF, the proper queue will be autoamtically selected.
552    
553     =item B<resource>
554    
555     The resources to be used within a LSF queue. Again, for CAF, the right one is selected.
556    
557    
558     =back
559    
560 nsmirnov 1.1 =head1 FILES
561    
562 slacapra 1.6 I<crab> uses a configuration file I<crab.cfg> which contains configuration parameters. This file is written in the INI-style. The default filename can be changed by the I<-cfg> option.
563 nsmirnov 1.1
564 slacapra 1.6 I<crab> creates by default a working directory 'crab_0_E<lt>dateE<gt>_E<lt>timeE<gt>'
565 nsmirnov 1.1
566     I<crab> saves all command lines in the file I<crab.history>.
567    
568     =head1 HISTORY
569    
570 fanzago 1.37 B<CRAB> is a tool for the CMS analysis on the Grid environment. It is based on the ideas from CMSprod, a production tools implemented originally by Nikolai Smirnov.
571 nsmirnov 1.1
572     =head1 AUTHORS
573    
574     """
575     author_string = '\n'
576     for auth in common.prog_authors:
577     #author = auth[0] + ' (' + auth[2] + ')' + ' E<lt>'+auth[1]+'E<gt>,\n'
578     author = auth[0] + ' E<lt>' + auth[1] +'E<gt>,\n'
579     author_string = author_string + author
580     pass
581     help_string = help_string + author_string[:-2] + '.'\
582     """
583    
584     =cut
585 slacapra 1.19 """
586 nsmirnov 1.1
587     pod = tempfile.mktemp()+'.pod'
588     pod_file = open(pod, 'w')
589     pod_file.write(help_string)
590     pod_file.close()
591    
592     if option == 'man':
593     man = tempfile.mktemp()
594     pod2man = 'pod2man --center=" " --release=" " '+pod+' >'+man
595     os.system(pod2man)
596     os.system('man '+man)
597     pass
598     elif option == 'tex':
599     fname = common.prog_name+'-v'+common.prog_version_str
600     tex0 = tempfile.mktemp()+'.tex'
601     pod2tex = 'pod2latex -full -out '+tex0+' '+pod
602     os.system(pod2tex)
603     tex = fname+'.tex'
604     tex_old = open(tex0, 'r')
605     tex_new = open(tex, 'w')
606     for s in tex_old.readlines():
607     if string.find(s, '\\begin{document}') >= 0:
608     tex_new.write('\\title{'+common.prog_name+'\\\\'+
609     '(Version '+common.prog_version_str+')}\n')
610     tex_new.write('\\author{\n')
611     for auth in common.prog_authors:
612     tex_new.write(' '+auth[0]+
613     '\\thanks{'+auth[1]+'} \\\\\n')
614     tex_new.write('}\n')
615     tex_new.write('\\date{}\n')
616     elif string.find(s, '\\tableofcontents') >= 0:
617     tex_new.write('\\maketitle\n')
618     continue
619     elif string.find(s, '\\clearpage') >= 0:
620     continue
621     tex_new.write(s)
622     tex_old.close()
623     tex_new.close()
624     print 'See '+tex
625     pass
626     elif option == 'html':
627     fname = common.prog_name+'-v'+common.prog_version_str+'.html'
628     pod2html = 'pod2html --title='+common.prog_name+\
629     ' --infile='+pod+' --outfile='+fname
630     os.system(pod2html)
631     print 'See '+fname
632     pass
633 slacapra 1.33 elif option == 'txt':
634     fname = common.prog_name+'-v'+common.prog_version_str+'.txt'
635     pod2text = 'pod2text '+pod+' '+fname
636     os.system(pod2text)
637     print 'See '+fname
638     pass
639 nsmirnov 1.1
640     sys.exit(0)