Reference implementation for a simple test iterator. A couple of concepts are demonstrated: 1) Using policy constraints and job arguments to restrict joblet execution to a specific resource, and 2) Scheduling joblets using a ParameterSpace.
> zos login --user vmmanager Please enter current password for 'vmmanager': Logged into grid as vmmanager > zos jobinfo --detail demoIterator Jobname/Parameters Attributes ------------------ ---------- demoIterator Desc: This example job is a reference for a simple test iterator. It is useful for demonstrating how policies and job args can be used to target the job to a particular resource. numJoblets Desc: joblets to run Type: Integer Default: 100 cmd Desc: Simple command to execute Type: String Default: os Desc: Regular expression match for Operating System Type: String Default: .* cpu Desc: Regular expression match for CPU architecture Type: String Default: .*
The files that make up the DemoIterator job include:
demoIterator # Total: 129 lines |-- demoIterator.jdl # 68 lines `-- demoIterator.policy # 61 lines
1 import time, random 2 3 # 4 # Add to the 'examples' group on deployment 5 # 6 if __mode__ == "deploy": 7 try: 8 jobgroupname = "examples" 9 jobgroup = getMatrix().getGroup(TYPE_JOB, jobgroupname) 10 if jobgroup == None: 11 jobgroup = getMatrix().createGroup(TYPE_JOB, jobgroupname) 12 jobgroup.addMember(__jobname__) 13 except: 14 exc_type, exc_value, exc_traceback = sys.exc_info() 15 print "Error adding %s to %s group: %s %s" % (__jobname__, jobgroupname, exc_type, exc_value) 16 17 class demoIteratorJob(Job): 18 19 def job_started_event(self): 20 print 'job_started_event' 21 self.completed = 0 22 23 # Launch the joblets 24 numJoblets = self.getFact("jobargs.numJoblets") 25 print 'Launching ', numJoblets, ' joblets' 26 27 pspace = ParameterSpace() 28 i = 1 29 while i <= numJoblets: 30 pspace.appendRow({'name':'joblet'+str(i)}) 31 i += 1 32 pspace.maxJobletSize = 1 33 self.schedule(demoIteratorJoblet,pspace,{}) 34 35 def joblet_completed_event(self, jobletnumber, node): 36 self.completed += 1 37 self.setFact("jobinstance.memo", "Tests run: %s" % (self.completed)) 38 39 class demoIteratorJoblet(Joblet): 40 41 def joblet_started_event(self): 42 print "Hi from joblet ", self.getFact("joblet.number") 43 time.sleep(random.random() * 15) 44 45 cmd = self.getFact("jobargs.cmd") 46 if len(cmd) > 0: 47 system(cmd) 48 49 # Example on more sophisticated exec 50 # e.g. e.signal("SIGUSR1") 51 """ 52 e = Exec() 53 e.setCommand(cmd) 54 #e.setStdoutFile("cmd.out") 55 e.writeStdoutToLog() 56 e.writeStderrtoLog() 57 #try: 58 e.execute() 59 #except: 60 #raise RetryElsewhere, "example error" 61 """
1 <policy> 2 <constraint type="accept" reason="Too busy for more work. Try again later!"> 3 <or> 4 <lt fact="job.instances.queued" value="4" /> 5 <contains fact="user.groups" value="superuser" /> 6 </or> 7 </constraint> 8 9 <constraint type="start" reason="Waiting on queue"> 10 <or> 11 <lt fact="job.instances.active" value="2" /> 12 <contains fact="user.groups" value="superuser" /> 13 </or> 14 </constraint> 15 16 <jobargs> 17 <fact name="numJoblets" 18 type="Integer" 19 description="joblets to run" 20 value="100" 21 visible="true" /> 22 23 <fact name="cmd" 24 type="String" 25 description="Simple command to execute" 26 value="" /> 27 28 <fact name="os" 29 type="String" 30 description="Regular expression match for Operating System" 31 value=".*" /> 32 33 <fact name="cpu" 34 type="String" 35 description="Regular expression match for CPU architecture" 36 value=".*" /> 37 </jobargs> 38 39 <constraint type="resource" reason="Does not match"> 40 <and> 41 <eq fact="resource.os.family" factvalue="jobargs.os" match="regexp" /> 42 <eq fact="resource.cpu.architecture" factvalue="jobargs.cpu" match="regexp"/> 43 44 <or> 45 <and> 46 <defined fact="env.VENDOR" /> 47 <eq fact="resource.os.vendor" factvalue="env.VENDOR" match="regexp" /> 48 </and> 49 <undefined fact="env.VENDOR" /> 50 </or> 51 </and> 52 </constraint> 53 54 <job> 55 <fact name="description" 56 type="String" 57 value="This example job is a reference for a simple test iterator. It is useful for demonstrating how policies and job args can be used to target the job to a particular resource." /> 58 </job> 59 </policy>
A representation of a running job instance.
Defines execution on the resource.
A representation of the matrix grid object, which provides operations for retrieving and creating grid objects in the system. MatrixInfo is retrieved using the built-in getMatrix() function. Write capability is dependent on the context in which getMatrix() is called. For example, in a joblet process on a resource, creating new grid objects is not supported.
A representation of Group grid objects. Operations include retrieving the group member lists and adding/removing from the group member lists, and retrieving and setting facts on the group.
Used to manage command line execution on resources.
Defines a parameter space to be used by the scheduler to create a Joblet set. A parameter space might consist of rows of columns or a list of columns that is expanded and can be turned into a cross product.
The following sections describe the DemoIterator job:
The deployment for the DemoIterator job is performed by lines 3-15 of demoIterator.jdl. When jobs are deployed into the grid, they can optionally be organized into and easy reference. In this case, the demoIterator job will be added to the group named examples, and will show up in the ZENworks Orchestrator Console in the Explorer panel at the location:
/ZOS/YOUR_GRID/Jobs/examples
For a general overview of how jobs are added to groups during deployment, see Walkthrough: Deploy a Sample Job
.
When the DemoIterator job receives a job_started_event, it creates a ParameterSpace object, and adds the number of rows as indicated by the value of the argument numJoblets (see lines 27-31 in demoIterator.jdl). A ParameterSpace object is like a spreadsheet, containing rows and columns of information that might all be given to one joblet or sliced up across many joblets at schedule time. In this case, the ParameterSpace is told that maxJobletSize is 1 (see line 32), meaning a joblet instance is created for each row in the ParameterSpace during job scheduling (see line 33).
Not shown in this example is the fact that a joblet can get access to this “spreadsheet” of information by calling self.getParameterSpace(), and calling hasNext() and next() to enumerate through each row of information. To learn more about putting information in a ParameterSpace object from a job and obtaining that information from the JobletParameterSpace object from a joblet, see ParameterSpace.
The resource that runs the joblet is determined from the resource constraint specified in lines 2-14 and 39-52 of demoIterator.policy, and from the values specified for the parameters os and cpu supplied on the command line. If these parameters are not specified on the command line, the default value for both is the regular expression .*, which means to include everything.
The constraints at lines 2-14 in demoIterator.policy define the work load for the resources. In this case, resources do not accept jobs if there are already four jobs queued up, and are not to run jobs if there are two or more jobs currently in progress.
To learn more about setting start, resource, or accept constraints in a policy file, see Defining Job Elements.
As the DemoIterator joblet is executed on a particular resource, it receives a joblet_started_event. When this happens, the DemoIterator joblet simply sleeps for a random amount of time to stagger the execution of the joblets, and then sends a command to the operating system, if one was supplied as a job argument. The command is executed on the target operating system using the built-in function system(), which is an alternative to using the more feature-rich class Exec.
For more information on sending commands to the operating system using the Exec class, see Exec .
After the joblet is finished running, a joblet_completed_event is sent to demoIteratorJob, which increments the variable completed, and posts the updated value to the job fact jobinstance.memo (see lines 35-37 in demoIterator.jdl). You can see the text for the memo displayed on the Job Log tab in the list of running jobs in the ZENworks Orchestrator Console.
For more information, see Starting and Stopping the ZENworks Orchestrator Console
.
Execute the following commands to deploy and run demoIterator.job:
Deploy demoIterator.job into the grid:
> zosadmin deploy demoIterator.job
Display the list of deployed jobs:
> zos joblist
should appear in this list.
Run the job on the first available resource without regard to OS or CPU, and use the default value for number of joblets, which is 100:
> zos run demoIterator
Run 10 joblets on Intel Windows resources, and launch the Notepad* application on each one:
> zos run demoIterator numJoblets=10 cmd=notepad os=Windows cpu=i386
Here is another example:
> zos run demoIterator numJoblets=3 cmd=pwd os=linux JobID: vmmanager.demoIterator.417 zos log vmmanager.demoIterator.417 job_started_event Launching 3 joblets [freeze] Hi from joblet 1 [freeze] /var/opt/novell/zenworks/zos/agent/node.default/freeze/vmmanager.demoIterator.417.1 [skate] Hi from joblet 0 [skate] /var/opt/novell/zenworks/zos/agent/node.default/skate/vmmanager.demoIterator.417.0 [melt] Hi from joblet 2 [melt] /var/opt/novell/zenworks/zos/agent/node.default/melt/vmmanager.demoIterator.417.2
Setting Constraints Using Policies (see Section 4.4, Policy Management and Section 5.0, Developing Policies).
Adding Jobs to Groups During Deployment (see how the JDL code can print the ID of group of jobs in factJunction.job).
quickie.job demonstrates how a job starts up multiple instances of a joblet on one or more resources. The Joblet class defines how a joblet is executed on a resource.
Setting default parameter values using policies
Configuring constraints in a policy file
Naming conventions for policy facts (see Section 3.1.1, Naming Orchestrator Job Files)
Facts provided by the ZENworks Orchestrator system that can be referenced within a JDL file
Using zos
Running commands using the Exec class