demoIterator.job

Reference implementation for a simple test iterator. Several concepts are demonstrated: 1) Using policy constraints and job arguments to restrict joblet execution to a specific resource, 2) Scheduling joblets using a ParameterSpace, and 3) An example of executing a command on a resource.

Usage

> zos login --user zenuser
Please enter current password for 'zenuser':
Logged into grid as zenuser

> zos jobinfo --detail demoIterator
Jobname/Parameters    Attributes
------------------    ----------
demoIterator       Desc: This example job is a reference for a simple test
                         iterator. It is useful for demonstrating how policies
                         and job args can be used to target the job to a                         particular resource.

    numJoblets     Desc: joblets to run
                   Type: Integer
                Default: 100

    cmd            Desc: Simple command to execute
                   Type: String                Default:

    os             Desc: Regular expression match for Operating System                   Type: String
                Default: .*

    cpu            Desc: Regular expression match for CPU architecture                   Type: String
                Default: .*

Description

The files that make up the DemoIterator job include:

demoIterator                                # Total: 156 lines
|-- demoIterator.jdl                        #   79 lines
`-- demoIterator.policy                     #   77 lines

demoIterator.jdl

 1  # -----------------------------------------------------------------------------
 2  #  Copyright © 2008 Novell, Inc. All Rights Reserved.
 3  #
 4  #  NOVELL PROVIDES THE SOFTWARE "AS IS," WITHOUT ANY EXPRESS OR IMPLIED
 5  #  WARRANTY, INCLUDING WITHOUT THE IMPLIED WARRANTIES OF  MERCHANTABILITY,
 6  #  FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGMENT.  NOVELL, THE AUTHORS
 7  #  OF THE SOFTWARE, AND THE OWNERS OF COPYRIGHT IN THE SOFTWARE ARE  NOT LIABLE
 8  #  FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
 9  #  TORT, OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE
10  #  OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
11  # -----------------------------------------------------------------------------
12  #  $Id: demoIterator.jdl,v 1.4 2008/03/05 20:05:48 ray Exp $
13  # -----------------------------------------------------------------------------
14
15  import time, random
16
17  #
18  # Add to the 'examples' group on deployment
19  #
20  if __mode__ == "deploy":
21      try:
22          jobgroupname = "examples"
23          jobgroup = getMatrix().getGroup(TYPE_JOB, jobgroupname)
24          if jobgroup == None:
25              jobgroup = getMatrix().createGroup(TYPE_JOB, jobgroupname)
26          jobgroup.addMember(__jobname__)
27      except:
28          exc_type, exc_value, exc_traceback = sys.exc_info()
29          print "Error adding %s to %s group: %s %s" % (__jobname__, jobgroupname, exc_type, exc_value)
30
31
32  class demoIteratorJob(Job):
33
34       def job_started_event(self):
35            print 'job_started_event'
36            self.completed = 0
37
38            # Launch the joblets
39            numJoblets = self.getFact("jobargs.numJoblets")
40            print 'Launching ', numJoblets, ' joblets'
41
42            pspace = ParameterSpace()
43            i = 1
44            while i <= numJoblets:
45                pspace.appendRow({'name':'joblet'+str(i)})
46                i += 1
47            pspace.maxJobletSize = 1
48            self.schedule(demoIteratorJoblet,pspace,{})
49
50       def joblet_completed_event(self, jobletnumber, node):
51           self.completed += 1
52           self.setFact("jobinstance.memo", "Tests run: %s" % (self.completed))
53
54
55  class demoIteratorJoblet(Joblet):
56
57       def joblet_started_event(self):
58           print "Hi from joblet ", self.getFact("joblet.number")
59           time.sleep(random.random() * 15)
60
61           cmd = self.getFact("jobargs.cmd")
62           if len(cmd) > 0:
63               system(cmd)
64
65
66
67               # Example of more sophisticated exec
68               # e.g.  e.signal("SIGUSR1")
69               """
70               e = Exec()
71               e.setCommand(cmd)
72               #e.setStdoutFile("cmd.out")
73               e.writeStdoutToLog()
74               e.writeStderrToLog()
75               #try:
76               e.execute()
77               #except:
78                   #self.retry("retryable example error")
79               """

demoIterator.policy

 1  <!--
 2   *=============================================================================
 3   * Copyright (c) 2008 Novell, Inc. All Rights Reserved.
 4   *
     5   * NOVELL PROVIDES THE SOFTWARE "AS IS," WITHOUT ANY EXPRESS OR IMPLIED
 6   * WARRANTY, INCLUDING WITHOUT THE IMPLIED WARRANTIES OF MERCHANTABILITY,
 7   * FITNESS FOR A PARTICULAR PURPOSE, AND NON INFRINGMENT.  NOVELL, THE AUTHORS
 8   * OF THE SOFTWARE, AND THE OWNERS OF COPYRIGHT IN THE SOFTWARE ARE NOT LIABLE
 9   * FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
10   * TORT, OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE
11   * OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
12   *=============================================================================
13   * $Id: demoIterator.policy,v 1.2 2008/02/27 20:49:34 john Exp $
14   *=============================================================================
15   -->
16
17  <policy>
18     <constraint type="accept" reason="Too busy for more work. Try again later!">
19        <or>
20           <lt fact="job.instances.queued" value="4" />
21           <contains fact="user.groups" value="superuser" />
22        </or>
23     </constraint>
24
25     <constraint type="start" reason="Waiting on queue">
26       <or>
27         <lt fact="job.instances.active" value="2" />
28         <contains fact="user.groups" value="superuser" />
29       </or>
30     </constraint>
31
32     <jobargs>
33       <fact name="numJoblets"
34             type="Integer"
35             description="joblets to run"
36             value="100"
37             visible="true" />
38
39       <fact name="cmd"
40             type="String"
41             description="Simple command to execute"
42             value="" />
43
44       <fact name="os"
45             type="String"
46             description="Regular expression match for Operating System"
47             value=".*" />
48
49       <fact name="cpu"
50             type="String"
51             description="Regular expression match for CPU architecture"
52             value=".*" />
53     </jobargs>
54
55    <constraint type="resource" reason="Does not match">
56      <and>
57        <eq fact="resource.os.family" factvalue="jobargs.os" match="regexp" />
58        <eq fact="resource.cpu.architecture" factvalue="jobargs.cpu" match="regexp"/>
59
60        <or>
61          <and>
62             <defined fact="env.VENDOR" />
63             <eq fact="resource.os.vendor" factvalue="env.VENDOR" match="regexp" />
64          </and>
65          <undefined fact="env.VENDOR" />
66        </or>
67      </and>
68    </constraint>
69
70    <job>
71      <fact name="description"
72            type="String"
73            value="This example job is a reference for a simple test iterator. It is useful for demonstrating how policies and job args can be used to target the job to a particular resource." />
74    </job>
75
76  </policy>
77

Classes and Methods

Definitions:

Job

A representation of a running job instance.

Joblet

Defines execution on the resource.

MatrixInfo

A representation of the matrix grid object, which provides operations for retrieving and creating grid objects in the system. MatrixInfo is retrieved using the built-in getMatrix() function. Write capability is dependent on the context in which getMatrix() is called. For example, in a joblet process on a resource, creating new grid objects is not supported.

GroupInfo

A representation of Group grid objects. Operations include retrieving the group member lists and adding/removing from the group member lists, and retrieving and setting facts on the group.

Exec

Used to manage command line execution on resources.

ParameterSpace

Defines a parameter space to be used by the scheduler to create a Joblet set. A parameter space might consist of rows of columns or a list of columns that is expanded and can be turned into a cross product.

Job Details

The following sections describe the DemoIterator job:

zosadmin deploy

The deployment for the DemoIterator job is performed by lines 3-15 of demoIterator.jdl. When jobs are deployed into the grid, they can optionally be organized for grouping. In this case, the demoIterator job is added to the group named examples, and can be displayed in the ZENworks Orchestrator Console in the Explorer view at the location:

/ZOS/YOUR_GRID/Jobs/examples

For a general overview of how jobs are added to groups during deployment, see Walkthrough: Deploy a Sample Job in the Novell ZENworks Orchestrator 1.3 Installation and Getting Started Guide.

job_started_event

When the DemoIterator job receives a job_started_event, it creates a ParameterSpace JDL class and adds the number of rows as indicated by the value of the argument numJoblets (see lines 27-31 in demoIterator.jdl). A ParameterSpace object is like a spreadsheet, containing rows and columns of information that might all be given to one joblet or sliced up across many joblets at schedule time. In this case, the ParameterSpace is told that maxJobletSize is 1 (see line 32), meaning a joblet instance is created for each row in the ParameterSpace during job scheduling (see line 33).

Not shown in this example is the fact that a joblet can get access to this “spreadsheet” of information by calling self.getParameterSpace(), and calling hasNext() and next() to enumerate through each row of information. To learn more about putting information in a ParameterSpace object from a job and obtaining that information from the JobletParameterSpace object from a joblet, see ParameterSpace.

The resource that runs the joblet is determined from the resource constraint specified in lines 2-14 and 39-52 of demoIterator.policy, and from the values specified for the parameters os and cpu supplied on the command line. If these parameters are not specified on the command line, the default value for both is the regular expression .*, which means to include everything.

The constraints at lines 2-14 in demoIterator.policy define the work load for the resources. In this case, resources do not accept jobs if there are already four jobs queued up, and are not to run jobs if there are two or more jobs currently in progress.

To learn more about setting start, resource, or accept constraints in a policy file, see Defining Job Elements.

joblet_started_event

As the DemoIterator joblet is executed on a particular resource, it receives a joblet_started_event. When this happens, the DemoIterator joblet simply sleeps for a random amount of time to stagger the execution of the joblets and then sends a command to the operating system, if one was supplied as a job argument. The command is executed on the target operating system using the built-in function system(), which is an alternative to using the more feature-rich class Exec.

For more information on sending commands to the operating system using the Exec class, see Exec .

After the joblet is finished running, a joblet_completed_event is sent to demoIteratorJob, which increments the variable completed, and posts the updated value to the job fact jobinstance.memo (see lines 35-37 in demoIterator.jdl). You can see the text for the memo displayed on the Job Log tab in the list of running jobs in the ZENworks Orchestrator Console.

For more information, see Starting and Stopping the ZENworks Orchestrator Console.

Configure and Run

Execute the following commands to deploy and run demoIterator.job:

  1. Deploy demoIterator.job into the grid:

    > zosadmin deploy demoIterator.job
    
  2. Display the list of deployed jobs:

    > zos joblist
    

    demoIterator should appear in this list.

  3. Run the job on the first available resource without regard to OS or CPU, and use the default value for number of joblets, which is 100:

    > zos run demoIterator
    
  4. Run 10 joblets on Intel Windows resources, and launch the Notepad* application on each one:

    > zos run demoIterator numJoblets=10 cmd=notepad os=Windows cpu=i386
    

NOTE:If a resource with the matching OS is not available, the job remains in the “waiting” state.

Here is an example that runs the pwd command on three joblets on the Linux operating system:

> zos run demoIterator numJoblets=3 cmd=pwd os=linux
JobID: zenuser.demoIterator.417

zos log zenuser.demoIterator.417
job_started_event
Launching  3  joblets
[freeze] Hi from joblet  1
[freeze] /var/opt/novell/zenworks/zos/agent/node.default/freeze/zenuser.demoIterator.417.1
[skate] Hi from joblet  0
[skate] /var/opt/novell/zenworks/zos/agent/node.default/skate/zenuser.demoIterator.417.0
[melt] Hi from joblet  2
[melt] /var/opt/novell/zenworks/zos/agent/node.default/melt/zenuser.demoIterator.417.2

See Also