5.2 BuildTest Job Examples

There are many available facts that you can use in creating your jobs. If you find that you need specific kinds of information about a resource or a job, such as the load average of a user or the ID of a job or joblet, chances are that it is already available.

If a fact is not listed, you can create your own facts by creating a <fact> element in the job’s policy. In addition, you can create a fact directly in the JDL job code. The fact does not need to be an argument; you can simply create the fact and write it to the fact database for use in specifying parameters for other objects as well.

If you want to remember something from one loop to the next or make something available to other objects in the grid, you can set a fact with your own self-defined names.

This section explains a relatively simple working job that performs a set (100) regression tests on three different platform types. A number of assumptions have been made to simplify this example:

To demonstrate the possible functionality for this example, we have invented some policies that might apply to this example:

5.2.1 buildTest.policy Example

Policies are typically spread over different objects, entities, and groups on the system. However, to simplify the concept, we have combined all policies into this one example that is directly associated with the job.

The arguments available to the job are specified in the in the <jobargs> section (lines 1-11). When the job is run, job arguments are made available as facts to the job instance. The default values of these arguments can be overridden when the job is invoked.

1 <policy>
2    <jobargs>
3        <fact name="buildId"
4            type="String"
5            value="02-24-06 1705"
6            description="Build Id to show in memo field" />
7        <fact name="testlist"8            type="String"
9            value="/QA/testlists/nightly.dat"
10           description="Path to testlist to use in tests" />
11    </jobargs>

The <job> section (lines 12-25) defines facts that are associated with the job. These facts are used in other policies or by the JDL logic itself. Typically, these facts are aggregated from inherited policies.

12    <job>
13        <fact name="max_queue_size"
14            type="Integer"
15            value="10"
16            description="Limit of queued jobs. Any above this limit are not accepted." />
17        <fact name="max_licenses"
18            type="Integer"
19            value="5"
20             description="License count to limit number of jobs to run simultaneously. Any above this limit are queued." />
21        <fact name="max_test_failures"
22            type="Integer"
23            value="50"
24            description="To decide to end the job if the number of failures exceeds a limit" />
25    </job>

The <accept> (line 26), <start> (line 31), and <continue> (line 40) constraints control the job life cycle and implement the policy outlined in the example. In addition, allowances are made for “privileged users” (lines 28 and 33) to bypass the accept and start constraints.

26    <constraint type="accept" reason="Maximum number of queued jobs has been reached">
27        <or>
28            <defined fact="user.privileged_user" />
28            <lt fact="job.instances.queued" factvalue="job.max_queue_size" />
29        </or>
30    </constraint>
31    <constraint type="start">
32        <or>
33            <defined fact="user.privileged_user" />
34            <lt fact="job.instances.active" factvalue="job.max_licenses" />
35        </or>
36    </constraint>

The <resource> constraint (lines 37 and 38) ensures that only resources that are members of the buildtest group are used by this job. As noted earlier, this is normally associated with the user or user group.

37    <constraint type="resource">
38        <contains fact="resource.groups" value="buildtest" reason="No resources are in the buildtest group" />
39    </constraint>
40    <constraint type="continue" >
41        <lt fact="jobinstance.test_failures" factvalue="job.max_test_failures" reason="Reached test failure limit" />
42    </constraint>

NOTE:Typically, resource usage restrictions are specified on the user or user group and not on the job.

5.2.2 buildTest.jdl Example

The following example shows how additional resource constraints representing the three test platform types are specified in XML format. These also could have been specified in ZENworks Orchestrator Console.

Setting Resource Constraints

The annotated JDL code represents the job definition, consisting of base Python v2.1 (and libraries) as well as a large number of added Orchestrator operations that allow interaction with the Orchestrator Server:

1  import sys,os,time

2  winxp_platform = "<eq fact=\"resource.os.name\" value=\"Windows XP\" />"
3  win2k_platform = "<eq fact=\"resource.os.name\" value=\"Windows 2000\" />"
4  win2003_platform = "<eq fact=\"resource.os.name\" value=\"Windows 2003 Server\" />"

Lines 2-4 specify the resource constraints representing the three test platform types (Windows XP, Windows 2000, and Windows 2003) in XML format. Alternatively, these constraints could be set programmatically as options in the Orchestrator console.

The job_started_event in line 6 is the first event delivered to the job on the server.The logic in this method performs some setup and defines the parameter space used to iterate over the tests.

5 class BuildTest(Job):

6    def job_started_event(self):
7       self.total_counts = {"failed":0,"passed":0,"run":0}
8       self.setFact("jobinstance.test_failures",0)

9       self.testlist_fn = self.getFact("jobargs.testlist")
10      self.buildId = self.getFact("jobargs.buildId")
11      self.form_memo(self.total_counts)

12      # Form range of tests based on a testlist file
13      filerange = FileRange(self.testlist_fn)

Parameter spaces (lines 14-16) can be multidimensional but, in this example, they schedule three units of work (joblets), one for each platform type, each with a parameter space of the range of lines in the (optionally) supplied test file (lines 21, 24 and 27).

14      # Form ParameterSpace defining Joblet Splitting
15      pspace = ParameterSpace()
16      pspace.appendDimension("cmd",filerange)

17      # Form JobletSet defining execution on resources
18      jobletset = JobletSet()
19      jobletset.setCount(1)
20      jobletset.setJobletClass(BuildTestJoblet)

Within each platform test, a joblet is scheduled for each test line item on each different platform. After they are deployed, these joblets can be viewed individually in various Orchestrator UI portals.

21      # Launch tests on Windows XP
22      jobletset.setConstraint(winxp_platform)
23      self.schedule(jobletset)

24      # Launch tests on Windows 2000
25      jobletset.setConstraint(win2k_platform)
26      self.schedule(jobletset)

27      # Launch tests on Windows 2003
28      jobletset.setConstraint(win2003_platform)
29      self.schedule(jobletset)

The test_results_event in line 32 is a message handler that is called whenever the joblets send test results.

30 # Event invoked when a Joblet has completed running tests.
31 #
32 def test_results_event(self,params):
33      self.form_memo(params)

Creating a Memo Field

In line 37, the form_memo method is called to form an informational string to display the running totals for this test. These totals are displayed in the memo field for the job (visible in all portal, Orchestrator Console, command line, SDK, and Web interface tools). The memo field is accessed through setting the String fact jobinstance.memo in line 55.

34 #
35 # Update the totals and write totals to memo field.
36 #
37 def form_memo(self,params):
38     # total_counts will be empty at start
39     m = "Build Test BuildId %s " % (self.buildId)
40     i = 0
41     for key in self.total_counts.keys():
42     if params.has_key(key):
43         total = self.total_counts[key]
44         count = params[key]
45         total += count
46         printable_key = str(key).capitalize()
47        if i > 0:
48            m += ", "
48        else:
49            if len(m) > 0:
50        m+= ", "
51        m += printable_key + ": %d" % (total)
52        i += 1
53        self.total_counts[key] = total
54     self.setFact("jobinstance.test_failures",self.total_counts["failed"])
55     self.setFact("jobinstance.memo",m)

Joblet Definition

As previously discussed, a joblet is the logic that is automatically shipped to any resource employed by this job, as defined in lines 56-80. The joblet_started_event in line 60 mirrors the job_started_event (line 6) but, of course, runs on a different resource than the server.

The portion of the parameter space allocated to this joblet in line 65-66 represents some portion of the total test (parameter) space. The exact breakdown of this is under full control of the administrator/job. Essentially, the size of the “work chunk” in line 67 is a compromise between overhead and retry convenience.

In this example, each element of the parameter space (a test) in line 76 is executed locally and the exit code is used to determine pass or failure. (The exit code is often insufficient and additional logic must be added to analyze generated files, copy results, or to perform other tasks.) A message is then sent back to the server prior to completion with the result counts.

56 #
57 # Define test execution on a resource.
58 #59 class BuildTestJoblet(Joblet):
60    def joblet_started_event(self):
61        passed = 0
62        failed = 0
63        run = 0
64        # Iterate over parameter space assigned to this Joblet
65        pspace = self.getParameterSpace()
66        while pspace.hasNext():
67            chunk = pspace.next()
68            cmd = chunk["cmd"].strip()
69            rslt = self.run_cmd(cmd)
70            print "rslt=%d cmd=%s" % (rslt,cmd)
71            if rslt == 0:
72                passed +=1
73            else:
74                failed +=1
75            run += 1
76         self.sendEvent("test_results_event",{"passed":passed,"failed":failed,"run":run})
77         def run_cmd(self,cmd):
78            e = Exec()
79            e.setCommand(cmd)
80            return e.execute()

5.2.3 Packaging Job Files

A job package might consist of the following elements:

  • Job description language (JDL) code (the Python-based script containing the bits to control jobs).

  • An optional policy XML file, which applies constraints and other job facts to control jobs. Constraints and facts are discussed Constraints and Facts.

  • Any other associated executables or data files that the job requires.

Section 7.2, JDL Package provides more information about job elements.

5.2.4 Deploying Packaged Job Files

After jobs are created, you deploy .jdl or multi-element .job files to the Orchestrator Server by using any of the following methods:

5.2.5 Running Your Jobs

After your jobs are deployed, you can execute them by using the following methods:

  • Command Line Interface: Nearly all Orchestrator functionality, including deploying and running jobs, can be performed using the command line tool, as shown in the following example:

    More detailed CLI information is available in the zos command line tool.

  • Web Portal interface: After Orchestrator is installed, the management console is available online to edit, deploy, and run all jobs you create. This process is discussed in Using the Orchestrator User Portal to Run VM Jobs in the Novell ZENworks Orchestrator 1.3 Virtual Machine Management Guide.

  • Custom Client: The Orchestrator toolkit provides an SDK that provides a custom client that can invoke your custom jobs. This process is discussed in Section B.0, Orchestrator Client SDK.

  • Built-in Job Scheduler: The Orchestrator Server uses a built-in job scheduler to run deployed jobs. This process is discussed in Section 8.0, Job Scheduling.

  • From Other Jobs: As part of a job workflow, jobs can be invoked from within other jobs. You integrate these processes within your job scripts as described in the Section 8.0, Job Scheduling.

5.2.6 Monitoring Job Results

ZENworks Orchestrator enables you to monitor jobs by using the same methods outlined in Section 5.2.5, Running Your Jobs. The following figure shows examples of the status of the job ray.buildtest.18 using different monitoring interfaces:

CLI Job Monitoring

Figure 5-1 CLI Job Monitoring Example

Notice in the bottom section that job costing metrics also can be monitored, which are quite minimal in this example. More sophisticated job monitoring is possible. For more information, see Verification at the Command Line and Verification at the Jobs Monitor in the Novell ZENworks Orchestrator 1.3 Installation and Getting Started Guide.

Web-Based Control

Figure 5-2 User Portal Job Monitoring Example

In this example, the job memo field is displayed.

5.2.7 Debugging Jobs

As a brief introduction to the built-in debugging capabilities of ZENworks Orchestrator, the following figure shows how to find that the buildTest job was not able to find or match any resources, because resources were not added to the buildtest group as required by the policy.

Figure 5-3 Debugging Jobs Using the Orchestrator Console

The policy debugger shows the blocking constraint, and the tooltip gives the reason why. Dragging and dropping the resources to the required group allows the job to quickly continue and no restart was necessary.