3.2 Scalability of the Primary Server

Understanding the scalability of the individual components that make up the ZENworks infrastructure is of the greatest importance. You need to understand the limitations and where you can expect to see performance degradation to ensure that you build an infrastructure that can perform well, regardless of the load that your end-user community places on it.

The first area that you need to consider is the scalability of the Primary Server. It is important to design your Primary Server placement based on the information you collected during your assessment phase and what you anticipate your overall design will require. You should design your infrastructure so that there are always Primary Servers available to service devices and the administrators that are managing the system:

3.2.1 Factors Influencing Scalability

The main physical factors that govern the scalability of the Primary Servers are:

  • RAM: The majority of operations are performed by zenserver and zenloader. Each of these services can consume approximately 1.2 GB of RAM on a 32-bit operating system. ZENworks 11 provides a 64-bit JVM for 64-bit operating systems and therefore these memory limitations are no longer applicable. For further details on how to tune the JVM and Tomcat for better performance, see Section 3.2.3, Achieving Scalability in the Real World.

  • Disk I/O: Disk I/O is used when serving content for applications and updates.

The minimum hardware recommendations are listed in Primary Server Requirements in the ZENworks 11 SP2 Server Installation Guide. If you can provide hardware that exceeds these recommendations, your system will perform better. Additional processing power and faster drives can make the systems more responsive, for example:

  • Using a quad core processor

  • Using 4 GB RAM or even more

  • Allocating as much disk space as you can (such as RAID 5 with separate physical drives to separate content and ZENworks Configuration Management from the OS)

There are other factors that you need to consider, including:

  • Device refresh frequency

  • Number of Primary Servers being used to deliver content to the managed devices (software, policies, images, patches, inventory collection, and so forth)

  • Number of administrators who have access to ZENworks Control Center

  • Frequency of uploading content in the ZENworks Content Repository

  • Number and frequency of reports run by administrators

3.2.2 Load Testing in the Novell SuperLab

ZENworks Configuration Management is tested in the Novell SuperLab in Provo, Utah to see how much load can be placed on the individual components, and more importantly, where the individual components start to break down and when performance is dramatically affected.

These tests provide insight on how far you can stretch the infrastructure design (for example, how many Primary Servers you need, based on the components and services you plan to deliver).

The following three tests show how Primary Servers react under different loads, and how quickly they can service individual requests when load is increased.

The tests included the following hardware and software:

  • A Dell 2950 Dual Quad Core 2.0Ghz, 4 GB RAM, RAID 5 (4 X 300 GB) server was used for the Primary Server.

  • The operating system was Windows Server 2003 Enterprise on a 64-bit device.

  • ZENworks Configuration Management shipping code was used.

    • The test deployed 100 MB of bundles (11 bundles) to an increasing number of devices.

  • All ZENworks Control Center settings used the default; after the 500-device test, retries were boosted to 800/10/20.

    • Three test passes for each test were run (for example, three test runs with 250 devices).

    • All devices were refreshed “simultaneously” (within 30 seconds).

    • The bundles were chained, with the first bundle being associated to a device group set to launch on refresh.

  • All tests were run on a full gigabit network.

Test 1: Average Time to Refresh

In this test, we used a single ZENworks Primary Server and refreshed all machines inside the lab at the same time. The devices then contacted the Primary Server at approximately the same time. We calculated the amount of time it took for the refresh of the ZENworks Adaptive Agent to complete. As the load increased, the average time also increased. In fact, the time to complete the refresh increased considerably at the 1,000 device mark. Between 1,000 and 2,000 devices, the amount of time it takes to complete the refresh more than doubled.

The Primary Server scaled well to this point; however, this does not mean that you should implement a single Primary Server to manage 3,000 managed devices. You must always consider the services that are being implemented, and most importantly, fault tolerance and load balancing.

The following graphic shows the average time to refresh managed devices:

Figure 3-1 Average Time to Refresh

Test 2: Average Time to Download

This test demonstrates the results of a download of 100 MB of bundle content, spread across 11 bundles that are chained together. The server load increased as the load (in terms of devices) increased.

The following graphic shows the number of minutes it took to download the content:

Figure 3-2 Average Time to Download

Test 3: Administrative Tasks Performed Under Load

This test used a series of administrative tasks performed on a server with no load and then with a load. These results demonstrate what to expect in a given environment when a certain load is placed on a Primary Server that is multi-tasking. This test provides insight into what you can expect if you do not properly distribute services across multiple Primary Servers. These are conditions that should not exist in a real-world environment; Novell runs these tests to see when the processes begin to break. A well-designed infrastructure should perform well for you regardless of the load you are placing on the servers.

The load consisted of 480 devices running the Daily Use Test.

The Daily Use Test environment in the SuperLab consisted of the following:

  • Single Server: Windows Server 2003 x86_64

  • Server hardware: Dell 2950, Quad Core 2.0 MHz, 8 GB RAM

  • External Microsoft SQL Server 2008 R2 database

Definition of the “Daily Use” included the following:

  • 24-hour period simulated in four hours

  • Three bundle pushes of 105 chained bundles (30 MB to 1 KB) per device

  • One full and three delta inventory scans per device

  • One patch distribution per device

  • Four device refreshes per device

  • Ten devices continually restoring images

This test environment stretched the limits of what the ZENworks Configuration Management system is able to achieve, giving us a good idea of where the system starts to reach its limitations. It comes close to a real-world simulation.

Table 3-1 Tests and Results.

Test Name

Test Description

Primary Server Under No Load

Primary Server Under Load

BOE Report

Run Predefined BOE report - Bundle Deployment Status (313 pages)

15 seconds

18 seconds

Inventory Report

Run canned report “Devices By Machine / Login Name”

7 seconds

8 seconds

Policy

Create policy, assign it to 60 devices, and perform a quicktask on all 60 devices to get the policy

7 minutes 43 seconds

7 minutes 45 seconds

Multicast

Create Multicast bundle and multicast to 60 devices

4 minutes 20 seconds

4 minutes 27 seconds

Bundle

Create an MSI Bundle of OpenOffice, assign it to 60 devices, and refresh all 60 by using a quicktask

25 minutes 30 seconds

1 hour 10 minutes 17 seconds

When the server is under load, and we created an MSI bundle and assigned it to 60 devices, there was a large increase in the amount of time it took to complete the task. In this situation, you should have a dedicated server for ZENworks Control Center so that there would be virtually no impact on performance.

The ZENworks Control Center is used by administrators to perform administrative tasks, including the management of content within the system. For environments that are managed by a small number of administrators (one or two), accessing ZENworks Control Center from a Primary Server that is also performing work might not cause performance issues. For larger implementations (several thousand devices, large amounts of daily content activity, and several administrators), having a dedicated Primary Server in place for ZENworks Control Center resolves potential performance issues.

3.2.3 Achieving Scalability in the Real World

Section 3.2.2, Load Testing in the Novell SuperLab discussed testing in the Novell SuperLab to determine the limits of the ZENworks system. Scalability, on the other hand, is achieved through the proper placement of services, a well thought-out design, and the proper configuration of services within the ZENworks Configuration Management system itself.

For example, even though we know that a Primary Server can manage 3,000 devices, you should never deploy only one Primary Server in a 3,000-device environment. For this situation, we recommend the following as a starting point:

  • Three Primary Servers to manage load and build a system that is fault tolerant.

  • A dedicated Database Server using Sybase, Oracle, or Microsoft SQL Server.

This system should be further enhanced by considering the following:

  • Using an L4 switch to manage fault tolerance and load balancing.

  • Using DNS and aliases for managing the load placed on Primary Servers during deployment and registration.

  • Using Closest Server Rules in ZENworks Locations to designate certain servers for specific functions (content, collection, etc.) and to exclude servers from specific functions, or all functions. If a Primary Server is not listed in the Closest Server Rule of a location for a particular role, then devices do not attempt to connect to it for that feature.

  • Using a dedicated Primary Server for reporting.

  • Using a dedicated Primary Server for imaging.

  • Using a dedicated Primary Server for ZENworks Control Center. This is done mainly to control where the administrators upload content to ensure that the load is dedicated to a single Primary Server.

  • Using Satellite devices for distribution of content.

The Primary Servers are the heart of your ZENworks Configuration Management environment. You want to protect these systems from major disruption. Primary Servers can be used for distribution of content, but this needs to be factored in to your design.

Some of the major factors that you need to consider with ZENworks Configuration Management and Primary Servers are the following:

  • Each Primary Server can handle 1,000 concurrent connections.

  • Each Primary Server can manage 3,000 devices that are registered with the ZENworks Management Zone.

  • A ZENworks Management Zone can scale to 40,000 devices. This has been validated in the SuperLab and is what Novell recommends as the upper limit to the Management Zone size.

We also recommend that Primary Servers and the Database Server be on the same network, in the same data center. We do not recommend spanning WAN links with Primary Servers because replication of the Content Repository can cause utilization issues. Placement of services in a multi-site environment is done by utilizing the Satellite role, which is discussed in more detail in Section 3.3, Scalability of Satellite Devices.

3.2.4 Tuning the JVM and Tomcat

To utilize the 64-bit JVM (Java Virtual Machine) of ZENworks 11 SP2 to the best possible extent, you can increase the following values beyond the defaults:

Maximum HTTP/HTTPS Tomcat Threads

ZENworks uses HTTPS threads to service incoming configuration, and authentication Web requests, and uses HTTP threads for content, and collection requests. By default, these threads are set to 200 for HTTP and 200 for HTTPS. This was originally a limitation of the 32-bit JVM, but it is no longer necessary, because ZENworks now uses the 64-bit JVM.

These values are found in the server.xml file in the following locations:

  • Windows: %ZENWORKS_HOME%\Share\tomcat\conf

  • Linux: /opt/novell/zenworks/share/tomcat/conf

To achieve further performance increases from your Primary Servers, you can change the thread values:

  1. Stop the ZENworks services (ZENmonitor, ZENserver, and ZENloader)

  2. Open the server.xml file for the operating system on which the Primary Server is running.

  3. Locate the line with the text <Connector port=”80”, and change the value for maxThreads as desired.

  4. Locate the line with the text<Connector port=”443”, and change the value for maxThreads as desired. Ensure that the values for the two ports are the same.

  5. Save the file.

  6. Start the ZENworks services again.

For the testing completed in our engineering labs, we used a server with 16 GB of RAM and found that increasing the thread counts to 600 each was the optimal point in terms of performance.

We suggest that you increment the threads in stages of 100 for both HTTP and HTTPS in order to monitor your performance.

Increasing Heap Memory Size for the ZENworks Services

ZENworks 11 SP2 uses the 64-bit JVM, so you can tune the Java memory allocations to allow ZENworks services to utilize more than the default 1 GB of memory. The sample settings in the following procedures were tested in the engineering lab on a server with 16 GB of RAM.

Configuring the Maximum Heap Memory Size on a Linux Primary Server

  1. Stop the ZENworks services (ZENmonitor, ZENserver, and ZENloader).

  2. Using a Linux text editor such as vi or gedit, create or modify the /opt/novell/zenworks/bin/zenserversettings.sh file with the following content:

    JAVA_MIN_HEAP="-Xms1024m"
JAVA_MAX_HEAP="-Xmx1024m"
JAVA_MIN_PERM_SIZE="-XX:PermSize=128m"
JAVA_MAX_PERM_SIZE="-XX:MaxPermSize=256m"
JAVA_THREAD_STACK_SIZE="-Xss1m"

    IMPORTANT:Change the JAVA_MAX_HEAP value from "-Xmx1024m” to “-Xmx<desired memory in MB>m”. For example, "-Xmx4096m" to configure 4GB.

    When you increment the JAVA_MAX_HEAP value, ensure that you make the same changes for the ZENserver and the ZENloader services, so that they always have the same value. For example, if you configure the ZENserver heap memory as 4096 (4GB), you must configure the same for ZENloader.

    You need to reserve RAM for the operating system, so change the values in increments that make sense.

  3. Save the file.

  4. Using a Linux text editor such as vi or gedit, create or modify the /opt/novell/zenworks/bin/zenloadersettings.sh file with the following content:

    JAVA_MIN_HEAP="-Xms256m"
JAVA_MAX_HEAP="-Xmx1024m"
JAVA_MIN_PERM_SIZE="-XX:PermSize=128m"
JAVA_MAX_PERM_SIZE="-XX:MaxPermSize=128m"

    IMPORTANT:Change the DEFAULT_MAX_HEAP value from "-Xmx1024m” to “-Xmx<desired memory in MB>m”. For example, “-Xmx4096m" to configure 4GB.

    When you increment the JAVA_MAX_HEAP value, ensure that you make the same changes for the ZENserver and the ZENloader services, so that they always have the same value. For example, if you configure the ZENserver heap memory as 4096 (4GB), you must configure the same for ZENloader.

    You need to reserve RAM for the operating system, so change the values in increments that make sense.

  5. Save the file.

  6. Run the following commands:

    chown zenworks.zenworks /opt/novell/zenworks/bin/zenserversettings.sh

    chown zenworks.zenworks /opt/novell/zenworks/bin/zenloadersettings.sh

  7. Run the following commands:

    chmod 755 /opt/novell/zenworks/bin/zenserversettings.sh

    chmod 755 /opt/novell/zenworks/bin/zenloadersettings.sh

  8. Start the ZENworks services.

Configuring the Maximum Heap Memory Size on a Windows Primary Server

To configure the maximum heap memory size on a Windows Primary Server:

  1. Stop the ZENworks services (ZENmonitor, ZENserver, and ZENloader).

  2. Run zenserverw.

  3. On the Java tab, change the Maximum memory pool from 1024 to a higher value. For example, 4096.

    You need to reserve RAM for the operating system, so change the values in increments that make sense.

  4. Click Apply, then click OK.

  5. Run zenloaderw.

    The Novell ZENworks Server Properties dialog box is displayed.

  6. On the Java tab, change the Maximum memory pool from 1024 to a desired value. For example 4096.

    When you increment the values, ensure that you make the same changes for both services, so they always have the same value. For example, if you increment the ZENserver settings to 4096, you must use the same settings for ZENloader.

  7. Click Apply, then click OK.

  8. Start the ZENworks services.