4.1 Tuning the ZENworks Primary Servers

Although with ZENworks 11 SP3 the server settings are tuned during an upgrade or new installation, it is valuable to understand the default settings for your server and how you can further tune these settings if required. The table below shows the default values based on the memory installed in your server:

Primary Server RAM

Tomcat Threads

Heap Memory

HTTP

HTTPS

ZENServer

ZENLoader

4 GB

200

200

2 GB

1 GB

8 GB

600

600

4 GB

2 GB

12 GB

600

1000

6 GB

3 GB

16 GB

600

1000

8 GB

4 GB

To utilize the 64-bit JVM (Java Virtual Machine) of ZENworks 11 SP3 to the best possible extent, you can increase the following values beyond the defaults:

4.1.1 Maximum HTTP / HTTPS Tomcat Threads

ZENworks uses HTTPS threads to service incoming configuration and authentication web requests; it uses HTTP threads for content and collection requests. By default, these threads are set to 200 for HTTP and 200 for HTTPS. This was originally a limitation of the 32-bit JVM, but it is no longer necessary because ZENworks now uses the 64-bit JVM only.

These values together dictate the maximum number of simultaneous web requests that the server will service. When these are exhausted, the server will begin returning a 503 error to agents indicating that it is busy. This will cause the busy retry logic to be executed. You can monitor the overall threads in use from the Diagnostics tab of the ZENworks Primary Server and, if necessary, adjust appropriately.

These values are found in the server.xml file in the following locations:

  • Windows: %ZENWORKS_HOME%\Share\tomcat\conf

  • Linux: /opt/novell/zenworks/share/tomcat/conf

To achieve further performance increases from your Primary Servers, you can change the thread values:

  1. Stop the ZENworks services (ZENmonitor, ZENserver, and ZENloader).

  2. Open the server.xml file for the operating system on which the Primary Server is running.

  3. Locate the line with the text <Connector port="80" and change the value for maxThreads as required. You should consider increasing this number if the server is servicing a large number of content or collection requests.

    As the ports in ZENworks can be customized, if you do not find a connector on port 80, search for the port number that you normally use to connect for content.

  4. Locate the line with the text<Connector port="443" and change the value for maxThreads as desired. Ensure that the values for the two ports are the same. You should consider increasing this number if the server is servicing a large number of configuration or authentication requests.

    As the ports in ZENworks can be customized, if you don’t find a connector on port 80, search for the port number that you normally use to connect to ZENworks Control Center.

  5. Save the file.

  6. Start the ZENworks services again.

If you increase the number of threads, you will also want to monitor the Java heap usage of the ZENServer service. If this begins to approach the allocated amount of memory, you will also want to look at increasing the heap size, or reducing the threads.

4.1.2 Heap Memory Size for ZENworks Services

ZENworks 11 SP3 uses the 64-bit JVM, so you can tune the Java memory allocations to allow ZENworks services to utilize more memory if required. The sample settings in the following procedures were tested in the engineering lab on a server with 16 GB of RAM.

Configuring the Maximum Heap Memory Size for ZENworks Server Service on a Linux Primary Server

The ZENworks Server service is the Tomcat instance used by the ZENworks Primary Server. It services agent requests, ZENworks Control Center, and the zenworks-setup page. If you find that ZENworks Server runs out of heap space, you can increase the heap size by doing the following:

  1. Stop the ZENworks services (ZENMonitor, ZENServer, and ZENLoader).

  2. Using a Linux text editor such as vi or gedit, create or modify the /opt/novell/zenworks/bin/zenserversettings.sh file with the following content:

      JAVA_MIN_HEAP="-Xms1024m"  JAVA_MAX_HEAP="-Xmx1024m"  JAVA_MIN_PERM_SIZE="-XX:PermSize=128m"  JAVA_MAX_PERM_SIZE="-XX:MaxPermSize=256m"  JAVA_THREAD_STACK_SIZE="-Xss1m"
    

    IMPORTANT:Change the JAVA_MAX_HEAP value from -Xmx1024m to -Xmx<desired memory in MB>m. For example, -Xmx4096m to configure 4GB.

    You need to reserve RAM for the operating system, so change the values in increments that make sense.

  3. Save the file.

  4. Run the following commands:

    chown zenworks.zenworks /opt/novell/zenworks/bin/zenserversettings.sh 
    chown zenworks.zenworks /opt/novell/zenworks/bin/zenloadersettings.sh 
    
  5. Restart the ZENworks Server service by running /etc/init.d/novell-zenserver restart.

Configuring the Maximum Heap Memory Size for ZENworks Loader Service on a Linux Primary Server

The ZENwork Loader service is responsible for performing background operations such as subscribing to external zones, packaging content, storing collection data in the database, processing audit events, and much more. If you have a server that is doing a lot of these background tasks and you see the heap memory decrease for the ZENLoader service, you can increase the heap size by doing the following:

  1. Using a Linux text editor such as vi or gedit, create or modify the /opt/novell/zenworks/bin/zenloadersettings.sh file with the following content:

    JAVA_MIN_HEAP="-Xms256m"
    	JAVA_MAX_HEAP="-Xmx1024m"
    JAVA_MIN_PERM_SIZE="-XX:PermSize=128m"  JAVA_MAX_PERM_SIZE="-XX:MaxPermSize=128m"
    

    IMPORTANT:Change the JAVA_MAX_HEAP value from -Xmx1024m to -Xmx<desired memory in MB>m. For example, -Xmx4096m to configure 4GB.

    You need to reserve RAM for the operating system, so change the values in increments that make sense.

  2. Save the file.

  3. Run the following commands:

    chmod 755 /opt/novell/zenworks/bin/zenserversettings.sh 
    chmod 755 /opt/novell/zenworks/bin/zenloadersettings.sh 
    
  4. Start the ZENworks Loader service by running /etc/init.d/novell-zenloader restart.

Configuring the Maximum Heap Memory Size for ZENworks Server Service on a Windows Primary Server

The ZENworks Server service is the Tomcat instance used by the ZENworks Primary Server. It services agent requests, ZENworks Control Center, and the zenworks-setup page. If you find that ZENworks Server runs out of heap space, you can increase the heap size by doing the following:

  1. Stop the ZENworks services (ZENMonitor, ZENServer, and ZENLoader).

  2. Run zenserverw.

  3. On the Java tab, change the Maximum memory pool from 1024 to a higher value. For example, 4096.

    You need to reserve RAM for the operating system, so change the values in increments that make sense.

  4. Click Apply, then click OK.

  5. Start the ZENworks services.

Configuring the Maximum Heap Memory Size for ZENworks Loader Service on a Windows Primary Server

The ZENworks Loader service is responsible for performing background operations such as subscribing to external zones, packaging content, storing collection data in the database, processing audit events and much more. If you have a server that is doing a lot of these background tasks and if you see the heap memory get low for the ZENLoader service, you can increase the heap size by doing the following:

  1. Stop the ZENworks services (ZENMonitor, ZENServer, and ZENLoader).

  2. Run zenloaderw.

  3. On the Java tab, change the Maximum memory pool from 1024 to a desired value. For example 4096.

    You need to reserve RAM for the operating system, so change the values in increments that make sense.

  4. Click Apply, then click OK.

  5. Start the ZENworks services.

4.1.3 Limiting Heap Memory Size for ZENServer Web Requests

In large ZENworks deployment zones, if you experience “OutofMemoryExceptions” frequently, you can limit the web service requests coming to the ZENworks primary server by configuring maximum permitted heap limit.

To configure maximum permitted heap limit:

  1. Stop the ZENworks services on the Primary Servers.

  2. Add the following line within the Engine properties of the server.xml file:

    <Valve className="com.novell.zenworks.tomcat.ZENRequestValve" debug="false" maxUsedHeapPercent="90"/>

    Path for server.xml on Linux: /opt/novell/zenworks/share/tomcat/conf/

    Path for server.xml on Windows: %ZENWORKS_HOME%\share\tomcat\conf\

    NOTE:It is not recommended to configure the maxUsedHeapPercent value for the ZENworks Primary Server, which is used for accessing ZENworks Control Center. This could result in blank pages during peak loads that exceed the configured permitted heap limit.

  3. You can enable valve debug logging by configuring the debug value as True.

  4. Start the ZENworks services.

    Valve logs can be found at following location:

    Linux: /opt/novell/zenworks/share/tomcat/logs/

    Windows: %ZENWORKS_HOME%\share\tomcat\logs\

NOTE:When the heap memory usage by the ZENServer process reaches the maximum permitted heap, web service requests coming from managed devices will be rejected with status 503. You can find the message Request would exceed the maximum permitted heap limit in valve debug logs.

Even after configuring the maxUsedHeapPercent value, if you observe that the ZENworks Primary Server is not responsive or is in a busy state for a long time, revert the configuration and consult the Novell Technical Support (NTS) team.

4.1.4 Connection Pool Tuning for the ZENworks Primary Server

Database connections are often expensive to create because of the overhead of establishing a network connection and initializing a database connection session in the back-end database. Hence, ZENworks uses the c3p0 library which helps in maintaining a pool of connections to the database to perform at optimal levels. Like any other pooling mechanism, you will need to tune the connections to your needs. This section will explain the various options available for your tuning needs.

Prior to ZENworks 11 SP3 there was minimal tuning you could do to the connection pools used by the ZENworks Server and Loader modules. For the most part you were limited to only a Maximum Pool Size and a Minimum Pool Size that impacted all of the connection pools used by ZENworks. With ZENworks 11 SP3 you now have significantly more flexibility over the connection pools used by ZENworks services.

Understanding the Different ZENworks Connection Pools

In ZENworks 11 SP3 you can tune each of the following connection pools independently:

  • ZENLoader Connection Pool: This connection pool is used for most of the ZENworks Loader functions that require access to the database, including storing inventory database; recording content repository data during packaging, patch subscriptions, zone-to-zone subscriptions, and all other tasks performed by the ZENworks Loader; and storing the device message and status.

  • ZENLoader Batch Connection Pool: This connection pool is used by ZENworks Loader for storing device message and status information such as install or launch bundle status. This status is stored in batches and, as such, uses a separate connection pool.

  • ZENServer Connection Pool: This connection pool is used by the ZENworks Server process to retrieve data from the database. This is most heavily used by Configuration and Authentication requests.

  • InvUI Connection Pool: This connection pool is used when browsing inventory data on devices or when running canned reports from ZENworks Control Center.

  • Audit ZENServer Connection Pool: This connection pool is used by the ZENworks Server process to retrieve data from the Audit database. This is used mainly by ZENworks Control Center servers.

  • Audit ZENLoader Connection Pool: This connection pool is used by the ZENworks Server process to store data in the Audit database. This is mainly used by Collection servers.

ZENworks Database Connection Pool Parameters (zdm.xml)

ZENworks 11SP3 provides the following parameters for tuning the connection pools used to connect to the ZENworks database.

  • MinPoolSize: This value specifies the minimum number of connections available for a given connection pool. For instance, if this value is set to 5, then the connection pool will always have at least 5 connections to the database.

  • ZENServer.MinPoolSize: This value specifies the minimum number of connections available for a ZENServer connection pool. If this value is not set, then the MinPoolSize parameter value is used to determine the minimum ZENServer connection pool size.

  • ZENLoader.MinPoolSize: This value specifies the minimum number of connections available for the ZENLoader connection pool. If this value is not set, then the MinPoolSize parameter value is used to determine the minimum ZENLoader connection pool size.

  • Batch.MinPoolSize: This value specifies the minimum number of connections available for a the ZENLoader batch connection pool. If this value is not set then the MinPoolSize parameter value is used to determine the minimum ZENLoader batch connection pool size.

  • MaxPoolSize: This value specifies the maximum number of connections available for a given connection pool. The pool size will grow dynamically from the minimum size to the maximum size as required by the requests being made to the server. Once the request volume reduces, the number of threads will reduce after the specified period of time.

  • ZENServer.MaxPoolSize: This value specifies the maximum number of connections available to the ZENServer connection pool. If this value is not set, the maximum number is determined by the MaxPoolSize parameter.

  • ZENLoader.MaxPoolSize: This value specifies the maximum number of connections available to the ZENLoader connection pool. If this value is not set, then the maximum number is determined by the MaxPoolSize parameter.

  • Batch.MaxPoolSize: This value specifies the maximum number of connections available to the ZENLoader batch connection pool. If this value is not set, then the maximum number is determined by the MaxPoolSize parameter.

  • MaxIdleTimeExcessConnections: This value is set in number of seconds and controls how long before the server should end unused connections. This is useful for situations such as first thing in the morning when a large number of clients are logging in, causing a lot of database activity. After this initial login, the number of requests typically drops dramatically. After the specified number of seconds passes, the database connections that are no longer required for active work will be closed.

  • ZENServer.MaxIdleTimeExcessConnections: This value controls the amount of time before the excess threads used by the ZENServer connection pool. If this value is not present in the zdm.xml file, then the MaxIdleTimeExcessConnections value is used to determine this value.

  • ZENLoader.MaxIdleTimeExcessConnections: This value controls the amount of time before the excess threads are used by the ZENLoader connection pool. If this value is not present in the zdm.xml file, then the MaxIdleTimeExcessConnections value is used to determine this value.

  • Batch.MaxIdleTimeExcessConnections: This value controls the amount of time before the excess threads are used by the ZENLoader batch connection pool. If this value is not present in the zdm.xml file, then the MaxIdleTimeExcessConnections value is used to determine this value.

The columns to the right of the parameter identify the recommended parameter value based on the role(s) the server will be performing in your organization. If a server will be acting in multiple roles, you should plan on setting the parameter to the highest value in the row.

XML Parameter

Config/Auth Server

Collection Server

ZCC Server

Other/Default

MinPoolSize

5

5

5

5

ZENServer.MinPoolSize

5

n/a

n/a

n/a

ZENLoader.MinPoolSize

5

n/a

n/a

n/a

Batch.MinPoolSize

5

n/a

n/a

n/a

MaxPoolSize

900*

10

100

100

ZENServer.MaxPoolSize

900*

10

100

n/a

ZENLoader.MaxPoolSize

5

30

10

n/a

Batch.MaxPoolSize

10

45

10

n/a

MaxInvUIPoolSize

10

10

100

100

MaxIdleTimeExcessConnections

300

120

120

120

ZENServer.MaxIdleTimeExcessConnections.

300

120

n/a

n/a

ZENLoader.MaxIdleTimeExcessConnections

120

300

n/a

n/a

Batch.MaxIdleTimeExcessConnections

120

n/a

n/a

n/a

NOTE:* 900 is the ideal pool size for Novell’s Performance and Scale testing. This is roughly 2/3 of the total number of permitted zenserver threads. If you have additional RAM and the database can support additional connections, you can increase the heaps, threadpools, and database connections accordingly, to further increase scalability.

These parameters can be set in the zdm.xml file in the <conf folder>\datamodel folder where the format of the entry is <entry key="paramater name">parameter value</entry>.

After you have determined the appropriate configuration for each server, you will need to calculate the total potential number of connections that all Primary Servers might require to the database. This will allow you to properly configure the number of connections the database will allow. To do this, simply add up the total number of Max Connections for each Primary Server in your environment. Then, if you are using ZENworks Reporting, add an additional 100 connections. An example is listed below.

Server Name

Roles

Max Number of Threads

Server1

Config/Auth

925 (900 ZENServer.MaxPool + 5 ZENLoader.MaxPool + 10 Batch.MaxPool + 10 InvUIMaxUIPool)

Server2

Config/Auth

925 (900 ZENServer.MaxPool + 5 ZENLoader.MaxPool + 10 Batch.MaxPool + 10 InvUIMaxUIPool)

Server3

Collection

95 (10 ZENServer.MaxPool + 30 ZENLoader.MaxPool + 45 Batch.MaxPool + 10 InvUIMaxUIPool)

Server4

Collection

95 (10 ZENServer.MaxPool + 30 ZENLoader.MaxPool + 45 Batch.MaxPool + 10 InvUIMaxUIPool)

Server5

ZCC

220 (900 ZENServer.MaxPool + ZENLoader.MaxPool + Batch.MaxPool+InvUIMaxUIPool)

Server6

Config/Auth/Collection

985 (900 ZENServer.MaxPool + 30 ZENLoader.MaxPool + 45 Batch.MaxPool + 10 InvUIMaxUIPool)

Adding these values, you get 925+925+95+95+220+985 = 3245 connections. If you are using ZENworks Reporting, you will get a maximum of 3245+100 = 3345 connections to the database. You should ensure that your database is configured to permit at least so many connections.

ZENworks Audit Database Connection Pool Parameters (zenaudit.xml)

Additionally, a separate set of connection pools is available when connecting to the ZENworks Audit database. Unlike the ZENworks database connection pools, there is only a ZENServer and a ZENLoader connection pool for Audit. The parameter names are the same as the ZENworks database equivalents. The table below shows the recommended values based on server role:

XML Parameter

Config Server

Collection Server

ZCC Server

Other/Default

ZENServer.MinPoolSize

5

n/a

n/a

n/a

ZENLoader.MinPoolSize

5

n/a

n/a

n/a

MaxPoolSize

10

10

50

100

ZENServer.MaxPoolSize

20

10

50

n/a

ZENLoader.MaxPoolSize

10

20

10

n/a

ZENServer.MaxIdleTimeExcessConnections.

n/a

120

n/a

n/a

ZENLoader.MaxIdleTimeExcessConnections

n/a

300

n/a

n/a

These parameters can be set in the zenaudit.xml file in the <conf folder>\datamodel folder, where the format of the entry is <entry key="paramater name">parameter value</entry>.

To calculate the required number of audit database connections, follow the same model as that shown at the end of ZENworks Database Connection Pool Parameters (zdm.xml).

4.1.5 Tuning the Threads Allocated to Loader Storer Processes

In ZENworks 11 SP3 you can tune each of the processes responsible to store data (status, audit, patch, inventory) by increasing or decreasing the number of threads allocated to the process. If you have more threads, it means that the server will attempt to store more items in parallel. In most cases, the default values should be acceptable. However, if you find that you are seeing a backlog of files in the collection folder on the server, consider increasing these values.

To tune the threads for these processes:

  1. To adjust the thread pool, open the file for the relevant loader module. The following files can be adjusted:

    • inventorystorer.xml: Controls the number of simultaneous inventory scans that can be stored in the database.

    • auditeventstorer.xml: Controls the number of simultaneous audit events that can be stored in the database.

    • statusstorer.xml: Controls the number of simultaneous status events that can be stored in the database.

    • AddPatchlinkAnalyzeReporting.xml: Controls the number of patch DAU results that can be stored simultaneously.

    • messageprocessor.xml: Controls the number of messages that can be stored simultaneously.

    These files can be found in the following locations:

    Windows: %zenworks_home%\conf\loader\inventorystorer.xml

    Linux: /etc/opt/novell/zenworks/loader/inventorystorer.xml

  2. Below the file you should find an entry that sets the ThreadPoolSize. For inventory, this looks similar to the following entry:

    <Parameter Name="InventoryStorerThreadPoolSize">10</Parameter>

    Set this value to the desired number of threads.

  3. Save the file and restart the ZENworks Loader service.

NOTE:If you increase the number of threads that you allocate for storing, it might also be necessary to increase the MaxPoolSize parameter for the Connection Pool used by the storer that you are increasing. To do this, see ZENworks Database Connection Pool Parameters (zdm.xml) and ZENworks Audit Database Connection Pool Parameters (zenaudit.xml).