5.1 Adding Servers to the Vertica, Kafka and ZooKeeper Cluster

This section details the procedure to move Vertica and its components from a single node cluster to a multiple node cluster.

NOTE:After adding Kafka and Vertica servers to their respective clusters, ensure that you run the Configure action to allocate appropriate memory for these services, on each of the servers added in the cluster. For more information, see Managing Vertica Memory Requirements.

5.1.1 Prerequisites

The prerequisites to be followed before adding servers to their individual clusters, are:

  • Ensure that the hardware and software configuration of the servers that are to be added are the same as the configuration of the existing server within that cluster.

  • Although there is no limit to the number of servers to be added in a cluster, ZENworks recommends a cluster size of 3 servers for ZooKeeper, Kafka and Vertica.

  • For Kafka, before adding servers, the Kafka cluster should already be configured. For more information, see Procedure to Configure Vertica and Kafkaon a Single Server.

  • For Vertica, before adding servers, the Vertica cluster should already be enabled. For more information, see Procedure to Configure Vertica and Kafkaon a Single Server.

  • The broker or server that is to be added to the Kafka or Vertica cluster should be an Appliance server and should not already be added to the cluster.

  • The server should be accessible and should have a specific host name.

5.1.2 ZooKeeper

Apache ZooKeeper is already packaged with the ZENworks 2020 build. When the build is deployed on the first Primary Server, ZooKeeper is automatically enabled in the zone. However, you can expand the ZooKeeper cluster as per your scalability requirements and to provide fault tolerance. It is recommended to have an odd number of servers in the cluster. A three node cluster still continues to function if one server goes down since the majority servers are still up. A four node cluster also can only survive the loss of a single node since if two nodes go down, the cluster loses its majority. Therefore, no additional fault tolerance that can be achieved by configuring the cluster with an even number of servers.

Adding Servers to the ZooKeeper Cluster

  • Execute the following command to add nodes or servers to the cluster:

    zman server-role-zookeeper-add-server (zman srzas)

    The parameter to be specified is:

    • --servers: Specify a comma separated list of Primary Servers on which ZooKeeper should be enabled. You can specify the DNS, GUID or path of the server object (server, server folder or server group) relative to /Devices/Servers.

For example: zman server-role-zookeeper-add-server --servers=server2.microfocus.com,server3.microfocus.com,server4.microfocus.com,server5.microfocus.com

For more information on debugging ZooKeeper configuration issues, see ZooKeeper.

5.1.3 Kafka

A summary of the steps to be followed to add brokers to the Kafka cluster are as follows:

NOTE:As Kafka requires client authentication to be enabled in the certificate, if you are using an external CA certificate, ensure that client authentication is enabled.

  1. Execute the command zman server-role-kafka-add-broker (zman srkab) to add brokers or servers to the configured Kafka cluster. Specify a comma separated list of the server DNS, GUID or path of the server object (server, server folder or server group) relative to /Devices/Servers.

    For example: zman server-role-kafka-add-broker --servers=server1.microfocus.com,server2.microfocus.com

  2. Execute the command zman server-role-kafka-update-cluster (zman srkuc) to modify the replication count.

    For example: zman server-role-kafka-update-cluster -c=3

Adding Servers

NOTE:The server that you are adding should be accessible and should have a specific host name.If a server has multiple host names, then the command to add the Kafka broker will fail.

  • Execute the following command to add brokers or servers to the configured Kafka cluster.

    zman server-role-kafka-add-broker (zman srkab)

    The parameters to be specified are:

    • --servers: Specify a comma separated list of appliance servers on which Kafka should be enabled. You can specify the DNS, GUID or path of the server object (server, server folder or server group) relative to /Devices/Servers.

    • -i --ignorewarning message (optional): As Kafka requires client authentication to be enabled in the certificate, if you are using an external CA certificate, ensure that client authentication is enabled and execute the command again with the option i set as true. Option i enables you to ignore the warning message and proceed with the execution of the command.

    For example: zman server-role-kafka-add-broker --servers=server1.microfocus.com,server2.microfocus.com

    IMPORTANT:When a broker is being added to the zone, tasks to remove or re-configure servers cannot be performed on other servers.

    When this command is executed for a server, the Kafka, Kafka connect and Schema Registry services are enabled and started on the specified server. The server is added to the cluster. When a broker is added in the cluster, ZENworks re-configures other existing brokers in the cluster. When re-configuration fails in one these brokers, you need to run the zman command (zman srkrcb) to re-configure the broker. For more information on this command, see Maintaining the Kafka Cluster.

Updating Replication Count

  • Execute the following command to modify the existing cluster parameters such as the replication count.

    zman server-role-kafka-update-cluster (zman srkuc)

    The parameters are:

    • -c --replication count: This ensures that messages remain available when a server in the cluster fails.Specify the number of copies to be maintained for each topic. For a three node cluster, this count can be set as 3, which indicates that a replica of the topic will be created on each server.

    • -l --logRetentionBytes: Specify the maximum permissible size of the log, beyond which, the existing data is overwritten with the new data. By default the log size is unlimited.

    • -t --zkSessionTimeout: Specify the ZooKeeper session timeout (in milliseconds), which is the maximum time that the server waits to establish a connection to ZooKeeper. If the server fails to signal a heartbeat to ZooKeeper within this specified time period, then the server is considered to be dead. A heartbeat request helps identify if the server is still connected to the Kafka cluster. The default value is 30000 milliseconds.

    • -r --retainDetectedLogsDuration: Specify the maximum time to retain deleted logs. The default value is 86400000 milliseconds (1 day).

    • -p --logCleanupPolicy: Specify the default cleanup policy for segments that exceed the maximum permissible retention window. The possible values are Delete and Compact. The default value is Delete. The Delete policy will remove old segments when the retention time or size limit has reached. The Compact policy will enable log compaction on the topic, which ensures that Kafka will always retain at least the last known value for each message key within the log of data for a single topic partition.

    • -s --schemaregistryport: Specify the port on which the schema registry is running. The default value is 8081.

    • -k --kafkaport: Specify the port on which Kafka listens. The default value is 9093.

    • -x --connectport: Specify the port on which Kafka connect listens. The default value is 8083.

      For example: zman server-role-kafka-update-cluster -c=3

To remove a broker from the cluster or to re-configure a broker, see Maintaining the Kafka Cluster.

After adding a broker to the Kafka cluster, ensure that you run the Configure action to allocate appropriate memory. For more information, see Managing Vertica Memory Requirements.

5.1.4 Vertica

A summary of the steps to add servers to the Vertica cluster are as follows:

  1. Execute the command zman server-role-vertica-add-server (zman srvas) to add servers to the cluster. Specify the DNS, GUID or path of the server object (server, server folder or server group) relative to /Devices/Servers.

    For example: zman server-role-vertica-add-server --servers=server2.microfocus.com,server3.microfocus.com

  2. Restart the ZENserver and ZENloader services.

  3. Execute the command zman server-role-vertica-update-cluster (zman srvuc) to update the K factor.

    For example: zman server-role-vertica-update-cluster -k=1

Adding Servers

  • Execute the following command to add servers to the existing cluster and specify a comma separated list of servers by specifying the DNS, GUID or path of the server object (server, server folder or server group) relative to /Devices/Servers.

    zman server-role-vertica-add-server (zman srvas)

    For example: zman server-role-vertica-add-server --servers=server2.microfocus.com,server3.microfocus.com

    IMPORTANT:After adding servers to the Vertica cluster, ensure that you restart the ZENserver and ZENloader services on all the Primary Servers.

Updating the K Factor

  • Execute the following command to update the K factor.

    zman server-role-vertica-update-cluster (zman srvuc)

    This command lets you update the K-safety factor in the existing cluster. The K-safety value determines the number of replicas of data that should exist in a cluster. The K safety sets the fault tolerance for the Vertica cluster, where K can be set as 0 or 1. Specify a value based on the number of nodes in the cluster, which is measured as 2k+1. For example: if the K factor is 1, then you need to have a minimum of 3 nodes. Vertica recommends a K-safety value of 1. For more information on K-safety, see Glossary.

    For example: zman server-role-vertica-update-cluster -k=1

    After adding a broker to the Vertica cluster, ensure that you run the Configure action to allocate appropriate memory. For more information, see Managing Vertica Memory Requirements.