Apache Kafka is already packaged in your ZENworks 2020 Appliance build and will initially be in a disabled state. To enable Kafka, you need to execute ZMAN commands in the command line utility at both the cluster level and broker level. You need to first configure the Kafka cluster settings and subsequently add a broker to it.
Configure Kafka Cluster: Execute the following command:
zman server-role-kafka-configure-cluster (zman srkcc)
While running this command, you need to specify the replication count. If the values for the remaining parameters are not specified, then the default values are considered. The parameters are:
-a --logRetentionPeriod: Specify the duration (in hours) for which the Kakfa logs should be retained.
-c --replication count: This ensures that messages remain available when a server in the cluster fails.Specify the number of copies to be maintained for each topic. For a single node cluster, this count can be set as 1.
-l --logRetentionBytes: Specify the maximum permissible size of the log, beyond which, the existing data is overwritten with the new data. By default the log size is unlimited.
-t --zkSessionTimeout: Specify the ZooKeeper session timeout (in milliseconds), which is the maximum time that the server waits to establish a connection to ZooKeeper. If the server fails to signal a heartbeat to ZooKeeper within this specified time period, then the server is considered to be dead. A heartbeat request helps identify if the server is still connected to the Kafka cluster. The default value is 30000 milliseconds.
-r --retainDetectedLogsDuration: Specify the maximum time to retain deleted logs. The default value is 86400000 milliseconds (1 day).
-p --logCleanupPolicy: Specify the default cleanup policy for segments that exceed the maximum permissible retention window. The possible values are Delete and Compact. The default value is Delete. The Delete policy will remove old segments when the retention time or size limit has reached. The Compact policy will enable log compaction on the topic, which ensures that Kafka will always retain at least the last known value for each message key within the log of data for a single topic partition.
-s --schemaregistryport: Specify the port on which the Schema Registry is running. The default value is 8081.
-k --kafkaport: Specify the port on which Kafka listens. The default value is 9093.
-x --connectport: Specify the port on which Kafka connect listens. The default value is 8083.
For example, zman server-role-kafka-configure-cluster -c=1
NOTE:Before adding a Kafka broker, you need to ensure that the following prerequisites are met:
The Kafka cluster should already be configured.
The broker or server that is to be added to the cluster should be an Appliance server and should not already be added to the cluster.
The server should be accessible and should have a specific host name. If a server has multiple host names, then the command to add the Kafka broker might fail.
As Kafka requires client authentication to be enabled in the certificate, if you are using an external CA certificate, ensure that client authentication is enabled in the Extended Key Usage (EKU) parameter of the server certificate.
Add Kafka brokers: Execute the following command:
zman server-role-kafka-add-broker (zman srkab)
This command adds brokers or servers to the configured Kafka cluster. The parameters to be specified are:
--servers: Specify the appliance server on which Kafka should be enabled. Specify the DNS, GUID or path of the server object (server, server folder or server group) relative to /Devices/Servers.
-i --ignorewarning message (optional): As Kafka requires client authentication to be enabled in the certificate, if you are using an external CA certificate, ensure that client authentication is enabled and execute the command again with the option i set as true. Option i enables you to ignore the warning message and proceed with the execution of the command.
For example: zman server-role-kafka-add-broker --servers=server1.microfocus.com
When this command is executed for a server, the Kafka, Kafka connect and Schema Registry services are enabled and started on the specified server.
For more information on debugging Kafka configuration issues, see Debugging and Log Location.
You can view the status of the Kafka Cluster configuration by navigating to Configuration > Performance Upgrade. For more information on monitoring the Kafka cluster, see Monitoring the Status of Vertica, Kafka and ZooKeeper.