10.6 Fixing a Messaging Service That Won’t Start

IMPORTANT:The following procedures assume that you know which two of your three Search appliances were configured to provide Messaging services.

Referring to Table 10-1, find the cause of the Messaging service failure and complete the steps shown.

Table 10-1 Fixing a Messaging Service That Won’t Start

Cause of Failure	Steps to Resolve
Power failure on both Messaging services nodes or other simultaneous shut down. When this happens, both Search appliances running Messaging services are set as the first node that shut down, meaning that neither Messaging service can start until the other is running.	Make sure to start the database server or PostgreSQL appliance first. When the database is running, start one of the Search appliances that is registered to run Messaging services. It doesn’t matter which of the two appliances you start first since they shut down at the same time. Access the appliance’s console and log in as the root user. Set the messaging service to start without timing out by entering the following command: rabbitmqctl force_boot Start the messaging service by entering the following command: systemctl start esn-rabbitmq-server Enter exit to log out as root. Start the other Search appliances and then the TeamWorks appliances as outlined in Startup Order Is Also Critical.
You start the wrong Messaging node (Search appliance) first and when you check the RabbitMQ status as directed in Step 2, service status is Stopped.	Click Home, then click the Configuration icon. On the Configuration Summary page, under Messaging Service, check to verify which two appliances are registered to run Messaging Services. The appliance currently running should be listed. The other appliance listed should be the last one shut down, and therefore the first one to start up. Click Home > Shutdown. When the Search appliance is completely shut down, start the other Search appliance that has Messaging services enabled. Check the RabbitMQ service status. If the status is Running, exit the Port 9443 console and continue wth appliance startup as outlined in Startup Order Is Also Critical. If the status is Stopped, do the following: Access the appliance’s console and log in as the root user. Set the messaging service to start without timing out by entering the following command: rabbitmqctl force_boot Start the messaging service by entering the following command: systemctl start esn-rabbitmq-server Enter exit to log out as root. Continue with appliance startup as outlined in Startup Order Is Also Critical.
You delete, recycle, etc. one of the Search appliances that was registered to run Messaging services and was not properly decommissioned. This causes the “Messaging cluster” configuration to become invalid and creates system problems. NOTE:The primary assumption here is that the deleted Search appliance was also the “last down” and this fact is preventing Messaging services from starting. However, the same steps should be completed if the “first down” Messaging node is deleted. This is because that would also cause the cluster configuration to become invalid and create system problems.	Repair the “Messaging cluster” configuration by removing the node that no longer exists. Access the console of the remaining appliance that is registered to run Messaging services and log in as the root user. Remove the appliance that no longer exists from the Messaging cluster by entering the following command: rabbitmqctl forget_cluster_node -offline rabbit@short_hostname_deleted_node Where short_hostname_deleted_node is the hostname of the appliance that was deleted but not decommissioned, for example tw-search-2. Start the messaging service by entering the following command: systemctl start esn-rabbitmq-server Enter exit to log out as root. Make sure that you restore the TeamWorks deployment to best practice status as soon as possible. In other words, run three Search Appliances, two of which are running Messaging services.

Cause of Failure

Steps to Resolve

Power failure on both Messaging services nodes or other simultaneous shut down.

When this happens, both Search appliances running Messaging services are set as the first node that shut down, meaning that neither Messaging service can start until the other is running.

Make sure to start the database server or PostgreSQL appliance first.
When the database is running, start one of the Search appliances that is registered to run Messaging services. It doesn’t matter which of the two appliances you start first since they shut down at the same time.
Access the appliance’s console and log in as the root user.
Set the messaging service to start without timing out by entering the following command:

rabbitmqctl force_boot
Start the messaging service by entering the following command:

systemctl start esn-rabbitmq-server
Enter exit to log out as root.
Start the other Search appliances and then the TeamWorks appliances as outlined in Startup Order Is Also Critical.

You start the wrong Messaging node (Search appliance) first and when you check the RabbitMQ status as directed in Step 2, service status is Stopped.

Click Home, then click the Configuration icon.
On the Configuration Summary page, under Messaging Service, check to verify which two appliances are registered to run Messaging Services.

The appliance currently running should be listed. The other appliance listed should be the last one shut down, and therefore the first one to start up.
Click Home > Shutdown.
When the Search appliance is completely shut down, start the other Search appliance that has Messaging services enabled.
Check the RabbitMQ service status.

If the status is Running, exit the Port 9443 console and continue wth appliance startup as outlined in Startup Order Is Also Critical.

If the status is Stopped, do the following:
1. Access the appliance’s console and log in as the root user.
2. Set the messaging service to start without timing out by entering the following command:
  
  rabbitmqctl force_boot
3. Start the messaging service by entering the following command:
  
  systemctl start esn-rabbitmq-server
4. Enter exit to log out as root.
Continue with appliance startup as outlined in Startup Order Is Also Critical.

You delete, recycle, etc. one of the Search appliances that was registered to run Messaging services and was not properly decommissioned.

This causes the “Messaging cluster” configuration to become invalid and creates system problems.

NOTE:The primary assumption here is that the deleted Search appliance was also the “last down” and this fact is preventing Messaging services from starting.

However, the same steps should be completed if the “first down” Messaging node is deleted. This is because that would also cause the cluster configuration to become invalid and create system problems.

Repair the “Messaging cluster” configuration by removing the node that no longer exists.

Access the console of the remaining appliance that is registered to run Messaging services and log in as the root user.
Remove the appliance that no longer exists from the Messaging cluster by entering the following command:

rabbitmqctl forget_cluster_node -offline rabbit@short_hostname_deleted_node

Where short_hostname_deleted_node is the hostname of the appliance that was deleted but not decommissioned, for example tw-search-2.
Start the messaging service by entering the following command:

systemctl start esn-rabbitmq-server
Enter exit to log out as root.
Make sure that you restore the TeamWorks deployment to best practice status as soon as possible. In other words, run three Search Appliances, two of which are running Messaging services.