7.4 Restoring Your System after a Primary Server Failure

7.4.1 Restoring a Primary Server for a Remote Data Store

This section is applicable only if the PostgreSQL or Oracle data store is hosted remotely. Before starting the process of converting a Secondary Server to a Primary Server, all the Secondary Servers that don't have a read/write replica of the ZENworks Object Store need to have eDirectory restarted. When it is restarted, eDirectory recognizes that the Primary Server is lost and no longer tries to communicate with it.

To restart eDirectory:

  1. Type /etc/init.d/ndsd restart on each Secondary Server that doesn't have a read/write replica of eDirectory.

Converting a Secondary Server to a Primary Server

To convert a ZENworks Secondary Server to a ZENworks Primary Server:

  1. Choose a Secondary Server with a read/write replica to be the new Primary Server and change the ZENworks Object Store replica on that server to be the master replica by typing ndsrepair -P -Ad.

  2. On all the Secondary Servers, copy var/opt/novell/zenworks/zfs/pds/security/csr/*.csr to var/opt/novell/zenworks/zfs/pds/security/*.cer.

  3. If required, restore the ZENworks Data Store (for instructions, see ZENworks Object Store and Data Store Maintenance in the Novell ZENworks 7.3 Linux Management Administration Guide).

  4. From within the /opt/novell/convertserver directory, type ./convertserver on the server that is being promoted to the Primary Server, then follow the prompts. You'll need the administrator password.

  5. If you want to have all the devices that were using the old Primary Server, change the address of the old Primary Server name on the DNS server to that of the new Primary Server.

  6. Restore or re-create any zlmmirror configuration files that existed previously.

  7. Restore the package and bundle directories if they have been backed up.

  8. If you haven't done so already, install ConsoleOne for Linux.

    1. Download the Linux version of ConsoleOne from the Novell Downloads Web site. You are prompted to enter your Novell account username and password. If you don't have a Novell account, use the “create new account” link.

    2. Save the ConsoleOne compressed file to disk and uncompress it.

    3. Run the cl-install installation script. Install all the snap-ins.

  9. Use ConsoleOne to remove the old Primary Server object from the ZENworks Object Store.

    1. Start ConsoleOne by running the ConsoleOne executable located under /usr/ConsoleOne/bin.

    2. Authenticate to eDirectory by clicking the tree icon. You are prompted with a login credential dialog box. Log in to your server with the following information:

      • Login name: admin

      • Password: Password from the ZENworks Linux Management installation

      • Tree: IP address of a Secondary Server that holds a read/write replica of the ZENworks Object Store

      • Context: system

      1. Under My World/NDS/.ZONENAME-TREE/system, select your old Primary Server object. Right-click and select Delete NDS object. Answer Yes.

  10. After a Secondary Server has been promoted to be the Primary Server, a read/write replica of the ZENworks Data Store needs to be created on one of the other Secondary Servers. Use ConsoleOne to create a new read/write eDirectory partition on one of the Secondary Servers that has no replica.

    1. In the left pane of ConsoleOne, expand NDS. The new zone name is appended with -TREE.

    2. Click the new zone name. Click View from the menu, then click Partition and Replica View. The servers and their replica types are listed.

    3. Select any server. Right-click it and select Add Replica.

    4. Select a server using the browse button. Navigate to the system container, then click the server object that will be your new read/write replica. Click OK.

    5. Click OK twice.

  11. Run the /opt/novell/zenworks/sbin/dbsync.sh script to synchronize the database with eDirectory.

7.4.2 Restoring a Primary Server for a Local Data Store

This section is applicable only if the PostgreSQL ZENworks data store is hosted locally.

To recover a crashed ZENworks 7.3 Linux Management Primary Server, you must restore the PostgreSQL database to a Secondary Server and then promote the server to be a Primary Server. The Secondary Server must have a read/write eDirectory replica for the ZENworks Object Store.

You must ensure that the following prerequisites are met before you begin the restoration process:

Do the following on the ZENworks Primary Server before you begin the restoration process:

  1. Stop all the ZENworks services by using the following command:

    zlm-config --stop

  2. Stop the PostgreSQL database by using the following command:

    /etc/init.d/postgresql stop

  3. Disconnect the device from the network.

Perform the following tasks in the order listed:

Recovering the ZENworks Data Store

Do the following on the Secondary Server to recover the Primary Server data store:

  1. Choose a Secondary Server with a Read/Write replica to be promoted as the new Primary Server. The Secondary Server must contain the Read/Write replicas of the master replica located on the Primary Server.

  2. Verify if the following RPM packages, which contain the PostgreSQL server and its client libraries, are installed on the Secondary Server. If the RPMs are not installed, you must install them.

    • postgresql-server

    • postgresql

    • postgresql-jdbc

    • postgresql-libs

  3. Stop all the ZENworks services by using the following command:

    zlm-config --stop

  4. Start the PostgreSQL service by using the following command:

    /etc/init.d/postgresql start

    If this is the first time you are starting the PostgreSQL service, the PostgreSQL default configuration and database files are created.

  5. In the PostgreSQL configuration file /var/lib/pgsql/data/postgresql.conf, edit or add the following parameters for the default PostgreSQL 8.0 database on a SLES 10 device. You need to optimize them based on the server load in your environment.

    • max_connections = 500

    • authentication_timeout = 1

    • shared_buffers = 2048

    • work_mem = 2048

    • max_fsm_pages = 40000

    • maintenance_work_mem = 16384

    • listen_addresses = '*'

    • timezone = GMT

    • unix_socket_permissions = 0770

    The PostgreSQL configuration settings such as sort_mem, vacuum_mem, and tcpip_socket in PostgreSQL 7.4 on SLES 9 devices have changed to work_mem, maintenance_work_mem, and listen_addresses in PostgresSQL 8.0 and above on SLES 10 devices. For SLES 9 servers with PostgreSQL 7.4, ensure to set the value of tcpip_socket to True.

  6. Edit the /etc/opt/novell/zenworks/hibernate.cfg.xml file as follows. These modifications point the ZENworks services to the newly created local PostgreSQL database.

    1. Change the value of the database connection.url parameter to jdbc:postgresql://127.0.0.1:5432/zenworks

    2. Change the value of the database connection.password parameter to the content of the file /etc/opt/novell/zenworks/serversecret.

  7. Restore the pg_hba.conf file that you backed up on the Primary Server to the following path on this Secondary Server:

    /var/lib/pgsql/data/pg_hba.conf

    This overwrites the default PostgreSQL client authentication configuration file.

    After the restoration process is complete, the database is locally available.

  8. From the pg_hba.conf file, remove the line containing the IP address of this Secondary Server.

  9. Ensure that you retain the pg_hba.conf file rights. The owner must be the user postgresql, the group must be postgresql, and the rights scheme for the file must be 0600.

  10. Edit the ~/.pgpass file to replace the IP address of the old Primary Server with localhost as follows:

    localhost:5432:zenworks:zenadmin:<serversecret>

    The value of serversecret must be the content of the file /etc/opt/novell/zenworks/serversecret, located on the promoted Secondary Server.

  11. Restart the postgreSQL service by using the following command:

    /etc/init.d/postgresql restart

  12. Rename the database backup file created on the Primary Server without the time stamp.

    For example, if the database backup filename is 11:59:08-zenworks-backup.tar.gz, rename the file as zenworks-backup.tar.gz.

  13. Copy the renamed database backup file to the /tmp directory.

  14. From within the /opt/novell/convertserver directory, run the script zlm_create_restore_pgdb as follows:

    ./zlm_create_restore_pgdb -F Path_to_zenworks_dbbackup_file

    This script creates the zenworks database schema on the Secondary Server. It also restores the database from the zenworks database backup file in verbose mode by default. If you are prompted for a password, use the value of /etc/opt/novell/zenworks/serversecret that is located on the Secondary Server.

  15. If there are additional Secondary Servers connected to this zone, do the following on each of the Secondary Servers:

    1. Edit the /etc/opt/novell/zenworks/hibernate.cfg.xml file as follows to direct all the Secondary Servers to the new Primary Server data store:

      • Edit the database connection.url parameter value to jdbc:postgresql://New_Primary_Server_IP_Address:5432/zenworks

      • Change the database connection.password parameter value to the content of the new Primary Server's file /etc/opt/novell/zenworks/serversecret

    2. Edit the ~/.pgpass file to point to the new Primary Server's IP address, and set the connection password to the value of the /etc/opt/novell/zenworks/serversecret file.

Re-configuring the ZENworks Object Store

The ZENworks Object Store is based on Novell's eDirectory. Before starting the process of converting a Secondary Server to a Primary Server, all the Secondary Servers that do not have a read/write replica of the ZENworks Object Store need to have eDirectory restarted.When it is restarted, eDirectory recognizes that the Primary Server is lost and no longer tries to communicate with it. To restart the eDirectory, run the /etc/init.d/ndsd restart command on each secondary server that doesn't have a read/write replica of eDirectory.

Do the following on the Secondary Server that you are promoting to a Primary Server:

  1. Start the ndsd daemon on the Secondary Server by using the following command:

    /etc/init.d/ndsd start

  2. Do the following during the NDS Repair process to change the ZENworks Object Store's Read/Write replica state to Master replica state:

    1. Run the ndsrepair -P –Ad command.

    2. Enter a replica number based on the partition name.

      For example, enter the replica number as 1 based on the partition name.

    3. Enter the replica operation option value for the option Designate this server as the new master replica.

      For example, the corresponding value of the option is 5.

      IMPORTANT:Do not start the old Primary Server back up as it might corrupt the Object Store information and might also conflict with another master replica in the same zone.

  3. Remove the old Primary Server from the eDirectory Replication schema:

    1. Run the ndsrepair -P –Ad command.

    2. Enter the replica number based on the partition name.

      For example, enter the replica number as 1 based on the partition name.

    3. Enter the replica operation option value for View Replica Ring.

      For example, enter the value as 5.

    4. Enter the number for the old Primary Server's server name.

    5. Enter the server option value for Remove this server from replica ring.

      For example, enter the value as 6.

  4. Restart the zlm services on the promoted Secondary Server by using the following command:

    zlm-config --restart

  5. Ensure that the local postgreSQL database service is running by using the following command:

    /etc/init.d/postgresql status

  6. Execute the following commands to check the status of the server back-end modules, health and datastore respectively:

    • zlman ping
    • zlman si
    • zlman sh server_name
  7. From within the /opt/novell/convertserver directory, run the ./convertserver command on the Secondary Server, and provide the administrator password to continue.

    The Secondary Server is promoted to a Primary Server and all the ZENworks services are automatically restarted.

  8. Restart the ZENworks Linux Management services on all the Secondary Servers by using the following command:

    zlm-config --restart

Re-configuring Content Replication

The content replication can be re-configured if no change has been made to the Primary Server’s DNS Name or IP address is unchanged.

Do the following to configure the content replication between the new Primary Server and the remaining Secondary Servers:

  1. Stop the ZENworks Server Management (TED) service on any additional Secondary Server by using the following command:

    /etc/init.d/novell-zented stop

  2. Delete the .keystore file from the /var/opt/novell/zenworks/zfs/pds/security/private/ directory located on the other Secondary Servers.

  3. From the new Primary Server, copy the /var/opt/novell/zenworks/zfs/pds/security/csr/*.csr files to the /var/opt/novell/zenworks/zfs/pds/security/on all other Secondary Servers.

  4. Rename the .csr files to .cer on all the Secondary Servers.

  5. In the etc/opt/novell/zenworks/zfs-startup.xml file, set the <Value> tag of the parameter <Parameter Name="eDirectory Servers"> to the IP address of the new Primary Server.

  6. Start the TED services by using the following command on all the Secondary Servers:

    /etc/init.d/novell-zented start