Access Manager 3.1.3 includes the High Availability (Active Standby) feature that creates a standby Linux Access Gateway process and improves the overall reliability. The High Availability feature reduces the downtime caused by Linux Access Gateway restarts and increases the availability of the system. A process that failed earlier comes up again automatically and serve as the standby process.
Do not install the High Availability feature on the SLES 9 Linux Access Gateway appliance.
This section provides information on hardware requirements and configuration information for the High Availability feature and some of the Linux Access Gateway server issues related to the ics_dyn processes.
Minimum 8 GB of RAM
Hard disk space for 3 COS partitions
The size of these partitions should be 10 times the maximum size of the files your users download. For example, if they download 4 GB files, the COS partitions should be approximately 40 GB.
You can enable the High Availability feature by using the installHA script to create COS partitions or by using the custom COS partitions.
The High Availability feature is not automatically enabled on a new installation of Access Manager or when Access Manager is upgraded from an earlier version to Access Manager 3.1.3. In both cases, you need to run the installHA script on the Linux Access Gateway server to enable the High Availability feature. The installHA script is located in the /chroot/lag/opt/novell/bin directory. The script creates three COS partitions in your Linux Access Gateway server and changes the configuration files to enable the High Availability feature.
Go to the cd /chroot/lag/opt/novell/bin/ location.
Run the command sh ./installHA.
After the installHA script is invoked from the /chroot/lag/opt/novell/bin directory, this script performs the following tasks:
Checks for RAM availability.
Ensure that your Linux Access Gateway machine has a minimum of 8 GB RAM.
Checks for the existing COS partitions and empty hard disks on your Linux Access Gateway machine.
NOTE:An empty hard disk indicates that there are no partitions on the hard disk. An empty hard disk does not indicate empty partitions.
Prompts the administrator to tell which partitions or unused space the Linux Access Gateway can use as COS partitions.
Formats the space and creates three COS file systems.
Changes the virtual.xml file and restarts novell-vmc.
If /dev/sdb is an empty hard disk on your Linux Access Gateway machine, the installHA script performs the following tasks in sequence:
Uses the entire empty disk partition to create an extended partition.
Uses the extended partition to create the required number of logical partitions.
Updates the file system of the newly created partitions as 0x68.
To view the newly created COS partitions, use the following command:
fdisk -l /dev/sdb
Before choosing an empty hard disk for creating partitions, the installHA script prompts you for confirmation.
If there are no empty hard disks in the system, the installHA script attempts to create three COS partitions in the existing COS partitions disk space. For example, if /dev/sda7 is an existing COS partition, the installHA script performs the following tasks in sequence:
Uses the /dev/sda7 partition to create an extended partition.
Uses this extended partition to create the required number of logical partitions.
Updates the file system of new partitions as 0x68.
To view the newly created COS partitions, use the following command:
fdisk -l /dev/sdsa7
Before using the existing COS partitions to create new partitions, the installHA script does not prompt for administration confirmation to choose COS partition for re-partitioning..
Before committing the COS partition changes, the installHA script displays the COS partition layout before and after the commitment, and asks for your confirmation to proceed with the changes. To reduce the risk of data or partition loss, you must verify the displayed partition layout before committing the COS partition changes.
The installHA script uses the parted and sfdisk commands to create COS partitions and to get partition details. When the installHA script is run, it might return some errors and warnings from the parted command.You can ignore the following warnings:
Error: Error informing the kernel about modifications to partition /dev/sda7p1 -- Invalid argument. This means Linux won't know about any changes you made to /dev/sda7p1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting.
After successfully creating the COS partitions, the installHA script updates the virtual.xml file so that the High Availabilty feature is enabled and uses the three newly created COS partitions. The installHA script asks for your confirmation to reboot the Linux Access Gateway server for the partition changes to take effect.You can choose to terminate the reboot process, and later manually reboot it before using the new partitions or starting novell-vmc.
For more information, see Section 2.8.8, Limitations of the Script that Installs the High Availability Feature.
You can use YaST to manually create the COS partitions for the High Availability feature.
In YaST, create atleast three COS partitions on your Linux Access Gateway machine.
You need at least three COS partitions for High Availability to work.
IMPORTANT:On YaST, the 0x68 file system should be assigned to the partitions to make them into COS partitions.
Run the enableHA script from the /chroot/lag/opt/novell/bin directory as follows:
sh ./enableHA
The enableHA script prompts you for confirmation to enable the High Availability feature on your Linux Access Gateway server and to stop novell-vmc. When novell-vmc is stopped, the enableHA script updates the configuration files to enable the High Availability feature and restarts novell-vmc.
To disable the High Availability feature, run the disableHA script from the /chroot/lag/opt/novell/bin directory as follows:
sh ./disableHA
The disableHA script asks for administrator’s confirmation to disable the High Availability feature on your Linux Access Gateway server. The disableHA script also asks for your confirmation to stop novell-vmc. When novell-vmc is stopped, the disableHA script updates the configuration files to disable the High Availability feature, then restarts novell-vmc.
The ics_dyn process is the core component of the proxy functionality. The High Availability feature requires three ics_dyn processes to run simultaneously. One ics_dyn process is the master, and the other two are slaves.
For the ics_dyn process to start, it needs to know the following configuration details of the Linux Access Gateway box:
The COS partition that it can use.
The size of RAM allotted to the ics_dyn process.
Other attributes including index and master.
These attributes are available in the virtual.xml file located at the var/novell/cfgdb/.current/ directory.
A sample DOM element looks like this:
<VM index=”0” master=”1” profileName=”default”> <Memory size=”25”/> <PartitionList> <Partition>/dev/sda5</Partition> </PartitionList> <CPUGroup> <CPU index=”0” make=”GenuineIntel” utilization=”3.141592”/> </CPUGroup> </VM>
The index parameter is used as <index> throughout the document.
If the High Availability feature is not installed or disabled, the ics_dyn holds the component that listens on port 8181. Port 8181 receives the setSessions from the eSP back channel communication. Enabling the High Availability feature activates a new process called broadcast that listens on port 8181.
The broadcast process on port 8181 performs the following tasks:
Accepts the connection on port 8181 from the embedded service provider (ESP).
Makes connection to the ics_dyn processes.
Reads data on port 8181 and transfers it to the ics_dyn process over the connections established in the ics_dyn processes.
Sends the data that is received from master ics_dyn back to the ESP over the connection on port 8181 from the ESP.
The following data is transferred to the 8181 listener port:
Session information after authentication is done that is setSessions.
Log out information.
With High Availability enabled, the broadcast process also owns the listener on port 41114.
The following commands flow on port 41114:
apply
esp-status
policy refresh
purge cache
The broadcast process listener on port 41114 performs the following tasks:
Accepts the connection on port 41114 from nash and reads the command.
Makes connection to the ics_dyn processes.
Sends the command and data that is received on port 41114 to the standby ics_dyn process, if all the connections are established.
If the response from the standby fails, this response will be sent back to nash and closes all the connections. So no more communications related to this command happen between ics_dyns nash and the broadcast processes.
If the response from the standby is successful, then the broadcast process sends the command and data that is received on port 41114 to the master ics_dyn process. The response received from master ics_dyn is sent to nash.
If the response from the master ics_dyn fails, then all the connections are closed and no further communication related to this command happens between ics_dyn and the nash and broadcast process.
If the response from the master ics_dyn is successful, then the broadcast process sends the command and data that is received on 41114 to the active ics_dyn. The broadcast process closes all the connections after the command is sent to the active ics_dyn.
Closes all the connections after the data is transferred to the active ics_dyn.
The High Availability feature, changes how the proxy accepts the connections from the browser changes. The master ics_dyn process performs the following tasks:
Accepts the connection from clients such as browsers.
Selects the slave to serve the accepted connection.
Sends the ownership of the accepted connection to the selected slave, if any. Otherwise, master ics_dyn displays an error in the browser.
The master ics_dyn process uses the following steps to select the slave:
Performs the simple failover load balancing across the two slave ics_dyn processes.
Finds the slave that is currently serving.
Checks whether a slave is up and ready to agree to the accepted connection. If yes, this slave is selected to serve the connection. If not, the slave ics_dyn process marks this slave as down.
Checks whether the next slave is up and ready to accept the connection. If yes, the slave ics_dyn process marks this slave as the current processing slave and marks the other slave as standby. The current processing slave is selected to serve the connection.
If none of the slaves are up and ready to accept the connection, then none of these slaves are selected to serve the connection
Health check requests from the Administration Console also follows the same process.
The master ics_dyn process accepts the connections and forwards the ownership to the slaves.
For example, consider a scenario where the configuration consists of a proxy service on port 80. The proxy service is present in the config.xml file. All the ics_dyn processes read the config.xml file to bring up the proxy services.
The following questions help you understand the communication that happens between the master and slave processes to execute a request.
'On reading the config.xml file, the master ics_dyn brings up the TCP listener service on port 80 and /var/novell/vms-80-0 UNIX path. The slave ics_dyn processes brings up the listener on /var/novell/vms-80-<index> UNIX path.' After slaves brings up their listener, they inform the maser ics_dyn on /var/novell/vms-80-0 that they are up so that the master ics_dyn knows that the slaves are up and ready to receive the requests..
The master knows which slaves are up and knows the UNIX path on which they listen. The master forwards the fd ownership to the slaves by using the UNIX domain sockets.
When master ics_dyn fails to forward the fd ownership to a slave, it knows that the slave is down.The master ics_dyn marks that slave as down. When the slave comes up again, it informs the master that it is ready to accept requests, and the master knows that the slave is up.
When you make changes through Administration Console, the ics_dyn listener service receives the change request on port 444. The master ics_dyn process forks and executes the preapply.sh script, forks and executes the nash apply vcdn command, and finally, forks and executes the postapply.sh script.
The nash process executes the os-conf which applies the network related configuration and sends the “apply” command to the broadcast process on port 41114.
Functionality of port 41114 listener of broadcast process.
For more information, see Section 2.8.4, High Availability Functionality.
After the master ics_dyn process receives and executes the apply command successfully, then the master and the standby ics_dyn processes are aware of the new configuraiton.
Functionality of master ics_dyn after receiving and executing the 'apply changes' command successfully, only master and the standby ics_dyn processes are aware of the new configuraiton.Active ics_dyn process is not aware of the new configuration. So, master ics_dyn process swaps its active and standby slaves pointers, so that standby slaves start serving the requests using a new configuration.The master ics_dyn process also sends the response back to the broadcast process, which then sends the response back to nash. Nash copies the config.xml from /var/novell/cfgdb/vcdn directory to the /var/novell/cfgdb/.current directory and sends that response back to Administration Console.The broadcast process sends the apply command to the other ics_dyn, so that all processes use the new configuration.
The High Availability feature allocates 10% of RAM to the master ics_dyn process and 25% of RAM to each slave.
Logging in to the ics_dyn process appends the device ID to the ics_dyn process ID in the virtual.xml file.
There is no change in the way proxy sends user session activity updates to ESP.
NOTE:Do not change the virtual.xml file contents manually unless you are told to do so by Novell Support.
Only one ics_dyn actively serves the browser requests at any point in time.
The system down alerts are available only if the master crashes. The core dumps for the master ics_dyn process are available if the .dumpcore touch file is present in the /tmp/ directory. The core dumps for the slave ics_dyn processes are available if the .slavedumpcore touch file is present in the /tmp/ directory. The system up alert is sent only by the master ics_dyn process.
If any ics_dyn process goes down, including the master, auto recovery of the entire system is available.
If two COS partitions exist in your Linux Access Gateway server, the installHA script chooses one of them to use for creating partitions. The installHA script does not have the intelligence to find the largest partition to use.If two COS partitions exist sequentially in your Linux Access Gateway server, the installHA script does not merge them. The installHA script selects one of the partitions and tries to create COS partitions in that partition.After the COS partitions are created or re-created, scripts are not provided to merge them into a single partition.You can merge the partitions by using YaST.