Logging HTTP transactions has associated costs. The Access Gateway is capable of handling thousands of transactions per second. If transaction volume is high and each log entry consumes a few hundred bytes, the Access Gateway can fill up the available disk space in a matter of minutes. HTTP logging also increases system overhead, which causes some degradation in performance. By default, the logging of HTTP transactions is turned off. Before enabling logging, you need to determine what needs to be logged and then plan a logging strategy.
Because logging requirements and transaction volume vary widely, Novell cannot make recommendations regarding a specific logging strategy. The following tasks guide you through the process of creating a strategy that fits your business needs.
Identify the reasons for tracking transactions such as customer billing, statistical analysis, or growth planning.
Determine which resources need logging.
You enable logging at the proxy service level. If you have a proxy service protecting resources whose transactions do not need to be logged, reconfigure your proxy services so that the proxy service you configure for logging contains only the resources for which you want to log transactions.
Determine what information you need in each log entry.
The common configuration for a log entry contains minimal information: the date, time, and client IP address for each entry. If you need more information, you can select the extended log configuration. Do not select all available fields, but carefully select what you really need. For example, you can include cookie information, but cookie information can consume a large amount of space and might not include any critical information you need.
You should log only the essential data because a few bytes can add up quickly when the Access Gateway is tracking thousands of hits every second. For information about what is available in an extended log profile, see Section 4.3.5, Configuring Extended Log Options.
Design a rollover strategy.
A log must be closed before it can be downloaded to another server for analysis or deleted. You specify either by time or size when the Access Gateway closes a log file and creates a new one. For each proxy service that you enable for logging, you need to reserve enough space for at least two files: one for logging and one for rollover. To calculate the best procedure, see Section 4.3.2, Calculating Rollover Requirements.
Design a log deletion strategy
The Access Gateway has a limited amount of disk space allocated for logging, and you need to decide how you are going to manage this space. You can limit the number of rollover files by number or age. To calculate the best procedure, see Section 4.3.2, Calculating Rollover Requirements.
You can have the Access Gateway roll over log files based on time or on size, but not both. If you already know which option you want to use, scan this section and then complete only the calculations pertinent to your choice. If you don’t know which option best matches your situation, completing the calculations in this section should help you decide.
The following variables are used in the formulas:
logpartition_size: The total disk capacity reserved for log files on the Access Gateway.
The Access Gateway Appliance reserves 4 GB to share between logging and system files. The system files do not grow significantly, so you can assume that you have about 2 GB for logging. To increase this size, see Section 4.3.6, Configuring the Size of the Log Partition.
logentry_size: The average log entry size.
You can determine this by configuring a proxy service to track the required information, generating traffic to the proxy service, downloading the log files, determining how large each entry is, and calculating the average.
request_rate: The peak rate of requests per second.
You can estimate this rate or place your Access Gateway in service and get more accurate data by accessing generated statistics. See Section 4.4, Viewing Access Gateway Statistics.
num_services: The number of proxy services for which you plan to enable logging.
logs_per_service: The number of log files, both active and closed, that you want the Access Gateway to generate for each proxy service before the disk fills.
You must plan to have at least two logs per proxy service, but you can have more.
The following formulas can help you estimate when the system would run out of resources:
Use the following formula to calculate how long it takes the Access Gateway to fill your logging disk space:
diskfull_time in seconds = logpartition_size / (request_rate * logentry_size * num_services)
For example, assume the following:
diskfull_time = (1 GB) / (1000 * 1 KB * 1) = 1048 seconds (17.47 minutes)
The logging disk space fills up every 17.47 minutes.
To calculate the diskfull_time for your Access Gateway:
Determine the values of the four variables listed above.
Use the diskfull_time formula to calculate how often you can expect your logging disk to fill, then use the result in Calculating max_roll_time.
If your diskfull_time interval is too short to be practical for your rollover schedule, the easiest option is to reduce the log entry size by configuring the proxy services to log less information per transaction.
Use the following formula to calculate the maximum rollover time value you should specify in the
fieldmax_roll_time = diskfull_time / logs_per_service
For example, assume the following:
max_roll_time = 12 / 2 = 6 hours
If you roll your logs over by time intervals, the maximum time should be less than six hours. Otherwise, scheduling the download and deletion of log files is much more complicated and the window in which this can be done is narrower.
To calculate the max_roll_time for your Access Gateway:
Determine how many log files you want the Access Gateway to generate per service before log space fills.
The minimum number is two.
Use the max_roll_time formula and the diskfull_time value obtained in Calculating diskfull_time to calculate how often you should have the cache device roll over the log files.
Record the max_roll_time result on your planning sheet.
Use the following formula to calculate the maximum log file size you should specify in the
field:max_log_roll_size = logpartition_size / (num_services * logs_per_service)
For example, assume the following:
max_log_roll_size = 600 MB / (2 * 3) = 100 MB
If you roll your logs over when they reach a specific size, the file size must be no more than 100 MB. Otherwise, the system runs out of disk space before you have three complete log files and scheduling the download and deletion of log files is much more complex.
To calculate the max_log_roll_size for your Access Gateway:
Determine the values of the three variables listed above.
Use the max_log_roll_size formula to calculate the maximum size a log file should reach before the cache device rolls it over.
Do not enable logging until you have designed a logging strategy. See Section 4.3.1, Determining Logging Requirements.
In the Administration Console, click
> > > > >Fill in the following fields:
Enable Logging: Select this field to enable logging.
Stop Service On Log Failure: (Access Gateway Appliance) Select this field if you want the Access Gateway to deny requests to this proxy service because the Access Gateway cannot log entries for it.
Log Directory: (Access Gateway Appliance) Displays the default location for the log files for this proxy service.
In the
, click one of the following options:New: Click this option to create a new logging profile. Then specify a name and select either
or .Default: Click
to modify or view the settings for the profile. The profile uses the common log options.A logging profile determines the type of information that is written to the log file; it also manages rollover and old file options.
Continue with one of the following:
Use the common log options page to control log rollover and old file options. The data included in a log entry is controlled by a default configuration that includes the following:
Date and time of the request
IP address of the client
Remote host name
The request line as it came from the client
The HTTP status code returned to the client
The number of bytes in the document transferred to the client
The Access Gateway does not allow active log files to be deleted. Only log files that have been closed can be deleted. The rollover options allow you to control when a file is rolled over and closed, and a new file is created. The old file options allow you to control when the rolled-over log files are deleted.
To configure a default log file for a selected proxy service:
Click
> > > > > .Select one of the following rollover options:
Rollover When File Size Reaches: Rolls the file when it reaches the specified number of megabytes.
Rollover every: Rolls the file at the specified interval. You can specify the interval in hours or days.
beginning: Specifies the day that the interval should begin. You can select a day of the week or the first of the month.
at: Select the hour of the day that the interval should begin and the time zone (either the local time zone or GMT).
(Access Gateway Appliance) Select one of the following old file options:
Limit Number of Files to: Allows you to limit the number of old log files on the system to the number specified in this option. The oldest file is automatically deleted when this number is reached. All logging data in deleted files is lost.
Delete Files Older Than: Allows you to configure the Access Gateway to delete files when they are older than the time you specify. All logging data in deleted files is lost.
Do Not Delete: Prevents the system from automatically deleting the log files.
Click
.Click the
link, then click > .Use the extended log options page to control log entry content, log rollover, and old file options. A log entry always includes the date, time, and client IP address for each entry, but with the log data options, you can add other fields such as the IP address of the server and the username of the client.
The Access Gateway does not allow active log files to be deleted. Only log files that have been closed can be deleted. The rollover options allow you to control when a file is rolled over and closed, and a new file is created. The old file options allow you to control when the rolled-over log files are deleted.
To configure an extended log file for a selected proxy service:
Click
> > > > > > .Select one or more of the log data options:
Select one of the following rollover options:
Rollover When File Size Reaches: Rolls the file when it reaches the specified number of megabytes.
Rollover every: Rolls the file at the specified interval. You can specify the interval in hours or days.
beginning: Specifies the day that the interval should be begin. You can select a day of the week or the first of the month.
at: Select the hour of the day that the interval should begin and the time zone (either the local time zone or GMT).
(Access Gateway Appliance) Select one of the following old file options:
Limit Number of Files to: Allows you to limit the number of old log files on the system to the number specified in this option. The oldest file is automatically deleted when this number is reached. All logging data in deleted files is lost.
Delete Files Older Than: Allows you to configure the Access Gateway to delete files when they are older than the time you specify. All logging data in deleted files is lost.
Do Not Delete: Prevents the system from automatically deleting the log files.
Click
.Click the
link, then click > .The size of the log partition should be configured as part of the installation process. The Access Gateway Appliance logs are stored in the /root partition by default. You can create a /var partition to store the logs. The size of this partition depends on your requirements. For more information on creating the /var partition, see Creating Custom Partitions
in the Novell Access Manager 3.1 SP2 Installation Guide.