Logging of appliance caching activity can be useful for a number of reasons, such as billing for services rendered. The iChain Proxy Server lets you specify how often a new log file will be started (rolled over), how long old log files will be retained, and the format of the log files.
iChain offers the following logging services:
The appliance can create logs using both the common and extended log formats. A wide variety of tools exists for manipulating and processing these files.
iChain Proxy Services provides a high performance proxy cache system capable of handling thousands of transactions per second. Although the iChain Proxy Server can log extensive details of each transaction, and although the disk space reserved for log files is quite generous on most appliances, if transaction volume is high and log entries consume a few hundred bytes each, iChain Proxy Services can fill up the available disk space in a matter of minutes.
This section explains how appliance logging works and presents management options you can use to ensure optimal use of the available log file disk space and timely migration (downloading) of log files to other storage devices.
The following table shows the transactions the appliance can log and the formats available for each service type.
Service | Common | Extended |
---|---|---|
Web Server Accelerator |
Yes |
Yes |
Dynamic Bypass |
No* |
Yes |
* These common log formats differ from the industry standard proxy cache common log format. |
Turning on logging for a given service increases system overhead and causes some degradation of performance. Therefore, logging should be used only when service transactions must be tracked for customer billing purposes or other compelling reasons.
Transaction volume and log entry size can cause available log disk space to fill up quickly. Proxy cache disk space is unaffected by log files.
See Planning Step 3: Calculating Log Rollover Requirements for formulas you can use to estimate how quickly your logging disk will fill.
To plan a logging strategy you must know the capacity and limitations of your appliance.
Logging disk space is not user-configurable; it is preset to 1 gigabyte on the iChain Proxy Server. Plan to download and delete log files before the disk space is filled.
iChain Proxy Services will not allow the deletion of active log files (files that are currently in use by the caching system). Only log files that have been rolled over and closed can be deleted.
You can ensure that there are closed files on the system by scheduling regular rolling over of log files. During each rollover, the current log file is closed and a new log file is opened.
You must plan for log files to roll over on a schedule that coincides with your download and deletion schedule. This is to ensure that there is at least one closed log file per service when the download and delete cycle starts.
NOTE: Although you can download active log files, this is normally useful only for periodic administrative checks.
Active files contain only the transaction data up to the moment of the download and are incomplete from customer-billing and other business standpoints.
When the appliance encounters a log disk full condition, it stops logging and closes all active log files. Information that would have been logged after that point is lost. Other appliance functions continue without interruption.
The appliance automatically generates log filenames as follows:
NOTE: The dash is not included after the letters double.
This naming convention accommodates up to 702 log files per service per day. If the rollover options are set so that all the possible filenames are used in one day, the log file with the ZZ letter identifier is not closed until the start of the next day (unless the logging disk becomes full).
Appliance log rollover options let you specify when the appliance closes active log files and opens new files so that the closed files can be downloaded and deleted.
Because of the limitations explained in this section, it is essential that you develop a solid log file rollover plan. This will ensure that your appliance doesn't run out of logging disk space or overwrite log files before they are downloaded and deleted.
You can have the appliance roll log files over according to time or file size as explained in the following table:
As explained in The Costs of Logging and System Constraints for Logging, logging of caching transactions involves system and maintenance overhead. If your situation requires logging, you should plan carefully so that the information you are tracking aligns with specific requirements. This will ensure optimal use of appliance resources.
Because logging requirements and transaction volume vary widely, it is impossible to make recommendations regarding specific logging strategies.
The following sections step you through the logging strategy planning process. We recommend you record the information you gather on a planning sheet of some kind.
To plan a logging strategy, you should first determine the requirements behind the need for logging. We recommend you complete the following steps:
Identify the business and other reasons for tracking service transactions.
Possible examples might include customer billing requirements, statistical analysis, growth planning, etc.
Determine which services you need to track.
Record this information for further reference.
If you use the common log format, log entry size is fixed. If you use the extended log format, log entry size depends on the number of log fields selected.
Complete the following steps for each service you need to track:
Referring to the Log Options Dialog Box, record which log fields must be tracked for each service to be logged.
Carefully scrutinize the information you plan to track to ensure that the log data collected is essential.
For example, if you select URI, also selecting URI- STEM and URI-QUERY would be redundant since URI = URI-STEM + URI-QUERY. Also, logging cookie information can consume a lot of space and might provide little, if any critical information.
A few bytes can add up quickly when the appliance is tracking thousands of hits every second.
As explained in Log Rollover Options, you can have the appliance roll over log files based on time or on size, but not both.
If you already know which option you want to use, scan this section and then complete the calculations pertinent to your choice.
If you don't know which option best matches your situation, completing the calculations in this section should help you decide.
The following variables are used in the formulas:
logvol_size: The total disk capacity reserved for log files on your appliance.
The default size is 1 gigabyte.
logentry_size: The average log entry size.
You can determine this by configuring your appliance to track the required information, generating traffic to the appliance, downloading the log files, determining how large each entry is, and calculating the average.
request_rate: The peak rate of requests per second.
You can estimate this rate or place your appliance in a service and get more accurate data by accessing the browser-based management tool's Monitoring tabs.
num_services: The number of services for which you plan to enable logging.
logs_per_service: The number of log files, both active and closed, that you want the appliance to generate for each service before the disk fills.
You must plan to have at least two logs per service before the disk is filled. See Rolling Over Log Files Before Deletion.
Using the following formula, you can calculate how long it will take the appliance to fill your logging disk space:
diskfull_time seconds = logvol_size / (request_rate * logentry_size * num_services)
For example, if you assume the following:
Then diskfull_time = (1 GB) / (1000 * 1KB * 1) = 1048 seconds (17.47 minutes).
The logging disk space will fill up every 17.47 minutes.
If this time is too short, you must reduce the log entry size by configuring the appliance to log less information per transaction. This is because you can't increase the disk space or limit the requests being logged.
To calculate the diskfull_time for your appliance:
Determine the values of the four variables listed in the bullet list above.
For more information, refer to Variable Definitions.
Using the diskfull_time formula, calculate how often you can expect your logging disk to fill, then use the result in Calculating MAX_ROLL_TIME.
Using the following formula, you can calculate the maximum roll-over time value you should specify in the Rollover Every field of the Log Options dialog box.
max_roll_time = diskfull_time / logs_per_service
For example, if you assume the following:
Then max_roll_time = 12 / 2 = 6 hours.
If you roll your logs over by time intervals, the maximum time should be less than six hours. Otherwise scheduling the download and deletion of log files is much more complicated and the window in which this can be done is narrower.
To calculate the max_roll_time for your appliance:
Determine how many log files you want the appliance to generate per service before log space fills.
The minimum number is two.
Using the max_roll_time formula and the diskfull_time value obtained in Calculating DISKFULL_TIME, calculate how often you should have the appliance roll over the log files.
Record the max_roll_time result on your planning sheet.
Using the following formula, you can calculate the maximum log file size you should specify in the Rollover When File Size Reaches field of the Log Options dialog box.
max_log_roll_size = logvol_size / (num_services * logs_per_service)
For example, if you assume the following:
Then max_log_roll_size = 600 MB / (2 * 3) = 100 MB.
If you roll your logs over when they reach a specific size, the file size must be no more than 100 MB. Otherwise, the system will run out of disk space before you have three complete log files and scheduling the download and deletion of log files will be much more complex.
To calculate the max_log_roll_size for your appliance:
Determine the values of the three variables listed in the bullet list above.
Using the max_log_roll_size formula, calculate the maximum size a log file should reach before the appliance rolls it over.
Record the max_roll_time result on your planning sheet.
Based on the planning you have completed in Planning Your Logging Strategy, you must now configure the log options for each affected service.
For each service you are logging, open the Log Options dialog box in the browser-based management tool.
Refer to the services you selected in Planning Step 1: Determining Your Logging Requirements.
The path for the Web Server Accelerator service is Configure > Web Server Accelerator > Insert > Enable Logging for This Accelerator > Log Options.
The following sections discuss each of the areas within the Log Options dialog box.
In the Log Options dialog box, specify the log format for the service based on the planning you did in Planning Step 2: Selecting a Log File Format and Optimizing the Log Entry Size.
Remember that each bit of information you log increases the size of each log entry, and affects the rate at which logging disk space is used.
In the Log Options dialog box, specify how the appliance rolls over the log files for the service based on the planning you did in Planning Step 3: Calculating Log Rollover Requirements.
You must schedule regular download and deletion of log files to avoid running out of log disk space.
Whenever possible, we recommend you use the FTP log push feature for this task. However, you can also manage log files manually using the browser-based management tool or the appliance's Mini FTP Server. See Manually Downloading and Deleting Log Files.
The appliance also provides three options for dealing with old files as a failover precaution.
Ideally, iChain Proxy Services will never actually use the old file option you select because you will schedule the downloading and deleting of log files so that the system never becomes full.
Two of these options automatically dispose of older files to avoid the disk full condition. The third option is not recommended for most situations.
Limit Number of Files To: This option lets you limit the total number of log files retained for each service. After the limit for each is reached, the oldest file for the service is deleted each time a new file is created. All logging data in deleted files is lost.
Delete Files Older Than: This option lets you configure the appliance to delete files when they are older than the time you specify. All logging data in deleted files is lost.
Do Not Delete: This option is not recommended because it can lead to a disk full condition if files are not manually downloaded and deleted. If, however, the older logging data has more value for some reason, this option will preserve the oldest log files unless you manually delete them or specify their deletion in the FTP Log Push Configuration dialog box.
To specify how the appliance handles older files:
In the Log Options dialog box, select an old file option that matches your requirements. (To review option specifics, see the bullet list above.)
As with log format and rollover options, you can specify different old file options for each service. We recommend, however, that you avoid potential confusion by using the same old file settings for each service.
(One time only) Click FTP Log Push > configure the FTP log push options.
For help with setting options in the FTP Log Push Configuration dialog box, refer to Using FTP Push to Automatically Download and Delete Log Files, then return to this procedure.
In the Log Options dialog box, double-check the Old File Options settings against either your FTP log push configuration or your schedule for manual download and deletion to ensure that log files won't reach the deletion threshold (number or age) prior to a scheduled download and deletion.
If you need to configure log options for other services, return to Configuration Step 1: Opening the Appropriate Log Options Dialog Box; otherwise, continue with the next section.
As with all appliance operations, you should monitor what is happening with your logging strategy over time and make adjustments and refinements if necessary.
The following are examples of monitoring you might want to consider.
The FTP Log Push feature lets you configure the appliance to push log files to an FTP server at specified intervals: on the first day of the month and/or on specified days of the week. However, log files cannot be pushed more often than once a day.
The feature operates within the following parameters:
For example, you specify that log files are to be pushed on every day of the week at 12 midnight. When the system clock reaches the target hour, iChain Proxy Services begins trying to establish a connection with the FTP server.
If a connection cannot be established before the hour changes to 1 a.m., iChain Proxy Services stops trying to connect and doesn't try again until 12 midnight the next day.
If a connection is established but an error occurs that prevents a successful push, the error is not detected, and iChain Proxy Services doesn't try to connect again until 12 midnight the next day.
To configure your appliance to use the FTP Log Push feature:
In the browser-based management tool, access the FTP Log Push Configuration dialog box by clicking FTP Log Push on any of the Log Options dialog boxes.
Paths to the dialog boxes are summarized under Configuring Logging Options.
IMPORTANT: Although the FTP Log Push Configuration dialog box is accessed through one of the service-specific Log Options dialog boxes, it is unaffected by the path used to reach it. The settings you specify affect all the log types you check in the box.
This lets you set the FTP push options for all log types in a single place.
In conformance with your logging strategy, specify the following information:
If your FTP server is always available, we recommend using the Push Logs When the Logs Roll Over option rather than setting specific download times. This will protect your appliance from sudden surges in traffic, which can fill the disk sooner than expected.
We recommend deleting log files after they have been pushed unless there is a compelling reason for manually deleting them. Automatic deletion also protects your appliance from sudden surges in traffic.
When you have configured your FTP log push options, click OK to return to the Log Options dialog box of the service you are configuring.
Whenever possible, we recommend that you use the FTP Log Push feature for downloading and deleting log files. See Using FTP Push to Automatically Download and Delete Log Files.
If you need to manage your log files manually, we recommend that you establish a regular schedule and ensure that all those responsible for downloading and deleting log files know the following information:
You will want to develop specific procedures for your situation. The following sections contain general ideas for accomplishing these tasks.
The primary consideration is that log files must be downloaded and deleted before the logging disk space fills up.
Before you can download or delete a log file you must know its exact name.
Appliance log filenames can be listed in the browser-based management tool in Monitoring > Cache Logs. They can also be listed from the command line, or through a Telnet session using the get command.
The appliance automatically generates log filenames as follows:
This naming convention accommodates up to 702 log files per day. If the rollover options are set so that all the possible filenames are used in one day, the log file with the ZZ letter identifier is not closed until the start of the next day.
To list log files using FTP, you must know the path to the files. Use the following table to determine the paths to various log files.
You can most easily view log filenames in the browser-based management tool. To do so, click Monitoring > click Cache Logs > select a log format > select a service.
The Mini FTP Server supports the CWD command for changing to the target log directories. You can also use the LS command in connection with full paths to list log files.
For example, the following command lists transparent and forward proxy log files in common format:
ls log:etc/proxy/data/logs/forward/common/
For a complete list of log file directory paths, see Getting Log Filenames.
You can also see a list of log filenames from the command line. However, you cannot download files from the command line.
The following table presents some command line/Telnet examples.
You can download the files in the browser-based management tool as you view them. After you click Download, when the browser asks what you want to do with the file, save it to your designated log file storage location.
You can use FTP from the storage location to retrieve the files with the get command. You must first obtain each filename by using one of the options explained in Getting Log Filenames.
After you have the log filename, you can transfer the file to your workstation. For example, to download a forward proxy common format log file, you would use the following command after starting an FTP session with the appliance:
get log:/etc/proxy/data/logs/forward/common/filename.log
The filename variable is the name of the log file you have previously obtained.
You can also use the mget command, but be aware that this command also downloads active log files that are not complete.
The appliance doesn't currently support the FTP server put command.
The following is a list of supported FTP commands:
TYPEUSERDELEQUITPWDSYST
After the log files have been downloaded and saved to another location, delete the files using one of the following options:
The following information about field values in extended log files might help you interpret the content of the files.
The information in the following table is supplementary to the W3C Extended Log Format Specification found on the Extended Log File Format Web site. You might find it useful for interpreting the content of extended log field headers.
The logs for authorized access attempts and unauthorized access attempts can be turned on or off globally. The rules for logging authorized access attempts for an individual access control list (ACL) can also be turned on or off. However, because unauthorized access attempts are usually the result of a user not being defined in any ACL rule, logging of unauthorized access attempts cannot be turned on or off for individual ACL rules.
To enable or disable ACL rule logging on a global level:
Access the URL of the iChain Proxy Server where you installed the iChain Proxy Services software to launch the proxy server browser-based administration tool.
For example, http://xx.xx.xx.xx:1959/appliance/config.html where xx.xx.xx.xx is the IP address.
Click Configure > Web Server Accelerator > Modify.
Check the Enable Logging check box.
To enable or disable ACL rule logging for an individual ACL rule:
From ConsoleOne, right-click an ACL Rule object.
Select Properties.
Check the Authorized Logging check box.
The ACL log files for each 24-hour period are saved to LOG:\ETC\PROXY\DATA\LOGS\REVERSE\EXTENDED\ACLCHECK\yymmdd-a.log, where yymmdd is today's date represented by two digits for the year, month and date. The default maximum size of the file is 1 MB. The default size can be changed; see Using ACLCHECK options for more information. If a log exceeds the maximum size, a new file named yymmdd-b.log is created.
Each file contains the following fields: