4.2 Configuring Proxy Service Logging

Logging HTTP transactions has associated costs. The Access Gateway is capable of handling thousands of transactions per second. If transaction volume is high and each log entry consumes a few hundred bytes, the Access Gateway can fill up the available disk space in a matter of minutes. HTTP logging also increases system overhead, which causes some degradation in performance. By default, the logging of HTTP transactions is turned off. Before enabling logging, you need to determine what needs to be logged and then plan a logging strategy.

4.2.1 Determining Logging Requirements

Because logging requirements and transaction volume vary widely, Novell cannot make recommendations regarding a specific logging strategy. The following tasks guide you through the process of creating a strategy that fits your business needs.

  1. Identify the reasons for tracking transactions such as customer billing, statistical analysis, or growth planning.

  2. Determine which resources need logging.

    You enable logging at the proxy service level. If you have a proxy service protecting resources whose transactions do not need to be logged, reconfigure your proxy services so that the proxy service you configure for logging contains only the resources for which you want to log transactions.

  3. Determine what information you need in each log entry.

    The common configuration for a log entry contains minimal information: the date, time, and client IP address for each entry. If you need more information, you can select the extended log configuration. Do not select all available fields, but carefully select what you really need. For example, you can include cookie information, but cookie information can consume a large amount of space and might not include any critical information you need.

    You should log only the essential data because a few bytes can add up quickly when the Access Gateway is tracking thousands of hits every second. For information about what is available in an extended log profile, see Section 4.2.5, Configuring Extended Log Options.

  4. Design a rollover strategy.

    A log must be closed before it can be downloaded to another server for analysis or deleted. You specify either by time or size when the Access Gateway closes a log file and creates a new one. For each proxy service that you enable for logging, you need to reserve enough space for at least two files: one for logging and one for rollover. To calculate the best procedure, see Section 4.2.2, Calculating Rollover Requirements.

  5. Design a log deletion strategy

    The Access Gateway has a limited amount of disk space allocated for logging, and you need to decide how you are going to manage this space. You can limit the number of rollover files by number or age. You can also select to copy the files to another server and then delete them. To calculate the best procedure, see Section 4.2.2, Calculating Rollover Requirements.

4.2.2 Calculating Rollover Requirements

You can have the Access Gateway roll over log files based on time or on size, but not both. If you already know which option you want to use, scan this section and then complete only the calculations pertinent to your choice. If you don’t know which option best matches your situation, completing the calculations in this section should help you decide.

The following variables are used in the formulas:

  • logpartition_size: The total disk capacity reserved for log files on the Access Gateway.

    The Access Gateway reserves 4 GB to share between logging and system files. The system files do not grow significantly, so you can assume that you have about 2 GB for logging. To increase this size, see Section 4.2.6, Configuring the Size of the Log Partition.

  • logentry_size: The average log entry size.

    You can determine this by configuring a proxy service to track the required information, generating traffic to the proxy service, downloading the log files, determining how large each entry is, and calculating the average.

  • request_rate: The peak rate of requests per second.

    You can estimate this rate or place your Access Gateway in service and get more accurate data by accessing generated statistics. See Section 4.3, Monitoring Access Gateway Statistics.

  • num_services: The number of proxy services for which you plan to enable logging.

  • logs_per_service: The number of log files, both active and closed, that you want the Access Gateway to generate for each proxy service before the disk fills.

    You must plan to have at least two logs per proxy service, but you can have three or more.

The following formulas can help you estimate when the system would run out of resources:

Calculating diskfull_time

Use the following formula to calculate how long it will take the Access Gateway to fill your logging disk space:

diskfull_time in seconds = logpartition_size / (request_rate *
   logentry_size * num_services)

For example, assume the following:

  • logpartition_size = 1 GB (1,073,741,824 bytes)
  • request_rate = 1000 requests per second
  • logentry_size = 1 KB (1,024 bytes)
  • num_services = 1
diskfull_time = (1 GB) / (1000 * 1 KB * 1) = 1048 seconds (17.47
   minutes)

The logging disk space will fill up every 17.47 minutes.

To calculate the diskfull_time for your Access Gateway:

  1. Determine the values of the four variables listed above.

  2. Use the diskfull_time formula to calculate how often you can expect your logging disk to fill, then use the result in Calculating max_roll_time.

If your diskfull_time interval is too short to be practical for your rollover schedule, the easiest option is to reduce the log entry size by configuring the proxy services to log less information per transaction.

Calculating max_roll_time

Use the following formula to calculate the maximum rollover time value you should specify in the Roll over every field 

max_roll_time = diskfull_time / logs_per_service

For example, assume the following:

  • diskfull_time = 12 hours
  • logs_per_service = 2
max_roll_time = 12 / 2 = 6 hours

If you roll your logs over by time intervals, the maximum time should be less than six hours. Otherwise, scheduling the download and deletion of log files is much more complicated and the window in which this can be done is narrower.

To calculate the max_roll_time for your Access Gateway:

  1. Determine how many log files you want the Access Gateway to generate per service before log space fills.

    The minimum number is two.

  2. Use the max_roll_time formula and the diskfull_time value obtained in Calculating diskfull_time to calculate how often you should have the cache device roll over the log files.

  3. Record the max_roll_time result on your planning sheet.

Calculating max_log_roll_size

Use the following formula to calculate the maximum log file size you should specify in the Maximum File Size field: 

max_log_roll_size = logpartition_size / (num_services *
   logs_per_service)

For example, assume the following:

  • logpartition_size = 600 MB
  • num_services = 2
  • logs_per_service = 3
max_log_roll_size = 600 MB / (2 * 3) = 100 MB

If you roll your logs over when they reach a specific size, the file size must be no more than 100 MB. Otherwise, the system runs out of disk space before you have three complete log files and scheduling the download and deletion of log files is much more complex.

To calculate the max_log_roll_size for your Access Gateway:

  1. Determine the values of the three variables listed above.

  2. Use the max_log_roll_size formula to calculate the maximum size a log file should reach before the cache device rolls it over.

4.2.3 Enabling Logging

Do not enable logging until you have designed a logging strategy. See Section 4.2.1, Determining Logging Requirements.

  1. In the Administration Console, click Devices > Access Gateways > Edit > [Name of Reverse Proxy] > [Name of Proxy Service] > Logging.

    Logging configuration

  2. Fill in the following fields:

    Enable Logging: Select this field to enable logging.

    Stop Service On Log Failure: Select this field if you want the Access Gateway to deny requests to this proxy service because the Access Gateway cannot log entries for it.

    Log Directory: Displays the default location for the log files for this proxy service.

  3. In the Logging Profile List, click one of the following options:

    • New: Click this option to create a new logging profile. Then specify a name and select either Common or Extended.

    • Default: Click Default to modify or view the settings for the Default profile. The Default profile uses the common log options.

    A logging profile determines the type of information that is written to the log file; it also manages rollover and old file options.

  4. Continue with one of the following:

4.2.4 Configuring Common Log Options

Use the common log options page to control log rollover and old file options. The data included in a log entry is controlled by a default configuration that includes the following:

  • Date and time of the request

  • IP address of the client

  • Remote host name

  • The request line as it came from the client

  • The HTTP status code returned to the client

  • The number of bytes in the document transferred to the client

The Access Gateway does not allow active log files to be deleted. Only log files that have been closed can be deleted. The rollover options allow you to control when a file is rolled over and closed, and a new file is created. The old file options allow you to control when the rolled-over log files are deleted.

To configure a default log file for a selected proxy service:

  1. Click Devices > Access Gateways > Edit > [Name of Reverse Proxy] > [Name of Proxy Service] > Logging > [Name of Common Log Profile].

    Configuring a common log profile

  2. Select one of the following roll over options:

    Rollover When File Size Reaches: Rolls the file when it reaches the specified number of megabytes.

    Rollover every: Rolls the file at the specified interval. You can specify the interval in hours or days.

    • beginning: Specifies the day that the interval should begin. You can select a day of the week or the first of the month.

    • at: Select the hour of the day that the interval should begin and the time zone (either the local time zone or GMT).

  3. Select one of the following old file options:

    Limit Number of Files to: Allows you to limit the number of old log files on the system to the number specified in this option. The oldest file is automatically deleted when this number is reached. All logging data in deleted files is lost.

    Delete Files Older Than: Allows you to configure the Access Gateway to delete files when they are older than the time you specify. All logging data in deleted files is lost.

    Do Not Delete: Prevents the system from automatically deleting the log files.

  4. Click OK.

  5. Click the Access Gateways link, then click Update > OK.

4.2.5 Configuring Extended Log Options

Use the extended log options page to control log entry content, log rollover, and old file options. A log entry always includes the date, time, and client IP address for each entry, but with the log data options, you can add other fields such as the IP address of the server and the username of the client.

The Access Gateway does not allow active log files to be deleted. Only log files that have been closed can be deleted. The rollover options allow you to control when a file is rolled over and closed, and a new file is created. The old file options allow you to control when the rolled-over log files are deleted.

To configure an extended log file for a selected proxy service:

  1. Click Devices > Access Gateways > Edit > [Name of Reverse Proxy] > [Name of Proxy Service] > Logging > [Name of Extended Log Profile].

    Configuring an extended log profile

  2. Select one or more of the log data options:

    Name

    Description

    User Name

    The name of the user sending the request.

    Server IP

    The IP address of the Access Gateway.

    Site Name

    The name of the reverse proxy.

    Method

    The HTTP method the browser sent to the Access Gateway.

    URI

    The HTTP URL the browser sent to the Access Gateway.

    URI Stem

    The stem portion of the HTTP URL the browser sent to the Access Gateway. The stem is everything in the URL up to the first question mark. If the URL has no question mark, the URI Stem field is the same as the URI field. URI Stem is redundant if URI is selected.

    URI Query

    The query portion of the HTTP URL the browser sent to the Access Gateway. The query is everything from the first question mark through the end of the URL. If the URL has no question mark, this field has no value. URI Query is redundant if URI is selected.

    Version

    The HTTP version specified in the URL the browser sent to the Access Gateway.

    Status

    The HTTP status code the Access Gateway sent to the browser.

    Bytes Sent

    The number of bytes of HTTP response data the Access Gateway sent to the browser.

    Bytes Received

    The number of bytes of HTTP request data the proxy service received from the browser.

    Time Taken

    The time in seconds it took the Access Gateway resources to deal with the request.

    User Agent

    The User-Agent HTTP request header value the browser sent to the Access Gateway.

    Cookie

    The Cookie HTTP request header value the browser sent to the Access Gateway. The Access Gateway doesn’t cache cookie information. Cookies can consume a lot of space. If you select this option, make sure it contains the critical information that you need.

    Referer

    The Referer HTTP request header value the browser sent to the Access Gateway.

    Cached Status

    The value indicates whether the request was filled from cache.

    • 1 = filled from cache
    • 0 = not filled from cache

    Fill Proxy

    The IP address of the upstream proxy.

    Origin Server

    The IP address of the Web server. This assumes the Access Gateway retrieved the requested information directly from the Web server.

    X-Forward-For

    The X-Forwarded-For HTTP request header value the browser sent to the Access Gateway. Do not confuse this with the X-Forwarded-For option, which causes the Access Gateway to generate or forward headers to upstream proxies or Web servers.

    Bytes Filled

    The total bytes filled in response to the request.

    Content Range

    The byte ranges sent from the Access Gateway to a requesting browser.

    E Tag

    The tag sent from the Access Gateway to a requesting browser.

    Completion Status

    The completion status for the transaction, indicating that it completed successfully or that it failed. Possible values: success, timeout, reset (the client terminated the connection), administrative (the Access Gateway terminated the connection).

    Reply Header Size

    The size in bytes of the HTTP header associated with a response to a client.

    X Cache Info

    Brief status statement for cached objects; brief reasons why an object was not cached.

    Range

    The Range header value.

    If Range

    The If Range header value, which indicates whether the browser request was a conditional range request.

    Content Length

    The size in bytes of the entire object delivered to a requesting browser.

    Request Pragma

    The pragma value associated with a browser request.

    Reply Pragma

    The pragma value associated with a server response to a requesting browser.

  3. Select one of the following rollover options:

    Rollover When File Size Reaches: Rolls the file when it reaches the specified number of megabytes.

    Rollover every: Rolls the file at the specified interval. You can specify the interval in hours or days.

    • beginning: Specifies the day that the interval should be begin. You can select a day of the week or the first of the month.

    • at: Select the hour of the day that the interval should begin and the time zone (either the local time zone or GMT).

  4. Select one of the following old file options.

    Limit Number of Files to: Allows you to limit the number of old log files on the system to the number specified in this option. The oldest file is automatically deleted when this number is reached. All logging data in deleted files is lost.

    Delete Files Older Than: Allows you to configure the Access Gateway to delete files when they are older than the time you specify. All logging data in deleted files is lost.

    Do Not Delete: Prevents the system from automatically deleting the log files.

  5. Click OK.

  6. Click the Access Gateways link, then click Update > OK.

4.2.6 Configuring the Size of the Log Partition

The size of the log partition should be configured as part of the installation process. The Access Gateway Appliance logs are stored in /root partition by default. You can create a /var partition to store the logs. The size of this partition depends on your requirements. For more information on creating the /var partition, see Customizing the Partitions in the Novell Access Manager 3.1 SP1 Installation Guide.