Message Processing

The core function of NetMail is message processing. Every message entering your messaging system---either inbound or outbound, from internal or external users---must be processed by NetMail agents. Depending on which agents are running on your messaging system, the message might be checked against a system blacklist, forwarded to another e-mail address, or filtered by user-defined rules.

The directory infrastructure that supports message processing comprises three basic components:

• Message Store
• Single Copy Message Store
• Message Queue

Everything that happens from the time a message enters the messaging system to when it is delivered to the user's mailbox occurs in these directories.

This section explains message processing both in terms of its directory infrastructure and the actual process. By understanding the internal structure of your NetMail system and how messages are processed, you can better manage your messaging system.

Message Store

The message store is the directory structure that holds users' mailboxes and messages. All other information required by your NetMail system is stored in eDirectory. Every server running an NMAP Agent has a message store. You can manually create additional message stores using Container objects or NetMail Parent objects. See Creating Separate Message Stores for Each Domain.

For information on mailbox file structure, see Mailbox File Structure.

Message Store Directory Structure

The default message store directories for each platform are as follows:

The NMAP Agent creates the following mailbox directory structure within the message store directory for every User object in its assigned contexts:

Single Copy Message Store

In cases when a message is sent to multiple recipients (by default, five or more) and the message is larger than a given size (by default, over 5 KB), NMAP does not replicate the message in all the users' mailboxes. Instead, a pointer message is sent to the individual mailboxes directing NMAP to the complete message in the Single-Copy Message Store (SCMS) directory. When the last user downloads or deletes the message, it is deleted from the shared directory.

For more information, see SCMS Message Structure.

Single Copy Message Store Directory Structure

The default single copy message store directories for each platform are as follows:

The SCMS directory structure stores messages in sorting directories that correspond to the last character in their hexadecimal filename (0-9 or A-F). Sorting messages by the last character in their hexadecimal filename allows the NMAP Agent to efficiently retrieve messages stored in the SCMS directories.

The SCMS directory structure is as follows:

Message Queue

Each message store has an associated message queue. The message queue is the directory where messages are processed as they enter or leave the NetMail messaging system.

For a complete discussion of how messages flow through the message queue, see Message Processing in the Message Queue.

Message Queue Directory Structure

The NMAP Agent is responsible for managing the message queue. When the NMAP Agent receives a message, it downloads the file to the message queue directory.

The default message queue directories for each platform are as follows:

Message Processing in the Message Queue

As the NMAP Agent downloads a message to the message queue directory, it generates two files: cxxxxxxx.in and dxxxxxxx.in. (The xxxxxxx represents a hashed value of the date and time the message was sent. The .in extension indicates the message is inbound.)

The cxxxxxxx.* file is the control file envelope. The control file is generated from information received in the initial SMTP exchange. It contains To and From information along with any message flags. All message processing functions are performed on the control file.

The dxxxxxxx.* file is the data file. It contains the message header and message text. Because most message processing is done through the control file, the data file is rarely opened.

NOTE:  For a complete breakdown of control file content and data file format, see Message Structure.

After the message is downloaded, NMAP gives the data file a .msg extension and then begins moving the control file through eight queues. The process of moving the control file between queues is a simple matter of changing its filename extension to reflect the current queue number.

The following table outlines the message-processing functions associated with each queue.

When the NMAP Agent moves a message into a queue, it notifies the agent registered on that queue. (Agents register on their queues when they first load.) The registered agent then scans the control file and performs its assigned function. If the agent modifies the message, NMAP returns the message to queue 0 for reprocessing. If no changes are made, NMAP advances the message to the next queue.

Message Processing: Step-by-Step Process

The following table provides a detailed, step-by-step breakdown of NetMail message processing. The outlined process includes all NetMail queue agents; however, on live systems the process includes only those agents actually loaded on messaging servers and registered with the NMAP Agent.

Message Processing Performance

NetMail processes messages very rapidly. The NMAP Agent is multi-threaded and can simultaneously process as many messages as memory will allow. To ensure optimal efficiency, NetMail automatically balances the NMAP Agent's thread usage with server utilization. When server resources are abundantly available, the NMAP Agent attempts to instantaneously fulfill all mail requests. However, when the mail load becomes heavy, NMAP tries to preserve server resources by limiting the number of threads that can be used for message delivery.

Under normal conditions, NMAP creates two threads to deliver every message; one to receive the message and one to push the received message through the queue. This state is called Concurrent Mode.

When the number of delivery threads exceeds the Concurrent Limit, new threads both receive messages and push them through the queue. This state is called Sequential Mode.

When the number of threads exceeds the Sequential Limit, new threads receive messages, but defer delivery until the total number of threads drops back below the Concurrent Limit. Only when NMAP drops back into Concurrent Mode will it deliver queued messages.

When the number of queued messages exceeds the NMAP Agent's load balancing settings, NMAP checks CPU utilization at a regular interval. If utilization exceeds the NMAP Agent's high threshold, NMAP lowers the Concurrent and Sequential Limits in an effort to lower utilization. If utilization drops below the NMAP Agent's low threshold, NMAP raises the Concurrent and Sequential Limits to make sure system resources are not wasted. The NMAP Agent's ability to self-adjust its load balancing settings ensures the NMAP Agent operates at optimal efficiency.

Magnifying the efficiency of the NMAP Agent, NetMail message processing is strategically designed to optimize performance. For example, separating each message into a control file and a data file speeds performance because most message processing is done using the small, compact control file. The larger data file is rarely opened.

The message queue itself is also designed for speed. Changing the control file's extension in a single directory rather than transferring the file between different queue directories makes advancing through the queue as efficient as possible.

Finally, having NMAP notify agents when a message is placed in their queues, rather than having the agents periodically scan their queues, means there is zero wait time between message-processing functions.