3.1 Retain Planning and Design Best Practices

There are so many variables that impact performance and design that no single document could adequately cover them. Instead, this document mostly discusses concepts so that the reader can apply them their needs. This document has four major sections:

  • Retain Architecture

  • Hardware Design

  • Retain Configuration

  • Backing Up Retain

This document's intent is to provide insight into how Retain works so you know how to design your system. It will provide some guidelines and best practices; however, it does not attempt to make hardware recommendations or provide specifications of such. It is anticipated that the reader will understand how Retain works, how the various hardware components play a role, and be able to make hardware decisions based on that understanding. Again, no two systems are alike, so one size does not fit all. And, keep in mind, Retain's minimum system requirements are MINIMUMS. Retain will work with those minimums; but, if performance is a priority, then read on and apply these concepts to your specific situation.

3.1.1 Retain Architecture

Retain can run on a bare metal server or on a VM running Windows Server or SuSE Linux. For backup purposes and flexibility, we recommend running it on a VM.

Retain consists of four major components:

  • Server

  • Worker

  • Indexer

  • Database


The Server is where the archive system is configured and maintained. It is responsible for storing, indexing, searching, and reading archived items.


The Worker is the component that interfaces with the messaging host/mail server containing the messages you are archiving. It retrieves the messages and passes them onto the Retain Server.


The indexer scans the messages and attachments to make them searchable, indexing each word and some phrases. When a user performs a search in his/her Retain mailbox, the list of messages returned is coming from the indexer, not the database. Understanding that difference helps you make decisions on memory configuration choices for Tomcat (which powers the indexer) and for the database. The index process is the most memory intensive (I.E. uses the most memory) part of Retain.


The database stores most of the Retain configuration as well as the message metadata, which is information about the messages being store (subject, sender, recipients, links to attachments, indexed state of messages, folder location of the message, etc). When a user logs in to his/her Retain mailbox, the list of folders and their messages are being retrieved from the database.

3.1.2 Archive Job Flow

When an archive dredge job is running the work flow follows this pattern.

  1. The Worker polls the Server every 10 minutes (default) for new queued jobs it may need to run. It launches the job.

  2. The Worker connects to the mail system and does a login to each mailbox. For every mailbox, it requests the items in that mailbox based on the settings in the profile.

  3. The mail server responds by sending the items to the Worker.

  4. The Worker sends smaller items to the Server. For large items, it sends the items' metadata and awaits instructions from the Server as to whether the item already exists.

    If it already exists, the Server notifies the Worker that it does not need to send the item over.

    If it does not exist, the Server notifies the Worker to send the item.

  5. The Server updates the Retain database with the item's metadata if it does not already exist.

    A good example of this is when User A sends message 1 to User B. When User A's mailbox is archived, Message 1 gets stored on disk with a record getting written in the database pointing to the file on disk and associating the message to User A's mailbox. When User B's mailbox gets archived, the Server notices that message 1 is already stored on disk and that a record in the database exists for that message. Now it updates that record and associates it with User B's mailbox.

  6. The Server adds the item to the storage area on disk.

  7. The Server launches the indexing process (if it is not already running) to begin the indexing process.

  8. The Indexer indexes any items that need to be indexed.

3.1.3 Hardware Design

There are four major considerations you need to take into account when designing the hardware for a Retain system:

  • Bandwidth

  • CPU

  • Storage/Disks

  • RAM

  • VM Configuration


The Worker will be querying your messaging system for messages to be sent to it and receiving every one of those items; however, not all of those items will be sent over to the Server. If the link between the Worker and the messaging system is slow, you should consider placing the Worker on the messaging system's server or on a server that has a fast link to the messaging system.

The only downside to this strategy is software updates. When upgrading Retain software, you will have to go to each server hosting a Worker and upgrade its software. If the Worker is running on the Retain Server itself, then the installer automatically upgrades every Worker on the Server.


Retain is multi-threaded and able to make use of multiple CPU cores. The base server uses 4 threads, and the Indexer starts with 3 threads. If more than 7 CPU cores are available, additional Indexer threads are spawned. The basic formula is [cores - 4 (minimum 3)].


Retain Server Threads

Indexer Threads




























We have found that 8 CPU cores provides a sweet spot for performance gains, allowing Retain 4 threads and the Indexer 4 threads.


Since Retain is an archiving solution, it will happily fill up your storage system. While it will warn you of impending doom, it is up to you to keep bad things from happening like the hard drive filling up completely. It is difficult to recover from that since not even the OS works very effectively under those conditions. You need to design in the ability to add extra storage easily. Retain stores data in the archive, database and indexes the data for search. Retain also logs what is happening for auditing and troubleshooting purposes.

Since the point of Retain is data safety, we will design around that priority. It is fine in a lab setting to just leave everything on a single drive, but in production, you should desire something more robust. At the very least put the OS on its own partition. It is much easier to recover if a data drive is full rather than a boot drive. It is also important to consider what needs to be done to backup RetainBacking Up Retain.

Size - Archive Files (BLOBs)

The trend with disk storage is that it is increasingly becoming less expensive. Micro Focus cannot tell a customer how much storage space will be needed - it is almost impossible to predict with any degree of accuracy. A general rule of thumb is current size + rate of yearly growth. Understanding the trends of your email growth, the current space consumed by it, and how Retain stores those messages can help you in your planning. If anything, you'll want at least enough storage space to handle your initial archive job of all your email to date and up to the next year or two. You can always add more disk space and Retain storage partitions after that.

Retain's storage design is efficient. Only one instance of any message or attachment is stored on disk, regardless of how many users received the same items and regardless of whether those users are on the same mail server.

Retain also allows you to expire and delete messages stored in the system after a specified time period, saving you disk space as well.

According to some research we found on the Internet, the average number of emails per user is currently 120 per day or 32,000 emails per year given an average workweek and increasing over the years. The average size of an email is 75kb. Do some math and the average is 2.4GB per user per year. From experience, we have seen typical customer email systems increase in size by about 11% per day.


There are a variety of factors that determine how much disk space Retain will consume. The biggest factors are number of messages per day and the length of time you intend to archive your data. It may be difficult to figure out either of those factors. If you have a virtualized environment, you can allocate more space than you think you will use and thin provision the disks. For our cloud customers, we typically set the db partition to 500 GB and go from there. If a partition runs low on disk space at any point, support can direct you on the proper steps to move the data to another partition if necessary.

The numbers provided in the following table are simply representations of three different systems. Remember, no two systems are alike. Two customers with the same number of messages in their system may have vastly different database sizes due to the difference in the message metadata. For example, Customer A may have short distribution lists while Customer B has a lot of emails with hundreds if not thousands of recipients associated with the messages; therefore, this sample data may or may not represent what you may experience with your system. The purpose of providing sample data is to get you into a possible ballpark range. If anything, err on the side of going higher on the storage need.

Example Systems

Server A

Server B

Server C

Message Count




Archive Size

5.3 TB

1 TB

115 GB

File Size per Message (archive)

4.54 KB

4.71 KB

6.21 KB

Database Size

455 GB

82 GB

16 GB

File Size per Message (database)

56.27 KB

64.02 KB

45.06 KB

File System

When selecting the type of file system on Linux, we recommend going with XFS because it dynamically creates iNodes whereas ext3 forces you to configure those up front. Once you run out of iNodes, nothing can be written to that disk, even if you have plenty of disk space left. XFS is also popularly regarding as performing faster as well.

It is NOT recommended to use ReiserFS, performance on that file system is much worse than XFS.

Other Disk Partitions

Let's talk concepts first, so you can determine if the following recommendation makes sense for you.

Retain archive jobs are disk I/O intensive. You have message content being stored in the archive directory, each message is getting indexed, you have the database getting updated with message metadata, and you have logs being written to continuously. That is a lot of disk access.

Now, let's apply that information to various disk configurations:

Physical ("bare metal") Server

This is your old-school standalone server not running as a VM. In this situation, Retain is installed on a physical machine, not a virtual one. The server has its own locally attached disk. If there is just one disk, you have a lot of disk I/O contention during a job and that will negatively impact performance. In such a case, the recommendations we will make in this section would apply.

VM Guest on Host With Local Disks

Your Retain server is a VM guest running on a physical server that has only local disks. Nothing is on a NAS or SAN. In such a case, the recommendations we will make in this section would apply. You would simply create multiple disks and just ensure that each one gets created on a different datastore.


This could be physical box where the storage is mounted/mapped to a NAS or SAN; or, this could be a VM guest where:

  • The VM guest itself is stored on a NAS/SAN; thus, the VM guest's "local disks" are also sitting on a NAS/SAN; or,

  • The VM guest itself is stored on the hosts local disks but the "local disks" of the VM guest are on datastores residing on a NAS/SAN; or,

  • The VM guest is mounting volumes stored on a NAS/SAN.

If the Retain storage will ultimately be on a NAS/SAN and if the volumes are expandable on the fly, then the recommendations we make don't necessarily apply. As you can see, there are so many configurations that a discussion on what exactly to do becomes impossible. The reader really needs to understand what Retain is trying to do and then see what can be done on the hardware end to facilitate best performance.

If it is a NAS/SAN, that doesn't necessarily translate into great disk performance. Many have a NAS but call it a "SAN". So then you have to consider the pipe speed to the storage: 1 gigabit/sec is considered very slow. On top of that, you have to consider how many disks are in the array, their RAID configuration, and the speed of the disks themselves.

For those situation where it makes sense to follow our recommendations:

There is a logical reason for separating your archive files from everything else and a performance reason.

If all the Retain storage is located on the same volume and you run out of space, Retain provides the ability to create additional storage volumes for the archive files. After an additional logical storage volume is created within Retain, all archive files from that point forward as archive jobs run will go to the new location. That works great.

However, the indexes will continue to grow and Retain doesn't have the ability to partition indexes. Some customers have run out of disk space, created new logical storage partitions that point to another volume, but then run into problems with their archive jobs because they are still out of disk space for the INDEXES. Thus, for logical reasons, you want to have your archive files on a separate volume to begin with unless the volume containing the Retain storage is expandable on demand.

If it makes sense to do so (based on all the concepts previously discussed), you'll want to separate your archive files from your indexes and from your database, which means two to three other partitions on your Retain Server in addition to your OS partition. If your database is on a separate server from Retain, then only two other partitions are needed; otherwise, you'll want three additional partitions.

Data Partitioning

We recommend dividing up your storage directories onto separate disks, so beyond the OS disk there should be:

  • Disk 1: Archive

  • Disk 2: Index (250G start). For best search performance, consider making this a solid state drive.

  • Disk 3: Logs, xml, ebdb, export, backup, and license (150 - 200G)

  • Disk 4: Database (if on-board)

Disk 2 should be expandable and you'll want to give it room for the indexes to grow; but, if you cannot do that, then when it runs out of space, you'll simply need to move your index files to another volume with more disk space in the future. For disk 2 - as mentioned previously - you may want to consider an SSD, as that would increase the search performance.

If disk 1 and disk 2 can literally be on different "spindles", then you will also get some performance gains from that because an archive job writes simultaneously to the archive directory, the index directory, and to the database. If each of those are on different physical disks (a.k.a., "spindles"), then this eliminates disk contention bottlenecks. Smaller systems may not need to be concerned with performance while larger systems that have archive jobs running for hours may want the performance gains.

Using disk 3 for logs is especially helpful for larger systems. If you have 6 Workers averaging 5 - 10 messages per second, expect a RetainServer log of around 60G unzipped. Plan for 150 - 200G for your logs directory. For the initial archive job, the rule of thumb is 10G per day per Worker. If you do not use a third disk, then the logs will go to the OS partition and that could spell trouble. Also, if users access their archives often and perform PDF exports, that can grow as well. The xml, ebdb, and license directories are pretty much static with minimal to no growth. The backup directory is a backup of the index directory and other important items. However, if the disk begins to run out of room, you can easily copy this data over to a larger disk at some future time and point retain to that new disk.

Finally, if your database is on the Retain Server, you'll want a third or fourth disk for it (depending on whether you decided to dedicate a disk for your Retain logs).

If performance is an issue, not only will you want all three of these partitions/disks to be on physically different spindles (or at least written to a NAS/SAN with many disks that it can swipe across), but you'll benefit from putting indexes and the database on high speed drives. Your archive directory does not need the performance and can be on less expensive disk media.

Make sure to set the permissions of the new disks correctly in Linux or the installation will fail.

Disk Performance

This comes up a lot; thus, it bears repeating what was mentioned previously. A customer will claim that they are on a fast SAN. Often, it is not a SAN but a NAS and there are many considerations that go into performance. So, knowing that disk I/O is the top issue with archive job performance, it is best to plan out your disk storage accordingly.

Storage design and disk I/O has everything to do with Retain performance as archive jobs are I/O intensive. You have the following processes writing to disk simultaneously:

  • The indexer to the [storage path]/index

  • The database (if on the Retain server)

  • The Retain Server to [storage path]/archive

  • The Retain Server to the logs directory:

    • Linux: /var/logs/retain-tomcat8

    • Windows: [drive]:\Program Files\Beginfinite\Retain\Tomcat8\logs

With all of that disk activity, if a single spindle (drive) is having to handle all of it, then you can see that the performance bottleneck would be disk I/O. However, many disk systems these days involve multiple disks using (i.e., RAID 5 or RAID 10) that write the data across multiple disks. The more disks involved, the more you are spreading the load and typically the faster the disk performance will be. You also have a difference in drives (SATA/SAS/SSD). In those cases, you now are looking at whether the disks are local to the server or in a SAN/NAS.

RAID Considerations

Let's say your server employs RAID 5, which provides better redundancy than, say, RAID 10. If there were 4 disks. As you know, RAID 5 uses an extra parity bit that consumes an entire disk, which leaves it with 3 drives on which to stripe across. If one of those drives becomes unavailable, that leaves you with 2. Striping across 2 or 3 drives doesn't lend for great speed, especially if the disks are lower end SATA drives.

SAN / NAS Considerations

If on a SAN/NAS, now you are looking at the network link speed as well. You could have very fast drives, but if your link speed is 1 Gb/s, your bottleneck is going to be your link.

The 1 GB/s network link is slower than a SATA 2 or 3 connection (AKA SATA 3 Gb/s and SATA 6 Gb/s.) Your SATA 2 connection (which is now getting to be a pretty old standard) is 3x faster than a 1000 Mb/s network link (or 1 Gb/s network connection). A fast single HDD can saturate a 1 Gb/s connection but not quite a 3 Gb/s connection (SATA 2.0, or SATA 3 Gb/s) with a sequential read/write. 7,200 RPM platter drives usually top out around 160-170 MB/s (or 1.28-1.36 Gb/s).

Measuring Disk Performance

It really comes down to IOPS. Here is a very simple IOPS calculator: http://www.thecloudcalculator.com/calculators/disk-raid-and-iops.html or you can find one of your own.

So, it really comes down to you understanding your underlying disk storage. This article just gives food for thought. If you are running Retain on a VM guest server like most customers do, then you need to also understand your VM host and VM infrastructure. Is the Retain storage viewed by the server OS running on the VM guest as "local" storage? If so, what type of disk system is holding your VM's datastore? If it is not local storage but the server is connecting to external storage, then you need to take a look at the external system's configuration.

Bottom line: Disk I/O performance is key to Retain's performance and there are several areas to investigate where the bottlenecks could be.

In addition to partition considerations, make sure that your storage is reliable. NFS mounts can be problematic, so you may want to shy away from those. NSS volumes are not supported, so do not use them.


The amount of memory depends on the number of active mailboxes you are archiving, the mail volume, your underlying hardware, and how your Retain system will be used.

We'll first discuss the concepts so that you can apply them to your system and we'll give you some general guidelines. In most instances, you will need to experiment with various memory configurations until you find what works best in your environment.


Retain runs under Tomcat as shown at the beginning of this article and Tomcat runs on Java. The Retain Server will use the Java "heap" for its memory and the indexer will use the OS memory as well as virtual memory (see the Virtual Memory subsection below). For this reason, you should configure Tomcat/Java with the bare minimum to have it run in an acceptable fashion for you. If logins or Retain in general seems sluggish when in the mailbox or using the web admin tool, you may need more heap. The sweet spot for most systems with a single Worker installed on the local Retain server is 8 GB minimum (xms) and maximum (xmx). You want to leave as much RAM as possible for the Indexer, which uses non-heap RAM.

The amount of Java heap you set will depend on total RAM on your system and the number of Workers you install in addition to the default single Worker. As we grow in customer experience with Retain 4, we adjust this article's memory recommendations accordingly.

Right now, development has suggested 1 - 2GB per additional Worker beyond the 8 GB you normally would give to the Java heap for a system with a single Worker local to the Retain server; however, we've had a customer with 110 million messages with 7 Workers local to the Retain server get away with 8 - 10 GB of RAM, but that is really pushing it. They didn't run under that configuration for more than 24 hours, so we cannot tell whether it would have been successful in the long run.

The installer for Retain 4.0.1 and later tunes Tomcat/Java memory based on total RAM and which Retain components are installed. See the online manual's topic, "Tomcat Memory tuning" (note: that link goes to the 4.0.1 documentation, so if the link doesn't exist in the future, go to the online manual and find that topic). Again, as we learn more from customer experience, the installer's default RAM configuration is subject to change.

If you really want the fastest search performance, load it up with RAM, like 64GB or more. Systems with large numbers of messages (100 million or more) seem to be needing 64 GB of RAM or more. If you have a database system running on your Retain Server as well and multiple local Workers, then those will cut away at available RAM for the indexer, so you need to take that into account. The indexer wants to cache indexing data into RAM and memory access is much quicker than disk.

General Guidelines

All of this really depends on the priority you place on Retain performance. If a customer is only interested in getting data into Retain and it doesn't matter how long the archive jobs take (as long as they finish within a 24-hour timeframe) nor does the customer care how long it takes to search for messages (because they do not do it that often), then none of this matters.

The key test is how quickly tomcat shuts down and how much memory the OS is sending to swap. If tomcat is shutting down slowly, that's probably an indication that it has code in swap memory that it is having to call off of disk in order to close out. Reserving more memory for the OS should alleviate that problem; thus, reserve a minimum of 4G for the server OS right up front. On some systems, we have had to allocate more, on others, less. So, the key is to try different configurations on your system to see what makes the difference.

Once you have subtracted the OS memory from your total memory, give 2 - 4G of RAM to the database (if the database is on the same server; otherwise, the remainder can go to Tomcat). Note that Tomcat will need a minimum of 2G.

For small systems (1 - 250 mailboxes), 8G of RAM might be OK if that's all you can afford to allocate that. It is true that a small Retain system can run on 4G, but performance will be awful in most cases. If even decent performance matters, you really should not go lower that 8G unless you are a very small business and have 0 - 50 mailboxes. You might even want to consider trying 12 - 16G and see what difference that makes for you. For some, it will make a big difference. For others, it may make no difference as the bottleneck is elsewhere.

For medium sized systems (250 - 750 mailboxes), 12 - 16G of RAM should be considered.

For larger systems, 16G should be considered a minimum. Many large systems range from 24 - 48G of RAM. The more mailboxes and mail volume, the more RAM you might consider giving your Retain server. But, again, we have to emphasize that every system is unique and RAM may not be the biggest performance factor for them.

Case in point: We have a customer with 700 users that found allocating 24G of RAM made a big difference. In another case, a customer that had 1,500 users needed only 12G. We have systems with thousands of mailboxes and those systems do benefit from increased memory allocation, but their needs vary.

Tomcat Memory Configuration

Tomcat memory is manually configured. The latest version of Retain sets it to 8G by default. It is an industry best practice to set the minimum and maximum memory values to the same value.

In Linux

You set the Tomcat memory parameters in a file called j2ee found at /etc/opt/beginfinite/retain/tomcat8. See Tomcat Memory Tuning for more detail. Tomcat must be restarted after configuring it.

In Windows

You can set Tomcat parameters by running Programs | Tomcat 8.0 | Configure Tomcat. Go to the "Java" tab to set them. Note, we also recommend setting the stack size to 256k (it defaults to 160k in Windows).

Database Memory Configuration

Since most organizations employing Oracle or MS SQL have someone designated as a database administrator (DBA), they typically understand memory configuration. What they need to know is that archiving speed and user mailbox browsing performance is affected by the amount of memory given to the Retain database.

Virtual Memory

If you have the available disk space, we recommend increasing the virtual memory to at least 50GB. In Linux, this is known as swap. In Windows, this is called the page file. Ideally, this swap or page file should be placed on a fast storage for performance reasons.

VM Configuration

VM (Virtual Machine) NIC Settings

We have found that using VMXNET3 for the network adapter in VMs helps performance.

Virtual Machine SnapShots

We have found that VM snapshots can reduce performance of the Retain Server. Keeping the number of snapshots to a minimum is highly recommended.

Resource Restrictions

See "Retain tomcat becomes unresponsive after several mailbox searches", which addresses a VM resource limitation setting that - if set - can affect the performance of Retain and cause it to freeze.

3.1.4 Retain Configuration

If your Retain system was installed with version 3.0 or older, we recommend checking some settings and consider making some adjustments:

Logging Level = Diagnostic

Both your Server and Worker logging levels should be set to "diagnostic" unless you do not have the available disk space on your root partition.

The Worker logging level is found under Data Collection | Workers | [select the Worker object] | Logging.

The Server logging level is found under Configuration | Server Configuration | Logging.

Indexing All Content

Indexing all content is now the default.


Under the Connection tab of the Worker object (Data Collection | Workers), be sure the port is set to 48080. For systems created with Retain 3.2 and later, it defaults to port 48080; however, if the Worker object was created with an earlier version of Retain than 3.2, the port default was 80. This causes the Worker traffic to the Server to pass through the web server, which slows the job down. Avoid SSL unless required by security requirements. It will cut the job speed in half.

For larger systems, you can divide up your jobs and assign each job to its own Worker. Only one job can run at a time for a given Worker; thus, if you install multiple Workers, you now can have those jobs run simultaneously in parallel. We recommend that each Worker object (created in the Retain Server administration interface) be given a name that describes its location and - if on the Retain server with other Workers - the directory to which it was installed. For multiple Workers on the same server, each Worker program directory will be named RetainWorker (the first installed Worker) and RetainWorkerX (where "X" is the Worker number assigned during the installation).

For installing additional workers on Linux, see Installing Additional Workers on Linux.

For installing additional workers on Windows, see Installing Additional Workers on Windows.

For Office 365 and Gmail, you will definitely want multiple Workers as archiving over the Internet from those systems is very slow - especially with Gmail. In those cases, create multiple distribution lists within those systems and create a separate job for each distribution list.

Archiving from Reload

Several years ago when GroupWise first supported the SOAP protocol, it was unstable. When Retain would run a job against a post office, the POA would often crash. Because this issue had such a big impact on customers, Micro Focus developed a method for archiving from a Reload backup designed to be a temporary solution until all of the SOAP issues in GroupWise were addressed. Novell eventually addressed the issues in their product and it became reliable and stable.

It is a recommended best practice to archive directly from the live post office and not from Reload. The archive job has a very light footprint on the post office - it only logs into one mailbox at a time and requests items. It would be like running the GroupWise client in caching mode and downloading mail on a daily basis. In addition to that, the performance and reliability of the archive jobs is much higher going straight to the production post office. There can be issues when archiving from Reload backups, especially if you don't stay right on top of the jobs to ensure they are completing properly.

We do not recommend running archive jobs against Reload unless there is a stability issue with your production post office. In such cases, that should be a temporary solution.




One of the main points of having Retain is to have a complete archive of messages in your system because of data retention policies under which your organization must operate. If your email system's retention functionality is not understood or properly configured, users could purge items from their trash folder before Retain gets a chance to archive it. See the KnowledgeBase article: "How Retention Services and Item Store Flags Work".

Message Placement

Also, users make take a few days to organize their mail into folders. When the Retain archive job runs, it will take a "snapshot" of the user's mailbox as it exists at the time of archiving that mailbox.

If an Item A is in the main mailbox folder at the time of the job, it gets placed into the main mailbox folder in the Retain mailbox. If the user subsequently moves the item to another folder, Retain will not remove the item from the main Retain mailbox folder; instead, Retain simply creates an additional pointer to the item in the folder to which the user moved the message. This means the item will be found in the main mailbox folder and in the folder the user eventually moved it to.

To avoid this, some customers set the Date Range to Scan to Number of days from job start (older) and set the number of days to a value between 7 - 21. This gives the users that extra time to organize their new email before Retain archives it.


To save disk space, enable the option Don't store MIME.822 attachments and then activating the settings:

  • Store/index Internet Headers

  • Include Routing Properties

These settings will grab the data that is in the MIME.822 attachments and store and index them.

We also recommend only including shared folders that are owned by the mailbox. The other setting can cause errors in the job due to circular folder references.

Under Disabled/Expired Users, set "When Retain encounters a disabled or expired mailbox" setting to "Ignore".



One of the main points of having Retain is to have a complete archive of messages in your system because of data retention policies under which your organization must operate. If your email system's retention functionality is not understood or properly configured, users could purge items from their trash folder before Retain gets a chance to archive it. See the KnowledgeBase article: "How Retention Services and Item Store Flags Work".

Message Placement

Also, users make take a few days to organize their mail into folders. When the Retain archive job runs, it will take a "snapshot" of the user's mailbox as it exists at the time of archiving that mailbox.

If an Item A is in the main mailbox folder at the time of the job, it gets placed into the main mailbox folder in the Retain mailbox. If the user subsequently moves the item to another folder, Retain will not remove the item from the main Retain mailbox folder; instead, Retain simply creates an additional pointer to the item in the folder to which the user moved the message. This means the item will be found in the main mailbox folder and in the folder the user eventually moved it to.

To avoid this, some customers set the Date Range to Scan to Number of days from job start (older) and set the number of days to a value between 7 - 21. This gives the users that extra time to organize their new email before Retain archives it.


Enable these settings:

  • Store/index Internet Headers

  • Include user's recoverable items.

We strongly recommend abandoning the practice of using journal mailboxes to meet retention requirements (ensuring nothing gets deleted before being archived). Journaling doubles the traffic on your Exchange server and can adversely impact performance. Instead, have Retain archive the Recoverable Items folder in each mailbox and configure Rolling In-Place Holds. See the KnowledgeBase article: "How to Transition from Journaling to Rolling In-Place Hold for Exchange Archiving" on how to configure this. It also provides links to articles on other recommended settings in Exchange.

Reporting & Monitoring Server

This tool provides archive stats as well as archive job stats. In Retain 4, it becomes a very critical tool for identifying errors in your jobs because it not only shows the error counts, but through drill downs on the error numbers, an administrator can see the exact message on which the error occurred, the folder in which it is stored, its delivered date, and the error description.

If you use this tool, make sure that when configuring it, you check the box under Core Settings that reads, Disable the disk usage statistic. This utility is too much of a drain on system resources and will be re-designed in the future. In the meantime, disable it. In Retain 4, this utility is disabled by default.

And, like the Worker, the port settings under the Connection tab should all be set to 48080 if possible so that the Server and the R&M Server are talking directly to one another's tomcat port.

3.1.5 How Retention Services and Item Store Flags Work

Retain keeps an "item store flag" to ensure that no item gets left behind.

With Exchange and O365 Holds and the Recoverable Items folder can be used for retention compliance. With On-Premise Exchange a journaling mailbox can be used but it is not recommended.

Gmail, by default, does not have a retention service.

GroupWise, on the other hand, has its own built-in feature called "Retention Services" that prevents items from being emptied from the mailbox until they have been successfully archived.

This article will first discuss Retain's support of the GroupWise Retention Services feature followed by a discussion of how Retain ensures that all items get archived in all other email systems.

GroupWise Retention Services

GroupWise has a feature that can be enabled in its GroupWise Administration option called Retention Services.

When enabled, GroupWise will prevent a user from emptying an item from Trash that has not yet been confirmed to have been archived. The way it does this is through a date/time field in each user database called the "digest retention time". It relies on third party archiving solutions like Retain to set that date/time, but GroupWise is the one that enforces it when set. What this does is it prevents any item newer than the date/time set in the "digest retention time" field from being emptied from Trash. This "digest retention time" is known in Retain as the "retention flag".

When Retain runs an archive job on a mailbox, it sets the digest retention time to the date/time of the newest/latest message it archived. However, if an error occurs on any item during that job which prevents Retain from archiving it or its attachment, Retain will set the digest retention time in the GroupWise user database for that mailbox to the date/time of the item that could not be archived due to an error.

And, even though Retain encounters an error on an item and cannot archive it, it moves beyond that item and continues to archive all other mailbox items; however, again, it will not advance the retention flag past the date/time of the FIRST error it encountered. Thus, when the next archive job gets run on that mailbox, Retain checks the digest retention time set in the user database and uses that date/time as its starting time for the new job.

Example: If today is September 17, 2014 but an item in the previous job produced an error, could not be archived because of that error, and had a delivered date/time of September 15, 2014 09:15, then when today's job runs, it will ask GroupWise for all items it has beginning with September 15, 2014 09:15 and on.

Now let's say that a month has passed and the problematic mail message has not been properly dealt with and we run a job. Even though Retain may have archived all items in the user's mailbox up to - let's say October 15th - it will still start the query with the digest retention time of September 15, 2014 09:15 because it could not advance the retention flag. If it were to do so, then they problem message would never get archived because Retain starts the query for items beginning with the digest retention time. Thus, if Retain were to advance the flag to the date/time of the newest/latest item it archived, then the problematic message would fail to fit within the query range and GroupWise would never send it to Retain.

So, when users complain they cannot empty trash, this typically has a few possible causes:

  • They have not exited the GroupWise client from the day before (or since a Retain job ran). The client checks the digest retention time upon loading but never checks it after that. Solution? Have the user close GroupWise and make sure that Notify also closes; otherwise, Notify keeps the current session open and relaunching the client is the same as not having exited the client in the first place; thus, it does not check the digest retention time because it is entering an already existing session. When the client launches and starts a new session, it checks the digest retention time and stores that for the session.

  • An archive job has not been run for a few days. Login to the RetainServer web UI and check the job status to see if it has run lately. If not, determine the cause (job disabled? license expired? Worker lost connection with Server? Out of disk space on Retain server? etc) and run the job.

  • The user that cannot empty trash had errors on an item or items in his/her mailbox during the archive job. Check Mailbox Error Monitoring in the Retain Server console.

  • Corruption/issues in the GroupWise user database for that mailbox (least likely of all scenarios). If this is the case, run GWCheck and rebuild the user database or contact Novell GroupWise support for help.

Exchange and Office 365

These email systems do not have a built-in retention service similar to GroupWise, there is no "digest retention time" field in any of their mail system databases that Retain can use; thus, Retain uses its own field in the "retain" database to keep track of its job starting point. This "item store flag" works just like the "retention flag" with GroupWise jobs. That date/time gets set to the date/time of the newest/latest item archived for a given mailbox; or, if an error(s) occurred during a job, the item store flag gets set to the date/time of the first item that had an error. That way, when the next archive job runs, it starts with the date/time of the item store flag, ensuring that Retain will see that item until it is properly archived. However, it is important to note that not advancing the item store flag does not prevent the user from emptying the item from their Trash in these email systems because they do not have a retention feature similar to GroupWise.

To prevent items from being deleted from Exchange/O365 a hold must be placed on the mailboxes. This can be an In-Place or Litigation hold. When a user deletes a message from Outlook the message is moved to the Trash, the user can then empty the trash. Exchange/O365 will then move the message to a Recoverable Items folder for 14 days before removing it from disk. However, a user can right-click on the trash and attempt to recover a deleted item, and at this point can purge an item immediately to remove it completely. This may be against your data retention policy, so to prevent the deletion a hold will then move the item to the hidden Purged folder, where the user cannot remove it but Retain can still archive it.

Alternatively, a journaling mailbox may be used on On-Premise Exchange. When a journaling mailbox is set up in Exchange, it can be configured in a way that redirects a copy of each message that is either sent or received throughout the entire mail system into they journaling mailbox. Retain can be configured to include the journaling mailbox in its archive job. Thus, even if a user empties an item from Trash, a copy of that item already exists in the journaling mailbox and remains in that mailbox until it is archived by Retain. If configured properly, Retain will remove that item from the journaling mailbox upon successfully archiving it. Items emptied from a user's Exchange mailbox but archived from the journaling mailbox do not appear in the user's Retain mailbox; however, they are searchable using the Retain search feature.

Because of the fact that duplicates of all email messages system wide get placed in the journaling mailbox, it can fill up fast. For this reason, we recommend that you not use the journaling mailbox feature and go with the Recoverable Items feature instead. If the journaling mailbox gets too big, Exchange will no longer be able to serve the mailbox. Thus, when Retain tries to run an archive job against it, it will fail because Exchange never responds back. This is why it is no longer recommended.


In the case of Gmail, it does not have retention services. It requires the purchase of their Vault service.