1.10 How Retention Services and Item Store Flags Work

Retain keeps an "item store flag" to ensure that no item gets left behind.

With Exchange and O365 Holds and the Recoverable Items folder can be used for retention compliance. With On-Premise Exchange a journaling mailbox can be used but it is not recommended.

Gmail, by default, does not have a retention service.

GroupWise, on the other hand, has its own built-in feature called "Retention Services" that prevents items from being emptied from the mailbox until they have been successfully archived.

The following sections explain Retain's support of the GroupWise Retention Services, followed by a discussion of how Retain ensures that all items get archived in all other email systems.

1.10.1 How Retain Works with GroupWise Retention Services

GroupWise has a feature that can be enabled in its GroupWise Administration option called Retention Services.

When enabled, GroupWise prevents a user from emptying an item from Trash that has not yet been confirmed to have been archived. The way it does this is through a date/time field in each user database called the "digest retention time". It relies on third party archiving solutions like Retain to set that date/time, but GroupWise is the one that enforces it when set. What this does is it prevents any item newer than the date/time set in the "digest retention time" field from being emptied from Trash. This "digest retention time" is known in Retain as the "retention flag" .

When Retain runs an archive job on a mailbox, it sets the digest retention time to the date/time of the newest/latest message it archived. However, if an error occurs on any item during that job which prevents Retain from archiving it or its attachment, Retain sets the digest retention time in the GroupWise user database for that mailbox to the date/time of the item that could not be archived due to an error.

And, even though Retain encounters an error on an item and cannot archive it, it moves beyond that item and continues to archive all other mailbox items; however, again, it doesn’t advance the retention flag past the date/time of the FIRST error it encountered. Thus, when the next archive job gets run on that mailbox, Retain checks the item store time set in its database of the user and uses that date/time as its starting time for the new job, minus one hour.

Example: If today is September 17, 2014 but an item in the previous job produced an error, could not be archived because of that error, and had a delivered date/time of September 15, 2014 09:15, then when today's job runs, it asks GroupWise for all items beginning with September 15, 2014 08:15 and on.

Now let's say that a month has passed and the problematic mail message has not been properly dealt with and we run a job. Even though Retain may have archived all items in the user's mailbox up to - let's say October 15th - it still starts the query with the item store time of September 15, 2014 08:15 because it could not advance the retention flag. If it were to do so, then the problem message would never get archived because Retain starts the query for items beginning with the digest retention time. Thus, if Retain were to advance the flag to the date/time of the newest/latest item it archived, then the problematic message would fail to fit within the query range and GroupWise would never send it to Retain.

1.10.2 How Retain Works with Exchange and Office 365

These email systems do not have a built-in retention service similar to GroupWise, there is no "digest retention time" field in any of their mail system databases that Retain can use; thus, Retain uses its own field in the "retain" database to keep track of its job starting point. This "item store flag" works just like the "retention flag" with GroupWise jobs. That date/time gets set to the date/time of the newest/latest item archived for a given mailbox; or, if an error(s) occurred during a job, the item store flag gets set to the date/time of the first item that had an error. That way, when the next archive job runs, it starts with the date/time of the item store flag, ensuring that Retain tracks the item until it is properly archived. However, it is important to note that not advancing the item store flag does not prevent the user from emptying the item from their Trash in these email systems because they do not have a retention feature similar to GroupWise.

Placing a Hold Prevents Loss of Unarchived Messages

To prevent items from being deleted from Exchange/O365 a hold must be placed on the mailboxes. This can be an In-Place or Litigation hold. When a user deletes a message from Outlook the message is moved to the Trash, the user can then empty the trash. Exchange/O365 then moves the message to a Recoverable Items folder for 14 days before removing it from disk. However, a user can right-click on the trash and attempt to recover a deleted item, and at this point can purge an item immediately to remove it completely. This may be against your data retention policy, so to prevent the deletion, a hold then moves the item to the hidden Purged folder, where the user cannot remove it but Retain can still archive it.

Journaling Mailbox, an Alternative to the Item Store Flag but Not Recommended

Alternatively, a journaling mailbox may be used on On-Premise Exchange. When a journaling mailbox is set up in Exchange, it can be configured in a way that redirects a copy of each message that is either sent or received throughout the entire mail system into they journaling mailbox. Retain can be configured to include the journaling mailbox in its archive job. Thus, even if a user empties an item from Trash, a copy of that item already exists in the journaling mailbox and remains in that mailbox until it is archived by Retain. If configured properly, Retain removes that item from the journaling mailbox upon successfully archiving it. Items emptied from a user's Exchange mailbox but archived from the journaling mailbox do not appear in the user's Retain mailbox; however, they are searchable using the Retain search feature.

Because of the fact that duplicates of all email messages system wide get placed in the journaling mailbox, it can fill up fast. For this reason, we recommend that you not use the journaling mailbox feature and go with the Recoverable Items feature instead. If the journaling mailbox gets too big, Exchange is no longer able to serve the mailbox. Thus, when Retain tries to run an archive job against it, it fails because Exchange never responds back. This is why it is no longer recommended.

1.10.3 How Retain Works with Gmail

Gmail does not have retention services, by default. That requires the purchase of their Vault service.