2.4 How Retention Services and Item Store Flags Work

Retain keeps an "item store flag" to ensure that no item gets left behind.

With Exchange and O365 Holds and the Recoverable Items folder can be used for retention compliance. With On-Premise Exchange a journaling mailbox can be used but it is not recommended.

Gmail, by default, does not have a retention service.

GroupWise, on the other hand, has its own built-in feature called "Retention Services" that prevents items from being emptied from the mailbox until they have been successfully archived.

This article will first discuss Retain's support of the GroupWise Retention Services feature followed by a discussion of how Retain ensures that all items get archived in all other email systems.

2.4.1 GroupWise Retention Services

GroupWise has a feature that can be enabled in its GroupWise Administration option called Retention Services.

When enabled, GroupWise will prevent a user from emptying an item from Trash that has not yet been confirmed to have been archived. The way it does this is through a date/time field in each user database called the "digest retention time". It relies on third party archiving solutions like Retain to set that date/time, but GroupWise is the one that enforces it when set. What this does is it prevents any item newer than the date/time set in the "digest retention time" field from being emptied from Trash. This "digest retention time" is known in Retain as the "retention flag".

When Retain runs an archive job on a mailbox, it sets the digest retention time to the date/time of the newest/latest message it archived. However, if an error occurs on any item during that job which prevents Retain from archiving it or its attachment, Retain will set the digest retention time in the GroupWise user database for that mailbox to the date/time of the item that could not be archived due to an error.

And, even though Retain encounters an error on an item and cannot archive it, it moves beyond that item and continues to archive all other mailbox items; however, again, it will not advance the retention flag past the date/time of the FIRST error it encountered. Thus, when the next archive job gets run on that mailbox, Retain checks the item store time set in its database of the user and uses that date/time as its starting time for the new job, minus one hour.

Example: If today is September 17, 2014 but an item in the previous job produced an error, could not be archived because of that error, and had a delivered date/time of September 15, 2014 09:15, then when today's job runs, it will ask GroupWise for all items it has beginning with September 15, 2014 08:15 and on.

Now let's say that a month has passed and the problematic mail message has not been properly dealt with and we run a job. Even though Retain may have archived all items in the user's mailbox up to - let's say October 15th - it will still start the query with the item store time of September 15, 2014 08:15 because it could not advance the retention flag. If it were to do so, then they problem message would never get archived because Retain starts the query for items beginning with the digest retention time. Thus, if Retain were to advance the flag to the date/time of the newest/latest item it archived, then the problematic message would fail to fit within the query range and GroupWise would never send it to Retain.

So, when users complain they cannot empty trash, this typically has a few possible causes:

  • They have not exited the GroupWise client from the day before (or since a Retain job ran). The client checks the digest retention time upon loading but never checks it after that. Solution? Have the user close GroupWise and make sure that Notify also closes; otherwise, Notify keeps the current session open and relaunching the client is the same as not having exited the client in the first place; thus, it does not check the digest retention time because it is entering an already existing session. When the client launches and starts a new session, it checks the digest retention time and stores that for the session.

  • An archive job has not been run for a few days. Login to the RetainServer web UI and check the job status to see if it has run lately. If not, determine the cause (job disabled? license expired? Worker lost connection with Server? Out of disk space on Retain server? etc) and run the job.

  • The user that cannot empty trash had errors on an item or items in his/her mailbox during the archive job. Check Mailbox Error Monitoring in the Retain Server console.

  • Corruption/issues in the GroupWise user database for that mailbox (least likely of all scenarios). If this is the case, run GWCheck and rebuild the user database or contact Novell GroupWise support for help.

2.4.2 Exchange and Office 365

These email systems do not have a built-in retention service similar to GroupWise, there is no "digest retention time" field in any of their mail system databases that Retain can use; thus, Retain uses its own field in the "retain" database to keep track of its job starting point. This "item store flag" works just like the "retention flag" with GroupWise jobs. That date/time gets set to the date/time of the newest/latest item archived for a given mailbox; or, if an error(s) occurred during a job, the item store flag gets set to the date/time of the first item that had an error. That way, when the next archive job runs, it starts with the date/time of the item store flag, ensuring that Retain will see that item until it is properly archived. However, it is important to note that not advancing the item store flag does not prevent the user from emptying the item from their Trash in these email systems because they do not have a retention feature similar to GroupWise.

To prevent items from being deleted from Exchange/O365 a hold must be placed on the mailboxes. This can be an In-Place or Litigation hold. When a user deletes a message from Outlook the message is moved to the Trash, the user can then empty the trash. Exchange/O365 will then move the message to a Recoverable Items folder for 14 days before removing it from disk. However, a user can right-click on the trash and attempt to recover a deleted item, and at this point can purge an item immediately to remove it completely. This may be against your data retention policy, so to prevent the deletion a hold will then move the item to the hidden Purged folder, where the user cannot remove it but Retain can still archive it.

Alternatively, a journaling mailbox may be used on On-Premise Exchange. When a journaling mailbox is set up in Exchange, it can be configured in a way that redirects a copy of each message that is either sent or received throughout the entire mail system into they journaling mailbox. Retain can be configured to include the journaling mailbox in its archive job. Thus, even if a user empties an item from Trash, a copy of that item already exists in the journaling mailbox and remains in that mailbox until it is archived by Retain. If configured properly, Retain will remove that item from the journaling mailbox upon successfully archiving it. Items emptied from a user's Exchange mailbox but archived from the journaling mailbox do not appear in the user's Retain mailbox; however, they are searchable using the Retain search feature.

Because of the fact that duplicates of all email messages system wide get placed in the journaling mailbox, it can fill up fast. For this reason, we recommend that you not use the journaling mailbox feature and go with the Recoverable Items feature instead. If the journaling mailbox gets too big, Exchange will no longer be able to serve the mailbox. Thus, when Retain tries to run an archive job against it, it will fail because Exchange never responds back. This is why it is no longer recommended.

2.4.3 Gmail

Gmail does not have retention services, by default. That requires the purchase of their Vault service.