5.7 Google Apps Module

The Google Apps module for G Suite allows Retain to archive Gmail data items. To configure Retain for Gmail archiving, Retain needs Gmail to be configured to allow Retain access, and the appropriate information entered into Retain.

Google Apps requires that a project be created, an OAuth key created and a Service Account specified and enabled before the Retain system can connect and archive mail.

To archive from Google Apps:

Create a project
Create a new Client ID key
Record client secret
Create a Service Account and Manage API Client Access
Generate a P12 key certificate

5.7.1 Prerequisite

Retain supports two-factor authentication with OpenId Connect for OAuth 2.0. To utilize OpenId the Retain Server needs to be accessible from the Internet. The URL through which the Retain Server is accessible from the Internet must be specified. For this, you must create a Project and Client ID.

NOTE:As a cloud service these screens may change at anytime.

To configure OpenID for Retain use:

Go to the Google API Console, and select 'Create a project'
In the sidebar under "API Manager", select 'Credentials' and then select 'OAuth consent screen tab
Choose an 'Email Address', specify a 'Product Name', and press 'Save'
In the 'Credentials' tab, select the 'Create credentials' drop-down list, and select OAuth client ID
Under 'Application type', select 'Web application' and Specify a name. The origin field should be the Retain Server's URL.

If using two-factor authentication for GMail login, enter an Authorized redirect URL (e.g. http://retain.gwava.com/RetainServer/Server/openIdConnect.jsp?).

When complete, select the 'Create' button.
The OAuth client dialog box should be displayed. This dialog box contains the Client ID and Client secret.
- Copy the Client ID and paste it into the Retain Client ID field in the Google Apps module configuration. Do not lose the client secret.
- OAuth access requires the secret and ID. This is the only time the secret will be displayed.
To attain the client secret for an existing project:
1. Select the Web Client and click the Edit OAuth Client button
2. Copy the Client ID and Client Secret which is required to configure OpenId tab for Google apps module.
  
  The OAuth Client needs to be enabled for domain-wide delegation to function. To enable domain-wide delegation, Retain requires a service account.
From the Products and Services hamburger menu at the top-left: Select IAM & Admin
Select the 'Service Accounts' and click the 'Create service account' button
Configure the Service Account Name and ID. No Role is needed.
- Select the option to Enable Google Apps Domain-wide Delegation.
- Enable "Furnish a new private key" and select the P12 file.
- Copy the Service Account name and ID. These are required by Retain.
- Click 'Create'.
The key will be automatically created and downloaded to the local machine. Do not lose the P12 file. This is the only copy and it is required to configure Retain.
You will need the Client ID from the Service Account
Go back to the API manager and enable: Calendar API, Gmail API and Admin SDK. Click on each one and click Enable on top.
The last step is to authorize the domain. Browse to the Google home page and select 'Admin' from the drop down menu at the top right.
Select 'Security' from the administration menu.
From the Security menu, click 'Show More'
Select the 'Advanced Settings' option.
On Advanced Settings, select 'Manage API client access'
In this window, enter the client ID created with the service account, and then input the entire domain the client will be accessing.

The Service Account name would be like Retain-Service as above and the API Scopes (listed in the module) would be:

https://mail.google.com/ , https://www.googleapis.com/auth/admin.directory.group ,https://www.googleapis.com/auth/admin.directory.user ,https://www.googleapis.com/auth/gmail.readonly
Once entered, click 'Authorize'.

5.7.2 Google Apps Module Setup

All configuration required is now complete in Google Apps, and the information and files keys are available for configuration of Retain.

5.7.3 Core Settings Tab

Once Google has been configured to allow access to Retain, the Google Apps Module may be configured.

The module needs to be enabled on this page to make it active in the Retain system.

The module can be given a name.

The Send Method option enables either the SMTP Forwarding or FTP features. For either feature to appear and function, the Module Forwarding tab must be configured on the Server Configuration page. See that section for more information.

Address book caching must be enabled to gather and maintain an updated list of users. Authentication is used to allow access to the Retain message store for users based on their existing Gmail account login. If the Enable Jobs option is not enabled, no jobs may be completed with the Google Apps module.

5.7.4 Settings Tab

In order to connect to the Google system, Retain requires the email address of the Admin user OAuth Service Account and a p12 Certificate for authentication. Retain archives the Gmail system through IMAP, and will login and download the message data to the Retain data store.

Test the connection to ensure that the configuration has been completed correctly.

Jobs and profiles for Gmail will not be visible until the address book has been cached. After the module has been cached, all configuration options for profiles, workers, schedules, jobs and data storage will be enabled and visible.

5.7.5 OpenID Tab

Configure the OpenId Connect tab in Retain by inputting the Client ID, the Client Secret, and the Public RetainServer URL and saving changes. NOTE: The public RetainServer URL should look something like http://<yourdomain.com>/RetainServer. Only specify to the ".../RetainServer" portion of the URL, the rest is automatically filled-in. Specifying the complete URL will result in a connection error.

If Retain has been configured with OpenId credentials, the login page will display an option to login with Google credentials. If a user is currently logged-in to their Gmail account, simply clicking the "Login with my Google account" button will automatically log them into Retain.

5.7.6 Google Apps Profile

To create a profile to archive email and data from Google Apps, select the add profiles button and name the profile then continue configuring as desired. The Google Apps profile must be configured and all settings saved before a job can be created and run.

5.7.7 Core Settings

The profile will not become active and will not allow jobs to be run unless the profile is enabled. Enable the profile by placing a check mark in the ‘Enable Archiving’ checkbox.

5.7.8 Message Settings

The Message Settings tab contains the source and status settings for the messages to be archived. Message types and sources which are checked will be archived in this profile.

Any item type or status which is not selected or specified will exclude qualifying items from being archived in Retain.

5.7.9 Scope tab

This is the most critical tab to fill out as it sets the limits on how much to archive.

Date Range to Scan

The Date Range determines which message items are collected, depending on the date of the message.

New Items: All items that have not been archived by Retain since the last time the job ran.

All Items in Mailbox: All items in the mailbox starting from 1/1/1970, duplicates will be processed but not stored if they already exist in the Retain archive.

Number of days before job start date and newer: Only items from the relative number of days from the time the job began will be archived. E.g. messages that came into the email system 7 days ago or less.

Number of days from job start date and older: Only items previous to the relative number of days from the time the job began will be archived. E.g. messages that came into the email system 7 days ago or more.

Specify custom date range: Only items between two absolute dates will be dredged.

Specify custom date range relative to job start: Only items between two relative dates will be dredged. E.g. messages that came into the email system between 7 and 5 days ago.

Advance Flags

Enabling "Don't Advance Timestamp" will not update the timestamp flag. Items that are dredged will still be considered new by Retain the next time the job runs.

This is useful when troubleshooting, but is generally not used for normal jobs.

5.7.10 Miscellaneous tab

The miscellaneous tab allows the configuration to allow or deny archiving and indexing of attachments in Retain. If attachments are to be archived, they may also be indexed to provide searching capability in the browse messages interface.

5.7.11 Advanced tab

The advanced tab allows the administrator to be even more specific in what to archive and what to exclude. The criteria added under the ‘advanced’ tab will limit associated jobs to only the items matching the criteria. Up to 6 lines of advanced criteria may be added to each profile. Each additional line will be logically AND-ed together. For example, the system will archive all items where the following is true: Criteria A AND Criteria B AND Criteria C AND etc.

You may select based on:

Subject
Sender
Recipient
Size
Attachment Name

Specify the criteria, whether it is: equal to or not equal to, whether they contain or do not contain the item.

This gives you great flexibility and granularity. It allows you to customize archival jobs and retention for many different groups, or even individuals.

5.7.12 Folder Scope

By default, we dredge items from all folders. You can specify one or more inclusions or exclusions.

Your choices are:

Dredge everything
Dredge only these listed folders
Dredge everything except these listed folders

How to specify the list of folders to dredge/exclude:

Select to only get items from the list, or all except items from the list, as desired
Select ‘Add’ to open a new selection
Specify a System Folder (mandatory). Example: Calendar
You may specify a subfolder of that folder (optional)

Example: entering “old” would mean the folder “old” under “Calendar”
You can have multiple hierarchies under that with the / delimiter

Example “old/mail” would mean the subfolder “mail” under “old” under “Calendar”
6.You specify if the option includes subfolder

Example: If you select “old” and “includes subfolder” is unchecked, “Calendar/mail” is selected. If “includes subfolder” is checked, “Calendar/old/mail” would also be selected

5.7.13 Google Apps Jobs

All data collection for Google Apps is configured through the Google Apps job interface. The jobs here combine the Profile, the Schedule, and a Worker together to archive a specified mailbox, distribution list, or domain. A Job must have a Profile, Schedule, Worker, and a target, mailbox, list, or domain, before it can be saved or run.

The Schedule, Profile, and Worker selected here will determine what is archived, when it is archived, and what worker does the actual archival work. The Job MUST be enabled to run. Only previously configured Schedules, Profiles, and Workers can be selected.

5.7.14 Core Settings tab

Core settings allows you to enable the job and set the Schedule, Profile and Worker needed to run the job.

The Data Expiration is an option to place a timestamp on the mail in the Retain database, which allows for ease of automation for the deletion manager. In addition, devices such as NetApp, Centera, and Hitachi HCAP may use this number to enforce hardware level protection of the stored item so that no one (including Retain) may delete the item before its expiration date.

Job Expiration is not retroactive for mail in the database, and only applies to mail archived by the job that it is active for. The base folders and criteria are specified under the custom expiration dates accessed through the ‘Add’ button at the bottom. All messages included in any specified folders will have a different date or be exempted from the standard expiration date. To have messages with custom job or folder expiration dates properly expire, the deletion management date scope must be set to delete messages with an Expiration Date older than 1 day.

5.7.15 Mailboxes tab

The mailboxes tab is where the administrator specifies which entities (mail server(s) and/or Distribution List(s)) are to be scanned.

Expand the Post Office and/or Distribution List trees, and check off the items you want to be dredged.

NOTE:If you desire to have a job backup a single user, or selected group of users, select the Users menu and assign the user(s) desired.

The users section allows you to select individual users to include, or exclude them from an archive job. For example: you can select an entire Mail Server to be archived, and then expand the users section to include or exclude users to the job.

This can also be used to select only certain users in the system for an archive job.

To add a user to the Include or Exclude list, select the respective ‘Add user’ button and search for the user. It can be helpful to unselect the ‘only show recently cached items’ option.

Add the selected users to the list in the search window, then select ‘Ok’ to add them to the include or exclude list.

5.7.16 Notification tab

When a job is run, the notification option allows the administrator to be emailed a summary of each running job if desired.

For notification to function correctly, the SMTP information for the desired SMTP server must be fully filled-out. How much information is required varied depending on the mail system used.

5.7.17 Status tab

The Status tab displays the status of any currently running jobs, as well as the stats of the last completed job.

This tab is informational only unless a job is currently running. If the selected job is running, an option to abort the job is displayed.

This window will refresh every ten seconds to keep you up to date as to the status of the selected job. The completed line is displayed during an active job. The completed status is a display of how many mailboxes have been completed, the job mailbox total, and gives an incrementing percentage for the amount completed. This amount is based entirely on the number of mailboxes, not the amount of mail. Because the last mailbox could be larger than the rest of the system, this percentage may not be accurate according to time.

You may now configure Schedules, Workers and Jobs.