Large number of messages in the Post Office\wpscsin\ofs\5 directory and they take 6 to 8 hours or longer to process

  • 7024972
  • 23-Dec-2020
  • 23-Dec-2020

Environment

GroupWise 18

Situation

When a Windows 2016 Server Post Office gets 200 or more messages in the Post Office\wpcsout\ofs\5\ directory, then it can take up to 6 to 8 hours or longer to process and to have status' on sent messages show up, like Transferred, Delivered, etc.  Initially, when the problem is occurring, the sent status is "Undelivered".  The recipients do in fact receive the message, so it is a status update problem.

Resolution

It was discovered that an automated process was sending a-lot of e-mail with a specific subject line to a specific GroupWise user who had GroupWise windows client rule that included an INVITE action ( same as delegate with appointments ) for mail messages.  

This user, we will call USER1, with this rule, would send the same message from the automated process to USER2 and USER3, that were on another Post Office from USER1, thus generating many status messages, coupled with the fact that it is normal for status messages to process with the lowest priority on the Post Office Agent.

Cause

As discussed in the Resolution and Additional Information sections, the use of an automated process
that was sending a-lot of messages to a user that was configured to request full status tracking and that had a
GroupWise rule that included the INVITE action that sent messages to 2 other users in other Post Offices,
coupled with the working as designed functionality that status messages have the lowest priority to process
on the Post Office Agent.

This problem likely would also occur with a Post Office on the linux platform also and could occur also in the GroupWise 14 version.

Additional Information

The use of proprietary GroupWise tools allowed us to decrypt a sampling of many of the status messages to determine that:

1.  An automated process was involved in sending alot of messages to USER1 with a specific subject line.
2.  That a "delegate" operation was involved, which is normally used only for appointments, which is essentially the same
as an INVITE action in a GroupWise rule.  The use of "Delegate" produces many status messages between the involved users on different Post Offices, about 30% more staus message traffic get generated with the use of "delegate" rather than replacing the rule action to "forward".

Another mitigating factor can be to increase the number of Message Worker Threads for the Post Office agent.
This can be set in the Post Office Agent Web console in a browser, Configuration, Performance Settings, Message Worker Threads.

If this setting is currently at 6 or 8, then double it.  Do not change the "Worker Yields to C/S Level" setting.
Then click submit.  When this issue is over be sure to set it back to it's original value and click submit.

Another change that can help to mitigate the issue, is to have the users involved in this message status storm, on the same Post Office so the status' do not traverse Post Offices, creating more traffic.

It was also discovered that the USER1 user, in the GroupWise Windows Client, Tools, Options, Send, Mail tab, the setting for "Create a sent item to track information" was set to "All information", which will additionally increase the number of status messages generated to also include deletions and purges from the trash, thus contributing to this status storm situation.

In this case, since USER1 represents an automated process that exists outside of GroupWise, it was recommended to change this setting to either "Delivered" or "Delivered and opened" only, thus reducing the number of status messages generated.

However the foundation of the problem in this case was not the build up of hundreds or thousands of status messages in the Post Office, but rather use of an automated process that sends a-lot of messages with the same subject line to USER1 that happens to have a GroupWise rule that uses the INVITE action that sends messages to 2 other users on other Post Offices. 

Once the Message Worker threads were "temporarily" increased and the USER1 GroupWise rule changed from using an INVITE action to a FORWARD action.  And the status tracking set just to "Delivered" then the issue subsided and went away.