Cool Solutions

GroupWise Mobility: Trouble shooting from the BEST!

Dean Lythgoe

By:

September 17, 2010 11:01 am

Reads: 16009

Comments:18

Score:0

Hello GroupWise friends… My name is Tony Anderson and I’ve been leading the QA effort for Novell Data Synchronizer Mobility Pack for the last year and a half or so. This has been a very challenging and rewarding project to work on. I hope the following insight will be helpful to you as you begin to roll this product into production.

During our own internal roll-out of Mobility Pack, which began late last year, a couple of the most common complaints we received right away were “I can’t connect” and “I don’t have all my items”. These types of questions continue, occasionally, for a few users and for a variety of reasons. The goal of this blog post is to help identify what types of things to look for should these complaints arise, and how to address some of them.

“I can’t connect”

Has the user entered the correct settings on the device?

This would include the correct IP/DNS address of the mobility server, SSL enablement (is it on or off on the device and does it match what the mobility connector expects), and the correct userid and password. It is important to note that the mobility server uses LDAP authentication, not GroupWise authentication, to authenticate the users.

As well, while some devices require something in the “domain” field, we don’t do anything with that setting. On Apple devices, it is recommended that, for now, it remain blank.

Also, users sometimes erroneously put their email address in the username field (username@novell.com as opposed to just username). This will also cause a connection failure. Username is all that is required for that field.

Is the LDAP server down?

One way to tell if the mobility server is having issues with the LDAP server is to query the mobility application interface log(s).

From the mobility server:

  1. cd /var/log/datasync/connectors
  2. grep “Can’t contact LDAP server” default.pipeline1.mobility-AppInterface.log | head

The output will look something like this:

2010-09-16 06:51:20.362 ERROR_VERBOSE [CP WSGIServer Thread-24] [DeviceInterface:838] [userID:] [eventID:] [objectID:] [Server] LDAP Authentication Exception: {‘desc’: “Can’t contact LDAP server”}

The first instance of this in the log file is a good indication of when the connection with the LDAP server began having issues. As all events that pass through the system must authenticate, you can pipe the output of the grep to wc -l (grep “Can’t contact LDAP server” default.pipeline1.mobility-AppInterface.log | wc -l) and get a pretty good idea of how much this is occurring on the system and then react accordingly by dealing with the LDAP server, the network connection between the LDAP server and Mobility server, or the Mobility server itself (restarting it if necessary).

Has the user’s password recently changed?

Security policies sometimes require users to change their password on a regular basis. One issue that is currently being looked at is when a user’s LDAP password changes, and they are using a device that has “push” enabled, the device may continue to attempt a repeated connection on its own using the previous password on the device, thus possibly locking out the user. If the user updates their password on the device, they still may not be able to connect until the lockout period on the LDAP server either expires or is reset by the admin.

In another case, users may be authenticating to one LDAP server, while their mobility server is pointing to another. If passwords between these servers are not kept in sync, users may see connection errors.

From the mobility server, the following will help to identify which users may be affected by a password change:

  1. cd /var/log/datasync/connectors
  2. grep -i <username> default.pipeline1.mobility-AppInterface.log | egrep ‘ldap_authenticate|LDAP authentication problem’

If a password change is the suspected culprit, but you’re unsure if LDAP is the real problem, have the user do the following from a browser:

  1. Go to https://<ip address of mobility server>:8120
  2. Have them log in with their LDAP credentials

If the user can successfully log in, then the problem may lie with the device, in which case it may be necessary to have the user delete and re-add their account on the device itself.

Is the user connecting in via a wi-fi or cellular connection (or both)?

Some devices will auto-switch on the fly to a wi-fi hot spot for data service, if one is available. There may be times when the Mail client will attempt a connection during this auto-switch between wi-fi and cellular and cause a connection error. Have the user force the device to use one or the other to determine if the data connection is the cause of the problem.

Are all Idle Threads busy?

By default, the number of Idle Threads that can handle incoming device connections is 15. Each device will use 2 – one for ping, and one for sync. These threads will fluctuate throughout the day on a busy system, and though it seems like a small number, they can usually handle between 100-150 active devices or more, depending on the amount of data coming through for each user. This number is configurable, but caution should be taken to not overtax the system by allowing too many connections in at once.

To modify this setting:

  1. Log in to the Web Admin UI (https://<address>:8120) as an administrator
  2. Edit the Mobility Connector
  3. Click “Edit XML Source”
  4. Add <threads>number-of-your-choosing</threads> between the <custom></custom> tag
  5. Click “Save Custom Settings”
  6. Restart the Mobility connector

Once the connector has restarted, you can verify the Idle Threads by tailing the mobility application interface log:

  1. cd /var/log/datasync/connector
  2. tail -f default.pipeline1.mobility-AppInterface.log | grep “Idle Threads”

Output will look something like this:

2010-09-15 16:38:51.319 DEBUG_VERBOSE [CP WSGIServer Thread-22] [__init__:1285] [userID:] [eventID:] [objectID:] [WSRThread] Socket 16: Total: 15 ,Idle Threads: 15

Note: “Total: 15 ,Idle Threads: 15″. “Total” is either the default (15) or whatever was set in the XML. “Idle Threads” is the number of threads that are NOT busy. So, if the number of Idle Threads is 1, for example, all threads are busy. If while tailing the log this remains at 1, then users may experience connection issues and it may be necessary to raise the number above 15.

Resync vs. Reinitialize vs. Delete/Re-add

If none of the above tips help to resolve a connection problem, it’s possible that there may be a problem with the user data coming from GroupWise. Whatever the case, the next step may be to reset the user. This is done in one of 3 ways:

Resync

A Resync may be necessary when data in the mobility database gets out of sync with the data on the device. In this case, the data is accurate in the db, but for whatever reason, the device isn’t picking it up. Though it’s a bit rare for this to cause a connection error, it can appear to end-users that the connection is dead since they don’t appear to be in sync with what is in GroupWise.

To Resync a user:

Log in to the Web Admin UI (https://<address>:8120) as an administrator.
Click on the mobility connector Monitor icon (right-most icon)
Find the affected user and click the Resync button (first button in the Device column)

This action will clear out the data on the device and force the device to come and get whatever data is waiting to be picked up in the mobility db for that user. This includes all email, folders, contacts, and calendar items.

If the newly synced data is still not accurate, or users are still having connection issues, it may be necessary to Reinitialize the user.

Reinitialize

Much like Resync, Reinitialize will force a resync of data to the affected device. However, in this case, the data is pulled from the Sync Engine (datasync) database and then synced to the mobility database for the device to pick up. Of course, in order for this to work correctly, the data in the datasync db must be accurate.

To Reinitialize a user:

Log in to the Web Admin UI (https://<address>:8120) as an administrator
Click on the mobility connector Monitor icon (right-most icon)
Find the affected user and click the Reinitialize button (under the Actions column)

Once again, if the newly synced data is still not accurate, and/or users are still complaining about connection errors, it may be necessary to Delete and Re-add the user.

Delete/Re-add

Every so often we see user accounts that have problems with the Initial Sync process (the sync process from GroupWise to the Mobility db only…. not to the device). Even though it appears that the initial sync was a success for a particular user (their status in Monitor is green), it may not always be true. Users in this situation will typically complain about being unable to connect to the server, or not having all their items or folders, etc. Currently, the best way to resolve this problem is to Delete the user from the mobility system and Re-add them, thus forcing an initial sync from top to bottom, replenishing the datasync and mobility databases with fresh data from GroupWise.

To Delete/Re-add a user:

If the user was added via an LDAP group, and that group is being used for both the groupwise and mobility connectors, delete the user from the group and wait until the user is completely deleted from both connectors. Note: on larger systems, it may take some time before the delete is complete.

To double check via the db, do the following:

  1. psql -U datasync_user mobility
  2. Enter your db password
  3. select userid from users order by userid;
  4. Verify that the affected user is NOT in the list
  5. \c datasync
  6. select dn,disabled,”connectorID” from targets order by dn;
  7. Verify that the user is listed twice, once for each connector (groupwise and mobility) and that the disabled flag is set to 1 for both entries
  8. If the above is true, the user is deleted and can now be Re-added back into the LDAP group
  9. Repeat steps 3-7, but reversing steps 4 and 7. The disabled flag should be set to 0 for both connectors when the user has been successfully added

If the user was added individually through the Web Admin UI, do the following:

  1. Log in to the Web Admin UI as administrator and click on the mobility connector
  2. Click on the Users tab
  3. Search for the affected user and click the Delete button (red X)
  4. Click Home and then on the groupwise connector
  5. Click on the Users tab
  6. Search for the affected user and click on the Delete button (red X)
  7. Verify the user has been completely deleted by checking the db using the steps above
  8. Re-add the user to the groupwise connector first and the mobility connector second

Once a user has been re-added to the system, and the connectors are running (they don’t need to be stopped in order to delete/re-add a user), the Initial Sync for the user will begin. Currently, it’s best to allow the initial sync for the user to complete before allowing them to connect in again. Having said that, their device may already be set up to connect into the system and may auto-connect during the initial sync. While this isn’t necessarily bad, it’s recommended that the user delete their account from the device and re-add it once the initial sync to complete.

To verify that the initial sync is complete:

  1. Log in to the Web Admin UI as administrator and click on the mobility Monitor icon (right-most icon) on the mobility connector line
  2. Search for the user in the list and verify that Sync State is set to “Synced” (it will be green in color)
    Note: If Sync State is green, it simply means that the mobility connector made the request for data and the Sync Engine responded that it received the request. Depending on the amount data being sent back, it may take a bit more time for the data to actually get written to the mobility db. Given that, from the Monitor page, click on the user to view the number of events flowing into the db for that user. If, after a refresh of this page, the number of events is no longer climbing, initial sync is most likely complete.
  3. At this point, have the user re-add their account to their device and attempt to connect

Other things to consider

Typically, when users complain of connection issues or missing items, a combination of 1 or more of the above tips (or clues) will help solve the problem. Obviously, there are other things to look for. Consider the following items while troubleshooting the Mobility server. Maybe we can touch on some of these next time…

  • Certificate issues – not all devices are created equal. Trusted root certificates may or may not be installed on certain devices. Either go with the popular bunch (Verisign and the like) or be prepared to export/import the cert onto the affected devices.
  • Performance and Scale – tailing the log files and looking for queue patterns is helpful when trying to understand scale and performance issues. Use top, or other Linux tools, to monitor the system.
  • Network outages – we’ve seen wi-fi outages cause a barrage of help-desk calls complaining that the mobility server was down, when it was the downed wi-fi connection that caused the connection errors.
  • POA outages – if users complain about items not syncing, check the POA. Devices only connect to the mobility connector and have no idea that their POA may be down or that there may be a connection problem between the POA and the GroupWise connector. Should sync completely stop for all users on the system, check SOAP threads on the POA(s). A POA with maxed out SOAP threads will need to be restarted, at which point sync will begin to flow again.

Excellent Resources

In Summary…

There is a lot to this great product. Hopefully these troubleshooting tips will help you whether you’re running in production, installing for the first time, or updating to the latest and greatest. We’re looking forward to frequent updates via our distribution channels and, of course, improved quality and new features. We simply would not be where we are today without your input (and help throughout the “Tech Preview” and private/public beta process). The feedback has been amazing. Thank you!

Tony

VN:F [1.9.22_1171]
Rating: 0.0/5 (0 votes cast)

Categories: Uncategorized

Disclaimer: This content is not supported by Novell. It was contributed by a community member and is published "as is." It seems to have worked for at least one person, and might work for you. But please be sure to test it thoroughly before using it in a production environment.

18 Comments

  1. By:ianfret

    Since we have been using this new mobility pack, we have been having problems with GroupWise. In general this is highlighted in the POA crashing. Having read the issues with Mobility server and version 8.0.2 we have nowhere to go without an update. This is so disappointing as we have waited so long for this tremendous business tool for our smartphone users.

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    • By:dlythgoe

      Your scenario does not match the experience of other users. It sounds like you need to contact NTS immediately and open an incident. There are probably some things they can help you figure out.

      Let us know…

      Dean

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
    • By:johnstonrd

      We are having similar issues as well. Narrowing it down to about the time we loaded the full release of Data Sync, our users have experienced issues with their client, client dropping off or disconnecting from the POA server, Client connection running slow even though on brand new Dell Hardware purchased back in July. Groupwise I am losing my faith as I try to keep you in my district.

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      • By:dlythgoe

        Please contact NTS immediately and provide them the requested information about your system, platforms, configuration and application versions.

        We need to understand what is going on so that if there is a problem, we can get it fixed ASAP.

        You can email me directly with your SR# – that would be great!

        dlythgoe@novell.com

        Dean

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)
  2. By:skapanen

    We had SOAP / POA crashing issues with GW801, but that got fixed with 802 update.

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  3. By:andersons81

    We waited to get the datasync until the full release came out. We were testing it with about 6 users. Everything was happy until about last Friday. My phone stopped syncing among others. I tried the suggested troubleshooting above. We even updated to Update 1 of the mobility pack. I had completely removed myself from the server and the settings from my phone. I am now not able to connect at all. On another phone, we set it to factory defaults with no luck in connceting. After entering in the credentials on the phone, it says unable to connect to excahnge server via IP address or server name. There have been no changes to our network to prompt the lack of communication between the server and our devices. Any suggestions would be appreciated.

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    • By:dlythgoe

      It is difficult to know from your comment exactly what may be happening. It sounds like a certificate issue or LDAP server connection problem.

      However, my best advice is for you to immediately contact support and open an incident. They will be able to help you track down exactly what is going on and get you back up and running.

      Please let us know how it goes.

      Dean

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      • By:andersons81

        It turned out that the IP address of the server was not bound to anything on the server. We had set this box up from scratch and the data sync is all that we have running on it. We had gone through all the configurations during initial setup and everything looked good on the connector side. Now, that the IP address is bound, we are working again. Hopefully, this helps anyone else who may have this same issue.

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)
  4. By:twa-asadi

    Hello, we have tested with the first data sync server on the address xx.xx.xx.10 and later we installed a new sync server on the address xx.xx.xx.11
    Data Sync works fine now with .11, but POA gives every second this error:
    GWEVENT notification 8926 http://xx.xx.xx.10:4500
    Can you tell me, how to remove this address from POA?
    Thanks

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    • By:dlythgoe

      If you are not going to use .10 anymore, then you need to clear out the event configurations for users that point to .10.

      You can do that from the POA web console (so long as you are using a later poa – 8.0.2).

      1. Go to the POA Web Console.
      2. Click on Configuration.
      3. Click on Event Configuration List.
      4. Click on the UserID of the affected user.
      5. Select Delete Events and Delete Event Configuration.
      6. Click Submit.

      In addition, If you did a proper uninstall of mobility pack on .10, that should have deleted the event configurations as well.

      Dean

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      • By:twa-asadi

        Hello,
        thank you for your quick answer.
        We use GW 8.0.2 POA in german and over web interface I can see only users with new address and they are not editable. The old users with .10 address are not visibable here.
        Any suggestion? How to edit the new users from POA web console?
        Thanks

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)
      • By:t1965a

        > I can see only users with new address

        Typically, the only reason the GWEvent Error occurs is if there is an event configuration for a user on an address that isn’t valid anymore (the user or address could be invalid).

        >The old users with .10 address are not visible here

        Yet the GWEvent Error is complaining about someone on the .10 address? If true, then you may need to log into the POA with the javaClient (found in the GroupWise Web Service (SOAP) SDK – ftp://sdk.provo.novell.com/ndk/gwsoap/builds/cross_platform/novell-gwsoap-devel-2008.12.23-1cross_platform.tar.gz) and see if the event configurations show up there. The POA should list all of the event configurations, even if they are disabled or not in use (and haven’t been deleted yet).

        > they are not editable

        If you click on the Event Configuration List link on the POA Configuration page in the POA Web Console, it should take you to an Event Configuration List page with a list of clickable UserID’s. Clicking a UserID should take you to an Event Configuration page with options to Delete Events and Delete Event Configurations. Are you saying the UserID’s are not clickable links?

        Let me know if you need more help.

        Tony
        tanderson@novell.com

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)
      • By:twa-asadi

        On my POA 8.0.2 Web Console (loged in with username/PW) I can not see and edit the evnet config. list. All other settings are editable!
        Best Regards

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)
      • By:t1965a

        > I can see only users with new address

        Typically, the only reason the GWEvent Error occurs is if there is an event configuration for a user on an address that isn’t valid anymore (the user or address could be invalid).

        >The old users with .10 address are not visible here

        Yet the GWEvent Error is complaining about someone on the .10 address? If true, then you may need to log into the POA with the javaClient (found in the GroupWise Web Service (SOAP) SDK – ftp://sdk.provo.novell.com/ndk/gwsoap/builds/cross_platform/novell-gwsoap-devel-2008.12.23-1cross_platform.tar.gz) and see if the event configurations show up there. The POA should list all of the event configurations, even if they are disabled or not in use (and haven’t been deleted yet).

        > they are not editable

        If you click on the Event Configuration List link on the POA Configuration page in the POA Web Console, it should take you to an Event Configuration List page with a list of clickable UserID’s. Clicking a UserID should take you to an Event Configuration page with options to Delete Events and Delete Event Configurations. Are you saying the UserID’s are not clickable links?

        Let me know if you need more help.

        Tony
        tanderson@novell.com

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)
      • By:t1965a

        > I can see only users with new address

        Typically, the only reason the GWEvent Error occurs is if there is an event configuration for a user on an address that isn’t valid anymore (the user or address could be invalid).

        >The old users with .10 address are not visible here

        Yet the GWEvent Error is complaining about someone on the .10 address? If true, then you may need to log into the POA with the javaClient (found in the GroupWise Web Service (SOAP) SDK – ftp://sdk.provo.novell.com/ndk/gwsoap/builds/cross_platform/novell-gwsoap-devel-2008.12.23-1cross_platform.tar.gz) and see if the event configurations show up there. The POA should list all of the event configurations, even if they are disabled or not in use (and haven’t been deleted yet).

        > they are not editable

        If you click on the Event Configuration List link on the POA Configuration page in the POA Web Console, it should take you to an Event Configuration List page with a list of clickable UserID’s. Clicking a UserID should take you to an Event Configuration page with options to Delete Events and Delete Event Configurations. Are you saying the UserID’s are not clickable links?

        Let me know if you need more help.

        Tony
        tanderson@novell.com

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)
  5. By:jflass

    Hello,

    Since we updated our datasync connectors (GroupWise v1.2.0.2567 and Mobility v1.2.0.2572), many users of our company tell us that the life battery of their iPhone is very short.
    At first, we thought that it was our 3G provider but, we have also users with personnal iPhone and other provider, they have the same problem.
    Can we change de timing of push or anything else ?

    Best regards,

    JFL.

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    • By:hme0

      I have the same issue

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      • By:jffgrnfld

        For iPhone battery life issues, note there is a fix reported in the “Patch File 1″ available as of Oct 19 in the Patch Finder downloads.

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)

Comment

RSS