Cool Solutions

GroupWise: Tune your GroupWise Linux Server!

Dean Lythgoe

By:

November 4, 2011 2:56 pm

Reads: 8943

Comments:11

Score:5

Our awesome consulting team has put together some great information on tuning a SLES server for GroupWise 8. A lot of this can also be applied to GroupWise 2012. Check out the details…

Novell consulting is always creating excellent tools, documentation and utilities to improve the their own engagements and on occasion we can convince them to share their insight with the community and with our GroupWise administrators.

This information can be used to get the most out of your SLES environment. Feel free to augment with your own experience and share back with the community how this advice and information has impacted your environment and configuration. Thanks Ed Hanley for the great details and real-world application.

How to tune a SLES 10+ server for GroupWise v8+

TCP/IP Stack
Disable IP V6 (if the GroupWise GWINTER will be running on the Linux server. See

http://www.novell.com/documentation/gw8/gw8_install/?

page=/documentation/gw8/gw8_install/data/b3kipez.html which has a section “2.3.1 IPV6 Support”

Add the following to /etc/sysconfig/network/ifcfg-lo (see TID 7007575)
MTU=’1500′
All GW services should use Bind Exclusive in Cluster enviornments
All GW services should use factory default IP Port numbers in Cluster environments

File System
In nsscon
nss /NoATime=GWVOL
nss /NoSalvage=GWVOL
nss /ReadAheadBlks=GWVOL:2 ← make sure this stays at 2 or lower (default is 2)
Add the following to /etc/opt/novell/nss/nssstart.cfg
/IDCacheSize=131072 (see TID 7006996)
/UnplugAlways
If using non-NSS file systems, the NoATime attribute applies there also
If using ext3 file system, the HTree feature must be disabled to run GW on it

NCP Protocol

Add the following to /etc/opt/novell/ncpserv.conf
CROSS_PROTOCOL_LOCKS 1 (see TID 7004594)
COMMIT_FILE 1
OPLOCK_SUPPORT_LEVEL 0
MAXIMUM_CACHED_SUBDIRECTORIES_PER_VOLUME 300000
MAXIMUM_CACHED_FILES_PER_VOLUME 120000
MAXIMUM_CACHED_FILES_PER_SUBDIRECTORY 6000

Core OS
Install libstdc++33-32bit (x86_64) to run 32-bit binary GW code on 64-bit Lnux server
NAMD – (from CoolSolutions article 12597 author peter6960)

To get the current “namcd” configuration execute:
namconfig get
If the preferred-server value does not point to the local ip address, or an ip address of a server on the same physical subnet, this can be changed with the following sequence of commands:
namconfig set preferred-server=[ndsd's ip address]
namconfig -k
namconfig cache_refresh

This step can also solve a lot of other OES related issues, and increase performance.

GroupWise

Add the following to /etc/init.d/grpwise
ulimit -c unlimited
export MALLOC_CHECK_=0
GWPOA
MAX APPLICATION CONNECTIONS = 4096 (4/user)
MAX PHYSICAL CONNECTIONS = 2048 (2/user)
MESSEGE HANDLER THREADS = 12 (the sweet spot)
C/S HANDLER THREADS = 50 (auto allocates up to 99 under load, then auto de-allocates back down to 50)

Other Items
Ensure that all AV scanners are disabled for scanning the GW PO, Domain and Gateways file structures. If your AV will support it – you can scan the temp storage area for WebAccess attachments.

All servers should have Hyperthreading disabled – the second pipe has only 3 steps as opposed to the primary that has 7 – net result for a server is ~17% slow down (Windows folk noticed this 4 years later) To disable Hyperthreading – modify the BIOS of the server

If your server is dedicated to GW, Set your hard disk controller Read/Write ratio to 80/20, if possible, most controllers can only be set to 75/25. Most disk controllers are set at 50/50 as their default R/W ratio. GW is mostly doing index read in small chunks of data. The GW average number of reads is 4 times more than writes.

I hope you find this information useful and applicable. If you have further ‘tuning’ suggestions, please let us know and we will update our own documentation and best practices.

Dean

VN:D [1.9.22_1171]
Rating: 5.0/5 (1 vote cast)
GroupWise: Tune your GroupWise Linux Server!, 5.0 out of 5 based on 1 rating

Tags: , ,
Categories: GroupWise, Technical

Disclaimer: This content is not supported by Novell. It was contributed by a community member and is published "as is." It seems to have worked for at least one person, and might work for you. But please be sure to test it thoroughly before using it in a production environment.

11 Comments

  1. By:jmarton

    With SLES I think everyone agrees ext3 is the best choice, but for OES servers should customers be putting their post offices on ext3 or nss?

    Joe

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  2. By:Schleissheimer

    First of all: Great Article, thanks for that!

    Suggestion: It be very cool to add the commands for reading
    the values to be changed. So you could verify what the actual
    value is and if the changes took effect.

    Question:
    I am struggeling on entering this:
    Add the following to /etc/init.d/grpwise

    ulimit -c unlimited
    export MALLOC_CHECK_=0

    Where should I add those two lines?

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    • By:dlythgoe

      Comments from Ed Hanley…

      Suggestion: It be very cool to add the commands for reading
      the values to be changed. So you could verify what the actual
      value is and if the changes took effect.

      I will need Dean to allocate a Linux coding guru to work with me , to help formulate the commands we need to find the current values (reading the values).

      I am struggeling on entering this:
      Add the following to /etc/init.d/grpwise

      ulimit -c unlimited
      export MALLOC_CHECK_=0
      Where should I add those two lines?

      In the file, the ulimit -c unlimited is commented out. Just remove the # character in front of it to enable this option.
      Then add a new line of export MALLOC_CHECK_=0 right after the line above (the ulimit -c unlimited line).

      Dean

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
  3. By:FlyingGuy

    What about single server installations where everything is being run on a single OES2 server?

    Yesterday I turned up GW on an OES2 server that is also running F/P web, etc.

    P.S. The Novell web site seems to be getting kinda slow.

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  4. By:mdallair

    Hi,

    I am not sure but i think there is a typo error inside the groupwise section of your tricks

    The line in error
    MESSEGE HANDLER THREADS = 12 (the sweet spot)
    Should be
    MESSAGE HANDLER THREADS = 12 (the sweet spot)

    Thank you for sharing with us your tricks

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  5. By:ukdtom

    And to add….

    When creating a dedicated NSS volume, I always do this from iManager, and make sure, that the namespace is UNIX, and not the default LONG, since that creates an additional overhead.

    /Tommy

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
  6. By:MARVHUFFAKER

    You say: “If using ext3 file system, the HTree feature must be disabled to run GW on it”

    What is the reason for having HTREE disabled? From the descriptions below, it would seem that even a reasonable sized GroupWise system would start to choke with HTREE disabled since you’re easily going to have more than 5000 files per directory in the offiles folders. Based on this fact, EXT3 would only be a viable option if in fact you could enable HTREE. Otherwise Reiser would be the preferred choice for performance, though the future of REISER is now in question.

    Please clarify. What is the optimal configuration on a SLES only (No OES) system?

    ——————-
    From http://serverfault.com/questions/10698/what-are-the-differences-between-ext3-ext4-reiserfs

    EXT3

    Most popular Linux file system, limited scalability in size and number of files
    Journaled
    POSIX extended access control
    EXT3 file system is a journaled file system that has the greatest use in Linux today. It is the “Linux” File system. It is quite robust and quick, although it does not scale well to large volumes nor a great number of files. Recently a scalability feature was added called htrees, which significantly improved EXT3′s scalability. However it is still not as scalable as some of the other file systems listed even with htrees. It scales similar to NTFS with htrees. Without htrees, EXT3 does not handle more than about 5,000 files in a directory.

    ReiserFS

    Best performance and scalability when number of files is great and/or files are small
    Journaled
    POSIX extended access controls
    The Reiser File System is the default file system in SUSE Linux distributions. Reiser FS was designed to remove the scalability and performance limitations that exist in EXT2 and EXT3 file systems. It scales and performs extremely well on Linux, outscaling EXT3 with htrees. In addition, Reiser was designed to very efficiently use disk space. As a result, it is the best file system on Linux where there are a great number of small files in the file system. As collaboration (email) and many web serving applications have lots of small files, Reiser is best suited for these types of workloads.

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    • By:dlythgoe

      Marvin,

      Here are some additional resources and information. I also sent this same information to Bret who I think you have been working with.

      Ext3 and the HTree feature

      If the Ext3 HTree directory indexing feature ( dir_index ) is enabled, GroupWise will have issues. Dean Lythgoe, Novell’s Director of GroupWise Engineering, recommends that HTree be DISABLED for the Ext3 file system. To accomplish this…

      To check if HTree is enabled, do this…

      tune2fs -l /dev/sda2 | grep features

      If you see dir_index, this means that HTree IS enabled.

      To Disable HTree, do this…

      umount /dev/sda2
      tunefs -O dir_index /dev/sda2
      e2fsck -FD /dev/sda2
      mount /dev/sda2

      The Ext3 file system performance will be slower, but you will not have any corruption now due to HTree being disabled.

      Read this Novell Cool Solutions article that explains why Ext3 should NOT be used at all for GW …

      http://www.novell.com/communities/node/4415/groupwise-7-and-oes2sles10-file-systems

      Details of the issue are next … If the below reason is incorrect, please tell me.

      EXT3

      EXT3 is slow without the H-Tree and so is discounted.

      With H-Tree, EXT3 becomes a very strong performer. However, there is a price to pay for the increased performanace. GroupWise uses telldir(), seekdir(), and readdir(), in the calling of files, all of which return a cookie that is nominally the position in the file. It’s unfortunately based an assumption that was true when the interface was designed but is not currently true. The assumption is that directories will always be laid out linearly, so a file offset will be enough to identify the directory entry. Unfortunately, this offset is a signed int, where only positive values are valid. This translates to a cookie size of 31 bits in which to uniquely identify the position.

      The problem arises with ext3 hashed directories, because they use a 64-bit hash that consists of a 32-bit major and a 32-bit minor hash. Ext3 returns the major hash, which is additionally truncated by one bit. This not only eliminates the collision handling that the kernel handles internally, but also creates even more collisions for telldir(). Discounted.

      Information provided by Ed Hanley

      VN:F [1.9.22_1171]
      Rating: 0.0/5 (0 votes cast)
      • By:bbecken

        From the GW2012 installation guide, page 25
        From the GW8 installation guide, page 22

        Linux File System Support.
        For best GroupWise performance on Linux, the ext3 file system is recommended. If you are running
        OES Linux and need the feature-rich environment of the NSS file system, GroupWise is also supported there. The reiser3 file system is also supported

        https://www.novell.com/documentation/groupwise2012/pdfdoc/gw2012_guide_install/gw2012_guide_install.pdf

        https://www.novell.com/documentation/gw8/pdfdoc/gw8_install/gw8_install.pdf

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)
      • By:MARVHUFFAKER

        BBECKEN — Why did you reply with a subject line of “Why Ext3 should NOT be used at all for GW” but then all the information you provided was from docs that say EXT3 file system IS IN FACT recommended.

        The intent of this original question I asked was not to question the validity of the EXT3 file system, it was usage of the htree feature/component of ext3.. There is nothing in the documentation that discusses this.. There are cool solutions that say turn HTREE off. Dean referenced one of them in his response.

        Furthermore, the underlying reason for probing into this issue is because I’ve noticed with multiple GroupWise systems on otherwise high performance hardware — having various “slowness” issues that I am trying to optimize and tune. I’ve noticed that I get the most performance related complaints on larger Post Offices (400-800GB).. And the common thread here is that with a system that size, there are generally millions of files throughout the /offiles/ sub-directory structure that must be managed by the file system.

        From a purely linux standpoint (non-GroupWise application), ext3 is known to be horrible at performance when directories have more than 5000 files. Google around and you will see (Also referenced in my original post). Specific to GroupWise, a larger GroupWise system could have 25,000+ files in a single offiles sub-directory. The logical conclusion would be that ext3 is a poor choice for file system use with larger GroupWise systems. That said, the “HTREE” feature of EXT3 is reported to improve performance of EXT3 — however Nobody at Novell will come forward and say YES or NO. You can google plenty of things that say Do Not use HTREE. But through various discussions and dialogs with Novell, I question whether this is based on actual technical knowledge and testing or just a lingering superstition from when GroupWise was originally ported over to Linux a few years ago. I do know that with the GW7 docs, it specifically said NO HTREE, but I do not believe any GW2012 docs or cool solutions or tids say it any more. I could be wrong.

        Furthermore, contrary to what many people believe, ReiserFS was developed AFTER ext3 to overcome some of EXT3′s limitations. As someone new to linux when Novell bought Suse, I always had the impression that ReiserFS was an old/legacy/obsolete file system as well. When in fact it performs quite well with the GroupWise architecture (Millions of very small files).. I realize that ReiserFS’s future is now in question and will likely go away soon, so it’s probably not a good option for a production system and not worth a lengthy discussion.

        So I’m really looking for someone with good firsthand knowledge of the reasons for using or not using HTREE.

        Marvin

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)
      • By:MARVHUFFAKER

        Dean, I wish I could say this helped, but unfortunately I cannot.

        Generally I find that HTREE is already disabled. I would expect that Enabling HTREE would improve performance based on what I understand about it from the Linux documentation. However, based on numerous tids and forum posts and cool solutions, everybody says to disable HTREE.

        So the Linux Architecture and GroupWise Recommendation completely contradict each other and I’m at a loss of what to do.

        Has the Developer team done any testing for performance on EXT3 and HTREE, and what technical reasons exist for Disabling HTREE despite the significant performance degradation? Is it due to the API’s that GroupWise uses? When you go with the 64bit only GroupWise, will it address these limitations?

        Thank you.

        Marvin

        VN:F [1.9.22_1171]
        Rating: 0.0/5 (0 votes cast)

Comment

RSS