Diagnosing Memory Heap Corruption in glibc with MALLOC_CHECK_

This document (3113982) is provided subject to the disclaimer at the end of this document.

Environment

Novell eDirectory 8.7.3 for Linux
Novell eDirectory 8.8 for Linux
Novell Open Enterprise Server (Linux based)
Novell SUSE Linux Enterprise Server 9
Novell SUSE Linux Enterprise Server 10

Situation

The GNU C Library (glibc) is the standard C library on Linux Systems. Other C libraries exist and are sometimes used for special purposes (such as very small subsets of the standard C libraries used for embedded systems and bootstrapping), but glibc is the standard C library on all Linux distributions.
The glibc includes three simple memory-checking tools. The two, mcheck() and MALLOC_CHECK_, enforce heap data structure consistency checking, and the third,mtrace(), traces memory allocation and deallocation for later processing.
When memory is allocated from the heap, the memory management functions need someplace to store information about the allocations. That place is the heap itself; this means that the heap is composed of alternating areas of memory that are used by the program and by the memory management functions themselves. This means that buffer overflows or underruns can actually damage the data structures that the memory management functions use to keep track of what memory has been allocated. When this happens, all bets are off, except that is pretty good bet that the memory management functions will eventually cause the program to crash.
If we set the MALLOC_CHECK_ environment variable, a different and somewhat slower set of memory management functions is chosen that is more tolerant of errors and can check for calling free() more than once on the same pointer for single-byte buffer overflows.

Resolution

Setting MALLOC_CHECK_ :

  1. If MALLOC_CHECK_ is set to 0 (zero), the memory management functions are simply more tolerant of error but do not give warnings.
    • Maybe be useful if we are prevented from finding one memory bug by another that is not convenient to fix at the moment; it might allow us to use other tools to chase down the other memory bug.
    • It may also be useful if you are running code that works on another system but not on Linux and we want a quick workaround that may allow the code to function temporarily, before you have the chance to resolve the error.
  3. If MALLOC_CHECK_ is set to 1 (one), the memory management functions print out warning messages on standard error when they notice problems.
    • It is useful if we are not aware of any problems and just want to be notified if any problem exist.
  5. If MALLOC_CHECK_ is set to 2 (two), the memory management functions call abort() when they notice problems.
    • This is most useful from inside the debugger or a shell starting an application or daemon, because it allows you to get a backtrace as soon as the memory management functions discover the error, which will get us closest to the point at which the error has happened.
    • It is useful if we get a core caused by a memory corruption, we would have more information about memory allocation therefore, making things better for troubleshooting where we need to find out which application overwrote a memory address.

Additional Information

Setting MALLOC_CHECK_ examples:
  1. eDirectory NDSD script (/etc/init.d/ndsd):
    ...
    initdir=$NDSHOME/etc/init.d

    trap 'rm -f /tmp/nds.stat.$$' 0 1 15

    PATH=$PATH:/usr/local/bin
    LD_LIBRARY_PATH=/usr/lib/nds-modules:$default_prefix/lib: $default_prefix/lib/nds-modules:/ $NDSHOME/opt/novell/lib:/opt/novell/xad/lib/nds-modules:/opt/novell/xad/lib: $default_prefix/lib/nds-modules/jre/lib/i386:$LD_LIBRARY_PATH

    export LD_LIBRARY_PATH

    export MALLOC_CHECK_=2

    #
    # Start the ndsd daemon
    #
    ...


  2. NAMCD script (/etc/init.d/namcd):
    ...
    # Template configuration variables

    prefix=/usr
    exec_prefix=/usr
    sbindir=/usr/sbin
    bindir=/usr/bin
    initdir=/etc/init.d

    export MALLOC_CHECK_=2

    userID=`id`
    ...
This setting should be used for debugging only so, as soon as we have the core file we need with MALLOC_CHECK_ set, comment this setting out avoiding slowness.

Document

Document ID:3113982
Creation Date:03-11-2008
Modified Date:02-13-2009
Novell Product:eDirectory
Novell Product:Open Enterprise Server
Novell Product:SUSE Linux Enterprise Server

Disclaimer

The Origin of this information may be internal or external to Novell. Novell makes all reasonable efforts to verify this information. However, the information provided in this document is for your information only. Novell makes no explicit or implied claims to the validity of this information.
Any trademarks referenced in this document are the property of their respective owners. Consult your product manuals for complete trademark information.