AppNote: Directions in NetWare Programming
Novell Cool Solutions: AppNote
By Russell Bateman
Digg This -
Posted: 13 Jul 2004
Senior Software Engineer
Core OS Team
This AppNote discusses the features and functionality of NetWare's programming environment for writing applications that run as NetWare-loadable Modules (NLMs) on NetWare.
- NKS for NLM Programming
- Process Management
- POSIX signals
- Threading and Synchronization
- Library Technology
- File and Directory I/O
- Screens on NetWare
- NetWare-specific Interfaces
- Development Environments
|Topics||application development tools, NLM development, kernel programming, server applications|
|Products||Libraries for C, NetWare 5.1, NetWare 6, NetWare 6.5|
A few years ago, starting in September 1999 and continuing through the end of the year with a couple of updates over the next two years, we published a series of articles on Novell's replacement for CLib, the environment and standard C runtime library for Novell's operating system, NetWare. We explored the reasons CLib was abandoned (but carefully preserved and maintained for the myriad applications that used it then and still use it today). We further explored in surface detail the various interfaces provided by the Novell Kernel Services (NKS) environment, then launched into the deeper details of multi-threaded programming correctness, latency and thread cancellation issues, and synchronization, covered various components of this environment including memory management, platform-dependent and debugging features, file and directory I/O, and time-outs. Then we talked about how LibC sits atop NKS and made comments about programming at both levels. We also made a few remarks about differences in programming between LibC and CLib.
These articles are available at the following links:
Before beginning, and in case there are any newcomers to NetWare reading this article, let's note that LibC is not merely the C runtime library for NetWare. It is also the execution kernel or environment in which NLMs run, even those that do not link any of LibC's publish preludes. Any NLM can may consume LibC since all threads on NetWare are LibC threads whether they care to be or not. Of course, being a LibC thread doesn't mean much until an interface in LibC is called and then, it just means that things will work (which is not, as all experienced NLM writers know, the case for CLib). LibC replaces CLib as the environment of choice for NetWare.
NetWare 5.0 had a performant, multiprocessing kernel that then became the impetus for applications, particularly Novell's applications, to recode in order to take advantage of the ability to run on multiple processors. CLib wasn't up to the task. It was multiprocessor safe, but not a good multiprocessing environment. The need to solve both the multithreading and multiprocessing issues, plus questions of portability prompted the creation of the Novell Kernel Services (NKS) environment. The hope was to make writing appliances and other applications much easier across Novell's platforms plus give them the power through sophisticate and highly orthogonal tools. Before the advent of Linux at Novell, a new 32/64-bit platform, code-named Modesto, was planned and mostly developed. The kernel interfaces were NKS and nothing else.
However, Novell hadn't yet embraced the Open Source community beyond a few internal projects. Since that time, the world has changed radically with the emergence of Linux. The original, industry standard portability solution, POSIX, has moved to the forefront although it can hardly be said to be much of a portability solution. It is more a statement of functionality common to Open Source applications and the extensions to the set of interfaces used in the community far outstrip the set established by ANSI and POSIX (ISO/IEC 9899:1999 and 9945:1996 with an update due out soon).
As it became clear that opening up to this larger community would be in our long-term best interest, the focus shifted away from providing the highly advanced NKS environment toward making NetWare an easier target of Open Source.
In a sense, therefore, NKS has been abandoned as the preferred interface consumed by NLMs. It persists, however, in that it is still used to neutralize NetWare filesystem platform peculiarities–a sort of metalayer that covers DOS/FATFS, NSS, traditional and remote NCP-based filesystems and provides an easier basis for the traditional POSIX open, read, lseek, fcntl, etc. interfaces to be implemented. This means that we sometimes write to NKS instead of directly to NetWare, especially when we know the NKS layer to be stable.
Second, it provides a opportunity to give a more NetWare-centric view of filesystem details–details that find no expression in industry-standard interfaces.
In summary, NKS interfaces are no longer the preferred interfaces for threading, synchronization and most other functionality for which it was originally destined, but it is still important to the library internally as a metalayer and is useful for filesystem access beyond what standard I/O operations provide.
Because of a lack of application or process containment in NetWare, NKS created the virtual machine (VM) concept for hosting applications. This brought to NetWare a containment model that existed on the Modesto platform. This was discussed in the article series years ago. Though there are vestiges in LibC naming practices, the concept of the virtual machine is largely an obsolete one that has yielded to the process in the UNIX sense.
Because programming to NetWare is still an exercise in writing kernel code, the NetWare meaning of process has its limits. However, containment has grown very much stronger these last couple of years because of the need to support advanced concepts such as fork which will make its appearance very soon. As can be imagined, it will be impossible to call fork from a kernel-resident NLM. Instead, an NLM must run in a protected address space AND be the only process running in that address space. (The nature of the protected address space that has existed on NetWare since version 5.0 is a protected version of what goes on traditionally in the kernel and, because more than one NLM can be loaded into a single protected address space, ill-resembles a UNIX process.) To distinguish such an address space from the purely NetWare variety, we call it the single-process address space and it is our process model. Indeed, in order to load into it, you must link an entirely new prelude object (called posixpre.o as of this writing). LibC will reject the loading of more than one NLM with a main function into this single-process address space. Any number of NLMs without main functions, what we call libraries, are allowed to load there.
Experimental versions of this are beginning to emerge in the NDKs; however, the concept won't be completely defined until fork is finished. By the next NDK, it will be possible to write one of these NLMs and see how they work, but fork won't be ready. Hence, an NLM in the special protected address space will not be able to launch a child into another, special protected address space.
In the meantime, the best way to launch another application is to use procle or procve. A minimal child relationship can be set up including use of wait or waitpid. Unfortunately, signals is not yet available. However, wiring up standard consoles between parent and child, managing environment variables, and even some descriptor sharing is possible.
These are under development and will doubtless appear in the same service pack as fork.
You should use pthread.h and semaphore.h interfaces for all threading and synchronization. Do not use NKS. We maintain UI threading and synchronization in thread.h and synch.h for programs ported from Solaris and other UNICES, but we consider pthreads to be our preferred interfaces.
- Data instancing
This problem still remains in LibC even if many of the context problems that afflicted CLib have been overcome in LibC. There are two types of data that must be instanced, thread-specific data and static or global data.
Thread-specific data is easy to instance and LibC does not make a mistake in doing it. The best example of thread-specific data would be errno.
Application globals, on the other hand, are tricky because while it is easy to find the thread-specific data from the executing thread itself, it is not always abundantly clear on NetWare what application or process owns that thread.
- Thread migration
This is still a problem for application code that expects to be called back by other code, but wants the effects of anything accomplished to be relative to the application. In other words, if code executed by a thread that does not belong to the application touches a variable that is instanced for the application, then there is a problem because LibC cannot know which instance of that variable to touch. This problem is solved by wrappering any function address that is used as a call-back. Call-backs are not common in a protected address space since call-backs that across ring or address space boundaries are very difficult to write. For more on this topic see http://developer.novell.com/ndk/doc/libc/libc_enu/data/a640kn9.html.
With the new concept of process on the NetWare, libraries can be written using the old, UNIX-style approach of _init and _fini. The modern, C++ approaches to this haven't been thought through yet and may simply be a function of the tools used (Watcom C++, g++, CodeWarrior, etc.) as they fall outside the domain of LibC. Additionally, and because it's a cleaner model, DllMain is also supported despite the obvious attempt on our part to look to UNIX and Linux. A good approach is still a good approach.
There is a difference, however, to loading libraries into the single-process address space. As they are loaded, their initialization code is not executed. This happens only after application–the NLM containing a main–loads. Subsequent libraries loaded are immediately initialized, however. Perhaps the reasons for this will be explored in a later article; for now, they would transcend the scope of the current one.
At this writing, it is planned that applications and libraries linking the new prelude object will still be loadable in the NetWare kernel. (We are nothing if not full of allegiance to the past.)
Originally, we had decided that all strings in NKS would be in Unicode, which we thought to be a forward-looking strategy. By default this is not true because few programs exist in Unicode, but we found that we could easily and usefully enhance all interfaces to accept either Unicode, UTF-8 or ASCIIZ where their signatures permit. We decided to add a flag to the NLM linker that will instruct LibC to use UTF-8 strings in place of ASCII in all its path-string oriented functions (LD_UTF8_STRINGS). The LD_UNICODE_STRINGS link flag was made obsolete because a) only NKS interfaces accept Unicode strings, b) few are going to program to them, and c) those few can use UTF-8 instead since the translation between Unicode and UTF-8 is completely algorithmic and non-lossy.
At this writing, there are few if any interfaces in LibC that cannot deal with a UTF-8 string passed as long as the underlying commodity can make heads or tails of them. For example, this means that passing CJK (Chinese, Japanese or Korean) content in UTF-8 to open will be meaningful if the underlying filesystem is NSS or remote NCP-based; however, if it is DOS or the traditional/legacy Novell file system, then we translate from UTF-8 into the server's local codepage before passing the string down. Such translation is grossly lossy and, in the case of heterogeneous locales where, for example, European and Asian content may be mixed together, potentially disastrous.
Since 7-bit ASCII is a perfect subset of UTF-8, linking the NLM as a UTF-8 NLM means little if all string content is English.
Wide character handling (wctype.h, wchar.h) in LibC means Unicode and most of the expected interfaces are there. However, if you are having trouble handling finer distinctions in Unicode, especially translation which the stdlib.h interfaces (wctomb, mbtowc, etc.) handle naively, unilib.h may be your best bet. There is nothing portable at all about these interfaces though the library is available on Linux. This latter library provides translation between Unicode and Macintosh codepages now too.
The concept behind file and directory I/O for NKS was that the interfaces be portable, that is, common to all platforms, inherently multi-threadable (atomic) and succinct (reduced semantics). Unfortunately, NKS did not achieve the status of standard and so the preferred interfaces for filesystem work remain ANSI and POSIX.
This area has been the focus of a lot of intensive work since LibC was first issued. As Open Source applications have been ported to NetWare, the list of supported interfaces continues to grow. Also, we've revamped some of their semantics.
- Path Strings
It is now possible to pass paths with complete POSIX appearance. For now, the signal to LibC that you are doing this is the existence of the LD_WANT_POSIX_SEMANTICS flag set on the NLM at link time. However, we may decide to replace the use of this flag with the linking of the POSIX prelude, posixpre.o. (There also exist posixpre.gcc.o and posixpre.obj.)
- Passing POSIX paths means not passing NetWare-style paths any more. To aid in this, some assumptions are made, and the whole notion is guided by the existence of a special file inspired by UNIX /etc/fstab. Ours is located on the path /etc/pathtab, which is really the POSIX equivalent of SYS:\ETC\pathtab.
- First, it is assumed that anytime a path is preceeded by the forward slash (/), it is a full path starting at the server's SYS: volume root.
- Volumes, directories and files on NetWare servers (except, as already noted, for directories and files on SYS:) can only be targetted by referencing them in /etc/pathtab. This file was inspired by UNIX' /etc/fstab, but the references are not the object of any genuine filesystem mounting process. Instead, this file serves as a database of path equivalents for use by the library. The following entries illustrate a path equivalent for both remote volumes and a local volume:
/remote/governator arnold/sys:\ # remote volume "ARNOLD/SYS:" mounted thus /krypton krypton/vol1:\ # remote volume "KRYPTON/VOL1:" mounted thus /datavol data:\ # local volume "DATA:" mounted thusThe path equivalents are fairly open as far as allowable characters. Indeed, except for using white space to parse on, all character validation is left to the underlying filesystem. Since only the NetWare "path rewrite" (here, arnold/sys:\ and krypton/vol1:\) actually reach the filesystem, the UNIX-style element could have contained any character except for space, tab, pound or the newline. This seems adequate except for space, but we will not attempt to close that hole unless and until someone requires it at which time we'll allow double-quoted paths.
- The choice of any equivalent is crucial to the continued access of the system-volume file or directory: if in the illustration above, data had been chosen instead of datavol, then a file or directory on the path named SYS:\data would become inaccessible to code written to LibC (with the full POSIX semantics flag linked).
By default, LibC uses the LONG namespace (and not NFS/UNIX). This creates a) a deficiency in dealing with the old primary namespace (DOS) and b) confusion over the case of a filesystem path or its elements because while case is significant on UNIX and Linux, it is ignored on Windows.
- LibC distributes fsio.h in an effort to deal with these and other issues. For example, set_pathname_format is a function that tells LibC to use a different namespace in dealing with the paths it returns in, for example, readdir. It is also possible to get the actual case of names by calling getstat or getstat_with_namespace. For details, see the LibC manual http://developer.novell.com/ndk/doc/libc/libc_enu/data/agtut92.html.
- Interprocess Communication (IPC)
While we still do not have fork, signals, memory-mapped I/O or shared memory, pipes do exist. Also, there is an implementation of System V semaphores.
Screens merit a word or two here. By default, an NLM application would have a screen just as a Linux application has the console of its command line. Screens range between two major types, the kind that come up automatically with your NLM and the type you created by hand.
An NLM's screen, as created by NetWare rather than explicitly by you, is wired up to stdin, stdout and stderr. When you link your NLM, you have the opportunity to name its screen with the SCREENNAME command, usually kept in a .def file associated with the linking process. If you give the screen a name, then from the NetWare console, holding down the ALT key while that screen is up will show you this name. Also, this is the name that is seen in a list when you hold down CTRL and hit the ESCAPE key. If you name your screen "System console", then no screen will be created, but stdout and stderr will be wired to the NetWare system console or logger screen (depending on the version of NetWare you are running), and stdin will not be wired since input from the system console or logger is denied to NLMs. If you name your screen "none", then you will get no screen and stdin, stdout and stderr will not be wired to any console.
It is possible to open another screen using calls directly into NetWare. These calls are prototyped in screen.h and documented. They are not nor can they be wired to the standard C consoles. They can be used in support of a NUT screen (see nwsnut.h interfaces in the documentation).
And yet, for all the POSIX, Linux, FreeBSD and Single UNIX Specification additions we are making to LibC, we continue to bulk accessibility to NetWare features and functionality. We have done this in two ways.
First, NetWare is a mature operating system now. We refused to certify NLMs that consumed OS interfaces directly in order to by-pass the approved ones in the library. While this is still certainly the case, a great deal of NetWare has been opened up via LibC headers event.h, netware.h and screen.h. In these headers, developers can find safe, documented and stable interfaces to the NetWare kernel or kernel features.
Second, we publish a number of interfaces to more complicated ones in the OS to cover functionality that has not heretofore been available or was available in CLib or XPlat in a different form. The headers esm.h, fshooks.h and monitor.h are good examples of this. The first interfaces 4Gb+ memory, the second, fshooks.h, is essential to antivirus and back-up software. monitor.h provides some information similar to what can be seen using monitor.nlm or the NetWare Remote Manager from a browser.
LibC includes headers for Berkeley Sockets (sys/socket.h, sys/select.h, etc.) and Winsock interfaces (novsock2.h). Both of these interfaces are the same ones available in CLib. You should select one of these and not use both types of sockets in the same applications.
This is something we've spent more time on for ourselves than for you, but you reap the benefits. The NetWare System Debugger has been greatly enhanced; this is a topic for an entirely different article, but let's note that there is better help for debugging in protected address spaces, soft breakpoints, and other cool things. To get to the NetWare System Debugger, hold down both SHIFT keys, press one of the ALT keys and hit ESCAPE. Once in this mode, use the command 'g' to get the server running again.
The NetWare Remote Manager (NoRM) continues to be enhanced as well and has a special LibC view that lets you look at pretty arcane things on every LibC NLM including lists of open files, standard console mappings, locale settings, and much more. Prior to CSP12, this only works for debug versions of LibC distributed in the NDK. To see the LibC pages, look for LibC View down the left-hand thumb of the NoRM page and click on it.
Starting in the CSP11 library, new debugger commands to list processes (LibC-only), display information about an open file, etc. have been included. As noted, most of this is for our team's benefit, but hopefully you will also find it helpful.
Probably the three most import NetWare System Debugger features for developers are displaying errno after an interface has failed, displaying the arguments to a function, and tracking down leaked memory.
- Displaying errno and function argument values
If a function is known to fail, but you don't have a print statement to reveal the value in errno, set a breakpoint on the function, go there, do a magic return, the examine errno. Assuming the failing function is fopen, the commands are listed in order below. Also, typing "parms" when on the first instruction of a function reveals all the arguments. This cannot be done accurately at any other instruction.
# b=fopen # g # parms # (press F3 here for the magic return–stops at return point in the caller) # errno
- This display will list not only what's in errno, but all the other error values as well including:
- filesystemerrno -- NetWare-level detail on a failed filesystem call (like fopen)
clienterrno -- from client.h interfaces
dlerror -- from dlfcn.h interfaces
h_errno -- communications interfaces (netdb.h)
- Tracking memory leaks
With developer options turned on, memory leaks still generate nasty grams at the console when an NLM unloads. Moreover, the NLM cannot pass certification as long as memory is leaked. LibC provides a way to track these down. The memory tracker works only with a debug LibC (available in the NDK). Basically, track memory leaks are listed out when the NLM is unloaded display a stack crawl for each allocation deep enough to be able to identify the place in your code that allocated the memory.
- In the debugger, type
# libc help
- This will show you a few commands, but in particular, we are interested in "nlm track memory" and "libc track memory" and status. To turn memory tracking on, which you must do before loading your NLM, enter the debugger and type
# nlm track memory on
- This will track memory your NLM allocates using malloc, etc. To see memory leaks indirectly caused by things your NLM does that it does not also clean up, use the "libc track memory" command.
- Both commands may be used together. Their status is listed by the "track memory status" command and this facility may turned off (because it's rather expensive) by replacing "on" in the initial command with "off."
Today NLM development mostly proceeds using one of the following tools.
- - Metrowerks CodeWarrior. This is still the premier development environment though there are no updates to it planned. It consists of an IDE and integrated debugger. It is also possible to use the tools (license.dat, lmgr326b.dll, mwasmnlm.exe, mwccnlm.exe and mwldnlm.exe) stand-alone on the command line. The cost for Metrowerks CodeWarrior is around US $400. It is best obtained from Novell because in addition to the actual Metrowerks software, there is a professional developers' kit (PDK) tailored to deal with NLMs including pre-written stationeries and understanding of the NDK as customarily downloaded. LibC itself is built using the command-line tools. http://developer.novell.com/ndk/cwpdk.htm.
- - Watcom 11.x. This was the last official release of Watcom C/C++ before it became Open Watcom. The linker suffers from not being able to handle certain details of modern NLM programming, but the Open Watcom environment solves all these problems. The main problems were the inability of the linker to deal with symbol prefixing and being able to link with the C++ class library without also requiring Watcom's own C runtime library. Many Novell components, like CLib, are still developed or maintained using this environment because it is still completely competent when new features are not needed.
- - Open Watcom. This environment is freely available at no cost. Sybase recognized with the end of supported Watcom development that an open community could be found to carry on development and benefit from this environment so it released it into the open. http://www.openwatcom.com.
- - gcc. It is already and has been for a long possible to create NLMs using gcc and and a utility named nlmconv. Nevertheless, we at Novell are in the process of refining this environment considerably, especially for use on Linux. There is nothing that can be announced at this writing. A good place to start in the meantime is the LibC newsgroup (news://developer-forums.novell.com/novell.devsup.libc) or Bernd Herd's pages at http://www.herdsoft.com/ti/netware/cross/.
The direction of NetWare with respect to C/C++ support is clear: it is toward making NetWare as easy an Open Source target as Windows with as much POSIX and Linux in its interface set as possible.
Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com