NetWare FTP is not displaying UNICODE / UTF8 / double-byte filenames

  • 7001283
  • 04-Sep-2008
  • 26-Apr-2012

Environment

Novell NetWare 6.5
Novell NetWare FTP Server (NWFTPD.NLM)

Situation

A directory on a NetWare server contains one or more filenames containing Unicode characters, or characters which require double-byte to represent.  These files (and possibly some or all other files in the same directory) are not displaying in a directory list displayed through NetWare FTP Server.

Resolution

There are limitations of the FTP Protocol itself which are usually coming into play here.  There are some ways to troubleshoot and workaround this, but fundamentally there is no smooth, 100% solution.  Here's a summary of the points involved:
 
Problems:
  • The FTP Protocol cannot easily deal with Unicode / UTF8 / other double-byte representations in filenames.  FTP clients and servers have no mechanism to negotiate what type of character representation to use.  They treat filenames as simple byte sequences, i.e. the client will simply assume that it is sending / receiving names in its own local code page, and the server will make a similar assumption on its own end.  Yet the 2 systems could easily be using different code pages.
  • Before the NetWare FTP Server can send a filename over the wire to a client, the OS must convert that name from Unicode (in which a NSS volume stores it) to the server's local code page.  There are many characters that exist within Unicode that won't exist within various local code pages.  When the OS attempts to convert a character and that conversion fails (due to no mapping into the local code page), an error is returned and FTP's retrieval of a directory list will stop at that point. 
Workarounds / Troubleshooting:
 
  1. Sometimes choosing a more appropriate local code page for the NetWare server will relieve this problem.  But no individual code page will contain mappings for all possible Unicode characters.  Some will be non-translatable.
  2. NSS can be told that when a non-translatable name is retrieved, instead of generating an error, a substitution should be made.  When this tool is activated, double-byte characters will be displayed as  a 6-character string, consisting of (for example): [5C42]   (This is a representation of 2 bytes, in hex.  Including the brackets, it takes 6 characters to represent this.)

    To activate this tool, give the following command at the console prompt:
    NSS /NCPDisplayNonTranslatableNames
    This tool is intended for troubleshooting only.  Once enabling it, all files can be seen and the ones containing non-translatable characters can be identified.  Those files should be renamed or removed (which might have to be done through a client other than FTP) so that non-translatable characters are no longer being used.  Once the filenames are corrected, this tool should be turned off again with:
    NSS /NoNCPDisplayNonTranslatableNames

Additional Information

This problem is not limited to FTP.  Some other protocols can experience the same thing in certain situations.  For example, if Novell Client32 for Windows has it's UTF8 support turned off, it will experience a similar symptom.  And the same NSS tool above can be used in that situation as well.
 
The NSS tool mentioned is not a full-fledged feature; it cannot seamlessly support all file system operations.  It is intended only for use as a troubleshooting tool.  Novell does not recommend or support normal production activities while this setting is active.
 
This NSS setting will not effect clients methods which already understand Unicode / UTF8 / etc.  For example, Novell Client32 for Windows, with UTF8 support turned on, causes NetWare to retrieve file names from Unicode into UTF8, sends UTF8 over the wire, then Client32 turns them into displayable names based on Window's own code page settings.  This completely avoids the step of converting Unicode to the NetWare Server's local code page.