Pure-FTPd does not handle ASCII Transfers in the same way as NetWare FTP

  • Document ID:7008378
  • Creation Date:15-Apr-2011
  • Modified Date:07-Jun-2013
    • Micro Focus Products:
      NetWare
      Open Enterprise Server

Environment

Novell NetWare
Novell Open Enterprise Server 2 (OES 2) Linux
NWFTPD.NLM
Pure-FTPd
Carriage Return
CR
<CR>
Decimal 13
Hex 0D
LF
Decimal 10
Hex 0A
\r
Control+M
^M

Situation

  • ASCII Files get "corrupted," (modified) binary files do not
  • ASCII FTP transfers to/from NSS Volumes migrated from NetWare to OES2 gain/lose Carriage Return characters
  • ASCII files downloaded from an OES2 Pure-FTPd Server to a *nix (Linux, Solaris or Unix) system have embedded Carriage Return <CR> characters; the same files downloaded from a NetWare FTP server do not
  • ASCII files uploaded to an OES2 Pure-FTPd Server from a *nix (Linux, Solaris or Unix) system have missing Carriage Return <CR> characters; the same files uploaded to a NetWare FTP server do not
Example
  • Downloading ASCII files from NWFTP on NetWare to a Solaris box DOES strip <CR> from files (desired behaviour)
  • Downloading ASCII files from pure-FTP on OES to a Solaris box appears NOT to strip <CR> from files  (not desired behaviour)

Resolution

This is working as designed, though it may not be desired.  FTP ascii mode transfers are intended to "pad" files to a certain standard when sending them over the wire, and then the receiving side is intended to set the line deliminters to those which are preferred by the OS.  Linux and Unix prefer just Line Feed.  DOS / Windows / NetWare prefer Carriage Return and Line Feed.
 
Since OES Linux is replacing NetWare in many cases, some processes which rely on the way NetWare did things are not having the desired result any longer.
 
Novell has enhanced pure-ftpd to have the option of silently disabling ascii modifications.  The enhancement is publically available in pure-ftpd 1.0.22-0.24.1 (or newer).  Put the newer version in place, then edit /etc/pure-ftpd/pure-ftpd.conf and find / set:
 
disable_ascii  yes
 
This will turn off all ascii modifications done by pure-ftpd.  FTP clients can still request ascii mode and will be answered positively.  They may still do their own modifications of ascii files, but pure-ftpd will not.  This option is very ideal for cases where previously NetWare FTP was in use; files were created by FTP or NCP clients; and OES Linux is now taking over for NetWare.  With this new feature, during an upload, pure-ftpd will no longer strip out Carriage Returns at the end of lines.  It will write the file exactly as received from the client.
 
However, this setting may not be ideal for files created directly on the OES Linux box by traditional Linux processes.  For example, if an ascii file is created by traditional Linux processes, it will only contain LF (Line Feed) at the end of each line.  Pure-ftpd (using this new setting) will not add a CR (Carriage Return) during download, so if is being downloaded by a DOS/Windows client, it will not have the format that some DOS/Windows applications expect.
 
 
Alternative Workarounds
1.  In some cases, altering the FTP sessions to use binary mode instead of ascii mode is sufficient to overcome these kinds of problems.  It just depends on how each system creates files and what line deliminters they prefer.  Binary mode transfers never alter the file, so if the "original" form of the file is good for all systems that must use it, binary transfers will be simple and safe.
 
2.  Use the Linux commands dos2unix and unix2dos (note that unix2dos may not be installed by default on some systems and will need to be installed separately) to convert between formats.  Refer to the man pages for details of these commands.

Additional Information

Summary of ASCII Transfer Process (per the official specification, i.e. without this new feature activated)
  1. The sending side of an FTP ASCII mode transfer find anything it considers to be its own "end of line" pattern, and change that to <CR><LF>.  The sending side may be a client (for uploads) or a server (for downloads).
  2. A *nix host considers <LF> to be its own pattern, so when sending, it will pad any <LF> to <CR><LF> to satisfy Point 1 (even if a <CR> was already present, since <CR> is not part of it's own pattern).
  3. A NW/DOS/Win host considers <CR><LF> to be its own pattern, so it does not have to make any alterations when that pattern is found (regardless of whether it is sending or receiving).
  4. A *nix receiver which sees <CR><LF> arrive will strip it to just <LF>, because that is its own pattern.
Consider how text files created on Windows and Linux are handled differently by notepad and vi.  The same rules are at work with FTP servers.
  • NetWare/DOS/Windows assume that their local files will have both <CR><LF> and most of their local utilities from notepad.exe to ftp assume this is so and will behave oddly if <CR> is not present
  • *nix assumes that its local files will only have <LF> and all of its local utilities from vi to ftp assume this is so and will behave oddly if <CR> is present
  • In other words, everything is behaving as expected and is consistent to each operating system
 
An example of the implications, using the example scenario from the situation section above:
  • A file exists on OES Linux which (for some reason) already contains <CR><LF>
  • Per Point 2, when pure-ftpd sends that file in ascii mode, it assumes that all <LF> should be changed to <CR><LF>.  It doesn't check whether <CR> is already there.  Its job is to find all occurances of it's own pattern <LF> and add a <CR> before it.  So the file crosses the wire with <CR><CR><LF>
  • Per Point 4, a *nix receiver notices the standard "over the wire" pattern <CR><LF> and changes is to just <LF>.  But the other <CR> is left intact.  The final result is <CR><LF>, creating the illusion that <CR> is not stripped.