Unicode-to-byte Conversion

Unmappable Unicode characters result in a call to a function handler, which forms the basis of lossless round trip conversion. The handler converts each unmappable Unicode character into a string of six byte characters as follows:

For example, if the character "#" is an unmappable Unicode "skull and crossbones" character (U+2620),

Scan/parse functions are disabled.

Byte-to-Unicode Conversion

Unmappable byte characters result in a substitution by the standard Unicode REPLACEMENT CHARACTER-0xFFFD.

The scan/parse functions are enabled, reversing the Unicode-to-byte function handler behavior. These scan/parse functions scan for the byte sequence "[NNNN]", where NNNN is a string of four hexadecimal digits. Scan/parse convert each such sequence to a single Unicode character whose value is U+NNNN. For example, if the character "#" were again the Unicode "skull and crossbones" character (U+2620),

Multiple Code Page Converters

Previously, an open code page had to be closed before a new code page could be opened. Using the new extended API functions, you can have multiple byte/Unicode converters loaded and active simultaneously, each with a different code page. For each converter, a handle is returned when the load function completes.

It is important to unload each converter when it is no longer needed, as explained in Unloading Converters. This helps avoid the possibility of tying up system resources needlessly.

Unmappable Characters

This option defines what action to take when an unmappable Unicode character or (less likely) an unmappable byte is encountered during conversion. The options are to

These options are set for individually for Unicode-to-byte conversion and byte-to-Unicode conversion. NWUXGetNoMapAction and NWUXSetNoMapAction specify both options. Refer to the function reference for details.

It is important to note that the default for Unicode-to-byte conversion is to call the handler function. That handler is described in Default Conversion Behavior.

Substitution Characters

If the NoMapAction is set to NWU_SUBSTITUTE, a substitute byte or Unicode character is output when an unmappable character is encountered. By default, NWU_SUBSTITUTE is set for Unicode-to-byte conversion and not set for byte-to-Unicode conversion.

The default substitution byte or Unicode character is determined by the converter, since different countries often have different preferences on what to display for undefined characters. For byte-to-Unicode conversion, the substitution character is U+FFFD, designated as the Unicode REPLACEMENT character. For Unicode-to-byte conversions, the converters generally set the default substitution byte to 0x03.

You can find out what the substitution characters is through NWUXGetSubByte or NWUXGetSubUni. You can set a new substituting character through NWUXSetSubByte or NWUXSetSubUni.

Fore related information, see:

Scan/Parse Action

Two scan action options are defined, one for converting Unicode-to-byte, and one for converting byte-to-Unicode. In the extended API, options are enabled or disabled through NWUXSetScanAction. By default, the scanAction is disabled for Unicode-to-byte and enabled byte-to-Unicode.

Enabling the option causes an automatic prescan of the input string for any special sequences and calls a parse function to replace such sequences with something else in the output string.

When the scanAction option is enabled, a Scan function is called internally to scan the input string before the conversion. If it finds a special sequence, the conversion is performed up to that point and then the Parse function is called internally. The functions are never called directly by the developer. Rather, they are set as explained in NoMap, Scan, and Parse Functions.

The system supplies default scan and parse functions for both byte-to-Unicode and Unicode-to-byte conversions. The byte-to-Unicode scan/parse functions operate as described in Default Conversion Behavior—where # is the Unicode "skull and crossbones character" (U+2620), the byte input string "abc[2620]def" becomes the Unicode output string "abc#def".

By default, scan/parse action is disabled for Unicode-to-byte conversion because the need for such action is very rare. If it is enabled, it operates in a similar way to byte-to-Unicode conversion. It scans for two hexadecimal digits surrounded by square brackets in a Unicode input string and converts them into a byte character of the same hexadecimal value in the byte output string.

For related information, seeSetting Scan/Parse Functions with an Extended Converter

NoMap, Scan, and Parse Functions

NWUXSetByteFunctions and NWUXSetUniFunctions set the NoMap, Scan, and Parse functions for the extended converter. The NoMap function is enabled if the NoMapAction is set to NWU_CALL_HANDLER, and the Scan and Parse functions are enabled if the ScanAction option is set to NWU_ENABLED.

The default behavior (the only behavior available for the standard converter) is to use the system supplied UniNoMap function and the ByteScan/Parse functions as described in Default Conversion Behavior. These functions implement round-trip conversion from Unicode to byte to Unicode.The developer may replace any of these functions with custom versions.

For related information, see Setting Scan/Parse Functions with an Extended Converter

Length-Specified Byte String Conversion

Unicode provides functions for converting a specified number of bytes from a byte string into Unicode characters.

For the standard converter, the functions are NWUSLenByteToUnicode and NWUSLenByteToUnicodePath.

For extended converters, the functions are NWUXLenByteToUnicode and NWUXLenByteToUnicodePath.

These functions behave exactly like their "Len" counterparts: NWUSByteToUnicode, NWUSByteToUnicodePath, NWUXByteToUnicode, and NWUXByteToUnicodePath, with the following exceptions:

If the length-specified function encounters a NULL before the specified number of bytes have been converted, it stops converting and returns NWU_EMBEDDED_NULL. However, it converts the bytes prior to the NULL, and returns the number of Unicode characters converted.

For example, consider the byte string abcdefgh for the following example:

ccode = NWUSLenByteToUnicode (&outbuf, MAX_LEN, inbuf, 5, &outlen);

On return ccode is zero, outbuf contains the Unicode string abcde, and outlen contains 5.

In contrast, given the byte string abc\0defg, the NWUSLenByteToUnicode function returns NWU_EMBEDDED_NULL. On return outbuf contains the Unicode string abc and outlen contains 3.

For related information, see