NAM Identity Server Error 300101017 "An Identity Provider response was received that failed to authenticate this session."

Access Manager Identity Server, installed on SLES11 SP2 OS on Microsoft Azure platform, is configured with SAML2 federation to 3rd party Identity (IDP) Server. Everything works fine for the most part ie. users can access the NAM SAML2 SP, get redirected to authenticate to the remote SAML2 IDP server which generates the required assertion. The 3rd party IDP server assertion includes an Authentication and Attribute Statement containing information about the user.

If users with extended characters (e.g. æ, ø, å and ö) are registered in AD, and the assertion authentication or attribute statement includes these extended characters, SSO to the NAM SAML2 SP server fails with the following error presented on the browser:

"Error on session id xxxxxxxxx,
error 300101017    -yyyyyyyyyyy, An Identity Provider response was received that failed to authenticate this session."

where
xxxxxx is the users JSESSIONID cookie value assigned by NAM and
yyyyyy is the device ID of the NAM server generating the error

The catalina log file on the NAM SAML2 SP server also indicates the following Signature validation problem:

<amLogEntry> 2013-12-04T12:48:46Z INFO NIDS IDFF: AM#500106006: AMDEVICEID#B27B49C42713A62D:  Validation failure on message from https://alta.novell.com/svc/trust : Signature validation failed </amLogEntry>

If a user authenticating at the remote IDP server did not have any extended characters, no errors would be reported and SSO would succeed without issues. In the example below, the user netiq2 had a surname of Tøster ... simply changing that to Tester would cause the SSO to succeed.

// sample AttributeStatement from Assertion when failing shows:

  <AttributeStatement>
            <Attribute Name="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress">
                <AttributeValue>netiq2@novell.com</AttributeValue>
            </Attribute>
            <Attribute Name="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname">
                <AttributeValue>netiq2</AttributeValue>
            </Attribute>
            <Attribute Name="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname">
                <AttributeValue>Tøster</AttributeValue>
            </Attribute>
        </AttributeStatement>

The same NAM setup going to the same SAML2 Identity Server has no issues when the NAM SAML2 Service Provider is configured on a local VMware host, and not Azure.

Make sure the local settings on the SLES host is set to UTF8. 

Customer installed the standard SLES11SP2 image provided in Windows Azure, where Northern Europe region set to Norwegian layout. There are some strange default settings in this image.

Modified the /etc/sysconfig/language and changed from LANG=”en_US” to “LANG=en_US.UTF-8” and problem went away.