VNC connection fails with "Invalid Protocol"

This document (7000039) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11 (SLES 11)
SUSE Linux Enterprise Desktop 11 (SLED 11)
Remote Administration (VNC) enabled.
VNC client or browser being used to establish the VNC connection.


Situation

After connecting through VNC 3 times successfully, and logging out each time, the fourth attempt to connect fails with an "Invalid Protocol" error when using Ultra VNC.  The failure will happen with any VNC client but may report a slightly different error.  For example, when using vncviewer from another SLED/SLES system the message indicates that the system being connected to is not a VNC server.

The error may go away after some time (15-30 minutes) or a reboot may clear the error.

Resolution

While there are no users logged into the desktop or connected remotely through VNC, stop the X server, remove the files indicated below, then start the X server back up again.  Use a SSH session or one of the VTs (Ctrl-Alt-F1 through F6).  Log in as the root user and run these commands:

rcxdm stop

rm /tmp/.X*-lock

rm /tmp/.X11-unix/X*

rcxdm start

Test the VNC connection to see if the problem is resolved.

Cause

The /tmp/.X*.lock are the lock files created by the X server when it starts up for the specific display number (the number after the X is the displaynumber).

The /tmp/.X11-unix/X* are the socket files created by the X server when it starts up for the specific display number.

When an instance of the X server is terminated properly it will remove these files, indicating that the display is no longer in use.  If for some reason the X server is not terminated properly then these files can be left behind and eventually cause the X server to report that there are no displays available.

The message reported by the VNC client does not give any indication of the problem.  The only place to see the display error is in a packet trace.

Additional Information


Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7000039
  • Creation Date: 21-Oct-2013
  • Modified Date:03-Mar-2020
    • SUSE Linux Enterprise Desktop
    • SUSE Linux Enterprise Server

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center