Administrator's Guide

CHAPTER 14

Troubleshooting

This chapter describes some techniques and procedures that you can use for troubleshooting the Novell exteNd Application Server. This chapter contains the following sections:

• Using error logging

• Low-level debugging

• Setting JDBC tracing

• Using the server's command shell

• Using the Watcher

• Common problems starting the server

• Using SilverMonitor

• Using the SilverMasterInit program

• Handling a stack overflow

• Miscellaneous issues

Using error logging

Consider turning on error logging at all times when running the server. Error logging is a lightweight process that prints detailed information about error messages either to the AgErrorLog table in the SilverMaster or to a file you designate. You can activate logging using the SMC. For more information, see Using server logging.

When logging to the SilverMaster, you can view the log in the SMC by selecting the Monitor icon from the toolbar and then selecting Logs.

    For more information, see Displaying logs.

Low-level debugging

The Debug options in the SMC enable the printing of server debug messages to the server console. Options include debugging client requests, Web applications, and SQL statements. You should activate debugging options only for application debugging purposes, as this activity can significantly inhibit server performance.

NOTE:   If you are running the server as a service in Windows, the output is printed to the error log instead of the console window.

To print debugging messages:

1. Start the SMC.

2. Select the Configuration icon from the toolbar.

3. Select Advanced.

4. Select the Debug tab:

   

5. Change the value for the type of activity you are debugging.

   The number you enter indicates the level of detail you want displayed. The value 0 means that messages are not printed. You have the following debugging options:

   Field	Description
   Client	If this parameter is set to 1, the server logs information for each client (such as http GETs, PUTs, and POSTs). If this parameter is set to 2 or greater, the server logs the complete request and reply message to the console window. TIP:   This option is useful for problems with HTTP, servlets, and other client-related issues.
   Web Application	If this parameter is set to 1 or higher, the server logs information about the execution of each running Web application as follows: If set to 1 or 2, provides minimal or maximal information about methods called in the public API. If set to 3 or 4, provides minimal or maximal information about all methods called, even methods not in the public API. If set to 5, echoes all output to ServletOutputStreams so that you can see what is being output from the business object. If set to 6, includes stack traces of various calls so that you can see where calls are being made.
   SQL	If this parameter is set to 1, the server logs each SQL statement executed against the database for client data. If this parameter is set to 2 or greater, the server logs additional information that Technical Support can use to track down server problems. TIP:   This option is useful for debugging database-related problems.
   Class Loader	If this parameter is set to 1 or higher, the server logs information about the J2EE ClassLoader (used for each J2EE application) as follows: 1—Shows only the list of repositories (places that the ClassLoader looks to find things) when they are added. Use this to find out where the ClassLoader will be looking for things. 2—Shows arguments to the basic API calls like findClass(), loadClass(), getResource(), and so on. Use this to find out when basic API calls are made. 3—Shows more internal information. Use this setting when tracing the search for a class or resource through the various search locations. 4—Shows stack traces in key areas. Use this when you need to know who is loading a particular class.

Setting JDBC tracing

If you have persistent problems with a database connection, use JDBC tracing.

To set JDBC tracing:

1. If necessary, create a log file for storing trace data.

2. Shut down the server (see Shutting down the application server).

3. Open the httpd.props file (located in server's \Resources directory).

4. Add an http-server.Jdbc.DriverManager.LogFile entry to the props file and point it to the log file. For example, if the log file is d:\test\jdbc.log, create this line in your httpd.props file:

     http-server.Jdbc.DriverManager.LogFile=d:\\test\\jdbc.log
   

5. Restart the server.

   NOTE:   Use JDBC tracing for troubleshooting only, as it will slow down the server and use considerable disk space.

Using the server's command shell

You can type commands into the server's console window (the window in which the server was started) to obtain diagnostic information about the state of the server. For example, you can get information on memory usage, threads, sessions, and server system properties. You can also enable tracing for different subsystems.

To learn more after the server has started:

• In the console window, enter:

    help
  

  All available commands are listed, and the command line is marked by a ! character (you can change the character using the prompt command).

To display help on a command:

• Enter:

    help command 
  

To disable the command shell:

1. Enter this line in the httpd.props file:

     http-server.com.sssw.srv.commandshell=false
   

2. Restart the server.

Using the Watcher

The Watcher tool helps you understand the state of the server in cases when the server becomes unresponsive. When it is activated, the Watcher logs the state of the server once a minute.

TIP:   You might find the Watcher valuable when faced with problems that are hard to debug.

To use the Watcher:

1. Add the following property to the httpd.props file:

     http-server.com.sssw.srv.httpdwatcher
   

2. Set the value of the property to the pathname of a watcher configuration file. For example:

     http-server.com.sssw.srv.httpdwatcher=c:\\temp\\watchconfig.txt
   

   TIP:   Remember to escape backslashes in the httpd.props file.

What happens

If this property is set when the server is started, the server will create a watcher thread. The watcher thread sleeps, waking up once a minute to check for the existence of the watcher configuration file supplied as the value of the httpdwatcher property.

If the watcher configuration file does not exist   As long as the configuration file doesn't exist, the Watcher does nothing and just goes back to sleep. Under these circumstances, the Watcher has minimal impact on server performance. And even if the server hangs, in most cases the Watcher will not hang.

If the watcher configuration file exists   When the Watcher discovers that the watcher configuration file exists, it reads the configuration file and uses it to control its further actions.

About the watcher configuration file   The watcher configuration file is an ASCII text file that must include:

• A flags value that tells the Watcher what information to print

  The flags value is a bit-coded integer in which the bits are defined as follows:

  • Bit 0 (== 0x1) dumps threads info

  • Bit 1 (== 0x2) dumps session info

  • Bit 2 (== 0x4) dumps database connection info

  • Bit 3 (==0x8) not used

  • Bit 4 (==0x16) dumps thread event log

• An (optional) output file name for the Watcher output (if not supplied, the output goes to the server console)

A typical watcher configuration file looks like this:

  7
  c:\temp\watchout.txt

This file tells the Watcher to dump information once every minute about threads, sessions, and database connections to the specified output file.

Common problems starting the server

This section describes some reasons the application server might fail to start, and how you can address the problem. For more information on troubleshooting server problems, see Using SilverMasterInit to recreate or refresh SilverMaster.

NOTE:   Server failure is often related to the specific database you are using. For database-specific information, see the Database Configuration Guide.

System resource problems

There are two causes of server failure that are due to inadequate system resources:

Database not synchronized

If you see errors related to database consistency, go to the SMC and execute the Synchronize database schema option, then restart the server.

    For more information, see Synchronizing the database schema.

Using SilverMonitor

SilverMonitor is a background process running on the server that monitors server status and attempts to restart the server if it terminates abnormally. By default, this process is activated when the server is started. SilverMonitor starts with default parameters, which you can modify. You can also run the server without SilverMonitor.

There are two ways you can modify the parameters:

• Specify parameter(s) as server startup options on the DOS command line. See Using startup options.

• Specify machine default values by editing the registry (Windows only).

Order of precedence

The order of precedence is as follows:

• Any values in the registry override the default values.

• Any values manually entered as startup options override registry values.

Summary of parameters

The following is a summary of SilverMonitor parameters:

To modify SilverMonitor parameters in the NT Registry:

1. From the Start menu, choose Run.

2. Type regedit.

   The Registry Editor displays:

   

3. Navigate the tree as follows:

   HKEY_LOCAL_MACHINE>SOFTWARE>Novell>exteNd>AppServer>version number

4. Double-click the SilverMonitor process.

5. When the following dialog displays, type one or more of the options described above. Separate each option with a space.

   The following shows the option for starting SilverMonitor:

   

6. Click OK.

What happens

SilverMonitor writes to the NT EventLog when it restarts the server. It also writes to a SilverMonitor.log file in the directory where it is run (usually the server's \bin directory). This log file gets an entry every time the monitor starts.

When you restart the SilverMonitor, the log file is emptied and restarted.

Using the SilverMasterInit program

The application server relies on the SilverMaster database for overall system management. SilverMasterInit is a command-line program that performs several types of processes on the SilverMaster database. The SilverMasterInit executable is located in the server's \bin directory. SilverMasterInit can:

• Recreate or refresh tables and properties used by the SilverMaster database

• Generate logs

• Display debug information

• Regain access to locked resources

This section contains the following topics:

• Command-line options

• Using SilverMasterInit to recreate or refresh SilverMaster

• Regaining access to SilverMaster

Command-line options

The table below describes how and when to run each of the SilverMasterInit command-line options. To see a list of options, type the following at the command prompt:

  SilverMasterInit -? 

Administration accounts   There are two administration accounts (database and server). Both accounts are defined during installation. The server administration account restricts who can log in and administer the application server. You define the server administration account using SilverMasterInit. After a default installation, the server administrator user account is part of the predefined Administrators group and has the Locksmith privilege.

The application server uses the database administration account when connecting to the SilverMaster database. The only time you need to specify the SilverMaster database account is when you are running SilverMasterInit.

Entering options   You must enter the database user account name and password for all command-line options. You also need to specify Full or Refresh mode for all SilverMasterInit options—except those noted in the following table. When you specify a Full mode database initialization, three options (-A, -n, and -W) require that you also define the server administration account name and password on the command line.

You specify parameter(s) as SilverMasterInit startup options on the command line.

  SilverMasterInit [-f or -r] +cp:a path -U dbusername -P dbpassword

  SilverMasterInit -f -U dbusername -P dbpassword -A adminusername -W adminpassword

  SilverMasterInit -a -U dbusername -P dbpassword

  SilverMasterInit [-f or -r] -b -U dbusername -P dbpassword

  SilverMasterInit [-f or -r] -c -U dbusername -P dbpassword

  <hostname>.<portnumber>

  <hostname>.<portnumber>.com.sssw.srv.nameServicePort=<nameServicePort>

  tundra.8080.com.sssw.srv.nameServicePort=55597

  <hostname>.<portnumber>.com.sssw.srv.port_iiop_ssl_min
  <hostname>.<portnumber>.com.sssw.srv.port_iiop_ssl_max

  SilverMasterInit -C port.props -U smbb -P password -A administrator -W admin -r

  SilverMasterInit -U dbusername -P dbpassword -D Agdb

  SilverMasterInit [-f or -r] -U dbusername -P dbpassword -e c:\ServerLogs\sminit.log

  SilverMasterInit -f -U dbusername -P dbpassword -A adminusername -W adminpassword

  SilverMasterInit [-f or -r] -L c:\ServerLogs\logs\jdbclogfile.log

  SilverMasterInit -l -U dbusername -P dbpassword

  SilverMasterInit -n -f -U dbusername -P dbpassword -A adminusername -W adminpassword

  SilverMasterInit [-f or -r] -U dbusername -P dbpassword -O tablespacename

  SilverMasterInit [-f or -r] -U dbusername -P dbpassword

  SilverMasterInit [-f or -r] -p c:\Program Files\Novell\exteNdn\AppServer\Resources\httpd.newprops -U dbusername -P dbpassword

  SilverMasterInit -r -U dbusername -P dbpassword

  SilverMasterInit -f -U dbusername -P dbpassword -A adminusername -W adminpassword

  SilverMasterInit [-f or -r] -U dbusername -P dbpassword

  SilverMasterInit [-f or -r] -v -U dbusername -P dbpassword

  SilverMasterInit -x

Using SilverMasterInit to recreate or refresh SilverMaster

The SilverMaster database, which is created during installation, can also be recreated or updated using SilverMasterInit. The SilverMaster database keeps track of all deployment databases used by the application server and also holds the application server's system tables, including those containing group, user, and licensing information. There is one SilverMaster catalog for each server or cluster.

    For more information about the SilverMaster database, see Configuring deployment databases.

If your SilverMaster database is damaged, you can run SilverMasterInit. If you cannot start the application server, try one of the following procedures if nothing else has worked:

• Refreshing the SilverMaster database with SilverMasterInit

• Creating a new SilverMaster database with SilverMasterInit

CAUTION:    Connection problems may be due to a corrupt driver connection, damaged deployment database, or network problems. If you have questions about what is causing your server problem, call Novell exteNd Technical Support before running SilverMasterInit. Running SilverMasterInit in Full mode will delete the contents of all your existing system tables and replace them with initialized data. Do not run in Full mode if you want to preserve existing system tables, including those that contain deployed J2EE archives, group, and user data.

Refreshing the SilverMaster database with SilverMasterInit

You can run SilverMasterInit in Refresh mode to upgrade or access SilverMaster properties. The refresh process skips some of the database installation steps used by Full mode. Run SilverMasterInit in Refresh mode to refresh SilverMaster system data and resources without deleting existing users, groups, and deployed J2EE archives.

NOTE:   As part of the application server installation process, SilverMasterInit upgrades resources. You typically upgrade the application server by running the installation program.

To run SilverMasterInit in Refresh mode:

1. Shut down the application server.

2. From server's \bin directory, enter:

     SilverMasterInit -r options
   

   You see the message Creating Resources will take a few minutes; please wait.

3. When SilverMasterInit completes without errors, restart the application server.

Creating a new SilverMaster database with SilverMasterInit

SilverMasterInit can often fix problems caused by someone removing or renaming a file or table that SilverMaster relies on. If you cannot start the application server or connect to the SilverMaster database, you may need to run SilverMasterInit.

While SilverMasterInit can reset corrupted SilverMaster properties, this program cannot repair a corrupted Registry key, configuration files, sample databases, or files associated with databases. To address these types of problems, run the installation program.

To avoid deleting all your database tables, try running SilverMasterInit in Refresh mode (before running it in Full mode) to see if that resolves the server problem.

If you run SilverMasterInit in Full mode to regenerate new SilverMaster properties, you will have to recreate any Silver Security users and groups, re-add any deployment databases, and redeploy any J2EE archives deployed to SilverMaster.

To run SilverMasterInit in Full mode:

1. Shut down the application server.

2. From the server's \bin directory, enter:

     SilverMasterInit -f options
   

   You see the message Creating Resources will take a few minutes; please wait.

3. Record any errors from this command.

4. Start the application server.

5. Add any deployment databases again.

6. Recreate any users and groups.

7. Redeploy any J2EE archives that were deployed to the SilverMaster.

NOTE:   If you do not want to lock down access to data and require user authentication, you can run the -n option in Full mode.

Regaining access to SilverMaster

You can use SilverMasterInit to regain access to locked resources. The SilverMaster database is where the application server stores all system resources and links to other databases. By default, any user with the Locksmith privilege has Read access permission to the SilverMaster. If all users are accidentally denied access to the SilverMaster database, no one will be able to access the application server through the SMC.

See the following sections if you suspect that Read access to SilverMaster has been restricted:

• Using the Locksmith option to access locked resources

• Using server authentication to access the login resource

Using the Locksmith option to access locked resources

By default, the administrator and any other user with Locksmith privilege can get and set data access permissions for any resource in any database, read all SilverMaster resources, and grant Locksmith privilege to users and groups. The only user able to grant the Locksmith privilege is someone who is already a Locksmith. If all accounts with the Locksmith privilege get deleted, use the SilverMasterInit Locksmith option to grant this privilege to a user to regain access to resources.

NOTE:   By default, after a new installation or after you run SilverMasterInit in Full mode, an administrator account is automatically created that has Locksmith privilege.

Once a user with the Locksmith privilege can access SilverMaster, that user can unlock resources and reset access privileges.

    For more information, see Using the Locksmith privilege.

To reset Locksmith privilege:

1. Shut down the application server.

2. From server's \bin directory, enter:

     SilverMasterInit -l -U dbusername -P dbpassword
   

   The Locksmith can now use the SMC to unlock resources.

Using server authentication to access the login resource

You can set server authentication from either the SMC or the SilverMasterInit command line. You should set server authentication if you accidentally restrict Read access to your SilverMaster database. If users cannot access SilverMaster, run the SilverMasterInit server authentication option to allow users to authenticate themselves when they initially connect to the server. When a user logs in to the application server from the SMC, a request for the login resource is issued. Users cannot access the login dialog if their access to SilverMaster is restricted, because they have no Read access to the database. This is because the SilverMaster has the login resource.

You do not need to specify Full or Refresh mode when you run the server authentication option. When you restart the server after setting server authentication, your first attempt to access the server will bring up the credentials dialog and you can log in.

To set server authentication:

1. Shut down the application server.

2. From the server's \bin directory, enter:

     SilverMasterInit -a -U dbusername -P dbpassword
   

3. Restart the application server.

   Users will now be prompted to log in.

Handling a stack overflow

In some obscure situations it is possible to exceed the limits of the stack, in which case the Java Virtual Machine (JVM) throws a java.lang.StackOverflowError.

About stacks

On a Windows system in the Java environment, there are at least two program stacks (and possibly more, depending on the JVM implementation), any one of which can overflow and cause a StackOverflowError to be thrown:

• There is always a hardware stack, which is used by native code in the JVM itself and by native code that is compiled from Java bytecodes by the Just In Time (JIT) compiler.

• There is always a Java bytecode stack, which is used for temporary storage of method call arguments and local variables for Java methods. This is a soft stack that is created and managed by the JVM.

Each thread created in the JVM has its own hardware and Java stacks.

What to do if you get a stack overflow

It is possible to alter the size of each of the stacks if it is determined that the default stack size is too small. However, the most common cause of stack overflow errors is a programming error where a method is called recursively a number of times. If this is the case, increasing the size of the stack will not fix the stack overflow problem. Before attempting to increase the stack size, verify that the code does not contain any errors of this nature. Assuming the stack overflow is not caused by an infinite recursion error, it should be possible to fix the stack overflow by increasing the stack size. To determine which stack overflowed is largely a matter of trial and error.

The size of the hardware stack is determined by the operating system using a value stored in the header of the executable. The executable (SilverServer.exe) specifies a default stack size of 256K.

Changing the stack size

In order to change the stack size, you must modify the executable header using Microsoft's EDITBIN utility. For example, to change the default stack size for SilverServer.exe to 512K, use the following command line:

  EDITBIN /STACK:0x80000 SilverServer.exe

TIP:   Of course, you should make a backup before modifying any executable.

Changing the Java stack size

If increasing the size of the hardware stack doesn't work, it is possible that the Java stack is the problem. In the JDK documentation, there are two command-line options affecting the stack size:

• -ss sets the maximum native stack size

• -oss sets the maximum Java stack size

The defaults were 128K and 400K respectively. Although these options are no longer documented in JDK 1.2 (Java 2), they appear to have been carried forward as nonstandard (-X) switches. To set these options for an application server executable, use +X instead of -X (the application server executables interpret + options as options to be passed to the JVM).

    For more information about application server startup options, see Using startup options.

Example

For example, to set both the native and Java stacks for the application server to a maximum of 512K, use the following command line:

  SilverServer +Xss512k +Xoss512k

NOTE:   Increasing any of the default stack size values will increase the amount of virtual memory allocated per thread. Virtual memory is a finite resource, albeit a large one (in a 32-bit operating system such as Windows NT, processes can address up to 2G of virtual memory). Increasing the per-thread virtual memory requirement will reduce the number of threads that can be created. It is important to realize that this could reduce the number of simultaneously connected users that the server is able to support (since the server uses one thread per connected client).

Miscellaneous issues

This section describes some uncategorized issues you may need to address.

Server appears to be hung

If the application server seems to be hung or in a loop, you can generate a listing for each thread with a stack trace. Doing this does not stop the server.

In NetWare   Make sure the Logger buffer size is set large enough to handle the thread stack information. (You'll want to set the Recall and Layout to large numbers.)

Determine the JVM ID of the application server process:

  java -show

To generate the stack trace:

  java -showstacksXXXX

where XXXX is the JVM ID of the application server process.

To see the stack trace, you can:

• Use the Logger screen

• Use the GUI version of the Console Log (where you can save to a file)

• Go to http://hostname:8008 on a browser and use the Remote Manager to view the logs

In UNIX   Determine the process that the application server is running under:

  ps -all | grep Silver

Issue the following command:

  kill -3 SilverServer_process_ID

The application server lists each thread with a stack trace in the window where the server was started from.

In Windows   In the window where you started the server, press Ctrl+Break. The application server lists each thread with a stack trace.

Socket exceptions

You may receive a Socket Exception message in your NT application log. Typically, this is not a problem: it usually indicates that a client has unilaterally closed a socket. Browsers such as Internet Explorer frequently do this when the connection has been idle for a while, and it will show up as a Socket Exception in the server's console when running with debugging.

You can usually ignore such warnings; they simply reflect a normal situation.

Copyright © 2004 Novell, Inc. All rights reserved. Copyright © 1997, 1998, 1999, 2000, 2001, 2002, 2003 SilverStream Software, LLC. All rights reserved.  more ...