Troubleshooting data migrations with migfiles from NetWare 6.5 to OES Linux

  • 7001767
  • 30-Oct-2008
  • 30-Jul-2014

Environment

Novell Open Enterprise Server 11 (OES 11) Linux
Novell Open Enterprise Server 2 (OES 2) Linux

Situation

NOTE: The content of this TID was originally written for OES 2, but applies to OES 11. Use zypper with OES11 instead of rug.

Migfiles is a command line utility used to migrate data in a variety of scenarios.  The focus of this document will be on data migrations in a single tree, from NetWare 6.5 NSS to OES 2 Linux NSS.

Note:
Since the initial release of this TID, migfiles has received a GUI interface.  Additional functionality has also been added.  The GUI wrapper for migfiles, and many other CLI tools related to migration, is miggui.  Miggui is available with OES2 SP1.   

Please see the miggui documentation:

https://www.novell.com/documentation/oes2/mig_tools_lx/index.html?page=/documentation/oes2/mig_tools_lx/data/bookinfo.html#bookinfo

Because miggui is a GUI wrapper around CLI tools, including migfiles, these troubleshooting steps listed below  apply for troubleshooting both data migration issues with the stand alone migfiles CLI and miggui.

The migfiles CLI uses nbackup and ncpmount as the principle tools to migrate data.  Migfiles is a ruby script that can be opened, as plain text, in any text editor for review.

Failed migrations are often related to configuration problems on the source server, the target server, or both servers.  This document will cover variables that should be considered on both the source NetWare NSS server as well as the target OES 2 Linux server.  Many of these steps also apply to tree to tree data migrations.

Source NetWare server:

The NetWare 6.5 server patch level should be current.  Any post support pack SMS updates should be installed.  Please do not skip this step.  A current code base is very important for successful migrations.

Confirm the SMDR.NLM  and TSAFS.NLM are loaded and current by typing these commands at the server console:

m smdr.nlm
m tsafs.nlm

Compare the dates with post support pack SMS patch dates on Novell's support site.

Smdr Configuration File

The SYS:\etc\sms\smdr.cfg file should have these entries:

SLP: enable
HOSTS: enable
IP: <Correct NetWare server IP address where SMDR will listen>

Hosts File

The SYS:\etc\hosts file should have entries for both the NetWare source server and Linux target server.

Syntax:

<SourceNetWareserverIPaddress><Sourceservername.serverdomain.com><Sourceservername>
<DestinationLinuxserverIPaddress><Destinationservername.serverdomain.com><Destinationservername>

Example:

151.155.XXX.XX3 Netwareserver.novell.com Netwareserver
151.155.XXX.XX2 Linuxserver.novell.com Linuxserver

Note:  Use the same server name case, as used in the hosts file, while authenticating with miggui.

Smdr Communication Port

SMDR listens for communication on port 40193.  Confirm the NetWare server is listening on this port.
Load TCPCON.NLM | Protocol Information | TCP | TCP Connections (Select to View or Modify)
There should be an entry showing the NetWare local host with port 40193.
Make sure SMDR.NLM is loaded if this entry does not exist.

Source NSS Volume / Name Space

Type from the NetWare console: volumes [ENTER]
Confirm the volumes involved in the migration have these name spaces:  DOS, MAC, NFS, LONG

The nbackup portion of the migration will fail with 0xfffdffdc, and 0xfffdffe5 if the NFS name space does not exist on the NetWare traditional source volumes.
Type the following at the NetWare server console to add NFS name space to the source NetWare Volume:
NFS [ENTER]  This will load NFS.NAM if it is not already loaded.

add name space NFS to volume <VOLUME NAME> [ENTER]

Dismount and remount the volume.

Type: volumes [ENTER] to confirm from the output that the name space added correctly and that the volume is mounted.


Target Linux/OES 2 server:

The Linux/OES 2 target server patch level should be current.  Please do not skip this step.  A current code base is very important for successful migrations.

Patching OES 2

Follow the steps to patch the server, if required.
Select "Updating an OES2 Linux Server" from this Link below:
https://www.novell.com/documentation/oes2/inst_oes_lx/index.html?page=/documentation/oes2/inst_oes_lx/data/front.html#front

Common patch issues:

Run this command:
# rug ca
Sub'd? | Name | Service
---------+-------------------------------------+------------------------------------
Yes | SUSE Linux Enterprise Server 10 SP1        | SUSE Linux Enterprise Server 10 SP1
Yes | Novell Open Enterprise Server 2                  | Novell Open Enterprise Server 2
Yes | SLES10-SP2-Updates                                  | https://nu.novell.com
        | SLE10-SP1-Debuginfo-Updates                  | https://nu.novell.com
Yes | OES2-SP1-Updates                                      | https://nu.novell.com

Confirm correct access to both the SLES and OES channels.  Note the items listed above in bold.
The patch versions shown in the rug ca output were current at the time this TID was authored.  However, at any point the patch versions may be superseded by the current released revision in the patch channel.
Please visit this site to determine the current patch release version:  https://download.novell.com/patch/psdb/

rug lu will not show any further installable migration related patches if the channels show correctly, and all available patches have been installed.

Please follow TID #3150078 if the OES2-Updates channel is not listed correctly.

Continue with the Novell OES 2 documentation to install all the channel updates once outstanding issues have been resolved.

Here is the rug CLI method to install all available OES and SLES updates:
Note:  This will likely require a server reboot.  To install all the updates without saying "yes" manually to license agreements, do the following:

rug up -t patch -y --agree-to-third-party-licenses oes2-sp1-updates sles10-sp2-updates

Contact the Novell ZEN support team for technical assistance if this TID is not resolving the problem.

Smdrd and Tsafs Load

Confirm smdrd and tsafs are loaded by typing:

rcnovell-smdrd status
This command should show the smdrd is running.

Confirm the process state is good:
 
ps aux | grep smdrd

A few bad state examples:
"X" - dead
"Z" - defunct
See "PROCESS STATE CODES" in the ps man page for further details.

Confirm tsafs is loaded by doing the following:

/opt/novell/sms/bin/smsconfig -t

The output should show tsafs is loaded.

Make sure it does not show --tsaMode=netware as this can cause 0xfffdffdc errors.  Reload tsafs without netware mode:

/opt/novell/sms/bin/smsconfig -u tsafs
/opt/novell/sms/bin/smsconfig -l tsafs

SMDRD Configuration File

The /etc/opt/novell/sms/smdrd.conf file should have these entries:

hosts: enable
slp: enable
ip: <Correct Linux server IP address where smdrd will listen>
autoload: tsafs
priority: slp, hosts
uds: enable
legacyconnections: enable
dataencryption: optional

Validate the SMS RPM

Output from this command does not necessarily catch problems.  It's a tool to help determine if the RPM is valid and which configuration files have been altered.

rpm -V novell-sms

Review output from this command to determine if any configuration files need to be examined more closely for possible syntax problems.
The values from the column to the left, if any, represent these things:
    
       S file Size differs
       M Mode differs (includes permissions and file type)
       5 MD5 sum differs
       D Device major/minor number mismatch
       L readLink(2) path mismatch
       U User ownership differs
       G Group ownership differs
       T mTime differs

Hosts File

The /etc/hosts file should have entries for both the Linux target server and the NetWare source server.

Syntax:

<DestinationLinuxserverIPaddress><Destinationservername.serverdomain.com><Destinationservername>
<SourceNetWareserverIPaddress><Sourceservername.serverdomain.com><Sourceservername>


Example:

151.155.XXX.XX2 Linuxserver.novell.com Linuxserver
151.155.XXX.XX3 Netwareserver.novell.com Netwareserver

Smdrd Communication Port

The smdrd listens for communication by default on port 40193.  Confirm the daemon is listening with the following command:

netstat -planet | grep 40193

If smdrd is listening, this command should return a line similar to this example (IP address, inode, user, and PID values will vary from system to system):

tcp          0          0<ipaddress>:40193          0.0.0.0:*          LISTEN          0          <inode #>         <pid#>/smdrd

Run this command to see netstat column header descriptions:

netstat -planet | less 

Correct the IP: <ip address> entry in the smdrd.conf file if the incorrect address is seen in the netstat output.

If the netstat command does not show the smdrd listening on port 40193, start the smdrd with this command:

rcnovell-smdrd start

Debugging SMDRD Load

If the output is still not showing the smdrd listening on port 40193, the smdrd load process needs to be debugged.  The strace utility is a good place to start.  This command below typically returns enough data to start the debugging process: 

strace -f -s 1000 -o output.txt rcnovell-smdrd start
Review the data logged in the output.txt file.

Entries in the smdrd.conf should also be examined  to confirm correct syntax.

If smdrd loads only when tsafs is not auto loaded (option in smdrd.conf),  confirm that the syntax in the /etc/opt/novell/ncpserv.conf is correct.  An incorrect volume entry in this file will cause smdrd to segfault when loading (seen in strace output).  The ncpserv.conf file should also not contain any NSS volume entries.

If smdrd loads and ./smsconfig -t shows the tsafs is loaded, but tsatest still reports the tsafs is not loaded, confirm the smdrd.conf file has the correct IP address entry. Putting in the correct IP entry resolves this issue.

Determine if SMS is Communicating Between the Source and Destination Servers

tsatest can be used to determine if you are able to connect to the smdr on the NetWare server.
cd /opt/novell/sms/bin
./tsatest --server <IP address of server to test connection> --volume <volume to test>:
Enter <user>.<context> when prompted for the user name.
Enter the password when prompted.
Statistics are logged to the counters on the screen when the connection is established and the test backup executes.

tsafs may return the error "Could not communicate with SMDR" upon loading if the ncpserv.conf file has an invalid NCP volume entry.  Correct any invalid entries in this file.  As previously reported in this document, this file should also not have NSS volume entries.

Authentication / eDir / LUM

The namcd daemon should be running.
Enter this command from a terminal to confirm the daemon status:

rcnamcd status

Confirm the process state is good:
 
ps aux | grep namcd

A few bad state examples:
"X" - dead
"Z" - defunct
See "PROCESS STATE CODES" in the ps man page for further details.


Edir should be running and healthy.
Enter this command to confirm the eDir daemon is running:

rcndsd status

Confirm the process state is good:
 
ps aux | grep ndsd

A few bad state examples:
"X" - dead
"Z" - defunct
See "PROCESS STATE CODES" in the ps man page for further details.

Run ndsrepair to confirm the health of eDirectory.

The account used for authentication during the migration should be LUM enabled. 
Enter this command from a terminal to determine if the user is LUM enabled:

id <account name used for migration>
(ie. id admin)
This command should return the uid, gid, and groups for the user if it is LUM enabled.

The UNIX Workstation - <server name> object should exist in the tree.
Open the UNIX Workstation object in iManager.
Select Groups under the Linux Profile tab.
Confirm the account used for authentication belongs to a group listed under "Groups with access to LUM-enabled Services."

NSS Load

Enter this command to confirm NSS is running:

rcnovell-nss status

Confirm the process state is good:
 
nsscon should show the Pool and Volume are active (In nsscon type:  nss pools and nss volumes)

-or-

Type mount from a terminal and verify the target volume is mounted.

Additional Information:

Documentation

Migfiles, nbackup, and ncpmount all have helpful man pages.

man migfiles
man nbackup
man ncpmount

Logs

Migfiles logs can be found at the default location:  /var/opt/novell/log/migration

Smsconfig

Smsconfig is used to load or verify the load status of tsafs.

cd
/opt/novell/sms/bin/
./smsconfig -t   (To confirm if tsafs is loaded and in which mode)
./smsconfig -u tsafs  (To unload tsafs)
Example tsafs load with NetWare mode:
./smsconfig -l tsafs --tsamode=netware

Migfiles Example

Example load of migfiles for data migration:

Note:  This example is for migfiles in OES2 shipping (without OES2 SP1). 

migfiles -ns <source ip address> -iv <source volume>: -x /media/nss/<nss volume name>/
Type man migfiles from a Linux terminal for other available switches and examples.

-n for NSS volume
-s for source server
-i  for verbose
-v for source path (My test migrations only worked when I supplied the colon on the source path)
-x for destination path

Here is an example from OES2 SP1.  The syntax is slightly different from migfiles in the OES2 first ship version.

migfiles -s <source ip address> -iV <source volume>: -x /media/nss/<nss volume name>/

-s for source server
-i  for verbose
-V for source path (My test migrations only worked when I supplied the colon on the source path)
-x for destination path

Tsafs Debug Mode

1.  cd to /opt/novell/sms/bin/
2.  Unload TSA with the following command:
      ./smsconfig -u tsafs
3.  Reload the TSA with the following command:
      ./ smsconfig -l tsafs --smsdebug fffffffc --smsdebug2 fffffffc
4.  Run any test that causes the failure that you want the debug log on.
5.  Unload the TSA with the following command:
      ./smsconfig -u tsafs
6.  Collect or view the log at /var/opt/novell/log/sms/tsafs_debug_xxxx.log (the xxxx represents some number)
7.  Reload the TSA with the following command:
      smsconfig -l tsafs


Smdrd Debug Mode

Smdrd can be loaded in debug mode on Linux:

Stop the smdrd daemon:
rcnovell-smdrd stop

Restart the smdrd with debug by doing the following:
/opt/novell/sms/bin/smdrd --smsdebug fffffffc --smsdebug2 fffffffc

The debug data can be seen in standard out and is also written to /var/opt/novell/log/sms/smdrd_debug_xxxx.log (xxxx is the process id).

CTRL C to end smdrd process when the applicable debug information has been gathered.

Migfiles Debug

Migfiles has a debug switch to increase debug log output.  Add --debug to the migfiles command line to turn on the additional debug information.

A ruby debugger can be used to view real time script execution, or to set breakpoints for debugging.
Do the following:

Go to:
http://rubyforge.org

Download:
ruby-debug 0.10.1.gem
ruby-debug-base-0.10.1.gem
columnize-0.1.gem
linecache-0.42.gem

gem install the columnize and linecache before the ruby-debug-base and ruby-debug

Launch the debugger with the following command:

rdebug migfiles --  <migfiles switches>

Example:

rdebug migfiles -- -ns <Source IP address> -iv <Source NSS VOLUME:> -x /media/nss/<Target NSS Volume>

Type help once you are in the debugger to see how to step through or over functions and set break points.

Nbackup

Using /opt/novell/sms/bin/nbackup manually from the command line may be helpful as a workaround to authentication failures, or to validate SMS components are functioning and communicating correctly.

Nbackup does backup trustees.

Backup example of a NetWare volume executed on the Linux server:
Note:  The < and > characters below denote where a value is entered.  They are not part of the actual command line.

Example backup

cd /opt/novell/sms/bin
./nbackup -cvf /media/nss/<VOLUMENAME on Linux server where you want to create the backup file>/<backup file name>.sidf -R<remote NetWare server's IP address> -U <admin user name.context> --target-type=netware --expand-compressed-data<VOLUMENAME you want to backup:>

Note:  If data is compressed on the source but compression is not enabled on the target volume, use this switch when doing the initial backup:  --expand-compressed-data

If all goes well, this should create a file.sidf on the Linux server in /media/nss/<VOLUMENAME>.  This file is similar in concept to a tar file.  It holds all the backup data.

Make sure the target volume will be large enough to hold the backup .sidf file as well as a restore of all the data it holds.
The .sidf file can be deleted to save disk space once the restore is complete.

Example restore

Restore the data from the file.sidf with this example command:

./nbackup -xvf /media/nss/<VOLUMENAME with backup file>/<backup file name>.sidf -r "<VOLUMENAME that had been backed up on source server:> (NOTE: there is a space here) /media/nss/<TARGET VOLUME WHERE YOU WANT TO RESTORE>" -U root

When the language of the workstation differs from the language of the Target Server
 
migfiles also uses ssh which requires authentication.  Make sure that /etc/ssh/sshd does not include LANG as one of the environment variables in the AcceptEnv line of this file.  If it does and the languages don't match the authentication will likely fail, which will prevent migfiles from running successfully.  If you change this file, be sure to restart sshd (rcsshd restart).