Novell Home

AppNote: Moving User Home Directories, the sed Way

Novell Cool Solutions: AppNote
By Jim Pye

Digg This - Slashdot This

Posted: 31 Aug 2005
 

"Very cool use of sed - modifying LDIF files is a perfect use for this tool, and something that any administrator who wishes to automate mass changes should get to know. Thumbs up!

Jim Henderson–
eDirectory Product Specialist
Novell, Inc.

Notes from the front line ...

Here are some additional notes from the front line after performing the directory migration on another customer's site.

Restoring Trustee Rights

Implementing a new cluster, we moved the data by backing up the original server and then restoring the data to the cluster resource, as opposed the doing a file copy. It was discovered that moving the data this way the trustee rights did not start to appear until after the files had been restored.

sed on Linux and the i\ command

I ran the sed script on a SUSE Linux Professional 9.2 machine and discovered that the syntax for the insert text command (i\) needed to be adjusted.

To run the script, use the following:

s/changetype\:\ add/changetype\:\ modify/ig
/ndsHomeDirectory/i\
replace: ndsHomeDirectory
s/DSR\-NW01_DATA,ou\=BNE/DSR\-CLUSTER1_USER,ou\=HQ/ig
s/USERS$/HOMES/ig

On the Linux machine, the script will need to look like this:

s/changetype\:\ add/changetype\:\ modify/ig
/ndsHomeDirectory/i\replace: ndsHomeDirectory
s/DSR\-NW01_DATA,ou\=BNE/DSR\-CLUSTER1_USER,ou\=HQ/ig
s/USERS$/HOMES/ig

Basically, the text to be inserted into the file - "replace: ndsHomeDirectory" - needs to be on the same line as the i\ command itself. The issue seems to be with the script and file being manipulated between Windows and Linux. The end of line character, which is different on these 2 OSs, was getting in the way.

Complexity of the Procedure

This article was written to show some commands and procedures that may be useful for other purposes. These commands and procedures could be done more effiently. One of my reasons for the complexity was to show what can be done, and as I said in the article there is more than one way to do it. For example, the following one-line command:

{ while read i;do grep -i $i server1.txt;done } < server2.txt

scans one file, looking for its contents in the other. It can be done with just switches on the grep command itself:

grep -i -f file1.txt file2.txt

but this is not nearly as much fun :-)


Editor's Note: In addition to the technical content, this AppNote has a generous sprinkling of humor. As you'll come to see, Pye R not square ...

Introduction

Let me start off by explaining the task at hand. With a new installation of a three-node NetWare 6.5 cluster with 1 TB of disk storage, we needed to move the user home directories from two older - and showing their age a little - NetWare servers. One was running NetWare 6.0 the other NetWare 5.1. The challenge was to do this for approximately 550 users.

My Real-World Scenario

All servers and the cluster are in the same eDirectory tree. The drive mapping to a user's home directory is via the login script variable %HOME_DIRECTORY. That means all users, when created, have their user object's HomeDirectory attribute populated. Templates were not used here - if they were, template definitions would need to be updated at task completion. User objects are spread through several different containers in the eDirectory tree.

The Sample Scenario

To protect the identity of the innocent, I have taken screen shots and other details for this article from a demonstration system. The company used in this demonstration system - DSR (The Department of Straight Roads) - is fictitious and is not intended to represent anyone now, in the past, or in the future. Also, even though I do not play guitar, I enjoy listening to blues guitarists. That's the reason for the selection of names for users working at DSR.

Users can be found in both OU=BNE.O=DSR and OU=SYD.O=DSR containers. The paths to the users home directories are DSR-NW01/DATA:USERS and DSR-NW02/USER:HOME respectively. The home directories, once moved, will be accessed through the cluster resource DSR-CLUSTER1/USER:HOMES.

This AppNote, with a little adaptation, can be applied to all versions of NetWare, OES, eDirectory (with LDAP support) - and not just to moving home directories. As all things should be in the computer world - there is more than one way to do it.

Tools Used

  • ConsoleOne
  • TRUSTBAR.NLM (fullback)
  • iManager
  • A backup solution that has a file copy function (ARCserve in this case)
  • sed, really the heart of this solution

Procedure Steps

Let me lay out the major steps in this procedure:

1. Get all the tools together. Don't let the list scare you. The majority of the tools come, and are installed, with NetWare/OES. The two main exceptions are sed and the backup solution.

I would like to also state here that these tools are not the only way of doing this task. I looked at several other tools that are on Cool Solutions, but did not feel that they gave me the flexibility that I needed for this particular task. This article might also give you a chance to get an insight to tools that can be used for other administration tasks.

2. Use ConsoleOne and its chilly partner - the ICE tool - to "suck" out of eDirectory the user objects, and in particular their current HomeDirectory attribute. With a little adaptation, this whole procedure could be used to change "en masse" any eDirectory attribute.

3. Use TRUSTBAR.NLM to extract file system trustee rights of the source server volumes.

This is a fallback approach (and when talking about clusters we all like to hear the term "fallback") just in case user trustee rights are not correctly copied between source and destination servers by the backup software.

4. Use the backup solution to copy files from the source servers to the cluster resource destination.

5. Use sed to manipulate the extracted eDirectory data.

6. Use ConsoleOne to import the manipulated data.

7. Use iManager to reapply volume space restrictions en masse if required.

8. Test the results.

If I still have your interest, let's get down to the "nitty gritty" of the solution.

Introduction to Tools

sed

"sed?, Who is sed?, What is sed?, When was sed?" You ask. "Ahhh", I reply, "FMTKYTFO (for me to know, you to find out)" - read on.

The tool you are not likely to have on hand is sed. I have installed sed on my Windows system not just for this task, but also because I use it all the time for other text manipulation tasks. Therefore it is well worth the effort to be included in any serious (or even non-serious - do we have any of these?) system administrator's tool belt. Instructions for getting and installing sed are below.

sed is a tool that makes its debut from the dark and dusty realms of UNIX administration and those arcane text manipulating rituals which were the domain of crusty beard-wearing, sandal-toting, gentle folk of days gone by ... DOH! I just looked up into the darkened monitor and I seem to be describing a reflection I see. But I digress.

sed is short for Stream EDitor and is a powerful text manipulator that allows information from a source to be fed into sed. Then sed then works its magic line by line on the input and outputs its findings. Usually this output goes to STDOUT (standard out, the screen). But with a little redirection we can output it to a file, to other commands to do their thing, or even - but hopefully not - to /dev/null (that famous round receptacle of all things messy).

As I said, this tool will make a welcome addition to any administrator's tool collection way beyond this present task. As for the others:

iManager and ConsoleOne

These are of course the current management tools for NetWare, OES, and eDirectory. I will put the usual proviso here: "Make sure these are the latest, and greatest" of their respective species to make sure things go smoothly.

ConsoleOne has two purposes here. The first is to provide a mechanism to export and import LDIF files, out of and into eDirectory. ConsoleOne is also used in this procedure to export a Trusted Root Certificate to allow for a secure SSL connection to the LDAP server. iManager could also be used for these two tasks, but I am more familiar with the workings of ConsoleOne.

iManager is used to apply Volume Space restrictions to multiple users. This functionality seems to be lacking in ConsoleOne, but iManager has a couple of options that make the job easier.

TRUSTBAR

TRUSTBAR is an interesting beast. It's a single NLM, first introduced as an aid to early cluster implementors (NetWare 5.1, etc.) to allow for the easy transition of File System Trustee rights. It name stands for Trustee Backup and Restore. However, it does not use some secret-, shrouded-in-mist type of file format for its output. Instead it uses XML. "Aha," I hear you say, "XML - isn't that a text format?" Now the penny begins to drop. Text file formats and a text manipulation program ... "I see" said the blind man..." (Ah! We won't go further down that track.)

Backup software

"Oooohhaaagh..." Sorry - I just had a creepy sensation run down my back. Not my favorite topic - Backup Software that is, not the need to do backups, you understand.

For this task we need a backup solution that does a file system copy - and hopefully along with that it will copy trustee rights and other NetWarey type file system info, such as ownership. So I use BrightStore ARCserve r11.1 which does all this, although it does not backup volume space restrictions. Or at least it does not copy them, as this is what we are doing here. That is where iManager comes in.

Getting and Installing the tools

sed

"How do we get sed?" I hear from the gentle murmur of the attending crowd. Well if you've jumped on the Linux bandwagon, it's already at your disposal - most *nix OS's come with this tool. If you are a little afraid of this Linux thing and have not taken the whole plunge, then there is an interesting piece of technology out there in the Open Source world called CYGWIN. This technology brings to the Windows world an insight to the wonders of the Unix/Linux shell without actually formatting disks. CYGWIN is a Windows .DLL that provides a Linux emulation on Windows. Effectively, this provides a BASH (Bourne Again SHell) prompt that allows Linux utilities to be run on Windows. (Note that the web site gives some warnings on what GYGWIN is and is not). CYGWIN comes with hundreds of tools and accessories allowing users and administrators to get a feel for, and use, the power of the command prompt way beyond the offerings of that Redmond company.

CYGWIN is licenced under the GNU General Public Licence (GPL), so it comes complete with the source code if you want to download it.

To get CYGWIN you need to access http://cygwin.com/ and proceed to the download of an installer. This installer allows you to choose what bits of CYGWIN and related utilities you want. It will then connect to the CYGWIN site, or mirror, and download those bits.

Once downloaded, the data will be installed. There is an option to keep the downloaded files after installation. It's your choice here, but if you want to install onto more than one machine, keep the files - this will save having to download them again.

For more exact details, check out: http://cygwin.com/cygwin-ug-net/setup-net.html Some of the major features of CYGWIN are, of course the BASH shell, an X server that allows connectivity between X applications on Unix hosts and Windows, and a whole bunch more. The sed utility is installed with the BASE package of CYGWIN so running the installation with the defaults should get the utility installed.

TRUSTBAR

TRUSTBAR is included in later versions on NetWare, but as I mentioned earlier, we have a NetWare 5.1 server with SP8, it did not seem to have the utility included. However all is not lost. The NetWare 6.0 server has a version, and a simple copy of the TRUSTBAR.NLM from SYS:SYSTEM to SYS:SYSTEM deals with this. However, please note that there have been reported sightings of that pesky predator - Mr. A. Bend - with some really early versions of TRUSTBAR. The version I used here was dated April 2004.

Preparation

One of the requirements of this procedure is to log in into the LDAP data source as a user with administrative privileges. To make sure the LDAP Authentication Credentials (a.k.a. the password) of this user are protected during login, an SSL connection to the LDAP data source should be used. This SSL connection requires that the Trusted Root Certificate for the server involved in the process be extracted and placed in a file ready for the ICE tool to use. Note that for performance reasons the server chosen to perform the export and import of LDIF files should have at least a R/W replica of the user objects involved. In the tree where I performed this task we had a DS Master server that provided copies of all partitions in the tree.

To extract the Trusted Root Certificate, follow these steps:

  1. Fire up ConsoleOne.
  2. Browse to the container that holds the server that will be used for the export/import.
  3. Select the IP version of the kmo (key material object) for the server. This will be the object named SSL CertificateIP - <servername>. In the figure below this is SSL CertificateIP - DSR-NW01


  4. Figure 1 - KMO used to generate Trusted Root Certificate


  5. Right-click and open the Properties.
  6. Click the Certificates tab and select the Trusted Root Certificate option.


  7. Figure 2 - SSL Certificate Properties


  8. Click Verify to make sure the certificate is valid. If not, head off to the Novell Knowledgebase and track down some TIDs on repairing broken server certificates.
  9. Once you've validated the certificate, click Export. The following dialog appears:


  10. Figure 3 - Export Private Key Option


  11. Make sure the No option is selected - we do not want the Private Key with this export.
  12. Click Next.
  13. Make sure the file is in the .der format.
  14. Change the name of the file to something a little shorter, without spaces. Some tools that use these files do not like the spaces in the name.
  15. Place the file on the workstation's file system (c:\ in the dialog below):


  16. Figure 4 - Save Certificate Option


  17. Click Next, and confirm the options:


  18. Figure 5 - Export Certificate Summary


  19. Click Finish.

You should now have a .der file sitting in the directory specified. We will use this file in the ConsoleOne's ICE tool during the LDIF export and import operations.

One more preparation task is needed before starting. Create a couple of test users within the source containers and make sure that they have a home directory located on the source servers. You can even put some files in these directories. These users will hopefully be carried along in the rush and can be used to test that all has gone well. We can log in and out using these accounts when finished.

Data Extraction

Now that we have the tools, let's go on to the extraction of the data.

The ConsoleOne management tool has a NDS Import/Export Wizard that is a front end to Novell's ICE tool. The ICE (Import/Convert/Export) tool allows for extraction of data from LDAP sources, conversion of data between LDAP data stores and/or insertion of data into LDAP data stores.

The ICE tool uses the LDIF (Lightweight Directory Interchange Format) format to achieve this functionality. LDIF is a text-file format that contains data and commands that the ICE tool uses to update, or insert new, information into LDAP data stores.

This AppNote will help you extract relevant information from eDirectory into an LDIF file. We will manipulate this information then Import it back into eDirectory by using the steps below.

  1. Open ConsoleOne, if it's not already open.
  2. Open the Wizard menu and select the NDS Import/Export option. The following dialog appears:


  3. Figure 6 - LDIF Export option


  4. Make sure you have the Export LDIF file option checked.
  5. Click Advanced.


  6. Figure 7 - Advanced Options


  7. Check the Run in Verbose Mode option.
  8. Click OK and then click Next.
  9. In the LDAP server dialog, fill out the details relevant to your environment. My demo in Figure 8 shows the details:


  10. Figure 8 - LDAP Server and Authentication Dialog


  11. Select the Trusted Root Certificate we exported earlier.
  12. Select the port the LDAP server is listening on for secure communications, 636 is the protocol's default.
  13. Supply the user login name. The syntax for the user login is in LDAP format, therefore there's no leading dot or comma. It must be a typefull name; cn= ou=, and commas must be used as separators.
  14. Click Advanced.


  15. Figure 9 - Advanced Options for the LDAP Server Dialog


  16. The defaults are good enough here, so click OK and then click Next.

Note that Display Search Operation but Do Not Perform is a handy way to test that all the options are correct. With this option checked, the operation will be performed without modifying any data.

Defining Search Criteria

We are now going to define our search criteria.

  1. Define the base DN/Scope for the search. This can be a sub-container on a large tree to narrow the search, and therefore lessen the cleanup needed to the file produced. My demo here has selected the o=dsr meaning I am searching the whole organization.
  2. Make sure that you check the Sub Tree option to scan from the Base DN down. These options are shown in Figure 10.


  3. Figure 10 - Search Criteria Base DN/Scope Dialog


  4. To define what objects you are interested in, click the Filter tab:


  5. Figure 11 - Search Criteria Filter Dialog


    Note that the ObjectClass is in LDAP terminology - an eDirectory User is equivalent to a LDAP inetOrgPerson.

  6. Click the Attributes tab so you can tell ICE what attributes of the User objects you are interested in.


  7. Figure 12 - Search Criteria Attributes Dialog


  8. Click the Attributes radio button. The List button will become active.
  9. Click List.
  10. Enter the ndsHomeDirectory attribute into the list. This is the LDAP designation for eDirectory's HomeDirectory attribute. Enter the text into the field at the top of the dialog and then click Add.
  11. Once the dialog looks like Figure 13 below, click Close.


  12. Figure 13 - Search Criteria Attributes To Include Dialog


    At the Set Search Criteria dialog, click Next.

  13. Specify the LDIF file the data is going to be sent to.
  14. A suggestion for the name might be something like rawoutput.ldif. This will make sure it is identified as the original data extraction. It is best once this file is generated that you work on copies for the actual manipulation. I'll name my copy of rawoutput.ldif as inputfile.ldif. This will be the input file to the sed tool.


    Figure 14 - Destination LDIF File Dialog


  15. Click Next to see and confirm your settings.
  16. Click Finish to start the extraction.

Figure 15 - Options Summary Dialog


If something goes wrong, the Wizard will display a Restart button. This is fortunate, as I find I always seem to get something wrong on the first run through. The Restart button restarts the wizard, remembering all the settings just entered. You can change the incorrect option and try again. Oh, and don't forget that the user password will need to be re-entered on the correct page in the Wizard.


Figure 16 - LDAP Operation Output Dialog


This should produce the LDIF file as specified in the Wizard. When you open the file, there are a couple of things to note:
  • The line length is truncated to a fixed width. The LDIF syntax defines that a line that starts indented directly below another is considered a continuation of the line above.
  • There is an empty line between each "entry" in the file. The dn: line defines the object we are dealing with. The changetype and the ndsHomeDirectory line, which are directly below and left-justified, define the commands and attributes for that DN.

From my demonstration system here is the raw LDIF exported file.

#This LDIF file was generated by Novell's ICE and the LDIF destination handler.
version: 1

dn: cn=admin,ou=BNE,o=DSR

dn: cn=NFAUUser,ou=BNE,o=DSR

dn: cn=publicUser,ou=Extend,ou=BNE,o=DSR

dn: cn=BBKing,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#USERS

dn: cn=EClapton,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#/USERS

dn: cn=JHealey,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#USERS

dn: cn=JHendrix,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#USERS

dn: cn=PTownsend,ou=SYD,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW02_USER,ou=SYD,o=DSR#0#/HOME

dn: cn=RDaltrey,ou=SYD,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW02_USER,ou=SYD,o=DSR#0#/HOME

dn: cn=BRubble,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#USERS

dn: cn=WFlintstone,ou=SYD,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW02_USER,ou=SYD,o=DSR#0#/HOME

To make this file ready for use as an LDIF import file, with modified home directory attributes, we need to manipulate it with sed and other editing tools. Before that, we will make a record of the Trustee Rights on the the volume.

Using TRUSTBAR to Record User Trustee Rights

The TRUSTBAR utility is an NLM, so it must be run on the server console. Basically, it reads through the volume specified in the command-line options, looking for directories and files that have been assigned trustees. Once it finds one, it records the file/directory, the trustee, and the trustee rights, in an XML format to a specified file.

The same TRUSTBAR tool with different command-line switches can be used to restore trustee rights back to a file system. Because this output file from TRUSTBAR is plain text, the manipulation of directory paths etc. can again be performed by sed and friends.

Note: If you're running backup software, you may need to wait until the TRUSTBAR restore finishes. Otherwise, the backup software might restore the files before their rights are applied.

To back up trustees,

  1. Go to the server console.
  2. Run the following command: TRUSTBAR DATA: sys:data-tr.xml -B -V

The command above will run TRUSTBAR over the DATA: volume. It's important to have the colon in the volume name. It will output the XML file to a file called data-tr.xml on the root of the SYS: volume. The -B option says do a backup of trustee rights; the -V option means to be verbose about it. Note that -V will fill the logger/console screen with a lot of information until it completes.

Because this tool reads all the file and directory entries looking for trusteeship, it can take a while scanning a large volume with lots of files. We'll cover the modification of this output file in a moment, but first let's start the data copy.

Starting the Data Copy

We want to multi-task while the file copy is happening, so here is were we have some fun trying to get our backup software to do what we want. As I mentioned, for this task I chose ARCserve - your mileage may vary on what backup software is available. What we need is to be able to copy files, directories, trustee rights, and NetWare-ish things like ownership, etc.

For ARCserve the job setup goes like this:

  1. Create a new Copy job from the Utilities menu.
  2. For the Source, drill down to the directory that contains the individual home directories (e.g., DATA:users).
  3. For the Destination, drill down to the equivalent directory on the destination server (e.g., USER:homes).
  4. The trick is in the options, so click Options now.
  5. In the dialog that appears, click the Destination tab. The settings should be as shown in Figure 17 below:


  6. Figure 17 - Global Options settings


  7. Click OK and submit the job.

If all is going according to plan, the file system will start to be copied along with trustee rights and other NetWare-ish stuff. I am not an expert when it comes to backup software, so there may be other options to make the copy job faster and more robust, but these seemed to work for me.

A Bit of a Time Saver

Because I was having to migrate two separate servers into one, I decided to set up a second file copy from the second source server to run at the same time. However, things could get a little messy if a user happened to have a home directory on both source servers. Even if they are only using one directory, there might be hangovers from file system cleanups being missed, etc. As it turned out, using the procedure below, I found home directories for two users that existed on both servers.

First off, we need to get a list of the user home directories into a file with each directory entry is on its own line. Here is where BASH (or more specifically the ls command) shines. Below are the steps to follow - note that the following commands are now case- sensitive.

  1. On the Windows machine where you have installed CYGWIN, map a drive to the home directory volumes on the servers in question. This example uses y and z drives.
  2. In CYGWIN, at the prompt, type: pwd. This is the "print working directory" command. The response, if you have just started CYGWIN, should be: /cygdrive/c
  3. Change directory to one of the mapped drives, by typing: cd ../z If that does not, work use this: cd /cygdrive/z
  4. Replace the z with the drive letter that you assigned to the mapped volume. When you do a directory listing (ls) you'll see the contents of the root of the drive that was mapped. Cool so far?
  5. Change directory to move above all the individual home directories by typing: cd HOME
    Note that we are now in Linux world, so case sensitivity is in effect:
  6. When you do an ls command here, you'll see the user's home directories. Now for the power of open source software. ls (the directory listing command) has about a million switches. The one we need here, of which I could not find a DOS dir command equivalent, is -1 (minus one). What this tells ls to do is send its output as one entry per line.

  7. To output the home directories in a format you can use, type this: ls -1
  8. Now redirect the output to a file: ls -1 > server1.txt
  9. To make sure you have something in the file, use this command: cat server1.txt
  10. Next, move that file into a safe place on the workstation local disk. Assuming you have a C:\DATA\HOMEDIRMOVE directory, use the command: mv server1.txt /cygdrive/c/DATA/HOMEDIRMOVE/
  11. Change directory to the second drive mapping on the second server (the y drive): cd /cygdrive/y/USERS
  12. Run the ls command again: ls -1 > server2.txt
  13. Move the file into the same directory as the server1.txt: mv server2.txt /cygdrive/c/DATA/HOMEDIRMOVE/
  14. CD into that directory: cd /cygdrive/c/DATA/HOMEDIRMOVE/
  15. Now for the real cooool part - run the following command to see if there are any directories duplicated on both servers:
{ while read i;do grep -i $i server1.txt;done } < server2.txt

If the above is typed in correctly, remembering case and space, the resulting output should list the directories that have entries in both files (on both source servers).

Now for an explanation of what this command does. The curly braces {} mean the contents are treated as one command. The < server2.txt means we are feeding the contents of the server2.txt file into this command. What is happening inside the {}, I hear you ask?

We are using a while loop to read each line of the server2.txt file. For each line of server2.txt, we are running the grep command, ignoring case (the -i), and looking for that line in the server1.txt file.

Now try to do that with one line in DOS!

Very cool.

Data Manipulation

Now that we have our source LDIF files, and the file copy is bubbling away in the background - and a nice warm cuppa (if not, reward yourself for getting this far and go get one) - we can settle down to some serious text file manipulation.

First, make a copy of both your exported LDIF and XML files to work on. If anything messes up, you can come back to the originals and try again.

Made the copies? Good.

The Home Directory Change

For my situation there were two types of entry in the raw data file that did not need to be present in the import file:

  • One was just a dn: line - this is the case when the object does not have a HomeDirectory attribute.
  • The other was an entry that did have an ndsHomeDirectory attribute, but it was not for either of the two source servers. In my case, the portion of the tree I extracted had three servers - we were only interested in 2 of these.

Warning: You must remove the empty data (the dn: entries with no ndsHomeDirectory attribute) or the ICE Import operation will fail. It's a matter of scanning down the file removing the dn: only entries. Hopefully there should not be too many. These entries are usually users created for processes to login in, and administrative users who do not need to have home directories.

As for the second issue - the naming standards used in this organisation made it relatively easy to work through the file deleting the entries not required for the merge. This is something to pass the time during the file copy, and it might pay to look it over and make sure all is well in that department.

Once you have the required entries in the LDIF file, you need to turn this into a LDIF import file.

  1. Make the LDIF file into a modify changetype and specify that you are modifying the HomeDirectory attribute. This is necessary because the default file generated from ICE is for an add operation.
  2. Change the values for the HomeDirectory attribute to the new cluster resource (the destination server).

Spotlight on sed

This is where sed makes its entry from stage right. Ta da!

As mentioned earlier, sed is a stream editor. This means it takes input from a file line by line, works its magic, and outputs that possibly modified line. The power of sed is it can search for text in each line - using a technique called regular expressions - and change that text to something else. So here is the magician's magic revealed.

In the same directory as the LDIF file you need to make another text file. A good name for this file would be along the lines of ldifchangescript. Ahh, isn't it nice to have long file names ...

One big boon with the BASH shell is the auto command completion function that has only recently been seen on Windows command lines. Command completion is where you enter a command and press the TAB key. BASH will try to complete the line for you. If there are multiple options it will beep, and if you press TAB again BASH will display all the options. Type more of the command until it is unique enough so that when you press TAB again, BASH can finish the line. This works on both the command executable part and on the file to be used as a parameter to the command.

ldifchangescript File

Into this ldifchangescript file you'll place the commands for sed to perform, one per line. Type in the following, again obeying case and space, I will explain after.

s/changetype\:\ add/changetype\:\ modify/ig
/ndsHomeDirectory/i\
replace: ndsHomeDirectory
s/DSR\-NW01_DATA,ou\=BNE/DSR\-CLUSTER1_USER,ou\=HQ/ig
s/USERS$/HOMES/ig
s/DSR\-NW02_USER,ou\=SYD/DSR\-CLUSTER1_USER,ou\=HQ/ig
s/HOME$/HOMES/ig

Now for the explanation.

The s/xx/yy/zz lines are the substitution command. It forces sed to substitute whatever is between the first // (the xx in this example), with what is between the second // (yy). The zz portion is the options to use in the substitution. For the script, "i" means to ignore case and "g" means do it globally (for each instance). So for the script above, the first line is replacing the "changetype: add" text with the text "changetype: modify".

Note that the slashes ("\") are required as "escape" characters in the text, because these have special meaning within the regular expression world. An example of where the \ slash must be used is before the "-". In a regular expression this is a sequence character (for example, a-z means all lowercase characters, A-Z means all upper case, etc.). As our example has a dash in the path, to replace it you must "escape" it so sed does not try to interpret it as a special character. If sed did interpret the "-" in this case, it would see "R-N". That's an illegal sequence, and sed would generate an error - not what we want sed to do in any case.

The second and third lines of the script are inserting a new line into the file. The second line of the script tells sed to insert (the i\) the third line of the script before the "ndsHomeDirectory" line in the input text.

The last four lines substitute the volume name and path. On line 5 and 7 there is a "$" in the text to be replaced. This indicates to sed that we are only interested in the text if it occurs at the end of the line. If you want to find text that's only at the beginning of a line, you could use a "^" before the text. These last four lines, of course, are where you need to change the script to match your source and destination requirements.

It is important to realize that sed only reads each line of the source file once. It compares this line with the commands in the script file to see if there is a match. If so, it operates on that line. When sed makes a change to a line, the new text it adds is not looked at for further changes, unless you tell sed to do so.

Once you have the script file completed and you have read over it a couple of times - and an innocent bystander has helped you check the syntax - you can save it and exit the editor.

Running sed

Now comes an exciting bit, hopefully one of many during this exercise - you get to run sed and watch the smoke clear :-)

In your BASH session in CYGWIN, type the command shown below. Make sure you are in the directory where the script and the LDIF file are located.

sed -f ldifchangescript inputfile.ldif > outputfile.ldif

Note that if you do not redirect the output to a file (using the ">"), the output will go to the screen.

If you now open outputfile.ldif, you should see an LDIF command file that has the correct commands to make all the users Home Directories the same. Below is what the input file and output file from my demonstration system look like after sed has had a look inside inputfile.ldif. When comparing this file to the raw data file, note that the empty entries have been stripped out, etc.

inputfile.ldif

#This LDIF file was generated by Novell's ICE and the LDIF destination handler.
version: 1

dn: cn=BBKing,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#USERS

dn: cn=EClapton,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#/USERS

dn: cn=JHealey,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#USERS

dn: cn=JHendrix,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#USERS

dn: cn=PTownsend,ou=SYD,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW02_USER,ou=SYD,o=DSR#0#/HOME

dn: cn=RDaltrey,ou=SYD,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW02_USER,ou=SYD,o=DSR#0#/HOME

dn: cn=BRubble,ou=BNE,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW01_DATA,ou=BNE,o=DSR#0#USERS

dn: cn=WFlintstone,ou=SYD,o=DSR
changetype: add
ndsHomeDirectory: cn=DSR-NW02_USER,ou=SYD,o=DSR#0#/HOME

outputfile.ldif

#This LDIF file was generated by Novell's ICE and the LDIF destination handler.
version: 1

dn: cn=BBKing,ou=BNE,o=DSR
changetype: modify
replace: ndsHomeDirectory
ndsHomeDirectory: cn=DSR-CLUSTER1_USER,ou=HQ,o=DSR#0#HOMES

dn: cn=EClapton,ou=BNE,o=DSR
changetype: modify
replace: ndsHomeDirectory
ndsHomeDirectory: cn=DSR-CLUSTER1_USER,ou=HQ,o=DSR#0#/HOMES

dn: cn=JHealey,ou=BNE,o=DSR
changetype: modify
replace: ndsHomeDirectory
ndsHomeDirectory: cn=DSR-CLUSTER1_USER,ou=HQ,o=DSR#0#HOMES

dn: cn=JHendrix,ou=BNE,o=DSR
changetype: modify
replace: ndsHomeDirectory
ndsHomeDirectory: cn=DSR-CLUSTER1_USER,ou=HQ,o=DSR#0#HOMES

dn: cn=PTownsend,ou=SYD,o=DSR
changetype: modify
replace: ndsHomeDirectory
ndsHomeDirectory: cn=DSR-CLUSTER1_USER,ou=HQ,o=DSR#0#/HOMES

dn: cn=RDaltrey,ou=SYD,o=DSR
changetype: modify
replace: ndsHomeDirectory
ndsHomeDirectory: cn=DSR-CLUSTER1_USER,ou=HQ,o=DSR#0#/HOMES

dn: cn=BRubble,ou=BNE,o=DSR
changetype: modify
replace: ndsHomeDirectory
ndsHomeDirectory: cn=DSR-CLUSTER1_USER,ou=HQ,o=DSR#0#HOMES

dn: cn=WFlintstone,ou=SYD,o=DSR
changetype: modify
replace: ndsHomeDirectory
ndsHomeDirectory: cn=DSR-CLUSTER1_USER,ou=HQ,o=DSR#0#/HOMES

Don't worry if your output is messed up at this point. We have not damaged the input file. You can go back to the script file, edit out the errors, and try the command again. The ">" redirection tells BASH to overwrite the outputfile.ldif file, wiping out any embarrassing errors you may have caused on the first try.

All Wrapped Up

One warning here - the LDIF export may have wrapped long lines. As mentioned previously, this is indicated by indenting a line directly below the line that it is a continuation of. This script and sample do not show how to deal with this issue. However, if you see wrapped lines you may want to try the following sed script over the input file first. What it attempts to do is join wrapped lines back together. I have not tried this in the real world, so some experimentation maybe needed.

/^ndsHomeDirectory/{
N
s/\n //
}

This script looks for lines beginning with ndsHomeDirectory, as these are the lines most likely to be wrapped. The "N" joins the next line with an embedded newline character. The substitution line replaces this inserted newline and a space with nothing, effectively joining the lines. Again, I have not tested this "in the wild," so there may be some gotchas. Maybe a topic for another article ...

One thing that I did on the production system was to break the output LDIF file up into chunks of 50 users or so. This was to make sure that I did not give eDirectory too much of a hard time when doing the import. By the time I had run through the ICE wizard setting up the next file for import, the updates from the previous file would have had a chance to synchronize out. Just remember to add the "version: 1" line to the beginning of each of the files if you decide to do this.

Making Sure the Rights are Coming Across

As the file copy is progressing, it's a good time to check in and see if the trustee rights are coming across. If not, you will need to fall back to the TRUSTBAR XML files to re-import them. Like the LDIF files, they are likely to need modification before they can be used.

To check the rights,

  1. Using Windows Explorer on a Windows workstation with the Novell client installed, map a drive letter to the volume that contains the new home directories.
  2. Drill down to a home directory.
  3. Right-click to open the context menu.
  4. Select the Trustee Rights option (it has a red "N" next to it).

The dialog that appears should state that the user who owns this directory is a trustee with either [RWCEMFA] or [RWCEMF] rights. The choice is a result of which tool was used to create the user. ConsoleOne and NWAdmin differ when giving the Access Control right.

If you are not seeing the rights correctly, you need to run the sed magic over TRUSTBAR's XML files and be prepared to restore the rights once the file copy has completed.

The following shows a TRUSTBAR file of a source home directory volume (data-tr.xml):

<trustees root="data:">
  <trustee path="data:\USERS\BBKing">
    <name>CN=BBKing.OU=BNE.O=DSR</name>
    <rights>251</rights>
  </trustee>
  <trustee path="data:\USERS\BRubble">
    <name>CN=BRubble.OU=BNE.O=DSR</name>
    <rights>251</rights>
  </trustee>
  <trustee path="data:\USERS\EClapton">
    <name>CN=EClapton.OU=BNE.O=DSR</name>
    <rights>251</rights>
  </trustee>
  <trustee path="data:\USERS\JHealey">
    <name>CN=JHealey.OU=BNE.O=DSR</name
    <rights>251</rights>
  </trustee>
  <trustee path="data:\USERS\JHendrix">
    <name>CN=JHendrix.OU=BNE.O=DSR</name>
    <rights>251</rights>
  </trustee>
</trustees>

To convert this file to match the volume and directory structure of the cluster system, use the following sed script (trmodscript):

s/data\:/USER\:/ig
s/USERS\\/HOMES\\/ig

The following command transforms the above data-tr.xml into user-tr.xml:

sed -f trmodscript data-tr.xml > user-tr.xml

The new user-tr.xml should look like this:

<trustees root="USER:">
  <trustee path="USER:\HOMES\BBKing">
    <name>CN=BBKing.OU=BNE.O=DSR</name>
    <rights>251</rights>
  </trustee>
  <trustee path="USER:\HOMES\BRubble">
    <name>CN=BRubble.OU=BNE.O=DSR</name>
    <rights>251</rights>
  </trustee>
  <trustee path="USER:\HOMES\EClapton">
    <name>CN=EClapton.OU=BNE.O=DSR</name>
    <rights>251</rights>
  </trustee>
  <trustee path="USER:\HOMES\JHealey">
    <name>CN=JHealey.OU=BNE.O=DSR</name>
    <rights>251</rights>
  </trustee>
  <trustee path="USER:\HOMES\JHendrix">
    <name>CN=JHendrix.OU=BNE.O=DSR</name>
    <rights>251</rights>
  </trustee>
</trustees>

If the trustee rights are not being copied with the data copy, use the following procedure to restore the trustee rights on the cluster volume, when the data copy has completed.

  1. Copy the user-tr.xml file to the root of the USER volume on the cluster.
  2. From the cluster node that has the user home directory volume (USER) currently mounted, run the following command:
TRUSTBAR USER:\user-tr.xml -R -V 

The -V will cause the console to show lots of output while the process is working. It may take a while if there are a lot of entries to be modified.

Changing the Home Directories

Once the data copy is completed and the trustee rights have been checked, and restored if needed, you can get down to the business of modifying the users HomeDirectory attribute. For this you need to import the modified LDIF file (outputfile.ldif), using the ICE tool in ConsoleOne.

  1. Start ConsoleOne.
  2. Start the NDS Import/Export Wizard from the Wizards menu.
  3. Make sure that the Import option is selected this time.


  4. Figure 18 - Import LDIF File Dialog


  5. Click Advanced.


  6. Figure 19 - LDIF Import Advanced Options Dialog


  7. In the Advanced options dialog, specify a file for any records that may be failed by the operation.
  8. The format of this file is another LDIF file. The idea of this file is to work out the issue with the objects that failed the operation requested and to reuse this log file to retry the operation. Again, verbose mode is probably a good idea so you can see what is going on.

  9. Click OK and then Next.


  10. Figure 20 - Source LDIF File Selection Dialog


  11. Enter the sed-modified LDIF file that you'll use to change the user's home directory attribute.
  12. Click Advanced.


  13. Figure 21 - LDIF Source File Advanced Options Dialog


    In the Advanced options dialog, uncheck "Add records without changetype". You should have removed these entries from the LDIF file, and we don't want new objects in the tree anyway.

    The "Display operations but do not perform" option allows you to test to see if things are going to work properly without actually writing to the LDAP directory. When the test import is finished, you can click Restart, which will start the Wizard again with all the same options. Just remember to come back into this dialog and turn off the testing mode to actually do the import.

  14. Click OK and then Next.


  15. Figure 22 - LDAP Server Selection Dialog


    These setting are the same as the ones used for the extract process. Again, using an SSL connection protects the administrative user's password.

  16. Click Advanced Options to check these settings.


  17. Figure 23 - LDAP Server Advanced Options Dialog


    The defaults here are usually good enough, especially if you have a connection to the server that contains a Read/Write or Master replica of the user objects you're modifying.

  18. Click OK and then Next.


  19. Figure 24 - Summary of Choices Dialog


  20. Click Finish to perform the modification of the user home directory attribute. If all is going well then the following dialog as shown below will appear.

Figure 25 - LDAP Operation Results Dialog


Remember that if there are any failed entries, these will be output to the log file specified during the configuration. This file can be used to perform the operation again once the issues are fixed.

If you divided the original LDIF file into smaller and more manageable sizes, repeat the above Wizard options until all files are imported. If you limit the LDIF file into 50 users or so, by the time you rerun the Wizard with the new file, the eDirectory synchronisation processes would have had a chance to catch up.

iManager Volume Space Restrictions

If you are restricting the space users are allowed to consume on their home directory volume, then these will need to be recreated. The ARCserve data copy function does not preserve these restrictions.

I found the functionality of ConsoleOne when trying to apply restrictions across multiple users, in different containers, to be a bit limited. For this task iManager was used. iManager has the ability to recognize the users with a trusteeship on the volume, and it allows them all to be selected and restricted in one go.

  1. Fire up a browser and head off to the iManager running on the cluster node server that currently has the users home directory volume mounted. In the tasks and Roles menu, select the Storage option and the Volumes sub-option. Once you select the server that has the volume mounted the right hand frame should change to look like the figure below:


  2. Figure 26 - iManager Storage Volumes


  3. From the list of volumes that appear, select the home directories volume (Data, in this case).
  4. Click User Quotas. The following listing will appear:


  5. Figure 27 - Current Quota Listing


  6. Click the All Users tab. This will give a listing of all users that are owners of files somewhere on the volume.


  7. Figure 28 - Selecting All Users


  8. Click the check box just below the New link (highlighted in Figure 28) to select all the users listed.
  9. Scroll down, de-selecting any users that you do not want restricted (e.g., .[Supervisor]).
  10. Click the Edit link. An additional dialog appears, as shown below:


  11. Figure 29 - Quota Edit Dialog


  12. Enter the value to restrict the users to and click the Finish button. The list of users should now reflect your settings.

Figure 30 - Figure 30: New Quotas Applied


If there are any users who do not have files on on the volume (in this example RDaltrey and WFlintstone), they will not appear in this list. The restrictions will need to be applied manually. The issue here is finding these users - this could be a topic for another article, if I hear enough feedback from this one ;-)

Test and Cleanup

Now that we can see the light at the end of the tunnel (and it does not seem to be an illuminated "No Exit" sign), we can test to see that the users' home directories are performing as they were on the original servers. The goal is to make sure you do not have any surprises during the next user logins.

  1. Rename the original home directories directory on the old server(s). This will ensure that if you missed modifying any user accounts that they will error out on their next login. You can then easily copy and modify their account as required.
  2. Try to log in as one of the test users you created at the beginning of the exercise. This will make sure that the HomeDirectory attribute is functioning correctly, that the user has the correct rights, and that the restrictions, etc., are working as expected.
  3. If template objects are used to create users, make sure to modify the home directory definition in these to match the new home directory location.
  4. Once you're happy that the testing has worked to satisfaction, you can delete the test users and their home directories.

Summary

This article has introduced the text manipulation tool sed, along with the CYGWIN environment. It has used them in a real-world, network administrator's task of moving data between servers in an eDirectory environment.

With some imaginative thought these tools, and procedures, could be adapted to other tasks without too much stress. I hope that this article has introduced you to some new tools, and maybe reintroduced some old ones, that you can place into your administration toolbox for the battles ahead.


Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com

© 2014 Novell