File System Factory: Identity-Based File System Management
Novell Cool Solutions: Feature
By Geoffrey Carman
Digg This -
Posted: 26 Apr 2006
Background - Home Directories and the University
In a university computer support environment, the view of the computing world is different than that of the small business workplace. Matters of scale and politics are obvious issues. One thing somewhat unique to education is the number of users. Unlike a place of business, a university can have 10,000 staff, and 60,000 students - two very large bodies of users with divergent needs.
Managing access for 70,000 users can be challenging, particularly when you want it to be seamless and scalable. Automated account management is pretty much a requirement, whether it be product-based, like Novell Identity Manager pulling data from a database somewhere, or home-grown.
But account management is only one piece of the puzzle. Students need space to store files they are working on, and floppies are NOT a good idea. (Of course, USB keys are so cheap now it almost seems irrelevant, but there are still benefits to providing space). Providing network home directories is very useful, regardless of size. In this day and age, a 50 meg home directory quota seems ancient, but if you are not using it for music or video - just school documents - it becomes very useful. However, graduate students may start to complain when their thesis documents start exceeding the quota!
In a school environment there can be hundreds to several thousands of workstations connecting (labs, teaching podiums, laptops, remote from home), and that load can be handled by single servers only up to a point. Once you cross that line of needing more than one server to handle the load, you run into a hugely complicated problem. How do you distribute, track, and balance users across multiple file stores?
Worse than that, there's the problem of school users and academic progress. They advance, whether from K-6 to 6-12 to university/college, from an undergraduate program to a graduate program, and so on. Often there is a desire to offer different services to different types of users.
Managing the location of home directories, or different quotas for different class of users who fluidly move between classes is a nightmare, especially if you are looking at writing an in-house solution. My advice to anyone considering it: you have underestimated the scope of the problem! It is much harder than it looks.
FSF to the Rescue
This is where File System Factory (FSF) comes to the rescue. FSF provides policy-driven home directory management, and much more. Policy-driven is the important part; it means that you set a policy that is stored in eDirectory, and that applies with inheritance to users, groups, and containers. FSF then watches for events on the objects being monitored and applies the rules that the policies define.
An example policy might be:
- All users of the .1stYear.myschool container get a 100 Meg quota. They all have home directories load-balanced across three volumes, based on free space.
- The .3rdYear.myschool container gets a 250 Meg quota with home directories on one specific volume.
If your provisioning system, be it LDAP, home-grown, NSure IDM, or even by hand, creates a user in these containers, FSF will catch the eDirectory event and generate a home directory according to the policy. If you move the user from one container to another, FSF will catch the event and apply the new policy. If needed, it will also move the home directory, adjust the quota, and do anything required for group storage.
The single nicest thing is that it abstracts the concept of file system access from your identity provisioning system! File system access is important and needed, but difficult in some ways from an identity management system viewpoint. (For example, LDAP does not understand the concept of file systems.)
That's just the simple stuff. Now for something completely different ...
Group Home Directories
Why no one at Novell thought of Group Home Directories decades ago is beyond me, but the crew at Condrey Consulting did, and boy is that a good thing! Why not extend the schema and add a Home directory attribute to Group objects, and try to offer it all the same properties of a user home directory?
This way, you can do course management by group, give each group some shared collaborative space, and since the group has membership that inherits file system rights, all members get rights to it. Now provide a tool to locate this space, and boy, isn't that interesting? FSF includes a Win32 application that can sit in the SysTray, called URAcess, that lists all home dirs a user has access to. (Groups and security equivalences are evaluated, the home_directory property returned and the list displayed. Elegant and simple).
A complementary product from the same company is Kanaka, which provides eDirectory integration for the Macintosh. It ships with an Applet called Dashboard that displays the same file spaces on the Macintosh. CCC also makes a web interface to the file system, called IUAdmin, which among other things shows all the file spaces via a web browser. This is truly cross-platform support and does not seem to have any browser dependencies.
But wait, there is more, so much more! Groups with a single shared directory is nice, but what if you could make it much more useful? For example, you could allow each group to have drop boxes, where students get W rights, the teaching assistants get RF, and only program directors get E rights. That means students cannot see what is there, but can put a file in the directory - as many files as the quota allows. The teaching assistant can read the files and see the list of them, but only the program director can delete a file. Thus a student can submit work, the assistant can read and mark it, but neither of them can attempt to fudge the results after the fact.
What about giving every member of the group personal storage inside the group home directory? For example, a specific course may require video work and huge files, but only for two semesters. If you have ever tried taking away space from users, good luck. So give them some course space, where each member of the group get a personal directory that only they have RWCEF rights to, but the TA's have RF or whatever you deem appropriate.
There is a very robust templating system in FSF that enables you to create a template directory for a policy in the file system. You create directories with special directory names (effectively variables), such as -OWNER-, -MEMBER-, and so on, and apply rights to those directories with special proxy users. Then when one of these templates is made part of an FSF policy applied to an object, FSF generates the matching structure. Best of all, FSF maintains it over time as users are added or deleted from groups, and other things change.
It is possible to make large scale changes in policies, and then run a backfill to reconcile the differences. For example, if you want to move thousands of home directories from an older server to a new cluster with 5 different home directory volumes using some load balancing algorithm, doing it by hand would be torturous. With FSF, you can take the existing system apply the policy you want, ask it to Backfill, and it will move the home dirs one at a time, until it is done. There you get a hands free migration, with almost no service interruption, beyond the user being moved at each moment in time. Even better, there is a Check Mode for backfills that tests the operation and reports obvious errors before moving any files.
FSF is an event monitoring system that can catch many events, such as adding or removing members of a group, creating a user or group, moving any of these objects, and applying policies with inheritance to them.
FSF also has a built-in message queueing system, seeing events on multiple replica servers and resolving them. That is a mouthful, but it means that since your eDirectory tree is loosely syncing, then an event on Replica server A won't be confused with the same event a few seconds later being seen on replica server B. FSF tracks both, reconciles them, and only processes it once. Just as importantly, FSF queues events to be processed, so that if something goes down and the even cannot be processed now, it will remain in the queue to be processed again later.
There are two NLM's that need to be deployed: an FSF Engine (FSFENGINE.NLM) and an FSF Event (FSFEVENT.NLM) listener. The Event listener needs to be deployed such that every object you care to monitor is represented in a local replica. For example, suppose you have a tree with 12 replicas. You only care about three of them, but there is no one server with all 3 replicas on it. You need to run enough Event listeners to cover all three replicas.
If one of the NLM's goes down, the events get queued and processed when the NLM resumes operations. This feature is quite unique (and astonishingly useful) to FSF and other Condrey Consulting products.
FSF is managed as a snapin to Novell Remote Manager (NoRM), the web server at port 8008/8009 (Secure) on all Netware servers since 5.x. You connect to NoRM as you normally would (https://server.ip.address:8009), and there will be a File System Factory item on the bottom of the list on the left.
The interface is very intuitive, allowing some features you might not have thought of. Backfill is a wonderful example - it enables you to take an existing environment, assign an FSF policy to it, and have the system go back and fill in what is missing or incorrect.
You can supply an interface to Helpdesk users to manage user quotas. You can choose to allow the Helpdesk staff to modify the quota, or to only bump it by a specified amount, and perhaps a limited number of times.
You can also use this to load balance accounts across servers. Several algorithms are provided, such as Random, Actual Space, and Percentage Space.
Just for fun, I tested the robustness by setting a test user's home directory to the root of a volume with 90GB of data in thousands of directories and millions of files. I then applied a policy that moved the home dir to a second server. FSF set the users home directory to a proxy directory that contains an HTML or TXT file that explains that your home directory is being moved, so please logout and try again in 5 minutes. Then it copied the files and reset the eDirectory Home directory property for the user to where it was now supposed to be. Performance was pretty good, considering both servers were in production on older hardware and 100 Meg links. Direct server-to-server copies are an interesting fringe benefit.
Often the technical merits of a product are not enough for management to agree to implement it. The goodies that come with it can sell the deal. FSF includes a number of reports on disk space usage which are immensely useful for planning and forecasting storage usage, growth, and abuse.
There is an Executive Dashboard that can show stats on the system, easily understandable to management. The items displayed are configurable and require specific eDirectory rights to access. Thus, you can tailor it to match the specific audience using it, from CIO to Helpdesk staffer. Items that can be displayed are things such as real time storage usage stats, Event stats, and Policy stats.
WorkflowWith version 1.21, FSF adds a workflow option to allow a procedure for handling deleted accounts. As part of the regular FSF policies, it allows you to:
- Immediately delete accounts when they fall out of a policy.
- Delete shared space when a member is deleted.
- Disable the space or delete it a settable number of days in the future.
But that may not be the only set of options you want. What if you need someone to review the space before it is deleted, to confirm there is no important data in it? (This makes a little more sense in a corporate environment than in an University setting, but it still has relevance). FSF relies on the same attributes for organizational reporting structure that eGuide uses - it enables you to define who takes over certain space when they are queued to be deleted. That user can then decide how to proceed.
Overall, this is a very useful product for medium to large environments, or anywhere that requires the use of collaborative storage. Once you have tried it, and thought about the possibilities, you will want it. More interestingly, try showing the collaborative group storage in action to your management, since that by itself is worth the price of the product. The product is written by Condrey Consulting and sold by Novell.
Note: At Brainshare in Salt Lake City in March 2006, Novell announced that next version of FSF will be called Novell Storage Manager 2.0. It will offer support for the event listener on SUSE Linux, with lots more interesting and new features. But the current version is worthwhile, right now!
Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions. www.webwiseone.com