Earlier in the year Novell Connection magazine gave you a sneak peek of the much-anticipated release of Novell Open Enterprise Server 2. In this first of a four-part series, Novell Connection magazine digs deep into some of the highlights of this upcoming blockbuster, starting with Dynamic Storage Technology—a new feature focused entirely on helping customers control rising data storage costs.
The idea behind Dynamic Storage Technology came as a result of significant research that showed customers have massive amounts of stale data consuming their network storage at exponential rates. Not only does this ever-increasing quantity of rarely used data have an extremely negative impact on backup windows and disaster recovery efforts, but it also increases storage costs by taking over valuable real estate on high-performing and expensive disk storage, such as SANs. The research also indicated that while Hierarchical Storage Management (HSM) solutions help address these problems, they don't do enough and often create problems of their own. (See HSM—The Third Tier.)
A cross-functional team at Novell determined that data should be partitioned into two categories—important data and less important data. (See Figure 1.) Active data would reside in the first partition, and less active data would reside in the second partition, but the user would see a virtual view of the data as if it were all in the same partition. As the team sketched the concept on a white board, Novell Distinguished Engineer Dana Henriksen had an epiphany. "I looked at the drawing and realized I knew how to do it," he said. "In fact, it could be done quite easily because we had already virtualized the file system in Open Enterprise Server."
With the foundation already laid, in just a few weeks Henriksen and his team had a proof of concept that grew into the revolutionary Dynamic Storage Technology feature in Novell Open Enterprise Server 2 on Linux.
> A Virtual View
The main concept behind Dynamic Storage Technology is that you create two independent partitions (or "tiers") for your users' unstructured data (such as documents, spreadsheets, photos, videos, mp3s, presentations and text files). Newly created data and data that is used more frequently is stored on a primary partition that will likely be located on your higher-performance disk storage devices. Based on policies you define, less active or static data will be moved to a secondary partition on less expensive storage. This secondary partition is referred to as a shadow volume or shadow tree.
Dynamic Storage Technology overlays these two independent partitions or directory trees to create one virtual volume that is presented to the user.
Four major architectural components make up Dynamic Storage Technology:
- NCP Engine creates the file virtualization view for NetWare Core Protocol (NCP) clients on Windows and handles the bulk of file movement between the two partitions.
- ShadowFS creates a local mount point for local services that need to access the partitions and provides the unified virtual view of user storage for clients using the Common Internet File System (CIFS) protocol. CIFS clients include Samba users, Mac users and Windows users using Domain Services for Windows.
- Policy Engine defines and enforces policies that dictate under what circumstances files will transfer from one partition to the other.
- Management tools facilitate the administration of Dynamic Storage Technology.
To give you an idea of how it works, let's say you create a new file called "IT budget presentation" in a new subdirectory called "presentations." The NCP engine will create the "presentations" subdirectory on both the primary and shadow partitions. However, the file "IT budget presentation" will only be stored in the "presentations" subdirectory on the primary partition. The subdirectory on the shadow partition will be empty. In preparation for giving your budget presentation, you make changes to the file over the course of the next month. Since you're regularly accessing or modifying the file, it remains on the primary partition. If the file lays dormant for several months after you give your presentation, the policy engine—based on your defined policies—will move your file to the "presentations" subdirectory on the shadow partition. (See Figure 2.)
Even though the file has physically moved, the virtual view provided by either the NCP engine or ShadowFS makes the file appear to be right where you left it. That's one of the most powerful aspects of this technology. All movement of data back and forth between the primary and shadow volumes happens in the background, completely transparent to users. They see their data the way it has always been, and they access their data in the same way they always have.
Furthermore, it doesn't require any user training or special software on the client, as other file virtualization solutions do. This level of transparency makes Dynamic Storage Technology simple to roll out. Users don't even need to know you've implemented the technology, but the impact to the bottom line will be significant.
HSM—The Third Tier
Even though Hierarchical Storage Management (HSM) solutions try to help organizations tackle the problem of enormous amounts of stale data taking up valuable storage, these solutions create problems of their own. The need to de-migrate near-line data upon access is one of these. Another—and perhaps bigger—problem is the fact that when HSM solutions move data from your primary online storage to near-line storage, they leave stub files behind. These stub files can slow down your backup efforts, especially if 85 percent of the data entries that your backup encounters are stub files. Furthermore, restoring stub files from backups can be complicated and problematic.
Still, HSM solutions play a complementary, third-tier role with Dynamic Storage Technology to help you control data growth and storage costs. Dynamic Storage Technology addresses the first two tiers of your data growth challenge. First, it keeps your more important and frequently accessed data on your high-performing online storage. And second, it moves data that is accessed less frequently onto less expensive online storage. At the third tier, you can use HSM for any data you need to save, but rarely, if ever, access.
The way to do this would be to use your HSM solution the same way you normally would, except instead of pointing it to your primary volume, you point it to your shadow volume. This keeps stub files from cluttering up your primary partition, while still allowing you to migrate from secondary storage to near-line storage those files that need to be stored, but will rarely be accessed.
> No Need to De-migrate
A major challenge that companies face when they try to use HSM solutions to solve the problem of data explosion is de-migration. When less frequently used files are migrated to near-line storage such as a CD jukebox, users can't access them until the desired files are de-migrated or moved back to online storage. HSM de-migration also causes headaches for users of indexing tools such as the Google desktop. When Google indexes all of a user's files, it also causes those files to be unnecessarily de-migrated. You don't experience any of these de-migration problems with Dynamic Storage Technology. It doesn't need to de-migrate any files since all of your user data is still online and easily accessible.
So going back to our previous example, months have passed and you need some budget information that you know was in your presentation. Even though it's stored on a shadow volume, the virtual view of your directory shows it to be just where you originally stored it, and you can immediately open it to get the information you need.
After you close the file, the file may or may not move from the shadow partition back to the primary partition. It depends on whether your policy dictates that files move back to the primary partition after file access or file modification.
To ensure that no data is lost when files move, Dynamic Storage Technology uses a transactional methodology. Data is copied between the two physical partitions using a read-after-write-verify method to ensure the integrity of the new copy before deleting the old one. A transactional audit log also notes every file that has moved or that attempted to move. If a move fails, the audit log also notes the reason.
> Storage and Backup Savings
Novell customer research indicates that approximately 85 percent of a company's unstructured data is infrequently modified or accessed by users. In those cases, Dynamic Storage Technology can automatically migrate that 85 percent to less expensive storage, while keeping the other 15 percent on higher performing storage. Several of the customers Novell has worked with during development and beta testing expect this technology to lower their overall storage costs by 75 to 80 percent per terabyte.
Dynamic Storage Technology can also streamline backup efforts. With the enormous amounts of data you probably have stored, backups take way too long to complete. Backup windows are such that even if you begin a backup at 6 p.m., after most everyone has gone home for the day, it will likely still be going when workers return in the morning. Likewise, since the backups are so large, disaster recovery efforts take an exorbitant amount time, causing unacceptable downtime. Dynamic Storage Technology solves both of these backup-related issues.
Even though Dynamic Storage Technology presents your users with a virtual overlay of both the primary and shadow volumes, for backup purposes you as an administrator can deal with those physical volumes separately. For example, if you're like most customers and can move 85 percent of your data to a shadow volume, you can schedule backups of the shadow volume to occur less frequently since the data there rarely changes. The 15 percent of your data that is actively used and residing on your primary volume can be backed up more regularly. And since it will be a fraction of what you once had to deal with, backup time will be significantly shorter. Likewise, restore times will be much faster, since you can first restore the 15 percent of data residing on your primary volume that has a higher demand for access.
With this in mind, when you define your policies that dictate when a file moves from primary to shadow partitions, or vice versa, you will want to do it in a way that coincides with your backup schedules and policies. For example, if your IT budget presentation file, and the other files stored on your shadow volume, haven't changed in months, weekly or bi-weekly backups have probably been sufficient. But after a year passes, it's time to update the presentation, so you access that file and make changes to it. Right after you save the file, your policy may dictate that it immediately move back to the primary partition so your nightly backup can pick up those changes.
After your presentation, and after the file has been dormant for a few months, you'll probably want it to move back to the shadow partition. To do this, you'll want to set your policy to move it, along with any other files needing to move, right before you have a scheduled backup of your shadow volume. This will ensure that you always have a copy of your files on your most recent backups. Failing to coordinate your policies in this manner could lead to data discrepancies and even lost files. Dynamic Storage Technology has tools to help solve data discrepancies encountered during file backup or restoration, but it doesn't provide a way to recover files that have not been backed up due to its policies being out of sync with your backup procedures.
> Creating Policies
Dynamic Storage Technology gives you a lot of flexibility in defining the policies that dictate whether a file should reside on the primary or shadow partition. All newly created files are automatically placed on the primary partition. But you can move files based on file owner, file size, file type and data parameters such as the last time the file was accessed, its contents modified or its metadata (for example, filename, owner or access control) changed. So, if you want to make sure mp3 files don't consume your SAN storage, you can set a policy to move all mp3s to the shadow volume. On the other hand, if you want to make sure that all of your CEO's files always reside on your high-performing storage, you can define a policy based on file ownership that accomplishes that.
In addition to moving files by policy, Dynamic Storage Technology has two Move to Primary server-wide settings. The first one is On Modify, which as the name indicates, automatically moves a file to the primary partition when it's modified. This setting is on by default. The second setting is On Access, which moves a file to the primary partition when it's accessed. This setting is off by default, but there are times when it is useful to have it on, such as when you want to use Dynamic Storage Technology to facilitate an in-place volume upgrade or on-demand migration of data from a remote NetWare volume to your Linux volume on Open Enterprise Server 2. (See On-Demand Migration and In-Place Upgrades.)
> More than Worth It
Whether it's compliance with regulations governing data retention, varying needs of the business, or changes in users' work habits, there are a variety of reasons for the exponential growth of network data storage. It's no longer uncommon to find large enterprises with terabytes of unstructured data taking up valuable storage space. Dynamic Storage Technology in Novell Open Enterprise Server 2 makes it a lot easier for you to control and lower the cost of storing and backing up the data. So as you contemplate whether or not to move to Open Enterprise Server 2 and take a closer look at the benefits of Dynamic Storage Technology, you might find yourself agreeing with one of the attendees at this year's BrainShare who said, "If Dynamic Storage Technology was the only thing that Novell shipped in Open Enterprise Server 2, it would be reason alone to buy it and much more than worth it."
"Novell has allowed me to provide the university with better products and greater computer access for our students, while achieving tremendous cost savings," said Saad. "Without Novell solutions, we would probably have to double our staff to do what we do now."
On-Demand Migration and In-Place Upgrades
The On Access setting in Dynamic Storage Technology can be an excellent tool to facilitate on-demand migration or in-place migrations to a new platform or piece of storage hardware. For example, if you want to migrate all of your user data from a NetWare volume to a Linux volume on Novell Open Enterprise Server 2, instead of having to bring your system down to make the move, you simply use the On Access setting to allow you to keep your system up and then move files upon access.
To do this, you would set up your new Linux-based storage as your primary partition and then set your NetWare-based storage as a shadow volume, connecting to it remotely using the Novell Client for Linux. With the "On Access" setting turned on, whenever users access one of their files, that file will automatically and transparently be moved from the shadow volume to its mirrored subdirectory on the primary volume. Not only does this allow the migration to happen without bringing the system down and restricting file access, but it enables the most frequently accessed files to be moved first.
A similar scenario can be employed for in-place upgrades, setting up an older partition as a shadow volume, while making the new replacement storage the primary volume. Once again, with the On Access setting turned on, data will be migrated without disrupting user productivity or system operations.