4.4 Files, Folders, and Net Folders

4.4.1 Assessing Your Files and Folders

It is critical to assess what data Filr is going to present to users along with the network that it will use for that purpose.

Do the following:

  1. Identify the files that need to be accessible through Filr.

    • Identify the types of servers that host these files.

    • Identify the path to each folder where the files are located.

  2. Identify the types of files that are involved, such as PDF, .doc, and so on.

  3. Identify the files by size and quantify how many there are of each size.

    • Small (less than 500 MB)

    • Medium (between 500 MB and 2 GB)

    • Large (over 2 GB)

  4. Identify which file folders will require regular synchronization between file servers and Filr clients.

  5. Identify the file directories that will need to be indexed for full text searching.

NOTE:Some organizations find that Novell File Reporter is very helpful in planning for Filr. Using File Reporter helps them quickly determine the number, sizes, types, access frequency, and so on for their organization’s files.

4.4.2 Making an Initial Net Folder Assessment

  1. Identify which of the non-home folders identified in Section 4.4.1, Assessing Your Files and Folders require access through Filr.

  2. Within each of these folders, identify how many files will require access.

    Depending on the number of files and the clients accessing the data, a regularly scheduled Net Folder sync might be required. The length of the synchronization process depends on the number of files to be synchronized. The database size also depends on the number of files.

  3. Consider the type and size of files that will be accessed through Filr, with a goal of keeping the number and size of files as low as possible.

  4. Assess how often ACL rights will change within Net Folders.

  5. Assess how often trustee rights will change on which files (number and type).

4.4.3 Identifying Each Net Folder

  1. Using the information that you gathered in Section 4.4.2, Making an Initial Net Folder Assessment, write down each hosting file server, volume/share name, and folder path that will need to be accessed through Filr.

  2. Initially, you should enable only the highest priority files and folders for access through Filr, to ensure a controlled and smooth implementation.

    The remaining files and folders can be enabled in a phased approach during the production rollout.

  3. Consider creating a map of the file structure that shows the order in which folders will be enabled through the testing, pilot, and production deployments.

4.4.4 Planning for Net Folder Synchronization

Before users can see files and folders in a Net Folder, a synchronization process must be run so that Filr has information about the files in its database. 

Synchronization Overview

Figure 4-1 illustrates at a high level the information and content that get synchronized in Filr. The table that follows the figure describes some of the results and implications of the processes that take place.

The figure does not illustrate functional details. For example, it does not attempt to show the flow of LDAP metadata and file/folder metadata to Filr services for storage and indexing.

Figure 4-1 What Gets Synchronized

Letter

Details

Desktop applications for Windows and Macintosh workstations must download local copies of files and folders before accessing them.

Before the files and folders in Net Folders (including in Home folders) can be downloaded, their metadata must be synchronized with Filr. (See the metadata synchronization process indicated in the section that contains letter D.) For this reason, many Filr administrators perform a manual synchronization when initially setting up a Net Folder so that when users access it from their desktops, they can download the files and folders as expected.

Note that the arrow that shows files and folders being synchronized from file servers to desktops bypasses the Net Folders icon. This is because Net Folder files and folders are not stored in Filr. Only their metadata is synced. The files and folders that are synchronized to the desktops come directly from the file servers. Net Folders only facilitate the transfer process, as shown by the dotted line.

Files and folders in Filr-based personal storage are automatically synchronized to the workstation’s file system. Net Folders are also configured by default to allow data synchronization with desktop applications.

After data is synchronized with the desktops, then as long as the desktops are connected with the network, changes to local copies are immediately synchronized with the file server. If the workstation is disconnected, then when the connection is restored, synchronization is automatic.

However, Net Folder download functionality can be disabled. If this occurs, then desktop application users will not be able to view or access the files and sub-folders in their assigned Net Folders from their workstations. (Unlike web and mobile access, desktop applications do not provide Net Folder browsing. Browsing within the desktop applications is limited to local downloaded copies of Net Folder files.)

One reason that Filr administrators might disable the download functionality would be to prevent anyone from getting a local copy of sensitive files that must be stored only on the organization’s file servers.

The MySQL database, the Lucene indexer, and Net Folders work together to provide access to data through Filr. MySQL houses the Filr database. The Lucene indexer indexes all of the LDAP and file storage metadata so that search functionality is available. It also indexes file content for searching in folders that have indexing enabled. Additionally, Net Folders provide users who have authenticated with proper credentials access to files and folders on the network through the file storage metadata that is associated with them.

Regular LDAP synchronization is essential to timely user access to their files and folders. Some organizations find it sufficient to synchronize LDAP once a day. Others require more frequent synchronization to keep Filr abreast of changes in their identity stores.

Because file and folder activity on network file servers can be in constant flux, and because synchronization needs vary greatly from server to server, Net Folder synchronization is multi-faceted and very flexible.

You can set synchronization schedules for each Net Folder Server. The Net Folders associated with that server are then synchronized according to the general nature of the volume or share where they reside.

You can also set synchronization schedules for individual Net Folders that will override the server schedules and synchronize the folders either more or less frequently than the server schedule dictates.

You can perform manual synchronizations (which is a very useful feature when creating Net Folders) to ensure that desktop application users can download local copies for access.

You can enable Just-in-Time Synchronization (JITS) so that each time a folder is accessed from the web or a mobile device, its content is synchronized with Filr. (The desktop applications do not trigger JITS.)

Planning Net Folder Synchronization Methods to Use

There are two mechanisms for synchronizing Net Folders.

For more information about these synchronization methods, see Planning the Synchronization Method in the Novell Filr 1.0.1 Administration Guide.

Full Synchronization

This process can be triggered by both Net Folder Server and Net Folder configuration settings.

If synchronization is run as part of a Net Folder Server configuration, then all of the Net Folders connected with the Server are synchronized.

If a need arises, Net Folder Server-level synchronization processes can be executed immediately.

Net Folder-level synchronizations, on the other hand, happen only when the process is scheduled as shown in Figure 4-2.

Figure 4-2 Net Folder Synchronization Schedule Dialog

NOTE:The synchronization schedule interval should allow enough time for the synchronization process to complete before another process is scheduled to start. There is no harm if this recommendation isn’t followed, because the currently running synchronization overrides any synchronizations that are scheduled to begin while it is still running. On the other hand, it doesn’t make sense to make synchronization intervals too short.

Just-In-Time Synchronization (JITS)

Just-in-time synchronization (JITS) events are not scheduled. Rather, they are triggered whenever users access Net Folders or Home Folders through the web or mobile clients, thus ensuring that users can view at least part if not all of the Net Folder’s contents.

The Windows and Mac desktop clients do not currently support JITS.

When a user accesses a directory on a Net Folder or Home Folder that is configured for JITS, the files within that directory are synchronized. Subdirectories are not synchronized until they are accessed.

Net Folder JITS processes run in accordance with a global timeout setting. Home Folder JITS processes are individually configurable.

The global setting is configured as shown in Figure 4-3, in the Filr Administration Console under Just-in-Time Sync).

Figure 4-3 Just-in_Time Synchronization Dialog

By default, JITS runs for 15 seconds and then returns as much information as it has been able to gather. Users only see the directory information after the timeout value has lapsed. If all files and directories are not found in the required time, then an incomplete list is displayed.

The timeout value can be any number, including 0. If the value is 0, then the JITS session runs in the background and a directory listing is returned as soon as possible. A refresh of the directory listing might be required.

Each Net Folder can have additional JITS-related parameters set, as shown in Figure 4-4, but the global timeout (Figure 4-3) still applies.

Figure 4-4 Just-in-Time Synchronization on a Net Folder

If you enable JITS on a Net Folder, you can specify the Maximum Age that synchronization results remain valid for future JITS requests. For example, if a user accesses a Net Folder and JITS has run within the last 30 seconds, then information from that JITS session is displayed, even if something changed in the folder since the JITS session.

The JITS maximum age setting controls how quickly a file change made on the file server through a non-Filr client becomes visible in the affected Net Folder.

The ACL age parameter serves a similar function. Because ACL data tends to change less frequently than the files themselves, this value can probably be increased to 86400 seconds (1 Day), meaning that JITS ACL information would only be updated to cache once per day.

What Happens During a Scheduled Net Folder Synchronization

Net Folders are only synchronized according to a schedule unless the entire Net Folder Server is synchronized manually.

When a Net Folder is synced, the following things happen:

  1. The Filr Appliance connects to the specified net folder server (and path).

  2. Filr then walks the directory structure, collecting metadata about each directory and file as it goes.

    For example, it collects the name, size, date, and ACL information.

  3. Filr then stores this metadata in the MySQL database.

  4. Next, Filr sends the name (along with ACL info) to the Filr Search appliances to be indexed.

    This information is used for future searches, and so forth.

    If there are multiple Filr Search appliances, the Filr appliance sends the information to each Search appliance.

    The majority of the synchronization work occurs on the Filr appliance.

  5. After the synchronization of metadata is complete, Filr checks whether the Net Folder is configured for content indexing.

    If so, Filr then walks the net folder path again, retrieves a copy of each file from the Net Folder, extracts text from each file, and then sends the text to the Filr Search appliance for indexing.

    Although indexing is very powerful functionality, it is obviously CPU- and IO-intensive because each file is processed separately.

  6. The next time the Net Folder is synced, Filr follows the same process for any changes that it finds, such as new, modified, or deleted files and folders.

Minimizing Content Indexing Impacts on Synchronization Speeds

  1. Assess which data must be searchable.

  2. Start with a subset (1 to 10 GB) of that data.

  3. Monitor how much time is required to complete the indexing process.

  4. Increase the amount and monitor the process again.

  5. Keep in mind that indexing impacts Filr in the following areas:

    • Time required to synchronize and index a Net Folder

    • Disk space usage in Filr’s filr/filerepository directory

    • Bandwidth usage between the Filr appliance and the target servers where Net Folders are located

    • CPU utilization on the Filr appliances

  6. Periodically reassess whether the balance of the above points is still meeting your organization’s needs.

Network Latency Impacts on Synchronization Speeds

Any form of network-connection latency has an impact on synchronization speeds. Local Net Folders are much faster at synchronization than remote. The speed difference tends not to be limited by raw network bandwidth, but is more a function of network latency.

The speed difference between OES and Windows 2012 also has an impact.

Your Filr Client Mix and Synchronization Planning

Before Filr clients can see the files and folders inside Net Folders, Filr must synchronize those files and folders.

For web, iOS, and Android clients, synchronization involves transferring file and folder metadata to the Filr appliance either through a scheduled synchronization operation, or by using just-in-time (on-demand) synchronization.

For desktop clients, synchronization involves copying the files and folders to the desktop, which requires enabling Data Sync.

Because of the differences and nuances involved with these various synchronization methods, it is important to understand them and the way they impact the user experience within different Filr clients.

To learn more and to devise a synchronization plan that will best meet the needs of your Filr client users, see the following sections in the Novell Filr 1.0.1 Administration Guide:

Implications of External Modification of Net Folder Data

If content is added to a Net Folder through a Filr client, then the content is automatically visible through Filr. On the other hand, if content is added to the Net Folder target folder on the file server itself (for example, from a mapped drive on a desktop PC), then either a regular sync interval or a JITS request will be necessary before that modified data is reflected in Filr.