Novell Filr 1.0.1 Planning and Deployment Best Practices Guide

IMPORTANT:This guide is a work in progress. Therefore, we recommend that you check back regularly for updates. The Filr development and sales teams are developing additional best practice guidelines that will be published in the coming months.

If you have specific content suggestions or requests, please submit them by using the User Comments feature at the bottom of each page of the online documentation.

Section 1.0, Small or Large Deployment?

Section 2.0, Deploying Filr as a Single Appliance

Section 3.0, Understanding Large Filr Deployments

Section 4.0, Planning a Large Filr Deployment

Section 5.0, Updating Filr

Section 6.0, Operational Administration

Section A.0, Documentation Updates

This guide documents best practices for planning and deploying Filr within a production environment. It supplements but doesn’t replace the Novell Filr documentation at http://www.novell.com/documentation/novell-filr1/.

Audience

This guide is intended for Filr administrators.

Feedback

We want to hear your comments and suggestions about this guide and the other documentation included with Novell Filr. Please use the User Comments feature at the bottom of each page of the online documentation.

Documentation Updates

For the most recent version of this guide, visit the Novell Filr 1.0.1 Documentation website.

Additional Documentation

The Novell Filr documentation on the web is constantly being updated. Therefore, you should always check the latest documentation when looking for detailed information on Filr.

The following guides are particularly relevant to the information in this guide:

Filr Overview “How Filr Works” Guide

Filr Installation and Configuration Guide

1.0 Small or Large Deployment?

Before deploying Filr, you must determine whether your organization’s long-term needs dictate a small deployment or a large deployment. The information in this guide can help you make that determination.

Figure 1-1 outlines basic capacity differences between small and large deployments.

Figure 1-1 Current and Future Load Requirements Determine Which Deployment Can Meet Your Organization’s Needs

For more information on small and large deployments, see Small Installation Vs. Large Installation in the Novell Filr 1.0.1 Installation and Configuration Guide.

2.0 Deploying Filr as a Single Appliance

IMPORTANT:The primary best practice for small Filr deployments is to ensure that a small deployment will meet your organization’s needs in the foreseeable future. A large percentage of past Filr support incidents come down to the simple fact that the service load was seriously underestimated. As a result, organizations have had to begin again with their deployments.

Novell strongly recommends that you thoroughly assess your organization’s needs before proceeding with a small deployment.

If you are certain that your organization’s file service load will never exceed the small deployment capacity limits outlined in Figure 1-1, then you can safely proceed with a small deployment as outlined in this section.

If you are not certain, complete the planning and other instructions in this guide, beginning with Section 3.0, Understanding Large Filr Deployments, before proceeding.

This Best Practices Guide Focuses on Large Deployments

Setting up a single Filr appliance is straight-forward and well documented in the Filr online documentation. The following sections provide only an overview of the basic processes you need to follow, and include links into the online documentation.

Getting Started with Your Small Deployment

The main processes involved in beginning a small deployment are illustrated and explained below.

Figure 2-1 A Small Deployment of Filr

Letter

Details

Download and deploy the software for your chosen virtualization platform as instructed in Downloading the Filr Appliance and Configuring the Virtual Environment in the Novell Filr 1.0.1 Installation and Configuration Guide.

IMPORTANT:The Novell best practices recommendation is to configure a small deployment appliance with 4 CPUs and 12 GB memory.

Specify the appliance’s basic configuration, which includes administrative users’ passwords, IP address settings, and the time zone and NTP time source, as instructed in Installing the Filr Appliance in the Novell Filr 1.0.1 Installation and Configuration Guide.

These settings are common to all Novell appliances.

Setting Up Your Small Deployment

After your Filr appliance is running on the network, you need to configure it with settings for the users and groups, the files and folders that they will access, and the permissions that they need in order to utilize Filr.

Be sure to complete the instructions in the following sections in the Novell Filr 1.0.1 Installation and Configuration Guide:

Small Deployment Configuration

It is especially important to adjust the Java Heap size to 8 GB as recommended in Small Installation (Filr Appliance): in the Novell Filr 1.0.1 Installation and Configuration Guide.

Setting Up the Filr Site

3.0 Understanding Large Filr Deployments

Section 3.1, Large Deployment Overview

Section 3.2, The Filr Appliances and Their Roles

Section 3.3, Appliance Disks and What They Store

Section 3.4, Understanding Filr Storage

Section 3.5, Clustering in Filr

Section 3.6, Virtualization Platform Support

3.1 Large Deployment Overview

No two organizations’ large Filr deployments are identical, but Figure 3-1 illustrates the components that are common to many of them.

Figure 3-1 Typical Components of a Large Filr Deployment

Letter

Details

Workstations and devices running Filr software can access Filr from anywhere on the Internet.

L4 switches provide load balancing and security.

Multiple Filr appliances service incoming requests for access to the organization’s files and folders. Because the appliances use shared storage, they all have access to the same data and configuration settings.

Two Filr Search appliances provide content indexing of data and folders. They also help with the synchronization of files between the file servers and various Filr clients.

Filr leverages existing MySQL database services for storing and retrieving configuration settings and user/group identities.

Filr doesn’t require any changes to the organization’s NetWare, OES, or Windows file servers.

3.2 The Filr Appliances and Their Roles

The appliances in a large Filr deployment perform three distinct roles that are essential to delivering Filr services: Filr, Filr Search, and MySQL. For more information, see Filr Applications and Appliances in the Novell Filr 1.0.1 Installation and Configuration Guide.

Table 3-1 Appliance Role Summary

Filr

Filr Search

MySQL

Role

File access and sharing

Content indexing for searching purposes

Database storage and retrieval of configuration and user/group identities

Number Required

At least one and as many as the service load demands

Two

One

Dependencies

At least one Filr Search appliance

Access to a MySQL database

None

None

Functions

Run a number of web-based services that are responsible for Filr client and web client (browser) interaction with the rest of the system.

Synchronize Net Folders.

When indexing applies, extract indexable content from synchronized files and send it to the Filr Search appliance to be indexed.

Accept information to be indexed

Respond to user-initiated search requests from Filr Appliances

Store and provide access to the following:

Configuration settings for all of the Filr-based clustered appliances

Information on the users and files within the system

Best Practices

Required:

Configure /vashare to point to a remote NFS disk that all Filr appliances share in common.

Strongly recommended: Deploy two Filr Search appliances to provide fault tolerance and prevent desktop clients from having to re-download all synchronized files when reindexing occurs.

Strongly recommended: Configure Filr to use a qualified DBA and high-availability in-house MySQL installation whenever possible.

3.3 Appliance Disks and What They Store

Table 3-2 summarizes the different appliance storage locations in a large, Filr-clustered deployment.

Table 3-2 Appliance Storage

Boot Partition (Hard Disk 1)

VM Hard Disk 2

Remote NFS-based Storage

Mount Point: /

Recommended Size: 100 GB of disk space

Required on: Filr, Filr Search, and MySQL appliances

Purpose: Stores the following:

All of the binary, configuration, and other files required for the operating system to run appliance services

Log files

Because log files are continually growing and can consume large amounts of disk space, you should regularly delete old log files.

Mount Point: /vastorage

Recommended Size: 25 GB of disk space

Required on: Filr, Filr Search, and MySQL appliances

Purpose: Stores the following:

Configuration files that are used for appliance upgrades

Ganglia files that are used to store monitoring information

Mount Point: /vashare

Recommended Size: Equal to the size of all files in the Personal Storage (My Files area), plus storage of HTML renderings and text extractions

Required on: Filr appliances in large, Filr-clustered deployments

Purpose: Stores the following:

Personal Storage files in My Files

Temporary files

Cached HTML renderings

Because cached files are not automatically deleted, you should regularly clean up the /vashare/cachefilestore area.

3.4 Understanding Filr Storage

Filr acts as an abstraction layer to the storage on your organization’s file servers and can also offer Personal Storage.

Net Folders: Pointers to data stored on your existing OES or Windows file servers

Personal Storage: Data that is stored on the Filr appliances themselves

For single Filr appliances, Personal Storage exists on the appliance’s /vastorage mount point.

For Filr-based clustered appliances, Personal Storage exists on a remote NFS directory that is mounted as /vashare on each Filr appliance that shares it.

3.5 Clustering in Filr

In this guide, the term clustering is used in two different ways:

Filr-Clustering: Filr appliances can be configured to store configuration settings and data that is not appliance-specific in the same storage location. Each appliance accesses the storage through a mount point named /vashare. The dialogs used to configure Filr to use shared storage refer to this as clustering. Shared storage provides fault tolerance and high availability in the sense that, if one Filr appliances goes down, other Filr appliances have access to the same settings and data.

This guide refers to shared storage configurations as Filr-clustering.

Of course, shared storage locations themselves are neither fault-tolerant protected nor highly available unless traditional cluster services are utilized. This leads to the second usage of "clustering" as it is used in this guide.

Figure 3-2

Traditional Clustering: In an enterprise environment, fault tolerance and high availability are important for the shared storage that Filr appliances access. The MySQL server and database that Filr appliances access should be protected as well.

The following design provides the maximum resilience and performance for enterprise environments. Although Figure 3-2 shows OES Servers and Clusters, these could be replaced by Microsoft Active Directory Servers or Linux Servers to provide the High Availability components for NFS and MySQL.

Figure 3-3 Filr HA Design

3.6 Virtualization Platform Support

Filr appliances are available for three virtualization platforms:

VMware

XEN

Microsoft Hyper-V

The examples and explanations in this document focus on VMware.

4.0 Planning a Large Filr Deployment

Section 4.1, Organization Needs

Section 4.2, Users and Groups

Section 4.3, Files and Folders

Section 4.4, Access and Security

Section 4.5, Net Folders

Section 4.6, Net Folder Synchronization

Section 4.7, Disk Space Requirements

Section 4.8, Content Indexing

Section 4.9, Memory Requirements

Section 4.10, Infrastructure

4.1 Organization Needs

A thorough assessment of your organization’s needs is key to ensuring that Filr meets expectations.

There is no absolute “one size fits all” approach for successful Filr deployments. Each organization has unique requirements, environments, resources, and project-management approaches. The following sections present general best practices for ensuring a successful Filr deployment.

Section 4.1.1, Stakeholder Participation Is Key to a Successful Rollout

Section 4.1.2, Meeting Organization Branding Requirements

Section 4.1.3, Assess the Business Requirements

4.1.1 Stakeholder Participation Is Key to a Successful Rollout

Stakeholder participation is key to a successful Filr deployment. Plan to involve those in your organization who are responsible for affected systems and services as early in the deployment process as possible.

Include All Functions

Consider involving representatives from as many of the following functions as applicable in your organization:

Management

User community

Server administration

Database administration

Network administration

Directory service administration

Security administration

Define and Test Use Case Scenarios

Write up use cases for how the people in your organization will use Filr.

For example:

Users accessing and viewing files through the Filr web client

Desktop users accessing and synchronizing files using the Filr desktop clients

Mobile app users accessing and viewing files

4.1.2 Meeting Organization Branding Requirements

Some organizations have branding requirements for any interfaces that employees and others use.

For information on the branding capabilities of Filr, see Setting Up Site Brandingin the Novell Filr 1.0.1 Administration Guide.

4.1.3 Assess the Business Requirements

Be sure to assess your organization’s business requirements first, then define the technical requirements for meeting the business objectives. Project stakeholder input can be extremely valuable in this process.

Discuss what is happening within the organization on the file-access and file-sharing fronts.

Identify key business drivers:

Is there a need to make corporate data available via mobile devices?

Is desktop access required as well?

What types of data does the organization need to expose?

Are you planning to allow storage and access to personal data?

Is there a need to share internally?

Is there a need to share externally? If so, are there any policies that apply?

Are there compliance and regulation standards that must be met?

4.2 Users and Groups

Filr leverages your organization’s identity stores. The following sections identify best practices for ensuring that Filr includes the users and groups that will use its services. For more information on managing users and groups in Filr, see Setting Up User Access to the Filr Site in the Novell Filr 1.0.1 Administration Guide.

Section 4.2.1, Identify User Sources

Section 4.2.2, Watch Out for Duplicate User Accounts

Section 4.2.3, Allow Enough Time to Import Users

4.2.1 Identify User Sources

Most Filr deployments use an LDAP source, such as eDirectory or Active Directory to control user access to the system. However, Google and Yahoo OpenID users can also access Filr, and you can define users by importing profile files or by creating users individually.

Identify the directory services that your organization currently uses.

Note important details about how the directory is configured, such as whether it is split over multiple sites.

Document each LDAP source’s access requirements, such as: Access ID, Password, and SSL certificate.  

Map out the LDAP structures to ensure that your import operations will find the target users and groups.

Define each required LDAP import task.

Identify how many LDAP users have file-server-based home directories that will be automatically configured for access through Filr.

Estimate how many files the home directories contain.

Document which users will be created manually or by importing profile files.

If you know of users who will need access to Filr through Google and Yahoo OpenID, identify these and record their account information.

Add up and record the number of users and groups that will be imported to Filr.

4.2.2 Watch Out for Duplicate User Accounts

Sometimes, organizations that utilize both eDirectory and Active Directory as identity stores have accounts for the same individuals in both directory services.

If you are importing users from eDirectory and from Active Directory, be aware that Filr doesn’t allow duplicate user accounts (for example, joe_user in both eDirectory and Active Directory).

4.2.3 Allow Enough Time to Import Users

Importing a large number of user and group objects takes more time than you might expect. For example, importing 25,000 objects could take from 1 to 4 hours, while importing 100,000 objects will take approximately 24 hours.

Depending on the number of users you need to import, you might need to consider running the process overnight or even over a weekend.

4.3 Files and Folders

It is critical to assess what data Filr is going to present to users along with the network that it will use for that purpose.

Do the following:

Identify the files that need to be accessible through Filr.

What types of servers host these files?

What is the folder path to where the files are located?

Identify the types of files that are involved, such as PDF, .doc, and so on.

Identify the files by size and quantify how many there are of each size.

Small (less than 500 MB)

Medium (between 500 MB and 2 GB)

Large (over 2 GB)

Identify which file folders will require regular synchronization between file servers and Filr clients.

Identify the file directories that will need to be indexed for full text searching.

NOTE:Some organizations find that Novell File Reporter is very helpful in gathering and organizing file data for their Filr deployments.

4.4 Access and Security

What are the file system access rights and privileges of the users and groups identified in Section 4.2.1, Identify User Sources for the files and folders identified in Section 4.3, Files and Folders.

Among the prioritized files, are there any file system restrictions in addition to the standard access rights and privileges?

4.5 Net Folders

Section 4.5.1, Initial Net Folder Assessment

Section 4.5.2, Identifying Each Net Folder

4.5.1 Initial Net Folder Assessment

How many of the non-home folders identified in Section 4.3, Files and Folders require access through Filr?

Within each of these folders, how many files will be accessed?

Depending on the number of files and the clients accessing the data, a regularly scheduled Net Folder sync might be required. The length of the synchronization process depends on the number of files to be synchronized. The database size is also dependant on the number of files.

It is important to consider the type and size of files that will be accessed through Filr. It is also important to keep the number and size of files as low as possible.

How often will ACL rights change within Net Folders?

How often will trustee rights change per number and type of files?

4.5.2 Identifying Each Net Folder

Using the information that you gathered in Section 4.5.1, Initial Net Folder Assessment, write down each hosting file server, volume/share name, and folder path that will need to be accessed through Filr.

Initially, you should enable only the highest priority files and folders for access through Filr, to ensure a controlled and smooth implementation.

The remaining files and folders can be enabled in a phased approach during the production rollout.

Consider creating a map of the file structure showing the order in which folders will be enabled through the testing, pilot, and production deployments.

4.6 Net Folder Synchronization

Before users can see files and folders in a Net Folder, a synchronization process must be run so that Filr has information about the files in its database. 

Section 4.6.1, Synchronization Overview

Section 4.6.2, What Happens During a Scheduled Net Folder Synchronization

Section 4.6.3, Network Latency and Synchronization Speeds

Section 4.6.4, Your Filr Client Mix and Synchronization Planning

Section 4.6.5, External Modification of Net Folder Data

Section 4.6.6, Planning for Net Folder Synchronization

4.6.1 Synchronization Overview

Figure 4-1 illustrates at a high level the information and content that get synchronized in Filr. The table that follows the figure describes some of the results and implications of the processes that take place.

The figure does not illustrate functional details. For example, it does not attempt to show the flow of LDAP metadata and file/folder metadata to Filr services for storage and indexing.

Figure 4-1 What Gets Synchronized

Letter

Details

Desktop applications for Windows and Macintosh workstations must download local copies of files and folders before accessing them.

Before the files and folders in Net Folders (including in Home folders) can be downloaded, their metadata must be synchronized with Filr. (See the metadata synchronization process indicated in letter D.) For this reason, many Filr administrators perform a manual synchronization when initially setting up a Net Folder so that when users access it from their desktops, they can download the files and folders as expected.

Note that the arrow that shows files and folders being synchronized from file servers to desktops bypasses the Net Folders icon. This is because Net Folder files and folders are not stored in Filr. Only their metadata is synced. The files and folders that are synchronized to the desktops come directly from the file servers. Net Folders only facilitate the transfer process, as shown by the dotted line.

Files and folders in Filr-based personal storage are automatically synchronized to the workstation’s file system. Net Folders are also configured by default to allow data synchronization with desktop applications.

After data is synchronized with the desktops, then as long as the desktops are connected with the network, changes to local copies are immediately synchronized with the file server. If the workstation is disconnected, then when the connection is restored, synchronization is automatic.

However, Net Folder download functionality can be disabled. If this occurs, then desktop application users will not be able to view or access the files and sub-folders in their assigned Net Folders from their workstations. (Unlike web and mobile access, desktop applications do not provide Net Folder browsing. Browsing within the desktop applications is limited to local downloaded copies of Net Folder files.)

One reason that Filr administrators might disable the download functionality would be to prevent anyone from getting a local copy of sensitive files that must be stored only on the organization’s file servers.

The MySQL database, the Lucene indexer, and Net Folders work together to provide access to data through Filr. MySQL houses the Filr database. The Lucene indexer indexes all of the LDAP and file storage metadata so that search functionality is available. It also indexes file content for searching in folders that have indexing enabled. And Net Folders provide users who have authenticated with proper credentials access to files and folders on the network through the file storage metadata that is associated with them.

Regular LDAP synchronization is essential to timely user access to their files and folders. Some organizations find it sufficient to synchronize LDAP once a day. Others require more frequent synchronization to keep Filr abreast of changes in their identity stores.

Because file and folder activity on network file servers can be in constant flux, and because synchronization needs vary greatly from server to server, Net Folder synchronization is multi-faceted and very flexible.

You can set synchronization schedules for each Net Folder Server. The Net Folders associated with that server are then synchronized according to the general nature of the volume or share where they reside.

You can also set synchronization schedules for individual Net Folders that will override the server schedules and synchronize the folders either more or less frequently than the server schedule dictates.

You can perform manual synchronizations, which is a very useful feature when creating Net Folders, to ensure that desktop application users can download local copies for access.

You can enable Just-in-Time Synchronization (JITS) so that each time a folder is accessed from the web or a mobile device, its content is synchronized with Filr. (The desktop applications do not trigger JITS.)

4.6.2 What Happens During a Scheduled Net Folder Synchronization

A Net Folder can be synchronized according to a schedule.

When a Net Folder is synced, the following things happen:

The Filr Appliance connects to the specified net folder server (and path).

Filr then walks the directory structure, collecting meta data about each directory and file as it goes.

For example, it collects the name, size, date, and ACL information.

Filr then stores this metadata in the MySQL database.

Next, Filr sends the name (along with ACL info) to the Filr Search appliances to be indexed.

This information is used for future searches, and so forth.

If there are multiple Filr Search appliances, the Filr appliance sends the information to each Search appliance.

The majority of the synchronization work occurs on the Filr appliance.

After the synchronization of metadata is complete, Filr checks whether the Net Folder is configured for content indexing.

If so, Filr then walks the net folder path again, retrieves a copy of each file from the Net Folder, extracts text from each file, and then sends the text to the Filr Search appliance for indexing.

Although indexing is very powerful functionality, it is obviously CPU- and IO-intensive because each file is processed separately.

The next time the Net Folder is synced, Filr follows the same process for any changes that it finds, such as new, modified, or deleted files and folders.

4.6.3 Network Latency and Synchronization Speeds

Any form of network-connection latency has an impact on synchronization speeds. Local Net Folders are much faster at synchronization than remote. The speed difference tends not to be limited by the raw network bandwidth, but is more a function of network latency.

The speed difference between OES and Windows 2012 also has an impact.

There are two modes of synchronization:

Full Synchronization

Just In Time Synchronization (JITS)

For information about these synchronization methods, see Planning the Synchronization Method in the Novell Filr 1.0.1 Administration Guide.

4.6.4 Your Filr Client Mix and Synchronization Planning

Before Filr clients can see the files and folders inside Net Folders, Filr must synchronize those files and folders.

For web, iOS, and Android clients, synchronization involves transferring file and folder metadata to the Filr appliance either through a scheduled synchronization operation, or by using just-in-time (on-demand) synchronization.

For desktop clients, synchronization involves copying the files and folders to the desktop, which requires enabling Data Sync.

Because of the differences and nuances involved with these various synchronization methods, it is important to understand them and the way they impact the user experience within different Filr clients.

To learn more and devise a synchronization plan that will best meet the needs of your Filr client users, see the following sections in the Novell Filr 1.0.1 Administration Guide:

Planning the Synchronization Method

Planning the Synchronization Schedule

Planning a Clustered Filr System to Support Net Folder Synchronization

Planning the Amount of Data to Synchronize

4.6.5 External Modification of Net Folder Data

If content is added to a Net Folder through a Filr client, then the content is automatically visible through Filr. On the other hand, if content is added to the Net Folder target folder on the file server itself (for example, from a mapped drive on a desktop PC), then either a regular sync interval or a JITS request will be necessary before that modified data is reflected in Filr.

4.6.6 Planning for Net Folder Synchronization

Full Synchronization

Just-In-Time Synchronization (JITS)

Full Synchronization

This process can be triggered from a Net Folder Server or a Net Folder.

If synchronization is run from a Net Folder Server, then all of the Net Folders connected with the Server are synchronized. Net Folder Server-level synchronization processes can be executed immediately when needed. Net Folder-level synchronizations, on the other hand, happen when the process is scheduled as shown in the following dialog:

NOTE:The synchronization schedule interval should allow enough time for the synchronization process to complete before another process is scheduled to start. There is no harm if that is not the case. The currently running synchronization overrides any synchronizations that are scheduled while it is still running. Still, it doesn’t make sense to make synchronization intervals too short.

Just-In-Time Synchronization (JITS)

JITS is an unscheduled synchronization that is triggered when users access Net Folders through the web or mobile clients. (The Windows desktop client does not currently support JITS.) JITS makes it possible for users to view the content of Net Folders that have not yet been synchronized.

When a user accesses a directory on a Net Folder that is configured for JITS, the files within that directory are synchronized. Subdirectories are not synchronized until they are accessed.

Net Folder JITS processes run in accordance with a global timeout setting. Home Folders JITS processes are individually configurable.

The global setting is configured as shown below:

This setting is found from the Admin Console under Just-In-Time Sync.

As shown, the setting means that JITS will run for 15 seconds and then return as much information as it has been able to gather.

The users sees the directory information after the timeout value has lapsed.

The timeout value can be any number including 0.

If all files and directories are not found in the required time, then an incomplete list is returned to the user.

If the value is 0, then the JITS session runs in the background and a directory listing is returned as soon as possible. A refresh of the directory listing might be required.

Each Net Folder can have additional JITS-related parameters set, as shown below, but the global timeout still applies.

JITS can be enabled or disabled on a Net Folder-by-Net Folder basis. If enabled, you can specify the Maximum Age for Just-In-Time results to control how quickly a file change on a Net Folder from a non-Filr client becomes visible to Filr users. For example, if a user accesses the directory and JITS has run in the last 30 seconds, then existing information is displayed instead of running a new JITS session.

The ACL age parameter serves a similar function. Because ACL data tends to change less frequently than the files themselves, this value can probably be increased to 86400 seconds (1 Day), meaning that JITS ACL information would only be updated to cache once per day.

4.7 Disk Space Requirements

Based on your analysis of files and folders, indexing requirements, and search requirements, define the estimated space needed for the Filr, MySQL, and Search servers. Check the Novell Filr 1.0.1 Installation and Configuration Guide for the latest data and start with the recommended minimums. Use the following as a guide:

Section 4.7.1, Sizing /vastorage

Section 4.7.2, Sizing the Remote NFS Share ( /vashare)

Section 4.7.3, Filr Search Sizing Guidelines

Section 4.7.4, MySQL Sizing Guidelines

Section 4.7.5, A Word About Inodes

4.7.1 Sizing /vastorage

/vastorage can be 5-10GB in size because only Ganglia and the appliance-specific files needed for updates are stored there.

4.7.2 Sizing the Remote NFS Share ( /vashare)

/vashare is used by all of the Filr appliances in a Filr Cluster for storing the following:

Personal Storage

The files that users store in their assigned My Files personal storage are stored in /vashare. Of course, Home directories are merely a special Net Folder and their data is stored on the file servers that host them.

HTML Renderings

HTML renderings and text extractions have a significant impact on the amount of storage required per file for the Filr appliance.

Each HTML rendering occupies disk space.

All HTML renderings are stored in /vashare.

Only one HTML rendering exists per document.

The disk space used by HTML renderings depends on the type of file that is rendered:

Microsoft Office and OpenOffice files use the same amount of disk space as the original file, so a rendered 1 MB file requires an additional 1 MB of disk space.

PowerPoint files require ~3 times the disk space as the original file, so a rendered 1 MB file requires an additional 3 MB of disk space.

PDF files require 10 times the disk space as the original file, so a rendered 1 MB file requires an additional 10 MB of disk space.

If HTML renderings are consuming more than 10 GB of disk space on your system, you can delete all HTML renderings by restarting the Filr appliance. To restart the Filr appliance, change any configuration option, then click Reconfigure Filr Server, as described in Changing Configuration Options for the Filr Appliance in the Novell Filr 1.0.1 Installation and Configuration Guide.

Indexing Text Extractions

When a file is added to Filr (either to Personal Storage or to a Net Folder), text from the file is extracted and added to the search index to be used for searching. By default, each text-extracted file is truncated to 1.1 MB.

Uploads to Net Folders

Based on these items and the number of users, /vashare should be at least 50 GB in size and perhaps much larger, depending on usage. For example, if there are 1,000 users uploading a 2MB file at the same time, that activity alone would consume up to 2 GB of disk space.

4.7.3 Filr Search Sizing Guidelines

Allocate ~5 KB disk space per file on non-indexed Net Folders.

Allocate ~11 KB of disk space per file on indexed Net Folders.

Initial synchronization requires more disk space, but Lucene optimizes space usage after the initial sync, thus reducing the space required. 

Because of the initial indexing overhead, Novell recommends waiting until the initial synchronization and optimization complete before enabling user access to the Filr system.

Formula

F=Number of Files        NF=Net Folders

Filr Search

F x 11 KB (NF Indexed)+

F x 5 KB (NF not indexed)

Organization Numbers

1,000,000 files (75% indexed; 25% non-indexed)

Filr Search Estimate

750,000 files * 11 KB = 8250000 KB

250,000 files * 5 KB = 1250000 KB

Initial Estimate of Lucene index is 9.05 GB.

4.7.4 MySQL Sizing Guidelines

Allocate 30 KB per file when indexing is enabled for Net Folders.

Allocate 20 KB per file when indexing is not enabled for Net Folders.

Allocate 10 KB per user.

Formula

U=Number of Users        F=Number of Files        NF=Net Folders

MYSQL

F x 30 KB (NF indexed)+

F x 20 KB (NF not indexed)+

U x 10 KB

Organization Numbers

1,000,000 files (75% indexed; 25% non-indexed) and 1,000 users

MySQL Database Estimate

750,000 files * 30 KB = 22,500,000k

250,000 files *20 KB =  5,000,000k

1,000 Users * 10 KB = 10,000k

Initial Estimate of Database size is 26 GB.

4.7.5 A Word About Inodes

Each file written to a disk consumes one inode, and Filr writes multiple files to /vashare for each file or directory that is added. For example, if a disk targeted by the /vashare mount point has 3 million inodes, and if 3 million files are written to the /vashare directory, then no more files can be written to that disk unless the amount of inodes is increased. See section The File System in Reality, in the The Linux Documentation Project.

4.8 Content Indexing

How many of the files need to have their content indexed? This impacts Filr in the following areas:

Time required to synchronize and index a Net Folder

Disk space usage in Filr’s filr/filerepository directory

Bandwidth between Filr install and the sources of the Net Folders

CPU utilization on the Filr appliances

4.9 Memory Requirements

Based on the defined files and folder, indexing, and search requirements, define the estimated space needed for the Filr, MySQL, and Search servers.  Check the Novell Filr 1.0.1 Installation and Configuration Guide for the latest data and start with the recommended minimums. Use the following as a guide:

When using multiple appliances for production deployments: 8 GB memory and 2 CPUs.  At least 66% of the memory should be dedicated to the Java processes [heap].  More memory might be required.

If memcached has been offloaded to the Filr Search appliances, or elsewhere, then it might be possible to dedicate as much as 75% of the memory to the Java processes.

4.10 Infrastructure

Section 4.10.1, VM Host Server Requirements

Section 4.10.2, Ensuring Fault Tolerance

Section 4.10.3, Network Infrastructure

4.10.1 VM Host Server Requirements

Determine the Filr component server hardware configurations and from this the hosting server requirements.  

Use the above disk space requirements and Filr installation manual server requirements as guides for determining memory, disk space, and processor requirements.

Ensure that the required disk space and memory are available for the Filr deployment.

Test and monitor your pilot deployment to identify connection usage patterns and determine how many Filr servers are required. Currently, a single Filr server can service approximately 500 concurrent user connections.

For example, you might find that two Filr servers would serve 3,000 users given that all of them are not concurrently connected.

VM Host Platform

One of the following software virtualization platforms:

VMware

One of the following VMware host servers for hosting the Filr appliance:

Installing VMware Tools on the host server is recommended.

ESXi 5.5.x with the latest update

ESXi 5.1.x with the latest update

ESXi 5.0 with the latest update

ESX 4.1.x and ESXi 4.1.x with the latest update

For the most up-to-date compatibility matrix of supported VMware host servers, refer to the VMware Compatibility Guide that is provided by VMware.

To generate the correct chart from the VMware Compatibility Guide:

Ensure that Guest OS is selected in the What are you looking for field.

In the OS Vendor field, select SUSE, then in the OS Family Name field, select SUSE Linux Enterprise 11.

Click Update and View Results, then look at the SUSE Linux Enterprise Server 11 Service Pack 2 64-bit line of the provided table.

The versions of ESX and ESXi that are listed in the Supported Releases column of this row are compatible with Filr 1.0.

A VMware vSphere client 4.x or VMware vSphere client 5.x for accessing the host server and appliance for initial configuration.

Not all versions of the vSphere client are compatible with versions of VMware ESX and ESXi. For a complete matrix of compatibility, refer to the VMware Product Interoperability Matrixes that are provided by VMware.

Xen

SLES 11 SP2, 64-bit

SLES 11 SP3, 64-bit

Hyper-V

Windows 2012 Server (as a Role)

Windows 2008 R2 Server (as a Role)

Hyper-V Server 2012 (standalone)

Hyper-V Server 2008 R2 (standalone)

Virtual Machine Memory Requirements

Filr Appliance (each): 8 GB of memory and 2 CPUs. At least half of the memory should be dedicated to the Java heap.

Database Appliance (if no in-house MySQL service is available): 4 GB of memory and 2 CPUs. (More memory is required if there are more than 1,000 users on the system.)

Search Index Appliance (2 required): 4 GB of memory and 2 CPUs. (More memory is required if there are more than 1,000 users on the system.) About 75% of the memory should be dedicated to the Java heap.

4.10.2 Ensuring Fault Tolerance

Filr Performance and High-Availability

Filr Search Redundancy

Filr Performance and High-Availability

Determine whether an L4 switch, NetIQ Access Manager, Apache, or DNS round-robin will be used to front the Filr servers.

Novell recommends having at least two clustered Filr servers for basic performance and high-availability. You can add more as your deployment grows through your initial pilot to a phased-in production rollout.

Novell recommends connecting a third Filr Appliance to the shared storage and not fronting it with a load balancer. This appliance can then be used for dedicated administration and file synchronization purposes.

Filr Search Redundancy

Novell recommends having two Search/Index servers for redundancy (fault tolerance) purposes. Having more than two is not recommended or needed, because there are no performance gains associated with multiple Filr Search appliances.

Having functional search appliances and the services they provide is vital to the health of the system. If you are using a single search appliance, avoid re-indexing when possible. Re-indexing causes all Windows and Macintosh desktop clients to delete local Filr files and re-download them from the server as the index is recreated.

If you are using two Search Appliances, re-indexing the appliances at separate times prevents the clients from re-downloading all their data.

4.10.3 Network Infrastructure

Where are the likely locations of the Filr appliances and the target data?

Do you have a geographically dispersed network?

If so, what is the bandwidth between sites (including latency)?

5.0 Updating Filr

This section provides best practice recommendations for updating Filr. For complete instructions on the update process, see Updating Filr in the Novell Filr 1.0.1 Installation and Configuration Guide.

Section 5.1, Updating from 1.0 to 1.0.1

Section 5.2, Do Not Use a Filr Rolling Update

Section 5.3, Update Order

Section 5.4, How the Update Process Works

5.1 Updating from 1.0 to 1.0.1

This document assumes you have a functioning version Filr 1.0  and are updating to version 1.0.1.

5.2 Do Not Use a Filr Rolling Update

Do not attempt a Filr rolling update.

Unless the Filr appliance is down, serious errors and a failed upgrade can result.

Update Filr only after all the Filr appliances and Filr Search servers have been stopped.

The MySQL database and the NFS server must be running during the update.

5.3 Update Order

To successfully update your version 1.0.1 cluster, you should update in the following order:

Update the Lucene Servers first.

Update each Filr appliance separately. Do not attempt to update more than one appliance at a time.

If required, the MySQL database is automatically updated when the first Filr appliance is updated.

As each Filr appliance is updated, log in directly to the appliance (not through a proxy).

If you do not see normal operations, use port 9443 and check the following log files for errors or other abnormal operations: ssf.log, ssr.log, and catalina.log.

5.4 How the Update Process Works

Each Filr appliance is newly installed.

The update process includes the use of the existing /vastorage disk from the version 1.0 appliances.

If applicable, the update process uses the /vashare NFS share as well as the existing /vastorage disk for VMWARE.

6.0 Operational Administration

This section doesn’t replace the Filr administration documentation; its only intent is to assist in day-to-day administration.

Section 6.1, Support for DFS

Section 6.2, Monitoring File Meta-Data Synchronization in a Filr-based Cluster

Section 6.3, Net Folders

Section 6.4, Backup and Restore of MySQL

Section 6.5, Backing up and Restoring Filr Data

Section 6.6, Setting the URL for Desktop Client Downloads and Updates

Section 6.7, Monitoring the Indexing Process

Section 6.8, The File Upload Process

6.1 Support for DFS

Section 6.1.1, Dynamic File Services (Windows)

Section 6.1.2, Distributed File Services (OES)

6.1.1 Dynamic File Services (Windows)

Novell Dynamic File Services (DFS) (also known as shadow volumes) on Windows servers is provided as a technical preview. That is, it can be used and demonstrated in a demonstration or pilot capacity, but it is not yet supported for production deployments.

IMPORTANT:The equivalent service on OES, Dynamic Storage Technology (DST), is fully supported by Filr.

6.1.2 Distributed File Services (OES)

Distributed File Services (DFS) for OES, not to be confused with DFS on Windows, involves the NSS Volume Location Database (VLDB) and is not supported by Filr 1.0.1.

6.2 Monitoring File Meta-Data Synchronization in a Filr-based Cluster

Synchronization requests can go to any of the Filr appliances in a Filr-based cluster, regardless of which appliance a user logs in to. This makes monitoring the status of outstanding synchronization requests a little more involved. Please note that with Filr 1.0.1 the logging of data mentioned here might actually take place in the ssr.log file, so all later references to ssf.log might need to change to ssr.log.

All synchronization status is logged to /var/opt/novell/tomcat-filr/logs/ssf.log. However, this is a per-server log. In a Filr-based clustered implementation, all of the Filr appliances’ ssf.logs need to be examined.

On each server, run the following command:

grep -i “(full) Starting” /var/opt/novell/tomcat-filr/logs/ssf.log

This lists all of the sync processes that have been started and have been recorded in the log file.

If the command finds any log entries, it returns something similar to:

2013-07-08 14:27:54,418 INFO  [Sitescape_Worker-1] [com.novell.teaming.module.folder.impl.PlusFolderModule] - (full) Starting synchronization on folder [/Home Workspace/Net Folders/shared] (id=41)

2013-07-08 09:05:25,969 INFO  [http-apr-8443-exec-1] [com.novell.teaming.module.folder.impl.PlusFolderModule] - (full) Starting synchronization on folder [/Home Workspace/Net Folders/support] (id=305)

2013-07-08 09:29:36,566 INFO  [http-apr-8443-exec-3] [com.novell.teaming.module.folder.impl.PlusFolderModule] - (full) Starting synchronization on folder [/Home Workspace/Net Folders/shared] (id=41)

2013-07-08 09:32:27,887 INFO  [http-apr-8443-exec-6] [com.novell.teaming.module.folder.impl.PlusFolderModule] - (full) Starting synchronization on folder [/Home Workspace/Net Folders/shared] (id=41)

This shows that on this particular server, four full synchronization tasks were started in the last 24 hours. Prior to this time, the ssf.log would have already been rolled over to a different file name. The lines extracted from the log show the folder name that is being synced.

After the Filr Appliance that executed the sync has been found, another grep can be used:

grep -i "(full) Completed synchronization" /var/opt/novell/tomcat-filr/logs/ssf.log

This shows the number of completed synchronization tasks within the scope of the log file.

Based on the previous output, the following text could be retrieved.

grep -i "(full) Completed synchronization" /var/opt/novell/tomcat-filr/logs/ssf.log

2013-07-08 08:29:39,215 INFO  [Sitescape_Worker-1] [com.novell.teaming.module.folder.impl.PlusFolderModule] - (full) Completed synchronization on folder [/Home Workspace/Net Folders/shared] (id=41) -

2013-07-08 09:05:26,084 INFO  [http-apr-8443-exec-1] [com.novell.teaming.module.folder.impl.PlusFolderModule] - (full) Completed synchronization on folder [/Home Workspace/Net Folders/support] (id=305) -

2013-07-08 09:31:03,127 INFO  [http-apr-8443-exec-3] [com.novell.teaming.module.folder.impl.PlusFolderModule] - (full) Completed synchronization on folder [/Home Workspace/Net Folders/shared] (id=41) -

2013-07-08 09:33:28,973 INFO  [http-apr-8443-exec-6] [com.novell.teaming.module.folder.impl.PlusFolderModule] - (full) Completed synchronization on folder [/Home Workspace/Net Folders/shared] (id=41)

This shows that each of the tasks that started on the server actually finished.

Further details of a completed synchronization can be extracted by using the vi or less commands on the log file in question.

In vi, issue the following command to see the first occurrence of the “Completed” string.

/Completed

This will find the text “Completed” in the log file and will show something similar to the text below:

2013-07-08 08:29:39,215 INFO  [Sitescape_Worker-1] [com.novell.teaming.module.folder.impl.PlusFolderModule] - (full) Completed synchronization on folder [/Home Workspace/Net Folders/shared] (id=41) -

  Sync time: 104.010406956 seconds

  Files: found=2484 add=0 modify=0 expunge=0 acl=0 ownership=0

  Folders: found=260 add=0 expunge=0 acl=0 ownership=0 processed=261 qsize=120

  Entries: expunge=0

  Failure count=0

If this is not the sync in question, then selecting “n” will page through the log file until you find the correct information.

6.3 Net Folders

Section 6.3.1, Net Folder Security

Section 6.3.2, Access to a Net Folder

Section 6.3.3, Understanding the Implications of Re-sharing

6.3.1 Net Folder Security

Within each Net Folder, security can be assigned to the following:

Internal Users and Groups

This refers to users that are created within Filr and to users that are imported to Filr from LDAP.

External Users

This refers to users who are configured via the external share mechanism and are referred to by their email address.

Public

The groups and users referred to here can be either Filr-based or derived from the LDAP import process.

6.3.2 Access to a Net Folder

Even though users can be granted access to a Net Folder from the dialogue above, the final arbiter of security is the file system that the Net Folder refers to. The user must also have rights to the back end NSS or NTFS file system before any content is seen through Filr.

6.3.3 Understanding the Implications of Re-sharing

The access granted to a user can also include the ability to re-share content. If re-sharing is allowed, then the re-share right could allow a user to share a file that the user has no existing file system permissions to in the first place.

Sharing access is provided by the Net Folder proxy. If shared content is subsequently modified by a user that is granted shared access to the file, the Net Folder proxy user is logged at the file system level as the last modifier of the file. In this case, even though the host file system shows the Filr Proxy User as the modifier, the person who shared the file might be able to deduce who modified the file, if it wasn’t shared with too many users.

However, if re-sharing is enabled, and the content has been re-shared more than one level, then the originator of the original share has no ability to determine who the file has been re-shared with. The only way to reference this information is by querying the back-end database.

6.4 Backup and Restore of MySQL

If you deploy the MySQL Novell appliance as part of your Filr solution, then the PHPADMIN web front end can be used.

If not, then you might need to back up and restore from the command line on the MySQL Server.

The backup and restore process discussed here is not a “Hot Backup” and would require shutting down the Filr appliance. A “Hot Backup” process for MySQL is available from Oracle as part of the paid-for version of MySQL.

Run the following command:

mysqldump -u root -p filr >/backupdir/filrback.sql

This creates a file named filrback that can be used to restore the database.

If the filr table does not exist, it must be recreated by logging in to mysql using the following command:

mysql -p

Then creating the database using

create database filr;

After the database file is present, quit mysql and run the following command:

mysql -p filr < /backupdir/filr-back.sql

This completes the restore of the Filr database and its associated tables.

6.5 Backing up and Restoring Filr Data

See Backing Up and Restoring Filr Data in the Filr 1.0.1 Administration Guide.

6.6 Setting the URL for Desktop Client Downloads and Updates

In order for desktop clients (Windows and Mac) to be deployed and updated, the Auto-update URL must be set to where the clients are hosted. This is a global setting and applies to all users in the Filr system.

Although the default URL points to the Filr appliance itself, Novell recommends hosting the desktop clients on a separate web server, as documented in Configuring a Separate Web Server to Deploy the Filr Desktop Application in the Novell Filr 1.0.1 Administration Guide.

For example, if the server host name is myserver.example.com, then a URL similar to the following would need to be set in the Web Administration Console (port 8443). Click Desktop Application in the left frame.

The desktop clients can also be hosted on Filr appliances, as documented in Configuring the Filr Desktop Application for All Users or for Individual Users and Groups in the Novell Filr 1.0.1 Administration Guide, provided that the appliances are not fronted by an L4 or L10 switch.

For more information on configuring the desktop applications, see Configuring the Filr Desktop Application to Access Files in the Novell Filr 1.0.1 Administration Guide.

6.7 Monitoring the Indexing Process

On the Filr appliance (or on any appliance in a Filr-based cluster), append the following line to the /opt/novell/filr/apache-tomcat/webapps/ssf/WEB-INF/log4j.properties file:

log4j.category.org.kablink.teaming.module.binder.impl.BinderModuleImpl=INFO

Restart the filr process with rcfilr restart.

After re-indexing has started, watch the ssf.log file for the following statement:

2013-06- INFO  [http-bio-8080-exec-1] [org.kablink.teaming.module.binder.impl.BinderModuleImpl] - indexTree took 1480.470827 ms

Check the results using grep indexTree /var/opt/novell/apache-tomcat/logs/ssf.log.

Look for the key words "indexTree took" as shown above.

This shows that the re-indexing previously triggered has now completed.

Please note that the indexing recommendations made elsewhere in this guide still apply, and a pair of Filr Search appliances should be deployed as part of your system. Start both Filr Search appliances as read/write and make them available to Filr Clients. Change one of the appliances to be indexed to Write and use that server for the re-index process.

This forces all the Filr clients to use the other indexing server.

After the reindexing process is complete, as indicated by the log file discussed earlier, the appliance can be changed back to Read/Write. Any deferred updates should be applied, and the second server can then be re-indexed by using the same process.

6.8 The File Upload Process

For the Filr desktop clients (Windows and Mac) and the web client, the file upload process is very similar. All file uploads are staged on the /vashare mount point in the /filr/temp directory. Within this directory, the upload process creates a subdirectory based on the Filr user ID.

Assuming a Filr-based clustered environment, the upload process follows the steps outlined below.

A file upload is initiated from a client.

The file is uploaded over a period of time to the /vashare/filr/temp/xx directory where xx is the Filr User ID. The file is created as a .tmp file, using one of two conventions depending on the source of the file upload.

If “.” exists in the file name, then the source of the upload was the web client using the HTML5 upload option.

If “_” exists in the file name, then the source of the upload was either a desktop client or the Java Applet from the web client.

After the file has been uploaded to /vashare/filr/temp/xx, the file is then moved via NCP or CIFS to the OES or Windows file server source of the Net Folder.

NOTE:All .tmp files in the /vashare/temp directory should be cleaned up automatically, even when an upload fails for some reason. However, if the move of the file to the OES or Windows file server starts but doesn’t finish, the incomplete file on the server is not cleaned up and must be cleaned up manually.

A.0 Documentation Updates

This section summarizes the changes made since version 0.8 of this guide.

January 31, 2014

Chapter or Section Changed

Summary of Changes

Throughout

Standardized fonts in illustrations for better PDF rendering.

B.0 Legal Notices

Novell, Inc., makes no representations or warranties with respect to the contents or use of this documentation, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this publication and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes.

Further, Novell, Inc., makes no representations or warranties with respect to any software, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to make changes to any and all parts of Novell software, at any time, without any obligation to notify any person or entity of such changes.

Any products or technical information provided under this Agreement may be subject to U.S. export controls and the trade laws of other countries. You agree to comply with all export control regulations and to obtain any required licenses or classification to export, re-export or import deliverables. You agree not to export or re-export to entities on the current U.S. export exclusion lists or to any embargoed or terrorist countries as specified in the U.S. export laws. You agree to not use deliverables for prohibited nuclear, missile, or chemical biological weaponry end uses. See the Novell International Trade Services Web page for more information on exporting Novell software. Novell assumes no responsibility for your failure to obtain any necessary export approvals.

Copyright © 2014 Novell, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the publisher.

Novell, Inc.

1800 South Novell Place

Provo, UT 84606

U.S.A.

www.novell.com

Online Documentation: To access the latest online documentation for this and other Novell products, see the Novell Documentation Web page.

Novell Trademarks

For Novell trademarks, see the Novell Trademark and Service Mark list.

Third-Party Materials

All third-party trademarks are the property of their respective owners.