This article introduces one of the little-known Novell gems, QuickFinder Server 4.0. You've probably used it many times because it's the search engine used, without alteration, on novell.com and the award-winning Novell Support Web sites. The best part is that you probably already own this product as it ships with Novell Open Enterprise Server on both NetWare and Linux platforms and the technology has been shipping with NetWare since version 5.1!
You need to read this article if you've ever found yourself in any of the following situations:
- You currently don't have site-wide search functionality installed on either your external, customer-facing Web site or your employee intranet.
- You have search capability installed, but often can't find the information you need.
- Your search engine indexes your Web sites, but it can't index the files stored on your LAN servers or vise versa.
- You're currently spending thousands of dollars for your search technology—unnecessarily.
- You want to learn more about QuickFinder Server and how it can improve both your Web site navigation and help your users find information across your enterprise.
- You're curious about how Novell set up QuickFinder Server on novell.com.
A Little Background
When NetWare 5.1 shipped, the name of this technology was NetWare Web Search Server. The name recently changed to QuickFinder Server for the following reasons: it ships with Open Enterprise Server, it is now fully ported to Linux and its broad scope also includes File Server content.
For additional detailed tips and tricks using QuickFinder Server 4.0, visit the companion article about QuickFinder on the Novell Cool Solutions Web site.
Now at version 4.0, QuickFinder Server is a remarkably feature-rich, high-end search engine. Although it costs practically nothing, it competes favorably with the best search engines available from IBM, Microsoft, Google, AltaVista, Autonomy, Verity and many others. The good news? You probably already own the technology that gives you the search engine functionality you need without spending thousands, even tens of thousands of dollars annually. It's QuickFinder Server!
All the Standard Features
QuickFinder provides all of the features you'd expect from just about any Web-based search engine. It downloads the files from your Web site, generates a highly organized index of those files, and then allows anyone to search against those indexes to quickly find the original documents on your Web site. Although I call these standard features, QuickFinder supports them better than most search engines.
- Query Language / QuickFinder provides all the standard search types: Boolean, phrase, fielded, filename, date, etc. It also goes well beyond the standards supporting almost 50 query parameters, switches and operators.
- Document Types / QuickFinder supports more than two dozen file formats: Word, WordPerfect, PDF, HTML, XML, PowerPoint, Excel, Open Office, etc. New Readers are regularly added and the existing ones are improved all the time.
- Languages / QuickFinder indexes documents written in almost any language: Arabic, Hebrew, Greek, Russian, Chinese, Japanese, Polish, English, French, German, etc. Besides the more than two dozen supported languages, QuickFinder also recognizes more than 50 character set encodings and more than 500 encoding aliases.
- Templates / QuickFinder uses a template-driven search results interface. The HTML interface can be easily modified to match any corporate look and feel. Templates can also be created in other file formats. For example, QuickFinder ships with templates in both HTML and XML formats.
- Administration / Like the search results interface, QuickFinder's Web-based admin interface is completely template driven providing almost 100 HTML templates to search administrators to easily configure all QuickFinder features. (See Figure 1.)
- Search Box / Search administrators can easily integrate QuickFinder's capabilities into their corporate headers. To see how, visit the Cool Solutions companion article online.
"An enterprise with 1,000 knowledgeable workers loses $48,000 per week—2.5 million per year—due to an inability to locate and retrieve information."
-The High Cost of Not Finding Information, IDC
A Host of Enterprise Features
QuickFinder also offers a rich assortment of high-end search features such as:
- Index File Servers / Index any mountable file system throughout your enterprise. (See Figure 2.)
- Private Web Sites / Besides indexing public Web sites anywhere in the world, QuickFinder can also index any of your private Web sites. Both basic and form-based authentication are supported when crawling a site.
- ASP/ISP Hosting Services / QuickFinder can host search services for multiple, independent organizations from a single installation using its Virtual Search Server capabilities.
- Clustering / QuickFinder synchronizes configuration settings, indexes and search templates across multiple machines in a QuickFinder Server cluster. It also functions within a Novell Cluster Services environment. (See Figure 3.)
- Highlighter / QuickFinder quickly highlights users' search terms within the original documents—even if they're rightsprotected.
- Installation / Quickly set up an Open Enterprise Server server as a QuickFinder Server via the Dedicated Server installation.
- Duplicate URLs / QuickFinder "fingerprint" technology detects and eliminates a wide variety of duplicate URLs while indexing.
- Relevance / QuickFinder gives search administrators several ways to promote specific search results such as relevance adjustments, Best Bets, synonyms, stop words and redirection.
- Unwanted Files / QuickFinder provides many ways to skip unwanted files. It also provides a unique ability to skip useless portions of files such as common headers, footers, side and navigation bars and server-side includes.
- Query Reports / Search administrators can receive detailed weekly and monthly query reports. Query logs can also be imported into high-end report generators for sophisticated analysis. (See Figure 4.)
- E-mail / QuickFinder integrates easily with your e-mail system to send errors, warnings and reports.
- Speller / If a user's search terms produce very few results, QuickFinder automatically suggests alternate spellings allowing the user to find what they were looking for. (See Figure 5.)
You probably already own Quickfinder. It ships with Novell Open Enterprise Server (Netware and Linux), and it has been shipping with Netware since version 5.1.
Beyond the Competition
Literally hundreds of search engines exist in the world today. They range in value from free to $100,000 USD annually. Most come with significant limitations. Some of the common limitations include:
- limited capabilities that don't offer the features an enterprise needs
- limits on the number of files, users, transactions or CPUs
- poor management or administration interfaces
- poor reporting or error handling
- escalating costs as corporate needs increase
- inability to scale to an enterprise level
QuickFinder Server doesn't have any of these limitations. It installs easily, manages easily and scales well. The robust QuickFinder Server Manager module gives search administrators complete control over every aspect of the product. Feel free to index millions of documents across any number of servers and provide search capabilities to all users without limits. Again, the good news? You probably already own it. You get all that on a shoestring budget since it ships with Novell Open Enterprise Server on both Linux and NetWare platforms at no additional cost.
Almost 70 percent of all moderately sized Web sites (up to 5,000 pages) don't have a search engine installed. Fully one half of the large sites don't either. According to searchenginewatch.com, the top two reasons Web site administrators reported not installing a search engine were because search engines are difficult to set up and administrators don't have the time it takes to do it. (See Figure 6.)
QuickFinder Server provides all of the features you'd expect from just about any web-based search engine, and then some.
In contrast, QuickFinder Server installs easily. Simply select it from the Open Enterprise Server Customized Server screen or choose the QuickFinder Server pattern deployment option from the initial install screen—and that's it. The installation automatically detects your server's network settings and configures itself to work with the installed Apache Web server, Tomcat servlet engine and local directory services.
It works the same way on either Linux or NetWare platforms and by the time the server is fully installed, QuickFinder has already defined and generated two indexes:
- DocRoot, an index of the current server's Web content accessible from port 80, and
- QuickFinder Documentation, an index of the help files that accompany the product.
After installation, you only need to define and generate any custom indexes you want and modify the sample set of search templates so they match the look and feel of your Web site. You can modify every aspect of the product with the user-friendly QuickFinder Manager module, which is accessed on your Web site at www.yourdomain.com/qfsearch/admin.
QuickFinder on novell.com
Let's take a look at how Novell has implemented QuickFinder on the Novell internal and external Web sites. (See Figure 7.) To my knowledge, novell.com uses just about every feature QuickFinder has to offer. When implemented together, these features are an outstanding solution to every enterprise that needs search functionality. (See Figure 8.)
Quickfinder installs easily, manages easily and scales well. The manager module gives admins complete control over every aspect of it.
Because of the large number of searches (altogether, the novell.com Web sites perform about 4,000,000 searches per month), Novell dedicates two servers to regenerate each index daily, four servers to provide searching for novell.com and the Novell forums, and two servers for the Novell innerweb. All six search servers are fronted by an iChain box to provide a good level of load balancing and session persistence. The indexing servers are relatively low-end boxes: dual 800 MHz CPUs, 1 GB RAM and 17 GB HD. Although indexing is a very resource-intensive process, you don't need high-end hardware since the indexing is done on dedicated servers. As long as each index regenerates daily, how long it takes isn't that important. Of course, most customers wouldn't dedicate a machine just for indexing; so Novell suggests you regenerate your indexes at low-utilization times, such as during the nighttime hours depending on your location.
Simply select Quickfinder server from the Open Enterprise Server, or the pattern deployment option from the installation screen—that's it.
Searching is entirely a CPU-intensive process. As long as you have enough RAM to keep the indexes cached in memory, all you really need is a fast CPU. The Novell search boxes are currently dual 1.2 GHz CPUs, 2 GB RAM with 34 GB hard drives. I know what you're thinking: the processors should be a lot faster. You're right; but those same machines have been there for three or four years, and since the search is fast enough, there's no compelling need to upgrade.
All of the machines are currently running SUSE Linux Professional 9.2 with Open Enterprise Server, SP1. Novell recently switched from NetWare because QuickFinder runs faster on Linux! Novell installed QuickFinder using the pattern deployment option, which means these machines were configured from the initial installation to be dedicated to searching. At the time of this writing, nothing has been patched or upgraded to anything other than shipping code. In other words, everything done on novell.com can be reproduced in your own environment! You should know, however, that Novell often uses its innerweb and novell.com servers to test beta releases before they ship in service packs. As the adage goes, it's a good idea to "eat your own dog food" first.
Monitoring for Optimal Performance
Each QuickFinder server can monitor its own state. If a catastrophic error occurs, and the box is still otherwise functioning, QuickFinder immediately sends an e-mail notifying the search administrators with the details.
Also, the index servers send newly generated indexes throughout the day. If an index ever fails to generate, or if the indexes cannot be synchronized to the search servers, QuickFinder again sends an e-mail notifying search administrators.
When implemented together, Quickfinder's features are an outstanding solution for every enterprise that needs search functionality.
Finally, the servers are also monitored by an external process that sends an "average query" every two minutes to each server to make sure the servers are still properly responding. If any server takes too long to respond or doesn't respond as expected, search administrators are notified.
Admittedly, there's a bit of overkill here. One of the innerweb search servers and one of the indexers exists only to ensure a faulttolerant system. Also, four search servers is more than what was needed, but Novell wants to make sure each search query is resolved as quickly as possible, even if it means the servers aren't operating at peak efficiency. Besides, Novell is using relatively inexpensive Intelbased hardware, not the insanely expensive UNIX servers of the past.
Quickfinder servers monitor their own state. If a catastrophic error occurs, and the box is still otherwise functioning, Quickfinder immediately sends an e-mail notifying the search administrators with the details.
QuickFinder is a fast search engine. Stress testing one of the search boxes against an index of about 100,000 documents showed that it can respond to several hundred queries per minute, more than enough for most Web sites. If needed, the entire system could respond to more than 2,000 searches per minute. The average query on novell.com takes only .8 seconds and covers about 400,000 documents!
An "average query" is sent to each server every two minutes to make sure they are still properly responding. If a server doesn't respond as expected, search administrators are notified.
Configuring QuickFinder for novell.com
Each of the QuickFinder servers is set up with three Virtual Search Servers: novell.com, innerweb and forums. (See Figure 9.) Each Virtual Search Server (VSS) has its own indexes, scheduled events, configuration settings, search results templates and administration interface. Each VSS is completely independent from the others. In fact, you can't search in more than one VSS at a time. Novell initially designed the VSS capability for ISPs and other hosted environments, but it's also a great way for Novell to keep its main search areas separate while taking advantage of the same hardware.
Now let's take a look at the novell.com VSS. It currently contains 47 indexes ranging in size from only a few hundred files to almost 300,000. Deciding how to split up a Web site into indexes can be a tough decision.
On one hand, searching a single index to find a specific file is faster than searching through dozens of indexes for the same file. On the other hand, if you have a general idea of where that file is located on the Web site, there's no faster way to isolate, or search in, a set of documents than to organize related files into specialized indexes. It's a decision you make based on your particular needs.
The novell.com web site could respond to more than 2,000 searches per minute. The average query on the site takes only 0.8 seconds and covers nearly 400,000 documents.
For example, suppose a Novell customer wants to search for a certain TID on the Novell Support Web site using the following three keywords: ZENworks PXE and Imaging. You can either create a single index for the entire Web site so this customer can search the whole site at once, or you can create several indexes organizing all of the site's files into logical groups of related documents so the customer can search a subset of documents. In this example, you could group the ZENworks portion of the Novell Support Web site in a separate index. The customer could then search just that index and have greater accuracy in the results; for this customer, a search on a ZENworks-specific index would produce more efficient results because only pertinent support documents would be listed. The site-wide search would return a much broader result, possibly including press releases, documentation and success stories.
Novell suggests that no more than 10 to 15 indexes be used in any single query and that no more than 1,000,000 files be included in a single index. Of course, you can go much higher than these recommendations, but you'll lose a bit of speed if you do.
Listed below are some other reasons you might want to have multiple indexes; but remember, the more indexes you include per query, the slower the search gets.
- You want to apply rights to some indexes, but not others. Rights-based searching can be an expensive proposition depending on how you implement this feature. For high-traffic situations, or if you have more than 100,000 rights-protected files, Novell suggests you use index-level restrictions instead of file-level or path-level rights controls. (See Figures 10 and 11.) (For more information, see the Cool Solutions companion article online.)
- The credentials that QuickFinder's indexer needs to access your private Web sites may differ from site to site. QuickFinder allows you to define authentication credentials only for an entire index rather than each "Web Site to Crawl" separately, which is better than having an entire Authentication section per "Web site to Crawl." (See Figure 12.)
- Some of the "Additional Settings" on the index definition pages apply to only some of the Web Sites to Crawl URLs or Server paths, but not others. (See Figure 13.) For example, you might need to configure the following settings differently depending on the Web site being crawled or the purpose of the index:
- index weight
- maximum index depth
- maximum file size to index
- index may be copied to other clustered servers
You might have noticed that 47 indexes are far past the 10 to 15 I recommended. When search users select the All of Novell choice in the Novell corporate header drop down list, they're really only choosing seven or eight of the 47 indexes. In reality, the selection sends no indexes at all; instead, it uses the indexes specified in the Default indexes to search option on the Search Settings configuration page. In other words, if a user doesn't select any indexes, QuickFinder either searches in all indexes or in the subset defined in the Default indexes to search option.
One of the more significant features of QuickFinder is its Best Bets ability. You might have noticed when searching novell.com that it actually provides two search results lists. There's the regular list from the indexes that the user specified (or the default set); but there's also another list before the regular list that shows hits from a carefully crafted Best Bets index. Best Bets is just another search in another set of indexes. The magic is in how you define the Best Bets index. (See Figure 14.)
So what's a Best Bet? That depends. For novell.com, it might be marketing or highlevel product information. For a support Web site, it's probably the most popular or the most recent information. For your sales site, it could be your current promotional items. Either way, it's up to you to get these files into your Best Bets index.
But Best Bets is only one way to spotlight important documents. QuickFinder gives you many other ways to emphasize certain documents. Check them out in the Beyond Best Bets sidebar and see if implementing them works for your situation.
|Logo||Click the QuickFinder logo to jump to the QuickFinder site.|
|Supported File Formats||novell.com/products/quickfinder/fileformats.html|
|Support||To see all QuickFinder-related TIDs, visit search.novell.com/qfsearch/search?query=QuickFinder+tids|
|Desktop Background||Download a QuickFinder-branded background image for your workstation and OES server|
Another new feature you'll want to use is Off-Site Links. To understand this feature, let's first look at how the QuickFinder crawler works.
Although QuickFinder is an enterprise search engine, we use the term "enterprise" rather loosely. The term clearly refers to your own Web sites and file servers; however, there's nothing preventing you from including your partners' sites, or any Web site in the world, public or private, in your QuickFinder search. Basically, QuickFinder will index anything you want.
Normally, QuickFinder starts indexing a Web site using the URLs defined in the Web Sites to Crawl and Additional URLs found in the index definition. As it encounters links within the files, it compares these against the original URLs to make sure they're still within the scope of the index definition. If the link falls within one of the Web Sites to Crawl, it's added to the list of URLs to be downloaded and indexed. If it's not within the scope, it's simply skipped. This new feature tells the crawler to allow a limited set of these off-site links.
So why would you want to index files that aren't on your Web site? The best answer is because your Web site points at them. Someone has obviously taken the time to create a link from your site to a file on another site. That other file probably relates to something on your site. For novell.com, examples include:
- news articles on CNN and Reuters Web sites
- links to third-party products that run on NetWare, SUSE Linux or with other Novell technologies
- magazine articles about Novell APIs, products, services, management, etc.
- Gartner studies about directories and servers
- links to channel partners and resellers
- links to benefit providers' Web sites and services from the internal innerweb (life insurance, health care, dental, 401K plans, etc.)
You control both the depth of off-site links and the Web sites that should be specifically excluded from the off-site ability. Although this new feature is not yet implemented on novell.com, the Novell innerweb is drastically better because of it. (See Figure 17.)
I hope you've gleaned some valuable information about how QuickFinder works and about how best to deploy this excellent technology within your internal and external Web sites. While this article discusses in-depth several important QuickFinder features, others are mentioned only in passing. For more information about some of QuickFinder's other features and for a few best-practices tips and tricks, visit the Cool Solutions companion article online.
QuickFinder is a fast and comprehensive search tool that you probably already own. If you run NetWare or Open Enterprise Server—in any flavor—take this information and put this excellent technology to work for your enterprise.
Down the road, Novell looks forward to even greater benefits from this technology as it maximizes its value for Novell customers in future product releases.