When defining and creating your indexes, start with the highest possible Web site URLs and file system paths.

If content is showing up in your search results that you don’t want included, try removing some paths or URLs from your defined indexes. Also, try excluding specific subdirectories that you know or suspect might contain content that you don’t want searched.

If you’ve indexed too many file types and cluttered your search results, try removing file types that you don’t want indexed by using the Extensions to Exclude option on the Define Index page. See Creating an Advanced Crawled Index and Creating an Advanced File System Index for more information.

Use the Robots meta tag in your Web site’s content. See Using the Robots Meta Tag.

Exclude documents or specific sections of documents, including headers, footers, and navigation bars. See Section 10.3, Excluding Documents from Being Indexed.

Use the Robots Exclusions standard (the /robots.txt files are found on almost all Web sites). The Index Definition’s excludePaths settings are automatically combined with the robots.txt settings when crawling and indexing. You can turn robots.txt support on or off.