Adding Search to Your Website

One of the most useful, and often underappreciated, application that you can put to work on a website is an internal site search box. The problem with search engine traffic and, in fact, any traffic source, is that most of your visitors land up on the home page. Once there, if they don't see what they're looking for, its goodbye. This problem can be solved, or at least minimized, by putting a search box, which searches pages of your site, and outputs pages which match the visitor's requirements.

There are several ways by which one can put an internal search box on every page of your site, easy and fast, without using too much hosting resources, and installation headaches or cost. Continue reading to get a full search engine working just for you....

Before we build a full fledged search engine for your site, with the power to search through thousands of pages on your website, maybe we should try out a few solutions which allow you to use existing search engines like Google to output pages only from your site, or a given keyword. Almost all search engines have API's which allow you to put such a search box.

Custom internal site search solutions:

• Google - http://google.com/coop/cse/
• Yahoo - http://builder.search.yahoo.com/m/promo
• MSN Live - http://search.live.com/siteowner

These are fast, easy and free solutions for internal site search. The only question is, do the search engines have current and uptodate caches of all your pages? If they don't have information about one of your pages, that page won't be appearing in the results. For sites which add new content daily, which the search engines may or may not be picking up on time, your search box will always be redundant.

So, use these search boxes, if either Google or Yahoo or MSN Live is picking up every page of your site, as soon as it is online. If it takes weeks for a page to enter search results, not much point.

In that case, what you need is a full fledged internal site search engine. Which brings us to XAV.com's FDSE, or Fluid Dynamics Search Engine. This script is a marvel of technology, which works as smooth as butter and is highly accurate and efficient. It also has custom features, like keyword based ad banner slots. So, if your visitor types in a keyword, the search engine will show relevant pages from your site in the results page, along with strategically placed banners which change as per the typed keyword. And you have two ways to install it. Either you can download the script and do the installation, or you can simply run it from the xav.com website, and it will install itself on your FTP location. There's a trial period for the full version, which means you can test run your search engine and see if its up to the task, and if its working, you can buy a license. Can't get better than that.

When I was testing it out, in addition to searching my entire site once a day for new or updated pages, I also set it up to spider a few sites which I considered important. That was until my web host called up and started screaming and threatening to shut down my little spider.

What happened, apparently, is that the FDSE spider automatically follows links. So if you set it to spider only pages on your site, it will follow links which lead only to pages on your site. But when you set it to spider links outside your website, and follow deep links, what happens is that the FDSE spider goes on a massive fishing expedition, spidering, indexing and creating caches of thousands and tousands of sites and pages. And since I set it to start searching everyday on its own, the FDSE spider used to wake up bright and early everyday and create havoc on the server.

Anyway, point is that its an overwhelmingly powerful script, and if you make correct use of it, it can prove to be an invaluable tool, and will definitely make for a better user experience.

What you need to look for in an internal site search engine:

• Accurate and uptodate results
• Keyword based ad banners on result pages
• Easy installation
• Low use of server resources
• Automatic spidering, meaning the search spider should be able to detect new or changed pages, and add them to results, or update results.
• Unlimited pages