In the past few years, comparison shopping engines have become increasingly popular among online shoppers. All the major internet companies, including Google, Yahoo, MSN, CNET and AOL offer comparison shopping. Services like Shopping.com, Shopzilla, Bizrate, and MySimon have become well known. There is also an ever increasing number of small startup companies that are trying to build better shopping engines in order to carve themselves a niche in this expanding market.

One of the reasons that there are so many different comparison shopping systems available is that it is harder to build a shopping engine than a general web search engine. So far, no single company has been able to create a technology to dominate comparison shopping in the way that Google has come to rule the web search market.

Usually, when a consumer uses a comparison shopping engine, what he or she really wants is one of two things:

1. To find the lowest possible price on a specific product (from a reputable merchant).
2. To find information and reviews about a certain range of products in order to make a selection.

Currently, most shopping engines are focused on finding pricing, although some also try to include information and reviews. However, there are also a few newer engines, primarily from small startup companies, that are focusing specifically on searching for product information and helping consumers choose the right product.

You might think that a strict price comparison would be a fairly exact science. After all, it should simply be a matter of comparing prices among vendors and listing the results. However, if you try searching for the same item on several price comparison engines, you'll find that the results can vary widely.

In fact, it quickly becomes clear that in order to have any confidence that you are finding the best value, you have to try more than one shopping engine. So it becomes a matter of comparing the comparison results. To understand why this is the case, it helps to take a look at how the shopping engines work.

If you set out to build a comparison shopping engine, there are basically two ways you can approach the problem. The first way is to "crawl" merchant (online store) websites in much the same way that a search engine like Google examines and indexes the information in web pages. This means that you build a software program to visit the merchant websites, analyze the HTML pages and find the information and prices for the products they offer.

While crawling works great for general web search engines, it much harder to build a comparison shopping engine this way. That's because the information on and the format of merchant sites varies widely. The product description and even the product title can vary among merchant websites, so that it can be difficult for the shopping engine to even be sure that it is comparing identical products. Also, any time that a merchant changes the format of its website, the crawler may stop working properly. Of course, a crawler needs to continuously visit the merchant websites in order to insure that information is up to date.

Some comparison shopping engines use crawling for at least some of their results, but the difficulties with this method have led most of the established shopping engines to use the second method: "data feeds" from the merchant stores. Data feeds are basically special data files that the merchant prepares and makes available to the shopping engines.

The advantage of using data feeds is that all the information is provided in a nice format which is easy for the shopping engine to understand and analyze. It's much easier, faster and more reliable for the shopping engine to get its information from a data feed than by trying to crawl a merchant website.

While data feeds have important advantages, they also have some big shortcomings. Since they are prepared by the merchant (and that can take a lot of work), they are only as up-to-date as the merchant makes them. In some cases, the information in the data feed may not be the same as on the merchant website. Also, in many cases, the merchant won't bother to include ALL the products it offers in the data feed, instead focusing on the most popular or profitable products.

Another issue is that there is currently no standard format for the data feeds; every search engine has its own data feed format. This makes a lot of work for the merchants, and also means that there needs to be a relationship between the shopping engine and the merchant in order for the proper data feed is made available. If a shopping engine is new or does not yet have a significant level of usage, some merchants may not be willing to expend the effort to make a data feed available.

So you can see that both crawling and using data feeds creates problems that can lead to widely differing results. Crawling because of technical problems with getting the information, and data feeds because the information may not be up-to-date, or complete, or even provided at all to some shopping engines.

A final factor that can cause results to vary is the sophistication of the shopping engine in dealing with issues like shipping charges and sales tax. Engines that do a good job of this, will of course, give more accurate total cost results. Given all this, it's not surprising that the different shopping engines may come up with completely different results for exactly the same search.

Once the shopping engine finds the information you are looking for, the next issue is how it will be presented to you. Again, you might think that the shopping engine will always list the lowest price first--after all, that's the point isn't it?

However, again you'll find that this is more than often not the case. The reason for this has a lot to do with the way that shopping engines make money. In the vast majority of cases, the shopping engine charges the merchant every time you click a link that takes you to the merchant website. And different merchants may be willing to pay different amounts to get that click. In these cases, you will very often (and very naturally) find that results from the merchants that are willing to pay the most are listed FIRST--regardless of the price.

Depending on the shopping engine, you may need to re-sort the results, or click a link to bring up more results before you really get to see the lowest price. And of course, if the merchant which truly offers the lowest price does not have a business relationship with the search engine, it may not appear in the results at all.

While this type of per-click payment is most often the rule, especially among the larger engines, some shopping engines are now using other models such as commissions on actual sales or revenue from advertising. Once again, the important lesson is that you really cannot rely on a single comparison shopping engine to give quality, unbiased results, You need to shop with multiple engines.

One of the hottest areas currently being pursued in the comparison shopping field is local search. Both formal studies and general common sense show that in many cases, consumers look online for information and prices, but then actually purchase at a local retail store. If the total cost difference is small, why wait for a product to be shipped if you can get it right now? And of course, buying online is great for a new digital camera, but maybe not so good for a big screen TV or a new desk.

While both established engines and startups are working hard in this area, the problems are significant. The issues with getting at the data are much more difficult than with online stores, and since the purchases occur offline, it's not clear how the shopping engine can make money at the game. Once again, especially since all the offerings in this area are really experimental, it pays to try more than one shopping engine.

Finally we arrive at the real point of all this: You really need to use more than one comparison shopping engine if you are serious about finding the right product and the best price. And that is the point of our service, RoboShopper.com!

You can do the work manually, finding and visiting a variety of product review sites and price comparison engines, and then typing in your search term at each site. That's a lot of manual effort.

With RoboShopper, you simply enter your search keyword(s) once and then click the Shop button. You can then use RoboShopper's navigation bar to automatically search multiple comparison engines without surfing around or re-entering your keywords.

RoboShopper will take you to sites which specialize in product information and reviews, as well as online price comparison services, and the new local shopping engines.

RoboShopper is unbiased in the sense that it simply acts as a navigation tool to help you work with comparison shopping engines. RoboShopper does not derive any revenue from links that you click or advertising on the search result pages (any revenue from clicks or ads goes to the specific shopping engine you are using).

We hope you will find RoboShopper to be a useful tool that will make online comparison shopping more rewarding and efficient.

Try RoboShopper!

More about RoboShopper