Selling Successfully (Your One-Stop Resource for Setting Up, Managing and Marketing)

Search Engines - An Intro To How They Work

For some reason, the subject of search engines and what they do reminds me of the old Tonight Show skit, "Karnak the Magnificent", in which the comedian Johnny Carson, dressed as a 'mystical soothsayer and seer', would touch a sealed envelope to his turbaned head and pretend to mysteriously divine the answer to a hidden question written on a page enclosed within. He would then solemnly pronounce a word or string of words, with eyes closed, all would hang expectantly in the air until he opened the flap of the envelope to reveal the secret question. Always, of course, the result would be a pun or a play on words. Had the answer he prognosticated been "Search Engine", for instance, the page within the envelope might have revealed the question, "What a man does after his wedding ring slips off while he's checking the oil."

How does this relate to the subject at hand? Even now, with all of the improvements and the continual upgrading being done by the various providers of search engine technology, a random word search can still often return humorous, unpredictable or undesirable results to the searcher. This is true because most searchers offer only a single word or a small combination of words and expect the search engine to 'divine' what relevancy the results returned will have in relation to what they actually had in mind. If the word "Apple" has been entered, how can the search engine possibly know if the searcher is looking for a reference to "Apple Computers", "Apple Growers", or a copy of the film "The Apple Dumpling Gang"?

Add the problem of deducing relevance to the fact that every search engine will retain stale references within its database until they can be updated, and that it is estimated that all of the many search engines combined are currently indexing only about 20% of the information available on the Web, and it does start to seem a bit like magic that a particular website or web page can get noticed at all, much less have references to it presented to the right searcher at the right time!

What can you do to improve the possibility of your own shop's pages being indexed and displayed as results to interested Web searchers and customers? Understanding even a little of how search engines work, can help. Trouble is, how any given engine works today is subject to change always, as the companies behind them seek to improve functionality and still stay one step ahead of those unscrupulous search-hog types who discover and abuse their search parameters. And as if that isn't bad enough, search engine companies are also busy acquiring each other, combining, and changing the way both they and the other guy used to work.

Some search engines which previously relied only on listings supplied by the human editors of directories, have begun moving away from that sole source in order to use combinations of algorithmic tools and, increasingly, information supplied by sites who have simply paid for search placement. In fact, paid placement and paid inclusion can, in some search engines, have such a dramatic effect on how results are displayed to consumers that the FTC has felt the need to post a consumer alert on their website. This recent rise in the popularity of paying for placement has many wondering about the future of search tools altogether and certainly does not appear to bode well for the operators of small stand alone retail or personal websites who have always relied on receiving search engine indexing for free. Presently this is still a given, but what the future holds in this regard is a question.

For now, as Steve Berkowitz, President of Web Properties at the search engine "Ask Jeeves" suggested in a recent interview with the Reuters news agency, Web search is still in its infancy and the technology remains imperfect. So a website owner's best chance of gaining and maintaining placement in a search engine database lies in trying to work within the current scope of search engine technology, while remaining flexible enough to be able to change methods in use at the drop of a hat. With this in mind, here is some of what you need to grasp in order to help yourself, for the present.

There are three sources from which search engines extrapolate, and usually combine data: These are through Directories, Algorithmic Device and Paid Placement. Here I am going to address the first two of the three.

As far as directories are concerned, you should first concentrate your efforts on getting your shop added to The Open Directory Project (ODP). The ODP is a library, if you will, of websites that have been reviewed by actual human beings(!). Said websites have been individually checked for accuracy of submission and content and have been 'chosen' for inclusion. ODP is the directory currently used by Google, and various data within it is used by a majority of other major search engines as well, including Netscape Search and AOL Search. For encouraging search engine visibility, then, submitting your shop to the ODP is perhaps the most important directory submission that you, as a shop owner can make.

The good thing about being in any directory is that once accepted, you don't usually have to do much else except continue to maintain your site relevant to the section within which you have been placed. The bad thing is that changing the content or wording within your submission usually can't be done after the fact. So do your directory submission right the first time; make it a brief, factual and accurate reflection of your shop, while at the same time utilizing as many keywords that apply to your items as possible. You can find detailed directions for how to submit your shop, by visiting the shop registration/information page here:,article=ss20030415.html The ODP is not a directory that Ruby Lane can submit for you, because they require individual submissions, so do take the time to make your own written submission to ODP (you may have to do it more than once - see the directions on the above linked page) in an effort to get reviewed and included.

It should be clearly understood that even after going through all the motions correctly, it is no easy task to get into these directories. The editors are overloaded, where there are editors, and it can take many months to get listed. You can imagine the literally millions of submissions they get!

Another source search engines employ to compile a database is the Algorithmic Device, which utilizes automated data retrieval programs, or 'robots', which 'spider' the Web by visiting previously visited pages, newly submitted URL's in queue and by following links between pages. This last method is one of the reasons why building links between your shop and similar sites and increasing your Link Popularity is important.

Search engine robots have, for the most part, limited functionality. There are some things they simply cannot do, and other things that confuse them, and certain aspects of a website that can stop them dead in their tracks. When a robot program does find access to a page, its function is to look at the visible text on the page itself, view the tags in the source code (hidden document code), and scrutinize any hyperlinks it finds. From all of this, in a seemingly magical way, the robot concludes what the page is all about and then stores the information. It then returns to its home base, bringing all the juicy tidbits of knowledge it has ingested, along with it.

The actual intelligence behind any search engine lies not in the robot, but in the method used to evaluate all the information it returns. Every search engine uses different calculations and parameters to decide what information presented by its robots matters - and how much it matters - relevancy being the God of all search engines, after all. The information thus deemed of significance is then entered into the search engine database, either as newly acquired knowledge or as an update to previously stored data about individual pages. It is then further subjected to the ranking process, after which it is finally ready for immediate retrieval and display on a results page, when requested by a search engine user.

Remembering how Algorithmic Devices are supposed to work should be of some value when it comes to getting your shop and its current items more readily displayed. But do keep these two things in mind: First, the World Wide Web is huge and all search engine robots take time to re-visit any given site. Many go out only on a set schedule. This is what necessitates the 'stocking of the shelves' approach used by search engine databases - it is simply the most expeditious way - working within the constraints of time itself - to supply information to their customers. Thus, any time you add or remove an item or a catalog, or have an item that happened to be "On Sale" when the robot initially visited, but which now is not, these new aspects about your shop or your items may not be apparent for a month or more, depending on when the robot returns. And in fact, if the new information is not chosen by the robot as significant enough to retain when it does return, or if for some reason it is unable to reach the information or store it, the old information will not be updated and the old information may continue to be given out in subsequent search results. You can hope this is taken care of in the next indexing cycle, whenever that is, but this is not guaranteed.

Also keep in mind that there is no guarantee that previously stored information will be kept within the search engines database on a month-to-month basis. As far as we can tell, search engines do not want to retain information on a month-to-month basis because the information on the web is constantly changing. If a search engine indexed an item page and left it there indefinitely, the search engine would be clogged with obsolete pages. Additionally, your ranking may have dropped for some reason or other sites with higher rankings are now indexed and their web pages are being displayed prior to your own on a search. So, it is possible that you may be able to see any particular item in your shop as ranked #5 in a search string one day, as returned by a given search engine, and then not be able to find that same item displayed by the same search engine the following month. As has been illustrated, there can be a multitude of reasons for this to occur. When and why any particular website or item is either indexed or displayed is strictly up to the search engine itself and neither you, nor Ruby Lane has any say in the process. What you can do is try to optimize your shop to make it more likely that your items will frequently appear.

What does all this suggest for those who seek higher search engine placement? That getting into Directories, however difficult, matters greatly. That linking from (not to) other similar sites out there on the Web is important. And that making your pages easy for robot data retrieval programs to access, and making the keywords or phrases necessary to help ascribe relevancy to your items handy for them to grab on their visit – are crucial.

Which means we've come back around, full circle, to the word or words a searcher is likely to put into the search box when seeking any information on the Web. All other aspects being equal, it is your use of keywords - both on your homepage and on your many item pages - that will help bump your site and your stuff upwards in any given search string. Often it is the first paragraph or so of the text on your homepage, for instance, that a search engine will use for its overall description of your website. You should try to utilize targeted keywords, therefore, within this area, while at the same time making the resulting text both legible and descriptive. In other words DO NOT needlessly repeat keywords, or list them merely as a string of unrelated words separated by commas. Some search engines perceive this practice to be an attempt to spam and it could affect your ranking. On the other hand, while telling your customers all about yourself and your family in the first paragraph on your homepage might lend you credibility as a dealer or make you seem friendly and approachable, such information imparted in that area is not likely to easily provide the keyword rich 'first touch' needed to feed the robots and help bring appropriate outside visitors into your shop.

Shari Thurow, author of "Search Engine Visibility", stresses again and again in her book the importance of keywords. Figuring out the ones that best relate to your business or to an item, for use in its title and description, is paramount (again, this is the reason why researching your items can be the key to selling them more quickly); but actually placing the keywords on the page in such a way that they will be easily accessed by a search engine is also very important. So think of text that you place on your web pages as a sort of electronic conversation - you are talking to the search engines. And just like a discourse between spouses, it isn't only the words that are spoken that have meaning, but the manner in which they are said, as well.

If you'd like more information or want to remain up to speed regarding how search engines work, here is an ongoing forum of potential interest: