Picture, if you will, a Rocky-style montage. A team is involved in website creation from the ground up. A driving, inspirational song begins. The first scene is a highly contentious meeting, with a sweating and nervous marketing executive frantically drawing away at a whiteboard in front of a hostile audience. Flash to copywriters, fingers cracked and bleeding, churning away at their keyboards. Jump to web designers, sporting blurred eyes rimmed with dark circles, peering into their monitors in obvious discomfort. Finally, we see signs of it all coming together. A beautiful home page briefly appears. The music ends. A bell rings. And…

Nothing happens. The website would seem to have tripped over its own shoelaces and fallen right through the canvas, disappearing into cyberspace. The credits roll, to the eternal shame of those whose names appear.

What went wrong?

The team engaged in website creation without any regard for the role of the search engine spider. You see, there is quite a difference in what is seen by humans on a website and what is seen by a search engine “spider” (a program that routinely combs the Internet indexing websites). There are an untold numbers of expensive websites out there that are beautiful to behold from a human perspective, yet all but invisible to search engine spiders (and thus searchers), just as there are untold numbers of expensive, beautiful yachts silently and pointlessly resting on the ocean bottom in the Bermuda Triangle.

What follows is a small list of common website elements, broken into two categories: what search engines can see, and what they can not.

Three Things a Search Engine Spider Can’t See

Graphic text - Most professionals involved in website creation take great pride in their work, which is obviously a desired trait. Occasionally, however, this can present problems. When a web designer decides to use text in a graphic form (meaning that the text is actually an image), the search engine spider can not read what that text says. A common reason for a designer to use text in a graphic is because he or she wants to use a rare font that most visitors won’t have on their machines. Another reason is that the designer wants to have absolute control over how the website text renders. When faced with the choice over which kind of text to use, it is important to weigh the aesthetic choice against the potential loss of search engine visibility.

Images - As touched upon above, a search engine spider is not yet able to look at images or pictures and determine what they are (although you can and should attach a tag to them which the spider can read - commonly referred to as an “alt” tag). A spider will skip directly over your logo and masthead, any pictures, and most other graphical ele

ments.

Flash - Search engine spiders will not read through the text in any Flash animation on your site (or any other animation). This does not mean that using Flash elements will render your site invisible; it merely means that you should not count on the text that appears in any Flash animation on your website to be indexed. If the team responsible for your website creation decides to build the entire site in Flash, however, you will encounter unique problems. While some search engines are getting better at trying to index websites built entirely on this platform, it is still an overall rankings killer. If you must have a website created entirely in Flash, it is wise to also have an alternate HTML version for search engines and people who prefer HTML sites.

Three Things a Search Engine Spider Can See

HTML text - A search engine spider relies heavily on HTML text to determine what a web page is about. Spiders, therefore, index HTML text and will even make distinctions between differences in how the text is presented. For example, text that is in a headline or is bolded is assumed to be slightly more important than regular text.

Links - Outgoing links on your pages are easily understood by the spider, especially if they are text links. The wording of these links (or the alt tags attached to them) can, like HTML text, give the spider an idea of what your page is about. However, there are certain types of links that are not easily indexed. If your website creation team embedded your links in a pull-down menu that utilizes JavaScript or other scripting language, most search engine spiders will not see them.

Tags - There are many kinds of tags, but not all are important for search engine optimization. Meta tags include the “keywords” tag, which should list keyphrases that describe the page. Another meta tag is the “description” tag, which should be one or two brief sentences that describe the page. Another tag, which is not actually a meta tag, but which has significant importance to search engine rankings, is the “title” tag. This tag contains the words that you will see in the blue bar at the top of your web page.

This list is by no means comprehensive - there are many other attributes that aren’t mentioned in this article. The primary message here is that companies should do their homework before engaging in website creation. There are thousands of resources available on the Internet that can answer your questions about any element you are considering adding before you build (or redesign) your site. Take the time to study each so that you can be sure you aren’t sacrificing your search engine rankings for the sake of something largely unnecessary.

As for the downtrodden website creation team mentioned earlier in the article, let’s remember that Rocky has been known to get beat down in the first fight but then to pull himself up from the floor and triumph in the end. The team will go back into training and eventually understand the Eye of the Spider. Cue the music…

source : http://www.isedb.com/db/articles/1706/1/Website-Creation-and-the-Eye-of-the-Spider/Page1.html

Not every site needs a site map, they can certainly be a good idea. Site maps provide a dual purpose: They provide search engine spiders easy access to all of your site pages and they provide site visitors easy access to all of your site pages. The difference is that search engines and visitors access your site map differently and therefore there are different methods that need to be applied to creating site map(s) that are friendly for both engines and search spiders.

Small sites typically don’t need a site map so long as all pages are linked in the main navigation. Once you get into main and sub-navigation menu’s then site maps are helpful in allowing search engines and visitors to quickly find anything they are looking for within just a couple of clicks. A single site map can be used for both purposes or multiple site maps can be created. Here we’ll address creating site maps for spiders and humans separately.

Site Map For Spiders

.xml file

An .xml document should be added to your site’s root directory containing links to all site pages. This .xml file should then be referenced should be compiled and placed into a proper .xml document which should be uploaded into the root directory.

Robots.txt file

With your .xml site map file in place you must then make it accessible to search spiders. Reference the site map in your robots.txt file by adding a line for sitemap: URL (example: sitemap: http://www.polepositionmarketing.com/sitemap.xml)

Keep current

Be sure that your .xml file is updated and uploaded frequently, or at least as often as pages are added or removed from your site. Large sites should i

mplement an automated site map update monthly or even weekly or daily.

Site Map For Visitors

Navigation links

A site map designed for human visitors is just like any other website page. Link to the site map page should be included in the primary navigation or the site’s global footer. Visitors should be able to find this link without too much searching.

Additional page links

Site map should also be linked from various pages within the site such as Help pages and your custom 404-redirect page. This helps point visitors to the site map as a quick and easy means to find what they need.

Overview

It’s helpful to provide a short overview paragraph at the top of your site map page. This can be a nice introduction should someone land on this page directly from a search engine or elsewhere.

Heading and layout

The layout of the site map should present a clear visual hierarchal structure or your website. Both headings and sub-headings should be used above properly grouped links.

Text links & descriptions

Site map should primarily use textual links and page should not be cluttered with images or other distractions. It is also a good idea to provide an additional short description (provided there is room) for each link that allows the visitor to better understand where each link will take them.

Keep current

Be sure that your site map is updated and uploaded frequently, or at least as often as pages are added or removed from your site.

As I said above, not every site needs a site map. But those that do should be sure that the site map(s) they create are actually benefiting them. And the best way to do that is to make sure your site maps provide maximum usability for visitors and search engine spiders.

source : http://www.isedb.com/db/articles/1800/1/9-Tips-for-Creating-a-Site-Map-for-Visitors-and-Spiders/Page1.html