OPTIMIZE

Introduction to common problems

There are a number of things that can cause a search engine to be unable to read and index your website or specific web pages correctly. In this section we will give you a brief introduction to some of the most important things to watch out for.

If you want to learn more about how to solve each of the issues, we highly recommend you visit Search Engine Watch [www.searchenginewatch.com]. If you do not have the time to learn how to optimize your website, or would rather have a professional do it for you, Position Tech has professional consultants [www.positiontech.com/consulting.htm] that can help you on a fixed budget or hourly rate.

Frames

Frames have always been a problem for search engines and some users to understand. Studies show that users have a problem performing even simple standard tasks that they would otherwise be able to solve with a similar website not using frames.

Jacob Nielsen, a well-known and respected usability guru, has some rather extreme views on the use of frames. He, however, does have some valid points, so we recommend you to read his article: "Why Frames Suck (Most of the time)" [www.useit.com/alertbox/9612.html] to understand the arguments against using frames.

With a website using frames, users can rarely bookmark specific pages deep inside the website. If they use the browser's bookmark function and click it later to return, they will end up on the front page - even if they did the book-marking from another web page deeper in the website.

If users can't bookmark pages within your site, neither can the search engines. If users can only bookmark your front page - as it is the case with most websites using frames - that is also the only page most search engines will link to.

Some search engines might try to browse and index individual pages on your website outside the normal frame set. However, that will leave you with a whole new set of problems to deal with. If the search engines send visitors directly to content pages that are supposed to be viewed inside a frameset it is usually difficult, if not impossible, for users to navigate further into your website.

To solve this problem, you can implement various sorts of scripts to re-fetch the content page in the right frameset if it is requested outside the frameset. You should refer to your programmers and JavaScript references for further instructions of how to do so. However, most of the related problems can be avoided by simply not using frames.

Some search engines read the content in the NOFRAMES section of a frameset. The NOFRAMES section is a special part of the HTML code that is presented to users with browsers that cannot read frames (e.g. search engines, very old browsers and homemade viewers).

The code looks like this:

<noframes>Place content here</noframes>

Browsers - and search engines - not supporting frames, can read the content you place between the two NOFRAMES-tags.

To use the space the best way you should:

• Fill the section with the same text that you have on the page visitors see. Just use the raw text without images - but never use text not visible to the users.
• Remember to add links in the NOFRAMES section too for the search engines to follow. Use the same navigational links as you do on the page you show your visitors but remember to keep them as normal text-links.

Using the two above tips does not in any way guarantee that your web pages will get indexed and rank well. Many search engines will not read the content in the NOFRAMES-section and follow links from there.

There are more advanced options available if you want a frames based website indexed and ranked well. The easiest solution, however, is still to get rid of the frames. So if you can do that, from a search engine point of view, you should do it.

If you want to use more advanced methods to make you frames based website more search engine friendly, we advise that you consult with an experienced search engine optimization expert or firm.

Two more ways of solving the problems of using frames are:

• Information pages
• Inclusion programs

Information pages are static HTML-pages (no frames) with information about key products or services provided on your website. There is one information page for each key topic. An information page is optimized with simple HTML, but otherwise looks and feels like the rest of the website. Creating such pages can, in some cases, produce search engine friendly entry points to your website.

If you can make additional entry points - except for your front page - you can use Pay For Inclusion programs to have the pages indexed in major search engines. You can use the inclusion programs even if the website is framed - as long as you can provide a unique URL for each target or entry page.

TIP:
Some search engines might compare the content you have in the NOFRAMES-section with what is found to be presented to normal users. If the search engines do not find the same text in the NOFRAMES-section as they do on the user-pages, you might get penalized for spam.

Dynamic Web Sites and Web Pages

Dynamic websites are very difficult for search engines to read. It is not because they are dynamic or driven by databases, but more because they use long and strange URL's (Internet addresses).

A traditional static website is made up of a number of individual files usually ending with .html - e.g. index.html, products.html etc. Each page is a unique file and usually has unique content.

On the other hand, a dynamic website very often has only one or a few files - so-called "templates." The template dictates how to present the content but holds no content itself. All the content is stored in a database. To show a page on a dynamic website you need to tell the template what to load from the database. Technically, that is done by adding so-called "parameters" to the file name.

If the template is called "pages.asp" and you want to load the content with ID #54, the URL could end up looking something like this:

www.YourDomain.com/pages.asp?page_id=54

That part is not too complicated. However, there is often more than one parameter attached to the URL, as the database needs to have more information about sort order, navigation, etc. This is when it gets complicated for the search engines.

There is simply no way for a search engine to be sure what parameters identify a new page and what parameters are just a sort order of the content - a navigation setting or something that does not justify indexing the new URL as a new unique web page.

There are many other problems related to having dynamic websites and websites built on content management systems indexed in search engines. It is unfortunately not possible to cover all of it in this tutorial.

The good news is that there are many ways and methods available for you to get your dynamic website indexed in search engines. Here is one method that is probably among the easiest to use:

• With Position Tech you can pay to have specific URL's indexed in Yahoo through Yahoo! Search Submit through the "trusted feed" services. That goes for dynamic URL's as well!
• You can have your dynamic website indexed simply by copying the URL of the page you want indexed into an e-mail, send it, and get to the same web page when you click on the link in the e-mail then you should be set.

For further information on how to get your dynamic website indexed, we recommend that you hire an experienced search engine optimization expert with expertise in these areas. Most often it takes a closer examination of your website to determine the best strategy.

Flash and Macromedia Director

Search engines do not read Flash and Macromedia files. So, content and links that you place in any of these formats will not be accessible to search engines.

You can read more about Flash in the Links and Navigation section.

Java Applets and Other Client Side Applications

Search engines basically only read regular text on a web page. They don't read text and follow links within special formats such as Java Applets or any other formats that require the user to have a certain program installed.

You can read more about Java Applets in the Links and Navigation section.

If you use image maps you should add the same links as you have in the image maps as regular text-based links.

JavaScript and DHTML navigation

Search engines do not read JavaScript. So, you should not put important content or navigation inside JavaScript unless you also have the same links and content duplicated as regular text and text-based links.

IP-delivery, Agent-delivery, Cloaking and Personalization

It has become more and more common for websites to implement some degree of personalization, regional targeting, or other forms of individualization in the way web pages are served to individual users.

In its simplest form, individualization can be a program on the server that checks what browser people are using, and serve up a specially tailored version for that browser. The same kind of program can also be used to check if people have specific plug-ins installed so that they get a version of the website that they can read.

A more advanced example would be to check what country a user is coming from in order to serve a local version. Some portals and search engines have been doing this all along, as well as other cross-national websites. There are many legitimate reasons why you'd want to account for this, including: business strategies, marketing, and legal issues with products only allowed in certain countries.

Furthermore, the same techniques can also be used to serve different content to search engine spiders and normal human visitors. The techniques used for this are usually referred to as "cloaking."

In most cases search engines do not like the use of cloaking. We recommend that you do not use cloaking unless you have a very good reason to do so, that you fully master the necessary techniques and that you understand the possible consequences of your work if you do not have the search engine's approval to do so.

Cookies

Spiders do not read cookies. Therefore, if your website uses cookies you have to make sure that all the web pages you want to have indexed can be accessed without accepting the cookie.

A cookie is a small text file that a web server can save in a users' browser for later retrieval when that same user comes back. It can be used to store login information or other preferences to make it easier or better for users to use a particular website.

Cookies are very safe to use in the sense that they cannot be read or shared across different users or websites. If a cookie is set on my browser the website that wrote it is the only website that can read it. Also, other users of a website cannot get to the information in my cookie - it can't be transferred or shared.

So even if the search engine spiders did accept cookies, there would be no way for them to re-fetch it in the users' browser when they click a link in a search result. Consequently, there is really no point in writing cookies for the search engines or sending cookies to the search engines.

Tip:
You can turn off cookies in your own browser to test if your website can be accessed without them. Refer to your manual or help files of the browser you are using for how to turn off cookies. Usually it will be somewhere under "advanced options."

Back to Optimize