Why
Your Website May Not Be Indexed
Why Your Website May Not Be Indexed
At one time or another you may have used a submission tool, or
submitted by hand and then wondered why you had not been indexed.
Unfortunately, there are many reasons that may delay or prevent
you from being indexed by a search engine. There's rarely one simple
answer for why you're not being found. Fortunately, there is generally
an explanation and a way to correct the problem if you know what
to look for.
Below are the Top 21 reasons we've compiled over the years as to
why you may not be finding your Web site or Web page in one or more
search engines:
1. INDEX TIME: First, make sure you've allowed
enough time to become indexed. The amount of time to allow is sometimes
listed on the search engine's submission page. Unfortunately, the
engine's own advertised times are often inaccurate or out of date.
WebPosition's Submitter report and WebPosition's URL Verification
report will both tell you how much time you should allow before
being concerned about not being indexed. Average index times often
range from one to eight weeks depending on the engine. Some engines
like AltaVista and Inktomi offer paid options if you wish to be
indexed more quickly.
2. ALREADY INDEXED: Be sure you're not already
indexed but just don't know it. Unfortunately, none of the major
engines are kind enough to e-mail or notify you as to if and when
you've been indexed.
In addition, you cannot simply do a search on a keyword that applies
to your Web site and expect it to pop up at the top. In fact, you
must take pro-active steps to optimize your pages for each search
engine. If you don't, it's very unlikely you'll find your Web site
except on the most specific of searches.
The method to determine if a page or domain has been indexed varies
from one engine to another, and in many cases, it's difficult to
tell for sure whether your pages are in fact in there. Never assume
that you're not indexed just because you searched for a bunch of
keywords and you never came up in the first few pages of results.
You could be there (i.e., indexed) but be buried near the bottom.
In addition, it's not very practical to check the status of a number
of pages on each major engine each week. Fortunately, WebPosition
Gold has a URL verification feature in the Reporter that makes this
process much easier. Each time you run a mission, it will report
which URLs exist and do not exist in each engine. If you're using
WebPosition and are not finding your URLs after submitting, be sure
to see this page for common pitfalls to watch out for:
http://www.webposition.com/urlnotfoundhelp.htm
3. MISSING PAGE: Make sure you have uploaded the
pages to your site before submitting them. This one will seem obvious
to many people, but submitting a page that does not exist or submitting
with a subtle typo in the URL is a goof we might all make at one
time or another. If you're using WebPosition's Submitter, there's
a checkbox on tab 2 labeled "Verify that each page exists on
Web site before submitting." This option defaults so that WebPosition
will verify that all your URLs are valid and actually exist before
submitting them. This is important since not all search engines
will notify you if the URL does not exist when you submit.
4. ROADMAP FROM HOME PAGE: Some engines have been
known to drop pages that cannot be traveled to from the home page.
HotBot has been rumored to do this. You may want to consider submitting
your home page that links either directly or indirectly to your
doorway pages. Think of your Web site as a series of roads (i.e.,
links) from one page to another. If there's no road from your home
page to the page you want indexed, a search engine may decide the
page is unimportant or of low-quality. You could submit the page
directly, but the engine may reject it or may drop it at a later
date when it finds no "road."
5. EXTERNAL LINKS: Some search engines such as
Google and HotBot have been known to refuse to index Web sites that
do not have any other Web sites linking to them. Or, they may index
your home page but refuse to index any other pages until you achieve
at least one or more links from another domain. Or, they may index
you for awhile but then "prune" their database later of
all Web sites that did not achieve any external links within a certain
period of time. However, do not worry! You simply need to establish
some links and when that's done, resubmit both your pages and the
pages that link to you. Once you have links to your Web site, it
becomes much easier to get indexed, stay indexed, and to achieve
top rankings.
To check your link popularity and for tips on how to increase the
links to your Web site see our free service at: Link Popularity
Tool
6. FRAMES: If you have content inside HTML frames,
this can cause problems with submissions. For example, the search
engine may index the main content of the page, but not the surrounding
menu frame. Visitors to your site will then find some information
but may not see the associated menu! It's generally best if you
can create non-framed versions of your pages. You should then submit
the non-frames versions of your pages, which can of course link
to your framed Web site. Alternatively, you can enter your relevant
text within the NOFRAMES area of a framed page that most search
engine spiders will read. However, don't expect to achieve high
rankings while optimizing the NOFRAMES area. Optimizing a NON-framed
page will often achieve better results.
7. SPIDER BLOCKS: Search engine spiders cannot
index sites that require any kind of registration or password. A
spider cannot fill out a form of any kind. The same rule applies
regarding indexing of content from a searchable database. That's
because the spider cannot fill out a form to query that database.
The solution is to create static pages that the engines will be
able to find and index without performing a special action on your
site. Depending on the database system you have, there are utility
programs out there that help you do this, as well as companies that
can assist you.
8. FREE SITES: Many engines no longer index pages
from free web sites or they limit the number of pages they will
index from these hosts. Sometimes they will get too many "junk"
submissions from free web site domains such as Geocities or others.
Therefore, some engines choose not to index anyone from some of
these domains. Or, more commonly, they limit the number of pages
they will accept.
It's always best to buy your own domain name (very important) and
place it on a respected, paid service to avoid being discriminated
against. The free traffic you can generate from the search engines
is just too valuable to be sacrificed for the small savings a free
hosting service provides. In addition, free hosts are often unreliable
or force you to display banners that send valuable visitors away
from your Web site soon after arriving. That can cost you sales.
9. GUILT THROUGH ASSOCIATION: If your Web site
shares the same IP address as many other Web sites on your host's
Web server, then you may find your IP quietly banned from something
another Web site on the same server did! It's always best to ask
your hosting service if your domain name has its own unique IP assigned
to it. If not, ask them to move it to its own IP to avoid the potential
of having your submissions ignored because of something that a site
sharing your IP did. We've heard from many people who tried everything
to be indexed only to find it was a snap once they changed hosting
services.
10. SUBMISSION LIMITS: Make sure you're submitting
within the recommended limits. Some engines do not like more than
a certain number of submissions per day for the same domain. If
you exceed the limit, you may find that all your submissions are
ignored. Fortunately, WebPosition's submitter will warn you regarding
current limits and help keep you within them. Some submission consultants
feel it is dangerous to submit more than ONE page a day to an engine
for a given Web site. For those who wish to be ultra-conservative
in their approach, the WebPosition Submitter includes a checkbox
to limit submissions to one URL per day per engine.
11. DYNAMIC PAGES: Dynamic pages are often ignored
by the search engine spiders. In fact, any URL containing special
symbols like a question mark (?) or an ampersand (&) will be
ignored by many engines. Pages generated on the fly from a database
often contain these symbols. In this situation, it's important to
generate "static" versions of each page you wish to be
indexed. In regard to the search engines, the simpler the page is,
the better. Does this mean, for example, having a javascript to
count visits to the page will prevent you from being indexed, or
lower your rankings? No. It simply means that the search engine
will most likely ignore the javascript and index the remaining areas
of the page. There is evidence that going too far with fancy scripts
and code on a page can hurt your rankings if the bulk of your page
consists of java or VB scripts.
12. NON-INDEXABLE CONTENT: It's important to know
the types of content that the average search engine cannot index.
Most engines cannot index text that is embedded in images. Text
that appears in multimedia files (audio and video) will not be indexed.
Most engines cannot index information that is generated by Java
applets or in XML coding.
13. LARGE PAGES: If your site has a slow connection
or the pages are very complex and take a long time to load, it might
time out before the spider can index all the text. For the benefit
of your visitors and the search engines, limit your page size to
50K or less. In fact, most Webmasters recommend that your page size
PLUS the size of all your images on the page should not exceed 50K-70K
total. If it does, many people on dial up connections will leave
before the page fully loads.
14. DEEP LINKS: If you submit just your home page,
don't expect a search engine to travel more than one or two links
away from the home page or from the page that you submitted. Over
time they may venture deeper into your site, but don't count on
it happening quickly. You'll often need to submit pages individually
that appear further down into your site or create more direct links
from the home page (either visible links or hidden links). This
way the search engines can find them. Visible links are preferable,
but when that's not practical, there are methods to create links
that won't be seen by the average visitor:
http://www.webposition.com/hiddenlinks.htm
The technique of submitting one page that then links to multiple
other pages you want found is called creating a "hallway page."
In many cases, this will not only get you indexed in cases where
they are ignoring you, but it will often improve your rankings.
That's because many engines assign "bonus points" to pages
they find on their own versus pages that were specifically submitted
to them.
15. UNRELIABLE HOSTS: If your Web site fails to
respond when the search engine spider pays a visit, you will not
be indexed. Even worse, if you are indexed and they pay a visit
when your site is down, you could be removed from their database!
Consequently, it pays to have a reliable hosting service that is
up 99.5% of the time. However, at some point a spider is going to
hit that other 0.5% and end up yanking your pages by mistake. Therefore,
it pays to keep a close eye on your listings and resubmit when needed.
16. SPAM: If you have ever used any questionable
techniques that might be considered an overt attempt at spamming
(i.e., excessive repetition of keywords, same color text as background,
or other things that the WebPosition Page Critic warns you about),
an engine may ignore or reject your submissions. If you're having
trouble getting indexed in the expected amount of time, make sure
your site is spam-free.
17. REDIRECTS: If your site contains redirects
or meta refresh tags these things can sometimes cause the engines
to have trouble indexing your site. Generally they will index the
page that it is redirecting TO, but if it thinks you are trying
to "trick" the engine by using "cloaking" or
IP redirection technology that it can detect, there is a chance
that it may not index the site at all.
18. PROPER DIRECTORY SUBMISSIONS: If you're submitting
to a directory site like Yahoo, Open Directory, Looksmart, or others,
then a human being will review your site. They must decide if the
site is of sufficient "quality" before they will list
it. I recommend you read the submission guide on the directory tab
of the WebPosition Submitter. It contains tips on how to improve
your chances of obtaining a good listing in these directories. Getting
listed in major directories first can help you get listed elsewhere.
19. INDEX TIMES CAN FLUCTUATE: WebPosition will
tell you the average index time of each search engine. However,
this is only an average. Sometimes engines will index sites every
30 days fairly consistently and then suddenly stop indexing most
sites for several months. This can be frustrating, but it does happen.
Generally a major engine will not go more than three to four months
without refreshing its index. If you're wondering if others are
experiencing trouble getting indexed on a particular engine, try
asking around. You can often find valuable real-world feedback on
a search engine discussion forum such as the one at:
http://www.marketposition.com
20. PAGE LIMITS: If you have many pages indexed
but are having a hard time getting new ones recognized, be aware
that there are limits. Each search engine will only spider so many
pages of your Web site. This may range from a few dozen or three
or four hundred depending on the engine. Some people have even been
successful in getting far more pages indexed depending on the engine.
Google is one engine that tends to crawl deeper into your site.
However, how deep they go may depend on factors like your link popularity.
Sites with higher link popularity are deemed "worthier"
of more thorough indexing.
21. RANDOM ERRORS: Last but not least, sometimes
the engines just lose submissions at random through technical errors
and bugs. After all, they are managing a database of hundreds of
millions of pages. Therefore, some people like to resubmit once
or twice a month for good merit in case they do lose a submission.
Certainly if you've followed all the "rules" and are still
not listed, by all means, re-submit! Sometimes a little persistence
is all that's needed.
TIP: Once your page achieves a desirable ranking,
it's best not to continue submitting it. You risk the engine re-evaluating
the page and possibly reducing your ranking.
If any of the above scenarios apply to your submission, you should
take the appropriate actions and then re-submit. If that still does
not work, you should consider e-mailing or calling the search engine
and asking them politely why you have not been indexed yet.
Sometimes they will reply back with "Sorry, there was a problem
with our system and I've now made sure you'll be indexed within
the next couple days." Or, sometimes they'll tell you why you
were not indexed so you might correct it. In other cases, they will
ignore your e-mail and you'll have to keep e-mailing or calling
them until they respond.
Still, it's definitely worth the effort to get your site listed
with the major engines. This assumes you also take the time to optimize
your pages so you'll achieve top rankings.
About the Author Robin Nobles is the
Director of Training of the Academy of Web Specialists, which teaches
online training in search engine marketing. She is also a trainer
with Search Engine Workshops, which presents on location workshops
in search engine marketing at various locations across the country.
Please visit our site for more information about online training
and other resources. http://www.marketposition.com/
|
