Are your web pages indexed?
A problem I run into once in awhile is not getting all of my web pages indexed by the search engines. Indexing means that a search engine has the web page listed in its database. If the page isn’t indexed it will not show up in a search for the topic, keywords or meta data. And, of course, that means no one will see it in their search results.
The first step is to check your sites to see if any pages are missing. Each search engine has its own index, so you will need to check each one separately. This could be a lengthy chore since there are hundreds of search engines. You should start with the big boys on the block; Google, Yahoo, MSN and Alta Vista.
Most search engines allow the use of a special operator or command to tell you if a page is in the search index. For example, if I wanted to know if a page from my web site was indexed by Google, I could search for “site:www.imjustsayingblog.com/about-me“. The search engine would show the page in its results if it is in the index.
Are Your Web Pages Indexed?
Let’s assume that your page is not found using this method, the next step is to try to figure out why. One possibility is that most of your site is missing, which you can determine by a similar search, such as “site:www.imjustsayingblog.com“–you can see how many pages are indexed. I just ran this and the index found 344 pages. I then compared it to the pages in my XML Site Map plugin which also shows 344. So, I’m good, today.
What to do about missing pages.
Since there could be several things I recommend you take it one step at a time.
Major site problems include:
- Spam penalties. If you get caught violating the search engines’ terms of service (spamming), they’ll drastically reduce your pages in the index. You can beg for re-inclusion. But, don’t expect a quick response if you get one at all. You may need to abandon the site and start a new one.
- Hidden links. If the navigation to your site is hidden within JavaScript, Flash, or other non-HTML code, the search engine spiders probably won’t be able to follow them.
- Dynamic URLs. If your URLs are excessively long, or have many parameters, or contain ID or session parameters, the search engines can choose not to index them.
- Incorrect robots.txt file. Your robots.txt file tells the search engine spider which pages to include and exclude when it crawls your site. If you’ve coded the file incorrectly, you might be excluding lots of pages you meant to include. Or, the robots.txt file is missing. The spider either crawls everything or nothing.
Minor site problems include:
What if it’s really just one or two pages that aren’t being indexed? Some problems are likely to be confined to a page or two:
- Incorrect robots tagging. Just like the robots.txt file, a robots metatag tells the spider to include or exclude an individual page. It could be telling the spider to exclude the page by mistake.
- User interaction required. If your page launches a pop-up window, or demands that a form be filled out, spiders can’t fill in the form or close the pop-up and therefore, won’t index it.
- Improper redirects. If your page uses a meta refresh or JavaScript redirect, spiders ignore them and don’t index the page.
- Poor quality pages. If your page is excessively long, contains HTML coding errors, is nothing but ads or uses frames, it probably won’t be indexed correctly.
Once the problem(s) is corrected, you can wait for the spiders to come back.
If it never gets indexed, it never shows up in the search results. And, whoa, no visitors.
(2 votes, average: 5.00 out of 5)
Loading ...
Related Posts -
Making Money With MySpace Making Money with a MySpace Page When using MySpace, you have to tread more carefully. You can’t blatantly market on MySpace or else they’ll ban you from the community. They dislike it when you add a certain number of friends to your list in one day because people have developed...... -
4 Critical Off-The-Page SEO Factors 4 Critical Off-The-Page SEO Factors In my previous post, we talked about the things you can do "on your website" to help it rank well in the search engines — in other words, the "on the page" factors. In this post we're going to talk about the external factors that...... - Is Microsoft's Bing Search Engine A Google Killer? Bing versus Google On June 3rd Microsoft launched its new search engine Bing.com. They spent tons of money on advertising it and a lot of hype and some criticism followed from the blogosphere. As with everything new some folks like it and some folks don't. The real question is "Will......
-
6 Critical On-Page SEO Factors 6 Critical On-Page SEO Factors Two main factor are considered when search engines review and decide what rank to give your web page: 1) The content on the page and how relevant it is to the search terms people use. This is called on-page factors. The nice thing about these...... - Website Link Building by Hal Major Building incoming website links of high quality can be a daunting task. You can search for other websites in your niche and contact the webmaster asking to exchange links. You can visit and post on a lot of blogs leaving your website URL in the comment. However,......
Related Websites - Popular Search Engine Optimization Misconceptions There is nothing more frustrating for a web designer or a website owner than spending an excessive amount of time ‘tweaking’ his website to optimize its page ranking – and then realizing that search engines are bypassing it, for reasons unknown. The problem is that many people are operating off......
- Site Indexation – Is Very Important Step Into Site Development. Before the site will appear in results of search, it should be indexed by search system. Indexation means that the search robot has visited your site, has analyzed it and has brought the information in a database of search system. If some page is brought in an index of the......
- Search Engine Optimization: Four Vital Steps For Optimizing Your Website There is a bit of confusion about search engine optimization. Some people think that SEO (the abbreviated form) is nothing more than tricking search engines into giving a high ranking for a particular site. Others think that search engine optimization is so complex that they could not possibly understand it.......
-
SEO Tips for Bloggers on WordPress There are several million WordPress blogs that you can find on the internet, but unfortunately the majority of them are not properly optimized for search engines. This article is going to touch on some SEO tips for bloggers using WordPress to host their blog, because with the right tips it...... - Search Engines, Seo, Optimization And Page Rank: Do You Understand Them? I have heard it said that your Page Rank is the most important aspect of your website SEO optimisation that gets you a high listing in the Search Engines. That gives me a problem! The first problem I have with that is why should Yahoo, MSN and ASK care about......
Comments
3 Comments on Are your web pages indexed?
-
Dante on
Sun, 1st Mar 2009 6:43 am
-
Hal Major on
Mon, 2nd Mar 2009 5:25 am
-
Noah from business development blog on
Fri, 28th Aug 2009 2:19 pm
Thanks so much for this info. I checked my site like you said and I found some pages listed in the search engines that I don’t want shown. How do I keep these pages from being crawled and indexed?
P.S. Please keep posting on these types of topics. Your explanations are easy to understand.
Dante’s last blog post..Dating, Attraction and Seduction Reviews
Hey Dante,
Excellent question. I’ll be posting on exactly that topic later this week. Probably on Wednesday. However, be sure to check back everyday or subscribe to my feed to get the updates sent to you.
Hal Major
You also might not have enough inbound links to support more of your pages being indexed. Might help to build some deep links to those pages that aren’t being indexed.
Noah@business development blog´s last blog ..Social Networking Opportunities for Entrepreneurs 
Tell me what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!
Translator
-
-
-
-
Mastermind Group
-
