Are your web pages indexed?
A problem I run into once in awhile is not getting all of my web pages indexed by the search engines. Indexing means that a search engine has the web page listed in its database. If the page isn’t indexed it will not show up in a search for the topic, keywords or meta data. And, of course, that means no one will see it in their search results.
The first step is to check your sites to see if any pages are missing. Each search engine has its own index, so you will need to check each one separately. This could be a lengthy chore since there are hundreds of search engines. You should start with the big boys on the block; Google, Yahoo, MSN and Alta Vista.
Most search engines allow the use of a special operator or command to tell you if a page is in the search index. For example, if I wanted to know if a page from my web site was indexed by Google, I could search for “site:www.imjustsayingblog.com/about-me“. The search engine would show the page in its results if it is in the index.
Are Your Web Pages Indexed?
Let’s assume that your page is not found using this method, the next step is to try to figure out why. One possibility is that most of your site is missing, which you can determine by a similar search, such as “site:www.imjustsayingblog.com“–you can see how many pages are indexed. I just ran this and the index found 344 pages. I then compared it to the pages in my XML Site Map plugin which also shows 344. So, I’m good, today.
What to do about missing pages.
Since there could be several things I recommend you take it one step at a time.
Major site problems include:
- Spam penalties. If you get caught violating the search engines’ terms of service (spamming), they’ll drastically reduce your pages in the index. You can beg for re-inclusion. But, don’t expect a quick response if you get one at all. You may need to abandon the site and start a new one.
- Hidden links. If the navigation to your site is hidden within JavaScript, Flash, or other non-HTML code, the search engine spiders probably won’t be able to follow them.
- Dynamic URLs. If your URLs are excessively long, or have many parameters, or contain ID or session parameters, the search engines can choose not to index them.
- Incorrect robots.txt file. Your robots.txt file tells the search engine spider which pages to include and exclude when it crawls your site. If you’ve coded the file incorrectly, you might be excluding lots of pages you meant to include. Or, the robots.txt file is missing. The spider either crawls everything or nothing.
Minor site problems include:
What if it’s really just one or two pages that aren’t being indexed? Some problems are likely to be confined to a page or two:
- Incorrect robots tagging. Just like the robots.txt file, a robots metatag tells the spider to include or exclude an individual page. It could be telling the spider to exclude the page by mistake.
- User interaction required. If your page launches a pop-up window, or demands that a form be filled out, spiders can’t fill in the form or close the pop-up and therefore, won’t index it.
- Improper redirects. If your page uses a meta refresh or JavaScript redirect, spiders ignore them and don’t index the page.
- Poor quality pages. If your page is excessively long, contains HTML coding errors, is nothing but ads or uses frames, it probably won’t be indexed correctly.
Once the problem(s) is corrected, you can wait for the spiders to come back.
If it never gets indexed, it never shows up in the search results. And, whoa, no visitors.
(2 votes, average: 5.00 out of 5)
Loading ...
Related Posts -
The Big, Bad List of Pre-SEO Questions You Need to Answer, Part VII Part VII of Pre-SEO questions you need to answer We tackle the thorny issue of cost and results in this section of the series on pre-SEO questions. One of the difficulties in determining how effective SEO tweaks are the unknown and unknowable issues of doing business on the web. For...... - XSitePro II vs. Firepow Website Design Software Review XsitePro II versus Firepow "What is the difference between XSitePro II and Firepow web software?" "And, Which is the better one?" A fellow attendee at SES NY posed these questions, which I thought was interesting, since the conference was about blogs and blogging and not website......
- Is Microsoft's Bing Search Engine A Google Killer? Bing versus Google On June 3rd Microsoft launched its new search engine Bing.com. They spent tons of money on advertising it and a lot of hype and some criticism followed from the blogosphere. As with everything new some folks like it and some folks don't. The real question is "Will......
-
Wordpress Plugins You Might Want by Hal Major In my post on Monday, Wordpress Plugins You Should Use, I listed and described the Wordpress plugins that are especially important for revenue generating blogs. Get those plugins before you get these. However, get the plugins from this post that fit into your niche. I use all...... - 14 Tips On Building Links Most people know that building links drives more traffic to their site, it will also improve your Google page rank (PR) and your Technorati authority. Both of these are critical factors when determining the value of your site.The greater the value of your site the more you can charge for......
Related Websites -
SEO and Corporate Blogging When it comes to trying to promote your business in an organic way, corporate blogging is possibly the best tool that you can use. This is one of the easiest ways to implement SEO on your site with little effort and a blog has the added benefit of being incredibly...... - Search Engine Optimization Techniques For Changing Times When web masters were unaware of the search engine’s potential in web site promotions, search engine optimization (SEO) was limited to integrating some keywords in the meta tags. Then, those who knew even a little about the impact of search engine queries on page popularity enjoyed unrivaled success in search......
- Five easy-to-make mistakes that keep search engine robots away from your website Many webmasters don't get high rankings on Google and other search engines just because Google's indexing robot has difficulty to index their web pages. Search engine robots are very simple software programs. If an indexing robot cannot find the content of your website immediately, it will skip your site and......
- Search Engine Optimization-Web Crawlers The terms web crawler, automatic indexers, bots, worms, web spiders, and web robots are programs or automated scripts with browse the World Wide Web in a methodical, automated manner. The term web crawler is the most commonly used term. Web crawlers are a tool used for search engine optimization.......
- Search Engine Optimization Guide for Webmasters [senior Hs Paper] Introduction
Search Engines have developed into the Internet's most popular and powerful source of information, accounting for an estimated 80% of the Internet's traffic (Heche, 2007, p. 1). As a result, website owners are realizing the power in such devises and are shifting marketing budgets into the optimization of their......
Comments
3 Comments on Are your web pages indexed?
-
Dante on
Sun, 1st Mar 2009 6:43 am
-
Hal Major on
Mon, 2nd Mar 2009 5:25 am
-
Noah from business development blog on
Fri, 28th Aug 2009 2:19 pm
Thanks so much for this info. I checked my site like you said and I found some pages listed in the search engines that I don’t want shown. How do I keep these pages from being crawled and indexed?
P.S. Please keep posting on these types of topics. Your explanations are easy to understand.
Dante’s last blog post..Dating, Attraction and Seduction Reviews
Hey Dante,
Excellent question. I’ll be posting on exactly that topic later this week. Probably on Wednesday. However, be sure to check back everyday or subscribe to my feed to get the updates sent to you.
Hal Major
You also might not have enough inbound links to support more of your pages being indexed. Might help to build some deep links to those pages that aren’t being indexed.
Noah@business development blog´s last blog ..Social Networking Opportunities for Entrepreneurs 
Tell me what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!
Translator
By N2H -
-
-
-
Mastermind Group
-
