robots.txt

[phuckbunny]

I'm full of confusion today. I know the robots disallow tag file has been brought up before. Now, would this be something good to include if you're being sploggy i.e. many posts with similar content and titles will end up in sups? Otherwise if each post is different enough as in a mainstream blog, wouldn't it be better to not use a robots file?

[mynameisjim]

I would never disallow a blog post or page. If you have a folder that contains your 400 recip galleries, then maybe no index that folder but I can't imagine a situation where you would not want a post indexed.

[phuckbunny]

Thx, Jim, that was my initial impression and that's why I was double checking. The reason I brought it up is that I'm doing well with one blog in SEs but I'm afraid of newly indexed pages falling into supplemental just because they share the same keywords or same words in the title. For this blog that I'm talking about I'm not too worried b/c every post is more or less unique but say in a situation like, you have a bangbros blog so say each of your post titles is something like

"candy from bangbros sucking cock"

"brandy from bangbros getting a facial"

"mandy from bangbros getting a DP"

I don't fully understand SEO yet or the mighty GOOGLE god itself, but from my primitive understanding, your pages will fall into the dreaded "supplemental results" without a robots no follow in such a situation b/c the post names are too similar.

Having said that, I am guessing that if someone googles "candy from bangbros" and I have a page that starts with that header it will indeed come up in the search results. Right?

[razor]

Maybe I am just being dense, but whats to stop you from changing the title of the posts? I am assuming that these are sponsor provided posts that you inport via rss.

[phuckbunny]

No, i actually write all my posts, but I've noticed I've been getting good SE results when including the model's names but it can get repetitive so I'm just apprehensive about these indexed posts falling into sups.

[phuckbunny]

Sorry I'm going to bump this one last time as I'm still worried about repetition in posts falling into supplemental since for some reason I'm on a decent google kick lately. Apologies if this is grade school material, but I've been getting the hang of writing a goodly amount of posts in a short enough amount of time that I figure why not keep up the momentum and blog my butt off so I can get a tonne of indexed pages - but if they start going into sups I feel like I'll be wasting my time instead of writing new splogs or trying my hand at freesites and what not so here goes:

I'll use a different analogy than Candy/Mandy/Fill in the blank-dy from Bangbros.

Let's say I had a political blog and I hate or love George Bush. And it's all about how Bush is great or Bush sucks.

So many of my posts would probably carry a subject header with the President's name in it, as in today for example I would write, "Bush Takes Health Care Away From Kids The Fucker" (btw that whole thing kind of reminded me 'Thatcher Thatcher, Milk Snatcher' if anyone from UK or Europe remembers how PM Thatcher started off her career in govt).

Anyway, so let's say today I wrote that subject heading. Yesterday I may have written "Bush is a Dick" or "Bush is the Best'. Tomorrow I may write "Bush is the Best President Ever" or "Bush is the Worst President Ever". What I'm trying to figure out is, is this a case to include a robots disallow because everything will just fall into supplemental because of "similar" posts just because I'm ranting and raving about my love or hate of Bush and he comes up in the subject of each post?

Any insight is appreciated, thanks.

PS I love Bush


PPS I hate Bush

[sunfunbill]

Hhhmmm, good question. But,, most blogs deal with a certain subject matter. Like a blog about cars will contain many post with cars, engine, ride, performance, etc in most post. So I assume google has taken that into account when spidering blogs. So I don't think I would worry about it, however I could be wrong.

[artwilliams]

You might want to read up on how to use that file to direct traffic to more important pages on your website. I've not done to but I know other do to "funnel" bots to more important pages on their sites.

[porn punk]

Originally posted by phuckbunny
No, i actually write all my posts, but I've noticed I've been getting good SE results when including the model's names but it can get repetitive so I'm just apprehensive about these indexed posts falling into sups.

You want repetitive in a 2% density ratio

[phuckbunny]

Originally posted by porn punk
You want repetitive in a 2% density ratio

PP- what does that mean?

Bill - you're right, I probably will just leave it be for now.

Art- Something to think about, good point. But I'm not advanced enough about that stuff yet.

[mynameisjim]

I heard keyword density should be at 6%..lol

Duplicate content will land you in supplemental but also poor inbound link profiles will as well.

In the cases you mentioned I doubt that will cause you problems.

But that's just my opinion.

[porn punk]

Originally posted by mynameisjim
I heard keyword density should be at 6%..lol

Duplicate content will land you in supplemental but also poor inbound link profiles will as well.

In the cases you mentioned I doubt that will cause you problems.

But that's just my opinion.

And there you have it?

[porn punk]

Originally posted by phuckbunny
PP- what does that mean?

Bill - you're right, I probably will just leave it be for now.

Art- Something to think about, good point. But I'm not advanced enough about that stuff yet.

It means if broken down, the keywords you are targeting should be 2% of your page, post whatever... Maybe it's 6% percent now. Who knows.

This is all theory and there are anal-izers out there to check your pages, but of course this becomes harder with blogs due to constant changing of pages and posts.

Which again in theory is a good thing.

Remember that game mad-libs. Try to find some top posts and play mad-libs and see what happens. Do it on a test (s)blog

[razor]

How do you know if a page is in the supplemental index and not the main?

[phuckbunny]

Go to goog, punch in site:yourdomain.com - you'll get a list of all the indexed pages but the ones that are in sup will be in the "we have omitted some pages in search results for duplicate content etc etc