I have some pages on my personal web site that are x-rated. What can I do so these pages aren't referenced by search engines?
I don't care if the rest of the pages are hit... in fact it is fun to see my host reports that show me which keywords and phrases are getting hits. But I'd prefer if some pages just weren't in the loop.
Thanks.
I don't care if the rest of the pages are hit... in fact it is fun to see my host reports that show me which keywords and phrases are getting hits. But I'd prefer if some pages just weren't in the loop.
Thanks.
posted by:
|
Seattle
|
-
Re: Avoiding Google and other search engines.
Sun, September 25, 2005 - 7:13 PMput the following into your robots.txt file:
User-agent: *
Disallow: /privatedata
cahnge privatedata to the name of the file you want to protect. leave off the file append (IE: HTM,PHP etc) so it will wildcard. Use a seperate line for each page.
HTH -
-
Re: Avoiding Google and other search engines.
Sun, September 25, 2005 - 8:41 PMThanks a lot Chris,
Sorry... I am a hobbyist, not a professional...
Are saying I can change my page name from "porno1.html" to "porno1.xyz" and it will work in a browser, be linkable... but won't be searched?
Also, I was unaware of the purpose of the robots.txt I have always seen it, my host provider put it there. I just added the line at the end:
# Dont allow search engines to index specific folder
User-agent: *
Disallow: /fpdb
Disallow: /stats
Disallow: /_borders
Disallow: /_derived
Disallow: /_fpclass
Disallow: /_overlay
Disallow: /_private
Disallow: /_contentindex
Disallow: /_themes
Disallow: /_vti_bin
Disallow: /_vti_cnf
Disallow: /_vti_log
Disallow: /_vti_map
Disallow: /_vti_pvt
Disallow: /_vti_txt
Disallow: /_vti_script
Disallow: /XXX
So an XXX folder anywhere in a folder tree will stop further searching in that tree? Or does the XXX have to be a root? (I currently have more folders below XXX -- I was doing that for my organization -- I didn't know it would help with this!)
Do I need to do both the robot.txt and renaming exentions to xyz?
Also, is there a way to get my current XXX pages out of Google?
Thanks again. -
-
Re: Avoiding Google and other search engines.
Mon, September 26, 2005 - 5:53 AMPut the full name of each page on each indvidual line. If you change the extension away from current accepted conventions the pages won't be readable by anyone (browser won't know what to do).
You already have them in their own folder and have thast in your disaalow file. You should be fine. They will disappear the next time the spiders browse them.
HTH. -
-
Re: Avoiding Google and other search engines.
Mon, September 26, 2005 - 7:29 AMTo quickly remove a page from Google index before the next spider search...
You need to put <meta name="ROBOTS" content="NOINDEX, NOFOLLOW"> in the head section. Then submit the page(s) for removal from Google.
services.google.com:8882/urlco...roller
It was painless to sign up and execute. It supposedly takes about 5 days for removal. I will let you know by Friday if has worked or not.
Thanks -
-
Re: Avoiding Google and other search engines.
Mon, September 26, 2005 - 12:56 PMhrm, didn't know that one. BOOKMARKED!!!!! Thanks! :-)
-
Re: Avoiding Google and other search engines.
Tue, September 27, 2005 - 11:05 AMThe removal worked! -
-
Re: Avoiding Google and other search engines.
Tue, September 27, 2005 - 2:36 PMFantastic! I will put that on my tech page
-
-
-
Re: Avoiding Google and other search engines.
Wed, April 26, 2006 - 5:59 AM"If you change the extension away from current accepted conventions the pages won't be readable by anyone (browser won't know what to do)"
that's not exactly true - it's the server that interprets the information. If you change the extension (e.g. you have the file cookies.mmm), you can change your server to interpret .mmm files as html, or php or asp or however you want. the server then sends the information to the browser as html (or xml, xhtml, opml, wml, etc.).
-
-
-
