Register  |  Login




Advertisement

Start Your Own Q&A Site

Create your own Q&A site easily, allowing you to quickly grow a new community around any subject matter or generate new organic traffic for your existing website.

Question

Status: Closed Points: 125 Time: 13:33 - Oct 12, 2006  

mary2

How can I block web crawlers and search engines from accessing and listing an entire website?

I have a new site that I just put up but also have a test site at a different URL. For some reason and somehow Yahoo has picked up the site and I just had somebody register to the test site yesterday. I need to stop that, first of all from having the public be able to access it and secondly allowing search engines to pick it up at all. Any suggestions? I dont know how it happened in the first place because there are no links pointing to my URL which is how I thought they accessed it in the first place. Thoughts? Answers? Expertice?

Answer Discussion
Tutorials

 

nhinds

Date:: Nov 01, 2006

Time:: 00:24

The search engines should be easy enough to stop, just put a file called "robots.txt" in the root directory with this in it:

User-agent: *
Disallow: /

For stopping users I would use whatever way your host offers to password-protect directories, or if you're using Apache on Linux you can create a .htaccess file in the root of the website containing:

AuthName "Members only in here"
AuthType Basic
AuthUserFile /path/to/your/.htpasswd
AuthGroupFile /dev/null
require valid-user

and then use the utility htpasswd (in the directory you want to create the .htpasswd in) like this:
htpasswd -c .htpasswd username (for the first time only. remove the -c if you want to add more users)

mary2

Date:: Nov 01, 2006

Time:: 05:03

Thanks a lot nhinds

suxrobuz

Date:: Sep 04, 2010

Time:: 07:50

but can you close the crawler, cause i can't get on a game that i downloaded, so can you close it please

Question Answered

This question has been closed, and points have been rewarded to the following experts:


nhinds: 125

You're welcome however to comment or give additional information or if you wish, you have the ability to write a Tutorial in the Tutorial Area.

Answer this Question

New User

Email:

Upon submission of this form, you will automatically be registered as a Quomon user and we will send your login information to this address

Registered User

Username:

Password:

Forgot Your Password?

No tutorials have been submitted yet. Want to be the first?

Answer this Question

New User

Email:

Upon submission of this form, you will automatically be registered as a Quomon user and we will send your login information to this address

Registered User

Username:

Password:

Forgot Your Password?

Ask a Question

Have a new question? Ask!

You have 100 characters to use



Top Experts

View More

Rank

Expert

Points

1.

nidhi

10354

2.

oracleofDelphi

6493

3.

rcastagna

5596

4.

LAGM

4848

5.

PeterNZ

3487

6.

gonzalo

2840

7.

Mason

2770

8.

jgivoni

2303

9.

xarcus

1820

10.

Anpanman

917

Become an Expert

Register today to share your knowledge with the community and be recognized and rewarded for your contributions.


Register Here




"Psst, Quomon is a great site. Pass it on."     Tell a Friend  |   Link To Us  |   Save to Delicious  |   Digg! Digg it



Language Options

English:

www.quomon.com

Español:

www.quomon.es