-
Fonts – Everything You Wanted To Know
All web browsers use standard fonts. Mostly two types, one for proportional, one for mono spaced fonts. Proportional (or variable width) characters adapt in width, an “m” uses more space than an “i”. Mono spaced (or fixed width) characters are all equal in width, typewriter style.
Most browsers use “Times” for proportional fonts. This font was originally developed by the London Times news paper. “Courier” is used for mono spaced. This was a very popular font used for mechanical typewriters. Both are normally set to 12 points (1/72nd of an inch).
Serif And Sans Serif.
Both these fonts are so called serif fonts. The French word serif indicates the little strokes at the outer ends of the characters. They are very old, you see them in old gothic handwriting, or Greek and Roman buildings. Partly used for ornamental reasons, partly because the characters are easier distinguished.
It’s a little odd they are used on computer screens. These are by nature quite coarse, which makes serif characters quite grainy and ugly. Sans serif characters generally display a lot better on screens. Sans serif literally translates to “without stroke”. Probably the most popular sans serif font is Helvetica or derivatives like Windows’ Arial.<FONT FACE=”Arial”>…</FONT>
The FONT Tag.
The html tag for fonts is a somewhat crude instrument. Most word processors let you use any font you like, as long as it’s on your system. And that’s the first big problem in web browsers. You have no control over other systems’ fonts. So you will have to choose a font which is likely to be on any system out there.<FONT FACE=”Arial,Helvetica,Sansserif”>…</FONT>
Typeface.
The font tag accommodates this by letting you specify several fonts in the FACE attribute. If the first one is not available, the second is used, and so on. The set above is often used. Arial is on all Windows systems, Helvetica on Macintosh, Sans serif on UNIX. The same is true for mono spaced fonts in the line below.<FONT FACE=”Courier New,Courier,Mono”>…</FONT>
Size.
Word processors let you specify font sizes in points exactly. No such luck in web browsers. There are seven sizes to choose from, denoted 1 (smallest) through 7 (largest). If this SIZE attribute is not used it defaults to 3. I think the default 12 point size is a bit big, so I use 2 for size, which gives you about a 10 point character. Some browsers let you set the overall font size smaller or larger. Which makes this issue even more awkward.<FONT SIZE=”2″>…</FONT>
There is a nasty bug in some browsers. When using a block of text with size 1, the last line skips a line. This bug can be squashed by putting a <BR> tag immediately after the block of text. If your browser has this bug it will show in the second text below.
There is a nasty bug in some browsers. When using a block of text with size 1, the last line skips a line. This bug can be squashed by putting a <BR> tag immediately after the block of text (with break).
Color.
Fonts can have any color you like, much like the colors in the body tag. Keep readability always in mind, avoid clashing colors and little contrast. You can create nice shading effects. But don’t make a Christmas tree out of your page by using too many colors.<FONT COLOR=”red”>…</FONT>
Style Sheets.
There is a chance all this soon will be replaced by style sheets. They do let you specify exact point sizes, even use downloadable fonts. But for now I would advise against that, since not all current browsers understand them. You could however use a combination of both, should you want to.Article written by Lee
-
Building A Surfer Trap – Stage 4
Building A Surfer Trap – Stage 4.
Here we go with stage 4 in our surfer trap.
We should now have a pretty basic surfer trap set up with consoles on all of the FPA’s all of the links on all of the FPA’s (Excluding the FPA graphics) should go to either another FPA or, the a POTD Program.
As mentioned in the last stage we are going to start building up some traffic to our surfer trap even though, we are not finished yet.
The first place where you can get some good free traffic would be from a toplist counter.
A good counter that I would recommend would be one of the following:
Outster http://www.outster.com (A Gay Specific Counter)
SexTracker http://www.sextracker.com (A General Adult Counter)You need to sign up to ONE of the above counters and only ONE.
Usually a counter will ask you for some details when signing up for them these will usually include:
Email Address.
Site Url
NameFor the email address, enter a REAL email.
For Your Name, Enter a name you wish to be known by to the counter owners.However, for the URL (The one to SEND traffic to FROM the counter) you should enter the index.html page of your surfer trap (The Multi-Site FPA).
You will now be given some coding once you complete the counter sign up process. This counter code you now need to place on EVERY page of your surfer trap.
Place the counter towards the bottom of your FPA’s (Including the Multi-Site FPA) this way, they will load after the rest of the graphics or at least, they should in theory.
Basically what you are doing by implementing this counter is as follows:
For each time the counter is displayed or clicked on your site (Depending on the counter you choose) you will earn one credit on the respective counters toplist.
Each credit you earn on the toplist will move you higher up the list.
Now, the higher up the list you get, the more visitors to the counter site that you are going to get visiting your surfer trap by clicking your link on the toplist.
In return, by clicking on your surfer trap link from the toplist, they are ‘registering’ another impression of your counter hence, getting you one more ‘credit’ on the counter toplist hence, moving you higher in the listings.
See where we are going with this?
The more clicks you get from the counter to your hub, the higher you will get on the toplist in the process earning you more and more traffic to filter through your surfer trap.
Adding these counter codes to your FPA’s as they stand at present should take you an hour or so to do so, for now this is where I will leave the traffic generation section of this tutorial.
Article written by Lee
-
Utilizing All The Content You Use On TGP Galleries!
Utilizing All The Content You Use On TGP Galleries!
For the last few months i have been experimenting with a few new (but old, if ya see what i mean) ideas.Basically i have adapted Voltars 3x3x3 project slightly and incorporated it into the TGP/TGP2 game.
The results have given me some of the best sales ratios ever! We’re talking under 1:60 almost consistently across all sponsors!!
What you did is thus….
Design a NICE 15 pic tgp gallery in a table preferably four rows high and 5 columns across.
In the first row place one thumb in each cell (5 pics).
In the second row place one odd sized banner in the first cell and merge cells 2,3 , 4 and, 5.
In the cells you have merged place some descriptive text about the pics you are using in under two paragraphs and include a line of text where you MENTION the sponsors site name.
In row 3 place one thumb in each cell (5 pics).
in row 4 place on thumb in each cell (5 pics).
under the table place a juicy text link.
below this juicy text link leave a line or two and add a money bar (one row by three columns using a contrasting color to the background of your gallery page).
ALL pics go on html pages using the same layout as your main TGP page but, without the table included.
Repeat this process 4 times once for four different niches, saving each niche in their own sub folder for the main site i.e:url/sub folder/niche 1
url/sub folder/niche 2
url/sub folder/niche 3
url/sub folder/niche 4You should now have 4 TGP galleries using 15 pics on each in four niches (60 pics in total).
At the top of each of these main pages place a three cell one row table.
in this table use text links to describe each niche i.e:
Teen
Gay
Lesbian
Maturein the template for the teen niche you use text for the gay, lesbian and, mature templates and link these to the appropriate TGP template.
This is where the fun begins…..
Take all four TGP niche galleries you have just made and do the following….
Delete the first row of cells (including pics) on all TGP galleries and save the new page layout in another folder using the same navigation structure as the original ie:
url/sub folder 2/niche1
url/sub folder 2/niche2
etc
etcYou should now have the following:
four TGP templates consisting of 15 pics each.
four TGP templates consisting of 10 pics each.Again, link these new 10 pic templates to each other in the same way as you did the 15 pic templates.
Repeat this procedure to make the 10 pic TGP page using the original TGP page BUT, this time, delete the bottom row of cells.
You now have the following:
4 niche 15 pic TGP pages
4 niche 10 pic TGP pages
ANOTHER 4 niche 10 pic TGP pagesStill with me?
Its gets better….
Take your original 15 pic TGP template and do the following…
Delete the first and the third row of cells including all the pics and save them in ANOTHER folder using the same structure as before i.e:
url/sub folder 3/niche 1
url/ sub folder 3/niche 2
etc
etcAgain link these in the same way as before using the text links at the top of the pages.
What do you have now?
4 TGP pages using 15 pics
8 TGP pages using 10 pics
4 TGP2 pages using 5 picsCan you guess what’s next?
Take that original template again and…
Remove the first and fourth row of cells do the same when you save it as you have done above…
What do you have?
4 TGP pages using 15 pics
8 TGP pages using 10 pics
8 TGP2 pages using 5 picsDo it once more using the original templates but removing the final set of 2 rows of cells you now have…
4 TGP pages using 15 pics
8 TGP pages using 10 pics
16 TGP2 pages using 5 picsIn the root of the domain where you are going to upload these pages do the following…
Create a simplistic warning page… link this to a multi site FPA… this multi site FPA links to each of the niche (15 pic) TGP templates you have made.
On this same fpa place an email erotica collection box, banner exchange code and a toplist counter code along with an exit console going to a MC POTD program.
Now, what you need to do is start submitting these pages to the TGP/TGP2’s submit one a day or as many as you can until you have submitted them all.
Ok still here?
Next you have to duplicate ALL of the above again.
Place the duplicate of the above into a separate set of folders/sub domains etc.
On the new root you have created, link to another multi site FPA AND the first root index.html page.
You now have the makings of an EXCLUSIVE 3x3x3 TGP hub which, you can leave alone and submit all of the pages intermittently to any TGP/TGP2 you wish for LIFE!!
All the time you are adding new sections to it and building your link pop in the search engines because you are continually interlinking to each of the separate niche and relevant sub niche folders.
Of course, you NEED to get the meta’s sorted on the first main template for this to work but, if you change all subsequent meta descriptions etc you’ll have a very reasonable listing in Google for the relevant search terms

Article written by Lee
-
Using .htaccess Effectively
The .htaccess file is an ASCII text document that can be placed in any directory on your site. It can be used to control access to files and directories, and customize some server operation in your site. A .htaccess file can be created in any word processor but must be saved as text only. You must use FTP software in ASCII mode to upload or edit your .htaccess file. For the examples provided here, place the .htaccess file in your root directory.
There are a variety of functions that you can control using .htaccess some of the more useful of these are explained below:
Custom Error Messages.
Add the following to the .htaccess file::
ErrorDocument 404 /notfound.html
After “ErrorDocument” specify the error code, followed by a space, and then the path and filename of the .html file you would like to be displayed when the specified error is generated, each specific error code is detailed below with the recommended codes to be used in the .htaccess file in bold :
200 OK
206 Partial content
301 Document moved permanently
302 Document found elsewhere
304 Not modified since last retrieval
400 Bad request
403 Access forbidden
404 Document not found
408 Request timeout
500 Internal server error
501 Request type not supportedUsing the codes above your error section of the .htaccess file should look like this:
ErrorDocument 301 /notfound.html
ErrorDocument 400 /notfound.html
ErrorDocument 403 /notfound.html
ErrorDocument 404 /notfound.html
ErrorDocument 500 /notfound.html
ErrorDocument 501 /notfound.htmlRedirect to a Different Folder.
Add the following to the .htaccess file:
RewriteEngine on
RewriteRule ^/oldfolder(.*)$ /newfolder/$1 [R]This redirects the user from /oldfolder/anyfile.html to /newfolder/anyfile.html, when the .htaccess file is uploaded to the otherwise empty “/oldfolder” directory.
Denying User Access.
Add the following to the .htaccess file:
<Limit GET>
order allow,deny
deny from 000.00.00.
deny from 000.000.000.000
allow from all
</Limit>This is an example of a .htaccess file that will block access to your site to anyone who is coming from any IP address beginning with 000.00.00 and from the specific IP address 000.000.000.000 . By specifying only part of an IP address, and ending the partial IP address with a period, all sub-addresses coming from the specified IP address block will be blocked. You must use the IP addresses to block access, use of domain names is not supported
Redirect a Machine Name.
Add the following to the .htaccess file:
RewriteEngine On
Options +FollowSymlinks
RewriteBase /
# Rewrite Rule for machine.domain-name.net
RewriteCond %{HTTP_HOST} machine.domain-name.net $
RewriteCond %{REQUEST_URI} !machine/
RewriteRule ^(.*)$ machine/$1This will redirect requests for the machine name machine.domain-name.net to the directory machine on the site domain-name.net.
Different Default Home Page.
Add the following to the .htaccess file:
DirectoryIndex filename.html
Then a request for http://domain-name.net/ would return http://domain-name.net/filename.html if it exists, or would list the directory if it did not exist.
To automatically run a cgi script, add the following to the .htaccess file:
DirectoryIndex /cgi-local/index.pl
This would cause the CGI script /cgi-bin/index.pl to be executed.
If you place your .htaccess file containing the DirectoryIndex specification in the root directory of your site, it will apply for all sub-directories at your site.
Preventing Hot Linking.
Add the following to the .htaccess file:
# Rewrite Rule for images
RewriteCond %{HTTP_REFERER} <URL of page accessing your domain>
RewriteRule ^(.*)$ http://<same as above URL>You would replace the <URL of page accessing your domain> above with the domain name and path of the page that is referring to your domain. For example: www.theirdomain.com/users/mypage/
The RewriteCond directive states that if the {HTTP_REFERER} matches the URL that follows, then use the RewriteRule directive. The RewriteRule directive will redirect any reference back to the referring web page.
Using the above you should, safely be able to publish your sites on the internet knowing that you will not be privy to bandwidth thieves via hotlinking and also, that you will not lose any traffic through pages that are ‘not found’.
Article Written By Lee
-
Building A Surfer Trap – Stage 5
So we hit stage 5 in this surfer trap tutorial.
It was brought to my attention this morning that we never added any ALT tags to our single FPA link so, in a change to the planned tutorial I am going to touch on this stage as, once the search engines get to our surfer traps this is going to be a crucial aspect on how highly we get ranked.
So what’s next?
Ok, now what you have to do is go back to manually editing the FPA’s (All of them!)
What you need to do is this…
Take the Multi-Site FPA first then, on ALL of the links that lead to the single site FPA’s you need to add the ALT tag. Again, in the same way as we did originally however, instead of using this tag on the images we will use this on the actual TEXT of the link so, for example the link which may be:
‘Voyeur Porn’ leading to the FPA you have for the Voyeur niche in the HTML coding will already look like this:
<a href=”mydomain.com”>Voyeur Porn</a>
Will get turned into:
<a href=”mydomain.com” ALT=”More Niche Related Keywords”>Voyeur Porn</a>
The reason we are going back over these links now and not earlier on is because you should hopefully have started to get a small amount of traffic from your counter impressions. These counters are virtually ALWAYS being crawled by the search engines due to the amount of people linking to them so, by optimizing our site at this stage, it makes it less work in getting into the search engines.
One other thing that we can now start to do (as we did a couple of stages back) is to create some more HTML pages with some tables on them, however, these will be HTML pages on their own with no images on them. Again however, you should make them 4 columns across and two rows high.
What you want to add into these tables are NICHE links so for example, taking the TEEN niche we would make eight links like:
Teen Sex
College Girls
Erotic Teens
Teen Porn
Etc
Etc
Etc
Etc…You should do this for each of the MAIN niches so you would have a table for Teen, Gay, Mature, Asian, Ebony, Fetish and one for General and again, these should link to the NICHE FPA’s that you already have created.
These tables will be used for another console on our surfer trap however, before we implement this console we are going to have some fun with them.
Article written by Lee
-
Converting Chinese Adult Traffic – The Niche Paysites That Work
Converting Chinese Adult Traffic – The Niche Paysites That Work.
To date we have looked at most of the types of country specific traffic available to webmasters every day and, more importantly, which niches they convert on. In this article we will finish up our look at the foreign traffic and how profitable it can be with the final main country on the net – China.Chinese Adult Surfers – Quick Analysis.
As with Japanese traffic, most webmasters seem to think that Chinese speaking adult surfers are pretty worthless when it comes to buying porn memberships and this is true – If you do not know what niches to offer your Chinese surfers that said, we found some interesting results given the three (3) month test we had on our Chinese traffic and, the results we observed might just make you change your mind about sending the Chinese surfers you get to your sites off to traffic trades.Chinese Porn Surfers – The Niches.
Because we knew relatively little about the Chinese traffic we had on our sites we decided that, rather send it to a specific language hub site we would send it to our main hub page and, allow the Chinese surfers to filter themselves through to the sites that interested them on their own, to our surprise, only 40% of the traffic we knew was Chinese chose to filter itself to a Chinese specific language hub site, the remainder, choosing to go through our main English hub site.Fetish.
Across all of the niches we offered the surfers, the Fetish niche was the one that converted most consistently month after month and, more than converted, actually retained members once they had signed up to the paysites.Gay.
Unlike the traffic we had that was Japanese a lot of the Chinese traffic that hit our site decided to visit the Gay niche paysites we had to offer them and, there was less than 10% difference between the Fetish and Gay signups we observed month after month. Quite a difference when compared to surfers from Japan and, a pleasant one at that.Amateur.
Specifically teen amateur niche sites or, at least sites that had amateur looking teen modes on the tours also did well on our Chinese traffic however, the conversion ratios for the recurring programs didn’t do as well as what we had expected with most memberships being cancelled within 2 months. Non the less, the Chinese surfers did join and most of them re-billed in excess of 1 month before leaving the site.Teen.
As mentioned above, it would appear that Chinese porn surfers are looking for teen niches again though, the teen paysites we used featured amateur looking teens prominently on the pages so whether this was a choice on the surfers part to join the teen sites or, because we only offered amateur looking teen paysites isn’t known.Asian.
Unlike Japanese surfers, Chinese surfers also joined the Asian niches that were offered up to them and, surprisingly enough, we noticed more signups to English Asian sites than localized Chinese pay sites.Chinese Adult Surfers – The Results.
Overall we were very impressed with how the Chinese traffic we had flowing through our sites both acted and, converted on the sites that we offered them to join, especially since the Japanese traffic we had didn’t seem to like the gay niche sites yet Chinese traffic worked wonderfully on exactly the same gay niche pay sites.Across the board I think a revisit of Chinese traffic is needed with more options on our part when it comes to the teen and amateur niches especially just to see if the results we experienced thus far were unique or, are truly indicative of how the Chinese surfer navigates and joins pay sites.
Article written by Lee
-
Designing A 404 Error Document
Designing A 404 Error Document.
Now that we know how to use .htaccess let’s get into how exactly to effectively design a 404 error document page. Here are a few tips when building your document:Keep The Design Constant.
That is, whatever design your web site supports, consider keeping that design when building your error document. This is not mandatory, however. If you have a good reason to stray from your design, then do so. Otherwise, keep it constant.
Use Attractive Dialog.
Do not use the word error. Error signifies something is terribly wrong, and while that may be so, you should give that information to your visitor in a very friendly way. Instead, consider writing “Sorry, the page you were looking for is not available at the moment”.
Include Contact Information.
Obviously, you want to fix the error, so give the user every opportunity to e-mail you about the error. Some users will not bother to do so, but you still need to give them the opportunity.
Keep The Visitor Moving.
Often, 404 error pages are like road blocks, or brick walls that the user cannot pass through. You need to break through the barriers and allow the visitor to keep traveling through your web site. If your web site supports a search capability, include a form on your error document to let them search for the information they were originally seeking. At the very least, include a link to your home page.
Give Them A Site Map.
When the user clicks upon a page that is no longer available, or was mistyped by the webmaster, consider adding a link to a site map page, or include your site map right on your 404 error document, so the user can get their bearings within your site, and can locate their desired content quickly and easily.
Give Them Help.
List tips on your error document page to help the visitor diagnose the error. For example, ask them to double check the URL in their browser’s address bar. If it’s correct, politely ask them to e-mail you to report the error. Also suggest that they visit your home page (or search utility) to find the information they are after.
List Some Popular Links.
Provide the user with a few links to your most popular content areas of your web site, as it is more likely they were searching for that content than anything else.
As you have seen, error documents are more important than you may think. Merely having a custom error page shows you have put forth effort on your web site and want to help the user find the information they need. Error documents are easy to create and adds professionalism to any web site.
Article written by Lee
-
Obscenity – Put It To The Test
Regardless of how long any of us have been an adult webmaster we all need to be
aware of obscenity laws and, in particular how they affect our businesses
whether we think a hardcore photoset is ‘obscene’ or not ultimately, if you get
taken to court on obscenity charges the one thing you should be aware of is how
the courts will decide whether the images you are using will be classified as
obscene or not.Testing Obscenity – The Miller Test.
The Miller test was developed in the 1973 court case of Miller vs. California
and in comprises of three parts ALL of which MUST be satisfied on order for
something to be deemed obscene by the courts. The Miller test is the ‘official’
method used by the United States Supreme Court for determining whether an
expression or a speech can be determined as obscene and, if deemed obscene, it
is not protected under the First Amendment and is therefore prohibited by law.The Miller Test – Part One.
Part one of The Miller Test states something may be obscene if ‘the average
person, applying contemporary adult community standards, would find that the
work, taken as a whole, appeals to the prurient interest’ In essence, this
means that if the ‘average’ person on a jury or on the bench finds the work to
be deemed obscene then, it is. However for the court to rule something as
obscene it also has to be deemed obscene by the standards set in part two and
part three below.The Miller Test – Part Two.
Part two of The Miller Test states that something is potentially obscene is
‘the average person, applying contemporary adult community standards, would
find that the work depicts or describes, in a patently offensive way, sexual
conduct’. Basically this is saying that if the images or speech is something
which is not practiced in a manner befitting your local community standards
then again, it may be obscene. However, as with part one of The Miller Test for
a court to find something obscene it needs to fall below the standards in part
three below.The Miller Test – Part Three.
Part three of The Miller Test states that something is potentially obscene if,
‘a reasonable person would find that the work, taken as a whole, lacks serious
literary, artistic, political, or scientific value.’ This is pretty much where
you could potentially come unstuck after all, everyone has different sexual
tastes and because of this, just because something that may be widespread such
as ‘bare backing’ (to use as an example) could potentially be considered as
obscene if you happen to have a jury who are devout practicers of safe sex.The Miller Test – Overview.
In essence The Miller Test is a useful guideline for webmasters when it comes
to operating our sites and specifically, when it comes to choosing the types of
content we utilize on them however, for the most part The Miller Test itself is
outdated. Since the early 70’s when this test was devised there have been many
sexual practices that were once deemed obscene that have become more a part of
everyday life and accepted in to society as whole thus, what once would (or
could) have been deemed obscene would no longer be in the same sense as
something that may be deemed obscene today could be found not to be in 5 years
time.Article written by Lee
-
Using Mainstream Sponsors On Adult Sites
With the adult industry moving more and more into the mainstream web every week that passes as late i thought it would be an idea to show you guys how it is possible to use mainstream sponsor programs on your adult sites to build up a steady commission check even when a surfer going to your adult site doesnt actually buy anything from your adult sponsor.
Mainstream Sponsor Programs – The Choices.
The first thing that you need to do is to choose a mainstream sponsor that offers you a good choice of products / services to market to your surfers mainstream affiliate programs such as Click Bank, Commission Junction and, Amazon.com are all good places to find a selection of mainstream products and services you can offer your surfers in addition to the adult sponsors you currently use.Mainstream Affiliate Programs – Setting Up.
The first thing that you need to do however in order to offer the mainstream affiliate program products to your surfers is to register a generic, non-adult domain name and, instantly get this domain setup on your current web server.This will be the domain name you use for all correspondence between the mainstream affiliate networks and yourself in addition, this will also be the ONLY domain that you utilize the mainstream linking codes on.
Using Mainstream Sponsors – Getting Started.
The first thing that you need to do now you have your domain and mainstream affiliate networks sorted out is to build a couple of specific ‘doorway pages’ on your new generic domain name. This could include something like an article or product review or perhaps something as simple as an image of the product and a short description which you can use to lead the surfer into the mainstream affiliate programs site, either way, the idea is to have the root of the domain 100% free of adult content and utilize specific doorway pages which you can link to from your adult sites.Generating Mainstream Traffic.
Onto the really easy part, your current free sites all have an enter and exit link correct? If so then you are already half way finished, all you now need to do is to ensure that EVERY SINGLE exit link goes to your mainstream doorway page for a specific product or group of products. Make sure that you do not link to the root of your generic domain at any time but only to specific doorways and, that each of these doorways has a link to the root domain.By linking to your generic site in this way, you are enabling the surfer to choose whether they first of all, want to look at your adult site or, they want to leave the site and go to your mainstream affiliate network sponsor in the process giving you the potential of making a sale at the mainstream affiliate program.
Mainstream Sponsor Programs – An Overview.
By utilizing mainstream affiliate programs in this way you are not only potentially increasing your bottom lines but, you are also utilizing every single click that you get both to and from your site, to often webmasters send their exit links to places like disney.com or google.com and in doing so, they lose out on potential incomes, hopefully when you start marketing your exit links in the manner explained above you will begin to see the benefit of how both adult and, mainstream sponsors can be used in conjunction with each other in order to increase your bottom line profits.Article written by Lee
-
Robots.txt – Control The Robots That Crawl Your Sites
By writing a structured text file you can indicate to robots that certain parts of your server are off-limits to some or all robots. It is best explained with an example:
# robots.txt file for general use on web servers.
User-agent: webcrawler
Disallow:User-agent: googlebot
Disallow: /User-agent: *
Disallow: /cgi-bin
Disallow: /logs
The first line, starting with ‘#’, specifies a comment.The first paragraph specifies that the robot called ‘webcrawler’ has nothing disallowed: it may go anywhere.
The second paragraph indicates that the robot called ‘googlebot’ has all relative URLs starting with ‘/’ disallowed. Because all relative URL’s on a server start with ‘/’, this means the entire site is closed off.
The third paragraph indicates that all other robots should not visit URLs starting with /cgi-bin or /log. Note the ‘*’ is a special token, meaning “any other User-agent”; you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines.
Two common errors:
Wildcards are not supported: instead of ‘Disallow: /tmp/*’ just say ‘Disallow: /tmp’.
You shouldn’t put more than one path on a Disallow line (this may change in a future version of the spec)
Ultimately, without the use of robots.txt files on your servers/domains, you are risking a variety of potential problems including, unauthorized access to your cgi directory, unauthorized viewing of your site stats, possible spamming of the search engines by accidental crawling of doorway pages.One distinct advantage however of having a robots.txt file on your server is that, quite simply, you will be able to tell when and where your site has been indexed or potentially indexed as, all robots will automatically call for the robots.txt file BEFORE any other page on your server so, as long as you keep an eye open for any calls of this file, you can see who is knocking at your site for indexing purposes.
Below is a robots.txt example that you can copy and paste into a text document to use on your own server:
<!–Start Copy Below This Line–>
User-agent: *
Disallow: /cgi-bin
Disallow: /logs<!–End Copy Above This Line–>
The above will allow all spiders to crawl all of your site except the subdirectory’s ‘cgi-bin’ and ‘logs’ which, may be altered to suit any subdirectory’s you do not wish the spiders to crawl on your server.
Article written by Lee
Premium Sponsors
Categories
- 2257
- Billing Solutions
- Blogging
- Branding
- Content
- Domain Names
- Employment
- Forms & Contracts
- General
- Hosting
- Link Lists
- Opt-in Mail
- Paid Traffic
- Pic Posts
- Promotion
- Scripts
- Search Engine Optimization
- Sponsors
- TGP
- Traffic
- Tutorials
- Viral Marketing
- WebDesign
- Writing