HTML Page Load Times – Making Them Quicker

Date: 2011.02.22 | Category: WebDesign | Response: 0

Surfers on the web are here for one thing and one thing only, they want information, they want it to be correct and, more importantly, they want it now. We cant always help when it comes to getting the information however we can help them so far as making sure that when one of the surfers clicks on our pages they load quickly.

Taking Control Of Speed.

For many webmasters hand coding their HTML pages is an often timely and costly method and, because of this they turned to the WYSIWYG (What You See Is What You get) editors such as Dreamweaver and Frontpage to create their pages for them however, even this in itself requires some manually intermission on the webmaster part. Many of the older style WYSIWYG Editors added a fair amount of un-needed code to the HTML so, spending a few minutes after you have your pages created to physically go through the HTML code and clean up the junk html can in effect, cut down your page size by up to 25% in some instances.

HTML Quotations And Hyphens.

In addition to un-needed HTML tags you may also want to consider removing any quotation marks or hyphens from your HTML code, both of these can quickly clutter up a page and laden it down with more code than is actually required.

For example, ‘HTML Coding’ makes just as much sense as me telling you HTML Coding when read in sequence. The trick with using quotations and hyphens is actually knowing at what point in your text you can ‘get away’ with removing them and, this is only something you will know yourself after proofing your completed page.

Hyperlinks.

If you are only going to provide links from one page of your site to another then, why use absolute paths? http://www.domain.com/linking.html is the same as /linking.html when you use it in a link from one page on your site to another and, well, in all honesty you have increased the load time of your page by only linking to the relative url instead of the absolute and that’s what we are trying to do here right? Make our pages load faster.

White Space.

White space on a web page can be a good marketing tool however, white space in your HTML code is a bad thing, placing spaces between certain parts of your HTML code can quickly bog down your load times instead, try to use the tab key to separate certain parts of code likewise, keep the use of &nbsp to a minimum, this can quickly become the most relevant term on your page if used in excess.

One other benefit of making your pages as small as possible is that a webmaster wanting to steal your HTML code is going to have a harder time finding the exact piece of coding if the HTML is all on a single line than they would if the code was laid out nicely however, this in itself is only a deterrent and it wont stop those webmasters determined to steal your HTML code.

Page Load Time Overview.

Hopefully this brief article has given you some idea of ways in which you can practically reduce the load time of your pages without affecting the overall look and feel of your site and, who knows, if your page loads quicker, then you have a better chance of your surfer getting to your sponsor quicker.

Article written by Lee

JavaScript Know How

Date: 2011.02.22 | Category: Scripts | Response: 0

JavaScript can be one of the most useful additions to any web page. It comes packaged as standard in Microsoft’s Internet Explorer and, Netscape Navigator and allows webmasters to perform field validations, mouse-over’s, pop ups and a whole entourage of other nifty little features on our sites.

In this article we will show you how to:

– Display the browser name and version number
– Change the text in the status bar of the browser
– Use an input box to get text from the user
– Use a message box to display text to the user
– Change the title of the browser window

Before that, however, we need to know how to setup our web page so that it can run the JavaScript. JavaScript code is inserted between opening and closing script tags: <script> and </script>, like this:

<script language=”JavaScript”>

–> JavaScript code goes here <–

</script>

These script tags can be placed anywhere on the page, however, it’s common practice to place them between the <head>and </head> tags. A basic HTML page that contains some JavaScript looks like this:

<html>
<head>
<title> My Test Page </title>
<script language=”JavaScript”>

function testfunc()
{
var x = 1;
}

</script>
</head>
<body>
<h1>Hello</h1>
</body>
</html>

For the examples in this article, you should use the basic document format I have just shown you, inserting the JavaScript code between the <script> and </script>tags. When you load the page in your browser, the JavaScript code will be executed automatically.

Displaying the browsers name and version number.

The “navigator” object in JavaScript contains the details of the user’s browser, including its name and version number. They can be displayed in a browser using the document.write function:

document.write(“Your browser is: ” + navigator.appName);
document.write(“<br>Its version is: ” + navigator.appVersion);

I run Windows 2000 and Internet Explorer version 6, so the output from the code above looks like this in my browser window:

Your browser is: Microsoft Internet Explorer
Its version is: 4.0 (compatible; MSIE 6.0b; Windows NT 5.0)

Changing the text in the status bar of the browser.

To change the text in the status bar of a browser window, just change the “status” member of the “window” object, which represents the entire browser window:

window.status = “This is some text”;

Using an input box to get text from the user.

Just like in traditional windows applications, you can use an input box to get some text input from the user. The “prompt” function is all you need:

var name = prompt(“What is your name?”);
document.write(“Hello ” + name);

The prompt function accepts just one argument (the title of the input box), and returns the value entered into the text box. In the example above, you get the users name and store it in the “name” variable. You then use the “document.write” function to output their name into the browser window.

Using a message box to display text to the user.

You can display a message box containing an OK button. These are great when you want to let the user know what is happening during their time on a particular page. You can use a message box to display the “name” variable from our previous example:

var name = prompt(“What is your name?”);
alert(“Your name is: ” + name);

The “alert” function takes one argument, which is the text to display inside of the message box.

Changing the title of the browser window.

To change the title of a web browser’s window, simply modify the “document.title” variable, like this:

document.title = “My new title”;

One bad thing about the “document.title” variable is that it can only be manipulated in Microsoft Internet Explorer. Netscape’s implementation of JavaScript doesn’t allow for modification.

In Closing.

As you can see from the examples in this article, JavaScript is a powerful scripting language that can be used to enhance a visitor’s experience with our site. However, you shouldn’t use JavaScript too much because in some cases it can annoy visitors and send them packing before your site even loads!

Article Written By Lee

How To Claim Back European V.A.T

Date: 2011.02.22 | Category: Billing Solutions, General | Response: 0

With the recent legislations requiring US businesses to charge Value Added Tax to EU Member state customers for purchases of digital product / image downloads there are many new questions going around enquiring how companies and small business owners can claim the VAT back this is what we will look at in this brief article – The basics of claiming back your VAT from Europe.

What Is V.A.T?

Value Added Tax (VAT) is a consumption tax levied on goods and services in many countries across the globe. Over the last decade, legislation in the tax systems of Europe, Canada, Japan and South Korea allow for a refund for the majority of non resident entities, worldwide to a refund of VAT in most instances however, especially in the business world, registration must be achieved prior to claiming back your VAT and, preferably before the purchase of Vat able goods takes place.

Who Is Entitled To A VAT Refund?

Most businesses will be entitled to receive a VAT refund however, as the laws vary from country to country, the best way for you as a business owner to find out the laws as they affect you is to contact your company accountant who, should be knowledgable enough to inform you of what you need to claim back from each of the EU member states where V.A.T has been charged.

What Products Can I Claim V.A.T Refunds For?

Most typically value added tax recoverable expenses can take form as one or more of the following items:

Business Property Expenses.
Transportation Costs.
Inter-company Billing.
Marketing + Promotional Services.
Exhibitions, Conferences + Business Meetings.
Importation V.A.T.
Digital Downloads (Business Related).

How Do I Register For V.A.T?

When a company inside the domestic US conducts transactions in the EU countries where value added tax registrations may be required the registration for the VAT refund should be done prior to the transaction taking place however, on occasions which vary from individual EU member states this registration process may be done retroactively. By becoming registered to claim your VAT back the company is legally required to d the following:

1) Register for the VAT in the EU country where the transaction takes place.
2) Account for VAT on all invoices or receipts issued to your European customers.
3) Deduct the V.A.T incurred from your European suppliers of goods or services.

Value Added Tax – An Overview.

When all is said and done, other than the obvious need to increase your pricing structure for certain global markets registering for and, charging value added tax actually has a great deal of benefits. You now get to claim addition refunds from any expenses that you incur from the day to day running of your business, your promotional activities become vat’able so you can now increase the level of global marketing you do and, claim full value back from the EU member states (where applicable). basically, registering in the EU member state will enable your business to grow on a global scale and, at the same time, afford you the luxury of being able to market your goods and services to a greater client base at virtually no cost to yourself this is a good thing especially when you take a look at the growth that international internet usage is making each month.

Article written by Lee

Niche Marketing – What Is A Niche?

Date: 2011.02.23 | Category: Promotion | Response: 0

The first step as webmasters we must take to become successful is to discover what our niche specialty is however, many webmasters still do not realize exactly what a niche is.

In fact, from the webmasters I have spoken to it would seem that a ‘niche’ is just another word for the main topics of pornography, Teen, Asian, Fetish, Gay, Straight, the list goes on and, whilst this list is never ending they are all niches in themselves however, you need to look beyond the top level niche and start to develop a feel, a stance of your own as to what niche you will work on for example, Asian Midgets, you probably cant get more niche than that in this day and age however, realistically, the chances of you a) being able to find the content for this niche and b) Being able to find a sponsor are remarkably thin.

That said, if you can find a niche that you like personally, whether that be she-male, midgets, straight or, gay, this will give you a solid foundation to market your sites on especially if you build on your already vast knowledge of what turns you on.

Now, onto breaking down the niche you are comfortable working in..

Lets say for arguments sake that you have chosen to concentrate on the ‘Gay’ market,, within this market (as with any other) there is a remarkable amount of ‘sub-niches’ and, these are what you need to build your business around whether you want to or not, the ‘sub-niches’ of any given adult market is where the $$$ can be earned.

Now using the gay market as an example the first few niches that come to mind are:

Twink’s – Young looking hairless 18 – 20 year olds.
Bear’s – Hairy men over the age of 20.
Hunk’s – Muscular looking guys in the age group of 20 – 30 .
Straight Guys – Yes, that’s right, gay men get off on looking at straight guys.

As you can see from the above four examples we can now start targeting our market audience a lot better.

The next thing we need to do now that we have found the niche and the sub-niche we will be working is to look for a sponsor that complements the niche of our choice, this is the hard part. Many webmasters will go directly to the highest paying sponsor and throw links up to their ‘sub-niche’ pay site not even looking at the sites tour or members area. This is a huge mistake, the first thing you NEED to do is to look over the sponsors paysites and, more importantly, the sponsors site tour AND members area. What point is there sending your surfers who are after Twink’s if, your sponsors tour has images of a load of guys over the Twink age bracket?

From this point on and, after you have chosen a suitable tour to send your traffic to you need to build up a flow of ‘niche’ traffic the chances are, if you submit your TGP gallery to an ‘all gay’ TGP that you will receive a substantial amount of unfiltered traffic making your hard work in choosing your sponsors tour worthless.

So how do you choose a traffic source that is right for you? Simple, you need to experiment, start of with your niche TGP’s, Top Lists, Links Lists etc and see what happens, how much traffic are you being sent from your chosen traffic source and, more importantly, how much of this traffic is actually clicking through to your sponsors tour. Once you can get an idea of what traffic you are being sent from what traffic source you have won more than half of the battle already.

Record EVERY possible detail about your traffic sources that you can, how much traffic they send a day, how many visits your sponsor gets, how many sales you made, what content you were using etc, etc.

Once you have a working list of data to use you can start to abuse and filter your traffic like you would have not thought imaginable, if one of your sponsors has a higher payout on Bear traffic, then you instantly know what traffic source gives you traffic for that niche and, you can utilize that information to your benefit.

The most important thing to remember however when marketing a niche is that you need to have patience, you need to use a process of trial and error and, more importantly, you need to experiment, just because TGP A sends you 100,000 hits of Twink traffic one day, it doesn’t mean you will get the same quantity of niche traffic the next.

Article written by Le

Thinking Of Starting An Adult Website In The UK?

Date: 2011.02.24 | Category: General | Response: 0

We all know that there are many foreign / international adult webmasters and there are some very well known ones based in England however, before you start building your online empire what are the legalities regarding the type of content you can and cant use on your adult pornographic sites?

To be honest with you, there are no ‘defined’ laws in England (At the time of writing this article) regarding the use of online media to promote adult activities however, it is a commonly held belief that as long as you stay to the same standard as what most ‘top shelf magazines’ or ‘R18 Videos’ have in them you shouldn’t go far wrong.

So what are these standards? That’s the information that I am about to give you.

Content that IS NOT acceptable.

1) Any material which is in breach of the criminal law.

2) Material (including dialogue) likely to encourage an interest in abusive sexual activity (e.g., pedophilia, incest) which may include depictions involving adults role-playing as non-adults.

3) The portrayal of any sexual activity, whether real or simulated, which involves lack of consent.

4) The infliction of pain or physical harm, real or (in a sexual context) simulated. Some allowance may be made for mild consensual activity.

5) Any sexual threats or humiliation which do not form part of a clearly consenting role-playing game.

6) The use of any form of physical restraint which prevents participants from withdrawing consent, for example, ball gags.

7) Penetration by any object likely to cause actual harm or associated with violence.

8) Activity which is degrading or dehumanizing (examples include the portrayal of bestiality, necrophilia, defecation, urolagnia).

As you can see from the above, there ARE very specific guidelines regarding water sports, animality and, some of the more ‘out there’ fetishes.

Now you know what you cant use on your adult sites, lets take a look at the list of sexual activities that you can use. Its no surprise that the list of what you can use is considerably shorter.

Content that MAY BE acceptable.

1) Aroused genitalia.

2) Masturbation.

3) Oral-genital contact including kissing, licking and sucking.

4) Penetration by finger, penis, tongue, vibrator or dildo.

5) Non-harmful fetish material.

6) Group sexual activity.

7) Ejaculation and semen.

It also worth noting that the above guidelines are not distinguished between heterosexual or homosexual activities.

So, that’s the low down on what the current status of affairs are with regards to the use of adult content on websites in England and Wales. Actually, that’s not to bad just ensure that you stay within these guidelines and, more importantly, that you do consult a solicitor before commencing any form of adult business after all, that’s what they are there for, to give you SOUND legal advice instead of basic ‘guidelines’.

Article written by Lee

Mirroring Adult Sites – Stage Two

Date: 2011.02.23 | Category: Promotion, Tutorials | Response: 0

Ok, in stage 1 of this tutorial we learnt the basic ‘setup’ for our multisite generation now we have to start putting the rest of our puzzle together.

At this stage in the tutorial we should now have a 50 pic free site, all of the images are in the appropriate folder on our HD along with the HTML in their own folder. If you have not already done so, you need to ensure that when you add/added the links to your individual HTML pages that you call the images like this:

/FreeSite/Images/pic1.jpg

The Thumbnails like this:

/FreeSite/Images/Thumbs/thumb1.jpg

The gallery links like this:

/FreeSite/Galleries/gallery1.html

And the FPA from the warning page like this:

/FreeSite/FPA/fpa.html

Why are we calling the links like this and not like, http://www.mydomain.com/FreeSite/Galleries/gallery1.html I hear you ask, well the answer is simple, in order to use this site TEMPLATE over and over again, we need to ensure that there is a standard way of calling the links, this way, once we decide duplicate this site on a different domain, all we do is upload the folder to our server with a different set of images and we have another set of sites built.

Ok, we now have our free site but, we have some empty folders that need filling up, here is how we are going to achieve that.

What we now have to do is re-open the Gallery Pages, DO NOT change the image calls or the thumbnail calls however, what you will need to do is select 15 TGP’s that you would like to submit to, download the recip buttons for these TGP’s and save them in the /FreeSite/Images/Recips/ folder.

Now we have the gallery pages open we need to modify them like this..

Take the top text link that you created and replace that with a 3 cell table. In this table for the first gallery, you need to call the first 3 recip links for the TGP’s you want to submit to and link them to the appropriate recip url for each of the TGP’s again, calling the recip images like this, /FreeSite/Recips/recip1.gif. Also, you will need to modify the text link at the bottom of your gallery pages, I would suggest creating an 8 cell table, containing 8 niche text links, four of which should go to your Single-Site FPA’s (as created for the surfer trap) and the remaining 4 text links should go directly to your sponsors site tour page.

Now save this newly created page as tgp1.html in the /FreeSite/TGP/ folder. Do the above again for the remaining 4 gallery pages but calling a different set of recip links each time. each time you modify a gallery page save it in the /FreeSite/TGP/ folder so gallery2.html would be renamed to tgp2.html, Gallery 3 would be called tgp3.html and so on.

What you should now have is a single 50 pic free site with 5 galleries of 10 pics, all pics going to the larger image on a HTML page and, 5 TGP galleries.

This is as far as we go with the tutorial today however, in the next stage we will continue to fill in the rest of the puzzle pieces.

Article written by Lee

Content – Keeping your members coming back for more

Date: 2011.02.22 | Category: Content | Response: 0

We know that the first battle for pay site owners is getting the traffic and the signups. But, sometimes this becomes such a focus that we forget the other battle: keeping members interested once you have them. When people pay for a membership to a site they are expecting to get their money’s worth. In return for their monthly fee they are looking for a site that encompasses quality, variety and originality.

As the web grows, so does the online entertainment industry. Four or five years ago there was only a handful of pay sites around compared to the thousands and thousands that are on the net today. Back then, you could literally put any kind of content on your sites and you would have membership, there just was not much choice out there for the connoisseur of porn. Now, potential members are more informed, they know what is out there, they know what they like, and they know what they want to see.

What has to be done to cater to these more discriminating porn shoppers? You have to give them what they want. Here are some useful tips for choosing a content provider and/or content for your pay sites that will keep the members coming back for more:

Go for the exclusive content.

You want to be able to provide your members with something that is fairly fresh and that is not pasted on every pay site on the Internet. The last thing a potential buyer wants to see is a site that is a carbon copy of half the sites out there.

Choose leased content.

Choosing leased content over buying CD’s is a good way to keep your site fresh. Most leased content providers will update their content on a bi-monthly or monthly basis. As well, with leased content, it is just a matter of adding the feed into your site. This alleviates the need for extensive extra graphic design work.

Have a wide variety of content.

Make sure that your site has a little bit of everything a potential member could want: pictures, videos, live feeds, etc. The more you have the better. Many content providers have great package deals available that will include all of these things all set up and ready to put on a site.

Choose a reputable content provider.

You want to make sure that when you choose a content provider that you do not just go for the best deal out there. You want to also look at things like: how much bandwidth they provide, what their technical support is like and last but not least, do they have good customer service.

These are just but a few helpful suggestions to follow when choosing content for your pay site. There are certainly many ways to achieve a high rate of member retention. One thing that many of our customers tell us is that they find that having a least one voyeur feed on their site great for keeping the members coming back for more. Laura’s Condo, one of our voyeur feeds, has one of the best member retention rates around. Visitors get attached to the girls, develop a rapport with them and they do not want to lose that. Hence, they will keep renewing to be able to keep their relationship going with their favorite girl.

The number one thing that pay site owners must remember is that members equal money and in order to keep the cash flow rolling in they must keep the members interested and give them what they want. By spending a little extra money to get high quality exclusive content, you will actually be putting more money in recurring memberships back in your pocket at the end of the day.

Article written by Meredith Murray

Why Go Adult?

Date: 2011.02.21 | Category: Hosting | Response: 0

When choosing a hosting company why choose an adult host over a mainstream one?

Simple, They provide above other things, a better quality of service, affordability and, most importantly, reliability.

Adult hosts face a variety of technical and support issues, this coupled with the points mentioned above makes them a far superior choice to standard mainstream hosting providers. This is one of the main factors why more and more mainstream internet based companies are now turning to the adult business forum to provide their hosting solutions.

Mainstream going adult? Yes that’s right, more and more mainstream companies are turning to our industries hosting providers for the fast speeds, security and, reliable infrastructures they can offer after all, adult sites are amongst the most prolific on the internet on average gaining more daily hits to their sites than mainstream websites so, they need to be cost efficient, not only to the hosting companies but, also to their clients, the webmaster.

Our industry strives on the use of heavy graphics, coupled with streaming media files to secure sales and longevity, this is why the adult hosting companies need to be able to provide not just a single, heavily laden server but tens of them, sometimes even hundreds all producing gigabytes of traffic on a daily basis.

Adult hosting companies are constantly upgrading their equipment in order to improve not just their servers, connectivity but also the smaller things that are often overlooked by webmasters, hubs, routers, switches and the likes along side the not so technical aspects of running a host, The staff.

Along with all of this technical equipment they also need to maintain a high level of technical knowledge, after all, not just anyone can install operating systems capable of running multi-million dollar industries. They need the knowledge to maintain, upgrade, repair and sell their services to the best of their capabilities.

What does this mean to you? Well, we all want to make money and, through the use of faster servers, e-commerce solutions and the technical support your hosting provider can give you, you will be fattening your wallet on a daily, weekly, monthly basis.

Adult hosting, quite simply, is the crème de la crème of the hosting world, they offer solid, solutions whether you are starting in the adult industry or, operate a mainstream e-commerce site.

So how do you choose a host that is right for you?

You need to ask some basic questions of the host and, do your homework. Not only should you find out what hosting solutions they can offer you but, you need to find out what their support service is like, how affordable they are in comparison to other companies who can offer similar or the same service. Whilst on the subject of hosting support, before you sign on the dotted line, give your host a call out of office hours, if they answer, then you can almost be certain they are a good hosting provider.

Of course, not all hosts will have 24 hour phone support, however, that said, some offer 24 hour icq support, email support and more.

Another sure fire way to find out if a hosting company is any good is to ask on the heavily traveled webmaster message boards, after all, you are not the only webmaster out there who will have tried this company, if they are no good, someone will know about it and point you in the direction of a reputable host like http://www.webair.com.

Hopefully, this article will have given you some insight into the world of adult hosting companies and, will assist you in making the right choice of host for your site, whatever type of website you might be hosting.

Article Written By Lee

Golden Rules For Finding A New Domain Name

Date: 2011.02.22 | Category: Domain Names | Response: 0

We all know by now that domain names are being registered at an unbelievable rate and, whilst the vast majority of ‘decent’ domain names have already been registered there are still some god domains to be registered and, this article will give you some insight into how you can choose an effective domain for your business purposes – regardless of what they are.

Avoid Play On Words.

Whilst sitting at a bar you come up with an idea for a domain www.4play.com and, at the time this might seem like a great domain name to buy however, before you head to your favorite register and purchase the domain stop to think about the long term implications.

Let me explain this a little more using the example www.4play.com lets think about this, every time you mention the domain in a conversation people are more likely to head across to www.foreplay.com or www.fourplay.com first as a type-in. This is also true of the other abbreviation styles of ‘internet slang’ remember the days of the ‘dot com’ registrations, you know domains such as www.dotsex.com looking back now, these seem ridiculous to have been registered however, at the time they were popular but they died their death years ago.

Think Investment.

A good ‘commercial’ domain must be able to distinguish itself from the competition, there are several ways to achieve this for example:

Consider Your Target Audience.

Aim your domain at a specific, yet broad market audience. use tools to narrow down what your customers (surfers) are looking for, narrow this list down to a select few domains and then, narrow the list down some more.

Follow Trends Or… Create Them.

If you are serious about picking a good domain name or two then be prepared to invest the time required to get a good domain name. Research the latest trends on the internet, see what products or services are ‘hot’ and will be around long term and also see which domains are just a flash in the pan.

A good example of ‘long term’ domains would be www.europeanwebmasters.com which i registered several years ago when, in all honesty, the foreign markets were not even thought about however, you cant go to any forum nowadays without someone asking about the international marketplace.

Act On A Whim.

In complete contrast to my opening statement in this article, if you think you have a solid business idea, act on it, the chances are someone else will also think of the same idea as you and, if you don’t get the domain name first, they will.

Alternative Buying.

So you have decided that you do want to buy www.4play.com before you make your purchase have a look at what other variants of this domain are available, for example, you might like to check registrations of www.4-play.com, www.for-play.com, www.forplay.com, etc etc.

If you have a great business idea, make sure you have all of your bases covered, just because you are starting a new craze, that doesn’t mean the domains will be available for registration tomorrow or next week.

By purchasing all the variants of the high quality domain you have thought of you are, in effect, eliminating all of the competition you could face long term.

Don’t Cyber-Squat.

If you are a serious domain buyer, avoid tailgating on other peoples domain names unless of course, it has enormous potential for example, if someone has registered a generic domain such as www.123456.com but failed to register www.123-456.com why not register the available domain? You are not impeding on anyone’s property rights and, you might benefit from some mis-typed traffic at the same time. As with any type of mis-typed domain, before you register it, seek legal advice, a few dollars spent before registering the domain could save you thousands afterwards.

Fools Rush In.

Don’t automatically buy the first domain name you think of, think on it some more, sleep on it if you have to, ask your family and friends on their opinions. Although the really good domain names have virtually almost all gone, its better to carefully think out each and every domain name purchase you make and save yourself some money long term than to rush in and buy a domain that wont be any use and have to register a new domain to take the place of the first.

In addition to the above don’t sit on a really good domain name idea forever because, forever comes a lot sooner than you may think.

Hopefully this article has given you some useful information in respect of registering new domain names and you can put this information to good practice when buying domains. Article written by Lee

Robots.txt – Control The Robots That Crawl Your Sites

Date: 2011.02.24 | Category: Search Engine Optimization, WebDesign | Response: 0

By writing a structured text file you can indicate to robots that certain parts of your server are off-limits to some or all robots. It is best explained with an example:

# robots.txt file for general use on web servers.

User-agent: webcrawler
Disallow:

User-agent: googlebot
Disallow: /

User-agent: *
Disallow: /cgi-bin
Disallow: /logs
The first line, starting with ‘#’, specifies a comment.

The first paragraph specifies that the robot called ‘webcrawler’ has nothing disallowed: it may go anywhere.

The second paragraph indicates that the robot called ‘googlebot’ has all relative URLs starting with ‘/’ disallowed. Because all relative URL’s on a server start with ‘/’, this means the entire site is closed off.

The third paragraph indicates that all other robots should not visit URLs starting with /cgi-bin or /log. Note the ‘*’ is a special token, meaning “any other User-agent”; you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines.

Two common errors:

Wildcards are not supported: instead of ‘Disallow: /tmp/*’ just say ‘Disallow: /tmp’.
You shouldn’t put more than one path on a Disallow line (this may change in a future version of the spec)
Ultimately, without the use of robots.txt files on your servers/domains, you are risking a variety of potential problems including, unauthorized access to your cgi directory, unauthorized viewing of your site stats, possible spamming of the search engines by accidental crawling of doorway pages.

One distinct advantage however of having a robots.txt file on your server is that, quite simply, you will be able to tell when and where your site has been indexed or potentially indexed as, all robots will automatically call for the robots.txt file BEFORE any other page on your server so, as long as you keep an eye open for any calls of this file, you can see who is knocking at your site for indexing purposes.

Below is a robots.txt example that you can copy and paste into a text document to use on your own server:

<!–Start Copy Below This Line–>

User-agent: *
Disallow: /cgi-bin
Disallow: /logs

<!–End Copy Above This Line–>

The above will allow all spiders to crawl all of your site except the subdirectory’s ‘cgi-bin’ and ‘logs’ which, may be altered to suit any subdirectory’s you do not wish the spiders to crawl on your server.

Article written by Lee