Mirroring Adult Sites – Stage One.

Date: 2011.02.23 | Category: Promotion, Tutorials | Response: 0

In this next series of tutorials I will be exploring the benefits of utilizing your existing content to build more sites in the least amount of time possible, in fact, after you have completed the steps in this tutorial series you should be able to build over 20 types of site within 5 minutes using a single set of 50 pictures.

So, onto the start of our tutorial.

Stage one in this tutorial is something we have ALL done before, so what you need to do is this…

Build a 50 pic free site, this is using a lot more content than a ‘standard’ free site however, what we are actually going to be making is a set of AVS sites and several TGP galleries, along with some SE pages and also some links list mirrored pages using this set of 50 images by building just ONE site! Sounds impossible? It isn’t… read on….

Your site should have the following on it:

1) Warning Page (index.html)
2) Menu Page (menu.html)
3) Gallery Pages (5 galleries of ten pictures called gallery1.html, gallery2.html etc)
4) HTML Large image pages x 50 (picpage1.html, picpage2.html, etc.)
5) Multi Site FPA (fpa.html)

Now lets break these individual pages down into sets of instructions for each:

Warning Page.

This should have your standard warning text placed on it, along with an enter AND an exit link. The ‘enter’ link should link to your Multi Site FPA and the ‘exit’ link should go to a program such as the ARS Discreet Browser tour page.

Menu Page.

This should be a page containing an odd sized banner, links to your 5 gallery pages, plus text links at the bottom of the page ideally, these text links should be in the same style as the niche tables we created for the surfer trap except they should only contain 4 cells instead of 8. These links should go to a different ‘niche’ as the one you are building your current site for and, link to the existing FPA’s of your surfer trap.

Gallery Pages.

On your gallery pages, ALL of the thumbnails should be linked to the relevant HTML page with a text link at the top and at the bottom of the HTML page (No Banners are to be used on the gallery pages!) Also, the file names of the images should be pic1.jpg, pic2.jpg, etc for the full sized images and, thumb1.jpg, thumb2.jpg etc for the thumb nailed images. your images MUST be named this way!!

Multi Site FPA.

This should link to the individual FPA’s that you should already have on your server if you followed the surfer trap tutorial series (If you did not create this surfer trap the tutorials can be found at this link) as well as linking to the Single Site FPA’s this should also have a ‘no thank you’ link which goes to your Menu Page.

Images.

You need to have 50 images, you also need to have 50 thumbnails for the content you will be using on this site. I usually select my content by the niche I am building for, rename the images using The Rename then, once renamed I use Thumbnailer from Smaller Animal to create the thumbnails for each of the renamed images. This will give you 50 full sized pictures with 50 thumbnails named respectively for the larger image.

They are the basics, now onto the good stuff…

Once you have created these pages you need to create some folders on your Hard Drive which will be the SAME structure you will have on your server. This folder structure should look somewhat like this once you have saved each of the pages we have just created into their respective sub folders:

FreeSite/index.html
FreeSite/AVS/
FreeSite/FPA/fpa.html
FreeSite/Galleries/gallery1.html, gallery2.html, gallery3.html, etc, etc.
FreeSite/Images/pic1.jpg, etc.
FreeSite/Images/Thumbs/thumb1.jpg, etc.
FreeSite/LinkList/
FreeSite/Menu/menu.html
FreeSite/PicPage/picpage1.html, picpage2.html, etc.
FreeSite/Recips/
FreeSite/Engine/
FreeSite/TGP/

This will give you 5 folders with HTML pages in them, one folder with the full sized images, one with thumbnails in it and five empty folders.

At this point we will end the first stage in this tutorial as this should take you a couple of hours to complete.

Article written by Le

The Next Big Traffic Craze – Thumbnail Link Lists

Date: 2011.02.24 | Category: Link Lists | Response: 0

As adult webmasters are constantly thriving to create the next craze in traffic sources to garner sales from their sites it comes as no surprise that there is already talk amongst the ranks of the next big thing, what is this new idea? The Thumbnail Link List.

Thumbnail Link Lists – Re-Inventing The Wheel.

As the past year and a half have taught us as webmasters, regular TGP’s have almost become a thing of the past with more and more thumbnail TGP’s being created and submitted to daily, why then, has it taken so long for someone to think up the idea of the Thumbnail Link List? Quite simply I think it has been a case of to much has happened over the last 12 months in the adult industry for webmasters to think this idea up.

So What Is A Thumbnail Link List?

A thumbnail link list is in effect a way of displaying free sites on your own standard Link List by utilizing small thumbnail samples of the images contained within the free sites galleries, in fact, much the same way that a thumbnail TGP works however, instead of only listing single image galleries your link list will consist of links to individual free sites.

Much like the Thumbnail TGP traffic source, the thinking behind the Thumbnail Link List is that by showing your surfers what type of pictures the sites within your link list contain they will be more likely to visit the free site thus improving your CTR to your webmaster submitted sites and, ultimately, your site overall allowing you to manipulate the traffic to garner more sales for yourself as the Thumbnail Link List owner.

Submitting Your Site To A Thumbnail Link List.

Looking at the Thumbnail Link List idea from a webmaster who would be submitting your free sites for listing (and ultimately traffic) how do the rules of the Thumbnail Link List differ to that of the regular link list which we are all used to seeing, well for a start, the main difference will be that you either need to submit a thumbnail yourself or, enter the url to one of your gallery pages in addition to the main entrance to your free site, this will actually be the only significant change in how you operate your link list submissions compared tot he current link list method, however, as each Thumbnail Link List owner will have their own individual rules you should check before submitting as to what is required from you as a submitter.

Thumbnail Link Lists – Overview.

The whole Thumbnail Link List idea is one that, in all seriousness should mean more traffic not only for the submitters of the free sites but, also for the Thumbnail Link List owner themselves, by offering the surfers a preview of the content contained within the free sites they will be visiting they can simply reduce the amount of sites that do not appeal to them and focus more on the sites that do meaning that the webmasters who submit highly niched free sites to the links lists will now be able to benefit from a larger traffic base and ultimately, more sales.

Article written by Lee

Traffic Filtering – Country Specific Redirects

Date: 2011.02.21 | Category: Traffic | Response: 0

To some, actually marketing to foreign surfers is a waste of time rather than a business practice. Unfortunately, they are missing out on additional revenue. We are already beginning to see some of the major sponsors in the adult industry embracing these foreign markets and, not just by utilizing dialers.

However, filtering your traffic base is often the hardest part of this money making equation. That is where the following piece of PHP coding can come in handy.

PHP FILTERING CODE

<?
$user_lan = $HTTP_ACCEPT_LANGUAGE;

if($user_lan==’de’) {
## German
$redir_url = “http://www.germanlanguagepageurl.com”;

} elseif($user_lan==’fr’) {
## French
$redir_url = “http://www.frenchlanguagepageurl.com”;

} elseif($user_lan==’it’) {
## Italian
$redir_url = “http://www.italianlanguagepageurl.com”;

} elseif($user_lan==’es’) {
## Spain
$redir_url = “http://www.spanishlanguagepageurl.com”;

## US traffic or Rest of world not defined above
} else {
$redir_url = “http://www.yourmainpageurl.com”;

}

header(“Location: $redir_url”);
exit;

?>

So, we know that the above piece of coding can redirect surfers based on their country of origin however, in order for you to maximize this to its full potential you will need to know the country specific codes (Also called ISO 639 codes) for each of the main browser languages. Some of these are as follows.

da | Danish
de | German
en | English
es | Spanish
fi | Finnish
fr | French
it | Italian
jp | Japanese

The problem you now have is finding where to send your filtered country specific surfers to. Of course, there is always the dialer option however, this is not going to be as lucrative to your wallet as what most people will have you believe.

In fact, I very rarely use a dialer on my foreign traffic instead, I tend to send them to a language specific tour page from one of the big sponsors and, if they do not sign up to that sponsors site, this is the point where I throw a dialer at them and, if the dialer still doesn’t make any money off the surfer I then recycle the surfer for a fresh one through a toplist or banner exchange heavy page.

In doing this I find it is often more productive than ‘regular’ English speaking traffic as, if you give the surfer something they are looking for, they will be more inclined to buy.

Hopefully this article has given you some insight into filtering and using your foreign traffic as oppose to just sending them off to a dialer program.

Article written by Lee

Are Your European Surfers Male Or Female?

Date: 2011.02.21 | Category: Promotion | Response: 0

In Europe, the domain of the Web still belongs to the younger generation, with nearly 80% of European surfers under the age of 30. This is not surprising, as Europe is culturally quite different from the US.

The older generation in Europe has typically been slower to adopt new technologies and habits compared to their American counterparts, and we can’t see this trend changing anytime soon. Cultural differences will limit Internet penetration among the older age groups.

Female Internet usage still lags in Europe, with slightly more than 20% of Web users being women. Although this difference is less pronounced in the UK, on the Continent, males still dominate the Web scene.

Our Target Audience

What does this tell us? Well, first of all, it tells us that the European audience is growing more and more each and every day but, we already knew that, what we didn’t know however, until now, was just what percentage of European surfers could be persuaded to buy porn memberships or related goods from one of our sites.

Most paysite owners will tell you that, on average the ages of their members range from 25-30 and, as such, the European market could potentially supply a vast majority of these signups if, they got the overall balance of their sites right.

The Solution

As you should already know by now, the European porn surfer has different expectations to its US counterpart, they wont give out personal information as readily as American surfers, this includes their email address so, this rules out sending them to ‘For Free’ programs, they are also getting really pissed-off at having to download dialer programs to access porn, heck, the average phone bill in Europe is almost double what it is in America, why antagonize these people even more? They know they are overlooked when it comes to porn surfing so we need to stop neglecting these surfers, get a site translated into a foreign language, broaden your horizons, tap into an as yet ‘untamed market’ I feel certain that if you take a few basic steps your bottom line can be increased significantly, it all comes down to this…..

Do you have the balls to try something new?

Article written by Lee.

International Billing Alternatives – Premium Phone Billing

Date: 2011.02.24 | Category: Billing Solutions | Response: 0

In the last article i wrote in respect of international billing options we took a closer look at the SMS Billing method and its pitfalls and benefits when charging our surfers for access to our sites. In this article we will take a look at another option we can offer our international surfer base – Phone Billing.

Phone Billing – What Is It?

Phone billing, as the name would suggest is a method of applying a ‘charge’ to a surfers normal land-line telephone. This charge is often around the cost of $35 (US).

Once the surfer has called the premium rate number displayed on your websites join page, they are given a code to enter into a form, again, this form could be on your join page or on a separate site.

Phone Billing – What Are The Costs.

To be perfectly honest with you this is all dependant on to many variable factors to give you a good solid answer. However, as mentioned above the standard cost would seem to be in the region of $35 (US) but, this can often vary depending on factors such as the country in which the surfer is calling the premium rate line from, How much the surfers telephone company charges for a call, How much the paysite charges for access, How much the premium rate phone line provider charges, etc etc.

Generally speaking however, the cost to the surfer is almost always made into profit in your pocket, if a call costs $35(US) you will almost certainly make $35(US) from that surfer minus a small percentage (depending on the provider) again however, this figure may vary slightly.

Phone Billing – Overview.

As with SMS Billing, Premium Rate Phone Billing offers a good alternative for your international surfers to access a paysite however, this doesn’t come without its drawbacks. Unless your members area is updated regularly and is of high quality you are going to make $35(US) approximately of each surfer unless, that is, they decide to call the premium rate number again for access to your site for another month.

That said, if you do not want to offer your foreign surfers the option of having credit card or debit card access to your sites Premium Phone Billing would almost certainly be my second choice to make money from them at the present time.

Article written by Lee

Adult Webmaster Conventions – Remember When?

Date: 2011.02.21 | Category: General | Response: 0

Most webmasters who have been working in the adult industry for a year or two have attended an adult webmaster convention of some sort and, those of us who had the privilege of attending our first webmaster show three to four years ago will know one thing that many others don’t, there used to be a time when you could attend a webmaster show for example, the one held at Disney World (how many of you remember that?) when apart from being relaxing, you would also get a lot of business done alas, times have changed.

Fighting, Drug Taking, Getting Drunk, Rape Accusations, Theft and Criminal damage would be the headline in the national press should one of these reporters attend today’s webmaster convention, instead of a relaxing atmosphere in which you can get business done you find a slew of webmasters who, well, lets be honest here, are only attending these shows for one reason, to get drunk (for free) and to have a laugh.

Of late there seems to have been an influx of people (notice I used the word people and not ‘adult webmasters’) joining the adult industry who would appear to think that our business model is one of all night partying, sordid sex sessions and drug taking, whilst, I will be the first to admit this has gone on in the past it would certainly seem that the last 2-3 years has seen more than its fair share of the negative events happening at these conventions.

Take for example the recent Hollywood Internext Expo show, one incident that comes to mind was the fool who decided it would be a good idea to smash the mezzanine canopy above the hotel lobby, to my knowledge the culprit was never found however, I am sure there are those out there who knows who it was and well, lets be honest, they quite possibly you could be reading this article.

So what happened? How did these shows change so drastically? Quite simply I think it was the sudden onslaught of press coverage about how the adult industry makes so much money for everyone and, whilst there is a lot of money to be made in the industry, it takes a lot of work to achieve making the reported figures by the press.

To many people are entering the adult industry thinking that the ‘norm’ is that of what you can find on your average day of visiting GFY and whist, this is perhaps true for the circle of webmasters who do frequent boards such as GFY for the vast majority of hard working webmasters in the industry, the behavior of those webmasters couldn’t be further from the truth.

So where is all the business being done at the shows? It would appear that event after event the open invite list of parties are being cut down to be taken over by more intimate and sociable private parties in fact, as a company we our ourselves holding several smaller invite only parties for those people that we actually want to do business with on both a personal and a professional level.

So what does the future have in hold for the adult industry events, in all honesty I do not know however, one thing is for certain, if things continue on their current route, things are going to become a whole lot worse before they get any better.

Article written by Lee.

Building Your Own Opt In Mailing List

Date: 2011.02.21 | Category: Opt-in Mail | Response: 0

Despite the constant launch of ‘new and improved’ email collection programs one of the more popular emails I receive each month ask how webmasters can actually start their own opt-in email list for surfers. Surprisingly enough, this is a fairly simple process and, should only realistically take a webmaster 10-15 minutes to have an opt-in email script installed.

However, the second type of email I get from webmasters who have already installed one of these scripts is asking how they can ensure that surfers sign up to their lists this is what I will approach in this article.

One of the easiest ways to make sure surfers join your opt0in list is to offer them a financial gain by doing so, I don’t mean you should literally pay them for their email address but, why not offer them something for free? A daily picture by email, free porn site access etc etc.

The next biggest mistake I find are that webmasters simply place a collection box saying something like ‘enter your email address here for our newsletter’ and, lets be honest, it isn’t very enticing. Ideally you need to have a ‘headline’ that will catch the surfers attention and actually ‘make’ them want to join your list. Be specific yet be vague (bit hard I know) and you’ll soon see your list grow from one new subscriber a week to hundreds of them!

The next thing you need to take into consideration is how quickly, not only your site, but the page with your collection box loads, if you are using a pop-up console, make sure it is heavy in text not graphics with the first words at the top not saying ‘free porn’ but a similar meaning text, everyone and their dog has seen the words ‘free porn’ on countless other porn sites. Yours NEEDS to stand out!

You should use an effective title if you are going to be using pop-up windows for your email collection. in the title bar don’t have the window called ‘pop-up’ have it named appropriately for the niche you are trying to collect the surfers email address from. Make it relevant, make it concise.

Balance the collection page, any type of page you build for your site should at least be appealing to the eye. Meaning that your fonts, colors and actual text need to contrast together perfectly. If you do use images on this page try to have them prominent yet not overbearing to the surfer.

Offer a way to close a pop-up if this is what you are using to collect the emails after all, they (the surfer) may decide they don’t want to give you their email until they have seen what your site can offer. In the same way, make sure you have a link from your sites to your collection page so, if a surfer decides they do want to give you their email address they can do it without the need of closing your site and reloading it afresh.

Overall however, the best rule you can follow is to keep to the point and keep it simple. This way you will gain the surfers attention quickly and easily and, the surfer will not be distracted from the main purpose of having them on your site – getting to your sponsor.

Hopefully the information above will enable you to start having some more success with your traffic and, once you have the surfers email address make sure you treat it with the respect you would want your own to be treat with, an email address is a sacred thing to most surfers, if you Spam them daily they will leave your opt-in list quickly however, if you send them a ‘newsletter’ once a week / month filled with relevant information you will soon find your list growing exponentially.

Article written by Lee.

Basic Bookkeeping For Adult Webmasters

Date: 2011.02.21 | Category: Forms & Contracts | Response: 0

Knowing how much you are spending on expenses for your online enterprise each month can be a very powerful tool. Taking out that shoe box at the end of the year and giving it to the Accountant is one way. But a very simple bookkeeping system can provide you with exactly how much you made each month after all expenses. In this article I will provide you with the knowledge to setup a simple system to track your revenues and expenses. Not only will you then know exactly what your Net Income or Loss is each month but at the end of the year you will have a nice neat little package to hand your Accountant. Which will result in lower accounting fees at tax time.

The first thing you need to do is find a way to file all receipts. Your receipts are the backup for all your transactions. Go to your local office supply store and buy an accordion file labeled with monthly slots. Each time you receive a business related receipt, file in correct month. Remember you need to file all your receipts for your numbers to be accurate. When you receive an email receipt for content or hosting print it out and file it. File sponsor cheque stubs, credit card receipts, ISP bill, etc. At the end of each month take out all your receipts and divide into revenue and expenses. Remember you must have a receipt in order to claim the expense. The receipt should show the detail of the purchase, vendor, amounts and date. Ensure that the expenses are needed or used to generate revenue for your business.

Now the next step depends on how much detail you would like.
Revenue-Expenses=Net Income or Loss. For a quick snapshot of the month add up the revenue receipts, staple all receipts together and put the total on the front either adding machine tape or handwritten. Repeat with all your expense receipts. You can put a quick summary of the month in a spreadsheet program. Or for a very small investment you can setup a computerized tracking system. I recommend buying a very inexpensive software such as Quicken. Most major banks, credit cards and even paypal are setup to download transactions to Quicken. You can purchase the Personal Finance version of Quicken for around $70. There are several other inexpensive personal finance software packages that you can purchase that will work as just well. But I personally prefer Quicken. The software you choose will have a list of business accounts already setup. For an adult enterprise you will need to customize the account names. This is very easy and quick to do. Add in accounts for content purchase, hosting, isp, traffic purchase, etc.

Once a week you will open up your program download your transactions from your bank account, credit card, paypal etc. If you have an expense that you pay cash for you will need to add that transaction in. I am assuming that all your sponsor cheques are being deposited into a bank account or paypal. If you receive a check and cash it, manually add the transaction in. After you have finished entering your transactions for the week ensure that the receipts that you have balance to what you have entered in the software.

You will be amazed at how simple it is to use a software package such as Quicken to track your revenue and expenses. These software packages come with pre-made report tools. Such as income statement, analysis of expenditures, cash flow projections. You can even setup budget numbers for your business.

If your business consists of a lot of transactions in a month you might want to consider using a bookkeeping/accounting service. There are many online and locally owned businesses that charge a very low rate for doing small business bookkeeping. Remember you don’t need an Accountant to do general bookkeeping. Many small business owners think they need an Accountant to do the monthly bookkeeping. This could result in unnecessary fees. Depending on the size of your enterprise and your passion for numbers sometimes employing a bookkeeper to do the data entry can be a blessing! Most bookkeeping providers will even provide customized reporting based on your needs.

Article written by Lorna

Writing A Business Plan – The First Step To Success

Date: 2011.02.21 | Category: Forms & Contracts, Writing | Response: 0

Regardless of what type of business you want to run be it a bakery, a funeral home or an online marketing empire the one thing that you will require is a business plan however, having never taken any business basics 101 classes what do you need to have and, what is required in your business plan?

That is what we will touch on in this brief article from the basics to the not-so-basics by using a few simple steps you can be sure your business plan will contain all the information you need to make it up the ladder of success.

Information Summary.

As the name suggest this is a simple summary of what you are hoping to achieve from the name of your business, the goals, the concept and your plans. This should actually be no longer than two pages so that you can fill the needed information out clearly and concisely.

Business Details.

This will be the founding information of your business again you need to mention your business name, what type of business you will be running (Sole Trader, Partnership, Corporation, etc) in addition you will also need to include details of who the company owners are, whether it be yourself alone or a list of your business partners, this information will be invaluable to you at a later date you also need to include a brief history of the business whether you have been running it already for a number of days, weeks, month or, years.

Products And Services.

Basically, you need to write what exactly it is you will be selling. Are you going to be selling baked goods, marketing services or custom solutions. Keep this section simple, if you need to go into depth then do so at the end of your business plan by way of an appendices. Once you have listed several of the products or services you will be selling you should also list the features and benefits (Marketing points) of the products and services in addition, why do you think your business idea will succeed, what do you offer your clients that no-one else currently does? In addition you also need to have a basic pricing structure planned out, this will assist you in the future as, from your initial plans you can see what, if anything needs to be changed.

Marketing.

A business is more than just an idea. You need to have good pricing, affordable suppliers and a great marketing plan. This is what is also needed in your overall business plan. You should include details of your competitors products and how they compare to your own or, if applicable why your product will be a benefit to the market place should it not already be readily available. In addition the marketing section of your business plan should include a list of advantages as well as some of the pitfalls you are likely to see. You also need to include a section detailing your specific marketing strategies regarding your overall and individual products and services.

Industry Economics.

This area should detail some of the more specific areas of the industry you are going to be working in for example, what type of industry are you entering, how does the future for this industry look and, more importantly, how many existing companies work this industry as well as how do you successfully launch yourself into the industry. Parts of this area you may have covered in the marketing and previous section non the less, go over them again here in more detail.

Target Market.

We have detailed the business ideas and products and services but, what about your potential clients? What does your product offer then that they need which no other company does? What is your target client base? How will you reach your customers? All of this and more needs to be addressed in this area of your business plan.

Competing Companies.

What do your competitors do and, more importantly, who are they? Make a list of your potential competitors and address their weaknesses and strengths ensure that you leave no stone unturned when getting this information together often, you will realize that there are things your competition is doing that you can do much better then they.

Don’t forget to address their marketing, what are they doing right and, more importantly to you, what are they doing wrong? How can you capitalize on this and make their errors benefit your long term business plans.

Location, Location, Location.

Location is everything and, this is particularly true in business for example, will you be working from an office, from your home or even your garage, do you require any special permits to operate your business? Can your customers reach you easily? All of these items and more need to be considered when writing this area of your business plan. By addressing any and all of these items you will find yourself with a much clearer picture of what is needed of you and your business.

Stock + Supplies.

Often overlooked in business plans the one thing you NEED to have is a list of potential suppliers and sub-contractors who can provide you with your products and services. What materials (if any) do you need? How much will these cost you? Where will you find your stock and supplies? By getting this information available now you will have a better idea once your business plan is laid out as to what you are able to offer easily and also it will enable you to lay your pricing structure out more concisely.

Labor.

So you have this great business idea all laid out but, who will be running this for you? Do you need additional staff or are you going it alone? Do you need to setup accountants, lawyers, insurance companies, consultants, bank accounts etc etc? All of these aspects of your business need to be laid out here.

Capital.

Money, as they say, makes the world go round. Your business is no different, how much financial outlay do you require to get your business off the ground? Do you need to ask for a loan from your banking facility to help you out? How much income are you planning on making within the next 6 to 12 months? How much of this income needs to be invested back into the business? All of these ‘financial’ question need to be answered in this area of your business plan only then will you begin to see the bigger picture of your business idea.

Hopefully this article has given you some solid advice and information regarding planning for your new business and, once you have all of the above laid out you should be able to see whether you will succeed or fail in your new idea.

Also, by planning now for the future of your business you should hopefully be able to avoid any hidden costs which can often push a company to the edge.

Article written by Lee

Robots.txt – Control The Robots That Crawl Your Sites

Date: 2011.02.24 | Category: Search Engine Optimization, WebDesign | Response: 0

By writing a structured text file you can indicate to robots that certain parts of your server are off-limits to some or all robots. It is best explained with an example:

# robots.txt file for general use on web servers.

User-agent: webcrawler
Disallow:

User-agent: googlebot
Disallow: /

User-agent: *
Disallow: /cgi-bin
Disallow: /logs
The first line, starting with ‘#’, specifies a comment.

The first paragraph specifies that the robot called ‘webcrawler’ has nothing disallowed: it may go anywhere.

The second paragraph indicates that the robot called ‘googlebot’ has all relative URLs starting with ‘/’ disallowed. Because all relative URL’s on a server start with ‘/’, this means the entire site is closed off.

The third paragraph indicates that all other robots should not visit URLs starting with /cgi-bin or /log. Note the ‘*’ is a special token, meaning “any other User-agent”; you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines.

Two common errors:

Wildcards are not supported: instead of ‘Disallow: /tmp/*’ just say ‘Disallow: /tmp’.
You shouldn’t put more than one path on a Disallow line (this may change in a future version of the spec)
Ultimately, without the use of robots.txt files on your servers/domains, you are risking a variety of potential problems including, unauthorized access to your cgi directory, unauthorized viewing of your site stats, possible spamming of the search engines by accidental crawling of doorway pages.

One distinct advantage however of having a robots.txt file on your server is that, quite simply, you will be able to tell when and where your site has been indexed or potentially indexed as, all robots will automatically call for the robots.txt file BEFORE any other page on your server so, as long as you keep an eye open for any calls of this file, you can see who is knocking at your site for indexing purposes.

Below is a robots.txt example that you can copy and paste into a text document to use on your own server:

<!–Start Copy Below This Line–>

User-agent: *
Disallow: /cgi-bin
Disallow: /logs

<!–End Copy Above This Line–>

The above will allow all spiders to crawl all of your site except the subdirectory’s ‘cgi-bin’ and ‘logs’ which, may be altered to suit any subdirectory’s you do not wish the spiders to crawl on your server.

Article written by Lee