Simple - it's a spider, a piece of software which grabs specific webpage data. It can grab data from tens of thousands of sites, and then present the data collected in nice, neat pages WITHOUT THE NEED OF ANY DATABASE. Yes NO database programming required!
If you haven't already downloaded it then you can do so from HERE.
When visiting a website, Niche Spider also leaves it's calling card in the site's visitor logs (the 'Referer' and 'User-Agent', which you can set), just like Googlebot or any other spider or web crawler. That, in itself, is a great way to announce yourself to the world, and to generate some curiosity/traffic at the same time.
However, Niche Spider is NOT Googlebot, or Yahoo's Slurp, or MSN's MSNBot. Google, Yahoo and MSN have access to computers, servers and databases us lesser mortals can only dream about. And the spiders they use crawl the web, looking for and following links to pages, which are then added to their respective databases. Such is their power they can traverse the entire web in a month or two, or less, adding and updating billions of pages in the process.
Niche Spider does NOT crawl the web, but targets a small piece of the web instead (a niche area). You decide what that niche is. It may, or may not be related to your website's theme. That's completely up to you.
And before we get started, there's a couple of other points. Niche Spider is FREE. However, in return, the odd link is added to the bottom of one or two of the pages produced (not every page). Of course, you can delete any such links if you wish. But if you keep any on, however, it would be very much appreciated.
It is also possible to make some cash by adding Google AdSense code to each template. Or you can include whatever affiliate products you may be aiming to promote. It's up to you. But they'll then appear on every page produced by Niche Spider. Enough said!
Check out this site, which was quickly thrown together to show Niche Spider in action. And a word of caution - don't go crazy producing hundreds/thousands of pages daily. Google won't thank you for it. Create a few dozen or so a day, every day, and Google will keep coming back for more. Remember, there are hundreds of keywords out there, and each one can be used to create 10/20/30 pages or more (soon adds up to an awful lot of pages). Be sensible!!!
Want to know if there are any updates? Send me an email (info at freebieads.co.uk) and I'll add you to my list. Or you can simply send me any comments.
A timeout bug, which caused the Niche Spider to traverse the URLs list without actually waiting to receive data, has been corrected. Now the default timeout value is set at 60. KEEP IT AT AROUND THE DEFAULT VALUE!
So if you see the error message "Error 12002 Timeout", that's what you are supposed to see. Just wait...and then after about 20 or 30 seconds, Niche Spider will continue on it's way!
Niche Spider is shown below. It may look complicated, but it's not. Just read this help file a couple of times and you'll soon get the hang of it.
 |
As can be seen, there are four main areas, moving from left to right. They are Keywords, URLs, Spider URLs, and Page Manager. There's one further area not shown, the Template area, accessed by means of the main menu running along the top of the program. The menu is self-explanatory, so no need to go into any great detail, except to say that |
| keyword lists, url lists, spider data and templates can all be saved and reused, again and again. | |
This is the first stage in the process. Enter a keyword in the text box just below the Input keyword and hit 'Enter' label, illustrated in the image below. Then wait for a number of seconds as Niche Spider queries Overture, then populates the listbox underneath.
 |
The number to the left of each keyword represents the number of searches carried out the previous month for that particular keyword. Note: you can change which Overture server to query for a more country-specific set of keywords. Sometimes the Overture server may be down, or there may be too many people trying to access it at the same time. As a consequence, no results may be returned. Simply repeat. If it's still down, simply click the 'Add' button to the right of the keyword entry box. The keyword will then appear in the listbox below, but without the number of searches figure. Save the list via the main menu - clicking on Keywords | Save Keywords. Name the list something meaningful, too. It all helps from an organizational point of view!
|
We've got our list of keywords (our niche) and completed the first stage. Now we grab some urls to spider - via Google (of course, being the Number 1 search engine). Niche Spider queries Google in a very considerate manner, with a random 5 to 15 second gap between each query. That may be slow, but it won't get you banned, either!
 |
Now examine the image to the left. The Grab urls area allows you to choose whether to add the home page of the website concerned to the list below, or the actual url of the page that appears in Google's search results. The former (the home page) is recommended (the default). But if you want the actual url of the page then click the Check for full url: checkbox. And you've also got a choice with regard to which Google server you want to query, google.com or google.co.uk. Google.com is the default. Then simply click the Go! button and watch as the listbox below fills up with urls. Notice the small square which changes constantly from blue to red, to let you know the query process is active. You can stop the process at any time by going to the main menu and clicking Stop | Stop All. Then click URLs in the main menu to save the list. |
|
You can, if you wish collect up to 30,000 urls for any one run! However, in reality, about 2,000 urls for any niche category is more than adequate, because you can have lots of related niches (in fact, as many as you want, e.g. secured loans, personal loans, car loans ... and so on). |
|
 |
IMPORTANT: notice the Randomize list button at the bottom of the listbox of urls? Click it and the list order is randomly mixed up. Do it again, and again, or as often as you want. Then save the list. That way, YOUR list (and therefore the pages YOU create) will always be different/unique from someone else targetting the same niche keywords! |
|
PLEASE MAKE USE OF THIS ASPECT OF NICHE SPIDER. DON'T IGNORE THIS! It can also be used to refresh data already spidered, enabling new content on pages already up on your website. Great for the spiders/crawlers - keeps them coming back! A further point - double-clicking any entry will delete it from the list. This is particularly useful if you come across the occasional site (and you do) which 'hangs' the program because it might be having a problem (internal server error). NOTE: some sites are very SLOW and may take up to a minute to actually respond! Don't panic - just give Niche Spider a decent amount of time to do its work. If you do come across a site which appears to 'choke' Niche Spider, try to delete it from the list by double-clicking, and then resave the list and try again. VERY IMPORTANT: once the offending site has been removed, and the list resaved (without it), you may have to press Ctrl, Alt + Delete on your computer to exit out of Niche Spider. Then reload the list and try again, from the beginning. It's a hassle, but there's little that can be done. The good thing is that that doesn't happen too often. And because Niche Spider is fairly quick, you shouldn't have lost too much time. |
|
Now to the third part of the process - and this is in two stages, the first being a 'dry run', and the second the creation of the actual pages themselves.
The 'dry run'
This is where Niche Spider goes through the WHOLE list of urls collected. Collect 1,000 urls to spider and maybe about 1 in 4 will actually end up on a page. Why? Lot's of reasons, the main one being the page title of the spidered page does not match the word you've added to the textbox below the Spidered page Title must contain label, as shown below.
 |
If the Title tag of the page being spidered doesn't contain the keyword (in this case the word mortgage) then it won't be added to the sites to be added to a page you'll be creating later. |
|
That way, the pages you'll be creating further on in the process will be tightly themed, will be more useful for your visitors, and be more acceptable to Google and other search engines. |
|
The listbox may have contained, say, 1,000 urls to be spidered. At the end of the run it may now contain only 200 urls.
Don't worry - this IS what you want. Save it to a directory called Niche Lists, or something like that, on your computer. Then when you want to respider, randomize the niche list. It'll take literally a minute or two to traverse the niche list and to create a brand new set of fresh pages! You can do this once a week/month or whatever.
Remember, too, with a list of 200 urls, and with five results per page (more on that later), you'll end up creating around 40 pages. Spider another four niche categories and the page count goes up to several hundred ... and so on!
This is the heart of the whole process, the creation of the pages themselves. Remember when the original list of urls was reduced (e.g. from 1,000 to maybe 200)? The reduced list was saved in our niche list directory on our computer. Now load the niche list by going to the main menu and clicking URLs | Open URLs and navigating to the niche list directory.
But before continuing, we need to talk about the Template we're going to use (main menu, Template | Show Template). You'll see that Niche Spider comes with a simple, illustrative template, already loaded which you would never dream of using! But it does give an idea of what is involved.
 |
Hopefully, it's all self-explanatory. Basically, in order to proceed with the creation of pages, Niche Spider looks for three symbols in every template -- !@@!, !%%! and !##!. When it does, it substitutes the spidered data for these symbols, thus creating a complete page, each containing details of x-number of sites (choose how many via the Results per page dropdown box within the page manager). Creating a template to match your own site is as easy as loading up your favourite HTML editor with any page from your site, clearing it of any text (leave the bare-bones HTML!), and then substituting the symbols in the appropriate places. Then save the template in order to use it as often as you like. It may seem tricky to do all of the above at first, but it's not. You'll get the hang of it. Once you've done it, save your work (main menu, Template | Save Template). |
Now click Spider on the main menu, then Show Spider to get back to the page manager.
 |
A little bit of explanation is required about exactly what data is collected by Niche Spider. Look at the graphic on the left, and at the textbox at the top. There's been a small update in this area so please check here before reading further. You'll see three lines, the first being the Title used in the webpage being spidered. The second line is the description used in the webpage's description meta tag. The third line is the url of the webpage. Notice how the word 'mortgage', contained in the text box below the Spidered page Title must contain label, is also found in the webpage's title. If it wasn't there then the webpage data would not be used. You've a choice whether to use title, description, and url, or description and url only. Just check the appropriate checkbox. And you've also the option of making the title into a clickable link, or the url itself. You can also choose the number of results to use in each page. The default, of 5 results, is more than adequate. |
Now let's look at the page manager settings when we come to creating the actual pages. Remember, the 'dry run' has been completed previously and the resulting smaller list of successfully spidered results saved to our niche list directory on our computer.
 |
This is where we create actual pages which we'll upload to our website. We've already explained the entries in the Spidered page Title must contain textbox and the Results per page dropdown box. The two boxes below these, Directory name and Page url, contain the name of the folder in which Niche Spider will save the newly created pages on your computer, and the actual name used for each page created - but with a different number added to each. So pages will be named (in this example) mortgage1.html, mortgage5.html, mortgage20.html etc.. |
|
Now, let's consider the Sitemap name and Sitemap <H1> heading boxes. If you know anything about websites and directories, then you'll know if a directory is created, in which there is a page called index.html, that page will be called automatically, without referring to the whole of the url. Click a link pointing to www.mywebsite.com/mortgages/ and www.mywebsite.com/mortgages/index.html will be loaded up in the visitor's web browser. SO IT'S A GOOD IDEA TO SIMPLY PUT 'INDEX' INTO THE SITEMAP NAME BOX! The other box is simply the heading you want included in the sitemap page created. The Font dropdown box and the font Size dropdown box are both self-explanatory. The Sitemap links vertical checkbox gives the choice of having the links on the sitemap page (pointing to the created pages) running either vertically (checked)or horizontally (unchecked). |
|
|
NOW WE ARE READY TO CREATE SOME PAGES! Check the Create pages checkbox and then click the Set Niche Spider loose! button. Within a minute, or two, or three... the pages will have been created. Simply upload to your website. IMPORTANT IMPORTANT IMPORTANT IMPORTANT IMPORTANT IMPORTANT IMPORTANT IMPORTANT Remember to include a link on your home page to the newly created sitemap page (which contains the links to all the pages newly created). Make the home page link point to http://www.mywebsite.com/mortgage/ which will automatically pull in the sitemap index.html page. One last point - why not simply ftp the whole of the newly created directory on your computer to your website. Then add a copy of all of the images used by the webpage template to that directory, too. Then all the paths to the images will be correct. Note: if you house all your images in a specific directory on your website (i.e. it may be called 'images', or something like that), as many sites do, then when you upload your newly created niche directory with sitemap and pages, upload your images directory within the niche directory. This will ensure the paths are all correct. |
Post freeads on freebieads.co.uk anytime!