Introduction of Webbot

What is the webbot?

An Internet Bot, also known as web robot, WWW robot or simply bot, is a software application that runs automated tasks (scripts) over the Internet. Typically, bots perform tasks that are both simple and structurally repetitive, at a much higher rate than would be possible for a human alone. The largest use of bots is in web spidering(web Crawler), in which an automated script fetches analyzes the files information from web servers at many times the speed of a human. More than the half of all web traffic is made up of bots. (According to Wikipedia)

So, you  are currently thinking that what is the webbot  and why you should learn this technology?

Basically, the browsers (like Firefox Mozila, Google chrome etc ) which we use in our day to day life is nothing but the simple manual tool  which only performs downloads and renders websites. Browsers can’t think for itself. You always have to decide that  webpage which you are surfing is contain relevant information. Browsers can’t notify when something important happens online, and it certainly won’t anticipate your actions, automatically complete forms, make purchase or download file for you.

To do these things, you’ll need the automation and intelligence only available with a webbot, or web robot. Once you start thinking about the inherent limitations of browsers, you start to see the endless opportunities that wait around the corner for webbot developer.

Inspirations from browser limitations:

A very useful method for generating ideas about webbot projects is to study what can’t be done by simply pointing a browser at a typical website. Browser is a simple manual tool so it can’t automate your internet experience. For example, it has the limitations like:

• ·       Browsers cannot aggregate and filter information for relevance.
• ·       Browsers cannot interpret what they find online.
• ·       Browsers cannot act on your behalf.

However, a browser may leverage the power of a webbot to do many things that it could not do alone. Let’s look at some real-life examples of how browser limitations were leveraged into actual webbot projects.

Webbots That Aggregate and Filter Information for Relevance:

TrackRates.com (http://www.trackrates.com, shown in Figure) is a website that deploys an army of webbots to aggregate and filter hotel room prices from travel websites. By identifying room prices for specific hotels for specific dates, it determines the actual market value for rooms up to three months into the future. This information helps hotel managers intelligently price rooms by specifically knowing what the competition is charging for similar rooms. TrackRates.com also reveals market trends by performing statistical analysis on room prices, and it tries to determine periods of high demand by indicating dates on which hotels have booked all of their rooms.

Webbots That Interpret What They Find Online:

WebSiteOptimization.com (http://www.websiteoptimization.com) uses a webbot to help web developers create websites that use resources effectively. This webbot accepts a web page’s URL (as shown in Figure) and analyzes how each graphic, CSS, and JavaScript file is used by the web page.

The WebSiteOptimization.com webbot analyzes the data it collects and offers suggestions for optimizing website performance. Without this tool, developers would have to manually parse through their HTML code to determine which files are required by web pages, how much bandwidth they are using, and how the organization of the web page affects its performance.

Help Out a Busy Executive:

Suppose you’re a busy executive type and you like to start your day reading your online industry publication. Time is limited, however, and you only let yourself read industry news until you’ve finished your first cup of coffee. Therefore, you don’t want to be bothered with stories that you’ve read before or that you know are not relevant to your business. You ask your developer webbots2e.book Page 18 Thursday, February 16, 2012 11:59 AM Ideas for Webbot Projects 19 to create a specialized webbot that consolidates articles from your favorite industry news sources and only displays links to stories that it has not shown you before. The webbot could ignore articles that contain certain key phrases you previously entered in an exclusion list 1 and highlight articles that contain references to you or your competitors. With such an application, you could quickly scan what’s happening in your industry and only spend time reading relevant articles. You might even have more time to enjoy your coffee.

Save Money by Automating Tasks:

It’s possible to design a webbot that automatically buys inventory for a store, given a predetermined set of buying criteria. For example, assume you own a store that sells used travel gear. Some of your sources for inventory are online auction websites.2 Say you are interested in bidding on under-priced Tumi suitcases during the closing minute of their auctions. If you don’t use a webbot of some sort, you will have to use a web browser to check each auction site periodically. Without a webbot, it can be expensive to use the Internet in a business setting, because repetitive tasks (like procuring inventory) are time consuming without automation. Additionally, the more mundane the task, the greater the opportunity for human error. Checking online auctions for products to resell could easily consume one or two hours a day—up to 25 percent of a 40-hour work week. At that rate, someone with an annual salary of $80,000 would cost a company $20,000 a year to procure inventory (without a webbot). That cost does not include the cost of opportunities lost while the employee manually monitors auction sites. In scenarios like this, it’s easy to see how product acquisition with a webbot saves a lot of money—even for a small business with small requirements. Additionally, a webbot may uncover bargains missed by someone manually searching the auction site.

There are many more webbot ideas and exist webbot projects available. Studying browser  limitation is one way to uncover ideas for new webbot designs.

In the next chapter we will learn basics of webbot development(Downloading webpages).

Thanks for reading,

Akshay Panchal.