root/gootrude/trunk/README

Gootrude - The Search Engine Results Trender

The purpose of this project is to accept a set of search terms to be input
into Yahoo and (optionally) Google and graph the numeric result returned over
time.  This allows changes in search results to be visualized graphically, and
large changes can indicate significant usage of related search terms in high
profile websites around the Internet.  Moving averages are supported, and
Gnuplot is used (currently) as the graphing software.  Gootrude is an open
source answer to the problem where search engines do not generally expose
trends in how they build their search indexes for arbitrary search terms.  All
information collected by Gootrude is returned by Yahoo through normal usage of
the Yahoo website - Gootrude just collects this information and graphs it over
time.

Gootrude is released under the GPL as free and open source software.  Please
email Michael Rash (mbr[at]cipherdyne[dot]org) with any questions or concerns.

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
WARNING:  According to the Google Terms of Service, automated web queries of
Google are not permitted.  Hence, the default search engine that Gootrude uses
is Yahoo for all queries, but Gootrude supports Google searches if you choose
to enable that feature.  However, you do this EXCLUSIVELY AT YOUR OWN RISK -
YOU HAVE BEEN WARNED - Google may choose to block your queries.  That said,
some things to keep in mind though are that Google is probably mostly
interested in making sure that the quality of its services is as high as
possible, and so it doesn't want huge numbers of automated queries against
its infrastructure.  So, if you choose to configure Gootrude to query Google,
then please make sure that Gootrude is not run more than once per day (as per
its design), and also make sure that the number of search terms is not high
(say, less than 20).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

QUICK START:

The basic deployment scenario for Gootrude on a Linux system is:

0) Gootrude requires the following software to be installed:

perl, and the Date::Calc CPAN module
wget
gnuplot

1) Download and extract the Gootrude source tarball within a directory, say
within /home/mbr/src:

$ cd /home/mbr/src
$ wget http://www.cipherdyne.org/gootrude/download/gootrude-0.1.tar.gz
$ tar xfz gootrude-0.1.tar.gz

2) Edit the searchterms.conf file:

$ cd gootrude-0.1
$ vim searchterms.conf

3) Within the searchterms.conf file, add search terms that you want to trend
over time accorind to the following format (with the brackets):

[search term] [search_engine:type] [file]

Here are two examples in the searchterms.conf file:

[Linux "highspeed firewalls"] [yahoo:count] [Linux_highspeed_firewalls.dat]
[http://www.slashdot.org] [google:link] [slashdot_links.dat]

The first [Linux "highspeed firewalls"] search term is input (with quotes)
into Yahoo as a normal search (as defined by [yahoo:count]), and the results
will be stored in the file [Linux_highspeed_firewall.dat] (within the
gootrude_plot/ directory - see the GOOTRUDE_PLOT_DIR variable in the
gootrude.conf file).

The second [http://www.slashdot.org] queries Google for the number of
backwards links to http://www.slashdot.org (see the [google:link] type) and
places the results in the [slashdot_links.dat] file (also within the
gootrude_plot/ directory).

All Gootrude data is stored within the gootrude_plot/ directory, and .png
graphics files are also created within this directory.

4) It is recommend to run the gootrude script on a regular basis out of cron
like so:

0 1 * * * cd /home/mbr/src/gootrude && ./gootrude

This will run Gootrude once per day at 1am.