Subversion Repositories SmartDukaan

Price Analyser
        It's basically a information retrieval and extraction tool. In our case this information is price and other features for a phone from different suppliers.
        The tool can also be used for comparison between the two supplier for a same phone.
        Brief description of its working:
                With the help of Turbogears2.0 forms are prepared for specifying certain parameters which would be helpful in which web page we want to retrieve
                and from that web page what we want to retrieve. These values are dumped into database.
                
                Now using Scrapy and Xpaths specifed as parameters in the forms, various information about the phones are captured and dumped into the database.

                Google charting API's are used for comparison and graphical viewing pleasure.

                The software also provides querying and searching functionality(a basic search engine functionality). Lucene is used for embedding this feature
                into our application.   



Detailled description:
        Forms part:
                VIEW    
                path to files : home/gaurav/code/TG2TEST/src/wiki20/templates
                codedata_forms.html: Give us the link for individual forms
                input_form.html : Form for a particular supplier, supplier depends on the link choosen in the codedata_forms.html

                MODEL   
                path to files : home/gaurav/code/TG2TEST/src/wiki20/widgets
                babuchak_form.py : Structure of the form for babuchak, contains the parameters that needs to be taken as input from the user 
                indiaplaza_form.py : Structure of the form for indiaplaza, contains the parameters that needs to be taken as input from the user 
                infibeam_form.py : Structure of the form for infibeam, contains the parameters that needs to be taken as input from the user 
                mobilestore_form.py : Structure of the form for mobilestore, contains the parameters that needs to be taken as input from the user 
                naaptol_form.py : Structure of the form for naaptol, contains the parameters that needs to be taken as input from the user 
                univercell_form.py : Structure of the form for univercell, contains the parameters that needs to be taken as input from the user 

                CONTROLLER
                path to files : home/gaurav/code/TG2TEST/src/wiki20/controllers
                root.py : All the processing part is done here, input is taken here, on submit this input is saved in database.
                There are two type of methods:
                1) form_infibeam : To create the structure it calls the template and pass it the values present in DB as default values
                2) save_infibeam : Take the values given by user, store it in DB and then then show the template for taking input again.
                Same template is used for both the type of methods.
        Scrapy: 
        Here individual spiders are written for each supplier, for some extra scripts are also needed because the number of pages they have for information
        is variable. These include infibeam, indiaplaza and mobilestore. These scripts are places in home/gaurav/code named runinfibeam, runindiaplaza and 
        runmobstore respectively.

        ******************************************INFIBEAM**********************************************************************************

        Documentation for script runinfibeam.py
        This is the script called by consetup.py
        First it will run the spider for infibeam dynamically (i.e for determining no of pages)
        Then, it will generate the csv file
        @param  path to the folder in which spider-projects reside (:/home/gaurav/code) but start with pathsep

        Documentation for class infi_spider
        This spider collects the information for the individual phones
        and store them in table datastore_datadefinition_infibeam_data  
        
        Documentation for class infibeam_data
        It represents database table for infibeam, it stores
        name = name of the phone
        shown_price = price offered by infibeam
        final_price = price which one has to pay, 
        final_price = shown_price + taxes + ship-price

        Documentation for method add_infiphone 
        This method is used to add a phone in infibeam's table

        Documentation for method get_all_infibeam_data 
        This method is used to retrieve all the phones in infibeam's table
        
        Documentation of various parameters     
        Xpath1 = Give us section for individual phone
        Xpath2 = Give us name for individual phone
        Xpath3 = Give us quoted price for individual phone
        vatplustax = to get final price from quoted price
        Removelist = To filer the prices so as to make them integer for eg remove ',' or 'Rs'
        URL = The start url for the supplier from this page we start crawling
        homepage = Homepage for the supplier
        referer = For some suppliers, the spiders were unable to fetch the data without setting this field, I set it to google/search
        domainname = Name by which this spider is known outside, passed as an argument when calling this spider

      **************************************************************************************************************************************


      ******************************************INDIAPLAZA**********************************************************************************
        
        Documentation for script runindiaplaza.py
        This is the script called by consetup.py
        First it will run the spiders for indiaplaza dynamically (i.e for determining no of pages)
        Then, it will generate the csv file
        @param  path to the folder in which spider-projects reside (:/home/gaurav/code) but start with pathsep  
        
        Documentation for class indiaplaza_spider
        This spider collects the url for the individual phones
        and store them in table datastore_datadefinition_indiaplaza_data.

        Documentation for class indiaplaza_extra
        This spider collects all the information for the individual phones
        and store them in table datastore_datadefinition_indiaplaza_items.

        Documentation for class indiaplaza_data
        It represents database table for indiaplaza, it stores
        v_name = name of the vendor
        v_site = url of the vendor

        Documentation for class indiaplaza_items
        It represents database table for indiaplaza, it stores
        p_name = name of the phone
        p_shown_price = price offered by indiaplaza
        p_final_price = price which one has to pay, 
        p_final_price = p_shown_price + taxes + ship-price
        p_guaranteeinfo = duaration of guarantee and whether guarantee is from vendor or manufacturer 
        p_shipinfo = how much time would be taken for shipping
        
        Documentation for method add_ipbasic 
        This method is used to add a url for phone in indiaplaza's table
        
        Documentation for method get_all_ipbasic 
        This method is used to retrieve all phone-urls in indiaplaza's table

        Documentation for method add_ipextra 
        This method is used to add a phone in indiaplaza's table

        Documentation for method get_all_indiaplaza_phones 
        This method is used to retrieve all the phones in indiaplaza's table

        Documentation of various parameters     
        Xpath1 = Give us section for individual phone
        Xpath2 = Give us name of individual phone
        Xpath3 = Give us url of individual phone
        Url1 = To get full url for individual phones
        Xpath4 = Give us name for individual phone
        Xpath5 = Give us quoted-price for individual phone
        Xpath6 = Give us ship-price for individual phone
        Xpath7 = Give us ship_price for individual phone, if not gettable form xpath6
        Xpath8 = Give us guarantee-info for individual phone
        Xpath9 = Give us guarantee-info for individual phone, if not gettable form xpath8
        Xpath10 = Give us ship-info for individual phone
        Removelist = To filer the prices so as to make them integer for eg remove ',' or 'Rs'
        URL = The start url for the supplier from this page we start crawling
        homepage = Homepage for the supplier
        referer = For some suppliers, the spiders were unable to fetch the data without setting this field, I set it to google/search
        domainname = Name by which the first spider is known outside, passed as an argument when calling this spider
        domainname1 = Name by which the second spider is known outside, passed as an argument when calling this spider
        var1 = To check for free shipping
        
      **************************************************************************************************************************************    
        
      ******************************************MOBILESTORE**********************************************************************************
        
        Documentation for script runmobstore.py
        This is the script called by consetup.py
        First it will run the spider for mobilestore dynamically (i.e for determining no of pages)
        Then, it will generate the csv file
        @param  path to the folder in which spider-projects reside (:/home/gaurav/code) but start with pathsep

        Documentation for class mobilestore_spider0
        This spider collects the information for the individual phones
        and store them in table datastore_datadefinition_themobilestorephones_new                        
        
        Documentation for class themobilestorephones_new
        It represents database table for themobilestore, it stores
        name = name of the phone
        shown_price = price offered by themobilestore
        final_price = price which one has to pay, 
        final_price = shown_price + taxes + ship-price
        extra_info = whether phone can be bought or not

        Documentation for method add_new_mobstorephone_new 
        This method is used to add a phone in themobilestore's table

        Documentation for method get_allmobstorephone_new 
        This method is used to retrieve all the phones in themobilestore's table

        Documentation of various parameters     
        Xpath3 = Give us name for individual phone
        Xpath4 = Give us price for individual phone
        Xpath5 = Give us name for individual phone, if its not gettable from xpath3
        Xpath6 = Give us name for individual phone, if its not gettable from xpath3 and xpath5
        Xpath7 = to check that the phone can be bought or not
        Xpath8 = to check that the item is mobile phone
        url1 and url2 = used for getting actual start urls
        homepage = Homepage for the supplier
        referer = For some suppliers, the spiders were unable to fetch the data without setting this field, I set it to google/search
        domainname0 = Name by which this spider is known outside, passed as an argument when calling this spider
        
        **************************************************************************************************************************************  
        
        ******************************************UNIVERCELL**********************************************************************************

        Documentation for class vendor_links
        This spider collects the url for the individual vendors 
        and store them in table datastore_datadefinition_univercell_data.

        Documentation for class univercell_price
        This spider collects the information for the individual phones
        and store them in table datastore_datadefinition_univercell_items
        
        Documentation for class univercell_data
        It represents database table for univercell, it stores
        v_name = name of the vendor
        v_site = url of the vendor

        Documentation for class univercell_items
        It represents database table for univercell, it stores
        p_title = name of the phone
        p_shown_price = price offered by univercell
        p_final_price = price which one has to pay, 
        p_final_price = p_shown_price + taxes + ship-price

        Documentation for method add_univervendor 
        This method is used to add a vendor in univercell's table
        
        Documentation for method get_all_univervendor 
        This method is used to retrieve all the vendors in univercell's table

        Documentation for method add_univerphone 
        This method is used to add a phone in univercell's table

        Documentation for method get_all_univercell_phones 
        This method is used to retrieve all the phones in univercell's table

        Documentation of various parameters     
        Xpath1 = Give us section for individual vendors
        Xpath2 = Give us name for individual vendors
        Xpath3 = Give us url for individual vendors
        Url1 = To get full url for individual vendors
        var1,var2,var3 and var4 are used to get proper url
        Xpath4 = Give us section for individual phone
        Xpath5 = Give us name for individual phone
        Xpath6 = Give us quoted-price for individual phone
        vatplustax = Give us final_price for individual phone on adding with quoted-price
        Removelist = To filer the prices so as to make them integer for eg remove ',' or 'Rs'
        URL = The start url for the supplier from this page we start crawling
        homepage = Homepage for the supplier
        referer = For some suppliers, the spiders were unable to fetch the data without setting this field, I set it to google/search
        domainname = Name by which the first spider is known outside, passed as an argument when calling this spider
        domainname1 = Name by which the second spider is known outside, passed as an argument when calling this spider
        Url1 = To get full url for individual vendor

        **************************************************************************************************************************************  
        
        ******************************************BABUCHAK**********************************************************************************

        Documentation for class babuchak1
        This spider collects the url for the individual vendors 
        and store them in table datastore_datadefinition_babuchak_urls.
        
        Documentation for class babuchak2
        This spider collects the url for the individual phones
        and store them in table datastore_datadefinition_babuchak_phoneurls.

        Documentation for class babuchak3
        This spider collects the information for the individual phones
        and store them in table datastore_datadefinition_babuchak_phones.

        Documentation for class babuchak_urls
        It represents database table for babuchak, it stores
        url = url for the vendors
        no_pages = number of pages for individual vendor

        Documentation for class babuchak_phoneurls
        It represents database table for babuchak, it stores
        url = url for the individual phones

        Documentation for class babuchak_phones
        It represents database table for babuchak, it stores
        name = name of the phone
        shown_price = price offered by babuchak
        final_price = price which one has to pay, 
        final_price = shown_price + taxes + ship-price

        Documentation for method add_babuchakurl 
        This method is used to add a url for vendor in babuchak's table

        Documentation for method get_allbabuchakurls 
        This method is used to retrieve all the vendor-urls in babuchak's table
        
        Documentation for method add_babuchakphoneurl 
        This method is used to add a url for phone in babuchak's table

        Documentation for method get_allbabuchakphoneurls 
        This method is used to retrieve all the phone-urls in babuchak's table

        Documentation for method add_babuchakphone 
        This method is used to add a phone in babuchak's table

        Documentation for method get_allbabuchakphones 
        This method is used to retrieve all the phone in babuchak's table

        Documentation of various parameters     
        Xpath1 = Give us section for individual vendors
        Xpath2 = Give us no of pages for individual vendors
        Xpath3 = Give us url for individual vendors
        Url1 = To get full url for individual vendors
        Xpath4 = Give us url for individual phone
        Url2 = To get full url for individual vendors
        Xpath5 = Give us name for individual phone
        Xpath6 = Give us quoted-price for individual phone
        Xpath7 = Give us final_price for individual phone
        Removelist = To filer the prices so as to make them integer for eg remove ',' or 'Rs'
        URL = The start url for the supplier from this page we start crawling
        homepage = Homepage for the supplier
        referer = For some suppliers, the spiders were unable to fetch the data without setting this field, I set it to google/search
        domainname = Name by which the first spider is known outside, passed as an argument when calling this spider
        domainname1 = Name by which the second spider is known outside, passed as an argument when calling this spider
        domainname2 = Name by which the third spider is known outside, passed as an argument when calling this spider

        **************************************************************************************************************************************  
        
        ******************************************NAAPTOL**********************************************************************************
                
        Documentation for class naaptol_spider
        This spider collects the url for the individual phones
        and store them in table datastore_datadefinition_naaptol_urls.  

        Documentation for class naaptol_price
        Since the urls collected in the previous spider for naaptol.com
        are redirected to get the data for individual phones.
        Some are of the form "http://www.naaptol.com/features/10417-Fly-E300.html"
        while others are of the form "http://www.naaptol.com/price/10417-Fly-E300.html".
        So to make data extraction symmetric, this spider will accomplish 2 tasks
        First, for the urls conatining 'features' it collects the information for the 
        individual phones and store them in table datastore_datadefinition_naaptol_phones
        for the ones conatining 'prices' in the url, a new url having 'price' repalced  
        with 'features' is framed and stored in the table datastore_datadefinition_morenaaptol_urls.

        Documentation for class naaptol_price
        Spider collects the information for the individual phones and store them in table 
        datastore_datadefinition_naaptol_phones

        Documentation for class naaptolurls
        It represents database table for naaptol, it stores
        url = url of the phones, which we got from sitemap.xml

        Documentation for class naaptolurls
        It represents database table for naaptol, it stores
        url = url of the phones, here urls are the ones which are redirected 
        and contained 'price' but before storing 'price' is replaced by 'features'      
        
        Documentation for class naaptolphones
        It represents database table for naaptol, it stores
        name = name of the phone, 
        range = price range for each phone
        range is in one of the 3 forms, i.e
        range = a to b
        range = a
        range = a(approx)
        here a,b are integers

        Documentation for class ntonlinesp
        It represents database table for naaptol, it stores
        nid = id of the phone in naaptolphones 
        name = name of the onlinesupplier, 
        price = price offered by the supplier for the phone

        Documentation for class ntofflinesp
        It represents database table for naaptol, it stores
        nid = id of the phone in naaptolphones 
        name = name of the offlinesupplier, 
        price = price offered by the supplier for the phone

        Documentation for method add_naaptolurl 
        This method is used to add a url for phone in naaptol's table
        These are taken from sitemap.xml

        Documentation for method get_allnaaptolurls 
        This method is used to retrieve all the url for phones in naaptol's table

        Documentation for method add_morenaaptolurl 
        This method is used to add a url for phone in naaptol's table
        These are the urls generated by replacing 'price' with 'features'
        which were redirected

        Documentation for method get_allmorenaaptolurls 
        This method is used to retrieve all the extra urls for phones in naaptol's table

        Documentation for method add_new_naaptolphone 
        This method is used to add a phone in naaptol's table

        Documentation for method get_naaptolphone 
        This method is used to retrieve a phone in naaptol's table
        given its name and range

        Documentation for method get_allnaaptolphones 
        This method is used to retrieve all the phone in naaptol's table

        Documentation for method add_new_ntonlinesp 
        This method is used to add a online-supplier for a particular phone
        in naaptol's table

        Documentation for method get_allntonlinesp 
        This method is used to retrieve all the online-suppliers in naaptol's table

        Documentation for method get_allntonlinespbynid 
        This method is used to retrieve all the online-supplier for a particular phone 
        in naaptol's table given the id of phone in naaptolphones

        Documentation for method add_new_ntofflinesp 
        This method is used to add a offline-supplier for a particular phone
        in naaptol's table

        Documentation for method get_allntofflinesp 
        This method is used to retrieve all the offline-suppliers in naaptol's table

        Documentation for method get_allntolinespbynid 
        This method is used to retrieve all the offline-supplier for a particular phone 
        in naaptol's table given the id of phone in naaptolphones

        Documentation of various parameters     
        Xpath1 = Give us url for individual phones
        chklist1 = elements in chk_list are specific to this site for determining valid sites
        Xpath2 = Give us price-range for individual phone
        Xpath3 = Give us price-range for individual phone, if unable to retrieve from xpath2
        Xpath4 = Give us number of onlinesellers for a particular phone
        Xpath5 = Give us price for a particular phone offered by onlinesellers
        Xpath6 and Xpath7 = Give us name of onlinesellers for a particular phone 
        Xpath8 = Give us number of offlinesellers for a particular phone
        Xpath9 = Give us price for a particular phone offered by offlinesellers
        Xpath10 = Give us name of offlinesellers for a particular phone
        Removelist = To filer the prices so as to make them integer for eg remove ',' or 'Rs'
        chklist2 = contains what needs to be replaced, presently it conatains 'price'
        part = contains 'features'
        URL = The start url for the supplier from this page we start crawling
        homepage = Homepage for the supplier
        referer = For some suppliers, the spiders were unable to fetch the data without setting this field, I set it to google/search
        domainname = Name by which the first spider is known outside, passed as an argument when calling this spider
        domainname1 = Name by which the second spider is known outside, passed as an argument when calling this spider
        domainname2 = Name by which the third spider is known outside, passed as an argument when calling this spider

        **************************************************************************************************************************************  
        
        ******************************************SUPPLIERS**********************************************************************************
        
        Documentation for class suppliers
        It represents database table for suppliers, it stores
        name = name of the supplier
        site = url of the supplier
        last_crawled = date of the last run for this supplier
        This table spans all the suppliers in our database
        
        Documentation for method add_suppliers 
        This method is used to add a supplier in supplier's table

        Documentation for method get_all_suppliers 
        This method is used to retrieve all the suppliers from the supplier's table

        Documentation for method get_suppId 
        This method is used to retrieve id of the supplier given his name
        from the supplier's table

        Documentation for method get_supp_byId 
        This method is used to retrieve a supplier given his id
        from the supplier's table

        Documentation for method get_suppbyName 
        This method is used to retrieve a supplier given his name
        from the supplier's table

        Documentation for method get_suppbySite 
        This method is used to retrieve a supplier given his url
        from the supplier's table

        **************************************************************************************************************************************  

        ******************************************MODELS**********************************************************************************

        Documentation for class models
        It represents database table for models, it stores
        brand = name of the brand for a particular phone
        model = name of the model for a particular phone
        This table spans all the phones-models in our database

        Documentation for method add_models 
        This method is used to add a model in model's table
        
        Documentation for method get_all_models 
        This method is used to retrieve all the model from the model's table

        Documentation for method get_modId 
        This method is used to retrieve the id of a model
        given its name from the model's table

        Documentation for method get_modId 
        This method is used to retrieve a model given its id from the model's table

        Documentation for method get_modId 
        This method is used to retrieve a model given its name from the model's table

        Documentation for method get_modId 
        This method is used to retrieve a model given its brand-name from the model's table

        **************************************************************************************************************************************  
        
        ******************************************PRICES**********************************************************************************
        
        Documentation for class prices
        It represents database table for prices, it stores
        supplier_id = id of the supplier who is selling this phone, from suppliers table
        mobile_id = id of the model of this phone, from models table
        quoted_price = price of the phone as offered by the supplier 
        final_price = price one has to pay to buy this phone, i.e
        it includes vat, tax and shippping charges
        extra_info = extra-info about this phone
        This table spans all the phones in our database

        Documentation for method add_prices 
        This method is used to add all the info for a phone in prices's table

        Documentation for method get_all_prices 
        This method is used to retrieve all the info for all the phones in prices's table

        Documentation for method get_prbyId 
        This method is used to retrieve all the info for a phone given its id in prices's table

        Documentation for method get_prbySid 
        This method is used to retrieve all the info for a phone in prices's table 
        given its supplier-id

        Documentation for method get_prbySid 
        This method is used to retrieve all the info for a phone in prices's table 
        given its model-id

        Documentation for method get_prbySid 
        This method is used to retrieve all the info for a phone in prices's table 
        given its model-name

        **************************************************************************************************************************************  

        ******************************************GUARANTEE and SHIPINFO**********************************************************************
        
        Documentation for class guarantee_info
        It represents database table for guarantee_info, it stores
        mid = id of the phone in models table
        guaranteeinfo = duaration of guarantee and whether guarantee is from vendor or manufacturer 
        shipinfo = how much time would be taken for shipping.
        This table spans all the phones in our database

        Documentation for method add_gs_info 
        This method is used to add guarantee and ship info for a phone

        Documentation for method get_all_gs_info 
        This method is used to retrieve guarantee and ship info for all the phones

        Documentation for method get_gs_bymid
        This method is used to retrieve guarantee and ship info for a phones
        given its phone-id

        **************************************************************************************************************************************  
        
        ******************************************extra_vars**********************************************************************************

        Documentation for class extra_vars
        It represents database table for extra_vars, it stores
        var = name of the variable
        val = value of the variable
        desc = description of the variable
        For some suppliers the number of pages to be crawled is not fixed,
        for them variables are created and based on the value of the variable 
        number of the pages to be crawled is determined dynamically

        Documentation for method set_extra_vars  
        This method is used to add a variable for a particular supplier.
        Some suppliers has no fixed count of number of pages on which they have data
        so to deal with that this function is used to set variables.

        Documentation for method get_extra_vars  
        This method is used to retrieve a variable's value given its name

        **************************************************************************************************************************************  

        ******************************************CRAWL**********************************************************************************

        Documentation for class crawl
        It represents database table for crawl, it stores
        crawled_date = date of crawling
        On each new crawl a new entry is made   
        Reason for creating this table is for retaining past data for comparison

        Documentation for method add_newcrawler 
        This method is used to add a new crawlid in crawl's table
        
        Documentation for method get_latestcrawler 
        This method is used to get the latest crawler from the crawl's table

        Documentation for method get_latestcrawlerid 
        This method is used to get the latest crawler-id from the crawl's table 

        **************************************************************************************************************************************  

        ******************************************OTHERS**************************************************************************************
        Documentation for class Datahelper
        This class contains various methods to access the database tables

        Documentation for method init
        Before using all the tables described in this module, one has to call this method

        Documentation for method initxy
        It calls a method init() so that when one needs to access the helper methods of 
        this class to access database, the database tables are in the scope

        **************************************************************************************************************************************

        ******************************************code_words**********************************************************************************

        Documentation for class code_words
        It represents database table for code_words used for parameters taken from forms, it stores
        key = name of the parameter
        value = the value of the parameter
        description = information about the parameter, for what purpose its stored

        Documentation for method init
        Before using all the tables described in this module, one has to call this method

        Documentation for method initialize_table
        It calls a method init() so that when one needs to access the helper methods of 
        this class to access database, the database tables are in the scope
        
        Documentation for method set_code_word 
        This method is used to set a code_word through input from form
        
        Documentation for method get_code_word_byId 
        This method is used to retrieve a code_word given its id

        Documentation for method get_code_word 
        This method is used to retrieve the value of a code_word given its key i.e name

        Documentation for method get_allcode_words 
        This method is used to retrieve all the code_words given the key i.e name

        Documentation for method get_code_word_byKey 
        This method is used to retrieve the code_word given its key i.e name

        
        **************************************************************************************************************************************