WebSVN – SmartDukaan – Diff – /trunk/PriceComparisonFramework/src/Scrapers/BaseScraper.py



        opener = urllib2.build_opener()
        try:
            response_data = opener.open(request).read()
            
        except urllib2.HTTPError as e:
            print 'ERROR: ', e
            print 'Retrying'
            self.count_trials += 1
            
            if self.count_trials < 3:
                return self.read(url)

Rev 4039	Rev 5291
Line 15...	Line 15...
15	`opener = urllib2.build_opener()`	15	`opener = urllib2.build_opener()`
16	`try:`	16	`try:`
17	`response_data = opener.open(request).read()`	17	`response_data = opener.open(request).read()`
18		18
19	`except urllib2.HTTPError as e:`	19	`except urllib2.HTTPError as e:`
20	`print 'ERROR:', e`	20	`print 'ERROR: ', e`
21	`print 'Retrying'`	21	`print 'Retrying'`
22	`self.count_trials += 1`	22	`self.count_trials += 1`
23		23
24	`if self.count_trials < 3:`	24	`if self.count_trials < 3:`
25	`return self.read(url)`	25	`return self.read(url)`

Subversion Repositories SmartDukaan

(root)/trunk/PriceComparisonFramework/src/Scrapers/BaseScraper.py – Rev 4039 → 5291