WebSVN – SmartDukaan – Blame – /trunk/Website/src/main/webapp/robots.txt

Rev	Author	Line No.	Line
1912	rajveer	1	`User-agent: *`
		2	`Disallow: /partners/phonecurry.xml`
		3
1090	vikas	4	`User-agent: OmniExplorer_Bot`
		5	`Disallow: /`
		6
		7	`User-agent: FreeFind`
		8	`Disallow: /`
		9
		10	`User-agent: BecomeBot`
		11	`Disallow: /`
		12
		13	`User-agent: Nutch`
		14	`Disallow: /`
		15
		16	`User-agent: Jetbot/1.0`
		17	`Disallow: /`
		18
		19	`User-agent: Jetbot`
		20	`Disallow: /`
		21
		22	`User-agent: WebVac`
		23	`Disallow: /`
		24
		25	`User-agent: naver`
		26	`Disallow: /`
		27
		28	`User-agent: dumbot`
		29	`Disallow: /`
		30
		31	`User-agent: Hatena Antenna`
		32	`Disallow: /`
		33
		34	`User-agent: grub-client`
		35	`Disallow: /`
		36
		37	`User-agent: grub`
		38	`Disallow: /`
		39
		40	`User-agent: WebZip`
		41	`Disallow: /`
		42
		43	`User-agent: larbin`
		44	`Disallow: /`
		45
		46	`User-agent: b2w/0.1`
		47	`Disallow: /`
		48
		49	`User-agent: Copernic`
		50	`Disallow: /`
		51
		52	`User-agent: psbot`
		53	`Disallow: /`
		54
		55	`User-agent: Python-urllib`
		56	`Disallow: /`
		57
		58
		59	`User-agent: NetMechanic`
		60	`Disallow: /`
		61
		62	`User-agent: URL_Spider_Pro`
		63	`Disallow: /`
		64
		65	`User-agent: CherryPicker`
		66	`Disallow: /`
		67
		68	`User-agent: EmailCollector`
		69	`Disallow: /`
		70
		71	`User-agent: EmailSiphon`
		72	`Disallow: /`
		73
		74	`User-agent: WebBandit`
		75	`Disallow: /`
		76
		77	`User-agent: EmailWolf`
		78	`Disallow: /`
		79
		80	`User-agent: ExtractorPro`
		81	`Disallow: /`
		82
		83	`User-agent: CopyRightCheck`
		84	`Disallow: /`
		85
		86	`User-agent: Crescent`
		87	`Disallow: /`
		88
		89	`User-agent: SiteSnagger`
		90	`Disallow: /`
		91
		92	`User-agent: ProWebWalker`
		93	`Disallow: /`
		94
		95	`User-agent: CheeseBot`
		96	`Disallow: /`
		97
		98	`User-agent: LNSpiderguy`
		99	`Disallow: /`
		100
		101	`User-agent: Alexibot`
		102	`Disallow: /`
		103
		104	`User-agent: Teleport`
		105	`Disallow: /`
		106
		107	`User-agent: TeleportPro`
		108	`Disallow: /`
		109
		110	`User-agent: Stanford Comp Sci`
		111	`Disallow: /`
		112
		113	`User-agent: MIIxpc`
		114	`Disallow: /`
		115
		116	`User-agent: Telesoft`
		117	`Disallow: /`
		118
		119	`User-agent: Website Quester`
		120	`Disallow: /`
		121
		122	`User-agent: moget/2.1`
		123	`Disallow: /`
		124
		125	`User-agent: WebZip/4.0`
		126	`Disallow: /`
		127
		128	`User-agent: WebStripper`
		129	`Disallow: /`
		130
		131	`User-agent: WebSauger`
		132	`Disallow: /`
		133
		134	`User-agent: WebCopier`
		135	`Disallow: /`
		136
		137	`User-agent: NetAnts`
		138	`Disallow: /`
		139
		140	`User-agent: Mister PiX`
		141	`Disallow: /`
		142
		143	`User-agent: WebAuto`
		144	`Disallow: /`
		145
		146	`User-agent: TheNomad`
		147	`Disallow: /`
		148
		149	`User-agent: WWW-Collector-E`
		150	`Disallow: /`
		151
		152	`User-agent: RMA`
		153	`Disallow: /`
		154
		155	`User-agent: libWeb/clsHTTP`
		156	`Disallow: /`
		157
		158	`User-agent: asterias`
		159	`Disallow: /`
		160
		161	`User-agent: httplib`
		162	`Disallow: /`
		163
		164	`User-agent: turingos`
		165	`Disallow: /`
		166
		167	`User-agent: spanner`
		168	`Disallow: /`
		169
		170	`User-agent: InfoNaviRobot`
		171	`Disallow: /`
		172
		173	`User-agent: Harvest/1.5`
		174	`Disallow: /`
		175
		176	`User-agent: Bullseye/1.0`
		177	`Disallow: /`
		178
		179	`User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0`
		180	`Disallow: /`
		181
		182	`User-agent: CherryPickerSE/1.0`
		183	`Disallow: /`
		184
		185	`User-agent: CherryPickerElite/1.0`
		186	`Disallow: /`
		187
		188	`User-agent: WebBandit/3.50`
		189	`Disallow: /`
		190
		191	`User-agent: NICErsPRO`
		192	`Disallow: /`
		193
		194	`User-agent: Microsoft URL Control - 5.01.4511`
		195	`Disallow: /`
		196
		197	`User-agent: DittoSpyder`
		198	`Disallow: /`
		199
		200	`User-agent: Foobot`
		201	`Disallow: /`
		202
		203	`User-agent: SpankBot`
		204	`Disallow: /`
		205
		206	`User-agent: BotALot`
		207	`Disallow: /`
		208
		209	`User-agent: lwp-trivial/1.34`
		210	`Disallow: /`
		211
		212	`User-agent: lwp-trivial`
		213	`Disallow: /`
		214
		215	`User-agent: BunnySlippers`
		216	`Disallow: /`
		217
		218	`User-agent: Microsoft URL Control - 6.00.8169`
		219	`Disallow: /`
		220
		221	`User-agent: URLy Warning`
		222	`Disallow: /`
		223
		224	`User-agent: Wget/1.6`
		225	`Disallow: /`
		226
		227	`User-agent: Wget/1.5.3`
		228	`Disallow: /`
		229
		230	`User-agent: Wget`
		231	`Disallow: /`
		232
		233	`User-agent: LinkWalker`
		234	`Disallow: /`
		235
		236	`User-agent: cosmos`
		237	`Disallow: /`
		238
		239	`User-agent: moget`
		240	`Disallow: /`
		241
		242	`User-agent: hloader`
		243	`Disallow: /`
		244
		245	`User-agent: humanlinks`
		246	`Disallow: /`
		247
		248	`User-agent: LinkextractorPro`
		249	`Disallow: /`
		250
		251	`User-agent: Offline Explorer`
		252	`Disallow: /`
		253
		254	`User-agent: Mata Hari`
		255	`Disallow: /`
		256
		257	`User-agent: LexiBot`
		258	`Disallow: /`
		259
		260	`User-agent: Web Image Collector`
		261	`Disallow: /`
		262
		263	`User-agent: The Intraformant`
		264	`Disallow: /`
		265
		266	`User-agent: True_Robot/1.0`
		267	`Disallow: /`
		268
		269	`User-agent: True_Robot`
		270	`Disallow: /`
		271
		272	`User-agent: BlowFish/1.0`
		273	`Disallow: /`
		274
		275	`User-agent: JennyBot`
		276	`Disallow: /`
		277
		278	`User-agent: MIIxpc/4.2`
		279	`Disallow: /`
		280
		281	`User-agent: BuiltBotTough`
		282	`Disallow: /`
		283
		284	`User-agent: ProPowerBot/2.14`
		285	`Disallow: /`
		286
		287	`User-agent: BackDoorBot/1.0`
		288	`Disallow: /`
		289
		290	`User-agent: toCrawl/UrlDispatcher`
		291	`Disallow: /`
		292
		293	`User-agent: WebEnhancer`
		294	`Disallow: /`
		295
		296	`User-agent: suzuran`
		297	`Disallow: /`
		298
		299	`User-agent: VCI WebViewer VCI WebViewer Win32`
		300	`Disallow: /`
		301
		302	`User-agent: VCI`
		303	`Disallow: /`
		304
		305	`User-agent: Szukacz/1.4`
		306	`Disallow: /`
		307
		308	`User-agent: QueryN Metasearch`
		309	`Disallow: /`
		310
		311	`User-agent: Openfind data gathere`
		312	`Disallow: /`
		313
		314	`User-agent: Openfind`
		315	`Disallow: /`
		316
		317	`User-agent: Xenu's Link Sleuth 1.1c`
		318	`Disallow: /`
		319
		320	`User-agent: Xenu's`
		321	`Disallow: /`
		322
		323	`User-agent: Zeus`
		324	`Disallow: /`
		325
		326	`User-agent: RepoMonkey Bait & Tackle/v1.01`
		327	`Disallow: /`
		328
		329	`User-agent: RepoMonkey`
		330	`Disallow: /`
		331
		332	`User-agent: Microsoft URL Control`
		333	`Disallow: /`
		334
		335	`User-agent: Openbot`
		336	`Disallow: /`
		337
		338	`User-agent: URL Control`
		339	`Disallow: /`
		340
		341	`User-agent: Zeus Link Scout`
		342	`Disallow: /`
		343
		344	`User-agent: Zeus 32297 Webster Pro V2.9 Win32`
		345	`Disallow: /`
		346
		347	`User-agent: Webster Pro`
		348	`Disallow: /`
		349
		350	`User-agent: EroCrawler`
		351	`Disallow: /`
		352
		353	`User-agent: LinkScan/8.1a Unix`
		354	`Disallow: /`
		355
		356	`User-agent: Keyword Density/0.9`
		357	`Disallow: /`
		358
		359	`User-agent: Kenjin Spider`
		360	`Disallow: /`
		361
		362	`User-agent: Iron33/1.0.2`
		363	`Disallow: /`
		364
		365	`User-agent: Bookmark search tool`
		366	`Disallow: /`
		367
		368	`User-agent: GetRight/4.2`
		369	`Disallow: /`
		370
		371	`User-agent: FairAd Client`
		372	`Disallow: /`
		373
		374	`User-agent: Gaisbot`
		375	`Disallow: /`
		376
		377	`User-agent: Aqua_Products`
		378	`Disallow: /`
		379
		380	`User-agent: Radiation Retriever 1.1`
		381	`Disallow: /`
		382
		383
		384	`User-agent: Flaming AttackBot`
		385	`Disallow: /`
		386
		387	`User-agent: Oracle Ultra Search`
		388	`Disallow: /`
		389
		390	`User-agent: MSIECrawler`
		391	`Disallow: /`
		392
		393	`User-agent: PerMan`
		394	`Disallow: /`
		395
		396	`User-agent: searchpreview`
		397	`Disallow: /`
		398
		399	`User-agent: sootle`
		400	`Disallow: /`
		401
		402	`User-agent: es`
		403	`Disallow: /`
		404
		405	`User-agent: Enterprise_Search/1.0`
		406	`Disallow: /`
		407
		408	`User-agent: Enterprise_Search`
		409	`Disallow: /`
		410
		411
		412	`# sitemap`
		413
		414	`Sitemap: http://www.saholic.com/sitemap.xml`

Subversion Repositories SmartDukaan

(root)/trunk/Website/src/main/webapp/robots.txt – Rev 1912