+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [Python] Simple Proxy Scraper. (/showthread.php?tid=5608560)
Simple Proxy Scraper. - Null - Apr 25, 2017
Hey,
I have been messing around with selenium and i made a really simple proxy scraper (credits to Piero-Bird) that will scrape off gatherproxy.com
you may need to change the chrome driver's path. (get it from here: https://chromedriver.storage.googleapis.com/index.html?path=2.29/ )
Here is the source code:
The program basically scrapes off gather proxies and export the proxies in a textfile in a IP:Port format (in the same directory of the script)
here is the original tutorial by Piero-Bird
* - LemonTea - Apr 25, 2017
Nice. Very simple and cool
Try using Phantomjs to make it a background process too!
* - SIM-1866423 - Apr 26, 2017
Try this,
PHP Code
<?php
import urllib
import re
data = urllib.urlopen("http://gatherproxy.com/proxylist/anonymity/?t=Elite").read()
proxy = re.findall(r"\d{1,3}\.\d{1,3}\.\d{1,3}.\d{1,3}", data)
port = [int(x, 16) for x in re.findall('PROXY_PORT":"(.+?)"', data)]
warehouse = ['{0}:{1}'.format(_[0],_[1]) for _ in zip(proxy, port)]
print(warehouse)
PHP Code
<?php
['192.241.129.174:3128', '112.82.190.235:8998',
'86.109.100.80:8080', '206.251.227.56:80',
'103.52.252.17:8080', '219.127.253.43:80',
'115.254.104.206:8080', '162.243.76.169:3128',
'159.203.189.158:3128', '104.236.174.88:3128',
'162.243.77.188:3128', '104.131.147.164:3128',
'54.174.144.206:80', '52.45.121.244:80',
'185.2.101.31:3128', '186.211.102.57:80',
'104.224.37.121:3128', '138.197.194.143:3128',
'119.252.163.66:8080', '93.51.247.104:80',
'138.197.194.7:3128', '47.52.29.37:3128',
'104.236.176.142:3128', '36.85.90.8:8080', '23.248.163.235:80']
[Finished in 1.9s]
* - lmorra - Apr 26, 2017
Selenium always works but it's really slow. Use the requests library and XPATHs, it's a lot faster.
* - Sapience - Apr 26, 2017
(Apr 26, 2017 12:28 AM)Mnemosyne Wrote: Try this,
PHP Code
<?php
import urllib
import re
data = urllib.urlopen("http://gatherproxy.com/proxylist/anonymity/?t=Elite").read()
proxy = re.findall(r"\d{1,3}\.\d{1,3}\.\d{1,3}.\d{1,3}", data)
port = [int(x, 16) for x in re.findall('PROXY_PORT":"(.+?)"', data)]
warehouse = ['{0}:{1}'.format(_[0],_[1]) for _ in zip(proxy, port)]
print(warehouse)
PHP Code
<?php
['192.241.129.174:3128', '112.82.190.235:8998',
'86.109.100.80:8080', '206.251.227.56:80',
'103.52.252.17:8080', '219.127.253.43:80',
'115.254.104.206:8080', '162.243.76.169:3128',
'159.203.189.158:3128', '104.236.174.88:3128',
'162.243.77.188:3128', '104.131.147.164:3128',
'54.174.144.206:80', '52.45.121.244:80',
'185.2.101.31:3128', '186.211.102.57:80',
'104.224.37.121:3128', '138.197.194.143:3128',
'119.252.163.66:8080', '93.51.247.104:80',
'138.197.194.7:3128', '47.52.29.37:3128',
'104.236.176.142:3128', '36.85.90.8:8080', '23.248.163.235:80']
[Finished in 1.9s]
Really good simple proxy scraper, nice work!
Personally I prefer having a file output rather than just putting the result in a terminal, so I took your code and "improved" it a bit:
It's still compact, while having a (imo) really useful feature.
* - ᏙictorҮ - Apr 26, 2017
I love Python and great simple code :")