+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Coders Lounge (https://hackforums.net/forumdisplay.php?fid=5)
+---- Thread: Best way to scrape protected website (/showthread.php?tid=6045808)
Best way to scrape protected website - arxadaer - Feb 1, 2020
Hi,
as the title says, I'm trying to scrape a website that block me.
To do this, I would like to use a simple http request library (like requests and session from python) and not a browser simulator (like Selenium, Puppeteer, PhantomJS, etc...). I already tried successfully with those, but I really want to do it with an http request.
I have already set the basic stuff, like changing user-agent, refer, and other headers. I have "top of the shelf" proxies, too. The website in question is very likely hosted by cloudflare, so they're protecting it in some other ways for sure.
My main question is: what should I focus on? Should I find someone path of the server that are not protected, or maybe find some work around with cookies?
Let me know if you have other advices, too :)
I would be very grateful if someone would help me for free, but since I need to find a serious and reliable solution, the help could also be paid.
Thank you
RE: Best way to scrape protected website - SIM-3991524 - Feb 20, 2020
If you're just going after some text you could take a take a "full page" screenshot and then pass that through ocr (see pytesseract). From there you could use a regex
RE: Best way to scrape protected website - hannalexise - Feb 24, 2020
You should also mimic the human behavior as well. For example, instead of scraping the same sort of sites, it is better to mix it with other unrelated stuff to prevent it from being detected. Some site has the mechanism to detect such abnormal behavior e.g., someone only request a certain sort of url, etc. Also give them an appropriate delay between requests.
RE: Best way to scrape protected website - Riԁdle - Feb 24, 2020
I don't know how to get around the cloudflare issue while using the python requests module, but if the reason you're not wanting to use selenium is because of selenium is popping up a visible browser on your screen you can do
Code
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chrome_options)or if your problem with using selenium is that you don't like the way you're scraping text using selenium's methods (I.E
Code
find_element_by_id, find_elements_by_class, find_element_by_xpathCode
driver.page_sourceRE: Best way to scrape protected website - gua - Feb 24, 2020
Idk what programming language u know but u can use delphi for this, and indy components. Changing proxy in every 5-10 requests, also changing requests that u are sending ( randomize time from-to ) ..
Also randomizing user agents etc.. By doing this u can basically scrape 24/7
if u are interested i can show u on pm some scraper that i made ✌
RE: Best way to scrape protected website - SIM-3991524 - Feb 24, 2020
(Feb 24, 2020 02:01 PM)Riԁdle Wrote: I don't know how to get around the cloudflare issue while using the python requests module, but if the reason you're not wanting to use selenium is because of selenium is popping up a visible browser on your screen you can do
and I believe it's different for other firefox, and other browsers, just type in python selenium [your browser name] headless into google and it'll show you how,Codechrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chrome_options)
or if your problem with using selenium is that you don't like the way you're scraping text using selenium's methods (I.E
ect..) you could always pass the page sourceCodefind_element_by_id, find_elements_by_class, find_element_by_xpath
to BeautifulSoup as you normally would do with requestsCodedriver.page_source
yep, selenium is extra easy in python, would recommend this method assuming the site is not too cryptic