+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: Ultimate scraping challenge: google. (/showthread.php?tid=6276394)
Ultimate scraping challenge: google. - TheSpecialOne97 - Jul 28, 2024
I wanna ask about how to scrape google.com and use their search results. not the basic wget or solanium solution. or the free proxy lists we know it doesnt work..
Can anyone teach me how to get past their blocks?
RE: Ultimate scraping challenge: google. - Klonopin - Jul 29, 2024
You need a residential proxy to avoid captchas. however, you can still get them if unlucky or flagged search
RE: Ultimate scraping challenge: google. - lasvegas - Aug 3, 2024
(Jul 28, 2024 05:34 PM)TheSpecialOne97 Wrote: I wanna ask about how to scrape google.com and use their search results. not the basic wget or solanium solution. or the free proxy lists we know it doesnt work..
Can anyone teach me how to get past their blocks?
Either residential proxies, or, iirc, they might have an API for that
RE: Ultimate scraping challenge: google. - Jessjones70 - Aug 27, 2024
Your best bet is to find a paid crawling service for that, if you have the money. Otherwise you will need to look into a setup using something like Playwright together with playwright-stealth and rotating through residential IPs to scrape results from Google.
RE: Ultimate scraping challenge: google. - banonkiel - Aug 27, 2024
- Browser Automation: Selenium or any alternative of choice (some options provide more stealthy options to reduce probability of challenges/captchas)
- Residential Proxy: Preferably rotating 3/4G proxies.
- Captcha Solver: like 2captcha
On the long run, this is probably the most reliable set-up you can make.