+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [Python] What kind of web-scraper should I create? (/showthread.php?tid=6116568)
Pages: Pages
1
2
What kind of web-scraper should I create? - Crypt - Oct 19, 2020
I'm trying to come up with a web-scraping program I want to make but I just have no idea what information I should search for.
RE: What kind of web-scraper should I create? - SecuroServ - Oct 19, 2020
You could do a fun project of scraping e.g. medium articles and their tags to train an AI, to then let an AI automatically tag documents.
Or anything else to train AI
RE: What kind of web-scraper should I create? - Crypt - Oct 19, 2020
(Oct 19, 2020 03:26 PM)Lesley Wrote: You could do a fun project of scraping e.g. medium articles and their tags to train an AI, to then let an AI automatically tag documents.
Or anything else to train AI
So store the tags on articles and then fitting certain characteristics let the AI decide what should be tagged in articles created?
RE: What kind of web-scraper should I create? - Harley Quinn - Oct 20, 2020
So the first web crawler i made did the following:
- Stored all links on website (with the same domain)
- Captured all mobile numbers, email addresses and social media links.
- Automatically downloaded all media, images, videos, etc.
Then displaying all this info nicely, then i built it so you can scrape any site using the same script.
It was a really fun project refining it. (The AI training sounds fun too)
RE: What kind of web-scraper should I create? - Snowwy - Oct 20, 2020
(Oct 20, 2020 12:45 AM)Harley Quinn Wrote: - Automatically downloaded all media, images, videos, etc.Can I ask - do you happen to still have that script around somewhere by any chance?
RE: What kind of web-scraper should I create? - Harley Quinn - Oct 20, 2020
(Oct 20, 2020 01:13 AM)Snowwy Wrote: Can I ask - do you happen to still have that script around somewhere by any chance?
I will have a look, it was on my old laptop. Is that something you are interested in, something to peel all media from an entire website?
RE: What kind of web-scraper should I create? - G.P.D. - Oct 20, 2020
(Oct 20, 2020 01:13 AM)Snowwy Wrote: Can I ask - do you happen to still have that script around somewhere by any chance?
I could do this for you mate.
Send me a PM with the website you want to scape.
RE: What kind of web-scraper should I create? - Snowwy - Oct 20, 2020
(Oct 20, 2020 01:20 AM)Harley Quinn Wrote: I will have a look, it was on my old laptop. Is that something you are interested in, something to peel all media from an entire website?Yeh more for just research and knowledge tbh.
(Oct 20, 2020 04:32 AM)Whait Wrote: I could do this for you mate.I don't want a particular website scapred - just want a script for it so I can play around with it if you have one?
Send me a PM with the website you want to scape.
RE: What kind of web-scraper should I create? - Harley Quinn - Oct 20, 2020
(Oct 20, 2020 04:36 AM)Snowwy Wrote: Yeh more for just research and knowledge tbh.
I don't want a particular website scapred - just want a script for it so I can play around with it if you have one?
Sure, let me take a look and see if i can find it, i'll send the source code over. If i can't i'll try through something together and send that :)
RE: What kind of web-scraper should I create? - Prawnster - Oct 20, 2020
I made something in the past that basically just scanned like big social media sites for information like emails, phone numbers and sensitive information and then looked it up in a DB and if they had a leaked password, log it and later email them with some free email service.
Was a nice little one to teach more about the language I was choosing to use at the time :)
RE: What kind of web-scraper should I create? - G.P.D. - Oct 20, 2020
(Oct 20, 2020 04:36 AM)Snowwy Wrote: Yeh more for just research and knowledge tbh.
I don't want a particular website scapred - just want a script for it so I can play around with it if you have one?
I don't have one for now but I can take my evening to create one.
I'll do it tonight or tomorrow.
RE: What kind of web-scraper should I create? - Crypt - Oct 20, 2020
(Oct 20, 2020 05:16 AM)Mazl Wrote: I made something in the past that basically just scanned like big social media sites for information like emails, phone numbers and sensitive information and then looked it up in a DB and if they had a leaked password, log it and later email them with some free email service.
Was a nice little one to teach more about the language I was choosing to use at the time :)
I really like this idea. I know how to open a specific web page and then look for emails but is there a better way to mass scan like tweets and look for emails and phone number?
This is a cool project.
RE: What kind of web-scraper should I create? - Prawnster - Oct 20, 2020
(Oct 20, 2020 12:20 PM)Crypt Wrote: I really like this idea. I know how to open a specific web page and then look for emails but is there a better way to mass scan like tweets and look for emails and phone number?
This is a cool project.
I just used google dorks as well as just massive amounts of checks (using public proxies) to facebook pages and shit to scrape all their users.
I don't remember the exact google dorks as it was such a long time ago, but it's an interesting one to learn as you get to learn working with site APIs, proper threading, proxy support, database usage if you wanna incorporate one of those and build up some form of database for each user, and also efficient scraping.
Goodluck mate!
RE: What kind of web-scraper should I create? - Crypt - Oct 20, 2020
(Oct 20, 2020 12:47 PM)Mazl Wrote: I just used google dorks as well as just massive amounts of checks (using public proxies) to facebook pages and shit to scrape all their users.
I don't remember the exact google dorks as it was such a long time ago, but it's an interesting one to learn as you get to learn working with site APIs, proper threading, proxy support, database usage if you wanna incorporate one of those and build up some form of database for each user, and also efficient scraping.
Goodluck mate!
Thanks man! This one interests me and I think I'd learn a lot from it. I'm going to work on this.
RE: What kind of web-scraper should I create? - Prawnster - Oct 20, 2020
(Oct 20, 2020 12:55 PM)Crypt Wrote: Thanks man! This one interests me and I think I'd learn a lot from it. I'm going to work on this.
Ayy glad to hear it mate, goodluck with it and keep us posted on how it goes :)