[Crypt]

I'm trying to come up with a web-scraping program I want to make but I just have no idea what information I should search for.

[SecuroServ]

You could do a fun project of scraping e.g. medium articles and their tags to train an AI, to then let an AI automatically tag documents.
Or anything else to train AI

[Crypt]

You could do a fun project of scraping e.g. medium articles and their tags to train an AI, to then let an AI automatically tag documents.
Or anything else to train AI

So store the tags on articles and then fitting certain characteristics let the AI decide what should be tagged in articles created?

[Harley Quinn]

So the first web crawler i made did the following:

- Stored all links on website (with the same domain)
- Captured all mobile numbers, email addresses and social media links.
- Automatically downloaded all media, images, videos, etc.

Then displaying all this info nicely, then i built it so you can scrape any site using the same script.

It was a really fun project refining it. (The AI training sounds fun too)

[Snowwy]

- Automatically downloaded all media, images, videos, etc.

Can I ask - do you happen to still have that script around somewhere by any chance?

[Harley Quinn]

Can I ask - do you happen to still have that script around somewhere by any chance?

I will have a look, it was on my old laptop. Is that something you are interested in, something to peel all media from an entire website?

[G.P.D.]

Can I ask - do you happen to still have that script around somewhere by any chance?

I could do this for you mate.
Send me a PM with the website you want to scape.

[Snowwy]

I will have a look, it was on my old laptop. Is that something you are interested in, something to peel all media from an entire website?

Yeh more for just research and knowledge tbh.

I could do this for you mate.
Send me a PM with the website you want to scape.

I don't want a particular website scapred - just want a script for it so I can play around with it if you have one?

[Harley Quinn]

Yeh more for just research and knowledge tbh.

I don't want a particular website scapred - just want a script for it so I can play around with it if you have one?

Sure, let me take a look and see if i can find it, i'll send the source code over. If i can't i'll try through something together and send that :)

[Prawnster]

I made something in the past that basically just scanned like big social media sites for information like emails, phone numbers and sensitive information and then looked it up in a DB and if they had a leaked password, log it and later email them with some free email service.
Was a nice little one to teach more about the language I was choosing to use at the time :)

[G.P.D.]

Yeh more for just research and knowledge tbh.

I don't want a particular website scapred - just want a script for it so I can play around with it if you have one?

I don't have one for now but I can take my evening to create one.
I'll do it tonight or tomorrow.

[Crypt]

I made something in the past that basically just scanned like big social media sites for information like emails, phone numbers and sensitive information and then looked it up in a DB and if they had a leaked password, log it and later email them with some free email service.
Was a nice little one to teach more about the language I was choosing to use at the time :)

I really like this idea. I know how to open a specific web page and then look for emails but is there a better way to mass scan like tweets and look for emails and phone number?

This is a cool project.

[Prawnster]

I really like this idea.  I know how to open a specific web page and then look for emails but is there a better way to mass scan like tweets and look for emails and phone number?

This is a cool project.

I just used google dorks as well as just massive amounts of checks (using public proxies) to facebook pages and shit to scrape all their users.
I don't remember the exact google dorks as it was such a long time ago, but it's an interesting one to learn as you get to learn working with site APIs, proper threading, proxy support, database usage if you wanna incorporate one of those and build up some form of database for each user, and also efficient scraping.
Goodluck mate!

[Crypt]

I just used google dorks as well as just massive amounts of checks (using public proxies) to facebook pages and shit to scrape all their users.
I don't remember the exact google dorks as it was such a long time ago, but it's an interesting one to learn as you get to learn working with site APIs, proper threading, proxy support, database usage if you wanna incorporate one of those and build up some form of database for each user, and also efficient scraping.
Goodluck mate!

Thanks man! This one interests me and I think I'd learn a lot from it. I'm going to work on this.

[Prawnster]

Thanks man!  This one interests me and I think I'd learn a lot from it.  I'm going to work on this.

Ayy glad to hear it mate, goodluck with it and keep us posted on how it goes :)