[Who Soup]

Hey, Here's a way you can make a simple website scraper in vb.net. The objective here will be to get the number of subscribers from a Youtube Page.



After opening a new console application, we first we want to import .net and regular expressions as we will be using a webclient to scrape and regex to sort out the text we want.

Code

Imports System.Net
Imports System.Text.RegularExpressions

Next create a new sub called scrape. This works by using the webclient to download the source as as a string.  

Code

Sub scrape()
       Dim wc As New WebClient
       Dim source As String = wc.DownloadString("https://www.youtube.com/channel/UCttMm3hHlDSqXGYmp7tqrEA")
       Dim subregex As New Regex("subscribers"">.*?</span>")
       For Each submatch As Match In subregex.Matches(source)
           Dim subscribers As String = submatch.ToString.Split(">")(1).Split("<")(0)
           Console.WriteLine("Subscribers: " & subscribers)
       Next
   End Sub

Once we have the source it looks for the key word, in this case it's  subscribers"">.*?</span>" .  The .*? in the center indicates the number of subscriber a user has.  It's marked that way because that number will change depending on which user is scraped. You may also notice 2 quotation marks  after the word "subscribers". This is because the key contained a single quotation mark; for regex to recognize it as part of the text you have to put two.



Now that we're able to download the source and find the key we now need to "clean it up" so we can have just the text that we want; in this case the subscriber count. As you can see in the above code we're able to achieve this by using .split to remove everything that's not between ">" and "<". The (1) and (0) represent the first instance of that character in the text and the last instance of that character in the text; this tells the program where to start and stop while looking for the text that we want.

If you want to tinker around with it here is the full code which you can copy and paste into a new console application; it should be working as is.

Full Code:

Code

Imports System.Net
Imports System.Text.RegularExpressions
Module Module1

   Sub Main()
       scrape()
       Console.ReadLine()
   End Sub

   Sub scrape()
       Dim wc As New WebClient
       Dim source As String = wc.DownloadString("https://www.youtube.com/channel/UCttMm3hHlDSqXGYmp7tqrEA")
       Dim subregex As New Regex("subscribers"">.*?</span>")
       For Each submatch As Match In subregex.Matches(source)
           Dim subscribers As String = submatch.ToString.Split(">")(1).Split("<")(0)
           Console.WriteLine("Subscribers: " & subscribers)
       Next
   End Sub

End Module

If you have any questions, comments, or get stuck feel free to drop a comment below  🤝

[Eternal Darkness]

If you have any questions, comments, or get stuck feel free to drop a comment below  🤝

A module with "2 methods", you should probably just post the full code, rather than posting the 1 custom method (separately). Kinda unnecessary repeat. Reference the method sure.

You should mention that there are other ways to extract information like split/substring before that becomes the point of this thread (lots of people do not appreciate Regex, and in this case I could "see what they mean"). Regex is best for complicated extractions (in comparison to manually devising an algorithm to do so, at-least when your throwing something together, no "one size fits all" for advice but hopefully you understand).

Your extract is bad, you can use regex to get only the # out via regex (\d+[K|M] as well, very rough/bad regex just threw it together as an example). You should avoid doing those splits. If you want to use the splits then you should avoid using Regex and prob just substring.

You should make scrape a function "Function scrape(ByVal string as String, ByVal pattern as String) as System.Text.RegularExpressions.MatchCollection" for example. Would make your function a lot more generic.
You should have another function handle getting the source string (so you can properly catch errors for a bad connection/unreachable site/whatever the case may be.

Know this is a tutorial and not a completed project... Not posting to shame you, making sure this doesn't end up being used specifically in a bunch of programs...

[Who Soup]

A module with "2 methods", you should probably just post the full code, rather than posting the 1 custom method (separately). Kinda unnecessary repeat. Reference the method sure.

You should mention that there are other ways to extract information like split/substring before that becomes the point of this thread (lots of people do not appreciate Regex, and in this case I could "see what they mean"). Regex is best for complicated extractions (in comparison to manually devising an algorithm to do so, at-least when your throwing something together, no "one size fits all" for advice but hopefully you understand).

Your extract is bad, you can use regex to get only the # out via regex (\d+[K|M] as well, very rough/bad regex just threw it together as an example). You should avoid doing those splits. If you want to use the splits then you should avoid using Regex and just IndexOf or something to get the substring.

You should make scrape a function "Function scrape(ByVal string as String, ByVal pattern as String) as System.Text.RegularExpressions.MatchCollection" for example. Would make your function a lot more generic. You should have another function handle getting the source string (so you can properly catch errors for a bad connection/unreachable site/whatever the case may be.

Know this is a tutorial and not a completed project... Not posting to shame you, making sure this doesn't end up being used specifically in a bunch of programs...

lmao, you're going no where in life kid.

[Eternal Darkness]

lmao, you're going no where in life kid.

Wat.

Code

Sub scrape()
       Dim wc As New WebClient
       Dim source As String = wc.DownloadString("https://www.youtube.com/channel/UCttMm3hHlDSqXGYmp7tqrEA")
       Dim subregex As New Regex("subscribers"">.*?</span>")
       For Each submatch As Match In subregex.Matches(source)
           Dim subscribers As String = submatch.ToString.Split(">")(1).Split("<")(0)
           Console.WriteLine("Subscribers: " & subscribers)
       Next
   End Sub

If you have any questions, comments, or get stuck feel free to drop a comment below  🤝

Good luck in life with this "super pro" code (which shows you don't understand what your doing). Funny how you can't take constructive criticism when you ask for it (enough to instantly neg rep me for suggesting improvements, jeez-us). 🤝

[Who Soup]

Wat.


Good luck in life with this "super pro" code (which shows you don't understand what your doing). Funny how you can't take constructive criticism when you ask for it (enough to instantly neg rep me for suggesting improvements, jeez-us). 🤝

This tutorial obviously is intended for beginners.  The fact that you’re attempting to flex your “knowledge” here is comical. I've created statistics based programs using this methodology that has made me close to $2m so far, how much have you made? The point of this thread is to get people to get excited about coding but sadly people like you will always exist in this world, looking to bring others down.

Very small pp

[Charger]

This tutorial obviously is intended for beginners.  The fact that you’re attempting to flex your “knowledge” here is comical.  I've created statistics based programs using this methodology that has made me close to $2m so far, how much have you made?  The point of this thread is to get people to get excited about coding but sadly people like you will always exist in this world, looking to bring others down.

Very small pp

Kinda shitty that someone like you, who is extremely successful outside of HF, takes time out of their day to post something simple and instead of providing you good feedback and leaving it at that, someone tries to rake you over the coals.

[Who Soup]

Kinda shitty that someone like you, who is extremely successful outside of HF, takes time out of their day to post something simple and instead of providing you good feedback and leaving it at that, someone tries to rake you over the coals.

Regex was used to parse my selection out of the source and split was used to clean up what was found; nothing is incorrect about that especially for a 20 line program.  He's salty because I didn't use syntax to scrape the subscriber count in one go but I felt that would be way too complicated for a beginner. You can copy and paste that code as is, and it works flawlessly. Don't worry I won't be posting any more tutorials or freebies, I almost forgot that HF is full of ungrateful people who don't want to learn.

[Charger]

Don't worry I won't be posting any more tutorials or freebies, I almost forgot that HF is full of ungrateful people who don't want to learn.

You do you but try to not let one clown ruin it for you. You only hurt people not involved (new people).

Hit me up some time, we haven't talked in a minute.

[Pawnhub]

and people wonder why nobody drops any tuts etc. People will jump to shit on code here.
Is it absolutely the most code efficient program?

Probably not but if it works that should be what matters.

Lucky 0x hasn't come in here to ruin the party.

[ZxfadexZ]

A module with "2 methods", you should probably just post the full code, rather than posting the 1 custom method (separately). Kinda unnecessary repeat. Reference the method sure.

You should mention that there are other ways to extract information like split/substring before that becomes the point of this thread (lots of people do not appreciate Regex, and in this case I could "see what they mean"). Regex is best for complicated extractions (in comparison to manually devising an algorithm to do so, at-least when your throwing something together, no "one size fits all" for advice but hopefully you understand).

Your extract is bad, you can use regex to get only the # out via regex (\d+[K|M] as well, very rough/bad regex just threw it together as an example). You should avoid doing those splits. If you want to use the splits then you should avoid using Regex and prob just substring.

You should make scrape a function "Function scrape(ByVal string as String, ByVal pattern as String) as System.Text.RegularExpressions.MatchCollection" for example. Would make your function a lot more generic.
You should have another function handle getting the source string (so you can properly catch errors for a bad connection/unreachable site/whatever the case may be.

Know this is a tutorial and not a completed project... Not posting to shame you, making sure this doesn't end up being used specifically in a bunch of programs...

agree with this. seems op just copied this tut from somewhere else not understanding what the code even does. as he got hurt once got suggestions on how to improve it. Just like he stole the proton VPN tut from somewhere else as well

[itcorp]

agree with this. seems op just copied this tut from somewhere else not understanding what the code even does. as he got hurt once got suggestions on how to improve it. Just like he stole the proton VPN tut from somewhere else as well

I think that you need to consider that he is trying to contribute and that he also emphasized that this was a simple solution and that it is not the most beautiful or the best solution.I personally would not do this in this way but it does not mean that we should condemn him for it,debates should be conducted culturally so that everyone benefits in the end but not this way.
I'm sorry but I just do not approve your comment on this thread.

[Who Soup]

agree with this. seems op just copied this tut from somewhere else not understanding what the code even does. as he got hurt once got suggestions on how to improve it. Just like he stole the proton VPN tut from somewhere else as well

lmfao you had to make a fake account because you couldn't face me on your main? Why don't you search the thread on google, nothing will show up; why would I need to copy 20 lines of code?  Also the ProtonVPN account creator uses httpwebrequest not a webclient so you're wrong there also. It was also a give away not a tutorial dumb ass.



Lastly, I clearly don't understand what I'm talking about.  It's not like I spent the first half of the thread explaining exactly what each line of code does and how you should implement it.

Look at all these bratty children misbehaving loooool

[BahNahNah]

Hell yeah bring back the .net section dick measuring contests. What a throwback 👹

For real tho dispose of the webclient instance please

[Who Soup]

Hell yeah bring back the .net section dick measuring contests. What a throwback  👹

For real tho dispose of the webclient instance please

7 years later and this section is as militant as ever lmao

[MavenGradle]

This tutorial obviously is intended for beginners.  The fact that you’re attempting to flex your “knowledge” here is comical.  I've created statistics based programs using this methodology that has made me close to $2m so far, how much have you made?  The point of this thread is to get people to get excited about coding but sadly people like you will always exist in this world, looking to bring others down.

Very small pp

If this tutorial was indeed intended for beginners you should have used a proper naming convention. Right now it looks like you can't even properly name your methods consistently, whereas this is for beginners, right? It seems like you can't even get the basics right.

Then again, reading this thread I'm pretty sure you got the shitty method name from Eternal Darkness which is great. Someone that's offering advice that contains a naming convention that is more in line of Java programming instead of C#. Absolutely wonderful (sarcasm, by the way).

Not to mention the fact that you have absolutely zero error handling in your code at all. Might I also recommend looking into HttpWebRequests for more fine-tuned control?
https://docs.microsoft.com/en-us/dotnet/...mework-4.8

Now, I'm sure you mean well but before you bash others because of a clear inability to take constructive criticism, keep in mind that your response to all this speaks volumes about your own ability to comprehend the basics of general programming. This is not at all about bringing people down but more about your attitude towards people who are trying to help YOU by improving your practices.

and people wonder why nobody drops any tuts etc. People will jump to Clay Davis on code here.
Is it absolutely the most code efficient program?

Probably not but if it works that should be what matters.

Lucky 0x hasn't come in here to ruin the party.

Right, and what exactly is wrong with pointing out flawed logic so people understand why they shouldn't use a specific tutorial for learning purposes because the tutorial itself is flawed? Anyway, 0xDEAD10CC doesn't have to show up at all to point those things out.

No one here is condemning anyone. People take criticism way to close to the heart which is why so many people are wrongfully becoming aggressive in this thread. There's nothing wrong with pointing out flaws and if it bothers people that they're being corrected, they should seriously reconsider their life choices and perhaps start sharing code that actually holds up to general coding standards, especially when it comes to tutorials which are intended to help out beginners.

All I see is beginners, who will instead be condemned for learning it's OK to use and implement bad practices and later on lose businesses millions of dollars because of a clear ineptitude of implementing basic error handling. This is all just very generalized anyway, but hopefully this post contains enough information to stop this nonsense.