[realsunjester]

Regular expressions can be a very useful tool in your arsenal when creating hacking tools. You should use regular expressions when split just won't cut it. You would normally use String.Split when there is only one occurrence of a string in the block your searching. Regular expressions is great for finding specific strings in a block of text, and it can be MUCH faster.

You can use Regex101 to help create your regular expression, it's what I normally use. I will go over a short intro to help get you started when creating a regular expression.

Match a single digit
\d

Match multiple digits
\d+

Match X amount of digits (if you want 3 digits; {3})
\d{x}

Match a single character
\w

Match any character or digit

.

Match any digit or character multiple times
.+

Match zero or more
?


So as an example, let's open the cracked.to website and view the source. We are going to use them as an example on how to scrape stuff. Let's scrape usernames and ID's.

Around line 2955 you will start to see the usernames



So copy that and paste it into the bottom textbox on regex101. The top box is where we will begin to write our regex. Each entry is an anchor link which has the username and their ID.

Code

<a href="https://cracked.to/member.php?action=profile&uid=103683" title="17MOD1997">17MOD1997</a>

In the top box put the following expression

Code

member[.]php[?]action=profile&uid=(\d+)"" title=""[a-zA-Z0-9&_\.-]+?"">([a-zA-Z0-9&_\.-]+?)<\/a>

Which should match that user (if they are logged in)



On the right in the image above you can see that we matched it. However, we only matched that one entry and we want them all. So now, let's replace the name and id with a regular expression.

Code

member[.]php[?]action=profile&uid=(\d+)" title=".+?">(.+?)<\/a>

So now let's turn this into something we can use in C#. You will need to include the RegularExpression namespace (https://docs.microsoft.com/en-us/dotnet/...work-4.7.2). You can do that by adding the following line to the head of your code

Code

using System.Text.RegularExpressions;

We are going to read a text file that has the source of the main cracked.to website, so we don't leave a footprint on their site while working with the regular expression demo. You can download the source I am using here (https://pastr.io/raw/NehxTX). We can read the file using the File.ReadAllLines() method. We will need to add the System.IO namespace for this.

Code

string src = File.ReadAllLines("testsrc");

So now let's apply our expression to the file we read above. Since the file we are using as a test is much different than how the source looks in your browser (in my case, Chrome) we are going to have to change our expression. Below is the image of the block of text on which we are going to use the expression.



Create a new Regex Object

Code

Regex r = new Regex(@"https:\/\/cracked[.]to\/member[.]php[?]action=profile&uid=(\d+)" title="([a-zA-Z0-9&_\.-]+)"><span");

Get the matches with MatchCollection

Code

MatchCollection mc = r.Matches(src);

Now we can loop through the matches in the MatchCollection, showing the capture groups. The capture groups is the data in parenthesis in the regular expression. 

Code

foreach(Match m in mc)
{
        Console.WriteLine("ID: " + m.Groups[1].Value + " \t Name: " + m.Groups[2].Value);
}

[Mr. Robot]

When's my thread coming OP.

[realsunjester]

When's my thread coming OP.

what thread is that? lemme know

[+Haze]

I understand you're trying to contribute to the forums, but this thread doesn't really have value. Not only do you misunderstand Regex and it's applications, but the examples you've shown are really bad.

1) In your example you used Regex to parse an HTML document, meanwhile, an HTML parsing library would be a lot better, not just because of performance, but because of maintainability. It's extremely difficult debugging fat regex expressions, and that's why you should them sparingly. What you have just shown can be done using AngleSharp parsing library and offer way more readability and maintainability than your regex expressions.

2) I've noticed that members on this forum seem to think that Regex is for parsing, but the truth is, it isn't. Regex is designed for pattern matching.

3)

Regular expressions is great for finding specific strings in a block of text, and it can be MUCH faster.

Nope, Regex is far slower than string functions. It's only in the past decade that Regex has been used more, and that's because machines are now more powerful, and can easily handle regex expressions. Back when regex was created in the 90s, they were very slow as computers then lacked power.

[realsunjester]

I understand you're trying to contribute to the forums, but this thread doesn't really have value. Not only do you misunderstand Regex and it's applications, but the examples you've shown are really bad.

1) In your example you used Regex to parse an HTML document, meanwhile, an HTML parsing library would be a lot better, not just because of performance, but because of maintainability. It's extremely difficult debugging fat regex expressions, and that's why you should them sparingly. What you have just shown can be done using AngleSharp parsing library and offer way more readability and maintainability than your regex expressions.

2) I've noticed that members on this forum seem to think that Regex is for parsing, but the truth is, it isn't. Regex is designed for pattern matching.

3)

Nope, Regex is far slower than string functions. It's only in the past decade that Regex has been used more, and that's because machines are now more powerful, and can easily handle regex expressions. Back when regex was created in the 90s, they were very slow as computers then lacked power.

thanks for your opinion

[vaiperboy]

using regex for parsing a html doc is by far the worst option

[realsunjester]

im sorry if you are all confused but its a tutorial on using regular expressions, not parsing html. thanks you for your opinions though.

[BahNahNah]

thanks for your opinion

Sorry boi but aint no opinion

https://stackoverflow.com/a/1732454/2691759

You would normally use String.Split when there is only one occurrence of a string in the block your searching.

Im not sure you understand what String.Split is used for either. One occurrence of what?

Yes, regex is a fast-to-prototype, hacky way to extract very small specific data. But it is not a substitute for a parser.

---

im sorry if you are all confused but its a tutorial on using regular expressions, not parsing html. thanks you for your opinions though.

If you want to make an educational post on how to use regex, you should probably use it correctly.

[vaiperboy]

Code

foreach(Match m in mc)
{
        Console.WriteLine("ID: " + m.Groups[1].Value + " \t Name: " + m.Groups[2].Value);
}

please, i hate foreach loops

use

Code

mc.ToList().ForEach(f => {Console.WriteLine($"ID: { m.Groups[1].Value } \t Name: { m.Groups[2].Value }");

no need for tolist i believe, idk if its enumerable or not

[Owl]

Match a single character
\w

Match any character or digit

.

lol

im sorry if you are all confused but its a tutorial on using regular expressions, not parsing html.

All you are doing is using regex to parse html but thx for your opinion.

please, i hate foreach loops

use

Code

mc.ToList().ForEach(f => {Console.WriteLine($"ID: { m.Groups[1].Value } \t Name: { m.Groups[2].Value }");

no need for tolist i believe, idk if its enumerable or not

Of course it's enumerable it's a MatchCollection??? The one thing OP got right and you call it out.

Code

mc.OfType<Match>().ToList().ForEach(f =>

To make that nonsense work.

[JMuller]

@OP, it's a good opportunity to update your knowledge, don't let your ego get in the way. These aren't opinions, they're widely accepted truths and best practices.

[vaiperboy]

Of course it's enumerable it's a MatchCollection??? The one thing OP got right and you call it out.

well to be fair I didn't read the thread
just got to the html part and ditched it


also i didnt sleep for 21 hours so far. give me a break

[HFDone]

lol


All you are doing is using regex to parse html but thx for your opinion.


Of course it's enumerable it's a MatchCollection??? The one thing OP got right and you call it out.

Code

mc.OfType<Match>().ToList().ForEach(f =>

To make that nonsense work.

linq is slower so its depends on what you using it for

[SneakyHF]

Thank you for this beautiful tutorial, I can't stress enough how useful this can be. Everyone needs to get on this yo

[itcorp]

linq is slower so its depends on what you using it for

Here's a small test of speed with good and bad examples of code

Code

class Program
    {
        static void Main(string[] args)
        {
            var src = new WebClient().DownloadString("https://pastr.io/raw/NehxTX");
            Regex r = new Regex(@"https:\/\/cracked[.]to\/member[.]php[?]action=profile&uid=(\d+)" title="([a-zA-Z0-9&_\.-]+)"><span");
            MatchCollection mc = r.Matches(src);

            Test1(mc);

            Test1a(mc);

            Test2(mc);

            Test3(mc);

            Test4(r,src);

            Console.ReadLine();
        }

        static void Test1(MatchCollection items)
        {
            Stopwatch s = new Stopwatch();
            s.Start();

            foreach (Match item in items)
            {
                Console.WriteLine("ID: " + item.Groups[1].Value + " \t Name: " + item.Groups[2].Value);
            }

            Console.WriteLine("Test1");
            Console.WriteLine("Time:" + s.ElapsedMilliseconds);
            Console.WriteLine("=====================================================");
        }

        static void Test1a(MatchCollection items)
        {
            Stopwatch s = new Stopwatch();
            s.Start();

            foreach (var item in items.OfType<Match>())
            {                
                Console.WriteLine("ID: " + item.Groups[1].Value + " \t Name: " + item.Groups[2].Value);
            }

            Console.WriteLine("Test1a");
            Console.WriteLine("Time:" + s.ElapsedMilliseconds);
            Console.WriteLine("=====================================================");
        }


        static void Test2(MatchCollection items)
        {
            Stopwatch s = new Stopwatch();
            s.Start();

            for (int i =0; i < items.Count; i++)
            {
                Console.WriteLine($"ID: {items[i].Groups[1].Value } \t Name: { items[i].Groups[2].Value }");
            }

            Console.WriteLine("Test2");
            Console.WriteLine("Time:" + s.ElapsedMilliseconds);
            Console.WriteLine("=====================================================");
        }
        
        static void Test3(MatchCollection items)
        {
            Stopwatch s = new Stopwatch();
            s.Start();

            items.OfType<Match>().ToList().ForEach( i =>
            
                Console.WriteLine($"ID: {i.Groups[1].Value } \t Name: { i.Groups[2].Value }")
            );

            Console.WriteLine("Test3");
            Console.WriteLine("Time:" + s.ElapsedMilliseconds);
            Console.WriteLine("=====================================================");
        }

        static void Test4(Regex matches,string src)
        {
            Stopwatch s = new Stopwatch();
            s.Start();

            matches.Matches(src).Cast<Match>().ToList().ForEach(i => Console.WriteLine($"ID: {i.Groups[1].Value } \t Name: { i.Groups[2].Value }") );
                                
            Console.WriteLine("Test4");
            Console.WriteLine("Time:" + s.ElapsedMilliseconds);
            Console.WriteLine("=====================================================");
        }
    }