[HardStoN]

Hi guys
the ive created a html request and got the response for getting the source code of the page
but the source that i got is not exactly the same as in the real page and it cause me some problem because
some element is not in my source but in the real page it is there.

Code

  string html = "";

           HttpWebRequest req = (HttpWebRequest)WebRequest.Create("my link");
           HttpWebResponse res = (HttpWebResponse)req.GetResponse();

           using (StreamReader reader = new StreamReader(res.GetResponseStream()))
               html = reader.ReadToEnd();

Console.WriteLine(html);

[Owl]

My guess is the data you want is delivered via ajax, which web requests don't execute. You will need to source and request from an API.

[HardStoN]

My guess is the data you want is delivered via ajax, which web requests don't execute. You will need to source and request from an API.

the missing line is
<div class="dbg0pd" aria-level="3" role="heading"><span>burger bar</span></div>
and if its got something to do with ajax.
can you guide me through on how to solve this problem.

Thanks for the help

[itcorp]

the missing line is
<div class="dbg0pd" aria-level="3" role="heading"><span>burger bar</span></div>
and if its got something to do with ajax.
can you guide me through on how to solve this problem.

Thanks for the help

Could you give us a url with which you work to check?Otherwise this line has nothing to do with AJAX.

[StarZ]

If you're performing a GET request, using a WebClient is quicker. It should return the same data as it's just a wrapper for a WebRequest.

Code

WebClient wc = new WebClient();
var pageData = wc.DownloadString("http://www.google.com/");

Off topic, but it does clean up code a fair bit.

[HardStoN]

I cant post links because i dont have 50 posts so you just need to search google restraunt near me and thats it

and im trying to get all of the restraunt that google shows

[StarZ]

You need to add a referrer. It works fine for me.

Let me pass down an old tool I wrote. It's a tester for WebRequests. Instead of Compiling > Testing > Changing > etc..., you can just change variables from a UI.

Here's a screenie of my test:
Spoiler (Click to View)

Download:

https://anonfile.com/g2q5Pcp3nc/WebRequester_exe

VT:

https://www.virustotal.com/#/file/af784d.../detection

[HardStoN]

Yeah your program do the same mistake, i dont know if the post data is wrong or we just cant get that spesific response but in the real site the response contains Invalid credantials...... but on our post data the response is the same 400 Bad request (thats what we need to get if the credentials are invalid) but the html source contains
Sorry, it appears that you have tried to submit the form twice. Please click only once and not invalid credentials.

[ＪＵＳＴＩＮ]

it's very unlikely that the issue is the need of a referrer. if you wan't to make sure it's not anything related to headers, simply look at the networking tab in chrome and copy every header your browser sends by default.

more likely than not, the issue is that the element you're looking for is being added to the page after the page is actually loaded via javascript or something. to confirm this, just view the source code of the page (view-source:http://example.com/whatever)

if what you're looking for isn't in there, it wont be in a standard get request either. the logical way to get around this would be to implement an actual browser. even something simple like the WebBrowser component may work, but there are also some more legit browsers that you can install via nuget such as CEFSharp. you'd want to load the page in there and after DocumentCompleted is triggered you can get the source code from the webBrowser.DocumentText property.

==== edit ====
i've tested with the example you mentioned (seeing nearby restaurants on google) and i can actually see the restaurant names by sending a simple get request. since this is the case, ignore what i said about using a browser. google actually might be searching for some type of header, like referrer or user agent.

[StarZ]

i've tested with the example you mentioned (seeing nearby restaurants on google) and i can actually see the restaurant names by sending a simple get request. since this is the case, ignore what i said about using a browser. google actually might be searching for some type of header, like referrer or user agent.

Yeah. I've noticed a few sites I've used check for a null referrer or uagent. Tbh nowadays I always just add a ref and assume it requires a uagent. Makes life simpler. In my screenie I was able to pull info using GET aswell. Not sure why he's trying to POST. If he's trying to perform a login to google, he'll need to read up on CSRF & reCaptcha, and give it another shot.

[HardStoN]

Yeah. I've noticed a few sites I've used check for a null referrer or uagent. Tbh nowadays I always just add a ref and assume it requires a uagent. Makes life simpler. In my screenie I was able to pull info using GET aswell. Not sure why he's trying to POST. If he's trying to perform a login to google, he'll need to read up on CSRF & reCaptcha, and give it another shot.

The other message that i sent is mentioned for another site a login site, and yeah i noticed there is a special header called x-xsrf-token which isnt stored in the html source as hidden. But for the first message ill try what you said.
Do you mean to just create a webbrowser and get the info from there?

[StarZ]

The other message that i sent is mentioned for another site a login site, and yeah i noticed there is a special header called x-xsrf-token which isnt stored in the html source as hidden. But for the first message ill try what you said.
Do you mean to just create a webbrowser and get the info from there?

Is it stored in a Cookie? Are you searching the html for the variable name or the value?

And I never recommend webbrowser controls for these tasks. They're slow and resource intensive. If you provide a little more information, we may be able to help you out a bit more. Y'know, like, a URL for example.

[ＪＵＳＴＩＮ]

The other message that i sent is mentioned for another site a login site, and yeah i noticed there is a special header called x-xsrf-token which isnt stored in the html source as hidden. But for the first message ill try what you said.
Do you mean to just create a webbrowser and get the info from there?

This is a common tactic used to make emulating login requests more difficult. They're specifically used to prevent CSRF (Cross-Site Request Forgery). Typically, when you visit a website for the first time, the website will set a cookie. Here's an example of the response headers when I go to the login page of a website I own:

Code

cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
content-type: text/html; charset=UTF-8
date: Sun, 12 May 2019 08:38:20 GMT
expires: Thu, 19 Nov 1981 08:52:00 GMT
pragma: no-cache
server: Apache
set-cookie: PHPSESSID=euvprk8bcbk7hb1qppit2fuqm1; expires=Tue, 11-Jun-2019 19:07:27 GMT; Max-Age=2629746; path=/
status: 200
x-powered-by: PHP/5.6.40

The important thing to look out for here is that set-cookie header. Response headers are used to tell a browser to store a cookie on that domain. It provides a name, a value, an expiration date, and some other information that we wont worry about right now. In this specific example, it sets a PHPSESSID cookie.
When I click the "login" button, after entering my username and password, here's the Form Data that gets posted:

Code

username: justin
password: [my password]
remember: on
v: MFBKLTlUWC2yMzE=

Now the odd one out here is that 'v' parameter. When the page is loaded, it's randomly generated and inserted into the html form as a hidden element. It's sent via POST behind the scenes, but the user never sees it. Behind the scenes, that randomly generated 'v' value, which we can refer to as a token, is stored in the PHPSESSID. Once the login request is submitted, it does 2 things:
1.) It confirms that the username/password are valid, and the user can login to their account.
2.) It confirms that the 'v' data matches whatever we stored in our PHP session.

So what does this do?
In conclusion, it makes it slightly harder for people to keep sending web requests. It's not fool proof, it's just very simple protection. The developer would need to load the login page before submitting a login request, and keep the generated cookie/stored value when POSTing their actual login request.

How can we get around it?
This is pretty simple. Take this class, for example:


This class works like a normal WebClient class, however it automatically stores and keeps cookies after each each request. Note that if you'd prefer to use HttpWebRequest objects you can do that too, the CookieContainer property is what you'll need to look out for.

Anyway, as I recommended in one of your other posts, it's important to have a complete understanding of HTTP and some other common web development concepts. You'll find that many of the problems you face are related to other protocols and techniques, not specifically the C# programming language.
HTTP, alongside plenty of other varying protocols, are managed by the IETF aka the Internet Engineering Task Force. On the site (ietf.org), you'll be able to find complete documentation regarding every prominent protocol/standard in existence. Here's a link to the RFC for HTTP: https://www.ietf.org/rfc/rfc2616.txt
Any question you'll have about HTTP requests can be answered by that page.

[HardStoN]

I wanted to send a link with it but i couldnt because i dont have enough posts.

I want to get the Value of the name X-XSRF-TOKEN which stored in a cookie.

My question is after i got the cookie container and set it equals to the request of the login page itself how do i get that specific element at the cookie.

There is no search by name or something like that there is only one thing that shows up its the GetCookies(Uri u) and i dont know what uri they want me to put in.


I think i should probably learn more about cookies but meanwhile can someone tell me how to search a specific header in a cookie and get his value.

Thanks for the help

[ＪＵＳＴＩＮ]

I think i should probably learn more about cookies but meanwhile can someone tell me how to search a specific header in a cookie and get his value.

The way that CookieContainer works is a bit funky, and afaik the only way to access all cookie information is by using reflection. The following code will get every cookie from the container regardless of path/domain.
You can make some small modifications to it, and return cookie.Value when cookie.Name == "X=XSRF=TOKEN", for example.

Not that it might look nicer if you just pass the Uri into GetCookies, you wont have to deal with all this reflection mumbo jumbo - but here's the code anyway.



edit:
Here's the example I'm talking about, without reflection.
You just need to specify the website the cookie will be on.

Code

IEnumerable<Cookie> cookies = container.GetCookiesByName(new Uri("https://example.com"), "X-CSRF-TOKEN");
foreach(Cookie cookie in cookies)
    Console.WriteLine(cookie.Name + " = " + cookie.Value);