[redmoon90211]

I am trying to scrape item numbers (eg. 33FR410) from this page https://www.tavor-trade.com/%D7%A7%D7%98...%95%D7%A8/

But I can't find them anywhere in the source HTML or HAR file
Completely puzzled 😧

Any help is appreciated

[Nachfolger]

I am trying to scrape item numbers (eg. 33FR410) from this page https://www.tavor-trade.com/%D7%A7%D7%98...%95%D7%A8/

But I can't find them anywhere in the source HTML or HAR file
Completely puzzled  😧

Any help is appreciated

You can't find them because they're loaded by a XHR request, which is not sent if you view the source or fetch it via cURL.

[redmoon90211]

You can't find them because they're loaded by a XHR request, which is not sent if you view the source or fetch it via cURL.

I can't find them in the HAR file saved from Chrome browser either

[Nachfolger]

I can't find them in the HAR file saved from Chrome browser either

They are all there, it's the XHR request made to "https://shop4.wizsoft.com/VSHOP/WSHOP.wzx". Block the request URI and the items will no longer load.

[redmoon90211]

They are all there, it's the XHR request made to "https://shop4.wizsoft.com/VSHOP/WSHOP.wzx". Block the request URI and the items will no longer load.

Thanks ! Still can't find them tho, may be encrypted or in base64 format, I will look into that

[Charger]

Thanks ! Still can't find them tho, may be encrypted or in base64 format, I will look into that

See if there’s any JavaScript that decodes it.

AFAIK you can see every JS call made in chrome profiles

[Nachfolger]

Thanks ! Still can't find them tho, may be encrypted or in base64 format, I will look into that

Based on what Charger just said, I decided to dig further. Here you go:

Code

eval("Tmp = " + unescape(response.responseText));

It produces a JSON object.

Aka:

Code

eval("Tmp = " + unescape(`%u007b%u0020%u0022%u0053%u0074%u0061%u0074%u0075%u0073%u0022%u003a%u0022%u004f%u004b%u0022%u002c%u0022%u004e%u006f%u0074%u0065%u0073%u0054%u0061%u0062%u0022%u003a%u005b%u005b%u0022%u0031%u0022%u002c%u0022%u05de%u05d5%u05ea%u05d2%u0022%u002c%u0022%u0030%u0022%u005d%u005d%u002c%u0022%u0043%u006f%u006c%u0075%u006d%u006e%u0073%u0022%u003a%u005b%u0022%u004e%u006f%u0074%u0065%u0049%u0044%u0022%u002c%u0022%u004e%u006f%u0074%u0065%u004e%u0061%u006d%u0065%u0022%u002c%u0022%u0049%u0073%u004e%u0075%u006d%u0065%u0072%u0069%u0063%u0022%u005d%u0020%u007d`))

Results in:

"{"Status":"OK","NotesTab":[["1","מותג","0"]],"Columns":["NoteID","NoteName","IsNumeric"]}"

[Charger]

Based on what Charger just said, I decided to dig further. Here you go:

Code

eval("Tmp = " + unescape(response.responseText));

It produces a JSON object.

Aka:

Code

eval("Tmp = " + unescape(`%u007b%u0020%u0022%u0053%u0074%u0061%u0074%u0075%u0073%u0022%u003a%u0022%u004f%u004b%u0022%u002c%u0022%u004e%u006f%u0074%u0065%u0073%u0054%u0061%u0062%u0022%u003a%u005b%u005b%u0022%u0031%u0022%u002c%u0022%u05de%u05d5%u05ea%u05d2%u0022%u002c%u0022%u0030%u0022%u005d%u005d%u002c%u0022%u0043%u006f%u006c%u0075%u006d%u006e%u0073%u0022%u003a%u005b%u0022%u004e%u006f%u0074%u0065%u0049%u0044%u0022%u002c%u0022%u004e%u006f%u0074%u0065%u004e%u0061%u006d%u0065%u0022%u002c%u0022%u0049%u0073%u004e%u0075%u006d%u0065%u0072%u0069%u0063%u0022%u005d%u0020%u007d`))

Results in:

I’m a genius I know

[Nachfolger]

I’m a genius I know

Compared to a aboriginal? Yes. Compared to me? No.

[redmoon90211]

Thank you all ! Especially Nachfolger, you rock !

[newQuery]

When it comes to scrapping, I'd always use python.
See: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

[Charger]

When it comes to scrapping, I'd always use python.
See: https://www.crummy.com/software/BeautifulSoup/bs4/doc/

simple_html_dom is available for PHP which is almost the same as BS.