[FlexagonV2]

I'm trying to do it for a few websites using demandware, yet simply using f12 isn't working. I don't want o parse html, since that's too slow. Is there another way for me to find the json file they're using to find products list?

[Nomad.Null]

You have to enumerate/spider/crawl the site if it is undocumented. I recommend using the proxy feature in burp suite to intercept requests and responses to view them easily. Unfortunately you're going to have a hard time regardless, but burp suite will make it infinitely easier for you.

[FlexagonV2]

You have to enumerate/spider/crawl the site if it is undocumented. I recommend using the proxy feature in burp suite to intercept requests and responses to view them easily. Unfortunately you're going to have a hard time regardless, but burp suite will make it infinitely easier for you.

I was able to find the target one by simply using f12, but not exactly sure on how to find it for hottopic .com or any demandware sites.

[WhiteGirl]

I was able to find the target one by simply using f12, but not exactly sure on how to find it for hottopic .com or any demandware sites.

You can't just find sources for all retailers. Like the other previous poster said, you have to scrape it yourself. This is coming from someone that has done so many spiders on python that I can do them with my eyes close.

[FlexagonV2]

You can't just find sources for all retailers. Like the other previous poster said, you have to scrape it yourself. This is coming from someone that has done so many spiders on python that I can do them with my eyes close.

I'm using python for this as well. Can you give me instructions on how I would go about finding the API? I sort of have an idea of what a spider is, but would you clarify?

---

Like what am I exactly looking for?

[WhiteGirl]

I'm using python for this as well. Can you give me instructions on how I would go about finding the API? I sort of have an idea of what a spider is, but would you clarify?

---

Like what am I exactly looking for?

I was referring to Scrapy spiders. I've spent so much time doing Scrapy , that I got really bored after a while. On top of that, 95% of those spiders were paid for clients. Now, I'm dedicating my time to reverse engineer a bigger target, one of the hardest known to botters.

Scraping retailers is not really difficult, some are super easy, others a little bit hard but nonetheless you can figure out how to automate just by looking at the http(s) requests. I got tired of doing that for hundreds of sites, as it got too easy for me, I wasn't learning anything new.

The API that you are looking for is mostly the json requests these ecommerce sites use to talk to their backend. It is not documented at all. Nobody provides documented API to pull their ecommerce data, that is why there is a huge market if you can curate the data from a lot of retailers and offer an API.

[FlexagonV2]

I was referring to Scrapy spiders. I've spent so much time doing Scrapy , that I got really bored after a while. On top of that, 95% of those spiders were paid for clients. Now, I'm dedicating my time to reverse engineer a bigger target, one of the hardest known to botters.

Scraping retailers is not really difficult, some are super easy, others a little bit hard but nonetheless you can figure out how to automate just by looking at the http(s) requests. I got tired of doing that for hundreds of sites, as it got too easy for me, I wasn't learning anything new.

The API that you are looking for is mostly the json requests these ecommerce sites use to talk to their backend. It is not documented at all. Nobody provides documented API to pull their ecommerce data, that is why there is a huge market if you can curate the data from a lot of retailers and offer an API.

Woah. Can you sort of teach me or guide me in how I can do this?

[Skidsuper]

Both chrome and firefox inspect element (developer tools) has a network tab which is helpful. Also you can try using fiddler.

[FlexagonV2]

Both chrome and firefox inspect element (developer tools) has a network tab which is helpful. Also you can try using fiddler.

I have tried inspect element, but even through network, I can't find where they get their products from. Also, can you tell me how to use fiddler to find hidden APIs?

[koolkid]

I have tried inspect element, but even through network, I can't find where they get their products from. Also, can you tell me how to use fiddler to find hidden APIs?

do you know how to use the developer tools that come with ur web browser ? more so firefox or chrome ?

[FlexagonV2]

do you know how to use the developer tools that come with ur web browser ? more so firefox or chrome ?

Yeah. I have tried using developer tools, but I can't find the stock of the items on sites with demandware.

[FlexagonV2]

I was referring to Scrapy spiders. I've spent so much time doing Scrapy , that I got really bored after a while. On top of that, 95% of those spiders were paid for clients. Now, I'm dedicating my time to reverse engineer a bigger target, one of the hardest known to botters.

Scraping retailers is not really difficult, some are super easy, others a little bit hard but nonetheless you can figure out how to automate just by looking at the http(s) requests. I got tired of doing that for hundreds of sites, as it got too easy for me, I wasn't learning anything new.

The API that you are looking for is mostly the json requests these ecommerce sites use to talk to their backend. It is not documented at all. Nobody provides documented API to pull their ecommerce data, that is why there is a huge market if you can curate the data from a lot of retailers and offer an API.

WhiteGIrl, can you PM me?