+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [Python] How can I find backend APIs that a website uses to fetch products and stock? (/showthread.php?tid=5862274)
How can I find backend APIs that a website uses to fetch products and stock? - FlexagonV2 - Jul 15, 2018
I'm trying to do it for a few websites using demandware, yet simply using f12 isn't working. I don't want o parse html, since that's too slow. Is there another way for me to find the json file they're using to find products list?
RE: How can I find backend APIs that a website uses to fetch products and stock? - Nomad.Null - Jul 16, 2018
You have to enumerate/spider/crawl the site if it is undocumented. I recommend using the proxy feature in burp suite to intercept requests and responses to view them easily. Unfortunately you're going to have a hard time regardless, but burp suite will make it infinitely easier for you.
RE: How can I find backend APIs that a website uses to fetch products and stock? - FlexagonV2 - Jul 16, 2018
(Jul 16, 2018 06:46 AM)True Demon Wrote: You have to enumerate/spider/crawl the site if it is undocumented. I recommend using the proxy feature in burp suite to intercept requests and responses to view them easily. Unfortunately you're going to have a hard time regardless, but burp suite will make it infinitely easier for you.I was able to find the target one by simply using f12, but not exactly sure on how to find it for hottopic .com or any demandware sites.
RE: How can I find backend APIs that a website uses to fetch products and stock? - WhiteGirl - Jul 16, 2018
(Jul 16, 2018 01:47 PM)FlexagonV2 Wrote: I was able to find the target one by simply using f12, but not exactly sure on how to find it for hottopic .com or any demandware sites.
You can't just find sources for all retailers. Like the other previous poster said, you have to scrape it yourself. This is coming from someone that has done so many spiders on python that I can do them with my eyes close.
RE: How can I find backend APIs that a website uses to fetch products and stock? - FlexagonV2 - Jul 16, 2018
(Jul 16, 2018 01:53 PM)WhiteGirl Wrote: You can't just find sources for all retailers. Like the other previous poster said, you have to scrape it yourself. This is coming from someone that has done so many spiders on python that I can do them with my eyes close.
I'm using python for this as well. Can you give me instructions on how I would go about finding the API? I sort of have an idea of what a spider is, but would you clarify?
Like what am I exactly looking for?
RE: How can I find backend APIs that a website uses to fetch products and stock? - WhiteGirl - Jul 17, 2018
(Jul 16, 2018 02:00 PM)FlexagonV2 Wrote: I'm using python for this as well. Can you give me instructions on how I would go about finding the API? I sort of have an idea of what a spider is, but would you clarify?
Like what am I exactly looking for?
I was referring to Scrapy spiders. I've spent so much time doing Scrapy , that I got really bored after a while. On top of that, 95% of those spiders were paid for clients. Now, I'm dedicating my time to reverse engineer a bigger target, one of the hardest known to botters.
Scraping retailers is not really difficult, some are super easy, others a little bit hard but nonetheless you can figure out how to automate just by looking at the http(s) requests. I got tired of doing that for hundreds of sites, as it got too easy for me, I wasn't learning anything new.
The API that you are looking for is mostly the json requests these ecommerce sites use to talk to their backend. It is not documented at all. Nobody provides documented API to pull their ecommerce data, that is why there is a huge market if you can curate the data from a lot of retailers and offer an API.
RE: How can I find backend APIs that a website uses to fetch products and stock? - FlexagonV2 - Jul 18, 2018
(Jul 17, 2018 04:41 AM)WhiteGirl Wrote: I was referring to Scrapy spiders. I've spent so much time doing Scrapy , that I got really bored after a while. On top of that, 95% of those spiders were paid for clients. Now, I'm dedicating my time to reverse engineer a bigger target, one of the hardest known to botters.
Scraping retailers is not really difficult, some are super easy, others a little bit hard but nonetheless you can figure out how to automate just by looking at the http(s) requests. I got tired of doing that for hundreds of sites, as it got too easy for me, I wasn't learning anything new.
The API that you are looking for is mostly the json requests these ecommerce sites use to talk to their backend. It is not documented at all. Nobody provides documented API to pull their ecommerce data, that is why there is a huge market if you can curate the data from a lot of retailers and offer an API.
Woah. Can you sort of teach me or guide me in how I can do this?
RE: How can I find backend APIs that a website uses to fetch products and stock? - Skidsuper - Jul 19, 2018
Both chrome and firefox inspect element (developer tools) has a network tab which is helpful. Also you can try using fiddler.
RE: How can I find backend APIs that a website uses to fetch products and stock? - FlexagonV2 - Jul 20, 2018
(Jul 19, 2018 11:58 PM)Skidsuper Wrote: Both chrome and firefox inspect element (developer tools) has a network tab which is helpful. Also you can try using fiddler.
I have tried inspect element, but even through network, I can't find where they get their products from. Also, can you tell me how to use fiddler to find hidden APIs?
RE: How can I find backend APIs that a website uses to fetch products and stock? - koolkid - Jul 20, 2018
(Jul 20, 2018 02:43 PM)FlexagonV2 Wrote: I have tried inspect element, but even through network, I can't find where they get their products from. Also, can you tell me how to use fiddler to find hidden APIs?
do you know how to use the developer tools that come with ur web browser ? more so firefox or chrome ?
RE: How can I find backend APIs that a website uses to fetch products and stock? - FlexagonV2 - Jul 20, 2018
(Jul 20, 2018 02:55 PM)koolkid Wrote: do you know how to use the developer tools that come with ur web browser ? more so firefox or chrome ?
Yeah. I have tried using developer tools, but I can't find the stock of the items on sites with demandware.
RE: How can I find backend APIs that a website uses to fetch products and stock? - FlexagonV2 - Aug 6, 2018
(Jul 17, 2018 04:41 AM)WhiteGirl Wrote: I was referring to Scrapy spiders. I've spent so much time doing Scrapy , that I got really bored after a while. On top of that, 95% of those spiders were paid for clients. Now, I'm dedicating my time to reverse engineer a bigger target, one of the hardest known to botters.
Scraping retailers is not really difficult, some are super easy, others a little bit hard but nonetheless you can figure out how to automate just by looking at the http(s) requests. I got tired of doing that for hundreds of sites, as it got too easy for me, I wasn't learning anything new.
The API that you are looking for is mostly the json requests these ecommerce sites use to talk to their backend. It is not documented at all. Nobody provides documented API to pull their ecommerce data, that is why there is a huge market if you can curate the data from a lot of retailers and offer an API.
WhiteGIrl, can you PM me?