![]() |
|
[Help] Web Automation Technologies? - Printable Version +- Hack Forums (https://hackforums.net) +-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444) +--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151) +---- Forum: Coders Lounge (https://hackforums.net/forumdisplay.php?fid=5) +---- Thread: [Help] Web Automation Technologies? (/showthread.php?tid=4310940) Pages: Pages
1
2
|
Web Automation Technologies? - Shahbazi - Jun 28, 2014 Im looking to create a program to perform some web automation to buy certain things from certain websites on my behalf. This will involve loading web pages, running site searches, and filling out web forms. Does anyone with experience in this field have any advice for me in terms of what technologies/frameworks are best to use? I have had a go with phantomjs which seemed extremely buggy and poor to me, and have also thought about using python but im not sure how easy it is to do. Im looking for something, fast, lightweight, and as easy as possible to program Any help appreciated, cheers * - Chickendipper - Jun 28, 2014 Any language is good. The easiest way would be VB.NET or C# with the WebBrowser control, since that keeps the session data like a normal browser. You can then invoke clicks, typing etc. * - ~Dec - Jun 28, 2014 PhantomJS is not buggy at all. The problem with it is that you interact with the site via JavaScript, and don't immediately have a means of waiting for pages to load (it is a browser, so you do have to wait for pages to load before you can do anything). I use a certain technique with PhantomJS that gets around this fairly easily. The idea is to detect if the page is currently loading, and set a global variable that you can reference if the page is loading or not, as well as an index counter that starts at 0. Then, you have an array of functions (which are your "steps") and have a main function that initiates a call to each of these steps in sequence, updating the counter of each one, and each time it runs it'll pause until the "load" global flag is set to finished, so you can prevent steps from being called too early. Also, you'll need to re-route the console messages so you can actually see the browser console output. Here's an example in code: PHP Code If what you want to do is much simpler, you could just use programmatic HTTP request with something like node.js and the `request` module on npm. I used it to create a Gulp plugin that logs into userscripts.org, and uploads/updates my script for me. You can see the code here: https://gist.github.com/declandewet/e3bcbde12c29acf5efe8 * - Shahbazi - Jun 28, 2014 Thanks a lot that was very helpful to me. It seems weird that these issues have not been addressed by the developers and end users are still required to create such "workarounds". I may give phantomjs another go, i was playing around trying to get it to login to my facebook and automatically post a status but could not get it working, nor could i find any sort of output explaining why my program was not working. Do you have any experience in using python with a framework like splinter? If so i wondered if you would recommend using phantomjs or something like splinter? Cheers! * - Dialog - Jun 28, 2014 (Jun 28, 2014 11:32 AM)Chickendipper Wrote: Any language is good. Seriously? Don't use a webbrowser for that shit. * - Shahbazi - Jun 28, 2014 You mention that phantomjs uses Javascript to interact with pages and must have measures to wait for a page to load, does this mean that using something such as splinter for python would be able to fill out a series of forms faster then phantomjs? speed is a necessary in the program i wish to make * - ~Dec - Jun 28, 2014 Not necessarily. If you do anything related to sending requests to other websites, you are always going to have to wait for the page to load. The difference with PhantomJS is that it is a web browser without an interface, so it has a different method of interacting with the data, but has the added benefit of being able to run JavaScript and therefore is more likely to interact with more interactive pages, where traditional methods would usually fail. Some people believe you need a browser emulation system of some kind if cookies are set by the particular target website, but that is not the case as simple HTTP request libraries or tools like cURL offer support for persistent cookie storage. So if you wanted something that feels faster (though not actually faster) and doesn't require the need to run javascript on the target website, then use Node.js and the request module, or just use cURL on it's own. I don't code in Python. * - Shahbazi - Jun 29, 2014 (Jun 28, 2014 11:50 AM)~Dec Wrote: PhantomJS is not buggy at all. The problem with it is that you interact with the site via JavaScript, and don't immediately have a means of waiting for pages to load (it is a browser, so you do have to wait for pages to load before you can do anything). At what point do you call/use the interval? * - Chickendipper - Jun 29, 2014 (Jun 28, 2014 01:00 PM)Dialog Wrote: Seriously? Don't use a webbrowser for that shit. A WebBrowser control works, and requires minimal programming. * - ~Dec - Jun 30, 2014 (Jun 29, 2014 08:03 AM)Shahbazi Wrote: At what point do you call/use the interval? It is already being called, if you pass a function and a timer to it (which I have done) then it immediately starts. The reason I store it in a variable is so that I can stop the interval: clearInterval(interval); * - Shahbazi - Jul 6, 2014 Thanks, btw, do you know how to click a link on a page using phantomjs? thanks * - ~Dec - Jul 7, 2014 Code page.evaluate(function() {That's how you click a link using PhantomJS. * - node.js - Jul 7, 2014 Do the web pages that you are loading require Javascript? If not you can us nodejs + request + cheerio If they do require Javascript, you can use Java's HtmlUnit. It's a very easy library to use. 🤝 * - Matadoer - Jul 7, 2014 PhantomJS code is hard to debug IMO, but it's also the best solution for these types of things especially if you are interacting with JavaScript. You could use C#, nodejs or whatever but I think PhantomJS is your best bet. (If you get it working). Someone should make a PhantomJS IDE, or at least some better debugging tools... * - ~Dec - Jul 7, 2014 Debugging in PhantomJS is actually rather straightforward: https://github.com/ariya/phantomjs/wiki/Troubleshooting |