[Shahbazi]

Im looking to create a program to perform some web automation to buy certain things from certain websites on my behalf. This will involve loading web pages, running site searches, and filling out web forms.

Does anyone with experience in this field have any advice for me in terms of what technologies/frameworks are best to use?

I have had a go with phantomjs which seemed extremely buggy and poor to me, and have also thought about using python but im not sure how easy it is to do. Im looking for something, fast, lightweight, and as easy as possible to program

Any help appreciated, cheers

[Chickendipper]

Any language is good.
The easiest way would be VB.NET or C# with the WebBrowser control, since that keeps the session data like a normal browser. You can then invoke clicks, typing etc.

[~Dec]

PhantomJS is not buggy at all. The problem with it is that you interact with the site via JavaScript, and don't immediately have a means of waiting for pages to load (it is a browser, so you do have to wait for pages to load before you can do anything).

I use a certain technique with PhantomJS that gets around this fairly easily. The idea is to detect if the page is currently loading, and set a global variable that you can reference if the page is loading or not, as well as an index counter that starts at 0. Then, you have an array of functions (which are your "steps") and have a main function that initiates a call to each of these steps in sequence, updating the counter of each one, and each time it runs it'll pause until the "load" global flag is set to finished, so you can prevent steps from being called too early. Also, you'll need to re-route the console messages so you can actually see the browser console output.

Here's an example in code:

PHP Code


<?php 
var page = new WebPage();
var index = 0;
var loadInProgress = false;

page.onConsoleMessage = function(msg) {
  console.log(msg);
};

page.onLoadStarted = function() {
  loadInProgress = true;
  console.log("load started");
};

page.onLoadFinished = function() {
  loadInProgress = false;
  console.log("load finished");
};

var steps = [
  function() {
    // load login page
    page.open("https://website.com/theformpage/");
  },
  function() {
    // enter credentials
    page.evaluate(function() {

      var arr = document.getElementsByClassName("login-form");
      var i;

      for (i=0; i < arr.length; i++) { 
      if (arr[i].getAttribute('method') == "POST") {

      arr[i].elements["email"].value="mylogin";
      arr[i].elements["password"].value="mypassword";
      return;
      }
      }
    });
  }, 
  function() {
    //Login
    page.evaluate(function() {
      var arr = document.getElementsByClassName("login-form");
      var i;

      for (i=0; i < arr.length; i++) {
      if (arr[i].getAttribute('method') == "POST") {
      arr[i].submit();
      return;
      }
      }

    });
  }, 
  function() {
    // Output content of page to stdout after form has been submitted
    page.evaluate(function() {
      console.log(document.querySelectorAll('html')[0].outerHTML);
    });
  }
];

var interval = setInterval(function() {
  if (!loadInProgress && typeof steps[index] === "function") {
    console.log("step " + (index + 1));
    steps[index]();
    index++;
  }
  if (typeof steps[index] !== "function") {
    console.log("test complete!");
    phantom.exit();
  }
}, 50); 


If what you want to do is much simpler, you could just use programmatic HTTP request with something like node.js and the `request` module on npm. I used it to create a Gulp plugin that logs into userscripts.org, and uploads/updates my script for me. You can see the code here: https://gist.github.com/declandewet/e3bc...29acf5efe8

[Shahbazi]

Thanks a lot that was very helpful to me. It seems weird that these issues have not been addressed by the developers and end users are still required to create such "workarounds". I may give phantomjs another go, i was playing around trying to get it to login to my facebook and automatically post a status but could not get it working, nor could i find any sort of output explaining why my program was not working.

Do you have any experience in using python with a framework like splinter? If so i wondered if you would recommend using phantomjs or something like splinter?

Cheers!

[Dialog]

Any language is good.
The easiest way would be VB.NET or C# with the WebBrowser control, since that keeps the session data like a normal browser. You can then invoke clicks, typing etc.

Seriously? Don't use a webbrowser for that shit.

[Shahbazi]

You mention that phantomjs uses Javascript to interact with pages and must have measures to wait for a page to load, does this mean that using something such as splinter for python would be able to fill out a series of forms faster then phantomjs? speed is a necessary in the program i wish to make

[~Dec]

Not necessarily. If you do anything related to sending requests to other websites, you are always going to have to wait for the page to load. The difference with PhantomJS is that it is a web browser without an interface, so it has a different method of interacting with the data, but has the added benefit of being able to run JavaScript and therefore is more likely to interact with more interactive pages, where traditional methods would usually fail. Some people believe you need a browser emulation system of some kind if cookies are set by the particular target website, but that is not the case as simple HTTP request libraries or tools like cURL offer support for persistent cookie storage. So if you wanted something that feels faster (though not actually faster) and doesn't require the need to run javascript on the target website, then use Node.js and the request module, or just use cURL on it's own.

I don't code in Python.

[Shahbazi]

PhantomJS is not buggy at all. The problem with it is that you interact with the site via JavaScript, and don't immediately have a means of waiting for pages to load (it is a browser, so you do have to wait for pages to load before you can do anything).

I use a certain technique with PhantomJS that gets around this fairly easily. The idea is to detect if the page is currently loading, and set a global variable that you can reference if the page is loading or not, as well as an index counter that starts at 0. Then, you have an array of functions (which are your "steps") and have a main function that initiates a call to each of these steps in sequence, updating the counter of each one, and each time it runs it'll pause until the "load" global flag is set to finished, so you can prevent steps from being called too early. Also, you'll need to re-route the console messages so you can actually see the browser console output.

Here's an example in code:

PHP Code


<?php 
var page = new WebPage();
var index = 0;
var loadInProgress = false;

page.onConsoleMessage = function(msg) {
  console.log(msg);
};

page.onLoadStarted = function() {
  loadInProgress = true;
  console.log("load started");
};

page.onLoadFinished = function() {
  loadInProgress = false;
  console.log("load finished");
};

var steps = [
  function() {
    // load login page
    page.open("https://website.com/theformpage/");
  },
  function() {
    // enter credentials
    page.evaluate(function() {

      var arr = document.getElementsByClassName("login-form");
      var i;

      for (i=0; i < arr.length; i++) { 
      if (arr[i].getAttribute('method') == "POST") {

      arr[i].elements["email"].value="mylogin";
      arr[i].elements["password"].value="mypassword";
      return;
      }
      }
    });
  }, 
  function() {
    //Login
    page.evaluate(function() {
      var arr = document.getElementsByClassName("login-form");
      var i;

      for (i=0; i < arr.length; i++) {
      if (arr[i].getAttribute('method') == "POST") {
      arr[i].submit();
      return;
      }
      }

    });
  }, 
  function() {
    // Output content of page to stdout after form has been submitted
    page.evaluate(function() {
      console.log(document.querySelectorAll('html')[0].outerHTML);
    });
  }
];

var interval = setInterval(function() {
  if (!loadInProgress && typeof steps[index] === "function") {
    console.log("step " + (index + 1));
    steps[index]();
    index++;
  }
  if (typeof steps[index] !== "function") {
    console.log("test complete!");
    phantom.exit();
  }
}, 50); 


If what you want to do is much simpler, you could just use programmatic HTTP request with something like node.js and the `request` module on npm. I used it to create a Gulp plugin that logs into userscripts.org, and uploads/updates my script for me. You can see the code here: https://gist.github.com/declandewet/e3bc...29acf5efe8

At what point do you call/use the interval?

[Chickendipper]

Seriously? Don't use a webbrowser for that shit.

A WebBrowser control works, and requires minimal programming.

[~Dec]

At what point do you call/use the interval?

It is already being called, if you pass a function and a timer to it (which I have done) then it immediately starts. The reason I store it in a variable is so that I can stop the interval: clearInterval(interval);

[Shahbazi]

Thanks, btw, do you know how to click a link on a page using phantomjs? thanks

[~Dec]

Code

page.evaluate(function() {
  document.querySelector('a.the-link').click();
});

That's how you click a link using PhantomJS.

[node.js]

Do the web pages that you are loading require Javascript? If not you can us

nodejs + request + cheerio

If they do require Javascript, you can use Java's HtmlUnit. It's a very easy library to use. 🤝

[Matadoer]

PhantomJS code is hard to debug IMO, but it's also the best solution for these types of things especially if you are interacting with JavaScript.

You could use C#, nodejs or whatever but I think PhantomJS is your best bet. (If you get it working).

Someone should make a PhantomJS IDE, or at least some better debugging tools...

[~Dec]

Debugging in PhantomJS is actually rather straightforward: https://github.com/ariya/phantomjs/wiki/Troubleshooting