Web scrapting dynamic JavaScript websites: Should I use PhantomJs or what?
I want to be able to do web scraping of websites that have dynamic
(JavaScript/ajax) content and I also want to be good at developing tests
for my own websites which are all Single Page Apps. This means that things
I have used in the past like beautifulsoup wont work for me because it
wont run the JavaScript.
Selenium IDE
PhantomJs
Karma
node.io
ghost
spyner
HtmlUnit
I want a tool which are looking for ease of use and expressive power
Want something that is well documented and in actively developed.
I want to be able to * navigate the dom and trigger and respond to events
as in Jquery. * simulate drag and drop operations. * do ajax requests. *
do i/o the file system * interface with a database (ideally postgres)
I want the code from the remote system to be sand boxed as in a real
browser, but the code we write for our test system to to unrestricted,
able to write to a database, etc.
What tool should I use?
No comments:
Post a Comment