IceScraper – Optimized Data sets provider

Abstract

Data is said to be the new oil for the coming decade. Every organization is after the data these days spending millions of dollars to be ahead in the game. Web scraping and bots are among one of the novel tools that businesses are focusing on. Commercial web scraping tools are every where. But there are lot of limitations while it can be used for a given scenario. Scraping from a single page & creating a web scraping agent are completely two different tasks. Scraping a single page is just an attribute of web scraping agent. Web scraping agent consist of techniques such as by pass IP blocks, deal with bandwidth & Ram , css changes etc. Web scraping agent should be able to scrape multiple pages by handling java script without failures , do that job in speed of single page scrapers’ speed, keep the efficiency & consistency. The proposed solution is a framework to scrape multiple web pages with an increased accuracy while maintaing the web elements like Javascript and CSS. The prototype presented would be gathering data sets for a given set of URLs by an user.