1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Web Crawling Service: looking for the opportunities

Discussion in 'General Business' started by Artyom, Apr 13, 2011.

  1. #1
    I'm building a clustered web-crawler that is able to crawl from many servers and easily scalable. Right now I'm in the test phase, everything basically just works, few things need a little adjustment.

    I'd really like to focus on this field full time, so I'm looking for opportunities to provide a paid service. I'd like to know your opinion if such service would be on demand on the market and what would be an appropriate pricing for it.

    What I can offer now is solving individual crawling tasks. A customer explains what he needs to crawl and the speed he needs to achieve, amount of data he wants to collect, etc and I'm solving this task. It can be a one time job or providing the fresh data on a regular basis.

    Basically it's good if you need to collect some content for your projects or to analyse some data, to get any statistics.

    Just a few examples of what can be done:

    • Collecting a database with the products from different sites
    • Collecting a database of product reviews
    • Collecting a database of jobs from different job sites
    • Collecting a database of real estate and property for rent
    • Crawling profiles from social networks and dating websites
    • Crawling the internet for some particular data
    • Different data mining tasks
    To solve such task now, people usually hire some free-lancer to write a simple PHP script that will crawl for ages. For now I have a crawling speed about 15mln pages per day from a single server, but it's a scalable number almost without limits (depends on amount of servers I'll add to the cloud)

    What do you guys think?
    I'm also open for any ideas and business opportunities related to field of web-crawling and processing large amounts of data.
     
    Artyom, Apr 13, 2011 IP