How much GB it would be to index all urls and links on the planet?

Hi,

How much GB it would be to index all urls and all
links to that urls on the planet?

If you want to write an ansewer please do it and include
way you calculate.

by the way if you know any software that would handle it
please write.

(all in all to do something like link:site.com inachor:keyword,
to know how many sites links with specific anchor to specifick site)

 

Are you for real?

 

Ask google

 

Its called the web graph by search engine researchers. They store the net as a graph. with pages being the nodes and links being the lines. Thas how they calculate precisely how many links you have.

And this graph is constantly (every second) changing as the bot crawls. apparently google has the biggest graph and thats not for sale. But i heard some minor search engines sell their web graphs so that other search engines can start with a good db. i dunno the cost or where to get it.do a google abt it.

Any way the graph is in TB range, some 2-3 TBs. so u can imagine what google has in store. They cache the pages too. So its HUGEEEEEEE. The web graph of google will be around 1000TB and the cache store i cant imagine.

 

eleventyoogle

 

So, you'd need a few million dollars to get started... and I doubt you can do it with some "software".

Time to stop dreaming and get back to work...

 

What ever you think it is, triple it, times it by 9999999999999999999999999 write down the awnser. It is alot more than that.

It will be massive, alot bigger than you would expect.

Greg

 

1000TB isnt even that much nowadays
If you get started now, it would be really difficult to catch up

 

Remember that they are not storing it the way we store. They have the best db optimisation. They use Oracle to its maximum. Its not that much huge, the web graph when its in their db. but the cache store will be massive.

 

great answer, i'll mail them.




king_cobra >> how you calculated?

and whats eleventyoogle?


l234244 - after minutes of lol yes i'm for real.

 

i've send mail to google and local search engine.

king_cobra - i think about web graph, only.

 

oh oh do let us know what their answer is!

 

yeah dude ask google or another major search engine

 

Ugh. I gotta engage brain. I think it is estimated that google index less than 10% of all web pages on the net. The number 8% sticks in my mind. That does not include dynamic generated pages which are a bugger to index. So, take the marketing lies about how many pages they index and multiply by ten or twelve and that is how many there are.

The problem is less storage and more bandwidth and intelligent retry strategy. Also, dealing with those nasty, NASTY dynamic pages will drive you nutz.

 

LMAO!

 

Great advise normanu, i've send w question to Microsoft Search. they got cool search builder better the google.

How much GB it would be to index all urls and links on the planet?

Not planet but still some data:
8 058 044 651 * 50 bytes * 16 = 5.86299913 terabytes

50 bytes for urls, 16 for links on each page, still mych then ther is.
when using Western Digital 180GB's , with prize $0,50/GB

that would be


$3001 and 85c for hard discs,


What software could do it?

 

Well, you'd want a distributed network of crawlers. Google's are written in Python I think; PHP & CURL would probably do the job too. I think 50 bytes/URL is a tad low though, even if you're only indexing static pages...

Example: http://www.llanfairpwllgwyngyllgogerychwyrndrobwyll-llantysiliogogogoch.com/ - that's 88 characters right there

 

Dayam son...are you dumb? Even if your 5.86299913 terabytes calculation is accurate (probably low IMHO) you would need more than 30+ hard drives because you need redundancy. Oh, and please don't tell me you plan on using IDE drives too...LOL. Oh, BTW, Dell doesn't sell computers that hold 30+ drives. And software that can index webpages, store them on a RAID 5 disk array with 30+ drives doesn't write itself and it's not sold online either.

You need to do alot more research and take some computer science classes because you are clueless.

 

I think that sums it up

 

Erm... while there's a degree of naivety here, I don't think calling him names is much smarter. Don't shoot someone down for being curious or asking interesting questions. And it's really not hugely complicated to build a network of storage devices to hold this kind of information. e.g. http://www.microsoft.com/windows2000/techinfo/howitworks/fileandprint/dfsnew.asp, http://www.google.com/search?hl=en&...l&q=unix+distributed+file+systems&btnG=Search, there's plenty of "out of the box" solutions to the storage question.

Maybe not both, but there's plenty of software that does either.