Web Crawler

Written By pcbolong on Wednesday, April 20, 2011 | 10:40 AM

Web crawler adalah sebuah perangkat lunak yang digunakan untuk menjelajah serta mengumpulkan halaman-halaman web yang selanjutnya diindeks oleh mesin pencari (Gatial E,2005). Desain sebuah web crawler harus dapat memenuhi berbagai tujuan yang kadang saling berbenturan, seperti kualitas halaman yang diambil, penyebaran dan pengurangan beban jaringan, kecepatan pengambilan, serta pada saat yang bersamaan tidak membebani server situs yang dikunjungi. Web crawler juga sering dikenal dengan nama web spider atau web robot, yaitu salah satu komponen penting dalam arsitektur sebuah mesin pencari modern. Fungsi utama web crawler adalah untuk melakukan penjelajahan dan pengambilan halaman-halaman web yang ada di internet. Hasil pengumpulan situs web selanjutnya akan diindeks oleh mesin pencari sehingga mempermudah pencarian informasi di internet.

Gambar di atas adalah merupakan arsitektur dari sebuah web crawler. Web crawler memulai kerjanya dengan mengunjungi situs yang sebelumnya alamat situs tersebut sudah diidentifikasikan di dalam sebuah database penyimpanan alamat atau yang disebut dengan seeds atau frontier url.

Dalam kerjanya, web crawler melakukan proses validasi terhadap HTML code dan tautan yang ditemukan pada situs yang dikunjungi. Setelah melakukan validasi, halaman web dan tautan tersebut didownload kemudian diparsing ke dalam tempat penyimpanan utama. Data yang dibawa oleh web crawler secara sederhana hanya berupa text dan metadata. Sedangkan data tautan yang ditemukan pada halaman web yang dikunjungi akan ditempatkan pada seeds (tempat penyimpanan data URL) masuk ke dalam antrian kunjungan selanjutnya web crawler. Secara simultan web crawler mengunjungi situs-situs yang alamatnya terdapat di dalam antrian sampai data URL habis atau dihentikan oleh administrator.

11 comments:

Anonymous said...

lagos nigeria dating marriage http://loveepicentre.com/faq.php dating in my 40s

Anonymous said...

remove drm from mobi ebook http://audiobookscollection.co.uk/fr/Web-Design-All-in-One-For-Dummies/p149685/ paper towns by john green ebook [url=http://audiobookscollection.co.uk/The-Nature-of-Leadership-Reptiles-Mammals-and-the-Challenge-of-Becoming-a-Great-Leader/p210128/]justin mamis ebook[/url] ebook cn

Anonymous said...

drawing a expression ebook http://audiobooksworld.co.uk/fr/Lavinia/p29131/ sherlock the doctor's case ebook [url=http://audiobooksworld.co.uk/es/G-H-Wright/m40384/]mainstreet by ann m martin ebook[/url] illustrated ebook

Anonymous said...

pdf moubray ebook http://audiobooksworld.co.uk/it/Dello-sport/c1070/?page=2 dan sherman above black ebook [url=http://audiobooksworld.co.uk/Quinn-s-Post-Anzac-Gallipoli/p212719/]ford f150 tech manual ebook[/url] download ebook mysql the complete reference
[url=http://audiobooksworld.co.uk/Thomas-E-Svarney/m124401/][img]http://audiobooksworld.co.uk/image/7.gif[/img][/url]

Anonymous said...

[url=http://onlinemedistore.com/products/effexor-xr.htm][img]http://onlinemedistore.com/2.jpg[/img][/url]
bioscrip pharmacy http://onlinemedistore.com/products/viagra-soft-flavoured.htm pharmacy logo pestle [url=http://onlinemedistore.com/categories/erection-packs.htm]k mart pharmacy primidone[/url]
establishment of the school of pharmacy http://onlinemedistore.com/products/plavix.htm drug world pharmacy [url=http://onlinemedistore.com/products/vermox.htm]vermox[/url]
cvs pharmacy orlando http://onlinemedistore.com/products/exelon.htm city market pharmacy silverthorne [url=http://onlinemedistore.com/products/herbolax.htm]costco on line pharmacy[/url]
communication servces in pharmacy http://onlinemedistore.com/faq.htm overseas online pharmacy [url=http://onlinemedistore.com/products/diabecon.htm]diabecon[/url]

Anonymous said...

[url=http://redbrickstore.co.uk/products/prevacid.htm][img]http://onlinemedistore.com/11.jpg[/img][/url]
primidone k mart pharmacy http://redbrickstore.co.uk/products/generic-imitrex.htm pharmacy board of victoria [url=http://redbrickstore.co.uk/products/compazine.htm]non prescription online pharmacy[/url]
mail order pharmacy http://redbrickstore.co.uk/products/endep.htm new jersey pharmacy dispense laboratory [url=http://redbrickstore.co.uk/products/sumycin.htm]sumycin[/url]
preckshot pharmacy peoria http://redbrickstore.co.uk/products/frozen--energy-and-libido-enhancer-.htm summit veterinary pharmacy inc [url=http://redbrickstore.co.uk/products/imuran.htm]barron pharmacy ohio[/url]
universal arts pharmacy 1550 west 84th street miami fl http://redbrickstore.co.uk/products/gift-kamagra-jelly.htm boise pharmacy [url=http://redbrickstore.co.uk/products/chloroquine.htm]chloroquine[/url]

Anonymous said...

Youг curгent article provides estаblished helpful to us.
It’s very helpful and yοu're simply clearly very well-informed of this type. You have got exposed our face for you to varying opinion of this kind of topic using interesting and strong articles.
Take a look at my web site buy viagra online

Anonymous said...

[url=http://certifiedpharmacy.co.uk/products/arimidex.htm][img]http://onlinemedistore.com/1.jpg[/img][/url]
online pharmacy turkey http://certifiedpharmacy.co.uk/products/mxman.htm aciphex pravachol internet pharmacy [url=http://certifiedpharmacy.co.uk/products/lozol.htm]how to designing a new pharmacy[/url]
current pharmacy issues http://certifiedpharmacy.co.uk/products/remeron.htm brooks pharmacy great barrington massachusetts [url=http://certifiedpharmacy.co.uk/products/floxin.htm]floxin[/url]
scholarships for pharmacy school http://certifiedpharmacy.co.uk/products/toprol-xl.htm stevens pharmacy [url=http://certifiedpharmacy.co.uk/products/baclofen.htm]genovis pharmacy[/url]
online pharmacy no prescriptions http://certifiedpharmacy.co.uk/categories/cardiovascular.htm apts pharmacy [url=http://certifiedpharmacy.co.uk/products/erectalis.htm]erectalis[/url]

Anonymous said...

The aгticlе provideѕ proven necesѕary tο
me. It’ѕ reаllу uѕeful and уou're certainly quite educated of this type. You have opened up my face for you to various opinion of this specific matter together with interesting and strong written content.
my web site - Buy Xanax

Anonymous said...

[url=http://fdaapproved.co.uk/products/revia.htm][img]http://onlinemedistore.com/9.jpg[/img][/url]
can pharmacist change the way pharmacy is prqacticed http://fdaapproved.co.uk/products/micardis.htm greens road pharmacy [url=http://fdaapproved.co.uk/products/tenormin.htm]andrews pharmacy wellesley[/url]
fort myer pharmacy http://fdaapproved.co.uk/products/tulasi.htm viagra cialis canadian pharmacy [url=http://fdaapproved.co.uk/products/purim.htm]purim[/url]
ub pharmacy law review http://fdaapproved.co.uk/products/parlodel.htm apex pharmacy [url=http://fdaapproved.co.uk/products/compazine.htm]link online pharmacy reciprocal[/url]
pharmacy colleges offering postgraduation in australia http://fdaapproved.co.uk/products/accupril.htm university of colorado pharmacy program [url=http://fdaapproved.co.uk/products/sublingual-viagra.htm]sublingual viagra[/url]

Anonymous said...

interatial dating website craigslist toronto couples seeking men wikipedia wildest dating show moments
hiv pos dating http://freeinternetdating.info/romance/free-inspirational-romance-fiction dating personals matchmaking
divorced dad kids and dating free pdx dating golf dating sites friendship friendly single