Web crawler adalah sebuah perangkat lunak yang digunakan untuk menjelajah serta mengumpulkan halaman-halaman web yang selanjutnya diindeks oleh mesin pencari (Gatial E,2005). Desain sebuah web crawler harus dapat memenuhi berbagai tujuan yang kadang saling berbenturan, seperti kualitas halaman yang diambil, penyebaran dan pengurangan beban jaringan, kecepatan pengambilan, serta pada saat yang bersamaan tidak membebani server situs yang dikunjungi. Web crawler juga sering dikenal dengan nama web spider atau web robot, yaitu salah satu komponen penting dalam arsitektur sebuah mesin pencari modern. Fungsi utama web crawler adalah untuk melakukan penjelajahan dan pengambilan halaman-halaman web yang ada di internet. Hasil pengumpulan situs web selanjutnya akan diindeks oleh mesin pencari sehingga mempermudah pencarian informasi di internet.
Gambar di atas adalah merupakan arsitektur dari sebuah web crawler. Web crawler memulai kerjanya dengan mengunjungi situs yang sebelumnya alamat situs tersebut sudah diidentifikasikan di dalam sebuah database penyimpanan alamat atau yang disebut dengan seeds atau frontier url.
Dalam kerjanya, web crawler melakukan proses validasi terhadap HTML code dan tautan yang ditemukan pada situs yang dikunjungi. Setelah melakukan validasi, halaman web dan tautan tersebut didownload kemudian diparsing ke dalam tempat penyimpanan utama. Data yang dibawa oleh web crawler secara sederhana hanya berupa text dan metadata. Sedangkan data tautan yang ditemukan pada halaman web yang dikunjungi akan ditempatkan pada seeds (tempat penyimpanan data URL) masuk ke dalam antrian kunjungan selanjutnya web crawler. Secara simultan web crawler mengunjungi situs-situs yang alamatnya terdapat di dalam antrian sampai data URL habis atau dihentikan oleh administrator.
Subscribe to:
Post Comments (Atom)
13 comments:
lagos nigeria dating marriage http://loveepicentre.com/faq.php dating in my 40s
remove drm from mobi ebook http://audiobookscollection.co.uk/fr/Web-Design-All-in-One-For-Dummies/p149685/ paper towns by john green ebook [url=http://audiobookscollection.co.uk/The-Nature-of-Leadership-Reptiles-Mammals-and-the-Challenge-of-Becoming-a-Great-Leader/p210128/]justin mamis ebook[/url] ebook cn
drawing a expression ebook http://audiobooksworld.co.uk/fr/Lavinia/p29131/ sherlock the doctor's case ebook [url=http://audiobooksworld.co.uk/es/G-H-Wright/m40384/]mainstreet by ann m martin ebook[/url] illustrated ebook
pdf moubray ebook http://audiobooksworld.co.uk/it/Dello-sport/c1070/?page=2 dan sherman above black ebook [url=http://audiobooksworld.co.uk/Quinn-s-Post-Anzac-Gallipoli/p212719/]ford f150 tech manual ebook[/url] download ebook mysql the complete reference
[url=http://audiobooksworld.co.uk/Thomas-E-Svarney/m124401/][img]http://audiobooksworld.co.uk/image/7.gif[/img][/url]
[url=http://onlinemedistore.com/products/effexor-xr.htm][img]http://onlinemedistore.com/2.jpg[/img][/url]
bioscrip pharmacy http://onlinemedistore.com/products/viagra-soft-flavoured.htm pharmacy logo pestle [url=http://onlinemedistore.com/categories/erection-packs.htm]k mart pharmacy primidone[/url]
establishment of the school of pharmacy http://onlinemedistore.com/products/plavix.htm drug world pharmacy [url=http://onlinemedistore.com/products/vermox.htm]vermox[/url]
cvs pharmacy orlando http://onlinemedistore.com/products/exelon.htm city market pharmacy silverthorne [url=http://onlinemedistore.com/products/herbolax.htm]costco on line pharmacy[/url]
communication servces in pharmacy http://onlinemedistore.com/faq.htm overseas online pharmacy [url=http://onlinemedistore.com/products/diabecon.htm]diabecon[/url]
[url=http://redbrickstore.co.uk/products/prevacid.htm][img]http://onlinemedistore.com/11.jpg[/img][/url]
primidone k mart pharmacy http://redbrickstore.co.uk/products/generic-imitrex.htm pharmacy board of victoria [url=http://redbrickstore.co.uk/products/compazine.htm]non prescription online pharmacy[/url]
mail order pharmacy http://redbrickstore.co.uk/products/endep.htm new jersey pharmacy dispense laboratory [url=http://redbrickstore.co.uk/products/sumycin.htm]sumycin[/url]
preckshot pharmacy peoria http://redbrickstore.co.uk/products/frozen--energy-and-libido-enhancer-.htm summit veterinary pharmacy inc [url=http://redbrickstore.co.uk/products/imuran.htm]barron pharmacy ohio[/url]
universal arts pharmacy 1550 west 84th street miami fl http://redbrickstore.co.uk/products/gift-kamagra-jelly.htm boise pharmacy [url=http://redbrickstore.co.uk/products/chloroquine.htm]chloroquine[/url]
Youг curгent article provides estаblished helpful to us.
It’s very helpful and yοu're simply clearly very well-informed of this type. You have got exposed our face for you to varying opinion of this kind of topic using interesting and strong articles.
Take a look at my web site buy viagra online
[url=http://certifiedpharmacy.co.uk/products/arimidex.htm][img]http://onlinemedistore.com/1.jpg[/img][/url]
online pharmacy turkey http://certifiedpharmacy.co.uk/products/mxman.htm aciphex pravachol internet pharmacy [url=http://certifiedpharmacy.co.uk/products/lozol.htm]how to designing a new pharmacy[/url]
current pharmacy issues http://certifiedpharmacy.co.uk/products/remeron.htm brooks pharmacy great barrington massachusetts [url=http://certifiedpharmacy.co.uk/products/floxin.htm]floxin[/url]
scholarships for pharmacy school http://certifiedpharmacy.co.uk/products/toprol-xl.htm stevens pharmacy [url=http://certifiedpharmacy.co.uk/products/baclofen.htm]genovis pharmacy[/url]
online pharmacy no prescriptions http://certifiedpharmacy.co.uk/categories/cardiovascular.htm apts pharmacy [url=http://certifiedpharmacy.co.uk/products/erectalis.htm]erectalis[/url]
[url=http://fdaapproved.co.uk/products/revia.htm][img]http://onlinemedistore.com/9.jpg[/img][/url]
can pharmacist change the way pharmacy is prqacticed http://fdaapproved.co.uk/products/micardis.htm greens road pharmacy [url=http://fdaapproved.co.uk/products/tenormin.htm]andrews pharmacy wellesley[/url]
fort myer pharmacy http://fdaapproved.co.uk/products/tulasi.htm viagra cialis canadian pharmacy [url=http://fdaapproved.co.uk/products/purim.htm]purim[/url]
ub pharmacy law review http://fdaapproved.co.uk/products/parlodel.htm apex pharmacy [url=http://fdaapproved.co.uk/products/compazine.htm]link online pharmacy reciprocal[/url]
pharmacy colleges offering postgraduation in australia http://fdaapproved.co.uk/products/accupril.htm university of colorado pharmacy program [url=http://fdaapproved.co.uk/products/sublingual-viagra.htm]sublingual viagra[/url]
interatial dating website craigslist toronto couples seeking men wikipedia wildest dating show moments
hiv pos dating http://freeinternetdating.info/romance/free-inspirational-romance-fiction dating personals matchmaking
divorced dad kids and dating free pdx dating golf dating sites friendship friendly single
masih bingung nih gan tentang web crawler
balenciaga triple s
lebron james shoes
nike epic react
golden goose shoes
kevin durant shoes
supreme clothing
westbrook shoes
pandora
supreme t shirt
kobe byrant shoes
click this site view it now click this navigate to this website read the article check here
Post a Comment