相似網(wǎng)頁去重算法的并行化研究與實現(xiàn).pdf

1、華中科技大學(xué)碩士學(xué)位論文相似網(wǎng)頁去重算法的并行化研究與實現(xiàn)姓名：丁益斌申請學(xué)位級別：碩士專業(yè)：計算機應(yīng)用技術(shù)指導(dǎo)教師：李瑞軒20090522II華中科技大學(xué)碩士學(xué)位論文AbstractDuetotheWebmirrsplagiarismduplicatenearduplicatewebpagesarecreatingaseriesofproblemsfsearchengines.Iteitherleadstotheextraduplic

2、atepageindexstageexertsgreatburdentothesearchengines.Atthesametimethesearchingresultsfillingwithduplicatecontentgivetheusersanannoyingfeedback.Thustheeffectiveduplicatepagedetectionalgithmsareneededtodiscoverremovemostof

3、theduplicatesreducetheburdenofthecrawlersofsearchenginesleadtobetterperfmance.ThesearchenginetechnologyadvancesquicklyinrecentyearsthenearduplicateremovaliscrucialfWebdatagathering.Firstlywebpage’stopiccontentblockshould

4、berecognizedwhilenoiseinfmationsuchasadvertisementshouldberemoved.Thenthecontentwillbetokenizedaccdingtothelexiconthepage’sfeaturesetwillbecomputedbasedonshinglealgithm.AtlastSimhash’algithmwillbeadoptedtocomputethefinge

5、rprintofthepage’featureset.Thisfingerprintcanrepresentwebpage’stopiccontentfeatureiftwohavenearduplicatecontenttheirfingerprintwillhavesmallHammingDistance.Meoverthethesisputsfwardtheimprovedparallelprocessingmethodftrad

6、itionalShinglemethodfnearduplicatepageremoval.AtlastthisthesispresentsthearchitectureoftheWebpagegatheringsystemimplementsanearduplicatepageremovalprototypesystem.Theprototypesystemhastwowkingmode:oneisonlinewkingmodewhi

7、chsinglefingerprintwillbecomparedwithhugenumberofpreviousfingerprintdatatheotherisparallelprocessingmodewhichasetoffingerprintswillbecomparedwithhugenumberofpreviousfingerprintdata.Todistinguishfromtheonlinemodepreviousf

8、ingerprintdataissplittofixedsizeddatablockswhicharestedonthedistributedcomputingplatfmthehammingdistancecomputingprocessbetweenthesetoffingerprintthepreviousfingerprintdataisbasedontheMapReduceprogrammingparadigm.Through