MapReduce框架下的任務(wù)調(diào)度算法研究.pdf

1、聲明本學(xué)位論文是我在導(dǎo)師的指導(dǎo)下取得的研究成果，盡我所知，在本學(xué)位論文中，除了加以標(biāo)注和致謝的部分外，不包含其他人已經(jīng)發(fā)表或公布過的研究成果，也不包含我為獲得任何教育機(jī)構(gòu)的學(xué)位或?qū)W歷而使用過的材料。與我一同工作的同事對(duì)本學(xué)位論文做出的貢獻(xiàn)均已在論文中作了明確的說明。研究生簽名：了＆凇∞c]年鄉(xiāng)月l。日學(xué)位論文使用授權(quán)聲明南京理工大學(xué)有權(quán)保存本學(xué)位論文的電子和紙質(zhì)文檔，可以借閱或上網(wǎng)公布本學(xué)位論文的部分或全部?jī)?nèi)容，可以向有關(guān)部門或機(jī)構(gòu)送交

2、并授權(quán)其保存、借閱或上網(wǎng)公布本學(xué)位論文的部分或全部?jī)?nèi)容。對(duì)于保密論文，按保密的有關(guān)規(guī)定和程序處理。研究生簽名：丑熬叢研究生簽名：J觸鹽加l夕年弓月f銦碩士學(xué)位論文AbstractRecently，bigdatacomputinghasbecomearesearchhotspot，HadoopandSparkclustersarebothbasedonMapReduceandmostcommonlyusedbig—dataclusters

3、processingframeworksResourceschedulingisanimportantfactorthataffectstheperformanceoflarge—scaledataprocessingofdistributedclusterframeworkTherefore，theresearchoftaskschedulingalgorithminMapReducebasedHadoopandSparkenviro

4、nmenthasimportanttheoreticalvalueandpracticalsignificanceThispaperexploresthebatchschedulingalgorithmunderHadoopenvironmentandtheresourceschedulingmethodwhenSparkistakenasaserviceInordertooptimizethemaximumcompletiontime

5、ofthebatchschedulingproblemunderHadoopenvironment，themodelofthisproblemistransformedintotwostagehybridflowtaskschedulingproblem謝thsetuptimeinthispaperAndheuristicalgorithms，DAGEA(DirectedAcyclicGraphEarliestAvailable)and

6、DAGEF(DirectedAcyclicGraphEarliestFinish)，whicharebasedonDAG(DirectedAcyclicGraph)modelareproposedExistingsolutionalgorithmsareoftenbasedonGanttchartstructureThesemethodsCan’teffectivelyconsidertheschedulingscopeofeachjo

7、bDifferentfromthis，DAGEAandDAGarebasedonDAGEFstructureTheycalculatetheschedulingrangeofeachjobbyDAGandadjustthestarttimeoftheoperation，SOastoeffectivelyimprovetheperformanceandefficiencyofthealgorithmSimulationexperiment

8、sverifythisconclusionThecomputingofSparkisbasedonmemories，whilethisprocessingofHadoopisbasedonthediskExistingSparkresourceschedulingmethodsjusttakethenumberofsparecoresandmemoryrequirementsintoconsiderationInthispaper，we

9、addtheclusternodeutilizationandtheprocessingabilityofeachnodeintoconsideration，re—evaluatetheresourceutilizationofeachnode，andallocateresourcestothetasksThenewschedulingalgorithmMEAN，reducesthegranularityofresourcepartit

10、ioningThereforeitCanimprovetheresourceutilization，increasethenumberofonlineWebrequests，andimprovetheconcurrencyTaskschedulingandresourceallocationisthekeyofbigdatacomputingplatform，anditsqualitydirectlydecidestheperforma

11、nceoftheplatformTheMapReduceschedulingalgorithmsinthispaperfocusonthebatchprocessingschedulingalgorithmunderHadoopenvironmentandtheresourceallocationmethodundertheenvironmentofSparkAlgorithmslikeDAGEA，DAGEFandMEANareprop