twitter大規(guī)模機器學習解決方案

1、LargeScaleMachineLearningatTwitterJimmyLinAlekKolczTwitterInc.ABSTRACTThesuccessofdatadrivensolutionstodifficultproblemsalongwiththepingcostsofstingprocessingmassiveamountsofdatahasledtogrowinginterestinlargescalemachine

2、learning.ThispaperpresentsacasestudyofTwitter’sintegrationofmachinelearningtoolsintoitsexistingHadoopbasedPigcentricanalyticsplatfm.Webeginwithanoverviewofthisplatfmwhichhles“traditional”datawarehousingbusinessintelligen

3、cetasksftheganization.TheceofthiswkliesinrecentPigextensionstoprovidepredictiveanalyticscapabilitiesthatincpatemachinelearningfocusedspecificallyonsupervisedclassification.Inparticularwehaveidentifiedstochasticgradientde

4、scenttechniquesfonlinelearningensemblemethodsasbeinghighlyamenabletoscalingouttolargeamountsofdata.Inourdeployedsolutioncommonmachinelearningtaskssuchasdatasamplingfeaturegenerationtrainingtestingcanbeaccomplisheddirectl

5、yinPigviacarefullycraftedloadersstagefunctionsuserdefinedfunctions.ThismeansthatmachinelearningisjustanotherPigwhichallowsseamlessintegrationwithexistinginfrastructurefdatamanagementschedulingmonitinginaproductionenviron

6、mentaswellasaccesstorichlibrariesofuserdefinedfunctionsthematerializedoutputofothers.CategiesSubjectDes:H.2.3[DatabaseManagement]:LanguagesGeneralTerms:LanguagesKeywds:stochasticgradientdescentonlinelearningensembleslogi

7、sticregression1.INTRODUCTIONHadooptheopensourceimplementationofMapReduce[15]hasemergedasapopularframewkflargescaledataprocessing.Amongitsadvantagesaretheabilitytohizontallyscaletopetabytesofdataonthoussofcommodityservers

8、easytounderstprogrammingsemanticsaPermissiontomakedigitalhardcopiesofallpartofthiswkfpersonalclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadedistributedfprofitcommercialadvantagethatcopiesbearthisnoticethefull

9、citationonthefirstpage.Tocopyotherwisetorepublishtopostonserverstoredistributetolistsrequiresprispecificpermissionafee.SIGMOD’12May20–242012ScottsdaleArizonaUSA.Copyright2012ACM97814503124791205...$10.00.highdegreeoffaul

10、ttolerance.AlthoughiginallydesignedfapplicationssuchastextanalysiswebindexinggraphprocessingHadoopcanbeappliedtomanagestructureddataaswellas“dirty”semistructureddatasetswithinconsistentschemamissingfieldsinvalidvalues.To

11、dayHadoopenjoyswidespreadadoptioninganizationsrangingfromtwopersonstartupstoFtune500companies.Itliesattheceofasoftwarestackflargescaleanalyticsowesalargepartofitssuccesstoavibrantecosystem.FexamplePig[37]Hive[47]provideh

12、igherlevellanguagesfdataanalysis:adataflowlanguagecalledPigLatinadialectofSQLrespectively.HBasetheopensourceimplementationofGoogle’sBigtable[13]providesaconvenientdatamodelfmanagingservingsemistructureddata.Wearealsowitn

13、essingthedevelopmentofhybriddataprocessingapproachesthatintegrateHadoopwithtraditionalRDBMStechniques[134330]promisingthebestofbothwlds.ThevalueofaHadoopbasedstackf“traditional”datawarehousingbusinessintelligencetaskshas

14、alreadybeendemonstratedbyganizationssuchasFacebookLinkedInTwitter(e.g.[2241]).Thisvaluepropositionalsoliesatthecenterofagrowinglistofstartupslargecompaniesthathaveenteredthe“bigdata”game.CommontasksincludeETLjoiningmulti

15、pledisparatedatasourcesfollowedbyfilteringaggregationcubematerialization.Statisticiansmightusethephrasedeivestatisticstodescribethistypeofanalysis.Theseoutputsmightfeedreptgeneratsfrontenddashboardsothervisualizationtool

16、stosupptcommon“rollup”“drilldown”operationsonmultidimensionaldata.Hadoopbasedplatfmshavealsobeensuccessfulinsupptingadhocqueriesbyanewbreedofengineersknownas“datascientists”.ThesuccessoftheHadoopplatfmdrivesinfrastructur

17、edeveloperstobuildincreasinglypowerfultoolswhichdatascientistsotherengineerscanexploittoextractinsightsfrommassiveamountsofdata.Inparticularwefocusonmachinelearningtechniquesthatenablewhatmightbebesttermedpredictiveanaly

18、tics.Thehopeistominestatisticalregularitieswhichcanthenbedistilledintomodelsthatarecapableofmakingpredictionsaboutfutureevents.Someexamplesinclude:IsthistweetspamnotWhatstarratingistheuserlikelytogivetothismovieShouldthe

19、setwopeoplebeintroducedtoeachotherHowlikelywilltheuserclickonthisbanneradThispaperpresentsacasestudyofhowmachinelearningtoolsareintegratedintoTwitter’sPigcentricanalyticsstackfthetypeofpredictiveanalyticsdescribedabove.F

20、ocus793encodeAretherebestpracticestoadoptWhilethispaperdoesnotdefinitivelyanswerthesequestionsweofferacasestudy.SinceTwitter’sanalyticsstackconsistsmostlyofopensourcecomponents(HadoopPigetc.)muchofourexperienceisgenerali

21、zabletootherganizations.3.TWITTER’SANALYTICSSTACKAlargeHadoopclusterliesattheceofouranalyticsinfrastructurewhichservestheentirecompany.DataiswrittentotheHadoopDistributedFileSystem(HDFS)viaanumberofrealtimebatchprocesses

22、inavarietyoffmats.Thesedatacanbebulkexptsfromdatabasesapplicationlogsmanyothersources.WhenthecontentsofarecdarewelldefinedtheyareserializedusingeitherProtocolBuffers3Thrift.4IngesteddataareLZOcompressedwhichprovidesagood

23、tradeoffbetweencompressionratiospeed(see[29]fmedetails).InaHadoopjobdifferentrecdtypesproducedifferenttypesofinputkeyvaluepairsfthemapperseachofwhichrequirescustomcodefdeserializingparsing.Sincethiscodeisbothregularrepet

24、itiveitisstraightfwardtousetheserializationframewktospecifythedataschemafromwhichtheserializationcompilergeneratescodetoreadwritemanipulatethedata.ThisishledbyoursystemcalledElephantBird5whichautomaticallygeneratesHadoop

25、recdreaderswritersfarbitraryProtocolBufferThriftmessages.InsteadofdirectlywritingHadoopcodeinJavaanalyticsatTwitterisperfmedmostlyusingPigahighleveldataflowlanguagethatcompilesintophysicalplansthatareexecutedonHadoop[371

26、9].Pig(viaalanguagecalledPigLatin)providesconciseprimitivesfexpressingcommonoperationssuchasprojectioniongroupjoinetc.Thisconcisenesscomesatlowcost:PigsapproachtheperfmanceofprogramsdirectlywritteninHadoopJava.Yetthefull

27、expressivenessofJavaisretainedthroughalibraryofcustomUDFsthatexposeceTwitterlibraries(e.g.fextractingmanipulatingpartsoftweets).FthepurposesofthispaperweassumethatthereaderhasatleastapassingfamiliaritywithPig.Likemanygan

28、izationstheanalyticswkloadatTwittercanbebroadlydividedintotwocategies:aggregationqueriesadhocqueries.Theaggregationqueriesmaterializecommonlyusedintermediatedatafsubsequentanalysisfeedfrontenddashboards.Theserepresentrel

29、ativelystardbusinessintelligencetasksprimarilyinvolvescansoverlargeamountsofdatatriggeredperiodicallybyourinternalwkflowmanager(seebelow).Runningalongsidetheseaggregationqueriesareadhocqueriese.g.oneoffbusinessrequestsfd

30、ataprototypesofnewfunctionalitiesexperimentsbyouranalyticsgroup.Thesequeriesareusuallysubmitteddirectlybytheuserhavenopredictabledataaccesscomputationalpattern.Althoughsuchjobsroutinelyprocesslargeamountsofdatatheyareclo

31、serto“needleinahaystack”queriesthanaggregationqueries.ProductionanalyticsjobsarecodinatedbyourwkflowmanagercalledOinkwhichschedulesrecurringjobsatfixedintervals(e.g.hourlydaily).Oinkhlesdataflow3:code.pprotobuf4:thrift.a

32、pache.g5kevinweilelephantbirddependenciesbetweenjobsfexampleifjobBrequiresdatageneratedbyjobAthenOinkwillscheduleAverifythatAhassuccessfullycompletedthenschedulejobB(allwhilemakingabestefftattempttorespectperiodicitycons

33、traints).FinallyOinkpreservesexecutiontracesfauditpurposes:whenajobbeganhowlongitlastedwhetheritcompletedsuccessfullyetc.EachdayOinkscheduleshundredsofPigswhichtranslateintothoussofHadoopjobs.4.EXTENDINGPIGTheprevioussec

34、tiondescribesamatureproductionsystemthathasbeenrunningsuccessfullyfseveralyearsiscriticaltomanyaspectsofbusinessoperations.InthissectionwedetailPigextensionsthataugmentthisdataanalyticsplatfmwithmachinelearningcapabiliti

35、es.4.1DevelopmentHistyTobetterappreciatethesolutionthatwehavedevelopeditisperhapshelpfultodescribethedevelopmenthisty.Twitterhasbeenusingmachinelearningsinceitsearliestdays.SummizeatwoyearoldstartupthatTwitteracquiredpri

36、marilyfitssearchproductin2008hadaspartofitstechnologyptfoliosentimentanalysiscapabilitiesbasedinpartonmachinelearning.AftertheacquisitionmachinelearningcontributedtospamdetectionotherapplicationswithinTwitter.Theseactivi

37、tiespredatedtheexistenceofHadoopwhatonemightrecognizeasamoderndataanalyticsplatfm.Sinceourgoalhasneverbeentomakefundamentalcontributionstomachinelearningwehavetakenthepragmaticapproachofusingofftheshelftoolkitswherepossi

38、ble.Thusthechallengebecomeshowtoincpatethirdpartysoftwarepackagesalongwithinhousetoolsintoanexistingwkflow.Mostcommonlyavailablemachinelearningtoolkitsaredesignedfasinglemachinecannoteasilyscaletothedatasetsizesthatouran

39、alyticsplatfmcaneasilygenerate(althoughmedetaileddiscussionbelow).Asaresultweoftenrestedtosampling.Thefollowingdescribesanotuncommonscenario:LikemostanalyticstaskswebeganwithdatamanipulationusingPigontheinfrastructuredes

40、cribedinSection3.TheswouldstreamoverlargedatasetsextractsignalsofinterestmaterializethemtoHDFS(aslabelsfeaturevects).Fmanytasksitwasaseasytogenerateamilliontrainingexamplesasitwastogeneratetenmilliontrainingexamplesme.Ho

41、wevergeneratingtoomuchdatawascounterproductiveasweoftenhadtodownsamplethedatasoitcouldbehledbyamachinelearningalgithmonasinglemachine.ThetrainingprocesstypicallyinvolvedcopyingthedataoutofHDFSontothelocaldiskofanothermac

42、hine—frequentlythiswasanothermachineinthedatacenterbutrunningexperimentsonindividuals’laptopswasnotuncommon.Onceamodelwastraineditwasappliedinasimilarlyadhocmanner.TestdatawerepreparedsampledusingPigcopiedoutofHDFSfedtot

43、helearnedmodel.TheseresultswerethenstedsomewhereflateraccessfexampleinaflatfilethatisthencopiedbacktoHDFSasrecdsedintoadatabaseetc.Therearemanyissueswiththiswkflowthefemostofwhichisthatdownsamplinglargelydefeatsthepointo