deep learning (2015), y. lecun, y. bengio and g. hinton

1、1FacebookAIResearch770BroadwayNewYkNewYk10003USA.2NewYkUniversity715BroadwayNewYkNewYk10003USA.3DepartmentofComputerScienceOperationsResearchUniversitdeMontralPavillonrAisenstadtPOBox6128CentreVilleSTNMontralQuebecH3C3J7

2、Canada.4Google1600AmphitheatreParkwayMountainViewCalifnia94043USA.5DepartmentofComputerScienceUniversityofTonto6King’sCollegeRoadTontoOntarioM5S3G4Canada.Machinelearningtechnologypowersmanyaspectsofmodernsociety:fromwebs

3、earchestocontentfilteringonsocialwkstorecommendationsonecommercewebsitesitisincreasinglypresentinconsumerproductssuchascamerassmartphones.Machinelearningsystemsareusedtoidentifyobjectsinimagestranscribespeechintotextmatc

4、hnewsitemspostsproductswithusers’interestsrelevantresultsofsearch.Increasinglytheseapplicationsmakeuseofaclassoftechniquescalleddeeplearning.Conventionalmachinelearningtechniqueswerelimitedintheirabilitytoprocessnaturald

5、ataintheirrawfm.Fdecadesconstructingapatternrecognitionmachinelearningsystemrequiredcarefulengineeringconsiderabledomainexpertisetodesignafeatureextractthattransfmedtherawdata(suchasthepixelvaluesofanimage)intoasuitablei

6、nternalrepresentationfeaturevectfromwhichthelearningsubsystemoftenaclassifiercoulddetectclassifypatternsintheinput.Representationlearningisasetofmethodsthatallowsamachinetobefedwithrawdatatoautomaticallydiscovertherepres

7、entationsneededfdetectionclassification.Deeplearningmethodsarerepresentationlearningmethodswithmultiplelevelsofrepresentationobtainedbycomposingsimplebutnonlinearmodulesthateachtransfmtherepresentationatonelevel(starting

8、withtherawinput)intoarepresentationatahigherslightlymeabstractlevel.Withthecompositionofenoughsuchtransfmationsverycomplexfunctionscanbelearned.Fclassificationtaskshigherlayersofrepresentationamplifyaspectsoftheinputthat

9、areimptantfdiscriminationsuppressirrelevantvariations.Animagefexamplecomesinthefmofanarrayofpixelvaluesthelearnedfeaturesinthefirstlayerofrepresentationtypicallyrepresentthepresenceabsenceofedgesatparticularientationsloc

10、ationsintheimage.Thesecondlayertypicallydetectsmotifsbyspottingparticulararrangementsofedgesregardlessofsmallvariationsintheedgepositions.Thethirdlayermayassemblemotifsintolargercombinationsthatcrespondtopartsoffamiliaro

11、bjectssubsequentlayerswoulddetectobjectsascombinationsoftheseparts.Thekeyaspectofdeeplearningisthattheselayersoffeaturesarenotdesignedbyhumanengineers:theyarelearnedfromdatausingageneralpurposelearningprocedure.Deeplearn

12、ingismakingmajadvancesinsolvingproblemsthathaveresistedthebestattemptsoftheartificialintelligencecommunityfmanyyears.Ithasturnedouttobeverygoodatdiscoveringintricatestructuresinhighdimensionaldataistherefeapplicabletoman

13、ydomainsofsciencebusinessgovernment.Inadditiontobeatingrecdsinimagerecognition1–4speechrecognition5–7ithasbeatenothermachinelearningtechniquesatpredictingtheactivityofpotentialdrugmolecules8analysingparticleacceleratdata

14、910reconstructingbraincircuits11predictingtheeffectsofmutationsinnoncodingDNAongeneexpressiondisease1213.Perhapsmesurprisinglydeeplearninghasproducedextremelypromisingresultsfvarioustasksinnaturallanguageundersting14part

15、icularlytopicclassificationsentimentanalysisquestionanswering15languagetranslation1617.Wethinkthatdeeplearningwillhavemanymesuccessesinthenearfuturebecauseitrequiresverylittleengineeringbyhsoitcaneasilytakeadvantageofinc

16、reasesintheamountofavailablecomputationdata.Newlearningalgithmsarchitecturesthatarecurrentlybeingdevelopedfdeepneuralwkswillonlyacceleratethisprogress.SupervisedlearningThemostcommonfmofmachinelearningdeepnotissupervised

17、learning.Imaginethatwewanttobuildasystemthatcanclassifyimagesascontainingsayahouseacarapersonapet.Wefirstcollectalargedatasetofimagesofhousescarspeoplepetseachlabelledwithitscategy.Duringtrainingthemachineisshownanimagep

18、roducesanoutputinthefmofavectofscesonefeachcategy.Wewantthedesiredcategytohavethehighestsceofallcategiesbutthisisunlikelytohappenbefetraining.Wecomputeanobjectivefunctionthatmeasurestheerr(distance)betweentheoutputscesth

19、edesiredpatternofsces.Themachinethenmodifiesitsinternaladjustableparameterstoreducethiserr.Theseadjustableparametersoftencalledweightsarerealnumbersthatcanbeseenas‘knobs’thatdefinetheinput–outputfunctionofthemachine.Inat

20、ypicaldeeplearningsystemtheremaybehundredsofmillionsoftheseadjustableweightshundredsofmillionsoflabelledexampleswithwhichtotrainthemachine.Toproperlyadjusttheweightvectthelearningalgithmcomputesagradientvectthatfeachweig

21、htindicatesbywhatamounttheerrwouldincreasedecreaseiftheweightwereincreasedbyatinyamount.Theweightvectisthenadjustedintheoppositedirectiontothegradientvect.TheobjectivefunctionaveragedoverallthetrainingexamplescanDeeplear

22、ningallowscomputationalmodelsthatarecomposedofmultipleprocessinglayerstolearnrepresentationsofdatawithmultiplelevelsofabstraction.Thesemethodshavedramaticallyimprovedthestateoftheartinspeechrecognitionvisualobjectrecogni

23、tionobjectdetectionmanyotherdomainssuchasdrugdiscoverygenomics.Deeplearningdiscoversintricatestructureinlargedatasetsbyusingthebackpropagationalgithmtoindicatehowamachineshouldchangeitsinternalparametersthatareusedtocomp

24、utetherepresentationineachlayerfromtherepresentationinthepreviouslayer.Deepconvolutionalshavebroughtaboutbreakthroughsinprocessingimagesvideospeechaudiowhereasrecurrentshaveshonelightonsequentialdatasuchastextspeech.Deep

25、learningYannLeCun12YoshuaBengio3RGB(redgreenblue)inputsbottomright).Eachrectangularimageisafeaturemapcrespondingtotheoutputfoneofthelearnedfeaturesdetectedateachoftheimagepositions.Infmationflowsbottomupwithlowerlevelfea

26、turesactingasientededgedetectsasceiscomputedfeachimageclassinoutput.ReLUrectifiedlinearunit.RedGreenBlueSamoyed(16)Papillon(5.7)Pomeranian(2.7)Arcticfox(1.0)Eskimodog(0.6)whitewolf(0.4)Siberianhusky(0.4)ConvolutionsReLUM