Deep learning - Wikipedia

Deep learning is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Deeplearning FromWikipedia,thefreeencyclopedia Jumptonavigation Jumptosearch Branchofmachinelearning Fordeepversusshallowlearningineducationalpsychology,seeStudentapproachestolearning.Formoreinformation,seeArtificialneuralnetwork. Representingimagesonmultiplelayersofabstractionindeeplearning[1] PartofaseriesonMachinelearninganddatamining Problems Classification Clustering Regression Anomalydetection DataCleaning AutoML Associationrules Reinforcementlearning Structuredprediction Featureengineering Featurelearning Onlinelearning Semi-supervisedlearning Unsupervisedlearning Learningtorank Grammarinduction Supervisedlearning(classification •regression) Decisiontrees Ensembles Bagging Boosting Randomforest k-NN Linearregression NaiveBayes Artificialneuralnetworks Logisticregression Perceptron Relevancevectormachine(RVM) Supportvectormachine(SVM) Clustering BIRCH CURE Hierarchical k-means Expectation–maximization(EM) DBSCAN OPTICS Meanshift Dimensionalityreduction Factoranalysis CCA ICA LDA NMF PCA PGD t-SNE Structuredprediction Graphicalmodels Bayesnet Conditionalrandomfield HiddenMarkov Anomalydetection k-NN Localoutlierfactor Artificialneuralnetwork Autoencoder Cognitivecomputing Deeplearning DeepDream Multilayerperceptron RNN LSTM GRU ESN reservoircomputing RestrictedBoltzmannmachine GAN SOM Convolutionalneuralnetwork U-Net Transformer Vision Spikingneuralnetwork Memtransistor ElectrochemicalRAM(ECRAM) Reinforcementlearning Q-learning SARSA Temporaldifference(TD) Multi-agent Self-play Learningwithhumans Activelearning Crowdsourcing Human-in-the-loop Modeldiagnostics Learningcurve Theory Kernelmachines Bias–variancetradeoff Computationallearningtheory Empiricalriskminimization Occamlearning PAClearning Statisticallearning VCtheory Machine-learningvenues NeurIPS ICML ML JMLR ArXiv:cs.LG Relatedarticles Glossaryofartificialintelligence Listofdatasetsformachine-learningresearch Outlineofmachinelearning vte PartofaseriesonArtificialintelligence Majorgoals Artificialgeneralintelligence Planning Computervision Generalgameplaying Knowledgereasoning Machinelearning Naturallanguageprocessing Robotics Approaches Symbolic Deeplearning Bayesiannetworks Evolutionaryalgorithms Philosophy Chineseroom FriendlyAI Controlproblem/Takeover Ethics Existentialrisk Turingtest History Timeline Progress AIwinter Technology Applications Projects Programminglanguages Glossary Glossary vte Deeplearning(alsoknownasdeepstructuredlearning)ispartofabroaderfamilyofmachinelearningmethodsbasedonartificialneuralnetworkswithrepresentationlearning.Learningcanbesupervised,semi-supervisedorunsupervised.[2] Deep-learningarchitecturessuchasdeepneuralnetworks,deepbeliefnetworks,deepreinforcementlearning,recurrentneuralnetworksandconvolutionalneuralnetworkshavebeenappliedtofieldsincludingcomputervision,speechrecognition,naturallanguageprocessing,machinetranslation,bioinformatics,drugdesign,medicalimageanalysis,climatescience,materialinspectionandboardgameprograms,wheretheyhaveproducedresultscomparabletoandinsomecasessurpassinghumanexpertperformance.[3][4][5] Artificialneuralnetworks(ANNs)wereinspiredbyinformationprocessinganddistributedcommunicationnodesinbiologicalsystems.ANNshavevariousdifferencesfrombiologicalbrains.Specifically,artificialneuralnetworkstendtobestaticandsymbolic,whilethebiologicalbrainofmostlivingorganismsisdynamic(plastic)andanalogue.[6][7] Theadjective"deep"indeeplearningreferstotheuseofmultiplelayersinthenetwork.Earlyworkshowedthatalinearperceptroncannotbeauniversalclassifier,butthatanetworkwithanonpolynomialactivationfunctionwithonehiddenlayerofunboundedwidthcan.Deeplearningisamodernvariationwhichisconcernedwithanunboundednumberoflayersofboundedsize,whichpermitspracticalapplicationandoptimizedimplementation,whileretainingtheoreticaluniversalityundermildconditions.Indeeplearningthelayersarealsopermittedtobeheterogeneousandtodeviatewidelyfrombiologicallyinformedconnectionistmodels,forthesakeofefficiency,trainabilityandunderstandability,whencethe"structured"part. Contents 1Definition 2Overview 3Interpretations 4History 4.1Deeplearningrevolution 5Neuralnetworks 5.1Artificialneuralnetworks 5.2Deepneuralnetworks 5.2.1Challenges 6Hardware 7Applications 7.1Automaticspeechrecognition 7.2Imagerecognition 7.3Visualartprocessing 7.4Naturallanguageprocessing 7.5Drugdiscoveryandtoxicology 7.6Customerrelationshipmanagement 7.7Recommendationsystems 7.8Bioinformatics 7.9Medicalimageanalysis 7.10Mobileadvertising 7.11Imagerestoration 7.12Financialfrauddetection 7.13Military 7.14Partialdifferentialequations 7.15ImageReconstruction 8Relationtohumancognitiveandbraindevelopment 9Commercialactivity 10Criticismandcomment 10.1Theory 10.2Errors 10.3Cyberthreat 10.4Relianceonhumanmicrowork 11Seealso 12References 13Furtherreading Definition[edit] Deeplearningisaclassofmachinelearningalgorithmsthat[8]: 199–200 usesmultiplelayerstoprogressivelyextracthigher-levelfeaturesfromtherawinput.Forexample,inimageprocessing,lowerlayersmayidentifyedges,whilehigherlayersmayidentifytheconceptsrelevanttoahumansuchasdigitsorlettersorfaces. Overview[edit] Mostmoderndeeplearningmodelsarebasedonartificialneuralnetworks,specificallyconvolutionalneuralnetworks(CNN)s,althoughtheycanalsoincludepropositionalformulasorlatentvariablesorganizedlayer-wiseindeepgenerativemodelssuchasthenodesindeepbeliefnetworksanddeepBoltzmannmachines.[9] Indeeplearning,eachlevellearnstotransformitsinputdataintoaslightlymoreabstractandcompositerepresentation.Inanimagerecognitionapplication,therawinputmaybeamatrixofpixels;thefirstrepresentationallayermayabstractthepixelsandencodeedges;thesecondlayermaycomposeandencodearrangementsofedges;thethirdlayermayencodeanoseandeyes;andthefourthlayermayrecognizethattheimagecontainsaface.Importantly,adeeplearningprocesscanlearnwhichfeaturestooptimallyplaceinwhichlevelonitsown.Thisdoesnoteliminatetheneedforhand-tuning;forexample,varyingnumbersoflayersandlayersizescanprovidedifferentdegreesofabstraction.[10][11] Theword"deep"in"deeplearning"referstothenumberoflayersthroughwhichthedataistransformed.Moreprecisely,deeplearningsystemshaveasubstantialcreditassignmentpath(CAP)depth.TheCAPisthechainoftransformationsfrominputtooutput.CAPsdescribepotentiallycausalconnectionsbetweeninputandoutput.Forafeedforwardneuralnetwork,thedepthoftheCAPsisthatofthenetworkandisthenumberofhiddenlayersplusone(astheoutputlayerisalsoparameterized).Forrecurrentneuralnetworks,inwhichasignalmaypropagatethroughalayermorethanonce,theCAPdepthispotentiallyunlimited.[12]Nouniversallyagreed-uponthresholdofdepthdividesshallowlearningfromdeeplearning,butmostresearchersagreethatdeeplearninginvolvesCAPdepthhigherthan2.CAPofdepth2hasbeenshowntobeauniversalapproximatorinthesensethatitcanemulateanyfunction.[13]Beyondthat,morelayersdonotaddtothefunctionapproximatorabilityofthenetwork.Deepmodels(CAP>2)areabletoextractbetterfeaturesthanshallowmodelsandhence,extralayershelpinlearningthefeatureseffectively. Deeplearningarchitecturescanbeconstructedwithagreedylayer-by-layermethod.[14]Deeplearninghelpstodisentangletheseabstractionsandpickoutwhichfeaturesimproveperformance.[10] Forsupervisedlearningtasks,deeplearningmethodseliminatefeatureengineering,bytranslatingthedataintocompactintermediaterepresentationsakintoprincipalcomponents,andderivelayeredstructuresthatremoveredundancyinrepresentation. Deeplearningalgorithmscanbeappliedtounsupervisedlearningtasks.Thisisanimportantbenefitbecauseunlabeleddataaremoreabundantthanthelabeleddata.Examplesofdeepstructuresthatcanbetrainedinanunsupervisedmanneraredeepbeliefnetworks.[10][15] Interpretations[edit] Deepneuralnetworksaregenerallyinterpretedintermsoftheuniversalapproximationtheorem[16][17][18][19][20]orprobabilisticinference.[8][10][12][21] Theclassicuniversalapproximationtheoremconcernsthecapacityoffeedforwardneuralnetworkswithasinglehiddenlayeroffinitesizetoapproximatecontinuousfunctions.[16][17][18][19]In1989,thefirstproofwaspublishedbyGeorgeCybenkoforsigmoidactivationfunctions[16]andwasgeneralisedtofeed-forwardmulti-layerarchitecturesin1991byKurtHornik.[17]Recentworkalsoshowedthatuniversalapproximationalsoholdsfornon-boundedactivationfunctionssuchastherectifiedlinearunit.[22] Theuniversalapproximationtheoremfordeepneuralnetworksconcernsthecapacityofnetworkswithboundedwidthbutthedepthisallowedtogrow.Luetal.[20]provedthatifthewidthofadeepneuralnetworkwithReLUactivationisstrictlylargerthantheinputdimension,thenthenetworkcanapproximateanyLebesgueintegrablefunction;Ifthewidthissmallerorequaltotheinputdimension,thenadeepneuralnetworkisnotauniversalapproximator. Theprobabilisticinterpretation[21]derivesfromthefieldofmachinelearning.Itfeaturesinference,[8][9][10][12][15][21]aswellastheoptimizationconceptsoftrainingandtesting,relatedtofittingandgeneralization,respectively.Morespecifically,theprobabilisticinterpretationconsiderstheactivationnonlinearityasacumulativedistributionfunction.[21]Theprobabilisticinterpretationledtotheintroductionofdropoutasregularizerinneuralnetworks.TheprobabilisticinterpretationwasintroducedbyresearchersincludingHopfield,WidrowandNarendraandpopularizedinsurveyssuchastheonebyBishop.[23] History[edit] SomesourcespointoutthatFrankRosenblattdevelopedandexploredallofthebasicingredientsofthedeeplearningsystemsoftoday.[24]Hedescribeditinhisbook"PrinciplesofNeurodynamics:PerceptronsandtheTheoryofBrainMechanisms",publishedbyCornellAeronauticalLaboratory,Inc.,CornellUniversityin1962. Thefirstgeneral,workinglearningalgorithmforsupervised,deep,feedforward,multilayerperceptronswaspublishedbyAlexeyIvakhnenkoandLapain1967.[25]A1971paperdescribedadeepnetworkwitheightlayerstrainedbythegroupmethodofdatahandling.[26]Otherdeeplearningworkingarchitectures,specificallythosebuiltforcomputervision,beganwiththeNeocognitronintroducedbyKunihikoFukushimain1980.[27] ThetermDeepLearningwasintroducedtothemachinelearningcommunitybyRinaDechterin1986,[28]andtoartificialneuralnetworksbyIgorAizenbergandcolleaguesin2000,inthecontextofBooleanthresholdneurons.[29][30] In1989,YannLeCunetal.appliedthestandardbackpropagationalgorithm,whichhadbeenaroundasthereversemodeofautomaticdifferentiationsince1970,[31][32][33][34]toadeepneuralnetworkwiththepurposeofrecognizinghandwrittenZIPcodesonmail.Whilethealgorithmworked,trainingrequired3days.[35] In1994,AndrédeCarvalho,togetherwithMikeFairhurstandDavidBisset,publishedexperimentalresultsofamulti-layerbooleanneuralnetwork,alsoknownasaweightlessneuralnetwork,composedofa3-layersself-organisingfeatureextractionneuralnetworkmodule(SOFT)followedbyamulti-layerclassificationneuralnetworkmodule(GSN),whichwereindependentlytrained.Eachlayerinthefeatureextractionmoduleextractedfeatureswithgrowingcomplexityregardingthepreviouslayer.[36] In1995,BrendanFreydemonstratedthatitwaspossibletotrain(overtwodays)anetworkcontainingsixfullyconnectedlayersandseveralhundredhiddenunitsusingthewake-sleepalgorithm,co-developedwithPeterDayanandHinton.[37]Manyfactorscontributetotheslowspeed,includingthevanishinggradientproblemanalyzedin1991bySeppHochreiter.[38][39] Since1997,SvenBehnkeextendedthefeed-forwardhierarchicalconvolutionalapproachintheNeuralAbstractionPyramid[40]bylateralandbackwardconnectionsinordertoflexiblyincorporatecontextintodecisionsanditerativelyresolvelocalambiguities. Simplermodelsthatusetask-specifichandcraftedfeaturessuchasGaborfiltersandsupportvectormachines(SVMs)wereapopularchoiceinthe1990sand2000s,becauseofartificialneuralnetwork's(ANN)computationalcostandalackofunderstandingofhowthebrainwiresitsbiologicalnetworks. Bothshallowanddeeplearning(e.g.,recurrentnets)ofANNshavebeenexploredformanyyears.[41][42][43]Thesemethodsneveroutperformednon-uniforminternal-handcraftingGaussianmixturemodel/HiddenMarkovmodel(GMM-HMM)technologybasedongenerativemodelsofspeechtraineddiscriminatively.[44]Keydifficultieshavebeenanalyzed,includinggradientdiminishing[38]andweaktemporalcorrelationstructureinneuralpredictivemodels.[45][46]Additionaldifficultieswerethelackoftrainingdataandlimitedcomputingpower. Mostspeechrecognitionresearchersmovedawayfromneuralnetstopursuegenerativemodeling.AnexceptionwasatSRIInternationalinthelate1990s.FundedbytheUSgovernment'sNSAandDARPA,SRIstudieddeepneuralnetworksinspeechandspeakerrecognition.ThespeakerrecognitionteamledbyLarryHeckreportedsignificantsuccesswithdeepneuralnetworksinspeechprocessinginthe1998NationalInstituteofStandardsandTechnologySpeakerRecognitionevaluation.[47]TheSRIdeepneuralnetworkwasthendeployedintheNuanceVerifier,representingthefirstmajorindustrialapplicationofdeeplearning.[48] Theprincipleofelevating"raw"featuresoverhand-craftedoptimizationwasfirstexploredsuccessfullyinthearchitectureofdeepautoencoderonthe"raw"spectrogramorlinearfilter-bankfeaturesinthelate1990s,[48]showingitssuperiorityovertheMel-Cepstralfeaturesthatcontainstagesoffixedtransformationfromspectrograms.Therawfeaturesofspeech,waveforms,laterproducedexcellentlarger-scaleresults.[49] Manyaspectsofspeechrecognitionweretakenoverbyadeeplearningmethodcalledlongshort-termmemory(LSTM),arecurrentneuralnetworkpublishedbyHochreiterandSchmidhuberin1997.[50]LSTMRNNsavoidthevanishinggradientproblemandcanlearn"VeryDeepLearning"tasks[12]thatrequirememoriesofeventsthathappenedthousandsofdiscretetimestepsbefore,whichisimportantforspeech.In2003,LSTMstartedtobecomecompetitivewithtraditionalspeechrecognizersoncertaintasks.[51]Lateritwascombinedwithconnectionisttemporalclassification(CTC)[52]instacksofLSTMRNNs.[53]In2015,Google'sspeechrecognitionreportedlyexperiencedadramaticperformancejumpof49%throughCTC-trainedLSTM,whichtheymadeavailablethroughGoogleVoiceSearch.[54] In2006,publicationsbyGeoffHinton,RuslanSalakhutdinov,OsinderoandTeh[55][56][57]showedhowamany-layeredfeedforwardneuralnetworkcouldbeeffectivelypre-trainedonelayeratatime,treatingeachlayerinturnasanunsupervisedrestrictedBoltzmannmachine,thenfine-tuningitusingsupervisedbackpropagation.[58]Thepapersreferredtolearningfordeepbeliefnets. Deeplearningispartofstate-of-the-artsystemsinvariousdisciplines,particularlycomputervisionandautomaticspeechrecognition(ASR).ResultsoncommonlyusedevaluationsetssuchasTIMIT(ASR)andMNIST(imageclassification),aswellasarangeoflarge-vocabularyspeechrecognitiontaskshavesteadilyimproved.[59][60]Convolutionalneuralnetworks(CNNs)weresupersededforASRbyCTC[52]forLSTM.[50][54][61][62][63]butaremoresuccessfulincomputervision. Theimpactofdeeplearninginindustrybeganintheearly2000s,whenCNNsalreadyprocessedanestimated10%to20%ofallthecheckswrittenintheUS,accordingtoYannLeCun.[64]Industrialapplicationsofdeeplearningtolarge-scalespeechrecognitionstartedaround2010. The2009NIPSWorkshoponDeepLearningforSpeechRecognitionwasmotivatedbythelimitationsofdeepgenerativemodelsofspeech,andthepossibilitythatgivenmorecapablehardwareandlarge-scaledatasetsthatdeepneuralnets(DNN)mightbecomepractical.Itwasbelievedthatpre-trainingDNNsusinggenerativemodelsofdeepbeliefnets(DBN)wouldovercomethemaindifficultiesofneuralnets.However,itwasdiscoveredthatreplacingpre-trainingwithlargeamountsoftrainingdataforstraightforwardbackpropagationwhenusingDNNswithlarge,context-dependentoutputlayersproducederrorratesdramaticallylowerthanthen-state-of-the-artGaussianmixturemodel(GMM)/HiddenMarkovModel(HMM)andalsothanmore-advancedgenerativemodel-basedsystems.[59]Thenatureoftherecognitionerrorsproducedbythetwotypesofsystemswascharacteristicallydifferent,[65]offeringtechnicalinsightsintohowtointegratedeeplearningintotheexistinghighlyefficient,run-timespeechdecodingsystemdeployedbyallmajorspeechrecognitionsystems.[8][66][67]Analysisaround2009–2010,contrastingtheGMM(andothergenerativespeechmodels)vs.DNNmodels,stimulatedearlyindustrialinvestmentindeeplearningforspeechrecognition,[65]eventuallyleadingtopervasiveanddominantuseinthatindustry.Thatanalysiswasdonewithcomparableperformance(lessthan1.5%inerrorrate)betweendiscriminativeDNNsandgenerativemodels.[59][65][68] In2010,researchersextendeddeeplearningfromTIMITtolargevocabularyspeechrecognition,byadoptinglargeoutputlayersoftheDNNbasedoncontext-dependentHMMstatesconstructedbydecisiontrees.[69][70][71][66] Advancesinhardwarehavedrivenrenewedinterestindeeplearning.In2009,Nvidiawasinvolvedinwhatwascalledthe“bigbang”ofdeeplearning,“asdeep-learningneuralnetworksweretrainedwithNvidiagraphicsprocessingunits(GPUs).”[72]Thatyear,AndrewNgdeterminedthatGPUscouldincreasethespeedofdeep-learningsystemsbyabout100times.[73]Inparticular,GPUsarewell-suitedforthematrix/vectorcomputationsinvolvedinmachinelearning.[74][75][76]GPUsspeeduptrainingalgorithmsbyordersofmagnitude,reducingrunningtimesfromweekstodays.[77][78]Further,specializedhardwareandalgorithmoptimizationscanbeusedforefficientprocessingofdeeplearningmodels.[79] Deeplearningrevolution[edit] Howdeeplearningisasubsetofmachinelearningandhowmachinelearningisasubsetofartificialintelligence(AI) In2012,ateamledbyGeorgeE.Dahlwonthe"MerckMolecularActivityChallenge"usingmulti-taskdeepneuralnetworkstopredictthebiomoleculartargetofonedrug.[80][81]In2014,Hochreiter'sgroupuseddeeplearningtodetectoff-targetandtoxiceffectsofenvironmentalchemicalsinnutrients,householdproductsanddrugsandwonthe"Tox21DataChallenge"ofNIH,FDAandNCATS.[82][83][84] Significantadditionalimpactsinimageorobjectrecognitionwerefeltfrom2011to2012.AlthoughCNNstrainedbybackpropagationhadbeenaroundfordecades,andGPUimplementationsofNNsforyears,includingCNNs,fastimplementationsofCNNsonGPUswereneededtoprogressoncomputervision.[74][76][35][85][12]In2011,thisapproachachievedforthefirsttimesuperhumanperformanceinavisualpatternrecognitioncontest.Alsoin2011,itwontheICDARChinesehandwritingcontest,andinMay2012,itwontheISBIimagesegmentationcontest.[86]Until2011,CNNsdidnotplayamajorroleatcomputervisionconferences,butinJune2012,apaperbyCiresanetal.attheleadingconferenceCVPR[3]showedhowmax-poolingCNNsonGPUcandramaticallyimprovemanyvisionbenchmarkrecords.InOctober2012,asimilarsystembyKrizhevskyetal.[4]wonthelarge-scaleImageNetcompetitionbyasignificantmarginovershallowmachinelearningmethods.InNovember2012,Ciresanetal.'ssystemalsowontheICPRcontestonanalysisoflargemedicalimagesforcancerdetection,andinthefollowingyearalsotheMICCAIGrandChallengeonthesametopic.[87]In2013and2014,theerrorrateontheImageNettaskusingdeeplearningwasfurtherreduced,followingasimilartrendinlarge-scalespeechrecognition. Imageclassificationwasthenextendedtothemorechallengingtaskofgeneratingdescriptions(captions)forimages,oftenasacombinationofCNNsandLSTMs.[88][89][90] SomeresearchersstatethattheOctober2012ImageNetvictoryanchoredthestartofa"deeplearningrevolution"thathastransformedtheAIindustry.[91] InMarch2019,YoshuaBengio,GeoffreyHintonandYannLeCunwereawardedtheTuringAwardforconceptualandengineeringbreakthroughsthathavemadedeepneuralnetworksacriticalcomponentofcomputing. Neuralnetworks[edit] Artificialneuralnetworks[edit] Mainarticle:Artificialneuralnetwork Artificialneuralnetworks(ANNs)orconnectionistsystemsarecomputingsystemsinspiredbythebiologicalneuralnetworksthatconstituteanimalbrains.Suchsystemslearn(progressivelyimprovetheirability)todotasksbyconsideringexamples,generallywithouttask-specificprogramming.Forexample,inimagerecognition,theymightlearntoidentifyimagesthatcontaincatsbyanalyzingexampleimagesthathavebeenmanuallylabeledas"cat"or"nocat"andusingtheanalyticresultstoidentifycatsinotherimages.Theyhavefoundmostuseinapplicationsdifficulttoexpresswithatraditionalcomputeralgorithmusingrule-basedprogramming. AnANNisbasedonacollectionofconnectedunitscalledartificialneurons,(analogoustobiologicalneuronsinabiologicalbrain).Eachconnection(synapse)betweenneuronscantransmitasignaltoanotherneuron.Thereceiving(postsynaptic)neuroncanprocessthesignal(s)andthensignaldownstreamneuronsconnectedtoit.Neuronsmayhavestate,generallyrepresentedbyrealnumbers,typicallybetween0and1.Neuronsandsynapsesmayalsohaveaweightthatvariesaslearningproceeds,whichcanincreaseordecreasethestrengthofthesignalthatitsendsdownstream. Typically,neuronsareorganizedinlayers.Differentlayersmayperformdifferentkindsoftransformationsontheirinputs.Signalstravelfromthefirst(input),tothelast(output)layer,possiblyaftertraversingthelayersmultipletimes. Theoriginalgoaloftheneuralnetworkapproachwastosolveproblemsinthesamewaythatahumanbrainwould.Overtime,attentionfocusedonmatchingspecificmentalabilities,leadingtodeviationsfrombiologysuchasbackpropagation,orpassinginformationinthereversedirectionandadjustingthenetworktoreflectthatinformation. Neuralnetworkshavebeenusedonavarietyoftasks,includingcomputervision,speechrecognition,machinetranslation,socialnetworkfiltering,playingboardandvideogamesandmedicaldiagnosis. Asof2017,neuralnetworkstypicallyhaveafewthousandtoafewmillionunitsandmillionsofconnections.Despitethisnumberbeingseveralorderofmagnitudelessthanthenumberofneuronsonahumanbrain,thesenetworkscanperformmanytasksatalevelbeyondthatofhumans(e.g.,recognizingfaces,playing"Go"[92]). Deepneuralnetworks[edit] Adeepneuralnetwork(DNN)isanartificialneuralnetwork(ANN)withmultiplelayersbetweentheinputandoutputlayers.[9][12]Therearedifferenttypesofneuralnetworksbuttheyalwaysconsistofthesamecomponents:neurons,synapses,weights,biases,andfunctions.[93]ThesecomponentsfunctioningsimilartothehumanbrainsandcanbetrainedlikeanyotherMLalgorithm.[citationneeded] Forexample,aDNNthatistrainedtorecognizedogbreedswillgooverthegivenimageandcalculatetheprobabilitythatthedogintheimageisacertainbreed.Theusercanreviewtheresultsandselectwhichprobabilitiesthenetworkshoulddisplay(aboveacertainthreshold,etc.)andreturntheproposedlabel.Eachmathematicalmanipulationassuchisconsideredalayer,[citationneeded]andcomplexDNNhavemanylayers,hencethename"deep"networks. DNNscanmodelcomplexnon-linearrelationships.DNNarchitecturesgeneratecompositionalmodelswheretheobjectisexpressedasalayeredcompositionofprimitives.[94]Theextralayersenablecompositionoffeaturesfromlowerlayers,potentiallymodelingcomplexdatawithfewerunitsthanasimilarlyperformingshallownetwork.[9]Forinstance,itwasprovedthatsparsemultivariatepolynomialsareexponentiallyeasiertoapproximatewithDNNsthanwithshallownetworks.[95] Deeparchitecturesincludemanyvariantsofafewbasicapproaches.Eacharchitecturehasfoundsuccessinspecificdomains.Itisnotalwayspossibletocomparetheperformanceofmultiplearchitectures,unlesstheyhavebeenevaluatedonthesamedatasets. DNNsaretypicallyfeedforwardnetworksinwhichdataflowsfromtheinputlayertotheoutputlayerwithoutloopingback.Atfirst,theDNNcreatesamapofvirtualneuronsandassignsrandomnumericalvalues,or"weights",toconnectionsbetweenthem.Theweightsandinputsaremultipliedandreturnanoutputbetween0and1.Ifthenetworkdidnotaccuratelyrecognizeaparticularpattern,analgorithmwouldadjusttheweights.[96]Thatwaythealgorithmcanmakecertainparametersmoreinfluential,untilitdeterminesthecorrectmathematicalmanipulationtofullyprocessthedata. Recurrentneuralnetworks(RNNs),inwhichdatacanflowinanydirection,areusedforapplicationssuchaslanguagemodeling.[97][98][99][100][101]Longshort-termmemoryisparticularlyeffectiveforthisuse.[50][102] Convolutionaldeepneuralnetworks(CNNs)areusedincomputervision.[103]CNNsalsohavebeenappliedtoacousticmodelingforautomaticspeechrecognition(ASR).[104] Challenges[edit] AswithANNs,manyissuescanarisewithnaivelytrainedDNNs.Twocommonissuesareoverfittingandcomputationtime. DNNsarepronetooverfittingbecauseoftheaddedlayersofabstraction,whichallowthemtomodelraredependenciesinthetrainingdata.RegularizationmethodssuchasIvakhnenko'sunitpruning[26]orweightdecay( ℓ 2 {\displaystyle\ell_{2}} -regularization)orsparsity( ℓ 1 {\displaystyle\ell_{1}} -regularization)canbeappliedduringtrainingtocombatoverfitting.[105]Alternativelydropoutregularizationrandomlyomitsunitsfromthehiddenlayersduringtraining.Thishelpstoexcluderaredependencies.[106]Finally,datacanbeaugmentedviamethodssuchascroppingandrotatingsuchthatsmallertrainingsetscanbeincreasedinsizetoreducethechancesofoverfitting.[107] DNNsmustconsidermanytrainingparameters,suchasthesize(numberoflayersandnumberofunitsperlayer),thelearningrate,andinitialweights.Sweepingthroughtheparameterspaceforoptimalparametersmaynotbefeasibleduetothecostintimeandcomputationalresources.Varioustricks,suchasbatching(computingthegradientonseveraltrainingexamplesatonceratherthanindividualexamples)[108]speedupcomputation.Largeprocessingcapabilitiesofmany-corearchitectures(suchasGPUsortheIntelXeonPhi)haveproducedsignificantspeedupsintraining,becauseofthesuitabilityofsuchprocessingarchitecturesforthematrixandvectorcomputations.[109][110] Alternatively,engineersmaylookforothertypesofneuralnetworkswithmorestraightforwardandconvergenttrainingalgorithms.CMAC(cerebellarmodelarticulationcontroller)isonesuchkindofneuralnetwork.Itdoesn'trequirelearningratesorrandomizedinitialweightsforCMAC.Thetrainingprocesscanbeguaranteedtoconvergeinonestepwithanewbatchofdata,andthecomputationalcomplexityofthetrainingalgorithmislinearwithrespecttothenumberofneuronsinvolved.[111][112] Hardware[edit] Sincethe2010s,advancesinbothmachinelearningalgorithmsandcomputerhardwarehaveledtomoreefficientmethodsfortrainingdeepneuralnetworksthatcontainmanylayersofnon-linearhiddenunitsandaverylargeoutputlayer.[113]By2019,graphicprocessingunits(GPUs),oftenwithAI-specificenhancements,haddisplacedCPUsasthedominantmethodoftraininglarge-scalecommercialcloudAI.[114]OpenAIestimatedthehardwarecomputationusedinthelargestdeeplearningprojectsfromAlexNet(2012)toAlphaZero(2017),andfounda300,000-foldincreaseintheamountofcomputationrequired,withadoubling-timetrendlineof3.4months.[115][116] Specialelectroniccircuitscalleddeeplearningprocessorsweredesignedtospeedupdeeplearningalgorithms.Deeplearningprocessorsincludeneuralprocessingunits(NPUs)inHuaweicellphones[117]andcloudcomputingserverssuchastensorprocessingunits(TPU)intheGoogleCloudPlatform.[118] Atomicallythinsemiconductorsareconsideredpromisingforenergy-efficientdeeplearninghardwarewherethesamebasicdevicestructureisusedforbothlogicoperationsanddatastorage. In2020,Maregaetal.publishedexperimentswithalarge-areaactivechannelmaterialfordevelopinglogic-in-memorydevicesandcircuitsbasedonfloating-gatefield-effecttransistors(FGFETs).[119] In2021,J.Feldmannetal.proposedanintegratedphotonichardwareacceleratorforparallelconvolutionalprocessing.[120]Theauthorsidentifytwokeyadvantagesofintegratedphotonicsoveritselectroniccounterparts:(1)massivelyparalleldatatransferthroughwavelengthdivisionmultiplexinginconjunctionwithfrequencycombs,and(2)extremelyhighdatamodulationspeeds.[120]Theirsystemcanexecutetrillionsofmultiply-accumulateoperationspersecond,indicatingthepotentialofintegratedphotonicsindata-heavyAIapplications.[120] Applications[edit] Automaticspeechrecognition[edit] Mainarticle:Speechrecognition Large-scaleautomaticspeechrecognitionisthefirstandmostconvincingsuccessfulcaseofdeeplearning.LSTMRNNscanlearn"VeryDeepLearning"tasks[12]thatinvolvemulti-secondintervalscontainingspeecheventsseparatedbythousandsofdiscretetimesteps,whereonetimestepcorrespondstoabout10ms.LSTMwithforgetgates[102]iscompetitivewithtraditionalspeechrecognizersoncertaintasks.[51] Theinitialsuccessinspeechrecognitionwasbasedonsmall-scalerecognitiontasksbasedonTIMIT.Thedatasetcontains630speakersfromeightmajordialectsofAmericanEnglish,whereeachspeakerreads10sentences.[121]Itssmallsizeletsmanyconfigurationsbetried.Moreimportantly,theTIMITtaskconcernsphone-sequencerecognition,which,unlikeword-sequencerecognition,allowsweakphonebigramlanguagemodels.Thisletsthestrengthoftheacousticmodelingaspectsofspeechrecognitionbemoreeasilyanalyzed.Theerrorrateslistedbelow,includingtheseearlyresultsandmeasuredaspercentphoneerrorrates(PER),havebeensummarizedsince1991. Method Percentphoneerrorrate(PER)(%) RandomlyInitializedRNN[122] 26.1 BayesianTriphoneGMM-HMM 25.6 HiddenTrajectory(Generative)Model 24.8 MonophoneRandomlyInitializedDNN 23.4 MonophoneDBN-DNN 22.4 TriphoneGMM-HMMwithBMMITraining 21.7 MonophoneDBN-DNNonfbank 20.7 ConvolutionalDNN[123] 20.0 ConvolutionalDNNw.HeterogeneousPooling 18.7 EnsembleDNN/CNN/RNN[124] 18.3 BidirectionalLSTM 17.8 HierarchicalConvolutionalDeepMaxoutNetwork[125] 16.5 ThedebutofDNNsforspeakerrecognitioninthelate1990sandspeechrecognitionaround2009-2011andofLSTMaround2003–2007,acceleratedprogressineightmajorareas:[8][68][66] Scale-up/outandacceleratedDNNtraininganddecoding Sequencediscriminativetraining Featureprocessingbydeepmodelswithsolidunderstandingoftheunderlyingmechanisms AdaptationofDNNsandrelateddeepmodels Multi-taskandtransferlearningbyDNNsandrelateddeepmodels CNNsandhowtodesignthemtobestexploitdomainknowledgeofspeech RNNanditsrichLSTMvariants Othertypesofdeepmodelsincludingtensor-basedmodelsandintegrateddeepgenerative/discriminativemodels. Allmajorcommercialspeechrecognitionsystems(e.g.,MicrosoftCortana,Xbox,SkypeTranslator,AmazonAlexa,GoogleNow,AppleSiri,BaiduandiFlyTekvoicesearch,andarangeofNuancespeechproducts,etc.)arebasedondeeplearning.[8][126][127] Imagerecognition[edit] Mainarticle:Computervision AcommonevaluationsetforimageclassificationistheMNISTdatabasedataset.MNISTiscomposedofhandwrittendigitsandincludes60,000trainingexamplesand10,000testexamples.AswithTIMIT,itssmallsizeletsuserstestmultipleconfigurations.Acomprehensivelistofresultsonthissetisavailable.[128] Deeplearning-basedimagerecognitionhasbecome"superhuman",producingmoreaccurateresultsthanhumancontestants.Thisfirstoccurredin2011inrecognitionoftrafficsigns,andin2014,withrecognitionofhumanfaces.[129][130] Deeplearning-trainedvehiclesnowinterpret360°cameraviews.[131]AnotherexampleisFacialDysmorphologyNovelAnalysis(FDNA)usedtoanalyzecasesofhumanmalformationconnectedtoalargedatabaseofgeneticsyndromes. Visualartprocessing[edit] Closelyrelatedtotheprogressthathasbeenmadeinimagerecognitionistheincreasingapplicationofdeeplearningtechniquestovariousvisualarttasks.DNNshaveproventhemselvescapable,forexample,of identifyingthestyleperiodofagivenpainting[132][133] NeuralStyleTransfer –capturingthestyleofagivenartworkandapplyingitinavisuallypleasingmannertoanarbitraryphotographorvideo[132][133] generatingstrikingimagerybasedonrandomvisualinputfields.[132][133] Naturallanguageprocessing[edit] Mainarticle:Naturallanguageprocessing Neuralnetworkshavebeenusedforimplementinglanguagemodelssincetheearly2000s.[97]LSTMhelpedtoimprovemachinetranslationandlanguagemodeling.[98][99][100] Otherkeytechniquesinthisfieldarenegativesampling[134]andwordembedding.Wordembedding,suchasword2vec,canbethoughtofasarepresentationallayerinadeeplearningarchitecturethattransformsanatomicwordintoapositionalrepresentationofthewordrelativetootherwordsinthedataset;thepositionisrepresentedasapointinavectorspace.UsingwordembeddingasanRNNinputlayerallowsthenetworktoparsesentencesandphrasesusinganeffectivecompositionalvectorgrammar.Acompositionalvectorgrammarcanbethoughtofasprobabilisticcontextfreegrammar(PCFG)implementedbyanRNN.[135]Recursiveauto-encodersbuiltatopwordembeddingscanassesssentencesimilarityanddetectparaphrasing.[135]Deepneuralarchitecturesprovidethebestresultsforconstituencyparsing,[136]sentimentanalysis,[137]informationretrieval,[138][139]spokenlanguageunderstanding,[140]machinetranslation,[98][141]contextualentitylinking,[141]writingstylerecognition,[142]Textclassificationandothers.[143] Recentdevelopmentsgeneralizewordembeddingtosentenceembedding. GoogleTranslate(GT)usesalargeend-to-endlongshort-termmemory(LSTM)network.[144][145][146][147]GoogleNeuralMachineTranslation(GNMT)usesanexample-basedmachinetranslationmethodinwhichthesystem"learnsfrommillionsofexamples."[145]Ittranslates"wholesentencesatatime,ratherthanpieces.GoogleTranslatesupportsoveronehundredlanguages.[145]Thenetworkencodesthe"semanticsofthesentenceratherthansimplymemorizingphrase-to-phrasetranslations".[145][148]GTusesEnglishasanintermediatebetweenmostlanguagepairs.[148] Drugdiscoveryandtoxicology[edit] Formoreinformation,seeDrugdiscoveryandToxicology. Alargepercentageofcandidatedrugsfailtowinregulatoryapproval.Thesefailuresarecausedbyinsufficientefficacy(on-targeteffect),undesiredinteractions(off-targeteffects),orunanticipatedtoxiceffects.[149][150]Researchhasexploreduseofdeeplearningtopredictthebiomoleculartargets,[80][81]off-targets,andtoxiceffectsofenvironmentalchemicalsinnutrients,householdproductsanddrugs.[82][83][84] AtomNetisadeeplearningsystemforstructure-basedrationaldrugdesign.[151]AtomNetwasusedtopredictnovelcandidatebiomoleculesfordiseasetargetssuchastheEbolavirus[152]andmultiplesclerosis.[153][154] In2017graphneuralnetworkswereusedforthefirsttimetopredictvariouspropertiesofmoleculesinalargetoxicologydataset.[155]In2019,generativeneuralnetworkswereusedtoproducemoleculesthatwerevalidatedexperimentallyallthewayintomice.[156][157] Customerrelationshipmanagement[edit] Mainarticle:Customerrelationshipmanagement Deepreinforcementlearninghasbeenusedtoapproximatethevalueofpossibledirectmarketingactions,definedintermsofRFMvariables.Theestimatedvaluefunctionwasshowntohaveanaturalinterpretationascustomerlifetimevalue.[158] Recommendationsystems[edit] Mainarticle:Recommendersystem Recommendationsystemshaveuseddeeplearningtoextractmeaningfulfeaturesforalatentfactormodelforcontent-basedmusicandjournalrecommendations.[159][160]Multi-viewdeeplearninghasbeenappliedforlearninguserpreferencesfrommultipledomains.[161]Themodelusesahybridcollaborativeandcontent-basedapproachandenhancesrecommendationsinmultipletasks. Bioinformatics[edit] Mainarticle:Bioinformatics AnautoencoderANNwasusedinbioinformatics,topredictgeneontologyannotationsandgene-functionrelationships.[162] Inmedicalinformatics,deeplearningwasusedtopredictsleepqualitybasedondatafromwearables[163]andpredictionsofhealthcomplicationsfromelectronichealthrecorddata.[164] Medicalimageanalysis[edit] Deeplearninghasbeenshowntoproducecompetitiveresultsinmedicalapplicationsuchascancercellclassification,lesiondetection,organsegmentationandimageenhancement.[165][166]Moderndeeplearningtoolsdemonstratethehighaccuracyofdetectingvariousdiseasesandthehelpfulnessoftheirusebyspecialiststoimprovethediagnosisefficiency.[167][168] Mobileadvertising[edit] Findingtheappropriatemobileaudienceformobileadvertisingisalwayschallenging,sincemanydatapointsmustbeconsideredandanalyzedbeforeatargetsegmentcanbecreatedandusedinadservingbyanyadserver.[169]Deeplearninghasbeenusedtointerpretlarge,many-dimensionedadvertisingdatasets.Manydatapointsarecollectedduringtherequest/serve/clickinternetadvertisingcycle.Thisinformationcanformthebasisofmachinelearningtoimproveadselection. Imagerestoration[edit] Deeplearninghasbeensuccessfullyappliedtoinverseproblemssuchasdenoising,super-resolution,inpainting,andfilmcolorization.[170]Theseapplicationsincludelearningmethodssuchas"ShrinkageFieldsforEffectiveImageRestoration"[171]whichtrainsonanimagedataset,andDeepImagePrior,whichtrainsontheimagethatneedsrestoration. Financialfrauddetection[edit] Deeplearningisbeingsuccessfullyappliedtofinancialfrauddetection,taxevasiondetection,[172]andanti-moneylaundering.[173] Military[edit] TheUnitedStatesDepartmentofDefenseapplieddeeplearningtotrainrobotsinnewtasksthroughobservation.[174] Partialdifferentialequations[edit] Physicsinformedneuralnetworkshavebeenusedtosolvepartialdifferentialequationsinbothforwardandinverseproblemsinadatadrivenmanner.[175]OneexampleisthereconstructingfluidflowgovernedbytheNavier-Stokesequations.UsingphysicsinformedneuralnetworksdoesnotrequiretheoftenexpensivemeshgenerationthatconventionalCFDmethodsrelieson.[176][177] ImageReconstruction[edit] Imagereconstructionisthereconstructionoftheunderlyingimagesfromtheimage-relatedmeasurements.Severalworksshowedthebetterandsuperiorperformanceofthedeeplearningmethodscomparedtoanalyticalmethodsforvariousapplications,e.g.,spectralimaging[178]andultrasoundimaging.[179] Relationtohumancognitiveandbraindevelopment[edit] Deeplearningiscloselyrelatedtoaclassoftheoriesofbraindevelopment(specifically,neocorticaldevelopment)proposedbycognitiveneuroscientistsintheearly1990s.[180][181][182][183]Thesedevelopmentaltheorieswereinstantiatedincomputationalmodels,makingthempredecessorsofdeeplearningsystems.Thesedevelopmentalmodelssharethepropertythatvariousproposedlearningdynamicsinthebrain(e.g.,awaveofnervegrowthfactor)supporttheself-organizationsomewhatanalogoustotheneuralnetworksutilizedindeeplearningmodels.Liketheneocortex,neuralnetworksemployahierarchyoflayeredfiltersinwhicheachlayerconsidersinformationfromapriorlayer(ortheoperatingenvironment),andthenpassesitsoutput(andpossiblytheoriginalinput),tootherlayers.Thisprocessyieldsaself-organizingstackoftransducers,well-tunedtotheiroperatingenvironment.A1995descriptionstated,"...theinfant'sbrainseemstoorganizeitselfundertheinfluenceofwavesofso-calledtrophic-factors...differentregionsofthebrainbecomeconnectedsequentially,withonelayeroftissuematuringbeforeanotherandsoonuntilthewholebrainismature."[184] Avarietyofapproacheshavebeenusedtoinvestigatetheplausibilityofdeeplearningmodelsfromaneurobiologicalperspective.Ontheonehand,severalvariantsofthebackpropagationalgorithmhavebeenproposedinordertoincreaseitsprocessingrealism.[185][186]Otherresearchershavearguedthatunsupervisedformsofdeeplearning,suchasthosebasedonhierarchicalgenerativemodelsanddeepbeliefnetworks,maybeclosertobiologicalreality.[187][188]Inthisrespect,generativeneuralnetworkmodelshavebeenrelatedtoneurobiologicalevidenceaboutsampling-basedprocessinginthecerebralcortex.[189] Althoughasystematiccomparisonbetweenthehumanbrainorganizationandtheneuronalencodingindeepnetworkshasnotyetbeenestablished,severalanalogieshavebeenreported.Forexample,thecomputationsperformedbydeeplearningunitscouldbesimilartothoseofactualneurons[190]andneuralpopulations.[191]Similarly,therepresentationsdevelopedbydeeplearningmodelsaresimilartothosemeasuredintheprimatevisualsystem[192]bothatthesingle-unit[193]andatthepopulation[194]levels. Commercialactivity[edit] Facebook'sAIlabperformstaskssuchasautomaticallytagginguploadedpictureswiththenamesofthepeopleinthem.[195] Google'sDeepMindTechnologiesdevelopedasystemcapableoflearninghowtoplayAtarivideogamesusingonlypixelsasdatainput.In2015theydemonstratedtheirAlphaGosystem,whichlearnedthegameofGowellenoughtobeataprofessionalGoplayer.[196][197][198]GoogleTranslateusesaneuralnetworktotranslatebetweenmorethan100languages. In2017,Covariant.aiwaslaunched,whichfocusesonintegratingdeeplearningintofactories.[199] Asof2008,[200]researchersatTheUniversityofTexasatAustin(UT)developedamachinelearningframeworkcalledTraininganAgentManuallyviaEvaluativeReinforcement,orTAMER,whichproposednewmethodsforrobotsorcomputerprogramstolearnhowtoperformtasksbyinteractingwithahumaninstructor.[174]FirstdevelopedasTAMER,anewalgorithmcalledDeepTAMERwaslaterintroducedin2018duringacollaborationbetweenU.S.ArmyResearchLaboratory(ARL)andUTresearchers.DeepTAMERuseddeeplearningtoprovidearobottheabilitytolearnnewtasksthroughobservation.[174]UsingDeepTAMER,arobotlearnedataskwithahumantrainer,watchingvideostreamsorobservingahumanperformataskin-person.Therobotlaterpracticedthetaskwiththehelpofsomecoachingfromthetrainer,whoprovidedfeedbacksuchas“goodjob”and“badjob.”[201] Criticismandcomment[edit] Deeplearninghasattractedbothcriticismandcomment,insomecasesfromoutsidethefieldofcomputerscience. Theory[edit] Seealso:ExplainableAI Amaincriticismconcernsthelackoftheorysurroundingsomemethods.[202]Learninginthemostcommondeeparchitecturesisimplementedusingwell-understoodgradientdescent.However,thetheorysurroundingotheralgorithms,suchascontrastivedivergenceislessclear.[citationneeded](e.g.,Doesitconverge?Ifso,howfast?Whatisitapproximating?)Deeplearningmethodsareoftenlookedatasablackbox,withmostconfirmationsdoneempirically,ratherthantheoretically.[203] OtherspointoutthatdeeplearningshouldbelookedatasasteptowardsrealizingstrongAI,notasanall-encompassingsolution.Despitethepowerofdeeplearningmethods,theystilllackmuchofthefunctionalityneededforrealizingthisgoalentirely.ResearchpsychologistGaryMarcusnoted:"Realistically,deeplearningisonlypartofthelargerchallengeofbuildingintelligentmachines.Suchtechniqueslackwaysofrepresentingcausalrelationships(...)havenoobviouswaysofperforminglogicalinferences,andtheyarealsostillalongwayfromintegratingabstractknowledge,suchasinformationaboutwhatobjectsare,whattheyarefor,andhowtheyaretypicallyused.ThemostpowerfulA.I.systems,likeWatson(...)usetechniqueslikedeeplearningasjustoneelementinaverycomplicatedensembleoftechniques,rangingfromthestatisticaltechniqueofBayesianinferencetodeductivereasoning."[204] Infurtherreferencetotheideathatartisticsensitivitymightbeinherentinrelativelylowlevelsofthecognitivehierarchy,apublishedseriesofgraphicrepresentationsoftheinternalstatesofdeep(20-30layers)neuralnetworksattemptingtodiscernwithinessentiallyrandomdatatheimagesonwhichtheyweretrained[205]demonstrateavisualappeal:theoriginalresearchnoticereceivedwellover1,000comments,andwasthesubjectofwhatwasforatimethemostfrequentlyaccessedarticleonTheGuardian's[206]website. Errors[edit] Somedeeplearningarchitecturesdisplayproblematicbehaviors,[207]suchasconfidentlyclassifyingunrecognizableimagesasbelongingtoafamiliarcategoryofordinaryimages(2014)[208]andmisclassifyingminusculeperturbationsofcorrectlyclassifiedimages(2013).[209]Goertzelhypothesizedthatthesebehaviorsareduetolimitationsintheirinternalrepresentationsandthattheselimitationswouldinhibitintegrationintoheterogeneousmulti-componentartificialgeneralintelligence(AGI)architectures.[207]Theseissuesmaypossiblybeaddressedbydeeplearningarchitecturesthatinternallyformstateshomologoustoimage-grammar[210]decompositionsofobservedentitiesandevents.[207]Learningagrammar(visualorlinguistic)fromtrainingdatawouldbeequivalenttorestrictingthesystemtocommonsensereasoningthatoperatesonconceptsintermsofgrammaticalproductionrulesandisabasicgoalofbothhumanlanguageacquisition[211]andartificialintelligence(AI).[212] Cyberthreat[edit] Asdeeplearningmovesfromthelabintotheworld,researchandexperienceshowthatartificialneuralnetworksarevulnerabletohacksanddeception.[213]Byidentifyingpatternsthatthesesystemsusetofunction,attackerscanmodifyinputstoANNsinsuchawaythattheANNfindsamatchthathumanobserverswouldnotrecognize.Forexample,anattackercanmakesubtlechangestoanimagesuchthattheANNfindsamatcheventhoughtheimagelookstoahumannothinglikethesearchtarget.Suchmanipulationistermedan“adversarialattack.”[214] In2016researchersusedoneANNtodoctorimagesintrialanderrorfashion,identifyanother'sfocalpointsandtherebygenerateimagesthatdeceivedit.Themodifiedimageslookednodifferenttohumaneyes.Anothergroupshowedthatprintoutsofdoctoredimagesthenphotographedsuccessfullytrickedanimageclassificationsystem.[215]Onedefenseisreverseimagesearch,inwhichapossiblefakeimageissubmittedtoasitesuchasTinEyethatcanthenfindotherinstancesofit.Arefinementistosearchusingonlypartsoftheimage,toidentifyimagesfromwhichthatpiecemayhavebeentaken.[216] Anothergroupshowedthatcertainpsychedelicspectaclescouldfoolafacialrecognitionsystemintothinkingordinarypeoplewerecelebrities,potentiallyallowingonepersontoimpersonateanother.In2017researchersaddedstickerstostopsignsandcausedanANNtomisclassifythem.[215] ANNscanhoweverbefurthertrainedtodetectattemptsatdeception,potentiallyleadingattackersanddefendersintoanarmsracesimilartothekindthatalreadydefinesthemalwaredefenseindustry.ANNshavebeentrainedtodefeatANN-basedanti-malwaresoftwarebyrepeatedlyattackingadefensewithmalwarethatwascontinuallyalteredbyageneticalgorithmuntilittrickedtheanti-malwarewhileretainingitsabilitytodamagethetarget.[215] In2016,anothergroupdemonstratedthatcertainsoundscouldmaketheGoogleNowvoicecommandsystemopenaparticularwebaddress,andhypothesizedthatthiscould"serveasasteppingstoneforfurtherattacks(e.g.,openingawebpagehostingdrive-bymalware)."[215] In“datapoisoning,”falsedataiscontinuallysmuggledintoamachinelearningsystem'strainingsettopreventitfromachievingmastery.[215] Relianceonhumanmicrowork[edit] Thissectionneedsadditionalcitationsforverification.Pleasehelpimprovethisarticlebyaddingcitationstoreliablesources.Unsourcedmaterialmaybechallengedandremoved.Findsources: "Deeplearning" – news ·newspapers ·books ·scholar ·JSTOR(April2021)(Learnhowandwhentoremovethistemplatemessage) MostDeepLearningsystemsrelyontrainingandverificationdatathatisgeneratedand/orannotatedbyhumans.Ithasbeenarguedinmediaphilosophythatnotonlylow-paidclickwork(e.g.onAmazonMechanicalTurk)isregularlydeployedforthispurpose,butalsoimplicitformsofhumanmicroworkthatareoftennotrecognizedassuch.[217]ThephilosopherRainerMühlhoffdistinguishesfivetypesof"machiniccapture"ofhumanmicroworktogeneratetrainingdata:(1)gamification(theembeddingofannotationorcomputationtasksintheflowofagame),(2)"trappingandtracking"(e.g.CAPTCHAsforimagerecognitionorclick-trackingonGooglesearchresultspages),(3)exploitationofsocialmotivations(e.g.taggingfacesonFacebooktoobtainlabeledfacialimages),(4)informationmining(e.g.byleveragingquantified-selfdevicessuchasactivitytrackers)and(5)clickwork.[217] Mühlhoffarguesthatinmostcommercialend-userapplicationsofDeepLearningsuchasFacebook'sfacerecognitionsystem,theneedfortrainingdatadoesnotstoponceanANNistrained.Rather,thereisacontinueddemandforhuman-generatedverificationdata toconstantlycalibrateandupdatetheANN.ForthispurposeFacebookintroducedthefeaturethatonceauserisautomaticallyrecognizedinanimage,theyreceiveanotification.Theycanchoosewhetherofnottheyliketobepubliclylabeledontheimage,ortellFacebookthatitisnottheminthepicture.[218]Thisuserinterfaceisamechanismtogenerate"aconstantstreamofverificationdata"[217]tofurthertrainthenetworkinreal-time.AsMühlhoffargues,involvementofhumanuserstogeneratetrainingandverificationdataissotypicalformostcommercialend-userapplicationsofDeepLearningthatsuchsystemsmaybereferredtoas"human-aidedartificialintelligence".[217] Seealso[edit] Applicationsofartificialintelligence Comparisonofdeeplearningsoftware Compressedsensing Differentiableprogramming Echostatenetwork Listofartificialintelligenceprojects Liquidstatemachine Listofdatasetsformachinelearningresearch Reservoircomputing Scalespaceanddeeplearning Sparsecoding References[edit] ^Schulz,Hannes;Behnke,Sven(1November2012)."DeepLearning".KI-KünstlicheIntelligenz.26(4):357–363.doi:10.1007/s13218-012-0198-z.ISSN 1610-1987.S2CID 220523562. ^Bengio,Yoshua;LeCun,Yann;Hinton,Geoffrey(2015)."DeepLearning".Nature.521(7553):436–444.Bibcode:2015Natur.521..436L.doi:10.1038/nature14539.PMID 26017442.S2CID 3074096. ^abCiresan,D.;Meier,U.;Schmidhuber,J.(2012)."Multi-columndeepneuralnetworksforimageclassification".2012IEEEConferenceonComputerVisionandPatternRecognition.pp. 3642–3649.arXiv:1202.2745.doi:10.1109/cvpr.2012.6248110.ISBN 978-1-4673-1228-8.S2CID 2161592. ^abKrizhevsky,Alex;Sutskever,Ilya;Hinton,Geoffry(2012)."ImageNetClassificationwithDeepConvolutionalNeuralNetworks"(PDF).NIPS2012:NeuralInformationProcessingSystems,LakeTahoe,Nevada.Archived(PDF)fromtheoriginalon2017-01-10.Retrieved2017-05-24. ^"Google'sAlphaGoAIwinsthree-matchseriesagainsttheworld'sbestGoplayer".TechCrunch.25May2017.Archivedfromtheoriginalon17June2018.Retrieved17June2018. ^Marblestone,AdamH.;Wayne,Greg;Kording,KonradP.(2016)."TowardanIntegrationofDeepLearningandNeuroscience".FrontiersinComputationalNeuroscience.10:94.arXiv:1606.03813.Bibcode:2016arXiv160603813M.doi:10.3389/fncom.2016.00094.PMC 5021692.PMID 27683554.S2CID 1994856. ^Bengio,Yoshua;Lee,Dong-Hyun;Bornschein,Jorg;Mesnard,Thomas;Lin,Zhouhan(13February2015)."TowardsBiologicallyPlausibleDeepLearning".arXiv:1502.04156[cs.LG]. ^abcdefDeng,L.;Yu,D.(2014)."DeepLearning:MethodsandApplications"(PDF).FoundationsandTrendsinSignalProcessing.7(3–4):1–199.doi:10.1561/2000000039.Archived(PDF)fromtheoriginalon2016-03-14.Retrieved2014-10-18. ^abcdBengio,Yoshua(2009)."LearningDeepArchitecturesforAI"(PDF).FoundationsandTrendsinMachineLearning.2(1):1–127.CiteSeerX 10.1.1.701.9550.doi:10.1561/2200000006.Archivedfromtheoriginal(PDF)on4March2016.Retrieved3September2015. ^abcdeBengio,Y.;Courville,A.;Vincent,P.(2013)."RepresentationLearning:AReviewandNewPerspectives".IEEETransactionsonPatternAnalysisandMachineIntelligence.35(8):1798–1828.arXiv:1206.5538.doi:10.1109/tpami.2013.50.PMID 23787338.S2CID 393948. ^LeCun,Yann;Bengio,Yoshua;Hinton,Geoffrey(28May2015)."Deeplearning".Nature.521(7553):436–444.Bibcode:2015Natur.521..436L.doi:10.1038/nature14539.PMID 26017442.S2CID 3074096. ^abcdefgSchmidhuber,J.(2015)."DeepLearninginNeuralNetworks:AnOverview".NeuralNetworks.61:85–117.arXiv:1404.7828.doi:10.1016/j.neunet.2014.09.003.PMID 25462637.S2CID 11715509. ^Shigeki,Sugiyama(12April2019).HumanBehaviorandAnotherKindinConsciousness:EmergingResearchandOpportunities:EmergingResearchandOpportunities.IGIGlobal.ISBN 978-1-5225-8218-2. ^Bengio,Yoshua;Lamblin,Pascal;Popovici,Dan;Larochelle,Hugo(2007).Greedylayer-wisetrainingofdeepnetworks(PDF).Advancesinneuralinformationprocessingsystems.pp. 153–160.Archived(PDF)fromtheoriginalon2019-10-20.Retrieved2019-10-06. ^abHinton,G.E.(2009)."Deepbeliefnetworks".Scholarpedia.4(5):5947.Bibcode:2009SchpJ...4.5947H.doi:10.4249/scholarpedia.5947. ^abcCybenko(1989)."Approximationsbysuperpositionsofsigmoidalfunctions"(PDF).MathematicsofControl,Signals,andSystems.2(4):303–314.doi:10.1007/bf02551274.S2CID 3958369.Archivedfromtheoriginal(PDF)on10October2015. ^abcHornik,Kurt(1991)."ApproximationCapabilitiesofMultilayerFeedforwardNetworks".NeuralNetworks.4(2):251–257.doi:10.1016/0893-6080(91)90009-t. ^abHaykin,SimonS.(1999).NeuralNetworks:AComprehensiveFoundation.PrenticeHall.ISBN 978-0-13-273350-2. ^abHassoun,MohamadH.(1995).FundamentalsofArtificialNeuralNetworks.MITPress.p. 48.ISBN 978-0-262-08239-6. ^abLu,Z.,Pu,H.,Wang,F.,Hu,Z.,&Wang,L.(2017).TheExpressivePowerofNeuralNetworks:AViewfromtheWidthArchived2019-02-13attheWaybackMachine.NeuralInformationProcessingSystems,6231-6239. ^abcdMurphy,KevinP.(24August2012).MachineLearning:AProbabilisticPerspective.MITPress.ISBN 978-0-262-01802-9. ^Sonoda,Sho;Murata,Noboru(2017)."Neuralnetworkwithunboundedactivationfunctionsisuniversalapproximator".AppliedandComputationalHarmonicAnalysis.43(2):233–268.arXiv:1505.03654.doi:10.1016/j.acha.2015.12.005.S2CID 12149203. ^Bishop,ChristopherM.(2006).PatternRecognitionandMachineLearning(PDF).Springer.ISBN 978-0-387-31073-2.Archived(PDF)fromtheoriginalon2017-01-11.Retrieved2017-08-06. ^Tappert,CharlesC.(2019)."WhoIstheFatherofDeepLearning?".2019InternationalConferenceonComputationalScienceandComputationalIntelligence(CSCI).IEEE.pp. 343–348.doi:10.1109/CSCI49370.2019.00067.ISBN 978-1-7281-5584-5.S2CID 216043128.Retrieved31May2021. ^Ivakhnenko,A.G.;Lapa,V.G.(1967).CyberneticsandForecastingTechniques.AmericanElsevierPublishingCo.ISBN 978-0-444-00020-0. ^abIvakhnenko,Alexey(1971)."Polynomialtheoryofcomplexsystems"(PDF).IEEETransactionsonSystems,ManandCybernetics.SMC-1(4):364–378.doi:10.1109/TSMC.1971.4308320.Archived(PDF)fromtheoriginalon2017-08-29.Retrieved2019-11-05. ^Fukushima,K.(1980)."Neocognitron:Aself-organizingneuralnetworkmodelforamechanismofpatternrecognitionunaffectedbyshiftinposition".Biol.Cybern.36(4):193–202.doi:10.1007/bf00344251.PMID 7370364.S2CID 206775608. ^RinaDechter(1986).Learningwhilesearchinginconstraint-satisfactionproblems.UniversityofCalifornia,ComputerScienceDepartment,CognitiveSystemsLaboratory.OnlineArchived2016-04-19attheWaybackMachine ^IgorAizenberg,NaumN.Aizenberg,JoosP.L.Vandewalle(2000).Multi-ValuedandUniversalBinaryNeurons:Theory,LearningandApplications.SpringerScience&BusinessMedia. ^Co-evolvingrecurrentneuronslearndeepmemoryPOMDPs.Proc.GECCO,Washington,D.C.,pp.1795-1802,ACMPress,NewYork,NY,USA,2005. ^SeppoLinnainmaa(1970).TherepresentationofthecumulativeroundingerrorofanalgorithmasaTaylorexpansionofthelocalroundingerrors.Master'sThesis(inFinnish),Univ.Helsinki,6-7. ^Griewank,Andreas(2012)."WhoInventedtheReverseModeofDifferentiation?"(PDF).DocumentaMathematica(ExtraVolumeISMP):389–400.Archivedfromtheoriginal(PDF)on21July2017.Retrieved11June2017. ^Werbos,P.(1974)."BeyondRegression:NewToolsforPredictionandAnalysisintheBehavioralSciences".HarvardUniversity.Retrieved12June2017. ^Werbos,Paul(1982)."Applicationsofadvancesinnonlinearsensitivityanalysis"(PDF).Systemmodelingandoptimization.Springer.pp. 762–770. ^abLeCunetal.,"BackpropagationAppliedtoHandwrittenZipCodeRecognition,"NeuralComputation,1,pp.541–551,1989. ^deCarvalho,AndreC.L.F.;Fairhurst,MikeC.;Bisset,David(8August1994)."AnintegratedBooleanneuralnetworkforpatternclassification".PatternRecognitionLetters.15(8):807–813.Bibcode:1994PaReL..15..807D.doi:10.1016/0167-8655(94)90009-4. ^Hinton,GeoffreyE.;Dayan,Peter;Frey,BrendanJ.;Neal,Radford(26May1995)."Thewake-sleepalgorithmforunsupervisedneuralnetworks".Science.268(5214):1158–1161.Bibcode:1995Sci...268.1158H.doi:10.1126/science.7761831.PMID 7761831. ^abS.Hochreiter.,"UntersuchungenzudynamischenneuronalenNetzenArchived2015-03-06attheWaybackMachine,"Diplomathesis.Institutf.Informatik,TechnischeUniv.Munich.Advisor:J.Schmidhuber,1991. ^Hochreiter,S.;et al.(15January2001)."Gradientflowinrecurrentnets:thedifficultyoflearninglong-termdependencies".InKolen,JohnF.;Kremer,StefanC.(eds.).AFieldGuidetoDynamicalRecurrentNetworks.JohnWiley&Sons.ISBN 978-0-7803-5369-5. ^Behnke,Sven(2003).HierarchicalNeuralNetworksforImageInterpretation.LectureNotesinComputerScience.Vol. 2766.Springer.doi:10.1007/b11963.ISBN 3-540-40722-7.S2CID 1304548. ^Morgan,Nelson;Bourlard,Hervé;Renals,Steve;Cohen,Michael;Franco,Horacio(1August1993)."Hybridneuralnetwork/hiddenmarkovmodelsystemsforcontinuousspeechrecognition".InternationalJournalofPatternRecognitionandArtificialIntelligence.07(4):899–916.doi:10.1142/s0218001493000455.ISSN 0218-0014. ^Robinson,T.(1992)."Areal-timerecurrenterrorpropagationnetworkwordrecognitionsystem".ICASSP.Icassp'92:617–620.ISBN 9780780305328.Archivedfromtheoriginalon2021-05-09.Retrieved2017-06-12. ^Waibel,A.;Hanazawa,T.;Hinton,G.;Shikano,K.;Lang,K.J.(March1989)."Phonemerecognitionusingtime-delayneuralnetworks"(PDF).IEEETransactionsonAcoustics,Speech,andSignalProcessing.37(3):328–339.doi:10.1109/29.21701.hdl:10338.dmlcz/135496.ISSN 0096-3518.Archived(PDF)fromtheoriginalon2021-04-27.Retrieved2019-09-24. ^Baker,J.;Deng,Li;Glass,Jim;Khudanpur,S.;Lee,C.-H.;Morgan,N.;O'Shaughnessy,D.(2009)."ResearchDevelopmentsandDirectionsinSpeechRecognitionandUnderstanding,Part1".IEEESignalProcessingMagazine.26(3):75–80.Bibcode:2009ISPM...26...75B.doi:10.1109/msp.2009.932166.hdl:1721.1/51891.S2CID 357467. ^Bengio,Y.(1991)."ArtificialNeuralNetworksandtheirApplicationtoSpeech/SequenceRecognition".McGillUniversityPh.D.thesis.Archivedfromtheoriginalon2021-05-09.Retrieved2017-06-12. ^Deng,L.;Hassanein,K.;Elmasry,M.(1994)."Analysisofcorrelationstructureforaneuralpredictivemodelwithapplicationstospeechrecognition".NeuralNetworks.7(2):331–339.doi:10.1016/0893-6080(94)90027-2. ^Doddington,G.;Przybocki,M.;Martin,A.;Reynolds,D.(2000)."TheNISTspeakerrecognitionevaluation±Overview,methodology,systems,results,perspective".SpeechCommunication.31(2):225–254.doi:10.1016/S0167-6393(99)00080-1. ^abHeck,L.;Konig,Y.;Sonmez,M.;Weintraub,M.(2000)."RobustnesstoTelephoneHandsetDistortioninSpeakerRecognitionbyDiscriminativeFeatureDesign".SpeechCommunication.31(2):181–192.doi:10.1016/s0167-6393(99)00077-1. ^"AcousticModelingwithDeepNeuralNetworksUsingRawTimeSignalforLVCSR(PDFDownloadAvailable)".ResearchGate.Archivedfromtheoriginalon9May2021.Retrieved14June2017. ^abcHochreiter,Sepp;Schmidhuber,Jürgen(1November1997)."LongShort-TermMemory".NeuralComputation.9(8):1735–1780.doi:10.1162/neco.1997.9.8.1735.ISSN 0899-7667.PMID 9377276.S2CID 1915014. ^abGraves,Alex;Eck,Douglas;Beringer,Nicole;Schmidhuber,Jürgen(2003)."BiologicallyPlausibleSpeechRecognitionwithLSTMNeuralNets"(PDF).1stIntl.WorkshoponBiologicallyInspiredApproachestoAdvancedInformationTechnology,Bio-ADIT2004,Lausanne,Switzerland.pp. 175–184.Archived(PDF)fromtheoriginalon2021-05-09.Retrieved2016-04-09. ^abGraves,Alex;Fernández,Santiago;Gomez,Faustino(2006)."Connectionisttemporalclassification:Labellingunsegmentedsequencedatawithrecurrentneuralnetworks".ProceedingsoftheInternationalConferenceonMachineLearning,ICML2006:369–376.CiteSeerX 10.1.1.75.6306. ^SantiagoFernandez,AlexGraves,andJürgenSchmidhuber(2007).AnapplicationofrecurrentneuralnetworkstodiscriminativekeywordspottingArchived2018-11-18attheWaybackMachine.ProceedingsofICANN(2),pp.220–229. ^abSak,Haşim;Senior,Andrew;Rao,Kanishka;Beaufays,Françoise;Schalkwyk,Johan(September2015)."Googlevoicesearch:fasterandmoreaccurate".Archivedfromtheoriginalon2016-03-09.Retrieved2016-04-09. ^Hinton,GeoffreyE.(1October2007)."Learningmultiplelayersofrepresentation".TrendsinCognitiveSciences.11(10):428–434.doi:10.1016/j.tics.2007.09.004.ISSN 1364-6613.PMID 17921042.S2CID 15066318.Archivedfromtheoriginalon11October2013.Retrieved12June2017. ^Hinton,G.E.;Osindero,S.;Teh,Y.W.(2006)."AFastLearningAlgorithmforDeepBeliefNets"(PDF).NeuralComputation.18(7):1527–1554.doi:10.1162/neco.2006.18.7.1527.PMID 16764513.S2CID 2309950.Archived(PDF)fromtheoriginalon2015-12-23.Retrieved2011-07-20. ^Bengio,Yoshua(2012)."Practicalrecommendationsforgradient-basedtrainingofdeeparchitectures".arXiv:1206.5533[cs.LG]. ^G.E.Hinton.,"LearningmultiplelayersofrepresentationArchived2018-05-22attheWaybackMachine,"TrendsinCognitiveSciences,11,pp.428–434,2007. ^abcHinton,G.;Deng,L.;Yu,D.;Dahl,G.;Mohamed,A.;Jaitly,N.;Senior,A.;Vanhoucke,V.;Nguyen,P.;Sainath,T.;Kingsbury,B.(2012)."DeepNeuralNetworksforAcousticModelinginSpeechRecognition:TheSharedViewsofFourResearchGroups".IEEESignalProcessingMagazine.29(6):82–97.Bibcode:2012ISPM...29...82H.doi:10.1109/msp.2012.2205597.S2CID 206485943. ^Singh,Premjeet;Saha,Goutam;Sahidullah,Md(2021)."Non-linearfrequencywarpingusingconstant-Qtransformationforspeechemotionrecognition".2021InternationalConferenceonComputerCommunicationandInformatics(ICCCI).pp. 1–4.arXiv:2102.04029.doi:10.1109/ICCCI50826.2021.9402569.ISBN 978-1-7281-5875-4.S2CID 231846518. ^Sak,Hasim;Senior,Andrew;Beaufays,Francoise(2014)."LongShort-TermMemoryrecurrentneuralnetworkarchitecturesforlargescaleacousticmodeling"(PDF).Archivedfromtheoriginal(PDF)on24April2018. ^Li,Xiangang;Wu,Xihong(2014)."ConstructingLongShort-TermMemorybasedDeepRecurrentNeuralNetworksforLargeVocabularySpeechRecognition".arXiv:1410.4281[cs.CL]. ^Zen,Heiga;Sak,Hasim(2015)."UnidirectionalLongShort-TermMemoryRecurrentNeuralNetworkwithRecurrentOutputLayerforLow-LatencySpeechSynthesis"(PDF).Google.com.ICASSP.pp. 4470–4474.Archived(PDF)fromtheoriginalon2021-05-09.Retrieved2017-06-13. ^YannLeCun(2016).SlidesonDeepLearningOnlineArchived2016-04-23attheWaybackMachine ^abcDeng,L.;Hinton,G.;Kingsbury,B.(2013)."Newtypesofdeepneuralnetworklearningforspeechrecognitionandrelatedapplications:Anoverview(ICASSP)"(PDF).Archived(PDF)fromtheoriginalon2017-09-26.Retrieved2017-06-12.{{citejournal}}:Citejournalrequires|journal=(help) ^abcYu,D.;Deng,L.(2014).AutomaticSpeechRecognition:ADeepLearningApproach(Publisher:Springer).ISBN 978-1-4471-5779-3. ^"DengreceivesprestigiousIEEETechnicalAchievementAward-MicrosoftResearch".MicrosoftResearch.3December2015.Archivedfromtheoriginalon16March2018.Retrieved16March2018. ^abLi,Deng(September2014)."Keynotetalk:'AchievementsandChallengesofDeepLearning-FromSpeechAnalysisandRecognitionToLanguageandMultimodalProcessing'".Interspeech.Archivedfromtheoriginalon2017-09-26.Retrieved2017-06-12. ^Yu,D.;Deng,L.(2010)."RolesofPre-TrainingandFine-TuninginContext-DependentDBN-HMMsforReal-WorldSpeechRecognition".NIPSWorkshoponDeepLearningandUnsupervisedFeatureLearning.Archivedfromtheoriginalon2017-10-12.Retrieved2017-06-14. ^Seide,F.;Li,G.;Yu,D.(2011)."Conversationalspeechtranscriptionusingcontext-dependentdeepneuralnetworks".Interspeech:437–440.doi:10.21437/Interspeech.2011-169.Archivedfromtheoriginalon2017-10-12.Retrieved2017-06-14. ^Deng,Li;Li,Jinyu;Huang,Jui-Ting;Yao,Kaisheng;Yu,Dong;Seide,Frank;Seltzer,Mike;Zweig,Geoff;He,Xiaodong(1May2013)."RecentAdvancesinDeepLearningforSpeechResearchatMicrosoft".MicrosoftResearch.Archivedfromtheoriginalon12October2017.Retrieved14June2017. ^"NvidiaCEObetsbigondeeplearningandVR".VentureBeat.5April2016.Archivedfromtheoriginalon25November2020.Retrieved21April2017. ^"Fromnotworkingtoneuralnetworking".TheEconomist.Archivedfromtheoriginalon2016-12-31.Retrieved2017-08-26. ^abOh,K.-S.;Jung,K.(2004)."GPUimplementationofneuralnetworks".PatternRecognition.37(6):1311–1314.Bibcode:2004PatRe..37.1311O.doi:10.1016/j.patcog.2004.01.013. ^"ASurveyofTechniquesforOptimizingDeepLearningonGPUsArchived2021-05-09attheWaybackMachine",S.MittalandS.Vaishay,JournalofSystemsArchitecture,2019 ^abChellapilla,Kumar;Puri,Sidd;Simard,Patrice(2006),Highperformanceconvolutionalneuralnetworksfordocumentprocessing,archivedfromtheoriginalon2020-05-18,retrieved2021-02-14 ^Cireşan,DanClaudiu;Meier,Ueli;Gambardella,LucaMaria;Schmidhuber,Jürgen(21September2010)."Deep,Big,SimpleNeuralNetsforHandwrittenDigitRecognition".NeuralComputation.22(12):3207–3220.arXiv:1003.0358.doi:10.1162/neco_a_00052.ISSN 0899-7667.PMID 20858131.S2CID 1918673. ^Raina,Rajat;Madhavan,Anand;Ng,AndrewY.(2009)."Large-scaleDeepUnsupervisedLearningUsingGraphicsProcessors".Proceedingsofthe26thAnnualInternationalConferenceonMachineLearning.ICML'09.NewYork,NY,USA:ACM:873–880.CiteSeerX 10.1.1.154.372.doi:10.1145/1553374.1553486.ISBN 9781605585161.S2CID 392458. ^Sze,Vivienne;Chen,Yu-Hsin;Yang,Tien-Ju;Emer,Joel(2017)."EfficientProcessingofDeepNeuralNetworks:ATutorialandSurvey".arXiv:1703.09039[cs.CV]. ^ab"MerckMolecularActivityChallenge".kaggle.com.Archivedfromtheoriginalon2020-07-16.Retrieved2020-07-16. ^ab"Multi-taskNeuralNetworksforQSARPredictions|DataScienceAssociation".www.datascienceassn.org.Archivedfromtheoriginalon30April2017.Retrieved14June2017. ^ab"Toxicologyinthe21stcenturyDataChallenge" ^ab"NCATSAnnouncesTox21DataChallengeWinners".Archivedfromtheoriginalon2015-09-08.Retrieved2015-03-05. ^ab"Archivedcopy".Archivedfromtheoriginalon28February2015.Retrieved5March2015.{{citeweb}}:CS1maint:archivedcopyastitle(link) ^Ciresan,D.C.;Meier,U.;Masci,J.;Gambardella,L.M.;Schmidhuber,J.(2011)."Flexible,HighPerformanceConvolutionalNeuralNetworksforImageClassification"(PDF).InternationalJointConferenceonArtificialIntelligence.doi:10.5591/978-1-57735-516-8/ijcai11-210.Archived(PDF)fromtheoriginalon2014-09-29.Retrieved2017-06-13. ^Ciresan,Dan;Giusti,Alessandro;Gambardella,LucaM.;Schmidhuber,Juergen(2012).Pereira,F.;Burges,C.J.C.;Bottou,L.;Weinberger,K.Q.(eds.).AdvancesinNeuralInformationProcessingSystems25(PDF).CurranAssociates,Inc.pp. 2843–2851.Archived(PDF)fromtheoriginalon2017-08-09.Retrieved2017-06-13. ^Ciresan,D.;Giusti,A.;Gambardella,L.M.;Schmidhuber,J.(2013)."MitosisDetectioninBreastCancerHistologyImagesusingDeepNeuralNetworks".ProceedingsMICCAI.LectureNotesinComputerScience.7908(Pt2):411–418.doi:10.1007/978-3-642-40763-5_51.ISBN 978-3-642-38708-1.PMID 24579167. ^Vinyals,Oriol;Toshev,Alexander;Bengio,Samy;Erhan,Dumitru(2014)."ShowandTell:ANeuralImageCaptionGenerator".arXiv:1411.4555[cs.CV].. ^Fang,Hao;Gupta,Saurabh;Iandola,Forrest;Srivastava,Rupesh;Deng,Li;Dollár,Piotr;Gao,Jianfeng;He,Xiaodong;Mitchell,Margaret;Platt,JohnC;LawrenceZitnick,C;Zweig,Geoffrey(2014)."FromCaptionstoVisualConceptsandBack".arXiv:1411.4952[cs.CV].. ^Kiros,Ryan;Salakhutdinov,Ruslan;Zemel,RichardS(2014)."UnifyingVisual-SemanticEmbeddingswithMultimodalNeuralLanguageModels".arXiv:1411.2539[cs.LG].. ^"WhyDeepLearningIsSuddenlyChangingYourLife".Fortune.2016.Archivedfromtheoriginalon14April2018.Retrieved13April2018. ^Silver,David;Huang,Aja;Maddison,ChrisJ.;Guez,Arthur;Sifre,Laurent;Driessche,Georgevanden;Schrittwieser,Julian;Antonoglou,Ioannis;Panneershelvam,Veda(January2016)."MasteringthegameofGowithdeepneuralnetworksandtreesearch".Nature.529(7587):484–489.Bibcode:2016Natur.529..484S.doi:10.1038/nature16961.ISSN 1476-4687.PMID 26819042.S2CID 515925. ^AGuidetoDeepLearningandNeuralNetworks,archivedfromtheoriginalon2020-11-02,retrieved2020-11-16 ^Szegedy,Christian;Toshev,Alexander;Erhan,Dumitru(2013)."Deepneuralnetworksforobjectdetection".AdvancesinNeuralInformationProcessingSystems:2553–2561.Archivedfromtheoriginalon2017-06-29.Retrieved2017-06-13. ^Rolnick,David;Tegmark,Max(2018)."Thepowerofdeepernetworksforexpressingnaturalfunctions".InternationalConferenceonLearningRepresentations.ICLR2018.Archivedfromtheoriginalon2021-01-07.Retrieved2021-01-05. ^Hof,RobertD."IsArtificialIntelligenceFinallyComingintoItsOwn?".MITTechnologyReview.Archivedfromtheoriginalon31March2019.Retrieved10July2018. ^abGers,FelixA.;Schmidhuber,Jürgen(2001)."LSTMRecurrentNetworksLearnSimpleContextFreeandContextSensitiveLanguages".IEEETransactionsonNeuralNetworks.12(6):1333–1340.doi:10.1109/72.963769.PMID 18249962.Archivedfromtheoriginalon2020-01-26.Retrieved2020-02-25. ^abcSutskever,L.;Vinyals,O.;Le,Q.(2014)."SequencetoSequenceLearningwithNeuralNetworks"(PDF).Proc.NIPS.arXiv:1409.3215.Bibcode:2014arXiv1409.3215S.Archived(PDF)fromtheoriginalon2021-05-09.Retrieved2017-06-13. ^abJozefowicz,Rafal;Vinyals,Oriol;Schuster,Mike;Shazeer,Noam;Wu,Yonghui(2016)."ExploringtheLimitsofLanguageModeling".arXiv:1602.02410[cs.CL]. ^abGillick,Dan;Brunk,Cliff;Vinyals,Oriol;Subramanya,Amarnag(2015)."MultilingualLanguageProcessingfromBytes".arXiv:1512.00103[cs.CL]. ^Mikolov,T.;et al.(2010)."Recurrentneuralnetworkbasedlanguagemodel"(PDF).Interspeech:1045–1048.doi:10.21437/Interspeech.2010-343.Archived(PDF)fromtheoriginalon2017-05-16.Retrieved2017-06-13. ^ab"LearningPreciseTimingwithLSTMRecurrentNetworks(PDFDownloadAvailable)".ResearchGate.Archivedfromtheoriginalon9May2021.Retrieved13June2017. ^LeCun,Y.;et al.(1998)."Gradient-basedlearningappliedtodocumentrecognition".ProceedingsoftheIEEE.86(11):2278–2324.doi:10.1109/5.726791.S2CID 14542261. ^Sainath,TaraN.;Mohamed,Abdel-Rahman;Kingsbury,Brian;Ramabhadran,Bhuvana(2013)."DeepconvolutionalneuralnetworksforLVCSR".2013IEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing.pp. 8614–8618.doi:10.1109/icassp.2013.6639347.ISBN 978-1-4799-0356-6.S2CID 13816461. ^Bengio,Yoshua;Boulanger-Lewandowski,Nicolas;Pascanu,Razvan(2013)."Advancesinoptimizingrecurrentnetworks".2013IEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing.pp. 8624–8628.arXiv:1212.0901.CiteSeerX 10.1.1.752.9151.doi:10.1109/icassp.2013.6639349.ISBN 978-1-4799-0356-6.S2CID 12485056. ^Dahl,G.;et al.(2013)."ImprovingDNNsforLVCSRusingrectifiedlinearunitsanddropout"(PDF).ICASSP.Archived(PDF)fromtheoriginalon2017-08-12.Retrieved2017-06-13. ^"DataAugmentation-deeplearning.ai|Coursera".Coursera.Archivedfromtheoriginalon1December2017.Retrieved30November2017. ^Hinton,G.E.(2010)."APracticalGuidetoTrainingRestrictedBoltzmannMachines".Tech.Rep.UTMLTR2010-003.Archivedfromtheoriginalon2021-05-09.Retrieved2017-06-13. ^You,Yang;Buluç,Aydın;Demmel,James(November2017)."ScalingdeeplearningonGPUandknightslandingclusters".ProceedingsoftheInternationalConferenceforHighPerformanceComputing,Networking,StorageandAnalysison-SC'17.SC'17,ACM.pp. 1–12.doi:10.1145/3126908.3126912.ISBN 9781450351140.S2CID 8869270.Archivedfromtheoriginalon29July2020.Retrieved5March2018. ^Viebke,André;Memeti,Suejb;Pllana,Sabri;Abraham,Ajith(2019)."CHAOS:aparallelizationschemefortrainingconvolutionalneuralnetworksonIntelXeonPhi".TheJournalofSupercomputing.75:197–227.arXiv:1702.07908.Bibcode:2017arXiv170207908V.doi:10.1007/s11227-017-1994-x.S2CID 14135321. ^TingQin,etal."AlearningalgorithmofCMACbasedonRLS."NeuralProcessingLetters19.1(2004):49-61. ^TingQin,etal."ContinuousCMAC-QRLSanditssystolicarrayArchived2018-11-18attheWaybackMachine."NeuralProcessingLetters22.1(2005):1-16. ^Research,AI(23October2015)."DeepNeuralNetworksforAcousticModelinginSpeechRecognition".airesearch.com.Archivedfromtheoriginalon1February2016.Retrieved23October2015. ^"GPUsContinuetoDominatetheAIAcceleratorMarketforNow".InformationWeek.December2019.Archivedfromtheoriginalon10June2020.Retrieved11June2020. ^Ray,Tiernan(2019)."AIischangingtheentirenatureofcomputation".ZDNet.Archivedfromtheoriginalon25May2020.Retrieved11June2020. ^"AIandCompute".OpenAI.16May2018.Archivedfromtheoriginalon17June2020.Retrieved11June2020. ^"HUAWEIRevealstheFutureofMobileAIatIFA". ^P,JouppiNorman;YoungCliff;PatilNishant;PattersonDavid;AgrawalGaurav;BajwaRaminder;BatesSarah;BhatiaSuresh;BodenNan;BorchersAl;BoyleRick(2017-06-24)."In-DatacenterPerformanceAnalysisofaTensorProcessingUnit".ACMSIGARCHComputerArchitectureNews.45(2):1–12.doi:10.1145/3140659.3080246. ^Marega,GuilhermeMigliato;Zhao,Yanfei;Avsar,Ahmet;Wang,Zhenyu;Tripati,Mukesh;Radenovic,Aleksandra;Kis,Anras(2020)."Logic-in-memorybasedonanatomicallythinsemiconductor".Nature.587(2):72–77.Bibcode:2020Natur.587...72M.doi:10.1038/s41586-020-2861-0.PMC 7116757.PMID 33149289. ^abcFeldmann,J.;Youngblood,N.;Karpov,M.;et al.(2021)."Parallelconvolutionalprocessingusinganintegratedphotonictensor".Nature.589(2):52–58.arXiv:2002.00281.doi:10.1038/s41586-020-03070-1.PMID 33408373.S2CID 211010976. ^TIMITAcoustic-PhoneticContinuousSpeechCorpusLinguisticDataConsortium,Philadelphia. ^Robinson,Tony(30September1991)."SeveralImprovementstoaRecurrentErrorPropagationNetworkPhoneRecognitionSystem".CambridgeUniversityEngineeringDepartmentTechnicalReport.CUED/F-INFENG/TR82.doi:10.13140/RG.2.2.15418.90567. ^Abdel-Hamid,O.;et al.(2014)."ConvolutionalNeuralNetworksforSpeechRecognition".IEEE/ACMTransactionsonAudio,Speech,andLanguageProcessing.22(10):1533–1545.doi:10.1109/taslp.2014.2339736.S2CID 206602362.Archivedfromtheoriginalon2020-09-22.Retrieved2018-04-20. ^Deng,L.;Platt,J.(2014)."EnsembleDeepLearningforSpeechRecognition".Proc.Interspeech.S2CID 15641618. ^Tóth,Laszló(2015)."PhoneRecognitionwithHierarchicalConvolutionalDeepMaxoutNetworks"(PDF).EURASIPJournalonAudio,Speech,andMusicProcessing.2015.doi:10.1186/s13636-015-0068-3.S2CID 217950236.Archived(PDF)fromtheoriginalon2020-09-24.Retrieved2019-04-01. ^McMillan,Robert(17December2014)."HowSkypeUsedAItoBuildItsAmazingNewLanguageTranslator|WIRED".Wired.Archivedfromtheoriginalon8June2017.Retrieved14June2017. ^Hannun,Awni;Case,Carl;Casper,Jared;Catanzaro,Bryan;Diamos,Greg;Elsen,Erich;Prenger,Ryan;Satheesh,Sanjeev;Sengupta,Shubho;Coates,Adam;Ng,AndrewY(2014)."DeepSpeech:Scalingupend-to-endspeechrecognition".arXiv:1412.5567[cs.CL]. ^"MNISThandwrittendigitdatabase,YannLeCun,CorinnaCortesandChrisBurges".yann.lecun.com.Archivedfromtheoriginalon2014-01-13.Retrieved2014-01-28. ^Cireşan,Dan;Meier,Ueli;Masci,Jonathan;Schmidhuber,Jürgen(August2012)."Multi-columndeepneuralnetworkfortrafficsignclassification".NeuralNetworks.SelectedPapersfromIJCNN2011.32:333–338.CiteSeerX 10.1.1.226.8219.doi:10.1016/j.neunet.2012.02.023.PMID 22386783. ^https://arxiv.org/abs/1404.3840SurpassingHumanLevelFaceRecognitionbyChaochaoLu&XiaoouTangRetrieved3/17/22 ^NvidiaDemosaCarComputerTrainedwith"DeepLearning"(6January2015),DavidTalbot,MITTechnologyReview ^abcG.W.Smith;FredericFolLeymarie(10April2017)."TheMachineasArtist:AnIntroduction".Arts.6(4):5.doi:10.3390/arts6020005. ^abcBlaiseAgüerayArcas(29September2017)."ArtintheAgeofMachineIntelligence".Arts.6(4):18.doi:10.3390/arts6040018. ^Goldberg,Yoav;Levy,Omar(2014)."word2vecExplained:DerivingMikolovetal.'sNegative-SamplingWord-EmbeddingMethod".arXiv:1402.3722[cs.CL]. ^abSocher,Richard;Manning,Christopher."DeepLearningforNLP"(PDF).Archived(PDF)fromtheoriginalon6July2014.Retrieved26October2014. ^Socher,Richard;Bauer,John;Manning,Christopher;Ng,Andrew(2013)."ParsingWithCompositionalVectorGrammars"(PDF).ProceedingsoftheACL2013Conference.Archived(PDF)fromtheoriginalon2014-11-27.Retrieved2014-09-03. ^Socher,Richard(2013)."RecursiveDeepModelsforSemanticCompositionalityOveraSentimentTreebank"(PDF).Archived(PDF)fromtheoriginalon2016-12-28.Retrieved2014-09-03.{{citejournal}}:Citejournalrequires|journal=(help) ^Shen,Yelong;He,Xiaodong;Gao,Jianfeng;Deng,Li;Mesnil,Gregoire(1November2014)."ALatentSemanticModelwithConvolutional-PoolingStructureforInformationRetrieval".MicrosoftResearch.Archivedfromtheoriginalon27October2017.Retrieved14June2017. ^Huang,Po-Sen;He,Xiaodong;Gao,Jianfeng;Deng,Li;Acero,Alex;Heck,Larry(1October2013)."LearningDeepStructuredSemanticModelsforWebSearchusingClickthroughData".MicrosoftResearch.Archivedfromtheoriginalon27October2017.Retrieved14June2017. ^Mesnil,G.;Dauphin,Y.;Yao,K.;Bengio,Y.;Deng,L.;Hakkani-Tur,D.;He,X.;Heck,L.;Tur,G.;Yu,D.;Zweig,G.(2015)."Usingrecurrentneuralnetworksforslotfillinginspokenlanguageunderstanding".IEEETransactionsonAudio,Speech,andLanguageProcessing.23(3):530–539.doi:10.1109/taslp.2014.2383614.S2CID 1317136. ^abGao,Jianfeng;He,Xiaodong;Yih,ScottWen-tau;Deng,Li(1June2014)."LearningContinuousPhraseRepresentationsforTranslationModeling".MicrosoftResearch.Archivedfromtheoriginalon27October2017.Retrieved14June2017. ^Brocardo,MarceloLuiz;Traore,Issa;Woungang,Isaac;Obaidat,MohammadS.(2017)."Authorshipverificationusingdeepbeliefnetworksystems".InternationalJournalofCommunicationSystems.30(12):e3259.doi:10.1002/dac.3259.S2CID 40745740. ^"DeepLearningforNaturalLanguageProcessing:TheoryandPractice(CIKM2014Tutorial)-MicrosoftResearch".MicrosoftResearch.Archivedfromtheoriginalon13March2017.Retrieved14June2017. ^Turovsky,Barak(15November2016)."Foundintranslation:Moreaccurate,fluentsentencesinGoogleTranslate".TheKeywordGoogleBlog.Archivedfromtheoriginalon7April2017.Retrieved23March2017. ^abcdSchuster,Mike;Johnson,Melvin;Thorat,Nikhil(22November2016)."Zero-ShotTranslationwithGoogle'sMultilingualNeuralMachineTranslationSystem".GoogleResearchBlog.Archivedfromtheoriginalon10July2017.Retrieved23March2017. ^Wu,Yonghui;Schuster,Mike;Chen,Zhifeng;Le,QuocV;Norouzi,Mohammad;Macherey,Wolfgang;Krikun,Maxim;Cao,Yuan;Gao,Qin;Macherey,Klaus;Klingner,Jeff;Shah,Apurva;Johnson,Melvin;Liu,Xiaobing;Kaiser,Łukasz;Gouws,Stephan;Kato,Yoshikiyo;Kudo,Taku;Kazawa,Hideto;Stevens,Keith;Kurian,George;Patil,Nishant;Wang,Wei;Young,Cliff;Smith,Jason;Riesa,Jason;Rudnick,Alex;Vinyals,Oriol;Corrado,Greg;et al.(2016)."Google'sNeuralMachineTranslationSystem:BridgingtheGapbetweenHumanandMachineTranslation".arXiv:1609.08144[cs.CL]. ^Metz,Cade(27September2016)."AnInfusionofAIMakesGoogleTranslateMorePowerfulThanEver".Wired.Archivedfromtheoriginalon8November2020.Retrieved12October2017. ^abBoitet,Christian;Blanchon,Hervé;Seligman,Mark;Bellynck,Valérie(2010)."MTonandfortheWeb"(PDF).Archivedfromtheoriginal(PDF)on29March2017.Retrieved1December2016. ^Arrowsmith,J;Miller,P(2013)."Trialwatch:PhaseIIandphaseIIIattritionrates2011-2012".NatureReviewsDrugDiscovery.12(8):569.doi:10.1038/nrd4090.PMID 23903212.S2CID 20246434. ^Verbist,B;Klambauer,G;Vervoort,L;Talloen,W;TheQstar,Consortium;Shkedy,Z;Thas,O;Bender,A;Göhlmann,H.W.;Hochreiter,S(2015)."Usingtranscriptomicstoguideleadoptimizationindrugdiscoveryprojects:LessonslearnedfromtheQSTARproject".DrugDiscoveryToday.20(5):505–513.doi:10.1016/j.drudis.2014.12.014.PMID 25582842. ^Wallach,Izhar;Dzamba,Michael;Heifets,Abraham(9October2015)."AtomNet:ADeepConvolutionalNeuralNetworkforBioactivityPredictioninStructure-basedDrugDiscovery".arXiv:1510.02855[cs.LG]. ^"Torontostartuphasafasterwaytodiscovereffectivemedicines".TheGlobeandMail.Archivedfromtheoriginalon20October2015.Retrieved9November2015. ^"StartupHarnessesSupercomputerstoSeekCures".KQEDFutureofYou.Archivedfromtheoriginalon24December2015.Retrieved9November2015. ^"Torontostartuphasafasterwaytodiscovereffectivemedicines".TheGlobeandMail.Archivedfromtheoriginalon2015-12-25.Retrieved2017-08-26. ^Gilmer,Justin;Schoenholz,SamuelS.;Riley,PatrickF.;Vinyals,Oriol;Dahl,GeorgeE.(2017-06-12)."NeuralMessagePassingforQuantumChemistry".arXiv:1704.01212[cs.LG]. ^Zhavoronkov,Alex(2019)."DeeplearningenablesrapididentificationofpotentDDR1kinaseinhibitors".NatureBiotechnology.37(9):1038–1040.doi:10.1038/s41587-019-0224-x.PMID 31477924.S2CID 201716327. ^Gregory,Barber."AMoleculeDesignedByAIExhibits'Druglike'Qualities".Wired.Archivedfromtheoriginalon2020-04-30.Retrieved2019-09-05. ^Tkachenko,Yegor(8April2015)."AutonomousCRMControlviaCLVApproximationwithDeepReinforcementLearninginDiscreteandContinuousActionSpace".arXiv:1504.01840[cs.LG]. ^vandenOord,Aaron;Dieleman,Sander;Schrauwen,Benjamin(2013).Burges,C.J.C.;Bottou,L.;Welling,M.;Ghahramani,Z.;Weinberger,K.Q.(eds.).AdvancesinNeuralInformationProcessingSystems26(PDF).CurranAssociates,Inc.pp. 2643–2651.Archived(PDF)fromtheoriginalon2017-05-16.Retrieved2017-06-14. ^Feng,X.Y.;Zhang,H.;Ren,Y.J.;Shang,P.H.;Zhu,Y.;Liang,Y.C.;Guan,R.C.;Xu,D.(2019)."TheDeepLearning–BasedRecommenderSystem"Pubmender"forChoosingaBiomedicalPublicationVenue:DevelopmentandValidationStudy".JournalofMedicalInternetResearch.21(5):e12957.doi:10.2196/12957.PMC 6555124.PMID 31127715. ^Elkahky,AliMamdouh;Song,Yang;He,Xiaodong(1May2015)."AMulti-ViewDeepLearningApproachforCrossDomainUserModelinginRecommendationSystems".MicrosoftResearch.Archivedfromtheoriginalon25January2018.Retrieved14June2017. ^Chicco,Davide;Sadowski,Peter;Baldi,Pierre(1January2014).DeepAutoencoderNeuralNetworksforGeneOntologyAnnotationPredictions.Proceedingsofthe5thACMConferenceonBioinformatics,ComputationalBiology,andHealthInformatics-BCB'14.ACM.pp. 533–540.doi:10.1145/2649387.2649442.hdl:11311/964622.ISBN 9781450328944.S2CID 207217210.Archivedfromtheoriginalon9May2021.Retrieved23November2015. ^Sathyanarayana,Aarti(1January2016)."SleepQualityPredictionFromWearableDataUsingDeepLearning".JMIRmHealthanduHealth.4(4):e125.doi:10.2196/mhealth.6562.PMC 5116102.PMID 27815231.S2CID 3821594. ^Choi,Edward;Schuetz,Andy;Stewart,WalterF.;Sun,Jimeng(13August2016)."Usingrecurrentneuralnetworkmodelsforearlydetectionofheartfailureonset".JournaloftheAmericanMedicalInformaticsAssociation.24(2):361–370.doi:10.1093/jamia/ocw112.ISSN 1067-5027.PMC 5391725.PMID 27521897. ^Litjens,Geert;Kooi,Thijs;Bejnordi,BabakEhteshami;Setio,ArnaudArindraAdiyoso;Ciompi,Francesco;Ghafoorian,Mohsen;vanderLaak,JeroenA.W.M.;vanGinneken,Bram;Sánchez,ClaraI.(December2017)."Asurveyondeeplearninginmedicalimageanalysis".MedicalImageAnalysis.42:60–88.arXiv:1702.05747.Bibcode:2017arXiv170205747L.doi:10.1016/j.media.2017.07.005.PMID 28778026.S2CID 2088679. ^Forslid,Gustav;Wieslander,Hakan;Bengtsson,Ewert;Wahlby,Carolina;Hirsch,Jan-Michael;Stark,ChristinaRunow;Sadanandan,SajithKecheril(2017)."DeepConvolutionalNeuralNetworksforDetectingCellularChangesDuetoMalignancy".2017IEEEInternationalConferenceonComputerVisionWorkshops(ICCVW).pp. 82–89.doi:10.1109/ICCVW.2017.18.ISBN 9781538610343.S2CID 4728736.Archivedfromtheoriginalon2021-05-09.Retrieved2019-11-12. ^Dong,Xin;Zhou,Yizhao;Wang,Lantian;Peng,Jingfeng;Lou,Yanbo;Fan,Yiqun(2020)."LiverCancerDetectionUsingHybridizedFullyConvolutionalNeuralNetworkBasedonDeepLearningFramework".IEEEAccess.8:129889–129898.doi:10.1109/ACCESS.2020.3006362.ISSN 2169-3536.S2CID 220733699. ^Lyakhov,PavelAlekseevich;Lyakhova,UlyanaAlekseevna;Nagornov,NikolayNikolaevich(2022-04-03)."SystemfortheRecognizingofPigmentedSkinLesionswithFusionandAnalysisofHeterogeneousDataBasedonaMultimodalNeuralNetwork".Cancers.14(7):1819.doi:10.3390/cancers14071819.ISSN 2072-6694.PMC 8997449.PMID 35406591. ^De,Shaunak;Maity,Abhishek;Goel,Vritti;Shitole,Sanjay;Bhattacharya,Avik(2017)."Predictingthepopularityofinstagrampostsforalifestylemagazineusingdeeplearning".20172ndInternationalConferenceonCommunicationSystems,ComputingandITApplications(CSCITA).pp. 174–177.doi:10.1109/CSCITA.2017.8066548.ISBN 978-1-5090-4381-1.S2CID 35350962. ^"ColorizingandRestoringOldImageswithDeepLearning".FloydHubBlog.13November2018.Archivedfromtheoriginalon11October2019.Retrieved11October2019. ^Schmidt,Uwe;Roth,Stefan.ShrinkageFieldsforEffectiveImageRestoration(PDF).ComputerVisionandPatternRecognition(CVPR),2014IEEEConferenceon.Archived(PDF)fromtheoriginalon2018-01-02.Retrieved2018-01-01. ^Kleanthous,Christos;Chatzis,Sotirios(2020)."GatedMixtureVariationalAutoencodersforValueAddedTaxauditcaseselection".Knowledge-BasedSystems.188:105048.doi:10.1016/j.knosys.2019.105048.S2CID 204092079. ^Czech,Tomasz."Deeplearning:thenextfrontierformoneylaunderingdetection".GlobalBankingandFinanceReview.Archivedfromtheoriginalon2018-11-16.Retrieved2018-07-15. ^abc"Armyresearchersdevelopnewalgorithmstotrainrobots".EurekAlert!.Archivedfromtheoriginalon28August2018.Retrieved29August2018. ^Raissi,M.;Perdikaris,P.;Karniadakis,G.E.(2019-02-01)."Physics-informedneuralnetworks:Adeeplearningframeworkforsolvingforwardandinverseproblemsinvolvingnonlinearpartialdifferentialequations".JournalofComputationalPhysics.378:686–707.Bibcode:2019JCoPh.378..686R.doi:10.1016/j.jcp.2018.10.045.ISSN 0021-9991.OSTI 1595805.S2CID 57379996. ^Mao,Zhiping;Jagtap,AmeyaD.;Karniadakis,GeorgeEm(2020-03-01)."Physics-informedneuralnetworksforhigh-speedflows".ComputerMethodsinAppliedMechanicsandEngineering.360:112789.Bibcode:2020CMAME.360k2789M.doi:10.1016/j.cma.2019.112789.ISSN 0045-7825.S2CID 212755458. ^Raissi,Maziar;Yazdani,Alireza;Karniadakis,GeorgeEm(2020-02-28)."Hiddenfluidmechanics:Learningvelocityandpressurefieldsfromflowvisualizations".Science.367(6481):1026–1030.Bibcode:2020Sci...367.1026R.doi:10.1126/science.aaw4741.PMC 7219083.PMID 32001523. ^Oktem,FigenS.;Kar,OğuzhanFatih;Bezek,CanDeniz;Kamalabadi,Farzad(2021)."High-ResolutionMulti-SpectralImagingWithDiffractiveLensesandLearnedReconstruction".IEEETransactionsonComputationalImaging.7:489–504.arXiv:2008.11625.doi:10.1109/TCI.2021.3075349.ISSN 2333-9403.S2CID 235340737. ^Bernhardt,Melanie;Vishnevskiy,Valery;Rau,Richard;Goksel,Orcun(December2020)."TrainingVariationalNetworksWithMultidomainSimulations:Speed-of-SoundImageReconstruction".IEEETransactionsonUltrasonics,Ferroelectrics,andFrequencyControl.67(12):2584–2594.arXiv:2006.14395.doi:10.1109/TUFFC.2020.3010186.ISSN 1525-8955.PMID 32746211.S2CID 220055785. ^Utgoff,P.E.;Stracuzzi,D.J.(2002)."Many-layeredlearning".NeuralComputation.14(10):2497–2529.doi:10.1162/08997660260293319.PMID 12396572.S2CID 1119517. ^Elman,JeffreyL.(1998).RethinkingInnateness:AConnectionistPerspectiveonDevelopment.MITPress.ISBN 978-0-262-55030-7. ^Shrager,J.;Johnson,MH(1996)."Dynamicplasticityinfluencestheemergenceoffunctioninasimplecorticalarray".NeuralNetworks.9(7):1119–1129.doi:10.1016/0893-6080(96)00033-0.PMID 12662587. ^Quartz,SR;Sejnowski,TJ(1997)."Theneuralbasisofcognitivedevelopment:Aconstructivistmanifesto".BehavioralandBrainSciences.20(4):537–556.CiteSeerX 10.1.1.41.7854.doi:10.1017/s0140525x97001581.PMID 10097006.S2CID 5818342. ^S.Blakeslee.,"Inbrain'searlygrowth,timetablemaybecritical,"TheNewYorkTimes,ScienceSection,pp.B5–B6,1995. ^Mazzoni,P.;Andersen,R.A.;Jordan,M.I.(15May1991)."Amorebiologicallyplausiblelearningruleforneuralnetworks".ProceedingsoftheNationalAcademyofSciences.88(10):4433–4437.Bibcode:1991PNAS...88.4433M.doi:10.1073/pnas.88.10.4433.ISSN 0027-8424.PMC 51674.PMID 1903542. ^O'Reilly,RandallC.(1July1996)."BiologicallyPlausibleError-DrivenLearningUsingLocalActivationDifferences:TheGeneralizedRecirculationAlgorithm".NeuralComputation.8(5):895–938.doi:10.1162/neco.1996.8.5.895.ISSN 0899-7667.S2CID 2376781. ^Testolin,Alberto;Zorzi,Marco(2016)."ProbabilisticModelsandGenerativeNeuralNetworks:TowardsanUnifiedFrameworkforModelingNormalandImpairedNeurocognitiveFunctions".FrontiersinComputationalNeuroscience.10:73.doi:10.3389/fncom.2016.00073.ISSN 1662-5188.PMC 4943066.PMID 27468262.S2CID 9868901. ^Testolin,Alberto;Stoianov,Ivilin;Zorzi,Marco(September2017)."Letterperceptionemergesfromunsuperviseddeeplearningandrecyclingofnaturalimagefeatures".NatureHumanBehaviour.1(9):657–664.doi:10.1038/s41562-017-0186-2.ISSN 2397-3374.PMID 31024135.S2CID 24504018. ^Buesing,Lars;Bill,Johannes;Nessler,Bernhard;Maass,Wolfgang(3November2011)."NeuralDynamicsasSampling:AModelforStochasticComputationinRecurrentNetworksofSpikingNeurons".PLOSComputationalBiology.7(11):e1002211.Bibcode:2011PLSCB...7E2211B.doi:10.1371/journal.pcbi.1002211.ISSN 1553-7358.PMC 3207943.PMID 22096452.S2CID 7504633. ^Cash,S.;Yuste,R.(February1999)."LinearsummationofexcitatoryinputsbyCA1pyramidalneurons".Neuron.22(2):383–394.doi:10.1016/s0896-6273(00)81098-3.ISSN 0896-6273.PMID 10069343.S2CID 14663106. ^Olshausen,B;Field,D(1August2004)."Sparsecodingofsensoryinputs".CurrentOpinioninNeurobiology.14(4):481–487.doi:10.1016/j.conb.2004.07.007.ISSN 0959-4388.PMID 15321069.S2CID 16560320. ^Yamins,DanielLK;DiCarlo,JamesJ(March2016)."Usinggoal-drivendeeplearningmodelstounderstandsensorycortex".NatureNeuroscience.19(3):356–365.doi:10.1038/nn.4244.ISSN 1546-1726.PMID 26906502.S2CID 16970545. ^Zorzi,Marco;Testolin,Alberto(19February2018)."Anemergentistperspectiveontheoriginofnumbersense".Phil.Trans.R.Soc.B.373(1740):20170043.doi:10.1098/rstb.2017.0043.ISSN 0962-8436.PMC 5784047.PMID 29292348.S2CID 39281431. ^Güçlü,Umut;vanGerven,MarcelA.J.(8July2015)."DeepNeuralNetworksRevealaGradientintheComplexityofNeuralRepresentationsacrosstheVentralStream".JournalofNeuroscience.35(27):10005–10014.arXiv:1411.6422.doi:10.1523/jneurosci.5023-14.2015.PMC 6605414.PMID 26157000. ^Metz,C.(12December2013)."Facebook's'DeepLearning'GuruRevealstheFutureofAI".Wired.Archivedfromtheoriginalon28March2014.Retrieved26August2017. ^Gibney,Elizabeth(2016)."GoogleAIalgorithmmastersancientgameofGo".Nature.529(7587):445–446.Bibcode:2016Natur.529..445G.doi:10.1038/529445a.PMID 26819021.S2CID 4460235.Archivedfromtheoriginalon2May2019.Retrieved30January2016. ^Silver,David;Huang,Aja;Maddison,ChrisJ.;Guez,Arthur;Sifre,Laurent;Driessche,Georgevanden;Schrittwieser,Julian;Antonoglou,Ioannis;Panneershelvam,Veda;Lanctot,Marc;Dieleman,Sander;Grewe,Dominik;Nham,John;Kalchbrenner,Nal;Sutskever,Ilya;Lillicrap,Timothy;Leach,Madeleine;Kavukcuoglu,Koray;Graepel,Thore;Hassabis,Demis(28January2016)."MasteringthegameofGowithdeepneuralnetworksandtreesearch".Nature.529(7587):484–489.Bibcode:2016Natur.529..484S.doi:10.1038/nature16961.ISSN 0028-0836.PMID 26819042.S2CID 515925. ^"AGoogleDeepMindAlgorithmUsesDeepLearningandMoretoMastertheGameofGo|MITTechnologyReview".MITTechnologyReview.Retrieved30January2016. ^Metz,Cade(6November2017)."A.I.ResearchersLeaveElonMuskLabtoBeginRoboticsStart-Up".TheNewYorkTimes.Archivedfromtheoriginalon7July2019.Retrieved5July2019. ^BradleyKnox,W.;Stone,Peter(2008)."TAMER:TraininganAgentManuallyviaEvaluativeReinforcement".20087thIEEEInternationalConferenceonDevelopmentandLearning:292–297.doi:10.1109/devlrn.2008.4640845.ISBN 978-1-4244-2661-4.S2CID 5613334. ^"TalktotheAlgorithms:AIBecomesaFasterLearner".governmentciomedia.com.Archivedfromtheoriginalon28August2018.Retrieved29August2018. ^Marcus,Gary(14January2018)."Indefenseofskepticismaboutdeeplearning".GaryMarcus.Archivedfromtheoriginalon12October2018.Retrieved11October2018. ^Knight,Will(14March2017)."DARPAisfundingprojectsthatwilltrytoopenupAI'sblackboxes".MITTechnologyReview.Archivedfromtheoriginalon4November2019.Retrieved2November2017. ^Marcus,Gary(November25,2012)."Is"DeepLearning"aRevolutioninArtificialIntelligence?".TheNewYorker.Archivedfromtheoriginalon2009-11-27.Retrieved2017-06-14. ^AlexanderMordvintsev;ChristopherOlah;MikeTyka(17June2015)."Inceptionism:GoingDeeperintoNeuralNetworks".GoogleResearchBlog.Archivedfromtheoriginalon3July2015.Retrieved20June2015. ^AlexHern(18June2015)."Yes,androidsdodreamofelectricsheep".TheGuardian.Archivedfromtheoriginalon19June2015.Retrieved20June2015. ^abcGoertzel,Ben(2015)."ArethereDeepReasonsUnderlyingthePathologiesofToday'sDeepLearningAlgorithms?"(PDF).Archived(PDF)fromtheoriginalon2015-05-13.Retrieved2015-05-10. ^Nguyen,Anh;Yosinski,Jason;Clune,Jeff(2014)."DeepNeuralNetworksareEasilyFooled:HighConfidencePredictionsforUnrecognizableImages".arXiv:1412.1897[cs.CV]. ^Szegedy,Christian;Zaremba,Wojciech;Sutskever,Ilya;Bruna,Joan;Erhan,Dumitru;Goodfellow,Ian;Fergus,Rob(2013)."Intriguingpropertiesofneuralnetworks".arXiv:1312.6199[cs.CV]. ^Zhu,S.C.;Mumford,D.(2006)."Astochasticgrammarofimages".Found.TrendsComput.Graph.Vis.2(4):259–362.CiteSeerX 10.1.1.681.2190.doi:10.1561/0600000018. ^Miller,G.A.,andN.Chomsky."Patternconception."PaperforConferenceonpatterndetection,UniversityofMichigan.1957. ^Eisner,Jason."DeepLearningofRecursiveStructure:GrammarInduction".Archivedfromtheoriginalon2017-12-30.Retrieved2015-05-10. ^"HackersHaveAlreadyStartedtoWeaponizeArtificialIntelligence".Gizmodo.Archivedfromtheoriginalon11October2019.Retrieved11October2019. ^"HowhackerscanforceAItomakedumbmistakes".TheDailyDot.18June2018.Archivedfromtheoriginalon11October2019.Retrieved11October2019. ^abcde"AIIsEasytoFool—WhyThatNeedstoChange".SingularityHub.10October2017.Archivedfromtheoriginalon11October2017.Retrieved11October2017. ^Gibney,Elizabeth(2017)."Thescientistwhospotsfakevideos".Nature.doi:10.1038/nature.2017.22784.Archivedfromtheoriginalon2017-10-10.Retrieved2017-10-11. ^abcdMühlhoff,Rainer(6November2019)."Human-aidedartificialintelligence:Or,howtorunlargecomputationsinhumanbrains?Towardamediasociologyofmachinelearning".NewMedia&Society.22(10):1868–1884.doi:10.1177/1461444819885334.ISSN 1461-4448.S2CID 209363848. ^"FacebookCanNowFindYourFace,EvenWhenIt'sNotTagged".Wired.ISSN 1059-1028.Archivedfromtheoriginalon10August2019.Retrieved22November2019. Furtherreading[edit] Goodfellow,Ian;Bengio,Yoshua;Courville,Aaron(2016).DeepLearning.MITPress.ISBN 978-0-26203561-3.Archivedfromtheoriginalon2016-04-16.Retrieved2021-05-09,introductorytextbook.{{citebook}}:CS1maint:postscript(link) vteDifferentiablecomputingGeneral Differentiableprogramming Informationgeometry Statisticalmanifold NeuralTuringmachine Differentiableneuralcomputer Automaticdifferentiation Neuromorphicengineering Cabletheory Patternrecognition Tensorcalculus Computationallearningtheory Concepts Gradientdescent SGD Clustering Regression Overfitting Adversary Attention Convolution Lossfunctions Backpropagation Normalization Activation Softmax Sigmoid Rectifier Regularization Datasets Augmentation Diffusion Autoregression Programminglanguages Python Julia Application Machinelearning Artificialneuralnetwork Deeplearning Scientificcomputing ArtificialIntelligence Hardware IPU TPU VPU Memristor SpiNNaker Softwarelibrary TensorFlow PyTorch Keras Theano ImplementationAudio–visual AlexNet WaveNet Humanimagesynthesis HWR OCR Speechsynthesis Speechrecognition Facialrecognition AlphaFold DALL-E Verbal Word2vec Transformer BERT LaMDA NMT ProjectDebater IBMWatson GPT-2 GPT-3 Decisional AlphaGo AlphaZero Q-learning SARSA OpenAIFive Self-drivingcar MuZero Actionselection Robotcontrol People AlexGraves IanGoodfellow YoshuaBengio GeoffreyHinton YannLeCun AndrewNg DemisHassabis DavidSilver Fei-FeiLi Organizations DeepMind OpenAI MITCSAIL Mila GoogleBrain MetaAI Architectures Recurrentneuralnetwork(RNN) Longshort-termmemory(LSTM) Gatedrecurrentunit(GRU) Echostatenetwork Multilayerperceptron(MLP) Autoencoder Variationalautoencoder(VAE) Generativeadversarialnetwork(GAN) Portals Computerprogramming Technology Category Artificialneuralnetworks Machinelearning Retrievedfrom"https://en.wikipedia.org/w/index.php?title=Deep_learning&oldid=1098889587" Categories:DeeplearningArtificialneuralnetworksArtificialintelligenceEmergingtechnologiesHiddencategories:WebarchivetemplatewaybacklinksCS1:longvolumevalueCS1errors:missingperiodicalCS1maint:archivedcopyastitleArticleswithshortdescriptionShortdescriptionmatchesWikidataAllarticleswithunsourcedstatementsArticleswithunsourcedstatementsfromNovember2020ArticleswithunsourcedstatementsfromMarch2022ArticleswithunsourcedstatementsfromJuly2016ArticlesneedingadditionalreferencesfromApril2021AllarticlesneedingadditionalreferencesCS1maint:postscriptArticlespronetospamfromJune2015 Navigationmenu Personaltools NotloggedinTalkContributionsCreateaccountLogin Namespaces ArticleTalk English Views ReadEditViewhistory More Search Navigation MainpageContentsCurrenteventsRandomarticleAboutWikipediaContactusDonate Contribute HelpLearntoeditCommunityportalRecentchangesUploadfile Tools WhatlinkshereRelatedchangesUploadfileSpecialpagesPermanentlinkPageinformationCitethispageWikidataitem Print/export DownloadasPDFPrintableversion Inotherprojects WikimediaCommons Languages العربيةবাংলাBân-lâm-gúБългарскиCatalàČeštinaDanskDeutschEestiEspañolEuskaraفارسیFrançais한국어ՀայերենBahasaIndonesiaItalianoעבריתമലയാളംBahasaMelayuМонголNederlands日本語NorskbokmålOccitanPolskiPortuguêsRomânăРусскийShqipSimpleEnglishSlovenščinaکوردیСрпски/srpskiSuomiSvenskaதமிழ்ไทยTürkçeУкраїнськаTiếngViệt粵語中文 Editlinks