數(shù)據(jù)挖掘ppt

1、智能信息處理32學(xué)時(shí),費(fèi)高雷fgl@uestc.edu.cn電子科技大學(xué) 通信與信息工程學(xué)院,教師信息,費(fèi)高雷電話：61830209郵箱：fgl@uestc.edu.cn地址：科研樓B325研究方向：網(wǎng)絡(luò)層析成像反演理論與方法復(fù)雜多維信息處理,提綱,引言數(shù)據(jù)挖掘概念及必要性數(shù)據(jù)挖掘的主要任務(wù)案列分析,Teaching Material,數(shù)據(jù)挖掘：概念與技術(shù)（原書第3版）[美]Jiawei Han ，

2、等 著范明 ，孟小峰 譯 機(jī)械工業(yè)出版社 特點(diǎn)：大量的圖解、實(shí)例和練習(xí)參考：第2版,Reference Books,考核方式,成績構(gòu)成：期末70%、期中5%、平時(shí)25%（平時(shí) ＝ 實(shí)驗(yàn)15% + 考勤10%）考試方式：期中隨堂開卷、期末開卷實(shí)驗(yàn)成績：結(jié)果分析50%、報(bào)告50%課堂講述、課后完成,Content,1、引言 2、認(rèn)識(shí)數(shù)據(jù) 3、數(shù)據(jù)預(yù)處理 4、數(shù)據(jù)倉庫與聯(lián)機(jī)分析

3、處理（自學(xué)）5、數(shù)據(jù)立方體技術(shù) （自學(xué)）6、挖掘關(guān)聯(lián)規(guī)則（重點(diǎn)）,7、高級(jí)模式挖掘（自學(xué)）8、分類：基本概念（重點(diǎn)）9、分類：高級(jí)方法（自學(xué)）10、聚類分析：基本概念和方法（重點(diǎn)）11、高級(jí)聚類分析（自學(xué)）12、離群點(diǎn)檢測(cè)13、智能信息處理技術(shù)前沿,數(shù)據(jù)挖掘技術(shù)不斷進(jìn)步、新方法不斷涌現(xiàn)，本課程注重基本概念和基本方法,開課必要性：工業(yè)界的角度,,,開課必要性：學(xué)術(shù)的角度,,提綱,引言數(shù)據(jù)挖掘概念及必要性數(shù)據(jù)挖掘的主要任

4、務(wù)案列分析,Trends leading to Data Flood,More data is generated:Bank, telecom, other business transactions ...Scientific Data: astronomy, biology, etcWeb, text, and e-commerce More data is captured:Storage technology fas

5、ter and cheaperDBMS capable of handling bigger DB,舉例,Europe's Very Long Baseline Interferometry (VLBI) has 16 telescopes, each of which produces 1 Gigabit/second of astronomical data over a 25-day observation sessio

6、n storage and analysis a big problemWalmart reported to have 24 Tera-byte DB AT&T handles billions of calls per daydata cannot be stored -- analysis is done on the fly,Growth Trends,Moore’s lawComputer Speed dou

7、bles every 18 monthsStorage lawtotal storage doubles every 9 monthsConsequencevery little data will ever be looked at by a humanKnowledge Discovery is NEEDED to make sense and use of data.,智能信息處理,為適應(yīng)信息時(shí)代信息處理的要求，當(dāng)前信息

8、處理技術(shù)向智能化發(fā)展，廣泛的模擬人的智能來處理各種信息智能信息處理是計(jì)算機(jī)科學(xué)中的交叉學(xué)科，其目標(biāo)是利用計(jì)算機(jī)技術(shù)處理海量復(fù)雜信息，研究新的、先進(jìn)的理論技術(shù),數(shù)據(jù)挖掘（Data Mining）?智能信息處理,,,,,,涉及的相關(guān)領(lǐng)域,,Statistics,MachineLearning,Databases,Visualization,Data Mining,,,,,,,,,,,,,Transformed Data,Pattern

9、sand Rules,Target Data,RawData,,,,Knowledge,Data Mining,Transformation,Interpretation& Evaluation,Selection& Cleaning,Integration,Understanding,數(shù)據(jù)挖掘的重要性,DATAWarehouse,Knowledge,提綱,引言數(shù)據(jù)挖掘概念及必要性數(shù)據(jù)挖掘的主要任務(wù)案

10、列分析,Data Mining Tasks: Classification,,,,,,,,,,,,,,,,,,,,,,Learn a method for predicting the instance class from pre-labeled (classified) instances,Many approaches: Statistics, Decision Trees, Neural Networks, ...,,,,

11、,,,,,,,Classification: Decision Trees,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,X,Y,,,,,,,if X > 5 then blueelse if Y > 3 then blueelse if X > 2 then greenelse blue,5,2,3,,,Classification: Neural Nets,Can select more comp

12、lex regionsCan be more accurateAlso can overfit the data – find patterns in random noise,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Data Mining Tasks: Clustering,,,,,cluster,,outlier,Linear Regression,,,,,,,,,,,,,,,,x,y,y = x + 1,,

13、,X1,Y1,(salary),(age),Example of linear regression,Major Data Mining Tasks,Classification: predicting an item classClustering: finding clusters in dataAssociations: e.g. A & B & C occur frequentlyVisualization

14、: to facilitate human discoveryEstimation: predicting a continuous valueDeviation Detection: finding changesLink Analysis: finding relationships…,實(shí)驗(yàn)工具,Weka主頁：http://www.cs.waikato.ac.nz/ml/weka/,提綱,引言數(shù)據(jù)挖掘概念及必要性數(shù)據(jù)挖

15、掘的主要任務(wù)案列分析,OD flow I-b,,案列：IP網(wǎng)絡(luò)流量異常行為檢測(cè),,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,OD flow I-b,Fri,S

16、at,Sun,案列：IP網(wǎng)絡(luò)流量分類(一),?,?,?,?,?,HTTP,Bittorrent,SMTP,,?,Traffic classification (TC),?,FeaturesReacting on application development,案列：IP網(wǎng)絡(luò)流量分類(二),29,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Training instances for class ATraining i