《大数据分析与处理》课程资料.pdfVIP

CS246: Mining Massive Datasets  Compress / reduce dimensionality: 106 rows; 103 columns; no updates Random access to any cell(s); small error: OK The above matrix is really “2-dimensional.” All rows can be reconstructed by scaling [1 1 1 0 0] or [0 0 0 1 1] 10/26/2018 Stanford C246: Mining Massive Datasets 2  Assumption: Data lies on or near a low d-dimensional subspace  Axes of this subspace are effective representation of the data 10/26/2018 Stanford C246: Mining Massive Datasets 3 Why reduce dimensions?  Discover hidden correlations/topics Words that occur commonly together  Remove redundant and noisy features Not all words are useful  Interpretation and visualization  Easier storage and processing of the data 10/26/2018 Stanford C246: Mining Massive Datasets 4 A[m x n] = U[m x r] Σ [ r x r] (V[n x r])T  A : Input data matrix m x n matrix (e.g., m documents, n terms)  U : Left singular vectors m x r matrix (m documents, r concepts)  Σ: Singular values r x r diagonal matrix (strength of each ‘concept’) (r : rank of the matrix A)  V : Right singular vectors n x r matrix (n terms, r concepts) 10/26/2018 Stanford C246: Mining Massive Datasets 5 T n n Σ VT m A ≈ m U 10/26/2018 Stanford C246: Mining Massive Datasets 6 T n σ u v σ u v