CMC702/Semester 7
Big Data Analysis
17 notes written · 55% syllabus coverage (16 of 29 topics)
CMC702 — Big Data Analysis
Detailed Syllabus
Module I — Introduction to Big Data and Hadoop (2 hours)
- 1.1 Introduction to Big Data — characteristics and types of Big Data
- 1.2 Traditional vs. Big Data business approach
- 1.3 Case Study of Big Data Solutions
- 1.4 Concept of Hadoop, Core Hadoop Components, Hadoop Ecosystem
Module II — Hadoop HDFS and MapReduce (8 hours)
2A. Distributed File Systems
- 2.1 Physical Organization of Compute Nodes
- 2.2 Large-Scale File-System Organization
2B. MapReduce
- 2.3 The Map Tasks, Grouping by Key, The Reduce Tasks
- 2.4 Combiners, Details of MapReduce Execution
- 2.5 Coping With Node Failures
2C. Algorithms Using MapReduce
- 2.6 Matrix-Vector Multiplication by MapReduce
- 2.7 Relational-Algebra Operations: Computing Selections, Projections, Union, Intersection, and Difference by MapReduce
- 2.8 Hadoop Limitations
Module III — NoSQL (10 hours)
- 3.1 Introduction to NoSQL, NoSQL Business Drivers
- 3.2 NoSQL Data Architecture Patterns
- Key-value stores
- Graph stores
- Column family (Bigtable) stores
- Document stores
- Variations of NoSQL architectural patterns
- 3.3 NoSQL Case Study
- 3.4 NoSQL solution for big data
- Understanding the types of big data problems
- Analyzing big data with a shared-nothing architecture
- Choosing distribution models: master-slave versus peer-to-peer
- NoSQL systems to handle big data problems
Module IV — Mining Data Streams (11 hours)
- 4.1 The Stream Data Model
- A Data-Stream-Management System
- Examples of Stream Sources
- Stream Queries
- Issues in Stream Processing
- 4.2 Sampling Data techniques in a Stream
- 4.3 Filtering Streams: Bloom Filter with Analysis
- 4.4 Counting Distinct Elements in a Stream
- Count-Distinct Problem
- Flajolet-Martin Algorithm
- Combining Estimates, Space Requirements
- 4.5 Counting Ones in a Window
- The Cost of Exact Counts
- The DGIM Algorithm (Datar-Gionis-Indyk-Motwani)
- Query Answering in the DGIM Algorithm
- Decaying Windows
Module V — Real-Time Big Data Models (4 hours)
- 5.1 A Model for Recommendation Systems
- Content-Based Recommendations
- Collaborative Filtering
- 5.2 Case Study: Product Recommendation
- 5.3 Social Networks as Graphs
- Clustering of Social-Network Graphs
- Direct Discovery of Communities in a social graph
Module VI — Data Analytics with R (4 hours)
- 6.1 Exploring Basic features of R
- Exploring RGUI, Exploring RStudio
- Handling Basic Expressions in R
- Variables in R, Working with Vectors
- Storing and Calculating Values in R
- Creating and using Objects
- Interacting with users, Handling data in R workspace
- Executing Scripts, Creating Plots
- Accessing help and documentation in R
- 6.2 Reading datasets and Exporting data from R
- 6.3 Manipulating and Processing Data in R
- 6.4 Using functions instead of script, built-in functions in R
- 6.5 Data Visualization: Types, Applications
Text Books
| # | Title | Author(s) | Publisher |
|---|---|---|---|
| 1 | Mining of Massive Datasets | Cre Anand Rajaraman, Jeff Ullman | Cambridge University Press |
| 2 | Hadoop in Practice | Alex Holmes | Manning Press / Dreamtech Press |
| 3 | Making Sense of NoSQL — A guide for managers and the rest of us | Dan Mcary, Ann Kelly | Manning Press |
| 4 | Big Data Black Book | DT Editorial Services | Dreamtech Press |
| 5 | Data Science and Big Data Analytics | EMC Education Services | Wiley |
References
| # | Title | Author(s) | Publisher |
|---|---|---|---|
| 1 | Taming The Big Data Tidal Wave: Finding Opportunities In Huge Data Streams With Advanced Analytics | Bill Franks | Wiley |
| 2 | Hadoop in Action | Chuck Lam | Dreamtech Press |
| 3 | Big Data, Data Mining, and Machine Learning: Value Creation for Business Leaders and Practitioners | Jared Dean | Wiley India Private Limited, 2014 |
| 4 | Data Mining: Concepts and Techniques (3rd ed.) | Jiawei Han, Micheline Kamber | Morgan Kaufmann |
| 5 | Data Mining and Knowledge Discovery Handbook (2nd ed.) | Lior Rokach, Oded Maimon | Springer |
| 6 | The Text Mining Handbook: Advanced Approaches in Analyzing Unstructured Data | Ronen Feldman, James Sanger | Cambridge University Press |
| 7 | Learning and Soft Computing | Vojislav Kecman | MIT Press |
Online References
| # | Resource |
|---|---|
| 1 | NPTEL: Big Data |
| 2 | Coursera: Big Data Specialization |
| 3 | NPTEL: Big Data Computing |
| 4 | Coursera: NoSQL Databases |
| 5 | Coursera: Basic Recommender Systems |
CML707 — Big Data Analytics Lab
List of Experiments
Select a case study to perform experiments 1–8. Experiments marked with * are compulsory.
| Sr. No. | Experiment Title | Hours |
|---|---|---|
| 1* | Hadoop HDFS Practical: HDFS Basics, Hadoop Ecosystem Tools Overview — Installing Hadoop, Copying/Moving/Deleting files in HDFS, Programming exercises on Hadoop | 2 |
| 2 | Use of Sqoop tool to transfer data between Hadoop and relational database servers — Sqoop installation and basic commands | 2 |
| 3* | Install and configure MongoDB / Cassandra / HBase / Hypertable to execute NoSQL commands | 2 |
| 4 | Hadoop MapReduce — Implement a word count program using MapReduce | 2 |
| 5 | Hadoop MapReduce — Implement simple algorithms: Matrix multiplication, Aggregates, Joins, Sorting, Searching, etc. | 2 |
| 6 | Create HIVE Database and Descriptive analytics — basic statistics | 2 |
| 7* | Data Stream Algorithms (any one) — Implement DGIM algorithm, Bloom Filter, or Flajolet-Martin algorithm | 2 |
| 8 | Social Network Analysis using R (e.g. Community Detection Algorithm) | 2 |
| 9 | Data Visualization using Hive / PIG / R / Tableau | 2 |
| 10 | Exploratory Data Analysis using Spark / PySpark | 2 |
| 11* | Mini Project: One real-life large data application — Streaming data analysis (Flume + HIVE/PySpark for twitter data, chat data, weblog analysis), Recommendation System (Health Care, Stock Market, Movie), or Spatio Temporal Data Analytics | 2 |
Assignments / Tutorials
| Sr. No. | Description | Hours |
|---|---|---|
| 01 | Assignment covering Modules I–III (limited to 3 questions) | 2 |
| 02 | Assignment covering Modules IV–VI (limited to 3 questions) | 2 |
Lab Text Books & References
(Same as theory — see Text Books and References sections above.)