Big Data in Data Science

Source: ttps://www.coursera.org/learn/data-scientiststools/ungradedWidget/8x1jm/bigdata

The Three Vs of Big Data:
Getting started with big data requires three key steps-

• Integrate
• Manage
•  Analyze

The definition of “Big” has evolved with the advancement of technology & data storage capability to be able to hold larger data sets. Also, our capacity to collect & store the data has upgraded with time such that the speed for data collection is unprecedented. The idea of “data” has developed gradually; as a result internet & technology have recognized to collect different categorical data for analysis. One of the main objectives in data science has been moving from structured data sets to tackling unstructured data.

Structured & Unstructured Data:

Structured data means our usual idea about data i.e. long tables, spreadsheets, or databases with columns and rows of information that one can sum or average or analyze. Unfortunately, we encounter the data sets which are much messier & the job of data scientist is to present the data sets into something neat & tidy format. With the advancement of internet & technology, many pieces of information that weren’t traditionally collected were suddenly able to be translated into a format that a computer could record, store, search, and analyze. Presently, the unstructured data being collected from all of our digital interactivity: Emails, Face book, Instagram, YouTube, Twitter, SMS, shopping habit, use of Smartphone, CCTV cameras & other video sources, etc.

Is Big Data a Volume or Technology?

The term “Big data” may seem to suggest large data set but it may refer to the technology when used by vendors. The technology incorporates the tools & processes to operate the massive volume of data and storage facilities.

Advantages and Disadvantages of Big Data:

The challenges of working with big data are-

• Big: Massive volume of raw data that you need to be able to store and analyze.
• Variety: Sometimes, it can be difficult to decide the source of data due to the variety of sources of information.
• Messy: In reality, the collected data can be messy. You need to turn the unstructured data into a format that can be analyzed.
• Update: The technology is changing at a rapid pace. First, Apache Hadoop & Apache Spark was used to solve big data problems. Now, the hybrid frameworks are used to be the best approach.

The advantage of working with big data are-

• Neglect error: There are many sources of error in data collection. If there are any errors in the data, the volume of the data set can negate the small errors in it.
• Accurate decision: Up to date information allows you to make analysis to the current state of the system & suggest rapid, informed predictions and decisions.
• Answer the unfeasible queries: The unconventional data sources may allow you to answer the previously inaccessible and formerly unfeasible questions. Big data can enable you to obtain more complete answers than before.
• Identify hidden correlations: The big data can recognize the hidden relation between outcome variable & input variable which may not be related to our outcome variable.

The application of Big Data:

Big Data can assist the company to identify a range of business ventures, from customer experience to analytics. Few examples are

• Product Development
• Customer Experience
• Fraud and Compliance
• Machine Learning
• Operational Efficiency

Conclusion:

A famous statistician, John Tukey, said in 1986, “The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.” Likewise, any big data set may not answer all the queries if it’s not the right data. So, we can conclude that data science is question-driven science and even the largest data sets may not be always suitable.