The article ‘Working with big data’ explains in detail the phenomenon that will become the essential basis for competition, innovation, and productivity, known as “big data”.
Big data is defined as a collection of large datasets that cannot be analyzed with normal statistical methods. A typical mp3 song is less than 1,000,000 bytes and a set of big data is usually measured in exabytes (1,000,000,000,000,000,000 bytes) to put in perspective the size of this data. The data can include numbers, videos, pictures, maps, words, etc. Big data can be broken down into two types: structured data and unstructured data. Structured data are numbers and words that can be easily categorized and analyzed (i.e. sales figures, account balances, etc.). Unstructured data refers to more complex information (i.e. customer reviews, photos, multimedia, etc.). This data cannot easily be separated into categories or analyzed numerically.
Even though most of the work done with this data is done with machines, workers are still involved in the collection, processing, and analysis of big data. These men and women, sometimes called ‘data scientists’, study big data using both conventional and newly developed statistical methods. They run computer programs or algorithms to detect patterns or to find usable information. Because there is so much data to sort through, finding a method to store and organize the data is often a complication. Summarizing the results of the data analysis into graphs, charts, or other tools is also a task the workers must complete to present to managers and/or clients.
Big data can be analyzed in virtually any field of work. The analysts’ job tasks differ based on the source of the data. The following are examples of specific kinds of big data and how workers are involved with them:
Business: Analysts collect product data and analyze it in the context of the industry. Big data can also be used to manage inventories to improve efficiency.
Electric Commerce: Data scientists can help a company improve customer service by looking at customer reviews, comments, and suggestions. They also search data to find trends in purchasing or website traffic.
Finance: Big data analysts study transaction data to look for fraud and other security breaches. Government: Analysts help governments serve their constituents better and improve policy decisions.
Healthcare: Estimating cost effectiveness of new drugs and tracking disease outbreaks in real time are part of an analyst’s job.
Science: Because many fields of science product huge datasets, data analysts are important because they can sort through them all and find trends.
Social Networking: Analyzing data from social networking websites can help businesses make better products or advertise more effectively.
Telecommunications: Analysts study huge amounts of data from phone records to try and minimize dropped calls and other problems.
Other: Big data is everywhere. Other fields where big data is increasingly used include politics, utilities, and smart meters on appliances.
The growth of big data has provided new insights but also has presented new challenges to those who work with big data. One of the biggest challenges is the availability of funding. New investments are often eliminated to cut costs. Because big data is a recent phenomenon, funds that were meant for analysis software, improved computing equipment, or hiring new data analysts often are targeted. Another challenge is storage. These data sets are huge and require many (sometimes hundreds) of servers. Finding the usable data among unusable information can often be a hurdle for big data analysts because of the volume of data there is. People who work with big data have to be logical and good at solving problems because with the amount and variety of unstructured data, it is often unclear how the data should be interpreted. Analysts mat also be responsible for devising methods of keeping big data secure.
A major barrier to the vast use of big data is the lack of workers with the appropriate training and skills. In addition to having a bachelor’s degree, most big data analysts have a master’s or higher degree with specialties in mathematics, statistics, or computer science. Math helps students develop logical thinking and problem solving skills they need. Statistics provides the analytical knowledge they need to properly study the data and to interpret the results in a meaningful way. Workers who use big data mat also need an education in the industry in which they work, especially in highly technical industries, such as physics and healthcare. It is also important that these men and women stay current with the fast-changing world of big data. Analysts often work in teams because the data is so complex. Each member of the team has a different responsibility. So communicating with people who don’t have technical backgrounds is helpful when trying to explain results to other workers. Most importantly, data analysts must possess intellectual curiosity. It is very valuable to always be exploring and investigating new ideas.