The Impact of Big Data

The New York Times recently published an interesting article, The Age of Big Data, on the topic of the Big Data phenomenon and the opportunities that accompany the surge of information from new sources. In last week’s blog, we noted that a McKinsey Institute Study found that the demand for skilled data analysts will significantly increase, and the article discusses how this is just the beginning of the shift toward the need for more analytical understanding of Big Data.

“It’s a revolution,” says Gary King, director of Harvard’s Institute for Quantitative Social Science. “We’re really just getting under way. But the march of quantification, made possible by enormous new sources of data, will sweep through academia, business and government. There is no area that is going to be untouched.”

The article examines the many reasons and examples of why Big Data has an influence on fields such as business, government, and economics, and more importantly why decision-making will increasingly be based on data analysis. The article presents a compelling comparison of the impact of big data to the invention of the microscope. Erik Brynjolfsson, an economist at Massachusetts Institute of Technology’s Sloan School of Management, explains how the microscope was a dramatic change in measurement by enabling things to be viewed at the cellular level.

Data measurement, Professor Brynjolfsson explains, is the modern equivalent of the microscope. Google searches, Facebook posts and Twitter messages, for example, make it possible to measure behavior and sentiment in fine detail and as it happens.

Along with data measurement, being able to integrate, visualize, and analyze data will be the key to accurate and timely decision-making for organizations tackling Big Data. As we’ve previously discussed on this blog, Big Data has a significant impact on social network behaviors. The article presents a good example of the opportunities that Big Data presents in being able to effectively understand key patterns and relationships in increasing amounts of data from various sources and of different types.

Researchers can see patterns of influence and peaks in communication on a subject — by following trending hashtags on Twitter, for example. The online fishbowl is a window into the real-time behavior of huge numbers of people. “I look for hot spots in the data, an outbreak of activity that I need to understand,” says Jon Kleinberg, a professor at Cornell. “It’s something you can only do with Big Data.”

Overall, the article is a good discussion on the impact of Big Data, with real-world examples of how Big Data is influencing society and business and the effect it has on critical decision-making.

Building Integrated Insights from Big Data

As we’ve discussed on this blog in previous posts, the growth of Big Data – data that is increasing in volume, rapidly changing pace, and includes varieties of data such as structured and unstructured – presents organizations with new challenges as well as new opportunities. One of the challenges organizations face is that as the rate of data creation and consumption continues to accelerate, so will the demand for business data analysis professionals. As explained in the McKinsey Institute Study, the demand for analysts will largely surpass supply. However, in addition to the challenges, a key opportunity remains for organizations to better understand Big Data on another level – creating actionable, integrated insights across multiple data sources. Key drivers of the demand for leveraging Big Data to improve decision-making can be addressed through greater focus on self-service analysis powered with visualization, along with methods for building and analyzing insights from disparate data sources.

For instance, the importance of creating integrated insights can be seen in recent studies that have shown that activities in social networks involving key individuals correlate with information transfers and investment activities in security markets. For example, consider the activities of a portfolio manager. Portfolio managers place larger bets on firms they are connected to through their social network, and perform significantly better on these holdings and on holdings relative to their non-connected holdings. Not surprisingly, VC firms who are better networked also experience significantly better fund performance, as measured by the proportion of investments that are successfully exited through an IPO or a sale to another company.

Source: GigaOm

SNA is based in modeling relationships between groups of people. Social network analysis uses graph theoretic ideas and applies them with the premise that the structure of the graph can be used to understand and identify critical relationships and influential people. However, networks modeling real world relationships often have different characteristics compared to relationships used in SNA. For instance, to accurately “connect the dots” involving large quantities of diverse information and see patterns involving multiple aliases of people, different approaches are needed to handle the misleading, inaccurate, or incomplete information. One of the issues with traditional SNA is that the people and entities in a network are often not treated as adaptive and capable of taking action, learning, or altering their networks over time.

Recent advancements in network analysis involving complex network topologies with multiple relationships between nodes, network behavior that is based on uncertain information, and time-based change of networks, have enhanced the value of incorporating advanced network analysis techniques as a key part of an analytics toolset to aid in better understanding data relationships. More organizations are beginning to understand that with advanced visual analysis technology, they can build integrated insights across all of their available data, enabling them to better understand emerging opportunities and threats.

Using advanced analytics, businesses can study Big Data to understand the current state of their business and track customer behavior. Security and law-enforcement organizations can use advanced visual analysis to build integrated views of emerging threats from all available data sources to take timely action. The combination of advanced visualization techniques, together with social network analysis techniques, will help bridge the emerging gap between the vast amounts of available information in Big Data and the available resources to better understand them.

In the next series of blog contributions, we will continue to explore the incredible opportunities that Big Data represents, and new products and technologies that can contribute to better understanding and new insights.

Visual Analysis: Beyond Social Network Analysis

Recent market dynamics continue to show an endless growth of data at a pace approaching 50% per year with the volume of digital content projected to grow to 2.7 zettabytes (ZB) by the end of 2012. With over 90% of the growth in information involving unstructured and semi-structured types of data, the demand for new types of analytic tools that help better understand hidden information in Big Data continues to increase.

As it has been noted on other posts of this blog, graph-based visual analysis is a highly effective method for capturing and understanding relationships between data that are not quantitative in nature. This method and technology has been used in diverse fields such as intelligence and law enforcement to customer sentiment and network topology analysis to uncover hidden insights in growing data that was not possible when relying only on traditional analytics.

This interest in visual analysis is due in part to the significant amount of data that are from new sources that are automated – capturing attributes such as time and geospatial locations on a routine basis. The richness of this new information creates an opportunity for understanding not only the traditional topological relationships between entities (such as social networks), but also for better understanding the behavior of networks in terms of change and pace of change, along with how specific changes are related to different types of data. As a result, we have the opportunity to better understand key trends and anomalies better than at any time in the past.

The combination of rich data collection, advanced analytics operating across both structured and unstructured data, and efficiently storing and analyzing information in quantities unimagined just a few years back, have created a new era of data analysis in general and visual analysis in particular. We can now look at the networks representing relationships between data as not just static topologies, but rather as “dynamic networks” with their own behavioral pattern in terms of change, sequence of change, and uncertainties of change, combined with the ability to integrate information from complex event processing engines and other “event driven” information sources. These new developments promise to bring about a new dawn of information use, enabling smarter, timelier decision-making in various fields of human endeavor.

New Blog Moderator and Direction

Hello Readers!

My name is Elizabeth Hefner, and as Stella Lau informed you, I’ll be the new blog moderator of Visual Insights. I want to thank all of you who have contributed to the blog already, and I’m looking forward to taking the blog in a new direction. As we all know, we are living in a data-driven world (see Accenture’s report on Big Data, and The Atlantic’s story on the “big data boom”). Not only are organizations bombarded with increasing volumes of data, but also data coming from new sources. As you can imagine, it is not enough to efficiently capture the data, but it must be better understood. Increasingly, many organizations are using visualization and analysis applications as part of their expanding portfolio of analysis software to discover new, valuable insights into their complex data.

Previously, Visual Insights focused more on data visualization. While data visualization plays a key part in insight discovery, I’ll be focusing this blog more on the topic of “advanced visual analysis”. If you are not familiar with the term, advanced visual analysis is what many analysts define as the new class of applications that tightly couple visualization and analysis, especially link analysis. These new applications go beyond pure visualization and traditional analytics, and reveal deeper, more complex relationships in data.

I’m excited to make this blog valuable to all data visualization and social network analysis enthusiasts, and I look forward to some insightful discussions! And remember, this community is an open forum for exchanging ideas, and comments and blog post submissions are greatly encouraged!

Elizabeth Hefner

Visual Insights Blog Moderator

Thank You!

Hello everyone! This will be my last post as the Visual Insights moderator. It has been a pleasure getting to know many of you. A big thank you to those who contributed to the conversation, whether it was through a blog post, comments, Twitter or Facebook shares, etc.

I’d like to introduce the new blog moderator, Elizabeth Hefner. Visual Insights started out as a platform to discuss all things data visualization, but since we’ve been seeing a growing demand and interest in visual analysis, Elizabeth will be taking this blog into this new direction. Please join me in welcoming Elizabeth!

– Stella Lau

Connect through LinkedIn

A Perspective of Why Financial Crises Happen, and How to Predict Them Before They Happen

The rise in the complexity and globalization of financial services has contributed to stronger linkages between various institutions. While this has led to smoother credit allocation and greater risk diversification among individual institutions, it has also increased the potential for disruptions spreading across banks and across borders as exemplified by the recent European banking crisis.  A financial contagion can take the form of shocks to haircuts, which trigger liquidity hoarding at some institutions that propagate across the interbank network. We live in a fascinating data-driven world full of challenges and opportunities – the world of Big Data. This is a world of unmatched availability of data to drive better decision making on a more timely basis but it is also a world where perceptions and panic can spread much faster than at earlier times.

There is a growing body of work studying the role of bank behavior in driving contagion as well as application of network techniques to better understand the probability and spread of contagion.  In these studies, the financial links between individual institutions are often represented as stable, embedded and long-term connections based on random graphs. However, one financial institution can be quite different from another in equity value and liabilities; the capacity of resisting risk impact is quite different. As a result, heterogeneous capacity of nodes is an important feature of real interbank networks versus a random network model. Also, the sequencing of transactions between institutions over time provides valuable insight into understanding the pattern of contagion spread across the network. Finally, the ability to analyze the behavior of interbank network in the context of other data in the media, development of events in regional politics, and impact of financial policy decisions in various countries can provide unforeseen insights.

New generation of advanced analytics, such as visual analysis, can be used to evaluate the resilience of the financial system to liquidity shocks affecting subset of banks. Advanced visual analysis applications can be used to explore the resilience of the financial system to liquidity shocks affecting a subset of banks under different network configurations, degree of connectivity between financial institutions, haircut assumptions, and balance sheet characteristics.  These tools combine advanced graph visualization techniques together with social and dynamic network analysis techniques to provide a comprehensive visual tool for analysts and policy makers.

“Big Data” and “Next-Generation Analytics” on Gartner’s Top 10 Strategic Technology Trends for 2012 List

On Monday at the Gartner Symposium, David Cearley presented Gartner’s annual list of the Top 10 Strategic Technology Trends for 2012. Among those that made the list are big data and next-generation analytics. Although the two items are listed separately, the two technologies go hand-in-hand and can provide the most compelling and effective experience for exploring and navigating through complex data.

[Source: PC Magazine]

It comes as no surprise that the two items were listed, as organizations are besieged by an explosive growth of information coming from disparate data sources every day. This data can be structured, semi-structured, and unstructured. With the annual growth rate well over 100%, analyzing and understanding big data has become a top priority.

For far too long, organizations have spent too much money and resources on collecting data, scrambling around to ensure data integrity, and finally, wondering how to make use of this data. More and more, organizations are starting to realize the need for technologies that can help them maximize the value of their data assets.

Traditional analytics that have relied on computers and algorithms to do the work for them are starting to be deemed limiting, especially with the growth of unstructured data. As Gartner continues to push big data and next-generation analytics, organizations need to adopt technologies that can drive business decision-making in ways not possible before. This new wave of analytic techniques, or advanced visual analysis, harnesses the most powerful pattern recognition system available — the human brain. Advanced visual analysis will help organizations discover key insights that were hidden in the past and turn the challenges of big data into opportunities.