Visual Analysis: Beyond Social Network Analysis

Recent market dynamics continue to show an endless growth of data at a pace approaching 50% per year with the volume of digital content projected to grow to 2.7 zettabytes (ZB) by the end of 2012. With over 90% of the growth in information involving unstructured and semi-structured types of data, the demand for new types of analytic tools that help better understand hidden information in Big Data continues to increase.

As it has been noted on other posts of this blog, graph-based visual analysis is a highly effective method for capturing and understanding relationships between data that are not quantitative in nature. This method and technology has been used in diverse fields such as intelligence and law enforcement to customer sentiment and network topology analysis to uncover hidden insights in growing data that was not possible when relying only on traditional analytics.

This interest in visual analysis is due in part to the significant amount of data that are from new sources that are automated – capturing attributes such as time and geospatial locations on a routine basis. The richness of this new information creates an opportunity for understanding not only the traditional topological relationships between entities (such as social networks), but also for better understanding the behavior of networks in terms of change and pace of change, along with how specific changes are related to different types of data. As a result, we have the opportunity to better understand key trends and anomalies better than at any time in the past.

The combination of rich data collection, advanced analytics operating across both structured and unstructured data, and efficiently storing and analyzing information in quantities unimagined just a few years back, have created a new era of data analysis in general and visual analysis in particular. We can now look at the networks representing relationships between data as not just static topologies, but rather as “dynamic networks” with their own behavioral pattern in terms of change, sequence of change, and uncertainties of change, combined with the ability to integrate information from complex event processing engines and other “event driven” information sources. These new developments promise to bring about a new dawn of information use, enabling smarter, timelier decision-making in various fields of human endeavor.

Advertisements

New Blog Moderator and Direction

Hello Readers!

My name is Elizabeth Hefner, and as Stella Lau informed you, I’ll be the new blog moderator of Visual Insights. I want to thank all of you who have contributed to the blog already, and I’m looking forward to taking the blog in a new direction. As we all know, we are living in a data-driven world (see Accenture’s report on Big Data, and The Atlantic’s story on the “big data boom”). Not only are organizations bombarded with increasing volumes of data, but also data coming from new sources. As you can imagine, it is not enough to efficiently capture the data, but it must be better understood. Increasingly, many organizations are using visualization and analysis applications as part of their expanding portfolio of analysis software to discover new, valuable insights into their complex data.

Previously, Visual Insights focused more on data visualization. While data visualization plays a key part in insight discovery, I’ll be focusing this blog more on the topic of “advanced visual analysis”. If you are not familiar with the term, advanced visual analysis is what many analysts define as the new class of applications that tightly couple visualization and analysis, especially link analysis. These new applications go beyond pure visualization and traditional analytics, and reveal deeper, more complex relationships in data.

I’m excited to make this blog valuable to all data visualization and social network analysis enthusiasts, and I look forward to some insightful discussions! And remember, this community is an open forum for exchanging ideas, and comments and blog post submissions are greatly encouraged!

Elizabeth Hefner

Visual Insights Blog Moderator

Thank You!

Hello everyone! This will be my last post as the Visual Insights moderator. It has been a pleasure getting to know many of you. A big thank you to those who contributed to the conversation, whether it was through a blog post, comments, Twitter or Facebook shares, etc.

I’d like to introduce the new blog moderator, Elizabeth Hefner. Visual Insights started out as a platform to discuss all things data visualization, but since we’ve been seeing a growing demand and interest in visual analysis, Elizabeth will be taking this blog into this new direction. Please join me in welcoming Elizabeth!

— Stella Lau

Connect through LinkedIn

A Perspective of Why Financial Crises Happen, and How to Predict Them Before They Happen

The rise in the complexity and globalization of financial services has contributed to stronger linkages between various institutions. While this has led to smoother credit allocation and greater risk diversification among individual institutions, it has also increased the potential for disruptions spreading across banks and across borders as exemplified by the recent European banking crisis.  A financial contagion can take the form of shocks to haircuts, which trigger liquidity hoarding at some institutions that propagate across the interbank network. We live in a fascinating data-driven world full of challenges and opportunities – the world of Big Data. This is a world of unmatched availability of data to drive better decision making on a more timely basis but it is also a world where perceptions and panic can spread much faster than at earlier times.

There is a growing body of work studying the role of bank behavior in driving contagion as well as application of network techniques to better understand the probability and spread of contagion.  In these studies, the financial links between individual institutions are often represented as stable, embedded and long-term connections based on random graphs. However, one financial institution can be quite different from another in equity value and liabilities; the capacity of resisting risk impact is quite different. As a result, heterogeneous capacity of nodes is an important feature of real interbank networks versus a random network model. Also, the sequencing of transactions between institutions over time provides valuable insight into understanding the pattern of contagion spread across the network. Finally, the ability to analyze the behavior of interbank network in the context of other data in the media, development of events in regional politics, and impact of financial policy decisions in various countries can provide unforeseen insights.

New generation of advanced analytics, such as visual analysis, can be used to evaluate the resilience of the financial system to liquidity shocks affecting subset of banks. Advanced visual analysis applications can be used to explore the resilience of the financial system to liquidity shocks affecting a subset of banks under different network configurations, degree of connectivity between financial institutions, haircut assumptions, and balance sheet characteristics.  These tools combine advanced graph visualization techniques together with social and dynamic network analysis techniques to provide a comprehensive visual tool for analysts and policy makers.

“Big Data” and “Next-Generation Analytics” on Gartner’s Top 10 Strategic Technology Trends for 2012 List

On Monday at the Gartner Symposium, David Cearley presented Gartner’s annual list of the Top 10 Strategic Technology Trends for 2012. Among those that made the list are big data and next-generation analytics. Although the two items are listed separately, the two technologies go hand-in-hand and can provide the most compelling and effective experience for exploring and navigating through complex data.

Gartner's Top 10 Strategic Technology Trends in 2012[Source: PC Magazine]

It comes as no surprise that the two items were listed, as organizations are besieged by an explosive growth of information coming from disparate data sources every day. This data can be structured, semi-structured, and unstructured. With the annual growth rate well over 100%, analyzing and understanding big data has become a top priority.

For far too long, organizations have spent too much money and resources on collecting data, scrambling around to ensure data integrity, and finally, wondering how to make use of this data. More and more, organizations are starting to realize the need for technologies that can help them maximize the value of their data assets.

Traditional analytics that have relied on computers and algorithms to do the work for them are starting to be deemed limiting, especially with the growth of unstructured data. As Gartner continues to push big data and next-generation analytics, organizations need to adopt technologies that can drive business decision-making in ways not possible before. This new wave of analytic techniques, or advanced visual analysis, harnesses the most powerful pattern recognition system available — the human brain. Advanced visual analysis will help organizations discover key insights that were hidden in the past and turn the challenges of big data into opportunities.

Key Features to Look for in Advanced Visual Analysis Software

“In 2009, more data will be generated by individuals than in the entire history of mankind through 2008.”
– Andreas Weigend, former Chief Scientist, Amazon

“Between the birth of the world and 2003, there were five exabytes of information created. We [now] create five exabytes every two days. See why it’s so painful to operate in information markets?”
– Eric Schmidt, CEO, Google

Enterprise data will grow 650% over the next five years. Of that, 80% of that will be unstructured. The amount of new information generated next year alone will amount to more than the previous 5,000 years. Finally, the number of text messages sent in the last 24 hours is one for every human on the planet.
– David Cappuccio, Analyst, Gartner Group

*~*~*~*~*~*~*

We all recognize that big data is only getting bigger, and with all the hoopla surrounding big data, what’s not to be concerned about? Big data is typically presented as a monstrous beast that is growing exponentially and spiraling out of control, and our ability to make sense of this data has not been keeping pace.

However, when equipped with the right tools to organize and manage your data, you can navigate seamlessly through the growing universe of data and turn the big data problem into big insights. Traditional analytics techniques are slowly starting to be deemed limiting. Many forward-looking organizations are incorporating advanced data visualization coupled with social network analysis capabilities, or advanced visual analysis, into their data analytics toolkit to proactively identify relevant actionable insights in a timely basis in ways not possible before.

Advanced visual analysis is the new generation of transforming potential liabilities caused by big data into invaluable business assets. Graph-based visualization, together with social network analysis capabilities, are most effective for seeing critical relationships, trends, anomalies, and patterns in your data that are not readable by computers but by the most powerful pattern-recognition system available – the human brain.

As you or your organization start to include advanced visual analysis into your analytics toolkit, here are some features to look for that are essential to maximizing the business value of your big data.

1. Access to Any Data Source, Any Time
Aside from traditional data sources such as Excel, XML, and text formats, new types of data are flowing into organizations at an exponential rate, such as clickstreams, GPS, biometric, cell phone intercepts, blog posts, Twitter feeds, point of sales, text, video, audio, seismic, Web logs, and RFID data. It is important that your advanced visual analysis application can access data from disparate data sources and integrate them into a single view.

2. Incremental loading
When you have a lot – even up to millions and billions – of data points, it is never effective to load or display all of this information at once. Loading all your data at once is slow and the amount of data presented might be overwhelming. With an incremental loading mechanism, you can incrementally load a subset of data from the data source that is relevant to your visualization purposes and analyzed in-memory for interactive analysis.

3. Filtering
Once you have your data mapped out, a key element of data and insight discovery through these applications requires sophisticated data filtering techniques. This enables you to focus on key trends, patterns, and outliers that matter to you for uncovering actionable insights without being distracted by other data.

4. Customizability
No matter what industry you’re in, you are the domain expert. Everyone’s usage models will differ from organization to organization. There are plenty of “out-of-the-box” solutions, but many of them don’t understand the unique context in which you are working in. Customizing your advanced visual analysis applications to fit your usage model will enable the most effective visual analysis experience, and help you identify insights that matter most to you.

How are you dealing with big data?

Congratulations to the Winner of the Graph Drawing 2011 Contest!

Last week, a few of us at Tom Sawyer Software attended the International Symposium on Graph Drawing in Eindhoven, Netherlands. The Graph Drawing Symposium is the main annual event that brings together top researchers and practitioners working in the area of graph drawing. The event is a forum where like-minded researchers gather to generate novel ideas, work together to create innovative solutions, and foster a forward-thinking community.

One of our core philosophies is to promote data visualization best practices and to provide the most compelling visualization techniques possible. At Graph Drawing, we contributed a challenge graph to the Graph Drawing Contest. For this particular topic, we challenged researchers to visually present a complex data set in the most compelling way possible by combining graph drawing algorithms and complexity reduction techniques, such as filtering and varying the graphical attributes. The graph is a composers graph with a musical theme, where the nodes represent Wikipedia articles about music composers, and the edges represent links between these articles.

A lot of great visualizations came out of this contest. Congratulations to the winner, the folks over at Meurs HRM! Check out the interactive visualization they produced for this challenge; awesome stuff.