A Perspective of Why Financial Crises Happen, and How to Predict Them Before They Happen

The rise in the complexity and globalization of financial services has contributed to stronger linkages between various institutions. While this has led to smoother credit allocation and greater risk diversification among individual institutions, it has also increased the potential for disruptions spreading across banks and across borders as exemplified by the recent European banking crisis.  A financial contagion can take the form of shocks to haircuts, which trigger liquidity hoarding at some institutions that propagate across the interbank network. We live in a fascinating data-driven world full of challenges and opportunities – the world of Big Data. This is a world of unmatched availability of data to drive better decision making on a more timely basis but it is also a world where perceptions and panic can spread much faster than at earlier times.

There is a growing body of work studying the role of bank behavior in driving contagion as well as application of network techniques to better understand the probability and spread of contagion.  In these studies, the financial links between individual institutions are often represented as stable, embedded and long-term connections based on random graphs. However, one financial institution can be quite different from another in equity value and liabilities; the capacity of resisting risk impact is quite different. As a result, heterogeneous capacity of nodes is an important feature of real interbank networks versus a random network model. Also, the sequencing of transactions between institutions over time provides valuable insight into understanding the pattern of contagion spread across the network. Finally, the ability to analyze the behavior of interbank network in the context of other data in the media, development of events in regional politics, and impact of financial policy decisions in various countries can provide unforeseen insights.

New generation of advanced analytics, such as visual analysis, can be used to evaluate the resilience of the financial system to liquidity shocks affecting subset of banks. Advanced visual analysis applications can be used to explore the resilience of the financial system to liquidity shocks affecting a subset of banks under different network configurations, degree of connectivity between financial institutions, haircut assumptions, and balance sheet characteristics.  These tools combine advanced graph visualization techniques together with social and dynamic network analysis techniques to provide a comprehensive visual tool for analysts and policy makers.


Key Features to Look for in Advanced Visual Analysis Software

“In 2009, more data will be generated by individuals than in the entire history of mankind through 2008.”
– Andreas Weigend, former Chief Scientist, Amazon

“Between the birth of the world and 2003, there were five exabytes of information created. We [now] create five exabytes every two days. See why it’s so painful to operate in information markets?”
– Eric Schmidt, CEO, Google

Enterprise data will grow 650% over the next five years. Of that, 80% of that will be unstructured. The amount of new information generated next year alone will amount to more than the previous 5,000 years. Finally, the number of text messages sent in the last 24 hours is one for every human on the planet.
– David Cappuccio, Analyst, Gartner Group


We all recognize that big data is only getting bigger, and with all the hoopla surrounding big data, what’s not to be concerned about? Big data is typically presented as a monstrous beast that is growing exponentially and spiraling out of control, and our ability to make sense of this data has not been keeping pace.

However, when equipped with the right tools to organize and manage your data, you can navigate seamlessly through the growing universe of data and turn the big data problem into big insights. Traditional analytics techniques are slowly starting to be deemed limiting. Many forward-looking organizations are incorporating advanced data visualization coupled with social network analysis capabilities, or advanced visual analysis, into their data analytics toolkit to proactively identify relevant actionable insights in a timely basis in ways not possible before.

Advanced visual analysis is the new generation of transforming potential liabilities caused by big data into invaluable business assets. Graph-based visualization, together with social network analysis capabilities, are most effective for seeing critical relationships, trends, anomalies, and patterns in your data that are not readable by computers but by the most powerful pattern-recognition system available – the human brain.

As you or your organization start to include advanced visual analysis into your analytics toolkit, here are some features to look for that are essential to maximizing the business value of your big data.

1. Access to Any Data Source, Any Time
Aside from traditional data sources such as Excel, XML, and text formats, new types of data are flowing into organizations at an exponential rate, such as clickstreams, GPS, biometric, cell phone intercepts, blog posts, Twitter feeds, point of sales, text, video, audio, seismic, Web logs, and RFID data. It is important that your advanced visual analysis application can access data from disparate data sources and integrate them into a single view.

2. Incremental loading
When you have a lot – even up to millions and billions – of data points, it is never effective to load or display all of this information at once. Loading all your data at once is slow and the amount of data presented might be overwhelming. With an incremental loading mechanism, you can incrementally load a subset of data from the data source that is relevant to your visualization purposes and analyzed in-memory for interactive analysis.

3. Filtering
Once you have your data mapped out, a key element of data and insight discovery through these applications requires sophisticated data filtering techniques. This enables you to focus on key trends, patterns, and outliers that matter to you for uncovering actionable insights without being distracted by other data.

4. Customizability
No matter what industry you’re in, you are the domain expert. Everyone’s usage models will differ from organization to organization. There are plenty of “out-of-the-box” solutions, but many of them don’t understand the unique context in which you are working in. Customizing your advanced visual analysis applications to fit your usage model will enable the most effective visual analysis experience, and help you identify insights that matter most to you.

How are you dealing with big data?

Congratulations to the Winner of the Graph Drawing 2011 Contest!

Last week, a few of us at Tom Sawyer Software attended the International Symposium on Graph Drawing in Eindhoven, Netherlands. The Graph Drawing Symposium is the main annual event that brings together top researchers and practitioners working in the area of graph drawing. The event is a forum where like-minded researchers gather to generate novel ideas, work together to create innovative solutions, and foster a forward-thinking community.

One of our core philosophies is to promote data visualization best practices and to provide the most compelling visualization techniques possible. At Graph Drawing, we contributed a challenge graph to the Graph Drawing Contest. For this particular topic, we challenged researchers to visually present a complex data set in the most compelling way possible by combining graph drawing algorithms and complexity reduction techniques, such as filtering and varying the graphical attributes. The graph is a composers graph with a musical theme, where the nodes represent Wikipedia articles about music composers, and the edges represent links between these articles.

A lot of great visualizations came out of this contest. Congratulations to the winner, the folks over at Meurs HRM! Check out the interactive visualization they produced for this challenge; awesome stuff.

Big Data = Big Opportunities through Advanced Visual Analytics

We live in a fascinating data-driven world full of challenges and opportunities – the world of Big Data. With the continuing, explosive growth of data in terms of quantity, diversity of sources and types, and speed, there are tremendous opportunities to explore this “Brave New World of Big Data” to discover new insights that were hard to uncover in the past.

In recognition of these new opportunities, more and more organizations are adding data visualization applications powered with advanced analytics, such as Social Network Analysis (SNA), into their portfolio of advanced analytics – what is often called advanced visual analysis. The growing recognition of the role of advanced visual analysis as part of analytics toolkits is highlighted by recent acquisition of i2 Technologies by IBM, growth of visualization based business intelligence products from QlikView, TIBCO/Spotfire, and Tableau as well as the recent move by Facebook to provide Open Graph as part of their makeover. This is indeed an exciting time for advanced visual analysis.

The growing use of advanced visual analysis applications, especially graph-based systems powered with SNA techniques, have led to remarkable recent track record of success in combating global terrorism, cyber warfare, and criminal fraud activities in various industries. New technologies such as pad computing devices, GPS and biometric based systems, and next generation network management systems for integrated communication of audio, data, and video, the continuing growth of online social communities, as well as integrated demand-supply chain systems have led to rise in demand for using these applications to discover insights from not just highly structured data in traditional enterprise repositories but from all data types across all types of information repositories. The increased success of these applications in different industries will continue to fuel the growth in advanced visual analysis applications of all types in the coming years.

The ongoing industry recognition of the power of advanced visual analysis will yield better integrated analytic systems in many cases, better defined visual analysis infrastructure in others, and overall a superior experience and results for organizations focused on maximizing the value of their Big Data.   This is a fascinating and exciting time for those of us involved in visualization, advanced visual analytics, and advanced data analytics of all types. We are entering an era where our ability to discover actionable insights in the right context is leading to a fundamental paradigm shift in how we leverage our data assets to create more agile, efficient, and intelligent organizations everywhere.

Visualizing the Network Map of the News Corp Scandal

The News Corp scandal is quickly snowballing as it spreads into different parts of the world, beginning with News of the World in London. In the past weeks, investigators have been analyzing the messy web of interconnections between organizations and key individuals involved with the scandal in order to indict the appropriate entities.

While there is a barrage of news articles providing the latest updates, it can become difficult to keep track of the exact causes and relationships with each new reported event. BusinessWeek and The Wall Street Journal released the following graphics to visually communicate these relationships in an easier-to-grasp form.

Data Visualization of the News Corp Scandal, Business Week

The Wall Street Journal:
Data Visualization of the News Corp Scandal, Wall Street Journal

In addition to the News Corp scandal, data visualization can often be used to help people better understand other real world developments that involve very complex subsets of relationships between various entities, such as the Galleon Group’s insider trading case and even finding Osama Bin Laden. Applying social network analysis and centrality measures can show who communicates with whom, how often, and how entities are tied together. This helps to reveal varying levels of influence of certain people or organizations that can provide critical actionable insights. Even if there are prominent perpetrators at the forefront of scandals, “hidden” individuals outside the immediate circle of their networks may interestingly come into question due to high centrality measures.

The Shift Towards a Data-Driven World and How to Navigate Through It

One of the latest buzz phrases in the information industry is that we are living in a data-driven world. Increasingly, the efficient operation of enterprise, government, and other organizations relies on the management, understanding, and effective use of vast amounts of data. One of my college professors, the late Nobel laureate and pioneer in artificial intelligence, Prof. Herbert Simon had a saying that “… What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it …”. His commentary about the need to more efficiently allocate attention to maximize the value of information best describes the growing interest in visualization and analysis applications.

The demand for data visualization and analysis applications is multifaceted and growing rapidly. This trend is consistent with the exponential growth of relevant data available to organizations as noted by former Amazon.com chief scientist Dr. Andreas Weigend, “… In 2009, more data will be generated by individuals than in the entire history of mankind through 2008 …”.  These visualization and analysis applications address a diverse universe of usages from business intelligence to network management and social network analysis. In all these cases, the common denominator is the recognition that visualization and visualization coupled with analysis which leverage people’s ability to better process dynamic images instead of long lists of text and numbers needs to be part of an organization’s tool kit for understanding information better.

Here at Tom Sawyer Software, as well as other companies in the data visualization and analysis market, we have witnessed a dramatic increase in demand for software for building these types of applications. For example, after the recent financial market situation in many countries, there is an increased interest in using graph visualization and social network analysis applications to proactively recognize trends and anomalies in financial institution networks for fund transfers and risk management markets. The interbank markets are pivotal for liquidity management purposes of financial institutions because they allow banks to buffer shocks by permitting rapid transfer of funds from surplus to deficit agents. The ability to recognize potentially dangerous situations and conducting forensic analysis on dangerous situations for root causes is becoming more important.

In the networking and telecommunications industries, there has been a major shift towards IP based, service centric, next generation networks in the last few years. The underlying drivers for the change include competitive pressures, game-changing technology innovations, change in purchasing behavior of consumers, and changes in regional regulations. The virtual layering involved in these networks require far more sophisticated network management tools and increasing number of companies have integrated visualization and analysis capabilities into their management systems. In addition, social network analysis is having an impact on how telecommunications companies leverage their data assets to improve their performance and increase competitive advantages. Social network analysis applications are increasingly being used to study how the network of customers, partners, and community members that exist outside the organization influence the organization to detect customer churn and to better target marketing efforts.

As recent world events have shown, there is a growing use of data visualization and social network analysis in the defense and law enforcement organizations to identify terrorist threats and analyze fraud situations in bank transactions to insider trading. With the growing recognition and proven ROI around these applications, we continue to expect a sustained growth of interest in sophisticated data visualization and analysis applications and software for building these applications.

Four Key Features to Look for when Choosing Data Visualization Software

As the volume of relevant information that organizations must sift through grows, organizations are increasingly finding it difficult to gain actionable insights from very large, complex data sets. Without the appropriate systems or applications to manage all this data, data itself can lead to counter-productivity.

Data visualization is named one of the (re)emerging trends in 2011 to help businesses better leverage their data. If your organization is considering on utilizing visualization technology, you’ll find that there are quite a few tools with different features out there. From twenty years of experience, here are four key things people have told us what they typically look for when choosing the right data visualization software to achieve their business goals.

1. Scalability and Complexity Management

Let’s say you are responsible for managing complex networks. Your data sets can easily contain tens of thousands to millions of objects. If the software takes minutes to recalculate your network map, your application may be useless and you can become inundated with your data. If it doesn’t provide tools like expanding and collapsing nested nodes, hiding, and filtering, then your users won’t be happy working with large data sets when they can’t choose see only what is relevant to them.

2. Stability

Visualization software can bring value in a huge variety of industries, and the way you want to use it may not be unique. Proven, well-tested software is more likely to be stable no matter how you use it.

3. Attention to Layout and Labeling

Well-designed visualization software enhances user comprehension of your data. It puts labels where you can read them, not on top of other objects. It always ensures links terminate right at the node, not somewhere close by. It ensures you can tell which label belongs with which object, and it makes the best use of the available space to ensure clarity in your visualization. A good tool does this automatically.

4. Customizability and Flexibility

A visualization may show link labels above the links, but what if you want them on the left? The icons might be a beautiful shade of blue, but your corporate color is green. What if you don’t want to display an overview window? Can you turn it off?  Having broad, well-documented, and well-supported APIs for all available functionality is essential if you want to be able to produce the results you want.