The Shift Towards a Data-Driven World and How to Navigate Through It

One of the latest buzz phrases in the information industry is that we are living in a data-driven world. Increasingly, the efficient operation of enterprise, government, and other organizations relies on the management, understanding, and effective use of vast amounts of data. One of my college professors, the late Nobel laureate and pioneer in artificial intelligence, Prof. Herbert Simon had a saying that “… What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention, and a need to allocate that attention efficiently among the overabundance of information sources that might consume it …”. His commentary about the need to more efficiently allocate attention to maximize the value of information best describes the growing interest in visualization and analysis applications.

The demand for data visualization and analysis applications is multifaceted and growing rapidly. This trend is consistent with the exponential growth of relevant data available to organizations as noted by former chief scientist Dr. Andreas Weigend, “… In 2009, more data will be generated by individuals than in the entire history of mankind through 2008 …”.  These visualization and analysis applications address a diverse universe of usages from business intelligence to network management and social network analysis. In all these cases, the common denominator is the recognition that visualization and visualization coupled with analysis which leverage people’s ability to better process dynamic images instead of long lists of text and numbers needs to be part of an organization’s tool kit for understanding information better.

Here at Tom Sawyer Software, as well as other companies in the data visualization and analysis market, we have witnessed a dramatic increase in demand for software for building these types of applications. For example, after the recent financial market situation in many countries, there is an increased interest in using graph visualization and social network analysis applications to proactively recognize trends and anomalies in financial institution networks for fund transfers and risk management markets. The interbank markets are pivotal for liquidity management purposes of financial institutions because they allow banks to buffer shocks by permitting rapid transfer of funds from surplus to deficit agents. The ability to recognize potentially dangerous situations and conducting forensic analysis on dangerous situations for root causes is becoming more important.

In the networking and telecommunications industries, there has been a major shift towards IP based, service centric, next generation networks in the last few years. The underlying drivers for the change include competitive pressures, game-changing technology innovations, change in purchasing behavior of consumers, and changes in regional regulations. The virtual layering involved in these networks require far more sophisticated network management tools and increasing number of companies have integrated visualization and analysis capabilities into their management systems. In addition, social network analysis is having an impact on how telecommunications companies leverage their data assets to improve their performance and increase competitive advantages. Social network analysis applications are increasingly being used to study how the network of customers, partners, and community members that exist outside the organization influence the organization to detect customer churn and to better target marketing efforts.

As recent world events have shown, there is a growing use of data visualization and social network analysis in the defense and law enforcement organizations to identify terrorist threats and analyze fraud situations in bank transactions to insider trading. With the growing recognition and proven ROI around these applications, we continue to expect a sustained growth of interest in sophisticated data visualization and analysis applications and software for building these applications.