DoDIIS and CTX

Hi everyone, It’s been a crazy past month, so sorry for the delay – I just recently attended the Counter Terrorism Expo in DC and noticed some themes that were similar to DoDIIS in Denver, so I wanted to comment … Continue reading →

DoDIIS Worldwide Conference

Hello there!

I’m going to be heading out in a couple of days to attend the annual DoDIIS Worldwide Conference from April 1-4th. It’ll be in Denver, CO this year. For those of you who may be unfamiliar with the conference, DoDIIS stands for the Department of Defense Intelligence Information Systems. This year’s theme is going to be “Advancing Mission Integration” and will be focusing on how defense intelligence infrastructure and information can be shared to provide a more integrated approach to analyzing information. Sounds like the perfect place to talk about how data visualization can help sort through all the complex data to draw out key insights in the government and financial districts and to check out the different approaches used in the field.

Check out their website: http://www.ncsi.com/dodiis12/

I’ll be back with a trip report in a few days, so be sure to check back for updates soon!

New Blog Moderator

Hello Readers!
My name is Andrew Tom, and as previously mentioned, I’ll be taking over as the new blog moderator from now on. I’m excited about driving the blog as I will take over where Elizabeth left off, not only in terms of discussions on advanced visual analysis, but also to bring more attention to how different companies in the information value chain associated with Big Data can work together to provide more compelling experience for end users, as well as how they further the development of visual analysis technology for our development partners.

I’m eager to see the topics we’ll be covering and looking forward to the kinds of discussions we’ll be having!

Thank You!

Hello readers! I wanted to let you know that we will be having a new blog moderator at the Visual Insights blog. Please join me in welcoming Andrew Tom as the new moderator, who will be continuing the discussion on all things advanced visual analysis.

It has been great connecting with you all, and thank you for your continuing interest in Visual Insights.

- Elizabeth Hefner

The Impact of Big Data

The New York Times recently published an interesting article, The Age of Big Data, on the topic of the Big Data phenomenon and the opportunities that accompany the surge of information from new sources. In last week’s blog, we noted that a McKinsey Institute Study found that the demand for skilled data analysts will significantly increase, and the article discusses how this is just the beginning of the shift toward the need for more analytical understanding of Big Data.

“It’s a revolution,” says Gary King, director of Harvard’s Institute for Quantitative Social Science. “We’re really just getting under way. But the march of quantification, made possible by enormous new sources of data, will sweep through academia, business and government. There is no area that is going to be untouched.”

The article examines the many reasons and examples of why Big Data has an influence on fields such as business, government, and economics, and more importantly why decision-making will increasingly be based on data analysis. The article presents a compelling comparison of the impact of big data to the invention of the microscope. Erik Brynjolfsson, an economist at Massachusetts Institute of Technology’s Sloan School of Management, explains how the microscope was a dramatic change in measurement by enabling things to be viewed at the cellular level.

Data measurement, Professor Brynjolfsson explains, is the modern equivalent of the microscope. Google searches, Facebook posts and Twitter messages, for example, make it possible to measure behavior and sentiment in fine detail and as it happens.

Along with data measurement, being able to integrate, visualize, and analyze data will be the key to accurate and timely decision-making for organizations tackling Big Data. As we’ve previously discussed on this blog, Big Data has a significant impact on social network behaviors. The article presents a good example of the opportunities that Big Data presents in being able to effectively understand key patterns and relationships in increasing amounts of data from various sources and of different types.

Researchers can see patterns of influence and peaks in communication on a subject — by following trending hashtags on Twitter, for example. The online fishbowl is a window into the real-time behavior of huge numbers of people. “I look for hot spots in the data, an outbreak of activity that I need to understand,” says Jon Kleinberg, a professor at Cornell. “It’s something you can only do with Big Data.”

Overall, the article is a good discussion on the impact of Big Data, with real-world examples of how Big Data is influencing society and business and the effect it has on critical decision-making.

Building Integrated Insights from Big Data

As we’ve discussed on this blog in previous posts, the growth of Big Data – data that is increasing in volume, rapidly changing pace, and includes varieties of data such as structured and unstructured – presents organizations with new challenges as well as new opportunities. One of the challenges organizations face is that as the rate of data creation and consumption continues to accelerate, so will the demand for business data analysis professionals. As explained in the McKinsey Institute Study, the demand for analysts will largely surpass supply. However, in addition to the challenges, a key opportunity remains for organizations to better understand Big Data on another level – creating actionable, integrated insights across multiple data sources. Key drivers of the demand for leveraging Big Data to improve decision-making can be addressed through greater focus on self-service analysis powered with visualization, along with methods for building and analyzing insights from disparate data sources.

For instance, the importance of creating integrated insights can be seen in recent studies that have shown that activities in social networks involving key individuals correlate with information transfers and investment activities in security markets. For example, consider the activities of a portfolio manager. Portfolio managers place larger bets on firms they are connected to through their social network, and perform significantly better on these holdings and on holdings relative to their non-connected holdings. Not surprisingly, VC firms who are better networked also experience significantly better fund performance, as measured by the proportion of investments that are successfully exited through an IPO or a sale to another company.

Source: GigaOm

SNA is based in modeling relationships between groups of people. Social network analysis uses graph theoretic ideas and applies them with the premise that the structure of the graph can be used to understand and identify critical relationships and influential people. However, networks modeling real world relationships often have different characteristics compared to relationships used in SNA. For instance, to accurately “connect the dots” involving large quantities of diverse information and see patterns involving multiple aliases of people, different approaches are needed to handle the misleading, inaccurate, or incomplete information. One of the issues with traditional SNA is that the people and entities in a network are often not treated as adaptive and capable of taking action, learning, or altering their networks over time.

Recent advancements in network analysis involving complex network topologies with multiple relationships between nodes, network behavior that is based on uncertain information, and time-based change of networks, have enhanced the value of incorporating advanced network analysis techniques as a key part of an analytics toolset to aid in better understanding data relationships. More organizations are beginning to understand that with advanced visual analysis technology, they can build integrated insights across all of their available data, enabling them to better understand emerging opportunities and threats.

Using advanced analytics, businesses can study Big Data to understand the current state of their business and track customer behavior. Security and law-enforcement organizations can use advanced visual analysis to build integrated views of emerging threats from all available data sources to take timely action. The combination of advanced visualization techniques, together with social network analysis techniques, will help bridge the emerging gap between the vast amounts of available information in Big Data and the available resources to better understand them.

In the next series of blog contributions, we will continue to explore the incredible opportunities that Big Data represents, and new products and technologies that can contribute to better understanding and new insights.

Visual Analysis: Beyond Social Network Analysis

Recent market dynamics continue to show an endless growth of data at a pace approaching 50% per year with the volume of digital content projected to grow to 2.7 zettabytes (ZB) by the end of 2012. With over 90% of the growth in information involving unstructured and semi-structured types of data, the demand for new types of analytic tools that help better understand hidden information in Big Data continues to increase.

As it has been noted on other posts of this blog, graph-based visual analysis is a highly effective method for capturing and understanding relationships between data that are not quantitative in nature. This method and technology has been used in diverse fields such as intelligence and law enforcement to customer sentiment and network topology analysis to uncover hidden insights in growing data that was not possible when relying only on traditional analytics.

This interest in visual analysis is due in part to the significant amount of data that are from new sources that are automated – capturing attributes such as time and geospatial locations on a routine basis. The richness of this new information creates an opportunity for understanding not only the traditional topological relationships between entities (such as social networks), but also for better understanding the behavior of networks in terms of change and pace of change, along with how specific changes are related to different types of data. As a result, we have the opportunity to better understand key trends and anomalies better than at any time in the past.

The combination of rich data collection, advanced analytics operating across both structured and unstructured data, and efficiently storing and analyzing information in quantities unimagined just a few years back, have created a new era of data analysis in general and visual analysis in particular. We can now look at the networks representing relationships between data as not just static topologies, but rather as “dynamic networks” with their own behavioral pattern in terms of change, sequence of change, and uncertainties of change, combined with the ability to integrate information from complex event processing engines and other “event driven” information sources. These new developments promise to bring about a new dawn of information use, enabling smarter, timelier decision-making in various fields of human endeavor.