The art of visual inspection: Spot the hidden story in your charts

You have a nice set of data, but how do you convey its meaning to an audience?

In the latest of a series of free technology talks, the Association for Computing Machinery presented noted researcher and author Angelica Lo Duca to talk about “How to Extract Meaningful Insights from Data.”

If the topic seems hopelessly broad for a one-hour tutorial, you’ll be pleasantly surprised by how effectively she boils it down to a few key principles, ones that could be used to give a narrative to any set of data.

Lo Duca offers several “heuristic techniques” to pull the insight from data. For this talk, she did not worry about the statistical unpinning; rather, she focused on what can be learned from eyeballing a chart (“visual inspection”). She developed these techniques by teaching data journalism students, she told the virtual audience.

Lo Duca is a researcher for Italy’s Institute of Informatics and Telematics of the National Research Council, and author of “Become a Great Data Storyteller: Learn How You Can Drive Change with Data” as well as the upcoming “Learning Generative AI Tools for Excel: Speed Up Your Everyday Tasks with Microsoft Excel, Copilot, ChatGPT, and Beyond.”

Any discovery you make in your data is an insight, she said. Data can contain multiple insights, depending on the analysis you perform. But when thinking about these things, you want to focus on one insight at a time.

She offered ways to analyze data in three formats: temporal, location and categorical.

Temporal analysis

Temporal analysis studies data patterns through time. On a simple X/Y analysis, a period of time can elapse along the horizontal Y axis  while an aggregation of all the events at a certain point of time are captured on the X axis:

The chart shows the number of times something happens each year, with a peak number of these events taking place in 2021.

Here, the insight comes not from the peak point, however, but from the range between the points where the events started increasing each year (2014), and the year they started declining (2021). That’s where the story lies.

One approach would be to explain what happened at the starting event that kicked off this trend, and then explain what happened at the ending event to bring the trend to a close.

But to find this story, you usually have to look outside the data itself, Lo Duca advised. The explanation can be in the public sphere, such as a news item of interest that took place in these inflection points. Or it could be an event that happened internally to the company itself.

For example, in the chart above, the data may have been about product sales. So, 2016 may have been the start of a new sales campaign, and 2021 was the year that a new competitor entered the market.

The same analysis can be done with a negative peak, or the lowest point in a chart.

If there are no peaks, then compare the starting and ending points of the line, calculate the difference between the two. There’s the story; The product may have an 186% increase in sales, or 50% decrease.

Everything is under control

But what if nothing is happening to your data? Your temporal line is neither on the ascension nor the descension? What if it is just bouncing around?

There’s a story there, too.

In these cases, Lo Duca advised, you need to set up a “threshold” either above or below all the data.

If the data line is under the threshold, then the insight is that the “situation is under control.”

Logging systems operate by this idea, with the user being alerted only when the data line breaks over the threshold. The situation is no longer under control.

Over the threshold? The situation remains out of control.

Spatial analysis

With spatial analysis, you are plotting points in space, rather than in time.

While temporal analysis looks for changes over time, spatial analysis seeks how data moves in space. It’s the same logic, but in a different dimension.  Both look for peaks, tends and stability.

In an example, Lo Duca showed a map of Europe dotted with data points.

Locations with many points could be called “hot spots.”

Insights can be gained by comparing different locations. Why does Italy have more points than France or Germany? What’s the story there?

You can also find insight in comparing neighboring regions. Why does Italy have this trait, but Spain not so much?

Locations with other commonalities can be compared as well, she said: “If I’m analyzing weather, I can look at locations with the same climate.”

For maps without hotspots, you can look for gradients, which highlight a trend in a particular direction.

A gradient can be drawn through Europe, from south to north, which will then allow you to measure the extremes. (In these cases, make sure all the other variables associated with areas being compared, such as time of the data sampling, are relatively equal.)

A gradient through Europe.

No gradients? No problem. Use your thresholds, as described earlier. Thresholds can be derived from historical model data, defined by laws or standards, or some other external factor.

Comparing categories

Another form of data comparison is that of categorization. Different types of a thing can produce different data points, and those can be compared.

An insight, then, may come from looking at the leading category, and seeing how far ahead it is from the others.

A group of categories, with a clear leading category.

In other cases, you may not have a dramatically leading category. But here, you can gain insight from looking at how the categories are distributed. What are the numerical differences between each category?  Are there a set of categories that exceed the values of the rest? Perhaps the first five categories have 90% of the values. There’s an insight.

Or, you can also use your old friend the threshold,  to separate categories that are doing well, for instance, from those that aren’t doing so well.

Other tricks

Temporal, spatial and categorical were the three types of analysis that Lo Duca concentrated on for her talk. Once she explained them, you could see how similar they are:

In the talk, she also discussed a few other forms of analysis and what to do with anomalies. Like all ACM Tech Talks, this one is free, so give the entire talk a listen if you want to learn more.

And, if you want to get fancy with your stories, also check out the talk Lo Duca gave earlier this year, “Applying Cinematic Techniques to Data Storytelling.”

The post The art of visual inspection: Spot the hidden story in your charts appeared first on The New Stack.