Data Experience

Do people believe in the data more if they are holding something in their hand? Do they, literally, give it more weight? According to recent Dutch research, it just might.

My interest was piqued by the recent episode of the Data Stories podcast (interestingly, they are back to being sponsored by Qlik again) where they interviewed Dani Llugany Pearson from Domestic Data Streamers.

Domestic Data Streamers create some wonderful installations, transforming data into art. These are installations that people can interact with and influence by adding to the data. It is a really marvelous concept. People can see, walk around, touch, and engage with data. Dani described it as an Info Experience as opposed to a static info graphic.

During the discussion, some recent research from the Netherlands was mentioned and they kindly shared the link to the research in the show notes. This research, performed by Nils B Jostmann, Daniël Lakens and Thomas Schubert, shows that people holding a heavier weight will effect cognition and lead people to assign more importance. It is an intriguing idea and you can read the research here:

    Weight as an embodiment of importance

So, if we attach more importance to more weight, do we see data visualizations on an iPad as being more important than the same data on an iPhone? Intriguing! How does that data seem on a desktop computer, where the only weight is in the mouse that we slide it across the desk?

Weight, of course, is only one facet of our data experience. The visuals must be important too, just like the research that shows that people eat less when they can't see food, they just don't enjoy it as much:

    Vision and eating behavior. (2002. Linné Y, Barkeling B, Rössner S, Rooth P.)

How about being able to touch and interact with the data? How does that make me feel about it?

In their 2011 paper, David Spiegelhalter et al discuss the ethical imperative to provide transparent information. This is because when we build dashboards for other people, "the desired outcome must be considered from the start" - we have to think about what we are trying to present before we design the dashboard. We are more persuading than informing where we should be more informing:

    Visualizing Uncertainty About the Future (2011. Spiegelhalter, Pearson, Short)

So, a visualization tool that I can hold in my hand and feel the importance of the data, that looks good enough to eat, and allows me to inform myself rather than being persuaded by someone else would be the ideal Data Experience. I wonder where I could get one of those?

---

Stephen Redmond is a Data Visualization professional. He is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
 

Crime and employment - Color as a trend

I was playing around with Qlik Cloud again. I grabbed some data from the Irish Central Statistics Office on employment statistics in Ireland and reported crime - both over a period from Q1 2003 up to, and including, Q1 2015.

It was easy enough to upload the data to the cloud and then load it into the application. A scatter chart seemed to be a good choice to see if there is any correlation between the two sets of data:

My first impression of the correlation is that there doesn't appear to be one. Perhaps there is a small negative correlation that crime incidents will reduce as unemployment increases (which is not what I would have thought before looking at the data).

The second thing that I thought about was that I couldn't quickly see the trend of the data in this chart. Now, I know that a line chart would be more useful for seeing any trend, but I felt that, with each dot on the scatter representing a calendar quarter, it would be useful to discern the trend in this chart. I thought that it might be useful to use color to achieve this.

I had a field called QuarterNumber, which is a 5 digit number representing the four digit year and the quarter number from 1 to 4. I created this color expression:

  ColorMix1(
    (QuarterNumber-Min(Total QuarterNumber))/
    (Max(Total QuarterNumber)-Min(Total QuarterNumber)),
    LightGray(),
    LightBlue()
  )

This results in a chart in which it is very easy to see where in time a point exists. This, I believe, adds more clarity to the chart:



As to the analysis, it appears that the negative correlation is true for many crimes, and that shows up in the overall figure. However, crimes that people are quite concerned about - theft, burglary and robbery - show a different correlation:

With those three types selected, there is a distinct positive correlation.

The good news for the Irish is that the highest quarter - Q4 of 2014 - was only 29,088 of such incidents. This is a very low percentage and Ireland is actually one of the safest countries in the world to live.

---

Stephen Redmond is a Data Visualization professional. He is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
 

Low down on Qlik Cloud 2.0

Qlik Cloud was launched along with Qlik Sense 1.0 at the end of last year. It allowed users of Qlik Sense Desktop to upload a document to the cloud and then share that application with other users. There were limitations. For example, the number of users that you could share with was limited to 3. Also, once they were up there, the applications could not be edited.

With the launch of Qlik Sense 2.0, we were promised new functionality, and this appears to have been made available in the last couple of days.

Now, as well as uploading applications that have been created from Qlik Sense Desktop, we can also upload files and create brand new applications directly on the service. Not only that, but we can make use of Qlik's new DataMarket to bring in curated data sources such as demographics, currencies and weather.

The number of users that we can share applications with has been increased to 5. But we have a brand new feature in that we can choose to share individual charts from these applications on our blogs and other media - like this:



Creating new applications is very straightforward. First, we need to provide some data (Excel, CSV, etc.):

We can now start to create new applications in our personal cloud:

When we create the application, we can choose to load in the data that we have uploaded:

And Qlik will parse it out for us:

Or we can choose to get data from the Data Market:

We can bring in multiple data sources and Qlik provides a profiler to suggest the correct data links.

Once we have loaded the data successfully, we can start creating content with the drag/drop interfaces:

When we have created a chart, there is a right-click (or tap-and-hold) option to share it. We can get a link that can be used to share via email, social media, etc., or an embed link to share via blogs and other web pages (as I have used above):

All very, very easy!

Part of the success of Tableau Software is that they have had Tableau Public, where users can create content and share it for free. Only time will tell whether the Qlik Cloud solution will challenge that, but not having to have an installed Windows application will certainly be interesting for many people.

So, that is the down-low on Qlik Cloud - don't keep it on the low down. Time for you to go and play and start creating data applications in the cloud!

---

Stephen Redmond is a Data Visualization professional. He is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
 

Invisible servants

The King woke in the morning and stretched. The sun streamed through the opened curtains.

He eased out of bed straight into his waiting slippers and dressing gown.

He made his way into his bathroom and eased himself into the bath - which was at just the right temperature for him.

After he dried off, using perfectly warmed towels, he made his way to his dressing room to don his pressed trousers and freshly ironed shirt.

He wandered down to the breakfast room to sit down in front of the exact breakfast that he wanted. Laid beside his breakfast plate was the days letters. Of course, only the ones of interest to him were there, there was no junk.

---

A lovely story, but what is missing? Of course, it is the servants. The servant who opened the curtains at the right time. The servant who laid out the slippers and dressing gown. The servant who prepared the bath at exactly the right temperature.

What on earth has this got to do with data visualization???

It has everything to do with data visualization! Your data visualization needs to be the servants. Your users are the King. The servants should present the right data in the right way so that the King doesn't even realize that the servants are there!

Remember your ink-to-data ratio and let the data dominate.

Let your servants fade into the background and crown your users.

---

Stephen Redmond is a Data Visualization professional. He is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
 

Are you answering the right question?

I bring before you the story of statistician, Abraham Wald.

During World War II, Wald was part of a team looking at the problem of bomber loss and to consider how they should reinforce the planes to better protect them. The problem was proposed that they should look at the frequency of damage sustained by returning bombers and to use that information to make recommendations on where the plane should be reinforced.

Wald's brilliant insight was to turn the problem on its head. He suggested that the places where the returning planes were being damaged most frequently were the places where those planes could actually sustain damage and, mostly, successfully return to base. What the question should really be is where the planes that weren't returning were being damaged which meant that they failed to return!

It seems so simple when you think about it, but sometimes we are so sure that we are looking at the problem the right way, that new insight that tells us that we are looking at it completely wrong is not always well received. But we should receive it and we should look at it and only dismiss it if we can logically decide that it should be dismissed.

So, think outside the box and solve this problem:

Here is a pattern of 9 dots, arranged in a 3x3 grid:

Now, I want you to connect the pattern of 9 dots using four straight lines drawn without lifting the pen from the paper or retracing any lines. Simple, eh?

Please don't post solutions below. If you discover it, just be happy that you have done so and feel good that you have thought differently and bring that skill to your daily work.

---

Stephen Redmond is a Data Visualization professional. He is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
 

Pick the right chart to tell the right story

A friend of mine recently shared the following chart on LinkedIn:

I believe that it may originate from the Centre for Learning and Teaching in Hong Kong and may have been a student project. I wonder what the grade was?

Unfortunately, this chart is not a great way to represent the data from a purist dataviz point of view. Polar charts like this are often a poor choice for most uses, in a very similar way to pie charts with lots of segments. In this case, the scales are all over the place so it is really difficult to use the graphic to tell me where the segments with the largest values are. For example, the largest segment seems to represent the 4 million Google searches which, by area, appears to be twice as big as the 3.3 million Facebook posts and it is also much bigger than the 50 billion Whatsapp messages! The 41,000 Instagram photos segment is bigger than the segment of 215,000 for the previous year! It is also way bigger than the 1.4million Skype calls right beside it. I can't really trust the size or position of any of the segments to relate any information to me.

The earliest recorded case of a Polar chart was the one created by Florence Nightingale in 1858 to demonstrate the causes of mortality in and around the Crimean campaign:

This was a revolutionary chart in its day. It was very easy to see that the largest cause of death to British servicemen was actually from preventable disease rather than from wounds or other causes of death. But, while we can see the story from the sheer amount of blue ink on the chart, can we discern anything else - is there any pattern or trend. I wonder if this is the best way to visually represent the data.

QlikView doesn't have a polar chart but we can approximate one with a radar chart:

Still not idea. We are still looking at lots of blue, but no trend. Perhaps a stacked bar chart would be useful?

This is good because we can see the overall trend. Although, one of the problems that stacked bar charts have is that we can only clearly discern the trend of either the whole or the bottom bar of the stack (Wounds). We can have difficulty seeing the trend of the middle (Other) or top (Disease) bars.

Another way might be to try a Redmond Profile chart:

The profile chart shows us the trend of the whole and then shows the % split of each of the parts. This is useful but maybe still not ideal for this data-set. There is a variation of the Redmond Profile chart that might work here:

In this case, the profile bars, instead of showing the % share of the values will show the actual value. In this way, we get to see the trend for the whole as well as the parts in a way that the stacked bar chart doesn't do.

If I was a gourmet chef, I might call it a deconstructed stacked bar chart.

There you go, gourmet data visualization.

---

Stephen Redmond is a Data Visualization professional. He is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
 

Poacher turned gamekeeper

Today is a day of mixed feelings. I am excited because on Monday I will start a new career away from consulting. I am also sad because I am leaving behind a company that I have worked for since 1999 - over 16 years!

The last 16 years have been quite a rollercoaster and I have met, worked with, and drank with, some really great people. I have interacted online with so many other nice people who I have yet to have had the pleasure of meeting face-to-face. I have got to travel far and wide - from Seattle to Seoul - and I have attended some great events - from the old SalesLogix partner events in the early days to the Qlik Qonnections events in more recent years.

I need to give special thanks to all my colleagues at Capventis over the years. All of them have helped me grow by challenging me to be the best that I could. It has been my pleasure to have worked with an incredible bunch of smart people.

I am leaving a much bigger and stronger organisation than the one that I joined so many years ago. I know that they will continue to grow and continue to succeed into the future.

---

Stephen Redmond is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
 

Data Preparation for Qlik Sense

Today, Capventis have made my latest book, Data Preparation for Qlik Sense Desktop using Pentaho Kettle, available for free download:

So, what is data preparation and why does anyone need it?

For those of us that have some expertise in QlikView and Qlik Sense development, we probably don't need to worry about this at all. This is because we already do our data preparation using the Qlik script. All of the data loading, joining, mapping that we do is all data preparation. A lot of us are really very good at using the script to manipulate data to meet the needs of business users.

There are, however, many potential users of QlikView and Qlik Sense who are not adept at scripting. To even tell them that they need to use script to load data will make them turn and run! But they are certainly happy to drag and drop files from one place to another and can handle setting properties in dialog.

For that population, the new feature in Qlik Sense Desktop of being able to drag desktop data-sources into an application makes it really easy to create the self-service analyses that they need to create. But that feature - even with announced changes to the data loader in Qlik Sense 2.0 - cannot really handle more complex loading and transformations, we need to start thinking of the script again.

That is where graphical data preparation tools come in. They enable business users to perform those more complex load and transforms in a graphical environment without having to learn any scripting. They can output a single file that can be dropped into a Qlik Sense app.

There are several Data Preparation tools on the market that have working plugins to extract data into QVX format that can be read into QlikView or Qlik Sense. Leading tools such as Lavastorm and Alteryx will also have server based options and integrations to advanced analytics engines like R.

I went for Pentaho Data Integration (PDI/Kettle) for this project because it is open source and the Community Edition is free - just like Qlik Sense Desktop. Once you have some experience with one, it makes it easier to transition to another. PDI doesn't have an out-of-box output to QVX, but output to Excel is usually good enough for most business users. For the more technical amongst you, there is Ralf Becher's excellent solution to stream data from Pentaho Kettle into QlikView via JDBC.

The eBook is about 80 pages and comes with support files to help you try out the exercises. Feel free to download it now.

---

Stephen Redmond is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
He is CTO of CapricornVentis a Qlik Elite Partner.
 

Explaining Pie-Gauges

Back in December 2013, I discussed different KPI approaches and introduced the Pie-Gauge as a form of representation. I used them in the dashboard of the winning app in the recent Qlik UK partner competition:

Last month, I described how happy I was that QlikView 11.2 SR10 was released so that we could change the segment color in a Pie chart - making Pie-Gauges look better.

Prior to that, and indeed in my entry to the partner app competition, I had used an extension object for creating Pie-Gauges. This is published on Branch.

So, what, exactly, is a Pie-Gauge anyway? It is a gauge because it is representing a KPI - one value versus a target value. However it solves one issue with gauges in that I don't ever have to worry about the scale. This is especially good when I may have several KPIs. Pie-gauges are all about ratios.

Like all pie charts, it is a part-to-whole comparison. In this case, there are always three segments, two of which are mutually exclusive, making up the whole.

The first segment represents the amount by which we have fallen short of the target. The second segment represents the lower of the target value or the actual value. The third segment represents the amount by which we have exceeded the target.

We can see that the whole is therefore the higher of the target or the actual value. We can also see that the first and third segments cannot exist together - we can't fall short and exceed the target at the same time.

The positioning of the segments is important. The first segment must be to the left of the top of the pie, and the third segment must be to the right - signifying below and above target.

The great thing about these gauges is that they will work no matter by how much we have fallen short or exceeded the target - they are always a part-to-whole comparison. Unlike gauges with fixed axes, they will just work.

The really important thing to grasp is that the actual % above or below target is not important!  It is the representation of the ratio that is important. It is whether we are above or below target, not by how much, that we are representing.

The three values can be very easily calculated using Qlik's RangeMin and RangeMax functions.

The first segment is:

RangeMax(Sum(Actual)-Sum(Target),0)

The second segment is:

RangeMin(Sum(Actual),Sum(Target))

The third segment is:

RangeMax(Sum(Target)-Sum(Actual),0)

In QlikView, we can have these as three separate expressions in a Pie chart. In Qlik Sense, we can only have one expression so I use a ValueList dimension and an expression like this:

If(ValueList(' ', '  ', '   ')=' ',
RangeMax(Sum(Actual)-Sum(Target),0)+0.001,
If(ValueList(' ', '  ', '   ')='  ',
RangeMin(Sum(Actual), Sum(Target))+0.001,
RangeMax(Sum(Target)-Sum(Actual),0)+0.001
))

Note the +0.001 on each - that stops Qlik Sense displaying the "chart contains zeros" message. The spaces in the value list are there just to stop additional text being displayed on the Pie.

The color can then be calculated like this:

If(ValueList(' ', '  ', '   ')=' ',
LightBlue(),
If(ValueList(' ', '  ', '   ')='  ',
RGB(240,240,240), LightGreen()
))

Have fun with Pie-gauges.

---

Stephen Redmond is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
He is CTO of CapricornVentis a Qlik Elite Partner.
 

The vaccine effect

Sparked by Alberto Cairo's tweet sharing a blog on Recreating a famous visualisation, I decided that I would go onto the Project Tycho website and grab the data myself to play with in Qlik Sense.

It was, of course, fairly quick to import an Excel file into Qlik Sense Desktop, adding a CrossTable command to break out the data by State.  First visualization, one that I usually turn to to see what I have, was the bar chart:

By adding a color expression to highlight bars up to 1963, when the measles vaccine was introduced, versus those after 1963, the data jumped out very quickly.

I used the new pivot table in Qlik Sense to recreate the WSJ heatmap, along with similar color rules to those used by Mick Watson:

I also decided to have a look at the geographic spread, both before and after the vaccine:

Vaccines are really amazing. Just to think that just over 50 years ago, people were dying from a disease that, for most of us now just doesn't exist.

Worth thinking about.

---

Stephen Redmond is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
He is CTO of CapricornVentis a Qlik Elite Partner.
 

Do your Apps win Michelin Stars? Do they need to?

Back in 2012, at the TED@SXSWi event in Austin, Texas, JP Rangaswami, then Chief Scientist at SalesForce, now Chief Data Officer at Deutsche Bank, challenged us to consider that information is food.  It is an interesting analogy. I especially liked the "Supersize Me" suggestion of having to watch Fox News for 30 days.

So, if information was food - what would you do differently?

If information was food, what kind of data visualization would you like to see?

Would you be happy with the daily stodge? Not too pretty to look at, and you are not 100% sure of where the ingredients come from (and you have only been ill a few times!). Perhaps of the variety sold to the residents of Ankh-Morpork by CMOT Dibbler?

Or would you be looking for the time-consuming, detail-attentive, incredibly beautiful and incredibly expensive, Michelin starred fare?

Or is it somewhere inbetween?

The reality is, boringly, that it doesn't matter how pretty nor how ugly the presentation layer is if the ingredients are suspect. As I said in a post last year, good governance prevents people from getting food poisoning.

The first step is to get the ingredients right (or as right as we possibly can!) - then we can focus on the presentation. And focus we must. A plate of great ingredients just mashed together will not encourage our diners to return. We mush present our ingredients as best we can with the tools that we have available and then the foodies will keep coming back.

You never know, one day there may be a Michelin judge with them.

---

Stephen Redmond is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
He is CTO of CapricornVentis a Qlik Elite Partner.
 

Segment color in pie chart

QlikView 11.2 SR10 includes one new feature change:

"Now possible to define color of the lines around the sectors in a Pie chart. In the properties Color tab of the chart, you can now find the Sector outline property. Please note that calculated colors do not work for this setting."

I know what you are thinking - "Yay!  Sector outline!"

Ok, ok, I know it is not the most fabulous thing to happen to QlikView since it was created, but it does help me with one area: Pie-Gauges.

When I first proposed the Pie-Gauge, it was difficult to get it looking really nice with the native Pie control:

For me, the black outlines detracted from the visualization.  So much so, that I ended up creating an extension object to do the job!

This new change in SR10 means that I can make one properties change, and the same visualization now looks way better.

We could argue for hours about whether this is more or less effective than the same gauge, but it is nice to see new features like this creep into the product.

---

Stephen Redmond is author of Mastering QlikView, QlikView Server and Publisher and the QlikView for Developer's Cookbook
He is CTO of CapricornVentis a Qlik Elite Partner.