By Per Pettersen on September 23, 2014
The advent of Big Data technologies has massively expanded the amount of data available and the associated analytics to an almost paralyzing level. To process the data and present it in a meaningful way requires highly sophisticated “Big Data” technology and analytics tools. With all this potentially valuable data, one question on my mind is what role good old common sense plays in analyzing and making decisions? What if the data tells you something that completely counters your intuition?
As a machine learning and Big Data fanatic, I believe that common sense should be combined with facts in making decisions. One of the first published “Big Data” discoveries was the suggestion that diapers located next to beer at 7 Eleven would be the ultimate pairing. Young fathers often are asked to bring diapers home and when there is a conveniently placed six pack, it is an easy decision (for some) to grab one of those as well. This is not an obvious combination of products — and that’s what makes it a good example of how machine learning can help uncover things that the normal human brain might find counter intuitive.
That said, I have seen people come to conclusions from Big Data and take for granted that they are right without going through a common sense step. If common sense tells you the numbers can’t be right, there is a good chance they are not. Common sense and data science can work together in tandem. Common sense is the control mechanism that keeps data science in check. Number crunching without the safety net of common sense can be worse than any data at all!
Common sense can also help guide the modeling and hypotheses analysis is supposed to test. Which data points are most likely relevant? Which modeling methodology makes sense? Where do we expect there to causal relationships? All of these are common sense inputs to the modeling and data science processes.
Good luck with your Big Data analytics efforts and make sure you don’t leave your common sense at the door!