I read Seth Stephens-Davidowitz’s Everybody Lies: Big Data, New Data, and What the Internet Can Tell Is About Who We Really Are to the very end, which is how I saw these sentences:
I hope there is some young person reading this right now who is a bit confused on what she wants to do with her life. If you have a bit of statistical skill, an abundance of creativity, and curiosity, enter the data analysis business.
The lines imparted a heady rush. Dr. Stephens-Davidowitz was talking about me! I’m not young, but then, I’m used to facing ageism in the technology world, so I simply ignored that part and set off on my journey. I had my eye on the Holy Grail: I’d use big data to find The Truth.
Obviously, The Truth is too important to locate with a simple internet search — someone would have found it by now. I understood that I had to use more powerful tools. I turned to Google Correlate.
Google Correlate can find search patterns that correspond with “real-world trends.” A related tool is Google Flu Trend, which once estimated the rate of Flu and Dengue fever based on the rate at which society searched for the symptoms. The fact that Google Flu Trend is no longer active did not deter me. If we could trace Flu by the rate at which its symptoms are queried, I could certainly trace The Truth by the words we also search for on the days we are curious about such matters.
In this way, I discovered that searches for ‘The Truth’ correlate to searches for the words ‘kush weed’, ‘screen capture mac’, ‘credit suisse careers’, ‘you type’, ‘how do you type’, ‘to pdf converter’, ‘oh how he loves’, ‘怎么 说’ ‘statement template’ and ‘more?’
As I looked through the results, I felt excited. ‘Statement template,’ in particular, seemed to scream ‘you are going in the right direction!’ Only, I also saw the fallacy in my thinking. I should not be searching for terms that correlated to the search term ‘The Truth’. Rather, I should be looking for search terms that correlated to The Truth itself.
Fortunately, I have a bit of statistical skill, an abundance of creativity, and curiosity and I knew what I had to do. I would take The Truth and feed it into Google Correlate via its ‘Enter your own data’ option:
I didn’t want to confuse Google Correlate by giving it two points for the same day, so I took the part of The Truth that I could capture with a single forward-moving line, converted the line into a series of points using WebPlotDigitizer and then popped that The Truth into Google Correlate.
PRESTO! That’s all it took, folks. Here are my findings:
Come! Let’s take an even deeper dive into the truth together.
DIVING DEEPER INTO THE TRUTH, A DEEPER DIVE
According to the Washington Post, Mary Anne Huntsman is an accomplished pianist who married a guy who wore gym shorts on their first date. Bazaar reports that at her wedding, guests enjoyed a fennel and orange salad, followed by a choice of either a filet or salmon.
Are you thinking what I’m thinking? That’s right! The Truth must have something to do with food. In fact, were I not a great big data scientist, I would have simply taken the top two results and made this claim. But no, I continued to dive deeper.
Pat Garrett and Billy the Kid is a movie directed by a man who, after discovering that his first week or footage was out of focus due to an “easily fixable lens malfunction…stood up on a chair and urinated all over the screen.”
One man down is also a movie. IMDB says this about it: “Two misplaced soldiers during WWII. They must unite to fight for their lives.” Clearly, The Truth ALSO has to do with movies, and perhaps even men who urinate on the screen.
nfl playoffs 2012. Apparently, this is some sort of tournament. I would have spent more time on this bit of The Truth, but I noticed that three of the remaining search terms were very similar and I turned to them instead: concordia accident, costa concordia accident and costa concordia sinking. According to Wikipedia, this was a ship wreck. Ship wreck resonates with what I’ve been feeling is true about the current administration, but don’t let me bias your thinking. I will, however, note that this result came up THREE times.
Wikipedia tells me that Ken Still was a professional golfer, and what does the pig say ended up being a youtube video about racism. But racism, too, is a part of The Truth, which we now know, thanks to big data.
Thanks to you as well, dear reader, for joining me on this journey.
May the truth be with you.