PreInvented Wheel

On the shoulders of giants

Python time series plotting

Whether with matplotlib or other python libraries, every article you need about data visualization

  • Stress-Free Reporting
  • Driving Organizational Change
  • Insightful Analytics
  • Quick Tips
  • Incomplete Articles
  • About

Let’s Stop Using The Phrase “Analysis Paralysis” – A Mathematical Explanation

I have come to hate the phrase “analysis paralysis”. Often used by people who are tired of thinking hard, as an excuse to end the discussion. They smugly throw it out with the implication that you are avoiding virtuous hard work with your insistence on a plausible plan. I would argue that in many cases, creating a plausible plan is hard work. It’s work that is easy to look down upon, because the results aren’t as clear. But if you remember that the vast majority of projects go over time and over budget, you can begin to appreciate the value of proper planning. I think one of the reasons the plane gets a bad rap, is that people assume that it will be perfect the first time around. Look at every thing you do in your organization, what do you do perfectly the first time around? Nothing! And that’s okay, as long as you quickly identify when you’re getting off track, and course correct. Planning is the exact same way, not something you do in a blank room before starting, and then blame when reality doesn’t match up. Your plan should include checkpoints for whether you’re still on track, it should include explicit preliminary stages where you gathering more information and testing assumptions, and it should include time for unforeseen circumstances.

It seems like a paradox plan for unforeseen circumstances, the basic recollection of the laws of statistics should help us understand how this can happen. You may remember a bell curve, also know as the normal distribution, also known as the guassian distribution. You can tell it’s important because so many people decided that it needed a name! A normal distribution happens when a large number of random variables are added together. For example, there are 100 tiny little steps that need to happen for your project to succeed, each of which will take a variable amount of time, the sum total of their durations will be a normal distribution. There is, one exception to this tantalizingly universal rule. If one of the steps that you’re adding together could take an amount of time defined by a power law, it has the potential to completely blow your estimate of the water. A power law is when the probability of it taking 10 times as long is not that much higher than the probability that taking 100 times as long or even 1000 times as long as average. Power laws give rise to what is known as the 80/20 principle, where 80% of the time is used to get 20% of the results. In practice, that 80% of the time is often the most frustrating part of your project, where you don’t even know what a proper result would look like. It’s often the least well-defined part of your project, where you’re thrown into a completely unknown situation that you just don’t know how to predict.

Using normal distributions for project planning
Using normal distributions for project planning

How can we make a plan when any one of our stuff to blow up to completely dominate the global timeline? The answer to this question strikes at the heart of why unsophisticated analysis and planning gives the whole process a bad name. What you need to do is identify these risk factors, these steps in their plan that won’t just get 10% better or 10% worse than expected, but which could be 10 times better or 10 times worse. Then you need to find a way to quickly and cheaply test your expectations, and make the truth reveal itself before you committed to a final deadline. This is what exploratory phases are for, and almost any nontrivial project should have one. This is where you reduce the unknown unknowns to known unknowns, with acceptable variances.

Power Law (80/20) Distribution for planning

With this perspective, you can see how people who objects to planning taken over simplistic view of what that process looks like. Of course you won’t be able to sit down with a blank piece of paper and predict the future, the planning isn’t just about thinking and calculating. It’s about designing tests and implementing them. It’s about taking that data, and understanding its implications for the project overall. In many cases, a preliminary experiment will highlight the need for another experiment, which also needs to be designed, executed and analyzed. An impatient team member may complain that you’re not making progress on your overall goal, but those people are forgetting that time spent in understanding the largest unknowns of your project are an investment that will pay off many times over in just the next few weeks. It is always cheaper, faster, and less demoralizing to have a experiment which does not go as planned and have a project that comes in overtime, over budget and with a burned-out team.

So be a leader, one who is brave enough to say “follow me, I don’t know the answers — but together we will be able to find them!”

Need Business Intelligence and Data Science consulting?

* indicates required

© Copyright 2016 PreInvented Wheel · All Rights Reserved