Deliver on time across your whole organization with 70% Engineering Estimates
 
 

The best algorithm for prioritizing work is value/cost. Long term I want to help your organization make clear and trustworthy estimates of both value and cost. We’re going to start with cost, because that’s the easy win.

Industry-wide only 16% of software projects ship on time and to spec. This hopelessly optimistic convention can stand, because few engineering organizations measure and have explicit expectations for their on time delivery rate. 70% goals (goals you expect to hit 70% of the time) are the standard for annual goals at leading companies like Google, Amazon, Airbnb, and Lyft. [1] Adopting a similar convention for engineering estimates as a whole is a crucial step towards moving your organization out of the tangled web of pain and confusion that chronic unreliability causes. In this post:

 
  1. I’ll show that deliberate practice with probabilities is an easy step towards making better bets across the board.
  2. I’ll explain why using probabilities can quickly make your engineering estimates dramatically clearer.
  3. I’ll show why every engineer is incentivized to work on making their estimates clear and reliable.
  4. I’ll show that a 20% improvement in identifying upfront which projects are destined to be failures based on cost is tractable (they were going to take so long that the organization would regret starting them if it’d known the true cost).
  5. I’ll provide a blueprint for delivering on time across your whole organization that any engineer can deploy.

 

 

Better bets through deliberate practice with probabilities

My favorite example of a good bet is when Jeff Bezos started Amazon even though he told his parents the company had a 70% chance of completely failing. He still thought the potentially massive upside made it a good investment, and he was confident in his abilities. He expected the average internet startup had a 90% chance of total failure.

Bezo’s has framed the problem in the right way, but there’s a big if. Can we trust his numbers? Amazon was a good investment if and only if the probability of massive success meant it actually had the high expected value he claimed. What would it take for us to trust Bezos, our fearless leaders, and ourselves when we made such bets? Some skepticism here is totally reasonable, because when the average person Douglas Hubbard tested was 90% sure of something it only corresponded to 55% in reality (author of How to Measure Anything).

Well, the good news is that in about 3 hours of training Hubbard found he could “calibrate” the vast majority of people. Calibrated people have the “correct amount of confidence,” things they say have a 90% chance happen approximately 9 times out of 10. Calibration training consists of trying to rapidly predict past outcomes with feedback. It is the deliberate practice for judgment based predictions. Poker players, weathermen, and other people who regularly get feedback on their predictions are mostly calibrated.

Neglecting probability or scope entirely is an even larger impediment to making good bets than being overconfident. People are often indifferent to a 70x or larger difference in expected value when making decisions in a variety of domains including avoiding electric shocks and saving birds from drowning in oil. Clearly a 99% chance of a painful shock is much much worse than a 1% chance of the shock, but subjects were only willing to pay 1.2x more to avoid the 99% electric shock than the 1% check  (Not 99x more as you’d expect). Biases of this magnitude are problematic if you want successfully make big bets like Bezos.

Luckily, Hubbard found that once people were calibrated their hesitations for using probabilities in decision making vanished. The training makes probabilities something tangible, something real, something people can rely on. So calibration training makes it harder to ignore probabilities, which in turn makes it harder to ignore scope.

And I’ve made the UX good. At Twitch I made calibration training 97% of staff would recommend to a colleague. Ok, so given people like the training that much, getting everyone in your org calibrated should be pretty tractable. Once your colleagues are proficient in using probabilities to communicate their expectations. Where should they start making predictions in practice?

 

 

Introducing 70% estimates: quick and clear

When you say you’ll “probably” ship something this quarter, it’s like saying there’s a 20-90% chance you’ll ship it this quarter.

sherman_kent_probability_range.png

Literally, the person you’re talking to will take your words to mean any point on the spectrum of wildly optimistic (20%) to promise (90%). It sounds totally ridiculous when you put it like that, but that’s the range people give when asked what “probably” means in terms of a probability.

It doesn’t take long to get a much clearer understanding of whether someone actually is on schedule. This is what a typical conversation looks like when I help an engineer make their first 70% estimate.

Engineer: I just started on the new login flow and I’m supposed to ship within a month.
Me: How surprised would you be if it wasn’t done in a month?
Engineer: Actually not that surprised. Some bugs came up on a past project, and similar projects have taken 4-6 weeks in the past.
Me: Sounds like there is like a 30% chance you finish it this month.
Engineer: yeah.
Me: When are you 70% confident it’ll be done by?
Engineer: I feel 70% confident I can deliver it within two months.
Me: That sounds right. You should update the team.

I generally extract 70% predictions from engineers for the first time in less than two minutes. An obvious solution to a critical problem, when to update your stakeholders, falls out of 70% estimates. If you now have a meaningfully different probability, say 40% that you’ll hit your date, let people know. When engineering risk isn’t explicit, it often doesn’t make sense to tell people you’re late until you’ve got no hope of being on time. If everything is a promise, why break your promise before you absolutely have to? Another thing I really like about 70% estimates for how much effort a project will be is that you can add them, because 70% cost estimates are close to the average time a project will take. For example: a 70% estimate for front end work + 70% estimate for backend  work ≈ estimate for engineering time on overall project. [2]

The right solution here is to get comfortable with numbers. Set the convention that all engineering imply a 70% chance of being completed before the specified date unless stated otherwise, you can get a quick and dramatic improvement in your communication. I recommend setting the same convention for cost estimates (how many weeks of work would this be for me type estimates). If the organization keeps track of these predictions it provides a strong incentive over time to be accurate rather than optimistic. Calibration training unlocks this much clearer, saner world.

Once you are calibrated, a 70% estimate is just the clearest fasted way to communicate. Before you are calibrated, 70% estimates are very misleading in an extremely precise way. If you’ve ever tried to change an organization or listened to any economists you know that an elegant solution is not enough. What incentive is there for individual engineers to improve the norms here?

 

 

5 reasons why engineers should want to make good estimates

Joel Spolsky (founder of StackOverflow and Trello) goes so far as to say any system where management writes a schedule and hands it off to programmers is doomed to fail.” That’s because the engineers implementing the work have the best information on cost, which is critical to prioritization. Here’s how I convinced hundreds of engineers at Twitch to care about making good estimates.

  1. It’s on the path to promotion. Ok to be clear I’m way more about empowerment and creating value than “career development.” But I want to shake engineers into paying attention here. Whether or not you care about your level of seniority within your org, promotion is a critical motivating factor across the board, engineers included. A lot of seniority is about soft skills not technical wizardry. And setting expectations well is key to taking on more responsibility and advancing. That's true for both the engineering management and the technical leadership path. Engineering leaders will be surprised and delighted if you tell them, you really want to level up here. And it’s the kind of thing you can observably do in a quarter.
  2. Get a seat at the table. Value divided by cost is the robust way to prioritize work. Engineers own the cost term at the roadmap conversation. So keep asking questions until the work is well defined enough for you to make a 70% estimate for the go no go decision. That’s your chance to drop “expensive” unimportant stuff and suggest better solutions. If the project gets approved with your 15% or 30% estimate, there is a huge risk the project will take twice as long and everyone will regret prioritizing it. If more influence on the roadmap sounds appealing, take some of the estimation work off your manager’s plate.
  3. Be a better architect. Cost estimation is core to architecting the right system. How long will it take you to ship with this language, framework, etc? How quickly will we be able to build future features on top of this work? Engineering estimation is engineering, and it merits deliberate practice. Engineering estimates is one of the two ways main ways senior engineers will interact with management (the other way is delivering on their estimate).
  4. Fewer fires, dead projects, and gutted specs. When schedules run late 37% of late projects are scrapped, so better estimates should save you a lot of time and frustration. A month into a six month engineering driven re-write of the web client at Twitch, there was quite a bit of disagreement as to just how big of a re-write to do (a hundred engineers eventually contributed to this rewrite). Emmett, our CEO, had been burned by massively delayed re-writes in the past and was skeptical the team would be able to execute on the proposed schedule. So we did something that felt totally obvious, but that I’d never seen before. We compared their estimates to actuals for each line item for the last month, and saw strong agreement. They had the track record they needed to make a case that the line items in their detailed schedule were reasonable, and they got their green light. In the end they delivered a massive project basically on time.
  5. Better overall company strategy. “Good strategy works by focusing energy and resources on one, or a very few, pivotal objectives whose accomplishment will lead to a cascade of favorable outcomes” —Good Strategy Bad Strategy. Engineers at Twitch basically all agreed we were doing too much, but they didn’t realize how much power they had to solve the problem. If it’s clear that everything will actually take 2-3x long that will make it much easier for your leadership to avoid committing to do too much, and focus on what matters.

 

Getting 20% more accurate is tractable

Informally, 20% improved accuracy in this context can be thought of as being 20% better at identifying which projects you would regret embarking on for being dramatically over budget and killing them right them.

Phillip Tetlock has run forecasting tournaments where over a thousand people would weigh in on predictions like “will there be a war in the Congo in 9 months.” Even give such difficult questions, an hour’s worth of training is enough to make 14% more accurate predictions.

hbr_training_improvement.png

And trained teams were 22% better than individuals without training.

Also, if the first hours goes well you can put in a couple more, so a big increase here is totally tractable. Here are two key teachings Tetlock got that improvement with.

  1. Outside View. How long have comparable projects taken? This question is critical for overcoming the optimism people have for how quickly a project will go, and pushing back against the pressure for unrealistic turnarounds. When teams using Agile assign points, they are using the outside view. Agile leverages this viewpoint to enable teams to have predictable output for short time horizons. The same perspective is critical for achieving medium and long term goals.
  2. Break things down into parts. Joel Spolsky, recommends breaking everything down into tasks less than 16 hours. I don’t go quite as far as Joel (especially for longer term planning). But you should keep in mind that it feels worth improving a cost estimate, breaking it down more is one of the best ways to do it.

Neither of these are super surprising. But it’s not just enough to know this is the right way to make good estimates. You have to remember to apply these techniques at the right time. Daniel Kahneman, the Nobel laureate who first discovered many of our reasoning shortcuts, put together a team to write a groundbreaking textbook. They budgeted 3 years for the project, and then one day someone asked if anyone knew how long such projects generally took. A pained look, a team member knew the answer, but had forgotten to ask himself the question: “7 years and 30% of the time they fail to finish.”

 

 

Deploy 70% estimates at your org: next steps

If you see the potential for massive upside then we’re on the same page. I see 70% estimates as the first step towards a 10x better organization. An org with a lot more autonomy and a lot less politics enabled by measuring and incentivizing good judgement. More on that in my next post, back to 70% estimates. Any engineer can start this deploy. Cultural heroes will be minted. If you are down, here are the next steps.

Step 1: Put on your oxygen mask on before assisting others
This training module, is dope, and only 10 minutes long. Few would dare to call a corporate training dope. I dare. You’ll test your knowledge on facts I believe every engineer (and engineering adjacent folks) should know, for example: What percent of data breaches involved phishing? So do it right now, before you forget.

The main novel thing I’ve done in this domain, is figure out that it’s important to have people estimate things that matter to them. The content is the key reason 97% of Twitch staff would recommend my training to a colleague. So like I said, do it. After that, if you want to bring this methodology to your own organization:

  1. Find an ally. A teammate or leader, that you think would be down to start trying 70% estimates out. Share this post with them and get them to take the training.
  2. Email the relevant group. It’s probably myteam@, engineering@, product@. Get a broader feel for interest and make 70% estimates a discussion point at your next team meeting.
  3. Find the internal champion. Who will eventually own pushing 70% estimates forward at the organizational level? Ideally this is you, but if it’s not bring them in early. They should at least be bought in on your path forward for trialing 70% estimates.
  4. If an internal champion that wants help: Bring me in. It generally takes face to to convince leaders to make meaningful change, and I’m a literal pro at batch convincing people this is the one true path. For more info on this service see my page on consulting. Either way, you should definitely do step 1, take and the short, engineering focused prediction, training I made.

Thanks to Devon Zuegel, Darragh Buckley, Alex Matzner, Ryan Matzner, Tom Brown, Hector Postigo, Josh Tabak, Michael Ossareh, and Ian David Moss for reading drafts of this.

 

 

Endnotes

  1. 70% is the explicit convention for goals at Amazon and Pinterest. And 70% is the implicit convention for goals at Google, Airbnb, Lyft, and everywhere that uses OKRs. OKRs use a grading system, where you are scored on 0-10 scale against all your objectives. The expectation is that you’ll average a score of 7/10. To see that that a 70% goal expectation consider binary outcomes (10 if you hit your binary objective and a 0 if you missed it). If you track true and hit 70% of your binary goals you’ll average a 7/10. Lists of companies using OKRs: a, b, c.
  2. That’s because projects that are late are often late by a multiple of 2-4x, so they have a large effect on the average completion time. If you tried to add a bunch of 30% estimates for sub projects into an overall estimate, you’ll hit that overall estimate much less than 30% of the time. The best case scenario gets more and more unlikely as the project increases in size. Say you made a bunch of 30% estimates for a project with 10 components, and then added them together as the overall estimate. You are expected to be late on 7 of components, and 3-4 of those you’ll probably get 2-4x delays so you probably have something more like a 5-10% chance of shipping on time without big cuts of your spec.
danny hernandez