Not only are software task completion estimates nearly always wrong, it’s often hard to know just how wrong they are.

In this article we’ll talk about why it’s so hard to estimate task completion, and why it’s important that we try. But we’ve also come up with a process to get things done, which you can read about here:The order of three (practical task estimation in software development)

This problem continues to be a source of widespread disagreement and controversy. It leads to a serious culture clash between software teams and teams focused on different aspects of the business.

Many software experts recommend weaning organisations off estimates altogether. Others argue that it’s impossible to run a business without them. There’s even a widely-used hashtag #NoEstimates which has been at the centre of an ongoing conversation dating back to 2012.

So why is it so hard?

Any innovative project includes unique work that goes beyond a team’s existing knowledge and experience. But it’s only possible to provide accurate estimates based on work that has been done before.

Even then, cognitive biases like the planning fallacy lead teams to be wildly over-optimistic about what they can achieve in a certain time-frame. At the beginning of a project, requirements are usually vague. That means that even the best possible estimates are also going to be vague.

The most useful thing to do in this situation is prototype or partially implement features to understand how complex they are. This helps uncover any hidden detail that wasn’t clearly defined at the outset.

The only way to precisely know how much something will cost and how long it will take is to build it. Understanding this leads many developers to view estimates as waste—work that delivers no value.

But coordination between departments with a schedule of delivery dates is unavoidable. This is particularly true when large investments or the future of a business is at stake.

This dysfunction is at the heart of the culture clash between software people and the rest of the business.

Read more: A few hard-earned lessons from startups

Process adherence rather than value creation

Agile software development hopes to deliver value to customers in small, continuous increments. This avoids committing to larger, more risky efforts. Achieving a flow of continuous delivery requires a high functioning team operating in sync with the business.

However, many software teams lose their way in a murky fog of Agile rituals and practices designed at reducing risk and micromanaging short-term outcomes.

In many of these situations, estimation still occurs. It’s just as problematic as quoting unrealistic person-hours, but it’s masked under the guise of relative measures like story points and game-ified rituals like planning poker.

The original intention of these measures was to help teams move away from fixed estimates, towards sustainable development. But a subtle consequence of this is a fixation on completing tasks rather than achieving goals. It further contributes to the cultural rift between the technical and non-technical sides of the business.

Crossing the cultural divide

If there’s any value in software estimation, it’s in pushing teams to engage with the problem and unpack their assumptions at the outset. Estimation involves working through the steps to understand and document the key assumptions, constraints and context of what a feature is and how it should work.

The time spent making estimates isn’t wasted if the people involved are creatively and critically engaged. The collaborative effort that it takes to figure out the scope of a project provides the energy and impetus to push that project into motion and make things happen.

We have to shift our perspective on what an estimate is and what it gives us. The order of three has helped me explore ways of reframing estimates to be more focused on the idea of budgeting and investment, rather than time and costs.

Read more: Are boards being let down by their senior leadership teams?

Software changes the world

For example, if we estimate that a task will take 3 days, then later learn that it’s going to take much more than that, framing this as a budgeting and investment question leads to more positive ways of dealing with the outcome. If 3 days is our agreed investment, what else could we do in that time to address the same need?

Software is valuable because it effects change in the world. This real-world impact should be what we’re focusing on and measuring with every piece of work that we commit to.

The best way for teams to effectively collaborate is to focus on building trust instead of haggling over tasks. Treating estimates as a tool for breaking down problems rather than forcing rigid commitments is a great starting point for achieving this.