Why Do Big Software Projects Fail?
A deep dive into why large software projects struggle, what the data tells us about success rates, and how modern delivery methods can help.
Big software projects frequently fail, and we've grown oddly accustomed to this reality. But we shouldn't accept it. Better ways to build software projects exist today, though implementing them requires change.
Project failures can have massive consequences, as demonstrated by last week's Sonos App rewrite failure, which may lead to the company being sold.
Failed software projects often make headlines—here are some notable examples:
Hertz paid $32 million for a website from Accenture that never went live
An abandoned NHS IT system that cost £10 billion
A US Census Bureau technical debacle costing more than $1.3 billion
Beyond these high-profile failures, many projects exceed their budgets or timelines. Some complex projects ship but later face quality issues so severe they must be withdrawn or replaced.
This pattern persists today, just as it did years ago. Despite decades of experience, we continue to encounter problems delivering software projects. To understand why, we can turn to data—particularly the CHAOS database, one of the most comprehensive studies of software project outcomes. Let's examine what this data reveals.
CHAOS
The Standish Group has maintained the CHAOS database since 1994, collecting data on project outcomes. Each year, they add 2,500 to 5,000 new project cases, creating a comprehensive picture of why projects fail or succeed.
The report costs $440—quite expensive—but most of the graphs are available in a 2021 paper.
The first graph shows three project categories:
Successful
Challenged
Failed
While Successful and Failed projects are self-explanatory, Challenged projects are those that are over budget, late, or don't meet customer expectations. Two notable trends emerge:
A spike and plateau in successful projects after 1994
Since 2000, there has been a consistent 20% failure rate alongside a high 45-50% rate of challenged projects.
The increase in successful projects can be attributed to improvements in project management, technological advances, and the adoption of agile methodologies. While these approaches have significantly reduced complete failures, progress has stalled in recent years. Looking at the latest data, 65% of projects are either challenged or failed—meaning your project's chances of success are worse than a coin flip.
Let's examine the data more closely by looking at project types and their success rates.
Projects built from scratch have the lowest success rates, which makes sense given their complexity—everything must be created anew. Projects using components—where pre-existing elements and third-party libraries are integrated into custom software—have better odds of success since they build upon established foundations.
The third category includes commercial off-the-shelf (COTS) software, such as WordPress, Microsoft Office, and Shopify. While these solutions can be customized to meet specific needs, projects can still fail due to misalignment with customer requirements, unexpected costs, or support challenges.
Infinite Flow
Now, let's examine the final category in the table: "Flow Like Modernization." What does this mean? It involves breaking down projects into very small deliverables, allowing for rapid delivery and iterative changes. Sound familiar? This approach closely aligns with DevOps principles, which emphasize this as a crucial factor for success. It also encompasses:
Continuous improvements
Agile Methodologies
User-Centric approach
Reduced decision latency
The last point presents an interesting contrast to traditional "good project management." Rather than spending excessive time deliberating each decision out of fear of making mistakes, you take action quickly. By making smaller decisions, you can adjust course when things aren't working. Traditional project management often resists changes because decisions involve such lengthy discussions that they become difficult to reverse. Smaller, faster decisions enable quicker adaptations.
But what about customer satisfaction and return on investment? Here, too, a Flow approach yields superior results.
Infinite Flow enables:
Faster delivery of products and features to customers
Quicker adaptation
Earlier dropping of unnecessary features
Reduced time invested in wasteful development
This waste reduction significantly impacts software delivery costs, as many features developed in large projects go unused while truly needed functionality often remains undelivered.
Accelerate
If you've read the book "Accelerate" or are familiar with DevOps and its annual reports, this will all make sense. Every DevOps status report has consistently shown that teams delivering smaller increments faster outperform their counterparts across all industries—including regulated ones.
Building software is complicated and difficult to compare to other endeavors. While we sometimes liken it to building a house for simplicity's sake, software development is even more complex (trust me, I've built both a house and software). Many aspects remain unpredictable, even when building something that already exists. Consider streaming services as an example: despite Netflix's established presence, Disney+, HBO, and others all encountered serious issues at launch. Building from scratch—even when solving a previously solved problem—is complex because:
There are countless ways to build it
Each decision creates new challenges and solutions
The same team can build the same product with different technology stacks and still face:
Complexity
Budget overruns
Delays.
So, what do you do when faced with something complex?
You break it down into smaller pieces your brain can process and understand. If that's not enough, you make it even smaller. This is precisely what these methods aim to achieve—solving a major industry issue by:
Making problems small enough to deliver something meaningful
Iterating on the solution
Implementing best practices like CI/CD to automate these processes
(because at this scale, manual operations become impractical)
I Cannot Do Continuous Deployment
A widespread misconception is that specific projects cannot do continuous deployment—and that's actually true. Not every software is a SaaS that can be updated and rolled back within seconds. There are also air-gap environments where continuous deployment is impractical, but frankly, this doesn't matter. What you can always do is continuous delivery. This means every change can be delivered and, if needed, deployed directly to a customer's system. With this approach, you can:
Simulate systems in-house
Implement continuous delivery internally
Summary and Takeaway
Writing software is complex, and delivering large projects on time and on budget is even more challenging. Industry best practices have evolved, and comprehensive studies like CHAOS and the DevOps Reports demonstrate that companies embracing small, iterative software releases consistently succeed. These companies not only deliver software faster but also achieve
Higher customer satisfaction
Greater software value
While implementing this approach in large software projects and enterprises may seem daunting, organizations that commit to this path will reap the benefits.
The notion that better planning alone will fix the next project is flawed because software development is inherently unpredictable. When facing unpredictability, the only practical solution is to reduce complexity until it becomes manageable—a principle well-supported by research.
Despite this evidence, we'll continue to see large projects fail because we're drawn to their grand scale and complexity. There's something alluring about tackling massive challenges and believing we can overcome them. When such projects partially succeed, they feel triumphant. However, from a business perspective, this approach wastes significant resources that could be saved through alternative methods.
To quote from the famous "Mythical Man-Month" book:
Question: How does a large software project get to be one year late? Answer: One day at a time!
References: Images from Endless Modernization: How Infinite Flow Keeps Software Fresh