Why Is Software Engineering So Hard?
This is a shorter newsletter than usual as I've been sick and recovering this week. So bear with me—it's more thoughts than research this time.
In this post, I want to explore why software engineering is so challenging. The numbers in one of my previous posts about project failure rates are striking. Software engineering is often compared to other engineering fields like construction, but if buildings failed at the same rate as software projects, we would have a devastated cityscape. So, is software engineering truly more difficult?
Is Software Engineering Complex?
To put it simply: yes, it is.
Creating any piece of software—whether small or large—involves almost unlimited variables:
Tech stack
Installation
Environments
Customers
Features
Service
...and the list goes on.
Success depends on managing these variables while adapting to an ever-evolving industry with changing user demands. The situation has become even more complex in recent years as we've added more possibilities rather than simplifying the equation. Failing in just one area can doom a project.
Consider a project with a solid tech stack and healthy test status that fails in its target environment due to excessive CPU and memory consumption. Or imagine software that can't be properly maintained because observability wasn't considered.
The Challenge of Rewrites
We often resort to rewrites to accommodate demanding new environments. But by then, our previous product line has accumulated numerous features that every customer expects in the new version. Not so easy, right?
The House-Building Analogy
Let's compare this to house building—a common analogy for software engineering.
With a large apartment building, it's obvious that changing the heating system requires significant investment. Even on a smaller scale, an apartment's floor plan clearly shows where the kitchen and bathroom are located. It's evident that converting a bathroom to a different type of room would require massive rework because of the plumbing. We understand these constraints because we can see the concrete walls.
In software, we lack these visible constraints, making it harder to understand why adding “just a small feature” takes so long. Even with good architecture that allows for quick changes, modifications can require considerable effort—especially since software can't be designed to accommodate every possible future direction.
The Cost of Shortcuts
Sometimes we take shortcuts, accumulating technical debt to make things possible. It's like breaking building codes with incorrect plumbing or wiring, leaving problems for others to handle later.
In a building, such shortcuts might take years to show their impact through mold or structural deterioration. In software, because of its rapid evolution, the consequences can surface with the very next feature request—causing everything to collapse and making changes even more time-consuming.
So, What to Do?
At least literature suggests one solution: be fast, iterate, and get your software out. Don't build a beast with no customer. This approach lets you iterate on smaller, manageable pieces where you can control the complexity.
Projects that try to do everything from the start often end up with an architecture so complex that only the creators can work on it.
We see this principle of small wins in action with startups that can deliver quickly. It's also easier to deliver when you have a focused list of requirements or can dictate which features customers get (like Apple does).
What About Large Companies?
Even in large companies that must deliver complex projects with numerous must-have features, starting small is key—build a working product first, then iterate. Spending too much time in the planning phase can blind you to critical issues that only surface once you have a testable product, even if it's just for internal testing.
Keep in mind that customers come in both external and internal varieties—getting feedback is much simpler when your product runs on more than just development machines.
Team Size Matters
Starting with a small group and growing over time is more effective. Smaller teams (5-25 members) consistently outperform larger groups.
The main reason is simple: more people create more communication overhead. While better planning and additional managers can help, there’s a natural limit. Beyond this threshold, adding more people slows down progress due to increased communication and tooling overhead.
Unfortunately, when projects slow down, organizations often respond by adding even more people—which only makes everything slower.
Skilled People Over Large Teams
Using fewer but more skilled people is often better.
Consider this comparison: for a complex medical procedure, you wouldn't put 20 junior doctors in the room—you'd want one specialist.
While specialists are more expensive, they can drive the project forward much more effectively.
Exceptions Exist, But They’re Rare
Of course, there are exceptions. Take Figma—it took 3.5 years to build their first live product and 5 years to get their first customer. But such exceptions succeed for very specific reasons.
Don't assume every project you undertake is an exception.
Instead, first consider if you can solve your problem like everyone else. Then evaluate whether your project is truly special enough to warrant extraordinary effort. Figma succeeded because they had a crystal-clear vision of both their problem and its technical challenges.