Self-Inflicted Technical Complexity

Learn how we overcomplicate software development and find ways to build simpler solutions that work better.

Before we start with this week's content, let's look at an interesting perspective from John Carmack, the creator of Quake. Microsoft recently published an AI demo that generates a Quake 2 game.

While the game is playable with major issues, it demonstrates an intriguing direction for AI-created games. Each frame is generated on the fly, opening endless possibilities for future gameplay.

While GenAI has sparked controversy in every industry it makes a mark, Carmack's response took an unexpected direction that surprised many fans.

I think you are misunderstanding what this tech demo actually is, but I will engage with what I think your gripe is — AI tooling trivializing the skillsets of programmers, artists, and designers. My first games involved hand assembling machine code and turning graph paper characters into hex digits. Software progress has made that work as irrelevant as chariot wheel maintenance. Building power tools is central to all the progress in computers. Game engines have radically expanded the range of people involved in game dev, even as they deemphasized the importance of much of my beloved system engineering. AI tools will allow the best to reach even greater heights, while enabling smaller teams to accomplish more, and bring in some completely new creator demographics. Yes, we will get to a world where you can get an interactive game (or novel, or movie) out of a prompt, but there will be far better exemplars of the medium still created by dedicated teams of passionate developers. The world will be vastly wealthier in terms of the content available at any given cost. Will there be more or less game developer jobs? That is an open question. It could go the way of farming, where labor saving technology allow a tiny fraction of the previous workforce to satisfy everyone, or it could be like social media, where creative entrepreneurship has flourished at many different scales. Regardless, “don’t use power tools because they take people’s jobs” is not a winning strategy.

Many argue that AI is just a blip that won't affect engineers' work. However, I've noticed that numerous senior engineers and leaders like John Carmack recognize that change is coming—the only questions are:

• How significant will it be?

• Where will AI progress ultimately end?

With that, let's dive into this week's topic.

Technical complexity

What is technical complexity? Non-technical people often get frustrated when they hear statements like "This isn't possible because our architecture wasn't designed for that" or, worse, "It will take months to untangle because our software is very complex."

But what defines complexity, and is it always necessary? When creating software, we must make technical decisions:

• Which framework to use

• Which database to choose

• How to make it scalable

• What future use cases to consider

• How to handle deployment

Spending weeks or even months planning the perfect architecture, testing approaches, and delivering technical milestones is tempting. The sad truth is that this level of complexity is often unnecessary, yet we create these complex monsters anyway.

As engineers, most of us love our craft. We enjoy solving problems, but we tend to overthink all possible scenarios. "What if we need to deploy this software across multiple clouds with global messaging capabilities?" I've caught myself diving deep into thoughts that have no direct impact on the current software and may never become relevant. These scenarios are interesting to solve, and we justify them by thinking we'll be prepared if we ever need them.

In this process, we often lose sight of what's truly important: quickly delivering what customers need. While we shouldn't ignore technical considerations, software must create value, especially in business. That's how it works in a capitalistic world.

We Excel at Creating Complex Things

As engineers, we often want to prove our capabilities by demonstrating that we understand complex systems. While technical prowess is important, the greatest challenge lies in building something that runs well, is easy to understand, and is simple to enhance. These two approaches—complexity versus simplicity—are often at odds with each other.

Simple solutions win on a global scale. Consider this: PHP powers approximately 77-79% of websites. Most use frameworks like WordPress because PHP is straightforward and works well for common use cases. It offers a unified ecosystem for front- and back-end development with a single language. Though some developers, particularly those working with higher-level languages, might dismiss PHP developers, this attitude is just plain wrong. PHP's dominance stems from its ability to meet most websites' needs efficiently.

Let's examine enterprise software.

In enterprise software, we frequently combine different technologies—for instance, pairing a Java backend with an Angular frontend. This combination inherently creates complexity since Java differs significantly from TypeScript or JavaScript.

While enterprise software often requires robust technology stacks for management purposes, we tend to overcomplicate things.

Often, a simpler technology stack would be more appropriate, but we stick with familiar frameworks. For example, an experienced Java developer naturally approaches problems from a Java perspective. This works well when Java suits the problem but can introduce unnecessary complexity when simpler solutions exist.

Consider data transformation or AI algorithms: Python dominates these fields not by chance but because it simplifies these tasks significantly. However, Python has limitations—building large-scale applications with robust security and performance requirements can be challenging.

C, C++, or Rust are more suitable choices for low-level memory management or maximum performance.

Web-based projects often benefit from a complete Node.js setup. Alternatives like Django, PHP, and Ruby can also create a cohesive ecosystem. In this case, the best choice for web-based projects often depends on your team's expertise and your company's strengths.

I encourage engineers to move beyond solving everything with a single language. Think of programming languages as tools in a toolbox. You wouldn't use an axe to dig a hole or a shovel to cut down a tree—while possible, it would be needlessly complex and time-consuming.

There are many advantages to selecting frameworks that match the specific challenges you need to solve.

Engineering complexity for everybody

So technical complexity has something to do with the problem, its solution, and the chosen technology stack. With that in mind it's more practical to carefully evaluate which components truly require complex enterprise frameworks and which could use simpler, more widely understood technologies.

This consideration is crucial for businesses since hiring talented developers is already challenging. Choosing overly complex technology combinations severely limits your talent pool, creating scaling issues for your team rather than technical limitations. Here's a real example I've witnessed:

Teams searching for an entire year to fill positions simply because their required technology combination was rare—and then they had to compete with big tech companies that could offer much higher salaries.

Therefore, adopting more accessible technologies reduces complexity and provides access to a larger pool of qualified developers.

Understanding on both sides

We need mutual understanding between both sides: engineers and those outside of engineering (whom I'll call "the business side"). A business that fails to understand how technology relates to complexity—and how this affects what a product can achieve—will struggle. Without this understanding, the relationship between business and engineering will always be strained.

While it's tempting to say engineers should explain technical complexity better, this oversimplifies a complex issue.

Key Points for Success:

• Engineers must learn to think more simply and adapt to business needs. I've noticed this adaptability, especially in engineers at early-stage startups, where it's essential for survival. This approach can work in larger companies, too, when applied appropriately.

• The business side must understand engineering, which often requires sitting down with engineers to discuss their daily challenges.

With frequent collaboration between both sides, mutual understanding will naturally develop.

GenAIs play in that

GenAI currently works more effectively with commonly used tech stacks. For example, a web page built with Next.js, where TypeScript handles both frontend and backend, will benefit more from GenAI assistance than a traditional "enterprise" architecture.

Why? Because there is significantly more training data available for popular tech stacks, while enterprise software—being private and inaccessible—provides limited training material.

The bottom line: While this dynamic may change in the future, having a simpler tech stack and architecture is advantageous.

Key Takeaways

Technical complexity often stems from our tendency as engineers to overcomplicate solutions. While we must consider technical requirements, we should prioritize simplicity and choose technologies that match our specific needs rather than defaulting to complex enterprise frameworks.

Here are the main points to remember:

• 📈 Simple solutions tend to be more successful globally - as demonstrated by PHP's dominance in web development

• 🎯 Choose technologies based on the actual problem, not just what you're familiar with or what seems most impressive

• 👥 Simpler tech stacks make it easier to hire developers and scale teams

• 🤝 Better collaboration between technical and business teams requires mutual understanding and frequent communication

• 🤖 GenAI tools currently work better with common, simpler tech stacks due to available training data

The key is finding the right balance between technical requirements and practical simplicity. As the software industry evolves, especially with the rise of AI tools, the ability to maintain this balance will become increasingly crucial for successful engineering practices.