Blue-Collar Engineering Dispatch #1: Microservices: Double-Edged Sword

Hi there, and welcome!

In our continuing journey on simplicity > complexity, I thought this edition we would discuss another overused (and misunderstood) paradigm in modern engineering: microservices.

First of all, do we use a hyphen in micro-services, is it one word? I really don’t know so we will go with one word for now. I digress. Like the last edition, I want to start off with a story.

Sliceify: Death by a Thousand microservices

Meet Sliceify, the "Uber for pizza." Their genius idea? Crowdsource pizza requests. No more calling the pizza joint - with Sliceify, complete strangers can pick up your pie on their way home and deliver it to you. Growth has been steady and investors have been surprised how the lack of safety has not bothered its user base.

Sean, Sliceify's lead architect, had just returned from MicroCon, a conference dedicated to all things microservices. Inspired, he decided Sliceify needed to break up their MVP (Minimum Viable Pizza) monolith into microservices. "Once we grow past 10 engineers, we will be able to deploy independently and deliver value to our customers faster!" he declared. "This will future-proof our stack"

The team got to work. They carved up the monolith like a Thanksgiving turkey:

• order-service for processing orders

• delivery-service for dispatching drivers

• topping-service to handle those extra pepperoni’s

• crust-service for the dietary restrictions and tastes

• authentication-service for authenticating the users

• orchestration-service to coordinate all inbound requests to the correct microservice

• notification-service to alert customers their pizza is en route

• reporting-service future customer support agents can view and query customer data

Before long, they had a Kubernetes cluster with tons of services and operators. Deploying a new feature was like playing 4D chess - the order-service needed to talk to the crust-service, but the crust-service was waiting on the topping-service, and the notification-service is in a crash loop backoff!

Debugging was a nightmare. When a customer complained their Hawaiian pizza arrived without pineapple, the team had to trace the request through a maze of services. "It left the topping-service…" one engineer said. "But it never arrived at the delivery-service." Sean, still sporting the kool-aid smile, injected into the Slack huddle, “We need distributed tracing!”

Once again the team got to work. First, they had to set up an Open Telemetry collector. Next, they needed to add all the OTEL dependencies to their services. Finally, the engineers added spans to instrument the microservices code paths. Because of a time constraint, they only added tracing to the happy paths. The team had every intention of coming back next sprint to finish the task properly.

As the service count grew, so did the problems. Deployments took hours. The AWS bill looked more like the U.S. national debt. The engineers spent more time writing API contracts than actual code. Jen, a senior engineer, started to wonder if they'd made a mistake when a new hire asked, "Which repo do I check out for the pepperoni feature?", and she didn't have an answer. Their microservices had become a macro-mess. ”We have create a distributed monolith! The only thing we have solved is added complexity.”

Finally, Rishi, the CEO, stepped in. "Why is everything taking so long?" he asked. "We're drowning in complexity," Jen admitted. "We thought microservices would solve all our problems, but they created new ones. We are lacking the skills and experience to build and manage this correctly.” The CEO sighed. "Let's go back to the basics. Can we put all the services back together in one?” The team happily responded, “yes, but it will take another quarter and 3 more headcount.”

As over the top as Slicify's story might be, the microservice mirage is all too real for many teams. The siren’s call of scalability and flexibility leads them to prematurely break their applications into increasingly smaller services, until they're left with a distributed monolith that's harder to manage than the one they started with. But I am getting ahead of myself. Where did this paradigm and term even come from?

The Rise of the Microservice

According to my deep and thorough investigation (cough: LLM and Wikipedia), microservices emerged in the early 2010s as a response to the challenges of maintaining and scaling large, complex monolithic applications. While the ideas behind microservices were catching on in the late 2000s under names like "micro web services (Peter Rodgers)", the term "microservice" itself was first coined in May 2011 at an architectural workshop. It quickly caught on as companies like Netflix and Amazon shared their success stories of adopting the new paradigm. The promise of microservices was alluring: break your monolith into small, independent services, and enjoy the benefits of flexibility, scalability, and autonomous teams. As an engineer or engineering leader, this sounds like a great tool to deliver for your customer, right? But as with anything in the technical field, the fine print often gets missed.

⚠SURGEON GENERAL WARNING:

Microservices are often the design of choice in modern software architecture—promising scalability, flexibility, and modularity. In fact, if you have done a system design interview in the last 5 years, your design must include microservices or you might get passed over. The idea is simple: break down a large application into smaller, independently deployable services that can evolve separately. It sounds elegant in theory, but in practice, microservices can introduce a level of complexity that that you and your team are not ready for. Before you become Sean at Slicefy, it’s worth considering the trade-offs.

Complexity Overload

Each microservice brings it’s own set of challenges. For example, they will need their own API contracts, monitoring, and deployment pipelines, which increases the overall system complexity. Suddenly, your engineering team is spending more time on the plumbing than on delivering features. The promised flexibility turns into operational overhead. Troubleshooting problems can be a challenge too. You now need to check logs and metrics across multiple services and code paths to find correlation.

I know you are saying, “But Bradley, they are doing microservices wrong!” That may be true, but why do a lot of engineering organizations struggle with this? It can’t be that they are all bad engineers or just “doing it wrong”. It is because microservice architecture is hard to get right and most teams do not have the time and experience to design it favorably from the start. Plus, most business domains do not fit nicely into clean boundaries…they are messy.

Deployment Challenges

In a monolith, deployments are generally straightforward. But with microservices, each service has its own deployment lifecycle. Updating a feature that spans multiple services requires careful choreography. Imagine trying to coordinate a new pizza topping feature across the order-service, topping-service, and delivery-service. You've got to deploy them in the right order and hope they all work together in production. What if they require database migrations? It can be a recipe for slower, riskier releases.

Troubleshooting

Debugging monoliths is no picnic, but microservices takes it to a whole new level. When an issue pops up, you've got to navigate a labyrinth of services to pinpoint the problem. Let's say a customer reports their pizza never arrived. Is the issue in the order-service, the delivery-service, or somewhere in between? You'll need to trace the request through multiple services, correlating logs across different systems, all while under pressure to resolve the issue ASAP. Distributed tracing tools like Jaeger or Zipkin can help, but they require a significant investment to set up and maintain. You'll also need robust logging, monitoring, and alerting across all your services to catch issues before they impact customers. It is not to say that non-microservice architectures are free from these pains, but this level of operational maturity is essential for success with microservices. Without it, troubleshooting can quickly become a nightmare for your customers with a lot of finger-pointing and guesswork.

When Microservices Make Sense

Up to this point, I have shouted from my front lawn that microservices are bad. But that is just me driving my newsletter theme home. Microservices are not always the wrong choice. In fact, many companies have delivered their products reliably because of them. They shine in specific scenarios. For example:

• Large, complex domains that can be cleanly divided into independent services

• Massive scale with high-volume, high-throughput requirements

• Mature organizations with multiple teams working on separate parts of the system

Netflix handles billions of requests daily across thousands of devices. Microservices allow teams to scale and deploy independently.

Here are some other well-known companies that have successfully implemented and shared their microservice architectures:

1. Amazon: Amazon was one of the early adopters of microservices. This allowed them to scale their business and achieve high levels of agility.

2. Uber: Uber's microservice architecture is designed to support their massive global operations. They have thousands of microservices that handle everything from ride requests to driver management to payment processing.

3. Spotify: Their architecture consists of hundreds of autonomous, independently deployable services. This has allowed them to scale to over 350 million active users.

The Takeaway

Start with a modular monolith. Identify natural service boundaries as you go. When the time comes, carve out services gradually. Let your architecture evolve with your needs, not your desire to be “future-proof”.

If you are tempted to go down the microservice path, you should ask yourself these five questions:

✅ Is my team large enough to handle the added complexity of microservices?

✅ Does my domain have clear bounded contexts that can be split into it’s own services?

✅ Do I have a pressing need to scale individual components independently?

✅ Am I okay with the additional overhead of inter-service communication and distributed debugging?

✅ Do I have robust testing and CI/CD practices in place to manage a microservice architecture?

If you answered "no" to most of these, it's probably best to stick with a monolith for now. Remember, you can always start with a modular monolith and move into microservices later as your needs evolve.

However, if you answered “yes” to most of them, head on over to the dispatch repository for some microservice tips.

Reader Challenge

Have you encountered a "microservice mirage" in your own projects? Reply and share your story! I'd love to feature it in a future edition.

Until next time,

Bradley

Chief Advocate for Keeping It Simple

P.S.

Despite the attraction of microservices, many successful companies still power their applications with monoliths:

1. Basecamp: The popular project management tool is built as a monolith in Rails. Basecamp's team has consistently advocated for the simplicity and productivity of monolithic architectures.

2. Stack Overflow: The go-to platform for developer questions runs on a .NET monolith. They've scaled to handle millions of requests per day.

3. Shopify: The e-commerce company that powers over a million businesses runs on a monolithic Rails application. Despite their scale, they've found the monolith to be the most efficient way to deliver new features.

4. GitHub: The world's largest code hosting platform was a Ruby on Rails monolith for most of its history. They only started breaking out microservices as they approached truly massive scale.

These companies demonstrate that monoliths are not just viable, but often the smart choice to start. They've been able to scale, maintain, and evolve their applications without the complexity of microservices. To be fair, these companies also have mature engineering teams and mechanisms. They've invested in practices like modular design, extensive testing, and continuous delivery to keep their monoliths manageable.