It turns out that corporate America was trying to do the same thing. Offshoring was in full swing. Companies were hiring on the other side of the world to save some of those almighty dollars, and it didn’t take long for them to realize that this came with some challenges. It turns out that communicating with people on the other side of the world is hard, even with email and telephone.

What made the ragtag band of gamers able to organize 40 people over months to finally knock down the final boss of Molten Core, but corporations struggle with even simple development workflows? Was it that the problems were more complex? Maybe, but after 13 years in the industry now, I don’t think so. Finishing these raids took 20-40 people executing mechanisms perfectly for hours on end, not to mention the time preparing and practicing. Different problems, certainly, but I’m not sure I’d buy an argument that they were more or less complex.

I think the biggest differentiating factor was being present at the same time, focused on the same goal, and communicating in real time.

The state of the industry

Looking at my time in the industry so far, I’ve seen this again and again. When I worked in Japan, we had offshore developers and operations staff in India. These were some incredibly talented people, and we still struggled even with that separation. Where we shined was on incident calls, when things were broken and we were all on a WebEx bridge at the same time, working on the same goal, and communicating in real time.

In 2020, when every office job went remote by force of circumstance, I saw a lot of people struggling, but my team mostly carried on almost as if nothing was happening. We had adapted quickly to the new situation despite hiding in our apartments. We had an existing culture and trust that allowed us to stay engaged and communicate in real time.

I’ve also worked on a remote team that is more than just remote, it is asynchronous. People worked when they want. Teams were split across the world. Our cycle time was slow and we had to accommodate this. Sometimes I needed PR reviews from someone when it’s 3AM their time. Sometimes, people were just not present unannounced or just didn’t answer messages for days.

All of this experience has led me to a conclusion that the industry needs to discuss more seriously: Remote work is not the same thing as asynchronous work. These are distinct modes of operating that are being conflated which muddies the discussion we desperately need to have.

What is remote and async work?

We will define remote work with a fairly simple definition: your team does not work in the same physical space. Employees are free to work from wherever, so long as they have a stable internet connection and are able to complete their work.

There are a lot of benefits to this.

Employees can choose where to live based on their needs. They may want to live near family instead of in the city. Maybe they need to be near certain medical specialists. Maybe they just like being able to sit on the beach while writing. They’re able to setup their work environment to suit them, be that ergonomics or accessibility needs. Remote workers are able to focus more without the distraction of an open office layout. Their commute involves walking to their home office, which is better for both their mental health and the environment.

The benefits extend to employers as well. You get a larger hiring pool since you can attract candidates with constraints that might otherwise keep them out. Turnover is lower since employees are happier. Perhaps the biggest advantage is facilities cost. No office leases, supplies, printers, chairs, desks, air conditioning, on-tap kombucha, etc. It’s just cheaper to run a remote business.

There are, of course, drawbacks as well. Your managers lose the direct visibility of seeing who is in the office and what they’re doing. Spontaneous collaboration is lower, since you’re not going to run into people at the water cooler or be able to pop into a conference room. Employees lose social connections, which for many are the only social connections they have. Mentoring and learning by osmosis is more difficult.

These drawbacks can be addressed, which we’ll get to later, but for now, let’s take a look at asynchronous work.

We’ll define asynchronous work as not working at the same time. For the sake of drawing this distinction, let’s imagine an office that is open 24/7 and employees are welcome to come in and work at any time they wish as it fits their schedule. Your night owls show up at 3 in the afternoon and work until midnight. The parents take off to drop the kids off and pick them up from school, then come back to the office after.

The benefits of this are clear. For employees, it allows work to fit around their schedule, which in modern adult life is often incredibly busy and complex to navigate, even more so if you have children. Employees are able to set aside time for deep work whenever they have it instead of forcing a day of shallow highly interrupted work. For employers, you can access a larger hiring pool, perhaps even global.

The drawbacks, though, are high. All communication has to be done via writing, recorded videos, or through meetings awkwardly shoved into time overlaps. Immediate feedback loops are completely gone, replaced by long feedback cycles of a day or more. You lose pretty much all capacity for high bandwidth communication and coordination. Brainstorming and collaborative creative work are drawn out for weeks instead of a good afternoon session. Process gets introduced to address this, and overhead increases.

And this is the crux of the issue. The problems we are seeing in modern remote organizations have very little to do with them being remote. They have to do with them being asynchronous. Asynchronous work has been conflated with remote work.

If your work is the type where you have long feedback cycles anyway, like journalism or other forms or writing, this is probably fine. As much as we try to pretend otherwise, software is a highly collaborative endeavor. It’s the rare piece of software that can actually be built and fully owned by a single person, and that takes an exceptionally isolated use-case and an exceptionally talented engineer to pull off.

The drawbacks of asynchronous work are intrinsic and incredibly difficult to mitigate, at least for the time being. To fix these issues, we need to solve for the need to collaborate with others entirely, which we are nowhere near accomplishing. Asynchronous work is simply a nonstarter right now. People need to be working at the same time on the same problem to support the high bandwidth communication needed. Period.

The drawbacks of remote work, however, are fixable.

Addressing the drawbacks of remote work

The things we lose from not being in the same physical space can be addressed with tooling, but more importantly culture. We need to bring the spontaneity in and encourage active collaboration. There is a tendency to self-isolate and over-focus on deep work when working remotely and we need to take active steps to counter that.

In a physical office, I can walk over to someone’s desk if I need an immediate response from them. On my team, we’re co-located, so I can just ask them immediately. This needs to be supported and encouraged on a remote team. When someone messages you during working hours, treat it as if they walked up to your desk. You wouldn’t ignore someone standing next to you for an hour, so why do it when they send you a message? You’re introducing asynchronous work patterns where they don’t need to exist. Odds are, your lack of a response is blocking someone else. Conversely, treat sending a message as walking up to their desk. You’re likely interrupting their deep work, so make sure the interruption is worth it.

On the topic of deep work, it’s undeniably critical, but we shouldn’t pretend like communication, mentorship, and guidance aren’t their own form of valuable work when done correctly. When you need dedicated time for deep work, schedule it and communicate it in advance so others can plan for your absence. Otherwise, you should accept that a large part of your work is communication. I would even argue that it’s the most important part, especially as you reach senior and principal levels.

Another drawback we can address is the real-time collaboration that a simple whiteboard brings. There’s some real collaborative magic that comes from being in a room with others and a whiteboard. Being able to draw, debate, and redraw strongly supports the rapid iteration loops that are needed. Fortunately, we have plenty of tooling solutions for this as well. Tools like Figma and Miro have become the defacto whiteboarding tools anyway, even when in the same physical space. They take the whiteboard experience and enhance it. Multiple times, I’ve been in physical meetings now where we are all at the same table, working on the same Figma. The whiteboard on the wall is just decoration.

The last big drawback is in management. One of the most common concerns I hear from managers about remote work is that they can't tell if people are actually doing work without being able to see them at their desks. Here’s the thing, though. If you are relying on visually ensuring butts are in seats to manage, you aren’t really managing at all. Truly great managers do their work by setting objectives, measuring outcomes, and ensuring their teams are able to operate without your constant guidance. They focus on building and maintaining the organizational substrate on which the team operates. The best manager I’ve ever had was able to disappear for three weeks and the team barely noticed, because he made sure we were self-sufficient and motivated. He didn’t “manage” us so much as keep the road we were traveling on clean. The worst manager I’ve ever had checked that butts were in seats, and if you were so much as going to the bathroom when he did his check, you’d get an earful.

The future

Remote work is here to stay in some capacity. For those of us who strongly believe in it, we need to admit the downsides that come with it and work to fix them. By far, the biggest downside the tendency for remote work to slowly morph into asynchronous work. They are not the same, and we cannot allow the slow feedback loops and destruction of collaboration that asynchronous work brings tarnish remote work as a concept.

Remote teams should expect members to be at the keys at the same time, most of the time. Setting core hours where people have to be available is the easiest approach. Obviously, there are exceptions for the basic life stuff like doctor appointments, but in general, we should be working at the same time. We are working together on the same problem, so we need to be working at the same time. This brings back the spontaneity and high bandwidth communication that allows for rapid innovation and progress.

We cannot allow a culture of asynchronous work to take over and damage the reputation of remote work as a whole.

More importantly, though, we need to have discussions about remote work that are free of the baggage of asynchronous work. The executives and managers aren’t wrong about “remote” work being slower - it is - but only because it has come hand-in hand with asynchronous work. Remote work could be a game changer for not just the employee, but employers, and we need to interrogate it as a solution honestly and precisely, which we are currently failing at spectacularly.

Before we start, let’s get a shared understanding of what a microservice is, though. We’ll define a microservice as a small independently-deployable unit of software. The exact implementation doesn’t really matter at all. The exact scale of “small” varies. I’ve seen some “microservices” that are a single AWS Lambda function and I’ve seen some “microservices” with half-a-million lines of code. This ambiguity in size is the first part of his argument. They’re not “microservices” they are just “services” and they come in a variety of sizes. There’s nothing about the definition that makes them “micro” just they are independently deployable.

We’ve been taught that microservices are where your architecture lives. When we draw out architecture diagrams on the whiteboard, our boxes are microservices with cute names like Galactus. This is deeply ingrained in the industry at this point and we’ve forgotten what software architecture actually is. Your architecture lies not in the microservices, but in the seams you define in your software.

Seams are the places where different pieces of software are joined together - clear integration points with defined contracts and behavior expectations. This concept goes by a few other names like “interfaces” and “ports” but they are all the same thing. Well-defined seams must be severable - I can take the software on each side of the seam and utilize it independently. Notice that we are still really abstract here. We’ve not talked about services, deployments, processes, or anything else. Just the abstract concept of a boundary between units of software.

Monolithic software has seams, operating systems have seams, embedded software has seams. A great example is in file systems. Linux can run on a variety of file systems because they all implement the same interface to the kernel (VFS). Do you need a basic fast file system? EXT4. Do you need high data integrity? ZFS. The seam allows you to choose the right tool for the job. As long as you abide by the behavior contract in the seam.

Well-defined seams offer you a really neat opportunity: you can split along them in a variety of ways that each confer different advantages and introduce different downsides.

You could simply split the seam as code packages. Create a different maven or npm module to keep the code isolated to prevent anyone from breaking the abstraction. The code still compiles into the same binary, but you get some measure of isolation without sacrificing performance.

You could split as individual processes on the same host using inter-process communication protocols. This allows you to have different modules in different runtimes or languages, with different OS-level configurations.

You could split as individual containers composed together as a single unit, using REST APIs to communicate between containers, commonly called sidecars. This enables each piece of the software to run independently to take advantage of differing container needs.

You could split as completely different services behind distinct load balancers, using REST APIs to communicate. This allows for independent scaling of each component, but incurs overhead in network transit time and request serialization/deserialization.

Regardless of how you split it, the seam is there. The contract is maintained.

A “microservice” is really just the last one. You’ve decided to split a seam using the highest level of separation you could: completely different services. This comes at a cost, but confers advantages.

In a high scale system, being able to independently scale different pieces of your software up or down is invaluable. Load is often inconsistent and unpredictable. Some pieces may be more popular in the morning, but less popular in the evening and vise-versa. It allows very clear boundaries on fault containers - you don’t have to worry about a botched process deployment killing a different process. You get memory and compute isolation - no more greedy processes taking all the host’s memory and CPU time.

But network traversal takes time, as does serializing your requests to and from JSON, as does the load balancer. It’s not a huge amount of time nowadays, but it’s still time. It also makes the seam behavior harder to enforce since there is no compile-time testing that the contract is being followed correctly. You have to deploy and pray or write massive suites of “contract tests” that assert the behavior at deploy-time. I’ve worked on teams with literally thousands of deploy-time tests to assert the behavior of a single microservice.

Microservices became popular because they solve a lot of the software challenges that come with monolithic single or multi-process software, but they do so at the cost of latency and contract enforcement. They have their place still, but they are not the one solution to rule them all. They are a tool by which you can split seams, but they are not your architecture. Your architecture is the seams.

We've all been in that meeting. Someone sends out a lengthy technical document or proposal beforehand. The meeting starts, and everyone nods along, occasionally offering vague comments that could apply to almost any document. "Interesting approach" or "I have some concerns about scale." Eventually, someone admits they "only had time to skim it," and like dominoes, others follow with similar confessions.

This theater of partial reading is toxic to organizations. We're normalizing a culture of incomplete understanding while pretending we're being transparent about it. "I skimmed it" has become an acceptable excuse, as if reading 10% of something gives you 90% of the understanding. It doesn't.

The Ownership Problem

Reading and understanding a document isn't some checkbox activity - it's a core part of owning a decision or contributing meaningfully to a discussion. When you say "I don't have time to read this properly," what you're really saying is "I don't have time to participate in this decision properly." That's fine, but we need to be honest about it.

AI summarization tools have made this worse, not better. They've given us a technological fig leaf to hide behind. "Oh, I had GPT summarize it for me" is the new "I skimmed it." Both are ways of pretending to have knowledge we don't actually possess. The AI doesn't understand your organization's context, history, or needs. It's giving you a book report without any of the insight that comes from truly engaging with the material.

Time Management is Risk Management

"No time to read" really means "no time to make an informed decision." We're sacrificing long-term value for short-term gain. Every time we make a decision based on partial information, we're taking on risk. The problem compounds because we often don't discover these risks until much later, when the context is forgotten and the original decision-makers may have moved on.

This pattern creates a form of hidden technical and organizational debt. Much like poor code documentation or insufficient testing, partial understanding creates gaps that someone will eventually have to pay for. The cost might come in the form of rework, missed requirements, or decisions that have to be revisited and reversed.

What makes this particularly insidious is that we rarely track or measure these costs. When issues surface months or years later, we treat them as new problems rather than connecting them back to their root cause: decisions made without full understanding of the available information.

Proper Delegation

There is a solution, but it requires us to be honest with ourselves and our organizations. If you don't have time to fully read and understand a document, you need to fully delegate it to someone who does. Not "skim it and give me the highlights" delegation, but real delegation with authority and ownership.

This means finding someone who:

Has the technical background to understand the content

Has the context to know what matters to your organization

Has the time to actually do the reading

Has your trust to make or recommend decisions

Most importantly, you need to actually listen to them. If you've delegated the deep understanding to them, you need to respect that understanding. Otherwise, you're just creating extra steps in the "pretending to read" theater.

Changing the Culture

This requires a cultural shift away from the idea that everyone needs to know everything about everything. We need to normalize saying "This isn't my area of expertise, and I don't have time to develop that expertise right now. I'm delegating this fully to Alice, and I'll trust her judgment."

This isn't a weakness - it's good organizational hygiene. It creates clear ownership, clear accountability, and better decisions. It also makes space for people to develop real expertise instead of spreading themselves thin trying to maintain surface-level knowledge of everything.

The next time you're tempted to skim a complex document or run it through ChatGPT, stop and ask yourself: Should I take the time to really understand this, or should I delegate it to someone who can? There's no wrong answer, but pretending to understand is always the wrong approach.

Your organization's decisions are only as good as the understanding behind them. Stop pretending to read documents. Either read them properly or delegate them properly. Anything else is just organizational theater, and the cost of admission is far higher than you think.

Simon Sinek, an author and speaker on business leadership, advocates for playing the infinite game in business. Finite games are played to win. These are things like Chess, Football, Cricket, and so on. Infinite games are played to continue playing. You need to use the right mindset for the right situation, and Sinek argues that Business is an infinite game, not a finite one. After all, what do you “win” in business? Where is the “end”? It doesn’t exist. You can only keep playing or not. Sinek is very critical of the finite mindset, and for good reason.

The technology industry, and American business as a whole, is rife with finite thinking. These companies are concerned about their next investment round or quarterly report. “Winning” is having a good investment round or beating your quarterly estimates. It’s shipping your project and marking it done. This finite mindset has its place in business, but it’s when things are dire. “Winning” is survival. If you’re a startup trying to get investment, you are laser focused on doing what’s needed to get that investment. Drive user aquisition and sales, no matter the cost. You are in an existential fight for survival, and thus have a finite goal.

The trap I see businesses and teams fall into quite often is not knowing when they’ve transitioned from a finite game to an infinite game and thus keep using a finite mindset to play an infinite game. This is the sickness at the heart of American business. We’re all playing the game like it’ll have a winner. This infects all levels of business, from CEOs to individual contributors.

My personal rule of thumb for recognizing when this transition has occurred is reaching profitability. Profitability indicates that you are self-sufficient. You are no longer reliant on investment capital to run the business. Your expenses to keep the business running are less than your income. You’ve won the fight for survival, now you need to keep playing.

Finite Mindset in Software Development

The Finite Mindset manifests in software development most commonly by overindexing on delivery versus quality. When you’re fighting for your life, delivering features is key, since each one could attract or retain customers (and thus investors). This is a process of rapid iteration and experimentation to find the right mix of features that your target customers will pay for. It’s natural that quality will suffer during this process, because quality isn’t the point. Who cares if the code is sloppy as long as customers (and thus investors) like it?

The answer to that question is: the team working on it five years from now. Each sacrifice in quality incurs technical debt and that debt will build up over time. Like financial debt, technical debt can completely cripple you. You may use a credit card to fund your way through an emergency, but eventually the banks will come calling for their payments. Software is much the same, as poor code quality can make the system brittle and difficult to modify. It can introduce weird edge cases and bugs that damage the customer experience. You will pay for those tradeoffs in one way or another. Thinking otherwise is a symptom of the finite mindset. You’re playing to win, not to keep playing.

Your customers will also notice this finite mindset in the features. You probably have some failed experiments in the product and some awkward or sharp edges in core features. For instance, maybe you don’t allow customers to rename a resource because you’re using the display name as the canonical identifier.

Making the Transition

Playing infinitely means building a sustainable organization system that balances delivery and quality. Once you’ve identified that you’re starting to play infinitely (that is, profitable or shortly thereafter), you should do two things: 1) polish core features and 2) pay off some of that technical debt.

Many companies reward larger features over smaller features, which builds products with many large pieces of functionality that kind-of sort-of work. Google, whose promotion process is contingent on large feature delivery, is infamous for this approach, producing a huge quantity of underwhelming products that no one will buy into and will eventually be deprecated. This only works because 80.2% of Google’s revenue is from advertising. Google’s core feature set as a company is advertising, not Gmail or even Search. Those are just vehicles by which to deliver the core feature. Their core product is extremely polished and effective. This is what you need to do as well.

Now that you’ve done that hard iteration process and found a feature set that works for a customer base, identify those core features and polish them to an unnatural shine. Identify any awkward issues that diminish your customers’ experience and fix them. As Ron Swanson puts it, “Don’t half-ass two things. Whole-ass one thing.”

The approach to clearing technical debt is well-understood, but many businesses hesitate to actually execute on it because of the aforementioned obsession with large feature delivery. The people who know your tech debt best are the engineers who built the software system. Allocate more time for them to work on this and let them decide the priority. You will probably want a high-allocation phase where major issues are fixed, then decrease allocation to a maintenance level.

By the time this transition is done, you’ll have a stable software system that delights customers and can continue making money into the immediate future.

Building for Infinity

Now, the true transition to playing the infinite game happens. At this point, your product teams should start looking at features that build on the core offering and you should be in a fairly stable technical state. The question becomes how to maintain this.

Many companies and teams will get a set of requirements and ask themselves, “What is the minimum amount of work needed to achieve these requirements?” This question sets you up for failure, because you’re looking for the bare minimum in terms of quality and feature set for every given deliverable. Over time, your product will thus be bare minimums built on top of bare minimums with an endless collection of technical debt and quirky behavior along the way. You are starting with a low bar and raising it just high enough to pass.

Instead, ask a different question: “What is the idealized solution for these requirements?” When answering this, dream big. Imagine you had all the money and time behind you to do this. What would you build? What would really delight customers? Go deep on requirements. Build the ideal architecture. Write it all down like you’re actually going to do it. Explore the edge cases in the features and the software design. Look around as many doors and corners as you can for anything waiting to get you.

Once that’s done and you have consensus on the idealized solution, start applying the constraints of reality to it. What features are high-cost and low-value? What features are low-cost and high-value? Where can you draw clean subdivisions of the product for incremental delivery? As you do this process, you’ll produce a roadmap for the next 1-2 years. You’ll have a clear vision of what you’re building and why, but it will be broken down into smaller steps and pieces. You can make conscious choices about the technical debt you’ll accept instead of letting it happen accidentally. Start with a high bar and gradually lower it until you can jump over it.

This may sound very waterfall-y and thus scare off any of you who are Agile™. It’s not, though. You aren’t committed to delivering this long-term vision, nor are you planning every little piece of it up front. The vision is just that, a vision. It’s a guiding light. A direction to unify everyone in the organization. It provides a critical reference point for making decisions.

If your environment changes or new business needs arise, you can adapt. This approach will leave you with clean places to cut off the product and software architecture. Just do the exercise again for your changed environment and keep going.

The key benefit of this approach is agency. Rather than let technical debt or shaky product decisions pile up accidentally, you are choosing what sacrifices to make. In a later post, I’ll go into how to build a decision-making framework that allows you to make such decisions wisely.

I use “DevOps” in quotes for a reason. Consultants have warped and twisted what was once a simple concept for the purposes of selling naive leaders tools and products. There are as many definitions of “DevOps” as there are consultants wanting to sell you something. I won’t be using this term except to refer to that vague nonsense.

In the modern world, for the vast majority of cases, software is now a service that requires ongoing development, maintenance, and support. Operating software is no longer a different concern, it’s part and parcel of the job of developing.

Throwing Things Over the Fence

Early in my career, I worked on an operations team where developers would throw their code over the fence. Predictably, these systems broke often and we built hours-long manual testing processes after every release to try to catch things. These ran well into the morning hours, with the operations and product team getting to bed around 2 or 3 AM. Even with all that testing, we still had bad code changes make it to production, and the developers would always complain when we had to pull them off of coding work to handle Incidents and fix their shoddy work.

This attitude was largely based in a feeling of developer supremacy - that developers are “too good” to be oncall or be paged. That’s for someone else, paid less, usually offshore, with a more limited skillset. Operations staff were warm bodies to throw at the soul-sucking task of Operations.

This was draining in a way I cannot describe in words. We were on the hook for someone else’s decision making with no real way of influencing those decisions. I’ve never been so stressed in my life as during those years. Turnover was high, morale was low.

This model simply does not work. It encourages a lack of ownership over your work, sets the wrong incentive structures, breaks key feedback loops, and grinds your organization to a halt.

Service Team

There is a better way, which I call the Service Team model. In this model, developers are responsible for designing and developing features, managing infrastructure, maintaining the long-term health of the system, handling customer requests, responding to Incidents, and maintaining a continuous improvement process. The buck stops with the service team. They own the software top-to-bottom, inside-and-out. I’ve been working under this model for years now with great success. We own everything related to our services. We are responsible for the victories and defeats. These are service teams, not development teams or operations teams.

This model is possible largely because of modern technologies that have lowered the technical breadth required to run a high performance software service, such as public clouds and containerization. No longer do you need an intricate knowledge of servers, load balancers, hypervisors, storage arrays, and networking to run a software service. These are now themselves software services provided by public clouds. Even if you are on-prem, a simple Kubernetes cluster abstracts almost all of this away from your service teams.

Feeling The Pain

Your development process directly dictates your operations pain. Missing edge cases, poorly-performing code, memory leaks, and insufficient logging and metrics all contribute to operations pain. If you’re throwing your code over the fence to an operations team, they are on the receiving end of this pain, but they have no tools to influence improvement here.

Some organizations try to build processes to resolve this gap. I’ve been in an organization where they tried allocating dev time every sprint for operations work prioritized by the ops team. That work was always the first to be deprioritized once someone demanded a new feature. I’ve been in an organization where the operations team was just cutting tickets into a blackhole, never seeing the improvements they needed actually get built. In another organization, operations teams were permitted to develop and submit PRs for their own needs, but those PRs often went ignored because they didn’t contribute towards any development team metrics and took forever because the operations team wasn’t familiar with the code-base.

The Service Team model embeds this feedback loop into the team itself. Developers are far more conscientious of their changes when they’ll be the ones woken up at 2AM when things go wrong and they have poor logging and no metrics to actually diagnose and fix the issue. It’s in the service team’s self-interest to address operational concerns as they are shipping code. This effect is so strong that I often see service teams pushing back on management and product teams that are trying to push them to launch without sufficient operational controls in place.

A Systems Frame

Developers being responsible for code, compute, and other infrastructure forces them to look at their software from a systems frame. They cannot afford to ignore the impact specific load balancer configurations may have on their service. They cannot live in a world of “worked on my machine.” They must face the reality that their software is a complex system that they must understand.

Working in high-scale cloud software, almost every scaling issue I’ve encountered is due to some odd interplay between system pieces. It’s very rarely just code or just infrastructure. Building a deep understanding of the whole system, not just the code or infrastructure, is what enabled us to find and fix these issues.

Implementation

Now that we’ve established why this model works better, let’s look at implementation. There are a few key considerations that need to be addressed:

• On-Call Rotation

• Ticketing Process

• Time Allocation for Ops Work

• Initial Investments

On-Call Rotation

All members of the service team should be in the on-call rotation, from juniors to seniors. Juniors or new hires who are too inexperienced to be on their own should be shadowing more experienced engineers. A rotation time of one week works very well for this and you should have a minimum of 5 engineers on the rotation to prevent burnout. During the rotation, the on-call is expected to respond to production incidents within a timely manner.

During this time, the engineer on-call has one priority: resolve Incidents. Everything else is secondary to that task. If they have no Incidents, should be working the operations backlog, not the feature backlog. This strict separation prevents operational issues from impacting feature capacity, leading to a more predictable feature development pace.

Ticketing Process

I’ll write a longer post on operations concepts and tickets, but at its core, the team needs a ticketing process that allows for three things:

Requests are customer tickets or tickets from other teams asking for support or some other help.

Incidents track instances service degradations. Outages, performance issues, and so on. An Incident is a single occurrence of this degradation. Incidents have different severities, with some being minor and some needing immediate attention.

Problems track ongoing issues with the service. For instance, a memory leak in the service code that occasionally causes Incidents where hosts run out of memory. Incidents and Problems are linked together to enable tracking the impact of a Problem over time, which aids in prioritization.

You need some ticketing system that can handle these three types of operational tickets. Ideally, this system should integrate with a paging tool that can contact the on-call automatically if an Incident of sufficient severity is opened.

Time Allocation for Ops Work

A portion of the regular sprint (or similar) capacity should be dedicated to working a ticket backlog fully owned by the service team. Product managers, team managers, and so on have no power here. The stakeholder is the service team and they get to decide what work needs done in what order. This is where tasks such as logging or metrics improvements, code refactors and cleanup, automated testing, automation work, and so on go.

Some points of caution:

• Do not put feature bug fixes in this backlog. Otherwise, the product teams are incentivized to drive shipping features with bugs and the ops backlog will eventually be inundated with the consequences of that behavior and turn into a bug backlog, choking out actual operational improvements.

• Do not save up for operations-dedicated sprints. Progress is best made slowly and steadily. Deferring operations capacity for shipping a feature will eventually result in the operations work never being done.

This backlog closes the loop between operational pain and impovement. It dedicates time for the team experiencing the pain to actually address it and generates a flywheel effect driving towards operational stability both for the team and the customer.

I recommend starting with 33% capacity dedicated to this work. That sounds like a lot, but you probably have more work here than you realize. As you continue this process, your needs may die down and there not be a need for this much capacity. This is a good opportunity for allowing your service team to experiment. They may even decide to pickup a feature with this time, at their discretion. It’s the service team’s time to do with what they want.

Initial Investments

Most organizations without a strong feedback loop are drowning, whether they realize it or not. Your initial goal is get your head above water and then to start swimming.

Phase I: Stop Drowning

Phase I focuses on two key types of work: improving observability and reducing Incidents.

By “observability” I’m not talking about the consultant-speak o11y where you have to buy a bunch of expensive tools. I’m specifically saying “do you even know what your application is doing?” The vendor tools have some great features and can definitely be used to great effect, but often times they are haphazardly implemented as an after-thought and end up being enormous piles of noise. If you’re so early in your operations journey that you don’t understand how your system works, you can’t possibly hope to utilize these tools effectively and they are not worth the financial investment, which tends to be substantial.

Logging and basic metrics can get you most of the way there. You should be able to trace a specific piece of work (request or message) through your system with some kind of unique identifier. Include that identifier in every log message and aggregate your logs in a single place. Congratuations, you now have distributed tracing. You can see what your application is doing. As your team goes through the service to implement logging and metrics, they will need to analyze the application code and infrastructure to understand what’s actually happening, what logging is useful, and what isn’t. This exercise is just as important as the logging itself and is the main reason the vendor tools fail. They promise that you don’t have to do the hard work of understanding, but the understanding is what makes the tooling functional in the first place.

At the same time, you should be reducing Incidents by driving root cause resolution. If you have a memory leak, you need to ruthlessly pursue it and eliminate it, not just write a script to bounce hosts every so often. Dive into root causes and fix them. The increased understanding granted by the logging and metrics work will help you drive these fixes, reducing the on-call burden and freeing up time for your service team to focus on the work in Phase II, which keeps you from drowning again.

Phase II: Start Swimming

Now that you’re not drowning, you need to invest in operational work that improves efficiency and reduces regressions. This is done by investing in Continuous Integration / Continuous Deployment (CI-CD) processes.

I have a whole post planned on CI-CD, but the short version here is to invest in automated deployment and testing. You should be able to merge a commit and have it in production within a few hours.

Automated deployment is trivial, especially in a cloud environment. Use Terraform or CDK to manage your infrastructure and code deployment processes and simply trigger them from a CI-CD pipeline tool such as Jenkins or GitLab.

Almost all testing can be automated nowadays, so there’s not really a reason to do major releases or gate your deployments based on a QA team. Instead, invest in automated tests using a testing framework of your choice and run those tests against a copy of your production environment. If the tests pass there, you’ll most likely be good in production.

Over time, you’ll find gaps in your CI-CD process that will enter the backlog and you’ll iteratively improve to the point you have almost no deployment failures or production regressions.

Making the Change

If you currently have a split development and operations team and want to make this change, there will be friction. Your developers are probably not used to being oncall and your operations team may feel threatened. You’ll need to address both of these concerns.

I recommend spreading the operations team across development teams and investing in them as developers. This was my career path and it worked out great for myself and my team. Having someone who had specialized in operations processes embedded in the team will help mentor the developers on good operations practices and enable sharing of their experience. Your operations personnel have valuable knowledge and shouldn’t be cast aside. Inversely, dedicated operations staff will be a dying breed in the coming years, so giving them the opportunity to learn software development and engineering will provide them with a career path.

Developers may be more problematic depending on the culture within the team.

If the developers are already accustomed to being involved in operations issues, you can leverage the rotation as a selling point. On-call rotations give your service team a break and a clear understanding of when they need to be engaged with work. They’re allowed to properly disconnect when they are not on-call, which goes a long way towards improving mental health and morale.

If the developers are not accustomed to being involved, you’re effectively asking them to do more work for the same pay. You can immediately solve this by giving an oncall stipend or some additional pay to account for the increase in responsibility. It’ll cost more up front, but will payoff in the long-run as the system becomes more stable and the team is able to automate their manual work, improving their efficiency. Ultimately, though, you’ll need to hire people with expectation that they own their service.

Questionably Functional is a reader-supported publication. If you found this article helpful and would like more, please subscribe below.

Some employers may allow you to expense subscriptions to industry newsletters under their Education or Development budgets. Click here for more information.

Subscribe now

If you have a question or topic that’s been on your mind, and you’d like my opinion on it, you can submit a question using the link in the navigation bar. Thank you!

Having been a junior myself and now having had several on my teams, I have some suggestions on what skills you can cultivate to help you succeed in your early career.

Abandon Your Absolutes

There are no absolutes. There is no “correct” code style. There is no “correct” editor or IDE. There is no “correct” language or framework. Reality is messy and everything is a tradeoff. Sometimes you may need to use a language or framework that you don’t really like because it has the best support internally at your company. Sometimes you need to scope down a project to deliver some pieces faster because customers need it. Sometimes you need to accept inefficiency in your code because your time is more valuable elsewhere.

This is likely a consequence of both the education system, which encourages thinking about things in terms of “right” and “wrong,” as well as pop culture around software development. People enter the industry parroting claims they’ve heard on social media because they don’t have enough experience or context to understand why opposing viewpoints are valid as well. Professors are notorious for this as well. Many have never written a line of production code in their life and they’re going to try to tell you what is and isn’t “correct.”

If you find yourself thinking or speaking in absolutes, stop and think it through. How do you know your claim is “correct?” Do you have evidence? As you dig in, you’ll start to find tradeoffs and nuance that you were previously glossing over. This process is recognizing and challenging your own claims is vital to developing a more nuanced understanding of things.

Avoid Seeking Perfection

You can spend months optimizing your code performance and design, seeking an ever more-perfect version. This is a trap that will lead you to poor productivity.

Learning how to see when something is “good enough” is a critical skill. Yes, your algorithm may not be fully optimized, but given the traffic needs, compute environment, and cost tradeoffs, is it “good enough?” Let’s say you’re able to manage a 2% performance improvement in an algorithm by spending two weeks on it. Are you actually going to save enough compute cost for that to be worth the cost of labor and opportunity cost of you working on something else? Often times, getting 80% or 90% of the way there is good enough, and it’s time to move on.

This is something I struggled with for years and frankly still do from time to time. I want to build something perfectly and can fall into a hole where I’m fussing over details that are ultimately unimportant. Recognizing when you’re doing this and allowing yourself to step away and say “This is good enough” will serve you well.

Learn to Self-Direct

Coming out of college, you’re probably used to the exact opposite of self-direction. Classes give you an assignment with a clear scope and a clear deadline. You do the work, you turn it in. Rinse and repeat for four years until you get the piece of paper. This situation is incredibly rare in the industry, and you should throw away any expectation of clear requirements and clear deadlines, even as a junior. We as seniors often try to give well-scoped tickets to juniors, but there’s a tradeoff for us as well. In the time it would take me to write up a precise spec for a junior, I could probably do the work myself. It also doesn’t push you to learn if you’re just a Written-Word-To-Code-Translator.

In reality, you will get vague requirements and vague deadlines. It’s up to you to figure out exactly how to solve the problem at hand and estimate how long it’s going to take you. Practice decomposing problems into smaller pieces. I use work breakdown structures for this, where I just create a bullet outline of the work as if I were writing a paper. It helps me identify the big pieces. Dive deep into each of those and work out any edge cases or odd behavior.

Once you have that done, send it along to someone to sanity check your work. Iterate until you’ve got it.

Recognize the Difference between Experimenting and Grinding

You’re given a task that’s beyond your current experience. You don’t already know how to solve it, so you go exploring. Reading documentation, trying different approaches, reading StackOverflow, and so on. You’ll find a lead and follow it. Maybe it works, maybe it doesn’t. If not, move to the next lead. This is experimentation and is a critical part of learning. You need to be able to explore an unknown space and try different things on your own as part of being self-directed.

The fatal mistake is doing this for too long. Once you’ve exhausted the leads you have and start trying random things, you’re no longer experimenting. You’re grinding. Grinding away at work will lead to burnout very quickly. This is when you should escalate to a more senior engineer for help.

It’s a bit of a meme in software that juniors ask senior engineers for help and get one of two answers: “Why are you bothering me with this?” or “Why didn’t you ask me sooner?” These answers depend on when you’re escalating. If you’ve spent 10 minutes on a problem, give up, and then message me, you’re likely to get a “Did you actually do anything?” kind of answer. If you wait two weeks, grinding away while your soul slowly disintegrates, then ask, you’re going to get a “Why didn’t you ask sooner?”

Here’s the trick to do this correctly:

1. Read any related documentation or training

2. Look for existing solutions if available by searching repositories, internal or external

3. Walk through the system and understand it. Reverse engineer if needed.

4. Once you’ve run out of leads, escalate.

I just used this process yesterday when trying to do something kind of unnatural with an internal Amazon library. I read the documents, found nothing. Asked in a couple slack channels and no one had tried this before. Started reverse engineering the library and found a specific Factory class that look encouraging. Worked to see if I could customize the Factory. Found someone else who had done it by searching the code repositories. A couple of hours of experimenting latter, I had a working solution.

If you’ve done all that and still run into a dead end, escalate. Use this template:

Hi <Senior Engineer>,

I’m trying to solve Task X, but I’ve hit a dead end. I’ve tried the following:

• A ended up not working because E

• B was giving me an error F that I couldn’t root cause

• C would be unsupported by the library vendor according to the documentation

Can you point me in the right direction?

By demonstrating that you’ve already done some things and are just at a dead end, you’re more likely to get helpful answers because we don’t have to walk you through the basics. We can also take a look at issues E and F to see if we can find a solution.

Beware the XY Problem

XY Problems are the bane of anyone asking for or giving help. XY Problems occur when the asker is trying to solve Problem X, does some exploring, and runs into Problem Y which prevents them from solving X.

Rather than ask for help about X, they go and ask for help about Y, which may or may not make any sense once given the full context. Instead, always communicate something along these lines:

I’m trying to solve problem X. I found this documentation which indicates if I do Z, I can solve X. Unfortunately, when I tried Z, I ran into problem Y.

This provides crucial context when asking for help, as your initial jump from X to Y may not make sense in the first place or there may be a better route. Just because you found Z first doesn’t mean it’s the right way. Even if I helped you with Y, your final solution Z may have some other issue.

Be Honest and Self-Aware

If you’re falling behind on a task, you need to clearly communicate that and why you are falling behind so you can get help. Yes, it can bruise the ego, but getting fired because you can’t do your work is going to bruise much more, I promise. If you’re on a supportive team, be honest about what’s up and what you need help with so we can provide it. Message people directly if you aren’t comfortable doing it in a daily meeting.

I’ve seen so many engineers go radio silent for weeks trying to do a single task. Meanwhile, resentment builds up in the rest of the team. “What is Joe even doing?” Then, we just go ask Joe what’s up, and it turns out he’s been grinding on something for a week that could’ve been fixed in a few minutes if he’d just asked.

Don’t be like Joe. Ask for help.

Breadth Before Depth

At this point of your career, you should be building breadth. If you’re doing feature work mostly, pickup an infrastructure task. If you’ve been working on synchronous API code, go work on something asynchronous. Try a new language. Go fix a security issue. Find some automation work that needs done. Shadow an on-call rotation. Pickup a front-end ticket. Write a product spec.

This approach serves you well in your early career because you probably don’t know what you actually enjoy. You’ve done some basic coding in school, but do you have any idea what actually interests you? What gets your brain going? If you’re stuck doing the same thing for years, you may end up with a specialization that you can’t stand and ultimately burn out. You may end up lacking skills that make you competitive in the job market and struggle to find work.

I started out doing infrastructure, automation, and monitoring. I had a broad scope and could pretty much get my hands dirty with whatever I wanted. I learned something important about myself in doing so: I really enjoy being a jack-of-all-trades that can bridge the gaps between specializations and I’m pretty good at it. I would never have found that out if all I was doing was configuring monitoring tools.

Continue Learning

It’s a hard fact that most degree programs don’t actually prepare you to be a software engineer. You’re learning the foundations: sorting algorithms, programming paradigms, different languages, data structures, and so on, but no one is teaching you how to breakdown and estimate work, write production-quality code, troubleshoot production issues, or learn how to use common third-party libraries.

You must continue learning on your own. Read books. Read blogs and newsletters. Ask questions. Your education didn’t stop once you got the piece of paper to hang on the wall. All that does is prove you know the absolute bare minimum about Computer Science, not that you’re an engineer.

Most of my career advancement has come from reading books to get the theory, then using opportunities at work to try those theories out. Some I found to work great, like Clean Architecture and Anti-Patterns. Some I found rather useless, like Test Driven Development. High Conflict taught me how to recognize and interrupt systemic cycles of conflict. Management taught me the principles underlying management as a practice so I could lead my team and interface with my managers more easily.

To keep advancing in your craft, you must keep learning, and you generally must do this on your own. There are no more classes with strict curricula to follow. You need to make your own. Be honest with yourself and find the things you aren’t great at. Take opportunities to fix those, be it a project in that space, reading a book, getting a certification, and so on. Step by step, you’ll turn yourself into a leader.

Questionably Functional is a reader-supported publication, providing a single free article monthly with weekly paid articles. If you found this article helpful and would like more, please subscribe below.

Some employers may allow you to expense subscriptions to industry newsletters under their Education or Development budgets. Click here for more information.

Subscribe now

If you have a question or topic that’s been on your mind, and you’d like my opinion on it, you can submit a question using the link in the navigation bar. Thank you!

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

— Melvin E. Conway

While not based on any mathematical theory of systems, this Law has proven true in the real world consistently enough that it’s safe to treat as an axiom of organizations.

This often manifests negatively, with people bemoaning Conway’s Law as the driver of their poor software architecture and ownership. It brings with it a rather sobering conclusion: the true software architects in your organization are not the expensive engineers or systems specialists… they are the HR department and executives. The problem is that they don’t generally understand that they are doing systems engineering when designing an organization, nor are they typically trained in it.

Fortunately, there is a simple approach we can implement that will instantly align your organization from top to bottom, enabling you to build a truly agile business. That tool is called domain modeling, as popularized in Domain-Driven Design.

Using Domain-Driven Design principles allows you to build a Domain-Driven Organization that uses Conway’s Law to constantly reinforce a strong socio-technical system aligned with your business needs.

• Use domain modeling to deeply understand the space in which you operate

• Using that new understanding, structure your organization along the domains and document the dependencies and communication lines

• Recurse into each domain and continue modeling

• Let Conway’s Law do the work of keeping your business aligned with the model from top to bottom, empowering your teams to have full ownership over their space of the domain model while still driving towards a common purpose

Domain Modeling

For the purposes of this article, we’re going to use a mock domain model I did recently for a genetics testing service similar to 23andMe. Let’s look at some of the functions we need:

• Sample Collection - Get samples from a customer

• Sample Processing - Process the physical samples to produce a genome

• Genome Analysis - Run tests against the genome data, checking for the presence of specific genes

• Reporting - Collect the results from the analysis and store them

• Customer Experience - Present the results to the customer

Sample Collection and Sample Processing are deeply intertwined, because the type of sample needed, how they are stored, how they are transported, what equipment is needed to process them, and so on, are all related. We’ll bundle these into a Sampling domain.

Genome Analysis is unique enough as a domain. It’s responsible for taking a specific input, the genome, and producing a specific output, the analysis results.

Reporting, similarly, is unique. What do we do with those results? How do we analyze them, share them, and do research against them?

Customer Experience manages the front-end customer experience, from the packaging of the kits to the front-end website.

Let’s map these out in a domain model, along with some of the basic entities inside each domain.

The dependencies are as follows:

• Customer Experience depends on Sampling for the sample containers and instructions for use.

• Customer Experience depends on Reporting for the analysis results so they can be presented to the customer.

• Sample Collection depends on Genome Analysis to specify the genome format, storage location, and upload procedure for the genome data.

• Genome Analysis depends on Reporting to specify the result format and storage location

That’s it. For this whole business, we have four dependencies and thus four organizational relationships to be managed.

Next up, let’s look at the skill sets we’ll need in each domain.

The Customer Experience domain needs designers, both computer interface and packaging designers, logistics specialists, manufacturing experts, and software engineers.

The Sampling domain needs lab specialists, people who know how to design a testing lab, lab technicians, equipment and facilities managers, sequencer device specialists, and software engineers.

The Genome Analysis domain needs scientists, bioinformatics engineers, and software engineers.

The Reporting domain needs researchers and software engineers.

Notice that, aside from software engineers, almost every domain has unique needs. This is a good sign that we’ve defined a strong domain model - specialist knowledge is contained within a single domain.

The Domain-Driven Organization

In order to weaponize Conway’s law and align our entire organization, we need to align the organization to the domain model. Let’s hire a Director for each area of the business. We also need to revisit our communication channels and make sure that those are all managed by a single owner. That will typically be a Product Manager for each domain who is responsible for ensuring that these domains all work well together. Aside from these interactions, the organizations can operate largely independently.

Now we need to break things down further, because you need more than two people for each domain. We’ll pick the Genome Analysis domain to look at as an example.

One of my favorite parts of the domain modeling approach is that it’s recursive. As the Director of Genome Analysis, I need to figure out what my organization structure looks like. Let’s do a domain model for this specific domain.

• Genome Storage - We need to securely ingest, store, and audit access to genome data, including custodianship requirements

• Bioinformatics - Specialists who understand genome data and can build the analysis jobs

• Analysis Platform - We need a way to run arbitrary analysis jobs against a genome

Mapping out their interactions, we get the following:

There are two primary interactions:

• The Analysis Platform domain needs to provide a common interface for the analysis jobs, as well as define a way to ingest and execute those jobs

• The Genome Storage domain needs to define how the Analysis Platform can access genome data

We’ll need Product Managers to manage those dependencies.

Now let’s build an organization structure.

Those accustomed to traditional organizations will notice something odd here. We have Product Managers and Security Engineers embedded into two of our domains.

This is intentional.

It’s been proven time and time again that building a multi-disciplinary team focused on one common goal drives results because it gives that team complete ownership. That stack of people under the Genome Storage manager is wholly and entirely in control of how genome data is stored, accessed, and audited. They are running their own little business-within-a-business and are empowered to do what’s necessary to succeed.

This is the end goal of the domain-driven organization. Ownership and empowerment for everyone with a clear scope and vision that aligns with the overall business, held in place with a seemingly inevitable law of organizations.

Mitigating the Side Effects

This model does have some downsides. Namely, your specialist groups of software engineers, product managers, and security engineers are isolated from each other. This could lead to drift or inconsistent standards across the organization, especially as it grows. In other words, how do we ensure that these job functions are staying unified in their approach and standards?

This is where treating your organization as a system instead of a hierarchy becomes valuable. The diagram above operates on a single vertical axis, enforcing superiority of one individual over another. Let’s get creative and add a second axis, this time looking at what job specializations there are.

We’ve added a second form of organization here, the Guild. While not the only approach to solving this problem, the Guilds serve as job function specific organizations that evolve independently of the business-domain oriented structures. This empowers the Security Guild to decide “We are going to deprecate TLS 1.2” and that decision have force across the organization. The Product Management guild may decide they want to track goals using a specific tool.

While the above chart is just the Genome Analysis domain, it should be understood that these guilds span the entire organization. Guilds are responsible for training and improving their job specialization, while the Managers operate their business function. As the organization grows, it may be wise to hire dedicated heads of these guilds whose entire function is to improve their job specialization across the organization.

Managing Growth

Eventually, your organization will scale or extend into some new line of business that doesn’t align with the previous model. Maybe we want to support Ancestry now instead of just simple gene presence checks.

When situations like this arise, start the process over with the top level domain model. Analyze how the additions impact the existing model and continue down recursively. This can happen at any level of the organization, and the owners of that domain must be empowered to do this analysis and modify their portion of the organization as needed.

How to Get There

This example is a new organization, but you may already have one in place that needs to be migrated. I’ve been through this process myself as an engineer who did the domain modeling that led to organizational change.

The first step is to model what you actually have. Map out your socio-technical system along with its communication lines. It is critical that you involve key people of all specialization at all levels in this exercise. You need depth and breadth here to produce an accurate model of where you’re at currently. You need to be self critical here. You’re going to find silly things. Write them down in the model and move on.

The second step is to model where you want to be. It’s important that this is the second step, as you don’t want your idealized model to bias your current state model from the first step.

The third step is to figure out what the minimal set of changes is to get from where you are to where you want to be. Prioritize these changes to target areas that are known problems. Communicate these changes broadly and explain them so that everyone understands what’s happening and what the goal is.

Finally, start making changes. Do one change at a time, see how it works, and continue. Do not attempt to reorganize your entire organization at once. This will cause massive amounts of friction and decrease morale and productivity. You may need to hire people or move people around. Do not be rash and layoff entire organizations. Be slow and strategic. You are playing an infinite game, so act like it.

Over time, you’ll slowly align with the idealized domain model.

Conclusion

Domain-Driven Organizations empower small units to fully own their space while still remaining aligned to the overall model of the business. It enables you to think about your organization as a complex system instead of as a hierarchy, allowing you to apply systems design and analysis to it. It produces high quality software and human systems with clear boundaries that align with the business.

But why? We’re humans. Having received the gift of fire from Prometheus, shouldn’t we be looking for a better way?

• Hierarchies enable accountability and are essential for humans to function at scale

• Organizations are complex systems of interdependencies and communication channels that do not align to a hierarchy, akin to organs in a living organism

• Only individual organs know how best to serve their functions and should be empowered to do so

Read more

You can use systems thinking and design to break down a project from technical design to tasks, organize those tasks into work streams, set up reporting, and provide accurate estimates to your stakeholders.

You can accomplish this by doing the following steps:

1. Find the work streams

2. Model dependencies between the work streams

3. Breakdown each work stream

4. Find the KPIs for each work stream

5. Build the project system

6. Monitor and adjust

We’re going to look at two examples inspired by projects I’ve led, modified to protect proprietary information. For the first example, we’ll discuss a service that consists of an API that takes in asynchronous work requests and a workflow that processes those requests. For the second example, we’ll talk about a major project that touches over a dozen services with hundreds of changes.

Find the work streams

Typically, your software will have boundaries that separate components and you can leverage these to separate your work streams as well. A good tech design will help guide you through this process. We’ll use this when we go through the first example above.

If there isn’t a tech design, you may need to look at the work itself. Assess the work that needs done and find any common patterns in how it will be done. Those patterns will help us define the work streams in the second example.

Model dependencies between the work streams

Work streams may have dependencies between them. These dependencies will eventually result in bottlenecks where one work stream is waiting on another. To find these, inspect the boundaries in your software. Are there API definitions that need completed? Database models? Message formats? Figure these out, and prioritize resolving these first.

Breakdown each work stream

For each work stream, start at a high level and find which pieces can be done independently. For instance, you may be building two sets of CRUD APIs for two different resources so you can build these separately. For each resource, you may need to define a data model which will prevent working on each of the Create, Read, Update, Delete APIs. You don’t want the Create and Delete APIs operating on completely different data models! Model these dependencies between high level tasks. As you do this, you’ll get a sense for what can be done in parallel and what tasks are blocking others.

Find the KPIs for each work stream

If you’re leading a project of any size, there will be someone wanting to know how progress is going. For some projects, it may make sense to assign task points and try to report the progress on those. For some projects, that could be a complete waste of time and something like ticket count would be a better fit. For others, you may report on the completion of each sub workstream, such as which APIs are done and which aren’t. Work with your stakeholders to find the right way to report progress.

KPI (Key Performance Indicator) is business word to many engineers, but it’s really just a metric to measure the success of a particular system. As software engineers, we know how to measure systems. Consider each of your work streams and what you would need to measure to report status to your stakeholders.

Build the project system

Now, we have the information we need to start figuring out how the project should function. We have our work streams, we understand the dependencies, and we know how many people we have. You’ll generally want to assign a single owner to each work stream, unless a stream is especially small and can be easily managed by someone with split attention. Model out the project as you would any other system, including reporting lines and work streams.

Monitor and adjust

No plan survives first contact with reality, and your project system is no exception. People will get sick, work streams will move more slowly or more quickly than expected, priorities will shift, and so on. Keep a close eye on the system, identify constraints, and eliminate them.

Example One: Async Workflow

For this example, we have a simple system where an API ingests work that a workflow completes in the background. Here’s the (very) high level technical design.

1. API writes the work request to a datastore

2. The datastore event triggers the workflow

3. The workflow performs the work and marks the request as completed

4. When the caller requests the status, the API reads from the datastore

Let’s go through the process outlined above to build a project system that can deliver this software system.

Work Streams

First, we can see three main parts of the system: The API, The Datastore, The Workflow. In this case, the work streams fall out of the architecture diagram. Let’s start by considering each of these a separate work stream and move to the next step.

Dependencies

Looking at the diagram, we can see that The API and The Workflow depend on The Datastore. The API and The Workflow are fairly independent, though. There may be some common code, but the bulk will be in how they interact with The Datastore. We can rework our diagram to show these dependencies.

We have a problem, though. According to this diagram, we need to complete The Datastore before we can start work on The API or The Workflow. This means those two work streams will be stalled waiting on the third, which isn’t great. We want to go fast.

Fortunately, with a strong architecture in our code, we can abstract away the implementation detail of The Datastore. I won’t get into code architecture in this post because it deserves its own, but just imagine The API’s code and The Workflow’s code operate against interfaces instead of making database queries.

With this abstraction, we can break the dependencies and work on all three work streams simultaneously.

Breakdown The Work

Now, we need to get some tasks. I recommend using a simple bullet list to start. For the API workstream, we might produce something like this:

• API Infrastructure

  • Create containers

  • Create load balancer

• CreateFoo

  • Define API model

  • Implement

  • Testing

• ReadFoo

  • Define API model

  • Implement

  • Testing

• UpdateFoo

  • Define API model

  • Implement

  • Testing

• DeleteFoo

  • Define API model

  • Implement

  • Testing

And so on and so forth. Once you have your bullet list, it’s worth doing a check for dependencies again. We want to make this maximally parallelizable. Fortunately, if you are following a strong testing pyramid design, these can all be done individually. You can build the CreateFoo API and test it without having any real containers running. You can continue breaking this bullet list down, perhaps a task for each test. At this point, we’re talking tasks and not work streams, so we’ll leave it here.

Reporting and KPIs

For this example, we can probably follow a traditional task pointing system. The work is fairly well understood and easy to estimate. Once you go through your planning sessions and assign points as a team, you can just report progress on those points as your KPI.

The Project System

Let’s say we have 5 engineers that can work on this project: Alice, Bob, Charlie, and Dani. The last one is the project lead: you.

We have three parallelizable work streams: The API, The Workflow, and The Datastore. We’ll assign Alice to The API and Charlie to The Workflow, as they need opportunities to lead a small project. You must delegate ownership of these streams to them. We’ll get into delegation in another post and how to use it to empower your fellow engineers and help them grow. The Datastore is especially concerning, as you need to make sure the data model is extensible. You’ll take that responsibility as the most experienced engineer on the team.

This leaves Bob and Dani unassigned. The Datastore doesn’t have a large volume of work, so you assign them to Alice and Charlie’s work streams respectively.

Since Alice and Charlie own their own workstreams, they only need to communicate with you insofar as escalating major issues or resolving conflicts on the boundary. You are the only one communicating with stakeholders, so they have a single point of contact and there is no confusion when communicating.

This diagram illustrates the system for the project team. You are not interacting with Bob and Dani since Alice and Charlie are directing them. The Stakeholder is not communicating with anyone else, ensuring clear communication from the project team. Any issues in the work streams are escalated through you, the project lead.

Monitor and adjust

When we executed this project, we encountered some unexpected complexity in the Datastore that resulted in almost two weeks of extra planning and another two weeks of implementation work. Once we realized that it was going to be more complex, we were able to pull a more experienced engineer off of the Workflow workstream to assist. The API also ended up being simpler than expected, so we were able to rebalance one engineer from that workstream to the Workflow.

Pay close attention to how your project system is behaving and adjust accordingly.

Example Two: Multi-Region Support

For Example Two, we’re going to look at a project with a larger scope and a different structure. This project is to take our system consisting of around a dozen microservices and enable it to launch in any AWS Region. Until now, the service had only needed to run in one region, so there are a lot of places in the code that just refer to the one region. Further, we need to consider data replication across regions and a failover mechanism should one region stop functioning.

Work Streams

For this project, we could structure the work in a few ways. For the first pass, we look at enabling this service-by-service, where we would define our work streams based on the service, so a dozen work streams. That’s a lot of parallel work to track and doesn’t really look at the entire system.

The first step we’ll take is to get an idea of the work that actually needs done. After completing an audit, we find the following:

• Hundreds of static references to a specific region

• Twelve datastores that need to support cross-region replication

• Three workflows that need to support some kind of cross-region exclusivity

• Updating deployment pipelines to support multiple regions

After doing this audit, we can really break it down into four major types of work:

• Static Configuration Fixes

• Datastore Replication

• Workflow Exclusivity

• Deployment Pipeline

Since these types are all likely to have similar solutions, we define these as the work streams. This will allow the people working each stream to focus within a particular problem/solution space, and we can assign them based on their skills.

Dependencies

Going through the dependency modeling exercise, we know that Datastore Replication and Workflow Exclusivity are going to be blocked on the Static Configuration Fixes. We can’t really test solutions for cross-region replication or exclusivity locks when the services can’t even be deployed in multiple regions due to hardcoded values. Updates to the Deployment Pipeline are going to have similar concerns.

Unlike Example One, we can’t really abstract the dependencies behind interfaces to break them during development. We can do some exploration and design in the later work streams, but not really much more. We need to figure out how to structure the work so that the blocked work streams can actually get started.

Looking at the work streams, we identify specific tasks along another dimension: the microservice. Let’s look at three hypothetical microservices: Thor, Odin, and Fafnir.

As we see, we actually have twelve divisions of work to go on. Further, we know that Fafnir needs Odin present to function, and Odin needs Thor. We model our dependencies like this, which gives us an ordering to do the workstreams in for each service.

Let’s model this as a directed graph, taking into account the dependencies in the actual work, and the order we want to do these services in. This will come in handy later.

Breakdown the Work

For each of these, we’ll do the standard work breakdown process as covered in Example One.

Reporting and KPIs

For a project like this, it may be tempting to report based on the work streams, but the reality is that your stakeholder doesn’t care about the implementation details of your services. They want to know how far you are through the service list that needs fixed and when you can launch in the new region. For this project, we actually want to report on that extra dimension we added above, the service. But we are planning to do the work based on the type of work, not the service. Our stakeholders are going to complain if we say no services are done, then suddenly they all are, so we need a way to approach this work so that we can report service-by-service, but allow our engineers to work in a specific problem space.

The Project System

In a case like this, we need to embrace a system inspired by graph traversal. We will walk the dependency graph, actively working on any nodes that have no blocking dependencies.

First, we’ll have our Static Configuration team tackle Thor’s configurations. When this is done, we have unblocked the Datastore Replication and Workflow Exclusivity work streams. We’ve also freed up the Static Configuration workstream to move on to Odin.

Now, we have three parallel work streams running: Odin Static Configuration, Thor Data Replication, Thor Workflow Exclusivity. Let’s say they all finish at roughly the same time, so we can update our graph. Now, we have all four work streams running.

In the next phase, we will have completed the Thor Deployment Pipeline work, which will allow us to report that Thor is completed to our stakeholders. The next round, we’ll finish Odin. The following, Fafnir. The project is done.

Now there are some caveats to this approach. First, we still have work streams waiting on others. This is just unavoidable in such a situation, so we may have later work streams doing design or investigation work while the Static Configuration team works on Thor. Alternatively, we can slowly ramp people into this project as their work streams become unblocked. They could also participate in some of the other work streams to move things along faster. Flexibility is the name of the game with this approach. Everyone should be ready to do what’s needed at any given time.

So where does that leave you, our project lead? You will need to wear two hats. First, manage the graph traversal. Keep track of the progress being made on each item and what downstream impacts it may have. You may need to shuffle people around to keep things balanced. Second, work through the more complex issues where your technical guidance may be needed.

Monitor and adjust

This project system is a complex dance of timings and downstream impacts, and needs close monitoring and rapid adjustment. If any one work stream starts lagging behind, it could delay the entire project significantly.

You will need to move people around and find creative ways to work around dependencies. This is a highly dynamic process that can only be managed through good KPIs and an attentive project leader.

Conclusion

Leading large projects can be overwhelming and intimidating at first, but at its core, you just need to find the right system. Example One shows us a basic system where the work aligns to the software being built. Example Two demonstrates a more complex arrangement where you opt to align the work to the problem or solution spaces at hand.

In both cases, you’re ultimately aligning to some understandable sequence implicit in the work itself. Learning to identify how the work is structured naturally will enable you to apply the right approach for any given project.

Questionably Functional is a reader-supported publication, providing a single free article monthly with weekly paid articles. If you found this article helpful and would like more, please subscribe below.

Some employers may allow you to expense subscriptions to industry newsletters under their Education or Development budgets. Click here for more information.

Subscribe now

If you have a question or topic that’s been on your mind, and you’d like my opinion on it, you can submit a question using the link in the navigation bar. Thank you!

This is the case with remote work and office work. One isn’t implicitly better than the other, but they do require different techniques. If you’re experiencing friction in a remote or hybrid team, the odds are high that you’re just trying to use the wrong techniques to build your organizational system. This is the core problem with many claims about remote work’s alleged ineffectiveness. By neglecting to look at organizations through a systems frame, it’s easy to believe that specific techniques are the only solution available. That is, that gas stoves are the only option to cook food effectively. Instead, you must look at the system and the needs it actually has. Once identified, think broadly and deeply about how you can resolve those needs in certain contexts and with certain constraints. To torture the metaphor one last time, an open fire is a delightful technique to cook food when out camping, but applying that technique indoors will lead to a visit from the fire department. Think critically and choose wisely.

In this post, we’ll take a look at some common needs working in a white-collar office job and explore how different techniques can be used for each situation. Having worked successfully both in-office and remote for many years, I’ve learned a few things about how to address common gaps. We’ll also touch on some of the unique benefits enabled by work flexibility that can be billed as employee perks to attract talent.

Information Sharing

Crossing Silos

Advocates of an office environment will claim that there is improved information sharing compared to remote work, specifically citing the “water cooler conversations” or “after meeting chats.” Water cooler conversations are typically short exchanges of information, such as a problem you’re having or a project you’ve started. These exchanges are incredibly valuable to an organization, because they help expose gaps in the formal communication mechanisms.

The error here is thinking these are ends in-and-of themselves. That is, “You can’t have water cooler conversations when there is no water cooler.” That’s obvious, but are water cooler conversations some implicit good? No, absolutely not. They are only useful as a technique for sharing information across typical boundaries and silos. If you have no water cooler, you need to address this need with another technique.

There are a few options here, the ones I’ve seen most often being sync meetings and adhoc notifications. I’m generally against sync meetings, even in the office. The cold hard reality is that people largely wait for their turn to speak in these meetings and don’t really engage consistently. We’ll get into meeting engagement in another post, but I think we can all agree this is more-or-less true. Doing these meetings over video conference only adds a layer of disconnection.

Instead, try adhoc notification mechanisms. Create a place where teams can send high-level notices and updates, such as a Slack channel, email list, etc. If you’re starting to explore a particular problem space, send a notice. If you’ve delivered something of high value, send a notice. If you need guidance on something others may have already solved, send a notice. This fully replaces the need for water cooler conversations, and actually performs the function better. It’s not just two or three people who all sit near each other getting the information, it’s dozens of people across the company. Managers, product managers, and senior engineers can watch these channels casually, looking for anything that needs their attention.

During my time at Amazon, this was used to great effect. First, it was interest mailing lists. By subscribing to lambda-interest , you’d get notifications about things happening with AWS Lambda. Later, this moved to Slack, but the function remained largely the same. These mechanisms have been critical for me when trying to keep tabs on what’s happening across an organization of that size.

Diving Deep

The “after-meeting chat” is another technique that is often cited as necessary. Reflecting on my experience with them, these chats were almost always deep dives into some specific area. Maybe someone brought up the need for encryption in a meeting, but wasn’t sure how to do it. You stay with them for 10 minutes afterwards to share your experiences.

There’s absolutely no reason this can’t be done remotely. Slack’s Huddle feature is perfect for quick chats like this, and I use it all the time. The only difference is that it requires hitting a button instead of awkwardly standing around a meeting room. You should encourage each other to be available for conversations like this, as they are critical to the “riffing” cited by certain executives. If people on your team are not making themselves available, this is a performance management problem, not a process problem.

The Rest

Any mechanism based on a meeting remains more-or-less the same when working remotely. Weekly or monthly updates are usually meetings. Sprint ceremonies are usually meetings. These are just as easy to do remotely as in-person. Encourage people to turn their cameras on during meetings to help with communication. We’ll get to that topic later.

Raw Productivity

Ultimately, we all have work to do. Write code, write design documents, meet with customers, look for candidates on LinkedIn, etc. This is all work that requires some degree of uninterrupted focus.

Modern open office floorplans, which I’ll address in another post, are probably the greatest sin committed against this need. People need to focus, and in order to do so they need workspaces that allow them to focus. Typically, this means noise cancelling headphones everywhere. Why are we building office environments that are so harmful to productivity that people feel the need to isolate themselves anyway?

Working remotely is clearly better in this instance. Employees are empowered to build workspaces that enable them to work most effectively. They can have as much cleanliness or distraction as is necessary. For me, I need to fidget when thinking, so I have an array of fidget toys on my desk that can make as much noise as I want. My cat also sleeps nearby, so I can get some ear scritches in during a stressful meeting. Empowering employees this way is absolutely critical if you are aiming to support a more diverse workforce. We’ll get into diversity in a later section.

Collaboration

Collaboration is the trickiest area for remote work. In-office advocates will rightly point out the power of whiteboarding sessions. Getting people in front a whiteboard together to brainstorm or work through technical designs can have a great effect, but that’s not the only way to collaborate. Let’s look at the needs that the whiteboarding technique addresses:

• Live feedback

• Diagramming and writing

• Broad input

Live feedback and broad input can be achieved with virtual meetings. The real challenge virtually is with diagramming and writing.

Virtual whiteboards exist and have been very effective for me, but ultimately what I’ve found to work best is shifting to a more asynchronous collaboration model inspired by the Japanese business practice of nemawashi. Nemawashi is, as defined by Wikipedia, the “informal process of quietly laying the foundation for some proposed change or project by talking to the people concerned and gathering support and feedback before a formal announcement." Have one or two people take an initial pass at the design, then share it out to collaborators for feedback. Your collaborators can take some time to review it, really think it through, then send you any changes they think need to be made. Update the design and send it back out. Do this until you have some broad agreement. If there are any sticking points between collaborators, you can schedule a short focused meeting to address that point.

I truly believe this model is better for collaborating on all manner of work. It enables deep thought on the topic, encourages feedback from everyone, and ultimately produces a better product. Companies like Amazon have been doing this very thing through their narrative process long before remote work was common; where someone writes a narrative document, gets it reviewed, updates it, gets it reviewed, and so on until there is agreement. This approach comes with the benefit of being self-documenting as well.

Team Building

Lack of connection to your team is another issue that comes up. The traditional team building techniques, such as lunches, events, and happy hours, are just not possible remotely. You need to take a compeltely different approach to team building when working remote, but it is completely possible. You can have human connection when remote.

Cameras on, folks! I know this is controversial, but I’ve never really understood the disdain for cameras in meetings. If you were in the office, you wouldn’t be able to hide behind an off switch. Having your camera on enables your team to see your body language and facial expressions, which are both critical components of human communication. Disembodied voices are for airport announcements. If you’re in a meeting, turn your camera on. It’s essential to how humans communicate. If you get self-conscious staring at yourself, most virtual meeting tools support turning off your own camera display.

In the office, teams rely on happy hours, game time, and so on to try to foster communication. Even before remote, this was getting less and less popular. My team had a standing happy hour starting around 3PM on Friday. Most people just showed up for the free food and left at 3:30PM for a start to an early weekend. This kind of team event is low effort and does nothing substantial. This is made even worse when it’s remote, because no one wants to sit in front of a camera doing nothing for two hours. Instead, invest in team events such as virtual escape rooms, or multiplayer games. We did Jackbox and Among Us on my team a few times, which were great experiences. These types of events are good because they are drop-in and people can come and go as needed. If anyone of you play video games, play them together. The key thing is to get people tackling problems of some kind together. Nothing builds community and trust like going through something challenging together, even if it’s just a game.

From the perspective of the company, the most vital thing you can do is to arrange offsites or conferences. Use all that money you’re saving on rent, leases, property tax, subsidies, maintenance, HVAC, and more to bring your teams together two or more times a year. It’s expensive, yes, but it will foster a sense of community like nothing else. You can get meals together, brainstorm, run lecture workshops, announce big initiatives, have cross-organization planning sessions, and so on. Events like this contribute to resolving pretty much every gap in remote work, and will accelerate your business and build your team. They are a great return on investment and also serve as a great employee perk to advertise, because who doesn’t like an all-expenses paid trip to different cities?

Work Flexibility

Remote work is itself a technique for your organization that comes with its own benefits, the first of which is flexibility.

Flexibility in work is one of the largest benefits to remote work. People need to get to the doctor, go to the bank, take a pet to the vet, handle a home repair, and so on. These are things that exist regardless of where the person works, but are so much easier to handle with remote work.

In the office, your employees’ only option is to take time off work to get things done. Even if they take a partial day, the office may be nowhere near their vet doctor, so they are commuting even more than usual. By enabling employees to work remotely, you empower them to balance their work and their life needs far more effectively.

A home repair could mean working 6 hours and dealing with the repair for 2 instead of taking a full day off. A couple of years ago, I had my home’s plumbing completely redone. It took several days with several plumbers. If I were mandated to be in the office, I would have had to take the days off, which would have meant no productivity for my employer. Since I was working remotely, I could keep working and just check in on them every couple of hours. It was actually beneficial for my employer because I was still working and still delivering to meet deadlines.

Child care is another great example. Working remotely, you can drop your kids off at school and pick them up on time, saving the cost of after-school care or babysitters. Is your child sick? You can keep an eye on them throughout the day while still getting work done. A child being sick may not mean a client meeting is cancelled.

These are major perks for employees that also benefit the employer because their workforce is more predictable and consistent.

Diversity, Equity, and Inclusion

I won’t pretend to be an expert on DEI, but I do my best. That said, I’ve seen the benefits that remote work can bring to organizations, and they cannot be overstated. Remote work is itself a technique to improving diversity.

Neurodivergent individuals may suffer from open office floor plans where distraction is high and social interaction is a constant consideration, but thrive when able to work in a space suited to them. Motherhood no longer precludes a woman from working, because work flexibility empowers her to do both. Those with medical conditions, such as the 15% of the population with IBS, are able to continue working without fear of no bathroom being available when lightning strikes. Those with mobility impairments don’t need to be concerned with crossing several streets to get to another building for a meeting. These are all benefits to improving the diversity of your workforce.

You can even enable diversity in your workforce in unexpected ways unrelated to the traditional protected classes. Remote work enables a geographically and culturally diverse team that can bring in new opinions and perspectives. For example, I worked for a team whose primary customer base consists of homeowners with a mind towards securing their property. I was shocked to discover that I was one of only a few people in my department who owned a single-family home in a city with one of the highest property crime rates in the country. How are you going to truly understand your customers if no one on your team is actually in that situation? How will people living in secured apartment buildings and condos understand the needs of someone who owns a single-family home? They can’t. They’re just making educated guesses. By enabling diversity of where people live and work, you can empower your business to understand customers in ways that were previously impossible.

If you’re serious about DEI and not just using it as marketing on your website, supporting remote work is imperative. Diversity is good for your business and for your employees.

Conclusion

Remote work isn’t ineffective, it’s just different. If you’re trying to use the same techniques as an in-office organization to run a remote team, you’re simply doing it wrong. You’re going to have friction and rough edges. Embrace remote work and learn the techniques needed to manage a distributed organization. Rather than lamenting what’s been lost, dive deeper and find a way to replace it or improve it.

The burden is on you, the employer, because employees absolutely will leave you for someone who embraces more flexible work options. Do you want to be left behind, fighting your employees, or do you want to lead the charge by applying systems thinking to your organization, doing the hard work, and building a talented and diverse workforce that gives you a competitive edge?

Questionably Functional is a reader-supported publication, providing a single free article monthly with weekly paid articles. If you found this article helpful and would like more, please subscribe below.

Some employers may allow you to expense subscriptions to industry newsletters under their Education or Development budgets. Click here for more information.

Subscribe now

If you have a question or topic that’s been on your mind, and you’d like my opinion on it, you can submit a question using the link in the navigation bar. Thank you!

Systems rule our lives in ways that aren’t always obvious. These can be massive systems, like the economy. These can be small systems, like the way your favorite restaurant runs their kitchen, or even your morning routine.

We live within these systems, but rarely stop to think about them deeply. To some extent, this makes sense. What impact are you going to have on the stock market’s trading system standing in the shower? We go along with our lives, but the harsh reality is that these systems are all questionably functional at best. That is, they rarely serve the people within them effectively.

By better understanding how systems work, we can reflect on the systems we live in and work to improve them. Helping people understand these systems is a primary goal of Questionably Functional.

Technology leadership

As I’ve progressed in my career, one thing became abundantly obvious: once you get out of your early career guidance in this industry dries up very quickly. Early on, there are dozens of books and YouTube channels you can learn from, but as you get into the complexities of organizing, inspiring, and developing people, they dry up. Sure, there are books like Radical Candor that can teach you some tricks, but at the end of the day, you’re on your own. I’ll be sharing strategies I’ve used personally when leading software teams to keep people engaged and growing.

Software quality

Building quality software is fairly well understood, but balancing it against competing demands of cost and time can be deeply challenging. The right tradeoff often depends on your existing software, your business, and your organization’s needs. Finding the right mixture is more an art than a science, and one I’ve become fairly adept at. I’ll be reviewing examples of tradeoffs I’ve had to make along with what led me to bias towards a particular solution.

If any of this sounds interesting, please subscribe using the button below. There will be one free post monthly, paid posts weekly, and the occasional bonus post if something strikes my fancy. Thank you for your support.

Subscribe now