Sunday, July 7, 2013

Signs your software process isn't working

The business has made it. The product has been well-received. New customers are coming in the door and partners are signing up. You're even profitable! But, something is rotten in Denmark.

Software releases feel slower. You haven't measured it, but you're pretty sure that the number of regressions is up. Emergency releases seem to be happening more often. New features aren't happening as quickly as they used to. Some features are failing now, which never happened before. And when new features do get out, it seems like everyone is exhausted. Some of the early employees are moving on. Some of the first customers are complaining. It's just not as sparkly.

The good news is that it's not in your mind. There are real issues that are causing all the pain. They can be identified, measured, and surfaced to the rest of the organization.

The other good news is that you can fix it. There are real and concrete things your groups can do to reduce the turmoil. Some of the problems can even turn into assets after some grooming.

But, that's about the extent of the good news.

The bad news is that there is no quick fix. Switching your technology stack isn't going to fix it. Hiring a rockstar developer or a great project manager isn't going to fix it. Adding "Agile" (or kanban or whatever) is only going to muddy the waters. And throwing more people at it is only going to exacerbate the problems.

Yes, problems. Because there isn't just one problem. There are multiple problems that contribute to this sense of unease and dread. Not everyone has the same set of problems, and some groups are unique snowflakes and have their own special brand of crazy. But most groups tend to run into some combination of the same sets of issues.
  • Measurement is haphazard.
  • Cross-checking is haphazard.
  • Responsibilities aren't clearly defined.
  • There is no assembly line.
  • All knowledge is internalized.
These issues tend to be carryovers from the attributes that made the company successful in the first place. One or two highly motivated people start a project. They know everything and communicate with each other constantly. Quality is high because it's small. Process is whatever gets something out the door. And, amazingly, everything scales. Going from two or three people to ten means 5x as much work is getting done. So, why did everything just fall over when going from ten people to fifty?

Fred Brooks, in The Mythical Man-Month, touches on one of the root issues. When dealing with three or four people, the number of 1-to-1 lines of communication is manageable (at six and ten, respectively). Everyone can get into one room and share all the knowledge about everything. The business is small enough that a single person can keep the entire thing in their head.

Going up to ten people increases the number of lines to 55. This is much larger, but still manageable because we have specialized. Everyone isn't expected to be able to do everything anymore, so some information can be limited to sharing with just some people. And one person can still keep everything in their head, even if they do less and less of the daily work.

The more astute reader is starting to see where the problem starts to form. Everyone has worked at a place where it's nearly impossible to find out what you need to know in order to do your job. Information is siloed. It is rare that someone is actively hiding the information. (If that does happen, the solution is very simple, if emotionally and politically difficult to do.) More often, the person who has the information you need doesn't know you need it. Sometimes, you don't know you need it, until you can't move forward without it. And one person can no longer keep the entire business in their head.

At the root, there is a limit to the amount of information and cross-references a single person can handle. There is also the limit of how much work a single person can accomplish. We organize groups and companies to exceed those limits. One person doesn't have to communicate with anyone. A few people, usually less than a dozen, can communicate together clearly together. One person can manage what a dozen or so people do. Beyond that, we need systems and processes in place to formalize how information and knowledge are organized, prioritized, and transferred between groups and people.

Alongside the problems communicating information dealing with today's tasks, we have to communicate yesterday's accomplishments and tomorrow's plans. New coworkers have to be trained and environments set up. Those who are leaving need to have their information retrieved. Everyone needs to know what will be coming. The number of information streams rapidly becomes unwieldy without explicit boundaries and organization around them.

In short, the corporate organism needs to learn how to think.