The Mikado Method

Copyright

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact

    Special Sales Department

    Manning Publications Co.

    20 Baldwin Road

    PO Box 261

    Shelter Island, NY 11964

    Email: 

[email protected]

©2014 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.

Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.

ISBN 9781617291210

Printed in the United States of America

1 2 3 4 5 6 7 8 9 10 – MAL – 19 18 17 16 15 14

Brief Table of Contents

Copyright

Brief Table of Contents

Table of Contents

Foreword

Preface

Acknowledgments

About this Book

About the Cover Illustration

About the Authors

1. The basics of the Mikado Method

Chapter 1. Meet the Mikado Method

Chapter 2. Hello, Mikado Method!

Chapter 3. Goals, graphs, and guidelines

Chapter 4. Organizing your work

2. Principles and patterns for improving software

Chapter 5. Breaking up a monolith

Chapter 6. Emergent design

Chapter 7. Common restructuring patterns

Appendix A. Technical debt

Appendix B. Setting the stage for improvements

Appendix C. Dealing with dynamically typed languages

Index

List of Figures

List of Tables

List of Listings

Table of Contents

Copyright

Brief Table of Contents

Table of Contents

Foreword

Preface

Acknowledgments

About this Book

About the Cover Illustration

About the Authors

1. The basics of the Mikado Method

Chapter 1. Meet the Mikado Method

1.1. What is the Mikado Method?

1.1.1. Basic concepts

1.1.2. When to use the Mikado Method

1.1.3. Benefits of the method

1.2. The Mikado Method recipe

1.2.1. How to work with the Mikado Method

1.3. Summary

Chapter 2. Hello, Mikado Method!

2.1. Your first Mikado Graph

2.1.1. Making changes without breaking code

2.2. A slightly more complicated change

2.2.1. Getting to know the code

2.2.2. Reverting code

2.3. Making the code configurable

2.3.1. Get going with the Naive Approach

2.3.2. Revert when there are errors

2.3.3. Select the next prerequisite

2.3.4. Create an interface

2.3.5. Restart from the goal

2.4. Summary

Chapter 3. Goals, graphs, and guidelines

3.1. The Mikado Method guidelines

3.1.1. Always start with the goal

3.1.2. Use experiments

3.1.3. Revert—the key to safe refactorings

3.1.4. Focus on the leaves

3.2. The Mikado Graph

3.2.1. The goal—business or technical?

3.2.2. The prerequisites

3.2.3. Prerequisite—step or decision?

3.2.4. How big should a step be?

3.2.5. More about the leaves

3.2.6. The dependency arrows

3.2.7. The check marks

3.3. Tips for using the Mikado Method

3.3.1. Focus on the goal

3.3.2. Follow the flow of the code

3.3.3. Keep it simple

3.3.4. Keep it visible

3.3.5. Iterate, increment, and evolve

3.3.6. Remember the state

3.3.7. Reflect on your work

3.4. Relation to other thinking models

3.4.1. Theory of Constraints

3.4.2. Empirical control systems

3.4.3. Pull systems

3.5. Summary

Chapter 4. Organizing your work

4.1. The scope of a change

4.1.1. Changes at different scales

4.1.2. When do I start drawing my graph?

4.2. How to approach a change

4.2.1. Working in different constellations

4.2.2. Where to put the focus in a team

4.2.3. Pain points when working as a team

4.3. Summary

2. Principles and patterns for improving software

Chapter 5. Breaking up a monolith

5.1. The code of an online loan institute

5.1.1. The existing functionality

5.1.2. The architecture and the code

5.2. Beheading the beast

5.2.1. Set a goal

5.2.2. Create an abstraction

5.2.3. More LoanRepository duties

5.2.4. Back to the business goal

5.2.5. Update the graph

5.2.6. Revert and restart from a clean slate

5.3. Getting to the true leaves of the graph

5.3.1. The first steps on the way back

5.3.2. Side effect–free programming

5.4. Summary

Chapter 6. Emergent design

6.1. Designing classes

6.1.1. Code that does too many things

6.1.2. Code that isn’t stable in the face of change

6.1.3. Code that violates its contract

6.1.4. Bulky interfaces

6.1.5. Code with rigid call-dependency chains

6.2. Designing packages

6.2.1. What to put in a package

6.2.2. Dependencies between packages

6.3. Summary

Chapter 7. Common restructuring patterns

7.1. Graph patterns

7.1.1. Group common prerequisites in a grouping node

7.1.2. Extract preparatory prerequisites

7.1.3. Group a repeated set of prerequisites as a templated change

7.1.4. Graph concurrent goals with common prerequisites together

7.1.5. Split the graph

7.1.6. Explore options

7.2. Scattered-code patterns

7.2.1. Merge, reorganize, split

7.2.2. Move code directly to a new package

7.2.3. Move code gradually to a new package

7.2.4. Create an API

7.2.5. Use a temporary shelter

7.3. Code tricks

7.3.1. Update all references to a field or method

7.3.2. Freeze a partition

7.3.3. Develop a mimic replacement

7.3.4. Replace configuration with code

7.4. Summary

Appendix A. Technical debt

A.1. How you get into debt

A.1.1. Acceptable debt builders

A.1.2. Unavoidable debt builders

A.1.3. Unnecessary debt builders

A.1.4. Bad debt builders

A.2. More about technical debt

A.2.1. Lactic acid

A.2.2. What is the interest?

A.3. Influence and type

A.3.1. Third party—technical and imposed

A.3.2. Bigwig—market and imposed

A.3.3. The boardroom—market and incurred

A.3.4. Propeller hat—technical and incurred

A.3.5. Borderliners

A.4. Attacking the technical debt source

A.4.1. Get to the bottom of the problem

A.4.2. Third party—create defensible space

A.4.3. Bigwig—probe and prepare

A.4.4. The boardroom—talk and learn

A.4.5. Propeller hat—form an opinion and align

A.5. The way out of technical debt

A.6. Summary

Appendix B. Setting the stage for improvements

B.1. Before your restructuring

B.1.1. Shorten your feedback cycles

B.1.2. Use a version control system

B.1.3. Use automated refactorings

B.1.4. Use one workspace

B.1.5. Use one repository

B.1.6. Merge frequently with the main branch

B.1.7. Scripting languages to the rescue

B.1.8. Use the best tools

B.1.9. Buy the books

B.1.10. Know the dynamic code

B.1.11. Consistency first, then improvements

B.1.12. Learn to use regular expressions

B.1.13. Remove unused code

B.1.14. Prepare yourself mentally

B.1.15. Prepare others

B.1.16. Measure code problems

B.1.17. Hire a critical mass of change agents

B.1.18. Encourage collective code ownership

B.2. During the restructuring

B.2.1. All things are relative

B.2.2. Be aware of the code

B.2.3. Create an opinion about what good code looks like

B.2.4. The road to better often runs through worse

B.2.5. Code of conduct

B.2.6. Find the ubiquitous language

B.2.7. Pay attention to your gut feelings

B.3. When the crisis is over

B.3.1. Refactor relentlessly

B.3.2. Design never stops

B.3.3. You ain’t gonna need it—YAGNI

B.3.4. Red-green-refactor

B.3.5. Continue using the Mikado Method

B.4. Summary

Appendix C. Dealing with dynamically typed languages

C.1. How to get feedback in a dynamic environment

C.2. The loan server (again)

C.2.1. A run-through of the Node.js code

C.2.2. Extending the test

C.3. Summary

Index

List of Figures

List of Tables

List of Listings

Foreword

Most software discussions, books, and articles seem to assume that development work starts with an empty codebase. The Agile literature usually presumes that the detailed learning about what is needed to solve a problem for a stakeholder community can be discovered iteratively and incrementally. This might be a valid assumption if one is starting fresh, but I seldom encounter teams with the opportunity of starting from scratch; there are always constraints. Most investment in software today involves modifications or extensions to existing applications and environments. Thus, in addition to discovering and implementing solutions to our organizational problems/opportunities, we have the constraint of fitting into the environment created by earlier development teams. We need to discover and codify not just the applicable domain knowledge, policies, and organizational goals that drive our current development work, but also to understand how the changes we make affect the existing application environment.

Everyone who has done this kind of work knows that the information we need is only weakly represented in the documentation left behind by preceding teams. One has to look at the source code to get reliable information about the constraints it imposes on additions and changes. If the source code includes an effective set of automated tests, we’re in luck because tests illustrate the behavior required of the code by previous implementers to solve previous problems. If these tests pass when executed, we know that the code behaves as they expected. More often, automated tests were never created, and we’re left with only the source code itself. The question then becomes how to learn what we need to know to avoid breaking previous work.

The barrier is sheer complexity. Analysis of the preexisting code has proven to be a weak tool for tackling this complexity. Much of the code we have to confront no longer communicates effectively to us the nuances we need to understand to avoid breaking it as we implement our changes, and it usually ends up taking far more time than we can afford. Therefore, after we exhaust our patience with analysis, we’re forced to go ahead and take a chance at breaking the fragile existing code. What we do next is critical.

Traditional approaches have ended with a long, tedious, unpredictable, test-and-fix period that often consumes the second and sometimes third 90% of our project schedule. Agile and lean thinking have taught us to test immediately, integrate continuously, and fix every problem as soon as we discover it. This works well with well-structured code that’s weakly dependent on its environment—provided we have an effective automated test suite. The teams who preceded us, however, have seldom had the perspective, skills, or inclination to leave us with such a foundation, because they were driven mostly by the need to get-it-done (project scope focus) even at the expense of keep-it-clean (longer-term product lifecycle focus). So we find ourselves in a hole!

The first rule of holes is this: When you find yourself in a hole, stop digging! Agile thinking suggests that for all the code you add to an application, you should use your refactoring, clean coding, and TDD practices to ensure that you don’t make the situation worse. You’ll probably have to add some higher-level automated functional tests to be able to do even this safely. Even then, complexity easily becomes overwhelming, and quality, speed, and cost all suffer. Managing this unavoidable complexity is what this book is about. Rather than permitting complexity to cascade, The Mikado Method enables you to discover what you need to know and address it in manageable pieces by maintaining your focus on always having a known good code base. Ultimately, this is the deepest core of Agile—always ensure that the work you have done so far is correct.

While the form of the Mikado Method is simple, it’s tied closely to and greatly helps with the application of principles and practices we’ve learned over the last decade. It addresses individual, pair, and team-wide practices to reliably tackle changes in small steps. Ubiquitous language and the SOLID and DRY principles characteristic of clean coding practices guide our Mikado Goals and the trees of prerequisites that structure our work. Mikado Graphs help keep our packaging, dependency, TDD, and refactoring work properly focused.

The initial focus of the Mikado Method is effectively dealing with a messy reality. The long-term goal, however, is to understand the forces that drive organizations to create bad code in the first place so we can avoid creating more code that our successors will have to struggle with unhappily. When we’re confronted with the normal levels of complexity in today’s software, we can’t keep everything we need in our heads at once. People have discovered many ways of effectively working together in complex tasks, starting with the scientific method and extending to the Theory of Constraints, empirical control systems, and pull/flow systems. The Mikado Method’s approach works with all these strategies to help us stay in control.

TOM POPPENDIECK

Preface

We once worked on an application that was used to configure the behavior of a specialized electronics module in an expensive hardware product. Our application was a part of the delivery of that product. One day, news reached our software development team that Sales had unexpectedly landed a contract with a new client. At first we didn’t know what to make of it, but in time we learned that the new client needed a version of our application that was almost identical to the one we had. This was fortunate. But after further investigation, we realized that there were certain parts of the application code that under no circumstances could be shared between the new client and our old ones. We had to come up with a solution to this problem, or we couldn’t deliver to the new client at all.

It finally boiled down to two possible solutions. We could duplicate the whole application, or we could restructure first and break out the parts that couldn’t be shared. Our team argued for well over a week before we reached consensus. We decided that the best solution was the restructuring approach: restructure first, and then add the new functionality. This shouldn’t take more than a month, or maybe two, we thought to ourselves.

We started working, and guess what? The work was more complex than we could imagine! The approach we chose was supposed to be simple, and we thought we could get away with moving some classes to new packages, fixing the compilation errors, and then adding the new functionality. But as we tried to split the application and separate sensitive information from the more general parts, we ran into trouble.

We had anticipated some trouble, but not that much. After every change we made, we got about 20 compiler errors, and as we tried to fix them, a myriad of new errors came up. When we were just about done with the new errors, the compiler moved ahead a bit, only to reveal another round of errors. Fixing those errors led to more restructurings and of course more errors.

More than two weeks went by, and we had well over a thousand files checked out and more than two hundred compiler errors. By that time, the high-level system tests, our indicator of a healthy application, had long since been put out of play because we couldn’t even compile the application.

To make matters worse, the rest of the team was moving along at their regular pace, and our efforts at keeping the code in sync became more and more difficult. It became obvious that we couldn’t keep up with the rest of the team, and that the complexity of the change was more than overwhelming. We were trying to behead a Software Hydra—for every head we cut off, two more grew out.

We still remember the day when we swallowed our pride and made the tough decision to revert everything we had done so far. Only later did it dawn on us that reverting the code was a defining moment in our software careers. Apart from realizing that it’s never too late to turn back, we had also discovered how to approach our particular problem, and other problems like it.

Our insight was that in order to get the system in shape, we had to do almost all of the things we had already done, but we needed to do them in the opposite order. You might think this sounds crazy, but the key was to take the errors and turn them into prerequisites.

Brownfield Development

You’ve likely worked with systems like the one we described—software hydras that aren’t easy to behead. For us, this is more or less the norm. We’ve had to clean up lots of messy code, refactor smaller and larger parts of systems, and restructure complex architecture. We’ve even tried rewriting code from scratch, also known as greenfield development. But we’ve noticed that the field doesn’t stay green very long. We’ve long since faced it; we’re stuck with brownfield development, whether we like it or not.

The Software Hydra—for every head we cut off, two more grew out.

We, the developers, hold the fate of the code in our hands, and we’re the only ones who have the power to improve it. It’s our responsibility to keep the code clean and fit for its purpose. This means we have to be able to add code and improve our own code and the code of others. If computer programs don’t improve, they’re doomed to a slow death.

How do we approach large improvement efforts? How do we deliver while the pressure is high, because we must deploy functionality once a month, or even more frequently than that? How do we deal with huge systems that are hard to comprehend, and whose complexity makes it hard to keep all the necessary details in our heads at once? We morph systems into a new shape!

The Goal: Morphing a System

Most organizations that deliver software don’t enjoy halting the development of new functionality. The stakeholders want to see value coming out as money is put in. As a result, software developers have to be able to cope with the current situation as future problems are encountered, solved, and sometimes avoided entirely.

If we want to be successful software developers, we need to learn how to morph an existing system into a desired new shape. When we say morph, we mean perform the necessary actions and steps that take us from a system of one shape to another without breaking things in between. The transitions between the different states need to be smooth and possibly add value themselves.

What complicates the process of morphing software even more is the fact that it’s more or less the norm to develop software as a team. Software is very seldom developed by a single individual, and working in groups makes distributing the work possible. But teamwork also adds another dimension of complexity, and it doesn’t take long to realize that the need to communicate necessary improvements within a team is essential for any real progress.

The Mikado Method

The Mikado Method is our guide to morphing. It helps us visualize, plan, and perform business-value-focused improvements over several iterations and increments of work, without ever having a broken codebase. It enhances communication, collaboration, and learning in software development teams. It also helps individuals and programming pairs stay on track while doing their day-to-day work. The framework that the method provides can help the whole team morph any system into the desired new shape.

The method itself is straightforward and simple: after you’ve decided on a goal, you start to build a model of required changes and dependencies by doing quick experiments.

We try to accomplish the goal in a straightforward fashion. When we can’t, we create a graph of prerequisite nodes. We use the information from our quick experiments to determine whether a change can or can’t be implemented right away. If a problem occurs, we roll back to the original state. By solving the prerequisites first, we work our way backwards, and the graph we build guides us toward our goal.

The Name of the Method

The Mikado Method gets its name from the Mikado game.

Mikado is a pick-up sticks game originating in Europe. In 1936 it was brought from Hungary to the USA and was mostly called pick-up sticks. This term is not very specific in respect to existing stick game variations. Probably the Mikado name was not used because it was a brand name of a game producer. The game got its name from the highest scoring (blue) stick Mikado (Emperor of Japan). The Buddhistic Chien Tung also contains a stick called emperor. (Wikipedia, February 2011, http://en.wikipedia.org/wiki/Mikado_(game))

To start the game, you drop the bunch of straw-sized sticks onto a surface, creating a pile. The goal is to pick up as many sticks as possible to score points, and preferably the Mikado stick, because it’s the highest-scoring stick. If you’re unable to pick up a stick without moving another one, the stick is put back and the turn goes to the next player. As long as you can pick up sticks without moving other sticks, your turn continues.

The trick is to pick up the easy sticks first—the sticks that have no other sticks on top of them. Eventually, by using that strategy, a player can pick up the Mikado stick and likely win the game.

The similarities between the Mikado game and restructuring software are striking. There are loads of dependencies and structural complexities in code that have to be taken into account. These dependencies have to be navigated and changed with care until you have an opportunity to do something without breaking stuff.

Characteristics

These are some characteristics of the Mikado Method:

It fits nicely in an incremental process.

It’s very lightweight (pen and paper, or whiteboard).

It increases the visibility of the work.

It provides stability to the codebase while you’re changing it.

It supports continuous deployments by finding a nondestructive change path.

It improves communication between people.

It enhances learning.

It aids reflection on the work done.

It leverages different competencies, abilities, and knowledge.

It helps collaboration within a team.

It scales by enabling distribution of the workload over the team.

It’s easy to use.

A Thinking Tool

When we introduce the Mikado Method to people who develop software, we often see how they change the way they look at, approach, and talk about large structural improvements. Their perspective shifts from a very analytical view of problems to a more practical approach, where the focus is on removing the minimum number of obstacles at a time in order to achieve real results.

The effect is shorter, focused conversations that are about finding the changes that can be made without breaking the code, instead of rigorous analyzing and guessing. They get more insight on how to deal with the unnecessary complexity of a software system, and the method serves as a thinking tool when they solve difficult problems.

Even though we’re becoming increasingly seasoned developers ourselves, we still find it challenging to decide where to start digging and where to take the code. The Mikado Method not only helps us find a starting point, it also shows us where to go and tells us when we’re done.