Taiichi Ohno is credited with the creation of the Toyota just-in-time production system, and his book “Toyota Production System: Beyond Large Scale Production”  is a surprisingly good read even today when many of these principles are considered well established.

My interest was in understanding how this philosophy applies to Agile/Lean Software Development.  I was pointed this way by Tom and Mary Poppendieck’s excellent book on the same subject, as well as an insightful paper from Fujitsu on the subject of Innovation in Software Development Process by Introducing Toyota Production System (pdf).   As I am reading I am surprised at how perfectly the advice from Mr. Ohno fits to the advice I would give to a software team for Lean/Agile software development

Mapping Concepts to Software

Before the advice applies, you need to understand how software maps to 20th century automobile manufacturing.  Ohno’s central theme is to eliminate waste.

One form of waste is the consuming of parts (which cost money) into incomplete products (which can not be sold yet).  The most wasteful form of mass productions would be one where you build 1000 car chassis, then you put 4000 wheels on them, then you put 1000 engines in them.  All the time you have this inventory of 1000 incomplete cars until they are finally finished.  He would prefer that there be only a single incomplete car at a time.  The reality is that it takes time to put the parts together, and so there is an assembly line with a number of cars being worked on at the same time, and the real goal is a very steady even stream of not only the finished cars, but also all the parts flowing into the process.  The total accumulation of incomplete cars, as well as the accumulation of incomplete parts, adds to the amount of waste, and should be eliminated to the degree possible.  If you have a significant amount of over production, then there is additional waste from the need to store the extra parts and handle them multiple times.

For software, the bytes that form the lines of code cost nothing.  You should not think of lines of code as being the parts of the system, and you should not think about the source code as being the output of the process.  This focus on the source code as the “product” is one of the biggest misconception in the software industry.  Reality is less tangible.  The “parts” that are consumed are simply the “person-hours” that are invested in discovering and developing a feature.  This includes time from not only programmers, but all the various roles on the team.   The output is best described as  “satisfied customers”.  While slightly inaccurate, this phase is meant to remind us that it is not enough to burn a CD with the program on it, but the final sale is for installed software which is successfully used to solve customer problems.  If you have done the job right, they will be “satisfied”.

Just as in a car factory, you do not want to build 1000 cars in parallel, similarly in software you do not want to create 1000 features in parallel.  As you start to build a car, the waste builds as you add more and more parts, but the waste disappears when the car is completed and ready for shipment.  The uncompleted cars are a liability on the organization, and building cars in parallel multiplies the amount of waste.  Similarly, once your start designing a feature, and start investing person-hours, the waste builds in direct proportion.  Finally, when the new feature is actually in a “customer-ready” shape, that waste disappears.  The started-but-uncompleted features are a liability.  Features designed but not coded is a waste of time.  Code that is written but not tested is a real liability because there is now a hidden amount of work that must be done to correct problems in the coding.  Completed features in the code, but can not be installed are still a waste.  Building a bunch of features in parallel multiplies the waste.

Careful readers will notice that the above discussion is slightly simplified.  Even a perfect factory would have cars in an incomplete state, and a perfect software team would have some feature in an incomplete state.  It is wrong to call all items in transit “waste”, rather it is waste only if it is more than the minimum necessary.  We want to minimize the number of incomplete features, not eliminate them entirely.

While you can envision a software development team as a factory that produces “features”, you need to be very careful.  When building 1000 cars/day, all of the various steps are highly repeatable.  The time for each step can be measured to a high precision, and the exact costs of parts can be predicted.  For software development no such predictability exists.  You start a software feature with no reliable estimate of the cost.  This does not change the goal of elimination of waste.  Even though you don’t know ahead of time the amount of waste, it still makes sense to runs thing in such a way as to minimize waste in whatever amount occurs.

Quotes from the Book

Understanding the mapping, we can now take quotes directly from the book (including page numbers), and translate them in to meaning for software development teams.

The loom stopped instantly if any one of the warp or weft threads broke.  Because a device that could distinguish between normal and abnormal conditions was build into the machine, defective products were not produced.  – p6

Automated testing and continuous testing are a central pillar of Agile development.  Ohno is talking here about looms that were perfected by Toyoda Sakichi from 1910 to 1926.  Amazing that the concept of continuous automated tests was in use a century ago.

Stopping the machine when there is trouble forces awareness on everyone. – p7

When a test fails, a software team should drop whatever it is doing and address the problem of the test.  This does force awareness on everyone.

…the workers themselves should push the stop button to halt production if any abnormality appears. – p7

Stopping a production line is expensive, but here we see that Ohno understand that the expense of an unfixed problem is far greater.  The biggest enemy of software quality is ignoring test failures.  Once I joined a team where the building of the product produced thousands of warnings from the compiler.  The developers felt that warnings were harmless, but they are not.  First thing I had them do was fix the code that caused the harmless warnings, so we could then see if any important warnings came out.  Allowing anyone to check in source with a broken test propagates those broken tests, and that causes a blindness to the real problems in the code.

Repeating why five times, like this, can help uncover the root of the problem and correct it.  -p17

So much of software is written to get around problems at other layers of software.  Code bases often seem like a huge elaborate house of cards.  Compensating code is distributed throughout to hold it together.  One should ask: “why is that routine written this way”, “why does that support routine return this result”, “why is this value needed”, etc.  Addressing the problem at the lowest level will eliminate the need to multiple copies of the compensating code, and usually reduce the potential for bugs.

We regard only work that is needed as real work, and define the rest as waste. … we must make only the amount needed. -p19

Here Ohno explains an important concept that overproduction is waste as well.  Well known in software methodology as YAGNI.  Don’t implement anything until you need it for an actual end-user use case.  Contrary to this, programmers often pride themselves on “complete” implementations of interior object classes, commonly adding a host of getters and setters which are not (yet) needed by anyone, but they anticipate will be needed at some time in the future.  This additional coding is precisely the kind of overproduction that Ohno is speaking of: not only does it waste the programmer’s time to implement it, but the excess lines of code increase the code review and maintenance burden in the way exactly analogous to the need to store the extra parts in a car factory.

I used to tell production workers one of my favorite stories about a boat rowed by eight men.  One rower might feel he is stronger than the next and row twice as hard.  This extra effort upsets the boat’s process and moves it off course. -p24

Lessons on teamwork are always useful, and this points out the danger of measuring anything intermediate to “customer satisfaction”.  The goal in rowing is to get the entire boat to the end line as quickly as possible, and a focus on anything else, such as individual force on the oar can distract from this goal.  In software, getting a workable feature to the final customer is the goal, and measuring anything else, such a number of lines produced, or bugs found at a particular phase, can distort the process in unacceptable directions.

The operating method of the Toyota production system is kanban.  This piece of paper carries information that be divided into three categories: (1) pickup information, (2) transfer information, and (3) production information. -p27

Kanban was a way of decentralizing the control of production.  The factory is oriented on pull-based production, and the kanban is a way to communicate the detailed need for production, in order to avoid over (or under) production.  The kanban might be considered to be analogous to storycards that are used to communicate details of a customer use case.  This is only the top level, because kanban is used to communicate within the factory within any producing/consuming relationship, and for the most part this is NOT done in a software team.  Sometimes a bug report can be used to allow one team to make a request for enhancement on another team, and to track the progress.  In this case the bug report is analogous to the kanban.  But caution should be used in the analogy: remember that software parts are unpredictable by nature.  The very regularity of car parts, and the known quantity and cost of the parts, makes it very easy to implement specific requests for production.  In software, however, every part is unique and custom made for the situation, and so a much richer, more interactive discussion is needed.  Kanban should not be implemented literally in a software development team to replace internal communications.  Still, the idea that performance level is tuned according to the need to produce a given use case is very important.

I have good reason for emphasizing the role of top management in discussing the first rule of kanban.  …management commitment and strong support are essential to the successful application of this first rule. -p31

Here he is talking about moving control of production from the central planners, and replacing that with the more “autonomic” method of Kanban.  It is interesting to see how this must have been very uncomfortable for upper management to “trust” that the people on the floor would regulate things correctly.  It is the same with moving software development to an agile approach, where individuals on the team self-organize to get the current sprint done.  The waterfall approach gave more control to external authority, and also provided a better way to measure progress.  Changing to a lean software approach means that this control and to some extent this ability to report progress is eliminated.   This top management support is essential.

Unless one completely grasps this method of doing work so that things will flow, it is impossible to go right into the kanban system when the time comes. -p33

The key word here is “flow”.  In a factory the flow of physical parts is paramount.  In software, the flow of feature ideas is paramount.    The flow of continual builds, and regular sprints, with customer quality code available at all times is the key.  Stockpiling a huge number of features for implementation in an 8 month project is precisely like the mass production of cars that was beginning to show signs of inefficiency in the 60’s and 70’s.  Mass production creates waste.  Waste is eliminated by a steady flow of product releases, each quickly generated from the last.  This is the central theme of the book, and it is the central theme of Lean Software Development.

In the beginning, everyone resisted kanban because is seemed to contradict conventional wisdom.   … To make kanban understood throughout the company, we have to involve everyone. If the manager of the production department understood it while the workers did not, kanban would not have worked.  -p35

How interesting that Mr. Ohno had so much pushback.  He admits else where that it was non-intuitive, but remember that conventional wisdom is based exclusively on what you used to do.  We see the same with software method, including the need to include everyone.  If you train part of a team on the method, you will surely fail, (and I have direct personal experience with this).

While traditional planned mass-production system does not respond easily to change, the Toyota production system is very elastic and can take the difficult conditions imposed by diverse marked demands and digest them. – -p37

This is the promise of Lean/Agile software development: the continuous flow of small features allows for much more rapid response to market changes.  It is ironic that cars with all their physicality can be developed in an agile way, while software which due its lack of physicality aught to be easy to change, but is still developed with a waterfall model making it impervious to change.

Production Leveling:  It had been a long accepted production fact that continuous punching with one die in the press beings the cost down.  It was considered common sense to product in the largest lots possible and punch continuously without stopping the press. -p38

In the 1940’s, Toyota’s die changes took two to three hours … By the late 1960s it was down to a mere three minutes. -p39

The idea that large production runs can reduce cost is well established, and is exactly counter to the idea of producing only as many of a thing that you need.  Not deterred by this, Toyota figured out how to eliminate much of the waste in small production runs.  Ohno had faith that die changes could be made faster, and without that faith it would never have been accomplished.

The same thinking perpetuates waterfall software development: the established “fact” that there is overhead in checking in and testing code, therefor, if you have to touch the code you should implement as many features at once as possible.  If you have the orientation that testing is long and difficult, and you do it only once a quarter, then the idea of doing it once a day is simply impossible to consider, and quick development sprints equally impossible.  However, if you understand the advantage of continuous flow of features, then you automate your tests, and make them so you can run them every day.  If you have faith that tests can be automated and run quickly, then you have a chance of realizing this potential.

To ensure that we have 100% defect-free products, we must set up a syste that automatically informs us if any process generates defective products -p41

Automated testing, continuous builds, with automated runs of tests.  Require each programmer to run the tests before checking in, don’t check in if the test does not run.  This is Agile/Lean software development, and Ohno is telling us about it in the 70’s.

Rule three of kanban prohibits picking up or producing goods without a kanban.  Overproduction is automatically checked, even if someone wants to make more. -p43

If software is developed only to meed the needs of a specific use case, and the use case is attached to the check in, we will eliminate over coding of things not needed for the use case.

If one sticks to the idea that, once set, a plan should not be changed, a business cannot exist for long.   Sticking to a plan once it is set up is like putting the human body in a cast.  It is not healthy.   -p46

This puts a dagger in the heart of the idea that plans should be perfectly formed ahead of time, and this is particularly true in software where nothing is predictable.  Instead, a lean/agile approach to software allows plans to be flexible over time.  Very important, and an idea I frankly did not expect to come from a car manufacturing giant.

The plant should be a place where such judgments can be made by workers autonomously. -p45

Again a common theme in lean/agile software development approaches, but not what I expected to hear from a manufacturing giant.  In this case he describes the business organization as having an “Autonomic Nervous System”.

Similarly we want information only when we need it.  Too much information induces us to procese ahead, and can cause mix-up in sequence.  In business, excess information must be suppressed. -p50

The kanban allows the information to be carried with the products/parts, one of the reasons why kanban works.  It is interesting to see the bias against too much information.  He talks about automatic fine tuning in this quote:

as long as we cannot accurately predict the future, our actions should change to suit changing situations. -p52

Again we see flexibility and adaptibility as a core value, the antithesis of putting together a perfect plan and then executing long term against that plan.  Instead, adapt as you go along.

In any manufacturing situation, we frequently see people working ahead.  Instead of waiting, the worker works on the next job, so the waiting is hidden.  If this situation is repeated, inventory begins to accumulate at the end of the production line or between lines. -p59

Here we touch upon a very pernicious area of waste in software products.  If the schedule is relatively fixed, when a programmer finds that an implementation went well and they get done early, they often turn to implementing an extra pet project they have been intending to implement.  It is routine for development team to report to product management that few extra features have been included “for free”.  The worst cases of these I have seen have actually caused delays in the product because once implemented we found flaws in the approach, but it was too much work to pull out the functionality, so instead extra work went into fixing it.  Extra work is not extra value, it is instead increased waste.

Making large lots of a single part — that is, punching out a large quantity of parts without a die change — is a common sense production rule even today. … The Toyota system takes the reverse course.  Our production slogan is “small lot sized and quick setups.” -p95

In this chapter he compares Ford production system to Toyota system in order to emphasize the differences.  The parallel of the Ford system to waterfall software method is strong, while the Toyota system is like Lean/Agile software development.  The biggest single driver of continued use of waterfall is the idea that creating a bunch of features at once will reduce overall cost.  There is a feeling that there is an “overhead” to the cycle that can not be avoided, and therefor do as much in a single cycle is the most efficient way.  Toyota’s concept of flow and production leveling is just as radical as the idea of continuous builds and fast sprint feature cycles.  Both are are not intuitive, and have to be seen to believe that it is possible.  It should be obvious, however, that large lot sizes will make changing the product more difficult and maybe nearly impossible.  Thus it is the method of manufacturing that causes the inability to respond.

For automation to be effective, we must implement a system in which the machines sense the occurrence of an abnormality and stop themselves.  In other words, we must give the automated machines a human touch — enough intelligence to make them autonomated and achieve “worker saving” rather than “labor saving”. -p113

In software development, of course, machines are replaced with automated script and programming that does steps of the process, and these must include tests at every step.  Ohno’s statement here is strong: the automation without the sensing causes more waste than it is worth.  Thus tests built into the automated process, or possibly into the product itself, becomes most important in automating builds and steps in the software cycle.

Conclusions

In 117 pages he sets out a clear description of how Toyota has accomplished the climb to be the world’s biggest car maker, and is a must read for anyone interested in industrial production strategy.

Then, if you understand the mapping to software development, you find that he sets out an equally clear description for Lean/Agile software development.  If Ohno was alive today, I am convinced he would have been an eloquent proponent for developing software in short iterations, with continuous builds, continuous testing, immediate response to build breakage, and guided by feature burn-down.  In other words, a proponent of Agile Software Development.