John R. D’Entremont interviewed me to put together an article called “Mastering the Unpredictable” on the Projects At Work website.  You have to register to read the entire article, but it is free, and John has done a nice job of putting all the information together into a compelling article about the genesis of the book by the same title.  Below is some of the questions and answers that we exchanged.

John: To help our readers gain some perspective on your background, could you share some thoughts on what you do when you are not writing?

Keith: I am really a software architect: I help design system and that requires that standard processes and procedures be put into place within the entire team of developers. To do that, I do a lot of thinking about how to design rules that people can work with. In the early 1990’s I got involved in a number of standards efforts: OMG, Case Communique, and Workflow Management Coalition. This was because I saw the need for design rules across the industry, and these organizations were the best ways to accomplish such things.

John: What inspired you to write your first book?

Keith: It was a long time coming. In the early 90’s I got my first papers published in conferences because Fujitsu had no real mechanism to publicize ideas, and conference proceedings seemed the best way to get in contact with others who care about the topic. From there I moved to tutorials and writing the materials for that. Then a German professor contacted me to let me know he was using my tutorial notes in his college class on Business Process Reengineering and Workflow. I self published four books from tutorial notes and from other documents that I had developed over the years for guidelines on software development techniques. In the mid 2000’s I participated in a chain of technology tutorials with the WfMC. Robert Shapiro and I recorded one of our sessions and turned that into a book on BPM standards. I spend so much time trying to get the concepts clear to me, that I really was compelled to write the stuff down and make it available in a reproducible form.

What was the motivation behind your research and writing on the subject of knowledge work and Adaptive Case Management (ACM)?

Keith: My goal has always been to find a way to support office workers. I thikn I was always considering the creative work the knowledge workers do, although I was not using that term then. If you read my papers in the 1990’s you will find discussion of how each team, and potentially each worker, needs to find the right fit to their own needs. The idea behind “Collaborative Planning” was that getting together to decide what to do was an important part of actually doing things. This is a BIG difference philosophically from those who believe that all work can be automated, and that you can eliminate humans from work entirely. The Workflow Management Coalition always included the idea that some work can be automated, but some work inherently must be done by humans, that is enshrined in the Workflow Reference Architecture produced in 1995 and still relevant today. I, and many others in the BPM field, were struggling to support humans along side the automatable data processing. It was in 2009 that it became clear that the definition of BPM had “collapsed” in the public eye to mean only automatable work, and specifically orchestrating data flow between servers. People felt that BPM was a part of SOA. Many of us were frustrated by that, because there is so much that can not be automated and required human intelligence. The key concept was that people were doing work that could not be predicted in advance, and so predictability became distinguishing aspect of what was and was not knowledge work. We had a meeting in Maidenhead England where the WfMC invited many of the top thinkers on this topic. Eleven of the 12 authors of Mastering the Unpredictable were at that meeting. We felt that the subject was so important, and so urgent, that we decided to share the effort and write it all down. Five months later Mastering the Unpredictable was released to the public.

Are there any sections (or elements) in “Mastering the Unpredictable” that you are particularly passionate about?

That is a little like asking which of your children you like the best! The book progressively discloses ideas, and so the first chapter covers the concepts at a high level, and chapters 4 and 5 get into a lot of important details. Then the rest of the book reflects on use cases and how the technology fits into different fields. We know so much more know about the field, but still I think the book stands on its own for giving a cogent picture of what is possible.

Are there any particular challenges that come into play with ACM? Can you recommend any best practices?

The biggest challenge is getting IT departments to realize that they can’t automate the creative parts of the organization. They have a “Jetson’s Mentality” thinking that all work will ultimately be reduced to a single button press. At the same time, the people who lead creative organizations, and know that it can not be automated, tend to shun al technology because of the way that the IT department tried to reduce all work to automation. Creativity comes so naturally to people that we don’t realize what we are doing and how we do it. So we find that we have to sell not a technology, but a management philosophy. Many manager believe that there is actually exactly one way to do things, even though their teams are constantly changing form and method. Because they are not faced with the reality that work processes are continually in flux, they tend to believe that there must be one single best way to do something. Getting past the “Newtonian Illusion” that an organization is base on fundamentally simple rule is the biggest challenges to deployment of ACM.

Do you have any advice on how to incorporate social media into managing the unpredictable?

I am always careful to distinguish “social media” from “social technology” – they are similar technology, but distinct uses and benefits. Social media is typically used to broadcast to — and from — masses of people. Social technology however is a set of capabilities that take social network relationships into account. In many ways ACM *is* social technology because it makes use of explicit representation of relationships between people to support the work on a particular case.

Do you have any case studies where you’ve combined social media with ACM?

I would say that ACM is about supporting case managers to accomplish projects and get things done. One of the things they may want to do is to engage the public using social media and their use of social media will be not significantly different from those who don’t use ACM. One of the use cases covered in the book is “New Product Development” and clearly when you release a product, you are going to want to leverage social media in the normal way. So they go together but I don’t see any specific dependence between the two subjects.

In hindsight, are there any features of your published works that you would change or build upon?

There have been way to many discussion of the comparison of ACM to BPM. This is because we accurately predicted that those who see the world as something to be automated, would perceive ACM as being the same as BPM, and we wanted to preempt the discussion and try to make it the distinction clear from the start. However, most of this discussion is in vein because those who try to automate all work tend to think only of automatable work cases are important, and dismiss the unpredictable work as either nonexistent or unimportant. If I were to do it over, I would probably simply ignore the BPM comparison, simply focus on those organization that depend upon innovation and creativity, and show how ACM can support that work, without any comparison with BPM. It seems that the knowledge work support, and routine work support, are distinct problem that can be handled by different support organizations.

What are your current/future projects?

Organization are realizing the value of supporting knowledge workers, particulary that knowledge work is the most important element of competition in the coming years. As routine work is automated, what is left is work that requires a human intelligence, and better you can leverage human intelligence the more competitive you will be. I am helping companies to select the right technology and configure it for use in supporting knowledge workers. I am trying to work with a group of people to define more clearly what is and is not essential as part of an ACM package, and if things work well I would like to develop interoperability standards to allow various ACM system to cooperate with each other.

Can you share some of the feedback that you’ve recieved for “Mastering the Unpredictable”? Have people in the field embraced the concepts outlined in the book?

There has been a big and positive response among those who have to deal with creative knowledge work. Forrester have published a rating of case management products (they call it Dynamic Case Management). IBM calls it “Advanced Case Management” and people from TJ Watson Research Center have published paper recently on similar topics. A proposal have been submitted for holding the first International workshop for Adaptive Case Management in Sept 2012. Even with this response, we are still just seeing the beginning of movement. Most IT Departments are still focused on automating routine work, and for good reason: there is still a lot of low-hanging fruit or routine processes that really need automating, and such automation is saving organizations tremendous amounts of needless human activity. But over the next 10 years those routine processes will be mostly automated. What will IT departments do when all the routine processes are automated? They will implement ACM system to facilitate knowledge workers, and such support will will make the difference between a winner and loser in the marketplace. It will become a key strategic factor for a company.

Any recommendations on how to get IT departments on board with ACM deployment?

The most important thing is to understand the principle of “unpredictability”. I use the example of a search and rescue team to demonstrate the kind of decisions-on-the-fly that are involved. A search and rescue team prepares in advance, but not by making specific pre-defined actions, but instead to practice “patterns” of working together that can be reused. Think of this like different “plays” in American football: the team practices how to interact using different plays, but the game is never scripted, and even a give play has variability that is decided at the time of the play, and by the players as the play works out. The launch of a new movie makes use of patterns of working together (booking first run theaters, billboard advertisements, promotional products at fast food restaurants, etc) but no two movies are ever launched in exactly the same way, because every launch needs to take into account many factors of current events, culture, fads, trends, as well as competition that is being launched at nearly the same time. IT departments need to acknowledge that in certain types of work, there is no single process, but instead the process followed is different every single time.

Another example I often use is Dr. House the television show because it demonstrates in nearly every episode that the information necessary to predict the correct treatment is not available up front. The patient is dying, and something has to be done, but nobody knows what. After treatment starts, strange responses to the treatment give additional clues about what is wrong, causing the doctor to decide to change the treatment as it progresses. This idea of insufficient information up front, and evolving the plan as the treatment is being given is an essential aspect of Adaptive Case Management. Many of the patients have combinations of problems. Statistical analysis shows that even combining only two illnesses can give you hundreds of millions of combinations, implying that every such patient that comes to a hospital is potentially unique. The idea that there are a small number of standard treatments that can be prescribed up front must be abandoned.

Once the IT department understands that things are not predictable up front, once they have let go of the Newtonian idea that there is a simple rule and the bottom of all behavior, then it is fairly easy to get them to see that ACM provides a way to put an intelligent human at the center of the work. Instead of a factory that is automated to eliminate all human involvement, the IT system becomes a kind of “bionic limb” that allows people to access information faster, sift and sort it more effectively, communicate to others more effectively, and to coordinate the tasks of other people more powerfully. The IT systems become extensions of the decision makers, not the replacement of them.

For many, when they realize this, they also realize that they have always known that executives would never be replaced by IT systems. For many, this is a realization that IT systems can support more kinds of work than they ever thought possible before. Instead of being an approach that competes with existing ways of designing and implementing applications, this become a way to extend IT systems to jobs that have never been able to be effectively supported before. It is a new opportunity for IT departments.

What is your vision of the long range future?

If you indulge me in a bit of wild speculation, we might glimpse at the far future. In 10 to 20 years we will see the transformation of business such that all workers are executives. What I mean by this is that work will consist of making decisions: someone will have to decide how much money and resources to invest in a particular initiative, but the actions resulting from that decision will be largely automated. I don’t mean to imply that we all will live like some futuristic Henry VIII. There will be hundreds of millions of other executives to deal with, so the chief decisions will be about strategies to get others on board with your initiatives — or more frequently on whether to get on board with other initiatives. Once the decision is made to do something with all the right people involved, all the rest will be essentially automated and will proceed without any “work”. Decide that you want the latest electronic gadget and it will be manufactured for you in Siberia and automatically shipped to your doorstep and possible even set up automatically for you. Decide that angioplasty is right treatment for this patient, and all the rest of the preparation, handling, exchange of money, and supply of materials is done automatically, and possibly even the surgery itself (although I expect innovative surgery will be one of the last areas of automation because surgeons need to be very adaptive as they work). Decide that a particular product line is unprofitable, but that there appears to be an uptick in demand for another product line, and this can be automatically communicated to all the right people, triggering a cascade of decision making through the organization, but much of what we consider today as “work” will be fully automated. It is not a panacea: decision making can be more stressful and more exhausting than what we call work today. Furthermore, the effect of poor decisions will be amplified the same way that good decision are, and so the pressure to make the right decision increases. Progressive companies are recognizing this trend today, and preparing by focusing on getting good decision makers, and on IT systems that support decision making such as ACM.

Again, thank you so much for your time on this.  It was a pleasure chatting with you, and I think our readers will really appreciate this opportunity to further connect with the concepts that you discuss in your writing.

My pleasure as well.