In a panel session this week at the eBizQ virtual conference (see here and here) I was asked “What common mistake do people make that causes unnecessary delay in BPM projects?” The answer: Many projects have a goal to implement too much at once. Some projects attempt to turn a manual process into a completely automated “straight-through” processes where there is no human interaction at all.
It is true that the more you can automate a process, the lower the cost or running the process. But the cost of automation can vary greatly from activity to activity. Usually it is the case that about half the steps can be automated fairly easily, but as you automate more and more, the cost starts to rise steeply. There is usually a step or two in the process which is very difficult to fully automate.
What are those difficult steps like? Usually it is an activity where a person accesses a third party system to manipulate the information held in that system. Unfortunately, there is no well designed API to manipulate the data in the same way.
The SOA folks say: “Wrap that application in a service”. Right they are, but that wrapping can be very very expensive. I have seen upwards of a programmer-month spend on creating such wrappers. It is a simple cost/benefit relation which depends upon the number of times that you need to do this. If a person can access the system through the user interface, and accomplish the activity in 30 seconds, and if you estimate you will need to do this 1000 times a month, then you will be saving ~10 hours of work per month, maybe 120 hours/year. How many hours will it take to create the wrapper? It can be more than this.
There is an emotional side. Some people are shocked when I suggest that we simply assign a task to a user, and have that user manually update the information in the third party system. It just seems wrong to automate the process so that people can manually do a task. This group-think often pushes a project to an all-or-nothing stance in process automation. Striving to automate that last 5% causes most of the delay and cost of a process automation project.
I suggest instead that you aim initially to automate 80% of the process. Look at it this way: before you start, the process is 100% manual. You have people doing manual tasks, coordinated only by their own diligence through the use of email, and without any reliable record of what has been done. If you can reduce the manual activity to 20%, you can cut the cost of the process to somewhere around 1/5 of what it was before. That can be a huge savings and can be relatively easy.
Get the process deployed into production with only 80% automation. This leaves 20% manual, but then it is 100% manual before you start, so why should anyone complain about leaving 20% manual? You will start to get a payback immediately on the 80% automated. You get a relatively quick win.
You don’t stop there. Use an Agile approach to automate another 5% and get that into production. Then another 5%, and so on. The ultimate goal is to automate 100% of the steps that can be automated. (Some steps require human decision making and can not be eliminated completely.) The key is that you don’t need to do this all at once.
Consider that as you automate a process, the automation will change the organization, which will in turn change the process. When certain steps get easier, you will find they are leveraged to replace harder steps. The process as it is today is a result of choices people have made in evaluating cost/return on individual steps, and when the cost of a step changes, it can change a process. This change is not intuitive, so when you deploy the process with 80% automation, you should investigate whether you process is still what you thought it would be. Luckily, if you are using a process support system which includes strong human level analytics you can easily re-evaluate the bottlenecks. You can focus you ongoing efforts on the parts of the process that will yield the most benefit.
There are benefits beyond the automation of tasks. It turns out that even if you left each activity completely manual, and just used the process technology as workflow to route and track the process, you would gain a tremendous benefit: you would be able to know the precise state of each process, and what has happened to it in the past. Usually half the activities are easy to automate, so there is no reason to stop short of that.
As you automate more and more of the process, you will find there are more and more exception conditions, each one of which is increasingly rare. The benefit of automation gets smaller as the number of cases gets smaller. You always reach a point where a case falls outside of the norm, and you simply give it to a person to “fix the problem”. That is the most cost effective way to design the process, but that is precisely where a lot of projects get tripped up. They underestimate the complexity of the process (it seems so simple) but on the course of complete automation, stopping short seems like failure, so they continue to implement and implement, through multiple delays and escalating costs.
The 80% solution is master of all this: start small, get a quick win, start seeing the benefit quickly, focus on the areas that need the most attention, continue to improve. After all, isn’t the point of “Business Process Management” all about continual improvement over time? You don’t need to do it all in the first step.
Thinking Matters 
Pingback: The 80% Solution | Adobe Tutorials
Pingback: Process for the Enterprise » Blog Archive » The Case Against Window Dressing
Automating processes perfectly kills them and the much asked for agility. All you need to do is to keep track of things … there is no need to control a process, not even 80%. The problem is that workflow systems can not do more than control a process and beyond that they jsut make things more difficult.
What you need is a kind of architectured collaboration and implement processes as you go along. That process of improving processes never stops which means that it must be a function of the process environment and not a design or execution environment such as BPMN or BPEL. If you have to go back to the blueprint you are not agile. Flowcharts are usable as an after the fact visualization tool only, not a means of design and control.
Why don’t you get it, people? Designed processes are rigid and NEVER agile. The user has to be able to work any way he sees fit to serve the customer and the process owner has to be able to monitor and change execution as needed without cmplex design work.
20% of processes will create 80% of revenue and and another 20% will create 80% of profit. These are not the 20% that have to be rigidily controlled. The Pareto Rule at work …
I agree with many of the things you say, you make here the mistake of assuming that there is a “one size fits all” approach to process. Processes vary across a spectrum of “repeatability”.
Consider a fast-food restaurant: you order, pay, pick up your food. It happens thousands of times a day at thousands of places across the country. There is not a lot of need for agility here.
Consider opening a bank account: there is a very rigid, very fixed process behind that. The business of the bank depends upon this process happening the same way every time regardless of who is involved, and regardless of which branch location. Like accounting practices, there are very rigid and agreed upon processes that need support.
The other extreme is fighting forest fires. In this case every day, sometimes every few hours, a new plan is formulate to meet the changing situation. In the worst part of the fight there is no need for a process diagram: the effort to draw one up is greater than the payback during the short time that it is needed.
There is a middle ground in “New Product Introduction”. This is a rare enough situation that depends heavily on the specifics of the product and the current market environment. But there is enough control, and there are enough people that need to know how the product will be introduced, that drawing up a formal plan for this product launch makes sense, to make sure that everyone is coordinated. It would not be successful if everyone just worked they way they wanted to. Time is limited, and efforts must be coordinated.
Across this spectrum of repeatability there are different techniques to use: at the highly repeatable end traditional modeling techniques are extremely useful. You can’t deny this. But traditional model becomes less useful as the process is less repeatable. Let’s not lump all process support into the same bucket.
Keith
Absolutely agree. I also noticed that many BPM novices are “integration maniacs”, especially those with IT rather than business background. Over time they either become pragmatic or don’t get what BPMS is good for and abandon “process toys”.
But after one’s got the 80% logic it becomes so seemingly trivial that nobody cares to leave a word of wisdom for followers. So thank you for putting it so clear and let me publish Russian translation of this article on bpms.ru.
We fall into this trap too often so I force myself to document such “trivial” hints on the blog.
Getting to the subject, I don’t agree with Max’s position – he seems to deny the whole idea of executable processes – yet there is a truth in his words. Those who proceed to integration too early are doomed to automate poorly understood or badly inefficient processes. Several cycles of modeling-execution-analysis are must; these cycles should be short; you can’t keep them short with too much integration and/or automation. So it’s not just about cost/benefit but also about agility, vague business requirements and process discovery.
Anatoly, as it happens your interpretatation is false. I do not deny the idea of executable processes. I deny the idea that the executable process can be defined once (whether it is 80% or more) and that this will be sufficient. It is at best 80% correct. Is it acceptable to execute processes that are 20% wrong? At most 20% of all processes are perfectly repeatable and not 80%.
What I disgree with the most is the blatant buzzword misuse by BPM proponents like agility. Once you defined a network of processes ALL agility is out the window. If not explain to me how the person doing the process can adapt it? That would be agility and true process discovery. The vagueness comes with the BPM ‘gurus’ requirement to certainly define the process when that is practically impossible.
Yes, I agree – BPM folks fall into a lot of traps and the biggest is that rigidly defined processes improve the way a business operates. I propose that processes can be executed perfectly and that is the way the process beneficiary needs them ‘at this point in time’. BPM reduces the agility of a business so stop misusing the term!
Max
Sorry for the misinterpretation, I can see from your clarification that our positions are actually very close.
You ask the right questions but it’s hard to answer “how the person doing the process can adapt it” within a comment. I can only assure that we aim exactly this target in our practice.
Again, you are right that there are BPM gurus of various kinds and some of them consider BPM as a one-time project. There is obviously no room for agility within this interpretation. Yet you are making too general statements about BPM. For me personally (and hopefully I’m not alone in this belief) if it isn’t agile then it isn’t true BPM.
From technological perspective, “preserving BPMS” (Keith’s term) is a pre-requisite for agile BPM.
Anatoly, thanks. Could you explain to me how a business user/process owner would have flexibility or adaptability even in simple things such as changing/adding content or data within a typcial BPMS workflow system – especially the BPMN/BPEL kind? It always requires to go back to the blueprint. Additionally all the pre- and post- processes are hardwired into entry points and if you have need to flexibly add additional events then the whole house of cards crumbles.
I am not making too general statements or interpret anything. Mabye a small percentage of BPM project offers some agility … MOST DONT!
Max, you really should get out more. 🙂
Imagine a system in which you draw a picture of the process by specifying for each activity “what is to be done” using the standard terminology recognized by the group, and you specify who it is that is going to do the activity. They system takes care of notifying those people when it is time to do something.
Say you have a four step process deployed, and you have a thousand cases running. But then comes an unusual case. The person assigned to step 3 in the process realizes that in this case the process is incorrect. That person can then add a step in the process TO THAT CASE and no other case. That step is added simply by specifying the activity using terminology that the people involved normally use, and by specify who is to do it. That case has an added step in it, it can be added instantly by the person doing the former process. There is no need to go back to the blueprint. In fact, there is no need for that person to even understand the completely process (but in any process agility situation, automated or not, it is necessary for them to understand their step and a few steps around it).
You are correct in pointing out that a BPEL based system can not do this. The world is not limited to BPEL based systems. I will also agree that such systems are rare. But they do exist.
Your statement was “It always requires to go back to the blueprint” but this is simply not true — there are business process systems designed for this kind of flexibility, that can be changed at any moment.
Imagine a system that can support “Mass Customization”. One customer of ours deployed Interstage BPM in such a way that 40% of all the processes were customized over the life of the case. That is, 40% of all cases had a process unique to that case. There was no technical limitation preventing agility. The “whole house of cards” did not crumble in the face of change. There is no limitation to the kinds of changes that can be made: anything from simply correcting the spelling, to removing all activities and designing a completely different one. Most would complain that it had “too much agility”, but that is for another discussion.
Ask yourself the question: do processes exist? This is a little like asking whether “beauty” exists, or “words”. Processes exist because we can talk about them. We have words for them, and conversations about them. If we have words for them, we can write those words down, and share them with others. The process is defined according to those words. A BP support system can communicate those words to others, and the organization (team if you will) can accomplish the process. If any member of the team wants to change the words, they do. There is no reason to believe that those words can not be changed at any time.
Some people believe that those words that describe the business process should be replaced with computer instructions that send and receive bytes. That is, the meaning of the words should be translated into operations that do the manipulation of the data itself. This is difficult, requires expertise, and causes the problems that you observe about making it difficult to change.
I would be happy to discuss how most of the systems out there are NOT designed for this kind of agility, but I would like to focus on the techniques that exist in products today that support this, in order to educate the market on the need, rather than blankly denying that any such capabilities exist.
Keith, no need for discussion! Simply … which system .. which software … which business … who can I talk to? I have stopped to believe all the great talk …
I absolutely agree that process is about communication and therefore about terminology, vocabulary and content – my words for the last 15 years! Like in physics where we are not looking for how the universe works, but what we can know about it. Therefore defining the entities and make them accesible to the business user is the core activity of process management.
Yes, I know about some (mostly hardcoded) case management products that have some flexibility built in and I know about Tibco’s Process Commander, where they offer to rewire process fragments in real-time … which is not very practical I am told and still limited. Obviously there is a lot of collaboration and ad-hoc human focused BPM that is more flexible and still they all end up with a lot of Java coding for content and data integration. That’s where it crumbles …
As I said I am in these businesses day by day and I see what they do AND NOT! You prefer to be very accurate and say, that there are these systems and yes, that is true. But the broad reality that is sold and implemented is not. And we can agree to disagree on that and let it be.
Max, I’m not sure it’d be politically correct to discuss specific BPMS pros and cons here. Just keep away from BPMN-BPEL translators if agility is your priority. They are good for high-performance STP processing yet agility isn’t their strong feature.
Keith said that others are rare – I’d not agree with that. It’s true that BPEL dominates in hype that big vendors produce yet it doesn’t necessary mean that they dominate in BPM projects. There is at least half a dozen of very decent products to choose from beyond BPEL league so if you know what you are looking for then you’ll get it. I’ve got positive results in reaching BPM+agility with several products, including Keith’s one. BTW, our Tibco experience was negative.
> It always requires to go back to the blueprint.
Keep away from systems that treat blueprints and executable diagrams as separate artifacts. The more agile alternative is these being two layers of a physically single model.
> Additionally all the pre- and post- processes are hardwired into entry points and if you have need to flexibly add additional events then the whole house of cards crumbles.
Of course integration wiring is a roadblock for agility and this is exactly the point of the article we comment on. But the vast majority of changes introduced into a typical cross-functional process that follows the way of constant improvements are not about integration. They are things like “let’s add another branch here or another control flow there”. It’s funny how differently IT and business people look at such changes: endless annoying routine for the former and the essence of day-to-day business operations for the latter.
Anatoly, you worry about political ocrrectness and I think about how solutions should be and what is wrong with BPM today.
IT should not even be involved in constant process improvements and thus it should not be annoying IT routine. I am still waiting to hear the names of those agile non-BPMN/BPEL systems that do not need to go back to the flowchart blueprint for any change. I still propose that there is a lot of hot air in BPM and you still have not offered any solid evidence to the opposite.
Max
OK, my personal favourites are Oracle/BEA/Fuego and Fujitsu. Lombardi seems to be a strong product too but I don’t have hands-on expereince with it. Please get BPTrends report (it’s free) for the comparative analysis.
Max,
Please see my post above from July 12.
Products: Fujitsu Intestage BPM, Appian, Handysoft, Savvion.
If you would like, not only do you not need to start all over, it is easy for me to demo an end user modifying a process while it is running. Adding nodes (or removing them) to a process instance, without having to stop the execution. Also, no limit on the number of versions of a process running simultaneously, and you can migrate any number of running processes to a new version if you like. This is all done through a web interface, no programming required.
Would that do it?
Keith, Anatoly – from my perspective the only one I sort of accept is Fujitsu because of the reusable data models. The others are human centric workflows that are quite limited. Are you saying that all the products can do all the things that you mention? That would truly surprise me because they are not being used this way, so even if they can do it there is something amiss.
Adding/removing decision nodes is no more than simple rules. But what happens when the backend data interfaces change, new data have to be prompted and interact with other processes, or documents with business data change or forms have to be changed? An enduser does that through the browser … never!
In most products forms suddenly carry a lot of extra programming that encodes process logic. Swtiching between user interactive and straight through processing is difficult or impossible, extracting data from content and to merge data into content is all programming, just as creating XSDs for forms or document mapping is a nightmare, particularly in maintenance.
And then again all of the above is not doable in BPMN/BPEL. Oracle/BEA/Fuego is a nightmare too and Lombardi is a CUTE product that trades function for simplicity (but apparently successfully).
Obviously the products are developing and I will take a closer look but sofar I just saw a lot of hacking for the backend data, forms, and content integration. And as it happens the content state IS THE PROCESS.
Max
The degree of your satisfaction depends on your expectations.
Do you wish EAI tasks to be solved by few clicks made by an end users withing a browser? That won’t happen. Building SOA framerwork with solid governance is a large project highly dependant on professional skills.
As for human activities, there is a dilemma: either users will accept rather simplistic web forms that can be easily modified say to introduce new piece of data or we’ll give them a sophisticated web application hand-made by programmer – superior in functionality and usability but more costly in development and maintenance. (The best choise is between: start with simple web forms and go to programming later when the rate of process changes goes down.)
You seemingly complain that BPM can’t resolve all processing issues as easily as a magic wand. Sure BPM isn’t a silver bullet.
For me BPMS is yet another handy tool. BPMS externalizes process logic from business application much the same way as DBMS externalized data logic. It makes process logic visible to business and makes business directly responcible for the process logic.
Yes sometimes changing this logic still needs programmer’s efforts. But the point is that programmer doesn’t interpret vague and incomplete process requirements any more – business owns the process diagram, business makes decisions and carries full responsibility.
Again, there are parallels with DBMS evolution. It was expected in early days that business people will fire SQL statements themselves. Well it didn’t happen – so should we throw away DBMS?
For some people it’s a disappointment to learn that BPMS is an addition to their toolset – we all wish life to become simpler. Does BPM replace any of the existing tool? No. Can it be replaced by existing tools? No. Is it justified to add another tool into your toolbox? It depends on your tasks. Probably you’ll have to do it sooner or later like DBMS finally became accepted everywhere. For some cases it’s reasonable to wait until BPMS become more standard/more mature/less expensive. For others it’ll pay for itself manyfold right now.
Anatoly, your comparison to SQL is somewhat valid but I am not so much complaining about what BPM is or is not but about as what it is being sold.
And yes, the same goes for SOA and yes, as you so surprisingly admit, both require immense IT efforts and huge amounts of governance to even get off the ground and hence, to use AGILITY when talking about BPM and SOA is worse than just stretching the truth – it is truly intentional misrepresentation.
Yes, there are always ways to calculate the ROI that will make an accountant happy. I believe that BPM/SOA hurts businesses when you realistically take the overall effort and the damage the lack of flexibility does to customer service into account. But I know you obviously you won’t stop because you make your money with it. We all believe in something …
I am just disappointed that so much money is spent selling BPM/SOA that it is impossible to get heard when you want to do something different. Much like the XML/Java sillyness. But let’s leave it at that … I have more important things to do than trying to convert the Pope …
Max
I would be happy to hear about more agile alternative to BPM. Exactly what “something different” you meant?
Anyway BPMS-enabled business apps are much more agile than traditional ones and that’s enough for me.
If price is #1 issue then jBPM 4 and other opensource BPMS may be worth to try.
Pingback: Keith Swenson’s Blog – On Collaborative Planning « Adam Deane
Hmm is anyone else encountering problems with the images on this blog loading?
I’m trying to find out if its a problem on my end or if it’s the blog.
Any feed-back would be greatly appreciated.