I did a web presentation with Sandy Kemsley on the subject of “Crossing the Next Frontier of Business Process Management: Introducing Process Intelligence.” The webcast went well, and I really appreciated Sandy’s clear and accurate descriptions of how process mining works.  It also touched on Fujitsu’s new Interstage Process Analytics product.  Access the webcast and related things at the BPM For Agile Enterprise site.

This image (click on it for larger view) helps to represent the problems that traditional manual process discovery runs up against.  When a consultant interviews people on what the process is, each person has their own view of the process, which may not in fact be accurate.  Then there is what I call the “bicycle riding problem”: The person may in fact have the ability to accurately perform the process, but can not explain how they do it.  Many work behaviors are like this where people operate on tacit knowledge that they can not put into words.   Finally, the person may have an accurate understanding of their part of the process, but they may not be willing to say exactly what it is.  They may feel that the process as it is performed is broken, and so they will report instead what they thinks that the process should be.  All of this gets in the way of discovering the real process.  Process mining cuts through all this by analyzing the real evidence behind the process, and giving you an ego-free picture of the process as it really is.

The webinar was coordinated with a white paper that Sandy wrote on the topic of process intelligence: “Enabling Process Intelligence Through Process Mining & Analytics” which can be downloaded from that link after registering.

Also this week, Vance McCarthy published an article at Integration Developer News on the product: “Fujitsu Interstage BPM Unites Automation, Analytics, SOA Loose-Coupling.”

Questions & Answers

A lot of questions asked in the webcast did not get answered there, so let me take the opportunity to answer them here:

Q: The solution relies heavily on event logs. What about other data, including unstructured (e.g. email)?

There is actually quite a lot of information that is structured appropriately to be mined.  Almost any application will keep a record of what people did, when they did it, and what it was done for.  It is possible to mine semi-structured information, like email messages, but it takes more work.  You have to process it to extract the key pieces of information from the text.  While this can be done, in my experience we have never needed to resort to this.  I always get asked this question, because people always are worried that they might not have the structured data, but in my experience there is a lot more structured information being saved that any of us realize on a daily basis.  Mining that usually gives you a very good picture of the process, and you usually don’t need to resort to mining the semi structured information.

Q: What are the advantages of the process flow diagram vs a Gantt diagram?

Sandy gave a great answer to this question which I did not capture here (you will have to listen to the recording to get that).  I think it is an excellent question because I am always talking about the advantages of a simple checklist over a flow diagram when it comes to processes which are expressed directly by workers themselves.  A Gantt chart is typically seen in project management software as a way to represent a sequence of tasks and their time line.  This is an approach that is good for planning a single instance of a process.  Built into the Gantt chart are some assumptions that have been made about the order of the tasks, and those assumptions are not explicit in the chart itself.  While Gantt charts are good for planning a single instance of a process in detail, it is not a useful approach for representing collections of processes.  When we analyze 4000 process instances, the result can be messy, and there are many different paths that are taken, with some paths taken quite infrequently.  It is very difficult to represent this in a Gantt chart, and so we draw tasks as boxes, and lines for transitions between them.  This allows us to represent any number of different flow paths on a single diagram, and we can include details such as how many times the path was taken, and how long it takes typically.  Thus, the flow chart is really the better way to represent aggregate information about collections of processes.

Q: Is Fujitsu technology using university research Sandy mentioned?

These actually were developed independently.  While a number of our competitors license the University of Eindhoven technology, our patented technology was developed within Fujitsu Labs over the past 15 years.  It is a slightly different approach from the other, although they both achieve process mining by consuming event logs and producing process diagrams.

Q: What is the difference between Process Intelligence and Business Intelligence?

Business Intelligence is a term that has been with us for 10 to 20 years and it refers to using analytical processing techniques on captured business data.  Generally in BI data values exist in time and space independently from other values.  Process Intelligence extends this idea by adding correlations of events with each other over time, and that adds an additional avenue of information.  Let me see if I can explain: a BI tool might analyze the number of bids you sent out, and the number of contracts awarded, the process intelligence tool will match individual bids with individual contract awards, and with other actions, and be able to report specific information about how things flow.

Q: Will “Process Intelligence” be the responsibility of process owners or IT staff, or both?

It has to be the responsibility of both, and we are seeing this increasingly to be the case.  Without an understanding of the meaning behind the numbers, you can not make meaningful reports.  It is the business person who can recognize that a particular value is troublesome, or a particular pattern is worth watching, and this will drive reports to cover those specific situations.

Q: Can you explain the difference between process mining and collaborative process design? They both claim to be doing process discovery.

Both are techniques for uncovering the process that exists.  Collaborative process design is an approach where all the people who are knowledgeable about the process collaborate online to define the process.   This is really quite similar to the traditional manual process discovery where consultants interview workers and document the process.  The difference is that instead of a consultant doing an interview, the worker or manager enters the information themselves using an online collaborative tool.  It still falls prey to the misunderstandings that people naturally have about their view of the process: they might not know how the process works, or they may know the process but be unable to explain it, or they may tell you not how the process works, but how they think the process should work.  What you end up with is a collective agreement about the process, but that is no guarantee that it is the real process.

Process mining is a very different approach.  Event records are taken from the systems, and a process diagram is produced that is an accurate depiction of what really happened.  It does not involve any interruption to the people on the process, and it does not fall prey to the limitations of understanding the process.  It is messy, because reality is messy, but it is much quicker and more accurate than collaborative process design.

Q: How long does it take to deploy it? How long before I see results?

It is very quick.  First of all, there is no need to “deploy” anything … it is not something that is installed into the network and watches traffic.  Instead, you extract data from existing databases, and analyze that.  The initial results can be available within hours of starting.  To get the best results, plan on spending 2 to 6 weeks in order to explore process space fully, and understand the meaning of the outlier process instances.

Q: What do I need to do to start using your product?

Interstage Process Analytics is available today.  Training is available regularly, or if you prefer we have consultants that can come and help with the extraction and analysis immediately.  You identify a process, and then you identify the systems that are used by the people in that process.  After that, it is a matter of extracting the event records from the database

Q: Process discovery sounds too good to be true… How can you convince me it works?

The results are pretty amazing.  We have some case studies available for the asking.  Some people believe that the cases have been cherry picked and are special situations, or at least are not comparable to their own situation.  This is a common misconception.  If you know you have a process problem, it is relatively inexpensive to make a trial run to see what can be discovered, and that is the only way to be assured that you have the event information necessary to make it all work.

Q: Why are you calling it the next generation of process intelligence?

This is the next logical step of process support: from initial orchestration, to elementary analysis of process records, to advanced process mining combined with real time alerting.

Q: Have you any numbers that speak to the costs involved in manual process discovery?

We don’t have any solid thoroughly researched estimates of what it costs on the average, but it is widely known and accepted that a modest manual process discovery can run into multiple person-years of effort.  Some estimate that 40% of a BPM project goes just to the process discovery phase.  Our own experience with discovering processes in house are consistent with it taking multiple person-years to achieve — this actually was the reason for developing the technique in the first place.

Q: You mentioned that workflow in ERP systems like SAP can be instrumented and visualized by Interstage. Don’t ERP suites already do this monitoring internally themselves?

It is very rare to have the capability to correlate separate events and to be able to analyze the results of that.  So, essentially, no, ERP suites do not have this capability built in.

Q: What are the advantages of combining process mining and analytics?

Process mining gives you an overview of the end to end process as it has been up to that point.  Process Analytics gives you access to real time data of things that are occurring in your system.  Combining these allows you to use the mined models to design alerts against transitions you need to be informed about.  Also, as more data arrives, the process mining can be run again, and always kept up to date.  As Sandy said, combining chocolate and peanut butter can produce a combination that is better than the two separately.

Q: Can I reuse the process model?

Interstage Process Analytics will convert the process model to an XPDL file which can be imported and used in a BPM implementation project.

Q: Is there any additional data I need to make the discovery work?

The requirement is that there are three critical data elements: the timestamp, the task being performed, and the case or context that the task is being performed in.  Bringing in additional data gives you additional analysis options.  For example, including the office location will allow you to compare how the process is performed in different offices. Including the product model name can allow you to compare processes across different model lines.  Each additional information column will give you another dimension upon which to separate out process instances for comparison,a nd can help you to isolate problems spots.

Q: Is process mining limited to certain data sources?

Interstage Process Analytics imports data from CSV files, so if you can extract the event records out into a CSV file, it can be used for analysis.

Q: Can you elaborate on how this figures into what Gartner is terming “Intelligent Business Process Management Suites?

Gartner is proposing a new category of products that combine analytics and traditional BPM techniques together into a single suite.  I believe is combines the active BPM process orchestration together with process intelligence and process analytics.