PCM In A Nutshell

Production Case Management (PCM) is programmed by specially trained technical people (programmers) to produce a case management application.  That application is deployed for use by knowledge workers to get their work done.  The application offers collections of operations that the knowledge worker can decide to use or not use depending on the specifics of the case.

A PCM application is used when there is a certain amount of unpredictability in the work, but still a large enough volume to make identifying and codifying regular patterns.  A workers who use a PCM application is less in control of their work patterns which are defined centrally.  Thus a worker using PCM will be involved in the outcome of a particular case, but not completely responsible for the ways that the case can evolve.

Four styles of BPM technology arranged by ability to handle upredictability and responsibility

The PCM application does not have a strong process orientation that a Human Process Management (HPM) system or Process Driven Server Integration (PDSI) system would have.  (See Seven Categories to Replace BPM)  A PCM application might have a set of high level states which the case can transition through, and each state can offer differing sets of options to the knowledge worker.  So a case can be seen as being progressed through a set of states, but those states are more used as a description of the status, and less as a definition of what operations must be done.

Because the PCM application is developed by programmers, it can makes use of more traditional mechanisms for data integration:  structured information can be read from some sources, transformed, and written to other destinations.  The sources and destinations can be web services or applications with an API.  Like a typical development model, once the application is coded, the design rationale behind a particular transformation is not included in the final produced code, because it is not needed.

Usage

PCM is used when the number of knowledge workers doing the same job is large, and the domain relatively well known, but the process is not entirely predictable.

You might use PCM for telephone system or cable TV repairmen.  These people need to visit the site, determine what the problem is, and then prescribe a resolution from a menu of well known operations.  It is hard to represent what these people do as a traditional HPM process because it is not predictable enough for a defined process.  The process unfolds at run time because the first resolution might not work, and that tells the repairman more about the situation, and possible leading to further action.  Yet at the same time the repairman is not in a position to invent entirely new procedures.  The phone/TV system is big and complex, and therefor the repairman’s options are necessarily restricted to those operations that are well known not to cause a problem with the operation of the system.

There are a lot of service businesses which can make use of PCM.  For example auto service: the car is brought in, there are a set of things to examine, there are decisions about what to repair or replace, and the are states of the process which, with any luck, end with the car being ready to be driven home.

The situation where PCM is most useful when the number of nearly identical offices is large, and the knowledge worker is a professional but not necessarily an owner of the process, and the process itself is not predictable enough to specify in advance.

Compare and Contrast

The biggest distinction between PCM and ACM is that PCM is not adaptive, and this can be see by three factors:

• The programmers uses formalism such as modeling or programming to put the application together
• It uses a standard application development lifecycle: the application is constructed, then tested, then deployed to non-programmers
• After deployment there is no ability to alter the structure of the application

There are some strong similarities:

• At run time the most important concept is that of a “case” which is primarily a folder to collect all the information around the case, accepting essentially any format of document
• Knowledge workers use their own expertise to control of the advancement of the case from state to state even though the states themselves are relatively fixed
• The resulting case file represents a system of record for the work that was done.

ACM is used for what I would call true knowledge workers: inventors, creative people, executives, managers, innovators, entrepreneurs, doctors, lawyers, etc.  These are people who really do need to decide their course of action every day, and the course of action might be to things that have never been done before.  A board of directors does not have a menu of options to pick from when it comes to actions to take.  Someone responsible for the merger of two companies will not have a system with all possible actions pre-programmed. A doctor responsible for the survival of a patient may prescribe radical and untested treatment if it seems like the only option.

PCM is for situations where the number of workers is high, and yet the course of action on a given case is not highly predictable.   The number of knowledge workers needs to be high so that the cost of developing a dedicated application can be justified.  Also, those knowledge workers are less responsible for the work.  For example, while a Doctor might be in a position to prescribe a radical treatment, there are many other who work in a health care facility who should not have that flexibility.  The routine care of a patient may still too unpredictable for a fixed process, may still require the judgement of a nurse or clinician, and still the options available may be restricted to a set of known actions.

As you see, PCM sits squarely at a mid-point between HPM and ACM.  It is developed and has a life cycle like HPM, it is deployed to potentially a large number of people in identical positions, and the cost of development is recouped from a small increment in efficiency across a large number of people.  At the same time the work being done is unpredictable like ACM, and depends upon the judgement of the workers who leverage their expertise in that particular domain.

Reflection

Two approaches to knowledge work implies that there are two kinds of knowledge worker distinguished by “responsibility.”

• knowledge worker for hire – someone is trained in a specific field, and learns to be an expert, but has little or no ownership of the overall process.  A car mechanic must make accurate suggestions on how to repair the car, but does not take responsibility for the business, and must work within the constraints set by others.
• knowledge worker with responsibility - someone who can plan and be responsible for the course of events.  This is the knowledge worker defined by Peter Drucker as “knowing more about their job than anyone else in the organization.”  These are the workers who handle the wicked problems and have to think outside of the box.

I am sure there are many more distinctions between types of workers, but for now this seems to be a determinant for whether you use PCM or ACM.

Yet BPM Lives On!

BPM is survived by a host of offspring which are all healthy and thriving.  These family members carry on the hopes and dreams that BPM held true to, and do it with many hands and many more forms than a single moniker could accomplish.  BPM has brought us to a level of maturity, where we no longer need the simplified concept that a single size will fit all.  Our IT culture has matured to appreciate the different forms of work that people do, and the differing needs they have for support.  Now we can appreciate the benefit of approaches tuned to a specific need.

Out of respect for the departed, we should no longer use the term “BPM” except for historical references in hallowed tones.

Terms to Use Instead

The biggest problem with the term BPM is that so many people saw it as meaning so many different things.  This causes unnecessary arguments between experts, like the blind men arguing over the shape of an elephant.  We can clarify this debate by naming the subcategories of BPM.

1. Management of Business Processes (MoBP) – This is the management practice, and outgrowth of the process oriented thinking and the Business Process Reengineering movement of the 1990s.  This is really just a management discipline which views tasks of various workers as being linked together into processes, and is informed by measures of process efficiency, and with a desire to continually improve the process.  Other names include LEAN, Six Sigma, etc.  There are many varied tools that people use, but nothing structured into specific applications.  The audience is primarily management.

2. Business Process Analysis (BPA) – this is a type of BPM where process experts map processes using graphical modeling tools.  This is the space where SAG’s ARIS business modeling tool fits (and others).  There is a strong modeling angle, with many different styles of modeling, but very little or no actual automation or enactment support for a process.  The audience again is primarily management and process analysts.

3. Process Driven Server Integration (PDSI) – This type of BPM is an information technology initiative to link together servers in a Service Oriented Architecture (SOA) environment.  A process designer will draw the process using a graphical language (probably BPMN) with the aim to produce an executable program that will send and retrieve information.  There is no “B” in this acronym.  This is the BPEL paradigm, also known as Straight Thru Processing (STP).  This environment is created for programmers who want to use diagrams so that they can explain what the process does to business people.

4. Social Content Management Systems (SCM) – Massively multi-user business systems for sharing and collaborating around content.  This is where all the Social Media activity is going.  It is far less goal oriented than a case management system.  All sorts of other names for closely related ideas: Social Business System, Enterprise Social Systems, Enterprise 2.0 systems, etc.  This is an environment primarily for business people and IT departments when they are not programming.

5. Human Process Management Systems (HPM) – This is the type of BPM where forms are routed through a set of people.  The route is typically determined by a graphical process with the boxes representing a human activity and assigned to a person (or group).  Details such as notification of responsibility, reminders, and reassignment of responsibility are NOT included in the diagram because those are just naturally assumed to be part of the HPMS infrastructure.  Also known as Human Workflow Systems.  Human Process Management applications are developed by programmers for business people to use.

6. Production Case Management Systems (PCM) – These systems are configured (programmed) by programmers for large numbers of professionals to use.  There is less of a process than HPMS.  Instead it has a set of well defined states, and a set of functions that can be done in each state.  Movement between states is pretty much the choice of the case managers, but adding states can only be done by the programmer.  Production Case Management applications are developed by programmers for business people to use.

7. Adaptive Case Management Systems (ACM) – These are malleable systems used by knowledge workers within a learning organization.  Here the case manager must both figure out how to do something, as well as do it.  Useful for managers, executives, board members, new product introduction, disaster response, emergencies, mergers, acquisitions, creative teams, innovation, research.  There are no fixed states, but case managers will copy on old case or a template to avoid having to re-design the states every time.  New states can be added at any time by the case manager without requiring any specialized tools.  Goals can be created and completed at any time, and this forms the primary organizing theme of the interaction.  Strong support for content management.

(8) Page Flow – This is technology that supports a series of screens to present to a user, or multiple users, to collect information.  Examples include the shopping cart handling for an e-commerce site.  There is a state machine, and sometimes a flow chart is used.  I would not include this as a style of BPM, but some people do, so I include it here for completeness.

(9) Workflow – While originally workflow was an expansive topic like BPM, it has most recently come to mean a style of handing data that must be processed by multiple applications.  Workflow is a technique for passing data from one application to another, or really invoking a sequence of applications to transform the data through a series of forms.  Implementation can be a simple batch script file.  Again, I would not include this as a style of BPM mainly because usually it is used by a single user, and BPM by definition must involve the coordination of a team of people.

Did I forget any?  Please comment.

Summary

Yes, all of these have been considered “BPM” at various time by various people.  I have witnessed sometimes energetic debates between people about the qualities of BPM, only to find out that they were intending different types of BPM all the time.  I have no idea if these names of the various subtypes of BPM will stick, but I am quite sure that continuing to confuse them is not doing anyone any good.  It is only natural that terms have a life cycle; they are invented, they are refined, and then finally they expire in favor of more precisely defined terms.  While the term “BPM” will certainly enjoy another decade of life in casual use, I think it is time for the leaders in the field to start being a little more specific about what they really mean.

Many thanks to the members of the ACM Linked-In group, and WfMC members, that helped in discussing and refining these categories.

Adaptive Case Management is an approach to work that supports knowledge workers to get their work done. A software system must support the following capabilities in order to be called an Adaptive Case Management System (ACMS).   The goal here is to lay out a set of criteria that can be used to clearly identify an ACMS.  

I broke the definition into three levels in roughly the same way that Max Pucher did in his post “Adaptive Case Management: Basic, Full, or Strategic?“  All ACM Systems MUST have level 1.  Many will have features at level 2, and only some advanced ones will have features at level 3.

I can not stress enough that the capabilities described below must be made available in a way that the actual case manager can use them.  The case manager is a professional, like a doctor, or a lawyer, and not a programmer.   When vendor makes a claim that a system has a capability, you need to dig below the surface, and make sure that it is available to the end user without programming.  The capability must be both safe and easy enough to use without specific training.   ACM supports knowledge workers, and having to go to a programmer will be too big a barrier.  Having to request help from an administrator, for example to define the members of the case team, is a similar barrier.  These capabilities need to be immediately available to the actual case manager.

Level 1 – Required of All ACM Systems

Team: It must organize the work of the knowledge worker (called a case manager) and also others involved in the case (called the case team). The case team is not predefined, and one of the jobs of the case manager is to identify people who are appropriate to help on the case.

Folder: The central concept is a case folder that is used to collect together all of the information and artifacts that might be needed by the case manager. There is no predefined collection of information; part of the job of the case team is to locate and collect the necessary information for the case, and put it in the folder.

Goals: To drive the work to completion, the case team communicates about goals (also called tasks), intended and actual timelines for performing activities toward those goals. None of the goals are fixed: any goal can be modified, added, or removed by the case manager without special skills.

History: The system keeps timestamped records of every change that happens in and around the case.

Security: The case manager is in complete control of who is authorized to access the case folder. Unauthorized users can not access the case folder.

Communication: The system must facilitate communication between case members through multiple channels: telephone, email, instant message, desktop conference, and other technologies.

Adaptive: The case manager can make new cases that build on structure of previous cases through some form of copy or reuse of the earlier cases without needing any special skills. Over time the case manager will adapt the system to their own style of working without needing the help of any specialist.

Reporting: A large variety of user configurable reports for communicating status, things that have happened within a given case, and aggregate information across cases.

Level 2 – Common Extensions in ACM Systems

Business Entities: Structured data with metadata behind it. Form capability to display and allow entry of business data.

Data Interchange: ability to access systems and internalize this within the business entity format for reuse, and the ability to transform the internalized data to a form to send to an external system.

Business Rules: Generalized rules mechanism that tests the business data for specific conditions that can either trigger activities, or effect the state of resources.

Resource State Model: Files and other resources can have a state model associated with them. Transitions between states can effect and be effected by other resource state changes.

Granular Access Control: Fine ability to control who can access what parts of the case information through the use of roles into which people can be assigned.

Sensors and Triggers: the ability to monitor external data / state, and the ability to trigger reactions that change the state of internal resources or goals.

Conformance Guiding: the ability to match the activities of a current state with a reference process, and provide a visual difference highlighting extra or missing activities, order disparity, and make recommendations on how to conform.

Level 3 – Advanced Extensions for ACM Systems

Process Mining: The ability to analyze the history records and to generate process maps showing the patterns of behavior across many cases.

Social Mining: The ability to scan networks of connections between people in order to determine skills and/or preferences of people, in order to guide assignment of activities.

Federated Case Folders: the ability to link folders in one case system to folders in another system, and synchronize content in both directions

Ontology / Taxonomy: represented in standardized form and able to exchange with knowledge management systems.

Resource Sharing: a given task or resource could be shared between multiple cases.

What An ACM System does NOT have

The hardest thing about recognizing a good ACM system is that it is not about what it has, but what it does not have.  To use an automobile analogy: how do you define a roadster?  A simple, two seater car, with plenty of power for a visceral driving experience more than anything else.  A roadster is more about what it does not have, than what it has.  It does not seat more than 2 people.  It does not have a huge trunk.  It does not have a large towing capacity.  It does not have a lot of creature-comfort features.  It does not have 14 cup-holders.  The dashboard is minimalist oriented.  If it had a dashboard like the cockpit of an airplane, it could never be a roadster.  With manual transmission, the roadster is enjoyed for the direct feel of the road, the enjoyment of driving.  So you can not define a roadster by listing all the things it has with the view that more things is better.

Adaptive Case Management is designed to be used by the knowledge worker directly, it will not have many of the more sophisticated features that require a professional to operate.  As you consider usability, think of a doctor using it.   Here are some thing that it must not have, or at the least it must be able to be completely functional without:

Graphical Modeling:  Experience has shown that professionals (like doctors) don’t have the time or desire to learn such graphical modeling. The ACMS must function without any graphical modeling.

Web Service Integration: This is a capability that is used by programmers, but the users of an ACMS are not programmers.  Thus, the ACMS has to work without any web service integration.

Of course, a system might have these capabilities, so that a programmer might occasionally help customize for a particular vertical.  The key here is that these capabilities are not central to the capabilities of the system.  Any system might have APIs for extending it, and that is OK as long as those APIs are not central to the features that are needed in the first place.  If, however, you see a vendor claiming that these are important features of the system, you should be very suspicious that it is not really an ACMS.

Acknowledgements

I would like to thank all the people who have helped to make this list somewhat complete on the original post and in the ACM Linked-In group.

75% of all content retrieved by IBM is tagged content retrieval (which indicates both how important tagging is, how effective it can be, and how mature IBM is in the use of tagging).

Content is right at the core of all information work.  It is the medium of exchange while mobile is the new face of engagement.  Your mobile app is in your customer’s pocket.  It won’t do to simply have a screen scrape of your web site ported to the mobile environment.  What are you going to do?

He presented a slide showing how dramatically the world has changed since 2007, only 5 years ago.  At that time there was no iPhone, no iPad, no app store.  The smart phone era has grown from essentially zero to 1 billion devices in 5 years!  700 million people participate in social networks.

Email is becoming your father’s Oldsmobile.

There was a perfect storm brewing.  Mobiles apps are at the center of it all.  Showed a blood pressure app that plugs into iPhone. Another of a bathroom scale that plugs into the iPhone.  It is a reformation of engagement.  We all need to be thinking differently how we engage customers and employees.  Stop thinking about it as a technology problem.

Architecture start with the systems of record in the middle,  building out from that.  They don’t host processes, but instead serve people.  That is where the benefit is, and where the loyalty lies.  Empowers with context rich apps.  Instead of accessing a web form and entering a bunch of information in order to get stuff, users will simply open phone and it will already know what it needs to know to get you want you want.  That is what is different about this new mobile centric approach.

He spoke about Trip-It and how useful it is for managing everything while traveling.

In the midst of all this amazing capability and advancement, beware the Jabberwok of success.  Cloud services (Facebook, Twitter, Pandora, Salesforce, and Box) are showing dramatic increases in percentage of traffic from smartphones.   When USAA launched their mobile app, they predicted 22 million mobile contacts a year, but it was so popular with this customer base, which includes a lot of service people around the world, that they got nearly six times that amount, 120 million, causing quite unexpected load on the servers.

We should not design the systems inside-out as they have been in the past.  Instead of designing processes around the way we think about the value and flow of information, we must instead think about how customers thinks of the value and flow of information.  Systems need to be designed primarily to support the mobile customer. Development processes need to change, be more agile.  Mobile will get people to think differently about those content systems, so they will be available on mobile devices.

Mobile is becoming the tail that wags the IT dog.

You choose: enabling or empowering.  Here are some steps or design criteria you should follow:

1. Form a center of excellence that bridges between business and It, Use mobile as the driver.
2. Design for mobile first.  Many have tried to take what they already had on the web site, and shrink down to the mobile format, but that did not work.  It did not leverage what is available in the mobile platform, nor meet the expectations of the users.  Instead, start with design for mobile.  It is then much easier to make available on bigger platforms.  It takes a different mindset.
3. Start with mapping the tasks of mobile users, and then match the resources to that.  He showed a graphic of processes in the travel industry.   On one side what the big thick heavy traditional processes.  The other side had a traveler timeline which had a multitude of touch points into the middle of the traditional processes.  Mobile requires you to think of the granularity of the task.  People check their status many many times, all the time. This hits a lot of back end systems.  Back end systems need to be re-conceived to make this information available in a more real time way.
   Can you do it on prem?  Not really.  Cloud is a huge benefit in offering all time on instant access.
4. Deploy you customer interface over the app internet to reach a billion mobile devices.
5. Task oriented design meets mobile reality.  He feels that hybrid of native capabilities along with HTML5 will win in the near term.
6. Assemble an Engagement Platform to support task oriented mobile apps.  Consider using technologies like Hadoop, OpenID Connect.

Key points to remember:

• Design for mobile first
• Collaboration should be built into every application, Social is part of the information workplace
• Get to the cloud
• BYOD (Bring Your Own Device) means lots of hassle, but it is the only way
• Social tagging may be best way to find and organize relevant content.  Leverage the social contribution to content. This has to happen to all content systems.

We live in an era when technology change is happening faster than ever before.  Mobile is rapidly becoming the new dominant form of personal communications.   The mobile device is not only portable, but is is always connected if we want them to be.  There is a tsunami of new devices.  It is the preferred way to access the cloud.  It rides the SaaS wave.  There is a big shift to “Do It Yourself” (DIY).  People are predominantly getting their own devices, provisioning themselves, and setting it up themselves.  This hooks into a flood of Big Data.  The mobile devices are packed with sensors: video, audio, location, etc and this can be leveraged in a multitude of ways.

For the first time in a generation we are moving away from mouse to a touch based user interface.    Email is now being challenged by multiple technologies.  Similarly, the “No SQL” movement is radically changing how we process data and knowledge, and this is seen as a response to the changes of the times, not the cause of anything.  Simply put: the old way does not work.

Mobile technology has been adopted faster than anything else in history.  Tablets are now the fastest growing segment of the tech market.  There will be 1 billion android devices sold in next 12 to 18 months.  Smart devices already rule the roost in corporations.  Laptops still growing, but eclipsed by smart devices.  The challenge of the technology crowd is to make it work and make it safe.

Social Net, Mobile, and Search are the three most important applications today.  We don’t do this well inside the corporation,  but we need to.  The change keeps coming.

The average age of a tablet: 2 years,  19% of workers now have one,  50% of phones are smart, 1M mobile applications.  This is not just gold rush of companies running to grab a new market, but people are demanding mobile devices and applets.  They expect and app for all information needs, and they expect a good one.  It is not acceptable to simply take a web application and make it fit the smaller form factor.

A smart device is not just laptop without keyboard.  The average one has 12 real world sensors which can capture all sort of other information about the user while using it.

80% of firms have some form of Bring Your Own Device (BYOD) policy.  Some companies are looking at radical new approaches: for example, a new hire will be offered $13,000 to spend on equipment.  Employee can get whatever they want, but there is no support from the IT department.

Companies have fallen behind this trend.  1 billion digital natives have moved to such mobile platforms, and are waiting for companies to catch up.  Younger employees will expect it as a natural part of their working life, and are looking for companies that understand.

There is a large amount of channel fragmentation.  Platform fragmentation is the norm.  Still, old systems are not being retired nearly as fast as new systems are being brought on board.

If you are going to provide access via mobile devices, the application has to be really easy and reliable.  If you don’t give them something like Evernote, they will use Evernote, much to the consternation of the IT department.  External technology is changing, we have to gear up for it within the enterprise.  Younger workers have a DIY mentality, and they don’t care what the rules are.  Disposable IT is the new thing: in order to accomplish a task, they might download an app, and then later discard it when the task is done.

Lifespan of average company is 15 years now, down from 75 years in 1940.  Driven, of course, by pace of technology adoption.

No small system can withstand sustained contact with a large system without being fundamentally disrupted.

Business value: mobile platforms allow you to be more connected than ever before, and this means customers more engaged than ever before.  For an application to be successful, it must much faster, easier to use, low barrier to access, fun to use, even though the IT guys say they are not in the fun business.  It is competition in the enterprise, and fun apps will win.  Employees want access to everything in a single place.  Best of all is the “one button control to your life.”   He gave examples of apps to control his sound system,  thermostat.

Expected functions:  Aggregations – show me what I want all in one place.  In-phone assistance concierge like Apple’s Siri which have built in access to Wolfram Alpha and it able to answer almost any question. For example “how many water molecules in a drop of water?”  Social Features, location awareness, native functionality (not HTML5 because it does not yet work as well)  One of the killer apps is self service CRM where customers can serve themselves.

Free apps to access your services are required, but they also have to be good.  He showed apps for a few finance/investment companies, but at the same time showed the “Pageonce” app which is not free — they charge $7 — but gives access to a collection of financial services and has nearly a five star rating.  These apps are getting the connection with customers.

Then there are some really cool innovative ideas like:

• the “star walk” app which has an augmented reality mode that points out constellations.
• Word lens is an augmented reality app with real time translation which actually inserts the translated text into the picture over where the other language had been.
• virtual assistant button: one button pizza – you enter you preferences, credit card, ahead of time.  Then, with a single click of the button a pizza is delivered to the location that your phone knows you are standing.
• Another really useful one is the taxi button.  One press orders a taxi, and then goes into a mode with regular updates on how close the taxi is,  and when it is finally there.  The taxi company already had the system in place for tracking taxis, but this app simply tapped into it allowing the customer the benefit.  Whatever your company does, you need to have “the button.”
• Multi-point video conference like Oovoo.
• Self-serve CRM applications
• Business intelligence

Not just about computing while you walk around.  Creates new possibilities representing a fundamental reinvention of our computing environment.  The bar is very high and you have to build competency quickly.  Users want the same good user experience as other applications.

Future:  New devices will come with near field communications which will allow the phone to interact with a multitude of near by sensors.  Location awareness in 3d space.  Battery life is getting longer.  He can use an iPad for entire flight while the laptop will only last a few hours.  The devices are always on, always available, so people can be productive whenever they want to be.   Business applications around touch.  Voice control is becoming bigger.

New operating systems are coming — for the first time windows is actually threatened by new suppliers without long experience.  The native OS is still very important. For simple applications HTML 5 is OK.  Still a lot of things that native OS does that HTML5 can not do, and the better app will be on the native OS.

App stores:  a bigger challenge than mobile.  There are 100 different companies are setting up app stores, but only a handful will remain 5 years from now.  Most IT departments are unable to block them.  This idea has spread to all software.

An open debate with whether the future applications will trend toward lots of small apps (atomization) or a few big powerful apps.  He thinks that atomization will be the future.

Q&A

How do you recommend for us to balance blazing change of IT with users who tire of changes?   Some people like change, and others don’t.  The slower parts slow the whole organization down.  Need to decentralize the adoption pattern, and try not to force the slow ones to keep up.

He started with a story of a collaboration challenge.  DARPA set out a challenge and offered a $40K prize for a team to find all 10 weather balloons that were distributed around the country.   They wanted to see how quickly it could be done, and to observe exactly how it was accomplished.  The MIT team decided to leverage social networks.  They offered to pay people who had seen the balloons, and also to pay people who introduce them to people who had seen them.  And even a third level of people who find the people who find the people who saw the balloons.  What they in fact set up was a distributed query through people across a very large, very badly formatted database.  It worked brilliantly.   Even more important, DARPA has set aside 30 days to accomplish the challenge, but the MIT solved it 9 hours.

What we are tapping into here is this extra capability we might call “cognitive surplus“.  People have time to think and work on thing, but there never before has been a network that would allow them to collaborate on this scale (and at any time or place they wanted).

What is the size of the cognitive surplus that is available?  One way to get a feeling of this is to compare this to another big job.  Start by asking the question: how many person-hours did it take to create Wikipedia?  A study was done to estimate this, and concluded that Wikipedia represents about 100 million hours of work — that is enormous.   But how does it compare to the amount of cognitive surplus that is available?  We might compare this to the amount of time watching TV.  It is estimated that America spends about 200 billion hours watching TV every year.  Think about that: a Wikipedia could be created in one year with just 1/2000 of the time spend watching TV.   In fact, a Wikipedia worth of time is wasted every weekend just watching ads.  Nobody has been previously networked at this scale, and it has never before been possible to easily share this information.

Information management and people management now overlap.   Knowledge management is not something that is extracted at the end of Friday and dumped into an archive.  Instead, knowledge management is a real time that people are doing all the time while they work.

This opens up some important new opportunities for collecting data on a scale, and with an ease that has not been possible before — and turns the traditional data collection ideas around.   Consider a project to distribute the collection of data for flu surveillance.  This is typically tracked by officials reporting for a county, to a region, and from there to a larger area and so on.  Someone decided to try something different;  go to each local clinic, and  just ask them to fax in a single number every day representing how many cases of flu they observed.  The data collection exported pointed out all sorts of problems: there were no controls to assure that data was consistent, or even was representing the same thing.  However, cleanliness of data does not matter, when you worry only about the trends.  If you are not trying to measure exactly how many cases there are, but instead just want to know if it is increasing or decreasing, then this is good enough.  The results clearly showed that the peaks of flu cases matched the official records.  The real advantage is how much faster this is collected.   The data comes in and can be viewed every day instead of a couple weeks later.  This is especially important in the case of a brewing epidemic, to identify trends as quickly as possible.

Consider what they do at Bit.ly, the URL shortening company.  They took a look at when a user clicks on X what also do they click on next?  Got a good map of strong correlations between topics.  This is a map showing real answers to one of the hardest questions to people who deal with information.  Normally you have to refine and collect and it takes a lot of time.  It makes a dramatic difference to be able to get this kind of instant information.

Told a story about the “stack overflow” web site.  Even though Microsoft has access to their own source code, many Microsoft engineers hang out at stack overflow because it is a reality that is best represented there.  Inside Microsoft they don’t have access to the real world of what users are doing and needing.

No matter who you are, most of the smart people work for someone else.        -Bill Joy

Value is building up outside the firewall.  Industry after industry will have to face this.  Stack overflow is a general purpose architecture;  they found they could re-purpose it for other things.  They started looking for groups of people who have intelligence of a particular domain or subject.  Find the group, then build a stack exchange site.   This is one of the big changes we are seeing.  If you as a company do not offer a way for such groups to collaborate around your products, someone else will.

Information Management has been often in practice a write-only medium.  Lot of stuff will be stored, but people almost never ask for anything back.  What is happening out in the world is a way of rethinking the feedback loop.

This is showing up in unusual places, like Foursquare.  They are able to collect incredible information on user behavior.  Presented graphically to finish the feedback loop.  Now, there is a whole generation has more accurate information about their drinking habits, than they do about their working habits.  What if this was done for workers?  It is ironic that our social life is better instrumented and better designed for feedback than what goes on inside the organization.  There is an opportunity to collect and give it to the user before they ask for it.  People don’t normally know what kind of questions they might even ask, and so you have to show them first.

Another example, “Patients Like Me” web site where people can share health information.  The outcome is information on what people report on what they are treating with, and how well it is doing.  This is hard and expensive to collect otherwise.  All of the data is self reported, but aggregated in near real time.  There was a large venting thread about depressed people needing support.  This aspect of the site is not very medical, but it turns out that this is an important part of of making the whole thing work.   People enter their data for aggregation BECAUSE of the venting threads.  Don’t make the mistake of eliminating the support for human interactions in a social environment.

All of this defies a lot of the traditional assumptions of aggregating and collecting data.  That is a real challenge.  It is a huge opportunity as well.  Managing knowledge, managing people, together.

Q&A

What about my company of 300 people:  Is there a critical threshold for making this kind of collection working?  Work with intensity rather than scale.  Show the partial results to employees.  This can generate value even inside the firewall with smaller groups.  Monitor data that allows you to see the trends:  Where are there an unmet needs?  Try to identify a small trend, try small interventions, instead of large interventions, and then monitor change.  For example hand washing and hand sanitation stations for flu prevention.  This can be effective because the feedback loops allow for detecting more detailed changes.  Start “small and good”  and not “large and mediocre”.

As a technology analyst my job requires me to sign up for and access hundreds of different web based applications, most of them cloud based.  Those that perform right are a pleasure.  Most, however violate one or more of these guidelines.  As far as I can tell, this comes from a failure to think through all the consequences of the design.  After a particularly egregious example this morning I compiled this “Bill of Rights” that a user should enjoy in order to have easy and satisfying access to your cloud service.  This is exclusively about how a user registers and maintains their profile.

Guidelines

 1) A user should not have to learn a new user name. A user should be able to bring their own user ID in the form of either an email address, or an OpenID. — The site should not allocate a user name to the user, nor force the user to choose a user name unique for that server.

Sites that violate this guideline will prompt you for your “user id” which is often restricted to alpha-numeric characters, and might have a length restriction 8 to 12 characters.  I have an identifier I usually use, but more often than not, it will respond with “That user id is already taken”.  I try again and again until I find a short sequence that is not already taken, but how do I remember that ID for that site?  I write it down in my log book, but this means that every time I want to access that site, I have to find the place in the log book where I recorded it!  That is so 20th Century.

This is completely unnecessary.  Let me use my email address as a login ID.  I know that is unique to me.  It is familiar enough to me – no trouble to remember.  Better yet is to use an OpenID because then I don’t have to specify a password.  Both of these are ways to “bring my own ID” to the site.  Don’t push the job of remembering a unique internal key for the profile on the user, instead allow the user to log in with their own login ID, and then find the internal key from that.

2) Each user must have the ability to change their password as desired. – A user should not be forced to use a system defined password.

This morning I had to sign up for a service, after registering I received an email with my user name, and another with my password.  After logging in, I searched for the way to change my password.  There was none.  (This was the incident that prompted me to write this blog post.)   The site had been implemented on SharePoint.   After searching around, I found you have to purchase an after-market add-in to SharePoint that allows users to change their passwords.  Ridiculous!  I can hardly believe that I have to send an email to the administrator in order to reset my password … and apparently I have no way to specify what that password should be.  Into my log book goes the password, right next to everything else you need to access this site.  So much for security.

The inability for a user to change their password as part of the site pretty much rules out SharePoint for any serious cloud site.  My guess is that Microsoft wants all the user profile stuff to be managed through Active Directory, and this makes sense for a on-premises environment, but unnecessarily complicates a simple cloud based service.  Paying extra for this capability is annoying … it really should be built in to the core capability.

3) Each user should be able to use whatever password they desire. – Passwords should not have arbitrary and unusual requirements.

Forcing the arbitrary use of classes of characters does not make password more secure, and often serves only to frustrate the user. I know that users in the past have chosen poor passwords: single words.  Some education is necessary.  It is a good idea to include some punctuation, but not always.  Studies have shown that remembering a phrase, and taking the first letter from each word (such as “General Tsu’s Chicken At Peking Palace Is Spicy” or  gtcappis) produces a string of characters effectively impossible to guess that is still easy to remember.  There are no punctuation marks, but that does not mean the password is easy to break.  Attempts to encourage better passwords by requiring certain types of characters can backfire, as it did one time for me: the requirements were so onerous that I figured out the simplest way to comply, and found out several others were using the same password.

A better approach is a password quality measure which gives you feedback on how good your password it.  This way you give a positive reinforcement to those who choose good passwords, but don’t prohibit any password based on the kinds of characters that are in it.

I use the same password for all these site.  Some people arrogantly suggest that I should use 100 different passwords at 100 different sites.  That would require that I write down all the passwords in a place that could be read by others.  The best solution is OpenID (or Facebook Connect or others) which I could use at 100 sites, and changing the one password on that one ID would effectively change it at all 100 sites.  OpenID makes it easy to implement the best security policies.

4) The site should show the user what it is they are logging into, before prompting for the userid and password. — The site should not pop up an authenticating box without any display of the context that user is logging into.

This happens to me all the time: I click on a link, the page clears, and I get a login prompt.  What cloud user account should I use?   The link was for a document, but the link does not always make it clear what authentication domain the document is protected by.

What worries me most is that this could be a phishing site.  I enter my username and password into that box, and who knows where it goes to.  Of course, a real phishing site might display something pretty convincing to fool me, but a completely blank page with no indication of where you are is simply too easy to fake.

5) If the user does not have a profile, the site should show to an unauthenticated user how to register, and how to get approval to access the site. — It should not simply show a blank screen, or a failure to authenticate message.

Just last night I was referred to a site that needed authentication.  I could see from the URL that I had never been there before.  But failing to authenticate left me with a completely blank page.  I had no idea how to register for an account.  I tried all the normal tricks: shortening the URL, etc.  Finally, I had to send email to someone, who sent me back a link to the registration page.  Why didn’t the failure to authentication give me this automatically?  Why do we make people be the critical path in giving users information about how to use the system?

6) A user registering with a particular email address must be accommodated whether there is a prior profile or not. — The site must not refuse access because there is already an account with that email address.

I recently tried to use a popular PHP-based forum.  Apparently, a couple of years ago, I registered for the forum, and created a user ID for my email address.  I have no idea today what that user ID was, but it absolutely would not allow me to make a new account with that email address.  Furthermore, it would not tell me the user ID, which was required to reset the password.  The whole thing is needlessly annoying.

Whatever account has my email address is clearly mine.  A user should be able to enter their email address, get the user ID, and reset the password, without needing an administrator.  Proving I own the email address is easy: send this information to that email address.  It’s as if the designers believe their system so special every user will remember a special ID just for it.  The email address seems to be an after thought.  More and more we are seeing the email address used for the login ID, eliminating this problem.

7) Every web site should offer a mechanism for password recovery, where the user enters an email address, and is sent a link that can be used to reset the password to a new value. — You should never need an administrator to reset the password.

Most cloud sites today offer password recovery, but still I see software inside the firewall lacking this feature – for example SharePoint.  The reliance on administrators makes this kind of software too expensive for the cloud.

Note that the service should never send the password in the email message itself.  Instead, it should send a link that allows you to reset the password.  This link should work only once, and should time out.  Passwords should never be sent by email.  Instead, a link, that offers one chance to change the password should be sent.  All traffic to and from the web site carrying the password should be HTTPS so that there is no possibility that the password will be seen by anyone without you knowing it.

8) The user should be able to specify an arbitrary display name that is used to indicate that user in the UI. — The email address should not be used for this purpose. The display name should be able to be changed at any time.

Once again, most cloud sites allow you to specify your display name.  Using your actual email address for display is rare, but some sites force you to specify a unique user ID and that is sometimes used for your display name.  The big problem is that the user ID can not be changed, because it is the key to access your information.  The solution is simple: allow the user to specify a display name, and allow them to change it any time they want to.  The best cloud software does this today.

9) The site should gracefully handle the situation when a user changes email address. – They should not be forced to create a whole new profile just because they are using a different email address.

Email addresses make good global IDs, but they are not permanent.  There are many reasons that a user may change email address.  The system has to be designed to support this.  Thus the email address can not be the key that identifies the user inside the system.  If done correctly, the user should be able to have any number of email addresses simultaneously.  The user simply adds the new address, and starts using that, ignoring the old email address, or deleting it if they wish.  Same should be true with OpenID or other ID systems like Facebook and Twitter.

Randy Farmer has a great write up on how this should be accomplished with a three part ID:   one part is the internal key, second part (multiple values) is the login globally unique ID like an email address, and the third part (multiple values) is the presentation or display name(s) that you may wish to use.  That design that allows all these guidelines to be met.

10) Web site should make a clear display of the information feeds that the user has subscribed to for email notifications. — User should never be prevented from unsubscribing from any notification streams.

Many, possibly most, cloud services include an unsubscribe link in any email message that is sent out.  This is just plain old web manners.  The list of subscriptions is less common, but something that a good cloud service should provide to the user.

11) A user should be able to deactivate their own account at any time, unsubscribing from all notification streams, and destroying all private data. — User should not be stuck with a zombie account.

The cloud service allows the user to register.  It stands to reason that it should allow them to un-register.  Safeguards should make sure that such a dangerous action is not taken accidentally, and there probably should be a period of time (a month?) within which this can be undone, but those are details.   Once deleted, it should indicate to others that the account is no longer active.

There is a lot of concern in the public that once you start a cloud account, there is no way to get rid of it. Note that deleting a profile does not necessarily means that all the posts you made are removed.  This will depend upon the type of service.  If you participated in a public discussion on a forum, then your forum questions and replies are part of the collaborative product of the site.  Notice that I said that “private” data is deleted.  Each site will have to determine what is private, and what is not, but if there is private information it should be deleted so that there is no way in the future to violate the privacy agreement.

12) There is no twelfth guideline

I have been reading way too much history recently, and it was the ancient Sumerians who introduced to the world the idea that things should add up to 12.  Eleven guidelines was simply not enough; one more was needed to fulfill the list.  However, this is a freebie:  if you satisfy the first 11 guidelines, you get the twelfth one automatically and for free.  Consider it a present from me to you, and a reward for getting the other guidelines correct!

Cloud User’s Bill of Rights – Summary

1. No New Name: A user should not have to learn a new user name.  A user should be able to bring their own user ID in the form of either an email address, or an OpenID.  — The site should not allocate a user name to the user, nor force the user to choose a user name that does not already exist on the server.
2. User Specified Password: Each user must have the ability to change their password as desired. — A user should not be forced to use a system defined password.
3. Unlimited Password: Each user should be able to use whatever password they desire.  — Passwords should not have arbitrary and unusual requirements. 
4. Context before Login: The site should show the user what it is they are logging into, before prompting for the userid and password. — The site should not pop up an authenticate box without showing any context that user is logging into.
5. Guidance for Registration: If the user does not have a profile, the site should show to an unauthenticated user how to register, and how to get approval to access the site.  — It should not simply show a blank screen, or a failure to authenticate message.
6. No Deadlocks: A user registering with a particular email address must be accommodated whether there is a prior profile or not.  — The site must not refuse access because there is already an account with that email address. 
7. Password Reset: Every web site should offer a mechanism for password recovery, where the user enters an email address, and is sent a link that can be used to reset the password to a new value.  — It should force you to get an administrator to reset the password.
8. Display Name: The user should be able to specify an arbitrary display name that is used to indicate that user in the UI.  — The email address should not be used for this purpose.  The display name should be able to be changed at any time.
9. Address Change: The web site should gracefully handle the situation when a user changed their email address.  – They should not be forced to create a whole new profile just because they are using a different email address.
10. Subscription List: Web site should make a clear display of the kinds of information feeds that the user has subscribed to received email notifications on.  — User should never be unable to unsubscribe from any or all notification streams.
11. Deactivation: A user should be able to deactivate their own account at any time, unsubscribing from all notification streams.  — User should not be stuck with a zombie account.
12. There is no twelfth guideline

This image (click on it for larger view) helps to represent the problems that traditional manual process discovery runs up against.  When a consultant interviews people on what the process is, each person has their own view of the process, which may not in fact be accurate.  Then there is what I call the “bicycle riding problem”: The person may in fact have the ability to accurately perform the process, but can not explain how they do it.  Many work behaviors are like this where people operate on tacit knowledge that they can not put into words.   Finally, the person may have an accurate understanding of their part of the process, but they may not be willing to say exactly what it is.  They may feel that the process as it is performed is broken, and so they will report instead what they thinks that the process should be.  All of this gets in the way of discovering the real process.  Process mining cuts through all this by analyzing the real evidence behind the process, and giving you an ego-free picture of the process as it really is.

The webinar was coordinated with a white paper that Sandy wrote on the topic of process intelligence: “Enabling Process Intelligence Through Process Mining & Analytics” which can be downloaded from that link after registering.

Also this week, Vance McCarthy published an article at Integration Developer News on the product: “Fujitsu Interstage BPM Unites Automation, Analytics, SOA Loose-Coupling.”

Questions & Answers

A lot of questions asked in the webcast did not get answered there, so let me take the opportunity to answer them here:

Q: The solution relies heavily on event logs. What about other data, including unstructured (e.g. email)?

There is actually quite a lot of information that is structured appropriately to be mined.  Almost any application will keep a record of what people did, when they did it, and what it was done for.  It is possible to mine semi-structured information, like email messages, but it takes more work.  You have to process it to extract the key pieces of information from the text.  While this can be done, in my experience we have never needed to resort to this.  I always get asked this question, because people always are worried that they might not have the structured data, but in my experience there is a lot more structured information being saved that any of us realize on a daily basis.  Mining that usually gives you a very good picture of the process, and you usually don’t need to resort to mining the semi structured information.

Q: What are the advantages of the process flow diagram vs a Gantt diagram?

Sandy gave a great answer to this question which I did not capture here (you will have to listen to the recording to get that).  I think it is an excellent question because I am always talking about the advantages of a simple checklist over a flow diagram when it comes to processes which are expressed directly by workers themselves.  A Gantt chart is typically seen in project management software as a way to represent a sequence of tasks and their time line.  This is an approach that is good for planning a single instance of a process.  Built into the Gantt chart are some assumptions that have been made about the order of the tasks, and those assumptions are not explicit in the chart itself.  While Gantt charts are good for planning a single instance of a process in detail, it is not a useful approach for representing collections of processes.  When we analyze 4000 process instances, the result can be messy, and there are many different paths that are taken, with some paths taken quite infrequently.  It is very difficult to represent this in a Gantt chart, and so we draw tasks as boxes, and lines for transitions between them.  This allows us to represent any number of different flow paths on a single diagram, and we can include details such as how many times the path was taken, and how long it takes typically.  Thus, the flow chart is really the better way to represent aggregate information about collections of processes.

Q: Is Fujitsu technology using university research Sandy mentioned?

These actually were developed independently.  While a number of our competitors license the University of Eindhoven technology, our patented technology was developed within Fujitsu Labs over the past 15 years.  It is a slightly different approach from the other, although they both achieve process mining by consuming event logs and producing process diagrams.

Q: What is the difference between Process Intelligence and Business Intelligence?

Business Intelligence is a term that has been with us for 10 to 20 years and it refers to using analytical processing techniques on captured business data.  Generally in BI data values exist in time and space independently from other values.  Process Intelligence extends this idea by adding correlations of events with each other over time, and that adds an additional avenue of information.  Let me see if I can explain: a BI tool might analyze the number of bids you sent out, and the number of contracts awarded, the process intelligence tool will match individual bids with individual contract awards, and with other actions, and be able to report specific information about how things flow.

Q: Will “Process Intelligence” be the responsibility of process owners or IT staff, or both?

It has to be the responsibility of both, and we are seeing this increasingly to be the case.  Without an understanding of the meaning behind the numbers, you can not make meaningful reports.  It is the business person who can recognize that a particular value is troublesome, or a particular pattern is worth watching, and this will drive reports to cover those specific situations.

Q: Can you explain the difference between process mining and collaborative process design? They both claim to be doing process discovery.

Both are techniques for uncovering the process that exists.  Collaborative process design is an approach where all the people who are knowledgeable about the process collaborate online to define the process.   This is really quite similar to the traditional manual process discovery where consultants interview workers and document the process.  The difference is that instead of a consultant doing an interview, the worker or manager enters the information themselves using an online collaborative tool.  It still falls prey to the misunderstandings that people naturally have about their view of the process: they might not know how the process works, or they may know the process but be unable to explain it, or they may tell you not how the process works, but how they think the process should work.  What you end up with is a collective agreement about the process, but that is no guarantee that it is the real process.

Process mining is a very different approach.  Event records are taken from the systems, and a process diagram is produced that is an accurate depiction of what really happened.  It does not involve any interruption to the people on the process, and it does not fall prey to the limitations of understanding the process.  It is messy, because reality is messy, but it is much quicker and more accurate than collaborative process design.

Q: How long does it take to deploy it? How long before I see results?

It is very quick.  First of all, there is no need to “deploy” anything … it is not something that is installed into the network and watches traffic.  Instead, you extract data from existing databases, and analyze that.  The initial results can be available within hours of starting.  To get the best results, plan on spending 2 to 6 weeks in order to explore process space fully, and understand the meaning of the outlier process instances.

Q: What do I need to do to start using your product?

Interstage Process Analytics is available today.  Training is available regularly, or if you prefer we have consultants that can come and help with the extraction and analysis immediately.  You identify a process, and then you identify the systems that are used by the people in that process.  After that, it is a matter of extracting the event records from the database

Q: Process discovery sounds too good to be true… How can you convince me it works?

The results are pretty amazing.  We have some case studies available for the asking.  Some people believe that the cases have been cherry picked and are special situations, or at least are not comparable to their own situation.  This is a common misconception.  If you know you have a process problem, it is relatively inexpensive to make a trial run to see what can be discovered, and that is the only way to be assured that you have the event information necessary to make it all work.

Q: Why are you calling it the next generation of process intelligence?

This is the next logical step of process support: from initial orchestration, to elementary analysis of process records, to advanced process mining combined with real time alerting.

Q: Have you any numbers that speak to the costs involved in manual process discovery?

We don’t have any solid thoroughly researched estimates of what it costs on the average, but it is widely known and accepted that a modest manual process discovery can run into multiple person-years of effort.  Some estimate that 40% of a BPM project goes just to the process discovery phase.  Our own experience with discovering processes in house are consistent with it taking multiple person-years to achieve — this actually was the reason for developing the technique in the first place.

Q: You mentioned that workflow in ERP systems like SAP can be instrumented and visualized by Interstage. Don’t ERP suites already do this monitoring internally themselves?

It is very rare to have the capability to correlate separate events and to be able to analyze the results of that.  So, essentially, no, ERP suites do not have this capability built in.

Q: What are the advantages of combining process mining and analytics?

Process mining gives you an overview of the end to end process as it has been up to that point.  Process Analytics gives you access to real time data of things that are occurring in your system.  Combining these allows you to use the mined models to design alerts against transitions you need to be informed about.  Also, as more data arrives, the process mining can be run again, and always kept up to date.  As Sandy said, combining chocolate and peanut butter can produce a combination that is better than the two separately.

Q: Can I reuse the process model?

Interstage Process Analytics will convert the process model to an XPDL file which can be imported and used in a BPM implementation project.

Q: Is there any additional data I need to make the discovery work?

The requirement is that there are three critical data elements: the timestamp, the task being performed, and the case or context that the task is being performed in.  Bringing in additional data gives you additional analysis options.  For example, including the office location will allow you to compare how the process is performed in different offices. Including the product model name can allow you to compare processes across different model lines.  Each additional information column will give you another dimension upon which to separate out process instances for comparison,a nd can help you to isolate problems spots.

Q: Is process mining limited to certain data sources?

Interstage Process Analytics imports data from CSV files, so if you can extract the event records out into a CSV file, it can be used for analysis.

Q: Can you elaborate on how this figures into what Gartner is terming “Intelligent Business Process Management Suites?

Gartner is proposing a new category of products that combine analytics and traditional BPM techniques together into a single suite.  I believe is combines the active BPM process orchestration together with process intelligence and process analytics.

Message senders do so with all the best intentions, but there is a better way to share documents, if only they would take the time to learn how.  Instead, they unwittingly propagate a pattern that wastes wastes time from everyone.  It is really not that hard to put the document in a document management system, and then mail a link to it.  Why is uptake so slow?  To help them along on the road to wisdom, here is the message that I would like to email back in response:

Dear Sender,

I writing in response to that document you just sent me by email. I wanted to let you know how much I appreciate it, which is not at all. I am sure that the document you sent might be useful to me someday, but for that to happen, I need to store it someplace where I will find it again when I need it. That means I have to read enough of the document in order to figure exactly how it should be classified. Even then, it can be ambiguous: do I put this under a particular customer name, and sometimes there are multiple, or under a salesperson’s name, which also might be multiple, or under the product name, which also might be multiple, or maybe a project name, or under the date I received it, or the date it was written, or possibly even the date on the title page of the document.

I think you can see that this only takes me a minute or two to decide where to save the document, but consider this: I receive dozens, if not hundreds, of documents every day, and this ends up taking a good part of an hour every day.

Wasting just my time being might be forgivable, but usually these documents sent by email are copied to tens or hundreds of other people, and every one of them goes through the same process. This means that across the company of 100 people several weeks of time is wasted every day, just to find the right place to store such documents.

That waste might be acceptable, except for how pointless it is. In most cases, the documents I receive are not the absolute final versions, and it is quite likely that before I get the chance to actually use that document, a newer version will be sent out. I will then waste the same amount of time finding a place to save that, but because it can be ambiguous where to save the document, and because priorities change over time, I am just as likely to save the new version in a different place. Then I will have two versions of the document floating around. Thus when I go to find the document, I am quite likely to find the wrong version, and not get the latest information, and that might be worse that not saving the document at all.

I am sure that you meant well, and imagine how awkward it is for me to complain about the document you are sharing with me. You are surely motivated by a desire to keep everyone informed, but probably don’t realize that this means of sharing causes a tremendous amount of work for everyone else.

May I suggest a better way: put the document in a repository or document management system that is accessible to everyone. Spend a few minutes tagging the document with the appropriate key words so that others can find it with a search. The message you were writing to me: instead put that in the description of the document. Then email me a link to the document letting me know it is there.

This way, I am informed about the new information, and can read it immediately if I want to. I can also be assured that it will be accessible in the future, if I need it, and I will be able to find it by searching at the normal document management system. If new versions of the document are produced, then put those new versions in the same place, on top of the older version, so when in the future I need this document, I will always get the latest one.

Yes, I know this is a little more work than just mailing the document as an attachment. But a couple of minutes of your time could save hours across the entire organization. If everyone does this, we all benefit. There will be fewer documents flowing through the email, we will all be spending less time saving and organizing documents, and we will be building a strategic library of documents that can be searched and found WHEN YOU NEED THEM.

As they say, the road to hell is paved with good intentions, and your intention to share important information with colleagues was honorable, it is precisely this behavior that makes us weaker and less effective. Please be more considerate in the future.

Yours Truly,  Recipient

That is it.  It would be so simple, and such organizations would be more effective.  I fear that the incentives are all wrong.  If I put a document in a DMS, I never quite know if everyone else can access it from there.  I never quite know if the link is going to work for them the same it does for me.  And, if this document is sensitive, I can never be sure that the access control is correct so that the right people can see it, and only the right people.

True, it does take a little more effort to put the document someplace.  It is soooooo easy to just attach it to the message I am sending.  Fewer systems to coordinate: everyone who gets the message gets the attachment (unless removed by anti-virous software).  Why should the sender worry about the time wasted by the recipient, anyway?

I am doomed to be an email attachment librarian for the foreseeable future.

John: To help our readers gain some perspective on your background, could you share some thoughts on what you do when you are not writing?

Keith: I am really a software architect: I help design system and that requires that standard processes and procedures be put into place within the entire team of developers. To do that, I do a lot of thinking about how to design rules that people can work with. In the early 1990′s I got involved in a number of standards efforts: OMG, Case Communique, and Workflow Management Coalition. This was because I saw the need for design rules across the industry, and these organizations were the best ways to accomplish such things.

John: What inspired you to write your first book?

Keith: It was a long time coming. In the early 90′s I got my first papers published in conferences because Fujitsu had no real mechanism to publicize ideas, and conference proceedings seemed the best way to get in contact with others who care about the topic. From there I moved to tutorials and writing the materials for that. Then a German professor contacted me to let me know he was using my tutorial notes in his college class on Business Process Reengineering and Workflow. I self published four books from tutorial notes and from other documents that I had developed over the years for guidelines on software development techniques. In the mid 2000′s I participated in a chain of technology tutorials with the WfMC. Robert Shapiro and I recorded one of our sessions and turned that into a book on BPM standards. I spend so much time trying to get the concepts clear to me, that I really was compelled to write the stuff down and make it available in a reproducible form.

What was the motivation behind your research and writing on the subject of knowledge work and Adaptive Case Management (ACM)?

Keith: My goal has always been to find a way to support office workers. I thikn I was always considering the creative work the knowledge workers do, although I was not using that term then. If you read my papers in the 1990′s you will find discussion of how each team, and potentially each worker, needs to find the right fit to their own needs. The idea behind “Collaborative Planning” was that getting together to decide what to do was an important part of actually doing things. This is a BIG difference philosophically from those who believe that all work can be automated, and that you can eliminate humans from work entirely. The Workflow Management Coalition always included the idea that some work can be automated, but some work inherently must be done by humans, that is enshrined in the Workflow Reference Architecture produced in 1995 and still relevant today. I, and many others in the BPM field, were struggling to support humans along side the automatable data processing. It was in 2009 that it became clear that the definition of BPM had “collapsed” in the public eye to mean only automatable work, and specifically orchestrating data flow between servers. People felt that BPM was a part of SOA. Many of us were frustrated by that, because there is so much that can not be automated and required human intelligence. The key concept was that people were doing work that could not be predicted in advance, and so predictability became distinguishing aspect of what was and was not knowledge work. We had a meeting in Maidenhead England where the WfMC invited many of the top thinkers on this topic. Eleven of the 12 authors of Mastering the Unpredictable were at that meeting. We felt that the subject was so important, and so urgent, that we decided to share the effort and write it all down. Five months later Mastering the Unpredictable was released to the public.

Are there any sections (or elements) in “Mastering the Unpredictable” that you are particularly passionate about?

That is a little like asking which of your children you like the best! The book progressively discloses ideas, and so the first chapter covers the concepts at a high level, and chapters 4 and 5 get into a lot of important details. Then the rest of the book reflects on use cases and how the technology fits into different fields. We know so much more know about the field, but still I think the book stands on its own for giving a cogent picture of what is possible.

Are there any particular challenges that come into play with ACM? Can you recommend any best practices?

The biggest challenge is getting IT departments to realize that they can’t automate the creative parts of the organization. They have a “Jetson’s Mentality” thinking that all work will ultimately be reduced to a single button press. At the same time, the people who lead creative organizations, and know that it can not be automated, tend to shun al technology because of the way that the IT department tried to reduce all work to automation. Creativity comes so naturally to people that we don’t realize what we are doing and how we do it. So we find that we have to sell not a technology, but a management philosophy. Many manager believe that there is actually exactly one way to do things, even though their teams are constantly changing form and method. Because they are not faced with the reality that work processes are continually in flux, they tend to believe that there must be one single best way to do something. Getting past the “Newtonian Illusion” that an organization is base on fundamentally simple rule is the biggest challenges to deployment of ACM.

Do you have any advice on how to incorporate social media into managing the unpredictable?

I am always careful to distinguish “social media” from “social technology” – they are similar technology, but distinct uses and benefits. Social media is typically used to broadcast to — and from — masses of people. Social technology however is a set of capabilities that take social network relationships into account. In many ways ACM *is* social technology because it makes use of explicit representation of relationships between people to support the work on a particular case.

Do you have any case studies where you’ve combined social media with ACM?

I would say that ACM is about supporting case managers to accomplish projects and get things done. One of the things they may want to do is to engage the public using social media and their use of social media will be not significantly different from those who don’t use ACM. One of the use cases covered in the book is “New Product Development” and clearly when you release a product, you are going to want to leverage social media in the normal way. So they go together but I don’t see any specific dependence between the two subjects.

In hindsight, are there any features of your published works that you would change or build upon?

There have been way to many discussion of the comparison of ACM to BPM. This is because we accurately predicted that those who see the world as something to be automated, would perceive ACM as being the same as BPM, and we wanted to preempt the discussion and try to make it the distinction clear from the start. However, most of this discussion is in vein because those who try to automate all work tend to think only of automatable work cases are important, and dismiss the unpredictable work as either nonexistent or unimportant. If I were to do it over, I would probably simply ignore the BPM comparison, simply focus on those organization that depend upon innovation and creativity, and show how ACM can support that work, without any comparison with BPM. It seems that the knowledge work support, and routine work support, are distinct problem that can be handled by different support organizations.

What are your current/future projects?

Organization are realizing the value of supporting knowledge workers, particulary that knowledge work is the most important element of competition in the coming years. As routine work is automated, what is left is work that requires a human intelligence, and better you can leverage human intelligence the more competitive you will be. I am helping companies to select the right technology and configure it for use in supporting knowledge workers. I am trying to work with a group of people to define more clearly what is and is not essential as part of an ACM package, and if things work well I would like to develop interoperability standards to allow various ACM system to cooperate with each other.

Can you share some of the feedback that you’ve recieved for “Mastering the Unpredictable”? Have people in the field embraced the concepts outlined in the book?

There has been a big and positive response among those who have to deal with creative knowledge work. Forrester have published a rating of case management products (they call it Dynamic Case Management). IBM calls it “Advanced Case Management” and people from TJ Watson Research Center have published paper recently on similar topics. A proposal have been submitted for holding the first International workshop for Adaptive Case Management in Sept 2012. Even with this response, we are still just seeing the beginning of movement. Most IT Departments are still focused on automating routine work, and for good reason: there is still a lot of low-hanging fruit or routine processes that really need automating, and such automation is saving organizations tremendous amounts of needless human activity. But over the next 10 years those routine processes will be mostly automated. What will IT departments do when all the routine processes are automated? They will implement ACM system to facilitate knowledge workers, and such support will will make the difference between a winner and loser in the marketplace. It will become a key strategic factor for a company.

Any recommendations on how to get IT departments on board with ACM deployment?

The most important thing is to understand the principle of “unpredictability”. I use the example of a search and rescue team to demonstrate the kind of decisions-on-the-fly that are involved. A search and rescue team prepares in advance, but not by making specific pre-defined actions, but instead to practice “patterns” of working together that can be reused. Think of this like different “plays” in American football: the team practices how to interact using different plays, but the game is never scripted, and even a give play has variability that is decided at the time of the play, and by the players as the play works out. The launch of a new movie makes use of patterns of working together (booking first run theaters, billboard advertisements, promotional products at fast food restaurants, etc) but no two movies are ever launched in exactly the same way, because every launch needs to take into account many factors of current events, culture, fads, trends, as well as competition that is being launched at nearly the same time. IT departments need to acknowledge that in certain types of work, there is no single process, but instead the process followed is different every single time.

Another example I often use is Dr. House the television show because it demonstrates in nearly every episode that the information necessary to predict the correct treatment is not available up front. The patient is dying, and something has to be done, but nobody knows what. After treatment starts, strange responses to the treatment give additional clues about what is wrong, causing the doctor to decide to change the treatment as it progresses. This idea of insufficient information up front, and evolving the plan as the treatment is being given is an essential aspect of Adaptive Case Management. Many of the patients have combinations of problems. Statistical analysis shows that even combining only two illnesses can give you hundreds of millions of combinations, implying that every such patient that comes to a hospital is potentially unique. The idea that there are a small number of standard treatments that can be prescribed up front must be abandoned.

Once the IT department understands that things are not predictable up front, once they have let go of the Newtonian idea that there is a simple rule and the bottom of all behavior, then it is fairly easy to get them to see that ACM provides a way to put an intelligent human at the center of the work. Instead of a factory that is automated to eliminate all human involvement, the IT system becomes a kind of “bionic limb” that allows people to access information faster, sift and sort it more effectively, communicate to others more effectively, and to coordinate the tasks of other people more powerfully. The IT systems become extensions of the decision makers, not the replacement of them.

For many, when they realize this, they also realize that they have always known that executives would never be replaced by IT systems. For many, this is a realization that IT systems can support more kinds of work than they ever thought possible before. Instead of being an approach that competes with existing ways of designing and implementing applications, this become a way to extend IT systems to jobs that have never been able to be effectively supported before. It is a new opportunity for IT departments.

What is your vision of the long range future?

If you indulge me in a bit of wild speculation, we might glimpse at the far future. In 10 to 20 years we will see the transformation of business such that all workers are executives. What I mean by this is that work will consist of making decisions: someone will have to decide how much money and resources to invest in a particular initiative, but the actions resulting from that decision will be largely automated. I don’t mean to imply that we all will live like some futuristic Henry VIII. There will be hundreds of millions of other executives to deal with, so the chief decisions will be about strategies to get others on board with your initiatives — or more frequently on whether to get on board with other initiatives. Once the decision is made to do something with all the right people involved, all the rest will be essentially automated and will proceed without any “work”. Decide that you want the latest electronic gadget and it will be manufactured for you in Siberia and automatically shipped to your doorstep and possible even set up automatically for you. Decide that angioplasty is right treatment for this patient, and all the rest of the preparation, handling, exchange of money, and supply of materials is done automatically, and possibly even the surgery itself (although I expect innovative surgery will be one of the last areas of automation because surgeons need to be very adaptive as they work). Decide that a particular product line is unprofitable, but that there appears to be an uptick in demand for another product line, and this can be automatically communicated to all the right people, triggering a cascade of decision making through the organization, but much of what we consider today as “work” will be fully automated. It is not a panacea: decision making can be more stressful and more exhausting than what we call work today. Furthermore, the effect of poor decisions will be amplified the same way that good decision are, and so the pressure to make the right decision increases. Progressive companies are recognizing this trend today, and preparing by focusing on getting good decision makers, and on IT systems that support decision making such as ACM.

Again, thank you so much for your time on this.  It was a pleasure chatting with you, and I think our readers will really appreciate this opportunity to further connect with the concepts that you discuss in your writing.

My pleasure as well.