http://www.processmodeling.info/posts/highlights-from-bpmn-2-0-activity-types/

http://www.processmodeling.info/posts/highlights-from-bpmn-2-0-artifact-shapes/

The new URL for this site is www.processmodeling.com.   Please update your feeds to the new site.

As a reminder I will be retiring the rickgeneva.com/wp URL January 1, 2010.

www.processmodeling.info/posts/recap-singapore-bpmn-training-event-2009/

I have mirrored all of the previous content before November 1st to the new site.  If you are a registered user you might receive some duplicate postings for a while.  I’ll be shutting down this URL January 1, 2009.   The new site is www.processmodeling.info.  For members of the old site, your registration information has already been copied to the new site.

I hope I didn’t lose any of my readers by moving the site to the new URL. This was necessary because I have to separate my personal site from my processmodeling.info site. www.process-modeling.com will redirect to processmodeling.info. So this is the official new home. Soon I’ll be back to writing again.

To others who blog, here’s what I’ve learned:

1. PHP is terribly insecure. If you use it, make sure it’s up to date. Many popular packages today are written in PHP. When a vulnerability is found, you need to update your software as soon as possible.
2. One password isn’t enough. Don’t rely on any sort of default security. Instead make it a complex maze of mixed types of security so that access to one area will not get far in the rest of the site. Yes, this is hard to manage, but so is rebuilding your precious website after some jerk (with obviously more time on his hands than you have) hacks you.
3. Learn every Apache server trick you can, and use it. I’m not going to give any specifics here (for security reasons). But just remember that your hosting company doesn’t provide anything but hosting. Security is your responsibility. If you get a good hosting company (I’m fortunate to have one) they can give you good, sound advice.
4. Host only what you need, and nothing extra. Anonymous directories will be eventually be found , and exploited. Get another account (it’s cheap) for anything not directly related to the site theme. Put extra passwords on things that you don’t think are very important. This is where the hackers look.
5. Everyone knows not to use their kids’ names or their pet names as a password (hopefully). I thought my passwords were pretty good. But then after some research I found out how easy it is to crack a typical password. But there’s hope. On an keyboard there are over 100 possible characters. Use a combination of the full range, and at least 10 characters. Again, a terrible pain to remember, but it’s your choice. Either protect it, or you’ll end up inadvertently helping to spread global terrorism, violence, or some quite offensive non-family oriented material.

I’ve been putting this off for a while but since I got hacked (Oct 25, 2009) I am raising the priority. See you there at the new URL.

Often I get an email from my boss that contains one or two short sentences. Sometimes I reply with a long description of my idea. He then replies to me saying “this is too long. Please revise and I’ll read your response”. The first time I saw this reply I was a bit surprised, and slightly offended. It took nearly a half hour to craft that wonderful email and he won’t even bother to read it. But then I realized how much work he must be handling on a daily basis. He’s constantly connected to people around the globe, with people in almost every time zone. So if I write an email that takes 5 minutes to read, this is only one of 100 other emails that also take 5 minutes to read. 500 minutes of emails a day? That’s more than 8 hours!

The point here is that the individual worker is hitting a point of information saturation. Not only are we asked to answer 100+ emails per day, but we are also tasked with meetings back-to-back all day long. So when do we have time to get anything done? I don’t know about you, but the only time I seem to get anything creative done I’m at cruising altitude on an airplane. Now my personal space at 33,000 ft (10,000 meters) is also being invaded (you have no personal space in coach class except in the confines of your mind, knowing that nobody will bother you for at least a few hours). Wi-Fi internet access is available on many flights. When airborne Wi-Fi is free someday, I’ll no longer have any excuse for not replying instantly to everyone who believes their request is the most important, above everyone else’s.

Process modeling 20 years ago was mostly about sequential activities. A worker would receive an assignment and complete it, then pass the file to the next person for subsequent activity. It was very natural to use flowcharts or swimlane diagrams to show the flow of process instance. But today we are more and more event driven. Processes are fragmented, distributed, globalized, and virtualized. Workers don’t work a regular 9 to 5 schedule. In fact, you cannot rely on someone even coming into the office because many people work from home one or more days per week. And they do this why? Because coming into the office often means being overrun by communication and unplanned events. There’s just no time to get anything done at the office.

Have you ever taken the time to model out a simple process that happens every day? For example, someone wants to create a proposal, and it requires feedback from 4 or 5 people. After feedback is collected, the manager will make the final approval or ask for someone to make some changes. How many times per day does this happen in your organization? How often does this occur over email? Try to model this process including all of the details, just to see how complicated it really is. Once you understand the simple process like this, it’s much easier to understand the complexity of a large process.

Most business analysts won’t take the time to do this sort of exercise. Instead, I see countless process models in a flowchart that is over-simplified and contains mixed objectives. I’m not suggesting that every process should include every intricate detail imaginable. I’m only suggesting that you try this exercise once, and see if it helps you understand the bigger picture.

The process called “Proposal gets approved” which leads to “was approved” and then loops back to “work on proposal” seems simple enough. But it does not consider the event based model that is the reality of the scenario.

Proposal Approval Fowchart

First, the process model assumes that everyone is available, waiting for the initial request (false). Most likely, the participants are in meetings, flying to meet a customer, or out of the office. Then the process assumes that everyone involved is well informed and will likely agree (false). When is the last time you asked for 5 people to agree on something? Five people from five different job positions and backgrounds are not going to agree on anything. Everyone has their own agenda, and perspective on the process. And finally it’s assumed that the manager will like what he sees and approve it most of the time (false). It’s much more likely that 3 or 4 revisions will be requested from 2 or 3 of the participants. In my experiences with this type of process, what usually happens is that the process never actually completes. Instead we run out of time. Then the manager ends up overriding some of the inputs and submitting the proposal. A few days later some brilliant feedback is submitted from one of the participants who was visiting a customer in another time zone on the other side of world. But it’s too late to include these ideas because the proposal has already been submitted.

Now that we have taken a look at the reality of the scenario, redraw your process model to accommodate the various events that can occur. Include the various deadlines (all of them). Include the various exceptional conditions that might occur (such as someone introduction of new information which might cause a complete revision). Include the assumption that people will not respond in time (take the pessimistic approach rather than the happy path). Also take into consideration that there might be more possible outcomes than what was originally planned for. Additional opportunities might arise during the creation of the proposal, or maybe the entire project must be scrapped due to unforeseen circumstances.

As a reminder, this is a very simple process… or is it?

This exercise is intended to be a warm-up to the real world. I often do this myself before I start on a large process. It gets me thinking about the exceptional conditions and helps me detect situations that were never discussed. It’s amazing how often I find organizations that have processes with no plans for the “what-if” scenario. Everything seems to route to one person whenever there is a problem. So what happens when that person is not available? Process deadlock? Maybe it’s ok to tell your customer that their request is being ignored by a middle manager. In my job, that could get me fired.

Process modeling is not about the ideal, optimized path where everything runs smoothly. That’s the easy part. Any middle manager with Visio can create a flowchart for that. Your job as a process analyst is to create a bullet-proof model that has a plan for everything, but simultaneously doesn’t bog the reader down with too many details. This is what is hard about process modeling.

- Rick Geneva

The event driven gateway is one of the most useful symbols that I know of in BPMN. Yet it’s often overlooked as a solution to common problems. It is a compound symbol, inheriting attributes of other BPMN shapes. First of all, it’s a gateway that is used to split sequence flow paths. The gateway is the diamond symbol. Inside this diamond shape you will see the intermediate event shape, which is the double thin line circle. Inside the intermediate event shape there is the pentagon, which is a symbol for multiple events. So when you put all of this together we have a gateway that deals with multiple intermediate events.

BPMN version 1.1 and higher

In all of my classes I tend to get more questions about the event based gateway than any other shape in BPMN. This is probably because there are so many use cases for it, and its pattern to many people appears to be drawn backwards. Most beginners in BPMN tend to draw the pattern with the intermediate event shapes to the left of the gateway. Let’s start by looking at the basic pattern and then discuss some of the potential business scenarios where you could use this notation.

The basic pattern that applies to the event based exclusive gateway

The main purpose of the event gateway is to illustrate that you must choose between one, and only one event attached to the right. Any other event would not correlate to the same process instance. For example, I send a message to a coworker, and he replies either by email or by phone. The message type is different for the reply, and requires a different action for me to process it. I don’t know in advance which message I’m going to receive. Therefore it is an event that I must react to and process accordingly.

The exclusive behavior of the event based gateway is due to the fact that only one path can be chosen. This is similar to the exclusive data based gateway, except that event-based means that we are reacting to events instead of data conditions. With data-based gateways, I already have the condition, and I must evaluate it at the relative point in time that it appears in the diagram. The event based behavior is expressing that we don’t yet have the condition at the specific point in time, and we must wait for an event to make the choice of which path should proceed.

There is a unique capability for this gateway to be used at the beginning of a process. It actually takes the place of the start event in this case. Often I encounter diagrams where there is additionally a start event to the immediate left of the event based gateway. This might actually be incorrect, depending on what you are trying to express. The literal interpretation when you use the event based gateway as a start event reads like this: My process starts when I receive one of the following events. The BPMN specification is very specific about using only intermediate events to the right of the event driven gateway. But if it makes more sense to you, think of the intermediate events acting as start events, with the gateway acting as the place where the choice of event is made.

The event based gateway acting as a start event expresses that multiple events can start a process.

When used in place of a start event, the event driven gateway is expressing that there are many events that can cause the process to begin. When used as an intermediate gateway, it expresses that the process is in a wait-state, and will proceed only after one of multiple events have occurred.

In either case of starting a process or continuing a process, the event based gateway merges with the exclusive data-based gateway. This might sound strange, but merging on gateways is found in a diagram when we want to express that a synchronizing flow exists. It’s similar to a traffic symbol. Exclusive merge means that only one path will ever be permitted through this point in the diagram. In contrast, parallel merge indicates that all paths are required in order to move forward, and will create a wait-state until all paths have reached the point of the gateway.

Using the exclusive data-based gateway for merging

Because per BPMN specification, the event based gateway is only permitted to have events to the right, we cannot use it for merging. But since we only need to express the merging behavior, the data-based exclusive gateway is a suitable substitute. Often a process modeler will omit the merging gateway. I am a proponent of always using a gateway for merging because it shows an explicit behavior, rather than leaving the diagram open to interpretation errors. For example, if I write a quote and don’t use the close quote symbol, doesn’t this leave you hanging a bit? So it’s better style to always use a merging gateway.

A best practice of using event based gateways is to use multiple event types whenever practical. If you have investigated the merging behavior of gateways, you might already know that it is possible to cause a deadlock or a replicated instance with improper usage of gateways.

Expecting the Unexpected Condition

A replication of sequence flow occurs when a parallel split occurs and there is no checkpoint or gateway to ensure that synchronization or merge of the parallel paths occur. For example, I split with a parallel gateway and I forget to merge with it. Somewhere downstream I could potentially end up with replication of effort on the same process instance. Suppose I send a new hire document to IT, human resources, and accounting departments in parallel. After these departments are complete with the document it is passed to another department who does a background check. The background check is expensive, and it is outsourced to an external agency. The external agency will not check to see if the same person is being processed more than once. There must be a point in the process that prevents the background check from occurring three times on the same person. Potentially every subsequent step downstream in the process might also occur three times until someone finally realizes the error and merges the files. Before the error is caught, duplication or triplication of effort occurs. This would be expensive and wasteful.

Without merging, a potential for replicating the downstream activities exists

Likewise with exclusive behavior it’s possible have sequencing and synchronization errors without properly handling the merging behavior. A deadlock condition occurs when a splitting gateway doesn’t match up with a corresponding merging gateway. Sometimes this can accidentally occur implicitly whenever the merging gateway is omitted from the diagram. For example, I send a document to one of three departments. A fourth department is expecting to receive documents from all three proceeding steps in parallel, but it only ever receives one. This can be a deadlock situation, meaning because of a merging flow error, the process will never proceed past the checkpoint.

Improper usage of gateways can lead to process deadlock

Deadlocks can occur in event based gateways as well. But in this case, it’s not because of using the improper merging shape, or from omitting the merging behavior. The deadlock is event based. Event based deadlocks occur when the primary means receiving the expected events has failed. For example, I send an email to my friend to have lunch with me. He might reply via email or via calling my mobile phone. But I don’t receive a reply so I go ahead and wait at the restaurant anyway. I assume he must have received the message, because email is so reliable, right? Meanwhile I’m waiting at the restaurant for him. I wait all day. I wait all week. Eventually, after about three months, the manager at the restaurant feels inclined to call the police to physically remove me from the restaurant. My friend actually did receive the message and left a note on my desk that he could not join me. Because the event (note on my desk) was outside of my processing capacity, I wasted three weeks of my life waiting. This was a terribly inefficient process, and could have benefitted from planning for the events based deadlock.

Obviously this is a fictional story. As human beings we have enough common sense to avoid this situation. But the elaborate computer systems we build don’t have this kind of common sense. A stuck automated process will wait forever until an administrator comes along and terminates the process or reboots the computer. Meanwhile, money was spent, customers were lost, paperwork piles up on someone’s desk. Eventually someone will realize the problem, but not until after the customer gets angry. Or maybe the error is realized after the board of directors becomes angry because your company keeps losing money.

The deadlock of events is usually caused by the assumption that things will always go as planned. The reality is that in business things go as planned only about 35% of the time. The other 65% is either the more expensive way to make money, or the ways that money is lost in a business transaction.

In the scenario of me waiting for my friend at the restaurant, my common sense tells me that I should only wait about an hour before I make the decision that he is not coming to meet me. For other friends this might be two hours because they might be an hour late on a regular basis. But the point is, I would not wait indefinitely. So the addition of an intermediate timer event to my process flow can save the day when I have an event based deadlock.

If you wish to be even more careful about event based deadlocks, try to consider more reasons why the process could fail. For example, as I’m waiting for my friend I hear the tsunami warning siren. The restaurant is by the beach, so I don’t think my friend will meet me. In fact, its time I consider evacuating to higher ground. In this case I would add a signal event to my process that listens to all “broadcasts” that might be interesting to me.

Full example of the event based gateway in action, using multiple events to avoid process deadlock.

A more realistic example

Now let’s apply the same pattern to a common business problem. Suppose that I send an order to my supplier. Do I wait indefinitely, assuming the supplier received the message? After all, they have been fairly reliable in the past, right? So what happens when I don’t receive the order? Should I continue waiting months later when I have no products to sell because I have no supplies to make anything? Maybe you assumed that someone else would realize this problem and call the manufacturer. But when will they make the call? Most likely it will be after supplies are low and we are at risk of shutting down manufacturing. In other words, it might be too late before we spot the problem. So instead, we model our process to include for provisions such as a timer that monitors when the product is received. Also we allow for the vendor to report an error in processing the order, which should prompt someone in my organization to reorder. And while we are at it, why not ask the supplier to broadcast system outage messages so that we know when to resubmit orders that might have been lost in transmission.

Ordering supplies with an error handling event and a timeout event

Hopefully this clarifies the usage of the event-based exclusive gateway. I hope you can now see that although the event based gateway uses a pattern difficult to understand, it’s well worth while learning. It’s one of the most powerful shapes in BPMN. Processes are not linear and don’t always function like clockwork. Modeling in the flowchart style with swimlanes and everything being perfectly sequential doesn’t depict the true nature of business processes. In the real world, processes are event based, and 65% of the time things don’t follow the optimal plan. By using an event based process model you are more likely to uncover the subtle details and pockets of inefficiency.

Application or Process?

A business process exists whether or not you automate it through a BPM system or a workflow tool. Many organizations choose not to use a formal BPM approach to process. Instead they use the traditional vertical market application that helps automate some of the process based on rules and logic provided by the software vendor. Some degree of process management exists with this approach. However the logic (the business know-how) is essentially outsourced to the software vendor. Often this requires the organization to undergo a massive customization effort in order to make the vertical market solution fully effective in the organization.

The application acts as a participant to the process. Without there being a business process there would be no need for the application. So you could say that what you use the application for is the process, and the application is the tool that helps you be more efficient at doing your part of the process.

Why BPM is a constant

Everything about business involves a process. Presenting a product to a customer, ordering supplies, and collecting money are all examples of processes. In a more simple term you could all each of these activities a workflow. The real benefit of BPM comes into play when you start to analyze the complex interaction between many of these individual workflows. Most likely the simple workflow evolves to include computer systems, probably just simple applications at first, gradually becoming more complex. These systems become participants of the process as well. Eventually the computer systems become an integral part of the process, often automating parts of the original process as well as enabling more efficiency as more people and systems are involved.

Let’s not forget that BPM is a management technique more than it is about technology. With BPM we are not talking about managing specifically people, systems, vendors, customers, or money. Instead we are talking about managing people, systems, vendors, customers, and money, as well as the complex interactions between them. The more complexity exists in a process the more efficient your organization becomes with proper business process management methodology and technique.

The next trend in IT

Enough about defending the need for BPM. Here’s what I see as the next emerging trend in information technology: Could Computing.

So what does cloud computing have to do with BPM? A lot, actually. Earlier in this post I contrasted the difference between applications and process. But we are now at a point in history where this line becomes even more blurred.

At the core of cloud computing is hardware; a lot of it. But instead of 4 or 5 servers to run one application we start to see a trend where 10 servers run 40 or 50 applications. The hardware virtualization means that CPUs, memory, and storage capacity is combined over multiple systems. The old days of redundant systems for maximum fault tolerance are coming to an end because the basic architecture of a cloud system is 3 and 4 times redundant in every way. In fact I’ve seen demonstrations where the plug is literally pulled out of the wall and the system keeps running. This is because there are dozens of power supplies with dozens of plugs, and often multiple complete systems in different data centers, all acting as one gigantic supercomputer.

The could computing design is more reliable, and it’s cheaper to operate. Moore’s law states that computer power doubles about every two years. The problem in data centers is that in most cases 95% of the CPU power is wasted sitting idle because it’s only utilized when someone is using that specific system. When people happen to be using another system, the CPU simply burns up kilowatts of power from the electric company as it remains idle, waiting to process the next request from a user; as quickly as possible. But with cloud computing, a cluster of computers are all operating as one. A peak load on one application can easily be absorbed across the entire system while the less frequently used applications.

The idea of cloud computing is undoubtedly inspired by the way the Internet works. At any specific point on the backbone of the Internet, if a failure occurs, there might be a localized disruption. But nobody has ever heard of the entire Internet going offline. It’s designed to be fault tolerant to a point where even nuclear war won’t take it offline.

Now back to my point about where BPM is involved in all of this. Prior to the cloud computing trend, systems were isolated in various localized data centers with virtually no way to communicate with each other besides for the system interfaces (API) that were designed to perform a specific function. Cloud computing brings systems previously separated by physical hardware together on one hardware platform. When you add service oriented architecture (SOA) to this combination, there starts to be less objections to using web services. Many software engineers and architects believe that a native interface (such as Java to Java or .NET to .NET) is better than web services for performance reasons. But in a Cloud environment when an application needs to talk to another system, often the other system is in the same virtual memory space on the same hardware cluster.

No longer do we have to worry about databases exceeding 50 terabytes. The cloud system can handle exabytes or even petabytes without even a flinch. So the notion of storing data once in a “normalized” database becomes too much effort to make it worth the effort of doing proper data modeling. Store it 1000’s of times in 100’s of formats to service dozens of applications because you can transform the data just as fast as you can store it.

Again, BPM is about management. If I can connect everything, nobody objects to connecting, and I have a virtually unlimited amount of computing power, imagine the complexity we can create! So what are you going to do with it? Burn up megawatts of power instead of the kilowatts of previous generations? Instead, how about getting smart about what you do with your computing power. All of the computing power in the world will not do anything for your organization if you don’t manage the processes that you are attempting to automate.

The trend as I see it will be for applications to be more process aware, and BPM systems to become more like applications. Many early attempts at BPM systems were nothing more than task state management for workflows. System to system integration was added, but this is still very task centric.

There are several categories of applications. Some examples include:

• Tools such as a calculator, disk defragmenter, backup utility, etc.
• Data origination tools such as a word processor, spreadsheet
• Information sharing tools such as email, screen sharing, etc.
• Collaboration tools such as groupware, and BPM automation systems.

The problem is that we still need one application for one job, and another application to do something else. For example, my word processor program is good for writing documents but doesn’t do so well at adding 2 + 2. My spreadsheet crunches numbers well but doesn’t manage people well (although some people insist that a spreadsheet is actually a database).

History hints at what’s next

If there is anything that history tells us about technology, it’s that consolidation of multiple systems is inevitable. Back in the 1970’s a CB radio or walkie talkie was all the rage in business communications. Then someone got the idea to combine a radio with a telephone and the cellular telephone was born. While we are at it, why not put a camera on it. Personally I couldn’t figure out this marriage of technologies out when it first emerged, but now I find myself sending pictures of my daughter to my friends and family on a regular basis. Oh, and while we are at it, why not hook the phone to the internet. For that matter, why not hook your refrigerator and toaster oven to the internet too? That way you can call up your appliances and tell them and make you breakfast before you get out of bed. Or better yet, as I sit here stretched out in business class on my favorite airline, I’m writing a blog post while connected to the Internet, powered by cellular phone technology.

In the above example there are a few major enablers of the merging technology. First there is a reliable cellular telephone network that is available virtually anywhere on earth people are found in mass. Next there is the Internet; always on; always ready to serve. This is the infrastructure. The telephone and the camera are the tools. The collaboration is when I hit the send button from 30,000 feet (9500 meters) above sea level, telling my wife how wonderful the remote control for the powered lay-flat seats are on this airliner, and she replies saying “that’s nice honey, enjoy. I have to put the little one to bed”. I’m constantly in touch with the world, powered by so much complexity that I never have to know about, or even know that the complexity exists.

The cloud , simply put, creates an enabling infrastructure that I never have to worry about. It’s there, it’s always on, and it would take a full-blown nuclear war or an asteroid hitting the earth to take it offline (in which case we’d all be dead anyway, so why worry about it). Data exists somewhere, but I no longer care where it is. Someone just added 5 more terabytes of RAM to the cloud and I didn’t even know it (yes, I said terabytes of RAM, not hard drives). My software got updated with a click of the button, and if I don’t like it I can click a button to switch back to the previous version without the tedious process of uninstall, reinstall.

So I’m on the internet writing on my blog page (the cloud enabled application) writing about the cloud, while I’m looking down at the clouds. Sorry to mention it, but I couldn’t resist pointing out the irony.

The green screen effect

It’s been said many times that we are coming full circle back to the days of the green screen. Ironically ‘green’ means something else today, which is causing the push to go back to the concept of the terminal attached to the mainframe. For those of you reading this who are too young to remember, the terminal was a green CRT screen. Remember the CRT? You know, that huge clunky tube screen. Before they CRTs were colored they displayed characters either green or yellow. Green today means using less power and being friendly to the environment. One of the most compelling arguments for moving to cloud computing is because it’s more environmentally friendly as well as easier to manage. It also means distributing computing power everywhere like a grid, and hosting applications online instead of installing them on your local machine.

Recently Google announced that they are releasing an operating system that is not much more than a window to the Internet. Anyone see where this is going yet? If you think about my example of how the cell phone enabled me to make toast in the morning without getting out of bed, what happens when the Internet meets cloud computing and both applications and processes are so intertwined that you can’t tell where one stops and the other begins? Applications? Process? Don’t know, don’t care. I have work to do so stop bothering me with such trivial things.

I don’t have to install applications anymore. I simply have to cache the data locally incase by odd occurrence that I cannot connect to the Internet. Even the green screen terminals of the 1960’s had this concept. They “buffered” the data in an 8 kilobyte local microchip in case the connection to the mainframe was severed momentarily. Well, the numbers certainly got bigger, but the concept is coming full circle. Want a word processor or spreadsheet? Try Google Docs. Want an image editor? Yep, you can do this online too. Why keep it local? Local storage is a single point of failure that packrats like me who never delete anything cannot afford to risk. Even this blog post is auto-saved out to the cloud somewhere and I don’t have to worry losing anything even if this plane I’m sitting on crashes. Google’s applications automatically store a temporary copy locally until the data is sent to the cloud for permanent storage. I bet some of you didn’t even know it works this way. That’s the point. It’s fault tolerant, it’s green, it’s empowering, and it’s transparent. At least to me, this sounds like a trend that is here to stay.

Person, system, or process? They are all process, out in the cloud. It won’t be long and I’ll be having a conversation with your virtual presence or avatar while you are out of the office. I’m hoping people will finally realize that sending spreadsheets over email is not process management – it’s completely wasteful. While we are at it, let’s get rid of email too and think of something more efficient because I’m tired of sifting through 200+ emails per day trying to follow a conversation. And this way I can ensure I never get another spreadsheet emailed to me. Please link it, don’t send it. This way I can actually find the correct version when I need it instead of sifting through 5 or 6 outdated versions.

Getting to the point

In case you don’t see where I’m going at this point with this post, let me spell it out very clear and concise. Show people they can things done without worrying about the technology that powers it and you will start to see things getting done. Take the complexity away from the worker and give them what they need, when they need it. Make it convenient and available wherever, whenever. Don’t bog them down with the complexity of technology. Likewise, don’t bog down the workforce of your organization with the complexity of business processes. They don’t need to know how it works, just that it does. Make it simple, seamless, and bullet-proof reliable. Don’t make them worry about whether or not the data is accurate and up to date.

So the trend I see for the future of BPM is that a new wave of process management will emerge that is not bogged down by legacy fears and horror stories of integration challenges. Applications that you install so support the business process will become merely windows to the data produced by the instances of processes. Synchronizing data to the local machine in a “buffer” will become a standard feature for all BPM systems, and local data storage will likely become the backup copy rather than permanent storage. The permanent storage is somewhere on the cloud but it will be nearly impossible to determine the precise physical location where it exists. In many ways, the technology world as we’ve known it for the past decade is taking a full reversal and going back to the concepts of the 1960’s. But Moore’s law of computing power will actually accelerate. And along with this, I believe BPM will become hundreds of times more effective when practiced religiously throughout the organization. Before this can happen I think that BPMN will have to evolve to deal with more complex event processing needs because the complexity will certainly be there, even if most of us no longer have to know the complexity exists.

You might have noticed that I haven’t written any posts in a while. The reason is because I had an epiphany. A light bulb turned on in my head (and yes, it was a compact florescent, because I’m on the green trend too). It was such an crazy thought that I started to wonder if my passion for BPM is going somewhere. Then just at my most critical moment, a dear friend asks me if I’m still doing “that process stuff”. I stand by my words. Yes, I do BPM. And I do it because there is more of a need for it now in the cloud computing age than ever before. Those of us who specialize in process management need to realize that we have invented a profession. We are not IT specialists, business analysts, efficiency experts, or project managers. We are a bit of all of these things in one. This is how I got the job title Process Expert, and I’m looking forward to a time in the not-so-distant future that this will be a job title I can find on the job search sites on a regular basis.

I see that this post is getting rather long so it’s time to wrap it up and open it for discussion. I’m looking forward to getting comments from my readers so that I can write more on this topic. But for now It’s time for me to do a video conference with my 11 month old child from 37,000 feet up in the air (camera + laptop + internet + airborne internet was a wonderful marriage of gadgets). I wonder what she’ll be doing when she’s my age? One thing is for sure, she’ll be online. Someone will probably have found a way to remove the thunderous background jet noise from the airborne video call. But I doubt by that time the FAA (Federal Aviation Administration for those of you outside the USA) will allow her to stay online during takeoff and landing. Until then, there is a lesson to be learned. Never forget to model the exceptional conditions into your processes, no matter how reliable the underlying technology becomes.

I’m using this class as a test run of the book content. Also there will be some hands-on exercises and real-world use cases to analyze. So if you happen to in the area during October, I’d love to have you in my class.

I just booked my trip to Singapore. What a trip this will be. I’ve been to Australia but this one is a much longer flying time. I will be in the air for 14 1/2 hours from Atlanta, GA USA to Tokyo. Then there’s another 7 1/2 hours to Singapore. Total flying time is 22 hours. I think the hardest part will be spending more than 24 hours in daylight. This will definitely throw off my sleep schedule. I suppose I’m whining because many of you hardcore travelers out there have probably done much worse than this. Besides the flight, I’m really looking forward to this trip. I always enjoy seeing other parts of the world on somebody else’s bill.

Many thanks to my colleague Arnaud Blandin in Singapore for organizing this event.

Included in this tour: Atlanta, Chicago, Vancouver, and then wrapping it up in San Francisco. The audience was fairly broad, including project managers, IT engineers/architects, business analysts, and even a few executives. There are several things I found in common from the questions at each of these sessions.

1. The IT trends are always changing, so why should I start now with BPM and SOA?

That’s a great point. Interestingly I get this question every time I present this topic. Yes, technology trends are always changing. But this is something different. Previously the right elements were not all in place to fully take advantage of the BPM paradigm. Today many BPM software vendors are creating SOA based products, and SOA software vendors are creating “orchestration” products which are very similar to BPM concepts. Basically out-of-the-box you get a whole lot of BPM and SOA from one product. Previously (2002 – 2005) this would have required several layers of middleware and custom connectors to achieve. So the best way to say this is that now BPM and SOA are consumable by an organization. It’s not quite to the point yet where the systems are such a commodity that it’s like a word processor. But even the common word processor is starting to have some collaboration capability built in. So where do you think this is going? In my opinion I think we are past the point of experimenting with BPM and SOA. It’s now become common practice in many organizations and has enough financial backing due to the customer demand that it’s here to stay.

While in my Vancouver session one of my attendees asked this same question in a way that I found interesting. Their organization had worked with various case tools and workflow automation tools over the years and they said that they have seen many “code generation” tools that have made the same claims as BPM promises to deliver but they never did. So it seems that the motivation for the skepticism is based on slick sales people offering that classic “80% out of the box solution” that you can “fire your IT people and do it yourself”. Well, you’ll never hear me telling you to fire your IT people. You might need to educate them on process modeling, but you still need them to create good SOA components for you.

The BPM vendors of the early decade (2002 – 2005) are partially the cause of the skepticism of the BPM potential. It used to be a hot buzzword back in 2003 but it went away for a while before coming back strong in 2008. Often a vendor would try to sell BPM to an organization. Who would have known that you have to actually do BPM, and you can’t actually buy it? Well, times have changed and it’s time to take a second look at what’s available.

Another reason why I see it’s time to get into BPM and SOA is that organizations tend to look at ways of being more efficient in times where cash is tight. In times of plenty, we just hire more warm bodies to throw at a problem. Later we can just fire them, right? Or on the other hand, we can learn to manage our processes and systems in a more efficient way so that we don’t have the huge downfall after times of rapid growth. After a period of recession there will be a period of growth. After you’ve already grown out of control it’s hard to get a handle on the problem. So maybe you can get started now before you miss this window of opportunity.

2. What are the most common problems I see in implementing BPM?

This one is easy. There is not a lot of BPMN process modeling skills in most organizations. The people that I find who know how to model a process well are usually consultants such as myself. A basic BPMN training course is a good start. But after the basics of BPMN are learned additional coaching/consulting is likely needed to do a full scale project. Process modeling for the purposes of automation/execution is much different then modeling the ‘as-is’ process. Different patterns are used, and sometimes these patterns are vendor specific for the BPM automation system. So it’s one thing to model your ‘as-is’ process. Learning how to make it actually work is another skill that you must learn by practice.

So the short answer to the question of what are implementation challenges is that there is no widely practiced methodology that works well with both SOA and BPM. Waterfall? Forget it. Scrum, Agile? Sort of, but not quite. Test driven? Mmm.. partially. PMF includes the best of all the above, and is totally oriented towards the business objectives and process flow. On one hand you have methodologies that are totally focused on data models and on the other you have high-level flowcharts. In the middle there is really nothing to bridge the gap.

For this reason I have created the Process Modeling Framework (PMF). PMF was briefly introduced in my book The Microguide to Process Modeling in BPMN but we did it no justice. The book’s title says “microguide” so please don’t’ be disappointed if the concept is a bit thin in this book. The intention of the book is to give you a quick start on a consumable, non-technical format. But for those of you who want to learn more I’m in the process of writing the next book that will cover PMF in great detail.

3. What are the most common challenges I see in implementing SOA?

Most of the time I see engineers creating data models as the first step to a system design. This I see as the greatest roadblock. What you have to understand is that data would not exist if there was no process first. What is the point of having data? Do you just create data as the objective of the process? Or is there a reason for doing so? In other words, stop modeling data and start modeling processes. The process will dictate what the data requirements are, and not the other way around.

I realize this is nearly treason for some of you on the SOA side of the fence. To the more technically minded the data is everything. And I totally agree with you that data is important. But let’s not get into a “chicken or the egg” debate. I you already have data, manage it, but in a way that supports the business objective. Don’t start going out of your way to create an SOA infrastructure when you don’t even have a solid process model to back your assumptions. Otherwise you might end up making the situation worse by creating an architecture that restricts the organization’s capability to grow.

4. How do I handle developing many services and many subprocesses in parallel?

For the most part you need to limit the scope of diagrams to a single business objective. Stop creating flow charts that wrap around the room. Stop modeling everything in terms of swimlanes. There is more going on here than one person passing a task to another person. Any process in a large organization likely interescts with other processes. So maybe what you should be drawing is how the processes interect with each other, and not how people interact. People are part of the organization, and the organization is made up of processes. This all relevant when determining what is the scope of a diagram. Once you start mixing objectives from processes that overlap/intersect/conflict with each other, then there is no way you can have multiple people work on it. It’s like saying that you are going to take 10 race car drivers, give one the steeing wheel, another to the brakes, another to the acellerator, etc. It’s just not going to work this way. Every driver needs to have their own car. So stop trying to model everything as if there is only one car with many lanes (swimlanes). Instead there are many cars, with many lanes, and not all of them flow in the same direction.

5. How much of the BPM/SOA methodology should I teach to the business stakeholders?

At first, as little as possible while still allowing yourself to do your work. Only explain as much as what is necessary to get the information you need from the stakeholders. Otherwise you’ll get into nearly religious debates about methodologies that evolved from the 1970’s. Gradually as you show results, the organization will adopt BPM/SOA because of the obvious gains in productivity, visibility accountability, and agility. So instead of upsetting the entire organization overnight, start slowly and show results, not radical concepts. Results in process modeling come from showing how to improve the organization. Results in SOA come from showing how quickly you can create a composite system from several existing components.

Today we have accelerated expectations yet we still use methodologies based on a time when computers were the size of refrigerators and access to a computer was only for those lucky enough to have one in their office. Yes, you are trying to overcome this problem. But you are fighting an uphill battle with “traditional wisdom”. Traditional wisdom doesn’t have a 3 GHz quad core CPU with 8 GB of RAM, connected to the internet at 10 gigabits per second. But I do, and you probably do too. Computers are just as much a part of the organization as its people are. 90% of corporate computer systems are sitting idle 90% of the time because we still haven’t figured out to efficiently use them yet as part of our business process. We’ve got way more computing power than we know what to do with, yet we still pass around pieces of paper or spreadsheets over email.

Let me ask you this: When is the last time you emailed a spreadsheet to a group of people and collected responses? Sorry, but this is not a process. It’s chaos in the making. Yet a majority of the organizations I work with use this technique of information mis-management as common practice. They know there is a better way, but at the same time they resist changing. After all, if it isn’t broke, why try to fix it, right? I’ve got a reason of you. It’s called financial meltdown – just incase you haven’t been watching the news lately.

The older methodologies never took this phenomenon of high speed computing into consideration because it didn’t exist yet. The world has changed dramatically in the past decade. Now I can connect to people via my “video phone” in a virtual world and I can book my own travel tickets faster than I can explain to someone where I want to go. Or better yet, I can schedule a trip to Chicago on my calendar and my saved preferences will automatically book my flight for me, and all I have to do is confirm it. This is an example of process oriented, as opposed to the way it’s always been done.

You have to consider that you cannot change the mindset of “the machine” of your large organization overnight. Some very respected people work in this same organization as you have built their reputation on these older methodologies. One tiny voice shouting BPM and SOA will not be heard over a hundred entrenched opinions based on traditions. By going on a crusade to change everything, all you are doing is shaking the tree of traditional wisdom, and whatever falls out of this tree is likely to come down hit you hard on the head. Many of these very respected people still call a travel agent over the phone and use a paper ticket at the airport. On the other hand, many of them ride around in a corporate jet and haven’t seen an airport security line since 2000 when it was still okay to meet your friends as they walk off the plane at the gate.

If you are lucky enough to have a CIO or CEO that mandates the organization will have a BPM/SOA center of excellence team (CoPE team) then you can go ahead and start a formal training program. But still, the details will be missed by most people in the organization. People tend to do 5 to 7 activities in their processes. Outside the scope of what they do on a day to day basis is hard to grasp. The bigger picture of a formal BPM/SOA practice is likely to be well received to a C-level executive, a VP, a director, etc. But anyone lower than this and it’s likely they’ll need a Business Analyst or IT architecture background to understand where you are going with these concepts.

But the good news is that over time as the organization transforms its self, every one else will eventually understand by seeing examples of good practices in action. I’ve seen even the people that have the hardest time changing eventually learn the process oriented approach. It’s contagous. And once you go process oriented, you’ll never go back.

The simple answer to which loop method to use is to express yourself. The specification leaves room for subtle differences, that when combined with some experience in process modeling and some modeling style, BPMN can be very expressive. In some cases there is definitely an incorrect choice for the process at hand, but somehow there never seems to be an absolutely correct one. So here is my style guide on how to pick the best expression for activities that loop.

Style 1: Upstream Loop-back Flow

Example:

Upstream Looping Process Flow

Style expression: Use when you want to do something over again (redo), but not when you want to repeat activities and preserve the previous results. If you want to preserve the previous results I would recommend using the looping subprocess instead.

Upstream looping flow with timer

Upstream flow is also very useful in high-level diagrams where you don’t wish to show all of the intricate details of process flow. I’ve found that if you are using a high-level view, it’s best to stick with subprocess shapes rather than using tasks. A task implies an atomic single activity where something is done. The subprocess expresses that there are several steps to complete a task, which can include the points of communication (message events, signals, or anything else required to complete the activity). I’m not saying that the subprocess is required here. But it might be a better option because it’s more likely to accurately depict the true nature of the process.

Upstream looping flow across participant lanes

Advantages:

• Simple to model and most people with no BPMN training can easily understand it.
• Great for showing high-level sequence flows across multiple participants (roles, systems, etc).
• Easily shows the loopback condition as part of the gateway, without using annotations.
• Allows you you to do a loop across multiple swimlanes, but be cautioned that this is not always a good idea (see my other post about swimlanes).

Disadvantages:

• Doesn’t always capture the detail of a process flow (see my other post about swimlanes) and limits ability to show explicit message interaction between participants.
• Not very easy to introduce exception conditions into the flow without adding a lot of extra shapes.
• Implicit way of showing multiple iterations of an activity might lead to inaccurate interpretation.

Style 2: Looping subprocess

Looping Subprocess

Style Expression: Use when you need to repeat one or more activities while preserving the data/results of the activities for future reference. Also very useful when you need to deal with multiple exceptional conditions that might interrupt the loop iterations.

In the following example I am using the simple “check mail” process. Every hour I check the mail, and if there is something in the mailbox I reply to the mail. When I’m done replying to the new mail the cycle repeats. Immediately after replying to mail I go back to check for new mail. This starts to optimize the process because if new mail arrives while I’m replying to other mail, as soon as I’m finished replying there is no wait state.

Looping Subprocess with Timer

Dealing with exceptional conditions in a subprocess is much easier to do than with straight-through linear flow. This is because you can place one or more exception handlers on the subprocess border. For example, I can catch an error condition if it occurs, but I’m not really expecting this to happen all the time. Or I can cause an alternate flow to occur when the activities don’t complete within a specified time period. Or, I can watch for all of the above conditions and deal with them accordingly. In contrast, the linear flowchart style diagram gets extremely complicated and difficult to read without the capability to “dynamically” create exception flow.

In a linear flowchart style, many conditional gateways or flows are used to check for current state. In the BPMN style, intermediate handler events are used on the subprocess border instead. In the following diagram I add a few layers that handle the exceptional conditions that could potentially occur. I’m not going to explicitly check for the errors all the time. It’s more of a passive monitoring for things that might occur.

Complex Nested Looping Subprocess with Error Handling

First I can receive the intermediate message labeled “cancel”. This is here because instead of watching for mail all day someone might give me a more meaningful way to spend my time. So maybe my boss sends me a message to go out for coffee or something. When this occurs I terminate this process and go do something else.

Next there is an intermediate event that catches any errors that might occur in my process. This error handler will catch anything that might occur on the check mail or reply to mail. Note that this is not necessarily a technical step geared toward the geek crowd. This gives me a “catch all” way of dealing with exceptional conditions, regardless of what the condition is. Most likely I’ll have to interrupt the normal pattern and do some improvising to get the process back on track. For example, one day someone plays mailbox baseball. For those of you who don’t know what this means, it’s when someone takes a baseball bat and smashed the box where I receive my mail. So I can’t receive mail. Sounds like an exceptional condition, right? I’m not planning for this specific event to happen, but anything can happen. So I don’t want to stop my process dead in its tracks just because I don’t have a way to deal with anything that might come up. The error handler is most likely a generic manual step.

Note the outer looping subprocess. This ensures that if an error condition occurs that I can easily jump back directly into a new instance of my inner loop, which is the main normal flow.

An finally, at 5 PM I’m going home, so I terminate the process. Because I’m a “clock puncher” type of person (How I wish this were true instead of me writing this post past midnight for three days now) it doesn’t matter what I’m doing. When the whistle blows at 5 PM (the timer event) I’m going home. Forget about replying, forget about checking for new mail.

So that’s the process. Now what would this same process look like if I didn’t have a subprocess and used upstream flow instead? Here is the equivalent diagram without using subprocesses, or should I say as close as you can get to the equivalent behavior.

Complex Upstream Flow with Error Handling

To achieve 100% of the same process flow behavior using upstream flow it’s incredibly difficult. In fact, this small example took several tries for me to get it right, and I’m still not totally convinced its the same. Feel free to comment and tell me how I did. The point is, when you start handling errors (as you should be doing in all processes anyway) in the upstream flow style, the diagram gets very complex to create, and even more difficult to read. Now just imagine this upstream flow example using swimlanes across multiple participants. You will probably end up with lines going all over the place and it’s a real eye sore to look at.

Just to put some context around this, all we are doing here is checking for mail and it’s already getting hard to read. Imagine if we were to model a real process! So for detailed process design including proper exception handling, the upstream flow doesn’t work so well.

Looping Subprocess Advantages:

• Explicit loop control
• Focus on a single process participant, which frequently yields a more accurate/detailed process diagram (see my other posting about swimlanes).
• Ability to easily deal with multiple exceptional conditions without complicating the normal flow.

Looping Subprocess Disadvantages:

• Slightly more difficult to understand for people that have not been trained on BPMN syntax. Note however that this can easily be explained to the BPMN newcomer in about two minutes, or you can just as easily add a text annotation to explain the loop.
• No way to determine what the loop condition is without annoations.

Style 3: Multiple Instance

Multiple Instance Subprocess

The multiple instance has two use cases. This sometimes happens in the BPMN specification. It’s not always clear in the diagram which use case is actually occuring. I suspect this will be cleared up in the 2.0 or later specification (or at least I hope so). Because of this current issue in the BPMN specification I recommend using a text annotation externally explain the sequence behavior. Note that the external text annotation is also a good practice for the looping subprocess as well.

Use case 1: Serial Execution

Use when you want to repeat some activities for a fixed number of times. There is no condition on breaking the loop. For example, “do this five times”. Each instance of the activity is performed in an iteration, meaning that each after the first iteration the subsequent iterations will not occur until the previous has completed. Essentially this is the same as the looping subprocess, except that there is no need to specify a condition that will cause the loop to break it’s loop. Only a number of iterations is provided.

Multiple Instance Serial Execution Mode

The same diagram can be expressed using only task shapes, but requires many more shapes to accomplish the same expression.

Multi Instance Serial Task Equivalent

Style expression for Serial Execution: Use in a “for each” loop when the number of iterations is known prior to execution. Not applicable to a “do while” or “do until”. Don’t use in any loop where a condition or normal flow might exist that would cause the loop to break before the number of iterations is complete. The usage of exception handlers is acceptable (and most often encouraged).

An important consideration for using this expression is whether or not you know in advance the number of iterations you will perform (If I am repeating myself here it must be because this is important). Otherwise you should use the looping subprocess instead. Conditions such as “do until the color is sufficient” or “keep checking the temperature every 5 minutes until the inner temperature is 200 degrees” is not a candidate for a multiple instance loop. Instead, a good example for a multiple instance loop is “Get 100 people to sign the petition”. Another example could be “take exactly 10 steps forward before turning left” if each step is an iteration of the looping activity. But then again, if you want to get very technical here one could argue that each step has a dependency on the other leg (left leg/right leg) to perform its duty, so this would disqualify the multiple instance for each step. Instead I could either hop 10 times or take 5 pairs of steps forward.

Use case 2: Parallel Execution

The basic use case is the similar to the serial execution use case, except that instead of iterations, X number of instances will instantly be launched, all running in parallel. Like the serial execution, there is no need to break the loop. In fact, it’s not really a loop at all. You are simply showing a different form of parallel flow in which you are launching X number of identical activities simultaneously.

Multiple Instance Parallel Mode

The same diagram using parallel gateway looks like this

Multiple instance parallel gateway equivalent

Style expression for parallel Execution: Another way to word this notation is “do something X number of times, right now, and don’t wait for anyone else to complete”. Use when you want to perform the same activity a fixed number of times, and each activity can be performed independent of the other with no dependencies of any other instances of the same activity to complete. Never use the parallel execution in situations where one activity might depend on another completing. Never use this expression when the activities in each instance might contain a different sub-sequence.

In the example above referring to “Get 100 people to sign the petition” we could apply the multiple instance. However, there is a subtle difference that occurs. We would have to get 100 people in action simultaneously. So in the serial execution example we are likely a single person going door to door for signatures. In the case of the parallel, it’s more likely that we blasted 100 emails out to people and we are awaiting 100 responses. So it might not be possible to actually receive 100 positive responses out of 100 emails sent unless the response is unanimous. Instead you might want to express that out of 1000 sent emails we want to count at least 100 positive responses, which causes a break of some sort. Normally I would suggest that you use the looping subprocess for this example. But in this case you have a unique problem that is very difficult to resolve with a normal loop; you have to correlate each response with the preceding request that is unique for each instance.

Looping Subprocess Emulating Parallel Multi-instance

It’s important to note that the serial execution and parallel execution appear identically in BPMN. According to the specification the execution mode is determined by the “attributes”. Now you see, this is what happens when a perfectly good modeling notation meddles in process automation and execution. I mentioned earlier that I hope this will be fixed in an upcoming version of BPMN. For example, vertical lines for parallel execution and horizontal lines for serial execution would have been nice. But the specification simply stops short of offering a graphical solution and suggests that execution engines use the “attributes” to specify behavior. This does nothing for creating a diagram. So I ask the folks at OMG to please fix this so that people who want to just create a diagram can have an easy way of doing so without having to read “attributes”. Now that I’ve vented my frustration on this subject, might I suggest for the rest of us to use a text annotation external to the subprocess to resolve the ambiguity (as in the “multiple instance parallel mode” example).

Multiple Instance (parallel) – Advantages:

• No need to specify a loop condition based on data. You can clearly tell that the activities will occur a fixed number of times, and this number of times is known before the loop begins.
• Has the ability to dynamically generate a bunch of parallel flows without having to draw all of them explicitly (parallel execution).

Multiple Instance (parallel) – Disadvantages:

• You cannot tell if this is a parallel or serial execution without any sort of annotations externally.
• You cannot tell exactly how many parallel instance or serial iterations are actually being performed. Again, to resolve this problem you should use external annotations.
• Hardest to understand, least documentation and examples in the BPMN 1.2 specification.