Key Learnings - Living Business Driven SOA

There are some significant challeges in getting business buy-in on supporting SOA. Back in August my work in obtaining this support was recorded in the following interview with Jennifer Zaino of bITa Planet.

The opening statements of the interview are as follows:

"Some of the Words of advice from the frontlines: One IT leader discusses the challenges of getting business buy-in for service-orientated architecture (SOA)."

Please read the complete list of this conversation:

http://www.bitaplanet.com/alignment/article.php/3694846

I would love to hear you views on this matter.
surekha -

Key Learnings - Profile of a Business Service

My experience has led to the creation of the following service based postulates. These were applied by me on major retail SOA projects.

You can find these and other laws posted on the SOA Blueprint site. http://www.soablueprint.com/soa_laws

1. A service should expose a single business function or business process in it’s' entirety
2. A service should not make assumptions of the validity of the request and should validate the request for accuracy including checking that the requestor is authorized properly prior to honoring the call
3. A service should delegate to the right enterprise resources to completethe business function
4. A service should be responsible for notifying and dispatching the right requests/ information to all related services and/or applications on behalf of the requestor
5. A service should be responsible for any call chaining, sequencing and aggregating of the requests and the intermediate responses in order to completely process the consumer’s original call
6. A service should provide business information as a canonical documentthat is consumable by both human actors and system actors
7. A service should return a response that has the business context built in along with the response to the question asked by the requestor
8. A service should rely on enterprise worthy metadata information including the application of any algorithms, enterprise-wide business rules and strategies in order to process the requestors' call

Your feedback is appreciated.

- surekha -

Key Learnings - Using EDA to implement the core SOA principle of "loose-coupling"!!!

A lot has been said about how SOA and EDA are unique "architecture styles". It seems like only one or the other architectural principle is considered in proposing architectural solutions. However, there is a distinct benefit to using both paradigms in unison in solving business problems!!


Business events are the "core concept" that drive any EDA implementation. On the other hand, decoupling business applications and the business functions/ business processes embedded in these business applications is the core theme of SOA. SOA implementations rely on the use of standards based web services technology stack and that of canonical business documents (XML) .


However, it is my contention that an enterprise that does not invest in the web services or ESB technology can still leverage EDA style business events to implement loosely coupled business services provided it makes an effort to analyze its' business events and creates canonical representaions of these key business events. This could mean defining business events that encapsulate a business concept that have an associated business concept state indicator or a business action indicator. Further, these business events can be used to trigger the business services or business action listeners that are key event handlers. These event handlers in turn invoke a business function by executing an application API.


It must be noted that the terms event producers and event consumers or publishers/ subcribers are being used loosely to denote the initiator of the event and the owner of the business behavior that "knows" how to deal with the event. In a SOA realm this would be the service consumer and the service provider respectively.


The key to this model in leveraging SOA is the use of self-describing canonical business events that are subscribed to by independant event listeners. These event listeners and/or event handlers that are delegated to by these event listerners help insulate the event producers and event consumers from the complexity of knowing how to interpret the events. Here the event producers/ publishers and event consumers/ subscribers are decoupled from one another via the use of canonical business events as well as messaging technologies.


Either of the two layers i.e. the event publisher or event subscribers can be altered long as the contract is adhered to in terms of the canonical business events. Also, messaging technology oriented configuration consoles allow the definition of the event publisher/ subsribers to be connected via metadata as opposed to hard-wiring these in code. Event handlers act as event adapters in that these could translate the event and invoke the required API call to deal with the event. The event handlers act as a business facade that hide the workings of the business application and allows the business application to change without affecting the event producers.


An example of a business event could be "approveCustomerLoan" that is subscribed to by the CRM application, credit application and the loans origination applications. All three can communicate with each other using events such as "getCreditRating", "getCustomerprofitability" etc.


To recap, if loose-coupling is a core SOA principle that promotes business agility then the use of event handlers invoked using messaging technology and the use of canonical events can be used to deliver this goal. This option also allows the service based techology to be used to turn the event handlers into web services. This model allows an enterprise to first articulate it's business architecture while pushing off iventment in the technology shock to a future phase without sacrificing business agility and/or offering novel business capabilities.

Selling SOA to business

One of the most important milestone in SOA adoption is getting the business to buy in. However, getting business buy in requires an approach that is sometimes not obvious. I had an interesting conversation with Jon Brodkin of Network World. Please check it out at



http://www.networkworld.com/supp/2007/ndc6/102207-avis-soa-case-study.html



and let me know what you think





Ashok Kumar

Challenges in implementing BAM - drawing a relationship between business processes and operational system exceptions

I have a question to pose to those of you that have end to end business processes that are composed of multiple legacy business applications/ business services that may be deployed on disparate platforms and technologies. I am curious about what architecture patterns have been deployed by you to report on the operational state of business processes. Also, I would love to hear about how you provide business executives visibility to the business process execution related issues specifically to those metrics that impact business SLAs and KPIs governing the core value streams.


Product vendors have touted Business Activity Monitoring (BAM) to be the magic bullet that allows the creation of intuitive executive dashboards. Most BAM tools that monitor KPIs expect to subscribe to business events/ business alerts being generated from the business process. There in lies the challenge. The reality of where to gather the relevant data for creating the executive dashboard is what complicates most BAM implementations!!! The going in assumption is that these business applications are not really conducive to refactoring and may be mission critical systems that cannot be re-architected to follow an Event Driven Architecture EDA model that expresses it's internal state via a series of events.


In my experience most of the legacy business applications are created with both the business logic and the process logic embedded in them. In addition, most of these business applications do not really communicate or interact using business events/ business alerts. Furthermore, business applications that encapsulate parts of a business process may leverage messaging to communicate states to the next step in the business process. These business applications may have custom message listeners and handlers that transform the message and execute pertinent APIs that represent the next activity of the business process. Given this, I am wondering how folks are dealing with abstracting and discerning between the business exceptions, system exceptions and business process exceptions. All these categories of exceptions and the internal state of the business information may need to be monitored in order to aggregate information and accurately report the health of the business process on an executive dashboard.


We have resorted to the following architecture pattern. The various form of exceptions and business entity information (whether they be business or system or process related) are extracted from the logs, business application operational tables or system monitoring data structures. These data points or the heart beats across the process are then persisted into a business process state-monitoring data-mart. The extraction process has transformation and reconciliation logic to correlate the exceptions with the state of the business information. The extraction process also has the logic build to deal with the after effects of data discrepancies and business information inaccuracies caused by communications disruptions and other system exceptions. The baseline threshold latency between the steps in the process is used to then report back on how the system alerts or data integrity issues may have led to missing the SLA for a particular process. It must be noted that the ETL tool being used is an industry strength ETL tool and not some data and log scrapping scripts.


Finally , vendor BAM tools could be used but the key challenge lies in the ability to reconcile between which system issue or data integrity issues caused the business alert. I would love to hear about other common patterns that are being used to address this challenge of gathering data to compute BAM metrics!!

Best Practices: Establishing Enterprise Architecture Teams

Recently I have had a number of conversation with some key decision makers of various IT organizations and the common questions I got asked was to explain the process of establishing enterprise architecture teams.


This first requires the understanding the various IT organization patterns (models) as well as the evolution (maturity*) of the EA teams. My initial thoughts on this are available here.

Based on the feedback / comments from the community, I plan to take it to the next level of document the entry and exit criteria as well as the deliverable by the EA teams though each of phases.










* As there are already a lot of maturity models out there have decided to stay away from that term

Key Learnings: Blueprinting Information Architecture is key successful adoption of SOA

Following is the high-level approach for Blueprinting the SOA Framework for the enterprise.

Understand the business process and identify the areas for improvement
„Understand overall business objectives
„Understand specific business pain points
„Understand priorities

Develop the Enterprise Information Models to identify the Information services needed to support business processes

Develop the SOA Framework foundation to identify the infrastructure needed to support the business process needs.





Currently there are a lot of content available on the topic of best practices, key learning, reference architectures, ROI, etc. on the topics of Business Processes and Infrastructure. However there does not seem to be a lot of publications around Blueprinting the Information Architecture for SOA.

Basically there are three areas of focus for Blueprinting the Enterprise Information Architecture and are as follows:

• Reference data: Examples: salutation, job title, contact type, partner type, region, state/province, country, currency, language/locale, and industry
• Master data: Examples: customers, contacts, products and orders
• Analytic data: Examples: marketing data mart, forecasting, customer Life cycle value and customer satisfaction

Categorizing the key enterprise information needs into these three areas enables IT to have an very fruitful discussion with the business. However, the first task is to get business to fund and participate in this Blueprinting effort. Following is a very high level three step approach to Blueprinting the Information Architecture.

1. Scope and Planning (Establish: Budget and timeline, Team, Deliverable)
2. Information Architecture Assessment (Current state: high-level business process, data flow, data models - reference, master and analytics)
3. Information Architecture Blueprinting (Enterprise Information Model, Data Governance, Recommendations and Actionable Road map)

Why is this necessary?

Most business processes cross business and application silos. The current best practice is to migrate/move the data across these silos which results in redundant data, systems and places a lot of burden on business operations teams to clean it manually. In addition, multiple business silos tend to implement the same business process in different applications.

Adopting SOA enables enterprises to reduce the overall cost by enabling business to implement their business processes by consolidating/reusing a single infrastructure/application. This is possible only by exposing the appropriate information services to support the business processes.

This can be achieved only by truly understanding the Enterprise Information Model. Please click here for a three slide presentation on this.

Overhauled the SOA Blueprint site

The previous SOA Blueprint site was not very well organized and nor was it easy to maintain. As I was browsing through the Yahoo! Web Hosting Console I came across the "Site Solution" tool which makes it easier to manage and publish the content. Hope you like the new layout and please do feel free to send me your feedback, comments and/or any questions related to SOA.

Following are the categories of content currently posted on the site.

• SOA Practitioners Guide
• SOA Case Studies - reference to case studies published by forums, analysts and vendors
• SOA Organization and Governance
• Enterprise Architecture Frameworks - links to the most common EA frameworks
• Enterprise Architecture Resources - Resource page with information relevant to the Enterprise Architecture teams
• SOA Reference Architecture - Reference to a list of various published SOA Reference Architectures
• SOA Maturity Models - There are multiple maturity model and this page attempts to list all the major ones along with some comments, if possible.

Please feel free to also send relevant links or documents that would be useful for adopting SOA.

Enterprise Architecture Survey Results

In the SOA Consortium: Enterprise Architecture 2010 Webcast both Ashok and I refer to the Enterprise Architecture survey which was conducted earlier this year. This was an unscientific and informal survey conductedand for those interested the survey results are available here.

SOA Consortium: Enterprise Architecture 2010 Webcast

One of the working groups in the SOA Consortium’s community of practice is the “EA2010” group. This group of seasoned enterprise architects from industry, government, systems integrators and vendors, has been actively discussing and defining the next generation role of enterprise architecture. Specifically, what enterprise architecture looks like – organization, practices and people – in a business-driven, service-oriented world.

We are excited to share the first release of “Enterprise Architecture 2010 – Where Enterprise Architecture Means Business” in a slide deck now available for download.

Prior to the release of this important deliverable, Brenda Michelson, SOA Consortium Program Director, sat down with the EA2010 working group leaders, Ashok Kumar and Yogish Pai, to discuss their working group’s motivation, findings and next steps. Their conversation is available as an on-demand webinar.

In this 31-minute webinar, Ashok and Yogish cover a wide range of enterprise architecture concerns, including catalyzing business change, gaining business-smarts, shifting focus to business architecture, managing enterprise architecture, participating in strategy and delivery, and winning enterprise constituents.

Click here for more details on Enterprise Architecture 2010 Webcast.

Semantics and its' role in Business Services

Role and importance of semantics in the context of services and SOA:

Semantics refer to interpretation of information and not the literal definition of information/ data. Applying semantics to information turns it into “knowledge”. Semantics is the act of applying references and drawing conclusions given a set of more scientific informational constructs. Typically semantics are derived using the context in which information is presented. Transposition on the other hand allows applies the rule of inference where in one can draw conclusions on the implication of truth based on some set of facts.


By deduction semantics and semantic transposition deals with alternate inferences being drawn from one or more data and informational constructs using business rules. Inferences of this nature are not completely predictable as the rules used to arrive at the interpretation may or may not be quantifiable and/or may not be codified into a business rule.


The biggest risk in dealing with semantics is the lack of predictability of the outcome of an interaction, whether manual or automated. The reason is that the rules driving the decision tree or the execution path are not quantifiable.


Despite the complexity of interactions brought about by semantics and the complexity in being able to interpret these to codify and execute alternate behavioral patterns, one can attempt to capture metadata and leverage naming conventions to make the service based interaction more predictable. These guidelines may either be applied within an enterprise or could even have applicability to govern service interactions within an industry vertical.


In the realm of services, semantics drive both the discovery of services and also the execution paths and alternate process logic that is undertaken by the service provider. Naming conventions and expected service behavioral guidelines govern candidate service descriptions. Having documented behavioral implications and constraints associated with these naming conventions make the service “discovery” of a service and the expected interaction with the service provider more predictable from the service consumers perspective. These service naming conventions also act as metadata that can be used to publish information about the service definition during the publication and registry process.


Furthermore, one could make the assumption that there are certain features of a service action that denote the type of expected behavior, notwithstanding the type of business concept or the business function being manipulated by the service invocation. The service business action verb component can be associated with the general description of the type of behavioral patterns a consumer can expect. Given that SOA style services follow design by contract principles the naming and the signature of the service definition or description does not bind or affect the manner in which the service is implemented. This metadata of the verb does not impact communications patterns internally used by the service or even the informational model or informational exchange pathways that might internally be used by the service. These details are left up to the service provider. Having action verb nails expected behavioral semantics (inquiry only vs. alteration of service information model state) and not implementation semantics. Also, the action verb points to expected interaction semantics (one way/ two way/ guaranteed delivery etc.) and not the implementation model used by the service provider.


The key is that having these kinds of action-verb semantics outlined makes the service usage more specific. The service based action/ verb may be used by the consumer to identify the right service to use during the process of discovery. Also, the provider can use the action/verb notation to inform the consumer about the implications of the information being returned.


For example, a service operation called “obtainLoanRate()” with a action/verb embedded as the service invocation metadata may inform the consumer that the information being returned in not legally binding on the service provider. On the other hand, using the “lock” action/verb in the service invocation metadata “obtainLoanRate()” implies that the service provider is locking the rate and that the consumer is also bound legally to non-repudiation of the “rate” being locked. In this case, the semantics of the verb can be used by the consumer context to pick the right service depending on the usage context on the consumer tier. In addition the action/verb pairing may be used by the provider to insure that the right application logic is invoked. For instance, when the service consumer calls with a “get” action/verb invocation context the service provider implementation layer may go through fewer rigors and processing cycles than would be the case when the service consumer calls with a “lock” action/ verb invocation context.

Another example of the metadata driving semantics is the use of business context field by the service provider. A service operation “getCustomerProfile()” that uses the context of “Customer Profitability” or “Customer Credit Rating” could alter the information being returned in the payload of Customer Profiles. The service provider uses the service usage contextual information provided by the consumer to decide what type of information is returned to the consumer. The consumer context allows the service provider to alter the information returned between providing “Customer Profitability Range” or the “Customer Credit Score” in the customer profile.


As has been mentioned semantics can be applied to both the service publication and service discovery process. As part of the service publication process (to the registry) the service provider may use metadata in order to express the need for more precise contextual information to drive the way the service provider interprets the request. The service provider may add this constraint to leverage this information for altering the returned payload. The service consumer may also be able to use this business context metadata provided by the service provider along with the service provider business action-verb name to derive information about expected behavior of the service provider and how the request might be interpreted. For instance, a service operation that has an action-verb called “preview” may inform the service consumer that the information may not be reliable or even persisted beyond the current session while having a service consumer invoke a service operation with an action/verb “reserve” indicates to the consumer that the information returned is persisted beyond the current session. This naming convention is indicative of the service provider confidence factor in the consumption of the information being returned by the service provider.


In addition having some area for capturing the concept of the consumer context may help the service provider apply this knowledge to execute alternate paths of the business use case scenario. To some small extent, semantic transposition can be achieved by incorporating rules that have been gleaned from observing human behavior. If these manual workflows and best practices can be captured then these business rules can be used to execute alternate flows in the service. Consumer context information can then be used to invoke alternate business functionality and/or pathways that are known at compile time but may be loaded at runtime or invoked at run time on an as-needed basis.


The behavior altering ability can help the provider to honor consumer requests with more precision and with greater compliance to the contracted SLAs. This may also enable the service provider to roll out more behavior without altering the information models being exchanged and without altering the service interface. This may be a mechanism to curb the service version sprawl!!


Finally, the business context that is expected by the service provider or the business context information and candidate service name as published by the service provider could be classified as "service usage semantics". The purpose of the service usage semantics is to inform the service a consumer about the possibility of semantic variations in the honoring of service call.


Issues in incorporating Semantics in the realm of services and SOA


Semantics and "Semantic Transposition" may be fairly complicated to achieve in business systems. Most applications/services are typically written to fall cleanly into the “known” process logic pathways with no ability of either interpreting independent unrelated pieces of information. All paths typically have to be known and defined at compile time. In most situations, semantic interpretation may be complicated enough or unique enough to only be undertaken manually by knowledge workers. However, there may be a small subset of cases where the interpretations and inferences that drawn by the knowledge workers may be codified into rules and contextual metadata to help in the execution of alternate flows of a business use case.


Impact of semantics on Web Services, Service Registry and Service Repositories


As the concept of SOA and SOA style services become more standardized service related metadata registries/ repositories such as UDDI hosting web service URLs, service contract definitions and service descriptors defined as WSDL may make provisions to allow a service provider to publish service usage semantics, standard classifications and ontologies. These sets of metadata may become a guide in aiding the service consumers to browse for and find services more intelligently.


Furthermore, having a standardized mechanism to express the semantic usage information may enable tools to automatically bring this information back when service consumers are browsing for potential services. It must be noted that if this information is just some comment or a open ended description field instead of it being a structured metatag with known enumeration of values then browsing and searching for services cannot be automated.


Each industry vertical may have different values that might be populated into the standard structure to help with automating the search. These fields and valid industry vertical enumeration of values could all be published into the UDDI registry to help the consumer in either formulating the request or in informing the consumer about how to process the response.Some sample metatags could include fields such as business area field (Credit/ Sales/ Logistics).One such specification that is addressing Semantics in the realm of services is the Semantic Annotations for WSDL specification. http://www.w3.org/2002/ws/sawsdl/.


Conclusion


Having some stringency bound in using naming convention and in creating the request canonical models allows execution of the right sub-set of the service behavior thus making the service interaction more optimal. The presence of semantic oriented metadata allows the service provider to alter the behavior being executed in response to a service invocation and/or alters the amount and type of information returned to the service consumer.


Although not applicable in all situations, having contextual information and action/verb usage metadata information enables semantic transposition and execution of alternative service provider behavior s without having to resort to the execution of a human workflow and manual intervention. The objective of associating semantics to action verbs or in associating consumer context is not to bypass human intervention always but to use the learning obtained from the knowledge worker and encapsulate it in process automation logic.

Enterprise Architecture: Making yourself recruitable...

A interested blog by James McGovern on "Enterprise Architecture: Making yourself recruitable..." for those enterprise architects looking for their next career opportunity.

Key Learnings: SOA Key Success Factors (BEA-IT 2002-2006)

Based on my conversation with my peers in the industry, there is still a lot of keen desire by IT Leadership teams to understand the key success factors for adopting SOA. Following are the list of key success factors we (IT leadership team) had identified while I was the CTO-IT at BEA Systems.

1. Build the right team
2. Organize for success
3. Build coalition with business partners
4. Maintain Flexibility

The slides of these are available here. I have created a link of all my key learning blogs at my structured blog.

SOA Practitioners Guide Part 4: SOA and ITIL Convergence

The SOA Practitioners have been working on the next set of the Practitioners' Guide with SOA and ITIL Convergence being the first (Part 4) of new set of Guides. Burc Oral has been leading this effort for the Practitioners' and is presentation (SOA Practitioners Guide Part 4: SOA and ITIL Convergence) this at the eGov meeting today (October 1st, 2007).