Issues with SOA Adoption

Here is my attempt to identify some of the reasons for failure to adopt SOA. This time the focus is on not having a holistic SOA enabling infrastructure.

Many large enterprises try to reduce vendor-lock by not having a single provider for their entire SOA development/ deployment stack. This philosophy works great from a risk management perspective. However, this risk management strategy directly competes with the “speed to market” gains promised by SOA.

1. Not having a unified platform that facilitates seamless integration across the service orchestration layer, the application layer, the data layer etc. leads to long system integration and debugging cycles
2. Not having a centralized facility for the end to end management and monitoring of services can cause long outages and hampers the ability to track information/ transactions flowing across the various layers of the service architecture (i.e. service orchestration layer, the application layer/ business logic layer, the data layer etc.)
3. Not having a holistic SOA governance suite that enable discovery of existing service assets at design time and that provides service utilization information at runtime causes service proliferation issues

The following blog by my colleague and co-blogger Yogish prompted me to address this issue as it speaks to Oracle/ BEA integration strategy two big names in the space of SOA infrastructure.

Analysis on Oracle’s BEA integration strategy

In lieu of the one year anniversary of Yogish’ blog I am being optimistic in assuming that this acquisition will lead to the creation of a holistic SOA platform that encompasses service interactions at design time and runtime. I am also hopeful that a single SOA platform will provide seamless integration across various layers of the architecture as in the service orchestration/ mediation layer, the business logic/ application logic layer and the data layer. In addition, my hope is that a stronger player such as Oracle (following the BEA acquisition) would start pushing for “SOA standards” and start holding other SOA players accountable for staying compliant with these (in much the same way that the other vendors would now be putitng more pressure on a stronger SOA contender such as Oracle (following the BEA acquisition) to abide by these same standards. This "peer pressure" will hopefully make interoperability an achievable goal.

In the event Oracle is able to pull together a holistic SOA stack then here are some advantages for its' customers -

1. Having one vendor support the end to end stack enables a customer to find the right support whether it be from the perspective of SOA product suite integration or from the perspective of availability of the right tooling to enable and enterprise to cut down its' service lifecycle timeline (service design, service deployment) and improve its' time to market.
2. Having a player such as Oracle that has traditionally focused on scalability, reliability and end to end monitorability will provide customers with a SOA platform that is robust enough to meet the stringent SLAs at runtime while making service availability more predictable and less of a guess work at runtime
3. Having one vendor provide a holistic service governance suite will allow an enterprise to reuse its enterprise service assets (if authored appropriately) thus enabling the business to compose existing services to offer new capabilities.

In conclusion, I want to state that I do not align with Oracle or any other SOA player but merely want to comment on issues that have hampered SOA adoption and how these might be addressed. I believe that a unified SOA infrastructure platform will be a key capability needed to truly realize the full potential of SOA.

Thanks for listening.
surekha -

Key Best Practices - REST Assured????

The purpose of the blog is to find out if there is a place for REST in the realm of “Business Services”.

First of my definition for REST – it is a "Get", "Put", "Post", "Delete" operations performed on "Resources" that are identified as URIs being transmitted over HTTP(S) as REST.

REST by nature has a very simple service operation set with the complexity all embedded in the Resource URI. The operations that REST allows are NOT business user-friendly and hence do not really belong on the Service Repository that an end user is referencing to discover business services. To accommodate complex business behavior and to compensate for the finite list of RESTful operations the resource identifiers have to be fairly complex.

SOA on the other hand, allows fairly diverse interface definition that resembles as closely to the business syntax as it can get. The canonical model that is shared is standardized and is not as diverse as are the resource URIs of the REST architecture. The canonical models for the most part are XML.

Another key difference lies in the protocol and messaging support for REST. REST is limited to only HTTP/ HTTPS which by nature is stateless and has no standardized protocol level interception model for the more enterprise category behavior such as guaranteed delivery of messages and asynchronous messaging semantics. SOAP and the standardized metadata associated with SOAP envelope on the other hand can be interpreted by the Web Services SOAP stack to enable behaviors such as Reliable Messaging, Transaction Management, Addressing, and Notification with a level of interoperability across-vendor platforms.

Having said that, it is possible to find a home for RESTful architecture and to leverage one of its strong points in terms of a simplified and standardized interface. REST may be a good architecture model or construct to follow in the service implementation layer or at the service adapter layer. This helps to keep the service implementation the same across any type of service interface definition. In the event there is a need for additional transactional integrity then these behaviors could be taken care of in the service facade or in the service mediation layer while keeping the actual implementation in the simplified RESTful architectural format.

If we are looking at SOA service operation being exposed to the world and using a RESTful service implementation layer then we would need to deal with translating canonical models and parameters of SOA style service into URIs. This translation behavior includes extracting the request parameters from the Request canonical model and transforming these to verbs and values in the format of a query string i.e. a URI.

One option is to have the specific URIs needed by the service implementation layer to be translated to the input URIs as specified by the RESTful service implementation layer. Here the service interface layer or the service mediation layer performs the translation of the input parameter into the specific URI and initiates the invocation of the appropriate service implementation layer.

The other option is to have the service interface layer or the service mediation layer translate the input into a generic URI for calling the RESTful implementation layer which in turn does the mapping internally to the required operational version of the URI. This improves the durability of the service interface layer or the service mediation layer. In addition, it provides a mechanism for doing internal operation overloading and polymorphism.

Regardless of which option is chosen, I would think that embedding RESTful services in the implementation layer would be a good option while not exposing painfully abstract behavior to the business user who is used to seeing more of English like business syntax.

Here is a link to another blog of mine on the topic in the context of Service Orientation!

Even though I agree that there is a place for REST as it may simplify (by providing a finite set of operations) there are infrastructural capabilities that do not exist with the REST and without these the business agility piece is going to be hard to deliver. Here are some additional thoughts on the topic that an enterprise architect may have to resolve prior to embarking on a RESTful implementation.

1. Identify which layer deals with the interpretation of the get(Resource) call? Is it the service provider (SP) interface or is it the service implmentation layer?
2. Would the SP be dealing with versioning of the Resource thereby creating a grand facade that does internal resource mapping via a URL redirect style substitution? This would be one way of protecting the service consumer (SC) and to accomplish backward compatibility.
3. Identify how the SC gets the handle or URL to the "right" resource? How does it "know" what to expect from the Resource and how does it "know" to express its' semantic expectations for the SP Resource that is being called upon? Does it do a "Get(SearchByCriteriaResource)" call with semantics embedded in the Request Resource?
4. Is the get(RightResource) another call and somehow the metadata held in the response Resource is expected to have the "right metadata" for each of the provider Resources (that meet the SC search criteria) in order to help the SC choose the right SP? If this is the case, then following this invocation would be a secondary SC to SP interaction to fetch the chosen SP Resource.
5. Identify if there would be a standardized URI with search criteria specification. This might be good to define up-front to enable the support of JINI-like or SaaS like interactions?
6. Identify and define how one would deal with asynchronous interactions? Would it be via the use of " ack/ no-ack post notifications" and "post response Resource" calls? How would this be to handled? The SC would have to provide a pre-set "call back" Resource identifier to which the SP would have to reply back to?
7. Would existing messaging infrastructures still work as most of them have proprietary protocols or else they support JMS/ RMI?
8. Identify a URI for brokering of the two-part synchronous post HTTP call to. This would be for enabling the pseudo-asynchronous interaction semantics. Is the centralized broker URI all that is needed to broker the two part synch call for faking the Asynch interaction semantics especially if the architecture does not have to deal with message delivery guarantee? I assume "post" might have void return type communication model to enable releasing of calling address space resources?
9. Would one have to leverage any "Registry" or ESB like middleware or "service grid" like infrastructures? If so, how does would resource proxying work? Would the proxy Resources’ "forward" their calls to the right Resource?
10. Would this mean that the middleware and mediation layer products need to know how to deal with this type of an interaction semantic where the "Registries", ESB, "service grids/ service marketplaces" are all metadata-driven Resource proxies with built-in support for REST - as opposed to service proxies as would be the case in SOA style architectural models.

Thanks in advance for your feedback.

surekha -

Extending the use of Service Enabled Interfaces for dealing with Business Events

Typically Business Services are considered to be invoked in a synchronous request-response mode. Business Applications are supposedly called into action by human actors who leverage user interfaces. On the other hand, Business Processes are typically assumed to be triggered by Business Events.

However, it is possible to extend the use of Business Services and/or Business Applications to deal with Business Events. This enables the reuse of existing business logic and business rules in order to deal with new buisness interactions which are introduced into the ecosystem without needing to undertake a large service implementation effort.

For the purpose of this discussion Business Services, Business Processes and Business Applications will collectively be named as Event Response Interfaces. Business Events can be routed to the receipient Event Respsonse Interfaces by an ESB or a Message Hub. Business Events may also be sent to a Complex Event Processing Engine for aggregating, filtering or de-duping business events prior to routing them to the apporpriate Event Response Interfaces. Having said that, how does one extend existing Service Enabled Interfaces to trigger them asynchronously?

Architecturally, adding a Message Listener that in turn calls the Service Interface (such as seen in a Service Activator JEE pattern) or leveraging a Service Mediator (such as an ESB or a Message Hub) can allow Business Events to invoke Business Services or Business Applications outside of their regular request-response mode of interaction. The use of this simple architectural strategy allows Business Events to be processed by existing well tested Business Services or Business Applications.

One additional point to keep in mind in extending the interaction context from synchronous mode to that of an asynchronous mode is with regards to dealing with business exceptions, faults and business responses. In a synchrounous communication mode both of these are returned to the invoker of the business service or the business application. In an asynchrounous mode there is a disjointed execution context where in the invoker is not blocking on the call and waiting for execution control to be returned. The execution path may or may not be re-established.

To deal with this asynchrounous communication model the service response and faults may either be logged in a common repository or may be returned back to the invoker by the service mediator. The service mediator may re-establish the execution path with the invoker. The service mediator such as the ESB may refer to the Registry to find the originator or may leverage the correlation identifier and originator address in the payload (a provision made by the WS-Addressing spec. via the use of wsa:From metadata element).

Regardless of which architecture component is used i.e. an ESB, a message hub or a message listerner that intercepts the Event Response Interface the result is the reuse of the service behavior. In addition to reusing the business service applying this simple service extension technique also enables leveraging existing investments in infrastructure components such as the ESB or message hub thus helping achieve business agility.

Please provide me with feedback as to whether or not you have leveraged existing IT service assets to introduce new communications pathways.

surekha -

Separation of service interface from service implementation

The pattern defined below attempts to increase an enterprise' chance of adopting SOA and insures the success of service orientation by reusing existing IT assets. These assets include leveraging existing legacy applications and well tested business logic or else could be the reuse of investments made in vendor package product installations.

________________________________________________________________________________

Separation of service interface from service implementation while leveraging existing IT assets

One of the key architecture tenants of SOA is the concept of separation of service interface from that of the service implementation. This concept although very simple is a very difficult one to put in practice. In this blog are discussed a few possible patterns that can be leveraged to loosely couple the service interface layer from the service implementation layer especially if these service interactions are within an industry vertical that are part of a business use case or interaction scenario common to that industry vertical. For instance, online auctioning and bidding interaction in the retail vertical. Loan approval based interactions in the financial services vertical.

In general, ESB vendors have tried to do this by proposing mediation flows that conform to the Service Broker and Service Gateway patterns. The Service Broker pattern leverages the ESB mediation flow to send requests to the appropriate service implementation(s) based on some type of mapping between the service interface and the service implementation. This information could either be wired into the ESB mediation flow or could be obtained from a service repository or metadata repository. On the other hand, the Service Gateway pattern leverages federated ESB mediation flows to send requests to the appropriate service implementation even when these implementations lie outside the firewalls or the enterprise boundaries. The Service Gateway pattern could be used for both extranet based service implementations and to communicate with external partners, agencies and SaaS style service providers. Both Service Broker and Service Gateway interactions may work well for an enterprise level service interaction or even for a specific extended enterprise level service interaction.

The following service pattern could fall into the category of industry vertical specific service interactions. The assumption is that both consumers and providers in a target industry are able to come to an agreement on common service interface definitions that apply to the interactions required by the industry vertical. However, the question would remain as to whether the service providers would be expected to re-architect their entire implementation layer, not to mention competitive advantage in having optimized product implementations.

An option would be to repurpose their service implementation layer by creating point to point adapters (such as the JCA style adapters). Our collective experience shows that the adapter soon becomes ridden with complex business logic such as business validation and transformation logic. An attempt is being made provide alternate options via the use of metadata and generic service patterns to deal with this predicament.

Use of industry specific schema based payload

This pattern proposes using industry-specific payload schemas that hold metadata such as a "service descriptor" and "properties descriptors" for service payloads. The “service descriptor” provides the definitions for a searchable metatag. Both the “service descriptor” and the “properties descriptors” are defined using industry-specific schemas and govern how this information is populated by service consumer.

The presence of the service descriptor metatag in the payload would allow the ESB mediation layer or a Service Router layer to generically route calls to the appropriate service implementation end-points. The service descriptor metatag can also be used to optimize Service Router specific "find" and "redirect to the service implementation layer" algorithms. Again, the “find” and “redirect” algorithms would be fine-tuned in keeping with the industry vertical specific interactions. Thus, any service implementation end-point that can map its' service semantics and service behavior to the value in the service descriptor can honor the service request even when its' own high level service facade does not conform to the "externally published" service interface.

Here the service implementation layer is expected to be able to satisfy a service request that takes the form of a business event or else would be in the form of an incoming message over a messaging pipeline. The key is that the service implementation layer call is not an RPC style API call (which would/ could require the use of a business adapter). The ESB mediator layer can at the most perform protocol translation but is not required to do more than a simple message format transformation (if needed) for interacting with a service implementation layer even though it does not map to the industry compliant service interface.

The pattern also prescribes the use of the industry-specific payload schema that has “property descriptors” that govern how the service consumer populates the payload. This schema not only provides placeholders for encapsulating service consumer business usage context but also defines the request parameter properties and the response payload properties. The property descriptors also have information on how to handle the payload semantically. The use of industry-specific payload schemas allows the transformation of the payloads to be optimized via the use of standard utilities or appliance plug-ins. These optimizations can be applied at the service mediation layer or at the service router layer. The model makes it possible to achieve processing efficiencies at the level of request and response payload creation and consumption.

In the event, the service implementation layer cannot deal with the industry-specific payload definitions then these optimized transformation algorithms could be leveraged to adapt the payload into a service implementation specific construct. This transformation can be undertaken either at the service router layer or ESB mediation layer or even at the service façade layer (i.e. at the entry-point into the service implementation layer).

Again, given that the interaction is going to be message-oriented and not an RPC based API call this pattern when leveraged would insure that the implementation of service provider does not have to be completely re-factored. This statement holds true as long as the service implementation layer honors the service behavior and is compliant with response semantics expected by the consumer.

Among the key benefits of this pattern is the fact that the industry vertical can agree to and publish service interaction semantics via the use of metadata that are embedded in the service payload. This enables all parties to interpret the service request and the service responses in an unambiguous manner.

________________________________________________________________________________
As always your feedback is invaluable.
surekha -

Key Learning: SOA forcing IT organization to focus on their core competencies

Over the last few months I have had the opportunity to talk to some of the IT organization in the silicon valley and am happy to report that most of them are either in the process or have already adopted SOA. By adopting SOA - it has forced the IT leadership to focus on their core competencies. Most of them are primarily focused on the fundamentals such as:

• Selecting Strategic Projects
• Aligning Project Portfolio of IT Strategy
• IT Organization Structure to for enabling Business Agility
• Establishing Enterprise Architecture Team
• Blueprinting Information Architecture and BPM and
• Service Engineering Process

What was surprising to me was that even though most of the us early adopters have shared our key learning and best practices on blogs such as this, most IT organizations are still struggling to focus on their core competencies - such as defining the roles and responsibilities of each teams, separations of development and support teams, communicating IT goals to all teams, establishing Enterprise Architecture teams and defining and enforcing the application life cycle (services life cycle come next).

If this is the state in the heart of the hi-tech industry, woder how it is out there in other industries. It would great to get the feedback on what the rest of you are observing .

- Yogish

SOA Consortium – Ground Floor SOA – Recap for 07

The most significant progress we on the SOA Consortium Working Group of "Executing Business Driven SOA" have made in the past year was in the definition of the SOA Planning Framework. Efforts undertaken as part of the SOA Planning Framework track will include classification and logical grouping of the collective body of work done thus far by the Strategy 2 working group; thereby allowing us to deliver a single document that can be used as a reference by anyone who embarks on the path to SOA-enable their enterprise. This includes work done under the headings of “SOA Readiness Assessment”, “SOA Opportunity Identification” and “High Level People, Process and Tools” matrix. All of this material will find a home in the various sub-components of the SOA Planning Framework. Once completed the SOA Planning Framework will be one of the most comprehensive guidelines for those looking to implement SOA.

The other key benefit of the SOA Planning Framework deliverable is that it allows an enterprise to start on one or more of these components without forcing it into having to define all components prior to getting started on the SOA Roadmap. This is where the SOA Planning Framework differs from that of a rigid SOA methodology - in that it is a loosely-woven collection of SOA enabling constructs.


Again, I would like to thank you of the members of Strategy 2 for your participation and invaluable input. I look forward to a very productive upcoming year.

surekha -

Selling SOA to business

One of the most important milestone in SOA adoption is getting the business to buy in. However, getting business buy in requires an approach that is sometimes not obvious. I had an interesting conversation with Jon Brodkin of Network World. Please check it out at



http://www.networkworld.com/supp/2007/ndc6/102207-avis-soa-case-study.html



and let me know what you think





Ashok Kumar