Overview of SeeBeyond for the BizTalk professional
The SeeBeyond e*Gate integrator is the core of the platform. The e*Gate Integrator enables the flow of information by providing connectivity to applications and datastores across a network. You can loosely think of them as a BizTalk 2006 Application than encompasses all the different artifacts. e*Gate is based on a distributed architecture with an open design that deploys load balancing options. e*Gate processes events according to user-defined business logic and integrates business processes between applications, ensuring end-to-end data flow into back-office systems.
The record structures of messages (called Events in SeeBeyond) that are transferred to and fro are called an "Object Type Definition" (OTD). OTDs are created from source types and can represent any record type, e.g. XML, BAPI, Flat File, Relational row, WSDL, etc. You can think of them as the canonical type definition used by the platform - or schemas in the BizTalk World. The OTD is a Java class. It does all the marshalling/unmarshalling to these various formats.
e*Way are key components of an e*Gate and provide application connectivity and data processing such as business collaborations, transformation logic, and publish/subscribe. They comprise of an executable and a configuration file. You can think of e*Ways as a combination of adapters and ports and their binding information. e*Way connections are analogous to BizTalk ports.
e*Ways have “External Slice” components which is analogous to the communications adapter and the “Work Slice” which can be thought of as an application adapter and some business logic.
Both BizTalk and SeeBeyond support wide accepted standards like XML, XSD, WDSL, SOAP, BPEL4WS and are geared towards messaging and loosely coupled integration. While the integration goals are the same, SeeBeyond has taken a different architectural approach than BizTalk. For example, SeeBeyond does not have a central MessageBox where all messages are delivered and which acts as the subscription engine (although it does have smaller and more numerous Intelligent Queues). SeeBeyond does not insist on XML conversion of Flat File messages (through a pre-processing disassembler).
SeeBeyond eBusiness Integration Suite
The SeeBeyond eBusiness Integration Suite includes the following components and sub-components:
· e*Gate Integrator:
· e*Way Intelligent Adapters
· IQ Intelligent Queues
· Business Object Brokers (BOBs)
· e*Insight Business Process Manager
· e*Xchange Partner Manager
· eSecurity Manager
· e*Index Global Identifier
A one to one comparison of artifacts will probably of much greater help:
| Architectural Parallels between BizTalk and SeeBeyond | |
| BizTalk | SeeBeyond |
| Messaging / routing | Collaboration (data routing) |
| Schema | Structure |
| BizTalk NT Service | E*Gate Registry Service |
| Host Instances | Participating hosts in a Control Broker |
| BizTalk Host | Registry Host |
| BizTalk Administration Console | E*Gate Enterprise Manager GUI |
| Group Hub / Health and Activity Tracking | E*Gate Monitor GUI |
| MessageBox | IQ (Intelligent Queue) datastore – different from MessageBox and is a very loose parallel |
| Immutablity of messages | Events are mutable even inside a IQ |
| Adapter | E*Way |
| Send and Receive Ports | E*Way Connections |
| BOB (Internal e*Gate component used for load balancing) | |
| BizTalk Application | E*Gate |
| Binding file (very loose analogy) | Schema (The parameters and relationships between e*Gate components) |
| Orchestrations | E*Insight BPM (Business Process Manager) |
| Pipelines | (no equivalent) No distict pre-processor artifact, although pre-processing can be impleted in code |
| Visual Studio IDE | SeeBeyond uses its own IDE unlike BizTalk which uses the common .NET IDE Visual Studio. Although SeeBeyond can be coded in J2EE, it does not provide Developer Tools that can be installed as plug-ins for standard IDEs like the Eclipse, Jbuilder, Java Studio Creator, etc. |
| Pub/sub managed by MessageAgent from the MessageBox | Events are published by Collaboration. The Subscription is by the EventType and the Publisher. |
| Business Rules / Policies | Property file – can be changed at runtime. Rules xan also be in the database |
| (no equivalent) – although Group Hub can add more MessageBoxes | IQ Manager controls IQs |
| Content Based Routing | Eventy Types / Collaboration Rules |
| Trading Partner Management | e*Xchange Partner Manager |
| Implemented in .NET Security infrastructure, PKI certificates | eSecurity Manager |
| Business Activity Services | e*Index Global Identifier (e*Index) |
| No direct analogy – implemented in BizTalk Group / Clustering | Business Object Broker (BOB) |
| BTS Management Database | e*Gate Registry |
| Port | E*Way Connnection |
Technical Parallels between SeeBeyond and Biztalk
Having discussed migration approaches, we want to highlight another challenge/dilemma that can come in the way of getting a quick start on the migration effort. The challenge is ability of a BizTalk trained team to be able to quickly understand the parallels between SeeBeyond and BizTalk.
Given this challenge and dilemma, we decided to dedicate a section to discussing these parallels and draw some analogies between the concepts, features, technologies and artifacts of the BizTalk and SeeBeyond platforms.
The best way to learn is by example. For the BizTalk architect or developer attempting a migration project - the first look into the SeeBeyond world is confusing and ambiguous – because the familiar concepts and artifacts they are familiar with in BizTalk are missing or fused with another feature. For some there are simply no parallels at all. There is little or no documentation available in this area on the internet and one has to rely on assistance from experts.
Please note that many parallels are mostly loose analogies to help the BizTalk developer gain a newcomer’s familiarity with SeeBeyond. Since the architecture of both platforms is different, drawing parallels between both is a hazardous task, fraught with shoe-horning and over-simplification. We have attempted it nevertheless - on the grounds of giving the migration team a first look / toehold into the world of SeeBeyond and dispel some of the fog surrounding it. Caution is advised against literal interpretation of these parallels – and no technical implementation based on it is recommended. A deeper and more thorough understanding of SeeBeyond Technical Architecture and concepts is advised before the migration project is undertaken.
In order to do the parallels justice, it is perhaps advisable to discuss what are the key components that make up the SeeBeyond suite and how they tie together
Posted in: Software Software Programming| Tags: Microsoft WCF BizTalk BizTalk Migration BizTalk Server EAI integration solutions Microsoft Biztalk SeeBeyond Technical ParallelsTechnical lessons in BizTalk Migration
1. Although SeeBeyond Collaborations can be translated to BizTalk Orchestrations, it makes better sense to separate an implementation into Orchestrations for business processes, Maps for data transformation, Business Rule policies for business logic, and Pipelines for low level message pre-processing. In a SeeBeyond implementation, the Collaboration is the main executive module where most logic resides - and all these sub-tasks are mixed together creating a delirious “code cocktail” we have witnessed first-hand. In BizTalk such collaboration can be cleanly split into functional modules which are easier to maintain.
2. Data conversion. Type system mismatch between Java and .Net, date and time formatting can be sources of subtle errors. For example, in Java the default DateTime string formatting is different from neutral culture formatted DataTime string in the .NET Common Type System. SeeBeyond code compare dates as strings which is culture specific in .Net, so it’s better to switch to a DateTime object.
3. Message size. In SeeBeyond messages are binary structures and strings. But all flat file and native messages entering BizTalk are converted into XML – blowing up their size 3 to 5, even 10 times! When these are mapped they are held in memory as is the transformed message – which is equally large perhaps. This can easily lead to performance problems due to memory pressure. One should pay close attention to the message size. Since messages in BizTalk are immutable developers may create duplicates in the orchestrations which will add to the memory overhead. Instead the BizTalk developer should chunk up the message into smaller (physical) sizes – as BizTalk is highly parallel and is best optimized for processing a high throughput of light-weight or medium-weight messages with remarkable speed – but the reverse (heavy messages and low throughput) is not true.
4. SeeBeyond’s data structures map pretty well to .Net typed DataSet. This fact can be used when converting component logic from Java to .Net. Entire structures can be mapped to a typed dataset which makes case for straightforward code conversion. There is a difference thoough – nullable types. .NET DataSet supports DbNull and this should be taken in account.
5. Centralized logging and tracing is critical in this type of mission. Microsoft Enterprise Logging Block or Log4Net library can help to tackle this task.
6. Sometimes it makes far more sense to engineer the logic in .NET components than BizTalk. Just because it’s a migration exercise does not mean BizTalk is a natural choice. A .NET library with a BizTalk wrapper can do the trick in many cases – do try to think out of the box when evaluating design.
7. Proper encoding should be used for data both in xml as well as in flat file formats. If this is not taken care of, it may result in output data mismatch and can lead to multiple test cycles and rework.
8. Automated test scripts also play a major role in making things easier since you must be prepared for many cycles of verification and overload testing.
Posted in: Software| Tags: Microsoft WCF BizTalk BizTalk Migration BizTalk Server EAI integration solutions Project ManagementMigration Challenges in BizTalk Migration
Integration migration projects are humbling in their complexity and over-whelmingly communication-intensive. You are more likely to run into communication, management and operational challenges which can conspire to defeat your march than any serious technical challenges. An adaptive, agile approach where assumptions are questioned daily and where close anticipation and planning discipline is instilled has a greater chance of success.
Here are pointers and advice about the challenges of an EAI migration project – right from the trenches.
Project Management lessons
1. First, if the “Replicating Behavior” migration approach is chosen, a pilot implementation of an interface would be beneficial. This allows learning a lot about the application scope, get the “big picture”, peek into many technical details and eccentricities early, and foresee upcoming challenges – not to mention planning ahead for infrastructure and performance. This Proof Of Concept is not required to be fully functional and can contain only limited amount of artifacts but should a variety of processes, workflows, operations, adapters etc. allowing to sound out different eccentricities.
2. In all large scale migration projects, it is very important to recreate the environment early. Solution partitioning, project structure, build, server topology, packaging and deployment guidelines should not be left for the last – we cannot overemphasize this. A controlled environment where SeeBeyond and BizTalk application can be run in isolation on the control data set. This is critical as there’s no other means to validate migration quality in this approach except to compare output for the known input. It takes considerable amount of discipline to stick to this plan as features get discovered, complexity grows and the team is crushed under this load – but it is important that this stays in focus as an early deliverable, not late one.
3. Early and continuous integration of the processes along with testing mentioned in 3) above helps to discover problems early and take mitigation actions. This is a key assumption that bedevils many teams who assume so-called isolated systems will not impact each other when run together on the same application server. Make sure to get the latest version of the shared code to avoid unpleasant surprises.
4. Some BizTalk features must be evaluated carefully before applying in implementation. There might be critical limitations that would slow down or even halt migration. For example, the Oracle DB adapter for BizTalk cannot return results of the joins from multiple tables. So, if you require this kind of result set then it’s time to turn back to the more flexible ADO.NET and bid goodbye to a code-free approach. Native SQL schema can be used instead of adapter generated schemas to generalize data access out of orchestrations and make them cleaner and easier to understand and maintain. When creating data access intensive orchestrations it often makes sense to consider delegating Oracle adapter task to a .Net component to get rid of large amount of transient messages and reduce performance overhead.
5. Do a careful analysis of the current code base before committing any timelines or budget. Do not assume the solution you are inheriting is either perfect or well-architected. We have found messy implementations where the management is delegated to UNIX/WMI scripts instead of being controlled from the GUI/e-Ways/Collaboration rules, resulting in business logic being distributed over areas not conducive to maintenance or good practice. When these problems are compounded by lack of documentation, then you need to increase project times by a significant amount to accommodate the pain and increased efforts of first collating logic in the code base into a sensible picture before the analysis and design can even begin.
Posted in: Software Programming| Tags: Microsoft WCF Challenge B2B BizTalk BizTalk Server EAI flagship integration solutions Migrating Windows Communication FrameworkRoll Out Strategy in BizTalk Migration
Rollout in an EAI migration project is always tricky – the goal should always be to cause as little disruption and downtime as is necessary. In order to achieve this goal of minimal disruption, an incremental phase approach is recommended.
To further illustrate our phased deployment approach, consider the following scenario which is typical in most EAI implementations in large enterprises - The existing SeeBeyond implementation has a history of having been built incrementally; new programs or upgrades were placed in service individually. Now compound that complexity with migrating the actual EAI logic to a new tool (BizTalk). Deploying such a migrated solution which includes a large number of sub-systems in service all at the same time, even if they are well tested, is an incredibly risky step for any business. Do not be deceived if each subsystem individually has a low bug rate – a single problem can manifest itself in ways across other systems that may not have been tested or accounted for. What typically occurs is that the number of such problems quickly overwhelms and cross-pollutes other systems, crippling the ability of the organization to deal with them. In other words - High Conversion Risk.
Incremental migration strategies entail the decomposition of multi-system implementations in phases which can be independently and selectively addressed and replaced. This reduces the costs and risks of a migration program. The original SeeBeyond sub-systems are retained and used until the BizTalk functions achieve an acceptable level of reliability, then they are replaced. In the beginning the smaller interfaces are piloted as the learning curve is steeper and the number of unknowns greater, so we do not want to overload with a complex sub-system. As we gain greater expertise and vital insights, we attempt to bite of larger pieces and tackle them until we meet with complete success.
In an incremental rewrite methodology, the SeeBeyond system continues to run as the new system is built, tested, and deployed. You will want to try to build the new system in a way that seamlessly integrates with the old system. The side-by-side environment maintains redundancy as functionality moves from the old to the new system.
Posted in: C# and .NET| Tags: Microsoft WCF B2B BizTalk BizTalk Server EAI flagship integration solutions Migrating Windows Communication FrameworkDiscovery and Development Strategy of BizTalk Migration
With any migration approach, discovery of SeeBeyond code and artifacts is necessary due to the invaluable wealth of information there which has survived due its robustness or been refined to its current state. Not to use it would be an enormous waste. Although a huge amount of it is redundant in BizTalk due to the pre-built infrastructure (ports, adapters, configuration) you can mine a lot of it outright – some of it a simple cut and paste – such as database queries. Hence knowledge of SeeBeyond is inescapable.
To delve into SeeBeyond or to have a meaningful exchange with the SeeBeyond analyst, it would benefit the BizTalk professional greatly if could gain a first look inside SeeBeyond with the concepts and mental models he is already familiar with. Getting sufficient time from the harried, expensive and stretched thin SeeBeyond analyst is likely not going to be easy to come by, so the problem gets worse.
We have painstaking compiled a reference section to help with this comprehension – you will find this in the Appendix A of this paper.
Once the migration is complete, it is necessary to ensure that the new destination data is semantically equivalent to the original destination data (that has been processed with the SeeBeyond implementation). For this to be so, the results of all operations on the new Biztalk solutions must be identical to those on the old SeeBeyond solutions. There is no better way to gain such assurance than to exhaustively perform the same operations on both solutions and to compare the results. We recommend a Cross-Talk approach for verification testing.
In the Cross-Talk QA approach, the data source is flipped from SeeBeyond to BizTalk receivers and the results compared. When verified with a credible set of test queries, a high degree of assurance can be obtained from this approach that the new data store is semantically equivalent to the old one.
Posted in: Software| Tags: Microsoft WCF B2B BizTalk BizTalk Server EAI flagship integration solutions Migrating Windows Communication FrameworkReArchitecture in BizTalk Migration
This rearchitecture approach is ideal in circumstances where the original implementation is sub-optimal (has performance issues or has been overtaken by new developments) or was developed in multiple layers laid on top over time of one another without an inherent design.
Even systems with well-defined architectures are prone to structural erosion. The relentless onslaught of changing requirements that any successful system attracts can gradually undermine its structure. Systems that were once tidy become overgrown as piecemeal growth gradually allows elements of the system to sprawl in an uncontrolled feature-creep. If such sprawl continues unabated, the structure of the system can become so badly compromised that it must be abandoned. The only way to sustain this system is by patches, and over time layers develop, until no shred of any original architecture or pattern remains.
Such systems are usually very difficult to maintain, document and perform poorly, making them good candidates for complete re-writes. In this case rearchitecture in a new system (BizTalk) from the ground up makes most sense as looking under the hood will confuse and provide no net benefit. There will also be many subtle, and not so subtle, differences between the old and new platforms and these make manual translation very difficult. A developer can quickly get led astray by the many subtle differences and become unproductive due to the overhead of constantly switching back and forth between old and new tools and platforms.
The newly designed system will accommodate new features, use the latest standards, and follow best practices for component reuse. The lessons from the previous implementation and hindsight will always be available, making the second generation solution a robust and optimized implementation.
Here are some tips for a rearchitecture approach
· Make use of accepted standards like HL7, HIPAA, RosettaNet, cXML, GS1, etc where applicable
· Loose coupling enabling other applications to be plugged in later
· Common components should be separated out into frameworks for reusability
· Avoid using custom connectors as these are difficult to develop and maintain
· Insist on automation of testing process to check for regression
· Run a performance lab and overload tests to make sure the application scales
· Ask for load balancing and fail over strategies for deployment
Posted in: Software Software Programming| Tags: Microsoft WCF B2B BizTalk BizTalk Server EAI flagship integration solutions Migrating Windows Communication FrameworkWhy migrate from SeeBeyond to BizTalk?
Migrating from a large EAI platform to another is always a tough decision and cannot be made in vacuum. Whatever EAI platform an enterprise has in place forms the backbone of their enterprise when it comes to their data communication needs. The EAI platform is what one would analogously consider the “plumbing” of a house. If the plumbing fails, the house is unlivable and alternate arrangements have to be made.
Given the business criticality of an EAI solution, it is essential for an enterprise to choose a tool that is scalable, robust, extensible and most importantly maintainable. Typically these are qualities one would look in any software tool, but nowhere are they more important then in an EAI tool.
Historically, the problem with EAI tools in the enterprise space has been their lack of ability to continue to evolve to the new standards and advancements in technology and also the ability to scale with the growing demands of business. The tools that did manage to provide patches to the tools for such advances were usually “kludgey” solutions that were not truly integrated with the tool. Given these challenges that are bound to occur with any EAI tool now or in the future, it is prudent to move with a tool that is supported by a financially stable company which will be there for the long haul and has the deep pockets to support the advancements in technology. The reasons discussed about inability to evolve and scale has plagued the EAI industry until now because typically all EAI vendors have been small companies with their own agendas dictated more by wall street than the needs of their customers and as a result virtually all the key players have either almost vanished (Vitria) or have been acquired by larger corporations to fill a niche in their overall solution portfolio (SeeBeyond being purchased by Sun is a prime example) when the focus of the company may not be EAI.
Biztalk is one of the few tools that has grown internally within Microsoft and has been nurtured over the past few years for it to mature and achieve enterprise status. It is supported by the most financially stable software corporation on the planet and has the deep pockets and resources to keep evolving BizTalk to the ever changing needs.
Regardless of what tool an enterprise may be utilizing currently, chances are the enterprise will hit a point where the tool is unable to evolve or scale as needed and the sooner an enterprise makes a decision to move with an industry leader, the safer its investment in time, resources and money will be.
Having discussed some benefits of moving to BizTalk as a tool for choice, here are some points to consider why it is beneficial to move away from other EAI tools such as SeeBeyond.
· Expensive to maintain and increases TCO (Total Cost of Ownership)
· Is no longer a market leader in EAI tools and will eventually phase out and disappear where components are cannibalized and utilized in the J2EE frameworks from Sun
· Product is not backwards compatible with older versions
· Lack of backward and forwards compatibility requires constant re-tooling of team
· Upgrade in SeeBeyond is resource intensive and time consuming
· Takes longer to make changes to functionality
Posted in: Software Software Programming| Tags: Microsoft WCF B2B BizTalk BizTalk Server EAI flagship integration solutions Migrating Windows Communication Framework