Integration Services and Strategy:  A journey into the quantum entanglement of systems servicing a logical business process

One of the most salient business needs in today’s world is seamlessly interconnecting two or more systems to allow business processes to flow effectively and efficiently. Most enterprise business systems are equipped with a configurable REST API¹ making interoperability and data flow control possible.

However, as the cliché statement goes, the devil is always in the details. Most of the details involve knowing your data flow and understanding where, why, and how processes will be spawned and what other processes they should spawn through their journey.

Data flow

Many times, there is a straight-forward, conceptually understood, and documented data flow. Still, when the implementation begins, there is a tremendous amount of complexity with the JSON² object property mappings and various systems that must be queried to make the proper REST API calls. If this is not done correctly, the logic fails to execute. This can lead to a Rube Goldberg model.

After performing many different integrations with various systems, ATR has found it best to ensure that defined processes are orthogonal³. Clear and concise boundaries for a specific service to have one job and one job only provides clear restrictions; it ensures that business logic will not become intermingled in the codebase.

The best practice is to map out all user scenarios that any system integration will need to handle, then walk through from start to finish the entire business process from the customer’s perspective. Ideally, a gateway service responsible for all transmission controls the desired user scenario flow.

Once this is done, all the touch points for all the different systems need to be denoted.

1. For example, a work order system spawns a request to perform a procedure only if a certain value or property is set on the work order system.
2. Then, if the procedure is being performed and there is a reading for a specific piece of equipment, it will need to query the value from another system (say PI Historian⁴)
3. Lastly, once the procedure is completed, a record will need to be sent to another system for archiving.

It is also imperative that the REST API routes follow a very logical path that allows for deep links so anyone can drill into objects/procedure/id/step/id, etc. Once the big picture is understood, modular integration services can be built.

Consistency

Consistent routing structures and nomenclature in the JSON object properties allow systems to connect without long, tedious meetings. Object transmission, call error handling, and retry intervals should all be handled by a separate service to stay modular and focused on their exact task.

So, in the case of PI Historian, calls will only perform the necessary GET call to obtain a value for an Equipment tag (if that is the primary use case), and the calls will not need to perform any other activity. Metadata services that can handle the object interpolation allow platforms such as a Maximo⁵ asset object to populate a global equipment object in ATR’s system and understand the object structure of what both systems expect.

It is also critical to always ensure a plan for what to do with the data once it gets to the other layers and ultimately clarify what system will own and control the data and define the “system of truth.”

Approach

Another important aspect is to build slowly with the essential building blocks first – sometimes, if we try to “boil the ocean,” we never finish the task. Set your expectation for round one to be the “minimal viable product” comprising what is needed for systems integration, and thoroughly document and map the system so that everyone understands and agrees on the scope. Then go back through and define a second or third phase to put in the nice to have items (gold plating).

If too much gold plating is put in upfront, the risk of delay increases. All good architectures are built up over time – the design should always be extensible, scalable, modular, and expandable as time goes on, and business needs evolve.

Key points

• Always keep the big data flow picture in mind for each use case scenario (user story).
• Always create modular services that concentrate on one specific task for one system.
• Have a separate flow control service responsible for handling errors, retry mechanisms, error notifications, etc. This will ensure consistency and ensure the same code for error handling doesn’t get duplicated into each service.
• Have a service designated for the JSON object property semantic mapper. This will ensure object equivalencies and proper interpolation when objects are boxed/unboxed and passed around. Don’t go property by property from an equipment object in one system to an equipment object in another system – instead, the mapper class should handle this for you.
• Have a dedicated service to handle the roles/security/authentication and authorization. You can divide authorization and rights from authentication, but they should be modular from the business processes. Keeping these modular ensures you don’t have duplicate code, which can be challenging to maintain later if a new security role gets added or if permissions sets get extended.

Final thoughts

Always map out your complete process flows for each user story and ensure you know the touch points and the journey from start to finish. Know what is getting spawned from where and what the rules will be. Keep it realistic and keep it simple. Walk through the user stories and remember to keep the use case scenarios clear, concise, and easy to understand and likewise keep the code equally modular and easy to understand.

Avoid duplication of code. Suppose you notice the same code for processing a user’s permission in more than one place. In that case, you should immediately stop and realize it may be time to break the decision structure into a REST API GET call because you do not want to intermingle responsibilities.

Last, don’t reinvent the wheel. If there is an existing REST API already available, then don’t recreate it if you don’t have to. Use existing libraries that are tried, true, and in use.

The possibilities are endless in terms of the types of integrations that can be achieved. Properly done, the results can increase productivity and gets your systems working in concert to produce results that help your business thrive and produce at optimum levels.