Scatter-Gather Integration Pattern (Mule 4) Part 2

GitHub repository with the Mule Project can be found at the end of the post.

In the previous post, Intro to Scatter-Gather Integration Pattern, we have discussed the basics of the Scatter-Gather message routing Enterprise Integration Pattern and also demonstrated a simple case of the pattern using MuleSoft Connector. Moving forward, from a happy path scenario, we will see the demonstration on how Mule 4 Scatter-Gather connector behaves in case of processing errors with or without Mule Error handlers. Additionally, we will explore the example of processing multiple services using the Mule 4 Scatter-Gather connector.

Gathering thoughts from the previous blog:

• Scatter-Gather processes messages in-parallel and the result is an aggregated response of the routes.

• To live up to its merits, Scatter-Gather needs more than one message processing route.

• When the process responses are independent of each other, it is efficient to use Scatter-Gather.

• Use Scatter-Gather for multicasting scenarios.

• Scatter-Gather is concurrent processing and there is no sequence of execution.

• Scatter-Gather will not stop processing messages to other successful routes during failures.

How does the input message get processed in Scatter-Gather?

The input payload/message is distributed or copied across all the routes of the Scatter-Gather. Our previous example of the MuleSoft Scatter-Gather is modified, by adding a new Set Payload component to check the status of the payload during the Scatter-Gather processing. As shown below from the log results, the payload is copied or distributed across all the routes of the Scatter-Gather. In the listed example, the payload value “result0: 42” is not honored by the three flows, as they have their payload values.

In short, the Set Payload value before the Scatter-Gather component is overwritten by the flow's payload value.

Fig 2.0: Input payload is copied to all the route processors.

However, the Set Payload information can be very well used to add business logic to the multiple routes accordingly. For demonstration purposes, Flow One payload is modified to retrieve the “result0” information.

Fig 2.1: Initial Set Payload result is displayed to include in the flow-one response.

Error Handling in Scatter-Gather

In our previous post, we stated that the Scatter-Gather avoids a complete meltdown, unlike sequential processing. But, without an effective error handling in place, you can question the merits of the latter statement. One can use a MuleSoft Try scope in the routers of a Scatter-Gather component for error handling purposes.

If a particular route encounters errors, the Try scope will take care of the error using the On-Error-continue error handler, which would make the route successful by handling the error logic. The Scatter-Gather route failures are handled well to process the error unless it is required to end the process abruptly. Let’s try to prove the same using some twists in our current Scatter-Gather example.

Happy Path ☺

Everything is green, life’s pretty smooth and happiness sways around. The Scatter-Gather route flows one, two, and three payloads have been modified to implement a simple addition, subtraction, and division of numbers (ex: 10+2, 16-2, 30/2) to be aggregated as an output result of each flow. In the screenshot below, results are displayed as expected, all looks good! Yay!!

Fig 2.2: Result of the happy path Scatter-Gather flow without any errors.

Not so Happy Path

Well, life isn’t all rainbows and unicorns as the above result made you believe for a while. If the flow-three fails due to a division by zero error, the Scatter-Gather will fail as there isn’t any mechanism to avert the catastrophe. The process will halt, and the error will be displayed as given below.

Fig 2.3: Mule debugger showing the Flow-Three failure due to error. No error handling is implemented.

Fig 2.4: Postman test of the Scatter-Gather Flow-Three failure due to error.

Although not so happy path the above case is, one would still want to see some result as an output instead of a long semi-cryptic error message. Here comes the error handling Try scope for the Scatter-Gather routes! All the three flows are wrapped with Try Scope to capture the errors in any of the routes if they failed while processing the message. The output is guaranteed to be some aggregated result instead of a long error description.

Fig 2.5: Try-Scope error handling implemented in all three processes of the Scatter-Gather.

From the above fig. 2.5, an error handling scope (Mule 4) of On-Error Continue is implemented which will make sure a result is derived from the Scatter-Gather. More information about Mule 4 error handling can be found at MuleSoft documentation. The transform message component, inside the On-Error Continue scope, is used to provide more information on the type of error as output. By adding this feature, we can now identify the failure reason for flow three. After all, our goal here is to achieve a more composed state of the output.

There will be valid business scenarios in real-time, where the output of the route in error is not required to be displayed. In such cases, the output will be an aggregate response of successful flows, as shown in Fig 2.5.1. In the On-Error continue component, the transform message payload is given as [ ] (empty array) to bypass error information.

Fig 2.5.1: Postman test of the Scatter-Gather flow with error handling implemented. Only success responses from the processes are displayed.

Fig.2.6 is showing results of the flow stepping into the error handling scope through Anypoint studio mule debugger. The error is bypassed, and the Scatter-Gather has aggregated the messages from successful processes.

Fig 2.6: Display of the Flow-Three error handling and log showing error information.

When the flow-three result is required irrespective of failure or success, a descriptive user-friendly error message can be added. This would help the user to identify the reason for Scatter-Gather route failures (if any). If a business logic needs to be invoked on flow-three failure, it can be incorporated in the On-Error continue error handler scope. The business logic can be anything specific to invoking another process upon failure or notifying the stakeholders through email.

Fig 2.7: Postman test of the Scatter-Gather flow with error handling implemented. All the route responses are displayed, including error.

Error message to be displayed as the output of the failed process flow-three.

Fig 2.8: Flow three error handling response returned.

Yay!! We searched for happiness, in the not so happy path of Scatter-Gather and we were able to strike the right balance. A thought to ponder and reflect upon, which brings us to our reflection questions, and I would encourage you to try out the code snippets in mule 4 to try out the below cases and get the answers.

Reflection Questions:

• What happens to the initial payload, when one of the processes in the Scatter-Gather fails?

• What happens if the variable in one process is modified and not in the other?

• What would be the result of Scatter-Gather if “On-Error propagates” is used instead of “On-Error continue” when one of the routes fails?

Multiple Services Call Using Mule 4 Scatter-Gather

Okay! Moving ahead, remember we are yet to see a case of multiple services being invoked through the Scatter-Gather Mule 4 component. The example demonstrated below is retrieved from the free Mule 4 developer course. It’s a very simple example to showcase the power of the Scatter-Gather component. The scenario is to search the flights of United Airlines and Delta airlines for a given airport code.

The United Airlines flight search is retrieved from REST API calls. Whereas Delta airlines information is retrieved through SOAP WSDL. The Set-Variable component is used to store the airport code value to retrieve the available flight information. Variable ‘code’ value is set as #[message.attributes.queryParams.code].

Fig 2.9: Scatter-Gather example showing multiple services (HTTP REST, SOAP WS) invoked.

The following result is derived from the Scatter-Gather processing of the two different service calls.

Fig 2.10: Postman test displaying the results of United and Delta flights.

You have made it to this part of the post, now the last stop in the journey is to add the missing part in the above demonstration, error handling. If you check Fig 2.9, Try scope hasn’t been implemented yet. Applying the learnings from our previous section, let’s add the Try scope to each of the Scatter-Gather routes.

We can add a flavor to the scenario by displaying only available flights. Meaning, we are looking to retrieve any flight information irrespective of route failure. You can most definitely hash out a test case to check the output if both the flights fail.

To demonstrate the error handling, the web services URL is tweaked as http://mu.learn.mulesoft.com/delta12?wsdl

Fig 2.11: Implemented Try-scope to all the processes.

Upon checking the flow through mule-debugger, we can see the getDeltaFlights process failed due to error type WSC: CONNECTIVITY.

Fig 2.12: getDeltaFlights process failure due to connectivity.

However, the failure was handled successfully by the MuleSoft Try scope and the On-Error continue to Transform message has an empty array returned as a response. Scatter-Gather has now aggregated the response from both the flows as output shown below:

Fig 2.13: Postman test of the Scatter-Gather flow with the final response from Scatter-Gather with only United flights.

Penultimate Thoughts

Finally, we have reached a point where we understand the ball bearings of Scatter-Gather through the Mule 4 connector example.

To reiterate:

• Scatter-Gather is best suited for Multicast scenarios.

• The Scatter-Gather messages are processed in-parallel with aggregate responses.

• More than one process is required for Scatter-Gather to function according to its discussed merits.

• The MuleSoft Scatter-Gather has many features to control the execution flow in case of route failures.

• In case of errors, a new business logic such as notifications can be invoked as required.

Now that you’re familiar with Scatter-Gather, it’s the time to check out the real code in action.

Configuration File Code:

Next Steps

It is highly recommended that you check the simple example code snippets and test out the concepts discussed in these pair of blogs. Download MuleSoft Anypoint Studio to test out the Scatter-Gather flows by goofing around in your unique ways. Test the above reflection questions to grasp an in-depth understanding of the Mule 4 Scatter-Gather component. Try out some DataWeave functions to make the error response interesting for the end-users.

For a better understanding of the Mule 4 error handling refer to the below amazing blogs:

• Mule 4 error handling demystified

• Mule 4 error handling deep dive

• Use case-specific error handling in Mule 4

I hope you enjoyed the post. Please subscribe to ProstDev for more exciting topics.

Hasta luego, amigos!

GitHub repository

ProstDev GitHub - Scatter-Gather Error Handling Example

Go to Previous Post: Part 1