While loop in Mule ESB, or not

It’s a common use case to need an automated job that run once a day, executing a list of queued requests received throughout the day. However, there is no direct support for while loops in mule. Google leads me to this article in dzone. It implements a while loop using subflows and recursion.

In this article I put this implementation of while loop in mule to the test.

<flow name="test-loop-flow" processingStrategy="synchronous">
  <poll doc:name="Poll">
    <schedulers:cron-scheduler expression="0/30 * 6-23 * * ?"/>
    <logger message="Scheduled job started" level="DEBUG" doc:name="Logger"/>
  </poll>
  <set-variable variableName="counter" value="40000" doc:name="Set max number to process"/>
  <flow-ref name="while_loop" doc:name="call while loop"/>
</flow>
<flow name="while_loop" processingStrategy="synchronous">
  <db:select config-ref="DATASOURCE_CONFIG" doc:name="Get Request">
     <db:parameterized-query><![CDATA[select top 1 id from RequestsToProcess]]></db:parameterized-query>
  </db:select>
  <logger message="#[payload]" level="DEBUG"/>
  <choice doc:name="Choice">
    <when expression="#[flowVars.counter == 0 || payload == empty]">
      <logger message="The loop breaks" level="DEBUG"/> 
    </when>
    <otherwise>
      <logger message="do something" level="DEBUG"/>
      <set-variable variableName="counter" value="#[flowVars.counter-1]" doc:name="count down"/>
      <db:delete config-ref="DATASOURCE_CONFIG" doc:name="Remove request">
        <db:parameterized-query><![CDATA[delete from RequestsToProcess where id = #[payload[0]['id']]]]
        </db:parameterized-query>
      </db:delete>
      <flow-ref name="while_loop"/>
    </otherwise>
  </choice>
</flow>

This above mule snippet sets up a job that starts every 30s between the hours of 06:00 and 23:00. It sets a counter to 40k, representing the maximum number of requests to process in each invocation. (The 40k is from an actual use case I’m working on. We are limited to 40k requests each day). The processing strategy for both flows are set to synchronous, where only one instance of the flow will be invoked at any time.

The database table RequestsToProcess has been populated with more than 40k entries. The time taken to select and delete 40k entries from a database table is significantly longer than 30s, the time between each invocation of the flow. Can you guess what happens?

06 08 2018 14:31:19 | DEBUG | org.mule.api.processor.LoggerMessageProcessor | [{id=106}]
06 08 2018 14:31:19 | DEBUG | org.mule.api.processor.LoggerMessageProcessor | do something
06 08 2018 14:31:20 | ERROR | org.quartz.core.JobRunShell | Job endpoint.polling.1912630717.test-loop-flow~
threw an unhandled Exception: 
java.lang.NoClassDefFoundError: Could not initialize class org.mule.config.ExceptionHelper
	at org.mule.api.MuleException.<init>(MuleException.java:56) ~[?:?]
	at org.mule.api.MessagingException.<init>(MessagingException.java:125) ~[?:?]
...
06 08 2018 14:31:29 | DEBUG | org.mule.api.processor.LoggerMessageProcessor | Scheduled job started
06 08 2018 14:31:30 | DEBUG | org.mule.api.processor.LoggerMessageProcessor | [{id=106}]

The log shows that at 30s, the cron scheduler starts a new invocation, causes an exception, halts the loop, and picks up request id 106 again. The reason a new invocation is started before the existing long running job has finished is because the previous instance is looping within the flow while_loop, while the cron scheduler invokes the calling test-loop-flow. In other words, if your job is still running when the next invocation starts, mule will throw a cryptic exception, and halts the execution of the existing job.

What if I set the cron scheduler to run the job only once a daily, eliminating the chance this particular job not finishing before the next invocation? The cron expression is edited as below, running once at 02:00 each day

<schedulers:cron-scheduler expression="0 0 2 * * ?"/>

This will lead to a stack overflow exception in mule, because of the recursion

Exception stack is:
1. null (java.lang.StackOverflowError)
  java.util.ResourceBundle$CacheKey:-1 (null)
2. null (java.lang.StackOverflowError). Message payload is of type: Integer (org.mule.api.MessagingException)
  org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor:32 
  (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/api/MessagingException.html)

From this little experiment, I would not recommend using recursion to implement a while loop in mule. I normally use a cron expression that runs the job multiple times for a period of time each day, processing multiple requests in each invocation.

<flow name="request-to-export-flow" processingStrategy="synchronous">
  <poll doc:name="Poll">
    <schedulers:cron-scheduler expression="0/5 * 6-23 * * ?"/>
    <logger message="Polling request table" level="DEBUG" doc:name="Logger"/>
  </poll>
  <db:select config-ref="DATASOURCE_CONFIG" doc:name="Get BT999 Request">
    <db:template-query-ref name="spPopRequest"/>
    <db:in-param name="topcount" type="INTEGER" value="100"/>
  </db:select>
  <foreach doc:name="For Each">
  </foreach>
</flow>

This is tested with Mule ESB 3.7 CE.