ION Dataflow Throttling

Jeff Busch · 2024-11-07T20:13:51+00:00

There was an error rendering this rich post.

Good Afternoon,

In certain situations, we have a bulk update done that generates a large number of BODs. These BODs call an external API via a dataflow. After a few process, we start getting errors due to a rate limit on the API. Any tips/tricks to throttling the BODs/Dataflow so that only a determined number get processed every 10 seconds?

Thanks

Find more posts tagged with

Accepted answers

Kevin Heiman

It is always best proactive to incorporate Jitt, Backoff and Retry patterns into any API client. Most gateway restrict intense volume in order to manage malicious intent, unfortunately we all have to live with that. THere is much on the web about each of the above, summarized here:

Jitter : Don't set a clock to always run at the same time , and in between calls randomize the time with a minim wait to ensure the target does not throttle.

Backoff : when you know your having errors that take time to resole like server unavailable, then exponentially wait on retries until the service becomes available again.

Retry : response status codes should always be checked and appropriate action taken. You could catch the throttled status and pause, / slowdown.

You can put a wait in the data flow, but that probably is not what you want to do.

All comments

Kevin Heiman

Jitter : Don't set a clock to always run at the same time , and in between calls randomize the time with a minim wait to ensure the target does not throttle.

Backoff : when you know your having errors that take time to resole like server unavailable, then exponentially wait on retries until the service becomes available again.

Retry : response status codes should always be checked and appropriate action taken. You could catch the throttled status and pause, / slowdown.

You can put a wait in the data flow, but that probably is not what you want to do.

Prabodha Amarapema

We also experienced the same issue and we had to change our solutions to accommodate these Infor OS limits. As per the Infor recommendations, it is not advisable generate payload's more than 5MB . However we got many integrations that generates more than 5 MB message paylaods.