Use of "Query Data Lake Activity" to obtain deltas in ION dataflow

Alistair Johnson
Alistair Johnson Member, Infor Partner Posts: 4
in ION

We have a dataflow that is currently using the "Retrieve from Data Lake" object to obtain deltas from M3 tables mirrored into Data Fabric.

We have a need to limit the result set being considered for this to a single company so have looked at the "Query Data Lake Activity" object. This allows us to add a filter to the dataset via AnySQL modeller e.g.

image.png

We can then consume that in the data flow and only company 200 data is retrieved.

However, rather than only sending deltas, each time the data flow triggers all the matching records are sent rather than just the new records matching the conditions.

Within AnySQL modeller I see the variation option which I've set on the VariationNumber field:

image.png

But that hasn't helped.

Any ideas on how to tweak this setup so deltas rather than full record sets are sent from the Query object?

Categories