Infor ETL

Tina Douglass
Tina Douglass Member Posts: 2
in Data Fabric

Good Morning,

Hoping I have asked this in the right forum.

After landing the data from the Data Lake into SQL Server using the ETL Tool (or whatever DB you use as a landing), what has others used to pull the data out of those landing tables into other applications? Do you repeat the same type of max(date), do you use a date processing table that has start and end dates with a new entry every pull? If this option, what is the best date to do a between start and end?

Best Answer

  • Kevin Heiman
    Kevin Heiman Member, Staff Posts: 16
    Answer ✓

    The default update in ION ETL is to do a upsert, ie. insert if new, update if exist. If you are using that default, then there is no need to use max variation logic. For incremental change you could track something that you know is the last processed marker (this all varies depending on design but variation might be a good option), or put a extra column in to denote if you picked it up.

    Best practice includes having an index over the variation added to the table.

Answers

  • Tina Douglass
    Tina Douglass Member Posts: 2

    Kevin, Do you happen to know what is mapped to the dldocument date out of queryall and into the upsert?

  • JillPapendick
    JillPapendick Member, Staff Posts: 1

    The DL Document Date is mapped to the dl_document_indexed_date.

Categories