DataLake record count vs ION ETL record count

PolarBear
PolarBear Member Posts: 2
edited September 25 in Data Fabric

I see this often where the DataLake will have XX number of records for a table, but IONETL provides XX - 1.

I'll re-run the translation and no new records are downloaded. I'll truncate the local table and re-run the translation and the count is still the same, smaller than the record count in the DataLake.

The only way I have found to reconcile is to purge the DataLake table, reset the replication table, replicate, and run the translation.

Some of these tables are in the tens of millions and frankly I'm getting tired of working every night fixing these issues. I'll get them reconciled and the next day different tables come up unreconciled.

I've attached a couple of screen shots from a small table that is one record off. I've also attached the report I create each time I run the translations, showing what tables are off.

Any help would be appreciated.

CashManagementAccount_Compass.png
Mismatched_Record_List.png
CashManagementAccount_SQL_Count.png
CashManagementAccount_Translation.png

Best Answer

  • Bojan Rafajloski
    Bojan Rafajloski Member, Staff Posts: 2
    Answer ✓

    Hi PolarBear

    I suggest to create a Data Fabric support case and our team will assist you. Few things to consider:

    Have you checked if there are duplicate records in Data Lake? Include DISTINCT in your query for counting or you turn on the Compass Analytical mode for the table (it will dedupe data in Compass).

    Have you been able to find the extra record in Data Lake? Would the same record miss when you are reloading data?

    Is there a chance that in Compass has more data because of incoming live data, data that hasn't been yet transferred in ETL?

    Which ETL Client version are you using?

Answers

Categories