How to ETL from Data Fabric into Azure Databricks

Erik van der Vlugt · 2025-02-17T15:07:40+00:00

There was an error rendering this rich post.

Hi,

I am looking for guidance how to build up "ETL" from Data Fabric / Atlas into Azure Databricks. Goal would be to use ION API's from a Python job (Databricks Workflows) to fetch the json data and work our way selecting the correct data to use for reporting from there.

Is there (easy to use) documentation on how to accomplish this
Has anyone done this willing to share some insights, perhaps even some code on how to do this.

Hope I am clear enough, not being the developer myself. Currently we're using Compass jdbc which is causing too heavy compute time on ION.

Gr, Erik

Find more posts tagged with

Accepted answers

All comments

Brandon Watson

Hello Erik - They have a tutorial for using Data Lake API's. This is going to be more difficult than the JDBC driver, due to limits and paging. You should be able to use JDBC, have you assessed what is causing the high compute time? Are you doing full extracts every time? Have you looked into infor.lastmodified for incremental loads (https://docs.infor.com/inforos/2025.x/en-us/useradminlib_cloud/default.html?helpcontent=datafabrug/wov1631199556283.html)?

https://developer.infor.com/tutorials/data-management/external-access-to-data-lake-apis/

Erik van der Vlugt

Hi Brandon, thanks for your quick answer. Our dev has used the infor.lastmodified. It is in combination with another filter though: something like this: WHERE (CONO= '1' AND infor.lastmodified() >= '2025-02-17 13:50:07.000')

We have only one (active) CMP so I suppose it could be left out. Would that influence compute? So answering your question: We're not doing full extract. Still this process brings us very high in Compute time, only part of the tables is loaded frequently.

This is why I am investigating if it is possible to get the json data out and into Databricks without (Compass) compute (as I am led to believe).

Brandon Watson

I would still have concerns about the API's also using compass compute time, since they are essentially running queries in the back end as well. Have you looked into using the Retreive method in an ION Data Lake Flow?

Erik van der Vlugt

I found some reference to this but it is hard to find documentation on this. Can you help me there?

Bojan Rafajloski

Hi Erik,

Compass SQL is intended for complex data analysis. I'll make an assumption that you are extracting replicated data as-is stored in Data Lake, 1:1. Please clarify if you are not. Compass is not the most optimal method for these scenarios, and there are alternatives that may fit that need better.

You are probably overusing your compute time by scheduling large amount of frequent single table incremental extraction queries, without any SQL transformation involved. Even though the individual queries may be running quickly, for example finishing in 2-5 seconds, if you include a lot of tables (e.g 300) and frequently schedule the extractions (e.g. every 15 minutes), you will easily overconsume your licensed compute time.

For 1:1 extraction, you may want to look into using the Data Fabric Object Retrieval APIs to pull the ndsjon data objects from Data Lake. You can use the dl_document_indexed_date for the incremental logic. Please also use variation handling concepts to ensure that when you are loading data you are updating target records only when the record variation is higher.

Or alternatively, if you prefer to use Compass SQL, you may want to transform the data in the queries to bring it as close to the final consumption data model.

Erik van der Vlugt

Hi Bojan,

Yes, your assumption is correct. Back in the days we set that up using jdbc towards compass not knowing any better. This as you say is now causing overconsumption of compute.

So now looking at changing that to fetching the ndjson data objects and trying to find usable documentation to convince and assist colleagues that will need to rebuild our tooling. Honestly, the documentation I can find up to now is highover or incomplete to make my case. But I am glad that here I get to meet the people that know the stuff. Any other documentation will be highly appreciated.