ETL Flows

@Tim P the interfaces factsheet is certainly a place where you can model ETL data interfaces. A couple of points to consider if you go down this road:

1 - Create a new subtype (or types) to differentiate and filter easily between APP interfaces and ETL interfaces. The meta-model v4 standard  interface sub-types (Logical, API) may confuse rather than help, it is OK to remove/change them.

2 - If using different sub-types, you can make use of the Conditional Attributes (leanix.net) feature, as likely the additional data attributes the Application versus ETL teams may have is going to be different in some areas.

Hope this helps.

Jumping on this thread…I am curious if anyone has done this.  Some of our applications share a database.  We have ETL processes that pull data from the shared database and move it to a target database.  Since the Interface fact sheet only allows one “Application” as its provider and there are more than one application sharing the source database, the only option I see is to create a “dummy” application (called something like “db X interfaces”) and attach the Interface fact sheet to that “dummy” as its Provider (btw, the same issue exists on the Consumer side).  That’s not ideal as 1) those “dummies” count towards licensing and 2) they skew my metrics on reports.  Anyone encounter and solve this delimma?  Thanks in advance.

@jeff.fellers the LeanIX documentation Interface Guidelines - Provider and Consumers (leanix.net) talks about the reasons for only having 1 provider and many consumers. You may or may not buy their arguments.

However, as we are talking about ETL’s as a distinct subtype it might be prudent to add a new (custom) relationship section with an alternative is to create a new relation titled something like ETL Source Applications and also one for the ETL Consumer(s) that could have many-many multiplicity.

This means that you could hide these “custom” relations in the other Interface sub-types, and the standard ones in the ETL subtype.

UPDATE: The new relations cannot be used in Data Flow Diagrams and the Interface Circle map report.

I do not think there is a single answer… This is how aI would approach it (just me, others may use different approaches):

• In the specific example, it looks like the interface is a “one to one” only. So we only have to determine who should be the “provider”. This is being described as a ETL process, so I’m assuming there are some Transformations involved, maybe some reformatting only or  some business logic. I would then ask the question of who “owns” the code/logic of that Transformation and would model it having the db you mentions as the component that implements the interface, and as such the “responsibility of the provider” (AppA). Note that this may be independent of the direction of the flow of data. So, we have AppA provider of InterfaceA (with the BD as the IT Component that implements the application) and AppB as the consumer.
• If it is not a “one to one”, and you have the similar scenario but multiple applications consuming the same data, then it would still fit the same modelling.
• In some scenarios like Pub/Sub or Data Hubs where there may be logic that “aggregates/composes” data from multiple sources I would consider using a distinct Application with multiple Interfaces as required.

Hope it makes sense.

Cheers

@jeff.fellers the LeanIX documentation Interface Guidelines - Provider and Consumers (leanix.net) talks about the reasons for only having 1 provider and many consumers. You may or may not buy their arguments.

However, as we are talking about ETL’s as a distinct subtype it might be prudent to add a new (custom) relationship section with an alternative is to create a new relation titled something like ETL Source Applications and also one for the ETL Consumer(s) that could have many-many multiplicity.

This means that you could hide these “custom” relations in the other Interface sub-types, and the standard ones in the ETL subtype.

If configured correctly the new relations can be used in Data Flow Diagrams and the Interface Circle map report too.

Hi @Justin Swift ,

Just a quick question on this non-OOB relation: Would this not break the Interface Circle Map report, and probably make some other reports more complex?

Now thinking more about this, the Interface Circle Map report could be fine, but don’t think so :-)

Cheers.

@jeff.fellers the LeanIX documentation Interface Guidelines - Provider and Consumers (leanix.net) talks about the reasons for only having 1 provider and many consumers. You may or may not buy their arguments.

However, as we are talking about ETL’s as a distinct subtype it might be prudent to add a new (custom) relationship section with an alternative is to create a new relation titled something like ETL Source Applications and also one for the ETL Consumer(s) that could have many-many multiplicity.

This means that you could hide these “custom” relations in the other Interface sub-types, and the standard ones in the ETL subtype.

If configured correctly the new relations can be used in Data Flow Diagrams and the Interface Circle map report too.

Hi @Justin Swift ,

Just a quick question on this non-OOB relation: Would this not break the Interface Circle Map report, and probably make some other reports more complex?

Now thinking more about this, the Interface Circle Map report could be fine, but don’t think so :-)

Cheers.

Hi @Helder.Luz we already have some extra non-OOTB relations, they don’t break the original Interface Circle report, as the other relationships are simply not shown in it, it is also the same on Data Flow Diagrams. I don’t think the logic considers them “interfaces”, even though they are in the filters.

Hi @Justin Swift ,

Hi @Helder.Luz we already have some extra non-OOTB relations, they don’t break the original Interface Circle report, as the other relationships are simply not shown in it, it is also the same on Data Flow Diagrams. I don’t think the logic considers them “interfaces”, even though they are in the filters.

That is what I meant by break: the Interface Circle Report will not be complete since those new interface will not show, which may mislead anyone relying on the report. Same for DFDs…

Cheers

HI @Helder.Luz in our use case that is the behaviour we wanted for the added relations, so that works for us. I have updated previous post with caution on this approach for others.