Why not doing Business Intelligence directly in dbt with Lightdash?
Hi all 👋
Hope you’re all doing well!
Today I want to discuss about one topic really important to me because I’m working on it since around 2 years. As you all already know, the ‘’modern data stack’’ is becoming one of the buzzwords in the past months. Trying to find the perfect stack for your company is a tricky game and almost everyday you will have a new great service incoming on the market.
Well, I will try to help you on one part of the stack called the Business Intelligence.
Context
You will find below a great summary of the modern data stack:
If you jump on the BI Tools part, you will have great service like:
- Looker
- Tableau
- Power BI
- Chartio
- And many more…
Some of them will propose to you their own modeling language in order to help you create your dimensions and measures. You will also have the faculty to create aggregations, tables, views, tests… on the fly without having to deal with code like Python or Java.
It is a great way to be agile and flexible. If you have to create in a hurry a really important dashboard for your managers, you can almost do everything you need to do in those kind of services.
So what’s the problem?
The problem
The problem (and especially in big company) having to deal with those kind of services will split the data warehouse in 2 parts:
- on one side you will have your data engineering team which will handle data ingestion, data lake and data warehouse
- and on the other you will have your analytics / BI team which will use the data warehouse to create dimensions and measures
In fact it’s not really a problem, but… after a couple of months you will have many tables and views created on the fly without having a clear understanding of when they are created, for what and by who…
Most of the time the solution is having your analytics team managing your data warehouse and also your business intelligence part. They have a strong understanding of how to manage and structure your data. If some strong evolutions are needed in the data warehouse they will certainly decide to repatriate the complexity in the data modeling.
But… in big companies dana engineering and analytics engineering are often not managed by the same team…
So what the solution?
The solution
Dbt is a great service to manage your data engineering using 90% of your time .sql files. You can create dependencies and tests on the fly. You will have a strong community and you can do almost everything you need to do on the data warehouse in the smoothest way possible.
But dbt can not handle business intelligence logic with dimensions and metrics. This is where Lightdash is coming.
Lightdash is a way to manage your business intelligence directly in dbt. In summary, you will create all of your dimensions and measures directly in dbt with .yml files. Syntax is really easy to understand and you can not be more close to all the transformations you handle for your company. Meaning all of your metrics management will be directly linked to your data engineering repositories. Metrics will be part of data engineering and it’s great news and for me it’s a game changer for a simple reason. It will force your company to manage BI directly with the data engineering team OR it will force your business intelligence team to work closely with the data engineering Team. They will be part of the repository, commit / pull request ceremony. They will be in sync with what is done by data engineering so there is absolutely zero possibility to work on their own.
What I think
Currently Lightdash cloud is in beta, you can have a look at their website. The team is really proactive and are waiting for feedback. So here is my thought. Building BI logic directly in dbt is clearly a really good idea. Thinking about splitting data analytics and business intelligence is not a great deal for me. If you want to design a great dashboard and build a strong explore, you need to understand the step behind and master almost all of the data pipeline. dbt is a great tool with a large community, being able to add business intelligence in it it’s clearly a game changer for me.
I’m convinced that Lighdash is on the right path for helping BI be less painful.
Michaël
