Ikim limited

About Project

Big data project for client energy company in Dublin. Designing distributed architecture to support hundreds of millions of records and implementing pipelines and solids in Dagster. Also, designing architecture to support Jupyter and Grafana as part of complete workflow.

What we did

We implemented data importing pipeline from scratch. It supports multi-processing and distributed importing of large amounts of data. Also we developed and integrated analytics engine with the imported data so that the data scientists can schedule asynchronous data analytics jobs on the imported data. Moreover, we integrated the results from the analytics jobs with Grafana and created a two-way data flow.