What we did
We implemented data importing pipeline from scratch. It supports multi-processing and distributed importing of large amounts of data. Also we developed and integrated analytics engine with the imported data so that the data scientists can schedule asynchronous data analytics jobs on the imported data. Moreover, we integrated the results from the analytics jobs with Grafana and created a two-way data flow.