MLGdańsk #132 – How to transform a notebook model into a system pipeline

MLGdańsk online meeting will be held on Monday, 2022.10.03, at 18:00 CEST.

link to the meeting:

Our speaker will be:
Francisco Pinto-Santos, MSc

The topic of the talk is:
How to transform a notebook model into a system pipeline

“In the world of data analysis, there is a lot of information on the Internet about building models of various kinds to solve all kinds of problems. However, when a complex model is developed, it is necessary to implement it in a software system.

In this step, most of the people dedicated to data science, need the help of software architects and engineers, MLOps, etc. Therefore, in this workshop, we intend to show the first steps to identify how to segment a model in processing stages, and implement a distributed pipeline, managed by basic continuous integration techniques, allowing the creation and maintenance of a comprehensive system that serves the model.

To do so, we will first introduce the concept of microservice architectures and event-driven architectures, and then explain what an event broker is and start working with a real one, Apache Kafka.

Once the work base is established, we will teach how to create microservices in Python integrated with Apache Kafka, with a model (or part of it) integrated in these, as well as the necessary scripts for the continuous integration of these.

Finally, we will explain the possibilities offered when integrating ingest, storage, etc. so that attendees can expand their knowledge in this area later.”

The meeting is open to all interested – feel free to join!