Meri Nova’s Post

To make machine learning useful, you need a Machine Learning Engineer. A data scientist (considering his background and education) is more focused on model development. He should be a "scientist", which means he is more focused on the math and algorithms. He spends lots of time on data preparation, because good data will yield good models. But once the model is developed and ready, it won't be useful without putting it into production. Putting a model into production involves many skills that are -in most cases- out of the scope of a data scientist's skills. What kind of infrastructure is required to support the model? Will the model be hosted on a separate machine? Will it be deployed to a cluster? Will it be deployed as a container to a Kubernetes cluster? Will it require a data pipeline and an orchestration tool to schedule when the model will run (this mode of deployment is sometimes called batch deployment)? How to monitor the infrastructure as well as the model in production? What to do when something goes wrong? How to secure the connection from and to the model? How to update the model in production without affecting the service? How to comply with regulations when the model is in production and apply governance? etc. Most (if not all) of these tasks should be handled by a Machine Learning Engineer. An MLE's experience should be at the intersection between Data Science, Software Development, and DevOps.

(Machine Learning Operations), data structures play a crucial role in managing and organizing data throughout the machine learning lifecycle. Some common data structures used in ML Ops include: 1. Dataset: A collection of data samples used for training, validation, and testing machine learning models. Datasets can be structured (e.g., tabular data) or unstructured (e.g., text, images). 2. Dataframe: A two-dimensional labeled data structure similar to a table in a relational database or a spreadsheet. Dataframes are commonly used in ML Ops for data manipulation and analysis, especially with libraries like Pandas in Python. 3. Feature Vector: A one-dimensional array or list containing the features (input variables) of a dataset. Feature vectors are often used as input to machine learning algorithms. 4. Model Parameters: Data structures containing the learned coefficients or weights of a machine learning model. These parameters are typically represented as vectors or matrices and are adjusted during the training process. 5. Pipeline: A data structure representing the workflow or sequence of steps involved in data preprocessing, feature engineering, model training, and evaluation. Pipelines help streamline the ML Ops process and ensure reproducibility. 6. Metadata: Information about the data, models, and experiments stored alongside the actual data and models. Metadata can include details such as data schema, model hyperparameters, training/validation/test splits, and performance metrics. 7. Artifact Store: A repository for storing artifacts generated during the ML Ops pipeline, such as trained models, intermediate data, evaluation results, and visualizations. Artifact stores ensure traceability and reproducibility of experiments. 8. Version Control System (VCS): A system for tracking changes to code, configurations, and other files. VCSs like Git are essential in ML Ops for managing collaboration, experimentation, and versioning of machine learning models and code. #mlops #mlengineer #dataengineer #devopscommunity #devops #devopsengineer #machinelearning #dataengineering #vertexai

(Machine Learning Operations), data structures play a crucial role in managing and organizing data throughout the machine learning lifecycle. Some common data structures used in ML Ops include: 1. Dataset: A collection of data samples used for training, validation, and testing machine learning models. Datasets can be structured (e.g., tabular data) or unstructured (e.g., text, images). 2. Dataframe: A two-dimensional labeled data structure similar to a table in a relational database or a spreadsheet. Dataframes are commonly used in ML Ops for data manipulation and analysis, especially with libraries like Pandas in Python. 3. Feature Vector: A one-dimensional array or list containing the features (input variables) of a dataset. Feature vectors are often used as input to machine learning algorithms. 4. Model Parameters: Data structures containing the learned coefficients or weights of a machine learning model. These parameters are typically represented as vectors or matrices and are adjusted during the training process. 5. Pipeline: A data structure representing the workflow or sequence of steps involved in data preprocessing, feature engineering, model training, and evaluation. Pipelines help streamline the ML Ops process and ensure reproducibility. 6. Metadata: Information about the data, models, and experiments stored alongside the actual data and models. Metadata can include details such as data schema, model hyperparameters, training/validation/test splits, and performance metrics. 7. Artifact Store: A repository for storing artifacts generated during the ML Ops pipeline, such as trained models, intermediate data, evaluation results, and visualizations. Artifact stores ensure traceability and reproducibility of experiments. 8. Version Control System (VCS): A system for tracking changes to code, configurations, and other files. VCSs like Git are essential in ML Ops for managing collaboration, experimentation, and versioning of machine learning models and code. #mlops #mlengineer #dataengineer #devopscommunity #devops #devopsengineer #machinelearning #dataengineering #vertexai

#Topics Unlock ML insights using the Amazon SageMaker Feature Store Feature Processor [ad_1] Amazon SageMaker Feature Store provides an end-to-end solution to automate feature engineering for machine learning (ML). For many ML use cases, raw data like log files, sensor readings, or transaction records need to be transformed into meaningful features that are optimized for model training. Feature quality is critical to ensure a highly accurate ML model. Transforming raw data into features using aggregation, encoding, normalization, and other operations is often needed and can require significant effort. Engineers must manually write custom data preprocessing and aggregation logic in Python or Spark for each use case. This undifferentiated heavy lifting is cumbersome, repetitive, and error-prone. The SageMaker Feature Store Feature Processor reduces this burden by automatically transforming raw data into aggregated features suitable for batch training ML models. It lets engineers provide simple data transformation functions, then handles running them at scale on Spark and managing the underlying infrastructure. This enables data scientists and data engineers to focus on the feature engineering logic rather than implementation details. In this post,...

Leaving this here for anyone who is learning Machine learning/Deep Learning/AI but does not know what the next step should be. It's not enough to just learn the models and how to apply them, but those models should be deployment-worthy whether online or in a dashboard. And for this to happen, the code has to go through several stages. Think Software development style Dev, Test, UAT & Prod sites/code-base

What does an 𝗘𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁 look like? MLOps practices are there to improve Machine Learning Product development velocity, the biggest bottlenecks happen when Experimentation Environments and other infrastructure elements are integrated poorly. Let’s look into the properties that an effective Experimentation Environment should have. As a MLOps engineer you should strive to provide these to your users and as a Data Scientist, you should know what you should be demanding for. 𝟭: Access to the raw data. While handling raw data is the responsibility of Data Engineering function, Data Scientists need the ability to explore and analyze available raw data and decide which of it needs to be moved upstream the Data Value Chain (2.1). 𝟮: Access to the curated data. Curated data might be available in the Data Warehouse but not exposed via a Feature Store. Such Data should not be exposed for model training in production environments. Data Scientists need the ability to explore curated data and see what needs to be pushed downstream (3.1). 𝟯: Data used for training of Machine Learning models should be sourced from a Feature Store if the ML Training pipeline is ready to be moved to the production stage. 𝟰: Data Scientists should be able to easily spin up different types of compute clusters - might it be Spark, Dask or any other technology - to allow effective Raw and Curated Data exploration. 𝟱: Data Scientists should be able to spin up a production like remote Machine Learning Training pipeline in development environment ad-hoc from the Notebook, this increases speed of iteration significantly. 𝟲: There should be an automated setup in place that would perform the testing and promotion to a higher env when a specific set of Pull Requests are created. E.g. a PR from feature/* to release/* branch could trigger a CI/CD process to test and deploy the ML Pipeline to a pre-prod environment. 𝟳: Notebooks and any additional boilerplate code for CI/CD should be part of your Git integration. Make it crystal clear where a certain type of code should live - a popular way to do this is providing repository templates with clear documentation. 𝟴: Experiment/Model Tracking System should be exposed to both local and remote pipelines. 𝟗: Notebooks have to be running in the same environment that your production code will run in. Incompatible dependencies should not cause problems when porting applications to production. It can be achieved by running Notebooks in containers. Did I miss something? 👇 #GenAI #LLM #LLMOps #MachineLearning

Aurimas Griciūnas Thanks for sharing. It's crucial for MLOps engineers and Data Scientists to provide effective Experimentation Environments for their users. Your breakdown of the properties an effective Experimentation Environment should have is insightful—it emphasizes the essential components for efficient Machine Learning product developmen

What does an 𝗘𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲 𝗠𝗮𝗰𝗵𝗶𝗻𝗲 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗘𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁 look like? MLOps practices are there to improve Machine Learning Product development velocity, the biggest bottlenecks happen when Experimentation Environments and other infrastructure elements are integrated poorly. Let’s look into the properties that an effective Experimentation Environment should have. As a MLOps engineer you should strive to provide these to your users and as a Data Scientist, you should know what you should be demanding for. 𝟭: Access to the raw data. While handling raw data is the responsibility of Data Engineering function, Data Scientists need the ability to explore and analyze available raw data and decide which of it needs to be moved upstream the Data Value Chain (2.1). 𝟮: Access to the curated data. Curated data might be available in the Data Warehouse but not exposed via a Feature Store. Such Data should not be exposed for model training in production environments. Data Scientists need the ability to explore curated data and see what needs to be pushed downstream (3.1). 𝟯: Data used for training of Machine Learning models should be sourced from a Feature Store if the ML Training pipeline is ready to be moved to the production stage. 𝟰: Data Scientists should be able to easily spin up different types of compute clusters - might it be Spark, Dask or any other technology - to allow effective Raw and Curated Data exploration. 𝟱: Data Scientists should be able to spin up a production like remote Machine Learning Training pipeline in development environment ad-hoc from the Notebook, this increases speed of iteration significantly. 𝟲: There should be an automated setup in place that would perform the testing and promotion to a higher env when a specific set of Pull Requests are created. E.g. a PR from feature/* to release/* branch could trigger a CI/CD process to test and deploy the ML Pipeline to a pre-prod environment. 𝟳: Notebooks and any additional boilerplate code for CI/CD should be part of your Git integration. Make it crystal clear where a certain type of code should live - a popular way to do this is providing repository templates with clear documentation. 𝟴: Experiment/Model Tracking System should be exposed to both local and remote pipelines. 𝟗: Notebooks have to be running in the same environment that your production code will run in. Incompatible dependencies should not cause problems when porting applications to production. It can be achieved by running Notebooks in containers. Did I miss something? 👇 #GenAI #LLM #LLMOps #MachineLearning