This is where visualizations in ML come in. Graphical representations of structures and data flow within a deep learning model make its complexity easier to comprehend and enable insight into its decision-making process. With the proper visualization method and a systematic approach, many seemingly mysterious training issues and underperformance of deep learning models can be traced back to root causes.

In this article, we’ll explore a wide range of deep learning visualizations and discuss their applicability. Along the way, I’ll share many practical examples and point to libraries and in-depth tutorials for individual methods.

Why do we want to visualize deep learning models?

Visualizing deep learning models can help us with several different objectives:

• Interpretability and explainability: The performance of deep learning models is, at times, staggering, even for seasoned data scientists and ML engineers. Visualizations provide ways to dive into a model’s structure and uncover why it succeeds in learning the relationships encoded in the training data.
  
• Debugging model training: It’s fair to assume that everyone training deep learning models has encountered a situation where a model doesn’t learn or struggles with a particular set of samples. The reasons for this range from wrongly connected model components to misconfigured optimizers. Visualizations are great for monitoring training runs and diagnosing issues.
  
• Model optimization: Models with fewer parameters are generally faster to compute and more resource-efficient while being more robust and generalizing better to unseen samples. Visualizations can uncover which parts of a model are essential  – and which layers might be omitted without compromising the model’s performance.
   
• Understanding and teaching concepts: Deep learning is mostly based on fairly simple activation functions and mathematical operations like matrix multiplication. Many high school students will know all the maths required to understand a deep learning model’s internal calculations step-by-step. But it’s far from obvious how this gives rise to models that can seemingly “understand” images or translate fluently between multiple languages. It’s not a secret among educators that good visualizations are key for students to master complex and abstract concepts such as deep learning. Interactive visualizations, in particular, have proven helpful for those new to the field.
  
  
  

How is deep learning visualization different from traditional ML visualization?

At this point, you might wonder how visualizing deep learning models differs from visualizations of traditional machine learning models. After all, aren’t deep learning models closely related to their predecessors?

Deep learning models are characterized by a large number of parameters and a layered structure. Many identical neurons are organized into layers stacked on top of each other. Each neuron is described through a small number of weights and an activation function. While the activation function is typically chosen by the model’s creator (and is thus a so-called hyperparameter), the weights are learned during training.

This fairly simple structure gives rise to unprecedented performance on virtually every machine learning task known today. From our human perspective, the price we pay is that deep learning models are much larger than traditional ML models.

It’s also much more difficult to see how the intricate network of neurons processes the input data than to comprehend, say, a decision tree. Thus, the main focus of deep learning visualizations is to uncover the data flow within a model and to provide insights into what the structurally identical layers learn to focus on during training.

That said, many of the machine learning visualization techniques I covered in my last blog post apply to deep learning models as well. For example, confusion matrices and ROC curves are helpful when working with deep learning classifiers, just as they are for more traditional classification models.

Who should use deep learning visualization?

The short answer to that question is: Everyone who works with deep learning models!

In particular, the following groups come to mind:

• Deep learning researchers: Many visualization techniques are first developed by academic researchers looking to improve existing deep learning algorithms or to understand why a particular model exhibits a certain characteristic.
  
• Data scientists and ML engineers: Creating and training deep learning models is no easy feat. Whether a model underperforms, struggles to learn, or generates suspiciously good outcomes – visualizations help us to identify the root cause. Thus, mastering different visualization approaches is an invaluable addition to any deep learning practitioner’s toolbox. 
• Downstream consumers of deep learning models: Visualizations prove valuable to individuals with technical backgrounds who consume deep learning models via APIs or integrated deep learning-based components into software applications. For instance, Facebook’s ActiVis is a visual analytics system tailored to in-house engineers, facilitating the exploration of deployed neural networks.
• Educators and students: Those encountering deep neural networks for the first time – and the people teaching them – often struggle to understand how the model code they write translates into a computational graph that can process complex input data like images or speech. Visualizations make it easier to understand how everything comes together and what a model learned during training.

Types of deep learning visualization

There are many different approaches to deep learning model visualization. Which one is right for you depends on your goal. For instance, deep learning researchers often delve into intricate architectural blueprints to uncover the contributions of different model parts to its performance. ML engineers are often more interested in plots of evaluation metrics during training, as their goal is to ship the best-performing model as quickly as possible.

In this article, we’ll discuss the following approaches:

• Deep learning model architecture visualization: Graph-like representation of a neural network with nodes representing layers and edges representing connections between neurons.
  
• Activation heatmap: Layer-wise visualization of activations in a deep neural network that provides insights into what input elements a model is sensitive to.
  
• Feature visualization: Heatmaps that visualize what features or patterns a deep learning model can detect in its input.
  
• Deep feature factorization: Advanced method to uncover high-level concepts a deep learning model learned during training.
  
• Training dynamics plots: Visualization of model performance metrics across training epochs.
  
• Gradient plots: Representation of the loss function gradients at different layers within a deep learning model. Data scientists often use these plots to detect exploding or vanishing gradients during model training.
  
• Loss landscape: Three-dimensional representation of the loss function’s value across a deep learning model’s input space. 
• Visualizing attention: Heatmap and graph-like visual representations of a transformer-model’s attention that can be used, e.g., to verify if a model focuses on the correct parts of the input data.
• Visualizing embeddings: Graphical representation of embeddings, an essential building block for many NLP and computer vision applications, in a low-dimensional space to unveil their relationships and semantic similarity.

Deep learning model architecture visualization

Visualizing the architecture of a deep learning model – its neurons, layers, and connections between them – can serve many purposes:

1. It exposes the flow of data from the input to the output, including the shapes it takes when it’s passed between layers.
2. It gives a clear idea of the number of parameters in the model.
3. You can see which components repeat throughout the model and how they’re linked.

There are different ways to visualize a deep learning model’s architecture:

1. Model diagrams expose the model’s building blocks and their interconnection.
2. Flowcharts aim to provide insights into data flows and model dynamics.
3. Layer-wise representations of deep learning models tend to be significantly more complex and expose activations and intra-layer structures.

All of these visualizations do not only satisfy curiosity. They empower deep learning practitioners to fine-tune models, diagnose issues, and build upon this knowledge to create even more powerful algorithms.

You’ll be able to find model architecture visualization utilities for all of the big deep learning frameworks. Sometimes, they are provided as part of the main package, while in other cases, separate libraries are provided by the framework’s maintainers or community members.

How do you visualize a PyTorch model’s architecture?

If you are using PyTorch, you can use PyTorchViz to create model architecture visualizations. This library visualizes a model’s individual components and highlights the data flow between them.

Here’s the basic code:

import torch
from torchviz import make_dot

# create some sample input data
x = torch.randn(1, 3, 256, 256)

# generate predictions for the sample data
y = MyPyTorchModel()(x)

# generate a model architecture visualization
make_dot(y.mean(),
         params=dict(MyPyTorchModel().named_parameters()),
         show_attrs=True,
         show_saved=True).render("MyPyTorchModel_torchviz", format="png")

The Colab notebook accompanying this article contains a complete PyTorch model architecture visualization example.

PyTorchViz uses four colors in the model architecture graph:

1. Blue nodes represent tensors or variables in the computation graph. These are the data elements that flow through the operations.
2. Gray nodes represent PyTorch functions or operations performed on tensors.
3. Green nodes represent gradients or derivatives of tensors. They showcase the backpropagation flow of gradients through the computation graph.
4. Orange nodes represent the final loss or objective function optimized during training.

How do you visualize a Keras model’s architecture?

To visualize the architecture of a Keras deep learning model, you can use the plot_model utility function that is provided as part of the library:

from tensorflow.keras.utils import plot_model

plot_model(my_keras_model,

           to_file='keras_model_plot.png',

           show_shapes=True,

           show_layer_names=True)

I’ve prepared a complete example for Keras architecture visualization in the Colab notebook for this article.

The output generated by the plot_model function is quite simple to understand: Each box represents a model layer and shows its name, type, and input and output shapes. The arrows indicate the flow of data between layers.

By the way, Keras also provides a model_to_dot function to create graphs similar to the one produced by PyTorchViz above.

Activation heatmaps

Activation heatmaps are visual representations of the inner workings of deep neural networks. They show which neurons are activated layer-by-layer, allowing us to see how the activations flow through the model.

An activation heatmap can be generated for just a single input sample or a whole collection. In the latter case, we’ll typically choose to depict the average, median, minimum, or maximum activation. This allows us, for example, to identify regions of the network that rarely contribute to the model’s output and might be pruned without affecting its performance.

Let’s take a computer vision model as an example. To generate an activation heatmap, we’ll feed a sample image into the model and record the output value of each activation function in the deep neural network. Then, we can create a heatmap visualization for a layer in the model by coloring its neurons according to the activation function’s output. Alternatively, we can color the input sample’s pixels based on the activation they cause in the inner layer. This tells us which parts of the input reach the particular layer.

For typical deep learning models with many layers and millions of neurons, this simple approach will produce very complicated and noisy visualizations. Hence, deep learning researchers and data scientists have come up with plenty of different methods to simplify activation heatmaps.

But the goal remains the same: We want to uncover which parts of our model contribute to the output and in what way.

For instance, in the example above, activation heatmaps highlight the regions of an MRI scan that contributed most to the CNN’s output.

Providing such visualizations along with the model output aids healthcare professionals in making informed decisions. Here’s how:

1. Lesion detection and abnormality identification: The heatmaps highlight the crucial areas in the image, aiding in the identification of lesions and abnormalities.
   
2. Severity assessment of abnormalities: The intensity of the heatmap directly correlates with the severity of lesions or abnormalities. A larger and brighter area on the heatmap indicates a more severe condition, enabling a quick assessment of the issue.
   
3. Identifying model mistakes: If the model’s activation is high for areas of the MRI scan that are not medically significant (e.g., the skull cap or even parts outside of the brain), this is a telltale sign of a mistake. Even without deep learning expertise, medical professionals will immediately see that this particular model output cannot be trusted.

How do you create a visualization heatmap for a PyTorch model?

The TorchCam library provides several methods to generate activation heatmaps for PyTorch models. 

To generate an activation heatmap for a PyTorch model, we need to take the following steps:

1. Initialize one of the methods provided by TorchCam with our model.
2. Pass a sample input into the model and record the output.
3. Apply the initialized TorchCam method.

from torchcam.methods import SmoothGradCAMpp

# initialize the Smooth Grad.CAM++ extractor
cam_extractor = SmoothGradCAMpp(my_pytorch_model)

# compute the model’s output for the sample
out = model(sample_input_tensor.unsqueeze(0))

# generate the class activation map
cams = cam_extractor(out.squeeze(0).argmax().item(), out)

The accompanying Colab notebook contains a full TorchCam activation heatmap example using a ResNet image classification model.

Once we have computed them, we can plot the activation heatmaps for each layer in the model: 

for name, cam in zip(cam_extractor.target_names, cams):
plt.imshow(cam.squeeze(0).numpy())
plt.axis('off')
plt.title(name)
plt.show()

In my example model’s case, the output is not overly helpful:

We can greatly enhance the plot’s value by overlaying the original input image. Luckily for us, TorchCam provides the overlay_mask utility function for this purpose:

from torchcam.utils import overlay_mask

for name, cam in zip(cam_extractor.target_names, cams):
result = overlay_mask(to_pil_image(img),
                to_pil_image(cam.squeeze(0), mode='F'),
                alpha=0.7)
plt.imshow(result)
plt.axis('off')
plt.title(name)
plt.show()

As you can see in the example plot above, the activation heatmap exposes the areas of the input image that resulted in the greatest activation of neurons in the inner layer of the deep learning model. This helps engineers and the general audience to understand what’s happening inside the model.

Feature visualization

Feature visualization reveals the features learned by a deep neural network. It is particularly helpful in computer vision, where it reveals which abstract features in an input image a neural network responds to. For example, that a neuron in a CNN architecture is highly responsive to diagonal edges or textures like fur.

This helps us understand what the model is looking for in images. The main difference to the activation heatmaps discussed in the previous section is that these show the general response to regions of an input image, whereas feature visualization goes a level deeper and attempts to uncover a model’s response to abstract concepts.

Through feature visualization, we can gain valuable insights into the specific features that deep neural networks are processing at different layers. Generally, layers close to the model’s input will respond to simpler features like edges, while layers closer to the model’s output will detect more abstract concepts.

Such insights not only aid in understanding the inner workings but also serve as a toolkit for fine-tuning and enhancing the model’s performance. By inspecting the features that are activated incorrectly or inconsistently, we can refine the training process or identify data quality issues.

In my Colab notebook for this article, you can find the full example code for generating feature visualizations for a PyTorch CNN. Here, we’ll focus on discussing the result and what we can learn from it.

As you can see from the plots above, the CNN detects different patterns or features in every layer. If you look closely at the upper row, which corresponds to the first four layers of the model, you can see that those layers detect the edges in the image. For instance, in the second and fourth panels of the first row, you can see that the model identifies the nose and the ears of the dog.

As the activations flow through the model, it becomes ever more challenging to make out what the model is detecting. But if we analyzed more closely, we would likely find that individual neurons are activated by, e.g., the dog’s ears or eyes.

Deep feature factorizations

Deep Feature Factorizatio (DFF) is a method to analyze the features a convolutional neural network has learned. DFF identifies regions in the network’s feature space that belong to the same semantic concept. By assigning different colors to these regions, we can create a visualization that allows us to see whether the features identified by the model are meaningful.

For instance, in the example above, we find that the model bases its decision (that the image shows labrador retrievers) on the puppies, not the surrounding grass. The nose region might point to a chow, but the shape of the head and ears push the model toward “labrador retriever.” This decision logic mimics the way a human would approach the task. 

DFF is available in PyTorch-gradcam, which comes with an extensive DFF tutorial that also discusses how to interpret the results. The image above is based on this tutorial. I have simplified the code and added some additional comments. You’ll find my recommended approach to Deep Feature Factorization with PyTorch-gradcam in the Colab notebook.

Training dynamics plots

Training dynamics plots show how a model learns. Training progress is typically gauged through performance metrics such as loss and accuracy. By visualizing these metrics, data scientists and deep learning practitioners can obtain crucial insights:

• Learning Progression: Training dynamics plots reveal how quickly or slowly a model converges. Rapid convergence can point to overfitting, while erratic fluctuations may indicate issues like poor initialization or improper learning rate tuning.
  
• Early Stopping: Plotting losses helps to identify the point at which a model starts overfitting the training data. A decreasing training loss while the validation loss rises is a clear sign of overfitting. The point where overfitting sets in is the optimal time to halt training.
  

See in the app Full screen preview

Related post

How to Improve ML Model Performance [Best Practices From Ex-Amazon AI Researcher]

Read more

Gradient plots

If plots of performance metrics are insufficient to understand a model’s training progress (or lack thereof), plotting the loss function’s gradients can be helpful.

To adjust the weights of a neural network during training, we use a technique called backpropagation to compute the gradient of the loss function with respect to the weights and biases of our network. The gradient is a high-dimensional vector that points in the direction of the steepest increase of the loss function. Thus, we can use that information to shift our weights and biases in the opposite direction. The learning rate controls the amount by which we change the weights and biases.

Vanishing or exploding gradients can prevent deep neural networks from learning. Plotting the mean magnitude of gradients for different layers can reveal whether gradients are vanishing (approaching zero) or exploding (becoming extremely large). If the gradient vanishes, we have no idea in which direction to shift our weights and biases, so training is stuck. An exploding gradient leads to large changes in the weights and biases, often overshooting the target and causing rapid fluctuations in the loss.

Machine learning experiment trackers like neptune.ai enable researchers, data scientists and AI/ML engineers to track and plot gradients during training.

See in the app Full screen preview

Editor’s note

Do you feel like experimenting with neptune.ai?

• 

  Request a free trial

• 

  Play with a live project

• 

  See the docs or watch a short product demo (2 min)

To learn more about vanishing and exploding gradients and how to use gradient plots to detect them, I recommend Katherine Li’s in-depth blog post on debugging, monitoring, and fixing gradient-related problems.

Related article

Understanding Gradient Clipping (and How It Can Fix Exploding Gradients Problem)

Read more

Loss landscapes

We can not just plot gradient magnitudes but directly visualize the loss function and its gradients. These visualizations are commonly called “loss landscapes.”

Inspecting a loss landscape helps data scientists and machine learning practitioners understand how an optimization algorithm moves the weights and biases in a model toward a loss function’s minimum.

In an idealized case like the one shown in the figure above, the loss landscape is very smooth. The gradient only changes slightly across the surface. Deep neural networks often exhibit a much more complex loss landscape with spikes and trenches. Reliably converging towards a minimum of the loss function in these cases requires robust optimizers such as Adam.

To plot a loss landscape for a PyTorch model, you can use the code provided by the authors of a seminal paper on the topic. To get a first impression, check out the interactive Loss Landscape Visualizer using this library behind the scenes. There is also a TensorFlow port of the same code.

Loss landscapes do not only provide insight into how deep learning models learn, but they can also be beautiful to look at. Javier Ideami has created the Loss Landscape project with many artistic videos and interactive animations of various loss landscapes.

Visualizing attention

Famously, the transformer models that have revolutionized deep learning over the past few years are based on attention mechanisms. Visualizing what parts of the input a model attends to provides us with important insights:

• Interpreting self-attention: Transformers utilize self-attention mechanisms to weigh the importance of different parts of the input sequence. Visualizing attention maps helps us grasp which parts the model focuses on.
  
• Diagnosing errors: When the model attends to irrelevant parts of the input sequence, it can lead to prediction mistakes. Visualization allows us to detect such issues.
  
• Exploring contextual information: Transformer models excel at capturing contextual information from input sequences. Attention maps show how the model distributes attention across the input’s elements, revealing how context is built and propagated through layers.
  
• Understanding how transformers work: Visualizing attention and its flow through the model at different stages helps us understand how transformers process their input. Jacob Gildenblat’s Exploring Explainability for Vision Transformers takes you on a visual journey through Facebook’s Data-efficient Image Transformer (deit-tiny).

Visualizing embeddings

Embeddings are high-dimensional vectors that capture semantic information. Nowadays, they are typically generated by deep learning models. Visualizing embeddings helps to understand this complex, high-dimensional data.

Typically, embeddings are projected down to a two- or three-dimensional space and represented by points. Standard techniques include principal component analysis, t-SNE, and UMAP. I’ve covered the latter two in-depth in the section on visualizing cluster analysis in my article on machine learning visualization.

Thus, it is no surprise that embedding visualizations reveal data patterns, similarities, and anomalies by grouping embeddings into clusters. For instance, if you visualize word embeddings with one of the methods mentioned above, you’ll find that semantically similar words will end up close together in the projection space.

The TensorFlow embedding projector gives everyone access to interactive visualizations of well-known embeddings like standard Word2vec corpora.

When to use which deep learning visualization

We can break down the deep learning model lifecycle into four different phases:

Each of these phases requires different visualizations.

Pre-training deep learning model visualization

During early model development, finding a suitable model architecture is the most essential task.

Architecture visualizations offer insights into how your model processes information. To understand the architecture of your deep learning model, you can visualize the layers, their connections, and the data flow between them.

Deep learning model visualization during model training

In the training phase, understanding training progress is crucial. To this end, training dynamics and gradient plots are the most helpful visualizations.

If training does not yield the expected results, feature visualizations or inspecting the model’s loss landscape in detail can provide valuable insights. If you’re training transformer-based models, visualizing attention or embeddings can lead you on the right path.

Post-training deep learning model visualizations

Once the model is fully trained, the main goal of visualizations is to provide insights into how a model processes data to produce its outputs.

Activation heatmaps uncover which parts of the input are considered most important by the model. Feature visualizations reveal the features a model learned during training and help us understand what patterns a model is looking for in the input data at different layers. Deep Feature Factorization goes a step further and visualizes regions in the input space associated with the same concept.

If you’re working with transformers, attention and embedding visualizations can help you validate that your model focuses on the most important input elements and captures semantically meaningful concepts.

Inference

At inference time – when a model is used to make predictions or generate outputs – visualizations can help monitor and debug cases where a model went wrong.

The methods used are the same as the ones you might use in the post-training phase but the goal is different: Instead of understanding the model as a whole, we’re now interested in how the model handles an individual input instance.

Conclusion

We covered a lot of ways to visualize deep learning models. We started by asking why we might want visualizations in the first place and then looked into several techniques, often accompanied by hands-on examples. Finally, we discussed where in the model lifecycle the different deep learning visualization approaches promise the most valuable insights.

I hope you enjoyed this article and have some ideas about which visualizations you will explore for your current deep learning projects. The visualization examples in my Colab notebook can serve as starting points. Please feel free to copy and adapt them to your needs!

Models like for example ChatGPT, Gopher **(280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) are predominantly very large and often addressed as large language models or LLMs. These models can easily have millions or up to billions of parameters making them financially expensive to deploy and maintain.

Such large natural language processing models require significant computational power and memory, which is often the leading cause of high infrastructure costs. Even if you are fine-tuning an average-sized model for a large-scale application, you need to muster a huge amount of data. 

Such scenarios inevitably lead to stacking new layers of neural connections, making it a large model, moreover, deploying these models will require fast and expensive GPU, which will ultimately add to the infrastructure cost. So is there a way to keep these expenses in check?

Sure there is.

This article aims to provide some strategies, tips, and tricks you can apply to optimize your infrastructure while deploying them. In the following sections, we will explore these:

Challenges of large NLP models

Computational resources

LLMs require is a significant amount of resources for optimal performance. Below are the challenges that are usually faced concerning the same. 

1. High computational requirements

Deploying LLMs can be challenging as they require significant computational resources to perform inference. This is especially true when the model is used for real-time applications, such as chatbots or virtual assistants. 

Consider ChatGPT as an example. It is capable of processing and responding to queries instantly within seconds (most of the time). But there are times when the user traffic seems to be higher, during those moments, the inference time gets higher. There are other factors that can delay the inference, such as the complexity of the question, the amount of information required to generate a response, et cetera. But in any case, if the model is supposed to serve in real-time, it must be capable of high throughput and low latency.

2. Storage capacity

With parameters ranging from millions to billions, LLM can pose storage capacity challenges. It will be good to store the whole model in a single storage device, but because of the size, it is not possible. 

For example, OpenAI’s GPT-3 model, with 175B parameters, requires over 300GB of storage for its parameters alone. Additionally, it requires a GPU with a minimum of 16GB of memory to run efficiently. Storing and running such a large model on a single device may be impractical for many use cases due to the hardware requirements. As such, there are three main issues around storage capacity with LLMs:

2.1 Memory limitations

LLMs require a lot of memory as they process a huge amount of information. This can be challenging, especially when you want to deploy them on a low-memory device such as a mobile phone. 

One way to deploy such models is to use a distributed system or distributed inference. In distributed inference, the model is distributed on multiple nodes or servers. It allows the distribution of the workload and speeds up the process. But the challenge here is that it may require significant expertise to set up and maintain. Plus, the larger the model, the more servers are required, which again increases the deployment cost. 

2.2 Large model sizes

The MT-NLG model released in 2022 has 530 billion parameters and requires several hundred gigabytes of storage. High-end GPUs and basic data parallelism aren’t sufficient for deployment, and even alternative solutions like pipeline and model parallelism have trade-offs between functionality, usability, and memory/compute efficiency. As the authors in the paper “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models put it, this, in turn, reduces the effectiveness of the model. 

For instance, a 1.5B parameter model on 32GB can easily run out of memory during inference if the input query is long and complicated. Even for basic inference on LLM, multiple accelerators or multi-node computing clusters like multiple Kubernetes pods are required. There are techniques discussed by researchers where they propose the idea of offloading parameters to the local RAM. But these techniques turned out to be inefficient in practical use-case scenarios. Users cannot download such large scaled models on their systems just to translate or summarise a given text. 

2.3 Scalability challenges 

Another area for improvement with LLMs is scalability. We know that a large model is often scaled using model parallelism (MP), which requires multiple storage and memory capacity. This involves dividing the model into smaller parts and distributing it across multiple machines. Each machine processes a different part of the model, and the results are combined to produce the final output. This technique can be helpful in handling large models, but it requires careful consideration of the communication overhead between machines. 

In Distributed inference, LLM is deployed on multiple machines, with each machine processing a subset of the input data. This approach is essential for handling large-scale language tasks that require input to pass through billions of parameters. 

Most of the time, MP works, but there are instances where it doesn’t. The reason being MP divides the model vertically, distributing the computation and parameters among several devices for each layer where the inter-GPU communication bandwidth is large. This distribution facilitates intensive communication between each layer in a single node. The limitation comes outside a single node which essentially leads to a fall in performance and efficiency.

3. Bandwidth requirements

As discussed previously, LLM has to be scaled using MP. But the issue we found was that MP is efficient in single-node clusters, but in a multi-node setting, the inference isn’t efficient. This is because of the low bandwidth networks. 

Deploying a large language model requires multiple network requests to retrieve data from different servers. Network latency can impact the time required to transfer data between the servers, which can result in slower performance, eventually leading to high latency and response time. This can cause delays in processing, which can impact user experience.

4. Resource constraints

Limited storage capacity can restrict the ability to store multiple versions of the same model, which can make it difficult to compare the performance of different models and track the progress of model development over time. This can be true if you want to adopt a shadow deployment strategy.

Energy consumption

As discussed above already, serving LLMs require significant computational resources, which can lead to high energy consumption and a large carbon footprint. This can be problematic for organizations that are committed to reducing their environmental impact.

Just for reference, below is the image showing the financial estimation of the LLMs, along with the carbon footprint that they produce during training.

What is more shocking is that 80-90% of the machine learning workload is inference processing, according to NVIDIA. Likewise, according to AWS, inference accounts for 90% of machine learning demand in the cloud.

Cost

Deploying and using LLMs can be costly, including the cost of hardware, storage, and infrastructure. Additionally, the cost of deploying the model can be significant, especially when using resources such as GPUs or TPUs for low latency and high throughput during inference. This can make it challenging for smaller organizations or individuals to use LLMs for their applications.

To put this into perspective, it is expected that the running cost of the chatGPT is around $100,000 per day or $3M per month.

Strategies for optimizing infrastructure costs of large NLP models

In this section, we will explore and discuss the possible solutions and techniques for the challenges discussed in the previous section. It is worth noting that when you deploy the model on the cloud, you choose the inference option and thereby create an end-point. See the image below. 

Keep that in mind, and with all the challenges we discussed earlier, we will discuss techniques that can be used to optimize the cost around this infrastructure for deploying LLMs. Below are some of the steps that you can follow to deploy your model as efficiently as possible. 

Smart use of cloud computing for computational resources

Using cloud computing services can provide on-demand access to powerful computing resources, including CPUs and GPUs. Cloud computing services are flexible and can scale according to your requirements. 

One of the important tips is that you should make a budget for your project. Making a budget always helps you find ways to optimize your project that will not exceed your financial limitation. 

Now when it comes to cloud services, there are a lot of companies that offer their platform. Cloud providers such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform offer a range of options for deploying LLMs, including virtual machines, containers, and serverless computing. But despite you must do your own research and calculation. For instance, you must know these three things:

Once you have the details, you can actually calculate how much-accelerated computing power you need. Based upon that, you can plan and execute your model deployment. 

Calculating model size

You can see the table below, which will give you an idea of how many FLOPs you might need for your model. Once you have an estimation, you can then go ahead and find the relevant GPU in your preferred cloud platform. 

A tool that I found under the blog post named “Estimating Training Compute of Deep Learning Models”  allows you to calculate the FLOPs required for your model both for training and inference. 

The app is based on the works of Kaplan et al., 2020 or Hoffman et al., 2022 where they show how to train a model on a fixed-compute budget. To understand more on this subject you can read the blog here.

Selecting the right hardware

Once you have calculated the required FLOPs, you can go ahead and choose the GPU. Make sure you are aware of the features that the GPU offers. For instance, see the image below to get an understanding.

Above you can see the list of specifications that NVIDIA offers. Similarly, you can compare different GPUs and see which one suits your budget. 

Choosing the right inference option

Once you have calculated the model size and selected the GPU, you can then proceed to choose the inference option. Amazon SageMaker offers multiple inference options to suit different workloads. For instance, if you require:

1. Real-time inference, which is suitable for low-latency or high-throughput online inferences and supports payload sizes up to 6 MB and processing times of 60 seconds.
2. Serverless inference, which is ideal for intermittent or unpredictable traffic patterns and supports payload sizes up to 4 MB and processing times of 60 seconds. In serverless inference, the model scales automatically based on the incoming traffic or requests. At times when the model is sitting idle you won’t be charged. It offers a pay-as-you-use facility. 
3. Batch transform is suitable for offline processing of large datasets and supports payload sizes of GBs and processing times of days. 
4. Asynchronous inference is suitable for queuing requests with large payloads and long processing times, supports payloads up to 1 GB and processing times up to one hour, and can scale down to 0 when there are no requests.

To get a better understanding and meet your requirement, look at the image below. 

When all the above points are satisfied, you can then deploy the model on any of the cloud services. 

To quickly summarize:

Optimizing the model for serving

In the last section, I discussed how the size of LLMs can pose a problem for deployment. When your model is too large, strategies like model compilation, model compression, and model sharding can be used. These techniques reduce the size of the model while preserving accuracy, which allows easier deployment and reduce the associated expenses significantly. 

Let’s explore each of those in detail. 

Model compression

Model compression is a technique used to optimize and transform an LLM into an efficient executable model that can be run on specialized hardware or software platforms–usually cloud services. The goal of model compression is to improve the performance and efficiency of LLM inference by leveraging hardware-specific optimizations, such as reduced memory footprint, improved computation parallelism, and reduced latency.

This is a good technique because it helps you to play with a different combination, set performance benchmarks for various tasks, and find a price that suits your budget.  As such, model compression involves several steps:

1. Graph optimization: The high-level LLM graph is transformed and optimized using graph optimization techniques such as pruning and quantization to reduce the computational complexity and memory footprint of the model. This, in turn, makes the model small while preserving its accuracy. 
2. Hardware-specific optimization: The optimized LLM graph is further optimized to leverage hardware-specific optimizations. For instance, Amazon Sagemaker provides model serving containers for various popular ML frameworks, including XGBoost, scikit-learn, PyTorch, TensorFlow, and Apache MXNet, along with software development kits (SDKs) for each container.

Here are a few model compression techniques that one must know.

Model quantization

Model quantization (MQ) is a technique used to reduce the memory footprint and computation requirements of an LLM. MQ essentially transforms the model parameters and activations with lower-precision data types. The goal of model quantization is to improve the efficiency of LLM during inference by reducing the memory bandwidth requirements and exploiting hardware-specific optimizations optimized for lower-precision arithmetic.

PyTorch offers model quantization, their API involves the reduction of model parameters by a factor of 4, while the memory bandwidth required by the model by the factor 2 to 4 times. As a result of these improvements, the inference speed can increase by 2 to 4 times, owing to the reduction in memory bandwidth requirements and faster computations using int8 arithmetic. However, the precise degree of acceleration achieved depends on the hardware, runtime, and model used.

There are several approaches to model quantization for LLMs, including:

Model quantization can be challenging to implement effectively, as it requires careful consideration of the trade-offs between reduced precision and model accuracy, as well as the hardware-specific optimizations that can be leveraged with lower-precision arithmetic. However, when done correctly, model quantization can significantly improve the efficiency of LLM inference, enabling better real-time inference on large-scale datasets and edge devices.

1. Post-training quantization: In this approach, the LLM is first trained using floating-point data types, and then the weights and activations are quantized to lower-precision data types post-training. This approach is simple to implement and can achieve good accuracy with a careful selection of quantization parameters.
2. Quantization-aware training: Here, the LLM is quantized during training, allowing the model to adapt to the reduced precision during training. This approach can achieve higher accuracy than post-training quantization but requires more computation during training.
3. Hybrid quantization: It combines both post-training quantization and quantization-aware training, allowing the LLM to adapt to lower-precision data types during training while also applying post-training quantization to further reduce the memory footprint and computational complexity of the model.

Model Pruning

Model pruning (MP) is again a technique used to reduce the size and computational complexity of an LLM by removing redundant or unnecessary model parameters. MP is to improve the efficiency of LLM inference without sacrificing accuracy.

MP involves identifying and removing redundant or unnecessary model parameters using various pruning algorithms. These algorithms can be broadly categorized into two categories:

1. Weight pruning: In weight pruning, individual weights in the LLM are removed based on their magnitude or importance, using techniques such as magnitude-based pruning or structured pruning. Weight pruning can significantly reduce the number of model parameters and the computational complexity of the LLM, but it may require fine-tuning of the pruned model to maintain its accuracy.
2. Neuron pruning: In neuron pruning, entire neurons or activations in the LLM are removed based on their importance, using techniques such as channel pruning or neuron-level pruning. Neuron pruning can also significantly reduce the number of model parameters and the computational complexity of the LLM, but it may be more difficult to implement and may require more extensive retraining and maybe fine-tuning to maintain accuracy.

Here are a couple of approaches to model pruning:

1. Post-training pruning: In this approach, the LLM is first trained using standard techniques and then pruned using one of the pruning algorithms. The pruned LLM is then fine-tuned to preserve its accuracy.
2. Iterative pruning: Here, the model is trained using standard training techniques and then pruned iteratively over several rounds of training and pruning. This approach can achieve higher levels of pruning while preserving accuracy.

You can explore this Colab notebook by PyTorch to better understand MP. 

Model distillation

(MD) is a technique used to transfer knowledge from an LLM called a teacher to a smaller, more efficient model called the student. It is used in the context of model compression. In a nutshell, the teacher model provides guidance and feedback to the student model during training. See the image below.

MD involves training a student, a more efficient model to mimic the behavior of a teacher, more complex LLM. The student model is prepared using a combination of labeled data and the output probabilities of the larger LLM. 

There are several approaches to model distillation for LLMs, including:

1. Knowledge distillation: In this approach, the smaller model is trained to mimic the output probabilities of the larger LLM using a temperature scaling factor. The temperature scaling factor is used to soften the output probabilities of the teacher model, allowing the smaller model to learn from the teacher model’s behavior more effectively.

2. Self-distillation: In this approach, the larger LLM is used to generate training examples for the smaller model by applying the teacher model to unlabeled data. The smaller model is then trained on these generated examples, allowing it to learn from the behavior of the larger LLM without requiring labeled data.

3. Ensemble distillation: In this approach, multiple smaller models are trained to mimic the behavior of different sub-components of the larger LLM. The outputs of these smaller models are combined to form an ensemble model that approximates the behavior of the larger LLM.

Optimizing hardware and software requirements

Hardware is an important area when it comes to deploying LLMs. Here are some useful steps you can take for optimizing the hardware performance:

1. Choose hardware that matches the LLM’s requirements: Depending on the LLM’s size and complexity, you may need hardware with a large amount of RAM, high-speed storage, or multiple GPUs to speed up inference. Opt for hardware that provides the necessary processing power, memory, and storage capacity, without overspending on irrelevant features.

2. Use specialized hardware: You can use specialized hardware such as TPUs (Tensor Processing Units) or FPGAs (Field-Programmable Gate Arrays) that are designed specifically for deep learning tasks. Similarly, accelerated linear algebra or XLA can be leveraged during inference time. 

Although such hardware can be expensive, there are smart ways to consume them. You can opt for charge-on-demand for the hardware used. For instance, elastic Inference from AWS Sagemaker helps you lower your cost when the model is not fully utilizing the GPU instance for inference. 

3. Use optimized libraries: You can use optimized libraries such as TensorFlow, PyTorch, or JAX  that leverage hardware-specific features to speed up computation without needing additional hardware. 

4. Tune the batch size: Consider tuning the batch size during inference to maximize hardware utilization and improve inference speed. This inherently reduces the hardware requirement, thus cutting the cost. 

5. Monitor and optimize: Finally, monitor the LLM’s performance during deployment and optimize the hardware configuration as needed to achieve the best performance.

Cost efficient scalability

Here’s how you can scale your large NLP models while keeping costs in check:

1. Choose the right inference option, that scales automatically like the serverless inference option. As it will reduce the deployment cost when the demand is less. 

A rigid architecture will always occupy the same amount of memory even when the demand is low thus the deployment and maintenance costs will be the same. On the contrary, a scalable architecture can scale horizontally or vertically to accommodate an increased workload and go back to its original configuration when the model lies in a dormant state. Such an approach can reduce the cost of maintenance whenever the additional nodes are not being used. 

2. Optimize inference performance, by using hardware acceleration, such as GPUs or TPUs, and by optimizing the inference code.

3. Amazon’s Elastic inference is yet another great option as it reduces the cost by up to 75% because the model no longer has extra GPUs to compute for inference. For more on Elastic inference, read this article here. 

Cutting energy costs

1. Choose an energy-efficient cloud infrastructure, that uses renewable energy sources or carbon offsets to reduce the carbon footprint of their data centers. You can also consider choosing energy-efficient GPUs. Check out this article by Wired to understand more. 

2. Use caching which helps reduce the computational requirements of LLM inference by storing frequently requested responses in memory. This can significantly reduce the number of computations required to generate responses to user requests. It also helps in addressing bandwidth issues as it reduces the time to access data. You can store frequently accessed data in cache memory so that it can be quickly accessed without the need for additional bandwidth. This allows you not to opt for additional storage and memory devices. 

Deploying large NLP models: other useful tips

Estimating the NLP model size before training

Keeping your model size in check could in turn keep your infrastructure costs in check. Here are a few things you can keep in mind while getting your large NLP model ready.

1. Consider the available resources: The size of the LLM for deployment should take into account the available hardware resources, including memory, processing power, and storage capacity. The LLM’s size should be within the limits of the available resources to ensure optimal performance.
2. Fine-tuning: Choose a model with optimal accuracy and then fine-tune it on a task-specific dataset. This step will increase the efficiency of the LLM and keep its size from spiralling out of control.
3. Consider the tradeoff between size and performance: The LLM’s size should be selected based on the tradeoff between size and performance. A larger model size may provide better performance but may also require more resources and time. Therefore, it is essential to find the optimal balance between size and performance.

Use a lightweight deployment framework

Many LLMs are too large to be deployed directly to a production environment. Consider using a lightweight deployment framework like TensorFlow Serving or TorchServe that can host the model and serve predictions over a network. These frameworks can help reduce the overhead of loading and running the model on the server thereby reducing the deployment and infrastructure costs.

Post-deployment model monitoring

Model monitoring helps optimize the infrastructure cost of deployment by providing insights into the performance and resource utilization of deployed models. By monitoring the resource consumption of deployed models, such as CPU, memory, and network usage, you can identify areas that can help you optimize your infrastructure usage to reduce costs. 

• Monitoring can identify underutilized resources, allowing you to scale back on unused resources, and reducing infrastructure costs. 
• Monitoring can identify resource-intensive operations or models, enabling organizations to optimize their architecture or refactor the model to be more efficient. This can also lead to cost savings. 

Key takeaways

Conclusion

In this article, we explored the challenges we face when deploying an LLM and the inflated infrastructural cost associated with them. Simultaneously, we also addressed each of these difficulties with the necessary techniques and solutions. 

Out of all the solutions we discussed, a couple of things that I would recommend the most when it comes to reducing infrastructure cost while deployment is elastic and serverless inference. Yes, model compression is good and valid, but when the demand is high, even the smaller model can act like a larger model, thus increasing the infrastructural cost. Thus, we need to have a scalable approach and pay-per-demand service. That’s where these inference services get handy. 

It goes without saying that my recommendation might not be the most ideal for your use case, and you can pick any of these approaches depending on the kind of problems you are dealing with. I hope what we discussed here will go a long way in helping you cut down your deployment infrastructure costs for your large NLP models. 

References

1. Large Language Model Training in 2023
2. https://d1.awsstatic.com/events/Summits/reinvent2022/AIM405_Train-and-deploy-large-language-models-on-Amazon-SageMaker.pdf
3. Top 10 AI Chip Makers of 2023: In-depth Guide 
4. https://www.nvidia.com/en-us/data-center/dgx-a100/
5. LLaMA: A foundational, 65-billion-parameter large language model
6. https://arxiv.org/pdf/2203.15556.pdf
7. https://huggingface.co/docs/transformers/model_doc
8. https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast
9. https://sunniesuhyoung.github.io/files/LLM.pdf
10. https://twitter.com/tomgoldsteincs/status/1600196995389366274?lang=en
11. https://arxiv.org/pdf/1910.02054.pdf
12. https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html
13. Jaime Sevilla et al. (2022), “Estimating Training Compute of Deep Learning Models”. Published online at epochai.org. Retrieved from: ‘https://epochai.org/blog/estimating-training-compute‘ [online resource]
14. https://arxiv.org/abs/2001.08361
15. https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf
16. https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html
17. https://aws.amazon.com/sagemaker/neo/
18. https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/7126bf7beed4c4c3a05bcc2dac8baa3c/pruning_tutorial.ipynb
19. https://towardsdatascience.com/distillation-of-bert-like-models-the-code-73c31e8c2b0a
20. https://aws.amazon.com/blogs/machine-learning/train-175-billion-parameter-nlp-models-with-model-parallel-additions-and-hugging-face-on-amazon-sagemaker/
21. Improving Language Model Behavior by Training on a Curated Dataset
22. https://towardsdatascience.com/how-to-deploy-large-size-deep-learning-models-into-production-66b851d17f33
23. https://huggingface.co/blog/large-language-models
24. https://aws.amazon.com/blogs/machine-learning/deploy-large-models-on-amazon-sagemaker-using-djlserving-and-deepspeed-model-parallel-inference/
25. Large Language Models Can Self-Improve
26. https://spot.io/resources/cloud-cost/cloud-cost-optimization-15-ways-to-optimize-your-cloud/
27. https://dataintegration.info/choose-the-best-ai-accelerator-and-model-compilation-for-computer-vision-inference-with-amazon-sagemaker
28. https://medium.com/data-science-at-microsoft/model-compression-and-optimization-why-think-bigger-when-you-can-think-smaller-216ec096f68b
29. https://medium.com/picsellia/how-to-optimize-computer-vision-models-for-edge-devices-851b20f7cf03
30. https://huggingface.co/docs/transformers/v4.17.0/en/parallelism#which-strategy-to-use-when
31. https://medium.com/@mlblogging.k/9-libraries-for-parallel-distributed-training-inference-of-deep-learning-models-5faa86199c1f
32. https://towardsdatascience.com/how-to-estimate-and-reduce-the-carbon-footprint-of-machine-learning-models-49f24510880

These systems can be broadly divided into five phases, keeping in mind that these phases contain various additional and repetitive tasks:

Executing these phases individually can take a lot of time and continuous human effort. These phases must be synchronized and sequentially orchestrated in order to get the best out of them. This can be achieved by task orchestration tools that enable ML practitioners to effortlessly bring together and orchestrate different phases of an AI system.

In this article, we will explore:

Task orchestration tools: What they are and how are they useful?

Orchestration tools enable various tasks in MLOps to be organized and sequentially executed. These tools have the capability to orchestrate different tasks at a given period. One of the key properties of these tools is the distribution of tasks. Most of the tools leverage what is known as the DAG or Directed Acyclic Graph, which you will often come across in this article. A DAG is a graph representation of the tasks that need to be executed. 

DAG enables tasks in a pipeline to be distributed parallelly to various other modules for processing, this offers efficiency. See the image above. DAG also enables tasks to be sequentially sound or arranged for proper execution and timely results.

Another important property that these tools have is adaptability to agile environments. This allows ML practitioners to incorporate various other tools that can be used to monitor, deploy, analyze and preprocess, test, infer, et cetera. If an orchestration tool can orchestrate various tasks from different tools, then it can be considered a good tool. But this is not the case every time, some of the tools are strictly contained within their derived environments, which does not bode well for users trying to integrate any third-party applications. 

In this article, we will explore three tools – Argo, Airflow, and Prefect, that incorporate these two properties and various others as well. 

TL;DR comparison table 

Here is a table inspired by Ian McGraw’s article, which provides an overview of what these tools offer for orchestration and how they differ from each other in these aspects.

	Features	Argo	Airflow	Prefect
1.	Fault-tolerant scheduling
2.	UI Support
3.	Workflow definition language	YAML	Python	Python
4.	3rd partyintegration	Since Argo is container-based it doesn’t come with pre-installed 3rd party systems.	Supports various 3rd party integration	Supports various 3rd party integration
5.	Workflows	Dynamic workflow	Static workflow	Dynamic workflow
6.	Accessibility	Open-sourced	Open-source	Hybrid (Open-sourced and subscription-based)
7.	Parametrized workflows	Have an extensive parameter-passing syntax.	Does not has a mechanism to pass parameter.	Supports parameters as first-class object
8.	Kubernetes support
9.	Scalability	Highly Parallel	Horizontal scalable	Parallel when using Kubernetes
10.	Community Support	Large	Large	Medium
11.	State storage	All states are stored within the Kubernetes workflow	Postgres DB	Postgres DB
12.	Ease of deployment	Medium	Medium	Difficult
13.	Event-driven workflows
14.	Scripts in DAG definition	Argo uses text scripts to pass in containers.	Airflow uses Python-based DAG definition language.	Perfect uses functional flow a Python-based API.
15.	Use Cases	– CI/CD- Data Processing- Infrastructure  Automation- Machine Learning- Stream Processing	– ELT – ML Workflow- ML Automation	– Automating Data Workflow (ELT)- ML Workflow and Orchestration- CI/CD

Now let’s explore each of these tools in more detail under three primary categories: 

Core concepts

All three tools are built on a set of concepts or principles around which they function. Argo is, for instance, built around two concepts: Workflow and Templates. Both of these make the backbone of its system. Likewise, Airflow is built around Webserver, Scheduler, Executor, and Database, while Prefect is built around Flows and Task. Now it is important for us to know what these concepts mean, what they offer, and how it is beneficial to us.

Before going into the details, here is a brief summary of the concepts. 

Summary of the concepts

Now, let’s understand these concepts in detail. 

Argo 

Argo uses two core concepts:

1. Workflow
2. Templates

Workflow

In Argo, the workflow happens to be the most integral component of the whole system. It has two important functions: 

1. It defines the tasks that need to be executed.
2. It stores the state of the tasks, which means that it serves as both a static and a dynamic object.

Workflow is defined in the workflow.spec configuration file. It is a YAML file that consists of a list of templates and entry points. The Workflow can be considered as a file that hosts different templates. These templates define the function that needs to be executed. 

As mentioned earlier that Argo leverages the Kubernetes engine for workflow synchronization, and the configuration file uses the same syntax as Kubernetes. The workflow YAML file has the following dictionaries or objects:

1. apiVersion: This is where you define the name of the doc or API.
2. kind: It defines the type of Kubernetes object that needs to be created. For instance, if you want to deploy an app you can use Deployment as one of a kind, at other times you can use service. But in this case, we will use Workflow.
3. metadata: It enables us to define unique properties for that object, that could be a name, UUID, et cetera. 
4. spec: It enables us to define specifications concerning the Workflow. These specifications would be entry points and templates. 
5. templates: This is where we can define the tasks. The template can contain the docker image and various other scripts. 

Templates 

In Argo, there are two types of templates which again are sub-classified into 6 types. The two major types are definition and invocators. 

Definition

This template, as the name suggests, defines the type of task in a Docker container. The Definition itself is divided into four categories:

1. Container: It enables users to schedule the workflow in a container. Since the application is containerized in Kubernetes, the steps defined in the YAML file are identical. It is also one of the most used templates.

#source: https://argoproj.github.io/argo-workflows/workflow-concepts/
- name: whalesay
    container:
      image: docker/whalesay
      command: [cowsay]
      args: ["hello world"]

2. Script: If you want a wrapper around a container, then the script template is perfect. The script template is similar in structure to the container template but adds a source field. The field allows you to define a script in place. You can define any variable or command based on your requirements. Once defined, the script will be saved into a file, and it will be executed for you as an Argo variable.

#source: https://argoproj.github.io/argo-workflows/workflow-concepts/
 - name: gen-random-int
    script:
      image: python:alpine3.6
      command: [python]
      source: import random

        	i = random.randint(1, 100)
        	  	print(i)

3. Resource: It allows you to perform operations like get, create, apply, delete et cetera on the K8 cluster directly.

#source: https://argoproj.github.io/argo-workflows/workflow-concepts/
- name: k8s-owner-reference
    resource:
      action: create
      manifest: |
        apiVersion: v1
        kind: ConfigMap
        metadata:
          generateName: owned-eg-
        data:
          some: value

4. Suspend: It basically introduces a time dimension to the workflow. It can suspend the execution of the workflow for a defined duration or till the workflow is resumed manually. 

#source: https://argoproj.github.io/argo-workflows/workflow-concepts/ 
 - name: delay
    suspend:
      duration: "20s"

Invocators

Once the templates are defined, they can be invoked or called on demand by other templates called invocators. These invocators are more of controllers templates that can control the execution of defined templates. 

There are two types of invocator templates:

1. Steps: It basically allows you to define the tasks in steps. All YAML files are enabled with the ‘steps’ template. 
2. Directed acyclic graph: Argo enables its users to manage steps with multiple dependencies in their workflow. This allows parallel execution of different workflows in their respective containers. These types of workflows are managed using a directed acyclic graph or DAG. For instance, if you are working on image segmentation and generation for medical purposes then you can create a pipeline that:
   • Processes the images.
   • Distributes the images (or dataset) to the respective DL models for image segmentation and generation pipeline.
   • Continuously predicts segmentation masks and updates the dataset storage with new images after proper inspection. 

Airflow

Apache Airflow consists of four main components:

1. Webserver
2. Scheduler
3. Executor
4. Database

Webserver

It provides the user with UI for inspecting, triggering, and debugging all DAGs and tasks. It essentially serves as the entry point for Airflow. The Webserver leverages Python-Flask to manage all the requests made by the user. It also renders the state metadata from the database and displays the same to the UI.

Scheduler

It monitors and manages all the tasks and DAGs. It examines the state of the tasks by querying the database to decide the order of the task that needs to be executed. The aim of the scheduler is then to resolve dependencies and submit the task instance to the executor once the dependencies are taken care of.

Executor

It runs the task instances which are ready to run. It executes all the tasks as scheduled by the scheduler. There are four types of executors:

1. Sequential Executor
2. Local Executor
3. Celery Executor
4. Kubernetes Executor

Metadata Database

It stores the state of the tasks and DAGs that can be used by the scheduler for proper scheduling of the tasks instance. It is worth noting that Airflow uses SQLAlchemy and Object Relational Mapping (ORM) to store the information. 

Prefect

Prefect uses two core concepts: 

1. Flows
2. Tasks

Flows

In Prefect, flows are the Python objects that can be interacted with. Here DAG is defined as flow objects. See the image below. 

Flow can be imported and can be used as a decorator, @flow, for any given function. Flows take an existing function and transform it into a Prefect flow function, with the following advantages:

• The function can be monitored and governed as it is now reported to the API.
• The activity of the function can be tracked and displayed in the UI.
• Inputs given to the function can be validated.
• Various workflow features like retries, distributed execution et cetera can be added to the function. 
• Timeouts can be enforced to prevent unintentional long-running workflows 

Here is a code block depicting the implementation of a flow object.

#Source: https://github.com/PrefectHQ/prefect
from prefect import flow

@flow(name="GitHub Stars")
def github_stars(repos: List[str]):
    for repo in repos:
        get_stars(repo)

In the code above, the function has been transformed into a flow which is named as “GitHub Stars”. This function is now within the constraints of Prefect orchestration laws. 

Now it must be noted that all workflows must be defined within the flow function. Likewise, all tasks must be called within the flow (function). Keep in mind that when a flow is executed, it is known as a flow run. 

Tasks

Tasks can be defined as specific work that needs to be executed, for instance, the addition of two numbers. In another word, tasks take an input, perform an operation and yield an output. Like flow, tasks can be imported and can be used as a decorator, @task, for a function. Once used for a function, it essentially wraps the function within the Prefect workflow and has similar advantages to the flow. For instance, it can automatically log information about task runs, such as runtime, tags, and final state. 

The code below demonstrates how a task is defined: 

#Source: https://github.com/PrefectHQ/prefect

@task(retries=3)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")

# run the flow!
github_stars(["PrefectHQ/Prefect"])

To sum up, the flow looks for any task that is defined within its body, and once found it then creates a computational graph in the same order. It then creates dependencies between the tasks whenever the output of one task instance is used to yield output by another. 

Features

All three provide more or less the same features, but some features are better than others, and it also boils down to users’ adaptability. Just like in the previous section, let’s begin with a summary of the features. 

Comparison of the features

Let’s start this section by exploring the User Interface. 

User Interface

Argo

For ease of use, Argo Workflow provides a web-based UI to define workflows and templates. The UI enables various purposes like:

• Artifact visualization 
• Using generated charts to compare Machine Learning pipelines
• Visualizing results 
• Debugging
• It can also be used to define workflows

Airflow

Airflow UI provides a clean and efficient design that enables the user to interact with the Airflow server allowing them to monitor and troubleshoot the entire pipeline. It also allows editing the state of the task in the database and manipulating the behaviour of DAGs and tasks. 

The Airflow UI also provides various views for its users, they include:

• DAGs View
• Datasets View
• Grid View
• Graph View
• Calendar View
• Variable View
• Gantt View
• Task Duration
• Code View

Prefect

Prefect like Airflow provides an overview of all the tasks, which helps you visualize all your workflow, tasks, and DAGs. It provides two ways to access UI:

1. Prefect Cloud: It is hosted on the cloud, which enables you to configure your personal accounts and workspaces. 
2. Prefect Orion UI: It is hosted locally, and it is also open-sourced. You cannot configure it the way you can with Prefect cloud. 

Some additional features of Prefect UI:

• Displaying run summaries
• Displaying flow details that are deployed
• Scheduled flow 
• Warnings notification for late and failed runs
• Details information of tasks and workflows
• Task dependency visualization and Radar flow
• Logs details

Deployment Style

Argo

It is a native Kubernetes workflow engine which means it:

On the downside:

• Implementation is hard since it uses configurational language (YAML).

Airflow

The downside of Airflow is:

• It is not parallel scalable.
• Deployment needs extra effort, which depends upon the cloud facility you choose. 

Prefect

Lastly, Prefect is a combination of both Argo and Airflow:

When it comes to the downside:

• It does not support open-source deployment with Kubernetes. 
• Deployment is difficult. 

Scalability

When it comes to scalability, Argo and Prefect are highly parallel, which makes them efficient and especially Prefect because it can leverage different third-party integrations support, making it the best of the three. 

Airflow, on the other, is horizontally scalable i.e., the number of active workers is equal to maximum task parallelism. 

Accessibility

All three are open-sourced, but Prefect also comes with a subscription-based service. 

Flexibility

Argo and Airflow aren’t that flexible when compared with Prefect as the former is Kubernetes-native it is confined in that environment, making it rigid, while the latter is complicated as it requires a well-defined and structured template, making itself not very well suited to an agile environment. 

Prefect, on the other hand, enables you to create dynamic dataflow in native Python, which does not require you to use DAG. All Python functions can be transformed to Prefect Flow and Task. This ensures flexibility.

Why use these tools?

So far, I’ve compared the basic concepts and features that these tools possess. Now let me give reasons as to why you can use any of these tools in your project.  

Argo

Here are some of the reasons why you should use Argo:

Airflow

Reasons for you to use Airflow:

Prefect

Prefect is one of the well-planned orchestration tools for MLops. It is Python-native and requires you to put effort into the engineering side of things. One of the areas where Prefect shines is in data processing and pipeline. It can be used to fetch the data, apply the necessary transformation, and monitor and orchestrate necessary tasks.

When it comes to tasks related to machine learning, it can be used to automate the entire data flow. 

Some other reasons to use Prefect are:

How to decide?

Choosing the right tool for your project depends on what you want and what you already have. But I can surely put some criteria that can help you decide which tool will be appropriate for you. You can use –

Argo

• If you want to set up a workflow based on Kubernetes.
• If you want to define your workflow as DAGs.
• If your dataset is huge and model training requires highly parallel and distributed training. 
• If your task is complex.
• If you are well-versed in YAML files. Even if you are not, learning YAML is not difficult.
• If you want to use a cloud platform like GCD or AWS, which is Kubernetes enabled. 

Airflow

• If you want to incorporate a lot of other 3rd party technology like Jenkins, Airbyte, Amazon, Cassandra, Docker, et cetera. Check the list of supported third-party extensions.
• If you want to use Python to define the workflow.
• If you want to define your workflow as DAGs.
• If your workflow is static.
• If you want a mature tool because Airflow is quite old. 
• If you want to run tasks on schedule.

Prefect

• If you want to incorporate a lot of other 3rd party technology.
• If you want to use Python to define the workflow.
• If your workflow is dynamic.
• If you want to run tasks on schedule.
• If you want something light and modern.

I found a thread on Reddit concerning the use of Airflow and Prefect. Maybe this can give you some additional information as to which tool to use.

“…The pros of Airflow are that it’s an established and popular project. This means it’s much easier to find someone who has done a random blog that answers your question. Another pro is that it’s much easier to hire someone with Airflow experience than Prefect experience. The cons are that Airflow’s age is showing, in that it wasn’t really designed for the kind of dynamic workflows that exist within modern data environments. If your company is going to be pushing the limits in terms of computation or complexity, I’d highly suggest looking at Prefect. Additionally, unless you go through Astronomer, if you can’t find an answer to a question you have about Airflow, you have to go through their fairly inactive slack chat.

The pros of Prefect are that it’s much more modern in its assumptions about what you’re doing and what it needs to do. It has an extensive API that allows you to programmatically control executions or otherwise interact with the scheduler, which I believe Airflow has only recently implemented out of beta in their 2.0 release. Prior to this, it was recommended not to use the API in production, which often leads to hacky workarounds. In addition, Prefect allows for a much more dynamic execution model with some of its concepts by determining the DAG that gets executed at runtime and then handing off the computation/optimization to other systems (namely Dask) to actually execute the tasks. I believe this is a much smarter approach, as I’ve seen workflows get more and more dynamic over the years.

If my company had neither Airflow nor Prefect in place already, I’d opt for Prefect. I believe it allows for much better modularization of code (which can then be tested more aggressively / thoroughly), which I already think is worth its weight in gold for data-driven companies that rely on having well-curated data in place to make automated product decisions. You can achieve something similar with Airflow, but you really need to go out of your way to make something like that happen, whereas in Prefect it kind of naturally comes out.” 

Here is a useful chart illustrating the popularity of different orchestration tools based on GitHub stars.

Conclusion

In this article, we discussed and compared the three popular tools for task orchestration, namely Argo, Airflow, and Prefect. My main aim was to help you understand these tools on the basis of three important factors i.e. Core concepts, Features offered, and why you should use them. The article also compared the three tools on some of the important features they offer, which could help you make the decision of choosing the most appropriate tool for your project.

I hope this article was informative and gave you a better understanding of these tools. 

Thanks!!! 

References

1. https://github.com/argoproj/argo-workflows 
2. https://argoproj.github.io/ 
3. https://codefresh.io/learn/argo-workflows/ 
4. https://hazelcast.com/glossary/directed-acyclic-graph/
5. https://towardsdatascience.com/mlops-with-a-feature-store-816cfa5966e9 
6. https://medium.com/arthur-engineering/picking-a-kubernetes-orchestrator-airflow-argo-and-prefect-83539ecc69b
7. https://argoproj.github.io/argo-workflows/artifact-visualization/#artifact-types 
8. https://airflow.apache.org/docs/apache-airflow/stable/concepts/overview.html 
9. https://spell.ml/blog/orchestrating-spell-model-pipelines-using-prefect-YU3rsBEAACEAmRxp 
10. https://github.com/PrefectHQ/prefect
11. https://www.datarevenue.com/en-blog/airflow-vs-luigi-vs-argo-vs-mlflow-vs-kubeflow 
12. https://hevodata.com/learn/argo-vs-airflow/#w6 
13. https://www.datarevenue.com/en-blog/what-we-are-loving-about-prefect 
14. https://github.com/PrefectHQ/prefect 
15. https://docs.prefect.io/ 
16. https://medium.com/the-prefect-blog/introducing-the-artifacts-api-b9e5972db043 
17. https://medium.com/the-prefect-blog/orchestrate-your-data-science-project-with-prefect-2-0-4118418fd7ce 
18. https://www.reddit.com/r/dataengineering/comments/oqbiiu/airflow_vs_prefect/

The difficulty arises when we want to put the model into production in the real world. The model often does not perform as well as it did during the training and evaluation phase. This happens primarily because of concept drift or data drift and issues concerning data integrity. Therefore, testing an ML model becomes very important so that we can understand its strengths and weaknesses and act accordingly. 

In this article, we will discuss some of the tools that can be leveraged to test an ML model. Some of these tools and libraries are open-source, while others require a subscription. Either way, this article will fully explore the tools which will be handy for your MLOps pipeline. 

Why does model testing matter?

Building upon what we just discussed, model testing allows you to pinpoint a bug or area of concern that might cause the prediction capability of the model to degrade. This can happen over time gradually or in an instant. Either way, it is always good to know in which area they might fail and which features can cause them to fail. It exposes flaws, and it can also bring new insights to light. Essentially, the idea is to make a robust model that can efficiently handle uncertain data entries and anomalies. 

Some of the benefits of model testing are:

Is model testing the same as model evaluation?

Model testing and evaluation are similar to what we call diagnosis and screening in medicine. 

Model evaluation is similar to diagnosis, where the performance of the model is checked based upon certain metrics like F1 score or MSE loss. These metrics do not provide a focused area of concern. 

Model testing is similar to diagnosis, where a certain test like the invariance test and unit test aims to find a particular issue in the model. 

What will a typical ML software testing suite include?

A machine learning testing suite often includes testing modules to detect different types of drifts like concept drift and data drift, which can include covariant drift, prediction drift, and so on. These issues usually occur within the dataset. Most of the time, the dataset’s distribution changes over time, affecting the model’s capability to accurately predict the output. You will find that the frameworks we will discuss will contain tools to detect data drifts. 

Apart from testing data, the ML testing suite contains tools to test the model’s capability to predict, as well as overfitting, underfitting, variance and bias et cetera. The idea of the testing framework is to inspect the pipeline in the three major phases of development:

• data ingestion,
• data preprocessing,
• and model evaluation.

Some of the frameworks like Robust Intelligence and Kolena rigorously test the given ML pipeline automatically in these given areas to ensure a production-ready model. 

In essence, a machine learning suite will contain:

1. Unit tests that operate on the level of the codebase,
2. Regression tests replicate bugs from the previous iteration of the model that is fixed,
3. Integration tests simulate conditions and are typically longer-running tests that observe model behaviors. These conditions can mirror the ML pipeline, including preprocessing phase, data distribution, et cetera. 

What are the best tools for machine learning model testing?

Now, let’s discuss some of the tools for testing ML models. This section is divided into three parts: open-source tools, subscription-based tools, and hybrid tools. 

Open-source model testing tools

1. DeepChecks

DeepChecks is an open-source Python framework for testing ML Models & Data. It basically enables users to test the ML pipeline in three different phases:

1. Data integrity test before the preprocessing phase.
2. Data Validation, before the training, mostly while splitting the data into training and testing, and
3. ML model testing.

These tests can be performed all at once and even independently. The image above shows the schema of three different tests that could be performed in an ML pipeline. 

Installation

Deepchecks can be installed using following the pip command:

pip install deepchecks > 0.5.0

The latest version of Deepcheck is 0.8.0. 

Structure of the framework 

DeepChecks introduces three important terms: Check, Condition and Suite. It is worth noting that these three terms together form the core structure of the framework. 

Check

It enables a user to inspect a specific aspect of the data and models. The framework contains various classes which allow you to check both of them. You can do a full check as well. Here are a couple of such checks:

1. Data inspecting involves inspection around data drift, duplication, missing values, string mismatch, statistical analysis such as data distribution et cetera. You can find the various data inspecting tools within the check module. The check module allows you to precisely design the inspecting methods for your datasets. These are some of the tools that you will find for data inspection:

•  ‘DataDuplicates’,
•  ‘DatasetsSizeComparison’,
•  ‘DateTrainTestLeakageDuplicates’,
•  ‘DateTrainTestLeakageOverlap’,
•  ‘DominantFrequencyChange’,
•  ‘FeatureFeatureCorrelation’,
•  ‘FeatureLabelCorrelation’,
•  ‘FeatureLabelCorrelationChange’,
•  ‘IdentifierLabelCorrelation’,
•  ‘IndexTrainTestLeakage’,
•  ‘IsSingleValue’,
•  ‘MixedDataTypes’,
•  ‘MixedNulls’,
•  ‘WholeDatasetDrift’

In the following example, we will inspect whether the dataset has duplicates or not. We will import the class DataDuplicates from the checks module and pass the dataset as a parameter. This will return a table containing relevant information on whether the dataset has duplicate values or not. 

from deepchecks.checks import DataDuplicates, FeatureFeatureCorrelation
dup = DataDuplicates()
dup.run(data)

As you can see, the table above yields relative information about the number of duplicates present in the dataset. Now let’s see how DeepChecks uses a visual aid to provide the concerning information. 

In the following example, we will inspect feature-feature correlation within the dataset. For that, we will import the FeatureFeatureCorrelation class from the checks module.

ffc = FeatureFeatureCorrelation()
ffc.run(data)

As you can see from both examples, the results can be displayed either in the form of a table or a graph, or even both to give relevant information to the user.  

2. The model inspection involves overfitting, underfitting, et cetera. Similar to data inspection, you can also find the various model inspecting tools within the check module. These are some of the tools that you will find for model inspection:

• ‘ModelErrorAnalysis’,
•  ‘ModelInferenceTime’,
•  ‘ModelInfo’,
•  ‘MultiModelPerformanceReport’,
•  ‘NewLabelTrainTest’,
•  ‘OutlierSampleDetection’,
•  ‘PerformanceReport’,
•  ‘RegressionErrorDistribution’,
•  ‘RegressionSystematicError’,
•  ‘RocReport’,
•  ‘SegmentPerformance’,
•  ‘SimpleModelComparison’,
•  ‘SingleDatasetPerformance’,
•  ‘SpecialCharacters’,
•  ‘StringLengthOutOfBounds’,
•  ‘StringMismatch’,
•  ‘StringMismatchComparison’,
•  ‘TrainTestFeatureDrift’,
•  ‘TrainTestLabelDrift’,
•  ‘TrainTestPerformance’,
•  ‘TrainTestPredictionDrift’,

Example of a model check or inspection on Random Forest Classifier:

from deepchecks.checks import ModelInfo
info = ModelInfo()
info.run(RF)

Condition 

It is a function or attribute that can be added to a Check. Essentially it contains a predefined parameter that can return a pass, fail, or warning results. These parameters can be modified as well accordingly. Follow the code snippet below to get an understanding. 

from deepchecks.checks import ModelInfo
info = ModelInfo()
info.run(RF)

The image above shows a bar graph of feature label correlation. It essentially measures the predictive power of an independent feature that can predict the target value by itself. When you add a condition to a check as in the example above, the condition will return additional information mentioning the features which are above and below the condition. 

In this particular example, you will find that the condition returned a statement stating that the algorithm “Found 2 out of 4 features with PPS above threshold: {‘petal width (cm)’: ‘0.9’, ‘petal length (cm)’: ‘0.87’}” meaning that features with high PPS are suitable to predict the labels. 

Suite 

It is a module containing a collection of checks for data and model. It is an ordered collection of checks. All the checks can be found in the suite module. Below is the schematic diagram of the framework and how it works. 

As you can see from the image above, the data and the model can be passed into the suites which contain the different checks. The checks can be provided with the conditions for much more precise testing. 

You can run the following code to see the list of 35 checks and their conditions that DeepChecks provides:

from deepchecks.suites import full_suite
suites = full_suite()
print(suites)
Full Suite: [
	0: ModelInfo
	1: ColumnsInfo
	2: ConfusionMatrixReport
	3: PerformanceReport
		Conditions:
			0: Train-Test scores relative degradation is not greater than 0.1
	4: RocReport(excluded_classes=[])
		Conditions:
			0: AUC score for all the classes is not less than 0.7
	5: SimpleModelComparison
		Conditions:
			0: Model performance gain over simple model is not less than
…]

In conclusion, Check, Condition, and Suites allow users to essentially check the data and model in their respective tasks. These can be extended and modified according to the requirements of the project and for various use cases. 

DeepChecks allows flexibility and instant validation of the ML pipeline with less effort. Their strong boilerplate code can allow users to automate the whole testing process, which can save a lot of time. 

Why should you use this?

• It is open-source and free, and it has a growing community.
• Very well-structured framework. 
• Because it has built-in checks and suites, it can be extremely useful for inspecting potential issues in your data and models.
• It is efficient in the research phase as it can be easily integrated into the pipeline.
• If you are mostly working with tabular datasets, then DeepChecks is extremely good. 
• You can also use it to check data, model drifts, model integrity, and model monitoring.

Key features 

Key drawbacks

2. Drifter-ML

Drifter ML is an ML model testing tool specifically written for the Scikit-learn library. It can also be used to test datasets similar to DeepChecks. It has five modules, each very specific to the task at hand.

1. Classification test: It enables you to test classification algorithms.
2. Regression test: It enables you to test classification algorithms.
3. Structural test: This module has a bunch of classes that allow testing of clustering algorithms.
4. Time Series test: This module can be used to test model drifts. 
5. Columnar test: This module allows you to test your tabular dataset. Tests include sanity testing, mean and median similarity, Pearson’s correlation et cetera. 

Installation

pip install drifter-ml

Structure of the framework

Drifter ML conforms to the Scikit-Learn blueprint for models, i.e., the model must contain a .fit and .predict methods. This essentially means that you can test deep learning models as well since Scikit-Learn has an integrated Keras API. Check the example below.

#Source: https://drifter-ml.readthedocs.io/en/latest/classification-tests.html#lower-bound-classification-measures

from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasClassifier
import pandas as pd
import numpy as np
import joblib

# Function to create model, required for KerasClassifier
def create_model():
   # create model
   model = Sequential()
   model.add(Dense(12, input_dim=3, activation='relu'))
   model.add(Dense(8, activation='relu'))
   model.add(Dense(1, activation='sigmoid'))
   # Compile model
   model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
   return model

# fix random seed for reproducibility
df = pd.DataFrame()
for _ in range(1000):
   a = np.random.normal(0, 1)
   b = np.random.normal(0, 3)
   c = np.random.normal(12, 4)
   if a + b + c > 11:
       target = 1
   else:
       target = 0
   df = df.append({
       "A": a,
       "B": b,
       "C": c,
       "target": target
   }, ignore_index=True)

# split into input (X) and output (Y) variables
# create model
clf = KerasClassifier(build_fn=create_model, epochs=150, batch_size=10, verbose=0)
X = df[["A", "B", "C"]]
clf.fit(X, df["target"])
joblib.dump(clf, "model.joblib")
df.to_csv("data.csv")

The example above shows the ease with which you can design your ANN model using drifter-ml. Similarly, you can also design a test case as well. In the test defined below, we will try to find the lowest decision boundary by which the model can easily classify the two classes. 

def test_cv_precision_lower_boundary():
   df = pd.read_csv("data.csv")
   column_names = ["A", "B", "C"]
   target_name = "target"
   clf = joblib.load("model.joblib")

   test_suite = ClassificationTests(clf,
   df, target_name, column_names)
   lower_boundary = 0.9
   return test_suite.cross_val_precision_lower_boundary(
       lower_boundary
   )

Why should you use this?

• Drifter-ML is specifically written for Scikit-learn, and this library acts as an extension to it. All the classes and methods are written in sync with Scikit-learn, so data and model testing become relatively easy and straightforward. 

• On a side note, if you like to work on an open-source library, then you can extend the library to other machine learning and deep learning libraries such as Pytorch as well. 

Key features 

Key drawbacks

Subscription-based tools

1. Kolena.io

Kolena.io is a Python-based framework for ML testing. It also includes an online platform where the results and insights can be logged. Kolena focuses mostly on the ML unit testing and validation process at scale. 

Why you should use this?

Kolena argues that the split test dataset methodology isn’t as reliable as it seems to be. Splitting the datasets provides a global representation of the entire population distribution and fails to capture the local representations at a granular level, this is especially true with label or class. There are hidden nuances of features that still need to be discovered. This leads to the failure of the model in the real world even though the model yields good scores in the performance metrics during training and evaluation. 

One way of addressing that issue is by creating a much more focused dataset that can be achieved by breaking a given class into smaller subclasses for focused results or even creating a subset of the features themselves. Such a dataset can enable the ML model to extract features and representation at a much granular level. This will increase the performance of the model as well by balancing both the bias and variance such that the model generalizes well in the real-world scenario. 

For example, when building a classification model, a given class in the dataset can be broken down into various subsets and those subsets into finer subsets. This can enable users to test the model in various scenarios. In the table below, the CAR class is tested against several test cases to check the model’s performance on various attributes. 

Another benefit is whenever we face a new scenario in the real-world, a new test case can be designed and tested immediately. Likewise, users can build more comprehensive test cases for a variety of tasks and train or build a model. The users can also generate a detailed report on a model’s performance in each category of test cases and compare it to the previous models with each iteration.

To sum up, Kolena offers:

• Ease of python framework
• Automated workflow testing and deployment
• Faster model debugging
• Faster model deployment

If you are working on a large-scale deep learning model which will be complex to monitor, then Kolena will be beneficial. 

Key features 

Key drawbacks

pip3 install --extra-index-url "$CR_URL" kolena-client

2. Robust intelligence

It is an E2E ML platform that offers various services in terms of ML integrity. The framework is written in Python and allows customizing your code according to your needs. The framework also integrates into an online dashboard that provides insights into various testing on data and model performance as well as model monitoring. All these services target the ML model and data right from training to the post-production phase. 

Why should you use this?

The platform offers services like:

1. AI stress testing, which includes hundreds of tests to automatically evaluate the performance of the model and identify potential drawbacks. 

2. AI Firewall, which automatically creates a wrapper around the trained model to protect it from bad data in real-time. The wrapper is configured based on the model. It also automatically checks both the data and model, reducing manual effort and time.

3. AI continuous testing, which monitors the model and automatically tests the deployed model to check for updates and retraining. The testing involves data drift, error, root cause analysis, anomalies detection et cetera. All the insights gained during continuous testing are displayed on the dashboard. 

Robust intelligence enables model testing, model protection during deployment, and model monitoring after deployment. Since it is an e2e-based platform, all the phases can be easily automated with hundreds of stress tests run on the model to make it production ready. If the project is fairly large, then Robust intelligence will give you an edge. 

Key features 

Key drawbacks

(Source)

Hybrid frameworks

1. Etiq.ai

​​Etiq is an AI-observability platform that supports AI/ML lifecycle. Like Kolena and Robust Intelligence, the framework offers ML Model testing, monitoring, optimization, and explainability. 

Etiq is considered to be a hybrid framework as it offers both offline and online implementation. Etiq has four tiers of usage:

1. Free and public: It includes free usage of the library as well as the dashboard. Keep in mind the results and metadata will be stored in your dashboard instance the moment you log in to the platform, but you will receive full benefits. 
2. Free and limited: If you want a free but private testing environment for your project and don’t want to share any information, then you can use the platform without logging into the platform. Keep in mind that you will not receive full benefits as would have received when you logged into the platform.  
3. Subscribe and private: If you want full benefits of Etiq.ai, then you can subscribe to their plan and make use of their tools in your own private environment. Etiq.ai is already available in the AWS market place which starts at around $3.00/hour or from $25,000.00/year. 
4. Personalized request: If you require functionality beyond what is provided by Etiq.ai, like explainability, robustness, or team share functionality, then you can contact them and get your own personalized test suite.  

Structure of the framework 

Etiq follows a structure similar to DeepChecks. This structure remains the core of the framework:

• Snapshot: It is a combination of dataset and model in the pre-production testing phase. 
• Scan: It is usually a test that is applied to the snapshot.
• Config: It is usually a JSON file that contains a set of parameters that will be used by the scan for running tests in the snapshot.
• Custom test: It allows you to customize your tests by adding and editing various metrics to the config file. 

Etiq offers two types of tests: Scan and Root Cause Analysis or RCA, the latter is an experimental pipeline. The scan type offers

• Accuracy: In some cases, high accuracy can indicate a problem just as low accuracy can. In such cases, an ‘accuracy’ scan can be helpful. If the accuracy is too high, then you might do a leakage scan, or if it is low, then you can do a drift scan. 
• Leakage: It helps you to find data leakage. 
• Drift: It can help you to find feature drift, target drift, concept drift, and prediction drift. 
• Bias: Bias refers to algorithmic bias that can happen because of automated decision making causing unintended discrimination. 

Why should you use this?

Etiq.ai offers a multi-step pipeline, which means you can monitor the test by logging the results of each of the steps in the ML pipeline. This allows you to identify and repair bias within the model. If you are looking for a framework that can do the heavy lifting of your AI pipeline, then Etiq.ai is the one to go. 

Some other reasons why you should use Etiq.ai:

All the points above are valid for free tier usage. 

One key feature of Etiq.ai is that it allows you to be very precise and straightforward in your model building and deploying approaches. It aims to give users the tools that can help them to achieve the desired model. At times, the development process gets drifted away from the original plan mostly because of the lack of tools needed to shape the model. If you want to deploy a model that is aligned with the proposed requirements, then Etiq.ai is the way to go. This is because the framework offers similar tests at each step throughout your ML pipeline. 

Key features 

Key drawbacks

Conclusion

The ML testing frameworks that we discussed are directed toward the needs of the users. All of the frameworks have their own pros and cons. But you can definitely get by using any one of these frameworks. ML model testing frameworks play an integral part in defining how the model will perform when deployed to a real-world scenario. 

If you are looking for a free and easy-to-use ML testing framework for structured datasets and smaller ML models, then go with DeepChecks. If you are working with DL algorithms, then Etiq.ai is a good option. But if you can spare some money, then you should definitely inquire about Kolena. And lastly, if you are working in a mid to large-size enterprise and looking for ML testing solutions, then hands-down, it has to be Robust Intelligence. 

I hope this article provided you with all the preliminary information needed for you to get started with ML testing. Please share this article with everyone who needs it. 

Thanks for reading!!!

Reference

1. https://www.robustintelligence.com/
2. https://aws.amazon.com/marketplace/pp/prodview-23bciknsbkgta
3. https://etiq.ai/
4. https://docs.etiq.ai/
5. https://arxiv.org/pdf/2005.04118.pdf
6. https://medium.com/kolena-ml/best-practices-for-ml-model-testing-224366d3f23c
7. https://docs.kolena.io/
8. https://www.kolena.io/
9. https://github.com/EricSchles/drifter_ml
10. https://arxiv.org/pdf/2203.08491.pdf
11. https://medium.com/@ptannor/new-open-source-for-validating-and-testing-machine-learning-86bb9c575e71
12. https://deepchecks.com/
13. https://www.xenonstack.com/insights/machine-learning-model-testing
14. https://www.jeremyjordan.me/testing-ml/
15. https://neptune.ai/blog/ml-model-testing-teams-share-how-they-test-models
16. https://mlops.toys

Because of this, transformers were able to achieve state-of-the-art results in various NLP tasks like machine translation, summary generation, text-generation, et cetera. It has also replaced RNN and its variants in almost all the NLP tasks. As a matter of fact, with its success in NLP, transformers are now being adopted in computer vision tasks as well. In 2020, Dosovitskiy and his team developed vision transformers (ViT), where they argued that reliance on CNN is not necessary. Based upon this premise, in this article, we will explore and learn how ViT can help in the task of image classification.  

This article is a guide aimed at building an MLOps pipeline for a computer vision task using ViT, and it will focus on the following areas with respect to a typical data science project:

1. Aim of the project
2. Hardware specification
3. Attention visualization 
4. Building the model and experiment tracking
5. Testing and inference
6. Creating a Streamlit app for deployment
7. Setting up CI/CD using GitHub actions
8. Deployment and monitoring

The code for this article can be found on this Github Link so that you can follow along. Let’s get started. 

MLOps pipeline for image classification: understanding the project

Understanding the requirements of the project or the client is an important step as it can help us brainstorm ideas and research various components that the project might require, such as the latest papers, repositories, relevant work, datasets, and even cloud-based platforms for deployment. This section will focus on 2 topics: 

Aim of the project: bird image classifier 

The aim of the project is to build an image classifier to classify different species of birds. Since this model will be later deployed in the cloud, we must keep in mind that the model must be trained to achieve a good accuracy score in both training and testing datasets. In order to do that, we should use metrics like precision, recall, confusion metrics, F1, and AUROC score to see how the model is performing on both datasets. Once the model achieves good scores on the test dataset, we will then create a web app to deploy it on a cloud-based server. 

In a nutshell, this is how the project will be executed:

This project will include some of the additional practices that you will find in this article, such as: 

• Live tracking to monitor metrics,
• Attention visualization,
• Directory structure,
• Code formatting for all the python modules. 

Hardware for accelerated training

We will conduct our experiment with two sets of hardware:

1. M1 Macbook: The efficiency of Apple’s M1 processors will allow us to quickly develop models and train them on a smaller dataset. Once the training is done, we can start building a web application on our local machine and create a small pipeline of data ingestion, data preprocessing, model prediction, and attention visualization before scaling up the model in the cloud. 

Note: if you have one of these M1 laptops, then make sure to check the installation process in my Github repo.

2. Kaggle or Google Colab GPUs: Once our code is working properly in our local machine and the pipeline is created, we can scale it up and train the whole model for a longer period in Google Colab or Kaggle which are free. Once the training is done, we can then download new weights and metadata to our local computer and test whether the web application is performing well in the unseen data before deploying it to the cloud. 

Now let’s start the implementation. 

MLOps pipeline for image classification: data preparation

The first step of implementing a deep learning project is to plan the different python modules that we are going to have. Although we will be using the Jupyter notebook for experimentation, it is always a good idea to have everything laid out before starting to code. Planning might include reference code repositories as well as research papers. 

It is always a good idea to create the directory structure for the project for efficiency and for ease of navigation.  

ViT Classification
├── notebooks
│   └── ViT.ipynb
└── source
    └──config.py

In our case, the main directory is called the ViT Classification, which contains two folders: 

1. Notebooks: This is where all the experimentation with jupyter notebook will reside.
2. Source: This is where all the Python modules will reside. 

As we progress, we will keep adding Python modules to the source directory, and we will also create different sub-directories for storing metadata, docker files, README.md files, et cetera. 

Building the image classification model

As mentioned before, research and planning is the key to implementing any machine learning project. What I usually do first is, create a config.py to store all the parameters with respect to data preprocessing, model training and inference, visualization, et cetera.

config.py

class Config:
   #Image configuration
   IMG_SIZE = 32
   PATCH_SIZE = 10
   CROP_SIZE = 100
   BATCH_SIZE = 1
   DATASET_SAMPLE = 'full'


   #opimizer configuration
   LR = 0.003
   OPIMIZER = 'Adam'

   #Model configuration
   NUM_CLASSES = 400
   IN_CHANNELS = 3
   HIDDEN_SIZE = 768
   NUM_ATTENTION_HEADS = 12
   LINEAR_DIM = 3072
   NUM_LAYERS = 12

   ATTENTION_DROPOUT_RATE = 0.1
   DROPOUT_RATE = 0.1
   STD_NORM = 1e-6
   EPS = 1e-6
   MPL_DIM = 128
   OUTPUT = 'softmax'
   LOSS_FN = 'nll_loss'

   #Device configuration
   DEVICE = ["cpu","mps","cuda"]

   #Training configuration
   N_EPOCHS = 1

The above code block gives a vague idea of what the parameters should look like. As we make progress, we can keep adding more parameters. 

Note: In the device configuration section, I have given a list of three hardware: CPU, MPS, and CUDA. MPS or Metal Performance Shaders is the hardware type to train on M1 Macbooks.  

Dataset

The dataset that we will use is the bird classification dataset which can be downloaded from Kaggle. The dataset consists of 400 classes of birds with three subsets: training, validation, and testing, each containing 58388, 2000, and 2000 images, respectively. Once the data has been downloaded, then we can then create a function to read and visualize the images. 

Preparing the data

We can move ahead to create a data loader that transforms the images into image tensors. Along with that, we will also perform resizing, image cropping, and normalizing as well. Once the preprocessing is done, we can then use the DataLoader function to automatically generate data for training in batches. The following pseudo function will give you an idea of what we are trying to achieve, you can find the full code in the link provided in the code heading:

preprocessing.py

#apply the desired transformations on dataset and split it into train, validation, and test set.

def Dataset(bs, crop_size, sample_size='full'):
      return train_data, valid_data, test_data

The above function has a sample size argument that allows the creation of a sub-set of the training dataset for testing purposes on your local machine. 

MLOps pipeline for image classification: building the vision transformer using Pytorch

I have created the full model as per the author’s description of ViT in their paper. This code is inspired by jeonsworld repo, I have added a few more details and edited some of the lines of code for the purpose of this task. 

The model that I have created is divided into 9 modules, and each module can be executed independently for various tasks. We will explore each section in order for ease of understanding. 

Embedding

Transformers and all the natural language model has an important component called embedding. Its function is usually to capture semantic information by grouping similar information together. Apart from that embeddings can be learned and reused across models. 

In ViT, embeddings serve the same purpose by retaining positional information which can be fed into the encoder. Again the following pseudo-code will help you to understand what’s going on and you can also find the full code in the link provided in the code heading. 

embedding.py

class Embeddings(nn.Module):

#Construct the embeddings from patch, position embeddings.
   def __init__(self, img_size:int, hidden_size:int, in_channels:int):

#create a CONV2D object for creation of embeddings 
   def forward(self, x):

#calculate and return embeddings
       return embeddings

Note that the embedding patches for the image can be created using convolution layers. It is quite efficient and easy to modify as well. 

Encoder

The encoder is made up of a number of attention blocks which itself has two important modules:

Self attention mechanism

Let’s start with the self-attention mechanism. 

The self-attention mechanism is the core of the whole system. It enables the model to focus on the important feature of the data. It does so by operating on a single embedding at different positions to compute the representation of the same sequence. You can find the link to the entire code below to get a deeper picture. 

attention.py

#Calculate the attention and return the attention output along with the weights

class Attention(nn.Module):
       return attention_output, weights

The output of the attention block will yield the attention output as well the attention weights. The latter will be used to visualize the ROI that is calculated using the attention mechanism. 

Multilayer perceptron

Once we receive the attention output, we can then feed it into the MLP, which will give us a probability distribution for the classification. You can get an idea of the entire process in the forward function. To see the full code click the link provided in the code heading below. 

linear.py

#Apply a linear transformation to the incoming attention output using the GELU activation function.

class Mlp(nn.Module):
   def __init__(self, hidden_size, linear_dim, dropout_rate, std_norm):
       return x

It is worth noting that we are using the GELU as our activation function. 

One of the pros of using GELU is that it avoids vanishing gradient, which makes the model easy to scale. 

Attention-block

The attention block is the module where we assemble both the modules: the self-attention module and the MLP modules. 

attention_block.py

#Returns the calculated sum of attention scores via MLP along with attention weights.

class Block(nn.Module):
       return x, weights

This module will also yield the attention weights directly from the attention mechanism along with the distribution yielded by MLP. 

Now let’s briefly understand the encoder. The Encoder essentially enables us to create multiple attention blocks that give the transformer more control over the attention mechanism. The three components: Encoder, Transformer, and ViT are written in the same module i.e., transformers.py.

#Creates multiple layers of attention blocks and returns encoded state and attention weights. 

class Encoder(nn.Module):
       return encoded, attn_weights

Transformer

After assembling the attention block we can then code our transformer. The attention block transformer is an assembly of the embedding module and encoder module. 

class Transformer(nn.Module):
   def __init__(self, img_size, hidden_size, in_channels, num_layers,
                num_attention_heads, linear_dim, dropout_rate, attention_dropout_rate,
                eps, std_norm):
       super(Transformer, self).__init__()
       self.embeddings = Embeddings(img_size, hidden_size, in_channels)
       self.encoder = Encoder(num_layers, hidden_size, num_attention_heads,
                              linear_dim, dropout_rate, attention_dropout_rate,
                              eps, std_norm)

   def forward(self, input_ids):
       embedding_output = self.embeddings(input_ids)
       encoded, attn_weights = self.encoder(embedding_output)
       return encoded, attn_weights

Vision transformer

Finally, we can code our vision transformer which involves two components: the transformer and the final linear layer. The final linear will help us to find the probability distribution over all the classes. It can be described as:

class VisionTransformer(nn.Module):
   def __init__(self, img_size, num_classes, hidden_size, in_channels, num_layers,
                num_attention_heads, linear_dim, dropout_rate, attention_dropout_rate,
                eps, std_norm):
       super(VisionTransformer, self).__init__()
       self.classifier = 'token'

       self.transformer=Transformer(img_size, hidden_size, in_channels,
                                    num_layers, num_attention_heads, linear_dim,
                                    dropout_rate, attention_dropout_rate, eps,
                                    std_norm)
       self.head = Linear(hidden_size, num_classes)

   def forward(self, x, labels=None):
       x, attn_weights = self.transformer(x)
       logits = self.head(x[:, 0])

       if labels is not None:
           loss_fct = CrossEntropyLoss()
           loss = loss_fct(logits.view(-1, 400), labels.view(-1))
           return loss
       else:
           return logits, attn_weights

Please notice that the network is going to consistently yield attention weights which will be useful for visualizing the attention maps. 

Here is a bonus tip. If you want to see the architecture of the model and how the inputs are being operated then use the following line of code. The code will generate a full operational architecture for you. 

from torchviz import make_dot
x = torch.randn(1,config.IN_CHANNELS*config.IMG_SIZE*config.IMG_SIZE)
x = x.reshape(1,config.IN_CHANNELS,config.IMG_SIZE,config.IMG_SIZE)
logits, attn_weights = model(x)
make_dot(logits, params=dict(list(model.named_parameters()))).render("../metadata/VIT", format="png")

You can find the image in the given link. 

But in nutshell, this how the architecture looks like. 

MLOps pipeline for image classification: training vision transformer using Pytorch

The training module is where we will assemble all the other modules like the config module, preprocessing module, and Transformer and log the parameters including the metadata into the neptune.ai API. One easiest way to log parameters is to use Config.__dict__. This automatically converts a class into a dictionary. 

Disclaimer

Please note that this article references a deprecated version of Neptune.

For information on the latest version with improved features and functionality, please visit our website.

You can later create a function that removes unnecessary attributes from the dictionary. 

def neptune_monitoring():
   PARAMS = {}
   for key, val in Config.__dict__.items():
       if key not in ['__module__', '__dict__', '__weakref__', '__doc__']:
           PARAMS[key] = val
   return PARAMS

Training 

The training function is quite straightforward and simple to write. I have included both training and evaluation in the pseudo-code. You can find the full training block here, or you can click the code heading below.

train.py

def train_Engine(n_epochs, train_data, val_data, model, optimizer, loss_fn, device,
                monitoring=True):

#Initiates the training procedure while tracking accuracy and loss over each iterations. 

Now our training loop is completed, we can then start the training and log the metadata into Neptune dashboard, which we can use for monitoring the training on the go, saving charts and parameters, and sharing them with teammates. 

train.py

if __name__ == '__main__':
   from preprocessing import Dataset
   from config import Config
   config = Config()
   params = neptune_monitoring(Config)

   run = neptune.init_run(project="nielspace/ViT-bird-classification",
                       api_token=API_TOKEN)
   run['parameters'] = params

   model = VisionTransformer(img_size=config.IMG_SIZE,
                num_classes=config.NUM_CLASSES,
                hidden_size=config.HIDDEN_SIZE,
                in_channels=config.IN_CHANNELS,
                num_layers=config.NUM_LAYERS,
                num_attention_heads=config.NUM_ATTENTION_HEADS,
                linear_dim=config.LINEAR_DIM,
                dropout_rate=config.DROPOUT_RATE,
                attention_dropout_rate=config.ATTENTION_DROPOUT_RATE,
                eps=config.EPS,
                std_norm=config.STD_NORM)

   train_data, val_data, test_data = Dataset(config.BATCH_SIZE, config.IMG_SIZE,
                                             config.DATASET_SAMPLE)

   optimizer = optim.Adam(model.parameters(), lr=0.003)
   train_Engine(n_epochs=config.N_EPOCHS, train_data=train_data, val_data=val_data,
               model=model,optimizer=optimizer, loss_fn='nll_loss',
               device=config.DEVICE[1], monitoring=True)

Note: The prototyping of this model was done in Macbook Air M1 on a smaller dataset with 10 classes. The prototyping stage is where I tried different configurations and played with the architecture of the model. Once I was satisfied I used Kaggle to train the model. Since the dataset has 400 classes, the model needed to be larger and trained for a longer period of time.  

Experiment tracking

In the prototyping stage, experiment tracking becomes a very handy and reliable source to make further changes to your model. You can keep an eye on your model’s performance during training and subsequently make necessary tweaks to it until you get a high-performing model.

The Neptune API enables you to:

• monitor the model’s training progress
• and simultaneously upload the metrics into the system.
• It also allows you to compare multiple runs involving different model configurations and simultaneously choose the best one.

If you want to log your metadata in the system, then import the Neptune API and call the init function. Following that, enter the API key provided for the project, and you are good to go. Get to know more about how to get started with Neptune here. Also, here is the Neptune dashboard, which has the metadata related to this project.

run = neptune.init_run(project="nielspace/ViT-bird-classification",
api_token="API_TOKEN")

Once you are done with the initialization, you can start logging. For instance, if you want to:

1. Upload the parameters, use: run[‘parameters’] = params.
   Note: make sure that the params are of dictionary class.
2. Upload metrics, use: run[‘Training_loss’].log(loss.item())and run[‘Training_loss’].log(loss.item())
3. Upload model weights, use: run[“model_checkpoints/ViT”].upload(“model.pt”)
4. Upload images, use: run[“val/conf_matrix”].upload(“confusion_matrix.png”)

Depending upon what you are optimizing your model for, there are plenty of things that you can log and track. In our case, we put an emphasis on training and validation loss and accuracy.

Logging metadata and dashboard

In the ongoing training process, you can then monitor the model’s performance. With each iteration, the graph will update. 

Along with the model’s performance, you will also find CPU and GPU performance as well. See the image below. 

You can also find all the model metadata as well. 

Scaling using Kaggle

Now, let’s scale the model. We will use Kaggle for this project because it is free and also because the dataset was downloaded from Kaggle so it will be easy to scale and train the model on the platform itself. 

1. The first thing we need to do is to upload the model and change the directory path to Kaggle-specific paths and enable the GPUs. 

2. Note that the model must be complex in order to capture relative information for prediction. You can start scaling the model by gradually increasing the number of hidden layers and seeing how the model behaves. You may not want to touch other parameters like the number of attention heads and hidden size because it may throw up arithmetic errors. 

3. For each change, you make the model run for at least two epochs in small data batches with all the 400 classes and observe if the accuracy is increasing. Typically, it will increase. 

4. Once satisfied, run the model for 10 to 15 epochs which would take around 5 hours for the subset of 30000 samples. 

5. After the training, check its performance on the test dataset, and if it performs well, then download the model weights. At this point, the size of the model should be around 650 MB for 400 classes. 

Attention visualization

As mentioned before, self-attention is the crux of the whole Vision Transformer architecture, and interestingly there is a way to visualize it as well. The source code of the attention map can be found here. I have modified it a bit and created it as a separate independent module that can use the output of the transformer to yield the attention maps. The idea here is to store the input image and its corresponding attention map image and display it in the README.md file. 

attention_viz.py (Source)

def attention_viz(model, test_data, img_path=PATH, device='mps'):

#Visualizes the attention mask of a given input (image) by comparing it with the original image. 

We can run this code by simply calling the attention_viz function and passing the corresponding arguments. 

if __name__ == '__main__':
   train_data, val_data, test_data = Dataset(config.BATCH_SIZE,config.IMG_SIZE, config.DATASET_SAMPLE)
   model = torch.load('metadata/models/model.pth', map_location=torch.device('cpu'))
   attention_viz(model, test_data, PATH)

Testing and inference

We can also use the attention_viz function in the test module, where we will test the model on the test data and measure the model’s performance on various metrics like confusion matrix, accuracy score, f1 score, recall score, and precision score.

test.py

def test(model, test_data):
   return logits_, ground, confusion_matrix

#Evaluates the model’s performance on the test dataset and returns the confusion matrix, logits and ground truth for further performance evaluation. 

We can easily generate a confusion matrix and visualize using heatmap from seaborn and save it in the results folder, which we can also use to display it on the README.md file. 

We can also generate the accuracy and loss graph and store it in the results folder as well. Consequently, we can use Sklearn to find other metrics, but before that, we must convert the tensors array into a NumPy array.

probs = torch.zeros(len(logits_))
y_ = torch.zeros(len(ground))
idx = 0
for l, o in zip(logits_, ground):
   _, l = torch.max(l, dim=1)
   probs[idx] = l
   y_[idx] = o.item()
   idx+=1

prob = probs.to(torch.long).numpy()
y_ = y_.to(torch.long).numpy()

print(accuracy_score(y_, prob))
print(cohen_kappa_score(y_, prob))
print(classification_report(y_, prob))

Once we are satisfied with the model’s performance, we can then do inference by simultaneously creating a Streamlit app. 

MLOps pipeline for image classification: creating the app using Streamlit

The Streamlit app will be a web app that we will deploy on the cloud. In order to build the app, we must first pip install streamlit followed by importing the library in the new module. 

The module will contain the same module as the inference module we just need to copy and paste the evaluation function as it is and then build the app using the Streamlit library. Below is the code of the app. 

app.py

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

from PIL import Image
import torch
from torchvision import transforms
import torch
import streamlit as st

from embeddings import Embeddings
from attention_block import Block
from linear import Mlp
from attention import Attention
from transformer import VisionTransformer, Transformer, Encoder

from config import Config
config = Config()

st.set_option('deprecation.showfileUploaderEncoding', False)
st.title("Bird Image Classifier")
st.write("")

# enable users to upload images for the model to make predictions
file_up = st.file_uploader("Upload an image", type = "jpg")


def predict(image):
   """Return top 5 predictions ranked by highest probability.
   Parameters
   ----------
   :param image: uploaded image
   :type image: jpg
   :rtype: list
   :return: top 5 predictions ranked by highest probability
   """
   model = torch.load('model.pth')

   # transform the input image through resizing, normalization
   transform = transforms.Compose([
       transforms.Resize(128),
       transforms.CenterCrop(128),
       transforms.ToTensor(),
       transforms.Normalize(
           mean = [0.485, 0.456, 0.406],
           std = [0.229, 0.224, 0.225])])

   # load the image, pre-process it, and make predictions
   img = Image.open(image)
   x = transform(img)
   x = torch.unsqueeze(x, 0)
   model.eval()
   logits, attn_w = model(x)

   with open('../metadata/classes.txt', 'r') as f:
       classes = f.read().split('n')

   # return the top 5 predictions ranked by highest probabilities
   prob = torch.nn.functional.softmax(logits, dim = 1)[0] * 100
   _, indices = torch.sort(logits, descending = True)
   return [(classes[idx], prob[idx].item()) for idx in indices[0][:5]]


if file_up is not None:
   # display image that user uploaded
   image = Image.open(file_up)
   st.image(image, caption = 'Uploaded Image.', use_column_width = True)
   st.write("")
   st.write("Processing...")
   labels = predict(file_up)

   # print out the top 5 prediction labels with scores
   for i in labels:
       st.write(f"Prediction {i[0]} score {i[1]:.2f}")

But before we deploy, we must test it locally. In order to test the app, we will run the following command:

streamlit run app.py

Once the above command is executed, you will get the following prompt:

You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
  Network URL: http://192.168.0.105:8501

Copy the URL and paste it into your browser, and the app is online (locally). 

Upload the image for classification. 

With the ViT model trained and the app ready our directory structure should look something like this now:

.
├── README.md
├── metadata
│   ├── Abbott's_babbler_(Malacocincla_abbotti).jpg
│   ├── classes.txt
│   ├── models
│   │   └── model.pth
│   └── results
│       ├── accuracy_loss.png
│       ├── attn.png
│       └── confusion_matrix.png
├── notebooks
│   ├── ViT.ipynb
│   └── __init__.py
└── source
    ├── __init__.py
    ├── app.py
    ├── attention.py
    ├── attention_block.py
    ├── attention_viz.py
    ├── config.py
    ├── embeddings.py
    ├── linear.py
    ├── metrics.py
    ├── preprocessing.py
    ├── test.py
    ├── train.py
    ├── transformer.py

Now we proceed toward deploying the app. 

MLOps pipeline for image classification: code formatting

First, let’s format our Python scripts. For that, we will use Black. Black is a Python script formatter. All you need to do is pip install black and then run `black ` following the name of the python module or even the whole directory. For this project, I ran black followed by the source directory which contains all the python modules. 

ViT-Pytorch git:(main) black source
Skipping .ipynb files as Jupyter dependencies are not installed.
You can fix this by running ``pip install black[jupyter]``
reformatted source/config.py
reformatted source/embeddings.py
reformatted source/attention_block.py
reformatted source/linear.py
reformatted source/app.py
reformatted source/attention_viz.py
reformatted source/attention.py
reformatted source/preprocessing.py
reformatted source/test.py
reformatted source/metrics.py
reformatted source/transformer.py
reformatted source/train.py

The advantage of using black is that it removes unnecessary spaces, adds double quotes instead of single quotes, and makes reviewing code faster and more efficient. 

Given below are the images of before and after using black to format the code. 

As you can see that unnecessary spaces have been removed. 

MLOps pipeline for image classification: setting up CI/CD 

For our CI/CD process, we will be using Github Actions, and Google Cloud Build to integrate and deploy our Streamlit app. The following are the steps that will help you to create a full MLOps pipeline. 

Creating the Github Repository

The first step is to create the Github repository. But before that we must create three important files:

requirements.txt

The requirements.txt file must contain all the libraries that the model is using. There are two ways in which you can create a requirements.txt file. 

1. If you have a dedicated working environment created specifically for this project, then you can run pip freeze>requirements.txt and it will create a requirements.txt file for you. 
2. If you have a general working environment, then you can run pip freeze and copy-paste the libraries that you have been working on.

The requirement.txt file for this project looks like this:

numpy==1.22.3
torch==1.12.0
torchvision==0.12.0
tqdm==4.64.0
opencv-python==4.6.0.66
streamlit==1.10.0
neptune-client==0.16.3

Note: Always make sure that you mention the version so that in the future, the app remains stable and performs optimally. 

Makefile

In a nutshell, Makefile is a command prompt file that automates the whole process of installing libraries, and dependencies, running a Python script, et cetera. A typical Makefile looks something like this:

#Makefile
setup:
   python3 -m venv ~/.visiontransformer
   source ~/.visiontransformer/bin/activate
   cd .visiontransformer
install:
   pip install --upgrade pip &&
       pip install -r requirements.txt
run:
   python source/test.py
all: install run

For this project, our Makefile will have three processes:

Essentially every time we make a new commit, the makefile will be executed, which will automatically run the test.py module generating the latest performance metrics and updating the README.md file.

But Makefile will only work if we create an action trigger. So let’s create that.

Action trigger: .github/workflow/main.yml

To create an action trigger, we need to create the following directory: .github/workflow, followed by creating a main.yml file. The main.yml will essentially create an action trigger whenever the repo is updated. 

Our aim is to continuously integrate any changes made in the existing build, like updating parameters, model architecture, or even the UI/UX. Once the change is detected, it will automatically update the README.md file. The main.yml for this project is designed to trigger the workflow on any push or pull request but only for the main branch.

At each new commit, the file will activate the ubuntu-latest environment, install the specific python version and then execute a specific command from the Makefile. 

main.yml

#main.yml
name: Continuous Integration with Github Actions

on:
 push:
   branches: [ main ]
 pull_request:
   branches: [ main ]

jobs:
 build:
   runs-on: ubuntu-latest
   # Steps represent a sequence of tasks that will be executed as part of the job
   steps:
     - uses: actions/checkout@v2
     - name: Set up Python 3.8
       uses: actions/setup-python@v1
       with:
         python-version: 3.8
     - name: Install dependencies
       run: |
         make install
         make run

Testing

After the files are created, you can push the entire codebase to Github. Once uploaded, you can click on the Actions tab and see the build-in progress for yourself. 

Deployment: Google Cloud Build

After the testing is done and all the logs and results are updated in the Github README.md file, we can move to the next step, which is to integrate the app into the cloud. 

1. First, we will visit: https://console.cloud.google.com/, and then we will create a new project in the dashboard and name it Vision Transformer Pytorch. 

Once the project is created, you can navigate into the project, and it will look something like this:

As you can see, google cloud build offers us various services right out of the box like a virtual machine, big query, GKE, or Kubernetes cluster on the project home page. But before we create anything in the cloud build we must enable the Kubernetes cluster and create a certain directory and their respective files in the project directory.

2. Kubernetes

Let’s set up our Kubernetes cluster before we create any files. To do that, we can search GKE in the google cloud console search bar and enable the API. 

Once the API is enabled, we will be navigated to the following page. 

But instead of creating the clusters manually, we will create them using the inbuild cloud shell. To do that, click on the terminal button on the top right hand, and check the image below. 

After activating the cloud shell, we can type the following command to create Kubernetes clusters:

gcloud container clusters create project-kube --zone "us-west1-b" --machine-type "n1-standard-1" --num-nodes "1"

This usually takes up to 5 minutes. 

After it is completed, it will look something like this: 

Now let’s set up the two files that will configure the Kubernetes clusters: deployment.yml and service.yml. 

The deployment.yml file allows us to deploy the model in the cloud. The deployment can be canary, recreate, blue-green or any other depending upon the requirement. In this example, we will overwrite the deployments. This file also helps in scaling the model efficiently using the arguments replicas. Here is an example of a deployment.yml file.

deployment.yml

#deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
 name: imgclass
spec:
 replicas: 1
 selector:
   matchLabels:
     app: imageclassifier
 template:
   metadata:
     labels:
       app: imageclassifier
   spec:
     containers:
     - name: cv-app
       image: gcr.io/vision-transformer-pytorch/vit:v1
       ports:
       - containerPort: 8501

The next file is the service.yml file. It essentially connects the app from the container to the real world. Notice the containerPort argument is specified as 8501, we will use the same number in our service.yml for the targetPort argument. This is the same number that Streamlit uses to deploy the application. Apart from that, the app argument is the same in both files. 

service.yml

#service.yml

apiVersion: v1
kind: Service
metadata:
 name: imageclassifier
spec:
 type: LoadBalancer
 selector:
   app: imageclassifier
 ports:
 - port: 80
   targetPort: 8501

Note: Always make sure that the name of the app and the version are in lower cases. 

3. Dockerfile

Now let’s configure the Dockerfile. This file will create a Docker container that will host our Streamlit app. Docker is very much required since it wraps the app in an environment that is easy to scale. A typical Dockerfile looks like this:

Dockerfile

FROM python:3.8.2-slim-buster

RUN apt-get update

ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./

RUN ls -la $APP_HOME/

# Install dependencies
RUN pip install -r requirements.txt

# Run the streamlit on container startup
CMD [ "streamlit", "run","app.py" ]

Dockerfile contains a series of commands that:

• Installs the Python version. 
• Copies the local code to the container image.
• Installs all the libraries.
• Executes Streamlit app. 

Note that we are using Python 3.8 as some of the dependencies use the latest Python version.

4. cloudbuild.yaml

In Google Cloudbuild cloudbuild.yml file stitches all the artefacts together to create a seamless pipeline. It has three primary steps:

• Build a Docker container using the Dockerfile from the current directory. 
• Push the container to the google container registry.
• Deploy the container in the Kubernetes engine. 

cloudbuild.yml

steps:
- name: 'gcr.io/cloud-builders/docker'
 args: ['build', '-t', 'gcr.io/vision-transformer-pytorch/vit:v1', '.']
 timeout: 180s
- name: 'gcr.io/cloud-builders/docker'
 args: ['push', 'gcr.io/vision-transformer-pytorch/vit:v1']
- name: "gcr.io/cloud-builders/gke-deploy"
 args:
 - run
 - --filename=kubernetes/ #this argument connects the files in kubernetes directory
 - --location=us-west1-b
 - --cluster=project-kube

Note: Please cross-check the arguments like the container name in deployment.yml and cloudbuild.yml file. Along with that also cross-check the cluster name that you created earlier with the cluster name in the clouldbuild.yml file. Lastly, make sure that the filename argument is as same as the Kubernetes directory where the deployment.yml and service.yml are present.  

After creating the files the file structure of the entire project should look like this:

.
├── Dockerfile
├── .github/workflow/main.yml
├── Makefile
├── README.md
├── cloudbuild.yaml
├── kubernetes
│   ├── deployment.yml
│   └── service.yml
├── metadata
│   ├── Abbott's_babbler_(Malacocincla_abbotti).jpg
│   ├── classes.txt
│   ├── models
│   │   └── model.pth
│   └── results
│       ├── accuracy_loss.png
│       ├── attn.png
│       └── confusion_matrix.png
├── notebooks
│   ├── ViT.ipynb
│   └── __init__.py
├── requirements.txt
└── source
    ├── __init__.py
    ├── app.py
    ├── attention.py
    ├── attention_block.py
    ├── attention_viz.py
    ├── config.py
    ├── embeddings.py
    ├── linear.py
    ├── metrics.py
    ├── preprocessing.py
    ├── test.py
    ├── train.py
    ├── transformer.py
    └── vit-pytorch.ipynb

5. Cloning and testing

Now let’s clone the GitHub repo in our google cloud build project, cd into it, and run the cloudbuild.yml file. Use the following commands:

• git clone https://github.com/Nielspace/ViT-Pytorch.git
• cd ViT-Pytorch

• gcloud builds submit –config cloudbuild.yaml

The deployment process will look something like this:

6. The deployment takes somewhere around 10 minutes, depending on various factors. And if everything is executed properly, you will see that the steps are color-coded with green ticks. 

7. Once the deployment is successful, you can find the endpoints of the app in the Services & Ingress tab in the Kubernetes Engine. Click on the endpoints, and it will navigate you to the Streamlit app. 

Additional tips:

1. Make sure that you use lowercases for app name and project id in all your *.yml config files.
2. Cross-check the arguments for all *.yml config files. 
3. Since you are copying your repo in a virtual environment, cross-check all the directory and file paths. 
4. In case of an error in the cloud build process, look for a command which will help you resolve the error you find in the error statement. See the image below for a better understanding; I have highlighted the command that needs to be executed before I re-run the cloud build command. 

Cloud build Integration

Now we will integrate the Google cloud build into the Github repo. This will create a trigger action that will update the build whenever a change is being made in the repo. 

1. Search Google Cloud Build in the Marketplace

2. Select the repo that you want to connect. In this case, it will be ViT-Pytorch and save it. 

3. In Google Cloud Build, we will go to the Cloud build page and click on the Triggers tab to create triggers. 

4. After clicking on create trigger, we will be navigated to the page below. There we will mention the trigger name, select the event which will trigger the cloudbuild.yml file, and select the project repository. 

5. Follow the authentication process. 

6. Connect the repository. 

7. Finally, create the trigger. 

Now that the trigger is created, all the changes that you make in the Github repo will be automatically detected, and the deployment will be updated. 

Monitoring the model-decay

Over time the model will decay, which will affect the prediction capabilities. We need to monitor the performance on a regular basis. One way to do that is to occasionally test the model on the new dataset and evaluate the same on metrics that I mentioned earlier, like F1 score, Accuracy score, Precision score, et cetera. 

Another interesting way to monitor the model’s performance is to use the AUROC metric, which measures the discriminative performance of the model. Because this project is a multiclassification project, you can convert it into a binary classification project and then check the model’s performance. If the performance of the model has decayed, then the model must be trained again with new samples and larger samples. And if it really required, then modify the architecture as well. 

Here is the link to the code, which will allow you to measure the AUROC score.

Conclusion

In this article, we learned to build an image classifier app with Vision Transformer using Pytorch and Streamlit. We also saw how we can deploy the app on the Google Cloud Platform using Github Actions and technologies like Kubernetes, Dockerfile, and Makefile. 

Important takeaways from this project:

1. Bigger data requires a larger model, which essentially requires training for more epochs. 
2. When creating a prototyping experiment, reduce the number of classes and test whether the accuracy increases with each epoch. Try different configurations till you are confident that the model’s performance is increasing before using GPUs on cloud services like Kaggle or Colab. 
3. Use various performance metrics like confusion metrics, precision, recall, confusion metrics, f1, and AUROC. 
4. Once the model is deployed, monitoring of the model can be done occasionally and not frequently. 
5. In order to monitor, using performance metrics like the AUROC score is good since it automatically creates threshold values and graphs the model’s True Positive rate and False Positive rate. With the AUROC score, the model’s previous and current performance can be easily compared. 
6. Re-training the model should be done only when the model has drifted significantly. Since a model like this requires a lot of computational resources, frequent retraining can be expensive.

I hope you found this article informative and practical. You can find the entire code in this Github repo. Feel free to share it with others as well. 

References

1. An Image Is Worth 16×16 Words: Transformers For Image Recognition At Scale
2. Are Transformers More Robust Than CNNs?
3. https://www.kdnuggets.com/2022/01/machine-learning-models-die-silence.html
4. https://github.com/jeonsworld/ViT-pytorch 
5. ​​https://gist.github.com/khizirsiddiqui/559a91dab223944fb83f8480715d2582
6. https://github.com/srivatsan88/ContinousModelDeploy 
7. Building MLOps Pipeline for NLP: Machine Translation Task

To manage this problem, a fast and efficient approach to designing and developing new models is needed. One cannot train these models on a single GPU because it will result in an information bottleneck. To solve the issue of information bottlenecks on a single core GPU we need to use multi-core GPUs. This is where the idea of distributed training comes into the picture.

In this article, we’ll look into some of the best frameworks and tools for distributed training. But before that, let’s have a quick overview of distributed training itself.

Distributed training

The DL training usually relies on scalability, which simply means the ability of the DL algorithm to learn or deal with any amount of data. Essentially the scalability of any DL algorithm depends on three factors:

Distributed training satisfies all three elements. It takes care of the model size and complexity, handles training data in batches, and it splits and distributes the training process among multiple processors called nodes. More importantly, it reduces the training time significantly making iteration time shorter and thus making experiments and deployment quicker.

Distributed training is of two types:

In data-parallel training, the data is divided into subsets based upon the number of nodes available for training. And the same model architecture is shared in all the available nodes. During the training process, all the nodes must communicate with each other to ensure that the training at each node is synced with each other. It is the most efficient way of training the model and the most common practice.

In model-parallel training, the DL model is split into segments based on the number of nodes available. Each node is fed with the same data. In model-parallel training, the DL model itself is split into different segments and each of the segments is then fed into different nodes. This type of training is possible if the DL model has independent components that can be trained individually. It is kept in mind that each of the nodes must be in sync with regard to the shared weights and biases of the different segments of the model. 

Among the two types of training, data-parallelism is quite commonly used and as we discover the frameworks for distributed training you will find that data-parallelism is offered by all of them irrespective of model-parallelism. 

Criteria for choosing the right framework for distributed training

Before we dive into the frameworks there are some points that one should consider while choosing the right framework and tools:

1. Computational graph type: The whole deep learning community is majorly divided into two factions, one that uses PyTorch or dynamic computational graph and the other that uses TensorFlow or static computational graph. Hence, it is not news that most of the distributed frameworks are built on top of these two libraries. So if you prefer one over the other then half of your decision is already made.
2. Cost of training: Affordability is a critical concern when you are dealing with distributed computing, e.g. a project involving the training of BigGAN can require a number of GPUs and the cost could scale up proportionally as this number increases. Hence, a tool with moderate pricing is always the right choice.
3. Type of training: Depending upon your training requirement i.e. data-parallelism or model-parallelism, you can prefer one tool over the other.
4. Efficiency: This basically refers to the number of lines you need to write to enable distributed training, the less the better.
5. Flexibility: Can the framework of your choice be used across different platforms? Especially when you need to train on-premise or on cloud platforms.

Frameworks for distributed training

Now, let’s discuss some of the libraries that offer distributed training. 

1. PyTorch

PyTorch is one of the most popular deep learning frameworks developed by Facebook. It is one of the most flexible and easy-to-learn frameworks. PyTorch allows you to create and implement neural network modules very effectively and with its distributed training modules you can easily implement parallel training with a few lines of code.  

PyTorch offers a number of ways in which you can perform distributed training:

1. nn.DataParallel: This package allows you to perform parallel training in a single machine with multiple GPUs. One advantage is that it requires a minimum code.
2. nn.DistributedDataParallel: This package allows you to perform parallel training across multiple GPUs within multiple machines. It requires a few more extra steps to configure the training process.  
3. torch.distributed.rpc: This package allows you to perform a model-parallelism strategy. It is very efficient if your model is large and does not fit in a single GPU.

Advantages

1. It is easy to implement.
2. PyTorch is very user-friendly.
3. Offers data-parallelism and model-parallelism methods out-of-the-box.
4. The majority of the cloud computing platforms support PyTorch.

When to use PyTorch?

You should opt for PyTorch when:

• You have a huge amount of data because data parallelism is easy to implement. 

Related information

• Repository Link
• Documentations
• Data-parallelism: tutorial
• Model-parallelism: tutorial

2. DeepSpeed

PyTorch distributed training specializes in data parallelism. DeepSpeed, which is built on top of PyTorch, targets other aspects i.e. model-parallelism. DeepSpeed is developed by Microsoft that aims to offer distributed training for large-scale models. 

DeepSpeed can efficiently tackle memory challenges when training models with trillions of parameters. It reduces memory footprint while maintaining compute and communication efficiency. Interestingly, DeepSpeed offers 3D parallelism through which you can distribute data, model, and pipeline, which basically means that now you can train a model which is large and consumes a huge amount of data, something like a GPT-3 or a Turing NLG.  

Advantages

1. Model scaling up to trillions of parameters.
2. Faster training up to 10X.
3. Democratize AI which means users can run bigger models on a single GPU without running out of memory.
4. Compressed training allows users to train attention models by reducing the memory required to compute attention operations. 
5. Easy to learn and use. 

When to use DeepSpeed?

You should opt for DeepSpeed when:

• You want to do data and model parallelism. 
• If your codebase is based on PyTorch. 

Related information

• Repository Link
• Documentations
• Tutorial

3. Distributed TensorFlow

TensorFlow is developed by Google and it supports distributed training. It uses data-parallel techniques for training. You can leverage the distributed training on TensorFlow by using the tf.distribute API. This API allows you to configure your training as per your requirements. By default, TensorFlow uses only one GPU but the tf.distribute allows you to use multiple GPUs.

TensorFlow provides three primary types of distributed training strategy:

1. tf.distribute.MirroredStrategy(): This simple strategy allows you to distribute training across multiple GPUs on a single machine. This method is also called Synchronous Data-Parallelism. It is worth noting that each worker node will have its own set of gradients. These gradients are then averaged and used to update the model parameters.

2. tf.distribute.MultiWorkerMirroredStrategy(): This strategy allows you to distribute training across multiple machines and multiple GPUs on a single machine. All the operations are similar to tf.distribute.MirroredStrategy(). It is also a Synchronous Data-Parallelism method.

3. tf.distribute.experimental.ParameterServerStrategy(): This is an Asynchronous Data-Parallelism method, it is common practice to scale-up model training on multiple machines. In this strategy, the parameters are stored in a parameter server and workers are independent of each other. This strategy scales up well because the worker nodes are not waiting for the parameter update from each other.

Advantages

1. Huge community support. 
2. It is a static paradigm of programming.
3. Very well integrated with Google Cloud and other cloud-based services. 

When to use Distributed TensorFlow?

You should use Distributed TensorFlow:

• If you want to do data parallelism. 
• If you like the static paradigm of programming compared to dynamic.
• If you are in the Google Cloud ecosystem since TensorFlow is very well optimized for TPUs. 
• Lastly, if you have huge data and need high processing power. 

Related information

• Repository Link
• Documentations
• Tutorial

4. Mesh TensorFlow

Mesh Tensorflow is again an extension of Tensorflow distributed training but is specifically designed to train large DL models on Tensor Processing Unit (TPUs) which AI accelerates like the GPUs but faster. Although Mesh TensorFlow can execute data-parallelism, it aims to solve distributed training for large models whose parameters cannot fit on one device. 

Mesh TensorFlow is inspired by a synchronous data-parallel method, i.e. every worker is involved in every operation. Apart from that, all the workers will have the same program and it uses collective communication like Allreduce. 

Advantages

1. It can train large models with millions and billions of parameters like: GPT-3, GPT-2, BERT, et cetera. 
2. Potentially low latency across the workers. 
3. Good TensorFlow community support. 
4. Availability of TPU-pods from Google. 

When to use Mesh Tensorflow?

You should use Mesh TensorFlow:

• If you want to do model parallelism. 
• If you want to develop huge models and practice rapid-prototyping.
• If you are especially working in the area of Natural Language Processing with huge data. 

Related information

• Repository Link

5. TensorFlowOnSpark

Apache Spark is one of the most well-known, open-source big data processing platforms. It allows users to do all kinds of data-related work like data engineering, data science, and machine learning. We already know what TensorFlow is. But if you wanna use TensorFlow on Apache Spark then you have to use TensorFlowOnSpark. 

TensorFlowOnSpark is a machine learning framework that allows you to perform distributed training on Apache Spark Clusters and Apache Hadoop. It was developed by Yahoo. The framework allows both distributed training and inference with minimum code changes to existing TensorFlow code on the shared grid. 

Advantages

1. Allows easy migration to Spark Clusters with existing TensorFlow programs. 
2. Fewer changes in the code. 
3. All TensorFlow functionalities are available. 
4. Datasets can be efficiently pushed and pulled by Spark and TensorFlow respectively. 
5. Cloud development is easy and efficient on CPUs or GPUs. 
6. Training pipelines can be created easily. 

When to use TensorFlowOnSpark?

You should use TensorflowOnSpark:

• If your workflow is based on Apache Spark or if you prefer Apache Spark.
• If your preferred framework is TensorFlow. 

Related information

• Repository Link
• Documentations
• Examples

6. BigDL

BigDL is also an open-source framework for distributed training for Apache Spark. It was developed by Intel to allow DL algorithms to run Hadoop and Spark clusters. One big advantage of BigDL is that it can help you to easily build and process production data in an end-to-end pipeline for both data analysis and deep learning applications. 

BigDL provides two options:

1. You can directly use BigDL as you would any other library that Apache Spark provides for data engineering, data analytics et cetera. 
2. You can scale out python libraries like PyTorch, TensorFlow, and Keras in the Spark ecosystem. 

Advantages

1. End-to-end pipeline: If your big data is messy and complex which is usually in the case of live data streaming, then adopting BigDL is appropriate because it integrates data analytics and deep learning in an end-to-end pipeline. 
2. Efficiency: With an integrated approach across different components of Spark BigDL makes development, deployment, and operations direct, seamless, and efficient across all components. 
3. Communication and Computing: Since all the hardware and software are stitched together, they run smoothly without any interruption, making communication with different workflows clear and computing faster. 

When to use BigDL?

You should use BigDL:

• If you want to develop an Apache Spark workflow, 
• If your preferred framework is PyTorch.
• If you want to have continuous integration of all the components like data mining, data analytics, machine learning et cetera. 

Related information

• Repository Link
• Documentations
• Tutorial

7. Horovod

Horovod was introduced by Uber in 2017. It is an open-source project that is specifically made for distributed training. It is an internal component of Michelangelo, a deep learning toolkit that Uber uses for implementing its DL algorithms. Horovod leverages data-parallel distributed training, which makes scaling very easy and efficient. It can also scale up to hundreds of GPUs in a matter of 5 lines of python code. The idea is to write a training script for a single GPU and Horovod can scale it to train on multiple in parallel. 

Horovod is built for frameworks like Tensorflow, Keras, Pytorch, and Apache MXNet. It is easy to use and fast. 

Advantages

1. Easy to learn and implement if you are familiar with Tensorflow, Keras, Pytorch, and Apache MXNet.
2. If you are using Apache Spark then you can unify all the processes on a single pipeline.
3. Good community support.
4. It is fast. 

When to use Horovod?

You should use Horovod:

• If you want to scale a single GPU script quickly across multiple GPUs.
• If you are using Microsoft Azure as your cloud computing platform. 

Related information

• Repository Link
• Documentations
• Tutorial

8. Ray

Ray is another open-source framework for distributed training built on top of Pytorch. It provides tools for launching GPU clusters on any cloud provider. Unlike any other libraries we have discussed so far, Ray is very flexible and can work anywhere like Azure, GCD, AWS Apache Spark, and Kubernetes. 

Ray offers the following libraries in its bundle for hyperparameter tuning, reinforcement learning, deep learning, scaling et cetera:

1. Tune: Scalable Hyperparameter Tuning.
2. RLlib: Distributed Reinforcement Learning.
3. Train: Distributed Deep Learning, currently in beta version. 
4. Datasets: Distributed Data Loading and Compute, currently in beta version. 
5. Serve: Scalable and Programmable Serving.
6. Workflows: Fast, Durable Application Flows.

Apart from these libraries, Ray also has integration with third-party libraries and frameworks which allows you to develop, train and scale your workloads with minimal code changes. Given below is the list of integrated libraries:

1. Airflow
2. ClassyVision
3. Dask
4. Flambe
5. Horovod
6. Hugging Face Transformers
7. Intel Analytics Zoo
8. John Snow Labs’ NLU
9. LightGBM
10. Ludwig AI
11. MARS
12. Modin
13. PyCaret
14. PyTorch Lightning
15. RayDP
16. Scikit Learn
17. Seldon Alibi 
18. Spacy
19. XGBoost

Advantages

1. It supports Jupyter Notebook 
2. It makes your code run parallel in single and multiple machines 
3. It integrates multiple frameworks and libraries. 
4. It works with all the major cloud computing platform

When to use Ray?

You should use Ray:

1. If you want to perform distributed reinforcement learning
2. If you want to perform distributed hyperparameter tuning
3. If you want to use distributed data loading and compute across different machines. 
4. If you want to serve your application.

Related information

• Repository Link
• Documentations
• Tutorial

Cloud platforms for distributed training

So far, we have discussed the frameworks and the libraries that can be used to enable distributed training. Now, let’s discuss and explore the cloud platforms where you can get into the hardware that will allow you to efficiently train your DL models. But before that, let’s lay out some criteria that will allow you to choose the best cloud platforms as per your requirements. 

1. Hardware and Software Support: It is important to learn and understand what hardware these platforms offer like GPUs, TPUs, storage units et cetera. Apart from that, one should also see the API that they offer so that (depending on your project) you can get access to hosting facilities, containers, tools for data analytics and so forth. 
2. Availability Zones: Availability zones are an important factor in cloud computing, it gives users the flexibility to set up and deploy their project anywhere in the world. Users can also shift their projects whenever they want to. 
3. Pricing: Whether the platform charges you based on your usage or do they offer a subscription-based model. 

Now, let’s discuss cloud computing options. We will discuss the two extremely feasible ready-to-use experimental notebook platforms and the three most popular cloud computing services. 

1. Google Colab

Google Colab is one most reliable and easy to use platforms for small scale to medium-scale projects. One good thing about Google Colab is that you can easily connect to Google Cloud with ease and you can work with any python library mentioned above. It offers three models:

1. Google Colab is free of cost and it gives you access to GPUs and TPUs. But you will get access to limited storage and memory. Once any of them exceeds, the program stops. 
2. Google Colab Pro is a subscription version of Google Colab where you have extra memory and storage. You can fairly run a heavy model but again is it limited. 
3. Google Colab Pro + is the new service which is a subscription based model and which is also expensive. It offers faster GPUs and TPUs plus extra memory so that you can run fairly larger models on fairly large datasets. 

Given below is the official comparison of all the three. 

2. Amazon Web Services: SageMaker

AWS SageMaker is one of the most popular and the oldest cloud computing platforms for distributed training. It is very well integrated with Apache MXNet, Pytorch, and TensorFlow and allows you to deploy deep learning algorithms with ease and less code modification. SageMaker API has 18+ machine learning algorithms, some of which are rewritten from scratch to make the whole process scalable and easy. These built-in algorithms are optimized to get the most out of the hardware. 

SageMaker also has an integrated Jupyter Notebook that allows Data-scientist and machine learning engineers to build and develop pipeline algorithms on the go and allows you to directly deploy them in a hosted environment. You can configure hardware and environments based on your requirements and preferences from SageMaker Studio or SageMaker console. All the hosting and development are billed according to the usage per minute. 

AWS SageMaker offers both data-parallelism as well as model-parallelism distributed training. In fact, SageMaker also offers a hybrid training strategy where you can use both model and data parallelism. 

3. Google Cloud Computing

Google Cloud Computing was developed by Google in 2010 to strengthen their own platforms like Google search engine and Youtube. Gradually, they started open-sourcing it to the public. Google Cloud Computing offers the same infrastructure that all Google’s platforms use. 

Google cloud computing offers in-built support for libraries like TensorFlow, Pytorch, Scikit-Learn, and many more. Furthermore, apart from configuring GPUs in your workflow, you can add TPUs as well to make the training process go much faster. Like I mentioned before you can connect your Google Colab to Google Cloud Platform and access all the features that it provides. 

Some of the features that it provides are: 

1. Compute (Virtual Hardwares like GPUs and TPUs)
2. Storage Bucket, 
3. Databases 
4. Networking
5. Management tools
6. Security
7. IoT
8. API platform
9. Hosting Services

It is worth noting that GCP has less availability zones but it is also less expensive compared to AWS. 

4. Microsoft Azure

Microsoft Azure is another very popular cloud computing platform. One of the popular language models GPT-3 from OpenAI was trained in Azure. It also offers both data parallelism and model parallelism methods and supports both TensorFlow and Pytorch. In fact, if you want to optimize computing speed then you can also leverage Horovod from Uber.

Azure machine learning service is for both coders and non-coders. It simply offers a drag and drop approach that can optimize your workflow. It also reduces manual work with automated machine learning that can help you to develop smarter working prototypes. 

The Azure Python SDK also allows you to interact in any Python environment like Jupyter Notebooks, Visual Studio Code, and many more. It is quite similar to both AWS and GCP in terms of offering services. These are the services that Azure offers:

1. AI, Machine Learning and Deep learning
2. Computing powers (GPUs) 
3. Analytics 
4. Blockchain
5. Containers
6. Databases
7. Developer Tools
8. DevOps
9. Internet of Things
10. Mixed Reality
11. Mobile
12. Networking et cetera

Let’s also compare the main 3 tools side-by-side to give you a better perspective about making the choice.

Comparison table for cloud platform

Final thoughts

In this article, we saw different Libraries and tools that can help you implement distributed training for your own deep learning application. Bear in mind that all the libraries are good and very effective in what they do, eventually, it all boils down to your preferences and requirements. 

You must have noticed that all the frameworks discussed have primarily Pytorch and TensorFlow integration in some way or another. This trait can easily help you isolate the framework of choice. Once your framework is decided you can then look at the advantages to decide which distributed training tool works the best for you. 

I hope you enjoyed this article. If you wanna try out all the frameworks we discussed then follow the tutorial link. 

Thanks for reading!

References

• https://neptune.ai/blog/ml-model-monitoring-best-tools
• https://neptune.ai/blog/best-mlops-tools
• https://towardsdatascience.com/how-to-train-your-deep-learning-models-in-a-distributed-fashion-43a6f53f0484
• https://analyticsindiamag.com/top-distributed-training-frameworks-in-2021/
• https://www.telesens.co/2017/12/25/understanding-data-parallelism-in-machine-learning/
• https://towardsdatascience.com/distributed-training-on-aws-sagemaker-8bcbea28466c
• https://docs.microsoft.com/en-us/azure/machine-learning/how-to-train-distributed-gpu
• https://www.telesens.co/2017/12/25/understanding-data-parallelism-in-machine-learning/
• https://arxiv.org/pdf/1811.02084.pdf
• https://developer.yahoo.com/blogs/157196317141/
• http://www.vldb.org/pvldb/vol13/p3005-li.pdf
• https://arxiv.org/pdf/1804.05839.pdf

We already know how efficient machine learning systems are to process the huge amount of data and based upon the task in hand, yield results in real-time as well. But these systems need to be curated and deployed properly so that the task at hand performs efficiently. This article aims to provide you with information on the model deployment strategies and how you can choose which strategy is best for your application.

We will cover the following strategies and techniques for model deployment:

These strategies can be broken down into two categories:

• Static deployment strategies: These are the strategies where the distribution of traffic or request are handled manually. Examples of this are shadow evaluation, A/B testing, Canary testing, Rolling deployment, Blue-green deployment et cetera. 
• Dynamic deployment strategies: These are the strategies where the distribution of traffic or request are handled automatically. Example of this is Multi Arm Bandits. 

To begin with, let’s have a quick overview of what model lifecycle and model deployment refers to. 

Lifecycle of an ML model

The lifecycle of a machine learning model refers to the entire process that structures the whole data science or an AI project. It is similar to the software development life cycle (SDLC) but differs in a few key areas such as the use of real-time data to evaluate the model performance before deployment. A life cycle of the ML model or model development life cycle (MDLC) primarily has five phases: 

Now, another term that you must be familiar with is MLOps. MLOps is generally a set of practices that enables ML Lifecycle. Its stitches machine learning and software applications together. Simply put, it is a collaboration between data scientists and the operations team that takes care of and orchestrates the whole ML lifecycle. The three key areas that MLOps focuses on are continuous integration, continuous deployment, and continuous testing.

What is model deployment (or model release)?

Model deployment (release) is a process that enables you to integrate machine learning models into production to make decisions on real-world data. It is essentially the second last stage of the ML life cycle before monitoring. Once deployed, the model further needs to be monitored to check whether the whole process of data ingestion, feature engineering, training, testing et cetera are aligned properly so that no human intervention is required and the whole process is automatic.  

But before deploying the model, one has to evaluate and test if the trained ML model is fit to be deployed into production. The model is tested for performance, efficiency, even bugs, and issues. There are various strategies one can use before deploying the ML model. Let us explore them.

Model deployment strategies

Strategies allow us to evaluate the ML model performances, capabilities and discover issues concerning the model. A key point to keep in mind is that the strategies usually depend on the task and resources in hand. Some of the strategies can be a great resource but computationally expensive while some can get the job done with ease. Let’s discuss a few of them.

1. Shadow deployment strategy

In shadow deployment or shadow mode, the new model is deployed with new features alongside the live model. The new deployed model in this case is known as a shadow model. The shadow model handles all the requests just like the live model except it is not released to the public. 

This strategy allows us to evaluate the shadow model better by testing it on real-world data while not interrupting the services offered by the live model. 

Methodology: champion vs challenger

In shadow evaluation, the request is sent to both the models running parallel to each other using two API endpoints. During the inference, predictions from both the models are computed and stored, but only the prediction from the live model is used in the application which is returned to the users.

The predicted values from both the live and shadow model are compared against the ground truth. Once the results are in hand, data scientists can decide whether to deploy the shadow model globally into production or not. 

But one can also use champion/challenger framework in a manner where multiple shadow models are tested and compared with the existing model. Essentially the model with the best accuracy or Key Performance Index (KPI) is selected and deployed. 

Pros:

• Model evaluation is efficient since both the models are running parallelly there is no impact on traffic.  
• No overloading irrespective of the traffic. 
• You can monitor the shadow model which allows you to check the stability and performance; this reduces risk. 

Cons:

• Expensive because of the resources required to support the shadow model. 
• Shadow deployment can be tedious, especially if you are concerned about different aspects of model performance like metrics comparison, latency, load testing, et cetera.
• Provides no user response data. 

When to use it?

• If you want to compare multiple models with each other then shadow testing is great, although tedious. 
• Shadow testing will allow you to evaluate the pipeline, latency while yielding results as well the load-bearing capacity. 

2. A/B testing model deployment strategy

A/B testing is a data-based strategy method. It is used to evaluate two models namely A and B, to assess which one performs better in a controlled environment. It is primarily used in e-commerce websites and social media platforms. With A/B testing the data scientists can evaluate and choose the best design for the website based on the data received from the users.  

The two models differ slightly in terms of features and they cater to different sets of users. Based on the interaction and data received from the users such as feedback, data scientists choose one of the models that can be deployed globally into production. 

Methodology

In A/B the two models are set up parallelly with different features. The aim is to increase the conversion rate of a given model. In order to do that data scientist sets up a hypothesis. A hypothesis is an assumption based on an abstract intuition of the data. This assumption is proposed through an experiment, if the assumption passes the test it is accepted as fact and the model is accepted, otherwise, it’s rejected. 

Hypothesis testing

In A/B testing there are two types of hypothesis: 

In hypothesis testing, the aim is to reject the null hypothesis by setting up experiments like the A/B testing and exposing the new model with a certain feature to a few users. The new model essentially is designed on an alternate hypothesis. If the alternate hypothesis is accepted and the null hypothesis is rejected then that feature is added and the new model is deployed globally. 

It is important to know that in order to reject the null hypothesis you have to prove the statistical significance of the test.

Advantages:

• It is simple. 
• Yields quick results and helps in the elimination of the low performing model.

Disadvantages:

• Models can be unreliable if the complexity is increased. One should use A/B testing in the case of simple hypothesis testing. 

When to use it?

As mentioned earlier, A/B testing is predominantly used for e-commerce, social media platforms, and online streaming platforms. In such a setting and if you have two models you can use A/B to evaluate and choose which one to deploy globally. 

3. Multi Armed Bandit

Multi-Armed Bandit or MAB is an advanced version of A/B testing. It is also inspired by reinforcement learning, and the idea is to explore and exploit the environment that maximizes the reward function.

MAB leverages machine learning to explore and exploit the data received to optimize the key performance index (KPI). The advantage of using this technique is that the user traffic is diverted according to the KPI of two or more models. The model which yields the best KPI is deployed globally. 

Methodology

MAB heavily depends on two concepts: exploration and exploitation. 

Exploration: It is a concept where the model explores the statistically significant results, as what we saw in A/B testing. The prime focus of A/B testing is to find or discover conversion rates of the two models. 

Exploitation: It is a concept where the algorithm uses a greedy approach to maximize conversion rates using the information it gained during exploring. 

MAB is very flexible compared to the A/B testing. It can work with more than two models at a given time, this increases the rate of conversion. The algorithm continuously logs the KPI score of each model based on the success with respect to the route from which the request was made. This allows the algorithm to update its score of which is best.  

Advantages:

• With exploring and exploiting the MAB offers adaptive testing.
• Resources are not wasted like in A/B testing. 
• Faster and efficient way of testing. 

Disadvantages:

• It is expensive because exploiting takes a lot of computing power which can be economically expensive. 

When to use it?

MAB is very helpful for scenarios where the conversion rate is all you care about and where the time to make a decision is small. For example, optimizing offers or discounts on a product for a limited period. 

4. Blue-green deployment strategy

Blue-green deployment strategies involve two production environments instead of just models. The blue environment consists of the live model whereas the green environment consists of the new version of the model. 

The green environment is set as a staging environment i.e. an exact replica of a live environment but with new features. Let us briefly understand the methodology. 

Methodology

In Blue-green deployment, the two identical environments consist of the same database, containers, virtual machines, same configuration et cetera. Keep in mind that setting up an environment can be expensive so usually, some components like a database are shared between the two. 

The Blue environment which contains the original model is live and keeps servicing requests while the Green environment acts as a staging environment for a new version of the model. It is subjected to deployment and final stages of testing against the real data to ensure that it performs well and is ready to deploy to production. Once the testing is successfully completed ensuring that all the bugs and issues are rectified the new model is made live. 

Once this model is made live, the traffic is diverted from the blue environment to the green environment. In most cases, the blue environment serves as a backup, in case something goes wrong the request can be rerouted to the blue model.

Pros:

• It ensures application availability round the clock.
• Rollbacks are easy because you can quickly divert the traffic to the blue environment in case of any issues. 
• Since both environments are independent of each other, deployment risk is less.

Cons:

• It is cost expensive since both models require separate environments.

When to use it?

In case your application cannot afford downtime then one should use the Blue-Green deployment strategy. 

5. Canary deployment strategy

The canary deployment aims to deploy the new version of the model by gradually increasing the number of users. Unlike the previous strategies that we’ve seen where the new model is either hidden from the public or a small control group is set up, the canary deployment strategy uses the real users to test the new model. As a result, bugs and issues can be detected before the model is deployed globally for all the users.

Methodology

Similar to other deployment strategies in canary deployment, the new model is tested alongside the current live model but here the new model is tested on a few users to check its reliability, errors, performance et cetera. 

The number of users can be increased or decreased based on the testing requirements. If the model is successful in the testing phase then the model can be rolled out and if it is not then it can be rolled back with no downtime but only a number of users will be exposed to the new model.

Canary deployment strategy can be broken down into three steps:

Pros:

• Cheaper compared to Blue-Green deployment.
• Ease to test the new model against real data.
• Zero downtime. 
• In case of failure, the model could be easily rolled back to the current version.

Cons:

• Rollouts are easy but slow.
• Since the testing takes place against the real data with few users, proper monitoring must be in place so in case of failure the users are effectively routed to the live version. 

When to use it?

Canary deployment strategy must be used when the model is to be evaluated against real-world real-time data. Also, it has advantages over A/B testing since it can take a long time to gather enough data from the user to find a statistically significant result. Canary deployment can do this in hours. 

6. Other model deployment strategies and techniques

Feature flag 

Feature flag is a technique rather than a strategy that allows developers to push or integrate code into the main branch. The idea here is to keep the feature dormant until it is ready. This allows developers to collaborate on different ideas and iterations. Once the feature is finalized it can be activated and deployed. 

As mentioned earlier feature flag is a technique so this can be used in combination with any deployment techniques mentioned earlier. 

Rolling deployment

Rolling deployment is a strategy that gradually updates and replaces the older version of the model. This deployment occurs in a running instance, it does not involve staging or even private development. 

The image above represents how the rolling deployment works. As you can see that the service is horizontally scaled this is the key factor. 

The image at the top left represents three instances. In the next step version 1.2 is deployed. With deployment of a single instance of version 1.2, one instance of version 1.1 is retired. The same trend follows for all other instances i.e. whenever a new instance is deployed the older instances are retired. 

Pros:

• It is faster than a blue/green deployment because there are no environmental restrictions. 

Cons:

• Although it is quicker, rollbacks can be difficult if further updates fail.

Recreate strategy

Recreate is a simple strategy where the live version of the model is shut down and then the new version is deployed. 

The image above depicts how the recreate strategy works. Essentially, the old instances namely V1’s are shut down and discarded while the new instances V2’s are deployed.

Pros:

• Easy and simple set-up.
• The entire environment is completely renewed.

Cons:

• Negative impact on users since it suffers from downtime as well as rebooting.

Comparison: which model release strategy to use?

There can be various metrics that one can use to determine which strategy will suit them the best. But it mostly depends on the project complexity and resource availability. The following comparison table gives some idea about when to use which strategy.

Key takeaways

Deployment strategies often help data scientists to figure out how their model is performing in a given situation. A good strategy depends upon the type of product and users it aims to target. To sum it up, here are the points one should keep in mind:

• If you want the model to be tested in real-world data then a shadow evaluation strategy or something similar to it must be considered. Unlike the other strategies where the sample of users are used, the shadow evaluation strategy uses live and real user requests. 
• Check the complexity of the task, if the model requires simple or minor tweaks then A/B testing is the way to go. 
• If there is time constraint and ideas are more, then one should opt for multiarm bandits since it gives you the best results in such a situation. 
• If your model is complex and needs proper monitoring before deploying then Blue-green strategy will help you analyse and monitor your model. 
• If you want no downtime and you are okay to expose your model to the public then opt for Canary deployment. 
• The rolling deployment must be used when you want to gradually deploy the new version of the model. 

Hope you guys enjoyed reading this article. If you want to read more about this topic, you can refer to the attached resources. Keep learning!

References

1. A/B Testing for Data Science using Python – A Must-Read Guide for Data Scientists
2. Deploying Machine Learning Models in Shadow Mode
3. The Machine Learning Lifecycle in 2021
4. Machine Learning Life-cycle Explained!
5. Automatic Canary Releases for Machine Learning Models
6. Intro To Deployment Strategies: Blue-Green, Canary, And More
7. Strategies To Deploy Your Machine Learning Models
8. Machine Learning Deployment Strategies
9. What is Blue Green Deployment ?
10. Safely Rolling Out ML Models To Production
11. Minimize Your A/B Test Losses Due to Low-Performing Variations
12. The Ultimate Guide to Deploying Machine Learning Models
13. Machine Learning Deployment: Shadow Mode
14. Multi-armed bandit
15. Sequential A/B Testing vs Multi-Armed Bandit Testing
16. What Is Blue-Green Deployment?
17. Pros and Cons of Canary Release and Feature Flags in Continuous Delivery
18. Rolling Deployments
19. Rolling Deployment: What This Is and How it De-Risks Software Deploys
20. Shadow Deployments of Machine Learning Models in AWS
21. Deploy shadow ML models in Amazon SageMaker
22. Shadow AB test pattern
23. MLOps Explained
24. MLOps: What It Is, Why It Matters, and How to Implement It
25. Dynamic A/B testing for machine learning models with Amazon SageMaker MLOps projects
26. Multi-Arm Bandits for recommendations and A/B testing on Amazon ratings data set
27. Automate Canary Testing for Continuous Quality
28. CanaryRelease
29. What Is the Best Kubernetes Deployment Strategy?

In deep generative modeling, deep neural networks learn a probability distribution over a given set of data points and generate similar ones. Since it is an unsupervised learning task, it uses no labels during the learning process. 

Since its release in 2014, the deep learning community has been actively developing new GANs to improve the field of generative modeling. This article aims to provide information on GANs, specifically Pix2Pix GAN, one of the most used generative models.

What is GAN?

GAN was designed by Ian Goodfellow in 2014. GAN’s main intention was to generate samples that were not blurry and had rich representations of features. Discriminative models were doing well on this front as they were able to classify between different classes. Deep generative models, on the other hand, were far less effective due to the difficulty in approximating many intractable probabilistic computations, which are quite evident in Autoencoders.  

Autoencoders and their variants are explicit likelihood models, meaning they explicitly compute the probability density function over a given distribution. GANs and their variants are implicit likelihood models, which means they don’t compute the probability density function but rather learn the underlying distribution. 

GANs learn the underlying distribution by approaching the whole problem as a binary classification problem. In this approach, the problem model is presented by two models: a generator and a discriminator. The job of the generator is to generate new samples and the job of the discriminator is to classify or discriminate if the sample produced by the generator is real or fake. 

The two models are trained together in a zero-sum game until the generator can produce samples that are similar to the real samples. In other words, they are trained until the generator can fool the discriminator. 

Architecture of a vanilla GAN

Let’s briefly understand the architecture of GANs. From this section onward most of the topics will be explained using code. So to begin with, let’s install and import all the required dependencies:

pip install torch torchvision matplotlib cv2 numpy

import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
import matplotlib.pyplot as plt
import torchvision
import torchvision.datasets as datasets
from torch.utils.data import DataLoader
import torchvision.transforms as transforms

Generator

The Generator is a component in GAN that takes in noise, which by definition follows a Gaussian distribution, and yields samples similar to the original dataset. As GANs have evolved over the years, they have adopted the use of CNNs, which is quite prominent in computer vision tasks. But for simplicity, we will define it with just linear functions using Pytorch.

class Generator(nn.Module):
   def __init__(self, z_dim, img_dim):
       super().__init__()
       self.gen = nn.Sequential(
           nn.Linear(z_dim, 256),
           nn.LeakyReLU(0.01),
           nn.Linear(256, img_dim),
           nn.Tanh(),  # normalize inputs to [-1, 1] to make outputs [-1, 1]
       )

   def forward(self, x):
       return self.gen(x)

Discriminator

The discriminator is simply a classifier that classifies whether the data yielded by the generator is real or fake. It does this by learning the original distribution from the real data and then evaluating between the two. We will keep things simple and define the discriminator using linear functions. 

class Discriminator(nn.Module):
   def __init__(self, in_features):
       super().__init__()
       self.disc = nn.Sequential(
           nn.Linear(in_features, 128),
           nn.LeakyReLU(0.01),
           nn.Linear(128, 1),
           nn.Sigmoid(),
       )

   def forward(self, x):
       return self.disc(x)

The key difference between the generator and the discriminator is the last layer. The former yields the same shape as that of the image while the latter yields only one output, either 0 or 1. 

Loss function and training

The loss function is one of the most important components in any deep learning algorithm. For instance, if we design a CNN to minimize the Euclidean distance between the ground truth, and predicted results, it will tend to produce blurry results. This is because Euclidean distance is minimized by averaging all plausible outputs, which cause blurring. 

Related

Understanding GAN Loss Functions

Read more

The above point is an important one that we must keep in mind. With that being said, the loss function that we will use for vanilla GAN will be binary cross-entropy loss or BCELoss because we are performing binary classification. 

criterion = nn.BCELoss()

Now let’s define the optimization method and other related parameters:

# Define hyperparameters
device = "cuda" if torch.cuda.is_available() else "cpu"
lr = 3e-4
z_dim = 64
image_dim = 28 * 28 * 1  # 784
batch_size = 32
num_epochs = 100

# Initialize Generator and Discriminator
gen = Generator(z_dim, image_dim).to(device)
disc = Discriminator(image_dim).to(device)

# Set up optimizers
opt_disc = optim.Adam(disc.parameters(), lr=lr)
opt_gen = optim.Adam(gen.parameters(), lr=lr)

# Prepare dataset and dataloader
transform = transforms.Compose([
   transforms.ToTensor(),
   transforms.Normalize((0.5,), (0.5,))
])
dataset = datasets.MNIST(root="dataset/", transform=transform, download=True)
loader = DataLoader(dataset, batch_size=batch_size, shuffle=True)

# Set up TensorBoard writers
from torch.utils.tensorboard import SummaryWriter
writer_fake = SummaryWriter(f"logs/fake")
writer_real = SummaryWriter(f"logs/real")
step = 0

Let’s understand the training loop. The training loop of GAN starts with:

1. Generating samples from the generator using Gaussian distribution 
2. Training the discriminator using real data and fake data produced by the generator
3. Updating the discriminator
4. Updating the generator

Here’s what the training loop looks like:

for epoch in range(num_epochs):
  # Loop over each batch of data
  for batch_idx, (real, _) in enumerate(loader):
      real = real.view(-1, 784).to(device)
      batch_size = real.shape[0]
      ### Train Discriminator: max log(D(x)) + log(1 - D(G(z)))
# Generate random noise as input to the generator
      noise = torch.randn(batch_size, z_dim).to(device)
      # Produce fake images from the generator using the random noise
      fake = gen(noise)
      # Get discriminator's predictions on real images
      disc_real = disc(real).view(-1)
      lossD_real = criterion(disc_real, torch.ones_like(disc_real))
      # Get discriminator's predictions on fake images
disc_fake = disc(fake).view(-1)
      # Calculate discriminator's loss on fake images
lossD_fake = criterion(disc_fake, torch.zeros_like(disc_fake))
      # Combine real and fake loss to get total discriminator loss
lossD = (lossD_real + lossD_fake) / 2
# Backpropagation and optimization step for discriminator
      disc.zero_grad()
      lossD.backward(retain_graph=True)
      opt_disc.step()
      ### Train Generator: min log(1 - D(G(z))) <-> max log(D(G(z))
      # where the second option of maximizing doesn't suffer from
      # saturating gradients
      output = disc(fake).view(-1)
      lossG = criterion(output, torch.ones_like(output))
# Backpropagation and optimization step for generator
      gen.zero_grad()
      lossG.backward()
      opt_gen.step()
      if batch_idx == 0:
          print(
              f"""Epoch [{epoch}/{num_epochs}] Batch {batch_idx}/{len(loader)}
                    Loss D: {lossD:.4f}, loss G: {lossG:.4f}"""
          )
          with torch.no_grad():
              # fixed_noise is not defined earlier in the code
              # It should be a constant noise vector used to generate consistent samples
              # Let's define it at the beginning of the training loop
              if 'fixed_noise' not in locals():
                  fixed_noise = torch.randn(64, z_dim, device=device)
              fake = gen(fixed_noise).reshape(-1, 1, 28, 28)
              data = real.reshape(-1, 1, 28, 28)
              img_grid_fake = torchvision.utils.make_grid(fake, normalize=True)
              img_grid_real = torchvision.utils.make_grid(data, normalize=True)
              writer_fake.add_image(
                  "Mnist Fake Images", img_grid_fake, global_step=step
              )
              writer_real.add_image(
                  "Mnist Real Images", img_grid_real, global_step=step
              )
              step += 1

There are some important considerations from the loop above:

1. The loss function for the discriminator is calculated twice: one for real images and another for fake images.
   • For real images, the ground truth is converted to ones using the torch.ones_like function which returns a matrix of ones of a defined shape.  
   • For fake images, the ground truth is converted to ones using the torch.zeros_like function which returns a matrix of zeros of a defined shape.  
2. The loss function for the generator is calculated only once. If you observe carefully, it is the same loss function that is used by the discriminator to calculate the loss for fake images. The only difference is that instead of using the torch.zeros_like function, torch.ones_like function is used. The interchanging of labels from 0 to 1 enables the generator to learn representations that will produce real images, therefore fooling the discriminator. 

Mathematically, we can define the whole process as:

Application of GANs

GANs are widely used for:

• Generating training samples: GANs are often used to generate samples for specific tasks like the classification of malignant and benign cancer cells, especially where the data is scarce to train a classifier. 
• AI Art or Generative Art: AI or Generative art is another new domain where GANs are extensively used. Since the introduction of non-fungible tokens, artists all over the world have been creating art in unorthodox fashion i.e. digital and generative. GANs like DeepDaze, BigSleep, BigGAN, CLIP, VQGAN, etc. are the most commonly used by creators. 

• Image-to-image translation: The idea here is to translate a certain type of image to an image in the target domain. For example a day-light image into a night image, or a winter image to a summer image (see the image below). GANs like pix2pix, cycleGAN, styleGAN are few of the most popular GANs used by digital creators.

• Text-to-image translation: Text-to-image translation is simply converting a text or a given string into an image. This is a very populardomain as of now and it is a growing community. As mentioned previously GANs such as DeepDaze, BigSleep and DALL·E from OpenAI are the most common tools for this.

Issues with GANs

Although GANs can produce images from random Gaussian distributions that are similar to real images, this process is not perfect most of the time. Here’s why:

• Mode Collapse:  This refers to the issue when the generator can fool the discriminator by learning with fewer data samples from the overall data. Because of mode collapse, the GAN is not able to learn a wide variety of distributions and remains limited to a few. 
• Diminished gradient: Diminished or vanishing gradient descent occurs when the derivative of the network is so small, that the update to the original weights is almost negligible. To overcome this issue, Wasserstein GANs (WGANs in short) are recommended. 
• Non-convergence: It occurs when the network is unable to converge to a global minimum. This results from unstable training, and it can be tackled with spectral normalization.

Related

Vanishing and Exploding Gradients in Neural Network Models: Debugging, Monitoring, and Fixing

Read more

Variations of GAN

Since the release of the first GAN, there have been many variants of GANs. Below are some of the most popular GANs:

• CycleGAN
• StyleGAN
• PixelRNN
• Text2image
• DiscoGAN
• IsGAN

Related

GAN Architectures You Really Should Know

Read more

This article solely focuses on Pix2Pix GAN. In the following section, we will understand some of the key components of the same like the architecture, loss function, etcetera. 

What is the Pix2Pix GAN?

Pix2Pix GAN is a conditional GAN (cGAN) that was developed by Phillip Isola, et al. Unlike vanilla GAN which uses only real data and noise to learn and generate images, cGAN uses real data andnoise as well as labels to generate images. 

In essence, the generator learns the mapping from the real data as well as the noise. 

The generator G combines the learnt real data x and the random noise z to output y, which is the fake data. 

Similarly, the discriminator not only learns from the “real data” example it has seen, but also from the labels that help it understand what is real and what is fake. 

The discriminator uses, then, two sources of information to improve its ability to tell real from fake: x (the real data) and y (the label saying “real” or “fake.”)

This setting makes cGAN to be suitable for image-to-image translation tasks, where the generator is conditioned on an input image to generate the corresponding output image. In other words, the generator uses a condition distribution (or data) such as a guide or a blueprint to generate a target image (see the image below).   

The idea with Pix2Pix relies on the dataset provided for the training. It is a pair-to-pair image translation with training examples {x, y} having a correspondence between them. 

Pix2Pix network architectures

The pix2pix has two important architectures, one for the generator and the other for the discriminator, namely U-net and patchGAN. Let’s explore both of them in more detail. 

U-Net generator 

As mentioned before, the architecture used in pix2pix is called U-net. U-net was primarily developed for biomedical image segmentation by Ronneberger et. al. in 2015. 

U-Net consists of two major parts: 

1. A contracting path made up of convolutional layers (left side) which downsamples the data while extracting information. 
2. An expansive path made of up transpose convolution layer (right side) which upsamples the information. 

Let’s say our downsampling has three convolutional layers C_l(1,2,3), then we have to make sure that our upsampling has three transpose convolutional layers C_u(1,2,3). This is because we want to connect the corresponding blocks of the same sizes using a skip connection. 

Downsampling

During downsampling, each convolutional block extracts spatial information and passes the information to the next convolutional block to extract more information until it reaches the middle part known as the bottleneck. Upsampling starts from the bottleneck. 

Upsampling

During upsampling, each transpose convolutional block expands information from the previous block while concatenating the information from the corresponding downsampling block. By concatenating information, the network can then learn to assemble a more precise output based on this information.

This architecture can localize, i.e. it can find the object of interest pixel by pixel. Furthermore, U-Net also allows the network to propagate context information from lower resolution to higher resolution layers. This allows the network to generate high-resolution samples. 

Markovian discriminator (PatchGAN)

The discriminator uses PatchGAN architecture. This architecture contains several transposed convolutional blocks. It takes an NxN part of the image and tries to find whether it is real or fake. N can be of any size. It can be smaller than the original image and it is still able to produce high-quality results. The discriminator is applied convolutionally across the whole image. Also, because the discriminator is smaller i.e. it has fewer parameters compared to the generator, it is faster. 

PatchGAN can effectively model the image as a Markov random field, where NxN is considered an independent patch. Therefore, PatchGAN can be understood as a form of texture/style loss.

Loss function

The loss function is: 

The equation above has two components: one for the discriminator and the other for the generator. Let’s understand both of them one by one. 

In any GAN, the discriminator is trained first in every iteration so that it can recognize both real and fake data. Essentially, 

D(x,y) = 1 i.e. real and, 

D(x,G(z)) = 0 i.e. fake. 

It is worth noting that G(z) will also produce fake samples and thus its value will be closer to zero. In theory, the discriminator should always classify G(z) as zero only. Therefore the discriminator should maintain a maximum distance between real and fake i.e. 1 and 0 in every iteration. In other words, the discriminator should maximize the loss function. 

After the discriminator, the generator is trained. The generator i.e. G(z) should learn to produce samples that are closer to the real samples. To learn the original distribution it takes help from the discriminator i.e. instead of D(x, G(z)) = 0, we change D(x, G(z)) = 1. 

With the alteration in labeling, the generator now optimizes its parameter concerning the parameter belonging to the discriminator with ground truth labels. This step ensures that the generator can now yield samples that are close to real data i.e. 1. 

The loss function is also mixed with an L1 loss so that the generator not only fools the discriminator but also produces images near the ground truth. In essence, the loss function has an additional L1 loss for the generator. 

Therefore, the final loss function is:

It is worth noting that the L1 loss can preserve low-frequency details in the image, but it will not be able to capture high-frequency details. Hence, it will still produce blurry images. To tackle this problem, PatchGAN is used. 

Optimization 

The optimization and training process is similar to vanilla GAN. However, the training itself is a difficult process since the objective function of GAN is more concave-concave rather than convex-concave. Because of this, it is difficult to find a saddle point and this is what makes training and optimizing the GANs difficult. 

As we saw previously, the generator is not trained directly but through the discriminator. This essentially limits the optimization of the generator. If the discriminator fails to capture high dimensional spaces then it is very certain that the generator will fail to produce good samples. On the other hand, if we can train the discriminator in a much more optimal way then we can be assured that the generator will be trained optimally as well. 

In the early stages of training, G is untrained and weak to produce good samples. This makes the discriminator very powerful, so instead of minimizing log(1 − D(G(z))), the generator is trained to maximize log D(G(z)). This creates some sort of stability in the early stages of the training. 

Other ways to tackle the instability are:

1. Using spectral normalization in every layer of the model
2. Using Wasserstein loss which calculates the average score for real or fake images.

Hands-on example with Pix2Pix

Let’s implement Pix2Pix with PyTorch and get an intuitive understanding of how it works and the various components behind it. This section will give you a clear understanding of how the Pix2Pix algorithm works. 

Let’s start by downloading the data. The following code can be used to download the data.

!wget http://efrosgans.eecs.berkeley.edu/pix2pix/datasets/facades.tar.gz
!tar -xvf facades.tar.gz

Data visualization

Once the data is downloaded, we can then visualize them to understand what are the necessary steps needed to format the data according to the requirement. 

We will import the following libraries for data visualization.

import matplotlib.pyplot as plt
import cv2
import os
import numpy as np

path = "facades/train/"
plt.imshow(cv2.imread(f"{path}91.jpg"))

From the image above, we can see that the data has two images attached together. If we then see the shape of the image above we find that the width is 512, which means that the image can be easily separated into two. 

print('Shape of the image: ',cv2.imread(f'{path}91.jpg').shape)

>> Shape of the image:  (256, 512, 3)

To separate the images we will use the following commands:

# Dividing the image by width
image = cv2.imread(f'{path}91.jpg')
w = image.shape[1]//2
image_real = image[:, :w, :]
image_cond = image[:, w:, :]
fig, axes = plt.subplots(1,2, figsize=(18,6))
axes[0].imshow(image_real, label='Real')
axes[1].imshow(image_cond, label='Condition')
plt.show()

Output:

The image on the left will be our ground truth while the image on the right will be our conditional image. We will refer to them as y and x respectively (from left to right). 

Creating dataloader

Dataloader is a function that will allow us to format the data as per the PyTorch requirement. This will involve two steps: 

1. Formatting the data, that is reading the data from the source, cropping them followed by converting them to Pytorch tensors. 

from glob import glob
from torch.utils.data import Dataset


class Data(Dataset):
   def __init__(self, path="facades/train/"):
       self.filenames = glob(path + "*.jpg")

   def __len__(self):
       return len(self.filenames)

   def __getitem__(self, idx):
       filename = self.filenames[idx]

       image = cv2.imread(filename)
       image_width = image.shape[1]
       image_width = image_width // 2
       real = image[:, :image_width, :]
       condition = image[:, image_width:, :]

       real = transforms.functional.to_tensor(real)
       condition = transforms.functional.to_tensor(condition)

       return real, condition

2. Loading the data by using Pytorch’s DataLoader function to create batches before feeding them into the neural nets. 

train_dataset = Data()
train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True)

val_dataset = Data(path="facades/val/")
val_loader = DataLoader(val_dataset, batch_size=4, shuffle=True)

Keep in mind that we will create a data loader for training and validation. 

Utils

In this section, we involve creating components that will be used to build the Generator and Discriminator. The components that we will create will be a convolutional function for downsampling and a transpose convolution function for upsampling which will be referred to as cnn_block and tcnn_block respectively. 

def cnn_block(
   in_channels, out_channels, kernel_size, stride = 1, padding = 0, first_layer = False
):

   if first_layer:
       return nn.Conv2d(
           in_channels, out_channels, kernel_size, stride = stride, padding = padding
       )
   else:
       return nn.Sequential(
           nn.Conv2d(
               in_channels, out_channels, kernel_size, stride = stride, padding = padding
           ),
           nn.BatchNorm2d(out_channels, momentum = 0.1, eps = 1e-5),
       )


def tcnn_block(
   in_channels,
   out_channels,
   kernel_size,
   stride = 1,
   padding = 0,
   output_padding = 0,
   first_layer = False,
):
   if first_layer:
       return nn.ConvTranspose2d(
           in_channels,
           out_channels,
           kernel_size,
           stride = stride,
           padding = padding,
           output_padding = output_padding,
       )

   else:
       return nn.Sequential(
           nn.ConvTranspose2d(
               in_channels,
               out_channels,
               kernel_size,
               stride = stride,
               padding = padding,
               output_padding = output_padding,
           ),
           nn.BatchNorm2d(out_channels, momentum = 0.1, eps = 1e-5),
       )

Defining parameters

In this section, we will define the parameters. These parameters will help us in training the neural network. 

# Define parameters
batch_size = 4
workers = 2

epochs = 30

gf_dim = 64
df_dim = 64

L1_lambda = 100.0

in_w = in_h = 256
c_dim = 3

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

Generator

Now, let’s define the generator. We will use the two components to define the same. 

class Generator(nn.Module):

class Generator(nn.Module):
 def __init__(self,instance_norm=False): #input : 256x256
   super(Generator,self).__init__()
   self.e1 = cnn_block(c_dim,gf_dim,4,2,1, first_layer = True)
   self.e2 = cnn_block(gf_dim,gf_dim*2,4,2,1,)
   self.e3 = cnn_block(gf_dim*2,gf_dim*4,4,2,1,)
   self.e4 = cnn_block(gf_dim*4,gf_dim*8,4,2,1,)
   self.e5 = cnn_block(gf_dim*8,gf_dim*8,4,2,1,)
   self.e6 = cnn_block(gf_dim*8,gf_dim*8,4,2,1,)
   self.e7 = cnn_block(gf_dim*8,gf_dim*8,4,2,1,)
   self.e8 = cnn_block(gf_dim*8,gf_dim*8,4,2,1, first_layer=True)

   self.d1 = tcnn_block(gf_dim*8,gf_dim*8,4,2,1)
   self.d2 = tcnn_block(gf_dim*8*2,gf_dim*8,4,2,1)
   self.d3 = tcnn_block(gf_dim*8*2,gf_dim*8,4,2,1)
   self.d4 = tcnn_block(gf_dim*8*2,gf_dim*8,4,2,1)
   self.d5 = tcnn_block(gf_dim*8*2,gf_dim*4,4,2,1)
   self.d6 = tcnn_block(gf_dim*4*2,gf_dim*2,4,2,1)
   self.d7 = tcnn_block(gf_dim*2*2,gf_dim*1,4,2,1)
   self.d8 = tcnn_block(gf_dim*1*2,c_dim,4,2,1, first_layer = True)#256x256
   self.tanh = nn.Tanh()

 def forward(self,x):
   e1 = self.e1(x)
   e2 = self.e2(F.leaky_relu(e1,0.2))
   e3 = self.e3(F.leaky_relu(e2,0.2))
   e4 = self.e4(F.leaky_relu(e3,0.2))
   e5 = self.e5(F.leaky_relu(e4,0.2))
   e6 = self.e6(F.leaky_relu(e5,0.2))
   e7 = self.e7(F.leaky_relu(e6,0.2))
   e8 = self.e8(F.leaky_relu(e7,0.2))
   d1 = torch.cat([F.dropout(self.d1(F.relu(e8)),0.5,training=True),e7],1)
   d2 = torch.cat([F.dropout(self.d2(F.relu(d1)),0.5,training=True),e6],1)
   d3 = torch.cat([F.dropout(self.d3(F.relu(d2)),0.5,training=True),e5],1)
   d4 = torch.cat([self.d4(F.relu(d3)),e4],1)
   d5 = torch.cat([self.d5(F.relu(d4)),e3],1)
   d6 = torch.cat([self.d6(F.relu(d5)),e2],1)
   d7 = torch.cat([self.d7(F.relu(d6)),e1],1)
   d8 = self.d8(F.relu(d7))

   return self.tanh(d8)

Discriminator

Let’s define the discriminator using the downsampling function. 

class Discriminator(nn.Module):
 def __init__(self,instance_norm=False):#input : 256x256
   super(Discriminator,self).__init__()
   self.conv1 = cnn_block(c_dim*2,df_dim,4,2,1, first_layer=True) # 128x128
   self.conv2 = cnn_block(df_dim,df_dim*2,4,2,1)# 64x64
   self.conv3 = cnn_block(df_dim*2,df_dim*4,4,2,1)# 32 x 32
   self.conv4 = cnn_block(df_dim*4,df_dim*8,4,1,1)# 31 x 31
   self.conv5 = cnn_block(df_dim*8,1,4,1,1, first_layer=True)# 30 x 30

   self.sigmoid = nn.Sigmoid()
 def forward(self, x, y):
   O = torch.cat([x,y],dim=1)
   O = F.leaky_relu(self.conv1(O),0.2)
   O = F.leaky_relu(self.conv2(O),0.2)
   O = F.leaky_relu(self.conv3(O),0.2)
   O = F.leaky_relu(self.conv4(O),0.2)
   O = self.conv5(O)

   return self.sigmoid(O)

Initializing the models

Let’s initialize both models and enable CUDA for faster training. 

G = Generator().to(device)
D = Discriminator().to(device)

We will also define the optimizers and the loss function. 

G_optimizer = optim.Adam(G.parameters(), lr=2e-4,betas=(0.5,0.999))
D_optimizer = optim.Adam(D.parameters(), lr=2e-4,betas=(0.5,0.999))

bce_criterion = nn.BCELoss()
L1_criterion = nn.L1Loss()

Training and monitoring our model

Training the model is not the last step. You need to monitor the training and track it to analyze the performance and implement changes if necessary. Given how taxing it can get to monitor the performance of a GAN with too many losses, plots, and metrics to deal with, we will use neptune.ai at this step.

Neptune allows the user to:

1. Monitor the live performance of the model
2. Monitor the performance of the hardware
3. Store and compare different metadata for different runs (like metrics, parameters, performance, data, etc.)
4. Share the work with others

Disclaimer

Please note that this article references a deprecated version of Neptune.

For information on the latest version with improved features and functionality, please visit our website.

To get started, just follow these steps:

1. Install the Python neptune library on your local system:

!pip install neptune

2. Sign up at neptune.ai.

3. Create a project for storing your metadata.

4. Save your credentials as environment variables.

For this project, we will log our parameters into the Neptune dashboard. For logging the parameters or any information into the dashboard, you can use a run object:

import neptune
import os

run = neptune.init_run(
   project=os.getenv(“NEPTUNE_PROJECT_NAME”),
   api_token=os.getenv("NEPTUNE_API_TOKEN")
)

A run object establishes a connection between your environment and the project’s dashboard you’ve created for this tutorial. To log metadata, like the dictionary below, you can use the following syntax:

# Logging parameter in Neptune
PARAMS = {'Epoch': epochs,
         'Batch Size': batch_size,
         'Input Channels': c_dim,

         'Workers': workers,
         'Optimizer': 'Adam',
         'Learning Rate': 2e-4,
         'Metrics': ['Binary Cross Entropy', 'L1 Loss'],
         'Activation': ['Leaky Relu', 'Tanh', 'Sigmoid' ],
         'Device': device}

run['parameters'] = PARAMS

To log the loss, generated images, and the model’s weights, we will use the run object again but with different methods like append or upload. Here is our training loop putting together everything we have along with Neptune logging:

# Define missing variables
epochs = 30  # Adjust as needed
L1_lambda = 100  # Adjust as needed
G_losses = []
D_losses = []
G_GAN_losses = []
G_L1_losses = []
img_list = []

# Assuming fixed_x and fixed_y are not defined, let's create them
fixed_x, fixed_y = next(iter(train_loader))
fixed_x = fixed_x.to(device)
fixed_y = fixed_y.to(device)

for ep in range(30):
    for i, data in enumerate(train_loader):

        y, x = data
        x = x.to(device)
        y = y.to(device)

        b_size = x.shape[0]

        real_class = torch.ones(b_size, 1, 30, 30).to(device)
        fake_class = torch.zeros(b_size, 1, 30, 30).to(device)

        # Train D
        D.zero_grad()

        real_patch = D(y, x)
        real_gan_loss = bce_criterion(real_patch, real_class)

        fake = G(x)

        fake_patch = D(fake.detach(), x)
        fake_gan_loss = bce_criterion(fake_patch, fake_class)

        D_loss = real_gan_loss + fake_gan_loss
        D_loss.backward()
        D_optimizer.step()

        # Train G
        G.zero_grad()
        fake_patch = D(fake, x)
        fake_gan_loss = bce_criterion(fake_patch, real_class)

        L1_loss = L1_criterion(fake, y)
        G_loss = fake_gan_loss + L1_lambda * L1_loss
        G_loss.backward()

        G_optimizer.step()

        # Neptune logging
        run["Gen Loss"].append(G_loss.item())
        run["Dis Loss"].append(D_loss.item())
        run["L1 Loss"].append(L1_loss.item())
        run["Gen GAN Loss"].append(fake_gan_loss.item())

        if (i + 1) % 5 == 0:
            print(
                "Epoch [{}/{}], Step [{}/{}], d_loss: {:.4f}, g_loss: {:.4f},D(real): {:.2f}, D(fake):{:.2f},g_loss_gan:{:.4f},g_loss_L1:{:.4f}".format(
                    ep,
                    epochs,
                    i + 1,
                    len(train_loader),
                    D_loss.item(),
                    G_loss.item(),
                    real_patch.mean(),
                    fake_patch.mean(),
                    fake_gan_loss.item(),
                    L1_loss.item(),
                )
            )
            G_losses.append(G_loss.item())
            D_losses.append(D_loss.item())
            G_GAN_losses.append(fake_gan_loss.item())
            G_L1_losses.append(L1_loss.item())

            with torch.no_grad():
                G.eval()
                fake = G(fixed_x).detach().cpu()
                G.train()
            figs = plt.figure(figsize=(10, 10))
            plt.subplot(1, 3, 1)
            plt.axis("off")
            plt.title("conditional image (x)")
            plt.imshow(
                np.transpose(
                    torchvision.utils.make_grid(fixed_x.cpu(), nrow=1, padding=5, normalize=True),
                    (1, 2, 0),
                )
            )

            plt.subplot(1, 3, 2)
            plt.axis("off")
            plt.title("fake image")
            plt.imshow(
                np.transpose(
                    torchvision.utils.make_grid(fake, nrow=1, padding=5, normalize=True),
                    (1, 2, 0),
                )
            )

            plt.subplot(1, 3, 3)
            plt.axis("off")
            plt.title("ground truth (y)")
            plt.imshow(
                np.transpose(
                    torchvision.utils.make_grid(fixed_y.cpu(), nrow=1, padding=5, normalize=True),
                    (1, 2, 0),
                )
            )

            run["epoch_results"].upload(figs)
            plt.close()
            img_list.append(figs)

run.stop()

Recommended

Explore the example project in neptune.ai

See more

Once the training is initialized, all the logged information will automatically log into the dashboard. Neptune fetches live information from the training which allows live monitoring of the entire process. 

Below are the screenshots of the monitoring process. 

See in the app Full screen preview

You can also access all metadata and see generated samples.

See in the app Full screen preview

Switching to the images panel, it will show you the generated samples:

See in the app Full screen preview

Key takeaways

• Pix2Pix is a conditional GAN that uses images and labels to generate images. 
• It uses two architectures:
  • U-Net for generator
  • PatchGAN for discriminator
• PatchGAN uses smaller patches of  NxN size in the generated image to discriminate it from real or fake instead of discriminating the entire image at once. 
• Pix2Pix has an additional loss specifically for the generator so that it can generate images closer to the ground truth. 
• Pix2Pix is a pairwise image translation algorithm. 

Other GANs that you can explore are:

1. CycleGAN: It is similar to Pix2Pix since most of the approach is the same except the data part. Instead of pair-image-translation, it is unpaired-image-translation. Learning and exploring CycleGAN will be much easier since it was developed by the same authors.
2. If you are interested in text-to-image translation then you should explore:
   • DeepDaze: Uses a generative model to create images from text prompts. Great for generating abstract or artistic images based on text descriptions.
   • BigSleep: Great if you want to discover unusual visualizations from prompts.
   • DALL:E: Developed by OpenAI, this model generates creative compositions with high level of detail directly from text descriptions.
3. Other interesting GAN projects you may want to try out:
   • StyleGAN: Generates realistic faces; ideal for style manipulation and creative blending.
   • AnimeGAN: Converts real photos into anime-style images.
   • BigGAN: Produces images with realistic textures.
   • Age-cGAN: Alters age in facial images.
   • StarGAN: Handles multiple transformations in faces, like hair color and expression changes.

In this article you will learn:

1. What is dimensionality reduction?
2. What is the curse of dimensionality?
3. Tools and libraries used for dimensionality reduction
4. Algorithms used for dimensionality reduction
5. Applications
6. Advantages and disadvantages

What is dimensionality reduction?

Dimensionality reduction is the task of reducing the number of features in a dataset. In machine learning tasks like regression or classification, there are often too many variables to work with. These variables are also called features. The higher the number of features, the more difficult it is to model them, this is known as the curse of dimensionality. This will be discussed in detail in the next section.

Additionally, some of these features can be quite redundant, adding noise to the dataset and it makes no sense to have them in the training data. This is where feature space needs to be reduced. 

The process of dimensionality reduction essentially transforms data from high-dimensional feature space to a low-dimensional feature space. Simultaneously, it is also important that meaningful properties present in the data are not lost during the transformation.

Dimensionality reduction is commonly used in data visualization to understand and interpret the data, and in machine learning or deep learning techniques to simplify the task at hand. 

Curse of dimensionality

It is well known that ML/DL algorithms need a large amount of data to learn invariance, patterns, and representations. If this data comprises a large number of features, this can lead to the curse of dimensionality. The curse of dimensionality, first introduced by Bellman, describes that in order to estimate an arbitrary function with a certain accuracy the number of features or dimensionality required for estimation grows exponentially. This is especially true with big data which yields more sparsity. 

Sparsity in data is usually referred to as the features having a value of zero; this doesn’t mean that the value is missing. If the data has a lot of sparse features then the space and computational complexity increase. Oliver Kuss [2002] shows that the model trained on sparse data performed poorly in the test dataset. In other words, the model during the training learns noise and they are not able to generalize well. Hence they overfit.  

When the data is sparse, observations or samples in the training dataset are difficult to cluster as high-dimensional data causes every observation in the dataset to appear equidistant from each other. If data is meaningful and non-redundant, then there will be regions where similar data points come together and cluster, furthermore they must be statistically significant. 

Issues that arise with high dimensional data are:

1. Running a risk of overfitting the machine learning model. 
2. Difficulty in clustering similar features.
3. Increased space and computational time complexity. 

Non-sparse data or dense data on the other hand is data that has non-zero features. Apart from containing non-zero features they also contain information that is both meaningful and non-redundant. 

To tackle the curse of dimensionality, methods like dimensionality reduction are used. Dimensional reduction techniques are very useful to transform sparse features to dense features. Furthermore, dimensionality reduction is also used to clean the data and feature extraction.

Tools and library

The most popular library for dimensionality reduction is scikit-learn (sklearn). The library consists of three main modules for dimensionality reduction algorithms:

1. Decomposition algorithms
   • Principal Component Analysis
   • Kernel Principal Component Analysis
   • Non-Negative Matrix Factorization 
   • Singular Value Decomposition 
2. Manifold learning algorithms
   • t-Distributed Stochastic Neighbor Embedding
   • Spectral Embedding
   • Locally Linear Embedding
3. Discriminant Analysis
   • Linear Discriminant Analysis

When it comes to deep learning, algorithms like autoencoders can be constructed to reduce dimensions and learn features and representations. Frameworks like Pytorch, Pytorch Lightning, Keras, and TensorFlow are used to create autoencoders.  

Algorithms for dimensionality reduction

Let’s start with the first class of algorithms.

Decomposition algorithms 

Decomposition algorithm in scikit-learn involves dimensionality reduction algorithms. We can call various techniques using the following command:

from sklearn.decomposition import PCA, KernelPCA, NMF

Principal Component Analysis (PCA)

Principal Component Analysis, or PCA, is a dimensionality-reduction method to find lower-dimensional space by preserving the variance as measured in the high dimensional input space. It is an unsupervised method for dimensionality reduction. 

PCA transformations are linear transformations. It involves the process of finding the principal components, which is the decomposition of the feature matrix into eigenvectors. This means that PCA will not be effective when the distribution of the dataset is non-linear. 

Let’s understand PCA with python code. 

def pca(X=np.array([]), no_dims=50):

    print("Preprocessing the data using PCA...")
    (n, d) = X.shape
    Mean = np.tile(np.mean(X, 0), (n, 1))
    X = X - Mean
    (l, M) = np.linalg.eig(np.dot(X.T, X))
    Y = np.dot(X, M[:, 0:no_dims])
    return Y

PCA implementation is quite straightforward. We can define the whole process into just four steps:

1. Standardization: The data has to be transformed to a common scale by taking the difference between the original dataset with the mean of the whole dataset. This will make the distribution 0 centered. 
2. Finding covariance: Covariance will help us to understand the relationship between the mean and original data. 
3. Determining the principal components: Principal components can be determined by calculating the eigenvectors and eigenvalues. Eigenvectors are a special set of vectors that help us to understand the structure and the property of the data that would be principal components. The eigenvalues on the other hand help us to determine the principal components. The highest eigenvalues and their corresponding eigenvectors make the most important principal components.
4. Final output: It is the dot product of the standardized matrix and the eigenvector. Note that the number of columns or features will be changed. 

Reducing the number of variables of data not only reduces complexity but also decreases the accuracy of the machine learning model. However, with a smaller number of features it is easy to explore, visualize and analyze, it also makes machine learning algorithms computationally less expensive. In simple words, the idea of PCA is to reduce the number of variables of a data set, while preserving as much information as possible.

Let’s also take a look at the modules and functions sklearn provides for PCA.

We can start by loading the most dataset:

from sklearn.datasets import load_digits
digits = load_digits()
digits.data.shape

(1797, 64)

The data consists of 8×8 pixel images, which means that they are 64-dimensional. To gain some understanding of the relationships between these points, we can use PCA to project them to lower dimensions, like 2-D:

from sklearn.decomposition import PCA

pca = PCA(2)  # project from 64 to 2 dimensions
projected = pca.fit_transform(digits.data)
print(digits.data.shape)
print(projected.shape)

(1797, 64)

(1797, 2)

Now, let’s plot the first two principal components.

plt.scatter(projected[:, 0], projected[:, 1],
            c=digits.target, edgecolor='none', alpha=0.5,
            cmap=plt.cm.get_cmap('spectral', 10))
plt.xlabel('component 1')
plt.ylabel('component 2')
plt.colorbar();

We can see that PCA optimally found the principal components that can quite effectively cluster similar distributions, for the most part. 

Kernel PCA (KPCA)

The PCA transformations we described previously are linear transformations that are ineffective with the non-linear distribution. To deal with non-linear distribution, the basic idea is to use the kernel trick. 

A kernel trick is simply a method to project non-linear data onto a higher dimensional space and separate different distributions of data. Once the distributions are separated we can use PCA to separate them linearly.

Kernel PCA uses a kernel function ϕ that calculates the dot product of the data for non-linear mapping. In other words, the function ϕ maps the original d-dimensional features into a larger, k-dimensional feature space by creating non-linear combinations of the original features.

Let assume a dataset x that contains two features x1 and x2:

After applying the kernel trick we get:

To get a more intuitive understanding of Kernel PCA let’s define a feature space that cannot be linearly separated. 

​​from sklearn.datasets import make_circles
from sklearn.decomposition import KernelPCA
np.random.seed(0)
X, y = make_circles(n_samples=400, factor=.3, noise=.05)

Now, let’s plot and see our dataset.

plt.figure(figsize=(15,10))
plt.subplot(1, 2, 1, aspect='equal')
plt.title("Original space")
reds = y == 0
blues = y == 1

plt.scatter(X[reds, 0], X[reds, 1], c="red",
           s=20, edgecolor='k')
plt.scatter(X[blues, 0], X[blues, 1], c="blue",
           s=20, edgecolor='k')
plt.xlabel("$x_1$")
plt.ylabel("$x_2$")

As you can see in this dataset the two classes cannot be separated linearly. Now, let’s define kernel PCA and see how it separates this feature space. 

kpca = KernelPCA(kernel="rbf", fit_inverse_transform=True, gamma=10, )
X_kpca = kpca.fit_transform(X)
plt.subplot(1, 2, 2, aspect='equal')
plt.scatter(X_kpca[reds, 0], X_kpca[reds, 1], c="red",
           s=20, edgecolor='k')
plt.scatter(X_kpca[blues, 0], X_kpca[blues, 1], c="blue",
           s=20, edgecolor='k')
plt.title("Projection by KPCA")
plt.xlabel(r"1st principal component in space induced by $phi$")
plt.ylabel("2nd component")

After applying KPCA, it is able to linearly separate the two classes in the dataset.  

Singular Value Decomposition (SVD)

The singular value decomposition or SVD is a factorization method of a real or complex matrix. It is efficient when working with a sparse dataset; a dataset having a lot of zero entries. This type of dataset is usually found in the Recommender Systems, rating, and reviews dataset, et cetera.   

The idea of SVD is that every matrix of shape nXp factorizes into A = USV^T, where U is the orthogonal matrix, S is a diagonal matrix and  V^T is also an orthogonal matrix. 

The advantage of SVD is that the orthogonal matrices capture the structure of the original matrix A which means that their properties do not change when multiplied by other numbers. This can help us approximate A. 

Now let’s understand SVD using code. To get a better understanding of the algorithm we will use a face dataset that scikit-learn provides.

from sklearn.datasets import fetch_lfw_people
lfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4)

Plot the images to get an idea of what we are working with. 

X = lfw_people.images.reshape(img_count, img_width * img_height)
X0_img = X[0].reshape(img_height, img_width)

plt.imshow(X0_img, cmap=plt.cm.gray)

Create a function for easy visualization of images. 

def draw_img(img_vector, h=img_height, w=img_width):
   plt.imshow( img_vector.reshape((h,w)), cmap=plt.cm.gray)
   plt.xticks(())
   plt.yticks(())
draw_img(X[49])

Before applying SVD it is better to standardize the data. 

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler(with_std=False)
Xstd = scaler.fit_transform(X)

After standardizing this is how the image looks. 

It is worth noting that we can always retrieve the original image by performing the inverse transformation. 

Xorig = scaler.inverse_transform(Xstd)
draw_img(Xorig[49])

Now, we can apply the SVD function from NumPy and decompose the matrix into three matrices. 

from numpy.linalg import svd

U, S, VT = svd(Xstd)

To check that the function works we can always perform a matrix multiplication of the three matrices. 

US = U*S
Xhat = US @ VT[0:1288,:]

# inverse transform Xhat to reverse standardization
Xhat_orig = scaler.inverse_transform(Xhat)
draw_img(Xhat_orig[49])

Now, let’s perform dimensionality reduction. To do that we just need to reduce the number of features from the orthogonal matrices. 

Xhat_500 = US[:, 0:500] @ VT[0:500, :]
# inverse transform Xhat to reverse standardization
Xhat_500_orig = scaler.inverse_transform(Xhat_500)
# draw recovered image
draw_img(Xhat_500_orig[49])

We can further reduce more features and see the results. 

Xhat_100 = US[:, 0:100] @ VT[0:100, :]
# inverse transform Xhat to reverse standardization
Xhat_100_orig = scaler.inverse_transform(Xhat_100)
# draw recovered image
draw_img(Xhat_100_orig[49])

Now let’s create a function that would allow us to reduce the dimensions of the image. 

def dim_reduce(US_, VT_, dim=100):

   Xhat_ = US_[:, 0:dim] @ VT_[0:dim, :]

   return scaler.inverse_transform(Xhat_)

Plotting images with a different number of features. 

dim_vec = [50, 100, 200, 400, 800]

plt.figure(figsize=(1.8 * len(dim_vec), 2.4))

for i, d in enumerate(dim_vec):
   plt.subplot(1, len(dim_vec), i + 1)
   draw_img(dim_reduce(US, VT, d)[49])

As you can see the first image contains the least number of features yet it can still construct the abstract version of the image and as we increase the features, we eventually obtain the original image. This proves that SVD can retain the basic structure of the data. 

Non-negative Matrix Factorization (NMF)

NMF is an unsupervised machine learning algorithm. When a non-negative input matrix X of dimension mXn is given to the algorithm, it is decomposed into the product of two non-negative matrices W and H. W is of the dimension mXp while H is of the dimension pXn.

Where Y = W.H

From the equation above you can see that to factorize the matrix, we need to minimize the distance. The most widely used distance function is the squared Frobenius norm; this is an extension of the Euclidean norm to matrices. 

It is also worth noting that this problem is not solvable in general which is why it is approximated. As it turns out, NMF is good for parts-based representation of the dataset i.e. NMF provides an efficient, distributed representation, and can aid in the discovery of the structure of interest within the data. 

Let’s understand NMF with code. We will use the same data that we used in SVD.

First, we will fit the model to the data. 

from sklearn.decomposition import NMF
model = NMF(n_components=200, init='nndsvd', random_state=0)
W = model.fit_transform(X)
V = model.components_

NMF takes a bit of time to decompose the data. Once the data is decomposed we can then visualize the factorized components. 

num_faces = 20
plt.figure(figsize=(1.8 * 5, 2.4 * 4))

for i in range(0, num_faces):
   plt.subplot(4, 5, i + 1)
   draw_img(V[i])

From the image above we can see that NMF is very efficient to capture the underlying structure of the data. It is also worth mentioning that NMF captures only the linear attributes. 

Advantages of NMF:

1. Data compression and visualization
2. Robustness to noise 
3. Easier to interpret

Manifold learning

So far we have seen approaches that only involved linear transformation. But what do we do when we have a non-linear dataset?

Manifold learning is a type of unsupervised learning that seeks to perform dimensionality reduction of a non-linear dataset. Again, scikit-learn offers a module that consists of various nonlinear dimensionality reduction techniques. We can call those classes or techniques through this command:

from sklearn.manifold import TSNE, LocallyLinearEmbedding, SpectralEmbedding

t-Distributed Stochastic Neighbor Embedding (t-SNE)

t-Distributed Stochastic Neighbor Embedding or t-SNE is a dimensionality reduction technique well suited for data visualization. Unlike PCA which simply maximizes the variance, t-SNE minimizes the divergence between two distributions. Essentially, it recreates the distribution of a high-dimensional space in a low-dimensional space rather than maximizing variance or even using a kernel trick. 

We can get a high-level understanding of t-SNE in three simple steps:

1. It first creates a probability distribution for the high-dimensional samples.  
2. Then, it defines a similar distribution for the points in the low-dimensional embedding.
3. Finally, it tries to minimize the KL-divergence between the two distributions. 

Now let’s understand it with code. For t-SNE, we will use the MNIST dataset again. Firstly, we import TSNE and then the data as well. 

from sklearn.manifold import TSNE
from sklearn.datasets import load_digits

digits = load_digits()
print(digits.data.shape)

# There are 10 classes (0 to 9) with almost 180 images in each class
# The images are 8x8 and hence 64 pixels(dimensions)

#Displaying what the standard images look like
for i in range(0,5):
   plt.figure(figsize=(5,5))
   plt.imshow(digits.images[i])
   plt.show()

We will then store the digits in order using np.vstack. 

X = np.vstack([digits.data[digits.target==i] for i in range(10)])
Y = np.hstack([digits.target[digits.target==i] for i in range(10)])

We will apply t-SNE to the dataset. 

digits_final = TSNE(perplexity=30).fit_transform(X)

We will now create a function to visualize the data. 

def plot(x, colors):
    palette = np.array(sb.color_palette("hls", 10))  #Choosing color palette

   # Create a scatter plot.
   f = plt.figure(figsize=(8, 8))
   ax = plt.subplot(aspect='equal')
   sc = ax.scatter(x[:,0], x[:,1], lw=0, s=40,c=palette[colors.astype(np.int)])
   # Add the labels for each digit.
   txts = []
   for i in range(10):
       # Position of each label.
       xtext, ytext = np.median(x[colors == i, :], axis=0)
       txt = ax.text(xtext, ytext, str(i), fontsize=24)
       txt.set_path_effects([pe.Stroke(linewidth=5, foreground="w"),
                             pe.Normal()])
       txts.append(txt)
   return f, ax, txts

Now we perform data visualization on the transformed dataset.

plot(digits_final,Y)

As it can be seen, t-SNE clusters the data beautifully. Compared to PCA, t-SNE performs well on nonlinear data. The drawback with t-SNE is that when the data is big it consumes a lot of time. So it is better to perform PCA followed by t-SNE. 

Locally Linear Embedding (LLE)

Locally Linear Embedding or LLE is a non-linear and unsupervised machine learning method for dimensionality reduction. LLE takes advantage of the local structure or topology of the data and preserves it on a lower-dimensional feature space. 

LLE optimizes faster but fails on noisy data. 

Let’s break the whole process into three simple steps:

1. Find the nearest neighbors of the data points. 
2. Construction of a weight matrix, by approximating each data point as a weighted linear combination of its k-nearest neighbors and minimizing the squared distance between them and their linear representation.
3. Map the weights into a lower-dimensional space by using the eigenvector-based optimization technique.

Spectral embedding

Spectral Embedding is another non-linear dimensionality reduction technique that also happens to be an unsupervised machine learning algorithm. Spectral embedding aims to find clusters of different classes based on the low-dimensional representations. 

We can again break the whole process into three simple steps:

1. Preprocessing: Construct a Laplacian matrix representation of the data or graph. 
2. Decomposition: Compute eigenvalues and eigenvectors of the constructed matrix and then map each point to a lower-dimensional representation. Spectral embedding makes use of the second smallest eigenvalue and its corresponding eigenvector.
3. Clustering: Assign points to two or more clusters, based on the representation. Clustering is usually done using k-means clustering. 

Applications: Spectral Embedding finds its application in image segmentation. 

Discriminant Analysis

Discriminant Analysis is another module that scikit-learn provides. It can be called using the following command:

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

Linear Discriminant Analysis (LDA)

LDA is an algorithm that is used to find a linear combination of features in a dataset. Like PCA, LDA is also a linear transformation-based technique. But unlike PCA it is a supervised learning algorithm. 

LDA computes the directions, i.e. linear discriminants that can create decision boundaries and maximize the separation between multiple classes. It is also very effective for multi-class classification tasks.

To have a more intuitive understanding of LDA, consider plotting a relationship of two classes as shown in the image below. 

One way to solve this problem is to project all the data points in the x-axis. 

This approach will lead to information loss and would be redundant. 

A better approach will be to compute the distance between all the points in the data and fit a new linear line that passes through them. This new line can now be used to project all the points. 

This new line minimizes the variance and classifies the two classes efficiently by maximizing the distance between them. 

LDA can be used for multivariate data as well. It makes data inference quite simple. LDA can be computed using the following 5 steps:

1. Compute the d-dimensional mean vectors for the different classes from the dataset.
2. Compute the scatter matrices (in-between-class and within-class scatter matrices). the Scatter matrix is used to make estimates of the covariance matrix. This is done when the covariance matrix is difficult to calculate or joint variability of two random variables is difficult to calculate. 
3. Compute the eigenvectors (e1, e2, e3…ed) and corresponding eigenvalues (λ1,λ2,…,λd) for the scatter matrices.
4. Sort the eigenvectors by decreasing eigenvalues and choose k eigenvectors with the largest eigenvalues to form a d×k dimensional matrix W (where every column represents an eigenvector).
5. Use this d×k eigenvector matrix to transform the samples onto the new subspace. This can be summarized by the matrix multiplication: Y=X×W (where X is an n×d dimensional matrix representing the n samples, and y are the transformed n×k-dimensional samples in the new subspace).

To know about LDA you can check out this article. 

Applications of dimentionality reduction

Dimensionality reduction finds its way in many real-life applications some of which are:

• Customer relationship management
• Text categorization
• Image retrieval
• Intrusion detection
• Medical image segmentation 

Advantages and disadvantages of dimentionality reduction

Advantages of dimensionality reduction:

• It helps in data compression by reducing features.
• It reduces storage.
• It makes machine learning algorithms computationally efficient.
• It also helps remove redundant features and noise.
• It tackles the curse of dimensionality

Disadvantages of dimensionality reduction:

• It may lead to some amount of data loss.
• Accuracy is compromised. 

Final thoughts

In this article, we learned about dimensionality reduction and also about the curse of dimensionality. We touched on the different algorithms that are used in dimensionality reduction with mathematical details and through code as well. 

It is worth mentioning these algorithms are supposed to be used based on the task at hand. For instance, if the nature of your data is linear then use decomposition methods otherwise use manifold learning techniques. 

It is considered to be a good practice to first visualize the data and then decide which method to use. Also, do not restrict yourself to one method but explore differently and see which one is the most suitable.

I hope you have learned something from this article. Happy learning.

 References

1. Introduction to Dimensionality Reduction – GeeksforGeeks
2. Introduction to Dimensionality Reduction for Machine Learning
3. Principal component analysis: a review and recent developments
4. Linear Discriminant Analysis In Python | by Cory Maklin
5. Working With Sparse Features In Machine Learning Models
6. Must-Know: What is the curse of dimensionality?
7. Curse Of Dimensionality And What Beginners Should Do To Overcome It
8. 6 Dimensionality Reduction Algorithms With Python
9. Sklearn API References
10. t-distributed stochastic neighbor embedding
11. Feature Selection and Extraction

• First, be careful—machine learning open source tools aren’t always 100% free all of the time. For example, Kuberflow has client and server components, and both are open. However, some tools might open-source only one of these components. The client is open, but the vendor controls everything server-side.
• Free open-source tools can cost you in other ways too. If you consider that you have to host and maintain the tool long-term, you’ll find that open-source can be quite costly after all. 
• Finally, if something goes awry, you probably won’t have 24/7/365 vendor support to rely on. Community can help you but, obviously, they don’t bear any responsibility for the result you’re left with.

Ultimately, open-source tools can be tricky. Before you choose the tool for your project, you need to carefully study its pros and cons. Moreover, you need to make sure that the tools work well with the rest of your stack. This is why I prepared a list of popular and community-approved MLOps platforms, tools, and frameworks for different stages of the model development process. 

If you’re exploring the possibility of integrating open source machine learning platforms into your workflow to simplify model development and model deployment, this article is tailored for you. Within these insights, you’ll discover a compilation of machine learning platforms, frameworks, and specialized tools designed to assist you in data exploration, deployment strategies, and testing procedures.

Furthermore, we’ve included a FAQ section towards the conclusion, offering comprehensive responses to the most commonly posed questions.

Interested in other MLOps tools?

When building their ML pipelines, teams usually look into a few other components of the MLOps stack.

If that’s the case for you, here are a few article you should check:

• 

  MLOps Landscape in 2023 [Tools and Platforms]

• 

  Building an ML Platform [Guide]

MLOps open source platforms

Let us start by exploring the open-source platforms first followed by frameworks and tools.

Full-fledged MLOps open source platforms

Full-fledged platforms contain tools for all stages of the machine-learning workflow. Ideally, once you get a full-fledged tool, you won’t have to set up any other tools. In practice, it depends on the needs of your project and personal preferences. 

Kubeflow

Almost immediately after Kubernetes established itself as the standard for working with a cluster of containers, Google created Kubeflow—an open-source project that simplifies working with ML in Kubernetes. It has all the advantages of this orchestration tool, from the ability to deploy on any infrastructure to managing loosely-coupled microservices, and on-demand scaling.

This project is for developers who want to deploy portable and scalable machine learning projects. Google didn’t want to recreate other services. They wanted to create a state-of-the-art open-source system that can be applied alongside various infrastructures—from supercomputers to laptops. 

With Kuberflow, you can benefit from the following features: 

• Jupyter notebooks

Create and customize Jupyter notebooks, immediately see the results of running your code, and create interactive analytics reports.

• Custom TensorFlow job operator

This functionality helps train your model and apply a TensorFlow or Seldon Core serving container to export the model to Kubernetes. 

• Simplified containerization

Kuberflow eliminates the complexity involved in containerizing the code. Data scientists can perform data preparation, model training, and deployment in less time.

All in all, Kuberflow is a full-fledged solution for the development and deployment of end-to-end machine learning workflows. 

Related post

The Best Kubeflow Alternatives

Read more

MLflow

MLflow is an open-source platform for machine learning engineers to manage the machine learning lifecycle through experimentation, deployment, and testing. MLflow comes in handy when you want to track the performance of your machine learning models. It’s like a dashboard, one place where you can: 

• monitor machine learning pipelines, 
• store model metadata, and 
• pick the best-performing model.

Right now, there are four components provided by MLflow:

• Tracking 

The MLflow Tracking component is an API and UI for logging parameters, code versions, metrics, and output files for running the code and visualizing the results. You can do log and query experiments using Python, REST, R API, and Java APIs. You can also record the results. 

• Project 

MLflow Project is a tool for machine learning teams to package data science code in a reusable and reproducible way. It comes with an API and command-line tools to connect projects into workflows. It helps you run projects on any platform. 

• Model

MLflow Model makes it easy to package machine learning models to be used by various downstream tools, like Apache Spark. With this, deploying machine learning models in diverse serving environments is much more manageable. 

Overall, users love MLflow because it’s easy to use locally without a dedicated server and has a fantastic UI where you can explore your experiments. 

Might be useful

Unlike manual, homegrown, or open-source solutions, neptune.ai is a scalable full-fledged component with user access management, developer-friendly UX, and advanced collaboration features.

That’s especially valuable for ML/AI teams. Here’s an example of how Neptune helped Waabi optimize their experiment tracking workflow.

The product has been very helpful for our experimentation workflows. Almost all the projects in our company are now using Neptune for experiment tracking, and it seems to satisfy all our current needs. It’s also great that all these experiments are available to view for everyone in the organization, making it very easy to reference experimental runs and share results.

James Tu, Research Scientist at Waabi

See in app Full screen preview

• 

  Full case study with Waabi

• 

  Dive into documentation

• 

  Get in touch if you’d like to go through a custom demo with your team

Metaflow

Netflix created Metaflow as an open-source MLOps platform for building and managing large-scale, enterprise-level data science projects. Data scientists can use this platform for end-to-end development and deployment of their machine-learning models. 

• Great library support

Metaflow supports all popular data science tools, like TensorFlow and scikit-learn, so you can keep using your favorite tool. Metaflow supports Python and R, making it even more flexible in terms of library and package choice. 

• Powerful version control toolkit 

What is excellent about Metaflow is that it versions and keeps track of all your machine learning experiments automatically. You won’t lose anything important, and you can even inspect the results of all the experiments in notebooks. 

As it was mentioned above, Metaflow was specifically created for large-scale machine learning development. The AWS cloud powers the solution, so there are built-in integrations to storage, compute, and machine learning services from AWS if you need to scale. You don’t have to rewrite or change the code to use any of it. 

Flyte

If you’re looking for a platform that will take care of experiment tracking and maintenance for your machine learning project, have a look at Flyte. It is an open-source orchestrator designed to simplify the creation of robust data and machine learning pipelines for production. Its architecture prioritizes scalability and reproducibility, harnessing the power of Kubernetes as its foundational framework.

Flyte offers a ton of features and use cases from a simple machine learning project to complex LLMs projects. To give you an I have distilled a some features and listed them below, but you check out their website and documentation. 

• Large-scale project support

Flyte has helped them to execute large-scale computing that’s crucial to their business. It’s not a secret that scaling and monitoring all pipeline changes can be pretty challenging, especially if the workflows have complex data dependencies. Flyte successfully deals with tasks of higher complexity, so developers can focus on business logic rather than machines.

• Improved reproducibility

This tool can also help you be sure of the reproducibility of the machine learning models you build. Flyte tracks changes, does version control, and containerizes the model alongside its dependencies.

• Multi-language support

Flyte was created to support complex ML projects in Python, Java, or Scala.

Flyte has been tested out by Lyft internally before they released it to the public. It has a proven record of managing more than 7,000 unique workflows totaling 100,000 executions every month. 

MLReef

MLReef is an MLOps platform for teams to collaborate and share the results of their machine learning experiments. Projects are built on reusable machine learning modules realized either by you or by the community. This boosts the speed of development and makes the workflow more efficient by promoting concurrency. 

MLReef provides tools in four directions:

• Data management

You have a fully-versioned data hosting and processing infrastructure for setting up and managing your machine learning models. 

• Script repositories

Every developer has access to containerized and versioned script repositories that you can use in your machine learning pipelines.

• Experiment management 

You can use MLReef for experiment tracking across different iterations of your project. 

• MLOps

This solution helps you optimize pipeline management and orchestration, automating routine tasks.

Moreover, MLReef feels welcoming to projects of any size. Newcomers can use it for small-scale projects, experienced developers―for small, medium-sized, and enterprise projects. 

• Newcomer

If you don’t have much experience developing machine learning models, you’ll find a user-friendly interface and community support for whatever problem you may face. 

• Experienced

MLReef lets you build your project on Git while taking care of all the DevOps mess for you. You can easily monitor progress and outcomes in an automated environment. 

• Enterprise

MLReef for enterprise is easy to scale and control on the cloud or on-premises.

All in all, MLReef is a convenient framework for your machine learning project. With just a couple of easy setups, you’ll be able to develop, test, and optimize your machine learning solution brick-by-brick. 

Seldon Core

Seldon Core is one of the platform for machine learning model deployment on Kubernetes. This platform helps developers build models in a robust Kubernetes environment, with features like custom resource definitions to manage model graphs. You can also merge your continuous integration and deployment tools with platform.

• Build scalable models

Seldon core can convert your model built on TensorFlow, PyTorch, H2O, and other frameworks into a scalable microservice architecture based on REST/GRPC. 

• Monitor model performance

It will handle scaling for you, and give you advanced solutions for measuring model performance, detecting outliers, and conducting A/B testing out-of-the-box. 

• Robust and reliable

Seldon Core can boast the robustness and reliability of a system supported through continuous maintenance and security policy updates. 

Optimized servers provided by Seldon Core allow you to build large-scale deep-learning systems without having to containerize them or worry about their security.

Sematic

Sematic stands as an open-source machine learning development platform. It grants ML Engineers and Data Scientists the ability to craft intricate end-to-end machine learning pipelines using straightforward Python code, which can then be executed on diverse platforms: their local machine, a cloud VM, or a Kubernetes cluster, harnessing the potential of cloud-based resources.

This open source platform draws upon insights amassed from leading self-driving car enterprises. It facilitates the seamless linking of data processing tasks (such as those powered by Apache Spark) with model training endeavors (like PyTorch or TensorFlow), or even arbitrary Python-based business logic. This amalgamation results in the creation of type-safe, traceable, and reproducible end-to-end pipelines. These pipelines, complete with comprehensive monitoring and visualization, are effortlessly managed through a contemporary web dashboard.

Here are some of the features that Sematic offers:

• Smooth Onboarding 

Embarking on your journey with Sematic is a breeze – no initial deployment or infrastructure requirements. Simply install Sematic locally and plunge into exploration.

• Parity from Local to Cloud

The same code that runs on your personal laptop can be seamlessly executed on your Kubernetes cluster, ensuring consistent outcomes.

• End-to-End Transparency

Every artifact of your pipeline is meticulously stored, tracked, and presented within a web dashboard, enabling comprehensive oversight.

• Harnessing Diverse Computing Resources

Tailor the resources allocated to each step of your pipeline, optimizing performance and cloud footprint through a range of options including CPUs, memory, GPUs, and Spark clusters.

• Reproducibility at the core

Rerun your pipelines with confidence from the intuitive UI, securing the assurance of reproducible results each time.

Sematic introduces an exceptional level of clarity to your machine learning pipelines, affording you an encompassing view of crucial aspects such as artifacts, logs, errors, source control, and dependency graphs. This robust insight is seamlessly coupled with an SDK and GUI that remain both straightforward and instinctive.

Sematic strikes an adept balance by offering a precisely calibrated level of abstraction. This equilibrium empowers ML engineers to concentrate on refining their business logic, all the while harnessing the power of cloud resources – all without the necessity of wielding intricate infrastructure expertise.

Data-processing MLOps open source platform

Data-processing platforms are ideally used to prepare a robust pipeline for any given application. These platforms are capable of scaling, optimizing, batching, distributing data streams et cetera. 

Apache Airflow

Apache Airflow emerges as an open-source platform tailored for the development, scheduling, and vigilant monitoring of batch-centric workflows. Airflow’s expansive Python foundation empowers you to forge intricate workflows, seamlessly bridging connections with a diverse spectrum of technologies.

A user-friendly web interface takes charge of workflow management, meticulously overseeing their state. From deploying as a singular process on your personal laptop to configuring a distributed setup capable of supporting the most intricate workflows, Airflow accommodates a plethora of deployment options.A distinctive hallmark of Airflow workflows is their anchoring within Python code. This “workflows as code” paradigm serves a multifaceted role:

• Dynamic Prowess

Airflow pipelines are molded through Python code, instilling the capability for dynamic pipeline generation.

• Inherent Extensibility

The Airflow framework houses a range of operators that seamlessly interface with a multitude of technologies. Every component of Airflow retains an intrinsic extensibility, seamlessly adapting to your unique environment.

• Supreme Flexibility

The fabric of workflow parameterization is woven into the system, harnessing the prowess of the Jinja templating engine for streamlined customization.

Apache Airflow is a versatile addition to any machine learning stack, offering dynamic workflow orchestration that adapts to changing data and requirements. With its flexibility, extensive connectivity, and scalability, Airflow allows machine learning practitioners to build custom workflows as code while integrating various technologies. Its monitoring capabilities, community support, and compatibility with cloud resources enhance ML reproducibility, collaboration, and efficient resource utilization in machine learning operations.

Monitoring MLOps open source platform

EvidentlyAI

EvidentlyAI is an open-sourced observability platform that allows you to evaluate, test, and monitor machine learning models. The platform covers the phase from validation to production. It offers services for tabular data, embeddings, and text-based models and data. It has also extended its services to cater to the needs of large language models or LLMs. 

These are some of the products that EvidentlyAI offers:

With these products, EvidentlyAI offers the following features:

• Build reports

The plug-and-play capabilities allow users to easily build reports for dataset and model performance. These reports are easy to share and appealing to interact. 

• Test your pipelines

EvidentlyAI test suites allow you properly create test pipelines for your machine learning models and data to see if there is any drift detected. 

• Monitoring

With dashboard capabilities and a wide range of testing methods Evidently makes monitoring and debugging machine learning models simple and interactive. 

• Data quality

With EvidentlyAI you can run various exploratory analyses to ensure that the data is of high quality and integrity. It enables you to spot issues in your data with ease with a single line of code. 

EvidentlyAI is an easy-to-use platform that offers great features with good capabilities. This testing platform is one of the best out there, and it is keeping up with the trend as it offers services towards LLMs. 

MLOps open source frameworks

Now that open-source platforms are covered let us dive into the frameworks. 

Workflow open source MLOps frameworks

The workflow frameworks allows you to provide a structural approach to streamline the different phases of your MLOps applications. You must keep in mind that some frameworks covers two phases while some may cover multiple phases. 

Kedro 

Kedro is a Python framework for machine learning engineers and data scientists to create reproducible and maintainable code. 

This framework is your best friend if you want to organize your data pipeline and make machine learning project development much more efficient. You won’t have to waste time on code rewrites and will have more opportunities for focusing on robust pipelines. Moreover, Kedro helps teams establish collaboration standards to limit delays and build scalable, deployable projects.

Kedro has many good features: 

• Project templates

Usually, you have to spend a lot of time understanding how to set up your analytics project. Kedro provides a standard template that will save you time. 

• Data management

Kedro will help you load and store data to stop being alarmed about the reproducibility and scalability of your code. 

• Configuration management

This is a necessary tool when you’re working with complex software systems. If you don’t pay enough attention to configuration management, you might encounter serious reliability and scalability problems.  

Kedro promotes a data-driven approach to ML development and maintains industry-level standards while decreasing operational risks for businesses. 

Related post

Building and Managing Data Science Pipelines with Kedro

Read more

ZenML

ZenML is an MLOps framework for orchestrating your machine learning experiment pipeline. It provides you with tools to:

• Preprocess data

ZenML helps you convert raw data into analysis-ready data. 

• Train your models

Among other tools for convenient model training, the platform uses declarative pipeline configs, so you can switch between on-premise and cloud environments easily. 

• Conduct split testing 

ZenML creators claim that the platform’s key benefits are automated experiment tracking and guaranteed comparability between experiments. 

• Evaluate the results

XML focuses on making ML development reproducible and straightforward for both individual developers and large teams. 

This framework frees you from all the troubles of delivering machine learning models with traditional tools. If you struggle with providing enough experiment data that prove the reproducibility of results, want to reduce waste and make the reuse of code simpler, ZenML will help. 

Deployment and serving open source MLOps framework

BentoML

BentoML is a framework that allows you to build, deploy and scale any machine learning application. BentoML provides a way to bundle your trained models, along with any preprocessing, post-processing, and custom code, into a containerized format. 

Some of the key features of BentoML include: 

• Model Serving

BentoML allows you to easily serve your machine learning models with a REST API. It abstracts away the complexities of serving machine learning models and managing infrastructure. 

• Model Packaging

You can package your trained models, along with dependencies and custom code, into a single deployable artifact. This makes it simple to reproduce your model deployments.

• Multi-Framework Support 

BentoML supports a variety of machine learning frameworks, such as TensorFlow, PyTorch, Scikit-learn, XGBoost, and more.

• Deployment Flexibility

You can deploy BentoML models in various environments, including local servers, cloud platforms, and Kubernetes clusters.

• Scalability

BentoML supports high-throughput serving, making it suitable for machine learning applications that require efficient and scalable model deployments.

• Versioning

BentoML allows you to version your model artifacts and easily switch between different versions for serving.

• Monitoring and Logging

BentoML provides features for monitoring the health and performance of your deployed models, including logging and metrics.

• Customization

You can customize the deployment environment, preprocessing, post-processing, and other aspects of your deployed model.

BentoML can be an important resource in your ML arsenal as it can essentially offer so much more with ease and reliability. With BentoML you can deploy your machine learning models as REST APIs, Docker containers, or even serverless functions. 

Workflow orchestration open source MLOps framework

Argo Workflow

Argo Workflow is a Kubernetes based orchestration tool which is based on YAML. It is lightweight and easy to use tool. Because it is based on YAML it is implemented as Kubernetes CRD (Custom Resource Definition). It is open-sourced, and it is trusted by a large community. 

Argo Workflow provides support to a wide range of ecosytem some of which are:

Argo Workflow also supports Python-based environments. Although Argo offers a quite a number of features I have listed out a few that attracted me a lot:

• UI

The workflow provides a user interface that allows users to manage their workflow with ease. 

• Artifact Support

You can integrate platforms such as S3, Azure Blob Storage, et cetera to store your metadata. 

• Scheduling

You can schedule your whole ML workflow using cron. This allows you to schedule jobs and tasks to run automatically at specific times, on specific days, or at regular intervals. 

• Kubernetes 

If you are well established on working with Kubernetes cluster then Argo is the go-to choice. One key feature is that Argo defines each step as a container. 

• Efficiency

It easily computes intensive jobs for data processing and ML making it efficient and reliable. 

You can find the full documentation here. 

MLOps open source tools

Open-source tools and libraries addresses one specific aspect in your machine learning applications. You can pick any of these tools and use it for your own application and fit them into a desired framework. One key advantage is that these tools are compatible with most of the working environments and they are compatible as well.

In this list, we will cover some of the major areas in ML lifecycle where open-source tools will get the job done for you. 

Development and deployment open source ML tools

MLRun

MLRun is a tool for machine learning model development and deployment. If you’re looking for a tool that conveniently runs in a wide variety of environments and supports multiple technology stacks, it’s definitely worth a try. MLRun offers a comprehensive approach to managing data pipelines.

MLRun has a layered architecture that offers the following powerful functionality: 

• Feature and artifact store

This layer helps you to handle the preparation and processing of data and store it across different repositories. 

• Elastic serverless runtimes layer

Convert simple code into microservices that are easy to scale and maintain. It’s compatible with standard runtime engines like Kubernetes jobs, Dask, and Apache Spark.

• Automation layer

For you to concentrate on model training the model and fine-tuning the hyperparameters, the pipeline automation tool helps you with preparing data, testing, and real-time deployment. You’ll only need to provide your supervision to create a state-of-the-art ML solution. 

• Central management layer

Here, you get access to a unified dashboard to manage your whole workflow. MLRun has a convenient user interface, a CLI, and an SDK that you can access anywhere.

With MLRun, you can write code once and then use automated solutions to run it on different platforms. The tool manages the build process, execution, data movement, scaling, versioning, parameterization, output tracking, and more. 

CML (Continuous Machine Learning)

CML (Continuous Machine Learning) is a library for continuous integration and delivery (CI / CD) of machine learning projects. The library was developed by the creators of DVC, an open-source library for versioning machine learning models and machine learning experiments. Together with DVC, Tensorboard, and cloud services, CML should facilitate the process of developing and implementing machine learning models into products.

• Automate pipeline building

CML was designed to automate some of the work of machine learning engineers, including training experiments, model evaluation, datasets, and their additions. 

• Integrate APIs 

The tool is positioned as a library that supports GitFlow for data science projects, allows automatic generation of reports, and hides complex details of using external services. Examples of external services include cloud platforms: AWS, Azure, GCP, and others. For infrastructure tasks, DVC, docker, and Terraform are also used. Recently, there is an infrastructural aspect of machine learning projects attracting more attention. 

The library is flexible and provides a wide range of functionality; from sending reports and publishing data, to distributing cloud resources for a project.

AutoML open source tools

AutoKeras

AutoKeras is an open-source library for Automated Machine Learning (AutoML). With AutoML frameworks, you can automate the processing of raw data, choose a machine learning model, and optimize the hyperparameters of the learning algorithm.

• Streamline machine learning model development

AutoML reduces the biases and variances that happen when humans develop machine learning models, and streamlines the development of a machine learning model. 

• Enjoy automated hyperparameter tuning

AutoKeras is the tool that provides functionality to match the architecture and hyperparameters of deep learning models automatically. 

• Build flexible solutions

AutoKeras is most famous for its flexibility. In this case, the code you write will be executed regardless of the backend. It supports Theano, Tensorflow, and other frameworks.

AutoKeras has several training datasets inside. They’re already put in a form that’s convenient for work, but it doesn’t show you the full power of AutoKeras. In fact, it contains tools for suitable preprocessing of texts, pictures, and time series. In other words, the most common data types, which make the data preparation process much more manageable. The tool also has built-in visualization for models.

H2O AutoML

H2O.ai is a software platform that optimizes the machine learning process using AutoML. H2O claims the platform can train models faster than popular machine learning libraries such as scikit-learn. 

H2O is a machine learning, predictive data analytics platform for building machine learning models and generating production code for them in Java and Python, all at the click of a button. 

• Implement ML models out-of-the-box

It has implementations of supervised and unsupervised algorithms such as GLM and K-Means, and an easy-to-use web interface called Flow. 

• Tailor H2O to your needs 

The tool is helpful for both beginner and seasoned developers. It equips the coder with a simple wrapper function that manages modeling-related tasks in a few lines of code. Experienced machine learning engineers appreciate this function, since it allows them to focus on other, more thought-intensive processes of building models (like data exploration and feature engineering). 

Overall, H2O is a powerful tool for solving machine learning and data science problems. Even beginners can extract value from data and build robust models. H2O continues to grow and release new products while maintaining high quality across the board.

EvalML 

EvalML is a library that offers multiple functionalities such building, optimizing, and evaluating machine learning pipelines. EvalML offers end-to-end supervised machine learning solutions that leverage Featuretools and Compose. The former is a framework that is used to perform automated feature engineering in relational datasets, and the latter is used to automate prediction engineering. 

With these automated capabilities, EvalML offers four important functionalities:

• Automation

It takes away the manual work from the picture. You make machine learning models with ease. The automation feature includes data quality check, cross-validation, and many other features. 

• Data Checks

As the name suggest it inspect data integrity and brings into light the issues and problems like duplicates, imbalance distribution et cetera before you can use them to train the model. 

• End-to-end

Offers end-to-end functionality that includes data-preprocessing, feature-engineering, feature-selection, and various other machine learning modeling techniques. 

• Model Understanding

It helps you to understand and inspect your machine learning model.

To conclude EvalML is an amazing tool that essentially automates two major phases of the ML lifecycle, data-preprocessing, and ML modeling. EvalML is has an active list of contributors, and the library is updated in a day-to-day basis. You can leverage this light-weight library to your own application with ease as the documentation is pretty straightforward and easy to understand. 

Neural Network Intelligence (NNI)

NNI or Neural Network Intelligence is a lightweight tool created by Microsoft for automating neural network optimization. This open-source toolkit allows users to automate feature engineering, neural architecture search or NAS, model compression, and hyper-parameter tuning. 

NNI offers simple to use function calling in Python. Similar to other Python libraries and frameworks NNI can be leveraged in an existing pipeline. All you need to have is a PyTorch working environment, and you are ready to plug-and-play and automate your optimization technique with a single function calling. For instance, if you want to perform:

• Hyperparameter tuning then simply call nni.get_next_parameter()
• Model pruning then call one of the pruning methods such as L1NormPruner(model, config)
• Model quantization then call any quantization function such as QAT_Quantizer(model, config)
• Neural architecture search then you can call a strategy and an evaluator like RegularizedEvolution() and FunctionalEvaluator() respectively.

There are other features as well One-shot neural architecture search and feature engineering. The idea that NNI is putting forward is to automate Neural Network model engineering. 

Essentially, NNI eases the model buildings and engineering phase while allowing you to manage AutoML machine learning experiments. Along with all the above it also provides a dashboard where you can monitor the tuning process which allows you to control the experiments. If you are someone who spends a lot of time building models and finetuning them then this tool is a necessity. 

Data validation open source ML tools

Data validation is the process of checking data quality. During this stage, you make sure that there are no inconsistencies or missing data in your sets. Data validation tools automate this routine process and improve the quality of data cleansing.  

Hadoop 

Hadoop is a freely redistributable set of utilities, libraries, and frameworks for developing and executing programs running on clusters. This fundamental technology for storing and processing Big Data is a top-level project of the Apache Software Foundation.

The project consists of 4 main modules:

• Hadoop Common

Hadoop Common is a set of infrastructure software libraries and utilities that are used in other solutions and related projects, in particular, for managing distributed files and creating the necessary infrastructure.

• HDFS is a distributed file system

Hadoop Distributed File System is a technology for storing files on various data servers with addresses located on a special name server. HDFS provides reliable storage of large files, block-by-block distributed between the nodes of the computing cluster.

• YARN is a task scheduling and cluster management system

YARN is a set of system programs that provide sharing, scalability, and reliability of distributed applications.

• Hadoop MapReduce

This is a platform for programming and performing distributed MapReduce calculations using many computers that form a cluster.

Today, there’s a whole ecosystem of related projects and technologies in Hadoop used for data mining and machine learning.

Apache Spark 

Apache Spark helps you to process semi-structured in-memory data. The main advantages of Spark are performance and a user-friendly programming interface.

The framework has five components: a core and four libraries, each solving a specific problem.

• Spark Core

This is the core of the framework. You can use it for scheduling and core I/O functionality.

• Spark SQL

Spark SQL is one of four framework libraries that comes in handy when working with processing data. To run faster, this tool uses DataFrames and can act as a distributed SQL query engine. 

• Spark Streaming

This is an easy-to-use streaming data processing tool. It breaks data into micro-batch mode. The creators of Spark claim that performance does not suffer much from this.

• MLlib

This is a high-speed distributed machine learning system. It’s nine times faster than its competitor, the Apache Mahout library when benchmarked against the alternating least squares (ALS) algorithm. MLlib includes popular algorithms for classification, regression, and recommender systems.

• GraphX

GraphX ​​is a library for scalable graph processing. GraphX ​​is not suitable for graphs that change in a transactional manner, for example, databases.

Spark is entirely autonomous but also compatible with other standard ML instruments, like Hadoop, if needed.

Great Expectations

For effective management of intricate data pipelines, data practitioners recognize the significance of testing and documentation. GX offers a solution for swift deployment of adaptable, expandable data quality testing within data stacks. Its user-friendly documentation ensures accessibility for both technical and non-technical users.

Great Expectations (GX) assists data teams in fostering a collective comprehension of their data by incorporating quality testing, documentation, and profiling.

Some of the key features are:

• Seamless Integration

GX seamlessly integrates into your current tech stack and can be linked with your CI/CD pipelines, enabling precise data quality enhancement. Validate and connect with your existing data, enabling Expectation Suites to perfectly address your data quality requisites.

• Quick Start

GX produces valuable outcomes promptly, even for large datasets. Its Data Assistants offer curated Expectations tailored for various domains, accelerating data discovery for rapid deployment of data quality across pipelines. Auto-generated Data Docs ensure ongoing up-to-date documentation.

• Unified Insight

Expectations serve as GX’s core abstraction, articulating anticipated data states. The Expectation library employs a human-readable vocabulary, catering to technical and non-technical users. Bundled into Expectation Suites, they excellently characterize your data expectations.

• Security and Transparency

GX preserves your data security by processing it within your own systems. Its open-source foundation ensures full transparency, allowing for complete control over insights.

• Data Contracts Support

Utilize Checkpoints for transparent, central, and automated testing of Expectations, producing readable Data Docs. Checkpoints can trigger actions based on evaluation results, bolstering data quality.

• Enhanced Collaboration

GX’s Data Docs are inspectable, shareable, and human-readable, fostering mutual understanding of data quality. Publish Data Docs in diverse formats to seamlessly integrate with existing catalogs, dashboards, and reporting tools.

Great Expectations aligns well with your MLOps tools by enhancing data reliability, reducing the risk of poor data quality impacting your machine learning models, and promoting a collaborative approach to data quality management within your team.

TensorFlow Extended (TFX)

TFX, short for TensorFlow Extended, presents a range of powerful features for effective machine learning operations:

• Scalable ML Pipelines

TFX offers a structured sequence of components tailored for scalable and high-performance machine learning tasks, streamlining the development of end-to-end ML pipelines.

• Component Modularity

TFX components are built using specialized libraries, providing both a cohesive framework and the flexibility to utilize individual components according to your needs.

• Data Preprocessing

TFX includes powerful tools for data preprocessing, transformation, and feature engineering, crucial for preparing data for model training.

• Model Training and Validation 

It supports model training using TensorFlow and facilitates model validation, ensuring the robustness and reliability of your machine learning models.

• Automated Model Deployment 

TFX simplifies the process of deploying models to various serving environments, enabling smooth integration with production systems.

• Artifact Tracking

TFX maintains track of experiment artifacts, aiding in tracking and managing the lifecycle of your ML models.

• Custom Component Development

It allows for the creation of custom components to meet specific requirements or integrate third-party tools.

• Integration with TensorFlow

As an extension of TensorFlow, TFX seamlessly integrates with TensorFlow ecosystem tools and technologies.

TFX is an excellent fit in your MLOps toolkit due to its focus on scalability, performance, and end-to-end ML pipeline management. It streamlines the development and deployment of machine learning workflows, ensuring efficient data preprocessing, model training, validation, and deployment. Its modularity and integration with TensorFlow make it a valuable asset in your quest for efficient and effective machine learning operations.

Data exploration open source ML tools

Data exploration software is created for automated data analysis that provides streamlined pattern recognition and easy insights visualization. Data exploration is a cognitively intense process, you need powerful tools that will help you track and execute code as you go.

Jupyter Notebook

Jupyter Notebook is a development environment where you can immediately see the result of executing code and its fragments. The difference from a traditional IDE is that the code can be broken into chunks and performed in any order. You can load a file into memory, check its contents separately, and also process the contents separately. 

• Multi-language support

Often when we talk about Jupyter Notebook, we mean working with Python. But, in fact, you can work with other languages, such as Ruby, Perl, or R. 

• Integration with the cloud

The easiest way to start working with a Jupyter Notebook in the cloud is by using Google Colab. This means that you just need to launch your browser and open the desired page. After that, the cloud system will allocate resources for you and allow you to execute any code.

The plus is that you don’t need to install anything on your computer. The cloud takes care of everything, and you just write and run code.

Data version control open source ML tools

There will be multiple machine learning model versions before you finish up. To make sure nothing gets lost, use a robust and trustworthy data version control system where every change is trackable.

Data Version Control (DVC)

DVC is a tool designed for managing software versions in ML projects. It’s useful both for experimentation and for deploying models to production. DVC runs on top of Git, uses its infrastructure, and has a similar syntax.

• Fully-automated version control

DVC creates metafiles to describe pipelines and versioned files that need to be saved in the Git history of your project. If you transfer some data under the control of DVC, it will start tracking all changes.

• Git-based modification tracking

You can work with data the same way as with Git: save a version, send it to a remote repository, get the required version of the data, and change and switch between versions. The DVC interface is intuitively clear. 

Overall, DVS is an excellent tool for data and model versioning. If you don’t need pipelines and remote repositories, you can version data for a specific project working on a local machine. DVC allows you to work very quickly with tens of gigabytes of data.

However, it also allows you to exchange data and models between teams. For data storage, you can use cloud solutions. 

Pachyderm

Pachyderm is a Git-like tool for tracking transformations in your data. It keeps track of data lineage and ensures that data is kept relevant. 

Pachyderm is useful because it provides:

• Traceability

You want your data to be fully traceable from the moment it’s raw to the final prediction. With its version control for data, Pachyderm gives you a fully transparent view of your data pipelines. It can be a challenge; for example, when multiple transformers use the same dataset, it can be hard to say why you get this or that result. 

• Reproducibility

Pachyderm is a step forward to the reproducibility of your data science models. You will always be assured that your clients can get the same results after the model is handed down to them.

Pachyderm stores all your data in one central location and updates all the changes. No transformation will pass unnoticed. 

Related post

Top Model Versioning Tools for Your ML Workflow

Read more

Data inspection open source ML tools

Alibi Detect 

Alibi Detect is an open-source Python library by SeldonIO which also provide Seldon Core which we discussed earlier. This library allows you to inspect your data’s integrity. It offers features like outlier, adversarial, and drift detection for tabular data, text, images and time series. It is compatible with TensorFlow and PyTorch backends. 

Abili Detect offers a variety of methods for inspecting your data’s integrity. The documentation is pretty neat and also offers a examples for better understanding. I highly recommend you to go through the documentation as it will be extremely beneficial. 

If you are using frameworks like TensorFlow and PyTorch then this would be the best reasons to Abili Detect as it will create a smooth transition in the machine learning pipeline. Another reason to use this library in your machine learning workflow is because it provides built-in preprocessing steps. This feature essentially enables you to detect drift while using the transformers library. It also helps you to extract hidden layer from machine learning models. 

Frouros

Frouros is an open source Python library aimed only to address drift detection. Unlike Abili Detect which offers inspection for outlier and adversarial detection, Frouros is only focused on drift detection. This library is special because it offers classical and more recent algorithms for detecting both data and concept drift. 

Frouros is also a lightweight library which works with Scikit-Learn, Numpy, PyTorch, and other frameworks. It offers a wide variety of methods which will be suitable for largely only univariant datasets and a few multivariate datasets as well. 

So as a final verdict, this library is good for people who want to explore the concept of data drifts in univariant datasets. But since this library offers a vast range of algorithms it is a good place to learn and even deploy in a fairly small project. 

Model serving open source ML tool

StreamLit

Streamlit is an open-source Python library that is used for creating interactive web applications mostly related to data science and ML projects. Streamlit comes under the framework category, but since it only allows you to deploy the ML application I have put it under the tools category. 

Anyways, StreamLit allows you to build web-based dashboards, visualizations, and applications with minimal effort. Some of the key features include:

• Rapid Prototyping

As mentioned previously you can create interactive applications by writing Python code that directly interacts with your data and visualizations

• Simplicity

The library is designed to be user-friendly, with a simple and intuitive API calling functions. You can create interactive widgets with just a few lines of code.

• Data Visualization

Streamlit supports integration with popular data visualization libraries like Matplotlib, Plotly, and Altair, enabling you to display charts and graphs in your web application.

• Customization

While Streamlit is straightforward to use out of the box, you can also customize the appearance and layout of your apps using CSS styling and additional layout components.

• Integration

You can integrate your Streamlit apps with machine learning models, data analysis scripts, and other Python-based functionalities to create cohesive data-driven machine learning applications.

• Interactivity

Streamlit’s widgets and features allow users to interact with data, adjust parameters, and see real-time updates in the app’s visualizations.

• Sharing and Deployment

You can deploy your Streamlit apps on various platforms, including cloud services, making it easy to share your work with others.

• Community and Extensions

Streamlit has a growing community and supports a range of extensions and integrations, allowing you to enhance the functionality of your apps.

Streamlit is particularly well-suited for scenarios where you want to create simple and interactive data visualization tools or prototypes without investing a significant amount of time in web development. It’s commonly used by data scientists and engineers who want to showcase their data analysis and machine learning results in an accessible and engaging manner.

TorchServe

TorchServe is an open-source model serving tool, made by Facebook AI. It is engineered to simplify the deployment and management of PyTorch models, aligning seamlessly with your MLOps workflows. Let’s delve into why TorchServe is a compelling choice for model management and inference in the MLOps landscape.

• Efficient Model Management

One of TorchServe’s standout features is its robust Model Management API. It empowers MLOps practitioners with multi-model management capabilities, allowing the allocation of models to workers in an optimized manner. This means you can effortlessly handle multiple models, versioning, and configurations while ensuring resource allocation is fine-tuned for peak performance.

• Versatile Inference Support

TorchServe extends its capabilities through its Inference API, offering support for both REST and gRPC protocols. But it doesn’t stop there; it’s equipped for batched inference, optimizing the prediction process for both single and multiple data points. This versatility ensures that your models can be integrated seamlessly into a wide array of applications.

• Complex Deployments Made Simple

For those tackling intricate deployments involving complex Directed Acyclic Graphs (DAGs) with interdependent models, TorchServe comes to the rescue. Its TorchServe Workflows feature enables the deployment of these intricate setups, giving you the flexibility needed to cater to demanding real-world scenarios.

• Wide Adoption in Leading MLOps Platforms

TorchServe’s reputation extends beyond its own ecosystem. It serves as the default choice for serving PyTorch models within platforms like Kubeflow, MLflow, SageMaker, Google Vertex AI, and Kserve, supporting both v1 and v2 APIs. This widespread adoption speaks volumes about its effectiveness and compatibility within the MLOps landscape.

• Optimized Inference Export

In the quest for optimized inference, TorchServe offers a suite of options. Whether it’s TorchScript right out of the box, ONNX, ORT, IPEX, or TensorRT, you have the freedom to export your model in a format that suits your specific performance requirements. This flexibility ensures that your models are primed for efficient execution.

• Performance at the Core

MLOps professionals know that performance is paramount. TorchServe recognizes this and provides built-in support to optimize, benchmark, and profile both PyTorch models and TorchServe itself. This means you can fine-tune your deployments for optimal throughput and responsiveness.

• Expressive Handlers for Custom Use Cases

Handling inferencing for diverse use cases is a breeze with TorchServe’s expressive handler architecture. It simplifies the process of customizing inferencing for your unique requirements, and it comes with a plethora of out-of-the-box solutions to cater to various scenarios.

• Comprehensive Metrics and Monitoring

Monitoring the health and performance of your models is vital. TorchServe comes with a Metrics API that offers out-of-the-box support for system-level metrics. It seamlessly integrates with Prometheus for metric exports, and it also supports custom metrics. Moreover, it aligns seamlessly with PyTorch’s profiler for in-depth performance analysis.

TorchServe seamlessly integrates with leading MLOps platforms and empowers you to deploy and manage models efficiently. If you’re seeking a robust solution to elevate your MLOps workflows, TorchServe deserves a prominent place in your toolkit.

Testing and maintenance open source ML tools

The final step of ML development is testing and maintenance after the main jobs are done. Special tools allow you to make sure that the results are reproducible in the long run.

Prometheus

Prometheus is an open-sourced monitoring toolkit built by Soundcloud. This toolkit has a very active community, and it is well-supported by a large number of organisation. The fundamental concept of Prometheus is that it stores all data and metrics in a time series format. This means that metrics collected during the monitoring phase is associated with a timestamp. 

This is a reason why Prometheus fits very well with timeseries data. It also supports multi-dimensional data collection along with querying the dataset. This means you can use Prometheus to log your ML system metrics. 

The image above represents how metrics can be logged into the time series database, and later the same can be retrieved via endpoints. 

Some of the highlighted key features are:

• Standalone servers

Each Prometheus server is a standalone server which means they are independent of others making it reliable. 

• PromQL

A powerful query language that allows searching, slicing, and dicing of time series data. With PromQL you can also generate graphs, table, and alerts on Prometheus’s expression browser.

• Efficient storage

The data is stored in memory and a local on-disk time series database in a custom format. This allows efficient scaling as well. 

• Dimensional Data

The keyconcept of Prometheus is storing data in a time series format. Because of this you can select any timeframe to understand the behaviour of your model. On top of that, you can create a visualization dashboard using Grafana.  

If you want a general-purpose lightweight tool to collect and log metrics about your system then Prometheus is a must. 

ModsysML

ModsysML is an extremely new MLOps tool that allows users to test, automate workloads and compare outputs, improve data quality, and catch regressions all in a single API. This enables you to automate, accelerate and backtest the entire process of running proactive intelligence and insights through testing data quality. 

ModsysML streamlines the process of meticulously refining AI systems across a diverse range of pertinent test cases. By meticulously scrutinizing and contrasting outputs, it constructs workflows that facilitate decision-making. Users can expedite the assessment of quality and promptly identify regressions.

The suite of tools we offer encompasses three fundamental functions:

• Conducting performance benchmarks for AI systems with respect to precise outcomes.
• Crafting automated tasks or revisiting established ones for a thorough evaluation.
• Detecting immediate fluctuations within data streams.

Empowered by a user interface (UI) and a Python library, you possess the means to intricately calibrate your AI systems for particular use cases. This encompasses the creation of automated workflows as well as deriving data-driven insights from real-time shifts within your datasets.

Deepchecks

We come across another open-source tool that allows you to thoroughly evaluate and test the integrity of the data as well the ML model. Deepchecks offer continuous evaluation from research to production. It has a strong and active community backing it up. 

Deepchecks cater to tabular, NLP, and computer vision (CV) datasets. It offers four solutions:

Deepchecks offer a convenient avenue for detecting imperfections within data and ML models, while also enabling proactive steps toward enhancement. Its features, the Suite, prove particularly advantageous by facilitating an in-depth assessment of diverse data and ML model facets, subsequently generating valuable reports.

To facilitate a clearer grasp, a selection of the predefined checks carried out within a suite, along with their functions, is outlined below:

• Dataset Integrity

Employed to ascertain the accuracy and comprehensiveness of the dataset.

• Train-Test Validation

A set of checks is devised to ascertain the appropriateness of the data split for the model training and testing phases.

• Model Evaluation 

A set of checks is performed to gauge model performance, its adaptability to diverse scenarios, and any indicators of overfitting.

One of the reasons why Deepcheck will fit in your workflow is because of the automated solutions it offers especially root cause analysis. It essentially expedites the process of grasping the fundamental source of the problem across the entire model lifecycle, allowing you to swiftly discern the underlying cause of the issue. It promises to give you granular details of the issue. 

Experiment tracking open source ML tools

Aim

Aim stands as an open-source, self-hosted AI Metadata monitoring solution tailored to manage vast volumes of tracked metadata sequences, numbering in the tens of thousands.

Aim presents an efficient and visually pleasing user interface (UI) that facilitates the exploration and juxtaposition of metadata, encompassing elements such as training runs or agent executions. What’s more, its software development kit (SDK) grants the capability for programmatic interaction with the tracked metadata—an ideal feature for streamlined automation and analysis within Jupyter Notebooks.

Some of the key features are:

• Streamlined Run Comparisons

Effortlessly contrast various runs to expedite the model-building process.

• In-Depth Run Inspection

Immerse yourself in the minutiae of each run, facilitating seamless troubleshooting.

• Centralized Repository of Pertinent Details

All pertinent information is centralized, ensuring hassle-free governance and management.

Aim can handle up to 100,000 metadata sequences which is why it can be one of the best fits in your ML stack. Apart from that, it has a beautiful UI which functional and appealing to the eyes. 

Guild AI

Guild AI serves as an open source toolkit that streamlines and enhances the efficiency of machine learning experiments. It stands as an all-encompassing ML engineering toolkit with an array of capabilities.

• Automated Experiment Tracking

Guild AI lets you run original training scripts, capturing unique experiment results and provides tools for analysis, visualization, and comparison.

• Hyperparameter Tuning with AutoML

Harness AutoML for hyperparameter tuning by automating trials with grid search, random search, and Bayesian optimization techniques.

• Comparison and Analysis 

Compare and analyze experiment runs to gain insights and enhance your model’s performance.

• Efficient Backup and Archiving

Secure training-related operations like data preparation and testing and archive runs to remote systems such as S3.

• Remote Operations and Acceleration 

Perform operations remotely on cloud accelerators, optimizing your workflow efficiency.

• Model Packaging and Reproducibility 

Package and distribute models for seamless reproducibility across different environments.

• Streamlined Pipeline Automation

Enable automated pipelines for smoother workflow execution.

• Scheduling and Parallel Processing

Utilize scheduling and parallel processing to optimize resource utilization.

• Remote Training and Management

Conduct remote training, backup, and restoration of experiments for enhanced flexibility.

If you are looking for a tool that offers automated experiment management, optimization, and insights that streamline and enhance machine learning workflows, then Guild AI is tool of choice. 

Model interpretability open source ML tools 

Alibi Explain

Alibi Explain is another tool from SeldonIO. It stands as an open-source Python library with a primary focus on the explainability and interpretation of ML models. The library is dedicated to furnishing top-tier implementations of explanation methods, encompassing black-box, white-box, local, and global approaches tailored for both classification and regression models.

Within Alibi Explain, a collection of algorithms or methodologies, termed explainers, are at your disposal. Each explainer serves as a conduit for obtaining insights into a model’s behavior. The array of insights attainable, contingent upon a trained model, is influenced by several variables. 

To know more please read their documentation here. It is one of the most pleasing tools in this list. 

Broadly speaking, the range of explainers available from Alibi is bounded by:

• The nature of the data the model handles, encompassing images, tabular data, or text.
• The task performed by the model, namely regression or classification.
• The specific model type employed, including neural networks and random forests.

Explainibility in general is one of the most sought features in the ML world. It is because as humans we are curious to know what is happening inside a closed. If you are working in medicine or healthcare or any sort of life-related industry then this tool is a must in your arsenal. 

Conclusion

Open source MLOps tools are necessary. They help you automate a large amount of routine work without costing a fortune. Fully-fledged platforms offer a wide selection of tools for different purposes, for whatever technological stack you might desire. In practice, however, it often turns out that you still need to integrate them with specialized tools that are more intuitive to use. Luckily, most open-source tools make the integration as seamless as possible. 

However, an important thing to understand about open-source tools is that you shouldn’t expect them to be completely free of charge: the costs of infrastructure, support, and maintenance of your ML projects will still be on you.

In this article, we’ll delve into the art of machine learning visualization, exploring various techniques that help us make sense of complex data-driven systems. I have also prepared a Google Colab notebook with visualization examples to try yourself.

So, without further ado, let’s get started.

What is visualization in machine learning?

Machine learning visualization (ML visualization for short) generally refers to the process of representing machine learning models, data, and their relationships through graphical or interactive means. The goal is to make comprehending a model’s complex algorithms and data patterns easier, making it more accessible to technical and non-technical stakeholders. 

Visualization bridges the gap between the enigmatic inner workings of ML models and our innate human capacity for understanding patterns and relationships through visuals.

Visualizing ML models can help with a wide range of objectives:

• Model structure visualization: Common model types, such as decision trees, support vector machines, or deep neural networks, often consist of many layers of computations and interactions that are challenging to grasp for humans. Visualization lets us see more easily how data flows through a model and where transformations occur.

• Visualizing performance metrics: Once we have trained a model, we need to assess its performance. Visualizing metrics such as accuracy, precision, recall, and the F1 score helps us see how well our model is doing and where improvements are needed.

• Comparative model analysis: When dealing with multiple models or algorithms, visualization of differences in structure or performance allows us to choose the best one for a particular task.

• Feature importance: It is vital to understand which features influence a model’s predictions the most. Visualization techniques like feature importance plots make identifying the critical factors driving model outcomes easy.

• Interpretability: Due to their complexity, ML models are often “black boxes” to their human creators, making it hard to explain their decisions. Visualizations can shed light on how specific features affect the output or how robust a model’s predictions are.

• Communication: Visualizations are a universal language for conveying complex ideas simply and intuitively. They are essential for effectively sharing information with management and other non-technical stakeholders.

Model structure visualization

Understanding how data flows through a model is essential in understanding how a machine learning model transforms the input features into its output.

Decision tree visualization

Decision trees have a flowchart-like structure that’s familiar to most people. Each internal node represents a decision based on the value of a specific feature. Each branch from a node signifies an outcome of that decision. The Leaf nodes represent the model’s outputs.

Visualization of this structure offers a straightforward representation of the decision-making process, enabling data scientists and business stakeholders alike to comprehend the decision rules the model has learned.

During training, a decision tree identifies the feature that best separates the samples in a branch based on a specific criterion, often the Gini impurity or information gain. In other words, it determines the most discriminative feature.

Visualizing decision trees (or their ensembles like random forests or gradient-boosted trees) involves a graphical rendering of their overall structure, displaying the splits and decisions at each node clearly and intuitively. The depth and width of the tree, as well as the leaf nodes, become evident at first sight. Moreover, decision tree visualization aids in identifying crucial features, the most discriminative attributes that lead to accurate predictions.

The path to accurate prediction can be summed in four steps:

• Feature Clarity: Decision tree visualization is like peeling back layers of complexity to reveal the pivotal features at play. It’s akin to looking at a decision-making flowchart, where each branch signifies a feature, and each decision node holds a crucial aspect of our data.
  
• Discriminative Attributes: The beauty of a decision tree visualization lies in its ability to highlight the most discriminative features. These factors heavily influence the outcome, guiding the model in making predictions. Through visualizing the tree, we can pinpoint these features and thus understand the core factors driving our model’s decisions.
  
• Path to Precision: Every path down the decision tree is a journey towards precision. The visualization showcases the sequence of decisions that lead to a particular prediction. This is gold for understanding the logic and criteria our model uses to reach specific conclusions.

• Simplicity Amidst Complexity: Despite the complexity of machine learning algorithms, decision tree visualization comes with an element of simplicity. It transforms intricate mathematical calculations into an intuitive representation, making it accessible to technical and non-technical stakeholders.

The diagram above shows the structure of a decision tree classifier trained on the famous Iris dataset. This dataset consists of 150 samples of iris flowers, each belonging to one of three species: setosa, versicolor, or virginica. Each sample has four features: sepal length, sepal width, petal length, and petal width.

From the decision tree visualization, we can understand how the model classifies a flower:

1. Root node: At the root node, the model determines whether the petal length is 2.45 cm or less. If so, it classifies the flower as setosa. Otherwise, it moves on to the next internal node.
   
2. Second split based on petal length: If the petal length is greater than 2.45 cm, the tree again uses this feature to make a decision. The decision criterion is whether the petal length is less than or equal to 4.75 cm.

3. Split based on petal width: If the petal length is less than or equal to 4.75 cm, the model next considers the petal width and determines whether it is above 1.65 cm. If so, it classifies the flower as virginica. Otherwise, the model’s output is versicolor.

4. Split based on sepal length: If the petal length is greater than 4.75 cm, the model determined during training that sepal length is best suited to distinguish versicolor from virginica. If the sepal length is greater than 6.05 cm, it classifies the flower as virginica. Otherwise, the model’s output is versicolor.

The visualization captures this hierarchical decision-making process and represents it in a way that is easier to understand than a simple listing of decision rules.

Ensemble model visualization

Ensemble approaches like random forests, AdaBoost, gradient boosting, and bagging combine multiple simpler models (called base models) into one larger, more accurate model. For example, a random forest classifier comprises many decision trees. Understanding the comprising models’ contributions and complex interplay is crucial when debugging and assessing ensembles.

One way to visualize an ensemble model is to create a diagram showing how the base models contribute to the ensemble model’s output. A common approach is to plot the base models’ decision boundaries (also called surfaces), highlighting their influence across different parts of the feature space. By examining how these decision boundaries overlap, we can learn how the base models give rise to the collective predictive power of the ensemble.

Ensemble model visualizations also help users better comprehend the weights assigned to each base model within the ensemble. Typically, base models have a strong influence in some regions of the feature space and little influence in others. However, there might also be base models that never contribute significantly to the ensemble’s output. Identifying base models with particularly low or high weights can help to make ensemble models more robust and improve their generalizability.

Visually building models

Visual ML is an approach to designing machine-learning models using a low-code or no-code platform. It enables users to create and modify complex machine-learning processes, models, and outcomes through a user-friendly visual interface. Instead of retroactively generating model structure visualizations, Visual ML places them at the heart of the ML workflow.

In a nutshell, Visual ML platforms offer drag-and-drop model-building workflows that allow users of various backgrounds to create ML models easily. They bridge the gap between the abstract world of algorithms and our innate ability to grasp patterns and relationships through visuals.

These platforms can save us time and help us build model prototypes quickly. Since models can be created in minutes, training and comparing different model configurations is easy. The model which performs best can then be optimized further, perhaps using a more code-centric approach.

Data scientists and machine learning engineers can make use of Visual ML tools to create:

Examples of Visual ML tools are TensorFlow’s Neural Network Playground and KNIME, an open-source data science platform built entirely around Visual ML and No-Code concepts.

Visualize machine learning model performance

In many cases, we do not care so much about how a model works internally but are interested in understanding its performance. For which kinds of samples is it reliable? Where does it frequently draw the wrong conclusions? Should we go with model A or model B?

In this section, we’ll look at machine learning visualizations that help us better understand a model’s performance.

Confusion matrices

Confusion matrices are a fundamental tool for evaluating a classification model’s performance. A confusion matrix compares a model’s predictions with the ground truth, clearly showing what kind of samples a model misclassifies or where it struggles to distinguish between classes. 

In the case of a binary classifier, a confusion matrix has just four fields: true positives, false positives, false negatives, and true negatives:

Equipped with this information, it’s straightforward to calculate essential metrics like precision, recall, F1 score, and accuracy.

The confusion matrix for a multi-class model follows the same general idea. The diagonal elements represent correctly classified instances (i.e., the model’s output matches the ground truth), while off-diagonal elements signify misclassifications.

Here is a small snippet to generate a confusion matrix for a sci-kit-learn classifier:

import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC


# generate some sample data
X, y = make_classification(n_samples=1000,
n_features=10,
n_informative=6,
n_redundant = 2,
n_repeated = 2,
n_classes = 6,
n_clusters_per_class=1,
random_state = 42
)


# split the data into train and test set
X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)


# initialize and train a classifier
clf = SVC(random_state=0)
clf.fit(X_train, y_train)


# get the model’s prediction for the test set
predictions = clf.predict(X_test)


# using the model’s prediction and the true value,
# create a confusion matrix
cm = confusion_matrix(y_test, predictions, labels=clf.classes_)


# use the built-in visualization function to generate a plot
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=clf.classes_)
disp.plot()
plt.show()

Let’s have a look at the output. As mentioned before, the elements in the diagonal represent the true class, and the off-diagonal elements represent cases where the model confuses classes – hence the name “confusion matrix.”

Here are three key takeaways from the plot:

1. Diagonal: Ideally, the matrix’s main diagonal should be populated with the highest numbers. These numbers represent the instances where the model correctly predicted the class, aligning with the true class. Looks like our model is doing pretty well here!
   
2. Off-diagonal entries: The numbers outside the main diagonal are equally important. They reveal cases where the model made errors. For example, if you look at the cell where row 5 intersects with column 3, you’ll see that there were five cases where the true class was “5”, but the model predicted class “3”. Perhaps we should look at the affected samples to better understand what’s going on here!
   
3. Analyzing performance at a glance: By examining the off-diagonal entries, you can see immediately that they’re quite low. Overall, the classifier seems to do a pretty good job. You’ll also notice that we have about an equal number of samples for each category. In many real-world scenarios, this is not going to be the case. Then, generating a second confusion matrix that shows the likelihood of a correct classification (rather than the absolute number of samples) can be helpful.

Visual enhancements like color gradients and percentage annotations make a confusion matrix more intuitive and easily interpretable. Confusion matrices styled like a heatmap draw attention to classes with high error rates and thus guide further model development.

Confusion matrices can also help non-technical stakeholders grasp a model’s strengths and weaknesses, fostering discussions about the need for additional data or cautionary measures when using model predictions for critical decisions.

Related post

Doing ML Model Performance Monitoring The Right Way

Read more

Visualizing cluster analysis

Cluster analysis groups similar data points based on specific features. Visualizing these clusters can bring to light patterns, trends, and relationships within the data.

Scatter plots where each point is colored according to its cluster assignment are a standard way to visualize the results of a cluster analysis. Cluster boundaries and their distribution across the feature space are clearly visible. Pair plots or parallel coordinates help to understand the relationships between multiple features.

One popular clustering algorithm, k-means, begins with selecting starting points called centroids. A simple approach is randomly picking k samples from the dataset.

Once these initial centroids are established, k-means alternates between two steps:

As this process continues, the centroids move, and the association of points with clusters is iteratively refined. Once the difference between the old and new centroids falls below a set threshold, signaling stability, k-means concludes. 

The result is a set of centroids and clusters that you can visualize in a plot like the one above.

For larger datasets, t-SNE (t-distributed Stochastic Neighbor Embedding) or UMAP (Uniform Manifold Approximation and Projection) can be employed to reduce dimensions while preserving cluster structures. These techniques aid in visualizing high-dimensional data effectively. 

t-SNE takes complex, high-dimensional data and transforms it into a lower-dimensional representation. The algorithm starts by assigning each data point a location in the lower-dimensional space. Then, it looks at the original data and decides where each point should really be placed in this new space, considering its neighboring points. Points that were similar in the high-dimensional space are pulled closer together in the new space, and those dissimilar are pushed apart.

This process repeats until the points find their perfect positions. The final result is a clustered representation where similar data points form groups, allowing us to see patterns and relationships hidden in the high-dimensional chaos. It’s like a symphony where each note finds its harmonious place, creating a beautiful composition of data.

UMAP also tries to find clusters in high-dimensional space but takes a different approach.

Here is how UMAP works:

• Neighbor finding: UMAP begins by identifying the neighbors of each data point. It determines which points are close to each other in the original high-dimensional space.
  
• Fuzzy simplicial set construction: Imagine creating a web of connections between these neighboring points. UMAP models the strength of these connections based on how related or similar the points are.
  
• Low-Dimensional Layout: After determining their closeness, UMAP carefully arranges the data points in the lower-dimensional space. Points strongly connected in the high-dimensional space are placed close together in this new space.

• Optimization: UMAP aims to find the best representation in lower dimensions. It minimizes the difference between the distances in the original high-dimensional space and the new lower-dimensional space.
  
• Clustering: UMAP uses clustering algorithms to group similar data points. Imagine gathering similar colored marbles together — this allows us to see patterns and structures more clearly.

You may also like

Dimensionality Reduction for Machine Learning

Read more

Comparative model analysis

Comparing different model performance metrics is crucial for deciding which machine learning model is best suited for a task. Whether during the experimental phase of an ML project or while re-training production models, visualizations are often necessary to turn complex numeric results into actionable insights.

Thus, visualizations for model performance metrics, such as ROC curves and calibration plots, are tools every data scientist and ML engineer should have in their toolbox. They are fundamental for understanding and communicating the effectiveness of machine learning models.

ROC curves

Receiver operating characteristic curves – ROC curves for short – are vital when analyzing machine-learning classifiers and comparing ML model performance.

A ROC curve plots a model’s true positive rate against its false positive rate as a function of the cutoff threshold. It depicts the trade-off between true and false positives we invariably have to make and offers insight into a model’s discriminative power.

A curve closer to the top-left corner signifies superior performance: The model achieves a high rate of true positives while maintaining a low rate of false positives. Comparing ROC curves helps us choose the best model.

Here is a step-by-step explanation of how the ROC curve works:

In binary classification, we are interested in predicting one of two possible outcomes, typically labeled as positive (e.g., presence of a disease) and negative (e.g., absence of a disease).

Remember that we can turn any classification problem into a binary one by selecting one class as the positive outcome and assigning all other classes as negative outcomes. Hence, ROC curves can still be helpful for multi-class or multi-label classification problems.

The axes of the ROC curve represent two metrics:

• True Positive Rate (Sensitivity): The proportion of actual positive cases correctly identified by the model.
• False Positive Rate: The proportion of actual negative cases incorrectly identified as positive.
  

A machine-learning classifier typically outputs the likelihood that a sample belongs to the positive class. For example, a logistic regression model outputs values between 0 and 1 that can be interpreted as the likelihood.

As data scientists, it’s up to us to select the threshold above which we assign the positive label. The ROC curve shows us the influence of that choice on our classifier’s performance.

If we set the threshold to 0, all samples will be assigned to the positive class – and the rate of false positives will be 1. Thus, in the upper right-hand corner of any ROC curve plot, you’ll see that the curve ends at (1, 1).

If we set the threshold to 1, no samples will ever be assigned to the positive class. But since, in this case, we never mistakenly assign a negative sample to the positive class, the rate of false positives will be 0. As you might have guessed already, that’s what we see in the lower left-hand corner of a ROC curve plot: The curve always begins at (0, 0).

The curve between those points is plotted by changing the threshold for classifying a sample as positive. The resulting curve – the ROC curve – reflects how the true positive rate and false positive rate change in relation to one another as this threshold varies.

But what do we learn from this? 

The ROC curve shows the trade-off we must make between sensitivity (the true positive rate) and specificity (1 – false positive rate). In more colloquial terms, we can either find all the positive samples (high sensitivity) or be sure that all samples our classifier identifies as positive actually belong to the positive class (high specificity).

Consider a classifier that can perfectly distinguish between positive and negative samples: Its true positive rate is always 1, and its false positive rate is always 0, independent of our chosen threshold. Its ROC curve would shoot straight up from (0,0) to (0,1) and then resemble a straight line between (0,1) and (1,1).

Thus, the closer the ROC curve follows the left-hand border and then the top border of the plot, the more discriminative the model and the better it can satisfy the sensitivity and specificity objectives.

To compare different models, we often don’t use the curve directly but compute the area under it. This quantifies the model’s overall ability to discriminate between positive and negative classes.

This so-called ROC-AUC (the area under the ROC curve) can take on values between 0 and 1, with higher values indicating a better performance. Indeed, our perfect classifier would reach a ROC-AUC of exactly 1.

When using the ROC-AUC metric, it’s essential to keep in mind that the baseline is not 0 but 0.5 – the ROC-AUC of a perfectly random classifier. If we use np.random.rand() as our classifier, the resulting ROC curve will be a diagonal line from (0,0) to (1,1).

Generating ROC curves and computing the ROC-AUC is straightforward using scikit-learn. It takes just a few lines of code in your model training script to create this evaluation data for each of your training runs. When you log the ROC-AUC and the ROC curve plot using an ML experiment tracking tool, you can later compare different model versions.

Might be useful

When visualizing, comparing, and debugging models, it’s really useful to keep an organized record of all experiments.

Media intelligence company Hypefactors is using neptune.ai for that.

We use Neptune for most of our tracking tasks, from experiment tracking to uploading the artifacts. A very useful part of tracking was monitoring the metrics, now we could easily see and compare those F-scores and other metrics.

Andrea Duque, Data Scientist at Hypefactors

See in app Full screen preview

• 

  Full case study with Hypefactors

• 

  Dive into available comparison features

• 

  Get in touch if you’d like to go through a custom demo with your team

Computing and logging the ROC-AUC

from sklearn.metrics import roc_auc_score


clf.fit(x_train, y_train)


y_test_pred = clf.predict_proba(x_test)
auc = roc_auc_score(y_test, y_test_pred[:, 1])


# optional: log to an experiment-tracker like neptune.ai
neptune_logger.run["roc_auc_score"].append(auc)

Creating and logging a ROC plot

from scikitplot.metrics import plot_roc
import matplotlib.pyplot as plt


fig, ax = plt.subplots(figsize=(16, 12))
plot_roc(y_test, y_test_pred, ax=ax)


# optional: log to an experiment tracker like neptune.ai
from neptune.types import File
neptune_logger.run["roc_curve"].upload(File.as_html(fig))

Calibration curves

While machine-learning classifiers typically output values between 0 and 1 for each class, these values do not represent a likelihood or confidence in the statistical sense. That’s perfectly fine in many cases because we’re only interested in obtaining the correct labels.

But if we want to report a confidence level along with the classification outcome, we must ensure our classifier is calibrated. Calibration curves are a helpful visual aid to understand how well a classifier is calibrated. We can also use them to compare different models or to check that our attempts to re-calibrate a model were successful.

Let’s again consider the case of a model that outputs values between 0 and 1. If we choose a threshold, say 0.5, we can turn this into a binary classifier where all samples for which the model outputs a higher value are assigned to the positive class (and vice versa).

A calibration curve plots the “fraction of positives” against the model’s output. The “fraction of positives” is the conditional probability that a sample actually belongs to the positive class given the model’s output (P(sample belongs to positive class|model’s output between 0 and 1)).

Does that sound way too abstract? Let’s look at an example:

First, have a look at the diagonal line. It represents a perfectly calibrated classifier: The model’s output between 0 and 1 is precisely the probability that a sample belongs to the positive class. For example, if the model outputs 0.5, there’s a 50:50 chance the sample belongs to either the positive or negative class. If the model outputs 0.2 for a sample, there is only a 20% chance that the sample belongs to the positive class.

Next, consider the calibration curve for the Naive Bayes classifier: You see that even when this model outputs 0, there is about a 10% chance that the sample is positive. If the model outputs 0.8, there’s still a 50% chance that the sample belongs to the negative class. Hence, the classifier’s output does not reflect its confidence.

Computing the “fraction of positives” is far from straightforward. We need to create bins based on the model’s outputs, which is complicated by the fact that the distribution of samples across the model’s value range is typically not homogeneous. For example, a logistic regression classifier typically assigns values close to 0 or 1 to many samples but rarely outputs values close to 0.5. You can find a more in-depth discussion of this topic in the scikit-learn documentation. There, you can also dive into possible ways to re-calibrate models, which is beyond the scope of this article.

For our purposes here, we’ve seen how calibration curves visualize complex model behavior in an easy-to-grasp fashion. From a quick glance at the plot, we can see whether models are well-calibrated and which comes closest to the ideal.

Visualizing hyperparameter tuning

Hyperparameter tuning is a critical step in developing a machine-learning model. The aim is to select the best configuration of hyperparameters – a generic name for parameters not learned by the model from the data but pre-defined by its human creators. Visualizations can aid data scientists in understanding the impact of different hyperparameters on a model’s performance and properties.

Finding the optimal configuration of hyperparameters is a skill on its own and goes far beyond the machine learning visualization aspect we will focus on here. To learn more about hyperparameter tuning in all its depth, I recommend this article on improving ML model performance by a former Amazon AI researcher. 

A common approach to systematic hyperparameter optimization is creating a list of possible parameter combinations and training a model for each. This is often referred to as “grid search.”

For instance, if you are training a Support Vector Machine (SVM), you might want to try out different values for the parameters C (the regularization parameter) and gamma (the kernel coefficient):

import numpy as np
 
C_range = np.logspace(-2, 10, 13)
gamma_range = np.logspace(-9, 3, 13)

param_grid = {“gamma”: gamma_range, “C”: C_range}

Using scikit-learn’s GridSearchCV, you can train models for each possible combination (using a cross-validation strategy) and find the best one with respect to an evaluation metric:

from sklearn.model_selection import GridSearchCV,

grid = GridSearchCV(SVC(), param_grid=param_grid, scoring=’accuracy’)
grid.fit(X, y)

After the grid search concludes, you can inspect the results:

print(
"The best parameters are %s with a score of %0.2f"
% (grid.best_params_, grid.best_score_)
)

But we’re usually not just interested in finding the best model but also want to understand the effect its parameters have. For example, if a parameter does not influence the model’s performance, we don’t need to waste time and money by trying out even more different values. On the other hand, if we see that as a parameter’s value increases, the model’s performance gets better, we might want to try even higher values for this parameter.

Here’s a visualization of the grid search we just performed:

From the plot, we see that the value of gamma greatly influences the SVM’s performance. If gamma is set too high, the influence radius of support vectors is minimal, potentially causing overfitting even with substantial regularization through C. Conversely, an extremely small gamma overly restricts the model, making it incapable of capturing the intricacies of the patterns within the data. In this scenario, the influence region of any support vector spans the entire training set, rendering the model akin to a linear one, using hyperplanes to separate dense areas of different classes.

The best models lie along a diagonal line of C and gamma, as depicted in the second plot panel. By adjusting gamma (lower values for smoother models) and increasing C (higher values for greater emphasis on correct classification), we can traverse this diagonal to achieve well-performing models.

Even from this simple example, you can see how helpful visualizations are for drilling down into the root causes of differences in model performance. This is why many machine-learning experiment tracking tools enable data scientists to create different types of visualizations to compare model versions.

May be useful

How to Compare Multiple Runs

Learn more

Feature importance visualization

Feature importance visualizations provide a clear and intuitive way to grasp the contribution of each feature in the model’s decision-making process. Understanding which features significantly influence predictions is paramount in many applications.

Plenty of different approaches to extracting insights about feature importance from machine-learning models exist. Broadly speaking, we can divide them into two categories:

• Some kinds of models, like decision trees and random forests, inherently contain feature importance information as part of their model structure. All we need to do is extract and visualize it.

• Most machine-learning models in use today do not provide feature importance information out of the box. We have to use statistical techniques and algorithmic approaches to uncover the importance of each of their input features on the model’s final output.

In the following, we’ll look at one example of each category: the mean decrease in impurity approach for random forest models and the model-agnostic LIME interpretability method. Other approaches you might want to look into comprise permutation importance, SHAP, and integrated gradients.

For the purpose of this article, we don’t care so much about how to obtain feature-importance data but about its visualization. To this end, bar charts are the top choice for structured data, with the length of each bar signifying the feature’s importance. Heatmaps are a clear favorite for images, and for text data, highlighting the most important words or phrases is typical.

In a business context, feature importance visualization is an invaluable tool for stakeholder communication. It provides a straightforward narrative, demonstrating the factors that predominantly influence predictions. This transparency enhances decision-making and can foster trust in the model’s outcomes.

Mean decrease in impurity

The mean decrease in impurity is a measure of each feature’s contribution to a decision tree’s performance. To understand this, we’ll first need to understand what “impurity” means in this context.

We’ll start with an analogy:

• Let’s say we have a fruit basket with apples, pears, and oranges. When the pieces of fruit are in the basket, they’re thoroughly mixed, and we could say this set has a high impurity.
• Now, our task is to sort them by kind. If we put all the apples into a bowl, place the oranges on a tray, and leave the pears in the basket, we would be left with three sets that have perfect purity.
• But here comes the twist: We cannot see the fruits while making our decision. For each piece of fruit, we are told its color, diameter, and weight. Then, we need to decide where it should go. Thus, these three properties are our features.
• The weight and the diameter of the pieces of fruit will be very similar. They won’t help us much in sorting – or, to say it differently, they are unhelpful in decreasing the impurity.
• But the color will be helpful. We might still struggle to distinguish between green or yellow apples and green or yellow pears, but if we learn that the color is red or orange, we can confidently make a decision. Thus, the “color” will give us the biggest mean decrease in impurity.

Now, let’s use this analogy in the context of decision trees and random forests:

When building a decision tree, we want each node to be as pure as possible regarding the target variable. In more colloquial terms, when creating a new node for our tree, we aim to find the feature that best splits the samples that reach the node into two distinct sets so that samples with the same label are in the same set. (For the full mathematical details, see the scikit-learn documentation).

Each node in a decision tree reduces the impurity – roughly speaking, it helps sort the training samples by their target label. Suppose a feature is the decision criterion in many nodes in the tree, and it’s effective in cleanly dividing the samples. In that case, it will be responsible for a large share of the reduction in impurity the decision tree achieves overall. That’s why looking at the “mean decrease in impurity” a feature is responsible for is a good measure of the importance of a feature.

Whew, that was a lot of complicated math and terminology!

Luckily, the visualizations are not quite that difficult to read. We can clearly identify our model’s primary drivers and use that information in feature selection. Reducing a model’s input space to just the most decisive features reduces its complexity and can prevent overfitting.

Additionally, understanding feature importance informs data preparation. Features with low importance might be candidates for removal or consolidation, streamlining the input data preprocessing.

There’s one important caveat, though, that I would like to mention before we move on. Since a node’s decrease in impurity is determined during training, using the training data set, the “mean decrease in impurity” doesn’t necessarily translate to previously unseen test data:

Consider the case that our training samples are numbered, and this number is an input feature for our model. Then, if our decision tree is complex enough, it can just learn which sample has which label (e.g., “fruit 1 is an orange”, “fruit 2 is an apple”, …). The mean decrease in impurity for the number feature will be massive, and it will appear as a highly important feature in our visualization, even though it’s entirely useless when applying our model to data it has not seen before.

Local interpretable model-agnostic explanations (LIME)

Local interpretability approaches aim to shed light on a model’s behavior in a specific instance. (The opposite is global interpretability, where a model’s behavior across its entire feature space is examined.) 

One of the oldest and still widely used techniques is LIME (Local Interpretable Model-agnostic Explanations). To uncover the contributions of each input feature to the model’s prediction, a linear model is fitted that approximates the model’s behavior in the particular area of the feature space. Roughly speaking, the linear model’s coefficients reflect the importance of the input features. The result can be visualized as a feature importance plot, highlighting the most influential features for a particular prediction.

Local interpretability techniques can extract intuitive insights from complex algorithms. Visualization of these results can support discussions with business stakeholders or be the foundation for cross-checking a model’s learned behavior with domain experts. They provide practical, actionable insights, enhance trust in a model’s intricate inner workings, and can be a vital tool for promoting machine learning adoption.

Related post

How to Improve ML Model Performance [Best Practices From Ex-Amazon AI Researcher]

Read more

How to adopt model visualization in machine learning?

In this section, I’ll share tips on seamlessly integrating model visualization into your daily data science and machine learning routines.

1. Start with a clear purpose

Before diving into model visualization, establish a clear purpose. Ask yourself, “What specific goals do I aim to achieve through visualization?”

Are you seeking to …

• … improve model performance?
• … enhance interpretability?
• … better communicate results to stakeholders?

Defining your objectives will provide the direction needed for effective visualization.

2. Choosing the appropriate visualization

Always have a top-to-bottom approach. This means you start at a very abstract level and then explore deeper for more insights.

For instance, if you are seeking to improve the model’s performance, then make sure that you start with simple approaches first, like plotting the model’s accuracy and loss using simple line plots.

Let’s assume that your model is overfitting. Then, you can use feature importance techniques to rank features based on their contribution to model performance. You can plot these feature importance scores to visualize the most influential features in the model. Features with high importance might point to overfitting and information leakage.

Likewise, you can create partial dependence plots for relevant features. PDPs show how the target variable’s prediction changes as a specific feature varies while keeping other features constant. You must look for erratic behavior or sharp fluctuations in the curve, which could indicate overfitting due to that feature.

3. Select the right tools

Selecting the right tools depends on the task at hand and the features the tools offer. Python offers a plethora of libraries like Matplotlib, Seaborn, and Plotly for creating static and interactive visualizations. Framework-specific tools, such as TensorBoard for TensorFlow and scikit-plot for scikit-learn, can be invaluable for model-specific visualizations.

Related post

The Best Tools for Machine Learning Model Visualization

Read more

4. Iterate and improve

Remember that model visualization is an iterative process. Continuously refine your visualizations based on feedback from your team and the stakeholders you present them to. The ultimate goal is to make your models transparent, interpretable, and accessible to all stakeholders. Their input and evolving project requirements might mean you need to reconsider and adapt your approach.

Incorporating model visualization into your daily data science or machine learning practice empowers you to make data-driven decisions with clarity and confidence. Whether you’re a data scientist, a domain expert, or a decision-maker, adopting model visualization as a routine practice is a pivotal step in harnessing the full potential of your machine-learning projects.

Conclusion

Effective machine-learning model visualization is an indispensable tool for any data scientist. It empowers practitioners to gain insights, make informed decisions, and communicate results transparently.

In this article, we covered a lot of information about how we visualize machine learning models. To conclude, here are some key takeaways:

Purpose of visualization in machine learning:

• Visualizations simplify complex ML model structures and data patterns for better understanding.
• Interactive visualizations and Visual ML tools empower users to dynamically interact with data and models. They can tweak parameters, zoom in on details, and better understand the ML system.
• Visualizations foster informed decision-making and effective communication of results.

Types of machine learning visualizations:

• Model structure visualizations help data scientists, AI researchers, and business stakeholders understand complex algorithms and data flows.
• Model performance visualizations provide insight into the performance characteristics of individual models and model ensembles.
• Visualizations for comparative model analysis aid practitioners in selecting the best-performing model or verifying that a new model version is an improvement.
• Feature importance visualizations uncover each input feature’s influence on a model’s output.

Best practices for adopting model visualization:

• Start with defined objectives and simple visualizations.
• Choose an appropriate visualization method that suits your needs and is accessible to the intended audience. 
• Select the proper tools and libraries that help you craft accurate visualizations efficiently.
• Continuously listen to feedback and adapt your visualizations to your stakeholders’ needs.

Deep learning is one of the main technologies that enabled self-driving. It’s a versatile tool that can solve almost any problem – it can be used in physics, for example, the proton-proton collision in the Large Hadron Collider, just as well as in Google Lens to classify pictures. Deep learning is a technology that can help solve almost any type of science or engineering problem. 

In this article, we’ll focus on deep learning algorithms in self-driving cars – convolutional neural networks (CNN). CNN is the primary algorithm that these systems use to recognize and classify different parts of the road, and to make appropriate decisions.  

Along the way, we’ll see how Tesla, Waymo, and Nvidia use CNN algorithms to make their cars driverless or autonomous. 

How do self-driving cars work?

The first self-driving car was invented in 1989, it was the Automatic Land Vehicle in Neural Network (ALVINN). It used neural networks to detect lines, segment the environment, navigate itself, and drive. It worked well, but it was limited by slow processing powers and insufficient data.

With today’s high-performance graphics cards, processors, and huge amounts of data, self-driving is more powerful than ever. If it becomes mainstream, it will reduce traffic congestion and increase road safety. 

Self-driving cars are autonomous decision-making systems. They can process streams of data from different sensors such as cameras, LiDAR, RADAR, GPS, or inertia sensors. This data is then modeled using deep learning algorithms, which then make decisions relevant to the environment the car is in. 

The image above shows a modular perception-planning-action pipeline used to make driving decisions. The key components of this method are the different sensors that fetch data from the environment. 

To understand the workings of self-driving cars, we need to examine the four main parts:

1. Perception 
2. Localization
3. Prediction
4. Decision Making
   1. High-level path planning 
   2. Behaviour Arbitration
   3. Motion Controllers

Perception 

One of the most important properties that self-driving cars must have is perception, which helps the car see the world around itself, as well as recognize and classify the things that it sees. In order to make good decisions, the car needs to recognize objects instantly.

So, the car needs to see and classify traffic lights, pedestrians, road signs, walkways, parking spots, lanes, and much more. Not only that, it also needs to know the exact distance between itself and the objects around it. Perception is more than just seeing and classifying, it enables the system to evaluate the distance and decide to either slow down or brake. 

To achieve such a high level of perception, a self-driving car must have three sensors:

1. Camera
2. LiDAR
3. RADAR

Camera

The camera provides vision to the car, enabling multiple tasks like classification, segmentation, and localization. The cameras need to be high-resolution and represent the environment accurately.

In order to make sure that the car receives visual information from every side: front, back, left, and right, the cameras are stitched together to get a 360-degree view of the entire environment. These cameras provide a wide-range view as far as 200 meters as well as a short-range view for more focused perception. 

In some tasks like parking, the camera also provides a panoramic view for better decision-making. 

Even though the cameras do all the perception related tasks, it’s hardly of any use during extreme conditions like fog, heavy rain, and especially at night time. During extreme conditions, all that cameras capture is noise and discrepancies, which can be life-threatening. 

To overcome these limitations, we need sensors that can work without light and also measure distance.

LiDAR

LiDAR stands for Light Detection And Ranging, it’s a method to measure the distance of objects by firing a laser beam and then measuring how long it takes for it to be reflected by something.

A camera can only provide the car with images of what’s going around itself. When it’s combined with the LiDAR sensor, it gains depth in the images – it suddenly has a 3D perception of what’s going on around the car. 

So, LiDAR perceives spatial information. And when this data is fed into deep neural networks, the car can predict the actions of the objects or vehicles close to it. This sort of technology is very useful in a complex driving scenario, like a multi-exit intersection, where the car can analyze all other cars and make the appropriate, safest decision.

In 2019, Elon Musk openly stated that “anyone relying on LiDARs are doomed…”. Why? Well, LiDARs have limitations that can be catastrophic. For example, the LiDAR sensor uses lasers or light to measure the distance of the nearby object. It will work at night and in dark environments, but it can still fail when there’s noise from rain or fog. That’s why we also need a RADAR sensor.

RADARs

Radio detection and ranging (RADAR) is a key component in many military and consumer applications. It was first used by the military to detect objects. It calculates distance using radio wave signals. Today, it’s used in many vehicles and has become a primary component of the self-driving car. 

RADARs are highly effective because they use radio waves instead of lasers, so they work in any conditions. 

It’s important to understand that radars are noisy sensors. This means that even if the camera sees no obstacle, the radar will detect some obstacles. 

The image above shows the self-driving car (in green) using LiDAR to detect objects around and to calculate the distance and shape of the object. Compare the same scene, but captured with the RADAR sensor below, and you can see a lot of unnecessary noise.

The RADAR data should be cleaned in order to make good decisions and predictions. We need to separate weak signals from strong ones; this is called thresholding. We also use Fast Fourier Transforms (FFT) to filter and interpret the signal. 

If you look at the below above, you’ll notice that the RADAR and LiDAR signals are point-based data. This data should be clustered so that it can be interpreted nicely. Clustering algorithms such as Euclidean Clustering or K means Clustering are used to achieve this task. 

Localization

Localization algorithms in self-driving cars calculate the position and orientation of the vehicle as it navigates – a science known as Visual Odometry (VO).

VO works by matching key points in consecutive video frames. With each frame, the key points are used as the input to a mapping algorithm. The mapping algorithm, such as Simultaneous localization and mapping (SLAM), computes the position and orientation of each object nearby with respect to the previous frame and helps to classify roads, pedestrians, and other objects around. 

Deep learning is generally used to improve the performance of VO, and to classify different objects. Neural networks, such as PoseNet and VLocNet++, are some of the frameworks that use point data to estimate the 3D position and orientation. These estimated 3D positions and orientations can be used to derive scene semantics, as seen in the image below. 

Prediction

Understanding human drivers is a very complex task. It involves emotions rather than logic, and these are all fueled with reactions. It becomes very uncertain what the next action will be of the drivers or pedestrians nearby, so a system that can predict the actions of other road users can be very important for road safety. 

The car has a 360-degree view of its environment that enables it to perceive and capture all the information and process it. Once fed into the deep learning algorithm, it can come up with all the possible moves that other road users might make. It’s like a game where the player has a finite number of moves and tries to find the best move to defeat the opponent. 

The sensors in self-driving cars enable them to perform tasks like image classification, object detection, segmentation, and localization. With various forms of data representation, the car can make predictions of the object around it.

A deep learning algorithm can model such information (images and cloud data points from LiDARs and RADARs) during training. The same model, but during inference, can help the car to prepare for all the possible moves which involve braking, halting, slowing down, changing lanes, and so on. 

The role of deep learning is to interpret complex vision tasks, localize itself in the environment, enhance perception, and actuate kinematic maneuvers in self-driving cars. This ensures road safety and easy commute as well.

But the tricky part is to choose the correct action out of a finite number of actions. 

Decision-making

Decision-making is vital in self-driving cars. They need a system that’s dynamic and precise in an uncertain environment. It needs to take into account that not all sensor readings will be true, and that humans can make unpredictable choices while driving. These things can’t be measured directly. Even if we could measure them, we can’t predict them with good accuracy. 

The image above shows a self-driving car moving towards an intersection. Another car, in blue, is also moving towards the intersection. In this scenario, the self-driving car has to predict whether the other car will go straight, left, or right. In each case, the car has to decide what maneuver it should perform to prevent a collision.

In order to make a decision, the car should have enough information so that it can select the necessary set of actions. We learned that the sensors help the car to collect information and deep learning algorithms can be used for localization and prediction. 

To recap, localization enables the car to know its initial position, and prediction creates an n number of possible actions or moves based on the environment. The question remains: which option is best out of the many predicted actions? 

When it comes to making decisions, we use deep reinforcement learning (DRL). More specifically, a decision-making algorithm called the Markov decision process (MDP) lies at the heart of DRL (we’ll learn more about MDP in a later section where we talk about reinforcement learning). 

Usually, an MDP is used to predict the future behavior of the road-users. We should keep in mind that the scenario can get very complex if the number of objects, especially moving ones, increases. This eventually increases the number of possible moves for the self-driving car itself. 

In order to tackle the problem of finding the best move for itself, the deep learning model is optimized with Bayesian optimization. There are also situations where the framework, consisting of both a hidden Markov model and Bayesian Optimization, is used for decision-making. 

In general, decision-making in self-driving cars is a hierarchical process. This process has four components:

• Path or Route planning: Essentially, route planning is the first of four decisions that the car must make. Entering the environment, the car should plan the best possible route from its current position to the requested destination. The idea is to find an optimal solution among all the other solutions.  
• Behaviour Arbitration: Once the route is planned, the car needs to navigate itself through the route. The car knows about the static elements, like roads, intersections, average road congestion and more, but it can’t know exactly what the other road users are going to be doing throughout the journey. This uncertainty in the behavior of other road users is solved by using probabilistic planning algorithms like MDPs.
• Motion Planning: Once the behavior layer decides how to navigate through a certain route, the motion planning system orchestrates the motion of the car. The motion of the car must be feasible and comfortable for the passenger. Motion planning includes speed of the vehicle, lane-changing, and more, all of which should be relevant to the environment the car is in.  
• Vehicle Control: Vehicle control is used to execute the reference path from the motion planning system. 

CNNs used for self-driving cars

Convolutional neural networks (CNN) are used to model spatial information, such as images. CNNs are very good at extracting features from images, and they’re often seen as universal non-linear function approximators. 

CNNs can capture different patterns as the depth of the network increases. For example, the layers at the beginning of the network will capture edges, while the deep layers will capture more complex features like the shape of the objects (leaves in trees, or tires on a vehicle). This is the reason why CNNs are the main algorithm in self-driving cars. 

The key component of the CNN is the convolutional layer itself. It has a convolutional kernel which is often called the filter matrix. The filter matrix is convolved with a local region of the input image which can be defined as:

Where: 

• the operator * represents the convolution operation,
• w is the filter matrix and b is the bias, 
• x is the input,
• y is the output. 

The dimension of the filter matrix in practice is usually 3X3 or 5X5. During the training process, the filter matrix will constantly update itself to get a reasonable weight. One of the properties of CNN is that the weights are shareable. The same weight parameters can be used to represent two different transformations in the network. The shared parameter saves a lot of processing space; they can produce more diverse feature representations learned by the network.

The output of the CNN is usually fed to a nonlinear activation function. The activation function enables the network to solve the linear inseparable problems, and these functions can represent high-dimensional manifolds in lower-dimensional manifolds. Commonly used activation functions are Sigmoid, Tanh, and ReLU, which are listed as follows:

It’s worth mentioning that the ReLU is the preferred activation function, because it converges faster compared to the other activation functions. In addition to that, the output of the convolution layer is modified by the max-pooling layer which keeps more information about the input image, like the background and texture. 

The three important CNN properties that make them versatile and a primary component of self-driving cars are:

• local receptive fields, 
• shared weights, 
• spatial sampling. 

These properties reduce overfitting and store representations and features that are vital for image classification, segmentation, localization, and more.

Next, we’ll discuss three CNN networks that are used by three companies pioneering self-driving cars:

1. HydraNet by Tesla
2. ChauffeurNet by Google Waymo
3. Nvidia Self driving car

HydraNet – semantic segmentation for self-driving cars 

HydraNet was introduced by Ravi et al. in 2018. It was developed for semantic segmentation, for improving computational efficiency during inference time.  

HydraNets is dynamic architecture so it can have different CNN networks, each assigned to different tasks. These blocks or networks are called branches. The idea of HydraNet is to get various inputs and feed them into a task-specific CNN network. 

Take the context of self-driving cars. One input dataset can be of static environments like trees and road-railing, another can be of the road and the lanes, another of traffic lights and road, and so on. These inputs are trained in different branches. During the inference time, the gate chooses which branches to run, and the combiner aggregates branch outputs and makes a final decision. 

In the case of Tesla, they have modified this network slightly because it’s difficult to segregate data for the individual tasks during inference. To overcome that problem, engineers at Tesla developed a shared backbone. The shared backbones are usually modified ResNet-50 blocks.

This HydraNet is trained on all the object’s data. There are task-specific heads that allow the model to predict task-specific outputs. The heads are based on semantic segmentation architecture like the U-Net. 

The Tesla HydraNet can also project a birds-eye, meaning it can create a 3D view of the environment from any angle, giving the car much more dimensionality to navigate properly. It’s important to know that Tesla doesn’t use LiDAR sensors. It has only two sensors, a camera and a radar. Although LiDAR explicitly creates depth perception for the car, Tesla’s hydranet is so efficient that it can stitch all the visual information from the 8 cameras in it and create depth perception. 

ChauffeurNet: training self-driving car using imitation learning

ChauffeurNet is an RNN-based neural network used by Google Waymo, however, CNN is actually one of the core components here and it’s used to extract features from the perception system. 

The CNN in ChauffeurNet is described as a convolutional feature network, or FeatureNet, that extracts contextual feature representation shared by the other networks. These representations are then fed to a recurrent agent network (AgentRNN) that iteratively yields the prediction of successive points in the driving trajectory.

The idea behind this network is to train a self-driving car using imitation learning. In the paper released by Bansal et al “ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst”, they argue that training a self-driving car even with 30 million examples is not enough. In order to tackle that limitation, the authors trained the car in synthetic data. This synthetic data introduced deviations such as introducing perturbation to the trajectory path, adding obstacles, introducing unnatural scenes, etc. They found that such synthetic data was able to train the car much more efficiently than the normal data. 

Usually, self-driving has an end-to-end process as we saw earlier, where the perception system is part of a deep learning algorithm along with planning and controlling. In the case of ChauffeurNet, the perception system is not a part of the end-to-end process; instead, it’s a mid-level system where the network can have different variations of input from the perception system. 

ChauffeurNet yields a driving trajectory by observing a mid-level representation of the scene from the sensors, using the input along with synthetic data to imitate an expert driver.  

In the image above, the cyan path depicts the input route, green box is the self-driving car, blue dots are the agent’s past route or position, and green dots are the predicted future routes or positions. 

Essentially, a mid-level representation doesn’t directly use raw sensor data as input, factoring out the perception task, so we can combine real and simulated data for easier transfer learning. This way, the network can create a high-level bird’s eye view of the environment which ultimately yields better decisions. 

Nvidia self-driving car: a minimalist approach towards self-driving cars

Nvidia also uses a Convolution Neural Network as a primary algorithm for its self-driving car. But unlike Tesla, it uses 3 cameras, one on each side and one at the front.  See the image below. 

The network is capable of operating inroads that don’t have lane markings, including parking lots. It can also learn features and representations that are necessary for detecting useful road features. 

Compared to the explicit decomposition of the problem such as lane marking detection, path planning, and control, this end-to-end system optimizes all processing steps at the same time. 

Better performance is the result of internal components self-optimizing to maximize overall system performance, instead of optimizing human-selected intermediate criteria like lane detection. Such criteria understandably are selected for ease of human interpretation, which doesn’t automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with a minimal number of processing steps.

Reinforcement learning used for self-driving cars

Reinforcement learning (RL) is a type of machine learning where an agent learns by exploring and interacting with the environment. In this case, the self-driving car is an agent. 

We discussed earlier how the neural network predicts a number of actions from the perception data. But, choosing an appropriate action requires deep reinforcement learning (DRL). At the core of DRL, we have three important variables:

1. State describes the current situation in a given time. In this case, it would be a position on the road. 
2. Action describes all the possible moves that the car can make. 
3. Reward is feedback that the car receives whenever it takes a certain action. 

Generally, the agent is not told what to do or what actions to take. So far as we have seen, in supervised learning, the algorithm maps input to the output. In DRL, the algorithm learns by exploring the environment and each interaction yields a certain reward. The reward can be both positive and negative. The goal of the DRL is to maximize the cumulative rewards. 

In self-driving cars, the same procedure is followed: the network is trained on perception data, where it learns what decision it should make. Because the CNNs are very good at extracting features of representations from the input, DRL algorithms can be trained on those representations. Training a DRL algorithm on these representations can yield good results because these extracted representations are the transformation of higher-dimensional manifolds into simpler lower-dimensional manifolds. Training on lower representation yields efficiency which is required at the inference. 

One key point to remember is that self-driving cars can’t be trained in real-world scenarios or roads because they will be extremely dangerous. Instead, self-driving cars are trained on a simulator where there’s no risk at all. 

Some open-source simulators are:

1. CARLA
2. SUMMIT​​
3. AirSim
4. DeepDrive
5. Flow

These cars (agents) are trained for thousands of epochs with highly difficult simulations before they’re deployed in the real world. 

During training, the agent (the car) learns by taking a certain action in a certain state. Based on this state-action pair, it receives a reward. This process happens over and over again. Each time the agent updates its memory of rewards. This is called the policy. 

The policy is described as how the agent makes decisions. It’s a decision-making rule. The policy defines the behaviour of the agent at a given time. 

For every negative decision the agent makes, the policy is changed. So in order to avoid the negative rewards, the agent checks the quality of a certain action. This is measured by the state-value function. State-value can be measured using the Bellman Expectation Equation.

The Bellman expectation equation, along with Markov Decision Process (MDP), makes up the two core concepts of DRL. But when it comes to self-driving cars, we have to keep in mind that the observations from the perception data should be mapped with the appropriate action and not just map the underlying state to the action. This is where a partially observed decision process or a Partially Observable Markov Decision Process (POMDP) is required, which can make decisions based on the observation. 

Partially Observable Markov Decision Process used for self-driving cars

The Markov Decision Process gives us a way to sequentialize decision-making. When the agent interacts with the environment, it does so sequentially over time. Each time the agent interacts with the environment, it gives some representation of the environment state. Given the representation of the state, the agent selects the action to take, as in the image below. 

The action taken is transitioned into some new state and the agent is given a reward. This process of evaluating a state, taking action, changing states, and rewarding is repeated. Throughout the process, it’s the agent’s goal to maximize the total amount of rewards. 

Let’s get a more constructive idea of the whole process:

1. At a give time t, the state of the environment is at St
2. The agent observes the current state St and selects an action At
3. The environment is then transitioned into a new state St+1, simultaneously the agent is rewarded Rt

In a partially observable Markov decision process (POMDP), the agent senses the environment state with observations received from the perception data and takes a certain action followed by receiving a reward. 

The POMDP has six components and it can be denoted as POMDP M:= (I, S, A, R, P, γ), where, 

• I: Observations 
• S: Finite set of states
• A: Finite set of actions
• R: Reward function
• P: transition probability function
• γ – discounting factor for future rewards. 

The objective of DRL is to find the desired policy that maximizes the reward at each given time step or, in other words, to find an optimal value-action function (Q-function).  

Q-learning used for self-driving cars

Q-learning is one of the most commonly used DRL algorithms for self-driving cars. It comes under the category of model-free learning. In model-free learning, the agent will try to approximate the optimal state-action pair. The policy still determines which action-value pairs or Q-value are visited and updated (see the equation below). The goal is to find optimal policy by interacting with the environment while modifying the same when the agent makes an error. 

With enough samples or observation data, Q-learning will learn optimal state-action value pairs. In practice, Q-learning has been shown to converge to the optimum state-action values for a MDP with probability 1, provided that all actions in all states are infinitely available. 

Q-learning can be described in the following equation: 

where:

α ∈ [0,1] is the learning rate. It controls the degree to which Q values are updated at a given t.

It’s important to remember that the agent will discover the good and bad actions through trial and error.

Conclusion

Self-driving cars aim to revolutionize car travel by making it safe and efficient. In this article, we outlined some of the key components such as LiDAR, RADAR, cameras, and most importantly – the algorithms that make self-driving cars possible. 

While it’s promising, there’s still a lot of room for improvement. For example, current self-driving cars are at level-2 out of level-5 of advancement, which means that there still has to be a human ready to intervene if necessary. 

Few things need to be taken care of:

1. The algorithms used are not yet optimal enough to perceive roads and lanes because some roads lack markings and other signs.
2. The optimal sensing modality for localization, mapping, and perception still lack accuracy and efficiency.
3. Vehicle-to-vehicle communication is still a dream, but work is being done in this area as well.  
4. The field of human-machine interaction is not explored enough, with many open, unsolved problems.

Still, the technology we’ve developed so far is amazing. And with orchestrated efforts, we can ensure that self-driving systems will be safe, robust, and revolutionary.

Further reading:

1. A Survey of Deep Learning Techniques for Autonomous Driving
2. A Survey of Autonomous Driving: Common Practices and Emerging Technologies
3. Decision Making for Autonomous Driving considering Interaction and Uncertain Prediction of Surrounding Vehicles
4. Autonomous car using CNN deep learning algorithm
5. Deep Reinforcement Learning for Autonomous Driving: A Survey
6. The WIRED Guide to Self-Driving Cars
7. Deep Learning for Self-Driving Cars
8. Training Self Driving Cars using Reinforcement Learning
9. A (Long) Peek into Reinforcement Learning
10. Tesla Statistics: What You Should Know About Safety, Pricing and More