The Open Neural Network Exchange (ONNX) format, released in 2017, is a portable file format for describing machine learning models. ONNX models are self-contained files (.onnx) that can be easily exported from all the major training frameworks and are supported by many hardware architectures and operating systems enabling improved efficiency at inference time. The ONNX Runtime in particular, developed in the open by Microsoft, is cross-platform and high performance with a simple API enabling you to run inference on any ONNX model exactly where you need it: VM in cloud, VM on-prem, phone, tablet, IoT device, you name it!

I thought it would be interesting to see how this might be applied to an image classification scenario in a serverless envionment. In this post we’ll go through how to run inference on an ONNX model using the .NET ONNX Runtime API in Azure Functions.

Prerequisites

Before we get started, we need to setup a few things first:

1. If you’re on Windows, you’ll need Node.js. Install the LTS release. Not required for macOS and Linux.
2. Visual Studio Code on one of the supported platforms.
3. .NET Core 3.1 for your OS (Mac, Windows or Linux).
4. The C# extension for Visual Studio Code.
5. The Azure Functions extension for Visual Studio Code.
6. An Azure account

Create functions project

Firstly, we need to create the Azure Functions .NET project:

1. Open a workspace in VS Code
2. Bring up the command palette (CTRL+SHIFT+P, CMD+SHIFT+P) and click Azure Functions: Create New Project
3. Select your workspace
4. Select C# and language
5. Select HTTPTrigger as binding
6. Enter Predict as the function name
7. Enter DotNetOnnxFunctions as the namespace
8. Clear out all the templated code in the Predict function
9. Remove the get trigger,
10. Inject ExecutionContext into Run()

We should have an function that looks like this:

[FunctionName("Predict")]
public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = null)] HttpRequest req,
    ExecutionContext context,
    ILogger log)
{
    // TODO: do stuff

    return new OkObjectResult("");
}

Great, we have our project setup. Now we can start adding the model to our project.

Get an ONNX model

We need a model in order to actually do predictions. Fortunately, the ONNX team provide a model zoo of models for various tasks. Let’s try the efficientnet-lite4.onnx model as this looks to be state-of-the-art. To enable our function to access the model, we need to:

1. Download the model and put it into our workspace
2. Add the following XML into an ItemGroup element in project file so the model is copied to the output directory:

<Project Sdk="Microsoft.NET.Sdk">
  ...
  <ItemGroup>
  ...
    <None Update="efficientnet-lite4.onnx">
      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
    </None>    
  </ItemGroup>
</Project>

Great, our function can now access our model. Now we can start adding the code to read an image.

Read the image

We need to read the image from the HTTP body, then provide a mechanism to read the pixel values. Let’s firstly install ImageSharp which is the go-to image library for cross-platform .NET development.

dotnet add package SixLabors.ImageSharp

Then we can add the following code to read the image from the body and into an image instance:

using var imageInStream = new MemoryStream();
req.Body.CopyTo(imageInStream);
using var image = Image.Load<Rgb24>(imageInStream.ToArray(), out IImageFormat format);

Here, we’re reading the body into a memory stream, then extracting the underlying bytes and creating an Image instance of type RGB24. Note that these objects require us to dispose of them, hence the using statements.

Now we’ve read the image, we need to resize it to the size the model is expecting.

Resize image

Like most image classification models, this model expects the image to be 224 pixels x 224 pixels. Let’s add the following code to centre crop and resize the image:

using var imageOutStream = new MemoryStream();
image.Mutate(i =>
{
    i.Resize(new ResizeOptions()
    {
        Size = new Size(224, 224),
        Mode = ResizeMode.Crop
    });
});
image.Save(imageOutStream, format);

Here, we’re creating a new memory stream for the resized imaged. We’re using the ResizeOptions to resize and centre crop the image. Then, we’re saving the result to the stream.

Now the image is resized, we can go ahead and preprocess it into a form that the model is expecting.

Preprocess image

The model we have chosen has been trained with TensorFlow and as such has specific preprocessing requirements. We need to ensure the input pixel values are normalised between [-1, 1], and that the input tensor has the shape (batch-size, height, width, channels). Models will have various preprocessing requirements depending on which framework they were trained in. Preprocessing has to be done correctly otherwise the model will produce rubbish (SISO).

We’ll need to install the ONNX Runtime package:

dotnet add package Microsoft.ML.OnnxRuntime

We can then add the following code to preprocess the image into the correct format:

var input = new DenseTensor<float>(new[] { 1, image.Height, image.Width, 3 });
for (int y = 0; y < image.Height; y++)
{
    var pixelRow = image.GetPixelRowSpan(y);
    for (int x = 0; x < image.Width; x++)
    {
        var pixel = pixelRow[x];
        input[0, y, x, 0] = (pixel.R - 127) / 128f;
        input[0, y, x, 1] = (pixel.G - 127) / 128f;
        input[0, y, x, 2] = (pixel.B - 127) / 128f;
    }
}

Here, we’re creating a .NET Tensor of the required shape (1, 224, 224, 3) where batch_size is 1 because we’re only allowing one image and channels is 3 for each colour channel (R=red, B=blue, G=green). Then, we’re looping through each pixel row of the image, extracting the RGB values, normalising them between [-1,1] and assigning them to the appropriate index in the tensor.

Now the image is preprocessed, we’re finally ready to run inference!

Run inference

Running inference is very simple with the Onnx Runtime API. It can be achieved with a few lines of code:

var inputs = new List<NamedOnnxValue>()
{
    NamedOnnxValue.CreateFromTensor("images:0", input)
};
var outputs = new List<string> { "Softmax:0" };
using var session = new InferenceSession(Path.Combine(context.FunctionAppDirectory, "efficientnet-lite4.onnx"));
using var results = session.Run(inputs, outputs);

Here, we’re creating a list of inputs that contains just one input for our tensor, targeting the node image:0 of the model. Then we’re creating an output list to tell Onnx Runtime where to read the output from; Softmax:0. Then, we’re setting up an inference session and running it by providing the inputs and outputs.

If you’re not sure what input or output you should be using, you can use the open source program called Netron to visualise and inspect the model:

Formatting the prediction

To display the results of the prediction, we can add the following code:

var prediction = (results.First().Value as IEnumerable<float>)
    .Select((x, i) => new { Label = LabelMap.Labels[i], Confidence = x })
    .OrderByDescending(x => x.Confidence)
    .Take(10)
    .ToArray();

return new OkObjectResult(prediction);

Here, we’re getting the first prediction (as there’s only one image) and returning the top 10 classes with labels and confidence scores.

Putting it all together

Run locally

To run this locally, simple hit F5, wait for the function app to start, then hit the endpoint with an image. For example, with Postman, using the following image:

I get the following response:

Great! Golden retriever was predicted with the highest confidence of 99.92%.

Run in Azure

To test this out in Azure, you can right click in your workspace and click “Deploy to Function App”. Then follow the prompts to deploy the app.

After deploying this (it took a little while as the model is quite big), I tested against the same image:

Summary

In this post, we developed an Azure Function to enable running image classification using a state-of-the-art ONNX model and the ONNX Runtime. We did this by reading the image, resizing, preprocessing, running inference and formatting the prediction results. I hope this post showed the ease and utility of ONNX and ONNX Runtime and how you can easily add AI into current or new applications. Will you think about exporting your models to ONNX for your next deployment?

Resources

• GitHub repo for this post
• ONNX
• ONNX Models
• ONNX Runtime
• Azure Functions

• Artificial Intelligence
• Machine Learning
• .NET
• Azure