Use AI on Tizen

Briefing

This guide is for Tizen developers who want to use AI in their application. First, we introduce ai-edge-torch, a python package tool converting PyTorch model to TensorFlow Lite (TFLite) model. Then, we will introduce how to use TFLite model in Tizen using Machine Learning Service APIs.

PyTorch is great for training and developing AI models. However, it's not so easy to deploy and use it in on-device. TFLite has great on-device Tizen performance and it's easy to use GPU, DSP, or NPU by its delegate feature. Thus we recommend using TFLite in Tizen, at least for PoC stage (after engineering you may find better solution).

Prepare your model

First, you should obtain your model. There are several ways:

Check NNStreamer/ML-API Tizen Examples
- There are many real examples you may find it useful.
Find web. There are many open models in the web.
- Official TFLite examples
- ultralytics offers many YOLO based vision models.
- huggingface or torchvision models
Make your own model (PyTorch)

In many cases, you need to use your own model. This model might be written from scratch or trained with your custom dataset. If your model is made with PyTorch, you can use ai-edge-torch to convert your model into TFLite format, so it can be used in Tizen.

Make PyTorch model

For brevity, this article use a pretrained MobileNet V3 small model from torchvision.

import torch
import torchvision

# Here we use mobilenet_v3_small with pre-trained weights.
torch_model = torchvision.models.mobilenet_v3_small(torchvision.models.MobileNet_V3_Small_Weights.IMAGENET1K_V1).eval()

# Finetune with your own data if you want.

sample_input = torch.randn(1, 3, 224, 224) # All pytorch CONV layer has NCHW format
sample_input.shape # torch.Size([1, 3, 224, 224])

sample_output = torch_model(sample_input)
saample_output.shape # torch.Size([1, 1000])

Obviously, you could/should finetune the model with your own data!

Convert PyTorch into TFLite with ai-edge-torch

ai-edge-torch is Google's open source project to convert PyTorch model into TFLite model. Check official guide for more details.

Install ai-edge-torch

pip install -r https://raw.githubusercontent.com/google-ai-edge/ai-edge-torch/main/requirements.txt
pip install ai-edge-torch-nightly

Export PyTorch model to TFLite model file.

import ai_edge_torch

# PyTorch uses NCHW format only, and TFLite uses NHWC format for default.
# We can convert the model to use NHWC format.
# This is optional and can be skipped if you don't need it (performance should be tested for both cases).
nhwc_torch_model = ai_edge_torch.to_channel_last_io(torch_model, args=[0]) # convert the 0-th input tensor to channel_last (NHWC) format
sample_nhwc_input = sample_input.permute(0, 2, 3, 1) # NCHW [1, 3, 224, 224] -> NHWC [1, 224, 224, 3]
edge_model = ai_edge_torch.convert(nhwc_torch_model, (sample_nhwc_input,)) # convert the model
edge_model_output = edge_model(sample_nhwc_input) # run the model
edge_model_output.shape # (1, 1000)
edge_model.export("mobilenet_v3_small.tflite") # save as tflite model file

Validate the TFLite model.

import numpy as np
import tensorflow as tf

# Load the TFLite model and allocate tensors.
interpreter = tf.lite.Interpreter(model_path="mobilenet_v3_small.tflite")
interpreter.allocate_tensors()

# Get input and output tensors.
input_details = interpreter.get_input_details()
#  'name': 'serving_default_args_0:0',
#  'index': 0,
#  'shape': array([  1, 224, 224,   3], dtype=int32),
#  'shape_signature': array([  1, 224, 224,   3], dtype=int32),
#  'dtype': numpy.float32,
#  'quantization': (0.0, 0),
#  'quantization_parameters': {'scales': array([], dtype=float32),
#    'zero_points': array([], dtype=int32),
#    'quantized_dimension': 0},
#   'sparsity_parameters': {}}


output_details = interpreter.get_output_details()
#  'name': 'StatefulPartitionedCall:0',
#  'index': 254,
#  'shape': array([   1, 1000], dtype=int32),
#  'shape_signature': array([   1, 1000], dtype=int32),
#  'dtype': numpy.float32,
#  'quantization': (0.0, 0),
#  'quantization_parameters': {'scales': array([], dtype=float32),
#    'zero_points': array([], dtype=int32),
#    'quantized_dimension': 0},
#  'sparsity_parameters': {}

# Test the model on random input data.
input_shape = input_details[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])
print(output_data)

Now you have tflite model file mobilenet_v3_small.tflite. You can use it in your Tizen app with ML Service APIs.

Use ML Service API on Tizen

Tizen ML API provides a set of APIs for AI applications to use its model in a convenient conf-file-based passion.

The conf file

A conf file mobilenet.conf should be something like this:

{
  "single" : {
    "model" : "mobilenet_v3_small.tflite",
    "framework" : "tensorflow-lite",
    "custom" : "Delegate:XNNPACK,NumThreads:2",
    "input_info" : [
      {
        "type" : "float32",
        "dimension" : "3:224:224:1"
      }
    ],
    "output_info" : [
      {
        "type" : "float32",
        "dimension" : "1000:1"
      }
    ]
  },
  "information" :
  {
    "label_file" : "mobilenet_label.txt"
  }
}

"model" : "mobilenet_v3_small.tflite" means this conf file uses the specified model.
"framework" : "tensorflow-lite" means this model should be invoked by TFLite.
"custom" : "Delegate:XNNPACK,NumThreads:2" sets custom options to the model.
"input_info" and "output_info" are used to specify input and output tensor information. This can be omitted.
"information" defines other information using this ml-service. Here label file path is specified.

Usage of ML Service API

A ml_service handle is created by API ml_service_new

ml_service_h mls;
char *conf_file_path = "path/to/mobilenet.conf";
ml_service_new (conf_file_path, &mls);

Check input and output of mls

ml_tensors_info_h input_info;
ml_service_get_input_information (mls, NULL, &input_info);
/*
- tensor count: 1
- tensor[0]
  - name: serving_default_args_0:0
  - type: 7
  - dimension[0]: 3
  - dimension[1]: 224
  - dimension[2]: 224
  - dimension[3]: 1
  - size: 150528 byte
*/

ml_tensors_info_h output_info;
ml_service_get_output_information (mls, NULL, &output_info);
/*
- tensor count: 1
- tensor[0]
  - name: StatefulPartitionedCall:0
  - type: 7
  - dimension[0]: 1000
  - dimension[1]: 1
  - dimension[2]: 0
  - dimension[3]: 0
  - size: 4000 byte
*/

Set callback. ML Service API provides async method for getting result value of mls.

void
_new_data_cb (ml_service_event_e event, ml_information_h event_data, void *user_data)
{
  ml_tensors_data_h new_data = NULL;

  if (event != ML_SERVICE_EVENT_NEW_DATA)
    return;

  // get tensors-data from event data.
  ml_information_get (event_data, "data", &new_data);

  // get the float result
  float *result;
  size_t result_size;
  ml_tensors_data_get_tensor_data (new_data, 0U, (void *) &result, &result_size);

  // result : float[1000]
  // result_size : 4000 byte
  // do something useful with the result.
}
...

// set event callback
ml_service_set_event_cb (mls, _new_data_cb, user_data);

All setup is done. Let's invoke mls with proper input data with API ml_service_request

ml_tensors_data_h input_data = NULL;
ml_tensors_data_create (input_info, &input_data); /* create input data layout of input_info */

// set data_buf as you want.
// It should be float value image data float[224][224][3][1].
uint8_t *data_buf = NULL;
size_t data_buf_size;

// set 0-th tensor data with user given buffer
ml_tensors_data_set_tensor_data (input_data, 0U, data_buf, data_buf_size);

// now request mls to invoke the model with given input_data
ml_service_request (mls, NULL, input_data);

// When mls get the result of model inference, `_new_data_cb` will be called.

Package your `mls` into RPK

Tizen Resource Package (RPK) is a package type dedicated to resources. Tizen ML Service API utilizes RPK to let app developers easily decouple their ML from their application, and upgrade their model without re-packaging/deploying their application. Please check this real example for details.

The results of the search are