.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "how_to/deploy_models/deploy_object_detection_pytorch.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        Click :ref:`here <sphx_glr_download_how_to_deploy_models_deploy_object_detection_pytorch.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_how_to_deploy_models_deploy_object_detection_pytorch.py:


Compile PyTorch Object Detection Models
=======================================
This article is an introductory tutorial to deploy PyTorch object
detection models with Relay VM.

For us to begin with, PyTorch should be installed.
TorchVision is also required since we will be using it as our model zoo.

A quick solution is to install via pip

.. code-block:: bash

    pip install torch==1.7.0
    pip install torchvision==0.8.1

or please refer to official site
https://pytorch.org/get-started/locally/

PyTorch versions should be backwards compatible but should be used
with the proper TorchVision version.

Currently, TVM supports PyTorch 1.7 and 1.4. Other versions may
be unstable.

.. GENERATED FROM PYTHON SOURCE LINES 42-56

.. code-block:: default


    import tvm
    from tvm import relay
    from tvm import relay
    from tvm.runtime.vm import VirtualMachine
    from tvm.contrib.download import download_testdata

    import numpy as np
    import cv2

    # PyTorch imports
    import torch
    import torchvision








.. GENERATED FROM PYTHON SOURCE LINES 57-59

Load pre-trained maskrcnn from torchvision and do tracing
---------------------------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 59-96

.. code-block:: default

    in_size = 300

    input_shape = (1, 3, in_size, in_size)


    def do_trace(model, inp):
        model_trace = torch.jit.trace(model, inp)
        model_trace.eval()
        return model_trace


    def dict_to_tuple(out_dict):
        if "masks" in out_dict.keys():
            return out_dict["boxes"], out_dict["scores"], out_dict["labels"], out_dict["masks"]
        return out_dict["boxes"], out_dict["scores"], out_dict["labels"]


    class TraceWrapper(torch.nn.Module):
        def __init__(self, model):
            super().__init__()
            self.model = model

        def forward(self, inp):
            out = self.model(inp)
            return dict_to_tuple(out[0])


    model_func = torchvision.models.detection.maskrcnn_resnet50_fpn
    model = TraceWrapper(model_func(pretrained=True))

    model.eval()
    inp = torch.Tensor(np.random.uniform(0.0, 250.0, size=(1, 3, in_size, in_size)))

    with torch.no_grad():
        out = model(inp)
        script_module = do_trace(model, inp)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Downloading: "https://download.pytorch.org/models/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth" to /workspace/.cache/torch/hub/checkpoints/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth

      0%|          | 0.00/170M [00:00<?, ?B/s]
      9%|8         | 14.8M/170M [00:00<00:01, 155MB/s]
     23%|##2       | 39.0M/170M [00:00<00:00, 213MB/s]
     37%|###7      | 63.3M/170M [00:00<00:00, 232MB/s]
     53%|#####2    | 89.5M/170M [00:00<00:00, 249MB/s]
     68%|######8   | 116M/170M [00:00<00:00, 259MB/s] 
     84%|########3 | 142M/170M [00:00<00:00, 263MB/s]
     99%|#########8| 168M/170M [00:00<00:00, 265MB/s]
    100%|##########| 170M/170M [00:00<00:00, 251MB/s]
    /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:3878: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      for i in range(dim)
    /usr/local/lib/python3.7/dist-packages/torchvision/models/detection/anchor_utils.py:127: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
      for g in grid_sizes
    /usr/local/lib/python3.7/dist-packages/torchvision/models/detection/anchor_utils.py:127: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      for g in grid_sizes
    /usr/local/lib/python3.7/dist-packages/torchvision/models/detection/rpn.py:73: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
      A = Ax4 // 4
    /usr/local/lib/python3.7/dist-packages/torchvision/models/detection/rpn.py:74: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
      C = AxC // A
    /usr/local/lib/python3.7/dist-packages/torchvision/ops/boxes.py:156: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      boxes_x = torch.min(boxes_x, torch.tensor(width, dtype=boxes.dtype, device=boxes.device))
    /usr/local/lib/python3.7/dist-packages/torchvision/ops/boxes.py:158: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      boxes_y = torch.min(boxes_y, torch.tensor(height, dtype=boxes.dtype, device=boxes.device))
    /usr/local/lib/python3.7/dist-packages/torchvision/models/detection/transform.py:293: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      for s, s_orig in zip(new_size, original_size)
    /usr/local/lib/python3.7/dist-packages/torchvision/models/detection/roi_heads.py:387: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      return torch.tensor(M + 2 * padding).to(torch.float32) / torch.tensor(M).to(torch.float32)




.. GENERATED FROM PYTHON SOURCE LINES 97-99

Download a test image and pre-process
-------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 99-110

.. code-block:: default

    img_url = (
        "https://raw.githubusercontent.com/dmlc/web-data/master/gluoncv/detection/street_small.jpg"
    )
    img_path = download_testdata(img_url, "test_street_small.jpg", module="data")

    img = cv2.imread(img_path).astype("float32")
    img = cv2.resize(img, (in_size, in_size))
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    img = np.transpose(img / 255.0, [2, 0, 1])
    img = np.expand_dims(img, axis=0)








.. GENERATED FROM PYTHON SOURCE LINES 111-113

Import the graph to Relay
-------------------------

.. GENERATED FROM PYTHON SOURCE LINES 113-117

.. code-block:: default

    input_name = "input0"
    shape_list = [(input_name, input_shape)]
    mod, params = relay.frontend.from_pytorch(script_module, shape_list)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /workspace/python/tvm/relay/build_module.py:411: DeprecationWarning: Please use input parameter mod (tvm.IRModule) instead of deprecated parameter mod (tvm.relay.function.Function)
      DeprecationWarning,




.. GENERATED FROM PYTHON SOURCE LINES 118-124

Compile with Relay VM
---------------------
Note: Currently only CPU target is supported. For x86 target, it is
highly recommended to build TVM with Intel MKL and Intel OpenMP to get
best performance, due to the existence of large dense operator in
torchvision rcnn models.

.. GENERATED FROM PYTHON SOURCE LINES 124-133

.. code-block:: default


    # Add "-libs=mkl" to get best performance on x86 target.
    # For x86 machine supports AVX512, the complete target is
    # "llvm -mcpu=skylake-avx512 -libs=mkl"
    target = "llvm"

    with tvm.transform.PassContext(opt_level=3, disabled_pass=["FoldScaleAxis"]):
        vm_exec = relay.vm.compile(mod, target=target, params=params)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /workspace/python/tvm/driver/build_module.py:268: UserWarning: target_host parameter is going to be deprecated. Please pass in tvm.target.Target(target, host=target_host) instead.
      "target_host parameter is going to be deprecated. "




.. GENERATED FROM PYTHON SOURCE LINES 134-136

Inference with Relay VM
-----------------------

.. GENERATED FROM PYTHON SOURCE LINES 136-141

.. code-block:: default

    dev = tvm.cpu()
    vm = VirtualMachine(vm_exec, dev)
    vm.set_input("main", **{input_name: img})
    tvm_res = vm.run()








.. GENERATED FROM PYTHON SOURCE LINES 142-144

Get boxes with score larger than 0.9
------------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 144-154

.. code-block:: default

    score_threshold = 0.9
    boxes = tvm_res[0].numpy().tolist()
    valid_boxes = []
    for i, score in enumerate(tvm_res[1].numpy().tolist()):
        if score > score_threshold:
            valid_boxes.append(boxes[i])
        else:
            break

    print("Get {} valid boxes".format(len(valid_boxes)))




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Get 9 valid boxes





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** ( 2 minutes  56.247 seconds)


.. _sphx_glr_download_how_to_deploy_models_deploy_object_detection_pytorch.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example


    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: deploy_object_detection_pytorch.py <deploy_object_detection_pytorch.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: deploy_object_detection_pytorch.ipynb <deploy_object_detection_pytorch.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_