.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "tutorial/uma.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note Click :ref:`here ` to download the full example code .. rst-class:: sphx-glr-example-title .. _sphx_glr_tutorial_uma.py: .. _tutorial-uma: Making your Hardware Accelerator TVM-ready with UMA =================================================== **Authors**: `Michael J. Klaiber `_, `Christoph Gerum `_, `Paul Palomero Bernardo `_ .. GENERATED FROM PYTHON SOURCE LINES 29-43 This is an introductory tutorial to the **Universal Modular Accelerator Interface** (UMA). UMA provides an easy-to-use API to integrate new hardware accelerators into TVM. This tutorial gives you step-by-step guidance how to use UMA to make your hardware accelerator TVM-ready. While there is no one-fits-all solution for this problem, UMA targets to provide a stable and Python-only API to integrate a number of hardware accelerator classes into TVM. In this tutorial you will get to know the UMA API in three use cases of increasing complexity. In these use case the three mock-accelerators **Vanilla**, **Strawberry** and **Chocolate** are introduced and integrated into TVM using UMA. .. GENERATED FROM PYTHON SOURCE LINES 43-46 .. code-block:: default .. GENERATED FROM PYTHON SOURCE LINES 52-58 Vanilla ------------- **Vanilla** is a simple accelerator consisting of a MAC array and has no internal memory. It is can ONLY process Conv2D layers, all other layers are executed on a CPU, that also orchestrates **Vanilla**. Both the CPU and Vanilla use a shared memory. .. GENERATED FROM PYTHON SOURCE LINES 60-64 .. image:: https://raw.githubusercontent.com/apache/tvm-site/main/images/tutorial/uma_vanilla_block_diagram.png :width: 100% :alt: A block diagram of Vanilla .. GENERATED FROM PYTHON SOURCE LINES 66-73 **Vanilla** has a C interface ``vanilla_conv2dnchw(...)``` for carrying out a Conv2D operation (including same-padding), that accepts pointers to input feature map, weights and result, as well as the dimensions of `Conv2D`: `oc`, `iw`, `ih`, `ic`, `kh`, `kw`. .. code-block:: c++ int vanilla_conv2dnchw(float* ifmap, float* weights, float* result, int oc, int iw, int ih, int ic, int kh, int kw); .. GENERATED FROM PYTHON SOURCE LINES 76-86 The script `uma_cli` creates code skeletons with API-calls into the UMA-API for new accelerators. For **Vanilla** we use it as follows: (``--tutorial vanilla`` adds all the additional files required for this part of the tutorial) .. code-block:: bash pip install inflection cd $TVM_HOME/apps/uma python uma_cli.py --add_hardware vanilla_accelerator --tutorial vanilla .. GENERATED FROM PYTHON SOURCE LINES 88-99 uma_cli.py generates these files in the directory ``vanilla_accelerator`` which we are going to revist. .. code-block:: bash backend.py codegen.py conv2dnchw.cc passes.py patterns.py run.py strategies.py .. GENERATED FROM PYTHON SOURCE LINES 102-105 Vanilla backend The generated backend for vanilla is found in `vanilla_accelerator/backend.py`: .. GENERATED FROM PYTHON SOURCE LINES 107-122 .. code-block:: python class VanillaAcceleratorBackend(UMABackend): """UMA backend for VanillaAccelerator.""" def __init__(self): super().__init__() self._register_pattern("conv2d", conv2d_pattern()) self._register_tir_pass(PassPhase.TIR_PHASE_0, VanillaAcceleratorConv2DPass()) self._register_codegen(fmt="c", includes=gen_includes) @property def target_name(self): return "vanilla_accelerator" .. GENERATED FROM PYTHON SOURCE LINES 126-130 Define offloaded patterns To specify that `Conv2D` is offloaded to **Vanilla**, it is described as Relay dataflow pattern (`DFPattern `_) in `vanilla_accelerator/patterns.py` .. GENERATED FROM PYTHON SOURCE LINES 133-139 .. code-block:: python def conv2d_pattern(): pattern = is_op("nn.conv2d")(wildcard(), wildcard()) pattern = pattern.has_attr({"strides": [1, 1]}) return pattern .. GENERATED FROM PYTHON SOURCE LINES 143-147 To map **Conv2D** operations from the input graph to **Vanilla**'s low level function call ``vanilla_conv2dnchw(...)``, the TIR pass *VanillaAcceleratorConv2DPass* (that will be discussed later in this tutorial) is registered in `VanillaAcceleratorBackend`. .. GENERATED FROM PYTHON SOURCE LINES 150-151 Codegen .. GENERATED FROM PYTHON SOURCE LINES 153-165 The file ``vanilla_accelerator/codegen.py`` defines static C-code that is added to the resulting C-Code generated by TVMś C-Codegen in ``gen_includes``. Here C-code is added to include **Vanilla**'s low level library``vanilla_conv2dnchw()``. .. code-block:: python def gen_includes() -> str: topdir = pathlib.Path(__file__).parent.absolute() includes = "" includes += f'#include "{topdir}/conv2dnchw.cc"' return includes .. GENERATED FROM PYTHON SOURCE LINES 168-174 As shown above in `VanillaAcceleratorBackend` it is registered to UMA with the `self._register_codegen` .. code-block:: python self._register_codegen(fmt="c", includes=gen_includes) .. GENERATED FROM PYTHON SOURCE LINES 177-208 Building the Neural Network and run it on Vanilla To demonstrate UMA's functionality, we will generate C code for a single Conv2D layer and run it on the Vanilla accelerator. The file ``vanilla_accelerator/run.py`` provides a demo running a Conv2D layer making use of Vanilla's C-API. .. code-block:: python def main(): mod, inputs, output_list, runner = create_conv2d() uma_backend = VanillaAcceleratorBackend() uma_backend.register() mod = uma_backend.partition(mod) target = tvm.target.Target("vanilla_accelerator", host=tvm.target.Target("c")) export_directory = tvm.contrib.utils.tempdir(keep_for_debug=True).path print(f"Generated files are in {export_directory}") compile_and_run( AOTModel(module=mod, inputs=inputs, outputs=output_list), runner, interface_api="c", use_unpacked_api=True, target=target, test_dir=str(export_directory), ) main() .. GENERATED FROM PYTHON SOURCE LINES 210-212 By running ``vanilla_accelerator/run.py`` the output files are generated in the model library format (MLF). .. GENERATED FROM PYTHON SOURCE LINES 214-219 Output: .. code-block:: bash Generated files are in /tmp/tvm-debug-mode-tempdirs/2022-07-13T13-26-22___x5u76h0p/00000 .. GENERATED FROM PYTHON SOURCE LINES 221-238 Let's examine the generated files: Output: .. code-block:: bash cd /tmp/tvm-debug-mode-tempdirs/2022-07-13T13-26-22___x5u76h0p/00000 cd build/ ls -1 codegen lib.tar metadata.json parameters runtime src .. GENERATED FROM PYTHON SOURCE LINES 240-251 To evaluate the generated C code go to ``codegen/host/src/default_lib2.c`` .. code-block:: bash cd codegen/host/src/ ls -1 default_lib0.c default_lib1.c default_lib2.c .. GENERATED FROM PYTHON SOURCE LINES 253-263 In `default_lib2.c` you can now see that the generated code calls into Vanilla's C-API and executes a Conv2D layer: .. code-block:: c++ TVM_DLL int32_t tvmgen_default_vanilla_accelerator_main_0(float* placeholder, float* placeholder1, float* conv2d_nchw, uint8_t* global_workspace_1_var) { vanilla_accelerator_conv2dnchw(placeholder, placeholder1, conv2d_nchw, 32, 14, 14, 32, 3, 3); return 0; } .. GENERATED FROM PYTHON SOURCE LINES 266-269 Strawberry --------------- Coming soon ... .. GENERATED FROM PYTHON SOURCE LINES 271-275 Chocolate -------------- Coming soon ... .. GENERATED FROM PYTHON SOURCE LINES 277-284 Request for Community Input ----------------------------- If this tutorial **did not** fit to your accelerator, lease add your requirements to the UMA thread in the TVM discuss forum: `Link `_. We are eager to extend this tutorial to provide guidance on making further classes of AI hardware accelerators TVM-ready using the UMA interface. .. GENERATED FROM PYTHON SOURCE LINES 286-293 References ----------- [UMA-RFC] `UMA: Universal Modular Accelerator Interface `_, TVM RFC, June 2022. [DFPattern] `Pattern Matching in Relay `_ .. _sphx_glr_download_tutorial_uma.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: uma.py ` .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: uma.ipynb ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_