tvm.runtime

TVM runtime namespace.

Classes:

PackedFunc

The PackedFunc object used in TVM.

Object

Base class for all tvm's runtime objects.

ObjectPath()

Path to an object from some root object.

ObjectPathPair

Pair of ObjectPaths, one for each object being tested for structural equality.

ObjectGeneric()

Base class for all classes that can be converted to object.

DataType(type_str)

TVM datatype structure

DataTypeCode()

DataType code in DLTensor.

Device(device_type, device_id)

TVM device strucure.

Module(handle)

Runtime Module.

Report(calls, device_metrics, configuration)

A container for information gathered during a profiling run.

String(content)

TVM runtime.String object, represented as a python str.

ShapeTuple(shape)

TVM runtime ShapeTuple object.

Functions:

num_threads()

Get the number of threads in use by the TVM runtime.

convert_to_object(value[, span])

Convert a Python value to corresponding object type.

convert(value[, span])

Convert value to TVM object or function.

const(value[, dtype, span])

construct a constant

device(dev_type[, dev_id])

Construct a TVM device with given device type and id.

cpu([dev_id])

Construct a CPU device

cuda([dev_id])

Construct a CUDA GPU device

gpu([dev_id])

Construct a CUDA GPU device

opencl([dev_id])

Construct a OpenCL device

cl([dev_id])

Construct a OpenCL device

vulkan([dev_id])

Construct a Vulkan device

metal([dev_id])

Construct a metal device

mtl([dev_id])

Construct a metal device

vpi([dev_id])

Construct a VPI simulated device

rocm([dev_id])

Construct a ROCM device

ext_dev([dev_id])

Construct a extension device

load_module(path[, fmt])

Load module from file.

enabled(target)

Whether module runtime is enabled for target

system_lib()

Get system-wide library module singleton.

load_static_library(path, func_names)

Load the .o library at path which implements functions with func_names.

save_param_dict(params)

Save parameter dictionary to binary bytes.

load_param_dict(param_bytes)

Load parameter dictionary to binary bytes.

class tvm.runtime.PackedFunc

The PackedFunc object used in TVM.

Function plays an key role to bridge front and backend in TVM. Function provide a type-erased interface, you can call function with positional arguments.

The compiled module returns Function. TVM backend also registers and exposes its API as Functions.

The following are list of common usage scenario of tvm.runtime.PackedFunc.

  • Automatic exposure of C++ API into python

  • To call PackedFunc from python side

  • To call python callbacks to inspect results in generated code

  • Bring python hook into C++ backend

See also

tvm.register_func

How to register global function.

tvm.get_global_func

How to get global function.

class tvm.runtime.Object

Base class for all tvm’s runtime objects.

class tvm.runtime.ObjectPath

Path to an object from some root object.

class tvm.runtime.ObjectPathPair

Pair of ObjectPaths, one for each object being tested for structural equality.

class tvm.runtime.ObjectGeneric

Base class for all classes that can be converted to object.

Methods:

asobject()

Convert value to object

asobject()

Convert value to object

class tvm.runtime.DataType(type_str)

TVM datatype structure

class tvm.runtime.DataTypeCode

DataType code in DLTensor.

class tvm.runtime.Device(device_type, device_id)

TVM device strucure.

Typically constructed using convenience function tvm.runtime.device().

Exposes uniform interface to device-specific APIs such as CUDA or OpenCL. Some properties may return None depending on whether an API exposes that particular property.

NOTE! The integer values in MASK2STR and STR2MASK must correspond to the values provided by the DLDeviceType and TVMDeviceExtType enums.

Attributes:

exist

Whether this device exists.

max_threads_per_block

Maximum number of threads on each block.

warp_size

Number of threads that execute concurrently.

max_shared_memory_per_block

Total amount of shared memory per block in bytes.

compute_version

Get compute version number as string.

device_name

Return the vendor-specific name of device.

max_clock_rate

Return the max clock frequency of device (kHz).

multi_processor_count

Return the number of compute units in the device.

max_thread_dimensions

Return the maximum size of each thread axis

api_version

Returns version number of the SDK used to compile TVM.

driver_version

Returns version number of the driver

Methods:

texture_spatial_limit()

Returns limits for textures by spatial dimensions

create_raw_stream()

Create a new runtime stream at the context.

free_raw_stream(stream)

Free a created stream handle.

set_raw_stream(stream)

Set a created stream handle.

sync([stream])

Synchronize until jobs finished at the context.

property exist

Whether this device exists.

Returns True if TVM has support for the device, if the physical device is present, and the device is accessible through appropriate drivers (e.g. cuda/vulkan).

Returns

exist – True if the device exists

Return type

bool

property max_threads_per_block

Maximum number of threads on each block.

Returns device value for cuda, metal, rocm, opencl, and vulkan devices. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

max_threads_per_block – The number of threads on each block

Return type

int or None

property warp_size

Number of threads that execute concurrently.

Returns device value for cuda, rocm, and vulkan. Returns 1 for metal and opencl devices, regardless of the physical device. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

warp_size – Number of threads that execute concurrently

Return type

int or None

property max_shared_memory_per_block

Total amount of shared memory per block in bytes.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

max_shared_memory_per_block – Total amount of shared memory per block in bytes

Return type

int or None

property compute_version

Get compute version number as string.

Returns maximum API version (e.g. CUDA/OpenCL/Vulkan) supported by the device.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

version – The version string in major.minor format.

Return type

str or None

property device_name

Return the vendor-specific name of device.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

device_name – The name of the device.

Return type

str or None

property max_clock_rate

Return the max clock frequency of device (kHz).

Returns device value for cuda, rocm, and opencl. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

max_clock_rate – The maximum clock frequency of the device (kHz)

Return type

int or None

property multi_processor_count

Return the number of compute units in the device.

Returns device value for cuda, rocm, and opencl. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

multi_processor_count – Thee number of compute units in the device

Return type

int or None

property max_thread_dimensions

Return the maximum size of each thread axis

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

dims – The maximum length of threadIdx.x, threadIdx.y, threadIdx.z

Return type

List of int, or None

property api_version

Returns version number of the SDK used to compile TVM.

For example, CUDA_VERSION for cuda or VK_HEADER_VERSION for Vulkan.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

version – The version of the SDK

Return type

int or None

property driver_version

Returns version number of the driver

Returns driver vendor’s internal version number. (e.g. “450.408.256” for nvidia-driver-450)

Returns device value for opencl and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns

version – The version string in major.minor.patch format.

Return type

str or None

texture_spatial_limit()

Returns limits for textures by spatial dimensions

Returns

limit – Maximum size of the texture by spatial dimensions

Return type

int or None

create_raw_stream()

Create a new runtime stream at the context.

User should free the stream after use.

Returns

stream – The created runtime stream.

Return type

TVMStreamHandle

free_raw_stream(stream)

Free a created stream handle.

Parameters

stream (TVMStreamHandle) – The stream which should to be released.

set_raw_stream(stream)

Set a created stream handle.

Parameters

stream (TVMStreamHandle) – The stream which should to be set to the device.

sync(stream=None)

Synchronize until jobs finished at the context.

Parameters

stream (TVMStreamHandle) – Jobs in this stream should be finished.

class tvm.runtime.Module(handle)

Runtime Module.

Attributes:

entry_func

Get the entry function

type_key

Get type key of the module.

format

Get the format of the module.

imported_modules

Get imported modules

is_dso_exportable

Returns true if module is 'DSO exportable', ie can be included in result of export_library by the external compiler directly.

Methods:

implements_function(name[, query_imports])

Returns True if the module has a definition for the global function with name.

get_function(name[, query_imports])

Get function from the module.

import_module(module)

Add module to the import list of current one.

get_source([fmt])

Get source code from module, if available.

save(file_name[, fmt])

Save the module to file.

time_evaluator(func_name, dev[, number, ...])

Get an evaluator that measures time cost of running function.

export_library(file_name[, fcompile, ...])

Export the module and all imported modules into a single device library.

property entry_func

Get the entry function

Returns

f – The entry function if exist

Return type

tvm.runtime.PackedFunc

implements_function(name, query_imports=False)

Returns True if the module has a definition for the global function with name. Note that has_function(name) does not imply get_function(name) is non-null since the module may be, eg, a CSourceModule which cannot supply a packed-func implementation of the function without further compilation. However, get_function(name) non null should always imply has_function(name).

Parameters
  • name (str) – The name of the function

  • query_imports (bool) – Whether to also query modules imported by this module.

Returns

b – True if module (or one of its imports) has a definition for name.

Return type

Bool

get_function(name, query_imports=False)

Get function from the module.

Parameters
  • name (str) – The name of the function

  • query_imports (bool) – Whether also query modules imported by this module.

Returns

f – The result function.

Return type

tvm.runtime.PackedFunc

import_module(module)

Add module to the import list of current one.

Parameters

module (tvm.runtime.Module) – The other module.

property type_key

Get type key of the module.

property format

Get the format of the module.

get_source(fmt='')

Get source code from module, if available.

Parameters

fmt (str, optional) – The specified format.

Returns

source – The result source code.

Return type

str

property imported_modules

Get imported modules

Returns

modules – The module

Return type

list of Module

property is_dso_exportable

Returns true if module is ‘DSO exportable’, ie can be included in result of export_library by the external compiler directly.

Returns

b – True if the module is DSO exportable.

Return type

Bool

save(file_name, fmt='')

Save the module to file.

This do not save the dependent device modules. See also export_shared

Parameters
  • file_name (str) – The name of the file.

  • fmt (str) – The format of the file.

See also

runtime.Module.export_library

export the module to shared library.

time_evaluator(func_name, dev, number=10, repeat=1, min_repeat_ms=0, limit_zero_time_iterations=100, cooldown_interval_ms=0, repeats_to_cooldown=1, f_preproc='')

Get an evaluator that measures time cost of running function.

Parameters
  • func_name (str) – The name of the function in the module.

  • dev (Device) – The device we should run this function on.

  • number (int) – The number of times to run this function for taking average. We call these runs as one repeat of measurement.

  • repeat (int, optional) – The number of times to repeat the measurement. In total, the function will be invoked (1 + number x repeat) times, where the first one is warm up and will be discarded. The returned result contains repeat costs, each of which is an average of number costs.

  • min_repeat_ms (int, optional) – The minimum duration of one repeat in milliseconds. By default, one repeat contains number runs. If this parameter is set, the parameters number will be dynamically adjusted to meet the minimum duration requirement of one repeat. i.e., When the run time of one repeat falls below this time, the number parameter will be automatically increased.

  • limit_zero_time_iterations (int, optional) – The maximum number of repeats when measured time is equal to 0. It helps to avoid hanging during measurements.

  • cooldown_interval_ms (int, optional) – The cooldown interval in milliseconds between the number of repeats defined by repeats_to_cooldown.

  • repeats_to_cooldown (int, optional) – The number of repeats before the cooldown is activated.

  • f_preproc (str, optional) – The preprocess function name we want to execute before executing the time evaluator.

Note

The function will be invoked (1 + number x repeat) times, with the first call discarded in case there is lazy initialization.

Returns

ftimer – The function that takes same argument as func and returns a BenchmarkResult. The ProfileResult reports repeat time costs in seconds.

Return type

function

export_library(file_name, fcompile=None, addons=None, workspace_dir=None, **kwargs)

Export the module and all imported modules into a single device library.

This function only works on host LLVM modules, other runtime::Module subclasses will work with this API but they must support implement the save and load mechanisms of modules completely including saving from streams and files. This will pack your non-shared library module into a single shared library which can later be loaded by TVM.

Parameters
  • file_name (str) – The name of the shared library.

  • fcompile (function(target, file_list, kwargs), optional) –

    The compilation function to use create the final library object during export.

    For example, when fcompile=_cc.create_shared, or when it is not supplied but module is “llvm,” this is used to link all produced artifacts into a final dynamic library.

    This behavior is controlled by the type of object exported. If fcompile has attribute object_format, will compile host library to that format. Otherwise, will use default format “o”.

  • workspace_dir (str, optional) – The path of the directory used to create the intermediate artifacts when exporting the module. If this is not provided a temporary dir will be created.

  • kwargs (dict, optional) – Additional arguments passed to fcompile

Returns

result of fcompile() – If the compilation function returns an artifact it would be returned via export_library, if any.

Return type

unknown, optional

tvm.runtime.num_threads() int

Get the number of threads in use by the TVM runtime.

Returns

Number of threads in use.

Return type

int

class tvm.runtime.Report(calls: Sequence[Dict[str, tvm.runtime.object.Object]], device_metrics: Dict[str, Dict[str, tvm.runtime.object.Object]], configuration: Dict[str, tvm.runtime.object.Object])

A container for information gathered during a profiling run.

calls

Per-call profiling metrics (function name, runtime, device, …).

Type

Array[Dict[str, Object]]

device_metrics

Per-device metrics collected over the entire run.

Type

Dict[Device, Dict[str, Object]]

Methods:

csv()

Convert this profiling report into CSV format.

table([sort, aggregate, col_sums])

Generate a human-readable table

json()

Convert this profiling report into JSON format.

from_json(s)

Deserialize a report from JSON.

csv()

Convert this profiling report into CSV format.

This only includes calls and not overall metrics.

Returns

csvcalls in CSV format.

Return type

str

table(sort=True, aggregate=True, col_sums=True)

Generate a human-readable table

Parameters
  • sort (bool) – If aggregate is true, whether to sort call frames by descending duration. If aggregate is False, whether to sort frames by order of appearancei n the program.

  • aggregate (bool) – Whether to join multiple calls to the same op into a single line.

  • col_sums (bool) – Whether to include the sum of each column.

Returns

table – A human-readable table

Return type

str

json()

Convert this profiling report into JSON format.

Example output:

Returns

json – Formatted JSON

Return type

str

classmethod from_json(s)

Deserialize a report from JSON.

Parameters

s (str) – Report serialize via json().

Returns

report – The deserialized report.

Return type

Report

tvm.runtime.convert_to_object(value, span=None)

Convert a Python value to corresponding object type.

Parameters
  • value (str) – The value to be inspected.

  • span (Optional[Span]) – The location of this itervar in the source code.

Returns

obj – The corresponding object value.

Return type

Object

tvm.runtime.convert(value, span=None)

Convert value to TVM object or function.

Parameters
  • value (python value) –

  • span (Optional[Span]) – The location of this statement in the source code.

Returns

tvm_val – Converted value in TVM

Return type

Object or Function

tvm.runtime.const(value, dtype=None, span=None)

construct a constant

Parameters
  • value (number) – The content of the constant number.

  • dtype (str or None, optional) – The data type.

  • span (Optional[Span]) – The location of the constant value in the source.

Returns

const_val – The result expression.

Return type

tvm.Expr

tvm.runtime.device(dev_type, dev_id=0)

Construct a TVM device with given device type and id.

Parameters
  • dev_type (int or str) – The device type mask or name of the device.

  • dev_id (int, optional) – The integer device id

Returns

dev – The corresponding device.

Return type

tvm.runtime.Device

Examples

Device can be used to create reflection of device by string representation of the device type.

assert tvm.device("cpu", 1) == tvm.cpu(1)
assert tvm.device("cuda", 0) == tvm.cuda(0)
tvm.runtime.cpu(dev_id=0)

Construct a CPU device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.cuda(dev_id=0)

Construct a CUDA GPU device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.gpu(dev_id=0)

Construct a CUDA GPU device

deprecated:: 0.9.0 Use tvm.cuda() instead.

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.opencl(dev_id=0)

Construct a OpenCL device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.cl(dev_id=0)

Construct a OpenCL device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.vulkan(dev_id=0)

Construct a Vulkan device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.metal(dev_id=0)

Construct a metal device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.mtl(dev_id=0)

Construct a metal device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.vpi(dev_id=0)

Construct a VPI simulated device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.rocm(dev_id=0)

Construct a ROCM device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

tvm.runtime.ext_dev(dev_id=0)

Construct a extension device

Parameters

dev_id (int, optional) – The integer device id

Returns

dev – The created device

Return type

Device

Note

This API is reserved for quick testing of new device by plugin device API as ext_dev.

tvm.runtime.load_module(path, fmt='')

Load module from file.

Parameters
  • path (str) – The path to the module file.

  • fmt (str, optional) – The format of the file, if not specified it will be inferred from suffix of the file.

Returns

module – The loaded module

Return type

runtime.Module

Note

This function will automatically call cc.create_shared if the path is in format .o or .tar

tvm.runtime.enabled(target)

Whether module runtime is enabled for target

Parameters

target (str) – The target device type.

Returns

enabled – Whether runtime is enabled.

Return type

bool

Examples

The following code checks if gpu is enabled.

>>> tvm.runtime.enabled("gpu")
tvm.runtime.system_lib()

Get system-wide library module singleton.

System lib is a global module that contains self register functions in startup. Unlike normal dso modules which need to be loaded explicitly. It is useful in environments where dynamic loading api like dlopen is banned.

To build system lib function, simply specify target option `llvm --system-lib` The system lib will be available as long as the result code is linked by the program.

The system lib is intended to be linked and loaded during the entire life-cyle of the program. If you want dynamic loading features, use dso modules instead.

Returns

module – The system-wide library module.

Return type

runtime.Module

tvm.runtime.load_static_library(path, func_names)

Load the .o library at path which implements functions with func_names. Unlike the generic load_module the result will remain as a static_library and will not be relinked on-the-fly into a .so library.

class tvm.runtime.String(content)

TVM runtime.String object, represented as a python str.

Parameters

content (str) – The content string used to construct the object.

class tvm.runtime.ShapeTuple(shape)

TVM runtime ShapeTuple object. :param shape: The shape list used to construct the object. :type shape: list[int]

tvm.runtime.save_param_dict(params)

Save parameter dictionary to binary bytes.

The result binary bytes can be loaded by the GraphModule with API “load_params”.

Parameters

params (dict of str to NDArray) – The parameter dictionary.

Returns

param_bytes – Serialized parameters.

Return type

bytearray

Examples

# set up the parameter dict
params = {"param0": arr0, "param1": arr1}
# save the parameters as byte array
param_bytes = tvm.runtime.save_param_dict(params)
# We can serialize the param_bytes and load it back later.
# Pass in byte array to module to directly set parameters
tvm.runtime.load_param_dict(param_bytes)
tvm.runtime.load_param_dict(param_bytes)

Load parameter dictionary to binary bytes.

Parameters

param_bytes (bytearray) – Serialized parameters.

Returns

params – The parameter dictionary.

Return type

dict of str to NDArray