tvm.runtime¶

TVM runtime namespace.

Classes:

`PackedFunc`	The PackedFunc object used in TVM.
`Object`	Base class for all tvm's runtime objects.
`ObjectPath`()	Path to an object from some root object.
`ObjectPathPair`	Pair of ObjectPaths, one for each object being tested for structural equality.
`ObjectGeneric`()	Base class for all classes that can be converted to object.
`DataType`(type_str)	TVM datatype structure
`DataTypeCode`()	DataType code in DLTensor.
`Device`(device_type, device_id)	TVM device strucure.
`Module`(handle)	Runtime Module.
`Report`(calls, device_metrics, configuration)	A container for information gathered during a profiling run.
`String`(content)	TVM runtime.String object, represented as a python str.
`ShapeTuple`(shape)	TVM runtime ShapeTuple object.

Functions:

`num_threads`()	Get the number of threads in use by the TVM runtime.
`convert_to_object`(value[, span])	Convert a Python value to corresponding object type.
`convert`(value[, span])	Convert value to TVM object or function.
`const`(value[, dtype, span])	construct a constant
`device`(dev_type[, dev_id])	Construct a TVM device with given device type and id.
`cpu`([dev_id])	Construct a CPU device
`cuda`([dev_id])	Construct a CUDA GPU device
`gpu`([dev_id])	Construct a CUDA GPU device
`opencl`([dev_id])	Construct a OpenCL device
`cl`([dev_id])	Construct a OpenCL device
`vulkan`([dev_id])	Construct a Vulkan device
`metal`([dev_id])	Construct a metal device
`mtl`([dev_id])	Construct a metal device
`vpi`([dev_id])	Construct a VPI simulated device
`rocm`([dev_id])	Construct a ROCM device
`ext_dev`([dev_id])	Construct a extension device
`load_module`(path[, fmt])	Load module from file.
`enabled`(target)	Whether module runtime is enabled for target
`system_lib`()	Get system-wide library module singleton.
`load_static_library`(path, func_names)	Load the .o library at path which implements functions with func_names.
`save_param_dict`(params)	Save parameter dictionary to binary bytes.
`load_param_dict`(param_bytes)	Load parameter dictionary to binary bytes.

class tvm.runtime.PackedFunc¶

The PackedFunc object used in TVM.

Function plays an key role to bridge front and backend in TVM. Function provide a type-erased interface, you can call function with positional arguments.

The compiled module returns Function. TVM backend also registers and exposes its API as Functions.

The following are list of common usage scenario of tvm.runtime.PackedFunc.

Automatic exposure of C++ API into python
To call PackedFunc from python side
To call python callbacks to inspect results in generated code
Bring python hook into C++ backend

See also

tvm.register_func: How to register global function.
tvm.get_global_func: How to get global function.

class tvm.runtime.Object¶: Base class for all tvm’s runtime objects.

class tvm.runtime.ObjectPath¶: Path to an object from some root object.

class tvm.runtime.ObjectPathPair¶: Pair of ObjectPaths, one for each object being tested for structural equality.

class tvm.runtime.ObjectGeneric¶

Base class for all classes that can be converted to object.

Methods:

asobject()

Convert value to object

asobject()¶: Convert value to object

class tvm.runtime.DataType(type_str)¶: TVM datatype structure

class tvm.runtime.DataTypeCode¶: DataType code in DLTensor.

class tvm.runtime.Device(device_type, device_id)¶

TVM device strucure.

Typically constructed using convenience function tvm.runtime.device().

Exposes uniform interface to device-specific APIs such as CUDA or OpenCL. Some properties may return None depending on whether an API exposes that particular property.

Attributes:

`exist`	Whether this device exists.
`max_threads_per_block`	Maximum number of threads on each block.
`warp_size`	Number of threads that execute concurrently.
`max_shared_memory_per_block`	Total amount of shared memory per block in bytes.
`compute_version`	Get compute version number as string.
`device_name`	Return the vendor-specific name of device.
`max_clock_rate`	Return the max clock frequency of device (kHz).
`multi_processor_count`	Return the number of compute units in the device.
`max_thread_dimensions`	Return the maximum size of each thread axis
`api_version`	Returns version number of the SDK used to compile TVM.
`driver_version`	Returns version number of the driver

Methods:

`texture_spatial_limit`()	Returns limits for textures by spatial dimensions
`create_raw_stream`()	Create a new runtime stream at the context.
`free_raw_stream`(stream)	Free a created stream handle.
`set_raw_stream`(stream)	Set a created stream handle.
`sync`([stream])	Synchronize until jobs finished at the context.

property exist¶

Whether this device exists.

Returns True if TVM has support for the device, if the physical device is present, and the device is accessible through appropriate drivers (e.g. cuda/vulkan).

Returns: exist – True if the device exists
Return type: bool

property max_threads_per_block¶

Maximum number of threads on each block.

Returns device value for cuda, metal, rocm, opencl, and vulkan devices. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: max_threads_per_block – The number of threads on each block
Return type: int or None

property warp_size¶

Number of threads that execute concurrently.

Returns device value for cuda, rocm, and vulkan. Returns 1 for metal and opencl devices, regardless of the physical device. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: warp_size – Number of threads that execute concurrently
Return type: int or None

property max_shared_memory_per_block¶

Total amount of shared memory per block in bytes.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: max_shared_memory_per_block – Total amount of shared memory per block in bytes
Return type: int or None

property compute_version¶

Get compute version number as string.

Returns maximum API version (e.g. CUDA/OpenCL/Vulkan) supported by the device.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: version – The version string in major.minor format.
Return type: str or None

property device_name¶

Return the vendor-specific name of device.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: device_name – The name of the device.
Return type: str or None

property max_clock_rate¶

Return the max clock frequency of device (kHz).

Returns device value for cuda, rocm, and opencl. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: max_clock_rate – The maximum clock frequency of the device (kHz)
Return type: int or None

property multi_processor_count¶

Return the number of compute units in the device.

Returns device value for cuda, rocm, and opencl. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: multi_processor_count – Thee number of compute units in the device
Return type: int or None

property max_thread_dimensions¶

Return the maximum size of each thread axis

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: dims – The maximum length of threadIdx.x, threadIdx.y, threadIdx.z
Return type: List of int, or None

property api_version¶

Returns version number of the SDK used to compile TVM.

For example, CUDA_VERSION for cuda or VK_HEADER_VERSION for Vulkan.

Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: version – The version of the SDK
Return type: int or None

property driver_version¶

Returns version number of the driver

Returns driver vendor’s internal version number. (e.g. “450.408.256” for nvidia-driver-450)

Returns device value for opencl and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.

Returns: version – The version string in major.minor.patch format.
Return type: str or None

texture_spatial_limit()¶

Returns limits for textures by spatial dimensions

Returns: limit – Maximum size of the texture by spatial dimensions
Return type: int or None

create_raw_stream()¶

Create a new runtime stream at the context.

User should free the stream after use.

Returns: stream – The created runtime stream.
Return type: TVMStreamHandle

free_raw_stream(stream)¶

Free a created stream handle.

Parameters: stream (TVMStreamHandle) – The stream which should to be released.

set_raw_stream(stream)¶

Set a created stream handle.

Parameters: stream (TVMStreamHandle) – The stream which should to be set to the device.

sync(stream=None)¶

Synchronize until jobs finished at the context.

Parameters: stream (TVMStreamHandle) – Jobs in this stream should be finished.

class tvm.runtime.Module(handle)¶

Runtime Module.

Attributes:

`entry_func`	Get the entry function
`type_key`	Get type key of the module.
`format`	Get the format of the module.
`imported_modules`	Get imported modules
`is_dso_exportable`	Returns true if module is 'DSO exportable', ie can be included in result of export_library by the external compiler directly.

Methods:

`implements_function`(name[, query_imports])	Returns True if the module has a definition for the global function with name.
`get_function`(name[, query_imports])	Get function from the module.
`import_module`(module)	Add module to the import list of current one.
`get_source`([fmt])	Get source code from module, if available.
`save`(file_name[, fmt])	Save the module to file.
`time_evaluator`(func_name, dev[, number, ...])	Get an evaluator that measures time cost of running function.
`export_library`(file_name[, fcompile, ...])	Export the module and all imported modules into a single device library.

property entry_func¶

Get the entry function

Returns: f – The entry function if exist
Return type: tvm.runtime.PackedFunc

implements_function(name, query_imports=False)¶

Returns True if the module has a definition for the global function with name. Note that has_function(name) does not imply get_function(name) is non-null since the module may be, eg, a CSourceModule which cannot supply a packed-func implementation of the function without further compilation. However, get_function(name) non null should always imply has_function(name).

Parameters

name (str) – The name of the function
query_imports (bool) – Whether to also query modules imported by this module.

Returns

b – True if module (or one of its imports) has a definition for name.

Return type

Bool

get_function(name, query_imports=False)¶

Get function from the module.

Parameters

name (str) – The name of the function
query_imports (bool) – Whether also query modules imported by this module.

Returns

f – The result function.

Return type

tvm.runtime.PackedFunc

import_module(module)¶

Add module to the import list of current one.

Parameters: module (tvm.runtime.Module) – The other module.

property type_key¶: Get type key of the module.

property format¶: Get the format of the module.

get_source(fmt='')¶

Get source code from module, if available.

Parameters: fmt (str, optional) – The specified format.
Returns: source – The result source code.
Return type: str

property imported_modules¶

Get imported modules

Returns: modules – The module
Return type: list of Module

property is_dso_exportable¶

Returns true if module is ‘DSO exportable’, ie can be included in result of export_library by the external compiler directly.

Returns: b – True if the module is DSO exportable.
Return type: Bool

save(file_name, fmt='')¶

Save the module to file.

This do not save the dependent device modules. See also export_shared

Parameters

file_name (str) – The name of the file.
fmt (str) – The format of the file.

See also

runtime.Module.export_library: export the module to shared library.

time_evaluator(func_name, dev, number=10, repeat=1, min_repeat_ms=0, limit_zero_time_iterations=100, cooldown_interval_ms=0, repeats_to_cooldown=1, f_preproc='')¶

Get an evaluator that measures time cost of running function.

Parameters

func_name (str) – The name of the function in the module.
dev (Device) – The device we should run this function on.
number (int) – The number of times to run this function for taking average. We call these runs as one repeat of measurement.
repeat (int, optional) – The number of times to repeat the measurement. In total, the function will be invoked (1 + number x repeat) times, where the first one is warm up and will be discarded. The returned result contains repeat costs, each of which is an average of number costs.
min_repeat_ms (int, optional) – The minimum duration of one repeat in milliseconds. By default, one repeat contains number runs. If this parameter is set, the parameters number will be dynamically adjusted to meet the minimum duration requirement of one repeat. i.e., When the run time of one repeat falls below this time, the number parameter will be automatically increased.
limit_zero_time_iterations (int, optional) – The maximum number of repeats when measured time is equal to 0. It helps to avoid hanging during measurements.
cooldown_interval_ms (int, optional) – The cooldown interval in milliseconds between the number of repeats defined by repeats_to_cooldown.
repeats_to_cooldown (int, optional) – The number of repeats before the cooldown is activated.
f_preproc (str, optional) – The preprocess function name we want to execute before executing the time evaluator.

Note

The function will be invoked (1 + number x repeat) times, with the first call discarded in case there is lazy initialization.

Returns: ftimer – The function that takes same argument as func and returns a BenchmarkResult. The ProfileResult reports repeat time costs in seconds.
Return type: function

export_library(file_name, fcompile=None, addons=None, workspace_dir=None, **kwargs)¶

Export the module and all imported modules into a single device library.

This function only works on host LLVM modules, other runtime::Module subclasses will work with this API but they must support implement the save and load mechanisms of modules completely including saving from streams and files. This will pack your non-shared library module into a single shared library which can later be loaded by TVM.

Parameters

file_name (str) – The name of the shared library.
fcompile (function(target, file_list, kwargs), optional) –
The compilation function to use create the final library object during export.

For example, when fcompile=_cc.create_shared, or when it is not supplied but module is “llvm,” this is used to link all produced artifacts into a final dynamic library.

This behavior is controlled by the type of object exported. If fcompile has attribute object_format, will compile host library to that format. Otherwise, will use default format “o”.
workspace_dir (str, optional) – The path of the directory used to create the intermediate artifacts when exporting the module. If this is not provided a temporary dir will be created.
kwargs (dict, optional) – Additional arguments passed to fcompile

Returns

result of fcompile() – If the compilation function returns an artifact it would be returned via export_library, if any.

Return type

unknown, optional

tvm.runtime.num_threads() → int¶

Get the number of threads in use by the TVM runtime.

Returns: Number of threads in use.
Return type: int

class tvm.runtime.Report(calls: Sequence[Dict[str, tvm.runtime.object.Object]], device_metrics: Dict[str, Dict[str, tvm.runtime.object.Object]], configuration: Dict[str, tvm.runtime.object.Object])¶

A container for information gathered during a profiling run.

calls¶

Per-call profiling metrics (function name, runtime, device, …).

Type: Array[Dict[str, Object]]

device_metrics¶

Per-device metrics collected over the entire run.

Type: Dict[Device, Dict[str, Object]]

Methods:

`csv`()	Convert this profiling report into CSV format.
`table`([sort, aggregate, col_sums])	Generate a human-readable table
`json`()	Convert this profiling report into JSON format.
`from_json`(s)	Deserialize a report from JSON.

csv()¶

Convert this profiling report into CSV format.

This only includes calls and not overall metrics.

Returns: csv – calls in CSV format.
Return type: str

table(sort=True, aggregate=True, col_sums=True)¶

Generate a human-readable table

Parameters

sort (bool) – If aggregate is true, whether to sort call frames by descending duration. If aggregate is False, whether to sort frames by order of appearancei n the program.
aggregate (bool) – Whether to join multiple calls to the same op into a single line.
col_sums (bool) – Whether to include the sum of each column.

Returns

table – A human-readable table

Return type

str

json()¶

Convert this profiling report into JSON format.

Example output:

Returns: json – Formatted JSON
Return type: str

classmethod from_json(s)¶

Deserialize a report from JSON.

Parameters: s (str) – Report serialize via json().
Returns: report – The deserialized report.
Return type: Report

tvm.runtime.convert_to_object(value, span=None)¶

Convert a Python value to corresponding object type.

Parameters

value (str) – The value to be inspected.
span (Optional[Span]) – The location of this itervar in the source code.

Returns

obj – The corresponding object value.

Return type

Object

tvm.runtime.convert(value, span=None)¶

Convert value to TVM object or function.

Parameters

value (python value) –
span (Optional[Span]) – The location of this statement in the source code.

Returns

tvm_val – Converted value in TVM

Return type

Object or Function

tvm.runtime.const(value, dtype=None, span=None)¶

construct a constant

Parameters

value (number) – The content of the constant number.
dtype (str or None, optional) – The data type.
span (Optional[Span]) – The location of the constant value in the source.

Returns

const_val – The result expression.

Return type

tvm.Expr

tvm.runtime.device(dev_type, dev_id=0)¶

Construct a TVM device with given device type and id.

Parameters

dev_type (int or str) – The device type mask or name of the device.
dev_id (int, optional) – The integer device id

Returns

dev – The corresponding device.

Return type

tvm.runtime.Device

Examples

Device can be used to create reflection of device by string representation of the device type.

assert tvm.device("cpu", 1) == tvm.cpu(1)
assert tvm.device("cuda", 0) == tvm.cuda(0)

tvm.runtime.cpu(dev_id=0)¶

Construct a CPU device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.cuda(dev_id=0)¶

Construct a CUDA GPU device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.gpu(dev_id=0)¶

Construct a CUDA GPU device

deprecated:: 0.9.0 Use tvm.cuda() instead.

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.opencl(dev_id=0)¶

Construct a OpenCL device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.cl(dev_id=0)¶

Construct a OpenCL device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.vulkan(dev_id=0)¶

Construct a Vulkan device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.metal(dev_id=0)¶

Construct a metal device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.mtl(dev_id=0)¶

Construct a metal device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.vpi(dev_id=0)¶

Construct a VPI simulated device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.rocm(dev_id=0)¶

Construct a ROCM device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

tvm.runtime.ext_dev(dev_id=0)¶

Construct a extension device

Parameters: dev_id (int, optional) – The integer device id
Returns: dev – The created device
Return type: Device

Note

This API is reserved for quick testing of new device by plugin device API as ext_dev.

tvm.runtime.load_module(path, fmt='')¶

Load module from file.

Parameters

path (str) – The path to the module file.
fmt (str, optional) – The format of the file, if not specified it will be inferred from suffix of the file.

Returns

module – The loaded module

Return type

runtime.Module

Note

This function will automatically call cc.create_shared if the path is in format .o or .tar

tvm.runtime.enabled(target)¶

Whether module runtime is enabled for target

Parameters: target (str) – The target device type.
Returns: enabled – Whether runtime is enabled.
Return type: bool

Examples

The following code checks if gpu is enabled.

>>> tvm.runtime.enabled("gpu")

tvm.runtime.system_lib()¶

Get system-wide library module singleton.

System lib is a global module that contains self register functions in startup. Unlike normal dso modules which need to be loaded explicitly. It is useful in environments where dynamic loading api like dlopen is banned.

To build system lib function, simply specify target option `llvm --system-lib` The system lib will be available as long as the result code is linked by the program.

The system lib is intended to be linked and loaded during the entire life-cyle of the program. If you want dynamic loading features, use dso modules instead.

Returns: module – The system-wide library module.
Return type: runtime.Module

tvm.runtime.load_static_library(path, func_names)¶: Load the .o library at path which implements functions with func_names. Unlike the generic load_module the result will remain as a static_library and will not be relinked on-the-fly into a .so library.

class tvm.runtime.String(content)¶

TVM runtime.String object, represented as a python str.

Parameters: content (str) – The content string used to construct the object.

class tvm.runtime.ShapeTuple(shape)¶: TVM runtime ShapeTuple object. :param shape: The shape list used to construct the object. :type shape: list[int]

tvm.runtime.save_param_dict(params)¶

Save parameter dictionary to binary bytes.

The result binary bytes can be loaded by the GraphModule with API “load_params”.

Parameters: params (dict of str to NDArray) – The parameter dictionary.
Returns: param_bytes – Serialized parameters.
Return type: bytearray

Examples

# set up the parameter dict
params = {"param0": arr0, "param1": arr1}
# save the parameters as byte array
param_bytes = tvm.runtime.save_param_dict(params)
# We can serialize the param_bytes and load it back later.
# Pass in byte array to module to directly set parameters
tvm.runtime.load_param_dict(param_bytes)

tvm.runtime.load_param_dict(param_bytes)¶

Load parameter dictionary to binary bytes.

Parameters: param_bytes (bytearray) – Serialized parameters.
Returns: params – The parameter dictionary.
Return type: dict of str to NDArray