tvm.runtime¶
TVM runtime namespace.
Classes:
The PackedFunc object used in TVM. |
|
Base class for all tvm's runtime objects. |
|
Path to an object from some root object. |
|
Pair of ObjectPaths, one for each object being tested for structural equality. |
|
A base class that enables the script() and show() method. |
|
Base class for all classes that can be converted to object. |
|
|
TVM datatype structure |
DataType code in DLTensor. |
|
|
TVM device strucure. |
|
Runtime Module. |
|
A container for information gathered during a profiling run. |
|
TVM runtime.String object, represented as a python str. |
|
TVM runtime ShapeTuple object. |
Functions:
Get the number of threads in use by the TVM runtime. |
|
|
Convert a Python value to corresponding object type. |
|
Convert value to TVM object or function. |
|
construct a constant |
|
Construct a TVM device with given device type and id. |
|
Construct a CPU device |
|
Construct a CUDA GPU device |
|
Construct a CUDA GPU device |
|
Construct a OpenCL device |
|
Construct a OpenCL device |
|
Construct a Vulkan device |
|
Construct a metal device |
|
Construct a metal device |
|
Construct a VPI simulated device |
|
Construct a ROCM device |
|
Construct a extension device |
|
Load module from file. |
|
Whether module runtime is enabled for target |
|
Get system-wide library module singleton. |
|
Load the .o library at path which implements functions with func_names. |
|
Save parameter dictionary to binary bytes. |
|
Load parameter dictionary from binary bytes. |
|
Save parameter dictionary to file. |
Load parameter dictionary from file. |
- class tvm.runtime.PackedFunc¶
The PackedFunc object used in TVM.
Function plays an key role to bridge front and backend in TVM. Function provide a type-erased interface, you can call function with positional arguments.
The compiled module returns Function. TVM backend also registers and exposes its API as Functions.
The following are list of common usage scenario of tvm.runtime.PackedFunc.
Automatic exposure of C++ API into python
To call PackedFunc from python side
To call python callbacks to inspect results in generated code
Bring python hook into C++ backend
See also
tvm.register_func
How to register global function.
tvm.get_global_func
How to get global function.
- class tvm.runtime.Object¶
Base class for all tvm’s runtime objects.
- class tvm.runtime.ObjectPath¶
Path to an object from some root object.
- class tvm.runtime.ObjectPathPair¶
Pair of ObjectPaths, one for each object being tested for structural equality.
- class tvm.runtime.Scriptable¶
A base class that enables the script() and show() method.
Methods:
script
(*[, name, show_meta, ir_prefix, ...])Print TVM IR into TVMScript text format
show
([style, black_format, name, show_meta, ...])A sugar for print highlighted TVM script.
- script(*, name: Optional[str] = None, show_meta: bool = False, ir_prefix: str = 'I', tir_prefix: str = 'T', buffer_dtype: str = 'float32', int_dtype: str = 'int32', float_dtype: str = 'void', verbose_expr: bool = False, indent_spaces: int = 4, print_line_numbers: bool = False, num_context_lines: int = - 1, syntax_sugar: bool = True, path_to_underline: Optional[List[tvm.runtime.object_path.ObjectPath]] = None, path_to_annotate: Optional[Dict[tvm.runtime.object_path.ObjectPath, str]] = None, obj_to_underline: Optional[List[tvm.runtime.object.Object]] = None, obj_to_annotate: Optional[Dict[tvm.runtime.object.Object, str]] = None) str ¶
Print TVM IR into TVMScript text format
- Parameters
name (Optional[str] = None) – The name of the object
show_meta (bool = False) – Whether to print the meta data of the object
ir_prefix (str = "I") – The prefix of AST nodes from tvm.ir
tir_prefix (str = "T") – The prefix of AST nodes from tvm.tir
buffer_dtype (str = "float32") – The default data type of buffer
int_dtype (str = "int32") – The default data type of integer
float_dtype (str = "void") – The default data type of float
verbose_expr (bool = False) – Whether to print the detailed definition of each variable in the expression
indent_spaces (int = 4) – The number of spaces for indentation
print_line_numbers (bool = False) – Whether to print line numbers
num_context_lines (int = -1) – The number of lines of context to print before and after the line to underline.
syntax_sugar (bool = True) – Whether to output with syntax sugar, set false for complete printing.
path_to_underline (Optional[List[ObjectPath]] = None) – Object path to be underlined
path_to_annotate (Optional[Dict[ObjectPath, str]] = None) – Object path to be annotated
obj_to_underline (Optional[List[Object]] = None) – Object to be underlined
obj_to_annotate (Optional[Dict[Object, str]] = None) – Object to be annotated
- Returns
script – The TVM Script of the given TVM IR
- Return type
- show(style: Optional[str] = None, black_format: bool = True, *, name: Optional[str] = None, show_meta: bool = False, ir_prefix: str = 'I', tir_prefix: str = 'T', buffer_dtype: str = 'float32', int_dtype: str = 'int32', float_dtype: str = 'void', verbose_expr: bool = False, indent_spaces: int = 4, print_line_numbers: bool = False, num_context_lines: int = - 1, syntax_sugar: bool = True, path_to_underline: Optional[List[tvm.runtime.object_path.ObjectPath]] = None, path_to_annotate: Optional[Dict[tvm.runtime.object_path.ObjectPath, str]] = None, obj_to_underline: Optional[List[tvm.runtime.object.Object]] = None, obj_to_annotate: Optional[Dict[tvm.runtime.object.Object, str]] = None) None ¶
A sugar for print highlighted TVM script.
- Parameters
style (str, optional) – Pygmentize printing style, auto-detected if None. See tvm.script.highlight.cprint for more details.
black_format (bool) – If true (default), use the formatter Black to format the TVMScript
name (Optional[str] = None) – The name of the object
show_meta (bool = False) – Whether to print the meta data of the object
ir_prefix (str = "I") – The prefix of AST nodes from tvm.ir
tir_prefix (str = "T") – The prefix of AST nodes from tvm.tir
buffer_dtype (str = "float32") – The default data type of buffer
int_dtype (str = "int32") – The default data type of integer
float_dtype (str = "void") – The default data type of float
verbose_expr (bool = False) – Whether to print the detailed definition of each variable in the expression
indent_spaces (int = 4) – The number of spaces for indentation
print_line_numbers (bool = False) – Whether to print line numbers
num_context_lines (int = -1) – The number of lines of context to print before and after the line to underline.
syntax_sugar (bool = True) – Whether to output with syntax sugar, set false for complete printing.
path_to_underline (Optional[List[ObjectPath]] = None) – Object path to be underlined
path_to_annotate (Optional[Dict[ObjectPath, str]] = None) – Object path to be annotated
obj_to_underline (Optional[List[Object]] = None) – Object to be underlined
obj_to_annotate (Optional[Dict[Object, str]] = None) – Object to be annotated
- class tvm.runtime.ObjectGeneric¶
Base class for all classes that can be converted to object.
Methods:
asobject
()Convert value to object
- asobject()¶
Convert value to object
- class tvm.runtime.DataType(type_str)¶
TVM datatype structure
- class tvm.runtime.DataTypeCode¶
DataType code in DLTensor.
- class tvm.runtime.Device(device_type, device_id)¶
TVM device strucure.
Typically constructed using convenience function
tvm.runtime.device()
.Exposes uniform interface to device-specific APIs such as CUDA or OpenCL. Some properties may return None depending on whether an API exposes that particular property.
NOTE! The integer values in MASK2STR and STR2MASK must correspond to the values provided by the DLDeviceType and TVMDeviceExtType enums.
Attributes:
Whether this device exists.
Maximum number of threads on each block.
Number of threads that execute concurrently.
Total amount of shared memory per block in bytes.
Get compute version number as string.
Return the vendor-specific name of device.
Return the max clock frequency of device (kHz).
Return the number of compute units in the device.
Return the maximum size of each thread axis
Returns version number of the SDK used to compile TVM.
Returns version number of the driver
Methods:
Returns limits for textures by spatial dimensions
Create a new runtime stream at the context.
free_raw_stream
(stream)Free a created stream handle.
set_raw_stream
(stream)Set a created stream handle.
sync
([stream])Synchronize until jobs finished at the context.
- property exist¶
Whether this device exists.
Returns True if TVM has support for the device, if the physical device is present, and the device is accessible through appropriate drivers (e.g. cuda/vulkan).
- Returns
exist – True if the device exists
- Return type
- property max_threads_per_block¶
Maximum number of threads on each block.
Returns device value for cuda, metal, rocm, opencl, and vulkan devices. Returns remote device value for RPC devices. Returns None for all other devices.
- property warp_size¶
Number of threads that execute concurrently.
Returns device value for cuda, rocm, and vulkan. Returns 1 for metal and opencl devices, regardless of the physical device. Returns remote device value for RPC devices. Returns None for all other devices.
Total amount of shared memory per block in bytes.
Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.
- property compute_version¶
Get compute version number as string.
Returns maximum API version (e.g. CUDA/OpenCL/Vulkan) supported by the device.
Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.
- property device_name¶
Return the vendor-specific name of device.
Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.
- property max_clock_rate¶
Return the max clock frequency of device (kHz).
Returns device value for cuda, rocm, and opencl. Returns remote device value for RPC devices. Returns None for all other devices.
- property multi_processor_count¶
Return the number of compute units in the device.
Returns device value for cuda, rocm, and opencl. Returns remote device value for RPC devices. Returns None for all other devices.
- property max_thread_dimensions¶
Return the maximum size of each thread axis
Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.
- Returns
dims – The maximum length of threadIdx.x, threadIdx.y, threadIdx.z
- Return type
List of int, or None
- property api_version¶
Returns version number of the SDK used to compile TVM.
For example, CUDA_VERSION for cuda or VK_HEADER_VERSION for Vulkan.
Returns device value for cuda, rocm, opencl, and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.
- property driver_version¶
Returns version number of the driver
Returns driver vendor’s internal version number. (e.g. “450.408.256” for nvidia-driver-450)
Returns device value for opencl and vulkan. Returns remote device value for RPC devices. Returns None for all other devices.
- texture_spatial_limit()¶
Returns limits for textures by spatial dimensions
- create_raw_stream()¶
Create a new runtime stream at the context.
User should free the stream after use.
- Returns
stream – The created runtime stream.
- Return type
TVMStreamHandle
- free_raw_stream(stream)¶
Free a created stream handle.
- Parameters
stream (TVMStreamHandle) – The stream which should to be released.
- set_raw_stream(stream)¶
Set a created stream handle.
- Parameters
stream (TVMStreamHandle) – The stream which should to be set to the device.
- sync(stream=None)¶
Synchronize until jobs finished at the context.
- Parameters
stream (TVMStreamHandle) – Jobs in this stream should be finished.
- class tvm.runtime.Module(handle)¶
Runtime Module.
Attributes:
Get the entry function
Get type key of the module.
Get the format of the module.
Get imported modules
Returns true if module is 'binary serializable', ie can be serialzed into binary
Returns true if module is 'runnable'.
Returns true if module is 'DSO exportable', ie can be included in result of export_library by the external compiler directly.
Methods:
implements_function
(name[, query_imports])Returns True if the module has a definition for the global function with name.
get_function
(name[, query_imports])Get function from the module.
import_module
(module)Add module to the import list of current one.
get_source
([fmt])Get source code from module, if available.
Get the runtime module property mask.
save
(file_name[, fmt])Save the module to file.
time_evaluator
(func_name, dev[, number, ...])Get an evaluator that measures time cost of running function.
export_library
(file_name[, fcompile, ...])Export the module and all imported modules into a single device library.
- property entry_func¶
Get the entry function
- Returns
f – The entry function if exist
- Return type
- implements_function(name, query_imports=False)¶
Returns True if the module has a definition for the global function with name. Note that has_function(name) does not imply get_function(name) is non-null since the module may be, eg, a CSourceModule which cannot supply a packed-func implementation of the function without further compilation. However, get_function(name) non null should always imply has_function(name).
- get_function(name, query_imports=False)¶
Get function from the module.
- Parameters
- Returns
f – The result function.
- Return type
- import_module(module)¶
Add module to the import list of current one.
- Parameters
module (tvm.runtime.Module) – The other module.
- property type_key¶
Get type key of the module.
- property format¶
Get the format of the module.
- get_source(fmt='')¶
Get source code from module, if available.
- property imported_modules¶
Get imported modules
- Returns
modules – The module
- Return type
list of Module
- get_property_mask()¶
Get the runtime module property mask. The mapping is stated in ModulePropertyMask.
- Returns
mask – Bitmask of runtime module property
- Return type
- property is_binary_serializable¶
- Returns true if module is ‘binary serializable’, ie can be serialzed into binary
stream and loaded back to the runtime module.
- Returns
b – True if the module is binary serializable.
- Return type
Bool
- property is_runnable¶
Returns true if module is ‘runnable’. ie can be executed without any extra compilation/linking steps.
- Returns
b – True if the module is runnable.
- Return type
Bool
- property is_dso_exportable¶
Returns true if module is ‘DSO exportable’, ie can be included in result of export_library by the external compiler directly.
- Returns
b – True if the module is DSO exportable.
- Return type
Bool
- save(file_name, fmt='')¶
Save the module to file.
This do not save the dependent device modules. See also export_shared
See also
runtime.Module.export_library
export the module to shared library.
- time_evaluator(func_name, dev, number=10, repeat=1, min_repeat_ms=0, limit_zero_time_iterations=100, cooldown_interval_ms=0, repeats_to_cooldown=1, f_preproc='')¶
Get an evaluator that measures time cost of running function.
- Parameters
func_name (str) – The name of the function in the module.
dev (Device) – The device we should run this function on.
number (int) – The number of times to run this function for taking average. We call these runs as one repeat of measurement.
repeat (int, optional) – The number of times to repeat the measurement. In total, the function will be invoked (1 + number x repeat) times, where the first one is warm up and will be discarded. The returned result contains repeat costs, each of which is an average of number costs.
min_repeat_ms (int, optional) – The minimum duration of one repeat in milliseconds. By default, one repeat contains number runs. If this parameter is set, the parameters number will be dynamically adjusted to meet the minimum duration requirement of one repeat. i.e., When the run time of one repeat falls below this time, the number parameter will be automatically increased.
limit_zero_time_iterations (int, optional) – The maximum number of repeats when measured time is equal to 0. It helps to avoid hanging during measurements.
cooldown_interval_ms (int, optional) – The cooldown interval in milliseconds between the number of repeats defined by repeats_to_cooldown.
repeats_to_cooldown (int, optional) – The number of repeats before the cooldown is activated.
f_preproc (str, optional) – The preprocess function name we want to execute before executing the time evaluator.
Note
The function will be invoked (1 + number x repeat) times, with the first call discarded in case there is lazy initialization.
- Returns
ftimer – The function that takes same argument as func and returns a BenchmarkResult. The ProfileResult reports repeat time costs in seconds.
- Return type
function
- export_library(file_name, fcompile=None, addons=None, workspace_dir=None, **kwargs)¶
Export the module and all imported modules into a single device library.
This function only works on host LLVM modules, other runtime::Module subclasses will work with this API but they must support implement the save and load mechanisms of modules completely including saving from streams and files. This will pack your non-shared library module into a single shared library which can later be loaded by TVM.
- Parameters
file_name (str) – The name of the shared library.
fcompile (function(target, file_list, kwargs), optional) –
The compilation function to use create the final library object during export.
For example, when fcompile=_cc.create_shared, or when it is not supplied but module is “llvm,” this is used to link all produced artifacts into a final dynamic library.
This behavior is controlled by the type of object exported. If fcompile has attribute object_format, will compile host library to that format. Otherwise, will use default format “o”.
workspace_dir (str, optional) – The path of the directory used to create the intermediate artifacts when exporting the module. If this is not provided a temporary dir will be created.
kwargs (dict, optional) – Additional arguments passed to fcompile
- Returns
result of fcompile() – If the compilation function returns an artifact it would be returned via export_library, if any.
- Return type
unknown, optional
- tvm.runtime.num_threads() int ¶
Get the number of threads in use by the TVM runtime.
- Returns
Number of threads in use.
- Return type
- class tvm.runtime.Report(calls: Sequence[Dict[str, tvm.runtime.object.Object]], device_metrics: Dict[str, Dict[str, tvm.runtime.object.Object]], configuration: Dict[str, tvm.runtime.object.Object])¶
A container for information gathered during a profiling run.
- calls¶
Per-call profiling metrics (function name, runtime, device, …).
- device_metrics¶
Per-device metrics collected over the entire run.
Methods:
csv
()Convert this profiling report into CSV format.
table
([sort, aggregate, col_sums])Generate a human-readable table
json
()Convert this profiling report into JSON format.
from_json
(s)Deserialize a report from JSON.
- csv()¶
Convert this profiling report into CSV format.
This only includes calls and not overall metrics.
- Returns
csv – calls in CSV format.
- Return type
- table(sort=True, aggregate=True, col_sums=True)¶
Generate a human-readable table
- Parameters
sort (bool) – If aggregate is true, whether to sort call frames by descending duration. If aggregate is False, whether to sort frames by order of appearancei n the program.
aggregate (bool) – Whether to join multiple calls to the same op into a single line.
col_sums (bool) – Whether to include the sum of each column.
- Returns
table – A human-readable table
- Return type
- json()¶
Convert this profiling report into JSON format.
Example output:
- Returns
json – Formatted JSON
- Return type
- tvm.runtime.convert_to_object(value, span=None)¶
Convert a Python value to corresponding object type.
- tvm.runtime.convert(value, span=None)¶
Convert value to TVM object or function.
- Parameters
value (python value) –
span (Optional[Span]) – The location of this statement in the source code.
- Returns
tvm_val – Converted value in TVM
- Return type
Object or Function
Note
This function is redirected to convert_to_object as it is widely used in the codebase. We can choose one to keep and discard the other one later.
- tvm.runtime.const(value, dtype=None, span=None)¶
construct a constant
- tvm.runtime.device(dev_type, dev_id=0)¶
Construct a TVM device with given device type and id.
- Parameters
- Returns
dev – The corresponding device.
- Return type
Examples
Device can be used to create reflection of device by string representation of the device type.
assert tvm.device("cpu", 1) == tvm.cpu(1) assert tvm.device("cuda", 0) == tvm.cuda(0)
- tvm.runtime.cpu(dev_id=0)¶
Construct a CPU device
- tvm.runtime.cuda(dev_id=0)¶
Construct a CUDA GPU device
- tvm.runtime.gpu(dev_id=0)¶
Construct a CUDA GPU device
deprecated:: 0.9.0 Use
tvm.cuda()
instead.
- tvm.runtime.opencl(dev_id=0)¶
Construct a OpenCL device
- tvm.runtime.cl(dev_id=0)¶
Construct a OpenCL device
- tvm.runtime.vulkan(dev_id=0)¶
Construct a Vulkan device
- tvm.runtime.metal(dev_id=0)¶
Construct a metal device
- tvm.runtime.mtl(dev_id=0)¶
Construct a metal device
- tvm.runtime.vpi(dev_id=0)¶
Construct a VPI simulated device
- tvm.runtime.rocm(dev_id=0)¶
Construct a ROCM device
- tvm.runtime.ext_dev(dev_id=0)¶
Construct a extension device
- Parameters
dev_id (int, optional) – The integer device id
- Returns
dev – The created device
- Return type
Note
This API is reserved for quick testing of new device by plugin device API as ext_dev.
- tvm.runtime.load_module(path, fmt='')¶
Load module from file.
- Parameters
- Returns
module – The loaded module
- Return type
Note
This function will automatically call cc.create_shared if the path is in format .o or .tar
- tvm.runtime.enabled(target)¶
Whether module runtime is enabled for target
- Parameters
target (str) – The target device type.
- Returns
enabled – Whether runtime is enabled.
- Return type
Examples
The following code checks if gpu is enabled.
>>> tvm.runtime.enabled("gpu")
- tvm.runtime.system_lib(symbol_prefix='')¶
Get system-wide library module singleton.
System lib is a global module that contains self register functions in startup. Unlike normal dso modules which need to be loaded explicitly. It is useful in environments where dynamic loading api like dlopen is banned.
To build system lib function, simply specify target option
`llvm --system-lib`
The system lib will be available as long as the result code is linked by the program.The system lib is intended to be linked and loaded during the entire life-cyle of the program. If you want dynamic loading features, use dso modules instead.
- Parameters
symbol_prefix (Optional[str]) – Optional symbol prefix that can be used for search. When we lookup a symbol symbol_prefix + name will first be searched, then the name without symbol_prefix.
- Returns
module – The system-wide library module.
- Return type
- tvm.runtime.load_static_library(path, func_names)¶
Load the .o library at path which implements functions with func_names. Unlike the generic load_module the result will remain as a static_library and will not be relinked on-the-fly into a .so library.
- class tvm.runtime.String(content)¶
TVM runtime.String object, represented as a python str.
- Parameters
content (str) – The content string used to construct the object.
- class tvm.runtime.ShapeTuple(shape)¶
TVM runtime ShapeTuple object. :param shape: The shape list used to construct the object. :type shape: list[int]
- tvm.runtime.save_param_dict(params)¶
Save parameter dictionary to binary bytes.
The result binary bytes can be loaded by the GraphModule with API “load_params”.
- Parameters
params (dict of str to NDArray) – The parameter dictionary.
- Returns
param_bytes – Serialized parameters.
- Return type
Examples
# set up the parameter dict params = {"param0": arr0, "param1": arr1} # save the parameters as byte array param_bytes = tvm.runtime.save_param_dict(params) # We can serialize the param_bytes and load it back later. # Pass in byte array to module to directly set parameters tvm.runtime.load_param_dict(param_bytes)
- tvm.runtime.load_param_dict(param_bytes)¶
Load parameter dictionary from binary bytes.
- Parameters
param_bytes (bytearray) – Serialized parameters.
- Returns
params – The parameter dictionary.
- Return type
dict of str to NDArray