tvm.testing

Utility Python functions for TVM testing

tvm.testing.local_run(mod: Module, device_type: str, args: list[np.ndarray | Tensor | int | float], evaluator_config: EvaluatorConfig | None = None, export_func: Callable[[Module, str], None] | Literal['tar', 'ndk'] = 'tar', output_format: str | None = None)

Run a TVM module on a local device.

Parameters:

  • mod (Module) – The TVM module to run.

  • device_type (str) – The device type to run the module on.

  • args (List[Union[np.ndarray, Tensor, int, float]]) – The arguments to be fed to the module.

  • evaluator_config (Optional[EvaluatorConfig]) – The evaluator configuration to use.

  • export_func (Union[Callable[Module, str], Literal["tar", "ndk"]]) – The function to export the module to a file. If callable, it must be a function that takes two arguments: the module to export and the path to export to. If “tar”, the module will be exported to a tar file. If “ndk”, the module will be exported to a shared library.

  • output_format (Optional[str]) – The format of the exported module. If not specified, it will be inferred from the export_func argument.

Returns:

  • args (List[Union[np.ndarray, Tensor, int, float]]) – The results of running the module.

  • profile_result (tvm.runtime.BenchmarkResult) – The profiling result of running the module.

tvm.testing.rpc_run(mod: Module, device_type: str, args: list[np.ndarray | Tensor | int | float], evaluator_config: EvaluatorConfig | None = None, rpc_config: RPCConfig | None = None, export_func: Callable[[Module, str], None] | Literal['tar', 'ndk'] = 'tar', output_format: str | None = None)

Run a TVM module on a remote device.

Parameters:

  • mod (Module) – The TVM module to run.

  • device_type (str) – The device type to run the module on.

  • args (List[Union[np.ndarray, Tensor, int, float]]) – The arguments to be fed to the module.

  • evaluator_config (Optional[EvaluatorConfig]) – The evaluator configuration to use.

  • rpc_config (Optional[RPCConfig]) – The RPC configuration to connect to the remote device. If not specified, the default RPC configuration will be used, which reads the following environment variables: - TVM_TRACKER_HOST - TVM_TRACKER_PORT - TVM_TRACKER_KEY

  • export_func (Union[Callable[Module, str], Literal["tar", "ndk"]]) – The function to export the module to a file. If callable, it must be a function that takes two arguments: the module to export and the path to export to. If “tar”, the module will be exported to a tar file. If “ndk”, the module will be exported to a shared library.

  • output_format (Optional[str]) – The format of the exported module. If not specified, it will be inferred from the export_func argument.

Returns:

  • args (List[Union[np.ndarray, Tensor, int, float]]) – The results of running the module.

  • profile_result (tvm.runtime.BenchmarkResult) – The profiling result of running the module.

class tvm.testing.Feature(name: str, long_name: str | None = None, cmake_flag: str | None = None, target_kind_enabled: str | None = None, compile_time_check: Callable[[], bool | str] | None = None, target_kind_hardware: str | None = None, run_time_check: Callable[[], bool | str] | None = None, parent_features: str | list[str] | None = None)

A feature that may be required to run a test.

Parameters:

  • name (str) – The short name of the feature. Should match the name in the requires_* decorator. This is applied as a mark to all tests using this feature, and can be used in pytests -m argument.

  • long_name (Optional[str]) –

    The long name of the feature, to be used in error messages.

    If None, defaults to the short name.

  • cmake_flag (Optional[str]) –

    The flag that must be enabled in the config.cmake in order to use this feature.

    If None, no flag is required to use this feature.

  • target_kind_enabled (Optional[str]) –

    The target kind that must be enabled to run tests using this feature. If present, the target_kind must appear in the TVM_TEST_TARGETS environment variable, or in tvm.testing.DEFAULT_TEST_TARGETS if TVM_TEST_TARGETS is undefined.

    If None, this feature does not require a specific target to be enabled.

  • compile_time_check (Optional[Callable[[], Union[bool,str]]]) –

    A check that returns True if the feature can be used at compile-time. (e.g. Validating the version number of the nvcc compiler.) If the feature does not have support to perform compile-time tests, the check should returns False to display a generic error message, or a string to display a more specific error message.

    If None, no additional check is performed.

  • target_kind_hardware (Optional[str]) –

    The target kind that must have available hardware in order to run tests using this feature. This is checked using tvm.device(target_kind_hardware).exist. If a feature requires a different check, this should be implemented using run_time_check.

    If None, this feature does not require a specific tvm.device to exist.

  • run_time_check (Optional[Callable[[], Union[bool,str]]]) –

    A check that returns True if the feature can be used at run-time. (e.g. Validating the compute version supported by a GPU.) If the feature does not have support to perform run-time tests, the check should returns False to display a generic error message, or a string to display a more specific error message.

    If None, no additional check is performed.

  • parent_features (Optional[Union[str,List[str]]]) –

    The short name of a feature or features that are required in order to use this feature. (e.g. Using cuDNN requires using CUDA) This feature should inherit all checks of the parent feature, with the exception of the target_kind_enabled checks.

    If None, this feature does not require any other parent features.

marks(support_required='compile-and-run')

Return a list of marks to be used

Parameters:

support_required (str) –

Allowed values: “compile-and-run” (default), “compile-only”, or “optional”.

See Feature.__call__ for details.

classmethod require(name, support_required='compile-and-run')

Returns a decorator that marks a test as requiring a feature

Parameters:

  • name (str) – The name of the feature that is used by the test

  • support_required (str) –

    Allowed values: “compile-and-run” (default), “compile-only”, or “optional”.

    See Feature.__call__ for details.

Examples

@Feature.require("cuda")
def test_compile_and_run():
    ...

@Feature.require("cuda", compile_only=True)
def test_compile_only():
    ...
class tvm.testing.Path(*args, **kwargs)

PurePath subclass that can make system calls.

Path represents a filesystem path but unlike PurePath, also offers methods to do system calls on path objects. Depending on your system, instantiating a Path will return either a PosixPath or a WindowsPath object. You can also instantiate a PosixPath or WindowsPath directly, but cannot instantiate a WindowsPath on a POSIX system or vice versa.

classmethod cwd()

Return a new path pointing to the current working directory (as returned by os.getcwd()).

classmethod home()

Return a new path pointing to the user’s home directory (as returned by os.path.expanduser(‘~’)).

samefile(other_path)

Return whether other_path is the same or not as this file (as returned by os.path.samefile()).

iterdir()

Iterate over the files in this directory. Does not yield any result for the special paths ‘.’ and ‘..’.

glob(pattern)

Iterate over this subtree and yield all existing files (of any kind, including directories) matching the given relative pattern.

rglob(pattern)

Recursively yield all existing files (of any kind, including directories) matching the given relative pattern, anywhere in this subtree.

absolute()

Return an absolute version of this path. This function works even if the path doesn’t point to anything.

No normalization is done, i.e. all ‘.’ and ‘..’ will be kept along. Use resolve() to get the canonical path to a file.

resolve(strict=False)

Make the path absolute, resolving all symlinks on the way and also normalizing it (for example turning slashes into backslashes under Windows).

stat(*, follow_symlinks=True)

Return the result of the stat() system call on this path, like os.stat() does.

owner()

Return the login name of the file owner.

group()

Return the group name of the file gid.

open(mode='r', buffering=-1, encoding=None, errors=None, newline=None)

Open the file pointed by this path and return a file object, as the built-in open() function does.

read_bytes()

Open the file in bytes mode, read it, and close the file.

read_text(encoding=None, errors=None)

Open the file in text mode, read it, and close the file.

write_bytes(data)

Open the file in bytes mode, write to it, and close the file.

write_text(data, encoding=None, errors=None, newline=None)

Open the file in text mode, write to it, and close the file.

Return the path to which the symbolic link points.

touch(mode=438, exist_ok=True)

Create this file with the given access mode, if it doesn’t exist.

mkdir(mode=511, parents=False, exist_ok=False)

Create a new directory at this given path.

chmod(mode, *, follow_symlinks=True)

Change the permissions of the path, like os.chmod().

lchmod(mode)

Like chmod(), except if the path points to a symlink, the symlink’s permissions are changed, rather than its target’s.

Remove this file or link. If the path is a directory, use rmdir() instead.

rmdir()

Remove this directory. The directory must be empty.

lstat()

Like stat(), except if the path points to a symlink, the symlink’s status information is returned, rather than its target’s.

rename(target)

Rename this path to the target path.

The target path may be absolute or relative. Relative paths are interpreted relative to the current working directory, not the directory of the Path object.

Returns the new Path instance pointing to the target path.

replace(target)

Rename this path to the target path, overwriting if that path exists.

The target path may be absolute or relative. Relative paths are interpreted relative to the current working directory, not the directory of the Path object.

Returns the new Path instance pointing to the target path.

Make this path a symlink pointing to the target path. Note the order of arguments (link, target) is the reverse of os.symlink.

Make this path a hard link pointing to the same file as target.

Note the order of arguments (self, target) is the reverse of os.link’s.

Make the target path a hard link pointing to this path.

Note this function does not make this path a hard link to target, despite the implication of the function and argument names. The order of arguments (target, link) is the reverse of Path.symlink_to, but matches that of os.link.

Deprecated since Python 3.10 and scheduled for removal in Python 3.12. Use hardlink_to() instead.

exists()

Whether this path exists.

is_dir()

Whether this path is a directory.

is_file()

Whether this path is a regular file (also True for symlinks pointing to regular files).

is_mount()

Check if this path is a POSIX mount point

Whether this path is a symbolic link.

is_block_device()

Whether this path is a block device.

is_char_device()

Whether this path is a character device.

is_fifo()

Whether this path is a FIFO.

is_socket()

Whether this path is a socket.

expanduser()

Return a new path with expanded ~ and ~user constructs (as returned by os.path.expanduser)

exception tvm.testing.TVMError
class tvm.testing.TestAttrs(name, axis=<object object>, padding=<object object>, func=<object object>)

Attrs used for testing purposes

property axis

axis field

property func

some random env function

property name
property padding

padding of input

tvm.testing.assert_allclose(actual, desired, rtol=1e-07, atol=1e-07, verbose=True)

Version of np.testing.assert_allclose with atol and rtol fields set in reasonable defaults.

Arguments actual and desired are not interchangeable, since the function compares the abs(actual-desired) with atol+rtol*abs(desired). Since we often allow desired to be close to zero, we generally want non-zero atol.

tvm.testing.assert_prim_expr_equal(lhs, rhs)

Assert lhs and rhs equals to each iother.

Parameters:

  • lhs (tvm.tirx.PrimExpr) – The left operand.

  • rhs (tvm.tirx.PrimExpr) – The left operand.

tvm.testing.check_bool_expr_is_true(bool_expr, vranges, cond=None)

Check that bool_expr holds given the condition cond for every value of free variables from vranges.

For example, 2x > 4y solves to x > 2y given x in (0, 10) and y in (0, 10). Here bool_expr is x > 2y, vranges is {x: (0, 10), y: (0, 10)}, cond is 2x > 4y. We create iterations to check:

for x in range(10):
    for y in range(10):
        assert !(2x > 4y) || (x > 2y)
Parameters:
tvm.testing.check_int_constraints_trans_consistency(constraints_trans, vranges=None)

Check IntConstraintsTransform is a bijective transformation.

Parameters:

  • constraints_trans (arith.IntConstraintsTransform) – Integer constraints transformation

  • vranges (Dict[tvm.tirx.Var, tvm.ir.Range]) – Free variables and their ranges

tvm.testing.check_numerical_grads(function, input_values, grad_values, function_value=None, delta=0.001, atol=0.01, rtol=0.1)

A helper function that checks that numerical gradients of a function are equal to gradients computed in some different way (analytical gradients).

Numerical gradients are computed using finite difference approximation. To reduce the number of function evaluations, the number of points used is gradually increased if the error value is too high (up to 5 points).

Parameters:

  • function – A function that takes inputs either as positional or as keyword arguments (either function(*input_values) or function(**input_values) should be correct) and returns a scalar result. Should accept numpy ndarrays.

  • input_values (Dict[str, numpy.ndarray] or List[numpy.ndarray]) – A list of values or a dict assigning values to variables. Represents the point at which gradients should be computed.

  • grad_values (Dict[str, numpy.ndarray] or List[numpy.ndarray]) – Gradients computed using a different method.

  • function_value (float, optional) – Should be equal to function(**input_values).

  • delta (float, optional) – A small number used for numerical computation of partial derivatives. The default 1e-3 is a good choice for float32.

  • atol (float, optional) – Absolute tolerance. Gets multiplied by sqrt(n) where n is the size of a gradient.

  • rtol (float, optional) – Relative tolerance.

tvm.testing.device_enabled(target)

Check if a target should be used when testing.

It is recommended that you use tvm.testing.parametrize_targets() instead of manually checking if a target is enabled.

This allows the user to control which devices they are testing against. In tests, this should be used to check if a device should be used when said device is an optional part of the test.

Parameters:

target (str or Dict[str, Any] or tvm.target.Target) – Target string to check against

Returns:

Whether or not the device associated with this target is enabled.

Return type:

bool

Example

>>> @tvm.testing.uses_gpu
>>> def test_mytest():
>>>     for target in ["cuda", "llvm"]:
>>>         if device_enabled(target):
>>>             test_body...

Here, test_body will only be reached by with target=”cuda” on gpu test nodes and target=”llvm” on cpu test nodes.

tvm.testing.enabled_targets()

Get all enabled targets with associated devices.

In most cases, you should use tvm.testing.parametrize_targets() instead of this function.

In this context, enabled means that TVM was built with support for this target, the target name appears in the TVM_TEST_TARGETS environment variable, and a suitable device for running this target exists. If TVM_TEST_TARGETS is not set, it defaults to variable DEFAULT_TEST_TARGETS in this module.

If you use this function in a test, you must decorate the test with tvm.testing.uses_gpu() (otherwise it will never be run on the gpu).

Returns:

targets – A list of pairs of all enabled devices and the associated context

Return type:

list

tvm.testing.exclude_targets(*args)

Exclude a test from running on a particular target.

Use this decorator when you want your test to be run over a variety of targets and devices (including cpu and gpu devices), but want to exclude some particular target or targets. For example, a test may wish to be run against all targets in tvm.testing.enabled_targets(), except for a particular target that does not support the capabilities.

Applies pytest.mark.skipif to the targets given.

Parameters:

  • f (function) – Function to parametrize. Must be of the form def test_xxxxxxxxx(target, dev):, where xxxxxxxxx is any name.

  • targets (list[str]) – Set of targets to exclude.

Example

>>> @tvm.testing.exclude_targets("cuda")
>>> def test_mytest(target, dev):
>>>     ...  # do something

Or

>>> @tvm.testing.exclude_targets("llvm", "cuda")
>>> def test_mytest(target, dev):
>>>     ...  # do something
tvm.testing.fixture(func=None, *, cache_return_value=False)

Convenience function to define pytest fixtures.

This should be used as a decorator to mark functions that set up state before a function. The return value of that fixture function is then accessible by test functions as that accept it as a parameter.

Fixture functions can accept parameters defined with tvm.testing.parameter().

By default, the setup will be performed once for each unit test that uses a fixture, to ensure that unit tests are independent. If the setup is expensive to perform, then the cache_return_value=True argument can be passed to cache the setup. The fixture function will be run only once (or once per parameter, if used with tvm.testing.parameter), and the same return value will be passed to all tests that use it. If the environment variable TVM_TEST_DISABLE_CACHE is set to a non-zero value, it will disable this feature and no caching will be performed.

Example

>>> @tvm.testing.fixture
>>> def cheap_setup():
>>>     return 5 # Setup code here.
>>>
>>> def test_feature_x(target, dev, cheap_setup)
>>>     assert(cheap_setup == 5) # Run test here

Or

>>> size = tvm.testing.parameter(1, 10, 100)
>>>
>>> @tvm.testing.fixture
>>> def cheap_setup(size):
>>>     return 5*size # Setup code here, based on size.
>>>
>>> def test_feature_x(cheap_setup):
>>>     assert(cheap_setup in [5, 50, 500])

Or

>>> @tvm.testing.fixture(cache_return_value=True)
>>> def expensive_setup():
>>>     time.sleep(10) # Setup code here
>>>     return 5
>>>
>>> def test_feature_x(target, dev, expensive_setup):
>>>     assert(expensive_setup == 5)
tvm.testing.get_dtype_range(dtype: str) → tuple[int, int]

Produces the min,max for a give data type.

Parameters:

dtype (str) – a type string (e.g., int8, float64)

Returns:

  • type_info.min (int) – the minimum of the range

  • type_info.max (int) – the maximum of the range

tvm.testing.identity_after(x, sleep)

Testing function to return identity after sleep

Parameters:

  • x (int) – The input value.

  • sleep (float) – The amount of time to sleep

Returns:

x – The original value

Return type:

object

tvm.testing.install_request_hook(depth: int) → None

Add a wrapper around urllib.request for CI tests

tvm.testing.is_ampere_or_newer()

Check if the target environment has an NVIDIA Ampere GPU or newer.

tvm.testing.known_failing_targets(*args)

Skip a test that is known to fail on a particular target.

Use this decorator when you want your test to be run over a variety of targets and devices (including cpu and gpu devices), but know that it fails for some targets. For example, a newly implemented runtime may not support all features being tested, and should be excluded.

Applies pytest.mark.xfail to the targets given.

Parameters:

  • f (function) – Function to parametrize. Must be of the form def test_xxxxxxxxx(target, dev):, where xxxxxxxxx is any name.

  • targets (list[str]) – Set of targets to skip.

Example

>>> @tvm.testing.known_failing_targets("cuda")
>>> def test_mytest(target, dev):
>>>     ...  # do something

Or

>>> @tvm.testing.known_failing_targets("llvm", "cuda")
>>> def test_mytest(target, dev):
>>>     ...  # do something
tvm.testing.parameter(*values, ids=None, by_dict=None)

Convenience function to define pytest parametrized fixtures.

Declaring a variable using tvm.testing.parameter will define a parametrized pytest fixture that can be used by test functions. This is intended for cases that have no setup cost, such as strings, integers, tuples, etc. For cases that have a significant setup cost, please use tvm.testing.fixture() instead.

If a test function accepts multiple parameters defined using tvm.testing.parameter, then the test will be run using every combination of those parameters.

The parameter definition applies to all tests in a module. If a specific test should have different values for the parameter, that test should be marked with @pytest.mark.parametrize.

Parameters:

  • values (Any) – A list of parameter values. A unit test that accepts this parameter as an argument will be run once for each parameter given.

  • ids (List[str], optional) – A list of names for the parameters. If None, pytest will generate a name from the value. These generated names may not be readable/useful for composite types such as tuples.

  • by_dict (Dict[str, Any]) – A mapping from parameter name to parameter value, to set both the values and ids.

Returns:

A function output from pytest.fixture.

Return type:

function

Example

>>> size = tvm.testing.parameter(1, 10, 100)
>>> def test_using_size(size):
>>>     ... # Test code here

Or

>>> shape = tvm.testing.parameter((5,10), (512,1024), ids=['small','large'])
>>> def test_using_size(shape):
>>>     ... # Test code here

Or

>>> shape = tvm.testing.parameter(by_dict={'small': (5,10), 'large': (512,1024)})
>>> def test_using_size(shape):
>>>     ... # Test code here
tvm.testing.parameters(*value_sets, ids=None)

Convenience function to define pytest parametrized fixtures.

Declaring a variable using tvm.testing.parameters will define a parametrized pytest fixture that can be used by test functions. Like tvm.testing.parameter(), this is intended for cases that have no setup cost, such as strings, integers, tuples, etc. For cases that have a significant setup cost, please use tvm.testing.fixture() instead.

Unlike tvm.testing.parameter(), if a test function accepts multiple parameters defined using a single call to tvm.testing.parameters, then the test will only be run once for each set of parameters, not for all combinations of parameters.

These parameter definitions apply to all tests in a module. If a specific test should have different values for some parameters, that test should be marked with @pytest.mark.parametrize.

Parameters:

  • values (List[tuple]) – A list of parameter value sets. Each set of values represents a single combination of values to be tested. A unit test that accepts parameters defined will be run once for every set of parameters in the list.

  • ids (List[str], optional) – A list of names for the parameter sets. If None, pytest will generate a name from each parameter set. These generated names may not be readable/useful for composite types such as tuples.

Returns:

Function outputs from pytest.fixture. These should be unpacked into individual named parameters.

Return type:

List[function]

Example

>>> size, dtype = tvm.testing.parameters( (16,'float32'), (512,'float16') )
>>> def test_feature_x(size, dtype):
>>>     # Test code here
>>>     assert( (size,dtype) in [(16,'float32'), (512,'float16')])
tvm.testing.parametrize_targets(*args)

Parametrize a test over a specific set of targets.

Use this decorator when you want your test to be run over a specific set of targets and devices. It is intended for use where a test is applicable only to a specific target, and is inapplicable to any others (e.g. verifying target-specific assembly code matches known assembly code). In most circumstances, tvm.testing.exclude_targets() or tvm.testing.known_failing_targets() should be used instead.

If used as a decorator without arguments, the test will be parametrized over all targets in tvm.testing.enabled_targets(). This behavior is automatically enabled for any target that accepts arguments of target or dev, so the explicit use of the bare decorator is no longer needed, and is maintained for backwards compatibility.

Parameters:

  • f (function) – Function to parametrize. Must be of the form def test_xxxxxxxxx(target, dev):, where xxxxxxxxx is any name.

  • targets (list[str], optional) – Set of targets to run against. If not supplied, tvm.testing.enabled_targets() will be used.

Example

>>> @tvm.testing.parametrize_targets("llvm", "cuda")
>>> def test_mytest(target, dev):
>>>     ...  # do something
tvm.testing.requires_cuda_compute_version(major_version, minor_version=0)

Mark a test as requiring at least a compute architecture

Unit test marked with this decorator will run only if the CUDA compute architecture of the GPU is at least (major_version, minor_version).

This also marks the test as requiring a CUDA support.

Parameters:

  • major_version (int) – The major version of the (major,minor) version tuple.

  • minor_version (int) – The minor version of the (major,minor) version tuple.

tvm.testing.requires_llvm_minimum_version(major_version)

Mark a test as requiring at least a specific version of LLVM.

Unit test marked with this decorator will run only if the installed version of LLVM is at least major_version.

This also marks the test as requiring LLVM backend support.

Parameters:

major_version (int)

tvm.testing.requires_nvcc_version(major_version, minor_version=0, release_version=0)

Mark a test as requiring at least a specific version of nvcc.

Unit test marked with this decorator will run only if the installed version of NVCC is at least (major_version, minor_version, release_version).

This also marks the test as requiring a CUDA support.

Parameters:

  • major_version (int) – The major version of the (major,minor,release) version tuple.

  • minor_version (int) – The minor version of the (major,minor,release) version tuple.

  • release_version (int) – The release version of the (major,minor,release) version tuple.

tvm.testing.requires_package(*packages)

Mark a test as requiring python packages to run.

If the packages listed are not available, tests marked with requires_package will appear in the pytest results as being skipped. This is equivalent to using foo = pytest.importorskip('foo') inside the test body.

Parameters:

packages (List[str]) – The python packages that should be available for the test to run.

Returns:

mark – The pytest mark to be applied to unit tests that require this

Return type:

pytest mark

tvm.testing.strtobool(val)

Convert a string representation of truth to true (1) or false (0). True values are ‘y’, ‘yes’, ‘t’, ‘true’, ‘on’, and ‘1’; false values are ‘n’, ‘no’, ‘f’, ‘false’, ‘off’, and ‘0’. Raises ValueError if ‘val’ is anything else.

tvm.testing.terminate_self()

Testing function to terminate the process.