TVM Runtime Device API, abstracts the device specific interface for memory management. More...

#include <device_api.h>

Collaboration diagram for tvm::runtime::DeviceAPI:

Public Member Functions
virtual	~DeviceAPI ()
	virtual destructor More...

virtual void	SetDevice (Device dev)=0
	Set the environment device id to device. More...

virtual void	GetAttr (Device dev, DeviceAttrKind kind, ffi::Any *rv)=0
	Get attribute of specified device. More...

virtual size_t	GetDataSize (const DLTensor &arr, ffi::Optional< ffi::String > mem_scope=std::nullopt)
	Get the physical memory size required. More...

virtual void	GetTargetProperty (Device dev, const std::string &property, ffi::Any *rv)
	Query the device for specified properties. More...

virtual void *	AllocDataSpace (Device dev, size_t nbytes, size_t alignment, DLDataType type_hint)=0
	Allocate a data space on device. More...

virtual void *	AllocDataSpace (Device dev, int ndim, const int64_t *shape, DLDataType dtype, ffi::Optional< ffi::String > mem_scope=std::nullopt)
	Allocate a data space on device with memory scope support. More...

virtual void	FreeDataSpace (Device dev, void *ptr)=0
	Free a data space on device. More...

virtual void	CopyDataFromTo (DLTensor from, DLTensor to, TVMStreamHandle stream)
	copy data from one place to another More...

virtual TVMStreamHandle	CreateStream (Device dev)
	Create a new stream of execution. More...

virtual void	FreeStream (Device dev, TVMStreamHandle stream)
	Free a stream of execution. More...

virtual void	StreamSync (Device dev, TVMStreamHandle stream)=0
	Synchronize the stream. More...

virtual void	SetStream (Device dev, TVMStreamHandle stream)
	Set the stream. More...

virtual TVMStreamHandle	GetCurrentStream (Device dev)
	Get the current stream. More...

virtual void	SyncStreamFromTo (Device dev, TVMStreamHandle event_src, TVMStreamHandle event_dst)
	Synchronize 2 streams of execution. More...

virtual void *	AllocWorkspace (Device dev, size_t nbytes, DLDataType type_hint={})
	Allocate temporal workspace for backend execution. More...

virtual void	FreeWorkspace (Device dev, void *ptr)
	Free temporal workspace in backend execution. More...

virtual bool	SupportsDevicePointerArithmeticsOnHost ()
	Whether pointer arithmetics on a device owned pointer may be performed on the host. More...

Static Public Member Functions
static DeviceAPI *	Get (Device dev, bool allow_missing=false)
	Get device API based on device. More...

static bool	NeedSetDevice (int device_type)
	Whether a certian device type requires set device device before launching the kernel function. More...

Protected Member Functions
virtual void	CopyDataFromTo (const void from, size_t from_offset, void to, size_t to_offset, size_t num_bytes, Device dev_from, Device dev_to, DLDataType type_hint, TVMStreamHandle stream)
	copy data from one place to another More...

Detailed Description

TVM Runtime Device API, abstracts the device specific interface for memory management.

Constructor & Destructor Documentation

◆ ~DeviceAPI()

virtual tvm::runtime::DeviceAPI::~DeviceAPI ( )

inlinevirtual

virtual destructor

Member Function Documentation

◆ AllocDataSpace() [1/2]

virtual void* tvm::runtime::DeviceAPI::AllocDataSpace	(	Device	dev,
		int	ndim,
		const int64_t *	shape,
		DLDataType	dtype,
		ffi::Optional< ffi::String >	mem_scope = `std::nullopt`
	)

virtual

Allocate a data space on device with memory scope support.

Parameters

dev	The device device to perform operation.
ndim	The number of dimension of allocated tensor.
shape	The shape of allocated tensor.
dtype	The type of elements.
mem_scope	The memory scope of allocated tensor.

Returns: The allocated device pointer.

◆ AllocDataSpace() [2/2]

virtual void* tvm::runtime::DeviceAPI::AllocDataSpace	(	Device	dev,
		size_t	nbytes,
		size_t	alignment,
		DLDataType	type_hint
	)

pure virtual

Allocate a data space on device.

Parameters

dev	The device device to perform operation.
nbytes	The number of bytes in memory.
alignment	The alignment of the memory.
type_hint	The type of elements. Only needed by certain backends such as OpenGL, as nbytes & alignment are sufficient for most backends.

Returns: The allocated device pointer.

◆ AllocWorkspace()

virtual void* tvm::runtime::DeviceAPI::AllocWorkspace	(	Device	dev,
		size_t	nbytes,
		DLDataType	type_hint = `{}`
	)

virtual

Allocate temporal workspace for backend execution.

Note: We have the following assumption about backend temporal workspace allocation, and backend will optimize for such assumption:

Only a few allocation will happen, and space will be released after use.
The release order is usually in reverse order of allocate (stack style).
Repeative pattern of same allocations over different runs.
Workspace should not overlap between different threads(i.e. be threadlocal)

Parameters

dev	The device of allocation.
nbytes	The size to be allocated.
type_hint	The type of elements. Only needed by certain backends such as OpenGL, as nbytes is sufficient for most backends.

◆ CopyDataFromTo() [1/2]

virtual void tvm::runtime::DeviceAPI::CopyDataFromTo	(	const void *	from,
		size_t	from_offset,
		void *	to,
		size_t	to_offset,
		size_t	num_bytes,
		Device	dev_from,
		Device	dev_to,
		DLDataType	type_hint,
		TVMStreamHandle	stream
	)

protectedvirtual

copy data from one place to another

Parameters

from	The source array.
from_offset	The byte offeset in the from.
to	The target array.
to_offset	The byte offset in the to.
num_bytes	The size of the memory in bytes
dev_from	The source device
dev_to	The target device
type_hint	The type of elements, only neded by certain backends. can be useful for cross device endian converison.
stream	Optional stream object.

◆ CopyDataFromTo() [2/2]

virtual void tvm::runtime::DeviceAPI::CopyDataFromTo	(	DLTensor *	from,
		DLTensor *	to,
		TVMStreamHandle	stream
	)

virtual

copy data from one place to another

Note: This API is designed to support special memory with shape dependent layout. We pass in DLTensor* with shape information to support these cases.

Parameters

from	The source array.
to	The target array.
stream	Optional stream object.

Note: The copy may happen asynchronously if it involves a GPU context. Call StreamSync to ensure the copy completes from host's pov.

◆ CreateStream()

virtual TVMStreamHandle tvm::runtime::DeviceAPI::CreateStream ( Device dev )

virtual

Create a new stream of execution.

Parameters

dev	The device of allocation.

◆ FreeDataSpace()

virtual void tvm::runtime::DeviceAPI::FreeDataSpace	(	Device	dev,
		void *	ptr
	)

pure virtual

Free a data space on device.

Parameters

dev	The device device to perform operation.
ptr	The data space.

◆ FreeStream()

virtual void tvm::runtime::DeviceAPI::FreeStream	(	Device	dev,
		TVMStreamHandle	stream
	)

virtual

Free a stream of execution.

Parameters

dev	The device of the stream
stream	The pointer to be freed.

◆ FreeWorkspace()

virtual void tvm::runtime::DeviceAPI::FreeWorkspace	(	Device	dev,
		void *	ptr
	)

virtual

Free temporal workspace in backend execution.

Parameters

dev	The device of allocation.
ptr	The pointer to be freed.

◆ Get()

static DeviceAPI* tvm::runtime::DeviceAPI::Get	(	Device	dev,
		bool	allow_missing = `false`
	)

static

Get device API based on device.

Parameters

dev	The device
allow_missing	Whether allow missing

Returns: The corresponding device API.

◆ GetAttr()

virtual void tvm::runtime::DeviceAPI::GetAttr	(	Device	dev,
		DeviceAttrKind	kind,
		ffi::Any *	rv
	)

pure virtual

Get attribute of specified device.

Parameters

dev	The device device
kind	The result kind
rv	The return value.

See also: DeviceAttrKind

◆ GetCurrentStream()

virtual TVMStreamHandle tvm::runtime::DeviceAPI::GetCurrentStream ( Device dev )

virtual

Get the current stream.

Parameters

dev	The device to get stream.

Returns: The current stream of the device.

◆ GetDataSize()

virtual size_t tvm::runtime::DeviceAPI::GetDataSize	(	const DLTensor &	arr,
		ffi::Optional< ffi::String >	mem_scope = `std::nullopt`
	)

virtual

Get the physical memory size required.

Parameters

arr	the tensor object.
mem_scope	the memory scope if any

Returns: the memory size.

◆ GetTargetProperty()

virtual void tvm::runtime::DeviceAPI::GetTargetProperty	(	Device	dev,
		const std::string &	property,
		ffi::Any *	rv
	)

inlinevirtual

Query the device for specified properties.

This is used to expand "-from_device=N" in the target string to all properties that can be determined from that device.

◆ NeedSetDevice()

static bool tvm::runtime::DeviceAPI::NeedSetDevice ( int device_type )

inlinestatic

Whether a certian device type requires set device device before launching the kernel function.

Parameters

device_type The device type.

◆ SetDevice()

virtual void tvm::runtime::DeviceAPI::SetDevice ( Device dev )

pure virtual

Set the environment device id to device.

Parameters

dev	The device to be set.

◆ SetStream()

virtual void tvm::runtime::DeviceAPI::SetStream	(	Device	dev,
		TVMStreamHandle	stream
	)

virtual

Set the stream.

Parameters

dev	The device to set stream.
stream	The stream to be set.

◆ StreamSync()

virtual void tvm::runtime::DeviceAPI::StreamSync	(	Device	dev,
		TVMStreamHandle	stream
	)

pure virtual

Synchronize the stream.

Parameters

dev	The device to perform operation.
stream	The stream to be sync.

◆ SupportsDevicePointerArithmeticsOnHost()

virtual bool tvm::runtime::DeviceAPI::SupportsDevicePointerArithmeticsOnHost ( )

inlinevirtual

Whether pointer arithmetics on a device owned pointer may be performed on the host.

◆ SyncStreamFromTo()

virtual void tvm::runtime::DeviceAPI::SyncStreamFromTo	(	Device	dev,
		TVMStreamHandle	event_src,
		TVMStreamHandle	event_dst
	)

virtual

Synchronize 2 streams of execution.

An event is created in event_src stream that the second then stream waits on. Neither event_src or event_dst need to be of the same device ID as the device, but they must be of the same device type.

Parameters

dev	The device of the streams.
event_src	The source stream to synchronize.
event_dst	The destination stream to synchronize.

The documentation for this class was generated from the following file:

include/tvm/runtime/device_api.h

Public Member Functions

Static Public Member Functions

Protected Member Functions

Detailed Description

Constructor & Destructor Documentation

◆ ~DeviceAPI()

Member Function Documentation

◆ AllocDataSpace() [1/2]

◆ AllocDataSpace() [2/2]

◆ AllocWorkspace()

◆ CopyDataFromTo() [1/2]

◆ CopyDataFromTo() [2/2]

◆ CreateStream()

◆ FreeDataSpace()

◆ FreeStream()

◆ FreeWorkspace()

◆ Get()

◆ GetAttr()

◆ GetCurrentStream()

◆ GetDataSize()

◆ GetTargetProperty()

◆ NeedSetDevice()

◆ SetDevice()

◆ SetStream()

◆ StreamSync()

◆ SupportsDevicePointerArithmeticsOnHost()

◆ SyncStreamFromTo()