Describes at compile time the constraints on where data is to be stored at runtime down to the (virtual) device and memory scope level, and how to compile code to compute that data. Used by the PlanDevices pass to collect and solve (virtual) device constraints for the whole Relay program. More...

#include <virtual_device.h>

Inheritance diagram for tvm::VirtualDeviceNode:

Collaboration diagram for tvm::VirtualDeviceNode:

Public Member Functions
DLDeviceType	device_type () const

bool	IsFullyUnconstrained () const
	Returns true if virtual device is 'fully unconstrained', ie no target/device type, device id or memory scope is specified. More...

bool	IsFullyConstrained () const
	Returns true if virtual device is 'fully constrained', ie target, device id and memory scope are all specified. More...

Device	ToDevice () const
	Returns the (virtual) `Device` implied by this `VirtualDevice`. Both the `device_type` and `virtual_device_must` be constrained. The returned `Device` may not correspond to any physical device available at compile time or even runtime: see "Virtual vs Physical Devices" above. More...

	TVM_DECLARE_ATTRS (VirtualDeviceNode, "VirtualDevice")

Public Member Functions inherited from tvm::AttrsNode< VirtualDeviceNode >
void	VisitAttrs (AttrVisitor *v)

void	VisitNonDefaultAttrs (AttrVisitor *v)
	Visit attributes that do not equal the default value. More...

void	InitByPackedArgs (const runtime::TVMArgs &args, bool allow_unknown) final
	Initialize the attributes by arguments. More...

bool	SEqualReduce (const VirtualDeviceNode *other, SEqualReducer equal) const

void	SHashReduce (SHashReducer hash_reducer) const

Array< AttrFieldInfo >	ListFieldInfo () const final
	Get the field information. More...

Public Member Functions inherited from tvm::BaseAttrsNode
virtual	~BaseAttrsNode ()
	virtual destructor More...

template<typename... Args>
void	InitBySeq (Args &&... args)
	Initialize the attributes by sequence of arguments. More...

void	PrintDocString (std::ostream &os) const
	Print readible docstring to ostream, add newline. More...

	TVM_DECLARE_BASE_OBJECT_INFO (BaseAttrsNode, Object)

Public Member Functions inherited from tvm::runtime::Object
uint32_t	type_index () const

std::string	GetTypeKey () const

size_t	GetTypeKeyHash () const

template<typename TargetType >
bool	IsInstance () const

bool	unique () const

	Object ()

	Object (const Object &other)

	Object (Object &&other)

Object &	operator= (const Object &other)

Object &	operator= (Object &&other)

Public Attributes
int	virtual_device_id
	The device identifier for the virtual device. This must be resolved to a physical device identifier either during compilation or at runtime. More...

Target	target
	The `Target` describing how to compile for the virtual device. More...

MemoryScope	memory_scope
	The scope of memory w.r.t. the virtual device which holds data. More...

Friends
class	VirtualDevice

Additional Inherited Members
Public Types inherited from tvm::BaseAttrsNode
using	TVMArgs = runtime::TVMArgs

using	TVMRetValue = runtime::TVMRetValue

Public Types inherited from tvm::runtime::Object
typedef void(*	FDeleter) (Object *self)
	Object deleter. More...

using	RefCounterType = std::atomic< int32_t >

Static Public Member Functions inherited from tvm::runtime::Object
static std::string	TypeIndex2Key (uint32_t tindex)
	Get the type key of the corresponding index from runtime. More...

static size_t	TypeIndex2KeyHash (uint32_t tindex)
	Get the type key hash of the corresponding index from runtime. More...

static uint32_t	TypeKey2Index (const std::string &key)
	Get the type index of the corresponding key from runtime. More...

static uint32_t	_GetOrAllocRuntimeTypeIndex ()

static uint32_t	RuntimeTypeIndex ()

Static Public Attributes inherited from tvm::BaseAttrsNode
static constexpr const bool	_type_has_method_sequal_reduce = true

static constexpr const bool	_type_has_method_shash_reduce = true

static constexpr const char *	_type_key = "Attrs"

Static Public Attributes inherited from tvm::runtime::Object
static constexpr const char *	_type_key = "runtime.Object"

static constexpr bool	_type_final = false

static constexpr uint32_t	_type_child_slots = 0

static constexpr bool	_type_child_slots_can_overflow = true

static constexpr bool	_type_has_method_visit_attrs = true

static constexpr bool	_type_has_method_sequal_reduce = false

static constexpr bool	_type_has_method_shash_reduce = false

static constexpr uint32_t	_type_index = TypeIndex::kDynamic

Protected Member Functions inherited from tvm::runtime::Object
void	IncRef ()
	developer function, increases reference counter. More...

void	DecRef ()
	developer function, decrease reference counter. More...

Static Protected Member Functions inherited from tvm::runtime::Object
static uint32_t	GetOrAllocRuntimeTypeIndex (const std::string &key, uint32_t static_tindex, uint32_t parent_tindex, uint32_t type_child_slots, bool type_child_slots_can_overflow)
	Get the type index using type key. More...

Protected Attributes inherited from tvm::runtime::Object
uint32_t	type_index_ {0}
	Type index(tag) that indicates the type of the object. More...

RefCounterType	ref_counter_ {0}
	The internal reference counter. More...

FDeleter	deleter_ = nullptr
	deleter of this object to enable customized allocation. If the deleter is nullptr, no deletion will be performed. The creator of the object must always set the deleter field properly. More...

Detailed Description

Describes at compile time the constraints on where data is to be stored at runtime down to the (virtual) device and memory scope level, and how to compile code to compute that data. Used by the PlanDevices pass to collect and solve (virtual) device constraints for the whole Relay program.

Is a quadruple of:

A device_type (DLDeviceType). May be kInvalidDeviceType if unconstrained.
A virtual_device_id (int). This allows us to distinguish distinct devices with the same Target, for example in a multi-GPU system. May be -1 if unconstrained. See "Virtual Devices" below.
A target (Target) describing how to compile code for the intended device. May be null if unconstrained.
A memory_scope (MemoryScope, which is currently just String) describing which memory area is to be used to hold data. May be "" if unconstrained. See "Memory Scopes and Devices" below.

Some or all of these fields may be unconstrained, signaling that device planning is free to choose a value consistent with the whole program. However if a target is given then the device_type must equal target->GetTargetDeviceType().

Note that currently we assume if a function returns its result on a particular (virtual) device then the function body is also executed on that device. See the overview comment in src/relay/transforms/device_planner.cc for more details.

By 'data' we include both tensors and additional supporting datastructures such as shapes, Relay ADT items (including tuples), Relay references, and Relay closures. Typically non-tensor data must reside on a 'CPU'-like host device with good support for scalars.

By 'execution' we include both (fused) primitive operators, and all the Relay expressions surrounding them which coordinates data and control flow. Again, typically non-primitive operators must be executed on a 'CPU'-like device with good support for control flow.

Since TVM targets such a wide range of systems it is not possible for VirtualDevice to impose much semantics on these fields, particularly for virtual_device_id and memory_scope. Instead we assume downstream passes and codegen will interpret an validate these fields appropriately.

Targets vs Devices

Generally Targets (a compile-time only datastructue) describe compiler options for a specific microarchitecture and toolchain, while Devices (a runtime datastructure also available at compile time) describe a physical device on the target system. Obviously the target must agree with the device's microarchitecture, but we otherwise don't impose any constraints between them:

It's ok to use different Targets for the same Device, eg to squeeze some extra perf out of a particular primitive using particular compiler flags.
It's ok to use the same Target for multiple Devices, eg if we have multiple CPUs.

Traditionally TVM assumes at most one Target per DLDeviceType. We are moving away from that assumption.

Virtual vs Physical Devices

The virtual_device_id may be used by downstream passes or the runtime to help decide which device_id to use for a particular physical runtime Device. For example:

Some runtimes may support passing in an array of actual device specifications, and the virtual_device_id can be used at runtime as an index into that array.
Some runtimes may support dynamically allocating computations to physical devices. On these systems a large space of virtual_device_ids could be used at compile time, even though at runtime only a few physical devices will be present.

The virtual_device_id may also be left unconstrained if not needed.

Memory Scopes and Devices

Multi-device systems can have complex memory hierarchies. For example

(kDLCPU, 0, "llvm", "global")

and

(kDLCPU, 1, "llvm", "global")

could denote:

The same memory area accessible from two separate CPUs without any CPU affinity;
Distinct memory areas in a NUMA architecture for which cross-device access is handled by the memory system;
Outright distinct memory areas, where one device cannot directly address the memory of another.

Similarly:

(kDLCPU, 0, "llvm", "global")

and

(kDLCUDA, 0, "cuda", "host")

could denote the same memory area, but with very different access costs.

Furthermore, not all memory scopes are accessible to all devices, and it is possible for a memory scope to only be accessible to a device when code is compiled with particular Target options.

VirtualDevices themselves have no system-level understanding. Currently the PlanDevices pass will simply insert "device_copy" operators wherever VirtualDevices are not exactly pointwise equal. We may revisit this in the future as the work on memory pools matures.

Joining and Defaulting

It is possible to 'join' two VirtualDevices to yield the most constrained VirtualDevice which agrees with both join arguments. Eg:

Join((kDLCPU, -1, "llvm", ""), (kInvalidDeviceType, 3, null, "global))
  => (kDLCPU, 3, "llvm", "global")
Join((kDLCPU, -1, "llvm", ""), (kInvalidDeviceType, 3, null, "local))
  => null (no join possible)

Related to 'join' is 'default', which only takes constrained fields from the rhs when the lhs is unconstrained:

Default(kDLCPU, -1, "llvm", "local"), (kDLCPU, 3, null, "global"))

=> (kDLCPU, 3, "llvm", "local")

These operations are needed during device planning.

Member Function Documentation

◆ device_type()

DLDeviceType tvm::VirtualDeviceNode::device_type ( ) const

inline

◆ IsFullyConstrained()

bool tvm::VirtualDeviceNode::IsFullyConstrained ( ) const

inline

Returns true if virtual device is 'fully constrained', ie target, device id and memory scope are all specified.

◆ IsFullyUnconstrained()

bool tvm::VirtualDeviceNode::IsFullyUnconstrained ( ) const

inline

Returns true if virtual device is 'fully unconstrained', ie no target/device type, device id or memory scope is specified.

◆ ToDevice()

Device tvm::VirtualDeviceNode::ToDevice ( ) const

inline

Returns the (virtual) Device implied by this VirtualDevice. Both the device_type and virtual_device_must be constrained. The returned Device may not correspond to any physical device available at compile time or even runtime: see "Virtual vs Physical Devices" above.

◆ TVM_DECLARE_ATTRS()

tvm::VirtualDeviceNode::TVM_DECLARE_ATTRS	(	VirtualDeviceNode	,
		"VirtualDevice"
	)

inline

Friends And Related Function Documentation

◆ VirtualDevice

friend class VirtualDevice

friend

Member Data Documentation

◆ memory_scope

MemoryScope tvm::VirtualDeviceNode::memory_scope

The scope of memory w.r.t. the virtual device which holds data.

Empty denotes unconstrained.

◆ target

Target tvm::VirtualDeviceNode::target

The Target describing how to compile for the virtual device.

Null denotes unconstrained. Note that if a target later becomes known for this VirtualDevice then it must be consistent with the device_type if already known. This is enforced by the Join and Default methods.

◆ virtual_device_id

int tvm::VirtualDeviceNode::virtual_device_id

The device identifier for the virtual device. This must be resolved to a physical device identifier either during compilation or at runtime.

-1 denotes unconstrained.

The documentation for this class was generated from the following file:

include/tvm/target/virtual_device.h

Public Member Functions

Public Attributes

Friends

Additional Inherited Members

Detailed Description

Targets vs Devices

Virtual vs Physical Devices

Memory Scopes and Devices

Joining and Defaulting

Member Function Documentation

◆ device_type()

◆ IsFullyConstrained()

◆ IsFullyUnconstrained()

◆ ToDevice()

◆ TVM_DECLARE_ATTRS()

Friends And Related Function Documentation

◆ VirtualDevice

Member Data Documentation

◆ memory_scope

◆ target

◆ virtual_device_id