ABI Overview

ABI Overview#

Hint

Authoritative ABI specifications are defined in

The TVM-FFI ABI is designed around the following key principles:

  • Minimal and efficient. Keep things simple and deliver close-to-metal performance.

  • Stability guarantee. The ABI remains stable across compiler versions and is independent of host languages or frameworks.

  • Expressive for machine learning. Native support for tensors, shapes, and data types commonly used in ML workloads.

  • Extensible. The ABI supports user-defined types and features through a dynamic type registration system.

This tutorial covers common concepts and usage patterns of the TVM-FFI ABI, with low-level C code examples for precise reference.

Important

C code is used for clarity, precision and friendliness to compiler builders. And C code can be readily translated into code generators such as LLVM IR builder.

Any and AnyView#

See also

Any and AnyView for Any and AnyView usage patterns.

At the core of TVM-FFI is TVMFFIAny, a 16-byte tagged union that can hold any value recognized by the FFI system. It enables type-erased value passing across language boundaries.

C ABI Reference: TVMFFIAny
tvm/ffi/c_api.h#
/*!
 * \brief C-based type of all on stack Any value.
 *
 * Any value can hold on stack values like int,
 * as well as reference counted pointers to object.
 */
typedef struct {
  /*!
   * \brief type index of the object.
   * \note The type index of Object and Any are shared in FFI.
   */
  int32_t type_index;
#if !defined(TVM_FFI_DOXYGEN_MODE)
  union {  // 4 bytes
#endif
    /*! \brief padding, must set to zero for values other than small string. */
    uint32_t zero_padding;
    /*!
     * \brief Length of small string, with a max value of 7.
     *
     * We keep small str to start at next 4 bytes to ensure alignment
     * when accessing the small str content.
     */
    uint32_t small_str_len;
#if !defined(TVM_FFI_DOXYGEN_MODE)
  };
#endif
#if !defined(TVM_FFI_DOXYGEN_MODE)
  union {  // 8 bytes
#endif
    /*! \brief integers */
    int64_t v_int64;
    /*! \brief floating-point numbers */
    double v_float64;
    /*! \brief typeless pointers */
    void* v_ptr;
    /*! \brief raw C-string */
    const char* v_c_str;
    /*! \brief ref counted objects */
    TVMFFIObject* v_obj;
    /*! \brief data type */
    DLDataType v_dtype;
    /*! \brief device */
    DLDevice v_device;
    /*! \brief small string */
    char v_bytes[8];
    /*! \brief uint64 repr mainly used for hashing */
    uint64_t v_uint64;
#if !defined(TVM_FFI_DOXYGEN_MODE)
  };
#endif
} TVMFFIAny;

Ownership. TVMFFIAny struct can represent either an owning or a borrowing reference. These two ownership patterns are formalized by the C++ wrapper classes Any and AnyView, which have identical memory layouts but different ownership semantics:

Note

To convert a borrowing AnyView to an owning Any, use TVMFFIAnyViewToOwnedAny().

Runtime Type Index. The type_index field identifies what kind of value is stored:

Important

The TVM-FFI type index system does not rely on C++ RTTI.

Construct Any#

From atomic POD types. The following C code constructs a TVMFFIAny from an integer:

TVMFFIAny Any_AnyView_FromInt(int64_t value) {
  TVMFFIAny any;
  any.type_index = kTVMFFIInt;
  any.zero_padding = 0;
  any.v_int64 = value;
  return any;
}

TVMFFIAny Any_AnyView_FromFloat(double value) {
  TVMFFIAny any;
  any.type_index = kTVMFFIFloat;
  any.zero_padding = 0;
  any.v_float64 = value;
  return any;
}

Set the type_index from TVMFFITypeIndex and assign the corresponding payload field.

Important

Always zero the zero_padding field and any unused bytes in the value union. This invariant enables direct byte comparison and hashing of TVMFFIAny values.

From object types. The following C code constructs a TVMFFIAny from a heap-allocated object:

TVMFFIAny Any_AnyView_FromObjectPtr(TVMFFIObject* obj) {
  TVMFFIAny any;
  assert(obj != NULL);
  any.type_index = kTVMFFIObject;
  any.zero_padding = 0;
  any.v_obj = obj;
  // Increment refcount if it's Any (owning) instead of AnyView (borrowing)
  if (IS_OWNING_ANY) {
    TVMFFIObjectIncRef(obj);
  }
  return any;
}

When IS_OWNING_ANY is true (owning Any), this increments the object’s reference count.

Destruct Any#

The following C code destroys a TVMFFIAny:

void Any_AnyView_Destroy(TVMFFIAny* any) {
  if (IS_OWNING_ANY) {
    // Checks if `any` holds a heap-allocated object,
    // and if so, decrements the reference count
    if (any->type_index >= kTVMFFIStaticObjectBegin) {
      TVMFFIObjectDecRef(any->v_obj);
    }
  }
  *any = (TVMFFIAny){0};  // Clears the `any` struct
}

When IS_OWNING_ANY is true (owning Any), this decrements the object’s reference count.

Extract from Any#

Extract an atomic POD. The following C code extracts an integer or float from a TVMFFIAny:

int64_t Any_AnyView_GetInt(const TVMFFIAny* any) {
  if (any->type_index == kTVMFFIInt || any->type_index == kTVMFFIBool) {
    return any->v_int64;
  } else if (any->type_index == kTVMFFIFloat) {
    return (int64_t)(any->v_float64);
  }
  assert(0);  // FAILED to read int
  return 0;
}

double Any_AnyView_GetFloat(const TVMFFIAny* any) {
  if (any->type_index == kTVMFFIInt || any->type_index == kTVMFFIBool) {
    return (double)(any->v_int64);
  } else if (any->type_index == kTVMFFIFloat) {
    return any->v_float64;
  }
  assert(0);  // FAILED to read float
  return 0.0;
}

Implicit type conversion may occur. For example, when extracting a float from a TVMFFIAny that holds an integer, the integer is cast to a float.

Extract a DLTensor. A DLTensor may originate from either a raw pointer or a heap-allocated TensorObj:

DLTensor* Any_AnyView_GetDLTensor(const TVMFFIAny* value) {
  if (value->type_index == kTVMFFIDLTensorPtr) {
    return (DLTensor*)(value->v_ptr);
  } else if (value->type_index == kTVMFFITensor) {
    return (DLTensor*)((char*)(value->v_obj) + sizeof(TVMFFIObject));
  }
  assert(0);  // FAILED to read DLTensor
  return NULL;
}

Extract a TVM-FFI object. TVM-FFI objects are always heap-allocated and reference-counted, with type_index >= kTVMFFIStaticObjectBegin:

TVMFFIObject* Any_AnyView_GetObject(const TVMFFIAny* value) {
  if (value->type_index == kTVMFFINone) {
    return NULL;  // Handling nullptr if needed
  } else if (value->type_index >= kTVMFFIStaticObjectBegin) {
    return value->v_obj;
  }
  assert(0);  // FAILED: not a TVM-FFI object
  return NULL;
}

To take ownership of the returned value, increment the reference count via TVMFFIObjectIncRef(). Release ownership later via TVMFFIObjectDecRef().

Object#

See also

Object and Class for the object system and reflection.

TVM-FFI Object (TVMFFIObject) is the cornerstone of TVM-FFI’s stable yet extensible type system.

C ABI Reference: TVMFFIObject
tvm/ffi/c_api.h#
/*!
 * \brief C-based type of all FFI object header that allocates on heap.
 */
typedef struct {
  /*!
   * \brief Combined strong and weak reference counter of the object.
   *
   * Strong ref counter is packed into the lower 32 bits.
   * Weak ref counter is packed into the upper 32 bits.
   *
   * It is equivalent to { uint32_t strong_ref_count, uint32_t weak_ref_count }
   * in little-endian structure:
   *
   * - strong_ref_count: `combined_ref_count & 0xFFFFFFFF`
   * - weak_ref_count: `(combined_ref_count >> 32) & 0xFFFFFFFF`
   *
   * Rationale: atomic ops on strong ref counter remains the same as +1/-1,
   * this combined ref counter allows us to use u64 atomic once
   * instead of a separate atomic read of weak counter during deletion.
   *
   * The ref counter goes first to align ABI with most intrusive ptr designs.
   * It is also likely more efficient as rc operations can be quite common.
   */
  uint64_t combined_ref_count;
  /*!
   * \brief type index of the object.
   * \note The type index of Object and Any are shared in FFI.
   */
  int32_t type_index;
  /*! \brief Extra padding to ensure 8 bytes alignment. */
  uint32_t __padding;
#if !defined(TVM_FFI_DOXYGEN_MODE)
  union {
#endif
    /*!
     * \brief Deleter to be invoked when strong reference counter goes to zero.
     * \param self The self object handle.
     * \param flags The flags to indicate deletion behavior.
     * \sa TVMFFIObjectDeleterFlagBitMask
     */
    void (*deleter)(void* self, int flags);
    /*!
     * \brief auxilary field to TVMFFIObject is always 8 bytes aligned.
     * \note This helps us to ensure cross platform compatibility.
     */
    int64_t __ensure_align;
#if !defined(TVM_FFI_DOXYGEN_MODE)
  };
#endif
} TVMFFIObject;

All TVM-FFI objects share these characteristics:

  • Heap-allocated and reference-counted

  • Layout-stable 24-byte header containing reference counts, type index, and deleter callback

  • Type index >= kTVMFFIStaticObjectBegin

Dynamic Type System. Classes can be registered at runtime via TVMFFITypeGetOrAllocIndex(), with support for single inheritance. See Type Checking and Casting for usage details.

A small static section between kTVMFFIStaticObjectBegin and kTVMFFIDynObjectBegin is reserved for static object types, for example,

Ownership Management#

Ownership is managed via reference counting, which includes both strong and weak references. Two C APIs manage strong reference counting:

The deleter callback (TVMFFIObject::deleter) executes when the strong or weak count reaches zero with different flags. See Reference Counting for details.

Move ownership from Any/AnyView. The following C code transfers ownership from an owning Any to an object pointer:

void Object_MoveFromAny(TVMFFIAny* any, TVMFFIObject** obj) {
  assert(any->type_index >= kTVMFFIStaticObjectBegin);
  *obj = any->v_obj;
  (*any) = (TVMFFIAny){0};
  if (!IS_OWNING_ANY) {
    TVMFFIObjectIncRef(*obj);
  }
}

Since AnyView is non-owning (IS_OWNING_ANY is false), acquiring ownership requires explicitly incrementing the reference count.

Release ownership. The following C code releases ownership of a TVM-FFI object:

void Object_Destroy(TVMFFIObject* obj) {
  assert(obj != NULL);
  TVMFFIObjectDecRef(obj);
}

Inheritance Checking#

TVM-FFI models single inheritance as a tree where each node points to its parent. Each type has a unique type index, and the system tracks ancestors, inheritance depth, and other metadata. This information is available via TVMFFIGetTypeInfo().

The following C code checks whether a type is a subclass of another:

int Object_IsInstance(int32_t sub_type_index, int32_t super_type_index, int32_t super_type_depth) {
  const TVMFFITypeInfo* sub_type_info = NULL;
  // Everything is a subclass of object.
  if (sub_type_index == super_type_index) {
    return 1;
  }
  // Invariance: parent index is always smaller than the child.
  if (sub_type_index < super_type_index) {
    return 0;
  }
  sub_type_info = TVMFFIGetTypeInfo(sub_type_index);
  return sub_type_info->type_depth > super_type_depth &&
         sub_type_info->type_ancestors[super_type_depth]->type_index == super_type_index;
}

Tensor#

See also

Tensor and DLPack for details about TVM-FFI tensors and DLPack interoperability.

TVM-FFI provides tvm::ffi::TensorObj, a DLPack-native tensor class that is also a standard TVM-FFI object. This means tensors can be managed using the same reference counting mechanisms as other objects.

C ABI Reference: tvm::ffi::TensorObj
tvm/ffi/container/tensor.h#
 class TensorObj : public Object, public DLTensor {
  // no other members besides those from Object and DLTensor
 };

Access Tensor Metadata#

The following C code obtains a DLTensor pointer from a TensorObj:

DLTensor* Tensor_AccessDLTensor(TVMFFIObject* tensor) {
  assert(tensor != NULL);
  return (DLTensor*)((char*)tensor + sizeof(TVMFFIObject));
}

The DLTensor pointer provides access to shape, dtype, device, data pointer, and other tensor metadata.

Construct Tensor#

Zero-copy conversion. The following C code constructs a TensorObj from a DLManagedTensorVersioned, which shares the underlying data buffer without allocating new memory.

TVMFFIObject* Tensor_FromDLPack(DLManagedTensorVersioned* from) {
  int err_code = 0;
  TVMFFIObject* out = NULL;
  err_code = TVMFFITensorFromDLPackVersioned(  //
      from,                                    // input DLPack tensor
      /*require_alignment=*/0,                 // no alignment requirement
      /*require_contiguous=*/1,                // require contiguous tensor
      (void**)(&out));
  assert(err_code == 0);
  return out;
}

Hint

TVM-FFI’s Python API automatically wraps framework tensors (e.g., torch.Tensor) as TensorObj, so manual conversion is typically unnecessary.

Allocate new memory. Alternatively, if memory allocation is intended, the following C code constructs a TensorObj from a DLTensor pointer:

TVMFFIObject* Tensor_Alloc(DLTensor* prototype) {
  int err_code = 0;
  TVMFFIObject* out = NULL;
  assert(prototype->data == NULL);
  err_code = TVMFFIEnvTensorAlloc(prototype, (void**)(&out));
  assert(err_code == 0);
  return out;
}

The prototype contains the shape, dtype, device, and other tensor metadata that will be used to allocate the new tensor. And the allocator, by default, is the framework’s (e.g., PyTorch) allocator, which is automatically set when importing the framework.

To override or explicitly look up the allocator, use TVMFFIEnvSetDLPackManagedTensorAllocator() and TVMFFIEnvGetDLPackManagedTensorAllocator().

Warning

In kernel library usecases, it is usually not recommended to dynamically allocate tensors inside a kernel, and instead always pre-allocate outputs, and pass them as TensorView parameters. This approach

  • avoids memory fragmentation and performance pitfalls,

  • prevents CUDA graph incompatibilities on GPU, and

  • allows the outer framework to control allocation policy (pools, device strategies, etc.).

Destruct Tensor#

As a standard TVM-FFI object, TensorObj follows the standard destruction pattern. When the reference count reaches zero, the deleter callback (TVMFFIObject::deleter) executes.

Export Tensor to DLPack#

To share a TensorObj with other frameworks, export it as a DLManagedTensorVersioned:

DLManagedTensorVersioned* Tensor_ToDLPackVersioned(TVMFFIObject* tensor) {
  int err_code = 0;
  DLManagedTensorVersioned* out = NULL;
  err_code = TVMFFITensorToDLPackVersioned(tensor, &out);
  assert(err_code == 0);
  return out;
}

Note that the caller takes ownership of the returned DLManagedTensorVersioned* and must call its deleter to release the tensor.

Function#

See also

Function for a detailed description of TVM-FFI functions.

All functions in TVM-FFI follow a unified C calling convention that enables ABI-stable, type-erased, and cross-language function calls, defined by TVMFFISafeCallType.

Calling convention. The signature includes:

  • handle (void*): Optional resource handle passed to the callee; typically NULL for exported symbols

  • args (TVMFFIAny*) and num_args (int): Array of non-owning AnyView input arguments

  • result (TVMFFIAny*): Owning Any output value

  • Return value: 0 for success; -1 or -2 for errors (see Exception)

See Calling Convention for more details.

Important

The caller must zero-initialize the output argument result before the call.

Memory layout. The FunctionObj stores call pointers after the object header.

C ABI Reference: TVMFFIFunctionCell
tvm/ffi/c_api.h#
/*!
 * \brief Object cell for function object following header.
 */
typedef struct {
  /*! \brief A C API compatible call with exception catching. */
  TVMFFISafeCallType safe_call;
  /*!
   * \brief A function pointer to an underlying cpp call.
   *
   * The signature is the same as TVMFFISafeCallType except the return type is void,
   * and the function throws exception directly instead of returning error code.
   * We use void* here to avoid depending on c++ compiler.
   *
   * This pointer should be set to NULL for functions that are not originally created in cpp.
   *
   * \note The caller must assume the same cpp exception catching abi when using this pointer.
   *       When used across FFI boundaries, always use safe_call.
   */
  void* cpp_call;
} TVMFFIFunctionCell;

Construct and Destroy#

Important

Dynamic function creation is useful for passing lambdas or closures across language boundaries.

The following C code constructs a FunctionObj from a TVMFFISafeCallType and a deleter callback. The deleter cleans up resources owned by the function; for global symbols, it is typically NULL.

TVMFFIObject* Function_Construct(void* self, TVMFFISafeCallType safe_call,
                                 void (*deleter)(void* self)) {
  int err_code;
  TVMFFIObject* out = NULL;
  err_code = TVMFFIFunctionCreate(self, safe_call, deleter, (void**)(&out));
  assert(err_code == 0);
  return out;
}

Release a FunctionObj using the standard destruction pattern.

Global Registry#

Retrieve a global function. The following C code uses TVMFFIFunctionGetGlobal() to retrieve a function by name from the global registry:

TVMFFIObject* Function_RetrieveGlobal(const char* name) {
  TVMFFIObject* out = NULL;
  TVMFFIByteArray name_byte_array = {name, strlen(name)};
  int err_code = TVMFFIFunctionGetGlobal(&name_byte_array, (void**)(&out));
  assert(err_code == 0);
  return out;
}

Note

TVMFFIFunctionGetGlobal() returns an owning handle. The caller must release it by calling TVMFFIObjectDecRef() when it’s no longer needed.

Register a global function. The following C code uses TVMFFIFunctionSetGlobal() to register a function by name in the global registry:

void Function_SetGlobal(const char* name, TVMFFIObject* func) {
  TVMFFIByteArray name_byte_array = {name, strlen(name)};
  int err_code = TVMFFIFunctionSetGlobal(&name_byte_array, func, 0);
  assert(err_code == 0);
}

Call Function#

The following C code invokes a FunctionObj with arguments:

int64_t CallFunction(TVMFFIObject* func, int64_t x, int64_t y) {
  int err_code;
  TVMFFIAny args[2];
  TVMFFIAny result = (TVMFFIAny){0};
  args[0] = Any_AnyView_FromInt(x);
  args[1] = Any_AnyView_FromInt(y);
  err_code = TVMFFIFunctionCall(func, args, 2, &result);
  assert(err_code == 0);
  return Any_AnyView_GetInt(&result);
}

Exception#

See also

Exception for detailed exception handling patterns.

Exceptions are a central part of TVM-FFI’s ABI and calling convention. When errors occur, they are stored as objects with a TVMFFIErrorCell payload.

C ABI Reference: TVMFFIErrorCell
tvm/ffi/c_api.h#
/*!
 * \brief Error cell used in error object following header.
 */
typedef struct {
  /*! \brief The kind of the error. */
  TVMFFIByteArray kind;
  /*! \brief The message of the error. */
  TVMFFIByteArray message;
  /*!
   * \brief The backtrace of the error.
   *
   * The backtrace is in the order of recent call first from the top of the stack
   * to the bottom of the stack. This order makes it helpful for appending
   * the extra backtrace to the end as we go up when error is propagated.
   *
   * When printing out, we encourage reverse the order of lines to make it
   * align with python style.
   */
  TVMFFIByteArray backtrace;
  /*!
   * \brief Function handle to update the backtrace of the error.
   * \param self The self object handle.
   * \param backtrace The backtrace to update.
   * \param update_mode The mode to update the backtrace,
   *        can be either kTVMFFIBacktraceUpdateModeReplace, kTVMFFIBacktraceUpdateModeAppend.
   */
  void (*update_backtrace)(TVMFFIObjectHandle self, const TVMFFIByteArray* backtrace,
                           int32_t update_mode);
  /*!
   * \brief Optional cause error chain that caused this error to be raised.
   * \note This handle is owned by the ErrorCell.
   */
  TVMFFIObjectHandle cause_chain;
  /*!
   * \brief Optional extra context that can be used to record additional info about the error.
   * \note This handle is owned by the ErrorCell.
   */
  TVMFFIObjectHandle extra_context;
} TVMFFIErrorCell;

Important

Errors from all languages (e.g. Python, C++) will be properly translated into the TVM-FFI error object.

Retrieve Error Object#

When a function returns -1, an error object is stored in thread-local storage (TLS). Retrieve it with TVMFFIErrorMoveFromRaised(), which returns a tvm::ffi::ErrorObj:

void Error_HandleReturnCode(int rc) {
  TVMFFIObject* err = NULL;
  if (rc == -1) {
    // Move the raised error from TLS (clears TLS slot)
    TVMFFIErrorMoveFromRaised((void**)(&err));  // now `err` owns the error object
    if (err != NULL) {
      PrintError(err);  // print the error
      // IMPORTANT: Release the error object, or gets memory leaks
      TVMFFIObjectDecRef(err);
    }
  } else if (rc == -2) {
    // Frontend (e.g., Python) already has an exception set.
    // Do not fetch from TLS; consult the frontend's error mechanism.
    return;
  }
}

This function transfers ownership to the caller and clears the TLS slot. Call TVMFFIObjectDecRef() when done to avoid memory leaks.

Frontend errors (-2). Error code -2 is reserved for frontend errors. It is returned when TVMFFIEnvCheckSignals() detects a pending Python signal. In this case, do not retrieve the error from TLS; instead, consult the frontend’s error mechanism.

Print Error Message

The error payload is a TVMFFIErrorCell structure containing the error kind, message, and backtrace. Access it by skipping the TVMFFIObject header via pointer arithmetic.

void PrintError(TVMFFIObject* err) {
  TVMFFIErrorCell* cell = (TVMFFIErrorCell*)((char*)err + sizeof(TVMFFIObject));
  fprintf(stderr, "%.*s: %.*s\n",                        //
          (int)cell->kind.size, cell->kind.data,         // e.g. "ValueError"
          (int)cell->message.size, cell->message.data);  // e.g. "Expected at least 2 arguments"
  if (cell->backtrace.size) {
    fprintf(stderr, "Backtrace:\n%.*s\n", (int)cell->backtrace.size, cell->backtrace.data);
  }
}

This prints the error message along with its backtrace.

Raise Exception#

The following C code sets the TLS error and returns -1 via TVMFFIErrorSetRaisedFromCStr():

int Error_RaiseException(void* handle, const TVMFFIAny* args, int32_t num_args, TVMFFIAny* result) {
  TVMFFIErrorSetRaisedFromCStr("ValueError", "Expected at least 2 arguments");
  return -1;
}

For non-null-terminated strings, use TVMFFIErrorSetRaisedFromCStrParts(), which accepts explicit string lengths.

Note

You rarely need to create a ErrorObj directly. The C APIs TVMFFIErrorSetRaisedFromCStr() and TVMFFIErrorSetRaisedFromCStrParts() handle this internally.

🚧 String and Bytes#

Warning

This section is under construction.

The ABI supports strings and bytes as first-class citizens. A string can take multiple forms that are identified by its type_index.

  • kTVMFFIRawStr: raw C string terminated by \0.

  • kTVMFFISmallStr: small string, the length is stored in small_str_len and data is stored in v_bytes.

  • kTVMFFIStr: on-heap string object for strings that are longer than 7 characters.

The following code shows the layout of the on-heap string object.

// span-like data structure to store header and length
typedef struct {
  const char* data;
  size_t size;
} TVMFFIByteArray;

// showcase the layout of the on-heap string.
class StringObj : public ffi::Object, public TVMFFIByteArray {
};

The following code shows how to read a string from TVMFFIAny

TVMFFIByteArray ReadString(const TVMFFIAny *value) {
  TVMFFIByteArray ret;
  if (value->type_index == kTVMFFIRawStr) {
    ret.data = value->v_c_str;
    ret.size = strlen(ret.data);
  } else if (value->type_index == kTVMFFISmallStr) {
    ret.data = value->v_bytes;
    ret.size = value->small_str_len;
  } else {
    assert(value->type_index == kTVMFFIStr);
    ret = *reinterpret_cast<TVMFFIByteArray*>(
      reinterpret_cast<char*>(value->v_obj) + sizeof(TVMFFIObject));
  }
  return ret;
}

Similarly, we have type indices to represent bytes. The C++ API provides classes String and Bytes to enable the automatic conversion of these values with Any storage format.

Rationales. Separate string and bytes enable clear mappings from the Python side. Small string allows us to store short names on-stack. To favor 8-byte alignment (v_bytes) and keep things simple, we did not further pack characters into the small_len field.

Further Reading#

Contents