Class Instance

TVM runtime instance.

All objects(NDArray, Module, PackedFunc) returned by TVM runtim function call and PackedFunc instance are tracked through a scope mechanism that will get auto-released when we call EndScope.

This is necessarily to be able to release the underlying WASM and WebGPU memory that are not tracked through JS native garbage collection mechanism.

This does mean that we have to get familar with the following functions:

Hierarchy

  • Instance

Implements

Constructors

  • Constructor

    importObject can also be a LibraryProvider object, a WASI object, or an object containing wasmLibraryProvider field.

    See

    Please use the async version instantiate when targeting browsers.

    Parameters

    • wasmModule: Module

      The input module or instance.

    • importObject: Record<string, any> = {}

      The imports to initialize the wasmInstance if it is not provided.

    • Optional wasmInstance: Instance

      Additional wasm instance argument for deferred construction.

    • Optional env: Environment

      Directly specified environment module.

    Returns Instance

Properties

cacheMetadata: Record<string, any> = {}
exports: Record<string, Function>
memory: Memory

Methods

  • Apply presence and frequency penalty. This is an inplace operation.

    Parameters

    • logits: NDArray

      The input logits before penalty.

    • token_ids: NDArray

      The appeared token ids.

    • token_freqs: NDArray

      The number of times each token has appeared since last PrefillStep. token_freqs[i] is the frequency of token_ids[i], for all i. And all token_freqs should be >= 1.

    • presence_penalty: number

      The penalty factor.

    • frequency_penalty: number

      The penalty factor.

    Returns any

  • Apply repetition penalty to the logits.

    Parameters

    • logits: NDArray

      The input logits before penalty.

    • token_ids: NDArray

      The appeared token ids.

    • penalty: number

      The penalty factor.

    Returns any

  • Apply softmax with temperature to the logits.

    Parameters

    • logits: NDArray

      The input logits before softmax w/ temperature.

    • temperature: number

      The temperature factor.

    Returns any

  • Asynchronously load webgpu pipelines when possible.

    Parameters

    • mod: Module

      The input module.

    Returns Promise<void>

  • Check whether we enabled asyncify mode

    Returns boolean

    The asynctify mode toggle

  • Attach a detached obj to the auto-release pool of the current scope.

    Note

    Normally user do not need to call this function explicitly, as all library call return values are explicitly attached to the current scope. You only need to do so when you call detachFromCurrentScope to create a detached object.

    Type Parameters

    Parameters

    • obj: T

      The input obj.

    Returns T

  • Begin a new scope for tracking object disposal.

    Returns void

  • Benchmark stable execution of the run function.

    Params

    run The run function

    Params

    dev The device to sync during each run.

    Number

    The number of times to compute the average.

    Repeat

    The number of times to repeat the run.

    Parameters

    • run: (() => void)
        • (): void
        • Returns void

    • dev: DLDevice
    • number: number = 10
    • repeat: number = 1

    Returns Promise<number[]>

  • Bind canvas to the current WebGPU context

    Parameters

    • canvas: HTMLCanvasElement

      The canvas.

    Returns void

  • Detach the object from the current scope so it won't be released via auto-release during endscope.

    User needs to either explicitly call obj.dispose(), or attachToCurrentScope to re-attach to the current scope.

    This function can be used to return values to the parent scope.

    Type Parameters

    Parameters

    • obj: T

      The object.

    Returns T

  • Create a new DLDevice

    Parameters

    • deviceType: string | number

      The device type.

    • deviceId: number = 0

      The device index.

    Returns DLDevice

    The created device.

  • Dispose the internal resource This function can be called multiple times, only the first call will take effect.

    Returns void

  • Create an empty NDArray with given shape and dtype.

    Parameters

    • shape: number | number[]

      The shape of the array.

    • dtype: string | DLDataType = "float32"

      The data type of the array.

    • dev: DLDevice = ...

      The device of the ndarray.

    Returns NDArray

    The created ndarray.

  • End a scope and release all created TVM objects under the current scope.

    Exception: one can call moveToParentScope to move a value to parent scope.

    Returns void

  • Given cacheUrl, search up items to fetch based on cacheUrl/ndarray-cache.json

    Parameters

    • ndarrayCacheUrl: string

      The cache url.

    • device: DLDevice

      The device to be fetched to.

    • cacheScope: string = "tvmjs"

      The scope identifier of the cache

    • cacheType: string = "cache"

      The type of the cache: "cache" or "indexedDB"

    Returns Promise<any>

    The meta data

  • Get global PackedFunc from the runtime.

    Parameters

    • name: string

      The name of the function.

    Returns PackedFunc

    The result function.

  • Get parameters in the form of prefix_i

    Parameters

    • prefix: string

      The parameter prefix.

    • numParams: number

      Number of parameters.

    Returns TVMObject

  • Get parameters based on parameter names provided

    Parameters

    • paramNames: string[]

      Names of the parameters.

    Returns TVMObject

    Parameters read.

  • Initialize webgpu in the runtime.

    Parameters

    • device: GPUDevice

      The given GPU device.

    Returns void

  • Check if func is PackedFunc.

    Parameters

    • func: unknown

      The input.

    Returns boolean

    The check result.

  • List all the global function names registered in the runtime.

    Returns string[]

    The name list.

  • Create a shape tuple to pass to runtime.

    Parameters

    • shape: number[]

      The shape .

    Returns TVMObject

    The created shape tuple.

  • Create an tuple TVMArray input array.

    The input array can be passed to tvm runtime function and needs to b explicitly disposed.

    Parameters

    Returns TVMArray

    The result array.

  • Move obj's attachment to the parent scope.

    This function is useful to make sure objects are still alive when exit the current scope.

    Type Parameters

    Parameters

    • obj: T

      The object to be moved.

    Returns T

    The input obj.

  • Update the ndarray cache.

    Returns void

  • Get NDArray from cache.

    Parameters

    • name: string

      The name of array.

    Returns NDArray

    The result.

  • Get NDArray from cache.

    Parameters

    • name: string

      The name of array.

    Returns NDArray

    The result.

  • Update the ndarray cache.

    Parameters

    • name: string

      The name of the array.

    • arr: NDArray

      The content.

    • override: boolean = false

    Returns void

  • Register an asyncfunction to be global function in the server.

    Note

    The async function will only be used for serving remote calls in the rpc These functions contains explicit continuation

    Parameters

    • name: string

      The name of the function.

    • func: Function

      function to be registered.

    • override: boolean = false

      Whether overwrite function in existing registry.

    Returns void

  • Register async function as asynctify callable in global environment.

    Note

    This function is handled via asynctify mechanism The wasm needs to be compiled with Asynctify

    Parameters

    • name: string

      The name of the function.

    • func: ((...args) => Promise<any>)

      function to be registered.

        • (...args): Promise<any>
        • Parameters

          • Rest ...args: any[]

          Returns Promise<any>

    • override: boolean = false

      Whether overwrite function in existing registry.

    Returns void

  • Register function to be global function in tvm runtime.

    Parameters

    • name: string

      The name of the function.

    • func: Function | PackedFunc
    • override: boolean = false

      Whether overwrite function in existing registry.

    Returns void

  • Register a call back for fetch progress.

    Parameters

    Returns void

  • Register an object constructor.

    Parameters

    • typeKey: string

      The name of the function.

    • func: FObjectConstructor

      Function to be registered.

    • override: boolean = false

      Whether overwrite function in existing registry.

    Returns void

  • Obtain the runtime information in readable format.

    Returns string

  • Sample index via top-p sampling.

    Parameters

    • logits: NDArray

      The input logits before normalization.

    • temperature: number

      The temperature factor, will take argmax if temperature = 0.0

    • top_p: number

      The top_p

    Returns number

    The sampled index.

  • Sample index via top-p sampling.

    Parameters

    • prob: NDArray

      The distribution, i.e. logits after applySoftmaxWithTemperature() is performed.

    • top_p: number

      The top_p

    Returns number

    The sampled index.

  • Create a new Scalar that can be passed to a PackedFunc.

    Parameters

    • value: number

      The number value.

    • dtype: string

      The dtype string.

    Returns Scalar

    The created scalar.

  • Set packed function arguments into the location indicated by argsValue and argsCode. Allocate new temporary space from the stack if necessary.

    Parma

    stack The call stack

    Parameters

    • stack: CachedCallStack
    • args: any[]

      The input arguments.

    • argsValue: number

      The offset of argsValue.

    • argsCode: number

      The offset of argsCode.

    Returns void

  • Set the seed of the internal LinearCongruentialGenerator.

    Parameters

    • seed: number

    Returns void

  • Show image in canvas.

    Parameters

    • dataRGBA: NDArray

      Image array in height x width uint32 NDArray RGBA format on GPU.

    Returns void

  • Get system-wide library module in the wasm. System lib is a global module that contains self register functions in startup.

    Returns Module

    The system library module.

  • Get type index from type key.

    Parameters

    • typeKey: string

      The type key.

    Returns number

    The corresponding type index.

  • Create am uniform NDArray with given shape.

    Parameters

    • shape: number[]

      The shape of the array.

    • low: number

      The low value.

    • high: number

      The high value.

    • dev: DLDevice

      The device of the ndarray.

    Returns NDArray

    The created ndarray.

  • Perform action under a new scope.

    Note

    For action to return a valid value, we will need to call moveToParentScope for the objects that are created in the scope.

    Type Parameters

    • T

    Parameters

    • action: (() => T)

      The action function.

        • (): T
        • Returns T

    Returns T

    The result value.

  • Wrap a function obtained from tvm runtime as AsyncPackedFunc through the asyncify mechanism

    You only need to call it if the function may contain callback into async JS function via asynctify. A common one can be GPU synchronize.

    It is always safe to wrap any function as Asynctify, however you do need to make sure you use await when calling the funciton.

    Parameters

    Returns AsyncPackedFunc

    The wrapped AsyncPackedFunc

Generated using TypeDoc