Frame Processors Tips

Avoiding Frame-drops

Frame Processors will be synchronously called for each frame the Camera sees and have to finish executing before the next frame arrives, otherwise the next frame(s) will be dropped. For a frame rate of 30 FPS, you have about 33ms to finish processing frames. At 60 FPS, you only have 16ms. If your Frame Procesor has not finished executing when the next frame arrives, the next frame will be dropped.

Some general tips:

• Use runAsync(..) if you don't need your Frame Processor to run synchronously
• Use runAtTargetFps(..) if you don't need your Frame Processor to run on every frame
• Use Shared Values (useSharedValue(..)) instead of React State (useState(..)) when sharing data
• Prefer native Frame Processor Plugins instead of pure JavaScript based plugins

FPS Graph

Use the FPS Graph to profile your Frame Processor's performance over time:

<Camera {...props} enableFpsGraph={true} />

Fast Frame Processor Plugins

If you use native Frame Processor Plugins, make sure they are optimized for realtime Camera use-cases. Some general tips:

• Prefer plugins that use the PixelFormat yuv instead of rgb, as yuv is more efficient in both memory usage and processing efficiency
• Prefer plugins that can work with the native Frame types (CMSampleBuffer and Image/HardwareBuffer) instead of passing the byte array (frame.toArrayBuffer()), as the latter involves a GPU -> CPU copy
• If you need to use the byte array (frame.toArrayBuffer()), prefer plugins that work with uint8 instead of float types, as uint8 is much more efficient
• Prefer plugins that support GPU acceleration. For Tensorflow, this might be the CoreML or Metal GPU delegates
• For operations such as resizing, prefer GPU or CPU vector acceleration (e.g. Accelerate/vImage) instead of just array loops

ESLint react-hooks plugin

If you are using the react-hooks ESLint plugin, make sure to add useFrameProcessor and useSkiaFrameProcessor to additionalHooks inside your ESLint config so dependencies are detected properly. (See "advanced configuration")

Technical

Frame Processors

Frame Processors are JS functions that will be workletized using react-native-worklets-core. They are created on a parallel camera thread using a separate JavaScript Runtime ("VisionCamera JS-Runtime") and are invoked synchronously (using JSI) without ever going over the bridge. In a Frame Processor you can write normal JS code, call back to the React-JS Thread (e.g. setState), use Shared Values and call Frame Processor Plugins.

In my benchmarks, Frame Processors can run more than 1000 times a second.

Frame Processor Plugins

Frame Processor Plugins are native functions (written in Objective-C, Swift, C++, Java or Kotlin) that are injected into the VisionCamera JS-Runtime. They can be synchronously called from your JS Frame Processors (using JSI) without ever going over the bridge. Because VisionCamera provides an easy-to-use plugin API, you can easily create a Frame Processor Plugin yourself. Some examples include Barcode Scanning, Face Detection, Image Labeling, Text Recognition and more.

Learn how to create Frame Processor Plugins, or check out the example Frame Processor Plugin for iOS or Android.

The Frame object

The Frame Processor gets called with a Frame object, which is a JSI HostObject. It holds a reference to the native (C++) Frame's GPU Buffer (~12 MB in size in YUV) and exposes properties such as width, height, bytesPerRow and more to JavaScript so you can synchronously access them. You can access the Frame data in JavaScript using frame.toArrayBuffer(), which copies over the GPU buffer to the CPU. The Frame object can be passed around in JS, as well as returned from- and passed to a native Frame Processor Plugin.

With 4k Frames, roughly 700 MB of Frame data flow through your Frame Processor per second.

See this tweet for more information.

🚀 Next section: Zooming (or creating a Frame Processor Plugin)