Android 7.0 Nougat (API level 24) introduced the native camera API, which finally allowed fine-grained control of camera directly from C++.

The new API allows you to access image data directly in C, without the need to pass them from Java. Therefore, there might be some (small) performance gain. I did not do any performance comparison myself, so I don't know for sure.

Using the new api gives you one additional benefit - you can reduce the JNI glue code. If your image processing is done mostly in C++, but you still have to jump back-and-forth between Java and C, you might be required to add a lot of JNI glue code for Java to C communication. Using the native camera api might help to reduce the unnecessary JNI parts.

In this post I would like to show how to use the native camera api to get image data for further processing (on CPU & GPU). You can find the sample project also on github - https://github.com/sixo/native-camera.

I also wrote another post where I show how to do high performance processing of images with C/C++ and RenderScript. And I wrote another blogpost which shows how to generate a video file from captured images using the MediaCodec API.

If you want to see some GPU-accelerated effects that you can do with OpengGL ES2, you can also checkout my Fake Snow Cam app. Even though in this app I'm using the Camera 2 Java API (I also wanted to support Android 6), the particle snow effect that I apply is used from C++ code in the same way I show in this post.

If you you're an Android enthusiast that likes to learn more about Android internals, I highly recommend to check out my Bugjaeger app. It allows you to connect 2 Android devices through USB OTG and perform many of the tasks that are normally only accessible from a developer machine via ADB directly from your Android phone/tablet.

LEGACY Hardware Level - HAL 1 vs HAL 3

Hardware Abstraction Layer (HAL) is the standard interface that Android forces hardware vendors to implement. There are 2 camera HALs supported simultaneously - HAL 1 and HAL 3. HAL2 was just a temporary step between the aforementioned versions.

HAL1 used operating modes to divide the functionality of the camera. According to documentation, these operating modes were overlapping and it was hard to implement new features. HAL3 should overcome this disadvantage and give applications more power to control the camera.

NDK's native camera is the equivalent of camera2 interface. But in comparison to camera2 api, the native camera doesn't support HAL1 interface.

This means that the native api won't list camera devices with LEGACY hardware level. Later in this post I'll show how to query for CameraMetadata.

Native Camera API Overview

The overall model is simple.

There is ACameraManager which gives you a list of available camera devices and allows you to query device features. The device features and settings that you can query are wrapped into ACameraMetadata.

You can use ACameraManager to open ACameraDevice, which you'll use to control the camera. Once ACameraDevice is open, you should use ACameraCauptureSession to configure where (to which ANativeWindow) the camera can send the outputs. There are multiple options where output images can be send, e.g. SurfaceTexture, Allocation, or AImageReader.

ACaptureRequest is then used to specify to which actual target should the captured images be sent.

At the end, the camera will give you the image data and some additional metadata that describes the actual configuration used by the camera for capturing (this might differ from what you requested, if you specified some incompatible configuration).

Configuring the Build System

You should make sure that you've set the proper platform level in build.gradle of your module. Camera is supported in NDK only since api 24.

android {
    defaultConfig {
    externalNativeBuild {
        cmake {
            arguments "-DANDROID_PLATFORM=android-24"
        }
    }
    ...

Additionally, you should link to camera lib in CMakeLists.txt. You might also need to link to media (for AImageReader), android (for ANativeWindow), and GLESv2 libraries, depending on what you are trying to do.

find_library( camera-lib camera2ndk )
find_library( media-lib mediandk )
find_library( android-lib android )
find_library( gl-lib GLESv2 )

target_link_libraries( native-lib ${camera-lib} ${media-lib} ${android-lib} ${gl-lib} )

Camera Permission

Change AndroidManifest.xml so that it contains the permission we require

<uses-permission android:name="android.permission.CAMERA"/>

When I was playing with the native camera api, I just requested the permission from Kotlin. If you don't want to initiate the request from Java (Kotlin), you can call back to Java from C++ in your native activity, similar to how it's done in the official samples.

val camPermission = android.Manifest.permission.CAMERA
if (Build.VERSION.SDK_INT >= 23 &&
        checkSelfPermission(camPermission) != PackageManager.PERMISSION_GRANTED) {
    requestPermissions(arrayOf(camPermission), CAM_PERMISSION_CODE)
}

Listing Available Cameras and Querying for Metadata

ACameraManager is the equivalent of CameraManager from the camera2 api (see the overview section). In the example below I use it to list the available camera devices.

To list available devices, first you should get a pointer to ACameraManager instance with ACameraManager_create() function. Then you pass this pointer to functions that query camera metadata (and other functions).

#include <camera/NdkCameraManager.h>
...
ACameraManager *cameraManager = ACameraManager_create();

Once you've done with the camera, you can then cleanup the manager with

ACameraManager_delete(camManager);

You can use ACameraManager_getCameraIdList() to get a list of camera device IDs. The IDs are stored inside of ACameraIdList struct. For the built-in cameras you get integer number starting from 0. The retrieved ID can then be used in ACameraManager_openCamera() to prepare the device for use.

In the example below I loop through each camera device and get metadata related to the device. You should use ACameraManager_getCameraCharacteristics() to store the read-only metadata inside of ACameraMetadata struct.

ACameraIdList *cameraIds = nullptr;
ACameraManager_getCameraIdList(camManager, &cameraIds);

for (int i = 0; i < cameraIds->numCameras; ++i) 
{
    const char* id = cameraIds->cameraIds[i];
    ACameraMetadata* metadataObj;
    ACameraManager_getCameraCharacteristics(camManager, id, &metadataObj);

    // Work with metadata here
    // ...

    ACameraMetadata_free(metadataObj);

}
ACameraManager_deleteCameraIdList(cameraIds);    

The ACameraMetadata struct is something like a list of properties. You understand what each list entry holds by getting its tag. I use ACameraMetadata_getAllTags() to get all the tags in the example bellow.

ACameraIdList *cameraIds = nullptr;
ACameraManager_getCameraIdList(cameraManager, &cameraIds);

std::string backId;

for (int i = 0; i < cameraIds->numCameras; ++i)
{
    const char* id = cameraIds->cameraIds[i];

    ACameraMetadata* metadataObj;
    ACameraManager_getCameraCharacteristics(cameraManager, id, &metadataObj);

    int32_t count = 0;
    const uint32_t* tags = nullptr;
    ACameraMetadata_getAllTags(metadataObj, &count, &tags);

    for (int tagIdx = 0; tagIdx < count; ++tagIdx)
    {
        // We are interested in entry that describes the facing of camera
        if (ACAMERA_LENS_FACING == tags[tagIdx]) {
            ACameraMetadata_const_entry lensInfo = { 0 };
            ACameraMetadata_getConstEntry(metadataObj, tags[tagIdx], &lensInfo);

            auto facing = static_cast<acamera_metadata_enum_android_lens_facing_t>(
                    lensInfo.data.u8[0]);

            // Found a back-facing camera
            if (facing == ACAMERA_LENS_FACING_BACK)
                ...

            break;
        }
        ...
    }

    ACameraMetadata_free(metadataObj);
}

ACameraManager_deleteCameraIdList(cameraIds);

ACameraMetadata_const_entry is a struct that contains a union of data. You use the tag and type members to decide which union member to use and how to interpret this data. The NdkCameraMetadataTags.h header contains multiple enums that describe camera properties. I checked for ACAMERA_LENS_FACING_BACK to get the back camera.

Enumerating the properties for each device might help you to pick the camera id you are interested in. For example, you can get the back-facing camera ID with the following function

std::string getBackFacingCamId(ACameraManager *cameraManager)
{
    ACameraIdList *cameraIds = nullptr;
    ACameraManager_getCameraIdList(cameraManager, &cameraIds);

    std::string backId;

    for (int i = 0; i < cameraIds->numCameras; ++i)
    {
        const char* id = cameraIds->cameraIds[i];

        ACameraMetadata* metadataObj;
        ACameraManager_getCameraCharacteristics(cameraManager, id, &metadataObj);

        ACameraMetadata_const_entry lensInfo = { 0 };
        ACameraMetadata_getConstEntry(metadataObj, ACAMERA_LENS_FACING, &lensInfo);

        auto facing = static_cast<acamera_metadata_enum_android_lens_facing_t>(
                lensInfo.data.u8[0]);

        // Found a back-facing camera
        if (facing == ACAMERA_LENS_FACING_BACK)
        {
            backId = id;
            break;
        }
    }

    ACameraManager_deleteCameraIdList(cameraIds);

    return backId;
}

The property you're querying might not be supported, therefore it's useful to check the result of ACameraMetadata_getConstEntry()

camera_status_t status = ACameraMetadata_getConstEntry(metadataObj, XY_TAG, &entry);
if (status == ACAMERA_OK) {
    // Great, entry available
}

To get exposure time range

ACameraMetadata_const_entry entry = { 0 };
ACameraMetadata_getConstEntry(metadataObj,
                              ACAMERA_SENSOR_INFO_EXPOSURE_TIME_RANGE, &entry);

int64_t minExposure = entry.data.i64[0];
int64_t maxExposure = entry.data.i64[1];

Similarly, you can use the ACAMERA_SENSOR_INFO_SENSITIVITY_RANGE to get the ISO range

ACameraMetadata_getConstEntry(metadataObj,
                              ACAMERA_SENSOR_INFO_SENSITIVITY_RANGE, &entry);

int32_t minSensitivity = entry.data.i32[0];
int32_t maxSensitivity = entry.data.i32[1];

As mentioned earlier in the overview section, camera will send recorded images to output surfaces that you specified. You might want to adjust the size of these surfaces to the size the camera supports, or find modes with same aspect ratio. You can use ACAMERA_SCALER_AVAILABLE_STREAM_CONFIGURATIONS to do this. For example, to get the supported width and height for JPEG stream

ACameraMetadata_getConstEntry(metadataObj,
                              ACAMERA_SCALER_AVAILABLE_STREAM_CONFIGURATIONS, &entry);

for (int i = 0; i < entry.count; i += 4)
{
    // We are only interested in output streams, so skip input stream
    int32_t input = entry.data.i32[i + 3];
    if (input)
        continue;

    int32_t format = entry.data.i32[i + 0];
    if (format == AIMAGE_FORMAT_JPEG)
    {
        int32_t width = entry.data.i32[i + 1];
        int32_t height = entry.data.i32[i + 2];
    }
}

Initializing Camera Device

Once I acquired a CameraManager instance and once I've got the ID of the camera device that I want to use, it's time to open a camera device.

All operations that could be potentially expensive are handled asynchronously. The camera api uses a set of callbacks that forward information back to your code.

You should use the ACameraManager_openCamera() function to initialize the camera. This function expects a ACameraDevice_StateCallbacks already initialized with the required callbacks.

The onDisconnected callback reports when the camera is no longer available (e.g., user withdrew camera permission, or external camera has been removed).

To understand what the error in onError callback means, check out the enum for ACameraDevice_ErrorStateCallback error code inside of NdkCameraDevice.h header.

#include <camera/NdkCameraManager.h>
#include <camera/NdkCameraMetadataTags.h>
#include <camera/NdkCameraMetadata.h>

static void onDisconnected(void* context, ACameraDevice* device)
{
    // ...
}

static void onError(void* context, ACameraDevice* device, int error)
{
    // ...
}

static ACameraDevice_stateCallbacks cameraDeviceCallbacks = {
        .context = nullptr,
        .onDisconnected = onDisconnected,
        .onError = onError,
};
    ...
    ACameraManager_openCamera(cameraManager, cameraId, &cameraDeviceCallbacks, &cameraDevice);

The callbacks have a context parameter which you can use to pass the this pointer when you wrap the functions into a C++ class. I the example above I just set it to null.

Getting Output Surface (ANativeWindow) for Capture Session with AImageReader (CPU Processing)

ACameraCaptureSession first needs a target surface (ANativeWindow) where it can send the pictures.

You have multiple options to choose from, depending on what you want to do with the captured data. For example, you can use SurfaceTexture for OpenGL processing, or Allocation for processing with Render Script.

In the code bellow I use AImageReader, which allows me to get access to RAW, uncompressed YUV, or compressed JPEG data.

AImageReader_new() expects width and height as the first two arguments. Here you should choose values compactible with the camera device (see previous section for how to query supported resolutions).

The third argument should be one of AIMAGEFORMAT values. I chose JPEG, which should be always supported. The forth argument determines the maximum number of images you want to access simultaneously. This will restrict how many times you can call AImageReader_acquireNextImage() before calling AImage_delete() inside of the imageCallback.

The recorded image is then processed in imageCallback asynchronously.

You should also note that the ANativeWindow* pointer that I acquired with AImageReader_getWindow() is managed by the AImageReader. Therefore, if you want to do cleanup later, you should not call ANativeWindow_release(), but only call AImageReader_delete() and let ImageReader do the cleanup.

static void imageCallback(void* context, AImageReader* reader)
{
    AImage *image = nullptr;
    auto status = AImageReader_acquireNextImage(reader, &image);
    // Check status here ...

    // Try to process data without blocking the callback
    std::thread processor([=](){

        uint8_t *data = nullptr;
        int len = 0;
        AImage_getPlaneData(image, 0, &data, &len);

        // Process data here
        // ...

        AImage_delete(image);
    });
    processor.detach();
}

AImageReader* createJpegReader()
{
    AImageReader* reader = nullptr;
    media_status_t status = AImageReader_new(640, 480, AIMAGE_FORMAT_JPEG,
                     4, &reader);

    //if (status != AMEDIA_OK)
        // Handle errors here

    AImageReader_ImageListener listener{
            .context = nullptr,
            .onImageAvailable = imageCallback,
    };

    AImageReader_setImageListener(reader, &listener);

    return reader;
}

ANativeWindow* createSurface(AImageReader* reader) 
{
    ANativeWindow *nativeWindow;
    AImageReader_getWindow(reader, &nativeWindow);

    return nativeWindow;
}

Camera sensor orientation is fixed. Therefore, if you want to display the captured image, you might need to rotate it first. You can use the following code to get the clockwise angle through which the captured image has to be rotated to be upright

ACameraMetadata_getConstEntry(metadataObj,
                              ACAMERA_SENSOR_ORIENTATION, &entry);

int32_t orientation = entry.data.i32[0];

Using SurfaceTexture as Output Surface with OpenGL (GPU Processing)

One way to do post-processing of recorded frames on GPU is to use a SurfaceTexture together with OpengGL.

There are multiple options to setup OpenGL rendering in Android. In this example I extended GLSurfaceView, so that I don't have to take care of OpenGL setup and thread control myself. You can however use a SurfaceView and manage the rendering thread in your code.

The NDK now contains some functions for working with SurfaceTexture, but these seem to be only available since Android P (platfrom 28).

Creating a SurfaceTexture still requires Java (Kotlin). Therefore, the first part of the example bellow still contains some Kotlin code.

First thing I did to configure target output Surface for camera images was to implement a GLSurfaceView.Renderer.

class CamRenderer: GLSurfaceView.Renderer {

    lateinit var surfaceTexture: SurfaceTexture
    val texMatrix = FloatArray(16)
    @Volatile var frameAvailable: Boolean = false
    val lock = Object()

    override fun onDrawFrame(p0: GL10?) {
        synchronized(lock) {
            if (frameAvailable) {
                surfaceTexture.updateTexImage()
                surfaceTexture.getTransformMatrix(texMatrix)
                frameAvailable = false
            }
        }

        onDrawFrame(texMatrix)
    }

    override fun onSurfaceChanged(p0: GL10?, width: Int, height: Int) {
        onSurfaceChanged(width, height)
    }

    override fun onSurfaceCreated(p0: GL10?, p1: EGLConfig?) {
        // Prepare texture and surface
        val textures = IntArray(1)
        GLES20.glGenTextures(1, textures, 0)
        GLES20.glBindTexture(GL_TEXTURE_EXTERNAL_OES, textures[0])

        surfaceTexture = SurfaceTexture(textures[0])
        surfaceTexture.setOnFrameAvailableListener {
            synchronized(lock) {
                frameAvailable = true
            }
        }

        // Choose you preferred preview size here before creating surface
        // val optimalSize = getOptimalSize()
        // surfaceTexture.setDefaultBufferSize(optimalSize.width, optimalSize.height)

        val surface = Surface(surfaceTexture)

        // Pass to native code
        onSurfaceCreated(textures[0], surface)
    }

    external fun onSurfaceCreated(textureId: Int, surface: Surface)
    external fun onSurfaceChanged(width: Int, height: Int)
    external fun onDrawFrame(texMat: FloatArray)
}

In the code above I create a new SurfaceTexture in the onSurfaceCreated() callback. I create a texture name with the usual OpenGL methods and then I use the name in SurfaceTexture's constructor.

Camera frames are generated on a different thread than OpenGL thread, therefore some synchronization is necessary. Once a frame is available, I update the SurfaceTexture with updateTexImage(). Here I make sure that it's called from the OpenGL thread.

Additionally, SurfaceTexture provides a matrix for transforming the texture's UVs. This matrix could change after the call to updateTexImage(), therefore I update the matrix with getTransformMatrix() and forward it to C++ code in the onDrawFrame() callback.

One important thing that I've skipped here is to call setDefaultBufferSize() before creating the Surface. Here you have to choose a preview size that the camera device offers for surface texture. Ideally, it would be something with same aspect ratio as the aspect ratio of your rendering surface (or display, if full screen). This will also affect the matrix that we later use to transform the texture with the preview to correct rotation and scale. You should be able to get the supported sizes in similar way as we got the supported width and height before for JPEG. Or you can just call getSupportedOutputSizes(SurfaceTexture.class) from the Java Camera 2 API.

Once I had a Renderer the next thing I did was to extend GLSurfaceView and set it up with my renderer

package eu.sisik.cam

import android.content.Context
import android.opengl.GLSurfaceView
import android.util.AttributeSet

class CamSurfaceView : GLSurfaceView {

    var camRenderer: CamRenderer

    constructor(context: Context?, attrs: AttributeSet) : super(context, attrs) {
        setEGLContextClientVersion(2)

        camRenderer = CamRenderer()
        setRenderer(camRenderer)
    }
}

Then on the C++ side I created some JNI glue code and set up OpenGL ES2 rendering.

I started with the onSurfaceCreated() callback that is called from the Renderer. Here I did all the camera initialization.

The code below is a bit lengthy because of the Open GL setup. You should focus on the camera initialization part at the bottom.

The important thing to note is that I passed the texture name from Kotlin, so that I can reference it in my drawing code.

Additionally, I also passed the surface created in Kotlin to C++ and used it to create ANativeWindow. This ANativeWindow can be used to create ACameraCapture session (see section below).

The createShader() and createProgram() functions are helper functions that I didn't include in the code bellow to make it a bit shorter. They just compile the shaders and do some checking and logging (regular OpenGL stuff).

// Variables for OpenGL setup
GLuint prog;
GLuint vtxShader;
GLuint fragShader;
GLint vtxPosAttrib;
GLint uvsAttrib;
GLint mvpMatrix;
GLint texMatrix;
GLint texSampler;
GLint color;
GLint size;
GLuint buf[2];

// The id is generated in Kotlin and passed to C++
GLuint textureId;

int width = 640;
int height = 480;

JNIEXPORT void JNICALL
Java_eu_sisik_cam_CamRenderer_onSurfaceCreated(JNIEnv *env, jobject, jint texId, jobject surface)
{
    /**
     * Basic OpenGL setup
     */

    // Init shaders
    vtxShader = createShader(vertexShaderSrc, GL_VERTEX_SHADER);
    fragShader = createShader(fragmentShaderSrc, GL_FRAGMENT_SHADER);
    prog = createProgram(vtxShader, fragShader);

    // Store attribute and uniform locations
    vtxPosAttrib = glGetAttribLocation(prog, "vertexPosition");
    uvsAttrib = glGetAttribLocation(prog, "uvs");
    mvpMatrix = glGetUniformLocation(prog, "mvp");
    texMatrix = glGetUniformLocation(prog, "texMatrix");
    texSampler = glGetUniformLocation(prog, "texSampler");
    color = glGetUniformLocation(prog, "color");
    size = glGetUniformLocation(prog, "size");

    // Prepare buffers
    glGenBuffers(2, buf);

    // Set up vertices
    float vertices[] {
        // x, y, z, u, v
        -1, -1, 0, 0, 0,
        -1, 1, 0, 0, 1,
        1, 1, 0, 1, 1,
        1, -1, 0, 1, 0
    };
    glBindBuffer(GL_ARRAY_BUFFER, buf[0]);
    glBufferData(GL_ARRAY_BUFFER, sizeof(vertices), vertices, GL_DYNAMIC_DRAW);

    // Set up indices
    GLuint indices[] { 2, 1, 0, 0, 3, 2 };
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, buf[1]);
    glBufferData(GL_ELEMENT_ARRAY_BUFFER, sizeof(indices), indices, GL_DYNAMIC_DRAW);

    /**
     * Camera initialisation
     */

    // Init cam manager
    cameraManager = ACameraManager_create();

    // Init camera
    auto id = getBackFacingCamId(cameraManager);
    ACameraManager_openCamera(cameraManager, id.c_str(), &cameraDeviceCallbacks, &cameraDevice);

    // Prepare surface
    textureId = texId;
    textureWindow = ANativeWindow_fromSurface(env, surface);

    // Prepare outputs for session
    ACaptureSessionOutput_create(textureWindow, &textureOutput);

    ACaptureSessionOutputContainer_create(&outputs);
    ACaptureSessionOutputContainer_add(outputs, textureOutput);

    // Prepare capture session, capture request, 
    // and start capturing (see section bellow)
    // ..
}

In the onSurfaceChanged() callback I just passed my window size to C++ code.

JNIEXPORT void JNICALL
Java_eu_sisik_cam_CamRenderer_onSurfaceChanged(JNIEnv *env, jobject, jint w, jint h)
{
    width = w;
    height = h;
}

In the onDrawFrame() callback I pass the transformation matrix (texMatArray) for SurfaceTexture's UVs to my shaders.

Additionally, you should note that I bind to GL_TEXTURE_EXTERNAL_OES to be able to use the SurfaceTexture with camera frames.

All the rest is just the usual OpengGL drawing stuff and passing of some uniforms (you will understand what those are for once you see the shaders).

JNIEXPORT void JNICALL
Java_eu_sisik_cam_CamRenderer_onDrawFrame(JNIEnv *env, jobject, jfloatArray texMatArray)
{
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
    glClearColor(0, 0, 0, 1);

    glUseProgram(prog);

    // Configure main transformations
    float mvp[] = {
            1.0f, 0, 0, 0,
            0, 1.0f, 0, 0,
            0, 0, 1.0f, 0,
            0, 0, 0, 1.0f
    };

    float aspect = width > height ? float(width)/float(height) : float(height)/float(width);
    if (width < height) // portrait
        ortho(mvp, -1.0f, 1.0f, -aspect, aspect, -1.0f, 1.0f);
    else // landscape
        ortho(mvp, -aspect, aspect, -1.0f, 1.0f, -1.0f, 1.0f);

    glUniformMatrix4fv(mvpMatrix, 1, false, mvp);

    /**
     * Prepare texture for drawing
     */
    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_EXTERNAL_OES, textureId);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

    // Pass SurfaceTexture transformations to shader
    float* tm = env->GetFloatArrayElements(texMatArray, 0);
    glUniformMatrix4fv(texMatrix, 1, false, tm);
    env->ReleaseFloatArrayElements(texMatArray, tm, 0);

    // I use red color to mix with camera frames
    float c[] = { 1, 0, 0, 1 };
    glUniform4fv(color, 1, (GLfloat*)c);

    // Size of the window is used in fragment shader
    // to split the window
    float sz[] = {0};
    sz[0] = width;
    sz[1] = height;
    glUniform2fv(size, 1, (GLfloat*)sz);

    // Set up vertices and indices
    glBindBuffer(GL_ARRAY_BUFFER, buf[0]);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, buf[1]);

    glEnableVertexAttribArray(vtxPosAttrib);
    glVertexAttribPointer(vtxPosAttrib, 3, GL_FLOAT, GL_FALSE, sizeof(float) * 5, (void*)0);

    glEnableVertexAttribArray(uvsAttrib);
    glVertexAttribPointer(uvsAttrib, 2, GL_FLOAT, GL_FALSE, sizeof(float) * 5, (void *)(3 * sizeof(float)));

    glViewport(0, 0, width, height);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    glDrawElements(GL_TRIANGLES, 6, GL_UNSIGNED_INT, 0);
}

Then in the vertex shader I transform the UVs of surface texture with the provided texMatrix

const char* vertexShaderSrc = R"(
    precision highp float;
    attribute vec3 vertexPosition;
    attribute vec2 uvs;
    varying vec2 varUvs;
    uniform mat4 texMatrix;
    uniform mat4 mvp;

    void main()
    {
        varUvs = (texMatrix * vec4(uvs.x, uvs.y, 0, 0)).xy;
        gl_Position = mvp * vec4(vertexPosition, 1.0);
    }
)";

The fragment shader divides the screen vertically into halves. In the left half I mix the provided color with the frame pixels. The right half shows a binary image.

With the directive at the top I specify that I require the GL_OES_EGL_image_external extension - #extension GL_OES_EGL_image_external : require. Then I use the special samplerExternalOES variable type to represent the SurfaceTexture.

const char* fragmentShaderSrc = R"(
    #extension GL_OES_EGL_image_external : require

    precision mediump float;
    varying vec2 varUvs;

    uniform samplerExternalOES texSampler;

    uniform vec4 color;
    uniform vec2 size;

    void main()
    {
        if (gl_FragCoord.x/size.x < 0.5) {
            gl_FragColor = texture2D(texSampler, varUvs) * color;
        }
        else {
            const float threshold = 1.1;
            vec4 c = texture2D(texSampler, varUvs);
            if (length(c) > threshold) {
                gl_FragColor = vec4(0.0, 0.0, 1.0, 1.0);
            } else {
                gl_FragColor = vec4(1.0, 1.0, 1.0, 1.0);
            }
        }
    }
)";

You can see the result in the video below

Creating ACameraCaptureSession

In the previous section I shoved you how to get ANativeWindow (Surface) which we will use here to create ACameraCaptureSession.

You can create a CaptureSession with or without initial ACaptureRequest parameters. In the following code I use ACameraDevice_createCaptureSession() to create a new session.

The ACameraDevice_createCaptureSession() function expects 0 or more Surfaces (ANativeWindows) (wrapped into ACaptureSessionOutput) to which the camera should send the output. Creating a session with an empty list, closes the previous session (there can be only one at a time).

The ACaptureSessionOutputContainer struct is used to hold a list of surfaces for the session. You should use ACaptureSessionOutputContainer_create/free/add/remove functions to manage the list.

static void onSessionActive(void* context, ACameraCaptureSession *session)
{}

static void onSessionReady(void* context, ACameraCaptureSession *session)
{}

static void onSessionClosed(void* context, ACameraCaptureSession *session)
{}

static ACameraCaptureSession_stateCallbacks sessionStateCallbacks {
        .context = nullptr,
        .onActive = onSessionActive,
        .onReady = onSessionReady,
        .onClosed = onSessionClosed
};

...

ACaptureSessionOutput* output = nullptr;
ACaptureSessionOutputContainer* outputs = nullptr;
ACameraCaptureSession* session = nullptr;
...

// Create session output from ANativeWindow (Surface)
ACaptureSessionOutput_create(window, &output);

// Create a container for outputs and add output
ACaptureSessionOutputContainer_create(&outputs);
ACaptureSessionOutputContainer_add(outputs, output);

// Create the session
ACameraDevice_createCaptureSession(device, outputs, &sessionStateCallbacks, &session);

Preparing Parameters for Capturing an Image - ACaptureRequest

ACaptureRequest contains parameters to capture one single image.

The session created previously was configured with a list of possible output surfaces, and the request specifies which surfaces will actually be used. The selected output surface is wrapped into ACameraOutputTarget and added to the request with ACaptureRequest_addTarget.

When you create a request with ACameraDevice_createCaptureRequest you specify a ACameraDevice_request_template with some request parameters. You can then specify additional parameters (like e.g., exposure, ISO) with ACaptureRequest_setEntry__ functions. The NdkCameraMetadataTags.h header contains more tags that you can use for specifying capture parameters.

ACaptureRequest* request = nullptr;
ACameraOutputTarget* jpegTarget = nullptr;

// Create request with preview template - high frame rate has priority
// over quality post-processing
ACameraDevice_createCaptureRequest(cameraDevice, TEMPLATE_PREVIEW, &request);

// Specify actual surfaces that should be used for output
ANativeWindow_acquire(jpegSurface);
ACameraOutputTarget_create(jpegSurface, &jpegTarget);
ACaptureRequest_addTarget(request, jpegTarget);

// Configure additional parameters here
uint8_t jpegQuality = 85;
ACaptureRequest_setEntry_u8(request, ACAMERA_JPEG_QUALITY, 1, &jpegQuality);

Capturing Image - One Image or Continuous Capture

Once you're done with all the setup shown in previous steps, you can capture an image with the ACameraCaptureSession_capture() function.

The function expects an array of one or more requests which it will try to process in the shortest time possible.

Again, you have the option to use callbacks, so that you are notified of state changes.

You can specify a sequence id to keep track of your submitted requests.

The result of capturing should produce buffers with the final image and metadata that stores the actual settings used (these settings can differ from what you specified in the request). In the example bellow, the onCaptureCompleted callback provides result metadata as the last parameter.

void onCaptureFailed(void* context, ACameraCaptureSession* session,
                     ACaptureRequest* request, ACameraCaptureFailure* failure)
{}

void onCaptureSequenceCompleted(void* context, ACameraCaptureSession* session,
                                int sequenceId, int64_t frameNumber)
{}

void onCaptureSequenceAborted(void* context, ACameraCaptureSession* session,
                              int sequenceId)
{}

void onCaptureCompleted (
        void* context, ACameraCaptureSession* session,
        ACaptureRequest* request, const ACameraMetadata* result)
{}

static ACameraCaptureSession_captureCallbacks captureCallbacks {
        .context = nullptr,
        .onCaptureStarted = nullptr,
        .onCaptureProgressed = nullptr,
        .onCaptureCompleted = onCaptureCompleted,
        .onCaptureFailed = onCaptureFailed,
        .onCaptureSequenceCompleted = onCaptureSequenceCompleted,
        .onCaptureSequenceAborted = onCaptureSequenceAborted,
        .onCaptureBufferLost = nullptr,
};
...
// Capture one image
ACameraCaptureSession_capture(session, &captureCallbacks, 1, &request, nullptr);

Alternatively, you can use ACameraCaptureSession_setRepeatingRequest(), which will capture images continuously until you call ACameraCaptureSession_stopRepeating().

ACameraCaptureSession_setRepeatingRequest(session, &captureCallbacks, 1, &request, nullptr);

Next Post