Cross-compiling OpenCV from sources and using it in my Android apps has long time been on my todo list.

I like to play with images and graphics stuff and OpenCV offers a lot of features related to image processing. Recently I found some free time and created my Pano Stitch & Crop app which uses OpenCV to create panoramic photos.

There's a lot of documentation and tutorials that show how to use OpenCV on Android. But I did have some specific requirements that forced me to use OpenCV in my Android app in a bit different way than usual.

As with most open source projects, I prefer to build the libraries from source, instead of downloading any prebuild binaries.

Libraries written in C++ often provide a JNI wrapper to access all possible APIs from Java/Kotlin code. Here I prefer to create my own wrapper that accesses only the minimal functionality I need. Therefore I decided to only use the native C++ part of OpenCV and did not build the Java wrapper.

Additionally, OpenCV on Android is usually used with OpenCVManager. With OpenCVManager you have the advantage that multiple apps can use one library to save space and you also get automatic updates with bugfixes. I still prefer when the user doesn't have to download and install something additional besides my app, so I decided to use OpenCV without OpenCVManager.

In this post I'll describe my experience on building and using OpenCV on Android.

If you you're an Android enthusiast that likes to learn more about Android internals, I highly recommend to check out my Bugjaeger app. It allows you to connect 2 Android devices through USB OTG and perform many of the tasks that are normally only accessible from a developer machine via ADB directly from your Android phone/tablet.

Cross-Compiling OpenCV from Source Code

The main requirements for building OpenCV from source is to have the NDK and CMake installed on your machine.

I cross-compiled the sources on Ubuntu 16. It might not be a requirement, but I only had good results cross-compiling on the Linux platform.

First thing you need to do is downloading the sources

git clone https://github.com/opencv/opencv.git opencv

Next you should create an output directory (inside of the directory where you've cloned OpenCV into) because OpenCV doesn't allow in-source builds

mkdir android_build
cd android_build

To create the necessary build structure with cmake, you need to provide the path to your NDK in ANDROID_NDK variable. Maybe this has already been fixed, but the .cmake file used by OpenCV wasn't working properly for Android. Luckily, the NDK provides android.toolchain.cmake file located somewhere in build/cmake. I forced cmake to use this file instead by defining the -DCMAKE_TOOLCHAIN_FILE variable.

Some additional tweaks were that I disabled building of the Java stuff and I also specified which STL runtime I want to use (although it's possible that the same one is already set by default).

I also specified the install prefix. This will be the directory where all the output files (including .so libraries and header files) will be copied.

One additional variable that you should specify is ANDROID_ABI, otherwise the deprecated armeabi might get selected by default. For other projects you would normally use CMAKE_ANDROID_ARCH_ABI, but this flag seems to be ignored when using the .cmake file provided by NDK.

Your cmake shell command should look something like this

cmake .. -DCMAKE_TOOLCHAIN_FILE=/path/to/ndk/build/cmake/android.toolchain.cmake -DANDROID_NDK=/path/to/ndk -DANDROID_NATIVE_API_LEVEL=android-21 -DBUILD_JAVA=OFF -DBUILD_ANDROID_EXAMPLES=OFF -DBUILD_ANDROID_PROJECTS=OFF -DANDROID_STL=c++_shared -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/opencv/android_build/out -DANDROID_ABI=arm64-v8a

Note here that I first changed current working directory to the directory we created in our previous step (android_build/) and then I executed the shell command.

In some cases you require only a small subset of the functionality specific to some modules. To reduce build time and final size, OpenCV's build system allows you to disable some modules and functionality during build

cmake .. \
-DBUILD_opencv_ittnotify=OFF -DBUILD_ITT=OFF -DCV_DISABLE_OPTIMIZATION=ON -DWITH_CUDA=OFF -DWITH_OPENCL=OFF -DWITH_OPENCLAMDFFT=OFF -DWITH_OPENCLAMDBLAS=OFF -DWITH_VA_INTEL=OFF -DCPU_BASELINE_DISABLE=ON -DENABLE_SSE=OFF -DENABLE_SSE2=OFF -DBUILD_TESTING=OFF -DBUILD_PERF_TESTS=OFF -DBUILD_TESTS=OFF -DCMAKE_BUILD_TYPE=RELEASE -DBUILD_EXAMPLES=OFF -DBUILD_DOCS=OFF -DBUILD_opencv_apps=OFF -DBUILD_SHARED_LIBS=OFF -DOpenCV_STATIC=ON -DWITH_1394=OFF -DWITH_ARITH_DEC=OFF -DWITH_ARITH_ENC=OFF -DWITH_CUBLAS=OFF -DWITH_CUFFT=OFF -DWITH_FFMPEG=OFF -DWITH_GDAL=OFF -DWITH_GSTREAMER=OFF -DWITH_GTK=OFF -DWITH_HALIDE=OFF -DWITH_JASPER=OFF -DWITH_NVCUVID=OFF -DWITH_OPENEXR=OFF -DWITH_PROTOBUF=OFF -DWITH_PTHREADS_PF=OFF -DWITH_QUIRC=OFF -DWITH_V4L=OFF -DWITH_WEBP=OFF \
-DBUILD_LIST=core,features2d,flann,imgcodecs,imgproc,stitching \
-DANDROID_NDK=/path/to/ndk -DCMAKE_TOOLCHAIN_FILE=/path/to/ndk/build/cmake/android.toolchain.cmake -DANDROID_NATIVE_API_LEVEL=android-21 -DBUILD_JAVA=OFF -DBUILD_ANDROID_EXAMPLES=OFF -DBUILD_ANDROID_PROJECTS=OFF -DANDROID_STL=c++_shared -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX:PATH=/absolute/path/to/opencv/android_build/out -DANDROID_ABI=arm64-v8a

This is how I disabled some of the modules and functionality when compiling OpenCV for my Pano Stitch & Crop App. Note how I disabled certain functionality with OFF and then I used -DBUILD_LIST to specify the modules I needed.

There might be some additional options that you want to change for your specific situation. You can use the following command to check which variables are available

cmake -LA

In case there are some errors and you changed some options to fix these errors, you might also need to first delete CMakeCache.txt file before you'll be able to successfully run cmake.

To finally build the project execute

make
make install

The .so libraries that we will use in our Android app should be located inside of the out/sdk/native/libs/ directory.

CMake makes it really easy to cross-compile projects for Android. You can use the same approach I showed above to cross-compile another related library - dlib.

Preparing Android Studio Project for Build

Here I assume that you've already created a new Android Studio project with C++ support enabled.

To properly link our native code with OpenCV libraries, you should change the CMakeLists.txt file inside of your app module folder.

# Configure path to include directories
include_directories(SYSTEM $ENV{VENDOR}/opencv/include )

# Set up OpenCV shared .so library so that it can
# be linked to your app
add_library( cv_core-lib SHARED IMPORTED)
set_target_properties( cv_core-lib
                      PROPERTIES IMPORTED_LOCATION
                      $ENV{VENDOR}/opencv/lib/${ANDROID_ABI}/libopencv_core.so )

add_library( cv_imgproc-lib SHARED IMPORTED)
set_target_properties( cv_imgproc-lib
        PROPERTIES IMPORTED_LOCATION
        $ENV{VENDOR}/opencv/lib/${ANDROID_ABI}/libopencv_imgproc.so )

add_library( cv_imgcodecs-lib SHARED IMPORTED)
set_target_properties( cv_imgcodecs-lib
        PROPERTIES IMPORTED_LOCATION
        $ENV{VENDOR}/opencv/lib/${ANDROID_ABI}/libopencv_imgcodecs.so )

...

# jnigraphics lib from NDK is used for Bitmap manipulation in native code
find_library( jnigraphics-lib jnigraphics )

# Link to your native app library
target_link_libraries( my_native-lib ${jnigraphics-lib} cv_core-lib cv_imgproc-lib cv_imgcodecs-lib other-libs...)

Here I linked with core, imgproc, and imgcodecs modules. You might need to add other OpenCV libraries depending on what functionality you use.

I decided to use the shared libraries in this case, but static linking should also work.

I also linked with jnigraphics library from the NDK. In my example code I use some Bitmap manipulation methods, so this is required.

Next you should change your module level build.gradle file

android {
    ...
    defaultConfig {
        ...
        sourceSets {
            main {
                jniLibs.srcDirs = [
                        System.getenv('VENDOR') + '/opencv/lib'
                ]
            }
        }
    }

This will will ensure that gradle packages the OpenCV .so libraries located in lib folder into the final APK. The lib/ folder should contain subfolders with each supported architecture, e.g., armeabi, armeabi-v7a,... I usually include at least armeabi-v7a and x86 to the final APK for publishing, but for testing you can only include the ABI supported by you testing devices. If you're not sure how to find the the supported ABIs for your test device, check out my Bugjaeger app which will show you the supported ABIs in Device section.

I stored the path to the folder with external libraries inside of VENDOR environment variable and I used System.getenv('VENDOR') to get the path to OpenCV. You should replace the path with wherever you store the previously built OpenCV libraries.

Additionally, in the module level build.gradle file I also usually specify which ABIs to build for, which C++ runtime to use, C++ exception support, and some other stuff

android {
    ...
    defaultConfig {
    ...
        externalNativeBuild {
            cmake {
                cppFlags "-std=c++14 -fexceptions"

                arguments "-DANDROID_TOOLCHAIN=clang",
                          "-DANDROID_STL=c++_shared",
                          "-DANDROID_PLATFORM=android-21"
            }
        }

        ndk {
            abiFilters 'armeabi-v7a'//, 'x86'
        }

        ...

Using OpenCV from Kotlin and Native C++ Code

In the following code I'll use Canny edge detection on one of my vacation photos from Krk Island. First I load the JPEG image into a Bitmap in Kotlin, then I'll use the pixels from the Bitmap in native code to perform the Canny edge detection.

This example might seem a bit lame considering all the stuff you can do with OpenCV, but I was trying to keep this simple. If you are interested in some low-latency/realtime Camera image processing on Android, you might also check out my previous post about Android Native Camera API.

It is always possible to open an image file directly in native C++ code. However, Android's Java API uses the Bitmap class in many other image processing and graphics related functions. Therefore I think it could be useful to show how to access Bitmap's pixel buffer in native code without doing unnecessary copies.

First we create a native C++ function that will be called from our Kotlin class via JNI. The function takes an input bitmap which will be used for canny edge detection. The function will write the detected edges into the given destination file.

#include <jni.h>
#include <string>

#include <opencv2/opencv.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <android/bitmap.h>

using namespace cv;

extern "C" {
JNIEXPORT void JNICALL
Java_eu_sisik_opencvsample_MainActivity_canny(
        JNIEnv *env,
        jobject /* this */,
        jobject bitmap,
        jstring destination) {

    // Get information about format and size
    AndroidBitmapInfo info;
    AndroidBitmap_getInfo(env, bitmap, &info);

    // Get pointer to pixel buffer
    void *pixels = 0;
    AndroidBitmap_lockPixels(env, bitmap, &pixels);

    // I create separate scope for input Mat here
    // to make sure it is destroyed before unlocking
    // pixels
    {
        // Check the format info before you pick the CV_ type
        // for OpenCV Mat
        // info.format == ANDROID_BITMAP_FORMAT_RGBA_8888 -> CV_8UC4

        // Now create the mat
        Mat input(info.height, info.width, CV_8UC4, pixels);

        // Perform canny edge detection
        Mat edges;
        Canny(input, edges, 200.0, 600.0, 600.0);

        // Save to destination
        const char *dest = env->GetStringUTFChars(destination, 0);
        imwrite(dest, edges);
        env->ReleaseStringUTFChars(destination, dest);
    }

    // Release the Bitmap buffer once we have it inside our Mat
    AndroidBitmap_unlockPixels(env, bitmap);
}

}

In the code above I construct an OpenCV Mat from Bitmap pixels buffer. I create a separate scope for the input Mat object to make sure it gets destructed before I call AndroidBitmap_unlockPixels().

In Kotlin code, you should make sure that you load your native .so library System.loadLibrary(). The native method is declared with external keyword

class MainActivity : AppCompatActivity() {
...
    external fun canny(src: Bitmap?, destinationPath: String): Void

    companion object {

        init {
            System.loadLibrary("native-lib")
        }
    }
...

To use our native C++ function from Kotlin

// Load my JPEG from assets
var bitmap: Bitmap? = null
assets.open("pinezici_krk_island.JPG").use {
    bitmap = BitmapFactory.decodeStream(it)
}

// Store result inside of app's cache folder
var dest = cacheDir.absolutePath + "/canny.JPG"

// Pass the bitmap to native C++ code and perform canny edge detection
canny(bitmap, dest)

// Show the processed image
ivPhoto.setImageBitmap(BitmapFactory.decodeFile(dest))

The resulting image looks like this

You can find the code for this blog post also on github.

OpenCV in My Projects

OpenCV offers a lot of cool features and using it on Android is really simple.

For now I only used OpenCV in one Android app - my Pano Stitch & Crop. If you're interested in photography, you might find this app useful.

I also created a simplified online version of this panorama tool - http://sisik.eu/pano. This web app uses OpenCV compiled into WebAssembly. Although this online version is not as optimized for mobile devices, I still think it's cool that you can embed OpenCV into web apps that easily.

As the next thing, I would like to use OpenCV for some object detection and real-time processing of camera frames. Maybe I'll publish some more info about this in my next posts.

More Useful Resources

I wrote another blogpost that uses the devices camera directly from Android NDK (C/C++). Combining it with the information you've found in this blogpost might be useful in some of your camera app projects.

OpenCV is often used for various image processing. I also show how to do efficient image processing by using C/C++ and RenderScript in this blogpost .

Even though it's only Java/Kotlin (no C++), you can process videos very efficiently by using OpengGL and MediaCodec API. I wrote another blogpost that shows how to do this.

Previous Post