Fun Stuff

Creating a Panorama Application

Note：This Tutorial and Sample Project is developed based on DJI Android SDK v2.4, an update version for Android SDK v3.1.1 will be published soon.

  • Setting up and Running the PanoDemo App

    • Setting up your Development Environment

    • Importing the Demo Project into Eclipse

    • Using the PanoDemo App

  • Creating your own Panorama App

  • Creating the User Interface

    • App permissions and basic layout

    • Creating the User Interface

  • Initialization and Setup

    • Initializing the Environment

    • Communicating with your drone

  • Taking Photos

    • Starting the Panorama Procedure

    • Capturing Photos

  • Downloading Images

    • Switching Camera Modes

    • Selecting Images

    • Downloading Images

  • Creating the Panorama

    • Stitching the images together

    • Displaying the results

  • Final Touches

    • Adding Phantom 3 Professional Support

    • Handling the Android Life Cycle

    • Bonus: Adding a Battery Indicator

  • Summary

If you come across any mistakes or bugs in this tutorial, please let us know using a Github issue, a post on the DJI forum, or commenting in the Gitbook. Please feel free to send us Github pull request and help us fix any issues. However, all pull requests related to document must follow the document style

Note: For the DJI Inspire 1 and the Phantom 3 Professional, using the DJI SDK and OpenCV Lib

In this tutorial, you will learn how to build a cool panorama app. With the help of the powerful DJI SDK and OpenCV libraries, this is actually quite easy. You will be using Intelligent Navigation's Waypoint feature and Joystick to rotate the aircraft to take photos. Let's get started!

Setting up and Running the PanoDemo App

Setting up your Development Environment

There are 8 separate required downloads to set up our development environment. Follow each link to download and install each package. The information in brackets next to each link details the version that we have used in creating our demo:

  1. Java Devlopment Kit (jdk 8u45)

  2. Eclipse (Eclipse Mars)

  3. Eclipse Android Development Tools (ADT 23.0.6)

  4. Android Native Development Kit (ndk r10e)

  5. Eclipse C/C++ Development Tooling (cdt8.7.0) Follow the "Installing and running the CDT" segment of this tutorial for further guidance: http://www.ibm.com/developerworks/opensource/library/os-ecc/?S_TACT=105AGX44&S_CMP=ART

  6. Cygwin (cygwin2.1.0)Follow the "Installing Cygwin" segment of this tutorial for further guidance: http://mindtherobot.com/blog/452/android-beginners-ndk-setup-step-by-step/

  7. OpenCV (OpenCV2.4.11) Make sure to download "OpenCV for Android"

  8. DJI Software Development Kit (DJI SDK 2.3.0)

Importing the Demo Project into Eclipse

We've already gone ahead and created a working panorama app for you to reference while you work through this tutorial. You can go ahead and download the package here:

https://github.com/DJI-Mobile-SDK/Android-PanoramaDemo.git

1.Open Eclipse->File->Import->General->Existing Projects into Workspace->Select root directory->Browse. Locate the downloaded folder "PanoDemo":

2.Select the following 3 projects:

3.Press Finish and wait for the project to build. In the Package Explorer on the left, right click on the "PanoDemo" package -> Properties -> Android. Check that under the "Libraries" section "OpenCV Library" and "DJI-SDK-LIB" have both been added. If not, press "Add..." to add them:

4.Navigate to the "Android.mk" file in the "PanoDemo" package. Update the highlighted line so that it includes the path of your own Android OpenCV package.

5.Ensure that the following folders have been added to your System Path. Some of them should have already been added in the process of following the tutorials linked in part 1.

  • C:\Program Files\Java\jdk1.8.0_45\bin;

  • D:\DJI\PanoramaDemo\release\NDK\android-ndk-r10e;

  • D:\DJI\PanoramaDemo\release\NDK\android-ndk-r10e\build;

  • D:\DJI\PanoramaDemo\release\NDK\android-ndk-r10e\prebuilt\windows-x86_64\bin

  • C:\OpenCV-2.4.11-android-sdk\OpenCV-android-sdk\sdk\native\jni;

Using the PanoDemo App

Run the "PanoDemo" project on an Android device (for instructions on how to do this click here) Once the project is installed on your device, connect it to a DJI remote controller via a USB cable, turn on the remote and its associated DJI drone, and start up the newly installed "PanoDemo" app.

--

When the app first starts up, you will be greeted with this page. Press the "Start" button to begin taking a panorama. If the "Start" button is grayed out, try unplugging and plugging your device back into the remote, and ensure that it is properly connected.

After pressing "Start", a live video feed from the drone will be displayed. Position your drone somewhere it can nice a nice photo, and press the "One key Panorama" button to start the automated panorama capturing process.

If you are connected to a Phantom 3 Professional, you will be greeted with an alert box like the one shown below. Choose either option to continue. You will learn what the difference between the two is later on in this tutorial.

The drone will start to take photos, either by rotating the gimbal (if you are using the Inspire 1), or by rotating the entire drone (if you are using the Phantom 3 Professional).

After the drone captures 8 images, each captured at a 45 degree angle differential from the last, you will be prompted to allow the app to stitch the photos together. Press "OK" to continue.

The app will display the photos onboard the drone's SD card, and automatically select the 8 it had just taken and download them. No user input is required.

The drone will then stitch the photos together. Video preview will be disabled during this period, and will instead display a static image. Please note that the stitching process will take several minutes to complete.

Once the photos have been stitched together, you will receive a dialogue box allowing you to view the finished result!

Creating your own Panorama App

Now that you've played around with the finished product, you can now have a go at creating your own app!

The demo app you just ran may seem intimidating to code yourself, but it can be broken down into several simple tasks:

We'll work our way slowly through each part of the above flowchart. Go ahead and create a new Android Application project in the Eclipse. Name your main activity "MainActivity".

You should have already imported the DJI SDK and OpenCV libraries in the previous section. If you haven't, import them now. The libraries can be found in the demo panorama app package (as shown in the previous section), or can be downloaded in part 1 from the provided links.

Right click on your project and select Properties -> Android. Check that under the "Libraries" section "OpenCV Library" and "DJI-SDK-LIB" have both been added. If not, press "Add..." to add them.

Creating the User Interface

App permissions and basic layout

Our app is going to need to use a few permissions in order to function. Let's get these all out of the way in one go. Add the following permissions and features into your Android Manifest file:

<uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
    <uses-permission android:name="android.permission.WAKE_LOCK" />
    <uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION" />
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
    <uses-permission android:name="android.permission.ACCESS_FINE_LOCATION" />
    <uses-permission android:name="android.permission.CHANGE_WIFI_STATE" />
    <uses-permission android:name="android.permission.MOUNT_UNMOUNT_FILESYSTEMS" />
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
    <uses-permission android:name="android.permission.READ_PHONE_STATE" />
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
    <uses-permission android:name="android.permission.CAMERA" />

    <uses-feature
        android:name="android.hardware.camera"
        android:required="false" />
    <uses-feature
        android:name="android.hardware.camera.autofocus"
        android:required="false" />
    <uses-feature
        android:name="android.hardware.camera.front"
        android:required="false" />
    <uses-feature
        android:name="android.hardware.camera.front.autofocus"
        android:required="false" />
    <uses-feature
        android:name="android.hardware.usb.accessory"
        android:required="false" />
    <uses-feature
        android:name="android.hardware.usb.host"
        android:required="false" />

The important permissions here allow us to connect via USB to the DJI remote controller, as well as write and read to and from external storage.

In the same AndroidManifest.xml file, alter your <application ... > section to look something like this:

<application
    // These 3 lines should have been automatically generated
    android:allowBackup="true"
    android:icon="@drawable/ic_launcher"
    android:label="@string/app_name"

    // This one too, but we're deleting it
    // android:theme="@style/AppTheme"

     // Add these 2 lines
    android:theme="@android:style/Theme.NoTitleBar.Fullscreen" 
    android:hardwareAccelerated="true">

    // Note that this line comes right after the opening tag, not inside of it
    <uses-library android:name="com.android.future.usb.accessory" />

The three lines we added remove the title bar from the layout, allow hardware acceleration, and allow us to connect via usb to the DJI remote controller respectively. Hardware acceleration will allow our app to use the GPU to help with processing intensive parts.

Within your MainActivity <activity ... > section, add the following code:

<activity
    android:name=".MainActivity"
    android:label="@string/app_name"

    ...

    // Add this line
    android:screenOrientation="landscape" >

    ...

    // Add these lines
    <intent-filter>
        <action android:name="android.hardware.usb.action.USB_ACCESSORY_ATTACHED" />
    </intent-filter>

    <meta-data
        android:name="android.hardware.usb.action.USB_ACCESSORY_ATTACHED"
        android:resource="@xml/accessory_filter" />

The first line added locks the app into a landscape orientation. The intent filter and meta data allow the app to start itself up when your device connects via USB to an external device.

Change your MainActivity (or your main activity) class so that it extends Activity.

Lastly, we're going to want our device's screen to stay on while our app goes through its automated process of creating a panorama. Add the following line of code to your onCreate() method to keep your device awake.

@Override
protected void onCreate(Bundle savedInstanceState)
{
    super.onCreate(savedInstanceState);

    // Keep screen on
    getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);

    setContentView(R.layout.activity_main);
}

Run your app. It doesn't contain anything right now, but you will notice that it starts up in landscape mode, and does not contain a title bar!

Creating the User Interface

Our app will consist of several view elements:

  • mDjiGLSurfaceView - A DjiGLSurfaceView view, which is a widget provided by the DJI mobile SDK. It acts as a specialized SurfaceView view that displays live video feed from the Drone's camera.

  • commonMessageTextView - The TextView element at the top of the app, used to display information on the app's current status.

  • startButton - The button used to start up the video feed.

  • stitchingButton - The button at the bottom of the app. Used to start the photo capture and stitching process.

There also exists a batteryTextView, but as this is not very relevant to the app's purpose, we'll be leaving this out for now. Stay tuned till the end of the tutorial for a bonus segment on how to add it though!

Add these elements to your activity_main.xml file. Here's our xml code as reference. All of this is contained within a relative layout:

<dji.sdk.widget.DjiGLSurfaceView
    android:id="@+id/mDjiSurfaceView"
    android:layout_width="match_parent"
    android:layout_height="match_parent" />

<LinearLayout
    android:id="@+id/centerLinearLayout"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_centerHorizontal="true"
    android:layout_centerVertical="true"
    android:orientation="vertical" >

    <Button
        android:id="@+id/startButton"
        android:layout_width="80dp"
        android:layout_height="75dp"
        android:layout_gravity="center_horizontal"
        android:background="@drawable/start_gray" />
</LinearLayout>

<TextView
    android:id="@+id/commonMessageTextView"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_alignParentTop="true"
    android:layout_centerHorizontal="true"
    android:gravity="center_horizontal"
    android:text="@string/commonMessageString"
    android:textColor="@android:color/white" />

<Button
    android:id="@+id/stitchingButton"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:layout_alignParentBottom="true"
    android:layout_centerHorizontal="true"
    android:text="@string/stitching" />

In our code we reference a start_gray resource. That's the image we use for our startButton. If you'd like your start button to look like ours, find the image files (PanoDemo\PanoDemo\res\drawable-mdpi) and copy them into your own image resource folder. Otherwise, you're going to have to create or find your own image files and copy them into your image resource folder.

You'll also notice that we reference three string values that don't exist for you yet. We're going to be referencing quite a few string resources, and instead of going through the hassle of adding them to our strings.xml resource file as we encounter them, let's just add them all right now. Copy the following code into your strings.xml file (res/values):

<!-- UI -->
    <string name="dji_sdk_activate_error">DJI SDK activation error</string>
    <string name="stitching">Stitching</string>
    <string name="one_key_panorama">One Button Panorama</string>
    <string name="commonMessageString">Common message</string>
    <string name="groundstation_take_control">Caution! GroundStation taking control now</string>
    <string name="init_gimabal_yaw">Initializing gimbal....</string>
    <string name="test">Test</string>
    <string name="capturing_image">Capturing</string>
    <string name="capture_image_complete">Capture complete</string>
    <string name="downloading">Downloading</string>
    <string name="battery">Battery</string>
    <string name="pressAgainExitString">Press again to exit</string>
    <string name="unsupported_drone">Unsupported drone, use Inspire1 or Phantom 3 pro and try again</string>
    <string name="video_preview_disabled_during_stitching">Warning! Video preview disabled during stitching</string>

Now that we have a UI ready to use, we have to initialize it. Head into your onCreate method and add a line calling initUIControls(), which is a method we're about to create. If your coding environment hasn't automatically created an onCreate method, add it as shown below:

@Override
    protected void onCreate(Bundle savedInstanceState)
    {
        super.onCreate(savedInstanceState);
        getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
        setContentView(R.layout.activity_main);
        //onCreate init
        initUIControls();
    }

Now add the following variables to your MainActivity class:

private DjiGLSurfaceView mDjiGLSurfaceView;
private TextView commonMessageTextView;
private LinearLayout centerLinearLayout;
private Button startButton;
private Button stitchingButton;
private ProgressDialog mDownloadDialog;

Create a method initUIControls() as shown below:

private void initUIControls()
{
    //Assign variables to their corresponding views
    mDjiGLSurfaceView=(DjiGLSurfaceView)findViewById(R.id.mDjiSurfaceView);
    commonMessageTextView=(TextView)findViewById(R.id.commonMessageTextView);
    centerLinearLayout=(LinearLayout)findViewById(R.id.centerLinearLayout);
    startButton=(Button)findViewById(R.id.startButton);
    stitchingButton=(Button)findViewById(R.id.stitchingButton);

    //Add Listeners for buttons
    startButton.setOnClickListener(this);
    stitchingButton.setOnClickListener(this);

    //Customize controls
    commonMessageTextView.setText("");
    startButton.setClickable(false);
    stitchingButton.setEnabled(false);
    stitchingButton.setText(getString(R.string.one_key_panorama));
}

We can split this function into three parts. The first two should look very familiar to you if you have experience with Android Development. We assign the variables we just created to their corresponding view elements defined in the activity_main.xml file, then set this class as the onClickListener for our buttons.

In the third part we customize our different view elements so that they appear as we'd like them to when the app just starts up. Our commonMessageTextView is set to contain no text, as there is no message we'd like to show the user. We keep both our buttons unclickable (we will enable the start button a bit later on), and set the text of our stitchingButton.

We're also going to initialize a download progress dialog box. We'll start and display this progress bar quite a bit later on, when we're downloading images from the drone to our device, but we're going to initialize it now:

private void initDownloadProgressDialog()
{
    mDownloadDialog = new ProgressDialog(MainActivity.this);
    mDownloadDialog.setTitle(R.string.downloading);
    mDownloadDialog.setIcon(android.R.drawable.ic_dialog_info);
    mDownloadDialog.setProgressStyle(ProgressDialog.STYLE_HORIZONTAL);
    mDownloadDialog.setCanceledOnTouchOutside(false);
    mDownloadDialog.setCancelable(false);
}

Call initDownloadProgressDialog() at the end of initUIControls():

private void initUIControls()
{
    ...

    initDownloadProgressDialog();
}

Lastly, because we're setting onClickListeners, we're going to need to implement OnClickListener:

public class MainActivity extends ActionBarActivity implements OnClickListener {
...
}

Build and run your project and give your new fancy UI a look. It doesn't do anything quite yet, but we'll soon fix that.

Initialization and Setup

Initializing the Environment

Although we've successfully initialized our UI, there's still more to our app that we have to set up. There are four functions we have to create:

initStitchingImageDirectory() will create directories for the source of our panorama, where we'll find the images to stitch together into a panorama, and for the result of our panorama. First add the following variables in the main class. Make sure to replace "APPNAME" with the name of your own app.

private final String STITCHING_SOURCE_IMAGES_DIRECTORY = Environment.getExternalStorageDirectory().getPath()+"/APPNAME/";
private final String STITCHING_RESULT_IMAGES_DIRECTORY = Environment.getExternalStorageDirectory().getPath()+"/APPNAME/result/";

Now create the function initStitchingImageDirectory().

private void initStitchingImageDirectory()
{
    //check if directories already exist. If not, create
    File sourceDirectory = new File(STITCHING_SOURCE_IMAGES_DIRECTORY);
    if(!sourceDirectory.exists())
    {
        sourceDirectory.mkdirs();
    }
    File resultDirectory = new File(STITCHING_RESULT_IMAGES_DIRECTORY);
    if(!resultDirectory.exists())
    {
        resultDirectory.mkdirs();
    }
}

initStitchingImageDirectory() will check your devices external storage for a folder named "APPNAME". If one does not exist, it will create said folder. It carries out the same process for the "result" folder, which exists inside "APPNAME".

Our app will store the raw images it takes in the "APPNAME" folder. Once the images are stitched together, the resultant panorama will be saved in the "result" folder.

initOpenCVLoader() loads and initializes the OpenCV library from the current application package through the initDebug() function. If initDebug() fails, we use the OpenCV manager to load the OpenCV library instead.

private boolean initOpenCVLoader()
{
    if (!OpenCVLoader.initDebug())
    {
        // Handle initialization error
        showLOG("init buildin OpenCVLoader error,going to use OpenCV Manager");
        OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this, mLoaderCallback);
        return false;
    }
    else
    {
        showLOG("init buildin OpenCVLoader success");
        return true;
    }
}

The above is a pretty standard usage of the OpenCV methods. Learn more at http://opencv.org. We load the OpenCV library because we will be using some of its provided functions to handle the bulk of our panorama construction.

Add the following into your class:

private static final String TAG = "APPNAMEMainActivity";  //debug TAG. Edit to suit the name of your own app

private void showLOG(String str)
{
    Log.e(TAG, str);
}

private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
    @Override
    public void onManagerConnected(int status) {
        switch (status) {
            case LoaderCallbackInterface.SUCCESS:
            {
                showLOG("OpenCV Manager loaded successfully");
                break;
            }
            default:
            {
                super.onManagerConnected(status);
                break;
            }
        }
    }
};

The constant TAG is just a string we'll use whenever we call Log.e() to help us locate relevant logcat messages. showLOG is just a simple helper function to display logcat messages. We implemented these here because mLoaderCallback, the callback function we use when trying to initialize OpenCV above, uses showLOG to confirm if OpenCV has been successfully loaded.

initDjiSDK() takes care of all the necessary setup required to use the DJI mobile SDK in your app, then initiates and connects to the drone.

private void initDJISDK()
{
    startDJIAoa();
    activateDJISDK();

    // The SDK initiation for Inspire 1
    DJIDrone.initWithType(this.getApplicationContext(), DJIDroneType.DJIDrone_Inspire1);
    DJIDrone.connectToDrone(); // Connect to the drone
}

You will notice that this code is made specifically for use with the Inspire 1. This tutorial also supports the use of the Phantom 3, but for simplicity's sake, we will be coding for the Inspire 1 and making some revisions at the end for those of you using a Phantom 3. In this particular part of the code, however, we will be leaving the drone type as it is, as the Inspire 1 and the Phantom 3 both use the same main controller.

initDjiSDK() calls the functions startDJIAoa() and activateDJISDK(), which we'll define below.

Add the code for startDJIAoa(), as well as the boolean isDJIAoaStarted.

private static boolean isDJIAoaStarted = false;  //DJIAoa

private void startDJIAoa()
{
    if(isDJIAoaStarted)
    {
        //Do nothing
        showLOG("DJIAoa aready started");
    }
    else
    {
        ServiceManager.getInstance();
        UsbAccessoryService.registerAoaReceiver(this);
        isDJIAoaStarted = true;
        showLOG("DJIAoa start success");
    }
    Intent aoaIntent = getIntent();
    if(aoaIntent != null)
    {
        String action = aoaIntent.getAction();
        if(action==UsbManager.ACTION_USB_ACCESSORY_ATTACHED || action == Intent.ACTION_MAIN)
        {
            Intent attachedIntent = new Intent();
            attachedIntent.setAction(DJIUsbAccessoryReceiver.ACTION_USB_ACCESSORY_ATTACHED);
            sendBroadcast(attachedIntent);
        }
    }
}

The purpose of the boolean is to ensure that we only start up DJIAoa if it isn't already started.

Add activateDJISDK():

pprivate void activateDJISDK()
{
    new Thread()
    {
        public void run()
        {
            try
            {
                DJIDrone.checkPermission(getApplicationContext(), new DJIGerneralListener()
                {
                    @Override
                    public void onGetPermissionResult(int result)
                    {
                        //result=0 is success
                        showLOG("DJI SDK onGetPermissionResult = "+result);
                        showLOG("DJI SDK onGetPermissionResultDescription = "+DJIError.getCheckPermissionErrorDescription(result));
                        if(result!=0)
                        {
                            showToast(getString(R.string.dji_sdk_activate_error)+":"+DJIError.getCheckPermissionErrorDescription(result));
                        }
                    }
                });
            }
            catch(Exception e)
            {
                showLOG("activateDJISDK() Exception");
                showToast("activateDJISDK() Exception");
                e.printStackTrace();
            }
        }
    }.start();
}

The purpose of both of these functions have been explained in our very first tutorial How to Create a Camera Application, and should be familiar to you if you have developed apps to use with DJI drones before. If you have not yet read or worked through this tutorial, we suggest you start there. If you are familiar with these functions, you will know that you will also have to register your app on the dev.dji.com website and add the APP key to your manifest file.

Lastly, add showToast(), a small helper function that displays toast messages:

private void showToast(String str)
{
    Toast.makeText(this, str, Toast.LENGTH_SHORT).show();
}

initDJICamera() creates a schedule so that CheckCameraConnectionTask(), a TimerTask, is carried out every 3 seconds:

private Timer checkCameraConnectionTimer = new Timer();
private void initDJICamera()
{
    //check camera status every 3 seconds
    checkCameraConnectionTimer.schedule(new CheckCameraConnectionTask(), 1000, 3000);
}

CheckCameraConnectionTask uses checkCameraConnectState to check if the app is connected to the drone's camera. If so, the start button turns green and will be enabled. If not, the start button remains gray and disabled. Add CheckCameraConnectionTask:

class CheckCameraConnectionTask extends TimerTask
{
    @Override
    public void run()
    {
        if(checkCameraConnectState()==true)
        {
            runOnUiThread(new Runnable() {
                public void run() {
                    startButton.setBackgroundResource(R.drawable.start_green);
                    startButton.setClickable(true);
                }
            });
        }
        else
        {
            runOnUiThread(new Runnable() {
                public void run() {
                    startButton.setBackgroundResource(R.drawable.start_gray);
                    startButton.setClickable(false);
                    stitchingButton.setEnabled(false);
                }
            });
        }
    }
}

checkCameraConnectState() uses the mobile SDK function getCamerConnectIsOk() to check the connection state. Add the function as shown below:

private boolean checkCameraConnectState(){
    //check connection
    boolean cameraConnectState = DJIDrone.getDjiCamera().getCameraConnectIsOk();
    if(cameraConnectState)
    {
        //showLOG("DJI Camera connect ok");
        return true;
    }
    else
    {
        //showLOG("DJI Camera connect failed");
        return false;
    }
}

Combined, these functions allow initDJICamera to start a schedule which will check, every 3 seconds, if the app has succesfully connected to the camera, and act accordingly. Conversely, if the app loses connection with the drone, it will turn the button gray and disable it.

Call these four functions in your onCreate() function, as shown below.

@Override
protected void onCreate(Bundle savedInstanceState)
{
    super.onCreate(savedInstanceState);
    // Keep screen on
    getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
    setContentView(R.layout.activity_main);

    //onCreate init
    initUIControls();
    initStitchingImageDirectory();
    initOpenCVLoader();
    initDJISDK();
    initDJICamera();
}

Go ahead and run your app. With any luck it'll look something like this:

And like this when it detects a connection to a drone's camera:

Communicating with your drone

At this point you should have an app that handles all the set up required, but leaves the user with start button that doesn't actually do anything. Let's fix that.

We're going to want our startButton to start the communication between the drone and our app. Specifically, we're going to want to communicate with the camera and the gimbal. We will do this through a set of callback functions.

Add the following variables to your class:

private DJIDroneType mDroneType;

//Callback functions to implement
private DJIReceivedVideoDataCallBack mReceivedVideoDataCallBack;
private DJIGimbalErrorCallBack mGimbalErrorCallBack;
private DJICameraPlayBackStateCallBack mCameraPlayBackStateCallBack;  //to get currently selected pictures count

private DJIGimbalCapacity mDjiGimbalCapacity;
private int numbersOfSelected = 0;  //updated from mCameraPlayBackStateCallBack
private final int COMMON_MESSAGE_DURATION_TIME = 2500;  //in milliseconds

First, add the following code. This is just to create a helper function showCommonMessage() that we will be using quite often in our code. It sets the commonMessageTextView element to display a given message, but only for a short period of time as allotted by COMMON_MESSAGE_DURATION_TIME.

private Timer commonMessageTimer = new Timer();

class commonMessageCleanTask extends TimerTask
{
    @Override
    public void run()
    {
        runOnUiThread(new Runnable()
        {
            public void run()
            {
                commonMessageTextView.setText("");
            }
        });
    }
}

private void showCommonMessage(final String message)
{
    runOnUiThread(new Runnable()
    {           
        @Override
        public void run()
        {
            if(message.equals(commonMessageTextView.getText()))
            {
                //filter same message
                return;
            }
            commonMessageTextView.setText(message);
            commonMessageTimer.schedule(new commonMessageCleanTask(), COMMON_MESSAGE_DURATION_TIME);
        }
    });
}

Now for the important bit. Create a method startDJICamera():

private void startDJICamera() {
    // check drone type
    mDroneType = DJIDrone.getDroneType();

    // start SurfaceView
    mDjiGLSurfaceView.start();

    // decode video data
    mReceivedVideoDataCallBack = new DJIReceivedVideoDataCallBack() {
        @Override
        public void onResult(byte[] videoBuffer, int size) {
            mDjiGLSurfaceView.setDataToDecoder(videoBuffer, size);
        }
    };

    mGimbalErrorCallBack = new DJIGimbalErrorCallBack() {
        @Override
        public void onError(final int error) {
            if (error != DJIError.RESULT_OK) {
                runOnUiThread(new Runnable() {
                    public void run() {
                        showCommonMessage("Gimbal error code=" + error);
                    }
                });
            }
        }
    };

    mCameraPlayBackStateCallBack = new DJICameraPlayBackStateCallBack() {
        @Override
        public void onResult(DJICameraPlaybackState mState) {
            numbersOfSelected = mState.numbersOfSelected;
        }
    };

    DJIDrone.getDjiCamera().setReceivedVideoDataCallBack(
            mReceivedVideoDataCallBack);
    DJIDrone.getDjiGimbal().setGimbalErrorCallBack(mGimbalErrorCallBack);
    DJIDrone.getDjiCamera().setDJICameraPlayBackStateCallBack(
            mCameraPlayBackStateCallBack);

    DJIDrone.getDjiGimbal().startUpdateTimer(1000);
}

startDJICamera() starts by setting mDroneType to the type of the drone currently connected to. Next, it starts up the Surface View element. It is crucial that we do this at the beginning of the function, before we set the ReceivedVideoDataCallBack function which will send data to the Surface View element. If the Surface View element is not yet started when the callback function is set, the drone will still attempt to send data to it, and the app may crash.

The rest of the function defines four different callback function, sets them, then starts an update timer for the gimbal to call its callback functions. Let's go through each callback function one by one.

mReceivedVideoDataCallBack uses setDataToDecoder() to send video data from the drone's camera to mDjiGLSurfaceView to be decoded and displayed.

mGimbalErrorCallback handles situations where the gimbal encounters an error, in which case the int error will not equal DJIError.RESULT_OK, and an error message should be printed out. This check occurs every second, as specified by startUpdateTimer(1000) at the bottom of startDJICamera(), which specifies how frequently the gimbal callback functions should be called.

mCameraPlayBackStateCallBack updates the numbersOfSelected variable, which keeps track of the number of currently selected photos. Users can select photos while in Multiple Edit mode, which we'll learn more about later on in this tutorial.

Finally, add to your onClick() function so that it calls startDJICamera when startButton is pressed.

@Override
public void onClick(View v) {
    switch (v.getId()) {
        case R.id.startButton:

            // start dji camera
            startDJICamera();

            centerLinearLayout.setVisibility(View.INVISIBLE); // hide startButton
            stitchingButton.setEnabled(true);
            break;
        default:
            break;
    }

}

Taking Photos

Starting the Panorama Procedure

At this point, our app is ready and connected to our drone. The callback functions we implemented allow us to view a live video feed from the drone, get the number of currently selected photos (this will make more sense later on), as well as receive and display gimbal error messages. Now that these capabilities are in place, we can move on to coding the Panorama capturing procedure.

In the previous section we added some code so that pressing startButton will call stitchingButton.setEnabled(true). If you recall, stitchingButton is the button at the bottom of the app that, when pressed, will start the panorama capturing process. Let's add some functionality to our onClickListener to support that. In our onClick() method, add the following switch case:

case R.id.stitchingButton:
    cleanSourceFolder();
    stitchingButton.setEnabled(false);
    stitchingButton.setText(getString(R.string.one_key_panorama));

    if(mDroneType==DJIDroneType.DJIDrone_Inspire1)
    {
        handler.sendMessage(handler.obtainMessage(HANDLER_INSPIRE1_CAPTURE_IMAGES,""));
    }
    else
    {
        showCommonMessage(getString(R.string.unsupported_drone));
    }
    break;

The first function we want to call when the user presses stitchingButton is cleanSourceFolder(), which clears all files (but not folders) out of our source image directory:

private void cleanSourceFolder()
{
    File sourceDirectory = new File(STITCHING_SOURCE_IMAGES_DIRECTORY);
    //clean source file, except folders
    for(File file : sourceDirectory.listFiles())
    {
        if(!file.isDirectory())
        {
            file.delete();
        }
    }
}

We get our source directory from the constant STITCHING_SOURCE_IMAGES_DIRECTORY, which we had already defined previously. Here it is again to refresh your memory:

private final String STITCHING_SOURCE_IMAGES_DIRECTORY = Environment.getExternalStorageDirectory().getPath()+"/APPNAME/";

Next, if we are connected to an Inspire 1 drone, we carry out some code which will kick off the whole panorama capturing process. This involves a lot of heavy work, including capturing and processing images. In order to keep the UI thread running smoothly throughout all this, we're going to be executing our code through a handler.

Create a new handler:

private Handler handler = new Handler(new Handler.Callback() {
    @Override
    public boolean handleMessage(Message msg)
    {
        // handleMessage code
    }
});

We're going to be sending our handler different messages for it to respond to by carrying out appropriate code. Add a switch statement to the handleMessage() method in our handler, with a single case HANDLER_INSPIRE1_CAPTURE_IMAGES:

switch (msg.what)
{
    case HANDLER_INSPIRE1_CAPTURE_IMAGES:
    // capture images code
    break;
}

Capturing Photos

To create a panorama, we're going to have our drone first capture 8 different photos, each taken 45 degrees away from the last. Later on we will be stitching these photos together, but lets just worry about taking the photos for now.

Add the following constants and variables to your class:

private final int HANDLER_SHOW_COMMON_MESSAGE = 1000;
private final int HANDLER_SET_STITCHING_BUTTON_TEXT = 1001;
private final int HANDLER_ENABLE_STITCHING_BUTTON = 1003;
private final int HANDLER_SHOW_STITCHING_OR_NOT_DIALOG = 1005;
private final int CAPTURE_IMAGE_GIMBAL_INIT_POSITION = -2300;  //-2300 for inspire1
private final int HANDLER_INSPIRE1_CAPTURE_IMAGES = 2000;

private final int CAPTURE_IMAGE_NUMBER = 8;  //number of images to take to form a panorama
private int captureImageFailedCount = 0;
private boolean isCheckCaptureImageFailure = false;  //check dji camera capture result

We have coded our handler so that if we send it the message HANDLER_INSPIRE1_CAPTURE_IMAGES it'll carry out our code in the appropriate switch case. Let's add the following code to that case:

case HANDLER_INSPIRE1_CAPTURE_IMAGES:
{
    new Thread()
    {
        public void run()
        {
            //rotate gimble to take photos
            int imgIndex=0;
            showCommonMessage(getString(R.string.init_gimabal_yaw));
            //init the gimbal yaw to Clockwise Min
            while(DJIDrone.getDjiGimbal().getYawAngle()>CAPTURE_IMAGE_GIMBAL_INIT_POSITION)
            {
                DJIGimbalRotation mYaw_relative = new DJIGimbalRotation(true,false,false, 1000);
                DJIDrone.getDjiGimbal().updateGimbalAttitude(null,null,mYaw_relative);
                try
                {
                    sleep(50);
                }
                catch(InterruptedException e)
                {
                    e.printStackTrace();
                }
            }
            DJIGimbalRotation mYaw_init_stop = new DJIGimbalRotation(true,false,false, 0);
            DJIDrone.getDjiGimbal().updateGimbalAttitude(null,null,mYaw_init_stop);
            try
            {
                sleep(50);
            }
            catch(InterruptedException e)
            {
                e.printStackTrace();
            }

            // Take specified number of photos
            for(int i=-180;i<180;i+=(360/CAPTURE_IMAGE_NUMBER))
            {
                imgIndex++;
                showCommonMessage(getString(R.string.capturing_image)+" "+imgIndex+"/"+CAPTURE_IMAGE_NUMBER);
                DJIGimbalRotation mYaw = new DJIGimbalRotation(true,true,true, i);
                DJIDrone.getDjiGimbal().updateGimbalAttitude(null,null,mYaw);
                try
                {
                    sleep(3000);
                }
                catch(InterruptedException e)
                {
                    e.printStackTrace();
                }
                DJICameraTakePhoto();
                try
                {
                    sleep(3000);
                }
                catch(InterruptedException e)
                {
                    e.printStackTrace();
                }
            }

            //gimbal yaw face front
            showCommonMessage(getString(R.string.capture_image_complete));
            DJIGimbalRotation mYaw_front = new DJIGimbalRotation(true,false,true, 0);
            DJIDrone.getDjiGimbal().updateGimbalAttitude(null,null,mYaw_front);
            try
            {
                Thread.sleep(3000);
            }
            catch(InterruptedException e)
            {
                e.printStackTrace();
            }
            if(captureImageFailedCount!=0)
            {
                showCommonMessage("Check "+captureImageFailedCount+" images capture failed,Task Abort!");
                captureImageFailedCount=0;
                handler.sendMessage(handler.obtainMessage(HANDLER_SET_STITCHING_BUTTON_TEXT,getString(R.string.one_key_panorama)));
                handler.sendMessage(handler.obtainMessage(HANDLER_ENABLE_STITCHING_BUTTON,""));
            }
            else
            {
                showCommonMessage("Check "+CAPTURE_IMAGE_NUMBER+" images capture all success,continue....");
                try
                {
                    Thread.sleep(3000);
                }
                catch (InterruptedException e)
                {
                    e.printStackTrace();
                }
                //show dialog
                handler.sendMessage(handler.obtainMessage(HANDLER_SHOW_STITCHING_OR_NOT_DIALOG, ""));
            }
        }
    }.start();
    break;
}

Create the method DJICameraTakePhoto():

private void DJICameraTakePhoto()
{
    CameraCaptureMode mode = CameraCaptureMode.Camera_Single_Capture;
    DJIDrone.getDjiCamera().startTakePhoto(mode, new DJIExecuteResultCallback()
    {
        @Override
        public void onResult(DJIError mErr)
        {
            if(mErr.errorCode==DJIError.RESULT_OK)
            {
                showLOG("take photo success");
            }
            else
            {
                if(isCheckCaptureImageFailure)
                {
                    captureImageFailedCount++;
                    handler.sendMessage(handler.obtainMessage(HANDLER_SHOW_COMMON_MESSAGE, "Capture image on error"));
                }
                showLOG("take photo failed");
            }
        } 
    });
}

Add the following switch cases to your handler's handleMessage() method:

case HANDLER_SHOW_COMMON_MESSAGE:
{
    showCommonMessage((String)msg.obj);
    break;
}
case HANDLER_SET_STITCHING_BUTTON_TEXT:
{
    stitchingButton.setText((String)msg.obj);
    break;
}
case HANDLER_ENABLE_STITCHING_BUTTON:
{
    stitchingButton.setEnabled(true);
    break;
}
case HANDLER_SHOW_STITCHING_OR_NOT_DIALOG:
{
    //capture complete, show dialog, user determines to continue or cancel
    break;
}

That was a huge amount of code to just throw at you, so let's break it down. The first part of the thread initializes the variables imgIndex, which we will use to keep track of how many images we have taken, displays a common message, then rotates the gimbal to a predefined angle given by the constant CAPTURE_IMAGE_GIMBAL_INIT_POSITION - we set CAPTURE_IMAGE_GIMBAL_INIT_POSITION to -2300, while the minimum rotation value that the gimbal can reach is -2400, leaving a small margin of error to work with. We rotate to this position using these lines of code contained in a while loop:

DJIGimbalRotation mYaw_relative = new DJIGimbalRotation(true,false,false, 1000);
DJIDrone.getDjiGimbal().updateGimbalAttitude(null,null,mYaw_relative);

We feed updateGimbalAttitude() a DJIGimbalRotation object. The parameters given to construct mYaw_relative are set so that the gimbal's movement is enabled and rotates in a negative direction, and rotates at a speed of 1000 units. We also call sleep(50) to allow the gimbal time to rotate before restarting the while loop and checking the position of the gimbal.

This rotates the gimbal to its most counter-clockwise position. This ensures that later on, when spinning the gimbal clockwise 360 degrees to take the photos for the panorama, that it does not reach the end of its rotational range.

Once the gimbal has reached its required position, we exit the while loop and execute the following two lines, setting the speed of the gimbal to 0 to halt its movement:

DJIGimbalRotation mYaw_init_stop = new DJIGimbalRotation(true,false,false, 0);
DJIDrone.getDjiGimbal().updateGimbalAttitude(null,null,mYaw_init_stop);

Once the gimbal is in its starting orientation, we take 8 photos, rotating the gimbal by 360/CAPTURE_IMAGE_NUMBER each time. This is carried out in this for loop:

for(int i=-180;i<180;i+=(360/CAPTURE_IMAGE_NUMBER))
{
    imgIndex++;
    showCommonMessage(getString(R.string.capturing_image)+" "+imgIndex+"/"+CAPTURE_IMAGE_NUMBER);
    DJIGimbalRotation mYaw = new DJIGimbalRotation(true,true,true, i);
    DJIDrone.getDjiGimbal().updateGimbalAttitude(null,null,mYaw);
    try
    {
        sleep(3000);
    }
    catch(InterruptedException e)
    {
        e.printStackTrace();
    }
    DJICameraTakePhoto();
    try
    {
        sleep(3000);
    }
    catch(InterruptedException e)
    {
        e.printStackTrace();
    }
}

In this case, we adjust the parameters when constructing mYaw so that we rotate the gimbal to an absolute position, rather than to set its speed. Photos are taken using the DJICameraTakePhoto() function. After sending a signal to the drone to either rotate the gimbal or take a photo, we call sleep(3000) so that the drone has time to carry our the command before it receives the next one. We will use this technique very frequently. Photos are taken through DJICameraTakePhoto, which will also increment captureImageFailedCount upon failing to capture an image.

We return the gimbal to a front facing position by feeding updateGimbalAttitude mYaw_front, which directs the gimbal to move to an absolute angle of 0. We check if captureImageFailedCount is a non-zero value, meaning that not all 8 photos have been captured succesfully. If so, the handler is sent messages to reset the stitching button text and reenable it so that the user can start the process over(HANDLER_SET_STITCHING_BUTTON_TEXT and HANDLER_ENABLE_STITCHING_BUTTON).

If all 8 pictures have been captured successfully, we send the handler HANDLER_SHOW_STITCHING_OR_NOT_DIALOG, which we've left empty for now.

Run your code and watch as your Inspire rotates its gimbal to take 8 photos, encompassing all 360 degrees!

Downloading Images

Switching Camera Modes

The photos your drone just took are stored on its SD card. Now we have to download those images onto our device, where they can be processed into a panorama. Here's a quick flowchart detailing how we're going to do that:

As we've just finished taking photos, the drone is in Capture Mode. From there we will need to navigate to Playback Mode, which is the general mode through which we can view and access media files. Within Playback Mode we have Multiple Preview Mode, which allows us to view multiple files at once in a grid layout. From there we can switch to Multiple Edit Mode which is the mode which allows downloading of media files. From there it's a simple matter of selecting and downloading the last 8 files on the SD card.

Let's confirm with the app user before setting off this chain of events. In our handler's HANDLER_INSPIRE1_CAPTURE_IMAGES case, upon successfully taking 8 photos it sends itself a HANDLER_SHOW_STITCHING_OR_NOT_DIALOG message, which we've left blank. Now we'll add the code to present a dialog box asking the user to confirm that they want to continue with the process:

case HANDLER_SHOW_STITCHING_OR_NOT_DIALOG:
{
    //capture complete,show dialog,user determing stitching or cancel
    DialogInterface.OnClickListener positiveButtonOnClickListener=new DialogInterface.OnClickListener()
    {
        @Override
        public void onClick(DialogInterface dialog, int which)
        {
            //set dji camera playback mode
            handler.sendMessage(handler.obtainMessage(HANDLER_SET_DJI_CAMERA_PALYBACK_MODE, ""));
        }
    };
    DialogInterface.OnClickListener negativeButtonOnClickListener=new DialogInterface.OnClickListener()
    {
        @Override
        public void onClick(DialogInterface dialog, int which)
        {
            handler.sendMessage(handler.obtainMessage(HANDLER_SET_STITCHING_BUTTON_TEXT,getString(R.string.one_key_panorama)));
            handler.sendMessage(handler.obtainMessage(HANDLER_ENABLE_STITCHING_BUTTON,""));
        }
    };
    break;
    new AlertDialog.Builder(MainActivity.this).setTitle("Message").setMessage("Capture complete,stitching?").setPositiveButton("OK", positiveButtonOnClickListener).setNegativeButton("Cancel", negativeButtonOnClickListener).show();
}

Clicking the positive button in the dialog box that is created here will send the handler a HANDLER_SET_DJI_CAMERA_PALYBACK_MODE message, while pressing the negative button resets the stitching button text and enables it, effectively returning the user back to the beginning. Let's add the code to handle HANDLER_SET_DJI_CAMERA_PALYBACK_MODE. First the constant:

private final int HANDLER_SET_DJI_CAMERA_PALYBACK_MODE = 2004;

Then the code in the handler:

case HANDLER_SET_DJI_CAMERA_PALYBACK_MODE:
{
    //set camera playback mode to pull back images
    showCommonMessage("Set camera playback mode");
    CameraMode mode_playback = CameraMode.Camera_PlayBack_Mode;
    DJIDrone.getDjiCamera().setCameraMode(mode_playback, new DJIExecuteResultCallback()
    {
        @Override
        public void onResult(DJIError mErr)
        {
            if(mErr.errorCode==DJIError.RESULT_OK)
            {
                //enter multi preview mode