Getting video stream from Android’s display

Getting video stream from Android’s display

This is something that has been tried to be achieved in various ways. What other people usually do is to take screenshots at regular intervals and stitch them together to make a video out of it. What I’m doing here is pretty different an much better than that approach.

And so I also present a way to capture video frames from Android’s default display and do further processing as one pleases.

I’ll broadly use two main APIs viz. MediaCodec (added in API level 16) and DisplayManager (added in API level 19). So this will limit our app to a minimum API level of 19 which is Kitkat. And further if you want to mirror the output of secure windows as well, then you’ll have to push your apk to /system/priv-app/ which will require having root access on your phone.

My logic is :

  • Create a video encoder.
  • Get an input Surface from the encoder using createInputSurface() method of the encoder object.
  • Pass this surface to DisplayManager so that the display manager will route it’s output to this surface.
  • Use the dequeOutputBuffer() method which will return you the H.264 encoded frames of your video.
  • And if you want to get raw video frames you can further pass these AVC encoded frames to a video decoder and get the raw frames. However as of now I’m not covering that in this blog post.

Let’s start with the code :

First we need to create an encoder, configure it and get an input surface out of it.

We will then pass the above created surface to createVirtualDisplay() method.

The DisplayManager will keep drawing the contents of the Android screen on our virtual display which in turn will feed the contents to video encoder through the surface.

The variable encodedData contains the AVC encoded frame and will be updated in every loop.
Now it’s upto you what you want to do with this. You can write it to a file. You can stream it over the network (Although that will require a lot more efforts).
You can further pass it to a video decoder to get raw frame and convert it to a bitmap or something. I’ve implemented the passing of video frames to video decoder in one of my projects here :
https://github.com/omerjerk/RemoteDroid/blob/master/app/src/main/java/in/omerjerk/remotedroid/app/ServerService.java#L285

6 thoughts on “Getting video stream from Android’s display

  1. Hi Rani,
    I’ve implemented this thing in my RemoteDroid app. Link to Github : https://github.com/omerjerk/RemoteDroid/
    Play Store link is down due to some issues. The app may not run as I’ve lately been doing major refactoring in the app. But you get a basic idea of how I’m doing stuffs.

    P.S. Don’t forget to start the repo. 🙂

    Thanks,
    Umair

  2. Hellow Omer,
    Is there anyway to open your project in AndroidStudio or Eclipse?? I have been trying to open it in both but it is giving Parsing Error every time.. Can you share any documentation of this Project… I want to develop it in some other language! but i need to see how it is working..have seen the youtube link you shared but wanna try for myself.. so please suggest anyway to open RemoteDroid in some IDE.
    Thanks in Advance!

  3. The project is very well buildable in Android Studio. It won’t work on eclipse as it’s based on the new gradle build system.
    I can help you if you can provide me more info as to what you’re facing via email.
    Thanks.

Leave a Reply

Your email address will not be published. Required fields are marked *