之前有介绍过这方面的知识内容
Android音视频-视频采集(系统API预览)
Android音视频-视频采集(Camera预览)
Android音视频-视频采集(Camera2预览基础)
Android音视频-视频采集(Camera2功能实现)
上面的都是基于Android的高级应用层API来实现的音视频的采集和编码,下面我们要打开摄像头通过OpenGL ES底层native代码来渲染视频画面。
简介
总体的思路是从摄像头采集到视频的数据,然后传递给底层的OpenGL ES来渲染显示到上层的SurfaceView上面,项目总体文件结构如下:
下面的实现列出关键的步骤和代码,完整的项目代码文章末尾列出。
Java上层代码实现
这里主要是Android应用层的Camera的一些配置切换等操作,并且通过native接口调用底层的实现代码
代码简单类图:
前面的一些权限配置等基础的代码就不列出。主要类是LPreviewScheduler.java文件
- LPreviewView:SurfaceView继承封装类
- LVideoCamera:封装操作Camera的一些方法
- LPreviewScheduler:总体调度器,包含上面两个类,并且对外提供应用接口
上面的操作Camera的类LVideoCamera.java是基于Camera来做的,详细可见以前文章的一些介绍。下面主要列出LPreviewScheduler的代码实现。
package com.lyman.camerapreview.preview;
import android.hardware.Camera.CameraInfo;
import android.util.Log;
import android.view.Surface;
import android.view.SurfaceHolder;
public class LPreviewScheduler
implements LVideoCamera.LVideoCameraCallback, LPreviewView.LPreviewViewCallback {
static {
System.loadLibrary("camerapreview");
}
private static final String TAG = "LPreviewScheduler";
private LPreviewView mPreviewView;
private LVideoCamera mCamera;
public LPreviewScheduler(LPreviewView previewView, LVideoCamera camera) {
isStopped = false;
this.mPreviewView = previewView;
this.mCamera = camera;
this.mPreviewView.setCallback(this);
this.mCamera.setCallback(this);
}
public int getNumberOfCameras() {
if (null != mCamera) {
return mCamera.getNumberOfCameras();
}
return -1;
}
/** 切换摄像头, 底层会在返回来调用configCamera, 之后在启动预览 **/
public native void switchCameraFacing();
private boolean isFirst = true;
private boolean isSurfaceExsist = false;
private boolean isStopped = false;
private int defaultCameraFacingId = CameraInfo.CAMERA_FACING_FRONT;
@Override
public void createSurface(Surface surface, int width, int height){
startPreview(surface, width, height, defaultCameraFacingId);
}
private void startPreview(Surface surface, int width, int height, final int cameraFacingId){
if (isFirst) {
prepareEGLContext(surface, width, height, cameraFacingId);
isFirst = false;
} else {
createWindowSurface(surface);
}
isSurfaceExsist = true;
}
public void startPreview(final int cameraFacingId){
try {
if(null != mPreviewView){
SurfaceHolder holder = mPreviewView.getHolder();
if(null != holder){
Surface surface = holder.getSurface();
if(null != surface){
startPreview(surface, mPreviewView.getWidth(), mPreviewView.getHeight(), cameraFacingId);
}
}
}
} catch (Throwable e) {
e.printStackTrace();
}
}
public native void prepareEGLContext(Surface surface, int width, int height, int cameraFacingId);
public native void createWindowSurface(Surface surface);
@Override
public native void resetRenderSize(int width, int height);
@Override
public void destroySurface(){
if(isStopped){
this.stopPreview();
} else{
this.destroyWindowSurface();
}
isSurfaceExsist = false;
}
public void stop() {
isStopped = true;
if(!isSurfaceExsist){
this.stopPreview();
}
}
private void stopPreview(){
this.destroyEGLContext();
isFirst = true;
isSurfaceExsist = false;
isStopped = false;
}
public native void destroyWindowSurface();
public native void destroyEGLContext();
/**
* 当Camera捕捉到了新的一帧图像的时候会调用这个方法,因为更新纹理必须要在EGLThread中,
* 所以配合下updateTexImageFromNative使用
**/
@Override
public native void notifyFrameAvailable();
public void onPermissionDismiss(String tip){
Log.i("problem", "onPermissionDismiss : " + tip);
}
private CameraConfigInfo mConfigInfo;
/** 当底层创建好EGLContext之后,回调回来配置Camera,返回Camera的配置信息,然后在EGLThread线程中回调回来继续做Camera未完的配置以及Preview **/
public CameraConfigInfo configCameraFromNative(int cameraFacingId){
defaultCameraFacingId = cameraFacingId;
mConfigInfo = mCamera.configCameraFromNative(cameraFacingId);
return mConfigInfo;
}
/** 当底层EGLThread创建初纹理之后,设置给Camera **/
public void startPreviewFromNative(int textureId) {
mCamera.setCameraPreviewTexture(textureId);
}
/** 当底层EGLThread更新纹理的时候调用这个方法 **/
public void updateTexImageFromNative() {
mCamera.updateTexImage();
}
/** 释放掉当前的Camera **/
public void releaseCameraFromNative(){
mCamera.releaseCamera();
}
}
从开始显示到停止的过程通过下面的图串一遍
上层代码还是比较简单的。
Native层代码分析
下面主要介绍底层的C++实现代码。
配置项目
首先看一下C++文件的目录结构:
上面我配置了两个输出so库,一个commontool和一个camerapreview库,其中camerapreview引用commontool中的实现代码。
CmakeLists.txt在项目中的配置略,看其中的配置C++代码如下:
cmake_minimum_required(VERSION 3.4.1)
find_library( # Sets the name of the path variable.
log-lib
# Specifies the name of the NDK library that
# you want CMake to locate.
log )
set(PATH_TO_ROOT ${CMAKE_SOURCE_DIR}/src/main/cpp)
include_directories(${PATH_TO_ROOT}/libcommon/)
file(GLOB FILES_LIB_COMMON "${PATH_TO_ROOT}/libcommon/*.cpp")
file(GLOB FILES_LIB_COMMON_EGL_CORE "${PATH_TO_ROOT}/libcommon/egl_core/*.cpp")
file(GLOB FILES_LIB_COMMON_MSG_Q "${PATH_TO_ROOT}/libcommon/message_queue/*.cpp")
file(GLOB FILES_LIB_COMMON_GL_MEDIA "${PATH_TO_ROOT}/libcommon/opengl_media/*.cpp")
file(GLOB FILES_LIB_COMMON_GL_MEDIA_RENDER "${PATH_TO_ROOT}/libcommon/opengl_media/render/*.cpp")
file(GLOB FILES_LIB_COMMON_GL_MEDIA_TEXTURE "${PATH_TO_ROOT}/libcommon/opengl_media/texture/*.cpp")
file(GLOB FILES_LIB_COMMON_GL_MEDIA_TEXTURE_COPIER "${PATH_TO_ROOT}/libcommon/opengl_media/texture_copier/*.cpp")
file(GLOB FILES_LIB_COMMON_SL_MEDIA "${PATH_TO_ROOT}/libcommon/opensl_media/*.cpp")
add_library( # Sets the name of the library.
commontool
# Sets the library as a shared library.
SHARED
# Provides a relative path to your source file(s).
${FILES_LIB_COMMON}
${FILES_LIB_COMMON_EGL_CORE}
${FILES_LIB_COMMON_MSG_Q}
${FILES_LIB_COMMON_GL_MEDIA}
${FILES_LIB_COMMON_GL_MEDIA_RENDER}
${FILES_LIB_COMMON_GL_MEDIA_TEXTURE}
${FILES_LIB_COMMON_GL_MEDIA_TEXTURE_COPIER}
${FILES_LIB_COMMON_SL_MEDIA}
)
# Include libraries needed for renderer lib
target_link_libraries(
commontool
${log-lib}
android
GLESv2
EGL
OpenSLES)
include_directories(${PATH_TO_ROOT}/camera_preview/)
file(GLOB FILES_LIB_CAMERA_PREVIEW "${PATH_TO_ROOT}/camera_preview/*.cpp")
add_library( # Sets the name of the library.
camerapreview
# Sets the library as a shared library.
SHARED
# Provides a relative path to your source file(s).
${FILES_LIB_CAMERA_PREVIEW}
${PATH_TO_ROOT}/LPreviewScheduler.cpp
)
# Include libraries needed for renderer lib
target_link_libraries(
camerapreview
commontool)
其中核心交互的类文件为mv_recording_preview_controller.cpp,它与上层的交互过程已经在上面列出来,下面主要分析这个类的实现。
实现代码
类文件mv_recording_preview_controller.cpp为和上层交互的主要集合类,串联了其他各个模块的实现。
主要的使用类如下:
- MVRecordingPreviewHandler:实现了消息机制,上层所有方法的调用通过它转发到消息队列调用
- EGLSurface:通过接受上层的Surface构造显示视图
- render_preview_renderer:渲染视图逻辑处理类,这个里面主要是OpenGL ES相关的实现代码
底层的实现的代码还是在OpenGL ES的渲染部分,里面的细节实现代码还没有完全弄明白,这里先串联出整个的渲染的流程,具体细节日后慢慢了解了再细究。下面是render_preview_renderer.cpp的实现大体流程:
其中渲染模块的OpenGL代码没有完全吃透。待以后补充。
本文Demo代码
原参考书籍代码