NXP

5.2 应用程序和驱动程序中buffer的传输流程

2019-07-12 13:41发布

摄像头采集的数据,是通过buffer的形式传到驱动程序中,然后驱动程序使能CSI设备来采集数据。之后将采集到的数据再次通过buffer的形式传递给应用程序中。这个过程中使用了VIDIOC_REQBUFSVIDIOC_QUERYBUFVIDIOC_QBUFVIDIOC_STREAMONVIDIOC_DQBUF这些ioctl调用。下面就来具体分析这个流程。
1. 内核中的数据结构 内核中有关V4L2的头文件是/include/uapi/linux/videodev2.h文件。
1.1 v4l2_requestbuffers结构体: enum v4l2_memory { V4L2_MEMORY_MMAP = 1, //内核空间开辟缓冲区 V4L2_MEMORY_USERPTR = 2, //用户空间的应用中开辟缓冲区 V4L2_MEMORY_OVERLAY = 3, V4L2_MEMORY_DMABUF = 4, };enum v4l2_buf_type { V4L2_BUF_TYPE_VIDEO_CAPTURE = 1, V4L2_BUF_TYPE_VIDEO_OUTPUT = 2, V4L2_BUF_TYPE_VIDEO_OVERLAY = 3, V4L2_BUF_TYPE_VBI_CAPTURE = 4, V4L2_BUF_TYPE_VBI_OUTPUT = 5, V4L2_BUF_TYPE_SLICED_VBI_CAPTURE = 6, V4L2_BUF_TYPE_SLICED_VBI_OUTPUT = 7, /* Experimental */ V4L2_BUF_TYPE_VIDEO_OUTPUT_OVERLAY = 8, V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE = 9, V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE = 10, /* Deprecated, do not use */ V4L2_BUF_TYPE_PRIVATE = 0x80, };
struct v4l2_requestbuffers { __u32 count; __u32 type; /* enum v4l2_buf_type */ __u32 memory; /* enum v4l2_memory */ __u32 reserved[2]; };
这个v4l2_requestbuffers结构体用在VIDIOC_REQBUFSioctl中,它代表想要申请内存的属性。 count代表想要申请的buffers的个数; type代表buffer的类型,对于capture设备来说,一般都是V4L2_BUF_TYPE_VIDEO_CAPTURE memory关于这个memory的类型,是传输视频帧的方法,主要有两种方法,readwrite方法和流I/O的方法。常用的是流I/O的方法,应用程序通过mmap系统调用映射到用户地址空间,从而使得用户和内核只需要交换缓冲区指针即可,不需要拷贝帧。 比如在应用程序中: struct v4l2_requestbuffers req; memset(&req, 0, sizeof (req)); req.count = TEST_BUFFER_NUM; req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; req.memory = V4L2_MEMORY_MMAP; ioctl(fd_v4l, VIDIOC_REQBUFS, &req)

1.2 v4l2_buffer结构体: 首先介绍一下v4l2_plane结构体,它如下所示,包含在v4l2_buffer结构体中: /** * struct v4l2_plane - plane info for multi-planar buffers * @bytesused: number of bytes occupied by data in the plane (payload) * @length: size of this plane (NOT the payload) in bytes * @mem_offset: when memory in the associated struct v4l2_buffer is * V4L2_MEMORY_MMAP, equals the offset from the start of * the device memory for this plane (or is a "cookie" that * should be passed to mmap() called on the video node) * @userptr: when memory is V4L2_MEMORY_USERPTR, a userspace pointer * pointing to this plane * @fd: when memory is V4L2_MEMORY_DMABUF, a userspace file * descriptor associated with this plane * @data_offset: offset in the plane to the start of data; usually 0, * unless there is a header in front of the data * * Multi-planar buffers consist of one or more planes, e.g. an YCbCr buffer * with two planes can have one plane for Y, and another for interleaved CbCr * components. Each plane can reside in a separate memory buffer, or even in * a completely separate memory node (e.g. in embedded devices). */ struct v4l2_plane { __u32 bytesused; __u32 length; union { __u32 mem_offset; unsigned long userptr; __s32 fd; } m; __u32 data_offset; __u32 reserved[11]; }; 这个plane我翻译成位面,这个结构体是用在多位面的情况下每个位面的一些信息。 bytesused表示已经使用的字节数(载重的大小)。 length表示这个位面所拥有的字节数(不是载重)。 m.offset表示从设备内存基址开始到这个位面的偏移值。 m.userptr表示用户空间的一个指针指向这个位面。 m.fd表示用户空间与此位面相关连的一个文件描述符。 data_offset表示这个位面中数据开始的偏移值,一般为0.
这个结构体中m那个联合中的参数与enumv4l2_memory结构体中的数据是息息相关的。这个v4l2_plane一般嵌入在v4l2_buffer结构体中,同时v4l2_buffer结构体中也有一个memory字段,即enumv4l2_memory memory== V4L2_MEMORY_MMAP的时候,则m.memoffset设置。 memory== V4L2_MEMORY_USERPTR的时候,则m.userptr设置。 memory== V4L2_MEMORY_DMABUF的时候,则m.fd设置。 /** * struct v4l2_buffer - video buffer info * @index: id number of the buffer * @type: enum v4l2_buf_type; buffer type (type == *_MPLANE for * multiplanar buffers); * @bytesused: number of bytes occupied by data in the buffer (payload); * unused (set to 0) for multiplanar buffers * @flags: buffer informational flags * @field: enum v4l2_field; field order of the image in the buffer * @timestamp: frame timestamp * @timecode: frame timecode * @sequence: sequence count of this frame * @memory: enum v4l2_memory; the method, in which the actual video data is * passed * @offset: for non-multiplanar buffers with memory == V4L2_MEMORY_MMAP; * offset from the start of the device memory for this plane, * (or a "cookie" that should be passed to mmap() as offset) * @userptr: for non-multiplanar buffers with memory == V4L2_MEMORY_USERPTR; * a userspace pointer pointing to this buffer * @fd: for non-multiplanar buffers with memory == V4L2_MEMORY_DMABUF; * a userspace file descriptor associated with this buffer * @planes: for multiplanar buffers; userspace pointer to the array of plane * info structs for this buffer * @length: size in bytes of the buffer (NOT its payload) for single-plane * buffers (when type != *_MPLANE); number of elements in the * planes array for multi-plane buffers * @input: input number from which the video data has has been captured * * Contains data exchanged by application and driver using one of the Streaming * I/O methods. */ struct v4l2_buffer { __u32 index; __u32 type; __u32 bytesused; __u32 flags; __u32 field; struct timeval timestamp; struct v4l2_timecode timecode; __u32 sequence; /* memory location */ __u32 memory; union { __u32 offset; unsigned long userptr; struct v4l2_plane *planes; __s32 fd; } m; __u32 length; __u32 reserved2; __u32 reserved; };
index鉴别缓冲区的序号。 type缓冲区类型,就是enumv4l2_buf_type里面所包含的类型。 bytesused缓冲区中数据的大小,单位是byte flags表示当前缓冲区的状态,有以下几种: #defineV4L2_BUF_FLAG_MAPPED 0x0001 /* Buffer is mapped (flag) */ #defineV4L2_BUF_FLAG_QUEUED 0x0002 /* Buffer is queued for processing */ #defineV4L2_BUF_FLAG_DONE 0x0004 /* Buffer is ready */ #defineV4L2_BUF_FLAG_KEYFRAME 0x0008 /* Image is a keyframe (I-frame) */ #defineV4L2_BUF_FLAG_PFRAME 0x0010 /* Image is a P-frame */ #defineV4L2_BUF_FLAG_BFRAME 0x0020 /* Image is a B-frame */ /*Buffer is ready, but the data contained within is corrupted. */ #defineV4L2_BUF_FLAG_ERROR 0x0040 #defineV4L2_BUF_FLAG_TIMECODE 0x0100 /* timecode field is valid */ #defineV4L2_BUF_FLAG_PREPARED 0x0400 /* Buffer is prepared for queuing */ /*Cache handling flags */ #defineV4L2_BUF_FLAG_NO_CACHE_INVALIDATE 0x0800 #defineV4L2_BUF_FLAG_NO_CACHE_CLEAN 0x1000 /*Timestamp type */ #defineV4L2_BUF_FLAG_TIMESTAMP_MASK 0xe000 #defineV4L2_BUF_FLAG_TIMESTAMP_UNKNOWN 0x0000 #defineV4L2_BUF_FLAG_TIMESTAMP_MONOTONIC 0x2000 #defineV4L2_BUF_FLAG_TIMESTAMP_COPY 0x4000 timestamp时间戳。 timecode时间编码,对于视频类应用有用。 memory即是enumv4l2_memory中的哪一种。 m.offsetmemory==V4L2_MEMORY_MMAP,表示从设备内存基址到当前缓冲区的偏移值。(同时这个值也是传给mmap函数的offset参数) m.userptrmemory==V4L2_MEMORY_USERPTR,表示用户空间的一个指针指向这个缓冲区。 m.fdmemory==V4L2_MEMORY_DMABUF,表示用户空间与此缓冲区相关联的一个文件描述符。 m.plane指向某一个structv4l2_plane结构体,在multiplanar情况下,用户指向位面数组的指针。 length缓冲区的大小non_multiplanar情况下);位面的个数(multiplanar情况下)。

2.VIDIOC_REQBUFS 应用程序中: struct v4l2_requestbuffers req; memset(&req, 0, sizeof (req)); req.count = TEST_BUFFER_NUM; req.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; req.memory = V4L2_MEMORY_MMAP; if (ioctl(fd_v4l, VIDIOC_REQBUFS, &req) < 0) { printf("v4l_capture_setup: VIDIOC_REQBUFS failed "); return 0; }
其中 #defineTEST_BUFFER_NUM 3 通过这个VIDIOC_REQBUFSioctl调用来申请内存空间,这里的关键结构体就是structv4l2_requestbuffers结构体,对应设置req中的几个参数,然后就会调用到内核驱动中。
驱动中: mxc_allocate_frame_buf(cam, req->count); static int mxc_allocate_frame_buf(cam_data *cam, int count) for (i = 0; i < count; i++) { cam->frame[i].vaddress = dma_alloc_coherent(0, PAGE_ALIGN(cam->v2f.fmt.pix.sizeimage), &cam->frame[i].paddress, GFP_DMA | GFP_KERNEL); cam->frame[i].buffer.index = i; cam->frame[i].buffer.flags = V4L2_BUF_FLAG_MAPPED; cam->frame[i].buffer.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; cam->frame[i].buffer.length = PAGE_ALIGN(cam->v2f.fmt.pix.sizeimage); cam->frame[i].buffer.memory = V4L2_MEMORY_MMAP; cam->frame[i].buffer.m.offset = cam->frame[i].paddress; cam->frame[i].index = i; } 首先在内核的cam_data结构体中有一个重要的成员:structmxc_v4l_frame frame[FRAME_NUM]; 这个成员structmxc_v4l_frame如下所示: struct mxc_v4l_frame { u32 paddress; void *vaddress; int count; int width; int height; struct v4l2_buffer buffer; struct list_head queue; int index; union { int ipu_buf_num; int csi_buf_num; }; }; 它包含一个很重要的结构体:structv4l2_buffer buffer,这个结构体正是我们前面所讲过的。它是本文要讨论的核心。
在这个函数中,通过dma_alloc_coherent函数申请req->count个缓冲区,之后设置cam->frame[i]中各个成员的值,其中最重要的是为structv4l2_buffer buffer赋值。 cam->frame[i].vaddress= dma_alloc_coherent(0, PAGE_ALIGN(cam->v2f.fmt.pix.sizeimage), &cam->frame[i].paddress,GFP_DMA | GFP_KERNEL); 这个dma_alloc_coherent函数,会生成一个虚拟地址,保存在cam->frame[i].vaddress,同时会生成对应的物理地址,保存在&cam->frame[i].paddress 然后将通过cam->frame[i].buffer.m.offset= cam->frame[i].paddress;将这个物理地址同样保存在cam->frame[i].buffer.m.offset 那么在应用程序中#defineTEST_BUFFER_NUM3,而在驱动程序中,通过一个for循环,所以会申请成功个缓冲区,每个buffer的物理地址都保存在cam->frame[i].buffer.m.offset中,虚拟地址都保存在cam->frame[i].vaddress

3.VIDIOC_QUERYBUF
在应用程序中: struct v4l2_buffer buf; for (i = 0; i < TEST_BUFFER_NUM; i++) { memset(&buf, 0, sizeof (buf)); buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; buf.memory = V4L2_MEMORY_MMAP; buf.index = i; if (ioctl(fd_v4l, VIDIOC_QUERYBUF, &buf) < 0) { printf("VIDIOC_QUERYBUF error "); return -1; } buffers[i].length = buf.length; buffers[i].offset = (size_t) buf.m.offset; buffers[i].start = mmap (NULL, buffers[i].length, PROT_READ | PROT_WRITE, MAP_SHARED, fd_v4l, buffers[i].offset); memset(buffers[i].start, 0xFF, buffers[i].length); } 应用程序中首先声明了一个structv4l2_buffer buf;这个buf是个structv4l2_buffer类型的,其中在前面的分析中,可以看到,这个structv4l2_buffer类型包含在structmxc_v4l_frame,即cam->frame[i]中。
驱动程序中: mxc_v4l2_buffer_status(cam, buf); static int mxc_v4l2_buffer_status(cam_data *cam, struct v4l2_buffer *buf) memcpy(buf, &(cam->frame[buf->index].buffer), sizeof(*buf)); 最终会调用到mxc_v4l2_buffer_status函数中,这个函数的核心就是一个memcpy函数,他将 cam->frame[buf->index].buffer里面的内容拷贝到buf中。 其中这个cam->frame[buf->index].buffer是在上面2中已经申请配置的。 同时,在应用程序中,是一个for循环,会将cam->frame[buf->index].buffer中申请的每个buffer都拷贝到buf中。
看应用程序中的for循环,它在调用VIDIOC_QUERYBUF之后,同时就会将这个buf中的值拷贝到buffers[i],而这个buffers[i]是应用程序中的一个全局变量,如下所示: struct testbuffer buffers[TEST_BUFFER_NUM]; struct testbuffer { unsigned char *start; size_t offset; unsigned int length; }; 注意这个赋值语句:buffers[i].offset= (size_t) buf.m.offset;这个buf.m.offsetVIDIOC_QUERYBUF里面,通过memcpy函数,将cam->frame[i]中的内容拷贝过来的,buf.m.offset保存的是申请内存的物理地址。参见上面标红的语句 这时候,应用程序这个structtestbuffer 结构体中各个元素的含义就清楚了 start保存的是mmap函数映射内存后的首地址。 offset保存的是VIDIOC_QUERYBUFbuf.m.offset的值,同样是cam->frame[i].buffer.m.offset,即申请内存的物理地址。 length保存的是申请内存的大小
到这以后,总结一下: 首先通过dma_alloc_coherent函数申请内存,将物理地址,虚拟地址都保存在cam->frame[i]中,同时为cam->frame[i].buffer赋值。 然后就会调用VIDIOC_QUERYBUF,来将cam->frame[i]里面的buffer都取出来,通过buf这个中间变量赋值给应用程序中的buffers[i]中。 这时候,申请的内存的物理地址,虚拟地址,大小等等信息保存在驱动程序中的cam->frame[i]中和应用程序中的buffers[i]中。

4.VIDIOC_QBUF 应用程序中: for (i = 0; i < TEST_BUFFER_NUM; i++) { memset(&buf, 0, sizeof (buf)); buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE; buf.memory = V4L2_MEMORY_MMAP; buf.index = i; buf.m.offset = buffers[i].offset; if (ioctl (fd_v4l, VIDIOC_QBUF, &buf) < 0) { printf("VIDIOC_QBUF error "); return -1; } } 应用程序中,通过这个for循环,依次从应用程序中的buffer[i]中取出buffer,保存在buf这个中间变量中,然后调用VIDIOC_QBUFioctl
驱动中: struct v4l2_buffer *buf = arg; int index = buf->index; if ((cam->frame[index].buffer.flags & 0x7) == V4L2_BUF_FLAG_MAPPED) { cam->frame[index].buffer.flags |= V4L2_BUF_FLAG_QUEUED; list_add_tail(&cam->frame[index].queue, &cam->ready_q); buf->flags = cam->frame[index].buffer.flags; }

驱动程序中会为cam->frame[index].buffer.flags添加上V4L2_BUF_FLAG_QUEUED属性,同时将这个cam->frame[index].buffer添加到cam->ready_q队列中,然后将cam->frame[index].buffer.flags复制给bufflags属性