Understanding Android Graphics Internals -SurfaceFlinger(V)

This SurfaceFlinger series turns lengthy. Let’s see if we can wrap it up in this post.

• FramebufferSurface and DisplayDevice

Jelly Bean supports external displays(e.g. HDMI, WiDi). Each display is represented by a DisplayDevice instance and associated with a FramebufferSurface instance which makes the mDisplaySurface field.

FramebufferSurface subclasses both ConsumeBase and DisplaySurface and wraps around a BufferQueue instance.

DisplayDevice  maintains a reference to  a vector of all sorted visible layers in mVisibleLayersSortedByZ, which is tracked in SurfaceFlinger::mDrawingState.

SurfaceFlinger tracks all DisplayDevice in an keyed vector mDisplays.  SurfaceFlinger::readyToRun() creates each DisplayDevice along with FramebufferSurface and adds each to mDisplays.

In SurfaceFlinger::doDisplayComposition(),when composition is complete and the display back frame buffer is drawn; then  DisplayDevice::swapBuffers() is invoked for a DisplayDevice instance. Within the latter call, eglSwapBuffers() is called. As a result,  FramebufferSurface::onFrameAvailable() is called.

In FramebufferSurface::onFrameAvailable(), HwComposer::fbPost() is called, in which for non HWC_DEVICE_API_VERSION_1_1, framebuffer_device_t::post in gralloc module is called to post the back frame buffer to frame buffer device driver.

• HWComposer

This class wraps framebuffer_device_t and struct hwc_composer_device_1 and manages the vsync with HWComposer::event_control(), HWComposer::sync().

SurfaceFlinger owns an instance of HWComposer.

• SurfaceFlinger

1.  Implements BnISurfaceComposer functionality

BnISurfaceComposerClient is implemented by Client class; responsible for creation of Layer instances.

2. Implements HWComposer::EventHandler

Accepts on vsync and hotplug in events from hwcomposer module; process vsync enable/disable in eventControl().

3.  Runs message queue on its own thread loop.

after initialization in readyToRun(), all handling is asynchronous and message driven.

4.  readyToRun()

Instantiates HWComposer instances, instantiates and Initializes DisplayDevice objects and corresponding FramebufferSurface objects, initializes egl config, context, etc.

5. Maintains the current and the drawing SurfaceFlinger::State

Drawing state is the state being rendered, the current state is open for modifications. mCurrentState is copied to mDrawingState  in CommitTransaction during handling of  MessageQueue::TRANSACTION and MessageQueue::INVALIDATE messages.

6. Several sources may trigger composition

new surface frame queued, changes in orientation and other flags, surface removal, vsync.

7. Composition and rendering

All layers are sorted by depth dimension Z, send layer state information to hwcomposer module to determine rendering by overlay or composition (by the prepare method in hwcomposer).

For layers selected for composition,  compute visible regions based on Z and alpha blending factor, and draw them to each DisplayDevice object’s back FramebufferSurface.

When composition on a display is complete, invoke eglSwapBuffers to switch back FramebufferSurface to the front and posted to frame buffer device.

In summary, SurfaceFlinger design is complex and has to deal with many usage scenarios and corner use cases.

Understanding Android Graphic- SurfaceFlinger (IV)

Having known that BufferQueue exposes the APIs for consuming surface frame while implementing BnGraphicBufferProducer, we continue our attempt to demystify surface compositions and rendering with an examination of a few more classes with respect to surface composing.

• SurfaceFlingerConsumer and GLConsumer class

Defined in framework/native/services/surfaceflinger/SurfaceFlingerConsumer.h, and framework/native/include/gui/GLConsumer.h, SurfaceFlingerConsumer and GLConsumer are the core classes to fetch the current buffer from BufferQueue to bind to a surface layer specific GL texture.  GLConsumer uses GLES functions to conduct the operation while SurfaceFlingerConsumer is a wrapper around it to safeguard the operation. The main APIs in SurfaceFlingerConsumer are  updateTexImage() and bindTextureImage().

Note that a surface frame will be drawn to the display frame buffer as a GL texture and GLConsumer inherits from ConsumerBase.

• Layer Class

It has been known that when a SurfaceComposerClient::createSurface  is invoked, Client in SurfaceFlinger posts a MessageCreateLayer to SurfaceFlinger which in turn calls SurfaceFlinger::createLayer() create a Layer instance to service the remote Surface instance.

Applying a lazy initialization, Layer delays until first reference to instantiate a BufferQueue object and SurfaceFlingerConsumer corresponding to the remote Surface object; meanwhile Layer links the BufferQueue object to SurfaceFlingerConsumer object and register itself as SurfaceFlingerConsumer::FrameAvailableListener listener among other procedures.

When Surface::queueBuffer() is invoked, BufferQueue::queueBuffer() and Layer::onFrameAvailable() call ensue ,as we elaborated in post SurfaceFlinger(III).

Subsequently , the Layer::draw() and Layer::onDraw() will be called to run SurfaceFlingerConsumer::bindTextureImage() and Layer::drawWithOpenGL() to FramebufferSurface current frame buffer.

Understanding Android Graphics Internals – SurfaceFlinger (III)

In this post, we discuss BufferQueue, BufferQueue::BufferSlot and BufferQueue::BufferState defined in frameworks/native/include/gui/BufferQueue.h.

• BufferQueue

We have learned the concepts of ANativeWindow and ANativeWindowBuffer defined in system/core/include/system/window.h with class Surface (in frameworks/native/include/gui/Surface.h) being concrete implementation of the former class and GraphicBuffer (in frameworks/native/include/ui/GraphicBuffer.h) being an implementation of the latter. IGraphicBufferProducer (in frameworks/native/include/gui/IGraphicBufferProducer.h) defines the Binder API interface for buffer interactions between a surface and SurfaceFlinger. Therefore a UI may draw on a GraphicBuffer and by invoking eglSwitchBuffesr() to render the current buffer; Within eglSwitchBuffers() calls queueBuffer(), dequeueBuffer(), and other APIs in ANativeWindow as necessary.

What is yet to learn is how a graphic buffer queued through eglSwitchBuffers() is composed into frame buffer or hardware-over-laid for rendering and then how the buffer is released to the surface. Here we attempt to uncover part of the myth.

IGraphicBufferProducer was named after the relation of GraphiBuffer content producer and consumer. When a surface finishes drawing on a graphic buffer and subsequent calling of the Surface::queueBuffer() in eglSwitchBuffers() will invoke BpGraphicBufferProducer::queueBuffer() to pass the graphic buffer  (through BufferQueue::queueBuffer())to a BufferQueue object in the Layer object corresponding to the Surface object.

BufferQueue implements BnGraphicBufferProducer and provides the graphic Buffer consumer APIs( e.g. acquireBuffer(), releaseBuffer(), etc.), Going forward, we use a more formal term frame to denote the content in the graphic buffer.

The Layer class implements SurfaceFlingerConsumer::FrameAvailableListener::onFrameAvailable() is set to be invoked in BufferQueue::queueBuffer(); Layer::onFrameAvailable () marks arrival of a new frame and signals to SurfaceFlinger by a signalLayerUpdate() call. The signalLayerUpdate() will either request a vsync on behalf of the surface or post a MessageQueue::INVALIDATE message.

• BufferQueue::BufferSlot and BufferQueue::BufferSlot::BufferState.

Within BufferQueue, a GraphicBuffer object is assigned a descriptor BufferSlot object to contain attribute metadata such as  associated EGLDisplay, timestamp, transform, cropping, framenumber, fence, buffer state, and so on.

BufferQueue::BufferSlot::BufferState  represents the different states in which a buffer slot can be. SurfaceFlinger recognizes four buffer states, namely, FREE, DEQUED, QUEUED, ACQUIRED, respectively. You may refer to the BufferQueue.h for further information.

BufferQueue tracks all BufferSlots in an array and index each slot by a cardinal slot number, which is reference as buffer in IGraphicBufferProducer.

Before ending this post, I want to note that in BufferQueue::acquireBuffer(), an argument type BufferItem is used, which is sort of equivalent to BufferQueue::BufferSlot.

Understanding Android Graphics Internals – SurfaceFlinger (II)

Prior to Jelly Bean, the threadLoop in SurfaceFlinger.cpp processes composition and rendering sequentially in a thread loop. Newly enqueued graphic buffer won’t be handled until the rendering of current frame buffer is complete. As a result, animation shown on display may appear jumpy or tearing in rendering. To address this issue, Android introduces vsync event into SurfaceFlinger.

In android, a vsync event relates to timestamp. For each display device, android assigns a DisplayEventReceiver::Event  ( defined in framework/native/include/gui/DisplayEventReceiver.h) structure, i.e. mVSyncEvent. mVSyncEvent.header.timestamp. Even though the timestamp value is refreshed with system time, the significance is whether it is non-zero. If it is nonzero, SurfaceFlinger will trigger either a refresh action or an invalidate action on the corresponding display and reset timestamp to zero.

vsync events can be generated from several sources.

• Periodical vsync event sources

If hwcomposer module is present, it will be generated either from the vsync signal emitted from the frame buffer device driver; or faked from the hwcomposer module.

If the hwcomposer module is not present, the VSyncThread will generate it.

• Spontaneous vsync event sources

A UI view may instantiate DisplayEventReceiver and thus obtain an IDisplayEventConnection interface through createDisplayEventConnection() call. Whenever the view has a graphic buffer to update, it may sound a requestNextVsync() call on the IDisplayEventConnection interface.

To establish concurrency between composition and rendering, SurfaceFlinger in Jelly Bean made two major changes in threading model and asynchrocy, adding a new EventThread thread to handle DisplayEventReceiver::Event events, adopted a Looper/Handler/MessageQueue paradigm in SurfaceFlinger main thread. As a side note, the EventThread also handles display hotplug event ( hwcomposer module may listen on uevent).

Understanding Android Graphics Internals – SurfaceFlinger (I)

Let’s recapture what we have known of SurfaceFlinger from previous posts on Android Graphics.
1. it is a native service implementing BnSurfaceComposer;
2. SurfaceFlinger administrates Surface and Graphic Buffer creation for all graphic clients;
3. Periodically, SurfaceFlinger composes surface layers to the back frame buffer in the frame buffer to the abstraction frame buffer device, and submit layers marked for hardware composition to hwcomposer hardware abstraction module for rendering in case of
4. SurfaceFlinger enforces vsync and triple buffering;
5. Supports snapshot of screen display;
6. Supports HDMI display and virtual display (e.g. Wireless Display);
7. It heavily relies on GLES and EGL.
With these knowledge in mind, we proceed to make some conjectures on it design.
1. To run surface composition, it has to own a dedicated thread;
2. It must maintain a context for each individual surface layer
This is done with Layer structure which tracks orientation, cropping, geometric parameters, color scheme, visible region, etc.
3. It must distinguish between the drawing state (front) and currently rendering state (back) for each layer
This is done with the Layer::State structure.
4. It must quarantine the set of currently active rendering layers from layers newly removed and added
This is done with SurfaceFlinger::state structure
5. vsync event is asynchronous of surface composition, we may need a thread somewhere to monitor vsync event

In next post, we will discuss vsync. In my view, vsync was the main force to drive the redesign of SurfaceFlinger in Jelly Bean.