Category Archives: StageFright

Understanding Android Stagefright Internals (VI) – ACodec Design Apsects

To begin with, ACodec is the asynchronous edition of OMXCodec, compliant to the same OMX Service specification.

  • Usage Model

ACodec recognizes that network delivery from content origin is the primary source in  introducing jittery and let the streaming source take place of the sink to drive the flow in the local device playback pipeline . Therefore, ACodec is no subclass of MediaSource.

All functionality APIs exposed in ACodec mark initiation of an act rather than completion of an act upon return.

Due to its innate support for AHandler,  AMessage takes place of  Metadata to carry  codec-related parameters in afore mentioned APIs. Parameters are stored in the item array field of AMessage.

For the same reason, ABuffer replaces MediaBuffer.

  • Identify the target codec component and create the node

ACodec extracts selection rules from an AMessage object and reuses OMXCodec::findMatchingCodecs() to locate the target codec.

  • Codec Configuration

Configuration info are retrieved from AMessage object rather MetaData.

  • States and Context

The figure quoted from OpenMAX spec V1.1.2 shows the component states and the transitions.

component_states

ACodec tracks the components states. Here a  compact, simple state transition framework in stagefright is utilized.

The state-of-art framework is formed with a few base classes: AHandler, ALooper, ALooperRoster,  AMessage,   AHierarchicalStateMachine and AState. It exemplifies the state design pattern with former playing the context role.

ACodec subclasses AHierarchicalStateMachine  and holds the responsibility for triggering actions occurs in a state transition.

The nine private state classes in ACodec are inherited from ACodec::BaseState, which subclasses AState and is enhanced with common routines required in interactions with the OMX Component instance.

  • Asynchronous Signaling

In addition to CodecObserver similar to OMXCodecObserver in OMXCodec , ACodec translates omx_message::EMPTY_BUFFER_DONE, omx_message::FILL_BUFFER_DONE, omx_message::EVENT into
AMessage objects describing ACodec events and dispatches them to currently active ACodec::BaseState object to process. Final processings of some events  like ACodec::kWhatDrainThisBuffer and  ACodec::kWhatFillThisBuffer occur outside of ACodec.

In usage, ACodec registers with a ALooper object which entails an ALooper::LooperThread instance thread and all internal messaging runs in that thread.

————————————————————————————-

End-of-Series Note:

My intent to write the Stagefright Internals series had two folds. The main motivation was to record my understanding for future memory refreshment; the second motivation was a sincere hope to share my understanding with folks whom I may know or may not know in person, so we can exchange knowledge and have discussions.

Understanding Android Stagefright Internals (V) – OMXCodec Design Aspects

In this post, we cover some design aspects that may shed light on OMXCodec source code.

  • Usage Model

An OMXCodec instance is modeled as a component in building a multimedia playback pipeline. In stagefright, once a pipeline is established, buffer flows are driven by the final sink/media renderer while controls like start, pause, resume, stop may be applied at the origin of the pipeline and propagate down to the sink. Under this model, all components can be viewed chained sources with the exception of the sink components, each pipeline component reads buffer passively from its preceding component, if it is not the origin and feeds output buffer to its succeeding component. Class MediaSource characterizes the generic source components, which contains a read routine and control routines.

In media playback pipeline, it is a standard practice to generate a pool of buffers at beginning of a session; during a session, buffers are requested from the pool and returned to the pool later for reuse after consumption. For sake of timely notification of a buffer for recycle, stagefright defines the signaling class MediaBufferObserver for MediaBuffer.

OMXCodec is a subclass inheriting from MediaSource and MediaBufferObserver.

As a side note,  the memory in MediaBuffer are allocated from kernel ashmem driver, hence referenceable /mappable across process.

  • Identify the target codec component and create the node

Stagefright adopts a factory method,  namely Create, to produce an OMXCodec instance.

Note that OMX components, software-based or hardware -accelerated, are manifested in “/etc/media_codecs.xml”.   OMXCodec relies on a helper MediaCodecList class to parse the xml file to produce a codec list, and rank software codecs higher over hardware codecs.

Four parameters are used in codec  matching and selection. the static Create routine  accepts them , subsequently forwards them to another static routine findMatchingCodecs() to undertake the selection.

The main parameter is  a mime description string, distinct for a codec type, the second parameter and the third parameters help to filter out results in regard to encoding/decoding function and  software/hardware codec preference; the optional fourth parameter is specify a particular component.

findMatchingCodecs() returns the matching codecs in an array; Thereafter, Create  instantiate an OMXCodec  instance linked to the first codec which it is able to configure with success.

  • Codec Configuration

Stagefright supports a selected subset of configuration parameters in OpenMAX standard; configuration is done in Create routine.

The OMXCodec user declares all configuration in a MetaData structure and passes it as the fiveth argument in the call to Create.

Create extracts configuration info in the MetaData structure and repacks them into OpenMAX compliant structures, then passes them to the component via OMX_Core APIs in OMX Service.

  • Signaling with OMX component

OMXCodec hides the asynchronous calls with the OMX  codec component from the media playback pipeline.

To receive the omx_message::FILL_BUFFER_DONE and omx_message::EMPTY_BUFFER_DONE message,  another IOMXObserver is defined and OMXCodecObserver is the implementing service class.

This OMXCodecObserver interface is established during OMXCodec::Create routine.

Understanding Android Stagefright Internals (IV) – Two client Access Paradigms: ACodec and OMXCodec

To simplify development of native libraries accessing OMX component via OMX service, stagefright supplies two useful client side classes, namely  OMXCodec and ACodec respectively, here we attempt to explain their usage difference.

In media playback on mobile devices, one of the major factors affecting viewing experience is latency in rendering audio/video frames. If the media content resides locally, the latency to obtain media frames is neglectable. In case of playing back streamed content originated on Internet, network traffic latency may vary unpredictably; If obtaining media frames, frame decoding and rendering occurs in single thread, latency at each step will accumulate to a significant magnitude to cause noticeable slowness in rendering and lengthier response time to user controls.

OMXCodec and ACodec cater to the two separate usage scenarios. The handling of media frames and user controls in the former is synchronic and operates in one single thread, thus OMXCodec suits well for local media content decoding.

ACodec may get its initial letter from the term asynchrony. In ACodec, the reception of data/control commands and actual handling of the request are carried out in two separate threads. An ACodec interface function converts a command into an AMessage, and returns immediately after posting it to a message queue. This part runs in the playback thread. The actual message handler runs in a second thread. This arrangement improves on responsiveness of Stagefright, it accompanies with tremendous design complexity in order to track codec states (e.g. unitialized, loaded, loaded to idle, executing and so forth). Worth of the design complexity, ACodec is the best effort to serve playback of streamed data.

In the next few blogs, we will explore the design of OMXCodec and ACodec.

Understanding Android Stagefright Internals (III) – The Core

In the heart of stagefright framework, sit several key classes.

  • Abstract interface class IOMX

Defines the API client sees over binder interface.

  • Class OMX

Subclass of IOMX, it allocates distinct nodes for connected OMX clients and generates a dedicated messaging dispatcher and thread, i.e. CallbackDispatcher and  CallbackDispatcherThread objects respectively, for each node.

In the communication between the remote client (represented by a node id) and the serving component objects, OMX is responsible to forward requests to the associated message dispatcher for handling.

  •  Struct OMXPluginBase

A skeleton descriptor for an OMX plug in component in “libstagefrighthw.so” or a software component. The OMX_CALLBACKTYPE and OMX_COMPONENTTYPE references  of a component can be obtained from its descriptor object,

  • Class OMXMaster

Manages the loading of OMX component plugins and undertakes the job of producing the OMX_CALLBACKTYPE and OMX_COMPONENTTYPE reference  when a client connects.

  • Class OMXClient

This is the OMX Binder client access agent to establish client side IOMX interface.

Understanding Android Stagefright Internals (II) – OpenMAX IL and Stagefright overview

OpenMAX IL provides a set of abstract C-language APIs for interfacing with multimedia components/Codecs, which aims to reduce efforts in integrating a component to diverse platforms.

Stagefright is a successor to OpenCore on Android platform compliant to OpenMAX IL, shipped in GB and later android distributions. The discussion here is aligned to Jelly Bean source code tag android-sdk-4.4.2 hosted at https://android.googlesource.com/platform/frameworks/av/+/android-sdk-4.4.2_r1.

Before diving into source code, let’s highlight some aspects of Stagefright framework

.   Take form of a binder service

.  OMX component SDKs are packed in plug-in modules and loaded when the service starts.

.  Component lists and profiles are manifested in xml files

.  Offer a variety of methods to access OMX component service

An application may either directly invoke OMX service, or indirectly through requesting media player service.

.  Optimized for streamed media decoding.

Understanding Android Stagefright Internals (I) – Asynchronous Processing

Stagefright is an implementation of OpenMAX IL on android platform, and contains two levels of asynchronous processing. Each is composed of four elements typically present in a reactor pattern in concurrent computing: a generic message data structure, a generic message handler and a message dispatcher, plus a messaging loop/thread.

At the lower level, stagefright implements Open MAX IL in class OMX, which defines struct CallbackDispatcher and struct CallbackDispatcherThread as follow:
struct OMX::CallbackDispatcherThread : public Thread {
CallbackDispatcherThread(CallbackDispatcher *dispatcher)
: mDispatcher(dispatcher) {
}
private:
CallbackDispatcher *mDispatcher;
bool threadLoop();
CallbackDispatcherThread(const CallbackDispatcherThread &);
CallbackDispatcherThread &operator=(const CallbackDispatcherThread &);
};

struct OMX::CallbackDispatcher : public RefBase {
CallbackDispatcher(OMXNodeInstance *owner);
void post(const omx_message &msg);
bool loop();
...
private:
OMXNodeInstance *mOwner;
bool mDone;
List mQueue;
sp mThread;
void dispatch(const omx_message &msg);
CallbackDispatcher(const CallbackDispatcher &);
CallbackDispatcher &operator=(const CallbackDispatcher &);
};

The message struct omx_message is defined in class IOMX as
struct omx_message {
enum {
EVENT,
EMPTY_BUFFER_DONE,
FILL_BUFFER_DONE,
} type;
IOMX::node_id node;
union {
// if type == EVENT
struct {
OMX_EVENTTYPE event;
OMX_U32 data1;
OMX_U32 data2;
} event_data;

// if type == EMPTY_BUFFER_DONE
struct {
IOMX::buffer_id buffer;
} buffer_data;
// if type == FILL_BUFFER_DONE
struct {
IOMX::buffer_id buffer;
OMX_U32 range_offset;
OMX_U32 range_length;
OMX_U32 flags;
OMX_TICKS timestamp;
OMX_PTR platform_private;
OMX_PTR data_ptr;
} extended_buffer_data;
} u;
};

At upper level, stagefright supplies a separate set of messaging data structures.
struct AHandler : public RefBase {
...
ALooper::handler_id id() const;
sp looper();
protected:
virtual void onMessageReceived(const sp &msg) = 0;
private:
friend struct ALooperRoster;
ALooper::handler_id mID;
void setID(ALooper::handler_id id);
};


struct ALooper : public RefBase {
typedef int32_t event_id;
typedef int32_t handler_id;
ALooper();
// Takes effect in a subsequent call to start().
void setName(const char *name);
handler_id registerHandler(const sp &handler);
void unregisterHandler(handler_id handlerID);
status_t start(
bool runOnCallingThread = false,
bool canCallJava = false,
int32_t priority = PRIORITY_DEFAULT
);
status_t stop();
private:
friend struct ALooperRoster;
struct Event {
int64_t mWhenUs;
sp mMessage;
};
AString mName;
List mEventQueue;
struct LooperThread;
sp mThread;
...
void post(const sp &msg, int64_t delayUs);
bool loop();
};


struct AMessage : public RefBase {
AMessage(uint32_t what = 0, ALooper::handler_id target = 0);
static sp FromParcel(const Parcel &parcel);
void writeToParcel(Parcel *parcel) const;
void setWhat(uint32_t what);
uint32_t what() const;
void setTarget(ALooper::handler_id target);
ALooper::handler_id target() const;
void setInt32(const char *name, int32_t value);
void setInt64(const char *name, int64_t value);
void setSize(const char *name, size_t value);
void setFloat(const char *name, float value);
void setDouble(const char *name, double value);
void setPointer(const char *name, void *value);
void setString(const char *name, const char *s, ssize_t len = -1);
void setObject(const char *name, const sp &obj);
void setBuffer(const char *name, const sp &buffer);
void setMessage(const char *name, const sp &obj);
void setRect(
const char *name,
int32_t left, int32_t top, int32_t right, int32_t bottom);
bool findInt32(const char *name, int32_t *value) const;
bool findInt64(const char *name, int64_t *value) const;
bool findSize(const char *name, size_t *value) const;
bool findFloat(const char *name, float *value) const;
bool findDouble(const char *name, double *value) const;
bool findPointer(const char *name, void **value) const;
bool findString(const char *name, AString *value) const;
bool findObject(const char *name, sp *obj) const;
bool findBuffer(const char *name, sp *buffer) const;
bool findMessage(const char *name, sp *obj) const;
bool findRect(
const char *name,
int32_t *left, int32_t *top, int32_t *right, int32_t *bottom) const;
void post(int64_t delayUs = 0);
// Posts the message to its target and waits for a response (or error)
// before returning.
status_t postAndAwaitResponse(sp *response);

// If this returns true, the sender of this message is synchronously
// awaiting a response, the “replyID” can be used to send the response
// via “postReply” below.
bool senderAwaitsResponse(uint32_t *replyID) const;

void postReply(uint32_t replyID);
enum Type {
kTypeInt32,
kTypeInt64,
kTypeSize,
kTypeFloat,
kTypeDouble,
kTypePointer,
kTypeString,
kTypeObject,
kTypeMessage,
kTypeRect,
kTypeBuffer,
};
size_t countEntries() const;
const char *getEntryNameAt(size_t index, Type *type) const;
private:
uint32_t mWhat;
ALooper::handler_id mTarget;

struct Rect {
int32_t mLeft, mTop, mRight, mBottom;
};
struct Item {
union {
int32_t int32Value;
int64_t int64Value;
size_t sizeValue;
float floatValue;
double doubleValue;
void *ptrValue;
RefBase *refValue;
AString *stringValue;
Rect rectValue;
} u;
const char *mName;
Type mType;
};
enum {
kMaxNumItems = 64
};
Item mItems[kMaxNumItems];
size_t mNumItems;
Item *allocateItem(const char *name);
void freeItem(Item *item);
const Item *findItem(const char *name, Type type) const;

void setObjectInternal(
const char *name, const sp &obj, Type type);

};

struct ALooperRoster {
...

ALooper::handler_id registerHandler(
const sp looper, const sp &handler);

void unregisterHandler(ALooper::handler_id handlerID);

status_t postMessage(const sp &msg, int64_t delayUs = 0);
void deliverMessage(const sp &msg);

status_t postAndAwaitResponse(
const sp &msg, sp *response);

void postReply(uint32_t replyID, const sp &reply);

sp findLooper(ALooper::handler_id handlerID);

private:
struct HandlerInfo {
wp mLooper;
wp mHandler;
};

Mutex mLock;
KeyedVector mHandlers;
ALooper::handler_id mNextHandlerID;
uint32_t mNextReplyID;

KeyedVector<uint32_t, sp > mReplies;
status_t postMessage_l(const sp &msg, int64_t delayUs);
};