Ideas, requests, problems regarding the Mobius site 
<subsystem or component name> Overview Authors/Editors: Version: ------------------------------------------------------------ Revision History: ------------------------------------------------------------ Related Documents: 1. Introduction =============== 2. Requirements =============== Primary goals of the connector framework. The framework *must*: o mandatory requirement 1 o mandatory requirement 2 Secondary requirements include: o potential feature 1 o potential feature 2 Non-requirements include: o mandatory non-requirement 1 o mandatory non-requirement 2 3. Overview =========== 4. Dictionary ============= o Concept 1 - Definition 1. 5. Example ========== 6. Bibliography ===============
---+ <subsystem or component name> Overview ---++ Authors/Editors Who wrote or edited this document and who did the analysis, design, development, and maintenance of this subsystem/component? ---++ Version What version number does this document have? ---++ Revision History Major milestones in the evolution of this document corresponding sometimes to major milestones in subsystem/component development and evolution. ---++ Related Documents What documentation, papers, etc. are related to this subsystem/component? ---++ Introduction A one paragraph summary of this subsystem/component. ---++ Requirements A quick summary to this subsystem/component's requirements. ---+++ Primary Goals This subsystem/component *must*: * mandatory requirement 1 * mandatory requirement 2 ---+++ Secondary Goals This subsystem/component *may*: * potential feature 1 * potential feature 2 ---+++ Non-Goals This subsystem/component will *not*: * mandatory non-requirement 1 * mandatory non-requirement 2 ---++ Overview A one page overview of this subsystem/component. ---++ Dictionary The primary concepts used in the analysis, design, and development of this subsystem/component. ---++ Example An example or two of the use of this subsystem/component. ---++ Bibliography Your directly relevant bibliography goes here.
Connector Architecture Overview
Authors: Joseph Kiniry
Version: $Revision: 1.1 $
------------------------------------------------------------
Revision History:
000/08/16 17:25:57 kiniry
Finished connector example.
Revision 1.10 2000/08/16 07:16:59 kiniry
Added the beginning (most of) a large example to this document to work
through the specification of interfaces and example of negotiation at
work.
Revision 1.9 2000/07/19 23:03:20 kiniry
First draft of connector overview diagram.
Revision 1.8 2000/07/19 22:09:32 kiniry
Made monitoring diagram.
Revision 1.7 2000/07/19 21:42:27 kiniry
Added two new classes to explicitly represent types and levels on events.
Wrote first draft of Monitoring overview. Made a small update to the
Connector document (cross-reference only).
------------------------------------------------------------
Related Documents:
Connector_Diagram.eps - A high-level diagram of the connector system.
1. Introduction
===============
The component architecture is broken down into a number of independent
components that need not communicate directly via method calls. These
parts are potentially written in different languages, might be running on
different platforms, and might have alternate implementations (i.e. 2D and
3D renderers). The connector architecture realizes the connections between
these components as first-class entities and provides a single
"plug-in/snap-on/strap-on" architecture for high-speed, reliable
inter-component communication.
2. Requirements
===============
Primary goals of the connector framework.
The framework *must*:
o Connect exactly two components with the "best" communication
mechanism available where "best" is a well-defined, analytical
built-in function that is provided as part of the framework.
o Provide a simple and convenient manner in which to provide a
stdio-based "wrapper" to a generic component.
o Provide wrapper functionality that does not necessitate inheritance
from a base class (i.e. any class can become a connector component).
o Support at least Java and one other programming language (likely
Eiffel or C++ originally).
o Support tagged data streams where the tags are both "types" and
"levels".
o Support multiple typed data streams.
o Support multiple data streams that have many different levels.
o Support user-defined tags on data streams.
o Support OS-native IPC mechanisms efficiently, transparently, and
completely. Possible mechanisms include shared memory, method invocation,
shared local files, anonymous and named pipes, shared database records,
shared tuple space locations, local loopback network, and remote network
connections.
Secondary requirements include:
o Connect more than two components together.
o Support archetypal connector-set topologies.
o Provide some form of query mechanism for determining the current
"best" connection mechanism, its reliability, data-rate, etc.
o Provide an XML representation of an arbitrary data stream (possibly
for persistence, archival, debugging, etc.)
Things that are _not_ goals of the connector framework include:
o Implementation of communication framework (that is the job of Dan's
Messaging framework).
o Provide "link-time" connectivity between components (i.e. translate
connector interface definitions and code into actual method calls with
zero overhead).
o Support non-core data type support for inter-language communication
in anything but a "native" manner.
3. Overview
===========
The most successful composition model in use today is the pipe architecture
on UNIX systems. The problem with pipes is twofold: (1) the definition of
what comes out of a component (a command) and what can go in a component
exists in exactly two places: the natural language documentation (e.g. the
man page) and the source code (which is not query-able), and (2) there is no
type scheme for the data even if the data format is well-understood.
To solve these two problems, the Connector architecture's API includes a
set of standard methods for querying a components interface and a set of
common data types to guarantee the components written in different languages
can communicate with each other.
An attractive generic communications architecture is Sims' Semantic Data
Streams [SDS] (Sims94, pg. 140). SDS, from the programmer point of view, are
just blobs of tagged, typed data. Such a structure decouples system
components because data type dependencies vanish, the structure of the
messages is irrelevant, the data size is irrelevant, and the content of the
data can change. The only thing information that two components that need to
communicate need to share are the SDS API and the set of data tags/labels.
Thus, a streaming architecture that is arbitrarily composable is very useful
(i.e. UNIX pipes), and a semi-structured data communication API improves upon
this metaphor.
We have one more constraint in our system: two components that communicate
with each other might only be interested in a subset of the information that
each can provide. For example, consider a display device that only renders
in black and white. Such a device would not be interested in the RGB values
of a shape, only the intensity of the image. We call this type of
communication "Multires Communication".
The multires communication API provides a notion of "resolution level" in the
SDS. Such levels are part of the published specification of the component
interface. Each level is associated with a set of data items and a set of
network utilization data that describes the typical characteristics of the
channel that is necessary to maintain this resolution level of
communication. Such data includes metrics like bandwidth and latency.
Every component has one or more input and output interfaces. Each interface
is described by a string name and a set of blank (dataless) SDOs.
The Negotiator, a major part of the Connector architecture is used to perform
the composition between two components. The Negotiator examines the connector
interfaces of the two components, determines the set of data channels over
which the components can communicate, decides which data channel is the most
useful for their communication "style" (captured by their level sets), and
sets up the actual communication.
This architecture facilitates the reuse of standard component services.
Services under consideration include authentication, encryption, data
compression, an "smart" composition (caching composition decisions).
Once a set of components are composed, they know that they are composed and
simply stream data to each other. They do not *have* to know (though they
might discover via their own mechanisms):
(a) who they are communicating with,
(b) where the component that they are communicating with is in the network,
(c) what mechanism is used for the communication,
(d) anything about the negotiation that went on to set up the channel, or
(e) what other components might be between the source and destination
components.
Support for higher level analysis of connectors at runtime is in the future
of the connector architecture. Ideas include topologies that reconfigure as
network behavior change, SDS implementation switches at runtime (i.e. switch
from UDP to TCP to a shared database as a component migrates through the
system), and others.
4. Dictionary
=============
o Channel - A potential communication medium for connectors.
o Channel Analyst - Aggregate data and associated analysis of channel
availability and conditions.
o Channel Information - describes the capabilities (current or potential) of a
channel.
o Component - a chunk of software (i.e. class, set of classes, etc.) that has
communication needs and can be used as a part of a larger architecture.
o Connector - a piece of software that sits between two or more components
and provides the communication channel on which they communicate.
o Core Data Types - a set of standard data types that appear in nearly all
programming languages. Examples include integers, floats, rationals,
strings, characters, and booleans.
o Debugging - Enable, disable, and fine-tune debugging information
within the Connector architecture.
o First-Class - a first class entity is something that has a
representation and one can get a "handle" to it. For example, class
is a first-class entity in Java (no pun intended) because you can
get a handle to its representation via the reflection packages, but
a for loop is not first-class in Java (but is in CLOS, for example).
o Interface - a string name and a set of blank (dataless) SDOs.
o Level - a number (either a float or an integer, TBD) associated with a data
item on an SDS that related to the "resolution" level of the data.
o Manager - Manages the set of all connectors in a VM.
o Negotiator - Negotiates the channel properties between components
and instantiates the appropriate connector.
o Semantic Data Stream (SDS) - a data stream that contains blobs of
structured data that are tagged with string "properties". In other words,
consider streaming instances of java.util.Property over a network
connection with an API that provides methods for manipulating and searching
the blobs in a very efficient fashion.
o Tag - a string associated with a data item on an SDS that "names" the
data. Tags are part of the interface of a component and manipulation of
the data blobs destined to or received from an SDS are performed through
the use of tags.
o Topology - a regular structure of a set of components and their
connectors. Typical topologies include 1-1 communications, 1-M networks,
M-N networks, rings, stars, etc.
o Type - a string associated with a data item on an SDS that specifies the
type of the data. The set of core data types each have such a name and the
component author can specify new data types for her own use. The type of
each data item is part of a component's interface.
5. Example
==========
Consider the following components.
Component HelloWorld provides information corresponding to the notion of
"Hello World" in a number of formats. It is expected that this data will be
rendered by some other component. HelloWorld has one input and one output
interface.
The input interface tells it when to send data on the output interface. It is
called "IN" and expects a blank, level 0 SDO. This is a standard technique in
distributed systems; often the reception of a message is sufficient signal for
some action, in this case, to send data on the output channel.
The output interface is called "HW_DATA". It provides three SDO types, all of
which store their data under a tag called "DATA":
1. A level 1 SDO that contains a String (which should, of course, be "Hello
World").
2. A level 2 SDO that contains a triangle mesh that, when rendered properly,
will spell out "Hello World" in three dimensions. This mesh is stored
in standard clockwise-rotation format a la OpenGL.
3. A level 3 SDO that contains a digitized sample of a person saying the
phrase "Hello World".
HelloWorld also specifies the requirements of each level used in its input and
output interfaces:
0. level 0 requires 0.015 kb/s at 100 ms latency.
1. level 1 provides data at an average of 1 kb/s at 100 ms latency (which
translates to 10 messages/s where each message is at most 100 bytes).
2. level 2 provides data at an average of 30 kb/s at 1000 ms latency (which
translates into 1 message/s where each messages is 30 kb).
3. level 3 provides data at an average of 89 kb/s at 100 ms latency (which
translates to 10 message/s where each message is 8.9 kb).
Component Trigger provides a trigger message to whichever component with which
it is composed. It has a single output interface called "OUT" that sends a
level 0 SDO with no data.
Since Trigger has no input interfaces it has no bandwidth or latency
requirements, but it can provide its level 0 messages at 100 ms latency with a
bandwidth use of 16 kb/s (which translates to 10 messages/s at 16 bytes per
message - got it yet?).
Component Printer provides a single input interface called "MYINPUT" which
expects a level 0 SDO with a single tag "DATA" which is a String. At level 0,
Printer can deal with data at 1 ms latency (i.e. 1000 messages/s) and a total
bandwidth of 100 kb/s (i.e. each message can be, at most, 100 bytes long).
Component Renderer provides a single input interface called "MESH_INPUT" which
expects a level 0 SDO with a single tag "DATA" which is a triangle mesh of the
form mentioned above in HelloWorld. At level 0, Renderer can deal with data
at 20 ms latency (i.e. 50 messages/s => 50 frames/s) and a total bandwidth of
1500 kb/s (i.e. each message can be, at most, about 30 kb long - or about
2,500 triangles/scene)(*).
Finally, component Speaker provides a single input interface called
"VOICE_INPUT" which expects a level 0 SDO with a single tag "DATA" which is a
digitized sample of the form mentioned above in HelloWorld. At level 0,
Speaker can deal with data at 1 ms latency (i.e. 1000 messages/s) and a total
bandwidth of 89 kb/s (i.e. aggregate messages result in a stereo data sample
at CD quality, regardless of whether the data arrives in 1000 very small
messages/s or 1 very large messages/s).
Now we wish to begin building our compositional application. We have three
channels available for communication:
1. A stream-based channel whose channel identifier is "JAVA_STREAM" This
channel has the following characteristics:
average/maximum bandwidth: 8000/10000 kb/s
average/minimum latency: 5/1 ms
1. A file-based channel whose channel identifier is "SHARED_FILE" This
channel has the following characteristics:
average/maximum bandwidth: 500/1000 kb/s
average/minimum latency: 9/1 ms
1. A network-based channel whose channel identifier is "RELI_MESS" This
channel has the following characteristics:
average/maximum bandwidth: 1000/8000 kb/s
average/minimum latency: 50/5 ms
First, we'll compose Trigger -> HelloWorld on the same host, thus all channels
are usable for this composition. If the components were on two different
hosts then only network-based communication mechanisms would be viable
connector alternatives.
The algorithm for determining which channel is the right one for a composition
is as follows:
Visualize each interface as a rectangle where the height of the rectangle is
the interface's bandwidth and the width is its average latency. So, taller
rectangles are an indicator of high-bandwidth communication, and shorter
rectangles are low bandwidth. Likewise, skinny rectangles represent low
latency communication, etc. Thus, a very tall and skinny rectangle is the
most difficult interface to fulfill given it is providing data at high data
rates with very low latency.
To determine if two interfaces are compatible, compare their width and
height. If the receiver (input) interfaces is wider *and* shorter than the
sender (inclusive), then the interfaces are compatible.
Next, determine which interfaces can support the data stream in question. For
each level, compute the area of a rectangle that visualizes the communication
demands as above. If the area of the level is smaller than the area of the
interface, then the interface can support the communication level.
Finally, to choose the final interface used for the connector, compute the
areas of all legal interfaces and pick the pair with the lowest summed area
for the composition.
So, using this algorithm, Trigger's OUT interface is compatible with
HelloWorld's IN interface because both specify the use of a blank SDO.
Additionally, the capabilities of Trigger's interface satisfy HelloWorld's
demands - IN requires at least 0.015 kb/s and Trigger can provide 16 kb
(i.e. it is ~1000 times more capable than necessary) and requires better than
100 ms latency and Trigger provides 1ms latency (100 times faster than
necessary).
Now, which channels are capable enough to support this data stream? Using the
above algorithm, we discover that Trigger's interface can be supported by all
channels since its data demands (at level 0, the only level available) are
wider and shorter than all interfaces' capabilities. Additionally,
HelloWorld's IN interface can be supported by all channels since the data
demands are wider and shorter than all channels' capabilities. Thus, we
should pick the channel with the smallest area, which in this example is the
SHARED_FILE channel (with an area of 4500 compared to JAVA_STREAM with an area
of 40000 and RELI_MESS with 50000). Thus, a connector will be created on top
of the SHARED_FILE channel for the Trigger <-> HelloWorld composition.
Turning to the second, and more complex, composition, we consider the
HelloWorld <-> Renderer composition (the other two are exercises for the
reader). It is obvious that the level 2 HW_DATA interface of HelloWorld is
compatible with the level 0 interface of Renderer, but which channel is the
right one? If the two components are on different hosts, then the RELI_MESS
is the only option. Considering its capabilities, we see that HW_DATA can fit
on any channel type; its the data demands are wider and shorter than all
channels' capabilities. MESH_INPUT will work with only the JAVA_STREAM
channel since the bandwidth of the SHARED_FILE and RELI_MESS are insufficient
to support its demands (i.e. the data is *taller* than both interfaces).
Thus, this composition will *only* work if the two components are on the same
host and the connector is constructed on the JAVA_STREAM channel.
This wraps up the Connector example.
(*) Back-of-the-napkin calculations for these numbers:
~30,000 bytes &
12 bytes/vertex &
1 vertex/triangle (since it is mesh format) =>
30,000/12 = ~2500 triangles/scene
6. Bibliography
============
Sims94. Sims, Oliver. Business Objects: Delivering Cooperative Objects for
Client-Server, McGraw-Hill, 1994.