A discussion on real-time implementations

 

In the case of continuously operating Active Vision systems, which interact with their environment (such as an industrial robot on a mobile platform), it is highly desirable that it is possible to achieve a "closed loop" of information flow. This may not be immediately achievable as a Real Time Application on a physical mobile platform, but it may be achievable in a testbed which sufficiently well simulates the real world components (environment, cameras, movement and so on).

We would like to re-emphasize this investigation into mechanisms and control structures for restrictive processing in continuous-operation vision systems. This is due to the computational demands resulting from performing robot vision in real time. It includes mechanisms for goal directed perceptual strategies such as focus of attention, fovea, and spatio-temporal scale-space, with variable space and time resolution sampling to provide sufficiently good representations of control mechanisms. The real-time restriction is interesting and important for a number of reasons:

1. Real-time vision allows for experimentation with a whole new set of applications not possible with simulations (simulations ("slowing down time") is only possible if you do not have to react to physical phenomena that have their own inherent timing).
2. Real-timing allows for creating vision applications that are responsive to the need of existing industrial settings (or in other word we can then create vision applications that respond to a world problem instead of the other way around).
3. Real-time vision is the "ultimate" version of "fast vision" (as real-time in a textbook refers to timing in fixed time-frame that is shorter than what is needed for a particular task). "Fast vision" has shown to provide at least partial answers to problems such as optical flow (due to the limitations of possible movements between frames) ...etc.

In order to be acceptable this real-time system is :

• Theoretically well founded :
  For example, algorithms provided by the system should have a formal specification (pre- and post-conditions). The complexity of the algorithms should be known. Data structures should also be described formally, with invariants. Concurrency and real-time control should be based on established theory, e.g. CSP.
• Open ended :
  There is no way that a system can provide support for every conceivable vision algorithm. Since such a system tries to cover applications, it will tend to be too large, monolithic, and hard to use and support if not modular.

  Therefore, such a system consists of a small kernel to which problem-oriented code are added as modules.

  However, it is crucial that the kernel is properly decomposed and layered, with well-defined interfaces to every part of the kernel, so that it will be possible to choose alternative implementations of one or several services in the kernel. One reason for choosing an alternative implementation is increased efficiency, another is integration and compatibility with other systems.

  For the same reason, the kernel must not contain anything that could just as well be implemented in add-on libraries. For example, support for the EUI data structures should be kept out of the kernel. For distributed systems, the naming of processes, objects and services could be implemented by a general, replaceable name service in user-space. Scheduling should be implemented a user-defined scheduler, and must not be hard-wired into the kernel. Support for the transportation and distribution of objects should be based on protocols, which allow for multiple programming languages, transport layers, and object persistence implementations.

• Based on main stream technology :
  Many existing systems contain deep type hierarchies with a single root. In such systems, all useful function tends to drift towards the root, whose interface quickly gets fat. The single root also makes it virtually impossible to combine the type hierarchy with other systems. In contrast, new object-oriented systems tend to contain small, shallow, orthogonal type hierarchies which are combined with multiple inheritance (mix-in techniques).

  Some systems invent their own specification languages, scripting languages, graphical interface, when there are existing, generally accepted alternatives. The use of mainstream components increases the probability that the system will be accepted and used.

  Many existing distributed image processing system (AVS, Khoros etc) are based on computational networks, consisting of filters which operate on data streams. While this abstraction might be appropriate for some image processing problems, reactive real-time systems needs more control over the computations. There is a strong trend towards distributed object-oriented systems, in which the programmer gets the feeling that all objects exists in a singe process, but where some method invocations are actually implemented by transparent remote procedure calls, shared buffers access, etc (CORBA, DOE, Spring and others).

• Well engineered :
  The aesthetic aspect of a system should not be underestimated. Most successful software packages are good compromises between generality, efficiency, and size. The concepts are often simple and straight-forward. The software are usually build on standard components, as far as possible, and use wide-spread, well suited programming languages.

  It is also important in order to create embedded applications where you do not need to "download" more than what is actually used. Moreover, it could enable a "progressive learning scheme" so that a newcomer does not need to learn more than is basically needed for his application to get started.

• With support for development :
  Several existing system lacks adequate tools for testing and debugging new applications. The system must be robust and e.g. detect crashed processes and communication problems. It must be possible to monitor program execution, especially in distributed systems, and to trace events.

The real-time restriction is interesting and important also for a number of reasons, related industrial applications :

1. Real-time vision allows for experimentation with a whole new set of applications not possible with simulations (simulations ("slowing down time") is only possible if you do not have to react to physical phenomena that have their own inherent timing).
2. Real-timing allows for creating vision applications that are responsive to the need of existing industrial settings (or in other word we can then create vision applications that respond to a world problem instead of the other way around).
3. Real-time vision is the "ultimate" version of "fast vision" (as real-time in my textbook refers to timing in fixed time-frame that is shorter than what is needed for a particular task). "Fast vision" has shown to provide at least partial answers to problems such as optical flow (due to the limitations of possible movements between frames) etc.

In terms of building real-time systems there are several commercially available operating systems, and recently tools for specification of such systems have also become available in the academic community. It is, however, characteristic that the real-time tools available today all assume the real-time systems are homogeneous, from a complexity point of view, and typically only a few selected routines require real-time response. In fully fledged computer vision systems the need for guaranteed response times varies from a few milliseconds (low-level control loops) to several seconds (symbolic interpretation and planning tasks). The requirements, in terms of response times, are consequently highly heterogeneous. In addition computer vision and image analysis is characterized by an excessive dataflow (6-20 MB/s), this in turn requires that an execution environment must be able to create and destroy large data structures without any effect on the performance of the system. On top of this facilities for control of processing, analysis and interpretation must be supported/provided to ensure limited size models.