Real-time and Embedded Systems Forum
San Jose, 6th - 8th February 2001
Summary
The Real-time and Embedded Systems Forum met from Tuesday afternoon February 6th through to Thursday February 8th. The forum was divided into three sessions, an open session on real-time and embedded Linux, the general open plenary session and the forum working group meetings. A joint meeting on quality of service for real-time systems was also held and that is minuted separately.
The first session was on the topic of real-time and embedded Linux. This included four presentations from speakers from FSM Labs, Linuxdevices.com, MontaVista and Lineo outlining the multiple approaches to achieving real-time capabilities with Linux and a market survey of the embedded marketplace. One approach to real-time is to run Linux as a subthread of a real-time executive (for example RTLinux), the other being to add real-time capabilities to Linux itself (MontaVista). The market survey from LinuxDevices.com showed that the major factors for the popularity of Linux as an embedded system are availability of source code, the lack of run-time royalties and the robustness of the product.
The second session was the general plenary with invited speakers. This included two keynote sessions from Gabriel Broner of SGIand Inder Singh from LynuxWorks. The sessions ranged from presentations on hard real-time operating systems, through security for real-time systems in the power industry, real-time Java, the Department of Defense Real-time Common Operating Environment, to the movement towards standards based solutions in the avionics industry.
The issues that kept recurring were security and standardisation. This was a reflection of how wide-open the market place is in open system considerations and user requirements vs. vendor solutions.
The third session was the working group meeting for the forum members. The objectives of this meeting were to review the output of the last meeting, to examine the charters of the established subgroups and to identify work items and volunteers . Further details are available in the minutes to that meeting.
The future meeting schedule for the forum is as follows:
Session Overview
The sessions lasted over a 1˝-day period starting in the afternoon of Tuesday 6th February and finishing in the evening of 7th February. They covered the following subjects:
Linux Interest Group session
General Open Session Plenary
Andrew welcomed the attendees to San Jose
In his introduction he gave an overview of why the forum has been setup, the mission of the forum, who the current members are and a brief look at the current activities.
RTLinux is a POSIX PSE51 (minimal) small hard real-time operating system that runs a PSE54 (full) operating system (Linux) as its lowest priority thread.
This session covered the technology that makes such an architecture possible, the performance, and the application programmers view of a system in which a real-time thread operates in an threads/signals environment that contains within it a general purpose kernel and POSIX processes. The RTLinux architecture was contrasted with the traditional "real-time UNIX" designs and with competing views of real-time in Linux.
He opened by comparing Real-time with time-sharing and showed two conceptual views of RTLinux. The importance of decoupling was stressed in that the Linux scheduler is not involved, and that Linux cannot delay or interrupt RTLinux. A history of RTLinux was presented. Version 3 is considered the first industrial strength version.
In general the programming model for RTLinux is to use as many of the standard interfaces from POSIX (threads & signals), and to keep them as standard as possible. Several examples of code were discussed, and some views on POSIX in general, and POSIX 1003.13 specifically were given . A full POSIX file system is impossible to make deterministic. Some particular areas of concern were the use of the pthread_cancel() function being able to leave several processes simply hanging, SMP was weakly specified and periodic scheduling was too wordy. An example comparing a Legacy network disk storage device to an RTLinux disk storage system was analysed with the RTLinux version being shown to be much simpler conceptually and smaller primarily due the collapsing of several layers of complexity.
The session closed with a presentation of the three sample applications of use of RTLinux.
Linux is rapidly emerging as an excellent Embedded Operating System. This session discussed the status of Linux as an embedded operating system and the good and bad points of using Embedded Linux. It also reviewed some interesting data collected in the LinuxDevices.com Embedded Linux Market Study about reasons why developers are interested in using Embedded Linux and open source software, etc.
Moore's law was shown as operational in that embedded systems are becoming ever more powerful and complex with the disadvantages of traditional operating systems being highlighted. The facts that Linux is open source and royalty free was seen as a major contributor to its success in this market.
Rick reported that incessant and accelerated chipset obsolescence and innovation resulted in embedded developers relying on well-supported Operating Systems because:
A dozen examples of devices with Embedded Linux inside were shown and a general discussion agreed that there was an urgent need for standards whilst it was clear that the definition of 'embedded' differed quite widely. The same applied to what 'proprietary' actually meant.
The session drew to a close with fifteen slides showing the results of a market survey analysis from some 500 or so self-selected people visiting the website.
Rick concluded with three major predictions:
This session discussed MontaVista's efforts to transform Linux into a fully pre-emptable kernel without compromising throughput-based performance. It presented MontaVista's Linux Real Time scheduler and modifications to the Linux kernel to enable full pre-emption. The talk covered the real-time scheduling algorithm along with priority inheritance semaphores. It also reviewed other protection methods used to dramatically improve system responsiveness as well as the impact on throughput.
George began by outlining other efforts to improve Linux latency and stated that MontaVista's goal was to move the standard Linux response times down with pre-emption times reflecting ~30 dispatch times and about 250 microseconds on a 4-500mhz ix86 processor.
The main points addressed were:
The session concluded with a lively and general discussion about coverage analysis. Latency was really the only thing that mattered and the management of what went in was the biggest problem in latency management.
This session discussed an add-on to Linux which is not part of the mainstream Linux source tree , the Real-Time Applications Interface (RTAI).
Dave started off by outlining RTAI's focus, showing it's architecture, explaining some of it's basics and discussing it's IPCs. RTAI is similar to RTLinux in implementation. It uses it's own separate API with 10 - 20 modules.
He continued by going through many RTAI features including some unique ones such as:During one of the subsequent supporting slides on Dynamic Memory Allocation slide a discussion developed about what 'safe' actually meant.
Dave felt that the three main reasons for a new API were:
He went on to talk about developing with RTAI and concluded his session with a synopsis of RTAI support elements.
A further general discussion ensued which addressed the need of separating the RT and non-RT elements.
Gabriel Broner, VP Strategic Software Organization, SGI
This session addressed the theme of real-time software development, the technologies, trends and challenges faced when tackling very demanding real-time computing problems and was based around the following main points:
Gabriel stated that the basic elements of a high-end OS were:
He cited as real-time application examples Man-in-the-loop simulators, Hardware-in-the-loop simulators, Telemetry, Radar, Data Acquisition, Command, Control, Communication, Computers and Intelligence (C4I) and Video Applications
He showed that the following were required for a real-time OS exhibiting a deterministic response time:
Gabriel explained that typical target customers in this environment were non-Enterprise ones. He demonstrated the architecture of SGI's Onyx hardware to be made up of configurable CPU, I/O and graphics bricks.
He concluded with an in depth feature array of the IRIX OS with particular emphasis on it's real-time features and a brief overview of SGI's real-time customers; these being:
This session explained how the electrical power industry is becoming more automated and electronically connected with much critical data and information being transmitted through corporate LANS and WANs; the alarming thing being that in many instances by direct connection to the Internet. It highlighted how there has been a significant amount of security work devoted to Internet and e-commerce applications yet very little corresponding work developed for real time control and sensing systems. These systems have unique attributes that need to be considered in their design over and beyond the existing efforts for the Internet and e-commerce applications and stressed the security needs for these systems. It also high-lighted lack of vulnerability tests on control equipment.
Joe had started out with certain expectations but found many of them to be specious. His observations in the Business Systems and Internet area were;
and in the Operational Systems were:
He stressed the extreme difficulty of laboratory tests because of the deeply integrated nature of the equipment.
He went on to outline the key security aspects of Process Control Systems and stated his concern over the vulnerability issues within them. They included systems not being designed with security as a driver, security and functional performance being mutually exclusive with trade-offs as necessary, electronic security vulnerabilities being able to exist at the network, system, and component level, internet-accessible hacking tools be able to be used and unique aspect of these systems not being addressed by IT Security procedures or technology.
He also addressed other issues such as security vendors and many systems vendors have not recognized need for security, information security concerns even with government agencies, any disgruntled employee is everyone's disgruntled employee, lack of information security at customer site could be major utility liability and UCITA.
Joe then presented EPRI EIS's program and concluded with an in-depth analysis of the interactions.
Dr. Inder M. Singh, Lynuxworks, Inc. And Chairman of the Embedded Linux Consortium
Dr. Singh opened with an overview of how major market trends were converging and that Linux was well placed to be at the centre of this. "Embedded developer seats will overtake desktop/server developer seats by 2005 - IDC (Feb. 2000)".
He stated that because of the complexity of embedded applications Linux was better placed to fit the demand of today's embedded applications than the traditional flat address space Real-time operating systems. He explored the post-PC consumer market ("Cahners In-Stat group: Internet appliances will grow by a factor of more than 15 by the end of 2004 to 37 million units, up from 2 million shipments in 1999) and went on to highlight the explosion of embedded products with intelligence being incorporated into more and more devices.
He felt that the five major reasons for Linux being the up and coming software enabler were:
He explored the issues of true real-time technology and addressed the three real-time approaches to Linux, putting the multiple approaches down to the fact that this is still a relatively young market. He expects a quick adoption of the 2.4 kernel due to its major new feature set in the embedded marketplace and continued with the challenges involved.
The business model for Linux is one of income stream rather than licensing the source.
He summarised by his assertion that embedded software was now poised on the magic spring board point and that Linux was able to provide the missing platform for the embedded industry, based on open standards and open sources.
The Java platform is poised to displace a multitude of existing real-time development platforms but there is considerable uncertainty as to what form a real-time Java platform might take. This talk surveyed existing real-time Java products, the NIST-drafted requirements for real-time Java extensions, current draft specifications for real-time Java extensions from a standards-setting perspective, and Dr. Nilsen's own experiences of bringing real-time Java technologies to the commercial marketplace.
Dr. Nilsen offered his recommendations for standards-setting bodies and end users seeking to leverage the new technologies enabled by the emerging standards.
He opened by explaining why RTJ was relevant supporting this with the history of Java. He continued with a detailed analysis of NIST goals and derived requirements. He recognised the importance of having the right tool for the right job ("is one language best for every application?") just so long as the infrastructure remained constant; the analogy given was that of different types of photography being supported by a constant infrastructure such as same film and print paper technologies, same processing chemicals etc.
He presented the criteria for comparison as:
He the proceeded an analysis of these criteria.
He observed that different technologies support different needs (horses for courses) and that potential users should be far more vociferous in stating how much a particular type of technology was worth to them.
In conclusion he stated that if there was truly a desire of industry and government for open systems and conformance then the community could help by rreviewing draft specifications and providing feedback, helping fund development of prototype implementations and trial case studies, guiding the development of certification and branding programs that suit their objectives.
This talk provided an overview/update of the ongoing efforts to extend the Defence Information Infrastructure Common Operating Environment (DII COE) in its support of real-time systems. Mr. Emery explained that a key goal for the RT Extensions is to be standards-based, and supporting multiple commercial implementations rather than to select a single commercial product. He presented the overall intent and structure of the DII COE, the RT extensions, and the current status of the effort. In particular, he identified where appropriate standards for adoption have been unable to be found.
David commenced with a DII COE overview including the relationship to DII, COE objectives, DII COE components, applications running on the COE and DII COE run-time overview. He moved on to consider the Kernel Platform Certification followed by DII COE Vision. There was an in depth review of POSIX profiles and a look at standardisation issues beyond POSIX.
He went on to talk about the issues and challenges now being faced which included:
and concluded by addressing the challenges faced by the real-time open systems community such as:
The concluding discussion highlighted the need for government funding of test suites and the potency of inviting both vendor and user to look at the same bug issues.
This session focused on the IRIX OS and stated that developing and testing real-time applications is often done by trial and error.
Mr. Stuart discussed real-time design choices and performance verification whilst considering in detail the development and monitoring tools available in the IRIX programming environment; many standard development and monitoring tools change the performance characteristics of the real time application being developed. Utilising special real-time features often eliminates standard monitoring tools from consideration. Utilizing text output often takes more system resources and time than the work performed by the real-time task. Many times the failure to meet a real-time response requirement has misunderstood underlying causes.
Dan opened with the following quote:
"A hard real-time system is one where the correctness of the system depends not only on the logical result of the computation, but also on the time at which the result is produced." Stankovic and Ramamritham (1988), "Hard Real-Time Systems", IEEE Computer Society Press.
He then showed the components of SGI's real-time system and moved immediately on to how IRIX handled the necessary components within a real-time system. These included:
He gave some examples on watching external interrupts with ULI and Stethoscope monitoring semaphore access time.
He concluded with the following:The vital importance of any RT system being a symbiosis of the hardware and software environments was again stressed.
This session outlined their real-time and embedded products and discussed their movement towards standards based solutions. Current barriers to adoption of standards based embedded solutions were identified including proposed areas the forum should address.
Raymond began with an overview of Rockwell Collins, followed by its mission, business units and core products. He continued with their embedded computer technologies which embraced:
He outlined the company's product trends and talked about four technology trends viz. target independent development, qualified open systems, object orientation/certification and modular programmable radio.
He concluded his presentation with what he thought the obstacles were that the Forum should address:
Several critical applications were discussed with particular emphasis being placed on medical systems.