Minutes of the 13 Jan 2011 Teleconference Austin-509 Page 1 of 1 Submitted by Andrew Josey, The Open Group. January 15 , 2010 Attendees Andrew Josey, The Open Group Don Cragun, PASC OR Mark Brown, IBM, TOG OR Geoff Clare, The Open Group Nick Stoughton, USENIX, ISO/IEC OR Eric Blake, Red Hat We had some troubles with the teleconference bridge and so had to use a temporary bridge for this call. A new number will has been issued for the next teleconference, check the events diary at: https://www.opengroup.org/platform/single_unix_specification/event.tpl?geid=13749 *Old Business carried forward: We discussed the status of the PASC chair. It was agreed that Nick (as PASC secretary) will schedule a conference call of the PASC SEC, with a tentative date of 24 February. Andrew has now started six interpretations and comments are due back by January 16 2011. Interps: 309 329 343 345 347 352 C1x ballot: There was some mail traffic about the comments due date for the ballot. Nick stated that comments are still due by the end of January - we may have to submit them via liaison rather than the US ballot. As a reminder Nick suggests that we use classic aardvark format, emailed to him. * We picked up on regular Bug processing Bug 0000336: Add recursive variable indirection in makefiles REOPENED http://austingroupbugs.net/view.php?id=336 (carried forward from last time --We left this open ) Bug 0000149: Add fdwalk system interface Accepted as marked http://austingroupbugs.net/view.php?id=149 Eric had written a proposal on changes to the existing close() and dup2() APPLICATION USAGE sections addressing why it is dangerous to close random file descriptors. Eric will also propose some new interfaces that have atomic handling of file descriptors (still to do). The first proposal was discussed, and the final changes are as follows. This was tagged for TC1-2008. This was discussed again during the 16 Dec 2010 meeting. Group consensus was that closefrom( ) cannot be standardized, because the standard explicitly permits the creation of file descriptors that are essential to the implementation (for example, posix_trace_* functionality is often implemented in terms of an inherited fd), and arbitrarily closing all file descriptors beyond a certain value could break the trace and render the application potentially non-conforming. There is no point in standardizing something while declaring that it cannot be portably used (but implementations like BSD can continue to provide it as an extension, as is true of any other non-standardized interface). The group also discussed that since fdwalk( ) has a callback function that operates in user-space, it is necessarily non-atomic. It is possible that closefrom() can be implemented in an atomic manner if it is not based on top of fdwalk( ). Meanwhile, because fdwalk( ) allows for discrimination on which file descriptors to close, it is possible that fdwalk( ) might be usable in a manner that does not invalidate implementation-specific file descriptors, so a future revision of the standard may indeed add fdwalk( ), although no one in the meeting was willing to draft a proposal for fdwalk( ) at this time (again, implementations like Solaris can continue to provide it as an extension). Another suggestion was made for adding a new fcntl( ) command, fcntl(fd,FD_NEXT,flags), which returns the next open file descriptor after fd that satisfies the conditions of flags (where flags could request the next fd in general, the next fd open for reading, the next fd open for writing, etc.). It has the same limitations as fdwalk( ) in being non-atomic, but is more efficient than iterating through every possible file descriptor value and checking to see if that value maps to an open file. However, unlike closefrom( ) and fdwalk( ), it lacks prior implementation practice at this time. A similar request for adding a command to the posix_spawn_file_actions namespace for closing all file descriptors beyond a certain range has the same drawbacks as calling closefrom( ) after fork( ). As a side note, it was mentioned that behavior differs on whether a posix_spawn_file_actions_addclose( ) that closes an already closed file descriptor should make posix_spawn( ) fail or should be ignored as a no-op, and 0000370 was raised to address that point; meanwhile, an application can use an extra posix_spawn_file_actions_adddup2( ) to guarantee that a file descriptor will not be already closed, but with the same risk of non-compliant behavior if doing so overwrites a file descriptor that must be inherited into the child for compliant behavior. Revisiting the original complaint: multi-threaded applications are already aware of the security implications of leaking unintended file descriptors to a child process. The race is possible when one thread creates a file descriptor, then another thread calls fork( ) and exec( ) before the first can use fcntl( ) to mark the file descriptor as close-on-exec. Since the second thread cannot know which file descriptors might have suffered from this race, a common approach has been to try and close all unknown file descriptors after the fork( ) and before the exec( ), rather than relying on the cloexec flag. But as pointed out above, closing a file descriptor that might be in use by the application leads to non-conforming results. Instead, if the standard were to provide ways to guarantee that file descriptors are atomically created as close-on-exec, then it is possible to write conforming software that need not iterate over file descriptors after fork( ), because there is no longer a close-on-exec race that could leak unintended file descriptors, and by avoiding the iteration altogether, there is no longer a risk of accidentally closing a file descriptor essential to the implementation. Therefore, the rest of this proposal seeks to document the problem with closing arbitrary file descriptors, and a new bugid will be opened to propose standardizing some recent interfaces and interface extensions first appearing in Linux (new interfaces such as pipe2( ), accept4( ), mkostemps( ), ..., and extensions like fopen(,"we")) to guarantee the atomic creation of file descriptors with the cloexec bit already set, as was already done in the standard with O_CLOEXEC in open( ) and F_DUPFD_CLOEXEC in fcntl( ). See also 0000368 for a related proposal to require CLOEXEC on hidden descriptors. Make the following changes in TC1: At line 22962 [XSH close APPLICATION USAGE], add a paragraph: Implementations may use file descriptors that must be inherited into child processes for the child process to remain conforming, such as for message catalog or tracing purposes. Therefore, an application that calls close( ) on an arbitrary integer risks non-conforming behavior, and close( ) can only portably be used on file descriptor values that the application has obtained through explicit actions, as well as the three file descriptors corresponding to the standard file streams. In multi-threaded parent applications, the practice of calling close( ) in a loop after fork( ) and before an exec call in order to avoid a race condition of leaking an unintended file descriptor into a child process, is therefore unsafe, and the race should instead be combatted by opening all file descriptors with the FD_CLOEXEC bit set unless the file descriptor is intended to be inherited across exec. also, at line 22968 [RATIONALE], add a new paragraph: The standard developers rejected a proposal to add closefrom( ) to the standard. Because the standard permits implementations to use inherited file descriptors as a means of providing a conforming environment for the child process, it is not possible to standardize an interface that closes arbitrary file descriptors above a certain value while still guaranteeing a conforming environment. At line 24917 [XSH dup2 APPLICATION USAGE], change: The dup( ) and dup2( ) functions are redundant. Their services are also provided by the fcntl( ) function. They have been included in this volume of POSIX.1-2008 primarily for historical reasons, since many existing applications use them. to: The dup( ) function is redundant. Its services are also provided by the fcntl( ) function. It has been included in this volume of POSIX.1-2008 primarily for historical reasons, since many existing applications use it. On the other hand, the dup2( ) function provides unique services, as no other interface is able to atomically replace an existing file descriptor. also, add a paragraph after line 24926: Implementations may use file descriptors that must be inherited into child processes for the child process to remain conforming, such as for message catalog or tracing purposes. Therefore, an application that calls dup2( ) with an arbitrary integer for filedes2 risks non-conforming behavior, and dup2( ) can only portably be used to overwrite file descriptor values that the application has obtained through explicit actions, or for the three file descriptors corresponding to the standard file streams. In order to avoid a race condition of leaking an unintended file descriptor into a child process, an application should consider opening all file descriptors with the FD_CLOEXEC bit set unless the file descriptor is intended to be inherited across exec. At line 46906 [XSH posix_spawn_file_actions_addclose], add a paragraph: Implementations may use file descriptors that must be inherited into child processes for the child process to remain conforming, such as for message catalog or tracing purposes. Therefore, an application that calls posix_spawn_file_actions_addclose( ) with an arbitrary integer risks non-conforming behavior, and this function can only portably be used to close file descriptor values that the application has obtained through explicit actions, or for the three file descriptors corresponding to the standard file streams. In order to avoid a race condition of leaking an unintended file descriptor into a child process, an application should consider opening all file descriptors with the FD_CLOEXEC bit set unless the file descriptor is intended to be inherited across exec. At line 46997 [XSH posix_spawn_file_actions_adddup2], add a paragraph: Implementations may use file descriptors that must be inherited into child processes for the child process to remain conforming, such as for message catalog or tracing purposes. Therefore, an application that calls posix_spawn_file_actions_adddup2( ) with an arbitrary integer for newfiledes risks non-conforming behavior, and this function can only portably be used to overwrite file descriptor values that the application has obtained through explicit actions, or for the three file descriptors corresponding to the standard file streams. In order to avoid a race condition of leaking an unintended file descriptor into a child process, an application should consider opening all file descriptors with the FD_CLOEXEC bit set unless the file descriptor is intended to be inherited across exec. Bug 0000368: Hidden file descriptors should be required to have the FD_CLOEXEC flag set and be closed when no longer needed. Accepted http://austingroupbugs.net/view.php?id=367 This item was tagged for issue 8. Bug 0000370: addclose should not cause posix_spawn to fail if closing an already closed fd Accepted http://austingroupbugs.net/view.php?id=370 This item was tagged for issue 8. Bug 346 ceil and floor Accepted as Marked http://austingroupbugs.net/view.php?id=346 An interpretation is required Interpretation response The standard states that checks for overflow should be made, even though they cannot happen , and conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor. Rationale: See Notes above Notes to the Editor (not part of this interpretation): See Note: 0000633 Next Steps ---------- The next call will be on January 20th at 09:00 Pacific and will continue processing defect reports. This call will be for the regular 90 minutes. *There will be new dialin numbers for the call* http://austingroupbugs.net See the calendar for the list of dialup numbers. An IRC channel will be available for the meeting irc://irc.freestandards.org #austin ICAL: http://www.google.com/calendar/ical/nvctqtstkuni3fab9k3jqtrt4g@group.calendar.google.com/public/basic XML: http://www.google.com/calendar/feeds/nvctqtstkuni3fab9k3jqtrt4g@group.calendar.google.com/public/basic