Minutes of the 4 August 2011 Teleconference Austin-533 Page 1 of 1 Submitted by Andrew Josey, The Open Group. August 6 , 2011 Attendees Andrew Josey, The Open Group Don Cragun, PASC OR Mark Brown, IBM, TOG OR Nick Stoughton, USENIX, ISO/IEC OR Eric Blake, Red Hat Apologies Geoff Clare, The Open Group Jim Pugsley, Oracle The invitation to join the IEEE ballot group has gone out, with copies of the invitation sent to the Austin Group, the PASC SEC and the PASC general list. It was noted that its not mandatory to sign up for IEEE balloting, and balloting occurs also in the Austin Group. Andrew noted that he still needs to complete the Mandatory Editorial Coordination with our IEEE project manager, who will check the document format for any style concerns. One item progressed in the last week has been the autogeneration of Rationale lines that now include a reference to the defect report and a URL. We are still on track to make TC1 Draft 3 available on August 15th. The final bugs from Mantis need to be applied. We also need to start any pending interpretations. * Old Business Bug 281 sort -M http://austingroupbugs.net/view.php?id=281 Don had followed up and completed his action to propose updates to the bug. There has been further discussion by email and we are currently trying to setup a discussion at the weekly teleconference with the submitter. * New Business We picked up on Bug processing for bugs reported against project TC1-2008. Bug 0000470: stdio.h function SEE ALSO updates Accepted as marked http://austingroupbugs.net/view.php?id=470 Eric had completed his action from last week. Delete the editorial note in lines 2673-2674: This applies to the fopen() function and all other functions listed in the header Then copy lines 2675-2676: In the SEE ALSO section, add a reference to XSH Section 2.5. Rationale: Editorial improvement. as the action for one change proposal for each of these interfaces: fclose page 806 line 26845 fflush page 846 line 28101 fgetc page 849 line 28206 fgetpos page 850 line 28261 fgets page 853 line 28316 fgetwc page 855 line 28381 fgetws page 856 line 28437 fprintf page 904 line 30229 fputc page 907 line 30319 fputs page 909 line 30379 fputwc page 911 line 30445 fputws page 912 line 30495 fread page 914 line 30558 freopen page 925 line 30983 fscanf page 935 line 31351 fseek page 938 line 31460 fsetpos page 941 line 31541 ftell page 956 line 32066 fwprintf page 979 line 32842 fwrite page 981 line 32908 fwscanf page 989 line 33199 getc page 992 line 33301 getc_unlocked page 994 line 33375 getchar page 995 line 33412 getdelim page 1006 line 33803 gets page 1070 line 35775 getwc page 1085 line 36186 getwchar page 1086 line 36218 popen page 1409 line 46171 putc page 1713 line 54800 putchar page 1715 line 54834 puts page 1724 line 55083 putwc page 1726 line 55126 putwchar page 1727 line 55157 rewind page 1786 line 57142 setbuf page 1855 line 59295 tmpfile page 2122 line 67175 ungetc page 2151 line 67946 ungetwc page 2153 line 67992 vfprintf page 2168 line 68459 vfscanf page 2170 line 68503 vfwprintf page 2171 line 68540 vfwscanf page 2172 line 68581 Bug 0000482: fclose change needs updating to match bug 87 change Accept http://austingroupbugs.net/view.php?id=411 ACTION: Andrew will apply these bugs to draft 3 which will complete the draft. * Issue 7 bugs Bug 0000411: adding atomic FD_CLOEXEC support Accepted as marked http://austingroupbugs.net/view.php?id=411 Eric had completed his action. Replace the original "Desired Action" for popen (affecting lines 46095 through 46167) with: At lines 45748-45763 [XSH pclose RATIONALE], replace the pclose( ) sample implementation with: See the RATIONALE for popen( ) for a sample implementation of pclose( ). At line 46093 [XSH popen DESCRIPTION], change: The popen( ) function shall ensure that any streams from previous popen( ) calls that remain open in the parent process are closed in the new child process. to: The popen( ) function shall ensure that any streams from previous popen( ) calls that remain open in the parent process are closed in the new child process, regardless of the FD_CLOEXEC status of the file descriptor underlying those streams. At lines 46095 and 46099 [XSH popen DESCRIPTION], change "If mode is" to "If mode starts with". After line 46102 [XSH popen DESCRIPTION], add the following: 3. If mode includes a second character of e, then the file descriptor underlying the stream returned to the calling process by popen( ) shall have the FD_CLOEXEC flag atomically set. Additionally, if the implementation creates the file descriptor for use by the child process from within the parent process, then that file descriptor shall have the FD_CLOEXEC flag atomically set within the parent process. If the second character is not e, the FD_CLOEXEC flag of the underlying file descriptor returned by popen( ) shall be clear. At line 46103 [XSH popen DESCRIPTION], change "3." to "4.". At line 46148 [XSH popen APPLICATION USAGE], change "r and w" to "r, w, re, and we". After line 46167 [XSH popen RATIONALE], add the following: The e mode modifier to popen( ) is necessary to avoid a data race in multi-threaded applications. Without it, the parent's file descriptor is leaked into a second child process created by one thread in the window between another thread creating the pipe via popen( ) then using fileno( ) and fcntl( ) on the result. Also, if the popen( ) implementation temporarily has the child's file descriptor open within the parent, then that file descriptor could also be leaked if it is not atomically FD_CLOEXEC for the duration in which it is open in the parent. The standard only requires that the implementation atomically set FD_CLOEXEC on file descriptors created in the parent process when the e mode modifier is in effect; implementations may also do so when the e modifier is not in use, provided that the FD_CLOEXEC bit is eventually cleared before popen( ) completes, however, this is not required because any application worried about the potential file descriptor leak will already be using the e modifier. Although the standard is clear that a conforming application should not call popen( ) when file descriptor 0 or 1 is closed, implementations are encouraged to handle these cases correctly. The following two examples demonstrate possible implementations of popen( ) using other standard functions. These examples are designed to show FD_CLOEXEC handling rather than all aspects of thread safety, and implementations are encouraged to improve the locking mechanism around the state list to be more efficient. Also, remember that other implementations are possible, including an implementation that uses an implementation-specific means of creating a pipe between parent and child where the parent process never has access to the child's end of the pipe. Both of these examples make use of the following helper functions, documented but not implemented here, to do the bookkeeping necessary to properly close all file descriptors created by other popen( ) calls regardless of their FD_CLOEXEC status: /* Obtain mutual exclusion lock, so that no concurrent popen( ) or pclose( ) calls are simultaneously modifying the list of tracked children. */ static void popen_lock(void); /* Release mutual exclusion lock, without changing errno. */ static void popen_unlock(void); /* Add the pid and stream pair to the list of tracked children, prior to any code that can clear FD_CLOEXEC on the file descriptor associated with stream. To be used while holding the lock. */ static void popen_add_pair(FILE *stream, pid_t pid); /* Given a stream, return the associated pid, or -1 with errno set if the stream was not created by popen( ). To be used while holding the lock. */ static pid_t popen_get_pid(FILE *stream); /* Remove stream and its corresponding pid from the list of tracked children. To be used while holding the lock. */ static void popen_remove(FILE *stream); /* If stream is NULL, return the first tracked child; otherwise, return the next tracked child. Return NULL if all tracked children have been returned. To be used while holding the lock. */ static FILE *popen_next(FILE *stream); The first example is based on fork( ): #include #include #include #include #include FILE *popen(const char *command, const char *mode) { int fds[2]; pid_t pid; FILE *stream; int target = mode[0] == 'w'; /* index of fds used by parent */ /* Validate mode */ if ((mode[0] != 'w' && mode[0] != 'r') || mode[1 + (mode[1] == 'e')]) { errno = EINVAL; return NULL; } /* Create pipe and stream with FD_CLOEXEC set */ if (pipe2(fds, O_CLOEXEC) < 0) return NULL; stream = fdopen(fds[target], mode); if (!stream) { int saved = errno; close(fds[0]); close(fds[1]); errno = saved; return NULL; } /* Create child process */ popen_lock(); pid = fork(); if (pid < 0) { int saved = errno; close(fds[!target]); fclose(stream); popen_unlock(); errno = saved; return NULL; } /* Child process. */ if (!pid) { FILE *tracked = popen_next(NULL); while (tracked) { int fd = fileno(tracked); if (fd < 0 || close(fd)) _exit(127); tracked = popen_next(tracked); } target = mode[0] == 'r'; /* Opposite fd in the child */ /* Use dup2 or fcntl to clear FD_CLOEXEC on child's descriptor, FD_CLOEXEC will take care of the rest of fds[]. */ if (fds[target] != target) { if (dup2(fds[target], target) != target) _exit(127); } else { int flags = fcntl(fds[target], F_GETFD); if (flags < 0 || fcntl(fds[target], F_SETFD, flags | FD_CLOEXEC) < 0) _exit(127); } execl("/bin/sh", "sh", "-c", command, NULL); _exit(127); } /* Parent process. From here on out, the close and fcntl system calls are assumed to pass, since all inputs are valid and do not require allocating any fds or memory. Besides, excluding failures due to undefined behavior (such as another thread closing an fd it knows nothing about), cleanup from any defined failures would require stopping and reaping the child process, which may have worse consequences. */ close(fds[!target]); popen_add_pair(stream, pid); popen_unlock(); if (mode[1] != 'e') { int flags = fcntl(fds[target], F_GETFD); if (flags >= 0) fcntl(fds[target], F_SETFD, flags & ~FD_CLOEXEC); } return stream; } The second example is based on posix_spawn( ): #include #include #include #include #include #include extern char **environ; FILE *popen(const char *command, const char *mode) { int fds[2]; pid_t pid; FILE *stream; int target = mode[0] == 'w'; /* index of fds used by parent */ const char *argv[] = { "sh", "-c", command, NULL }; posix_spawn_file_actions_t actions; int saved; FILE *tracked; /* Validate mode */ if ((mode[0] != 'w' && mode[0] != 'r') || mode[1 + (mode[1] == 'e')]) { errno = EINVAL; return NULL; } /* Create pipe and stream with FD_CLOEXEC set */ if (pipe2(fds, O_CLOEXEC) < 0) return NULL; stream = fdopen(fds[target], mode); if (!stream) { saved = errno; close(fds[0]); close(fds[1]); errno = saved; return NULL; } /* Create child process */ if (posix_spawn_file_actions_init(&actions)) { saved = errno; goto spawnerr1; } popen_lock(); tracked = popen_next(NULL); while (tracked) { int fd = fileno(tracked); if (fd < 0 || posix_spawn_file_actions_addclose(&actions, fd)) goto spawnerr2; tracked = popen_next(tracked); } if (posix_spawn_file_actions_adddup2(&actions, fds[!target], !target)) goto spawnerr2; if (posix_spawn(&pid, "/bin/sh", &actions, NULL, (char **)argv, environ)) { spawnerr2: saved = errno; posix_spawn_file_actions_destroy(&actions); popen_unlock(); spawnerr1: close(fds[!target]); fclose(stream); errno = saved; return NULL; } /* From here on out, system calls are assumed to pass, since all inputs are valid and do not require allocating any fds or memory. Besides, excluding failures due to undefined behavior (such as another thread closing an fd it knows nothing about), cleanup from any defined failures would require stopping and reaping the child process, which may have worse consequences. */ posix_spawn_file_actions_destroy(&actions); close(fds[!target]); popen_add_pair(stream, pid); popen_unlock(); if (mode[1] != 'e') { int flags = fcntl(fds[target], F_GETFD); if (flags >= 0) fcntl(fds[target], F_SETFD, flags & ~FD_CLOEXEC); } return stream; } Both examples can share a common pclose( ) implementation. int pclose(FILE *stream) { int status; popen_lock(); pid_t pid = popen_get_pid(stream); if (pid < 0) { popen_unlock(); return -1; } popen_remove(stream); popen_unlock(); fclose(stream); /* Ignore failure */ while (waitpid(pid, &status, 0) == -1) { if (errno != EINTR) { status = -1; break; } } return status; } Note that, while a particular implementation of popen( ) (such as the two above) can assume a particular path for the shell, such a path is not necessarily valid on another system. The above examples are not portable, and are not intended to be. Bug 0000465: is the list of special built-ins exhaustive (is "local" special)? OPEN http://austingroupbugs.net/view.php?id=465 This was discussed on the conference call, 2011-08-04. We agreed that the problem was a serious one, but no consensus was reached with respect to the proposed actions. One possibility that was discussed was using some form of reserved namespace for application writers to use to avoid such problems, such as the use of uppercase letters or underscores in any function name. Bug 0000469: asctime and integer overflow OPEN http://austingroupbugs.net/view.php?id=469 It was agreed there is a problem although we did not reach closure on the action. Any major change would be for Issue 8. One change that could be done as an errata would be 1990->1900 as in: Change from: if timeptr->tm_year exceeds {INT_MAX}-1990, to: if timeptr->tm_year exceeds {INT_MAX}-1900, It looks as if the C1X wording might be a candidate for replacing the text. Eric took an action to add a note to the bug (Completed) Next Steps ---------- The next call is on August 11th at 08:00 Pacific and will continue processing defect reports. This call will be for the regular 90 minutes. http://austingroupbugs.net See the calendar for the list of dialup numbers. An IRC channel will be available for the meeting irc://irc.freestandards.org #austin ICAL: http://www.google.com/calendar/ical/nvctqtstkuni3fab9k3jqtrt4g@group.calendar.google.com/public/basic XML: http://www.google.com/calendar/feeds/nvctqtstkuni3fab9k3jqtrt4g@group.calendar.google.com/public/basic