Minutes of the 18 December 2014 Teleconference Austin-686 Page 1 of 1 Submitted by Andrew Josey, The Open Group. 19 December 2014 Attendees: Mark Ziegast, SHware Systems Don Cragun, IEEE PASC OR Joerg Schilling FOKUS Fraunhofer Geoff Clare, The Open Group Andrew Josey, The Open Group Martin Rehak, Oracle, The Open Group OR Eric Blake, Red Hat Richard Hansen, BBN Apologies Nick Stoughton, USENIX, ISO/IEC JTC 1/SC 22 OR David Clissold, IBM Mark Brown, Canonical * General news On the PASC PAR approval progress - still open. Don currently has the action to progress. * Outstanding actions +Bug 0000251: Forbid newline, or even bytes 1 through 31 (inclusive), in filenames OPEN http://austingroupbugs.net/view.php?id=251 Don has an action to produce a proposal. +Bug 0000561: NUL-termination of sun_path in Unix sockets OPEN http://austingroupbugs.net/view.php?id=561 Eric has an action to update the proposal. +Bug 0000573: Please add '+' to the portable filename character set OPEN http://austingroupbugs.net/view.php?id=573 Joerg has an action to prepare a proposed change. +Bug 0000592: consistent use of struct timespec OPEN http://austingroupbugs.net/view.php?id=592 Jim had provided additional information in bugnote 1627. This was discussed and Jim took an action to provide further information. +Bug 0000598: OH shading and new interfaces OPEN http://austingroupbugs.net/view.php?id=598 Eric has an action to propose a new solution with self-contained headers. +Bug 0000517: EBNF support OPEN http://austingroupbugs.net/view.php?id=517 Action on Joerg to look at this. +Bug 0000633: SIGEV_THREAD delivery renders many signal interfaces unsafe OPEN http://austingroupbugs.net/view.php?id=633 We noted that feedback has settled down on the mailing list, and will discuss next session. +Bug 0000657: Conditions under which fmemopen() write a NUL to the buffer are insufficiently specified OPEN http://austingroupbugs.net/view.php?id=657 Eric has an action to propose wording to clarify the behavior for fmemopen(), and also to contact the glibc developers to get their feedback. +Bug 0000658: Undefined/unspecified behavior clauses in description of open have race conditions OPEN http://austingroupbugs.net/view.php?id=658 It was noted that there is some overlap with changes in TC1. Eric took an action to update the proposal to resolve the overlaps appropriately. +Bug 0000615: pthread_setcancelstate should be async-signal-safe OPEN http://austingroupbugs.net/view.php?id=615 We now have reports on AIX and Apple. Jim to report back on whether pthread_cancelstate() is async-signal-safe on Solaris. Andrew to ask HP whether pthread_cancelstate() is async-signal-safe on HP-UX. +Bug 0000672: Necessary step(s) to synchronize filename operations on disk OPEN http://austingroupbugs.net/view.php?id=672 Geoff has a new proposed resolution in note 1618. Decided to solicit input from FS developers. Eric to go to Linux, David to AIX and Jim to Solaris. Jim has completed his action (see bugnote 1691). Andrew should chase HP and Apple for input. +Bug 0000663: Specification of str[n]casecmp is ambiguous reopened http://austingroupbugs.net/view.php?id=663 Action on David to follow up with the IBM developers about the EBCDIC collation sequence. Bug 696 either NAME_MAX shouldn't be optional, or readdir_r() needs clarification http://www.austingroupbugs.net/view.php?id=696 Don has an action to propose a resolution. Bug 0000721: Internal storage vs static storage OPEN http://austingroupbugs.net/view.php?id=721 This item is still open. Bug 0000375: Extend test/[...] conditionals: ==, <, >, -nt, -ot, -ef OPEN http://austingroupbugs.net/view.php?id=375 This is still left open due to discussions pending on the reflector. Bug 0000789: Add set -o pipefail OPEN http://austingroupbugs.net/view.php?id=789 * Current Business - Bug processing Bug #854: requirement for additional built-in utilities to be searched for via $PATH was not and is not existing practice Accepted as marked http://austingroupbugs.net/view.php?id=854 The issue is tagged for Issue 8. An interpretation is required. Interpretation response: The standard states that if a PATH search for a utility is successful, the built-in version of that utility (if present) will always be executed, and if the PATH search is not successful then attempts to execute the utility must fail with exit status 127. Conforming implementations must conform to this. However, concerns have been raised about this which are being referred to the sponsor. Rationale: Many existing implementations execute a regular built-in without performing a PATH search. This behavior does not match the normative text, and it does not allow script authors to override regular built-in utilities via a specially crafted PATH. In addition, the rationale explains that the intention is to allow authors to override built-ins by modifying PATH, but this is not what the normative text says. Notes to the Editor (not part of this interpretation): On page 46 lines 1440-1443 (XBD 3.83 definition of Built-In Utility), change: A utility implemented within a shell. The utilities referred to as special built-ins have special qualities. Unless qualified, the term "built-in" includes the special built-in utilities. Regular built-ins are not required to be actually built into the shell on the implementation, but they do have special command-search qualities. Note: Special Built-In Utilities are defined in detail in XCU Section 2.14 (on page 2356). Regular Built-In Utilities are defined in detail in XCU Section 2.9.1.1 (on page 2339). to: A utility implemented within a shell. There are two main types of built-in utilities: special built-ins and regular built-ins. Unless qualified, the term "built-in" includes both types. The utilities referred to as special built-ins have special qualities. Regular built-ins are not required to be actually built into the shell on the implementation, but they usually have special command-search qualities or affect the current execution environment. Note: Special Built-In Utilities are defined in detail in XCU Section 2.14 (on page 2356). Regular Built-In Utilities are defined in detail in XCU Section 1.6 (on page 2318). On page 65 after line 1901 insert a new section: 3.201 Intrinsic Utility A utility that is not subject to a PATH search during command search, usually implemented as a regular built-in utility. Note: Intrinsic Utilities are defined in detail in XCU Section 1.7 (on page 2318). On page 84 after line 2374 insert a new section: 3.318 Regular Built-In Utility (or Regular Built-In) See Built-In Utility in Section 3.83 (on page 46). Change the description of PATH in XBD section 8, P178, L5711-5729 from: This variable shall represent the sequence of path prefixes that certain functions and utilities apply in searching for an executable file known only by a filename. The prefixes shall be separated by a (’:’). When a nonzero-length prefix is applied to this filename, a shall be inserted between the prefix and the filename if the prefix did not end in . A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent characters ("::"), as an initial preceding the rest of the list, or as a trailing following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH. The list shall be searched from beginning to end, applying the filename to each prefix, until an executable file with the specified name and appropriate execution permissions is found. If the pathname being sought contains a , the search through the path prefixes shall not be performed. If the pathname begins with a , the specified path is resolved (see Section 4.12, on page 111). If PATH is unset or is set to null, the path search is implementation-defined. Since is a separator in this context, directory names that might be used in PATH should not include a character. to: This variable shall represent the sequence of path prefixes that certain functions and utilities apply in searching for built-ins and executable files known only by a filename. The prefixes shall be separated by a (’:’). When a nonzero-length prefix is applied to this filename, a shall be inserted between the prefix and the filename if the prefix did not end in . A zero-length prefix is a legacy feature that indicates the current working directory. It appears as two adjacent characters ("::"), as an initial preceding the rest of the list, or as a trailing following the rest of the list. A strictly conforming application shall use an actual pathname (such as .) to represent the current working directory in PATH. The list shall be searched from beginning to end, applying the filename to each prefix, until either an executable file with the specified name and appropriate execution permissions is found or, if the shell is performing the search, a built-in associated with that prefix and filename is found. If the pathname being sought contains a , the search through the path prefixes shall not be performed. If the pathname begins with a , the specified path shall be resolved (see Section 4.12, on page 111). If PATH is unset or is set to null, or if a path prefix in PATH contains a character ('%'), the path search is implementation-defined. Since is a separator in this context, directory names that might be used in PATH should not include a character. Since may have an implementation-defined meaning when searching for built-in utilities, directory names in PATH that will be used to search for non-built-in utilities should not contain a character. On page 2318 lines 73544-73553 (XCU 1.6 Built-In Utilities), change: The utilities named in Table 1-5 are frequently provided in built-in form. All of the utilities named in the table have special properties in terms of command search order within the shell, as described in Section 2.9.1.1 (on page 2339). Table 1-5 Regular Built-In Utilities alias bg cd command false fc fg getopts jobs kill newgrp pwd read true umask unalias wait However, all of the standard utilities, including the regular built-ins in the table, but not the special built-ins [...] to: The intrinsic utilities described in Section 1.7 (on page 2318) are frequently provided as regular built-ins. However, all of the standard utilities, including the regular built-ins but not the special built-ins [...] On page 2318 after line 73556 insert a new section: 1.7 Intrinsic Utilities As described in Section 2.9.1.1 (on page 2339), intrinsic utilities are not subject to a PATH search during command search and execution. The utilities in Table 1-5 shall be intrinsic utilities. Table 1-5 Intrinsic Utilities alias bg cd command fc fg getopts hash jobs kill read [XSI]type[/XSI] [XSI]ulimit[/XSI] umask unalias wait Whether any additional utility is considered an intrinsic utility is implementation-defined. Because applications are unable to override an intrinsic utility with a utility from PATH, implementations should not make any utility an intrinsic utility beyond the utilities in Table 1-5. On page 2339 lines 74388-74389 (XCU 2.9.1.1 bullet 1.c.), change: If the command name matches the name of a utility listed in the following table, that utility shall be invoked. alias bg cd command false fc fg getopts jobs kill newgrp pwd read true umask unalias wait to: If the command name matches the name of an intrinsic utility (see Section 1.17 on page 2318), that utility shall be invoked. On page 2340 lines 74397-74398 (XCU 2.9.1.1 bullet 1.d.i.a.), change: If the system has implemented the utility as a regular built-in or as a shell function, it shall be invoked at this point in the path search. to: If the system has implemented the utility as a built-in or as a shell function, and the built-in or function is associated with the directory that was most recently tested during the successful PATH search, that built-in or function shall be invoked. Add the following paragraphs to the end of the rationale for the PATH environment variable in XRAT subclause A.8.3 after P3499, L118222: The standard specifies that (when no character is included in a command pathname) special built-in utilities and intrinsic utilities are not subject to a search using PATH. All other standard utilities, even if implemented as shell built-ins, are required to be found by searching PATH. This means that if a shell includes a built-in for a standard utility that is not intrinsic, a user can write a utility that will override that built-in. The standard also requires that all standard utilities can be executed by commands like: find . -type d -exec printf 'Found directory: %s\n' '{}' + So, other than differences caused by using different shell execution environments, a standard utility that is implemented as a built-in and the non-built-in version of that standard utility are both required to behave as the standard specifies. But, if a non-standard utility is found in PATH before the standard utility's location in PATH, the non-standard utility must be invoked rather than the built-in. For instance, if the shell includes a built-in printf utility (which most shells do), PATH is initialized using: PATH="$HOME/bin:$(command -p getconf PATH)" and $HOME/bin/printf is an executable file containing: command -p printf 'In %s with args:\n' "${0##*/}" >&2 command -p printf '\t%s\n' "$@" >&2 command -V printf >&2 command -Vp printf >&2 command -p printf "$@" then the command: printf '%s %s\n' HOME "$HOME" PATH "$PATH" should produce output similar to: In printf with args: %s %s\n HOME /Users/dwc PATH /Users/dwc/bin:/usr/bin:/bin:/usr/sbin:/sbin printf is a tracked alias for /Users/dwc/bin/printf printf is a shell builtin HOME /Users/dwc PATH /Users/dwc/bin:/usr/bin:/bin:/usr/sbin:/sbin The current version of the Korn shell installs built-ins into the shell using a builtin utility that allows the built-in to be associated with the pathname of the non-built-in version of that utility. (Unfortunately, some implementations that use ksh93 as their standard sh utility do not make use of this feature and install built-ins for standard utilities that are not associated with a PATH search. And, most other shells incorrectly always use a built-in utility if one is installed, even when it should be overridden by a PATH search that should find the non-standard version of a utility with the name of that built-in.) Some other shells use a character in a directory pathname in PATH to indicate one or more directories that should be used when processing PATH to determine when non-intrinsic standard utilities should be found. The POSIX.1-201x revision of the standard allows either of these methods to be used to install built-ins that meet the requirements stated in XCU subclause 2.9.1.1 by making the behavior of the built-in path search implementation-defined when a character is found in PATH. On page 3674 lines 125546-125552 (XRAT C.1.7 rationale for XCU 1.6 Built-In Utilities), change: All of these utilities can be exec-ed. There is no requirement that these utilities are actually built into the shell itself, but many shells need the capability to do so because XCU Section 2.9.1.1 (on page 2339) requires that they be found prior to the PATH search. The shell could satisfy its requirements by keeping a list of the names and directly accessing the file-system versions regardless of PATH. Providing all of the required functionality for those such as cd or read would be more difficult. There were originally three justifications for allowing the omission of exec-able versions: to: Other than the special built-in utilities, there is no requirement to build utilities into the shell itself. However, many shells implement certain utilities as regular built-ins for the following reasons: To improve performance, especially for frequently used lightweight utilities (such as test, true, and false). To eliminate the need for some sort of interprocess communication between the shell and those utilities that read or modifiy the shell's execution environment (such as cd). To make it easier to satisfy the command search and execution requirements in XCU Section 2.9.1.1 (on page 2339) for intrinsic utilities. Intrinsic utilities must be found prior to the PATH search. The shell could satisfy this requirement by keeping a list of the intrinsic utility pathnames and directly accessing the file-system versions regardless of PATH, but these utilities usually need to read or modify the shell's execution environment anyway. With the exception of the intrinsic utilities, all regular built-in utilities are subject to the PATH search and can be overridden by a specially crafted PATH environment variable. All of the regular built-in utilities can be exec-ed. There were originally three justifications for allowing the omission of exec-able versions: On page 3675 between lines 125572 and 125573, insert a new section heading: C.1.8 Intrinsic Utilities On page 3675 line 125573, change: There were varying reasons for including utilities in the table of built-ins: to: There were varying reasons for including utilities in the table of intrinsic utilities: On page 3675 line 125580, change: cd, getopts, newgrp, read, umask, wait to: cd, getopts, hash, read, [XSI]type, ulimit,[/XSI] umask, wait Remove page 3675 lines 125599-125602 (true, false). On page 3675 after line 125602 insert a new paragraph: The following utilities are frequently implemented as intrinsic (and built-in) utilities. Future versions of this standard might not allow these utilities, or any other standard utility not in Table 1-5 in XCU 1.7 on page 2318, to be intrinsic; implementations are encouraged to implement these as non-intrinsic utilities instead (but still built-in if they were previously built-in). [ echo false newgrp printf pwd test true On page 3694 lines 126360-126366 (rationale for XCU 2.9.1.1), change: The sequence selected for the Shell and Utilities volume of POSIX.1-2008 acknowledges that special built-ins cannot be overridden, but gives the programmer full control over which versions of other utilities are executed. It provides a means of suppressing function lookup (via the command utility) for the user's own functions and ensures that any regular built-ins or functions provided by the implementation are under the control of the path search. The mechanisms for associating built-ins or functions with executable files in the path are not specified by the Shell and Utilities volume of POSIX.1-2008, [...] to: The sequence selected for the Shell and Utilities volume of POSIX.1-2008 acknowledges that special built-ins cannot be overridden, but gives the programmer full control over which versions of other utilities are executed (with some exceptions). It provides a means of suppressing function lookup (via the command utility) for the user's own functions and, with the exception of the intrinsic utilities (see XCU Section 1.7), ensures that any regular built-ins or functions provided by the implementation are under the control of the path search. The mechanisms for associating non-intrinsic built-ins or functions with executable files in the path are not specified by the Shell and Utilities volume of POSIX.1-2008, [...] Bug #879: strptime is missing conversion specifiers described in strftime OPEN http://austingroupbugs.net/view.php?id=879 This was discussed during the call and notes made in the etherpad. http://posix.rhansen.org:9001/p/2014-12-18 We will continue this item on the next call. Next Steps ---------- The next call is on January 8, 2014 (a Thursday) The timetable for upcoming calls is Dec 18, and then Jan 8 2015 Calls are anchored on US time. (8am Pacific) This call will be for the regular 90 minutes. http://austingroupbugs.net An IRC channel will be available for the meeting irc://irc.freenode.net/austingroupbugs An etherpad is usually up for the meeting, with a URL using the date format as below: http://posix@posix.rhansen.org:9001/p/201x-mm-dd password=2115756#