Minutes of the 9th September 2021 Teleconference Austin-1161 Page 1 of 1 Submitted by Andrew Josey, The Open Group. 10th September 2021 Attendees: Don Cragun, IEEE PASC OR Nick Stoughton, USENIX, ISO/IEC JTC 1/SC 22 OR Joerg Schilling Eric Blake, Red Hat, The Open Group OR Geoff Clare, The Open Group Eric Ackermann, HPI, University of Potsdam Mark Ziegast Apologies: Richard Hansen Andrew Josey, The Open Group * General news This was a call dedicated to general bugs and C17 integration. * Outstanding actions (Please note that this section has been flushed to shorten the minutes - to locate the previous set of outstanding actions, look to the minutes from 13th June 2019 and earlier) Bug 1254: "asynchronous list" description uses "command" instead of "AND-OR list" OPEN https://austingroupbugs.net/view.php?id=1254 Action: Joerg to investigate how his shell behaves. Bug 713 - Nick to raise with the C committee. This item raised in Austin/1129 and submitted to the C committee in May 2021. * Current Business Bug 1437: make: (document .NOTPARALLEL and .WAIT special targets) in RATIONAL Accepted as Marked https://austingroupbugs.net/view.php?id=1437 This item is tagged for Issue 8. On page 2975, insert before line 98708: .NOTPARALLEL The application shall ensure that this special target is specified without prerequisites or commands. When specified, make shall update one target at a time, regardless of whether the -j maxjobs option is specified. If the -j maxjobs option is specified, the option shall continue to be passed unchanged to sub-make invocations via MAKEFLAGS. On page 2975, insert before line 98742: .WAIT The application shall ensure that this special target, if specified as a target, is specified without prerequisites or commands. When .WAIT appears as a target, it shall have no effect. When .WAIT appears in a target rule as a prerequisite, it shall not itself be treated as a prerequisite; however, make shall not recursively process the prerequisites to the right of the .WAIT until the prerequisites to the left of it have been brought up-to-date. Implementations may also enforce the same ordering between the affected prerequisites while processing other target rules that have some or all of the same affected prerequisites. On page 2975, change line 98742 from: The special targets .IGNORE, .POSIX, .PRECIOUS, .SILENT, and .SUFFIXES shall be specified without commands. to: The special targets .IGNORE, .NOTPARALLEL, .POSIX, .PRECIOUS, On page 2975 lines 98744-98747, change: Targets with names consisting of a leading followed by the uppercase letters "POSIX" and then any other characters are reserved for future standardization. Targets with names consisting of a leading followed by one or more uppercase letters are reserved for implementation extensions. to: Targets and prerequisites consisting of a leading followed by the uppercase letters "POSIX" and then any other characters are reserved for future standardization. Targets and prerequisites consisting of a leading followed by one or more uppercase letters, that are not described above, are reserved for implementation extensions. On page 2985 after line 99167 (make EXAMPLES) append a new example: 7. With the following makefile, make -j 10 all may bring one and two up-to-date in parallel despite the .WAIT in the prerequisite list for foo. This is because the .WAIT does not stop make from recursively processing bar and its prerequisites in parallel. However, if only foo is specified (make -j 10 foo), make will wait for one to be brought up-to-date before bringing two up-to-date. Note also that the .WAIT does not create a prerequisite relationship between one and two, so make -j 10 two will not build one. all: foo bar foo: one .WAIT two bar: one two foo bar one two: ; @echo $@ Bug 1436: make: add "-j max_jobs" option to support simultaneous rule processing Accepted as Marked https://austingroupbugs.net/view.php?id=1436 This item is tagged for Issue 8. Updated changes after resolving 0001437 After page 2969 line 98472, make OPTIONS, add: -j maxjobs Set the maximum number of targets that can be updated concurrently. If this option is specified multiple times, the last value of maxjobs specified shall take precedence. If this option is not specified, or if maxjobs is 1, only one target shall be updated at a time (no parallelization). If the value of maxjobs is non-positive, the behavior is unspecified. When maxjobs is greater than 1, make shall create a pool of up to maxjobs - 1 tokens. (Note that implementations are not required to create a pool of exactly maxjobs - 1 tokens. For example, an implementation could limit the pool size based on the number of processors available.) If the size of the token pool would be 0, make need not implement a token pool. When all of the following are true: there is a target with commands that is not up-to-date the target's prerequisites (if any) are up-to-date make is not waiting to bring the target up-to-date (see .WAIT) make is currently bringing a different target with commands up-to-date make is not currently bringing maxjobs targets up-to-date in parallel the special target .NOTPARALLEL is not specified the token pool is not empty then make may attempt to remove one token from the pool. If a token is successfully removed, it shall attempt to bring this target up-to-date in parallel, and after this processing completes shall return the token to the pool. When make is bringing a target without commands up-to-date, it need not remove a token from the pool. If a rule invokes a sub-make either via the MAKE macro or via a command line that begins with '+', the sub-make is the same implementation as the make that invoked the sub-make, and the -j option is passed to the sub-make via the MAKEFLAGS environment variable with the same maxjobs value and is not overridden by a maxjobs value from another source (even if it has the same value), the sub-make shall use the same token pool as its invoking make rather than create a new token pool. Otherwise, it is unspecified whether the sub-make uses the same token pool as its invoking make or creates a new token pool. If a rule executes multiple sub-make processes asynchronously the behavior is unspecified. On page 2969 line 98476 change: However, lines with a ('+') prefix shall be executed. to: However, lines with a ('+') prefix and lines that expand the MAKE macro shall be executed. On page 2986 line 99178, delete the bullet item: Specifying that command lines containing the strings "${MAKE}" and "$(MAKE)" are executed when the −n option is specified (GNU and System V). On page 2987, delete lines 99229 - 99235 (the paragraph beginning "Early proposals ..."). On page 2990, after line 99387 add the following new paragraphs: The standard specifies a way for portable applications to request parallel updating of targets with commands by using the -j maxjobs option. This feature is described in terms of a token pool initially containing up to maxjobs - 1 tokens. Note that this is not intended to prescribe a particular implementation design; the usual "as if" rule applies. Implementations are permitted to silently limit the pool size for a few reasons, including: Implementations that do not support parallelism can support the -j option by simply ignoring the option (other than passing it to sub-make invocations via the MAKEFLAGS environment variable). In effect, such an implementation silently restricts the size of the token pool to zero (and therefore need not create a token pool). Some historical implementations dynamically limit the token pool size based on current system load to avoid overloading the system. Implementations may want to limit the token pool size based on the number of processors available. Implementations may want to limit the token pool size based on resource limits. Limiting the pool size does not change the value of maxjobs that is passed to sub-make invocations via the MAKEFLAGS environment variable. When a different maxjobs value is passed to a sub-make, some historical make implementations created a separate pool of tokens while other historical make implementations continued to obtain tokens from the invoking make but limited the number of tokens held at a time to the new value of maxjobs - 1. Both behaviors are believed to have merit in different situations: The former gives a sub-make complete control the amount of parallelism while the latter allows the user to control overall system load. The standard permits either behavior. The standard calls for a token pool of size maxjobs - 1, and for removal from that pool only for the second and subsequent tasks in a set of parallel tasks. This design was chosen because this is effectively what existing implementations do, and also because the token consumed by a parallel task that invokes a sub-make is effectively lent to the sub-make. Lending the token to the sub-make has the following advantages: It prevents the sub-make from being completely idle due to token starvation, allowing it to always make some progress regardless of how many tokens other sub-make invocations have consumed. It prevents token pool exhaustion caused by a long chain of sub-make invocations. If the token consumed by the invoking rule was not effectively lent to the sub-make, then the pool would be exhausted by a chain of sub-make invocations that is maxjobs long. Such a chain would never accomplish any work, and would thus never complete. When a rule invokes multiple sub-make processes asynchronously (for example by using an asynchronous list in the shell), some implementations allow each sub-make to execute at least one rule even though this would cause the total number of parallel rule executions across all make instances to exceed maxjobs (after discounting the rules that execute sub-make processes). This behavior may not be ideal, but it is easier to implement and is unlikely to cause problems in practice because applications typically do not have any rules that invoke multiple sub-make processes asynchronously. For this reason the behavior is unspecified if a rule executes multiple sub-make processes asynchronously. When multiple sub-make processes are running in parallel there is no requirement placed on the ordering of output from these processes. Some implementations of make attempt to serialize output from each sub-; others make no such attempt. If diagnostic messages from failed commands are intermixed, the usual way to deal with this is to repeat the make without -j (or with -j 1) so that intermixing will not occur. Bug 1302: Alignment with C17 https://austingroupbugs.net/view.php?id=1302 https://austingroupbugs.net/file_download.php?file_id=55&type=bug We started looking at the C17 alignment changes (bug 1302 attachment C17_alignment_20210813.pdf) and have made notes in the etherpad of proposed changes to be made to that document. To be continued next time. Next Steps ---------- The next calls are on: Thu 2021-09-13 (gettext) Thu 2021-09-16 (general bugs & C17 integration) Calls are anchored on US time. (8am Pacific) Please check the calendar invites for dial in details. Apologies in advance: Geoff Clare 2021-09-16, 2021-09-20, 2021-09-23 Bugs are at: https://austingroupbugs.net An etherpad is usually up for the meeting, with a URL using the date format as below: https://posix.rhansen.org/p/20xx-mm-dd (For write access this uses The Open Group single sign on, for those individuals with gitlab.opengroup.org accounts. Please contact Andrew if you need to be setup)