10/5/04 Meeting agenda (Taking minutes: HP - JR) y HP Jim Hamrick (JH) afer 15 minutes Y HP Jay Rosser (JR) Y HP Fred Worley (FW) Y IBM Fredy Neeser (FN) N NetApp Arkady Kanevksy (AK) N Sun Matt Pearson (MP) cascading ascii art attendance diagram. (if you have more than 1 minus visible, you are not eligible to vote.) hp ibm netapp sun ---- ----- ------ --- m-3 + + - + m-2 + + + - m-1 + + - - m-0 + + - - o Agenda bashing, approve minutes o Minutes to approve: draft ICSC ITWG minutes for 9/21/04, sent by Jay Rosser o Action item review o FN - create additional text for Global Behaviors section to clarify what an asynchronous call is in the IT-API. o MM group to create text describe potential hazard of use of stale STags with fast re-registration use model. o FN - propose PBL extension to allow variable-size page lists, e.g., for IBVE o FN - determine what PBL type checking needs to be done depending on pbl_types_supported IA attribute. See, MM-4.0.D4.4 and MM-8.3.D1 o Continue discussion of MM based on v095a o Handling of Privileged and Non-Privileged QPs o Security concerns o Reasons for a Privileged IT-API Consumer to selectively enable/disable Privileged Mode on Endpoints (or their underlying QPs)? o LMR Unlink (Local Invalidation of Memory Resources) o Can an RMR or LMR be linked or unlinked through an Endpoint that is in unconnected state (it_rmr_bind and it_rmr_unbind currently say no)? o Can a QP be destroyed or reset if MWs are associated with it? In IBVE (p. 61), there are differences between Type 2A/2B MWs o Treatment of MW Bind Enable flags (see email by Fredy Neeser, 10/04/2004, "Re: Memory Management Detailed Requirements v095a for Review") o Man page generation o Feedback on draft CM man pages o Clarification on extended state machine (email from Caitlin Bestler, 09/30/2004) o Feedback on draft MM man pages o Difficulties with MM-9.5.D1: Remote protection errors should generate local completion errors o Allowable RMR types (email from Caitlin Bestler, 09/30/2004) o Next steps o Focus on man page generation, occupy spare time in telecons with errata review, next round of detailed requirements to be prioritized. o Any other business o Respond to RNIC-PI WG questions (see RNIC-PI Minutes from 09/30/2004) * Agenda bashing, approve minutes o 9/21/04 minutes approved o FN - does have comment on minutes, though. Removal of physical memory is not handled by the API - O/S has no way to revoke registration of a region. o FW - SysAdmin use model may be simply to remove memory abruptly. Impacts range from tearing down connections to killing application. Might be nice to have an API that could notify Consumer of loss of memory. o FN - thinks that if the API existed that revokes registrations, applications could be written to deal with this. Could be desirable to add to API in future - for instance, if a single applications is over-commiting physical memory, or too many applications are contending for physical memory, it might be desirable to add this API. o FW - mentions that mlock in POSIX has a concept of minimum amount of physical memory that needs to be conserved for O/S use. However, this is very coarse. Thinks that having an API where applications could be requested to return memory to the O/S could add a graceful model into the API. o FN - thinks this should be proposed as a possible new requirement for phase 3 + AI - JR to find the text required to identify new requirements (as proposed by Dick) and send to FN/FW so they can issue new proposed requirement. * AI review o Global behavior text for asynchronous behavior (Closed) + FN - describes the approach + JH - okay with FN's text + FN - asks about it_get_pathinfo as an example - notes that this man page does NOT identify that it can block (!!). + AI - Everyone - look at IT API 1.0 man pages, etc., to look for and identify calls that may block on a remotely generated event (e.g. it_get_pathinfo). + FN - any disagreement that identifying local blocking behavior (on acquiring resources, etc) is unnecessary since it will typically be for a "short" length of time. # JH - agrees that we should not need to identify all of these issues # JH - agrees that remote interaction causing blocking should be identified # All - agreed + AI - FN - request Global Behavior section from editor and update as above. o Stale STags (Open) + Still pending o PBL extension for variable sized page lists (Closed) + See email from FN subject "AI - Variable-size page lists", sent 10/5/04 9:52AM PT + FN - describes proposal + JR - asks about the IB VE attributes - how would they be exposed in the IT API? + FN - the set of PBL types will be exposed via the it_ia_info_t # This handles the case for BMM + FN - describes requirement on implementation to ensure pbl-type is supported. Remainder of checks done by hardware. # FN - notes that exposing of all errors is still a TBD + FN - since IB has a completion error defined for passing in a pbl-type that is not supported, such should appear in IT API. # FN - AI on MM group to think about corresponding errors. (FN - notes that MM already does actually have an AI for this). o PBL type checking (Open) + Pending (same as previous noted AI) * MM discussion based on v095a o Privileged QP discussion + FN - current assumption is when a privileged Consumer creates a QP, the underlying QP is automatically privileged. # FN - is there a need to allow the privileged Consumer ability to choose to create a non-privileged QP? # JR - postulates that the use model of the privileged QP may be restricted by the Consumer so that no hazard exists * JR - the STag of 0 equivalent is a special case that Consumer can simply avoid (so can be ignored) * JR - the other issue is fast register - is there a hazard here? * FN - yes - say SDP library does privileged registration (i.e. fast register) on behalf of user-space consumer (for, say private buffers). o FN - also say there is a protection zone shared between user-space apps and a privileged consumer entity o JR/FW - think that this is an implementation defect potentionally. + FN - summary - we believe it is sufficient simply to state that privileged Consumers can only create privileged QPs by default - they have no mechanism to create non-privileged QPs. # FW - we will welcome use-cases that can find problems with the above o LMR Unlink + FN - a.k.a "local invalidation". + FN - this is section MM-10 (page 19 of v095a) + FN - MM-10.0.1.D0 allows non-privileged consumers to unlink LMRS # JR - what is the impact here - Consumer will leave a MMU resource on card allocated but pointing at nothing? # FN - yes # JR - are we sure we want to let them do this? # FN - we want to do this to avoid a nasty asymmetry in the API by allowing this - we will advise unprivileged Consumer not to do this * JR - this is simply a debugging problem since the Consumer can leak all their MMU resources on card. # Everyone okay with this + FN - MM-10.0.1.D1 # FN - not sure of issue with MM-10.0.1.D1.2.1 (b/w compatibility) + FN - MM-10.0.1.D1.3 (errors) # FN - D1.3.5 "due to the way it was linked" * Assumption here is there may be IT API implementations that use the "register virtual memory region" feature of InfiniBand which passes in only a Vaddr and len (not PBL) which means that translation and pinning is implemented by verbs provider. o In this case, it may or may not be possible to unlink an LMR o This is only an issue for InfiniBand. * AI - FN to give a deeper explanation of this next time + FN - MM-10.2 (query interfaces) # FN - need additional query interfaces - need to know whether or not an LMR is linked. * FN - D1.1.2 "REMOTE ACCESS FLAG" o JH - wonders how this differs from existing REMOTE ACCESS FLAG? Is it with respect to remote read/remote write privileges? o FN - You can enable remote access on or off on creation of the LMR in RDMAC verbs. If on, you then can turn on or off remote access via modify - if off, you can never give remote access o AI - JH to determine if this flag is truly necessary or can be handled with existing flags. + FN - MM-10.3.1 and 10.3.2 # FN - completion error is defined for InfiniBand which surfaces an attempt to invalidate a memory region that was created via register virtual memoy region. # FN - this requires a query attribute (IT_LMR_PARAM_UNLINKABLE) to determine whether or not the LMR will have this attribute * FN - MP and he had discussed forbidding IT API implementations from using register virtual memory region to avoid this wart. o AI - Everyone - FN suggests we all think about this issue + FN - MM-10.4 (RDMA Read with invalidate) # FN - notes that this requires the flag "rdma_read_local_extensions". # FN - notes that the requirement to deal with unlinking narrow RMRs with RDMA Read forced the definition of a new API call "it_post_rdma_read_to_rmr". + FN - MM-10.4.D3.6.2 # FN - worried about unlink operation getting an error in the middle of invalidating # JR - concerned that constraining an RDMA read to use only a single scatter element seems problematic * AI - FN will add text stating why the restriction is necessary. o Questions on MM Man page generation + AI - FN/JR - Can an RMR or LMR be linked or unlinked via an Endpoint in the unconnected state on InfiniBand? Current IT API pages deny this - see if InfiniBand verbs allow it. + AI - MM group - determine if QP can be destroyed or reset if MWs associated with it (for IB, IB VE, and iWARP). * Man pages o CM draft pages + JR - working through JH's comments, then FNs, and CBs to come. + FN - CB had some interesting comments on flags to it_socket_convert - she advised flags be supplied as "hints" rather than directives. Thus administrative controls could over-ride the "hints". # FN - default could be redefined by the administrative setup. # JR - still thinking about this. o MM draft pages + FN - has issued draft pages including many questions # What should be done about remote protection errors (MM-9.5.D1)? # JH - states that you will get a completion that indicates success for Sends and RDMA Writes and a terminate message later. Only for RDMA Read will you see a completion error. Thus must surface an Affiliated Asynchronous Error for the Send and RDMA Writes (corresponding to the terminate message). * FN - notes that this is a difference between the transports. * JH - agrees - this is unavoidable. * JH - clarifies that RDMA Read does not exhibit this difference from IB. * Any other business o Maybe no meeting next week since Fredy is on holidays. Meeting adjourns at the 2hr 15min mark.