From: "Wittle, Mark" Date: 10 June 2003 11:06:46 am EDT To: icsc-nativewg@opengroup.org Subject: ICSC ITWG 6/9 part 1 draft minutes 6/9 ICSC ITWG Meeting Minutes, 2pm - 4pm (Taking minutes: Netapp - wittle) ========== Attendees ========== y HP Jim Hamrick (JH) Y HP Jay Rosser (JR) y IBM Peter Hochschild (PH) y IBM Kevin Reilly (KR) y Infiniswitch Ken Walker (KW) y NetApp Arkady Kanevksy (AK) y NetApp Mark Wittle (MW) y Sun Jim Roberts (JRb) y Sun Matt Pearson (MP) y Sun Steve Sistare (SS) ========== Action Items ========== ========== Agenda ========== 6/9 ICSC ITWG Meeting Agenda 1. Agenda bashing, get connected, approve minutes (5 minutes) -----Original Message----- From: Kevin Reilly Sent: Thursday, June 05, 2003 7:15 PM Subject: 06/02/2003 minutes 2nd half 2. defect review level 5,4 & 3. - DTO -----Original Message----- From: Jay Rosser Sent: Sunday, June 01, 2003 2:00 PM Subject: Round 2 - DTO resolutions and proposals - EP -----Original Message----- From: Jay Rosser Sent: Monday, June 02, 2003 12:50 PM Subject: Round 2 - EP resolutions and proposals - Connection -----Original Message----- From: Kevin Reilly Sent: Sunday, June 01, 2003 8:59 PM Subject: 90% of the connect defect resolution Next Meetings: Weds 6/11 Minutes: HP? EVD issues 2 hrs or 4 hrs? ========== Minutes ========== - DTO -----Original Message----- From: Jay Rosser Sent: Sunday, June 01, 2003 2:00 PM Subject: Round 2 - DTO resolutions and proposals JR: #10 - having to check flag is a tax on data path? 2 proposals (see spreadsheet). JR: can we vote? MW: let's vote. if contention, we'll do it again w/all 5 members present. if strong concensus, we'll check with InfiniSwitch offline and see if they concur. AK: need standard flags rules. either need to say which flags are ignored, or say what the default value of each will be. JR: problem is the default value on barrier fence flag. documented that if transport doesn't support rdma-read, then this flag is ignored. special case. so we can't get complete uniformity. AK: if we can't get uniformity, i'm uncomfortable. perhaps we should decide whether we want uniformity first. MW: vote - HP, IBM, Sun for #1, Ntap for #2. AK: what about barrier fence and solicted wait. JR: solicited flag only supported for send. PH: or we could remove default flag. MW: is there a tax involved in the other flags SS: atleast one flag is checked on each call. JR: so its cheap to do these checks. what about default concept? get rid of it? PH: yes. AK: consumer must put the values in? SS: hard to define default. AK: except for notify bit. AK: we should define that barrier has no meaning for UD, and define a default value. MW: Proposal - 1. implementation must check that flag has a valid value 2. we need to describe the required settings for each call. 3. we should not define a default flag value MW: vote: all 4 in favor. JR: #11 - text implies more than the hw can always deliver. propose additional text to resolve. PH: it_post_send looks good. it_post_rdma_write ... relevant text near line 70. ... PH: need cleanup from 78 - 81. should be consolidated with proposal text. AK/JR: Proposal - 1. add it_post_send change as stated, and restore text describing order of buffers. 2. add paragraph 1 and 2 of rdma_write proposal, and add buffer ordering info 3. rephrase Applicate Usage as stated in post_send. 4. cleanup lines 75 - 77; current text is too strong. MW: vote - all 4 in favor. SS: will a 2nd RDMA-write force the 1st RDMA-write to finish? my question is not about wire ordering, it is about delivery. AK: not sure if that's true for iwarp. JR: #11.1 - new related defect for rdma-write on non-coherent memory. SS: not true for non-coherent LMR. would need the followup msg receive to include the RMR/address range, so that the target can sync the address range. we should clarify the solution to #11 with this info. coherency is under consumer control. MW: propose add ptr in the post_rdma_write page to the coherency man pages ... eg, proper handling ... see ... AK: rdma_read call also need reference? SS: no. they've already created the lmr, and it includes a ptr to the sync pages. JR: #353.1 - transfer length undefined on error. propose adding text. MW: all agreed. JR: #362 - incompatible flag settings. resolved by resolution to #10. MW: all agreed. JR: #362.1 - need to resolve conflict between solicited wait and notify flag. propose adding text to clarify. MW: all agreed. SS: make sure stated as a result of the "or" rule that causes notification, not as an exception. JR: yes, fix here, and point to definitive evd page explaining the "or" rule. MW: all agreed. JR: #354 - need to clarify invalid flags error. resolved by #10. MW: all agreed. JR: #129 - clarify behavior of specifying an AH that is not associated with the spigot. propose new text. MW: move IB reference to Implementors Guide. MW: all agreed. JR: #129.1 - additional sendto error behavior note. propose adding new text. MW: all agreed. JR: #355, #356, #357, #358 - description seems to relate too much to C language. 2 proposals: ignore, change text. SS: prefer ignore, at this point, fewer changes are better. MW: #1: HP, IBM, Sun #2: Ntap JR: #152 - clarify completion behavior proposed new text. MW: all agreed. JR: #153 - clarify for dto size error handling proposed new text. MW: all agreed. JR: #154, #155 - clarify ordering of send/recv wrt rdma ops proposed new text. MW: all agreed. - EP -----Original Message----- From: Jay Rosser Sent: Monday, June 02, 2003 12:50 PM Subject: Round 2 - EP resolutions and proposals JR: #6 - need signaling suggestion for implementors. propose adding text to implementor's guide for ep_create. MW: all agreed. JR: #6.1 - EVD/threshold issue - postpone for now. [Kevin Reilly, IBM, joins] JR: #13 - need to define IT_EP_NO_FLAG. propose add it. MW: all agreed. JR: #313.1 - RC EP attributes should be modifyable in unconnected state. AK: propose specifying that segments, sizes, and number of recv_dtos are modifyable in unconnected state. JH: would have to be implemented under the covers. issue in some cases. could be done after a reset [remote side would get involved? - mw] AK/JH: just for RC. AK: for RC, don't allocate until ready to move out of this state. JH: but, if you reset and then want to change some of these could fail. AK: create first, then allocate PH: need a new error code. create might work, and then allocate might fail. JR: why would you want to do this? a lot of implementation. why not just destroy and recreate? MP: very narrow case ... hard to document. AK: return resource error specific to each attribute SS: confusing to consumer ... if QP allocation failed, but we have to return an error on a different resource. AK: trying to cover the case of a pool allocated, and then modified for each connection. JRb: consumer would have to destroy and recreate JH: might be necessary. what if you have unreaped cookies? after reset, QP ... AK: described in reset. can't count on getting cookies after reset. JH: but this is completions for a difference QP. if the QP has been reallocated, tough for implementation to manage. AK: DAT provides this. reset destroys the resources. JH: hard to do right. AK: reference implementation relies on HW to destroy resources. MW: higher level issue. this is different than most other api's. there is a customer for this resource allocation model, so question is where to implement it. JH: if you can't give the consumer exactly what they want, why go to a lot of trouble to give a partial solution. issue requires going beyond iwarp and ib published interfaces. SS: i'm also sympathetic. but i don't see a way to make this work in all cases. would require global mutex between ep_create and ep_modify. AK: should we discuss after the thread discussion ... could be supported. SS: thread safety doesn't help. this would have to be done across all consumers. calls for a larger resource management solution. MW: workaround is that when the passive consumer receives a connection request, to destroy ep and recreate new ep with proper attributes. AK: yes, but connection request has timeout. MW: vote on proposal (both parts) - against: HP, IBM, Sun - for: Netapp JR: #29 - clarify payload resource error. for RC only. propose new text. MW: all agreed. JR: #30 - clarify payload resource error. for UD only. propse new text. JH: if transport can handle upto 2048, you ask for 1, but send 5, is that an error? SS: that's a new defect. AK: didn't we agree that the implementation can allocate more than the consumer requested. does that apply here? SS: #30 addresses the case where the consumer has exceeded the HW max. MW: all agreed. JR: new case is more than they asked for, but less than HW limit. UD man page says that the implementation can allocate more than the consumer has asked for. JH: for rdma_reads, the consumer must have control over the number. JR: we don't specify which attributes the implementation is allowed to overallocate. he has to check [using query? - mw] AK: CM has to match attributes on the two sides. JH: consumer must have control over this ... downsize if necessary. AK: solution is the require the implementation to set Payload and RDMA-credits (IRD and ORD). JH: only way to do that on IB is to force to the IB size. AK: enforced by the create and/or modify call. JH: forces consumer to do transport-specific programming. JH: real constraint is can't establish connection unless MTU number is same on passive and active sides. the question is whether the transport must enforce this for data transfer, or just for connection setup. if connection time checking only, then we would get TI behavior. [Ken Walker, InfiniSwitch joins] AK: do we require implementation to check, or not? JH: error_local_length ... refers to spigot not ep. this is correct. SS: this is new defect... let's postpone this one, get it written up, etc. JR: #46 - need to clarify semantics of EVD when freeing and EP. propose additional text. PH: what does 'inability to retreive' mean? AK: the events associated with the EP might not be on the EVD. SS: should say events might be lost. AK: reuse of EP handle? a new defect? we might need a general solution, not just EP. [switch to part 2 of minutes, from Arkady Kanevsky] - Connection -----Original Message----- From: Kevin Reilly Sent: Sunday, June 01, 2003 8:59 PM Subject: 90% of the connect defect resolution