Resent-From: icsc-nativewg@opengroup.org From: Matthew Pearson Date: June 25, 2004 3:52:51 PM EDT Resent-To: icsc-nativewg@opengroup.org To: icsc-nativewg@opengroup.org Subject: minutes 6/23 DRAFT 6/23 ICSC ITWG Meeting Minutes Taking minutes: Sun - mp Attendees: y jh Jim Hamrick, HP - ak Arkady Kanevsky, NetApp [ partial attendance ] y mp Matt Pearson, Sun y jr Jay Rosser, HP - rs Rajeev Sivaram, IBM [ missed first 20 minutes ] y dt Dick Treumann, IBM y fw Fred Worley, HP y hx Hanhong Xue, IBM • Agenda bashing, approve minutes ◦ Minutes to approve: ▪ Email from Dick Treumann, subject "ITWG Minutes 6/09/2004 (draft)", sent 6/15/04 3:02PM PT so approved • Action item review ◦ MP assemble and maintain list of high-level requirement additions. mp i have a list, consists of jr's two. not sure if ibm has any, will do a quick check on the reflector. ◦ JR send proposed high-level requirement to reflector regarding ORD changes on connected EP. ▪ See email from Jay Rosser, subject "Action Item - New IT API Phase 2 requirement - ORD modification on connected EP", sent 6/17/04 6:45PM PT jr Here is the new proposed requirement: DTO-7.0 Support for modifying ORD (IT_EP_PARAM_MAX_ORD) on an EP in the IT_EP_STATE_CONNECTED state shall be provided. jr would like to vote on this and add it into the specification to support iSER. objection to voting now? mp would like to talk about ib state transitions first. ◦ JR write new high level requirement from "Proposal - add dynamic RDMA Read/write enable change requirement", sent 5/20/04 10:49AM PT ▪ See email from Jay Rosser, subject "Action Item - New IT API Phase 2 requirement - RDMA Read/Write enable change", sent 6/17/04 6:45PM PT jr Here are the new proposed requirements: DTO-8.0 The API shall identify whether or not the IA supports modifying RDMA Read enable (IT_EP_PARAM_RDMA_RD_ENABLE) and RDMA Write enable (IT_EP_PARAM_RDMA_WR_ENABLE) on an EP. DTO-8.1 The API shall reject attempts to modify RDMA Read/Write enables on an EP on an IA that has no such support. ◦ JR to investigate whether IB state transition from RTS -> SQD -> RTS to allow this change to see if there are any side effects a consumer could see. ▪ See email thread started by Jay Rosser, subject "Action Item - RTS->SQD->RTS side effects", sent 6/17/04 6:45PM jr did some investigation on whether the required state transitions would be user visible. fundamentally it's an issue as to whether moving from sqd to rts state would appear - there's an errata here, ORD can only be changed from sqd to sqd state and can only be changed when the SQ is empty. there are three visible things. async event from draining to drained could be consumed by the implementation, so this could be hidden. second, send queue is no longer processed during this - but there is an immediate error already for this, because it happens elsewhere. third issue is this could take time. mp brought up an issue here - it could take quite a while for the sq to drain and ramifications are that the potential for this to happen should be identified - users should really drain their sq before trying to change ord if they can, else they could be held off for some time. mp good summary jr don't think this is a show stopper mp agreed, just need to inform users that this is not a lightweight operation. jr 7.0 required for iSER support. 8.0 is really required, as the current spec is broken. feel 8.0 is not controversial. ready to vote for next wednesday? [ no objections ] jr there will be a call for vote on 7.0, 8.0, 8.1 next meeting. jh live vote next wednesday? ai jr electronic ballot due by next meeting. due COB tuesday 29th of june [pacific time]. • Discuss plans for 1.0 errata ◦ Propose folding errata fixes into the initial phase 2 release ◦ Propose consuming telecon time (once all other agenda items resolved) discussing errata and fixes jr propose that we consider it part of our phase 2 release that we fold in errata in 1.0. need to bring latest errata spreadsheet to the table. ai jh to send errata spreadsheet to reflector jh process for going through the spreadsheet? suggest - 1. discuss errata as to whether it needs to be fixed. 2. assign errata to someone to go off and fix it. 3. suggested fix evaluated by group. jr prioritization? jh yes there is some severity level info in this document, currently. could go with my order or evaluate order as a group. jr think your order will do, more lightweight. jh will publish spreadsheet and can begin discussions next week. how to assign? should there be a job bucket - volunteering - or assign bugs to specific people? jr not sure if hoping for ownership is a good step. think only way is to assign. as we go through them we may find must-do's. jh could round-robin it. we're going to need to have some method of selecting a victim. jr some of them we might generate resolution text right in the telecon, so the actual work involved is editorial in nature. might not be too painful. possibility there is to dump them on the editor. jh mechanical issue - do we want to try to differentiate between feature change-bars and errata change-bars? jr&dt sounds desirable jh need to delegate someone to see if this can be done. jr change fonts? switch to italics? given we have a primitive toolset? jh color might be simplest mp or underlines, they print better... • iWARP CM detailed requirements ◦ Email thread started by Jay Rosser, subject "Draft 1 of iWARP CM detailed requirements ready for review", sent 6/10/04 9:37AM PT ◦ Email thread started by Fredy Neeser, subject "Feedback on iWARP CM detailed requirements", sent 6/23/04 8:47AM PT jr anyone have a chance to look at these aside from those who have sent comments to the reflector? jh big issue that jr and caitlin are batting about could bear discussion. jr kind of a meta-issue, that one. fundamental issue is whether to support ietf spec ability to turn markers on and off. the current state is to expose consumer control to modify those. caitlin advocates that we not make this consumer visible. leave it up to the negotiation between the devices, essentially. i am concerned that the rdmac spec says no such control of markers is allowed, and that is the lowest common denominator - seems the reasonable thing to do for compatibility is to require markers. jr rdmac mpa mandates markers and crc. the ietf version allows crc administratively controlled option and markers is implementation choice. ak in userlevel api not clear users would care about it. kernel folk might. jh are markers and crc independently enabled? jr yes. mp reason you would turn these off is for performance? jh caitlin believes that some implementations may not need markers, those that do not place out of order. jr what happens to those implementations if they have markers? jh caitlin said nothing specific, issue is complexity in discarding them. jh people seem to think we shouldn't expose these in the api level but are coming to opposite opinions on which way we should go under the covers. question is, how hard are we willing to work for what level of interoperability. interface becomes a lot less complex if you are willing to lop off some interoperability switches. do we want to simplify the api at the expense of interoperability if some other api other than ours was used? ak if you implement ietf but without the verbs layer, you don't need to include markers. that wouldn't work with ours, is that right? fw if you expose the ability to select or deselect markers then it's not a problem. the request to remove those simplifies the api - if we do this and require markers then there are/could be devices that don't work with itapi... jh flipside is if markers are always disabled then we will not work with some rdmac implementations that require them. ak if it's an option, then consumer needs a way to find out whether remote side supports markers. can't be done through our api. jh if itapi is deployed in a unified environment, then it's reasonable to assume that in a single datacenter, there's a way to find this out about the remote side. in the case of a WAN, then you've got a problem... but I'm not sure this is our target environment. fw not sure I want to restrict the api use now. can't tell what people will be doing with this in future. ak possible compromises - assume for a moment we have an api that allows a kernel consumer to set up the mode of the rnic - uses markers or not. there could be a per-ia attribute to indicate what the rnic is in, but it isn't modifiable - jh biggest problem there is deploying in a heterogeneous environment. mp can't you do this on a per connection basis? fw gets down to the question of what's a kernel app and not a kernel app, maybe privileged/nonprivileged. ak believe that mostly this would be used by people who consume the whole server. fw boils down to the issue that the server vendor and storage vendors might choose different ones, this is an issue regardless of whether you own the machine. fw caitlin's argument is that rdmac implementations that don't support mpa markers off will stop existing. and she believes that ietf devices might exist that can't work with markers. from my perspective i see a need to support rdmac-compliant devices within the industry. i see uncontrollable heterogeneity. how do we address that within the api so that application writers will want to use it in this environment? jh think the kneejerk reaction that app writers don't want to deal w/ this is probably correct. jr where this has come up is in the transport-dependent model. we have no idea yet how this will play out in the transport-independent model. jh if we're in this from the app writer perspective, we shouldn't expose this. mp could expose it in the TD model and not in the TI model? jr if we're going to punt on this in the TI mode you can either support RDMAC devices or IETF devices but not both. tough choice here. fw how bad would this be to app developers to expose this? if we do expose it then the issue of interoperability is about as addressed as we can make it. jh think it is onerous - most app vendors are thinking about it in a much higher level than this. they worry about address spaces and so on, not bits on the wire. jr although to be fair this is just a CE issue. ak could we have a default behavior? and then consumers could change it as needed. jh you mean slant the api toward the rdmac model, but make it possible for the user to control ietf-like behavior. ak still not sure of granularity required here but that is the general idea. jh think to have the api maximally interoperable need per-ep control of these. ak not sure there is such a thing as an rnic that can do that. jh ietf rnics should be able to do this. ak required to support both or can you just choose one. jh receiver is supposed to decide and can force sender to send or not send markers. jr so it needs to be able to send in both ways, not necessarily variation in receive. jh do we want to see this level of control in the TI model? fw personal feeling is that since we are mandating a protocol already (CEP) i see no problem in mandating everything. jr means that you couldn't talk to a non-marker IETF remote side. fw how many such nics do we expect? caitlin expects a lot. ak in ib we don't have anything like this. fw you can only negotiate if the remote side is IETF. can't do it if remote is RDMAC - no negotiation is possible there. so our ulp needs to pick one. jr and, an rdmac person cannot ever talk to an ietf person who doesn't support markers... jh given that, it might be necessary to expose what the local ia is capable of to explain why this is the case. jr could make no statement of what the transport is capable of and just rely on negotiation. in that case the only failure would be if an RDMAC person talks to an IETF who doesn't support markers. [ general agreement that this would work, as a rough guide, for TI case ] jh that covers the TI model, back to TD. expose the capabilities here? feeling of the group is don't expose complexity if they don't want it. jr and we're giving them a swiss army knife... jh we just want to make it so if people don't want to worry about this they don't have to. fw make a default behavior that can be changed via some interface? ak default needs to be the same across all connections [... crosstalk...] jr not just a default but the ability to change the default globally? so there should be a baseline default - and you're wondering how to change this? change it per ia or per ep? jr need to investigate whether we can change defaults on a per ep basis. ak not sure we can do this. jh do we want this default mechanism that's global written into the api - eg default to markers on all the time - or make it an implementation/installation default. ak thought default would be rdmac behavior (markers). jh it would be really annoying if the library were configured one way and the rnic underneath were configured the other way. ak consumers would have to change it every time - not much of a default. jh on the other hand we haven't gotten very far if we allow the global default to be different on each nic. jh going to guess the market is going to require rnic vendors to support markers on recv. fw believe that more likely than that rdmac-compliant vendors will suddenly become ietf compliant. ak performance implications if you don't have markers? jr markers fundamental to direct placement. jh hw folks probably think this is braindead. if we default in the direction we think things are going and we are right we've done a service. but should make sure we're functional if we're wrong. fw real possibility that over time things may shift from one to another. 5-10 years from now everything may be ietf and a markerless model may prevail. jh need to be useful now to have trouble with that later. jr am pretty sure this has been discussed by RDDP many times over. ai jr to ask caitlin for more info on this issue. jr so our plan is to default to markers (RDMAC model) but allow switching to other mode. ak if you use default and get an error you can change the value and try again, in the case of an IETF that doesn't support markers. mp isn't this something a computer could do, then? try with markers, and if it fails, you immediately try again with the exact arguments with one different. jr requires user cooperation though, ulp needs to negotiate and so on. ak receiver tells you whether it wants markers or not. so the implementation could reject that if it can't support it or take it if it can. implementation says ask me to do certain things, i can't do it, i reject automatically - don't need to bother consumer with it. jr non peer reject? ak something that says the two sides are not compatible. jr if you're doing TD programming you should negotiate outside the api how you're going to communicate, will this happen? fw how do we handle the case where the sender is RDMAC and the receiver is IETF and can't accept markers? jr you would have to stick with tcp/ip in that case. fw how does app determine this? jr think model would be, outside of api, in TD - use streaming mode to determine whether markers are OK or not, then switch up. fw couldn't we do this in streaming mode, a handshake that validates whether or not there is a compatibility story here and if so what it is, it could be done in streaming mode as part of the conversion... jr sounds like a ULP to me. fw yes, it would be a new ulp. after all, we've already got one... jr gut reaction is, this is transport dependent, and if you want to do it you take the burden upon yourself - roll your own. fw depends on what you mean by transport dependent too. fw might want a new error code to indicate, in the TI case, that the error was because the other transport absolutely can't talk to you. jr still have consensus in the TI model to default to RDMAC mode? mp yes. any other things to discuss for posterity? jr TI model is to default to RDMAC mode (markers on) and there are no capabilities to modify that behavior. Is that what we agreed on? ak proposal on table, people want more time to think about it. fw think it's reasonable, having agreed that there is a potential class of devices that simply could not use the ITAPI if we do that. note, i do not necessarily have a problem with that. could use a model like the streaming mode ulp to negotiate this on behalf of the user, for example. fw information from caitlin could tell us whether this corner case is even worth addressing. if it isn't, we are fine using RDMAC as default. mp so this new ulp is to determine whether or not you can negotiate? jr this facility already exists. if you send a req and it is rej'd, just send a req again? jr TD model is to specify a default which is RDMAC and add an option that specifies "no markers". there is the option of having a new ulp that negotiates this more cleanly. mp not sure if i like this in the TD space - isn't TD how you talk to a non-ITAPI consumer? not in favor of a ulp outside ITAPI space. jr moving on to other issues jr caitlin's suggestions for adding class of service args and other things to the iwarp conn qual seems reasonable. it's not just an ip address - could include qos and a local addr modifier. jr anything else? -- meeting adjourned around 6:00p EDT • Next steps ◦ Generate S-RQ man pages, continue review of detailed requirements as available. • Any other business