1SAS Layer
2---------
3
4The SAS Layer is a management infrastructure which manages
5SAS LLDDs.  It sits between SCSI Core and SAS LLDDs.  The
6layout is as follows: while SCSI Core is concerned with
7SAM/SPC issues, and a SAS LLDD+sequencer is concerned with
8phy/OOB/link management, the SAS layer is concerned with:
9
10      * SAS Phy/Port/HA event management (LLDD generates,
11        SAS Layer processes),
12      * SAS Port management (creation/destruction),
13      * SAS Domain discovery and revalidation,
14      * SAS Domain device management,
15      * SCSI Host registration/unregistration,
16      * Device registration with SCSI Core (SAS) or libata
17        (SATA), and
18      * Expander management and exporting expander control
19        to user space.
20
21A SAS LLDD is a PCI device driver.  It is concerned with
22phy/OOB management, and vendor specific tasks and generates
23events to the SAS layer.
24
25The SAS Layer does most SAS tasks as outlined in the SAS 1.1
26spec.
27
28The sas_ha_struct describes the SAS LLDD to the SAS layer.
29Most of it is used by the SAS Layer but a few fields need to
30be initialized by the LLDDs.
31
32After initializing your hardware, from the probe() function
33you call sas_register_ha(). It will register your LLDD with
34the SCSI subsystem, creating a SCSI host and it will
35register your SAS driver with the sysfs SAS tree it creates.
36It will then return.  Then you enable your phys to actually
37start OOB (at which point your driver will start calling the
38notify_* event callbacks).
39
40Structure descriptions:
41
42struct sas_phy --------------------
43Normally this is statically embedded to your driver's
44phy structure:
45	struct my_phy {
46	       blah;
47	       struct sas_phy sas_phy;
48	       bleh;
49	};
50And then all the phys are an array of my_phy in your HA
51struct (shown below).
52
53Then as you go along and initialize your phys you also
54initialize the sas_phy struct, along with your own
55phy structure.
56
57In general, the phys are managed by the LLDD and the ports
58are managed by the SAS layer.  So the phys are initialized
59and updated by the LLDD and the ports are initialized and
60updated by the SAS layer.
61
62There is a scheme where the LLDD can RW certain fields,
63and the SAS layer can only read such ones, and vice versa.
64The idea is to avoid unnecessary locking.
65
66enabled -- must be set (0/1)
67id -- must be set [0,MAX_PHYS)
68class, proto, type, role, oob_mode, linkrate -- must be set
69oob_mode --  you set this when OOB has finished and then notify
70the SAS Layer.
71
72sas_addr -- this normally points to an array holding the sas
73address of the phy, possibly somewhere in your my_phy
74struct.
75
76attached_sas_addr -- set this when you (LLDD) receive an
77IDENTIFY frame or a FIS frame, _before_ notifying the SAS
78layer.  The idea is that sometimes the LLDD may want to fake
79or provide a different SAS address on that phy/port and this
80allows it to do this.  At best you should copy the sas
81address from the IDENTIFY frame or maybe generate a SAS
82address for SATA directly attached devices.  The Discover
83process may later change this.
84
85frame_rcvd -- this is where you copy the IDENTIFY/FIS frame
86when you get it; you lock, copy, set frame_rcvd_size and
87unlock the lock, and then call the event.  It is a pointer
88since there's no way to know your hw frame size _exactly_,
89so you define the actual array in your phy struct and let
90this pointer point to it.  You copy the frame from your
91DMAable memory to that area holding the lock.
92
93sas_prim -- this is where primitives go when they're
94received.  See sas.h. Grab the lock, set the primitive,
95release the lock, notify.
96
97port -- this points to the sas_port if the phy belongs
98to a port -- the LLDD only reads this. It points to the
99sas_port this phy is part of.  Set by the SAS Layer.
100
101ha -- may be set; the SAS layer sets it anyway.
102
103lldd_phy -- you should set this to point to your phy so you
104can find your way around faster when the SAS layer calls one
105of your callbacks and passes you a phy.  If the sas_phy is
106embedded you can also use container_of -- whatever you
107prefer.
108
109
110struct sas_port --------------------
111The LLDD doesn't set any fields of this struct -- it only
112reads them.  They should be self explanatory.
113
114phy_mask is 32 bit, this should be enough for now, as I
115haven't heard of a HA having more than 8 phys.
116
117lldd_port -- I haven't found use for that -- maybe other
118LLDD who wish to have internal port representation can make
119use of this.
120
121
122struct sas_ha_struct --------------------
123It normally is statically declared in your own LLDD
124structure describing your adapter:
125struct my_sas_ha {
126       blah;
127       struct sas_ha_struct sas_ha;
128       struct my_phy phys[MAX_PHYS];
129       struct sas_port sas_ports[MAX_PHYS]; /* (1) */
130       bleh;
131};
132
133(1) If your LLDD doesn't have its own port representation.
134
135What needs to be initialized (sample function given below).
136
137pcidev
138sas_addr -- since the SAS layer doesn't want to mess with
139	 memory allocation, etc, this points to statically
140	 allocated array somewhere (say in your host adapter
141	 structure) and holds the SAS address of the host
142	 adapter as given by you or the manufacturer, etc.
143sas_port
144sas_phy -- an array of pointers to structures. (see
145	note above on sas_addr).
146	These must be set.  See more notes below.
147num_phys -- the number of phys present in the sas_phy array,
148	 and the number of ports present in the sas_port
149	 array.  There can be a maximum num_phys ports (one per
150	 port) so we drop the num_ports, and only use
151	 num_phys.
152
153The event interface:
154
155	/* LLDD calls these to notify the class of an event. */
156	void (*notify_ha_event)(struct sas_ha_struct *, enum ha_event);
157	void (*notify_port_event)(struct sas_phy *, enum port_event);
158	void (*notify_phy_event)(struct sas_phy *, enum phy_event);
159
160When sas_register_ha() returns, those are set and can be
161called by the LLDD to notify the SAS layer of such events
162the SAS layer.
163
164The port notification:
165
166	/* The class calls these to notify the LLDD of an event. */
167	void (*lldd_port_formed)(struct sas_phy *);
168	void (*lldd_port_deformed)(struct sas_phy *);
169
170If the LLDD wants notification when a port has been formed
171or deformed it sets those to a function satisfying the type.
172
173A SAS LLDD should also implement at least one of the Task
174Management Functions (TMFs) described in SAM:
175
176	/* Task Management Functions. Must be called from process context. */
177	int (*lldd_abort_task)(struct sas_task *);
178	int (*lldd_abort_task_set)(struct domain_device *, u8 *lun);
179	int (*lldd_clear_aca)(struct domain_device *, u8 *lun);
180	int (*lldd_clear_task_set)(struct domain_device *, u8 *lun);
181	int (*lldd_I_T_nexus_reset)(struct domain_device *);
182	int (*lldd_lu_reset)(struct domain_device *, u8 *lun);
183	int (*lldd_query_task)(struct sas_task *);
184
185For more information please read SAM from T10.org.
186
187Port and Adapter management:
188
189	/* Port and Adapter management */
190	int (*lldd_clear_nexus_port)(struct sas_port *);
191	int (*lldd_clear_nexus_ha)(struct sas_ha_struct *);
192
193A SAS LLDD should implement at least one of those.
194
195Phy management:
196
197	/* Phy management */
198	int (*lldd_control_phy)(struct sas_phy *, enum phy_func);
199
200lldd_ha -- set this to point to your HA struct. You can also
201use container_of if you embedded it as shown above.
202
203A sample initialization and registration function
204can look like this (called last thing from probe())
205*but* before you enable the phys to do OOB:
206
207static int register_sas_ha(struct my_sas_ha *my_ha)
208{
209	int i;
210	static struct sas_phy   *sas_phys[MAX_PHYS];
211	static struct sas_port  *sas_ports[MAX_PHYS];
212
213	my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0];
214
215	for (i = 0; i < MAX_PHYS; i++) {
216		sas_phys[i] = &my_ha->phys[i].sas_phy;
217		sas_ports[i] = &my_ha->sas_ports[i];
218	}
219
220	my_ha->sas_ha.sas_phy  = sas_phys;
221	my_ha->sas_ha.sas_port = sas_ports;
222	my_ha->sas_ha.num_phys = MAX_PHYS;
223
224	my_ha->sas_ha.lldd_port_formed = my_port_formed;
225
226	my_ha->sas_ha.lldd_dev_found = my_dev_found;
227	my_ha->sas_ha.lldd_dev_gone = my_dev_gone;
228
229	my_ha->sas_ha.lldd_execute_task = my_execute_task;
230
231	my_ha->sas_ha.lldd_abort_task     = my_abort_task;
232	my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set;
233	my_ha->sas_ha.lldd_clear_aca      = my_clear_aca;
234	my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set;
235	my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2)
236	my_ha->sas_ha.lldd_lu_reset       = my_lu_reset;
237	my_ha->sas_ha.lldd_query_task     = my_query_task;
238
239	my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port;
240	my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha;
241
242	my_ha->sas_ha.lldd_control_phy = my_control_phy;
243
244	return sas_register_ha(&my_ha->sas_ha);
245}
246
247(2) SAS 1.1 does not define I_T Nexus Reset TMF.
248
249Events
250------
251
252Events are _the only way_ a SAS LLDD notifies the SAS layer
253of anything.  There is no other method or way a LLDD to tell
254the SAS layer of anything happening internally or in the SAS
255domain.
256
257Phy events:
258	PHYE_LOSS_OF_SIGNAL, (C)
259	PHYE_OOB_DONE,
260	PHYE_OOB_ERROR,      (C)
261	PHYE_SPINUP_HOLD.
262
263Port events, passed on a _phy_:
264	PORTE_BYTES_DMAED,      (M)
265	PORTE_BROADCAST_RCVD,   (E)
266	PORTE_LINK_RESET_ERR,   (C)
267	PORTE_TIMER_EVENT,      (C)
268	PORTE_HARD_RESET.
269
270Host Adapter event:
271	HAE_RESET
272
273A SAS LLDD should be able to generate
274	- at least one event from group C (choice),
275	- events marked M (mandatory) are mandatory (only one),
276	- events marked E (expander) if it wants the SAS layer
277	  to handle domain revalidation (only one such).
278	- Unmarked events are optional.
279
280Meaning:
281
282HAE_RESET -- when your HA got internal error and was reset.
283
284PORTE_BYTES_DMAED -- on receiving an IDENTIFY/FIS frame
285PORTE_BROADCAST_RCVD -- on receiving a primitive
286PORTE_LINK_RESET_ERR -- timer expired, loss of signal, loss
287of DWS, etc. (*)
288PORTE_TIMER_EVENT -- DWS reset timeout timer expired (*)
289PORTE_HARD_RESET -- Hard Reset primitive received.
290
291PHYE_LOSS_OF_SIGNAL -- the device is gone (*)
292PHYE_OOB_DONE -- OOB went fine and oob_mode is valid
293PHYE_OOB_ERROR -- Error while doing OOB, the device probably
294got disconnected. (*)
295PHYE_SPINUP_HOLD -- SATA is present, COMWAKE not sent.
296
297(*) should set/clear the appropriate fields in the phy,
298    or alternatively call the inlined sas_phy_disconnected()
299    which is just a helper, from their tasklet.
300
301The Execute Command SCSI RPC:
302
303	int (*lldd_execute_task)(struct sas_task *, gfp_t gfp_flags);
304
305Used to queue a task to the SAS LLDD.  @task is the task to be executed.
306@gfp_mask is the gfp_mask defining the context of the caller.
307
308This function should implement the Execute Command SCSI RPC,
309
310That is, when lldd_execute_task() is called, the command
311go out on the transport *immediately*.  There is *no*
312queuing of any sort and at any level in a SAS LLDD.
313
314Returns: -SAS_QUEUE_FULL, -ENOMEM, nothing was queued;
315	 0, the task(s) were queued.
316
317struct sas_task {
318	dev -- the device this task is destined to
319	task_proto -- _one_ of enum sas_proto
320	scatter -- pointer to scatter gather list array
321	num_scatter -- number of elements in scatter
322	total_xfer_len -- total number of bytes expected to be transferred
323	data_dir -- PCI_DMA_...
324	task_done -- callback when the task has finished execution
325};
326
327DISCOVERY
328---------
329
330The sysfs tree has the following purposes:
331    a) It shows you the physical layout of the SAS domain at
332       the current time, i.e. how the domain looks in the
333       physical world right now.
334    b) Shows some device parameters _at_discovery_time_.
335
336This is a link to the tree(1) program, very useful in
337viewing the SAS domain:
338ftp://mama.indstate.edu/linux/tree/
339I expect user space applications to actually create a
340graphical interface of this.
341
342That is, the sysfs domain tree doesn't show or keep state if
343you e.g., change the meaning of the READY LED MEANING
344setting, but it does show you the current connection status
345of the domain device.
346
347Keeping internal device state changes is responsibility of
348upper layers (Command set drivers) and user space.
349
350When a device or devices are unplugged from the domain, this
351is reflected in the sysfs tree immediately, and the device(s)
352removed from the system.
353
354The structure domain_device describes any device in the SAS
355domain.  It is completely managed by the SAS layer.  A task
356points to a domain device, this is how the SAS LLDD knows
357where to send the task(s) to.  A SAS LLDD only reads the
358contents of the domain_device structure, but it never creates
359or destroys one.
360
361Expander management from User Space
362-----------------------------------
363
364In each expander directory in sysfs, there is a file called
365"smp_portal".  It is a binary sysfs attribute file, which
366implements an SMP portal (Note: this is *NOT* an SMP port),
367to which user space applications can send SMP requests and
368receive SMP responses.
369
370Functionality is deceptively simple:
371
3721. Build the SMP frame you want to send. The format and layout
373   is described in the SAS spec.  Leave the CRC field equal 0.
374open(2)
3752. Open the expander's SMP portal sysfs file in RW mode.
376write(2)
3773. Write the frame you built in 1.
378read(2)
3794. Read the amount of data you expect to receive for the frame you built.
380   If you receive different amount of data you expected to receive,
381   then there was some kind of error.
382close(2)
383All this process is shown in detail in the function do_smp_func()
384and its callers, in the file "expander_conf.c".
385
386The kernel functionality is implemented in the file
387"sas_expander.c".
388
389The program "expander_conf.c" implements this. It takes one
390argument, the sysfs file name of the SMP portal to the
391expander, and gives expander information, including routing
392tables.
393
394The SMP portal gives you complete control of the expander,
395so please be careful.
396