1Copyright (C) 2015 Freescale Semiconductor Inc.
2
3DPAA2 (Data Path Acceleration Architecture Gen2)
4------------------------------------------------
5
6This document provides an overview of the Freescale DPAA2 architecture
7and how it is integrated into the Linux kernel.
8
9Contents summary
10   -DPAA2 overview
11   -Overview of DPAA2 objects
12   -DPAA2 Linux driver architecture overview
13        -bus driver
14        -dprc driver
15        -allocator
16        -dpio driver
17        -Ethernet
18        -mac
19
20DPAA2 Overview
21--------------
22
23DPAA2 is a hardware architecture designed for high-speeed network
24packet processing.  DPAA2 consists of sophisticated mechanisms for
25processing Ethernet packets, queue management, buffer management,
26autonomous L2 switching, virtual Ethernet bridging, and accelerator
27(e.g. crypto) sharing.
28
29A DPAA2 hardware component called the Management Complex (or MC) manages the
30DPAA2 hardware resources.  The MC provides an object-based abstraction for
31software drivers to use the DPAA2 hardware.
32
33The MC uses DPAA2 hardware resources such as queues, buffer pools, and
34network ports to create functional objects/devices such as network
35interfaces, an L2 switch, or accelerator instances.
36
37The MC provides memory-mapped I/O command interfaces (MC portals)
38which DPAA2 software drivers use to operate on DPAA2 objects:
39
40         +--------------------------------------+
41         |                  OS                  |
42         |                        DPAA2 drivers |
43         |                             |        |
44         +-----------------------------|--------+
45                                       |
46                                       | (create,discover,connect
47                                       |  config,use,destroy)
48                                       |
49                         DPAA2         |
50         +------------------------| mc portal |-+
51         |                             |        |
52         |   +- - - - - - - - - - - - -V- - -+  |
53         |   |                               |  |
54         |   |   Management Complex (MC)     |  |
55         |   |                               |  |
56         |   +- - - - - - - - - - - - - - - -+  |
57         |                                      |
58         | Hardware                  Hardware   |
59         | Resources                 Objects    |
60         | ---------                 -------    |
61         | -queues                   -DPRC      |
62         | -buffer pools             -DPMCP     |
63         | -Eth MACs/ports           -DPIO      |
64         | -network interface        -DPNI      |
65         |  profiles                 -DPMAC     |
66         | -queue portals            -DPBP      |
67         | -MC portals                ...       |
68         |  ...                                 |
69         |                                      |
70         +--------------------------------------+
71
72The MC mediates operations such as create, discover,
73connect, configuration, and destroy.  Fast-path operations
74on data, such as packet transmit/receive, are not mediated by
75the MC and are done directly using memory mapped regions in
76DPIO objects.
77
78Overview of DPAA2 Objects
79-------------------------
80The section provides a brief overview of some key objects
81in the DPAA2 hardware.  A simple scenario is described illustrating
82the objects involved in creating a network interfaces.
83
84-DPRC (Datapath Resource Container)
85
86    A DPRC is an container object that holds all the other
87    types of DPAA2 objects.  In the example diagram below there
88    are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC)
89    in the container.
90
91    +---------------------------------------------------------+
92    | DPRC                                                    |
93    |                                                         |
94    |  +-------+  +-------+  +-------+  +-------+  +-------+  |
95    |  | DPMCP |  | DPIO  |  | DPBP  |  | DPNI  |  | DPMAC |  |
96    |  +-------+  +-------+  +-------+  +---+---+  +---+---+  |
97    |  | DPMCP |  | DPIO  |                                   |
98    |  +-------+  +-------+                                   |
99    |  | DPMCP |                                              |
100    |  +-------+                                              |
101    |                                                         |
102    +---------------------------------------------------------+
103
104    From the point of view of an OS, a DPRC is bus-like.  Like
105    a plug-and-play bus, such as PCI, DPRC commands can be used to
106    enumerate the contents of the DPRC, discover the hardware
107    objects present (including mappable regions and interrupts).
108
109     dprc.1 (bus)
110       |
111       +--+--------+-------+-------+-------+
112          |        |       |       |       |
113        dpmcp.1  dpio.1  dpbp.1  dpni.1  dpmac.1
114        dpmcp.2  dpio.2
115        dpmcp.3
116
117    Hardware objects can be created and destroyed dynamically, providing
118    the ability to hot plug/unplug objects in and out of the DPRC.
119
120    A DPRC has a mappable mmio region (an MC portal) that can be used
121    to send MC commands.  It has an interrupt for status events (like
122    hotplug).
123
124    All objects in a container share the same hardware "isolation context".
125    This means that with respect to an IOMMU the isolation granularity
126    is at the DPRC (container) level, not at the individual object
127    level.
128
129    DPRCs can be defined statically and populated with objects
130    via a config file passed to the MC when firmware starts
131    it.  There is also a Linux user space tool called "restool"
132    that can be used to create/destroy containers and objects
133    dynamically.
134
135-DPAA2 Objects for an Ethernet Network Interface
136
137    A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX
138    queuing mechanisms, configuration mechanisms, buffer management,
139    physical ports, and interrupts.  DPAA2 uses a more granular approach
140    utilizing multiple hardware objects.  Each object has specialized
141    functions, and are used together by software to provide Ethernet network
142    interface functionality.  This approach provides efficient use of finite
143    hardware resources, flexibility, and performance advantages.
144
145    The diagram below shows the objects needed for a simple
146    network interface configuration on a system with 2 CPUs.
147
148              +---+---+ +---+---+
149                 CPU0     CPU1
150              +---+---+ +---+---+
151                  |         |
152              +---+---+ +---+---+
153                 DPIO     DPIO
154              +---+---+ +---+---+
155                    \     /
156                     \   /
157                      \ /
158                   +---+---+
159                      DPNI  --- DPBP,DPMCP
160                   +---+---+
161                       |
162                       |
163                   +---+---+
164                     DPMAC
165                   +---+---+
166                       |
167                    port/PHY
168
169    Below the objects are described.  For each object a brief description
170    is provided along with a summary of the kinds of operations the object
171    supports and a summary of key resources of the object (mmio regions
172    and irqs).
173
174       -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a
175        hardware device that connects to an Ethernet PHY and allows
176        physical transmission and reception of Ethernet frames.
177           -mmio regions: none
178           -irqs: dpni link change
179           -commands: set link up/down, link config, get stats,
180             irq config, enable, reset
181
182       -DPNI (Datapath Network Interface): contains TX/RX queues,
183        network interface configuration, and rx buffer pool configuration
184        mechanisms.
185           -mmio regions: none
186           -irqs: link state
187           -commands: port config, offload config, queue config,
188            parse/classify config, irq config, enable, reset
189
190       -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue
191        packets and do hardware buffer pool management operations.  For
192        optimum performance there is typically DPIO per CPU.  This allows
193        each CPU to perform simultaneous enqueue/dequeue operations.
194           -mmio regions: queue operations, buffer mgmt
195           -irqs: data availability, congestion notification, buffer
196                  pool depletion
197           -commands: irq config, enable, reset
198
199       -DPBP (Datapath Buffer Pool): represents a hardware buffer
200        pool.
201           -mmio regions: none
202           -irqs: none
203           -commands: enable, reset
204
205       -DPMCP (Datapath MC Portal): provides an MC command portal.
206        Used by drivers to send commands to the MC to manage
207        objects.
208           -mmio regions: MC command portal
209           -irqs: command completion
210           -commands: irq config, enable, reset
211
212    Object Connections
213    ------------------
214    Some objects have explicit relationships that must
215    be configured:
216
217       -DPNI <--> DPMAC
218       -DPNI <--> DPNI
219       -DPNI <--> L2-switch-port
220          A DPNI must be connected to something such as a DPMAC,
221          another DPNI, or L2 switch port.  The DPNI connection
222          is made via a DPRC command.
223
224              +-------+  +-------+
225              | DPNI  |  | DPMAC |
226              +---+---+  +---+---+
227                  |          |
228                  +==========+
229
230       -DPNI <--> DPBP
231          A network interface requires a 'buffer pool' (DPBP
232          object) which provides a list of pointers to memory
233          where received Ethernet data is to be copied.  The
234          Ethernet driver configures the DPBPs associated with
235          the network interface.
236
237    Interrupts
238    ----------
239    All interrupts generated by DPAA2 objects are message
240    interrupts.  At the hardware level message interrupts
241    generated by devices will normally have 3 components--
242    1) a non-spoofable 'device-id' expressed on the hardware
243    bus, 2) an address, 3) a data value.
244
245    In the case of DPAA2 devices/objects, all objects in the
246    same container/DPRC share the same 'device-id'.
247    For ARM-based SoC this is the same as the stream ID.
248
249
250DPAA2 Linux Driver Overview
251---------------------------
252
253This section provides an overview of the Linux kernel drivers for
254DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure"
255drivers and 2) functional object drivers (such as Ethernet).
256
257As described previously, a DPRC is a container that holds the other
258types of DPAA2 objects.  It is functionally similar to a plug-and-play
259bus controller.
260
261Each object in the DPRC is a Linux "device" and is bound to a driver.
262The diagram below shows the Linux drivers involved in a networking
263scenario and the objects bound to each driver.  A brief description
264of each driver follows.
265
266                                             +------------+
267                                             | OS Network |
268                                             |   Stack    |
269                 +------------+              +------------+
270                 | Allocator  |. . . . . . . |  Ethernet  |
271                 |(dpmcp,dpbp)|              |   (dpni)   |
272                 +-.----------+              +---+---+----+
273                  .          .                   ^   |
274                 .            .     <data avail, |   |<enqueue,
275                .              .     tx confirm> |   | dequeue>
276    +-------------+             .                |   |
277    | DPRC driver |              .           +---+---V----+     +---------+
278    |   (dprc)    |               . . . . . .| DPIO driver|     |   MAC   |
279    +----------+--+                          |  (dpio)    |     | (dpmac) |
280               |                             +------+-----+     +-----+---+
281               |<dev add/remove>                    |                 |
282               |                                    |                 |
283          +----+--------------+                     |              +--+---+
284          |   mc-bus driver   |                     |              | PHY  |
285          |                   |                     |              |driver|
286          | /fsl-mc@80c000000 |                     |              +--+---+
287          +-------------------+                     |                 |
288                                                    |                 |
289 ================================ HARDWARE =========|=================|======
290                                                  DPIO                |
291                                                    |                 |
292                                                  DPNI---DPBP         |
293                                                    |                 |
294                                                  DPMAC               |
295                                                    |                 |
296                                                   PHY ---------------+
297 ===================================================|========================
298
299A brief description of each driver is provided below.
300
301    mc-bus driver
302    -------------
303    The mc-bus driver is a platform driver and is probed from an
304    "/fsl-mc@xxxx" node in the device tree passed in by boot firmware.
305    It is responsible for bootstrapping the DPAA2 kernel infrastructure.
306    Key functions include:
307       -registering a new bus type named "fsl-mc" with the kernel,
308        and implementing bus call-backs (e.g. match/uevent/dev_groups)
309       -implemeting APIs for DPAA2 driver registration and for device
310        add/remove
311       -creates an MSI irq domain
312       -do a device add of the 'root' DPRC device, which is needed
313        to bootstrap things
314
315    DPRC driver
316    -----------
317    The dprc-driver is bound DPRC objects and does runtime management
318    of a bus instance.  It performs the initial bus scan of the DPRC
319    and handles interrupts for container events such as hot plug.
320
321    Allocator
322    ----------
323    Certain objects such as DPMCP and DPBP are generic and fungible,
324    and are intended to be used by other drivers.  For example,
325    the DPAA2 Ethernet driver needs:
326       -DPMCPs to send MC commands, to configure network interfaces
327       -DPBPs for network buffer pools
328
329    The allocator driver registers for these allocatable object types
330    and those objects are bound to the allocator when the bus is probed.
331    The allocator maintains a pool of objects that are available for
332    allocation by other DPAA2 drivers.
333
334    DPIO driver
335    -----------
336    The DPIO driver is bound to DPIO objects and provides services that allow
337    other drivers such as the Ethernet driver to receive and transmit data.
338    Key services include:
339        -data availability notifications
340        -hardware queuing operations (enqueue and dequeue of data)
341        -hardware buffer pool management
342
343    There is typically one DPIO object per physical CPU for optimum
344    performance, allowing each CPU to simultaneously enqueue
345    and dequeue data.
346
347    The DPIO driver operates on behalf of all DPAA2 drivers
348    active in the kernel--  Ethernet, crypto, compression,
349    etc.
350
351    Ethernet
352    --------
353    The Ethernet driver is bound to a DPNI and implements the kernel
354    interfaces needed to connect the DPAA2 network interface to
355    the network stack.
356
357    Each DPNI corresponds to a Linux network interface.
358
359    MAC driver
360    ----------
361    An Ethernet PHY is an off-chip, board specific component and is managed
362    by the appropriate PHY driver via an mdio bus.  The MAC driver
363    plays a role of being a proxy between the PHY driver and the
364    MC.  It does this proxy via the MC commands to a DPMAC object.
365