1 The Linux RapidIO Subsystem 2 3~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4 5The RapidIO standard is a packet-based fabric interconnect standard designed for 6use in embedded systems. Development of the RapidIO standard is directed by the 7RapidIO Trade Association (RTA). The current version of the RapidIO specification 8is publicly available for download from the RTA web-site [1]. 9 10This document describes the basics of the Linux RapidIO subsystem and provides 11information on its major components. 12 131 Overview 14---------- 15 16Because the RapidIO subsystem follows the Linux device model it is integrated 17into the kernel similarly to other buses by defining RapidIO-specific device and 18bus types and registering them within the device model. 19 20The Linux RapidIO subsystem is architecture independent and therefore defines 21architecture-specific interfaces that provide support for common RapidIO 22subsystem operations. 23 242. Core Components 25------------------ 26 27A typical RapidIO network is a combination of endpoints and switches. 28Each of these components is represented in the subsystem by an associated data 29structure. The core logical components of the RapidIO subsystem are defined 30in include/linux/rio.h file. 31 322.1 Master Port 33 34A master port (or mport) is a RapidIO interface controller that is local to the 35processor executing the Linux code. A master port generates and receives RapidIO 36packets (transactions). In the RapidIO subsystem each master port is represented 37by a rio_mport data structure. This structure contains master port specific 38resources such as mailboxes and doorbells. The rio_mport also includes a unique 39host device ID that is valid when a master port is configured as an enumerating 40host. 41 42RapidIO master ports are serviced by subsystem specific mport device drivers 43that provide functionality defined for this subsystem. To provide a hardware 44independent interface for RapidIO subsystem operations, rio_mport structure 45includes rio_ops data structure which contains pointers to hardware specific 46implementations of RapidIO functions. 47 482.2 Device 49 50A RapidIO device is any endpoint (other than mport) or switch in the network. 51All devices are presented in the RapidIO subsystem by corresponding rio_dev data 52structure. Devices form one global device list and per-network device lists 53(depending on number of available mports and networks). 54 552.3 Switch 56 57A RapidIO switch is a special class of device that routes packets between its 58ports towards their final destination. The packet destination port within a 59switch is defined by an internal routing table. A switch is presented in the 60RapidIO subsystem by rio_dev data structure expanded by additional rio_switch 61data structure, which contains switch specific information such as copy of the 62routing table and pointers to switch specific functions. 63 64The RapidIO subsystem defines the format and initialization method for subsystem 65specific switch drivers that are designed to provide hardware-specific 66implementation of common switch management routines. 67 682.4 Network 69 70A RapidIO network is a combination of interconnected endpoint and switch devices. 71Each RapidIO network known to the system is represented by corresponding rio_net 72data structure. This structure includes lists of all devices and local master 73ports that form the same network. It also contains a pointer to the default 74master port that is used to communicate with devices within the network. 75 762.5 Device Drivers 77 78RapidIO device-specific drivers follow Linux Kernel Driver Model and are 79intended to support specific RapidIO devices attached to the RapidIO network. 80 812.6 Subsystem Interfaces 82 83RapidIO interconnect specification defines features that may be used to provide 84one or more common service layers for all participating RapidIO devices. These 85common services may act separately from device-specific drivers or be used by 86device-specific drivers. Example of such service provider is the RIONET driver 87which implements Ethernet-over-RapidIO interface. Because only one driver can be 88registered for a device, all common RapidIO services have to be registered as 89subsystem interfaces. This allows to have multiple common services attached to 90the same device without blocking attachment of a device-specific driver. 91 923. Subsystem Initialization 93--------------------------- 94 95In order to initialize the RapidIO subsystem, a platform must initialize and 96register at least one master port within the RapidIO network. To register mport 97within the subsystem controller driver's initialization code calls function 98rio_register_mport() for each available master port. 99 100After all active master ports are registered with a RapidIO subsystem, 101an enumeration and/or discovery routine may be called automatically or 102by user-space command. 103 104RapidIO subsystem can be configured to be built as a statically linked or 105modular component of the kernel (see details below). 106 1074. Enumeration and Discovery 108---------------------------- 109 1104.1 Overview 111------------ 112 113RapidIO subsystem configuration options allow users to build enumeration and 114discovery methods as statically linked components or loadable modules. 115An enumeration/discovery method implementation and available input parameters 116define how any given method can be attached to available RapidIO mports: 117simply to all available mports OR individually to the specified mport device. 118 119Depending on selected enumeration/discovery build configuration, there are 120several methods to initiate an enumeration and/or discovery process: 121 122 (a) Statically linked enumeration and discovery process can be started 123 automatically during kernel initialization time using corresponding module 124 parameters. This was the original method used since introduction of RapidIO 125 subsystem. Now this method relies on enumerator module parameter which is 126 'rio-scan.scan' for existing basic enumeration/discovery method. 127 When automatic start of enumeration/discovery is used a user has to ensure 128 that all discovering endpoints are started before the enumerating endpoint 129 and are waiting for enumeration to be completed. 130 Configuration option CONFIG_RAPIDIO_DISC_TIMEOUT defines time that discovering 131 endpoint waits for enumeration to be completed. If the specified timeout 132 expires the discovery process is terminated without obtaining RapidIO network 133 information. NOTE: a timed out discovery process may be restarted later using 134 a user-space command as it is described below (if the given endpoint was 135 enumerated successfully). 136 137 (b) Statically linked enumeration and discovery process can be started by 138 a command from user space. This initiation method provides more flexibility 139 for a system startup compared to the option (a) above. After all participating 140 endpoints have been successfully booted, an enumeration process shall be 141 started first by issuing a user-space command, after an enumeration is 142 completed a discovery process can be started on all remaining endpoints. 143 144 (c) Modular enumeration and discovery process can be started by a command from 145 user space. After an enumeration/discovery module is loaded, a network scan 146 process can be started by issuing a user-space command. 147 Similar to the option (b) above, an enumerator has to be started first. 148 149 (d) Modular enumeration and discovery process can be started by a module 150 initialization routine. In this case an enumerating module shall be loaded 151 first. 152 153When a network scan process is started it calls an enumeration or discovery 154routine depending on the configured role of a master port: host or agent. 155 156Enumeration is performed by a master port if it is configured as a host port by 157assigning a host destination ID greater than or equal to zero. The host 158destination ID can be assigned to a master port using various methods depending 159on RapidIO subsystem build configuration: 160 161 (a) For a statically linked RapidIO subsystem core use command line parameter 162 "rapidio.hdid=" with a list of destination ID assignments in order of mport 163 device registration. For example, in a system with two RapidIO controllers 164 the command line parameter "rapidio.hdid=-1,7" will result in assignment of 165 the host destination ID=7 to the second RapidIO controller, while the first 166 one will be assigned destination ID=-1. 167 168 (b) If the RapidIO subsystem core is built as a loadable module, in addition 169 to the method shown above, the host destination ID(s) can be specified using 170 traditional methods of passing module parameter "hdid=" during its loading: 171 - from command line: "modprobe rapidio hdid=-1,7", or 172 - from modprobe configuration file using configuration command "options", 173 like in this example: "options rapidio hdid=-1,7". An example of modprobe 174 configuration file is provided in the section below. 175 176 NOTES: 177 (i) if "hdid=" parameter is omitted all available mport will be assigned 178 destination ID = -1; 179 (ii) the "hdid=" parameter in systems with multiple mports can have 180 destination ID assignments omitted from the end of list (default = -1). 181 182If the host device ID for a specific master port is set to -1, the discovery 183process will be performed for it. 184 185The enumeration and discovery routines use RapidIO maintenance transactions 186to access the configuration space of devices. 187 188NOTE: If RapidIO switch-specific device drivers are built as loadable modules 189they must be loaded before enumeration/discovery process starts. 190This requirement is cased by the fact that enumeration/discovery methods invoke 191vendor-specific callbacks on early stages. 192 1934.2 Automatic Start of Enumeration and Discovery 194------------------------------------------------ 195 196Automatic enumeration/discovery start method is applicable only to built-in 197enumeration/discovery RapidIO configuration selection. To enable automatic 198enumeration/discovery start by existing basic enumerator method set use boot 199command line parameter "rio-scan.scan=1". 200 201This configuration requires synchronized start of all RapidIO endpoints that 202form a network which will be enumerated/discovered. Discovering endpoints have 203to be started before an enumeration starts to ensure that all RapidIO 204controllers have been initialized and are ready to be discovered. Configuration 205parameter CONFIG_RAPIDIO_DISC_TIMEOUT defines time (in seconds) which 206a discovering endpoint will wait for enumeration to be completed. 207 208When automatic enumeration/discovery start is selected, basic method's 209initialization routine calls rio_init_mports() to perform enumeration or 210discovery for all known mport devices. 211 212Depending on RapidIO network size and configuration this automatic 213enumeration/discovery start method may be difficult to use due to the 214requirement for synchronized start of all endpoints. 215 2164.3 User-space Start of Enumeration and Discovery 217------------------------------------------------- 218 219User-space start of enumeration and discovery can be used with built-in and 220modular build configurations. For user-space controlled start RapidIO subsystem 221creates the sysfs write-only attribute file '/sys/bus/rapidio/scan'. To initiate 222an enumeration or discovery process on specific mport device, a user needs to 223write mport_ID (not RapidIO destination ID) into that file. The mport_ID is a 224sequential number (0 ... RIO_MAX_MPORTS) assigned during mport device 225registration. For example for machine with single RapidIO controller, mport_ID 226for that controller always will be 0. 227 228To initiate RapidIO enumeration/discovery on all available mports a user may 229write '-1' (or RIO_MPORT_ANY) into the scan attribute file. 230 2314.4 Basic Enumeration Method 232---------------------------- 233 234This is an original enumeration/discovery method which is available since 235first release of RapidIO subsystem code. The enumeration process is 236implemented according to the enumeration algorithm outlined in the RapidIO 237Interconnect Specification: Annex I [1]. 238 239This method can be configured as statically linked or loadable module. 240The method's single parameter "scan" allows to trigger the enumeration/discovery 241process from module initialization routine. 242 243This enumeration/discovery method can be started only once and does not support 244unloading if it is built as a module. 245 246The enumeration process traverses the network using a recursive depth-first 247algorithm. When a new device is found, the enumerator takes ownership of that 248device by writing into the Host Device ID Lock CSR. It does this to ensure that 249the enumerator has exclusive right to enumerate the device. If device ownership 250is successfully acquired, the enumerator allocates a new rio_dev structure and 251initializes it according to device capabilities. 252 253If the device is an endpoint, a unique device ID is assigned to it and its value 254is written into the device's Base Device ID CSR. 255 256If the device is a switch, the enumerator allocates an additional rio_switch 257structure to store switch specific information. Then the switch's vendor ID and 258device ID are queried against a table of known RapidIO switches. Each switch 259table entry contains a pointer to a switch-specific initialization routine that 260initializes pointers to the rest of switch specific operations, and performs 261hardware initialization if necessary. A RapidIO switch does not have a unique 262device ID; it relies on hopcount and routing for device ID of an attached 263endpoint if access to its configuration registers is required. If a switch (or 264chain of switches) does not have any endpoint (except enumerator) attached to 265it, a fake device ID will be assigned to configure a route to that switch. 266In the case of a chain of switches without endpoint, one fake device ID is used 267to configure a route through the entire chain and switches are differentiated by 268their hopcount value. 269 270For both endpoints and switches the enumerator writes a unique component tag 271into device's Component Tag CSR. That unique value is used by the error 272management notification mechanism to identify a device that is reporting an 273error management event. 274 275Enumeration beyond a switch is completed by iterating over each active egress 276port of that switch. For each active link, a route to a default device ID 277(0xFF for 8-bit systems and 0xFFFF for 16-bit systems) is temporarily written 278into the routing table. The algorithm recurs by calling itself with hopcount + 1 279and the default device ID in order to access the device on the active port. 280 281After the host has completed enumeration of the entire network it releases 282devices by clearing device ID locks (calls rio_clear_locks()). For each endpoint 283in the system, it sets the Discovered bit in the Port General Control CSR 284to indicate that enumeration is completed and agents are allowed to execute 285passive discovery of the network. 286 287The discovery process is performed by agents and is similar to the enumeration 288process that is described above. However, the discovery process is performed 289without changes to the existing routing because agents only gather information 290about RapidIO network structure and are building an internal map of discovered 291devices. This way each Linux-based component of the RapidIO subsystem has 292a complete view of the network. The discovery process can be performed 293simultaneously by several agents. After initializing its RapidIO master port 294each agent waits for enumeration completion by the host for the configured wait 295time period. If this wait time period expires before enumeration is completed, 296an agent skips RapidIO discovery and continues with remaining kernel 297initialization. 298 2994.5 Adding New Enumeration/Discovery Method 300------------------------------------------- 301 302RapidIO subsystem code organization allows addition of new enumeration/discovery 303methods as new configuration options without significant impact to the core 304RapidIO code. 305 306A new enumeration/discovery method has to be attached to one or more mport 307devices before an enumeration/discovery process can be started. Normally, 308method's module initialization routine calls rio_register_scan() to attach 309an enumerator to a specified mport device (or devices). The basic enumerator 310implementation demonstrates this process. 311 3124.6 Using Loadable RapidIO Switch Drivers 313----------------------------------------- 314 315In the case when RapidIO switch drivers are built as loadable modules a user 316must ensure that they are loaded before the enumeration/discovery starts. 317This process can be automated by specifying pre- or post- dependencies in the 318RapidIO-specific modprobe configuration file as shown in the example below. 319 320 File /etc/modprobe.d/rapidio.conf: 321 ---------------------------------- 322 323 # Configure RapidIO subsystem modules 324 325 # Set enumerator host destination ID (overrides kernel command line option) 326 options rapidio hdid=-1,2 327 328 # Load RapidIO switch drivers immediately after rapidio core module was loaded 329 softdep rapidio post: idt_gen2 idtcps tsi57x 330 331 # OR : 332 333 # Load RapidIO switch drivers just before rio-scan enumerator module is loaded 334 softdep rio-scan pre: idt_gen2 idtcps tsi57x 335 336 -------------------------- 337 338NOTE: In the example above, one of "softdep" commands must be removed or 339commented out to keep required module loading sequence. 340 341A. References 342------------- 343 344[1] RapidIO Trade Association. RapidIO Interconnect Specifications. 345 http://www.rapidio.org. 346[2] Rapidio TA. Technology Comparisons. 347 http://www.rapidio.org/education/technology_comparisons/ 348[3] RapidIO support for Linux. 349 http://lwn.net/Articles/139118/ 350[4] Matt Porter. RapidIO for Linux. Ottawa Linux Symposium, 2005 351 http://www.kernel.org/doc/ols/2005/ols2005v2-pages-43-56.pdf 352