Growth Networks Inc

Report
Design of a
Diversified Router:
Dedicated CRF
plus IPv4 Metarouter
John DeHart
[email protected]
http://www.arl.wustl.edu/arl
Revision History

5/22/06 (JDD):

6/1/06 (JDD):

6/2/06 (JDD):

6/5/06 (JDD):

6/6/06 (JDD):
» Created
» Buffer descriptor stuff probably needs updating.
» Updating data going between blocks, still in progress.
» More cleanup of data going between blocks.
» Buffer descriptor details still need updating.
» Slight change to format for Lookup Key and defining what goes in each word in the NN ring.
» Add IP Pkt Length to data Demux passes to Parse
» Reorganized the Lookup Result given to Hdr Format to distinguish between MR portion and
Substrate portion.
» Clean up labeling of data to Parse (MN vs. IP Pkt)

Output from Parse is still IP Pkt Offset and Length.
» Data from Parse to Lookup needs update to reflect case where lookup is just for Substrate
mapping of MI to LC.

6/7/06 (JDD):

6/15/06 (JDD):
» Updated notes about Parse block’s input/output and functionality
» Removed CRC from Rx to Demux data.

MSF does not pass us a CRC like we thought so we will skip the CRC checking.
» Updated data going from Demux to Parse, Parse to Lookup and Lookup to Hdr Format
‹#› - JDD - 7/17/2015
Revision History

6/19/06 (BDH):
» Split Header Format into MR Header Format and Substrate Encap
» Demux is now Substrate Decap
» Reorganization of all slides into logical and physical formats, coloring scheme
» IPv4 MR now has own section, integrated JL’s internal format slides
‹#› - JDD - 7/17/2015
Dedicated CRF Slide Organization
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
L2
L2
L1
Substr
Encap
QM
Tx
Metarouter
Input Data
Block
Output Data

In the “at-a-glance” format, all blocks are logical

In the detailed format, all blocks are physical

Color scheme
Substrate
» Logical inputs and outputs
» High-level overview of processing
» Each logical block is like an Intel microblock not necessarily an ME.
» Physical inputs and outputs
» Specific functionality and implementation notes
» Blue = Substrate, should not change!
» Green = Metarouter, different for each MR
‹#› - JDD - 7/17/2015
Logical Formats
Receive
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
L2
L2
L1
Substr
Encap
QM
Tx
Buffer Handle
RBUF
Rx
Ethernet Frame Len
Port

Coordinate transfer of packets from RBUF to DRAM
‹#› - JDD - 7/17/2015
Substrate Decapsulate
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
L2
L2
L1
Substr
Encap
QM
Tx
Buffer Handle
Substrate Type
Buffer Handle
Ethernet Frame Len
Port
Block
Src ID
MN Frame Length
MN Frame Offset




Read and validate Ethernet header from DRAM
Read and validate substrate header from DRAM
Extract Rx MI or Source MPE
Calculate MN frame length and offset
‹#› - JDD - 7/17/2015
Metarouter Parse
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
Src ID (RxMI/SrcMPE)
MN Frame Length
MN Frame Offset
L2
L1
Substr
Encap
QM
Tx
Buffer Handle
Buffer Handle
Parse Flags
L2
MR
Parse
Lookup Flags
Lookup Key
to Lookup
Src ID
MN Pkt Length
to MR Hdr Format
MR Data


Read and align MN header (includes IPv4 Hdr) from DRAM
MR-specific
»
»
»
»
»

Consume internal header (if packet from other MPE of MR)
Header validation
Header modification
Exception checks
Extract lookup key and set lookup flags
Write aligned modified IPv4 header back to DRAM
‹#› - JDD - 7/17/2015
Lookup
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
L2
L2
L1
Substr
Encap
QM
Tx
Buffer Handle
Buffer Handle
Lookup Input Flags
Lookup Key



Lookup Result Flags
Lookup
Dest Addr
Output Port
Src ID
QID
MN Pkt Length
MR Lookup Result
Perform lookup in TCAM
Increment counters based on Stats Index
Priority resolution of results from multiple databases,
if needed
‹#› - JDD - 7/17/2015
Metarouter Header Format
L1
L2
Rx
Substr
Decap
from Lookup
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format



L2
L1
Substr
Encap
QM
Tx
Buffer Handle
Buffer Handle
Lookup Result Flags
MN Frame Length
Dest Addr
Output Port
from MR Parse
L2
MR Hdr
Format
MN Frame Offset
Dest Addr
QID
Output Port
MR Lookup Result
QID
MR Data
Substrate Type
Substr. Type-dep. Data
Process Lookup result
For exceptions, generate internal header
Decide substrate type
‹#› - JDD - 7/17/2015
Substrate Encapsulation
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
L2
L2
L1
Substr
Encap
QM
Tx
Buffer Handle
MN Frame Length
MN Frame Offset
Dest Addr
Output Port
Buffer Handle
Substr
Encap
Output Port
QID
MN Frame Length
QID
Substrate Type
Substr. Type-dep. Data

Write substrate and ethernet headers
‹#› - JDD - 7/17/2015
Queue Manager
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
L2
L2
L1
Substr
Encap
QM
Tx
Buffer Handle
Output Port
QID
QM
Buffer Handle
Output Port
MN Frame Length



CRF queue management for Meta Interface
queues
WRR?
Details
‹#› - JDD - 7/17/2015
Transmit
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
Buffer Handle
Output Port


Tx
L2
L2
L1
Substr
Encap
QM
Tx
TBUF
Coordinate transfer of packets from DRAM to TBUFs
Recycle buffer handle
‹#› - JDD - 7/17/2015
Physical Formats
Receive
RBUF
Buffer
Descriptor
Buf Handle(32b)
Eth. Frame
Len (16b)
Reserved
(8b)
Port
(8b)
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
TxMI



RBUF format details here
Buf Handle details here
Notes:
» We’ll pass the Buffer Handle which contains the
SRAM address of the buffer descriptor.
» From the SRAM address of the descriptor we
can calculate the DRAM address of the buffer
data.
‹#› - JDD - 7/17/2015
VLAN
Packet_Next
Substrate Decapsulate
Buf Handle(32b)
Eth. Frame
Len (16b)
Reserved
(8b)
Buf Handle(32b)
Port
(8b)
P
Flags MR ID (VLAN) (12b)
(4b)
Rx MI(16b)
MN Frm Length(16b) MN Frm Offset (16b)
Buffer
Descriptor
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID

P Flags: indicate RxMI vs. SrcMPE
» bit0: 0: RxMI , 1: ScrMPE
‹#› - JDD - 7/17/2015
TxMI
VLAN
Packet_Next
Metarouter Parse
Buf Handle(32b)
P
Flags MR ID (VLAN) (12b)
(4b)
RxMI/SrcMPE
(16b)
MN Pkt Length (16b) MN Pkt Offset (16b)
Buf Handle(32b)
L
Flags
(4b)
MR Passthrough (28b)
MR Passthrough (32b)
Buffer_Next
Lookup Key[143-112] MR/MI (32b)
Buffer_Size
Lookup Key[111-80] (32b)
Lookup Key[ 79-48] (32b)
Lookup Key[ 47-16] (32b)
Lookup Key
[15- 0] (16b)





Offset
Free_List
Packet_Size
MR_ID
TxMI
bit 0: 0: Normal, 1: Substrate Lookup
bit 1: 0: Normal, 1: NH MN Address present in Key Word[1]


Reserved (16b)
L Flags:
»
»
Buffer
Descriptor
Key Word[0] = MR/MI
Bit 1 should never be set without bit 0 also being set.
Hdr Format needs to start at the beginning of the IP Header and re-write
headers upward in the Buffer.
Can Parse adjust the buffer/packet size and offset?
Can Parse do something like, terminate a tunnel and strip off an outer
header?
Rx MI needs to be passed to Header Format (through Lookup) so that
Header Format can include it in the shim of packets that end up on the
slow path. This will allow the Control Processor know what interface the
exception packets arrived on.
‹#› - JDD - 7/17/2015
VLAN
Packet_Next
Lookup
Buf Handle(32b)
L
Flags
(4b)
MR Passthrough (28b)
MR Passthrough (32b)
Buf Handle(32b)
H
Flags
(4b)
MR Passthrough (28b)
Buffer_Next
MR Passthrough (32b)
Lookup Key[143-112] MR/MI (32b)
Lookup Key[111-80] (32b)
Lookup Key[ 79-48] (32b)
Lookup Key[ 47-16] (32b)
Lookup Key
[15- 0] (16b)
Buffer
Descriptor
Buffer_Size
MR Lookup Result (32b)
Offset
MR Lookup Result (32b)
Free_List
Port
DA(8b) (4b)
QID(20b)
Reserved (16b)
Packet_Size
MR_ID
TxMI

L Flags:
» bit 0: 0: Normal, 1: Substrate Lookup
» bit 1: 0: Normal, 1: NH MN Address present in
Key Word[1]


Key Word[0] = MR/MI
Bit 1 should never be set without bit 0 also being
set.
‹#› - JDD - 7/17/2015
VLAN
Packet_Next
Metarouter Header Format
Buf Handle(32b)
H
Flags
(4b)
Buffer Handle(32b)
MR Passthrough (28b)
Port(8b)
MR Passthrough (32b)
MR Lookup Result (32b)
MR Lookup Result (32b)
Port
DA(8b) (4b)
QID(20b)
Rsv
(4b)
QID(20b)
Text
SH Type SH Len
(8b)
(8b)
Rsv(16b)
Substrate Header Data (LWO)
Offset
Free_List
Packet_Size
MR_ID
TxMI
Substrate Header Data (LW2)
VLAN
Substrate Header Data (LW3)
Packet_Next
Substrate Header Data (LW4)
‹#› - JDD - 7/17/2015
Buffer_Next
Buffer_Size
MN Pkt Offset (8b) MN Pkt Length (16b)
Substrate Header Data (LW1)

Buffer
Descriptor
Substrate Encapsulation
Buffer Handle(32b)
Port(8b)
Rsv
(4b)
QID(20b)
MN Pkt Offset (8b) MN Pkt Length (16b)
Buffer Handle(32b)
Port(8b)
Rsv
(4b)
QID(20b)
Reserved (16b) MN Pkt Length (16b)
Buffer
Descriptor
Buffer_Next
Buffer_Size
Offset
SH Type SH Len
(8b)
(8b)
Rsv(16b)
Substrate Header Data (LWO)
Substrate Header Data (LW1)
Substrate Header Data (LW2)
Substrate Header Data (LW3)
Substrate Header Data (LW4)

Substrate header types/formats here?
‹#› - JDD - 7/17/2015
Free_List
Packet_Size
MR_ID
TxMI
VLAN
Packet_Next
Queue Manager
Buffer Handle(32b)
Port(8b)
Rsv
(4b)
QID(20b)
Reserved (16b) MN Pkt Length (16b)
Buffer Handle(32b)
Reserved (24b)
Port(8b)
Buffer
Descriptor
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
TxMI

Text
‹#› - JDD - 7/17/2015
VLAN
Packet_Next
Transmit
Buffer Handle(32b)
Reserved (24b)
Port(8b)
TBUF
Buffer
Descriptor
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
TxMI

Text
‹#› - JDD - 7/17/2015
VLAN
Packet_Next
IPv4 Metarouter
MR Parse and MR Header Format
Logical and Physical Data Formats
IPv4 Logical Formats
Exception Bits
Rx MI / Source MPE
MN Frame Length
MN Frame Offset
MR
Parse
MR Hdr
Format
Lookup
Buffer Handle
Buffer Handle
Buffer Handle
Buffer Handle
Parse Flags
Lookup Flags
Lookup Result Flags
MN Frame Length
Rx MI / Source MPE
Lookup Key
Dest Addr
MN Frame Offset
MN Frame Length
Rx MI / Tx MI
Output Port
Dest Addr
QID
Output Port
IPv4 Lookup Flags
QID
TxMI
Substrate Type
IPv4 Lookup Result
Substr. Type-dep. Data
MN Frame Offset
‹#› - JDD - 7/17/2015
IPv4 Physical Formats
Buf Handle(32b)
P
Flags MR ID (VLAN) (12b)
(4b)
Buffer Handle(32b)
RxMI/SrcMPE
(16b)
Port(8b)
Exception
Bits (12b)
IP Pkt Length (16b)
Rx MI/SrcMPE
(16b)
IP Pkt Offset (16b)
Buf Handle(32b)
H
Flags
(4b)
Lookup Key[ 47-16] Ports (32b)
Lookup Key
Proto/TCP_Flags
[15- 0] (16b)
‹#› - JDD - 7/17/2015
Reserved (16b)
Exception
Bits (12b)
IP Pkt Length (16b)
Lookup Key[143-112] MR/MI (32b)
Lookup Key[111-80] DA (32b)
Lookup Key[ 79-48] SA (32b)
IP Pkt Length (16b)
MR Hdr
Format
Lookup
Buf Handle(32b)
L
Flags
(4b)
QID(20b)
Reserved (16b)
MN Pkt Length (16b) MN Pkt Offset (16b)
MR
Parse
Rsv
(4b)
HD
NM
L
A
H D
(1b)(1b)
C (1b)
(1b)(1b)
Reserved
(11b)
Rx MI/SrcMPE
(16b)
IP Pkt Offset (16b)
TxMI(16b)
MrBits[31:0](32b)
Port
DA(8b) (4b)
QID(20b)
H: Hit
D: Drop
IPv4 Parse and Header Format
 Parse
»Consume internal header
»Verify IP header (per RFC1812 5.2.2)
»Decrement TTL
»Recalc IP checksum
»Write updated checksum to DRAM
»Store drop/exception/pass statistics
»Extract lookup key, set lookup flags
 Header
Format
‹#› - JDD - 7/17/2015
IPv4 Flag Formats
 IPv4
Lookup Flags
»H:
Hit
»D:
Drop
»NH: NH MN Address present
»MAC: MAC Address needed
»LD: Local Delivery
 IPv4
Exception Bits
»Bit 0: TTL = 0 or 1
»Bit 1: Options
»Bit 2?: RxMI or SrcMPE
‹#› - JDD - 7/17/2015
IPv4 Internal Header Formats
 For
packets going from IPv4 MPE to IPv4 MPE
•
 Packets from Ingress LC or to Egress LC don’t have
Metanet Internal Header
Type (1B)
Length (1B)
RxMI (2B)
Type Dependent Data (2-6B)
‹#› - JDD - 7/17/2015
IPv4 Internal Header Types

Packets entering routing MPE
» Routing MPE: MPE that does routing lookup
» FwdKey = [TxMI + MnNhAddr if multi-access link] (**)
Source
Category
Ingress
LC
Normal
Fwd
MPE
No Classify
(w/
FwdKey**)
0x00
Classify
(w/o
FwdKey)
0x01
‹#› - JDD - 7/17/2015
Type
Value
Reason
Incoming
MR
Internal
Hdr
RMPE Action
No MR Int
Hdr
Classify and fwd
If from Control MPE, it
is an original pkt that
is reinjected back to
data path
RxMI +
FwdKey
Perform substrate
lookup to resolve
LCAddr, port and
QID
New pkt generated by
MPE. If from Control
MPE, it is ICMP or
local traffic
RxMI
Classify and fwd
IPv4 Internal Header Types

Packets exiting Routing MPE
» If FwdKey in MR Internal hdr is invalid, lookup raises error flag and
hdr_fmt sends pkt to CMPE for debug (***)
Destination
Category
Egress LC
Fast path
Regular
MPE
Fast path
0x02
Control
MPE
Exception
0x03
No route
RxMI
0x04
TTL expire
RxMI
0x05
IP w/ options
RxMI + FwdKey
0x06
Redirect due to RxMI =
TxMI
RxMI + FwdKey
0x07
Inspect
RxMI + FwdKey
0x08
Local delivery
RxMI
0x09
Monitor
RxMI
0x0a
Log due to error in pkts***
RxMI
Control
Debug
‹#› - JDD - 7/17/2015
Type
Value
Reason
Outgoing MR Internal
Hdr
No MR Int Hdr
RxMI + FwdKey
Extra
 The
next set of slides are for templates or extra
information if needed
‹#› - JDD - 7/17/2015
Text Slide Template
‹#› - JDD - 7/17/2015
Image Slide Template
‹#› - JDD - 7/17/2015
At-a-glance Block Template
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
L2
L2
L1
Substr
Encap
QM
Tx
Buffer Handle
RBUF
Block
Ethernet Frame Len
Port

Text
‹#› - JDD - 7/17/2015
Detailed Block Template
RBUF
Buffer
Descriptor
Buf Handle(32b)
Eth. Frame
Len (16b)
Reserved
(8b)
Port
(8b)
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
TxMI

Text
‹#› - JDD - 7/17/2015
VLAN
Packet_Next
QM/Scheduler on Multiple MEs
Header
Format
MR-1
...
MR-n
Input
Hlpr
(1 ME)
Buffer Handle(32b)
Port(8b)
Rsv
(4b)
Reserved (16b)
QID(20b)
IP Pkt Length (16b)
QM/Schd
(1 ME)
Tx
QM/Schd
(1 ME)
Tx
NN/Scratch Rings
NN Ring
Buffer Handle(32b)
Reserved (24b)
Port(8b)

QID(32b):

Input Hlpr would use QM ID to select Scratch ring on which to put
request.
QM/Sched then sends on its output NN/scratch ring to its associated Tx
With 64 entries in Q-Array and 16 entries in CAM, max number of
QM/Schds is probably 4 (2 bits).


» Reserved (8b)
» QM ID (3b)
» QID(17b): 1M queues per QM
» We’ll set aside 3 bits to give us flexibility in the future.
‹#› - JDD - 7/17/2015
Packet Buffer Descriptor Tradeoffs

Why use a Buffer Descriptor at all?
» QM needs something to link packets/buffers in queues
» ME-to-ME communications costs vs. SRAM access costs
‹#› - JDD - 7/17/2015
Packet Buffer Descriptor def

Meta Data structure of Packet Buffers (LSB to MSB)
» buffer_next
» offset
» BufferSize
» header_type
» rx_stat
» free_list
» packet_size
» output_port
» input_port
» nhid_type
» reserved
» fabric_port
» nexthop_id
» color
» flow_id
» reserved
» class_id
» packet_next
‹#› - JDD - 7/17/2015
32 bits
16 bits
16 bits
8 bits
4 bits
4 bits
16 bits
16 bits
16 bits
4 bits
4 bits
8 bits
16 bits
8 bits
24 bits
16 bits
16 bits
32 bits
Next Buffer Pointer (in a chain of buffers)
Offset to start of data in bytes
Length of data in the current buffer in bytes
type of header at offset bytes in to the buffer
Receive status flags
Freelist ID
(Total packet size across multiple buffers)
Output Port on the egress processor
Input Port on the ingress processor
Nexthop ID type.
Reserved
Output port for fabric indicating blade ID.
NextHop IP ID
Qos Color
QOS flow ID or MPLS label/flow id
Reserved
Class ID
pointer to next packet (unused in cell mode)
Packet Buffer Descriptor Gets















buffer_next: tx
Offset: rx, tx, fwd
BufferSize: tx, fwd
header_type: tx, fwd
rx_stat: NONE
free_listpacket_size: NONE
output_port: qm(?), tx
input_port: rx, fwd
nhid_type: NONE
fabric_port: qm(?), tx
nexthop_id
color
flow_id
class_id
packet_next
‹#› - JDD - 7/17/2015
Meta Data Caching
 Meta
Data can be cached in one of three places:
»SRAM Xfer Registers
»DRAM Xfer Registers
»GPR Registers
 Size
of Meta Data Cache is controlled by #define
META_CACHE_SIZE
 Macro dl_meta_load_cache[] loads meta data cache
»buffer_handle: buffer handle for which meta data is to be
fetched
»dl_meta: read transfer register prefix

Xbuf_alloc[] should be used to allocate the needed registers
»signal_number:
»START_LW: starting long word for fetch
»NUM_LW: number of long words to fetch
 Each
microengine (microblock?) can use Meta Data
Caching differently.
‹#› - JDD - 7/17/2015
Meta Data Caching

In the ipv4_v6_forwarder sample app,
» dl_meta_load_cache() used in:

Egress





ethernet_arp.uc
pkt_tx_16p.uc
statistics_util.uc
tx_helper.uc
Ingress
ethernet_arp.uc
pkt_tx_16p.uc

statistics_util.uc

tx_helper.uc
» dl_meta_get_*[] used in:



Egress




ethernet_arp.uc
pkt_tx_16p.uc
tx_helper.uc
Ingress
Ether.uc
Ipv4_fwder.uc

Ipv4_fwder_util.uc

Ipv6_fwder.uc

V6v4_tunnel_decap.uc

V6v4_tunnel_encap.uc

pkt_tx_16p.uc

tx_helper.uc
» dl_meta_set_*[] used in:



Egress




ethernet_arp.uc
pkt_rx_init.uc
pkt_rx_two_me_util.uc
Ingress









pkt_rx_init.uc
pkt_rx_two_me_util.uc
Ether.uc
Ipv4_fwder_util.uc
Ipv6_fwder.uc
V6v4_tunnel_decap.uc
V6v4_tunnel_encap.uc
pkt_tx_16p.uc
tx_helper.uc
‹#› - JDD - 7/17/2015
Buffer Handle
‹#› - JDD - 7/17/2015
Buffer Descriptor Usage


Is there a different Buffer Descriptor defn for LC and PE?
Will we support Multi-Buffer Packets?
» If not, we do not need buffer_next(32b) or buffer_size(16b)







QM uses packet_next for its packet chaining in qarray.
Output Port and Input Port probably translate to TxMI and RxMI
Next Hop fields (nhid_type(4b) and nexthop_id(16b)) probably can
go away.
QOS fields (color(8b) and flow_id(24b)) probably can go away.
Two reserved fields 4b and 16b can go away.
class_id(16b) (virtual queue id?) can probably go away.
fabric_port can probably go away.
‹#› - JDD - 7/17/2015
Buffer Descriptor Usage

PE Buffer Descriptor:
» MR_ID
» TxMI
» VLAN
(16b)
(16b)
(16b)
» buffer_next
» offset
» BufferSize











Next Buffer Pointer (in a chain of buffers)
Offset to start of data in bytes
Length of data in the current buffer in bytes
header_type
8 bits
type of header at offset bytes in to the buffer
rx_stat
4 bits Receive status flags
» free_list
» packet_size

32 bits
16 bits
16 bits
output_port
input_port
nhid_type
reserved
fabric_port
nexthop_id
color
flow_id
reserved
class_id
4 bits
16 bits
16 bits
16 bits
4 bits
4 bits
8 bits
16 bits
8 bits
24 bits
16 bits
16 bits
» packet_next 32 bits
‹#› - JDD - 7/17/2015
Freelist ID
(Total packet size across multiple buffers)
Output Port on the egress processor
Input Port on the ingress processor
Nexthop ID type.
Reserved
Output port for fabric indicating blade ID.
NextHop IP ID
Qos Color
QOS flow ID or MPLS label/flow id
Reserved
Class ID
pointer to next packet (unused in cell mode)
Buffer Descriptor Usage

PE Buffer Descriptor:

Leave multi-buffer fields there as a template for the dedicated blade
implementation of a jumbo-frame MR.
» LW0:
» LW1:
» LW1:
» LW2:
» LW2:
» LW2:
» LW2:
» LW3:
» LW3:
» LW4:
» LW4:
» LW5:
» LW6:
» LW7:
buffer_next
offset
BufferSize
reserved
reserved
free_list
packet_size
MR_ID
TxMI
VLAN
reserved
reserved
reserved
packet_next
32 bits Next Buffer Pointer (in a chain of buffers)
16 bits Offset to start of data in bytes
16 bits Length of data in the current buffer in bytes
8 bits reserved/unused
4 bits reserved/unused
4 bits Freelist ID
16 bits (Total packet size across multiple buffers)
16 bits Meta Router ID
16 bits Transmit Meta Interface
16 bits VLAN
16 bits reserved/unused
32 bits reserved/unused
32 bits reserved/unused
32 bits pointer to next packet (unused in cell mode)
» Also reduces changes to Rx, Tx, and QM and reduces potential problems.
‹#› - JDD - 7/17/2015
Multicast Alternatives

At least Three Options
» Force MRs that need Multicast to be Dedicated Blade MRs and do their
own Multicast


For our short term goals this is probably sufficient and the best course.
Perhaps longer term we can look at adding it to the CRF
» Treat as exception and send to Xscale
» Provide support in CRF for Multicast


Use Multi-Hit Lookup capability of the TCAM
MI Bit mask defined in Lookup Result




Will put a bound on the number of MIs that can be supported on an MR because of the
size of the lookup result.
Has issues of mapping bits in the bit mask to actual MIs.
Lookup Result contains an index into a table containing MI bit masks
Allow but do not force MRs to provide code to interpret Lookup Result.
– This would also allow other possible extensions on an MR-specific basis
– This carries with it the problem of bounding the execution time of the MR-specific
code in the Lookup block. For general multicast, this could be a serious issue.
– There are also issues with generating a QID based on an MI when the QID is not
included in the Lookup Result.
» Other options?
‹#› - JDD - 7/17/2015
CRF Support for Multicast
Lookup
Parse
MR-1
...
MR-n
MR-Specific
Path
MR
Interp
Post Process
Default/Unicast path
Header
Format
MR-1
...
MR-n
DRAM Buf Ptr
DRAM Buf Ptr
DRAM Buf Ptr
DRAM Buf Ptr
MR Id
MR Id
MR Id
MR Id
Input MI
MR Lookup Key
Output MI
Output MI
MR Ctrl Blk Ptr
MR Ctrl Blk Ptr
MR Ctrl Blk Ptr
QID
MR Mem Ptr
MR Mem Ptr
MR Mem Ptr
Buffer Offset
QID
Stats Index
MR Specific
Lookup Result
‹#› - JDD - 7/17/2015
CRF Support for Multicast
MR-Specific
Path
MR
Interp
DRAM Buf Ptr
MR Id
MR Lookup Key
MR Ctrl Blk Ptr
MR Mem Ptr

We will need some kind of copy
count or multicast bit and last copy
bit to let TX know when it can
release the DRAM buffer that holds
the packet.
‹#› - JDD - 7/17/2015
Post Process
Lookup
Default path
DRAM Buf Ptr
MR Id
Output MI
Copy Cnt=1
MR Ctrl Blk Ptr
MR Mem Ptr
QID
Stats Index
MR Specific
Lookup Result
DRAM Buf Ptr
MR Id
Output MI
Copy Cnt
MR Ctrl Blk Ptr
MR Mem Ptr
QID
Stats Index
MR Specific
Lookup Result
CRF Support for Multicast
Lookup
MR-Specific
Path
DRAM Buf Ptr
MR Id
MR Lookup Key
MR Ctrl Blk Ptr
MR Mem Ptr

MR
Interp
DRAM Buf Ptr
MR Lookup Key
MR Specific
Lookup Result
Output MI
Output MI
Copy
Output
MI Cnt
Copy Cnt
Copy Cnt
MR Ctrl Blk Ptr
MR Mem Ptr
We will need some kind of copy count or
multicast bit and last copy bit to let TX know
when it can release the DRAM buffer that holds
the packet.
‹#› - JDD - 7/17/2015
Post Process
Default path
DRAM Buf Ptr
DRAM Buf Ptr
MR Id
DRAM Buf Ptr
MR Id
MR Id Output MI
Output MI
Copy Cnt
Output MI
Copy Cnt
Copy MR
Cnt Ctrl Blk Ptr
MR Ctrl Blk Ptr
MRPtr
Mem Ptr
MR Ctrl Blk
MR Mem Ptr
MR Mem PtrQID
QID
QID Stats Index
Stats MR
Index
Specific
StatsMR
Index
Lookup Result
Specific
MRLookup
SpecificResult
Lookup Result
OLD
 The
rest of these are old slides that should be deleted
at some point.
‹#› - JDD - 7/17/2015
Common Router Framework (CRF) Functional Blocks
Parse
Rx
RBUF
DeMux
MR-1
...
MR-n
Lookup
Header
Format
MR-1
...
MR-n
Buf Handle(32b)
QM
Tx
Buffer
Descriptor
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size

Rx:
MR_ID
» Function

Coordinate transfer of packets from RBUF to DRAM
» Notes:


TxMI
VLAN
Packet_Next
We’ll pass the Buffer Handle which contains the SRAM address of the buffer
descriptor.
From the SRAM address of the descriptor we can calculate the DRAM address
of the buffer data.
‹#› - JDD - 7/17/2015
Common Router Framework (CRF) Functional Blocks
Parse
Rx
DeMux
Buf Handle(32b)
MR-1
...
MR-n
Lookup
Buffer Offset(16b)
MR Id(16b)
Input MI(16b)
MR Mem Ptr(32b)
» Function



Tx
Buffer
Descriptor
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
VLAN
Packet_Next
Read Pkt Header from DRAM
Use VLAN from Ethernet header to determine destination MR in order to locate:


QM
TxMI
DeMux

MR-1
...
MR-n
Buf Handle(32b)
DRAM Buf Ptr(32b)

Header
Format
MR Parse code
MR specific memory pointers
Write MR Id to Buffer Descriptor
Write VLAN to Buffer Descriptor
‹#› - JDD - 7/17/2015
Common Router Framework (CRF) Functional Blocks
Header
Format
Parse
Rx
DeMux
MR-1
...
MR-n
Lookup
MR-1
...
MR-n
QM
Tx
Buf Handle(32b)
Buf Handle(32b)
DRAM Buf Ptr(32b)
DRAM Buf Ptr(32b)
Buffer
Descriptor
Buffer Offset(16b)
Buffer Offset(16b)
Buffer_Next
MR Id(16b)
MR Id(16b)
Buffer_Size
Offset
Input MI(16b)
Input MI(16b)
Free_List
MR Mem Ptr(32b)
MR Mem Ptr(32b)
Packet_Size
MR_ID

» Function




MR Lookup Key(16B)
Parse
MR-specific header processing
Generate MR-specific lookup key (16 Bytes) from packet
Need CRF functionality to managed multiple MRs in shared PE.
Notes:
» Can Parse adjust the buffer/packet size and offset?
» Can Parse do something like, terminate a tunnel and strip off an outer header?
‹#› - JDD - 7/17/2015
TxMI
VLAN
Packet_Next
CRF Wrapper Around Parse
Parse
MR
Selector
Buf Handle(32b)
MR-1
...
MR-n
Buf Handle(32b)
DRAM Buf Ptr
DRAM Buf Ptr
Buffer Offset
Buffer Offset
MR Id
DRAM Buf Ptr
MR Lookup Key
Buffer Offset
Buffer Offset
Input MI
MR Mem Ptr
Input MI
MR Mem Ptr
‹#› - JDD - 7/17/2015
MR Id
Input MI
MR Mem Ptr
MR Lookup Key
Common Router Framework (CRF) Functional Blocks
Parse
Rx

DeMux
MR-1
...
MR-n
Lookup
» Function

Perform lookup in
TCAM based on MR Id
and lookup key
» Result:




Output MI
QID
Stats index
MR-specific Lookup
Result (flags, etc. ?)

How wide can/should
this be?
‹#› - JDD - 7/17/2015
Lookup
Header
Format
MR-1
...
MR-n
QM
Tx
Buf Handle(32b)
Buffer Handle(32b)
DRAM Buf Ptr(32b)
DRAM Buf Ptr(32b)
Buffer
Descriptor
Buffer Offset(16b)
Buffer Offset(16b)
Buffer_Next
MR Id(16b)
MR Id(16b)
Input MI(16b)
MR Mem Ptr(32b)
MR Mem Ptr(32b)
Lookup Result(Nb)
MR Lookup Key(16B)
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
TxMI
VLAN
Packet_Next
Common Router Framework (CRF) Functional Blocks
Parse
Rx

DeMux
Header Format
» Function


MR specific packet
header formatting
MR specific Lookup
Result processing


MR-1
...
MR-n
Drop and Miss bits
Need CRF functionality
to managed multiple
MRs in shared PE.
» Pulls out QID, Length
and Port from MR
Result, etc.
» Checks for Drop and
Miss bits and deals with
those actions.
‹#› - JDD - 7/17/2015
Lookup
Header
Format
MR-1
...
MR-n
QM
Tx
Buffer Handle(32b)
Buffer Handle(32b)
DRAM Buf Ptr(32b)
QID(16b)
Buffer
Descriptor
Buffer Offset(16b)
Size (16b)
Buffer_Next
Buffer_Size
MR Id(16b)
Port(8b)
Offset
Free_List
MR Mem Ptr(32b)
Packet_Size
Lookup Result(Nb)
Includes drop
and miss bits
MR_ID
TxMI
VLAN
Packet_Next
CRF Wrapper Around Header Format
Header
Format
Buffer Handle
MR
Selector
DRAM Buf Ptr(32b)
MR-1
...
MR-n
Buffer Offset
Buffer Handle
QID
Size
MR Id
Port
DRAM Buf Ptr
MR Mem Ptr
Buffer Offset
Lookup Result
Output MI
MR Mem Ptr
MR Specific
Lookup Result
‹#› - JDD - 7/17/2015
Buffer Offset
Gets written to Buffer Descriptor
May also cause size(s) in
Descriptor to be updated.
(what about trimming data,
What if it is a buffer’s worth
Which would change the chaining,
Can they add/trim at either end?
Common Router Framework (CRF) Functional Blocks
Parse
Rx

DeMux
MR-1
...
MR-n
Lookup
» Function

MR-1
...
MR-n
QM
Buffer Handle(32b)
QM

Header
Format
CRF queue management for Meta
Interface queues
For performance reasons, QM
may actually be implemented as
multiple instances
» Each instance on a separate ME
would support a separate set of
Meta Interfaces.
» See next slide for more details…
‹#› - JDD - 7/17/2015
Tx
Buf Handle(32b)
QID(16b)
Size (16b)
Port(8b)
Buffer
Descriptor
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
TxMI
VLAN
Packet_Next
QM/Scheduler on Multiple MEs
Header
Format
MR-1
...
MR-n
Input
Hlpr
(1 ME)
Buffer Handle(32b)
QM/Schd
(1 ME)
...
Tx
Output
Hlpr
(1 ME)
QM/Schd
(1 ME)
QID(32b)
Size (16b)
Buf Handle(32b)
Scratch Rings
NN Ring
NN Ring
Port(8b)

QID(32b):

Input Hlpr would use QM ID to select Scratch ring on which to put request.
Output Hlpr would process all Scratch rings coming from QM/Schd MEs and
multiplex onto one NN ring to TX
With 64 entries in Q-Array and 16 entries in CAM, max number of QM/Schds is
probably 4 (2 bits).


» Reserved (8b)
» QM ID (4b)
» QID(20b): 1M queues per QM
» We’ll set aside 4 bits to give us flexibility in the future.
‹#› - JDD - 7/17/2015
Common Router Framework (CRF) Functional Blocks
Parse
Rx
DeMux
MR-1
...
MR-n
Lookup
Header
Format
MR-1
...
MR-n
Tx
QM
Buffer Handle(32b)
TBUF
Buffer
Descriptor
 Tx
»Function

Coordinate transfer of packets from DRAM to
TBUF
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
TxMI
VLAN
Packet_Next
‹#› - JDD - 7/17/2015
Old Template
Rx
DeMux
Parse
Lookup
Header
Format
QM
Buffer Handle(32b)
Reserved (24b)
Tx
TBUF
Port(8b)
Buffer
Descriptor
 Tx
»Function

Coordinate transfer of packets from DRAM to
TBUF
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
TxMI
VLAN
Packet_Next
‹#› - JDD - 7/17/2015
Old Rejected Overly Busy Slide
L1
L2
Rx
Substr
Decap
L3
L3
L3
MR
Parse
Lookup
MR Hdr
Format
L2
L2
L1
Substr
Encap
QM
Tx

Input Data
Input Data
‹#› - JDD - 7/17/2015
Block
Output Data
Output Data
Buffer
Descriptor
Buffer_Next
Buffer_Size
Offset
Free_List
Packet_Size
MR_ID
TxMI
VLAN
Packet_Next
 Logical interface
» Data passing between
layers

Notes here

 Physical format
» Actual Format of data
» Shows type of
communication




Scratch Ring
NN Ring
SRAM Rings
 Buf Descriptor
» shows fields
read/written

similar documents