Interconnection Simulation

LegoSim applies Popnet as the interconnection simulator to provide the latency of each transaction. Popnet simulates the interconnection behavior according to the benchmark file provided by Phase 1. Popnet generates one latency information file, providing the latency of each package.

The latency of packages is related to the source-destination address and the network congestion. The source-destination address does not change crossing iterations. The same degree of network congestion leads to similar latency information. Hence, the latency information generated by Phase 2 can be used in Phase 1 of the next iteration.

Each pair of WRITE and READ commands generates one transaction in the interconnection. In particular, WRITE commands with LOCK, UNLOCK, and BARRIER descriptors generate transactions in the interconnection without pairing READ commands.

File format

LegoSim uses the trace file and the latency information file to communicate with Popnet.

Trace file format

The trace file is one text file. Each line presents one transaction in the interconnection, as shown below.

<src_cycle> <dst_cycle> <src_x> <src_y> <dst_x> <dst_y> <flit_num> <desc>

Fields in the trace file are filled by the value from the WRITE and READ commands.

Field in trace file	Field in WRITE command	Field in READ command	Description
`src_cycle`	`cycle`		Start cycle of the transaction from the source's view.
`dst_cycle`		`cycle`	Start cycle of the transaction form the destination's view.
`src_x`, `src_y`	`src_x`, `src_y`	`src_x`, `src_y`	Source address.
`dst_x`, `dst_y`	`dst_x`, `dst_y`	`dst_x`, `dst_y`	Destination address.
`flit_num`			Flit number. \(\text{ceil}(nbytes / payload\#) + 1\).
`desc`	`desc`	`desc`	Behavior of transaction.

The trace file is generated by interchiplet. interchiplet always keeps packages in the order of the package injection cycle. For example,

2846470 0 0 0 0 1 1251 0
2847814 0 0 0 1 0 1251 0
2849309 0 0 0 1 1 1251 0
2850905 2847725 0 0 0 1 1251 0
2852501 2849069 0 0 1 0 1251 0
2854098 2850569 0 0 1 1 1251 0
2875272 2855527 0 1 0 0 14 0
2876868 2875644 1 0 0 0 14 0
2878470 2877240 1 1 0 0 14 0

TODO: flexible format for address. Flexible format for different kinds of PComps and SComps.

Latency information file format

The latency information file is also one text file. Each line provides the latency information for one transaction in the interconnection, as shown below.

<cycle> <src_x> <src_y> <dst_x> <dst_y> <desc> <lat_num> [<lat_0> <lat_1> ...]

cycle presents the start cycle of one transaction from the source's view as the src_cycle field in the trace file. src_x, src_y, dst_x, and dst_y present the source and destination address as the trace file. desc describes the behavior of the transaction as the trace file.

For each transaction, latency information provides the latency of each package related to one transaction. For example, normal communication transactions only provide the latency of one package. BARRIER/LOCK/UNLOCK/LAUNCH transactions provide the latency of one request package and one acknowledge package.

lat_num presents the number of provided latency values. Each package has two latency values. lat_{2i} and lat_{2i+1} are the latency of the i-th package from the source's and destination's views.

Transaction's behavior	`lat_num`	`lat_0` and `lat_1`	`lat_2` and `lat_3`
Normal	2	Normal transfer package
LAUNCH	4	Request package	Acknowledge package
BARRIER	4	Request package	Acknowledge package
LOCK	4	Request package	Acknowledge package
UNLOCK	4	Request package	Acknowledge package

Popnet does not need to sort packages. interchiplet will reorder packages according to the package injection cycle when loading files. One example of latency information files is shown below:

2847106 0 0 0 1 0 2 1250 1255
2848612 0 0 1 0 0 2 1250 1255
2850107 0 0 1 1 0 2 1250 1260
2851703 0 0 0 1 0 2 1250 1255
2853299 0 0 1 0 0 2 1250 1255
2854896 0 0 1 1 0 2 1250 1260
2876066 0 1 0 0 0 2 13 18
2877662 1 0 0 0 0 2 13 18
2879259 1 1 0 0 0 2 13 23

TODO: flexible format for address. Flexible format for different kinds of PComps and SComps.

Sorting Latency Information

interchiplet needs to rebuild the order of transactions according to the latency information file.

Transactions are organized by a two-level structure. The first level is a map of addresses and ordered queues of latency information. The second level is one queue of latency information ordered by cycles.

	Ordered content	Key of map	Key for ordering
Order of Transaction	All latency informations	Source addresses	Transaction start cycle
Order of Launch	Latency information of Launch transactions	Destination addresses	Cycle when the request package arrives the destination
Order of Lock and unlock	Latency information of Lock/unlock transactions	Destination addresses	Cycle when the request package arrives the destination

TODO

The network latency is provided by Phase 2 of the previous iteration. interchiplet will load all delay information before starting the simulation processes in Phase 1. When it receives the paired READ and WRITE commands, it will search for the first delay information message with the same source and destination. The matched delay information message will be dropped after use.

There is no implicit guarantee that the number and the order of transactions do not change crossing the iteration. Because the simulation flow is based on iteration, the difference crossing iteration should reduce as the simulation continues.

If the network delay from SComps is missing, the network delay only considers the propagation delay, which equals the data amount divided by network bandwidth. In the first iteration, the network delay is determined in the same way.