# Co-Packaging of Optics for HPC (High-Performance Computing) and Datacenters

https://www.youtube.com/watch?v=eDdFtvokijU

[00:00] this came about.
[00:04] this came about in part because of what was probably a chance meeting uh at Ecock I think.
[00:16] Elijah came up to me and asked if I'd be willing to do it so it's one of the benefits of attending in-person conferences I think.
[00:24] but uh so I'm gonna um give this seminar on on co-packaging of Optics in particular for high performance Computing and data centers.
[00:36] and then throughout this talk I'm going to kind of show the Highlight the differences that now exists between HPC and data centers.
[00:49] so the outline will be.
[00:52] I'm going to talk about you know what are the driving forces for HPC and data centers HPC being mostly driven by
[01:01] governments data centers
[01:03] mostly driven by ethernet switches
[01:07] I give some specific examples and then
[01:09] show how HPC is different from Data Center
[01:13] um and then and with a focus on on the optics for both of these uh and then I'm going to go and talk about uh various packaging form factors
[01:24] um for the the two types of machines
[01:30] um I'll start with high performance Computing
[01:32] high performance Computing has been tracked by
[01:37] the uh top 500 uh dot org list and and
[01:42] this is their
[01:44] um they're tracking over a
[01:49] uh like almost a 30-year period now 2023 would be a 30-year period
[01:56] um and so what they're plotting is they're plotting the performance of the the number one machine and then the
[02:03] plotting the performance of the number 500 machine um and we have a period here up to about 2008 where all of the high performance machines had no fiber optics in them whatsoever and then starting with 2008 IBM started using fiber and they had the first petaflop machine so this was the IBM Roadrunner it had 40000 fibers this was all vixel based running at uh five gigabits per second per fiber um and then uh as the years progress in what I'm calling the fiber error um the various number one machines kind of go in and out of using fiber like the Japanese K Machine um that was all copper but then you know it was followed by Blue Jean Q Sequoia and then also I put in question marks some of the Chinese machines because
[03:04] they didn't really publish we know they were using fiber optics but they didn't really publish clearly what it was.
[03:12] um cray cray tends to use all copper but they'll use Fiber when they have a government-sponsored program.
[03:22] um and then uh lately you know we had the IBM Summit machine and then this year we have some extra scale machines.
[03:33] um but I sort of put this in a double asterisks because um because of the tensions with China.
[03:39] um the Chinese basically stopped reporting their high performance Computing progress a number of years ago.
[03:49] so China actually might have gotten to exascale First there's some people who think they did but we just don't know because of this tension they they just said hey we're not going to participate anymore.
[04:03] um HPC historically drove new Optical
[04:06] Technologies it was the big you know sort of gorilla in the room for a while.
[04:12] um it was the first to deploy active optical cables the first to deploy co-packaging.
[04:18] um it's primarily driven by governments.
[04:21] um this exponential 25-year exponential trend is not uh slowing down.
[04:28] and it so what you find for the i o is a mixture of infinity band ethernet and proprietary solutions.
[04:37] um a bit about Infinity band.
[04:40] um infiniband has now a bunch of specifications uh going all the way up to 400 gig.
[04:48] um I think the 200 gig and the 400 gig are still work in progress.
[04:52] a number of years ago they stopped specifying the optical interface so basically Infinity band specifies an electrical interface and they use active optical cables most likely those have.
[05:07] vixels inside but in with an active optical cable anything could be inside
[05:13] what makes Infinity Band popular is it has much lower latency than ethernet and that's very important for high performance computing
[05:27] so ethernet I think the ethernet roadmap is familiar to a lot of people but uh ethernet right now has got 800 gig
[05:37] some people are probably even shipping 800 gig prototypes right now certainly 400 gig is shipping they're working on 200 gigabit electrical signaling uh and of course Optics would follow that
[05:51] and then the Beyond 800g there's a 1600 G
[05:59] the for ethernet it's sort of performance is Paramount
[06:04] um power consumption is is not has not
[06:08] traditionally been a very big consideration within the ethernet space.
[06:12] I think now it's starting to be um but this is this is the road map and and this road map seems to be going.
[06:21] It was going in factors of 10 at one point but now it's going in like factors of two.
[06:27] um so this is a big eye chart of all the different kind of ethernet standards and ethernet form factors and.
[06:40] um I'm not expecting anybody to sort of take away anything from this other than what I want to point out is that it's a mixture of copper and vixel solutions and then in this this other space there's a wide variety of Technologies directly modulated lasers externally modulated lasers silicon photonics uh lithium nib a thin film lithium niobate there's there's other sorts of Technologies now.
[07:10] being sort of mixed into this space.
[07:14] ethernet has gone parallel.
[07:17] while the electronics catches up and and then primarily they switch back to serial.
[07:23] but there's a number of of 16 lane four-lane and uh and then multi-wavelength solutions in in the single mode space.
[07:34] foreign is somewhat driven by the the merchant uh switch silicon.
[07:47] and uh here's sort of a map of what the merchant ethernet switch silicon is doing over time but basically every two years there's been a doubling of performance so we're right now here in 2022 and broadcom just released their
[08:10] 51 terabit Tomahawk 5.
[08:14] Um switch uh certainly 25.6 has been shipping for a while now.
[08:20] People are working on the 100 terabit switch versions.
[08:24] Um I've also plotted the power consumption and bandwidth on a log scale versus time.
[08:31] They're both scaling exponentially.
[08:37] And then of course what's concerning is the power consumption.
[08:41] I have drawn in sort of this dotted line which is a um uh an estimate of like sort of the limit for air cooling a single chip module.
[08:56] Um they've gone way past that and so what you're going to see is a lot of water cooled systems now to handle these high powers.
[09:05] I don't think I updated this chart but maybe I did Tomahawk 5 is a 700 watt.
[09:13] solution so maybe that's like one of these data points here.
[09:20] um the other thing that we're seeing in the sort of ethernet industry is the time it takes to develop a new 30s is shortening.
[09:29] um and so you know 10 gig 30s had sort of a a four gig four year run 25 gig 30s maybe a three year run maybe similarly for the 50 gig 30s and now people are working on 100 gigs 30.
[09:51] so um right now in our time frame we've got a mixture of probably 10 25 50 and 100.
[09:57] um and even people are working on 200 gig 30s.
[10:01] um so things are things are accelerating.
[10:05] this is another chart I got from from Andy uh bechtelstein at Arista it's just another way of saying the same thing.
[10:13] that the time between these silicon switch speed uh 30s is is growing shorter.
[10:25] so at the same time I want to look at fiber cabling because you hear a lot about data centers you know data centers are now the sort of 800 pound gorilla in the room driving a lot of of specifications and in particular you know saying that that their Solutions need to be sort of 500 meters to 2000 meters.
[10:49] but we took a look at at what they're actually ordering.
[10:54] um and the thing that's kind of striking and these are sort of histograms of cable length um in in data centers is that they're buying a lot of really really short cables.
[11:08] um here's a histogram from one supplier uh representing over 600
[11:15] um duplex and trunk cables
[11:18] more than ten thousand kilometers of
[11:20] fiber and 92 percent of these were all
[11:23] less than 28 meters so you know asking
[11:26] for
[11:28] you know a single mode solution and then
[11:31] deploying it at 28 meters is probably
[11:33] not economical uh I wouldn't say that
[11:36] all data center operators do that there
[11:39] are some that that will have a mixture
[11:42] of copper and multi-mode
[11:45] and single mode in the same Data Center
[11:48] um and and
[11:50] sort of what you hear them say in the
[11:52] Press
[11:54] um and where they Place their purchase
[11:56] orders are are not are not the same
[11:59] thing so a lot of short cables still
[12:02] being used in data centers
[12:04] and you talk about a couple of IBM uh
[12:08] high performance
[12:10] Computing systems that did use Optics
[12:13] this is our blue jean Q Sequoia system
[12:16] uh it was number one in um in 2012 so um it's 10 years old.
[12:25] it would have been 10 years old except it got decommissioned in 2020.
[12:30] and one thing I want to say about high performance Computing is uh every four years you can get a machine that's about 10 times the performance at the same power that you sort of expended and and so you know to keep a machine running when you could get something 10 times faster uh just doesn't make economic sense.
[12:54] and I would say that this machine probably would have been decommissioned you know 2018 or or so but for for other reasons it was it was kept running longer.
[13:07] it had an eight-year life um this machine had 620000 uh vixels and fibers and here's a
[13:16] distribution of the fiber links this is a picture of the entire machine room.
[13:22] by the way this is I think approximately 10 meters by 30 meters.
[13:28] um and and this entire system was cabled up.
[13:32] the longest cable was 22 almost 23 meters.
[13:35] we had a huge spike in cables at two meters so this is also kind of a consequence of using a 5D Taurus topology.
[13:46] um here's another example this is our our Summit IBM Summit system.
[13:56] um this was number one uh like in 2019 I guess 2018.
[14:04] um and uh today it's dropped to number four but its performance is is a factor of eight behind the number one machine.
[14:13] um this was a 200 petaflop machine it was water cooled so you can see these
[14:21] Copper water heat spreaders.
[14:29] It used about eighty thousand uh vixels and multimode fiber.
[14:31] This is the the distribution of cable lengths and what I'm showing in green are fiber cable lengths and then in Red.
[14:39] So where is copper red is copper so where everything that was two meters or less we did in Copper that's for cost reasons.
[14:47] Uh originally we thought we could cable the entire system up with just 30 meter cables but we would actually reach the floor we had to get some cables that were 50 meters.
[15:00] But the message here is you know number one HBC system put together with with very short cable links.
[15:08] And then moving past uh IBM programs um we have the uh in the U.S we have the Coral 2 program.
[15:17] And so the frontier system this is the
[15:19] current number one machine uh it came online this year.
[15:22] it's 1.7 exaflops.
[15:26] It's a combination of AMD CPUs and gpus.
[15:29] it uses the create slingshot interconnect.
[15:33] and it is water cooled and it costs the government about 600 million dollars.
[15:39] Aurora is an Intel machine that is I believe it's being assembled right now.
[15:46] it may power on in December of this year or or very early next year.
[15:53] it's being installed at Argonne National Labs.
[15:56] um and then El Capitan uh it's another AMD system uh similar to the frontier.
[16:02] it'll be greater than two exaflops.
[16:05] and that'll come online on 2023.
[16:10] um what I'm showing here is several of these have publicly publicly announced that they are water cooled.
[16:18] um and so just a statement about water.
[16:21] um this is from Alan Benner but water cooling has a tremendous number of benefits.
[16:27] um one of the big benefits is that it's super quiet.
[16:31] if you've ever walked into a data center that's air-cooled you know you have to uh put on ear protection.
[16:38] it's it's just it's dangerous because of the acoustic levels in these machine rooms.
[16:42] uh water is very quiet.
[16:45] um it takes up a lot less uh space.
[16:49] um and you can put like entire data centers or HPCS in very tiny rooms like you can tuck them away in a closet for example.
[16:59] um the HPC Community has been using water cooling for more than 10 years now.
[17:03] and data centers are are slowly moving the water cooling.
[17:10] um I want to talk about you know the the sort of challenges that HPC has especially for optical interconnects.
[17:19] um and and going forward.
[17:22] we have this sort of rough rule 15 rule which says that our HBC customers are willing to pay 10 to 15 percent of the system cost on their interconnect.
[17:34] and so you know for a 400 million dollar system that means they'll spend 60 million on the interconnect.
[17:43] um and and you can go through all the math and project where you need to be.
[17:47] but the bottom line is um you know future Optics need to for HPC need to be in sort of less than the 15 cent per gigabit range.
[18:00] um and even lower for 100 gigabit signaling.
[18:03] and right now Optics is sort of in the dollar per gigabit range.
[18:08] um and and so that's that's problematic for for HPC systems.
[18:13] uh moving forward.
[18:16] another sort of issue that HPC faces is power consumption.
[18:23] um so most of these machines are built
[18:27] you know even Frontier that I showed you
[18:31] 've got sort of a power envelope of sort
[18:34] of less than 30 megawatts
[18:37] um and again if you allocate 15 of that
[18:40] for the network and then you make some
[18:43] assumptions uh about scaling It Forward
[18:46] uh what what you get is that Optical
[18:50] links will need to be a Pico Joule per
[18:52] bit
[18:53] and today's commercial devices are are
[18:56] sorted in the 10 Pico Joule per bit
[18:59] range right now so we're sort of
[19:01] um we're at a sort of a disconnect uh in
[19:05] energy as well as
[19:07] um in cost
[19:10] so let me talk about uh Optics now uh
[19:14] packaging for HPC and data center
[19:19] um you know what we see mostly in the
[19:22] industry today are are pluggable Optics
[19:25] and so we've got a switch Asic or it could be a processor and an optical module be at the far end of the board be externally pluggable.
[19:35] um there's Kobo that was sort of putting Optical modules in the middle of the board that didn't really catch on.
[19:42] um co-packaging is is the main topic of this talk and this is what a lot of people are working on is where we move the Optics right onto the first level package.
[19:54] Beyond co-packaging.
[19:56] um we see sort of optical switching so companies like IBM and Google have had research efforts into all Optical switching.
[20:04] it's um not a topic of this talk.
[20:10] uh so Kobo.
[20:13] um the Consortium for onboard Optics.
[20:17] they came out with.
[20:20] so at the time Koba was being worked on people didn't think they could get much.
[20:26] power into a plugable module they didn't
[20:28] think they could do it
[20:30] um they didn't think they could get the
[20:31] high speeds to it and so they said you
[20:34] know as a compromise we can put these
[20:37] modules out in the middle of the board
[20:39] and they can be larger and they can have
[20:41] higher power dissipation
[20:44] um and and so what they did is they came
[20:45] up with a a rather large number of
[20:48] specifications for both 8 and 16 Lane
[20:51] and different Power consumptions the
[20:55] thinking was that coherent needed
[20:58] coherent does need more power but it
[21:00] needed to go into a larger module
[21:02] although now I will tell you that people
[21:04] have are putting coherent into plugables
[21:07] and and the need for the for Kobo type
[21:11] modules
[21:13] has really gone way down
[21:16] um
[21:18] and then I didn't have a chart on
[21:19] pluggables but I just assumed that the
[21:21] audience knows uh about pluggables
[21:25] um
[21:26] so let's talk about co-packaging and and
[21:29] why we're why why you should be
[21:30] interested in co-packaging and why we're
[21:32] interested in it
[21:34] um this is primarily to increase
[21:36] bandwidth coming from the A6 so
[21:39] the packages for large A6 are pin
[21:42] constrained if you can take i o off the
[21:45] top of the package you've basically
[21:47] broken a constraint and you can get more
[21:50] bandwidth out of your Asic
[21:53] when you put the Optics right next to
[21:56] your Asic it doesn't need all that
[22:00] electrical power it doesn't need very
[22:01] high voltage swings that are needed to
[22:03] overcome like 30 DB cable loss you can
[22:07] greatly reduce the
[22:09] power consumption by reducing the
[22:11] voltage swing maybe even taking out some
[22:15] cdrs
[22:17] um
[22:19] this is you should get a reduction in
[22:21] cost from co-packaging because basically
[22:23] you're stripping down the optical uh
[22:26] packages and you have a reduced function
[22:29] in the ICS and then you can also if
[22:32] you're going to have less powerful i o
[22:34] cells you can reduce your Asic area
[22:37] reducing your Asic area always increases
[22:39] your yield
[22:41] but instead of reducing the Asic area
[22:44] you might choose to use your extra area
[22:47] for other functions so add more i o
[22:50] cells or add more memory hubs for
[22:53] example the other thing about
[22:55] co-packaging is it reduces a human
[22:58] handling
[23:00] um
[23:01] we had a guy from Microsoft say that the
[23:05] number one reliability problem that they
[23:06] had in data centers was human handling
[23:09] of pluggables so all of our efforts to
[23:12] make lasers more reliable
[23:13] uh
[23:15] kind of went to knot when the humans
[23:17] break everything and co-packaging
[23:20] somewhat relieves you of that
[23:25] from an electrical
[23:27] the the electrical interface is really
[23:29] important to understand
[23:32] uh one aspect of co-packaging and that's
[23:35] how you save uh energy and I'm just
[23:38] using this this fairly old chart from
[23:41] the oif on their 56 gig uh common
[23:45] electrical interface specs but they had
[23:48] five different specifications from Ultra
[23:51] short reach to Long Reach
[23:53] and and
[23:55] um VSR and Long Reach were sort of the
[23:58] two popular ones for ethernet
[24:01] but if we use the ultra short reach you
[24:03] know sort of a 2db loss interconnect as
[24:06] a baseline uh what I've done is as I've
[24:10] sort of shown in red how the energy goes
[24:13] up for the various lengths so as as the
[24:15] loss of the electrical link goes up
[24:18] The energy needed to overcome that loss
[24:20] goes up as well and and the power
[24:23] consumption overall goes up and so this
[24:25] is
[24:26] you know
[24:27] one way to illustrate the benefits of of
[24:30] how co-packaging can reduce your energy
[24:32] if
[24:34] you're willing to get somebody to design
[24:35] a USR or an xsr interface for you and
[24:39] that's that's a big if a lot of people
[24:42] want to do a one-size-fits all uh chip
[24:45] and and when they do that they go for
[24:48] VSR or LR so you don't quite get the
[24:50] benefits
[24:52] um
[24:53] that you could get
[24:55] um another aspect of large Asic
[24:58] packaging and and I try to show this
[25:00] down here
[25:02] um here's a whole bunch of examples of
[25:04] single chip and dual chip and multi-chip
[25:07] packages
[25:10] um chip scale packages
[25:12] um
[25:14] even as the technology node is improving
[25:19] um and and so the IC pad pitch is going
[25:21] down as the technology node improves
[25:25] which means that the number of pads on
[25:27] the chip goes way up
[25:29] um but the number of pads on the package
[25:32] doesn't doesn't go up that much and so
[25:36] there's this sort of five to one ratio
[25:38] between pads on your IC and and and pads
[25:43] on your package and so that's one issue
[25:45] and that's where if you can sort of
[25:47] break this by taking i o off the top as
[25:51] well as off the bottom
[25:53] I can get some benefit there
[25:56] so I'm going to talk about uh an IBM
[26:01] system
[26:02] uh it's called the power 775 it's
[26:04] something we shipped
[26:06] in 2010 this is the very first system
[26:09] very first commercial system to actually
[26:11] use co-packaging
[26:13] you might get the impression
[26:16] um that um
[26:18] you know co-packaging is a new thing but
[26:21] uh it's not it's it's over
[26:24] um uh 10 years old
[26:27] um
[26:29] and
[26:31] um so this drawer uh this is about one
[26:33] meter by 1.8 meters it was water cooled
[26:37] uh in the center you had all your
[26:40] processors on either side we actually
[26:42] had water cooled dims
[26:45] um so we packed the memory as tight as
[26:47] we could and then we had these switch
[26:49] chips
[26:50] um these were
[26:51] um
[26:52] uh uh custom switch chips and then we
[26:56] surrounded all these switch chips with
[26:57] Optics so I'll go take a closer look at
[27:00] that
[27:02] um
[27:04] so here's here's a blow up of our our
[27:06] sort of switch ceramic with this large
[27:08] switch here and we surrounded it uh with
[27:12] 56 of these Optical modules that we
[27:15] co-developed with avago this is called
[27:18] the micropod these things were about
[27:21] nine millimeters on a side and and we've
[27:24] got a sub millimeter Gap
[27:27] um in between them so packed them very
[27:29] tightly
[27:31] um we packed them so tight that the
[27:32] fibers had to exit at a slight angle
[27:35] they couldn't exit
[27:38] parallel to the to the ceramic or they'd
[27:41] hit the module in front of it so we had
[27:43] all these stacked uh fiber ribbons like
[27:46] this
[27:47] um one thing I want to point out is even
[27:50] though we had socketed these things and
[27:53] put on a water cooling plate these were
[27:55] not field replaceable
[27:57] um and so our strategy you know if the
[28:00] lane went down it was a fail-in-place
[28:02] strategy and we had a redundancy
[28:07] um in in all of all of our links we had
[28:10] spare links uh in 2010 we bought about
[28:15] half a million of these and we did drive
[28:18] the price down to a dollar per gigabit
[28:20] and sort of that's the first
[28:22] demonstration that that co-packaging can
[28:24] reduce the cost of Optics
[28:29] um excuse me yeah how much redundancy is
[28:32] required to meet the lifetime of this
[28:34] thing what uh what is the expected
[28:36] Lifetime and how much redundancy you
[28:38] need to cover that lifetime
[28:42] so these things uh
[28:45] IBM systems or
[28:48] a lot of times we look at 60 000 hour uh
[28:52] Power on uh life so that's that's six
[28:55] years other times we look at 10 years we
[28:58] don't really look uh Beyond 10 years
[29:00] because
[29:01] um
[29:03] Beyond 10 years or even Beyond six years
[29:06] the technology as I showed before is
[29:08] moving so quickly it's really obsolete
[29:10] by then on and it's not energy efficient
[29:14] to keep keep system like that powered on
[29:17] um what we did so we had
[29:21] um we had these d-links d-links were 10
[29:25] channels wide so they were 100 gigabit
[29:27] but we implemented a D-Link as 12 wide
[29:31] so basically a D-Link had two spares
[29:35] um LR links were half as wide so an LR
[29:38] link nominally was five channels wide
[29:42] but it was half of an optical module so
[29:45] it had one uh spare Lane
[29:48] and where does the electronics sit
[29:51] relative to these it it sits right here
[29:55] Tia and the driver RX Tia for the
[29:59] optical modules yeah they're they're
[30:01] completely inside these things uh don't
[30:05] know
[30:06] if I have a chart showing the breakout
[30:09] of that
[30:10] um
[30:12] yeah they're completely inside this this
[30:15] little thing little nine by nine
[30:17] millimeter thing
[30:19] uh not sure if I have a chart showing
[30:22] how that broke out
[30:24] one one thing I want to talk about with
[30:26] lgas and this comes up a lot in
[30:28] co-packaging the question is
[30:30] do I sock it or do I solder now in this
[30:34] very first imp instant instantiation we
[30:39] went with the socket but the socket
[30:41] introduces a lot of problems it
[30:44] introduces mechanical issues introduces
[30:46] thermal issues so so here's our LGA for
[30:51] all of our Optics
[30:53] um the optical modules sit on here and
[30:55] then to get the heat out we put what we
[30:57] call a saddle so the saddle sat on top
[31:01] of the Optics so the heat came out of
[31:03] the Optics through the sides we had a
[31:06] first layer of thermal interface
[31:07] material we had a second layer of
[31:10] thermal interface material we had this a
[31:13] giant heat spreader which has some
[31:15] initial springs for pre-loading the LGA
[31:19] on onto this Frame
[31:22] um but this Frame also had a giant LGA
[31:25] underneath it for all the electrical
[31:29] connections so there's massive LGA
[31:31] underneath and then we had the cold
[31:34] plate way up here on top of this
[31:37] pre-loaded thing with a third layer of
[31:40] thermal interface material and that
[31:42] actuated everything at once so so
[31:46] sockets might seem like a good idea but
[31:50] they make the thermal path and the
[31:51] mounting more challenging as speeds go
[31:55] up uh sockets present a signal Integrity
[31:58] issue as well
[32:01] um
[32:04] I showed you that oif chart on the
[32:07] various standards of USR xsr and VSR and
[32:12] we had this very large ceramic substrate
[32:14] it had 89 layers
[32:18] um and our switch Asic was mounted here
[32:21] but only six of our modules could sort
[32:23] of meet the USR spec the bulk of them
[32:26] were xsr
[32:28] and then the corner ones they were far
[32:30] enough away
[32:32] um some of them were six centimeters
[32:34] away so this is I'm showing you like a
[32:36] straight line
[32:37] four centimeter distance but but when
[32:39] you route it out and you go through all
[32:42] the different layers
[32:44] ended up with like six centimeters so
[32:46] it's very tricky
[32:48] uh to meet a USR specification or even
[32:51] an xsr specification on on large
[32:54] laminates when you when you finally sit
[32:57] down and Route things out
[33:01] um I'm gonna sort of switch gears here
[33:04] now and talk about
[33:06] um the project I'm working on in
[33:08] research as a government sponsored
[33:10] project by
[33:12] depart U.S department of energy they're
[33:16] um
[33:16] Advanced research projects Agency energy
[33:20] part and this is work that IBM has
[33:24] teamed up with two six and if you
[33:26] followed the news 2-6 acquired coherent
[33:28] and decided to change their name to
[33:31] coherent
[33:33] but it's really it's 2-6 finaster
[33:39] um so so motion is a co-packaging
[33:41] project
[33:42] and you know one of the reasons IBM is
[33:45] interested in it is we feel that
[33:47] co-packaging
[33:49] can alleviate a lot of bandwidth
[33:51] bottlenecks
[33:52] um so I kind of already talked about
[33:55] um this one here where Optics on this
[33:58] next to the CPU or GPU can get you
[34:02] um higher bandwidth
[34:05] um but also co-packaging up into
[34:08] ethernet switches uh can enable higher
[34:11] Radix switches give you a flatter
[34:13] Network
[34:15] um and then you can also disaggregate
[34:17] things such as memory uh or connections
[34:21] to accelerators for example so a lot of
[34:24] good reasons for for pursuing uh
[34:27] co-packaged
[34:28] Optics in particular because IBM IBM is
[34:33] a data center company but we're also a
[34:35] server company so we have a focus on
[34:39] um doing co-packaging to CPUs
[34:42] our program has uh two phases phase one
[34:46] has has been completed uh it developed
[34:49] this module
[34:52] um which was 56 gigabit nrz
[34:55] uh error free with without feck
[34:59] um and and one of the reasons why it has
[35:00] an nrz interface is because CPUs and
[35:03] gpus uh still have nrz interfaces and
[35:07] and CPUs of yet to you know even reach
[35:10] sort of 50 gigabit nrz signaling
[35:14] um another reason for pursuing nrz is is
[35:18] we can be error free without FEC and so
[35:21] nrz is much lower latency than than uh
[35:24] pam4
[35:25] uh one of the things that we're doing
[35:27] differently than than the prior project
[35:29] that I showed you from 10 years ago is
[35:32] is we're soldering now uh onto the first
[35:35] level package this sort of the feeling
[35:37] is that we could always fall back to
[35:40] sockets if we needed to
[35:43] um soldering is a harder thing to do and
[35:45] want to work it out because we feel that
[35:48] in the long run this is the preferred
[35:51] solution
[35:52] um because we're sponsored by rpe
[35:55] there's an energy goal an aggressive
[35:56] energy goal which was less than uh four
[35:59] picajoules per bit
[36:02] so let me go so this is one where I
[36:05] actually have details on the package
[36:07] um
[36:08] this whole package here is built up on a
[36:11] glass substrate
[36:13] we take our transmitter and receiver ICS
[36:17] which are us
[36:18] um a city by CMOS process and we flip
[36:22] chips solder those to this glass along
[36:25] with
[36:26] um 16 photodiodes and well 32 pixels
[36:33] um there's a heat spreader which we call
[36:34] the Keel underneath it carries the heat
[36:37] out to the sides to this copper heat
[36:39] spreader
[36:40] um and then
[36:42] um
[36:43] there's a first lens and then there's
[36:46] there's a second lens here and some
[36:48] turning mirrors and a special ferrule
[36:51] that plugs in
[36:53] um
[36:57] we in this project since we're teamed up
[37:00] with finisar uh finisar did a market
[37:03] study
[37:04] and it kind of came back with a funny
[37:07] result which was
[37:09] um hey IBM you're the only company
[37:11] that's willing to do soldered uh all the
[37:14] rest of our customers are too too
[37:17] concerned to do about that and they'd
[37:19] like a socketed version so we
[37:22] uh we do have a way to to build this
[37:25] part with both we can attach this inner
[37:28] poser for soldering or we have this
[37:31] spring set us removable springs for a
[37:34] socketed version so we're we're pursuing
[37:37] both
[37:39] um our architecture is is very simple uh
[37:42] it's just basically bits in bits out
[37:44] there's no cdrs it's it's Broadband it
[37:47] can operate sort of from zero up to 56
[37:51] um gigabit nrz uh because we stripped a
[37:54] lot of stuff out we have a nickname for
[37:57] these ICS we call it safe simplified
[38:00] analog front end
[38:03] um it is 16 channels wide it does have
[38:06] um built-in
[38:08] um prbs and air detectors
[38:12] um
[38:13] and it operates off of two power
[38:15] supplies uh 1.8 and 3.3
[38:19] um
[38:20] some of the pushback that we got from
[38:22] our server uh group is they only wanted
[38:26] it to be one power supply and and I
[38:29] thought two was pretty good
[38:32] the the reason we have two is we get a
[38:35] lot of power savings from using the 1.8
[38:38] um uh we need the 3.3 for the Optics but
[38:42] I also point out that the oif CPO
[38:44] specifications have seven power supplies
[38:48] so quite quite different there
[38:51] um here's a die photo of the transmitter
[38:54] I see it's about 1.6 by 4.6
[38:58] millimeters the receiver photo looks
[39:01] very similar to this here's a breakdown
[39:04] of of the current draw from the various
[39:08] voltage domains and and you can see what
[39:11] I said we've we've talked a lot of
[39:13] function into the lowest voltage domain
[39:16] and that helps us reduce the energy
[39:19] consumption which is quite important to
[39:21] us
[39:23] uh the vixels and photodiodes are are
[39:27] designed by finisar now coherent
[39:31] um
[39:32] they're 940 nanometer vixels
[39:35] um and here's like a representative 56
[39:38] gig I and 112 gig pam4i
[39:43] um
[39:44] we put down a raise of four but actually
[39:47] what we're doing is we're putting down a
[39:49] raise of two by four for redundancy and
[39:52] I'm going to say something on the next
[39:54] slide but basically in this project
[39:57] um because we're soldered and and for
[39:59] other reasons we actually have one for
[40:01] one laser sparing
[40:03] um the photo diode looks like this so
[40:05] there's a high speed portion with a very
[40:07] tiny photo diode and then there's a
[40:09] monitor diode function which is a larger
[40:12] area
[40:15] um so we did do a reliability
[40:17] calculation
[40:19] um in this particular calculation we
[40:21] looked out to a 10-year period
[40:24] um
[40:25] our our IBM server like to have less
[40:29] than 200 PPM
[40:31] uh at the end of 10 years and so you
[40:34] know without sparing there's no way we
[40:36] would get there
[40:38] um but with sparing uh we can get to
[40:40] about 100 PPM at the end of 10 years so
[40:43] you have a period here
[40:45] where
[40:47] um your fixed fit components are
[40:50] dominating and then laser wear out
[40:52] starts to kick in noticeably around
[40:54] five-year time frame and this is where
[40:56] your spares are helping you out
[41:00] um and this is this is how we
[41:02] implemented it so
[41:04] um I like to say it's the equivalent of
[41:06] having four spare tires in your trunk
[41:08] right you've got a spare for every every
[41:10] Tire
[41:13] um
[41:13] one of the sort of interesting things
[41:16] about this was
[41:18] um
[41:19] and we didn't think about it at the time
[41:20] but our server group guys came back and
[41:22] said hey how quickly can you switch
[41:24] these spares
[41:26] and we said what do you mean they said
[41:29] well can you get to hit list sparing
[41:30] like like the system doesn't even know
[41:32] you know you've detecting that a laser
[41:35] is on its way out you flick in the spare
[41:37] the system doesn't know
[41:40] so we didn't design it for that but we
[41:42] did characterize it and what we found is
[41:45] right now it takes about 100 nanoseconds
[41:49] um before you know you have an open eye
[41:51] over here and the eye is open again over
[41:53] here
[41:54] um
[41:56] having looked at this we realized
[41:58] you know one of the reasons for the the
[42:02] 100 nanosecond time was that all of the
[42:05] current sources for the spare Channel
[42:08] um needed to be charged up they all had
[42:10] capacitors in their uh for their current
[42:12] sources to stabilize them and it took
[42:15] some time so we feel like you could in a
[42:18] different design get to like a Hit List
[42:20] sparing but it wouldn't be necessarily a
[42:23] low power consumption point
[42:27] our phase two of this program yeah
[42:31] um so uh yeah at any given time there's
[42:34] only light in half of the fibers or
[42:36] somehow you couple like from two pixels
[42:39] into one fiber yeah that's exactly what
[42:42] we do
[42:43] um each pixel is coupled uh into one
[42:46] fiber
[42:48] um what's not shown here or what's not
[42:50] mentioned here
[42:53] um these two devices actually have
[42:55] different polarization
[42:56] and so we're actually using a
[42:58] polarization mux
[43:00] and with the polarization mux we can
[43:02] couple both into one fiber so yeah so
[43:07] it wouldn't be economical to do separate
[43:09] fibers that way
[43:12] um what does a mux look like for a multi
[43:15] mode
[43:16] it it looks like a 45 degree
[43:20] um
[43:21] turning mirror okay
[43:23] it it looks and is
[43:27] this
[43:29] um so the polarization reflects off of
[43:32] one polarization reflects off the first
[43:34] surface the other polarization goes
[43:36] right through and hits the metal and
[43:39] reflects off the back surface and they
[43:41] both then travel into the same fiber
[43:43] there's a lens that you're not seeing
[43:46] right here
[43:47] that's on the end of of this
[43:51] XI okay
[43:53] yeah
[43:56] so we're in the middle of our phase two
[43:59] right now
[44:00] um and I just want to highlight the
[44:01] differences what we're doing in phase
[44:03] two uh we're doubling the number of
[44:05] channels
[44:06] but we're not doubling the number of
[44:08] fibers so so basically keeping the
[44:10] number of fibers the same so we're going
[44:13] to do two wavelengths we're also
[44:15] doubling the data rate so to double the
[44:18] data rate
[44:19] uh we are going to Pam for uh to do that
[44:24] um
[44:25] the uh the government gave us a power
[44:28] another energy challenge as they said
[44:30] cut the energy down in half
[44:33] um so now we're switching from sigi to
[44:36] CMOS so we're using a five nanometer uh
[44:39] CMOS node uh to do that
[44:42] those are the primary differences
[44:45] um the other thing that's going on in
[44:47] our second phase is our IBM systems
[44:49] group is taking our phase one hardware
[44:52] and they're doing a technology
[44:54] evaluation on it so basically they're
[44:56] mounting our phase one Parts
[44:59] um onto a server laminate so this looks
[45:03] um almost identical to an IBM processor
[45:05] laminate
[45:07] and they're going through all of the
[45:11] evaluations that they would normally do
[45:13] to qualify an IBM processor module I'm
[45:18] sorry would it be okay if I interrupt
[45:19] right now sure I was just wondering if
[45:22] uh multiple wavelengths is used for
[45:24] pixels and what are the challenges for
[45:27] that
[45:28] yeah so
[45:30] so there are products today
[45:33] um using multiple wavelengths uh for
[45:36] sure
[45:37] um
[45:38] there are ethernet Parts uh that use
[45:41] multiple wavelengths
[45:43] um
[45:44] there are challenges in how in the space
[45:48] it takes to do the multiplexer and the D
[45:51] multiplexer
[45:53] um
[45:54] and and I would say be if you look at
[45:57] the commercial Parts they're they're
[45:58] quite a bit larger than what what we're
[46:00] doing in co-packaging so in principle
[46:02] there's no challenge because you know
[46:05] people are shipping product with two and
[46:07] and four-way links
[46:09] um
[46:11] uh but
[46:13] um but to do it in co-packaging and to
[46:15] do it in the tiny space that we're doing
[46:18] it in and we're attempting to do it in
[46:20] um that is
[46:22] um that is challenging
[46:26] thank you so much
[46:29] um so here's another example so this is
[46:32] an IBM sort of processor laminate
[46:34] without the heat spreader on it or
[46:37] internally we call it the lid this is
[46:40] what it looks like when the lid is put
[46:41] on
[46:43] um and then this is what it looks like
[46:44] when
[46:45] um when the fibers are inserted and they
[46:48] have some strain relief
[46:50] we are doing an experiment
[46:53] you know this is an experiment like our
[46:55] power 775 where the ribbons go one on
[46:58] top of the other
[47:00] um you know one advantage is that this
[47:02] takes up less space on this side but the
[47:05] disadvantage is we removed a lot of
[47:08] uh heat spreader material to do this
[47:12] um
[47:13] and then another experiment and the
[47:15] reason why we got two different
[47:16] directions is
[47:18] I think ideally we would like our fibers
[47:20] to be sort of parallel
[47:23] even though we're water cooled there's
[47:26] still air blowing in these systems for
[47:29] other components that are
[47:31] less power hungry but still need air
[47:33] Cooling and we'd rather have our fibers
[47:37] parallel to air flow rather than
[47:38] perpendicular to it
[47:40] I'm showing an example here of I think
[47:44] senko put this together of a sort of a
[47:46] mock-up of a data center switch but you
[47:49] know with 16 modules using up all four
[47:51] sides then you basically can't
[47:54] of you can't make your fibers all
[47:57] parallel to the airflow
[47:59] so that's that's a concern
[48:04] um and then we've done a lot of thermal
[48:06] mechanical modeling of this package
[48:10] um you know the question is when we add
[48:12] these Optical modules to a processor
[48:14] laminate do we do anything that affects
[48:18] the Integrity of you know the precious
[48:20] silicon the processor silicon and what
[48:23] we find is the package does a warp a
[48:27] little bit almost 200 microns in the
[48:29] center
[48:31] um but when the lid is put on that's
[48:33] flattened out and this is considered
[48:35] acceptable uh pre-workage before the lid
[48:39] is put on
[48:41] um we did run through a completed a
[48:44] series of reliability tests so these
[48:47] were tests
[48:48] to see
[48:50] if the Integrity of the processor die
[48:53] was affected At All by the presence of
[48:56] the optical modules it's different from
[48:58] reliability of the optical modules and
[49:02] so what we found is basically the
[49:04] processor laminate the processor was not
[49:06] affected
[49:08] um up to a thousand hours of DTC and
[49:12] then we started seeing some fails
[49:15] um Beyond a thousand I didn't say a
[49:17] thousand hours I made a thousand Cycles
[49:20] um and then so beyond a thousand Cycles
[49:23] that's actually acceptable because DTC
[49:25] is a highly accelerated test
[49:29] um
[49:31] we've done some sort of system modeling
[49:33] to show the benefits of
[49:36] of using higher rated switches enco
[49:40] packaging uh and we took a summit we did
[49:43] an analysis of a summit like system
[49:45] with 600 1620 36 by 36 switches and then
[49:51] compared it to another system that had
[49:52] 1280 128 by 128 switches and we find you
[49:58] know that was basically wired up with
[50:01] co-packaging
[50:03] and we find that we can get sort of
[50:04] three times more Network endpoints for
[50:07] 20 percent fewer switches almost three
[50:10] times higher bisection bandwidth
[50:13] um and you know basically once you go
[50:16] into Optics you open up the path to
[50:18] direct network attached accelerators
[50:23] in doing this modeling
[50:26] we looked at different
[50:28] traffic patterns
[50:31] so here's just I think four examples of
[50:33] traffic patterns that we looked at and
[50:36] we looked at throughput and and delay
[50:39] and so we're finding uh you know in our
[50:42] co-packaging scenario that there's
[50:44] benefits in both throughput
[50:47] and delay for the various different
[50:50] types of traffic that the calculations
[50:53] might encounter
[50:58] foreign
[51:01] challenges uh for co-packaging uh
[51:05] certainly reliability is always going to
[51:07] be there
[51:08] um
[51:09] there's a field replacement
[51:11] serviceability issue
[51:14] um I would argue that once you go
[51:15] co-packaging even if you're socketed
[51:19] um there's no way you're doing field
[51:20] replacement uh if if the people in your
[51:23] field are already breaking your
[51:24] plugables
[51:26] you certainly don't want to expose them
[51:28] to or have your co-packaged Optics
[51:31] exposed to that you would need sort of a
[51:34] um higher skilled Workforce I just don't
[51:38] see it happening I do see like a a
[51:41] fail-in-place strategy is something
[51:42] worthwhile to pursue
[51:45] yield is always a yield and assembly
[51:47] together you know who's responsible at
[51:51] the end of the day for the final yield
[51:53] and who does what and when and where
[51:57] um you know IBM's approach to this has
[52:00] been to do uh sort of all the assembly
[52:02] uh in-house so we have our IBM Bromont
[52:05] packaging facility and our server
[52:08] manufacturing facility
[52:11] um but if you don't have those
[52:14] um then I think it becomes it becomes
[52:16] tricky
[52:17] um so there are some standards and msas
[52:20] in the works right now the oif certainly
[52:22] is doing one
[52:24] um but the minimum time for standards is
[52:27] at least two years we're likely to see
[52:30] proprietary Solutions emerge first
[52:33] um there is a question about compatible
[52:35] Technologies what I've been showing you
[52:36] is all multi-mode fiber
[52:39] what a lot of the industry says that
[52:42] they want is single mode
[52:45] um although I'm hearing some
[52:47] you know undercurrents of Desiring
[52:49] multimode again
[52:51] um and then
[52:53] something that that Optics vendors don't
[52:57] traditionally offer but it's very
[52:58] important especially
[53:00] uh when you can't replace things is you
[53:03] need field upgradable firmware
[53:05] um
[53:06] my I have yet to see Optics firmware
[53:09] that was bug free
[53:12] it's a couple examples of what other
[53:15] companies uh are doing other than IBM so
[53:18] here's something from Japan Japan has
[53:22] this research organization called Petra
[53:24] and then they spun out a company called
[53:27] AIO core they're doing silicon photonics
[53:31] um co-packaged Optics they got a little
[53:35] tiny five millimeter by five millimeter
[53:37] transceiver a hundred gig uh but it's
[53:40] actually silicon photonics with
[53:41] multi-mode fiber uh interesting solution
[53:46] um rockly photonics so I showed this I
[53:50] like to show this because actually
[53:52] this package was was packaged for
[53:55] Rockley by IBM Bromont
[53:58] um but if you've been following the news
[53:59] Rockley has exited the Datacom business
[54:02] all together and they're pursuing a
[54:05] silicon photonic wearables
[54:08] um but um
[54:10] anyway I like to show this at 96 fibers
[54:12] a 25 gig it was a silicon photonics
[54:15] co-package solution
[54:19] um
[54:20] so Cisco showed this a number of years
[54:23] ago uh but the key word here is mock-up
[54:28] um I haven't seen anything from them
[54:30] since uh but but the idea here for them
[54:34] if I zoom in on this is that they would
[54:38] have like a switch Asic and they'd have
[54:40] a photonic die with an electrical die
[54:42] mounted on top and they'd have a
[54:44] separable
[54:45] optical interface kind of like this
[54:50] [Music]
[54:51] um
[54:53] broadcom uh so broadcast been doing some
[54:57] very recent work
[54:59] um they've had some press releases so
[55:01] this is stuff that came from their ofc
[55:03] 2023 uh Booth
[55:06] um where they have a comment they sort
[55:09] of wired up a 25 Terra switch with half
[55:13] of co-packaged Optics and half
[55:15] electrical
[55:17] um
[55:17] as I showed you before you know these
[55:20] fibers uh you want to protect them
[55:23] Against the Wind essentially
[55:25] and so they very carefully have routed
[55:28] the cables under here
[55:30] they've got external lasers so what's
[55:32] going under
[55:33] Neath this cover is probably
[55:37] some splicing of the fibers that come
[55:39] from their external light sources and
[55:42] then go back into their photonics and
[55:44] then a nice neat cable management
[55:47] solution here this is a mock-up of what
[55:50] a 51 Tera
[55:53] uh switch might look like
[55:58] um and then Intel's been doing some work
[55:59] in this space they had a demo using a
[56:02] 12.8 Tera switch where they wired up
[56:06] um four of their
[56:08] uh sort of 400 gig co-packaged Optical
[56:11] modules
[56:13] um
[56:14] I I think you know was surprising for me
[56:17] to see these these heat pipes because it
[56:21] you know it's hard to say low energy
[56:23] when you have such a intense cooling
[56:27] solution
[56:28] um but they had this uh this demo
[56:31] running for a while
[56:34] um so Facebook and Microsoft teamed up a
[56:37] couple years ago to announce a big sort
[56:40] of MSA if you will or they put out a set
[56:44] of specifications that they called CPO
[56:47] and that launched a whole bunch of
[56:49] activity it got the oif working on
[56:52] specifications
[56:54] um
[56:55] and I think the oif is actually almost
[56:59] completely done with their
[57:00] specifications but uh Facebook and
[57:04] Microsoft seem to have
[57:06] uh lost interest in the project it
[57:08] appears to be on hold and they've gone
[57:10] back to plugable Optics which is kind of
[57:14] unfortunate I still think that a number
[57:18] of companies that worked on this are um
[57:22] are still moving ahead with their plans
[57:25] um and and we'll see where that that
[57:27] goes
[57:28] this is a duplicate of a slide I had
[57:31] before so
[57:34] um
[57:34] I want to talk about some competing
[57:36] alternatives to co-packaged Optics and
[57:39] that's basically copper copper cable
[57:42] and you know if you're working in Optics
[57:44] I I would advise you to never
[57:47] underestimate the capabilities of of
[57:50] copper cables
[57:52] um
[57:53] and I will tell you in this industry
[57:55] that for every person like myself
[57:57] working on fiber optics there's
[58:00] 10 Engineers working on copper cable
[58:04] and doesn't matter how good you are you
[58:06] can't compete with that level of
[58:08] resource
[58:09] uh you're just out out resourced
[58:14] um but um so
[58:17] so bringing copper off the top of a
[58:20] package just the same way I described as
[58:22] bringing Optics off this top
[58:25] um is a competing solution
[58:28] um there's talker bringing DC power to
[58:30] the top of the packages a lot of the i o
[58:32] pins on the bottom of a package are
[58:34] devoted to power
[58:36] um
[58:37] and uh and there's other things going on
[58:41] where they can make the high performance
[58:44] Asic you know which is five nanometer
[58:46] going to three nanometer
[58:48] go into two nanometer they can just take
[58:50] the i o drivers out of it and use
[58:53] additional chips so sort of like
[58:54] satellite surges if you will
[58:57] um and and that's a way that they can
[58:59] sort of
[59:00] still have a large Asic but not have
[59:04] some of its area devoted to IO
[59:08] um so these there's some competing
[59:10] Technologies out there and actually it
[59:12] wouldn't surprise me like I showed you
[59:14] the broadcom and the Intel
[59:17] uh demos if you will but they were half
[59:19] Optics half copper
[59:21] so so in summary uh you know tried to
[59:25] make the point that that HPC and data
[59:28] centers requirements have sort of
[59:30] diverged a lot at this point uh HPC
[59:33] systems are still put together with very
[59:36] short distance uh links they have a need
[59:40] for very low power and very low cost
[59:43] um data centers are put together with
[59:45] short to long distance
[59:47] interconnect power to date has not been
[59:50] a primary consideration but I think it's
[59:52] it's becoming a primary consideration
[59:56] um we're still seeing vixels in
[59:58] multimode fiber continue to have a large
[01:00:01] volume presence in both these spaces
[01:00:06] um
[01:00:06] and ethernet specifications are driving
[01:00:09] the data center industry
[01:00:11] but in a manner that's not energy
[01:00:13] efficient or low latency
[01:00:16] um I hope I've convinced you of the case
[01:00:18] for co-packaged Optics that
[01:00:21] it supports bandwidth increases it can
[01:00:23] lower power consumption and have a lot
[01:00:26] of sort of other packaging benefits
[01:00:30] um I do need to acknowledge my sponsor
[01:00:32] rpe and the roughly 100 people within
[01:00:37] IBM finisar and Texas A M University who
[01:00:41] are contributing to the motion project
[01:00:44] and thank you very much
[01:00:49] all right thank you very much
[01:00:53] um at this point I would like to open it
[01:00:56] to uh audience if they have any
[01:00:58] questions that they haven't asked
[01:01:00] throughout this talk
[01:01:10] I do have a question in uh in general uh
[01:01:16] when we put all this electronic and
[01:01:18] Optics in the same package what does the
[01:01:21] testing before putting it on the board
[01:01:24] look like how do you make sure that the
[01:01:26] Optics and the fiber and the electronics
[01:01:28] are all good
[01:01:30] and what is the overall yield after
[01:01:34] doing all this
[01:01:38] yeah
[01:01:39] um
[01:01:40] you definitely need to have known good
[01:01:42] parts uh going into this
[01:01:46] um
[01:01:47] and
[01:01:49] and I think what I didn't point out
[01:01:53] in in what we're doing in IBM
[01:01:56] um
[01:01:58] I didn't point out that that this
[01:02:02] um all five of these components are put
[01:02:04] down in a single
[01:02:06] um soldering step
[01:02:09] um and so
[01:02:11] basically we're not doing so we have
[01:02:14] known good die we've got known good
[01:02:16] Optics we've got known good laminate
[01:02:20] um but we don't know that the whole
[01:02:22] thing is good until we build it
[01:02:24] um
[01:02:27] I mean it's a fair question if if we
[01:02:30] don't underfill
[01:02:33] um you can see the the underfill here
[01:02:35] uh don't think you can see the underfill
[01:02:39] around the Optics but you'll have to
[01:02:41] trust me it's it's there I think if you
[01:02:44] don't underfill
[01:02:46] you have an opportunity to maybe do one
[01:02:48] one or two reworks but I think once you
[01:02:51] underfill your your that's it
[01:02:56] um and and so you really
[01:02:58] um
[01:02:59] you really need a high yield you really
[01:03:02] need
[01:03:03] um
[01:03:04] processes that are that are not going to
[01:03:06] result in things like um
[01:03:09] opens in shorts on all your solder
[01:03:11] joints and stuff and you need
[01:03:14] um you know packages that are going to
[01:03:15] stand up to
[01:03:17] um
[01:03:18] all the soldering operations
[01:03:22] it's not
[01:03:23] I don't want to trivialize this problem
[01:03:26] um
[01:03:28] how how does the testing look like in
[01:03:31] the end though like a machine plucks
[01:03:34] every fiber into the tester yeah yeah
[01:03:38] that's right I mean you could do
[01:03:40] loopback testing on the Optics or you
[01:03:43] could do through testing as well
[01:03:47] um but
[01:03:49] um that's that's the sort of in general
[01:03:51] what it would look like these so so for
[01:03:54] an IBM P series the processor module is
[01:03:59] a is a field replaceable unit and it is
[01:04:02] um fully testable on its on its own
[01:04:06] and so this would add a another level of
[01:04:09] tests
[01:04:11] Optics parts
[01:04:13] I see
[01:04:16] another question uh in the figure you
[01:04:21] showed uh with the tias and the PDS that
[01:04:24] there was a block diagram of the all the
[01:04:26] area that you had there is there any
[01:04:29] Equalization
[01:04:31] uh which ones the it was a sketch of the
[01:04:36] PDS in the PDS and vxl's in the middle
[01:04:40] and the area of RX is ntx
[01:04:45] this one yeah okay
[01:04:48] um yeah this one could work so is there
[01:04:51] any Equalization on the RX or the TX uh
[01:04:56] yeah yeah I know there there is there's
[01:04:58] a very
[01:04:59] [Music]
[01:04:59] um
[01:05:02] there's
[01:05:03] so for driving the vixel there's uh
[01:05:07] we've got
[01:05:08] um
[01:05:09] we've got three tops for driving the
[01:05:11] vixel
[01:05:13] um we do have a ctle on the input of the
[01:05:19] um transmitter and then we have
[01:05:22] um a very simple uh
[01:05:25] Equalization on the output of the
[01:05:27] receiver die
[01:05:29] and this Equalization is good up to what
[01:05:33] speed
[01:05:34] what's and what's the length
[01:05:38] so the equalization is not to equalize
[01:05:42] out the fiber length it's to equalize
[01:05:44] out the bandwidth of the vixels and the
[01:05:47] photodiodes so
[01:05:49] in the in the distances so I showed you
[01:05:52] how HPC systems are sort of wired up
[01:05:54] with less than 50 meters
[01:05:57] um we sort of targeted a 30 meter uh
[01:06:00] link for this technology 30 meters of
[01:06:04] fiber has far more bandwidth than 56
[01:06:07] okay good bit so that's we're not
[01:06:09] equalizing fiber we're equalizing
[01:06:12] the uh
[01:06:14] um
[01:06:15] the bandwidth of the vixel on the photo
[01:06:17] down and these are typical om4 fibers it
[01:06:21] yes that's right that's right okay thank
[01:06:24] you very much thank you very much
[01:06:27] okay
[01:06:29] if there are no other questions I would
[01:06:32] like to thank our speaker for for his
[01:06:34] time and this wonderful talk
[01:06:38] and we wrap it up
[01:06:41] um so the recording of this meeting will
[01:06:44] be available on our uh YouTube channel
[01:06:47] for people to view later on
[01:06:51] all right thank you very much and
[01:06:54] everyone have a good evening Take Care
[01:06:56] thank you bye