Essential Scale-Out Computing by James Cuff

Essential Scale-Out Computing by James Cuff


>>JAMES CUFF: Hi, good
afternoon, everyone. My name is James Cuff. I’m the Assistant Dean for Research
Computing here at Harvard University. And today I’m going to talk to you about
why scale-out computing is essential.>>So I guess, first up, who is this guy? Why am I here? Why am I talking to you? I have a background in scientific
computing and research computing, stretching back to the United
Kingdom– The Wellcome Trust Sanger Institute for the human genome– and
then more recently in the United States working at the Broad and other esteemed
places of learning, such as Harvard.>>I guess what that really means is
that I’m a recovering molecular bio physicist. So what right have I got to tell
you about scale-out computing? There’s a however. 18 years or so I’ve just seen the most
dramatic increases in scale complexity and overall efficiency
of computing systems.>>When I was doing my PhD at Oxford, I
was pretty excited with a 200 megahertz Silicon Graphics machine with 18
gigabytes of storage and a single CPU. Times have changed. If you fast forward now, we’re spinning
over 60,000 CPUs here at Harvard. Many other organizations
are spinning many more.>>The important takeaway from this is
that scale is now not only inevitable, it’s happened and it’s
going to continue to happen. So let’s, for a moment, kind
of rewind and talk very quickly about science, my favorite
subject, the scientific method.>>If you are to be a scientist,
you have to do a few key things. If you don’t do these things you can
not consider yourself a scientist and you will struggle being able to
understand your area of discipline.>>So first of all, you would formulate
your question, you generate hypotheses, but more importantly, you
predict your results– you have a guess as to
what the results will be. And then finally, you test your
hypothesis and analyze your results.>>So this scientific method is
extremely important in computing. Computing of both the prediction
and being able to test your results are a key part of what we need
to do in the scientific method. These predictions and testings
are the real two cornerstones of the scientific method, and each
require the most significant advances in modern computation.>>The two pillars of science are that
of theory and that of experimentation. And more recently,
computing is often mentioned as being the third pillar of science. So if you students are watching this,
you have absolutely no pressure. Third pillar of science– no big
deal– computing, kind of important. So glad this is the computing part
of computer science course 50.>>So enough of the background. I want to tell you the plan of what
we’re going to talk about today. I’m going to go over some history. I’m going to explain why we got here. I’m going to talk about some
of the history of the computing here at Harvard, some
activities around social media, green things– very
passionate about all things green– storage– computer storage–
how chaos affects scale-out out systems, and distributive systems in particular.>>And then I’m going to touch on some
of the scale-out hardware that’s required to be able to
do computing at scale. And then finally, we’re going to
wrap up with some awesome science.>>So, let’s take a minute to
look at our actual history. Computing has evolved. So since the ’60s, all
the away through to today, we’ve seen basically a change of
scope from centralized computing to decentralize computing, to
collaborative and then independent computing and right back again.>>And let me annotate that a little bit. When we first started off with
computers, we had mainframes. They were inordinately
expensive devices. Everything had to be shared. The computing was complex. You can see, it filled rooms and
there were operators and tapes and all sorts of whirry,
clicky, spinny devices.>>Around the ’70s early ’80s, you started
to see an impact of the fax machines. So you’re starting to see computing
start to appear back in laboratories and become closer to you. The rise of the personal
computer, certainly in the ’80s, early part of the
decade, really changed computing.>>And there’s a clue in
the title, because it was called the personal computer,
which meant it belonged to you. So as the evolution of
computing continued, people realized that their personal
computer wasn’t really big enough to be able to do anything of any merit,
or significant merit, in science.>>And so folks started to
develop network device drivers to be able to connect PCs
together to be able to build clusters. And so this begat the era
of the Beowulf cluster. Linux exploded as a response to
proprietary operating system, both cost and complexity.>>And then, here we are today,
where, yet again, we’re faced with rooms full of computer
equipment and the ability to swipe one’s credit card and get
access to these computing facilities, remotely.>>And so you can then see, in
terms of history impacting how we do computing
today, it’s definitely evolved from machine
rooms full of computers through some personal computing
all the way right back again to machine rooms full of computers.>>So this is my first cluster. So 2000, we built a
computer system in Europe to effectively annotate
the human genome. There’s a lot of technology
listed on the right hand side there that, unfortunately,
is no longer with us. It’s passed off to the
great technology in the sky.>>The machine itself is probably
equivalent of a few decent laptops today, and that just kind of shows you. However, we did carefully annotate
the human genome and both protected it with this particular paper in
Nature from the concerns the data being public or private.>>So this is awesome, right? So we’ve got a human genome. We’ve done computing. I’m feeling very pleased myself. I rolled up to Harvard in 2006,
feeling a lot less pleased with myself.>>This is what I inherited. This is a departmental
mail and file server. You can see here there’s
a little bit of tape that’s used to hold the system together. This is our license and print server. I’m pretty sure there maybe passwords
on some of these Post-it Notes. >>Not awesome. Pretty far from awesome. And so, I realize this little chart
that I showed you at the beginning from sharing to ownership
back to sharing, that we needed to change the game. And so we changed the game
by providing incentives. And so human beings, as this
little Wikipedia article says here, our purposeful creatures. And the study of incentive
structures is essential to the study of economic activity.>>So we started to incentivize
our faculty and our researchers. And so we incentivized them with
a really big computer system. So in 2008, we built a 4,096
processor machine– 10 racks, couple hundred kilowatts of power.>>What I think is
interesting is it doesn’t matter where you are in the cycle. This same amount of power and
compute, the power is the constant. It was 200 kilowatts when we
were building systems in Europe. It’s two hundred kilowatts
in 2008, and that seems to be the [? quanter ?] of small
university-based computing systems.>>So Harvard today– fast forward, I’m no
longer sad panda, quite a happy panda. We have 60-odd thousand load balanced
CPUs, and their climbing dramatically. We have 15 petabytes of
storage, also climbing. Again, this 200 kilowatt
increment, we seem to be adding that
every six or so months. Lots and lots of virtual machines. And more importantly,
about 1.8 megawatts of research computing equipment.>>And I’m going to come
back to this later on, as to why I now no longer necessarily
count how much CPU we have, but how big is the electricity bill. 20 other so dedicated
research computing staff. And more importantly, we’re
starting to grow our GPGPUs. I was staggered at how much of this
is being added on a day-to-day basis. So, history lesson over, right?>>So how do we get there from here? Let’s look at some modern
scale-out compute examples. I’m a little bit obsessed with the
size and scale of social media. There are a number of extremely
successful large scale computing organizations now on the planet,
providing support and services to us all. So that’s the disclaimer.>>And I want to start with a
number of ounces in an Instagram. It’s not actually a
lead-in to a joke, it’s not even that funny,
actually, come to think of it. But anyway, we’re going to
look at ounces in Instagram. And we’re going to start
with “My bee and a flower.” I was at [INAUDIBLE] Village
and I took a little picture of a bee sitting on a flower. And then I started to think about
what does this actually mean. And I took this picture off my phone
and counted how many bytes are in it, and it’s about 256 kilobytes. Which when I started, would basically
fill a 5 and 1/4 inch floppy. And started to think, well, that’s cool.>>And I started to look and do
some research on the network. And I found out that Instagram
has 200 million MAUs. I wasn’t actually that
sure what a MAU was. And a MAU, down here, is
a monthly active user.>>So, 200 million MAUs– pretty cool. 20 billion photographs– so
quite a lot of photographs. 60 million new photographs
each and every day coming out at about .002 gig per photo. That’s about five petabytes
of disk just right there. And that’s really not the central part
of what we’re going to talk about. That is small potatoes. Or as we say in England, tiny spuds.>>So let’s look at the real elephant
in the room– unique faces. Again, let’s measure in
this new quanta call a MAU. Facebook itself has 1.3 billion MAUs. WhatsApp, which I hadn’t even
heard of until recently, it’s some sort messaging service,
is 500 million MAUs. Instagram, which we just
talked about, 200 million MAUs. And Messenger, which is
another messaging service, is also 200 million MAUs.>>So total that up, it’s about
2.2 billion total users. Clearly there’s some overlap, but that’s
equivalent to a third of the planet. And they send something in the
region of 12 billion messages a day. And again, there’s only 7
billion people on the planet. Not everyone has a smartphone. So this is insane numbers.>>And I’m going to argue that it’s not
even about the storage or the compute. And to quote the song,
it’s all about that graph. Here’s our lovely Meghan Trainor down
here, singing about all the bass. Note, she also has quite a
bit of bass herself– 207, well 218 million people have seen
this young lady singing her song.>>So my argument is it
it’s all about the graph. So we took some open source software
and started to look at a graph. And this is LinkedIn, so this
is a Facebook for old people. And so, this is my LinkedIn graph. I have 1,200 or so nodes,
so-called “Friends.” And here’s me at the top. And here’s all of the interconnections.>>Now, think back to the Instagram story. Each one of these is
not just the photo, it has a whole plethora of connections
between this particular individual and many others. This is central piece is either a
bug in the graph drawing algorithm, or this maybe David
Malan, I’m not sure yet. So you can redraw the
graphs in all sorts of ways– gephi.gihub.io is where
you can pull that software from. It’s really cool for being
able to organize communities. You can see here, this is Harvard and
various other places that I’ve worked, because this is my work-related data.>>So just think about the complexity
of the graph and all of the data that you pull along with. So meanwhile, back at FriendFace, right? We looked at the Instagram data that
was of the order of five petabytes. No big deal. Still quite a lot of data, but no big
deal in the greater scheme of things.>>From this article on the old internet,
“Scaling the Facebook data warehouse to 300 petabytes.” That’s a whole different
game changer now, when you’re starting to
think of data and the graph and what you bring along with. And their high data is growing of
the order of 600 terrabytes a day.>>Now, you know, well, then–
I mean, 600 terrabytes a day, 300 petabytes– they’re
also now starting to get very concerned about
how to keep this stuff and to make sure this data stays around. And this gentleman here,
Jay Parikh, is looking at how to store an exabyte of data.>>Just for those of you
who are watching along at home, an exabyte– 10 to the 18. It’s got its own Wikipedia
page, it’s that big of a number. That is the size and scale of what we’re
looking at, to be able to store data. And these guys aren’t mucking around,
they’re storing that amount of data. So one of the clues that
they’re looking at here is data centers for
so-called cold storage. >>Which brings me to being green. And here is Kermit. He and I agree– it’s extremely
difficult to be green, but we give it our best try. Kermit can’t help it, he has
to be green all the time, can’t take his green-ness off at all.>>So, being concepts– a
few kind of core concepts of greenness, when it
relates to computing. The one that is the most important
is the longevity of the product. If your product has a short lifetime,
you cannot, by definition, be green. The energy taken to manufacture a
disk drive, a motherboard, a computer system, a tablet, whatever it may
be, the longevity of your systems are a key part of how green you can be.>>The important part, as all of you
are building software algorithms– algorithm’s a partial
word for software, right? So, your algorithm design is
absolutely critical in terms of how you are going to be able to make
quick and accurate computations to use the least amount of energy possible. And I’ll get to this in a little bit.>>Data center design– you’ve seen
that we already have thousands upon thousands of machines, sitting
quietly in small, dark corners of the world, computing. Resource allocation– how to get
to the compute, to the storage, through the network. Operating systems are a key part of
this, and a lot of virtualization to be able to pack more and
more compute into a small space.>>I’ll give you a small example
from research computing. We needed more ping, more
power, and more pipe. We needed more bigger,
better, faster computers, and needed to use less juice. And we couldn’t work out how to do this. I don’t know if the hashtag gowest as
probably been used by the Kardashian, but anyway, gowest. And we did.>>We picked up our operation
and we moved it out to Western Massachusetts
in a small mill town called Holyoke, just north
of Chikopee and Springfield. We did this for a couple of reasons. The main one was that we
had a very, very large dam. And this very large dam is able to
put out 30 plus megawatts of energy, and it was underutilized at the time.>>More importantly, we also had
a very complicated network that was already in place. If you look at where the network
goes in the United States, it follows all the train tracks. This particular piece of network was
owned by our colleagues and friends at Massachusetts
Institute of Technology, and it was basically built
all the way out to Route 90.>>So we had a large river tick, Route 90
tick, we had a short path of 100 miles, and a long path of about 1,000 miles. We did have to do a very large
network splice, as you can see here, to basically put a link in, to
be able to connect to Holyoke, but we had all of the requisite
infrastructure– ping, power, pipe. Life was good. And again, big dam.>>So we built basically the Massachusetts
Green High Performance Computing Center. This was a labor of love through five
universities– MIT, Harvard, UMass, Northeastern, and BU. Five megawatt day one connected load. We did all sorts of cleverness
with airside economizers to keep things green. And we built out 640-odd racks,
dedicated for research computing.>>It was an old brownfield site, so we
had some reclamation and some tidy-up and some clean-up of the site. And then we started
to build the facility and, boom– lovely facility with the
ability to run sandbox computing, to have conferences and seminars,
and also a massive data center floor.>>Here is my good self. I’m obviously wearing the same jacket. I maybe only have one
jacket, but there’s me and John Goodhue– he’s the
executive director of the Center– standing in the machine room
floor, which, as you can see, is pretty dramatic, and it
goes back a long, long way.>>I often play games driving
from Boston out to Holyoke, pretending that I’m a TCP/IP packet. And I do worry about my latency
driving around in my car. So that’s the green piece. So let’s just take a minute
and think about stacks. So we’re trying very carefully to
build data centers efficiently, computing efficiently, make good
selection for the computing equipment and deliver, more
importantly, our application, be it a messaging service
or a scientific application.>>So here are the stacks. So physical layer, all the
way up through application– hoping that this is going to
be a good part of your course. OSI seven layer model is basically,
you will live, eat, and breathe this throughout your computing careers. This whole concept of physical
infrastructure– wires, cables, data centers, links. And this is just describing the network.>>Up here is, well, obviously,
this is an old slide, because this should say HTTP, because
nobody cares about simple mail transport protocols, anymore. It’s all happening in the HTTP space. So that’s one level of stack.>>Here’s another set of stacks, where you
have a server, a host, a hypervisor, a guest, binary library,
and then your application. Or, in this case, the device
driver, a Linux kernel, native c, Java virtual machine,
Java API, then Java applications, and so on and so forth. This is a description
of a virtual machine.>>Holy stacks, Batman! Think about this in
terms of how much compute you need to get from
what’s happening here, all the way up to the top
of this stack, to then be able to do your actual
delivery of the application.>>And if you kind of
rewind and start to think about what it takes to provide
a floating point operation, your floating point operation is a sum
of the sockets, the number of cores in the socket, a clock, which is
how fast can the clock turnover– four gigahertz, two
gigahertz– and then the number of operations you can
do in a given hertz.>>So those microprocessors today
do between four and 6 FLOPs per clock cycle. And so a single-core 2.5 gig clock
has a theoretical performance of about a mega FLOP, give or take.>>But, as with everything,
we have choices. So and Intel Core 2, Nehalem
Sandy Bridge, Haswell, AMD, take your choices– Intel Atom . All of these processor architectures
all have a slightly different way of being able to add
two numbers together, which is basically
their purpose in life. Must be tough. There’s millions of them sitting
in data centers, now though.>>Sor, flops per watt–
this is the big thing. So if I want to get more of this
to get through this stack, faster, I’ve got to work on how many
floating point operations a second, I can do, and then give them watt. And fortunately, folks
have thought about this.>>So there’s a large
contest every year to see who can build the fastest computer
that can diagonalize a matrix. It’s called the Top 500. They pick the top from
the best 500 computers on the planet that can
diagonalize matrices. And you get some amazing results.>>A lot of those machines are
between 10 and 20 megawatts. They can diagonalize matrices
inordinately quickly. They don’t necessarily diagonalized
them as efficiently per watt, so there was this big push to look at
what a green 500 list would look like. And here is the list from June. There should be a new one very shortly.>>And it calls out– I’ll take
the top of this particular list. There’s two specific machines– one
from the Tokyo Institute of Technology and one from Cambridge
University in the United Kingdom. And these have pretty staggering
mega flops per watt ratios. This one’s 4,389, and the
next one down is 3,631.>>I’ll explain the difference between
these two, in the next slide. But these are these are
moderately sized test clusters. These are just 34
kilowatts or 52 kilowatts. There are some larger ones
here– this particular one at the Swiss National
Supercomputing Centre. The take home message
for this is that we’re trying to find computers
that can operate efficiently.>>And so, let’s look at this top
one, cutely called, the KFC. And a little bit of advertising here. This particular food company
has nothing to do with this. It’s the fact that
this particular system is soaked in a very
clever oil-based compound. And so they got their
chicken fryer moniker when they first started to
build these types of systems.>>But basically what they’ve taken
here is a number of blades, put them in this
sophisticated mineral oil, and then worked out how to get all
the networking in and out of it. Then, not only that,
they’ve put it outside so that it can exploit outside air cooling. It was pretty impressive. So you have to do all
of this shenanigans to be able to get this amount of
compute delivered for small wattage.>>And you can see this is the shape
of where things are heading. The challenge is that regular air
cooling is the economy of scale and is driving a lot of the
development of both regular computing, and high performance computing. So, this is pretty disruptive. I think this is fascinating. It’s a bit messy when you
try to swap the disk drives, but it’s a really cool idea.>>So not only that, there’s
a whole bunch of work being built around what we’re
calling the Open Compute Project. And so, more about that
a little bit later. But the industry’s starting to
realize that the FLOPs per watt is becoming important. And you, as folks here, as
you design your algorithms and you design your
code, you should be aware that your code can
have a knock-on effect.>>When Mark was sitting here in his
dorm room writing Facebook 1.0, I’m pretty sure he had a view
that it was going to be huge. But how huge it would be on the
environment is a big dealio. And so all of ya’ll could
come up with algorithms that could be the next challenging
thing for folks like me, trying to run systems.>>So let’s just think about
real world power limits. This paper by Landauer–
is not a new thing. 1961 this was published
in the IBM Journal. This is the canonical
“Irreversibility and Heat Generation in the Computing Process.” And so he argued that
machines inevitably perform logistic functions that
don’t have single-valued inverse.>>So that the whole part of
this is that back in the ’60s, folks knew that this was
going to be a problem. And so the law of limits said 25
degrees C, a sort of canonical room temperature, the limit
represents 0.1 electron volts. But theoretically, this is
the theory, computer memory, operating at this limit could be
changed at one billion bits a second.>>I don’t know about you, but not
come across many one billion bits a second data rate exchanges. The argument there was that only
2.8 trillions of a watt of power ought to ever be expanded.>>All right, real world example–
this is my electric bill. I’m 65% percent of
that lovely data center I showed you, in this particular time. This is back in June last year. I’ve taken an older version so that we
can and sort of anonymize a little bit. I was spending $45,000 a
month for energy there.>>So the reason being there is that we
have over 50,000 processes in room. So could you imagine your own
residential electricity bill being that high? But it was for a 199 million
watt hours over a month.>>So the question I pose is, can you
imagine Mr. Zuckerberg’s electric bill? Mine is pretty big, and I struggle. And I’m not alone in this is. There’s a lot of people
with big data centers. And so, I guess, full disclosure–
my Facebook friends a little bit odd.>>So my Facebook friend is
the Prineville data center, which is one of Facebook’s largest,
newest, lowest energy data center. And they post to me, things like
power utilization effectiveness, as in how effective is the data
center versus how much energy you’re putting into it, how much
water are they using, what’s the humidity and temperature.>>And they have these
lovely, lovely plots. I think this is an
awesome Facebook page, but I guess I’m a little bit weird.>>So one more power thing,
research computing that I do is significantly different to
what Facebook and Yahoo and Google and other on-demand, fully,
always available services. And so I have the advantage that when
ISO New England– and ISO New England helps set the energy
rates for the region.>>And it says it’s extending
a request to consumers to voluntarily conserve high energy,
because of the high heat and humidity. And this was back on the 18th of July. And so I happily Tweet back, Hey,
ISO New England, Green Harvard. We’re doing our part over
here in research computing. And this is because we’re doing science.>>And as much as people say science
never sleeps, science can wait. So we are able to quiesce our
systems, take advantage of grade rates on our energy bill, and
help the entire New England region by shedding
many megawatts of load. So that’s the unique thing that
differs about scientific computing data centers and those that are
in full production 24/7.>>So let’s just take another gear here. So, I want to discuss
chaos a little bit. And I want to put it in
the auspices of storage. So for those that kind
of were struggling getting their head around what petabytes
of storage look like, this an example. And this is the sort of stuff
I deal with all the time.>>Each one of these little fellas
is a four terabyte hard drive, so you can kind of count them up. We’re getting now between
one to 1 and 1/2 petabytes in a standard industry rack. And we have rooms and rooms, as you
saw in that earlier picture with John and I, full of these racks of equipment. So it’s becoming very, very easy
to build massive storage arrays>>It’s mostly easy inside of Unix to
kind of count up how things are going. So this is counting how many
MAU points have I got there. So that’s 423 intercept points. And then if I run some sketchy awk, I
can add up, in this particular system, there was 7.3 petabytes
of available storage.>>So that’s a lot of stuff. And storage is really hard. And yet, for some reason,
this is an industry trend. Whenever I talk to our researchers
and our faculty and say, hey, I can run storage for you. Unfortunately, I have to
recover the cost of the storage. I get this business. And people reference Newegg
or they reference Staples or how much they can buy a
single terabyte disk drive for.>>So this, you’ll note
here, that there’s a clue. There’s one disk drive here. And if we go back, I have many. Not only do I have many, I have
sophisticated interconnects to be able to stitch
these things together. So the risk associated with these large
storage arrays is not insignificant.>>In fact, we took to the
internet and we wrote a little story about a well-meaning,
mild-mannered director of research computing– happens to have a
strange English accent– trying to explain to a researcher what the
no underscore backup folder actually meant. It was quite a long, little story,
a good four minutes of discovery.>>And note, I have an awful
lot less space than the lady that sings about all the bass. We’re quite a few accounts lower. But anyway, this is an
important thing to think about, in terms of what could go wrong. So if I get a disk drive, and
I throw it in a Unix machine, and I start writing things to it,
there’s a magnet, there’s a drive head, there’s ostensibly, a one or a zero
being written down on to that device.>>Motors– spinny, twirly
things always break. Think about things that break. It’s always been spinny, twirly things. Printers, disk drives,
motor vehicles, etc. Anything that moves is likely to break.>>So you need motors, you
need drive firmware, you need SAS/SATA controllers, wires,
firmware on the SAS/SATA controllers, low level blocks. Pick your storage controller file
system code, whichever one it may be, how you stitch things together. And your virtual memory manager
pages, DRAM fetch and stores. Then, you get another
stack, which is kind of down the list on this
one, algorithms, users.>>And if you multiply this
up, I don’t know how many, there’s a lot of places
where stuff can go sideways. I mean, that’s an example about math. But it’s kind of fun to think of
how many ways things could go wrong, just for a disk drive. We’re already at 300 petabytes, so
imagine the number of disk drives you need at 300 petabytes
that can go wrong. Not only that– so that’s storage. And that alludes to the
person I’d like to see enter stage left, which
is the Chaos Monkey. So at a certain point, it gets even
bigger than just the disk drive problem.>>And so, these fine ladies and gentleman
that run a streaming video service realized that their computers were
also huge and also very complicated and also providing service
to an awful a lot of people. They’ve got 37 million members– and
this slide’s maybe a year or so old– thousands of devices. There are billions of hours of video. They log billions of events a day.>>And you can see, most people watch
the telly later on in the evening, and it far outweighs everything. And so, they wanted to
be able to make sure that the service was up and
reliable and working for them. So they came up with this
thing called Chaos Monkey. It’s piece of software
which, when you think about talking about the title
of this whole presentation, scale-out means you
should test this stuff. It’s no good just having
a million machines. So the nice thing about
this is, Chaos Monkey is a service which
identifies groups of systems and randomly terminates one
of the systems in a group. Awesome. So I don’t know about
you, but if I’ve ever built a system that relies on other
systems talking to each other, you take one of them out, the
likelihood of the entire thing working, diminishes rapidly.>>And so this piece of software runs
around Netflix’s infrastructure. Luckily, it says it runs only in
business hours with the intent that engineers will be
alert and able to respond. So these are the types
of things we’re now having to do to perturb our computing
environments, to introduce chaos and to introduce complexity.>>So who, in their right
mind, would willingly choose to work with a Chaos Monkey? Hang on, he seems to be pointing me. Well, I guess I should– cute. But the problem is you
don’t get the choice. The Chaos Monkey, as you
can see, chooses you.>>And this is the problem with computing
at scale is that you can’t avoid this. It’s an inevitability of complexity
and of scale and of our evolution, in some ways, of computing expertise. And remember, this is
one thing to remember, Chaos Monkeys love
snowflakes– love snowflakes. A snowflake– we’ve explained the
Chaos Monkey– but a snowflake is a server that is unique and
special and delicate and individual and will never be reproduced.>>We often find snowflake
service in our environment. And we always try and
melt snowflake service. But if you find a server
in your environment that is critical to the longevity
of your organization and it melts, you can’t put it back together again. So Chaos Monkey’s job was to
go and terminate instances. If the Chaos Monkey melts the
snowflake, you’re over, you’re done. I want to talk about
some hardware that we’re seeing in terms of sort of
scale-out activities too. And some unique things that are in
and around the science activity. We are now starting to see, remember
this unit of issue, this rack? So this is a rack of GPGPUs– so general
purpose graphics processing units.>>We have these located in our data
center, 100 or so miles away. This particular rack
is about 96 tera FLOPS of single-precision math able
to deliver out the back of it. And we have order 130-odd
cards in an instance that we– multiple
racks of this instance.>>So this is interesting in the sense that
the general purpose graphics processes are able to do mathematics incredibly
quickly for very low amounts of energy. So there’s a large uptick in
the scientific computing areas, looking at graphics
processing units in a big way.>>So I ran some Mcollective
through our puppet infrastructure yesterday, very excited about this. just short of a petaflop
of single-precision. Just to be clear here , this
little multiplier is 3.95. Double-precision math
would be about 1.2, but my Twitter feed
looked way better if I said we had almost a petaflop
of single-precision GPGPUs.>>But it’s getting there. It’s getting to be
very, very impressive. And why are we doing this? Because quantum chemistry,
among other things, but we’re starting to design
some new photovoltaics.>>And so Alan Aspuru-Guzik, who’s a
professor in chemistry– my partner in crime– for the last few years. We’ve been pushing the
envelope on computing. And the GPGPU is ideal
technology to be able to do an awful lot of complicated
math, very, very quickly.>>So with scale, comes new challenges. So huge scale– you have to be
careful how you wire this stuff. And we have certain levels of
obsessive compulsive disorder. These pictures probably
drive a lot of people nuts. And cabinets that aren’t
wired particularly well drive our network and
facilities engineers nuts. Plus there’s also airflow
issues that you have to contain.>>So these are things that I
would never have thought of. With scale, comes more complexity. This is a new type of file system. It’s awesome. It’s a petabyte. It can store 1.1 billion files. It can read and write to 13 gigabytes
and 20 gigabytes a second– gigabytes a second. So it can unload terabytes
in no time at all.>>And it’s highly available. And it’s got amazing lookup
rates– 220,000 lookups a second. And there are many different people
building these kind of systems. And you can see it here graphically. This is one of our file systems
that’s under load, quite happily reading at just short
of 22 gigabytes a second. So that’s cool– so complexity.>>So with complexity and scale,
comes more complexity, right? This is one of our many,
many network diagrams, where you have many different
chassis all supporting up into a main core switch,
connected to storage, connecting to low latency interconnects. And then all of this side of the
house, is just all of the management that you need to be able to address
these systems from a remote location. So scale has a lot of
complexity with it. >>Change gear again, let’s go back
and have a little spot of science. So, remember, research
computing and this little shim– little pink shim between the
faculty and all of their algorithms and all of the cool science and all of
this power and cooling and data center floor and networking and big computers
and service desks and help desks and so forth– and so, we’re just
this little shim between them.>>What we’ve started to
see is that the world’s been able to build
these large data centers and be able to build
these large computers. We’ve gotten pretty good at it. What we’re not very good at is this
little shim between the research and the bare metal and the technology. And it’s hard.>>And so we’ve been able to hire
folks that live in this world. And more recently, we spoke to the
National Science Foundation and said, this scale-out stuff is great,
but we can’t get our scientists on to these big complicated machines. And so, there have been a
number of different programs where we really were mostly
concerned about trying to see if we could transform
the campus infrastructure.>>There are a lot of programs
around national centers. And so, ourselves, our
friends at Clemson, University of Wisconsin Madison,
Southern California, Utah, and Hawaii kind of got together to
look at this problem. And this little graph here
is the long tail of science.>>So this is– it doesn’t
matter what’s on this axis, but this axis is actually number
of jobs going through the cluster. So there’s 350,000 over
whatever time period. These are our usual suspects
along the bottom here. In fact, there’s Alan Aspuru-Guzik,
who we were just talking about– tons and tons of compute, really
effective, knows what he’s doing.>>Here’s another lab that I’ll talk
about in a moment– John Kovac’s lab . They’ve got it. They’re good. They’re happy. They’re computing. Great science is getting done . And then, as you kind
of come down here, there are other groups that
aren’t running many jobs.>>And why is that? Is it because the computing is too hard? Is it because they don’t know how to? We don’t know, because
we’ve gone and looked. And so that’s what this
project is all about, is locally, within
each of these regions, to look to avenues where we can engage
with the faculty and researchers actually in the bottom end of the tail,
and understand what they’re doing.>>So that’s something that we’re
actually passionate about. And that’s something that
science won’t continue to move forward until we solve
some of these edge cases. Other bits of science
that’s going up– everyone seen the Large Hadron Collider. Awesome, right? This stuff all ran out at Holyoke. We built– the very first
science that happened in Holyoke was the collaboration between
ourselves and Boston University. So it’s really, really cool.>>This is a fun piece
of science for scale. This is a digital access to
a sky century at Harvard. Basically, it’s a plate archive. If you go down Oxford–
Garden Street, sorry, you’ll find one of the observatory
buildings is basically full of about half a million plates.>>And these are pictures of the
sky at night, over 100 years. So there’s a whole rig set up
here to digitize those plates, take pictures of them, register
them, put them on a computer. And that’s a petabyte and a half,
just right there– one small project.>>These are other projects. This Pan-STARRS project is doing
a full wide panoramic survey, looking for near Earth asteroids
and transient celestial events. As a molecular biophysicist, I love
the word transient celestial event. I’m not quite sure what it is, but
anyway, we’re looking for them.>>And we’re generating 30 terabytes
a night out of those telescopes. And that’s not really a bandwidth
problem, that’s like a FedEx problem. So you put the storage on the van
and you send it whatever it is.>>BICEP is really interesting–
so background imaging of cosmic extra galactic polarization. When I first started working
at Harvard seven or so, eight years ago, I remember
working on this project and it didn’t really sink
home as to why polarized light from the cosmic microwave
background would be important, until this happened.>>And this was John Kovac,
who I talked to before, using millions upon millions of CPU
hours, in our facility and others, to basically stare into the inside
of the universe’s first moments after the Big Bang, and
trying to understand Einstein’s general theory of relativity. It’s mind blowing that our computers
are helping us unravel and stare into the very origins of why we’re here.>>So when you talk about scale,
this is some serious scale. The other thing of scale is, that
particular project hit these guys. And this is the response curve for BICEP
[INAUDIBLE] This was our little survey.>>And you can see here, life
was good until about here, which was when the
announcement came out. And you have got literally
seconds to respond to the scaling event which
corresponds to this little dot here, which ended up shifting
four or so terabytes of data through the web server
that day– pretty hairy.>>And so, these are the
types of things that can happen to you in your infrastructure
if you do not design for scale. We had a bit of a
scramble that day, to be able to span out enough web service
to keep the site up and running. And we were successful. This is a little email
that’s kind of cute. This is a mail to Mark Vogelsberger,
and Lars Hernquist, who’s a faculty member here at Harvard. More about Mark later. But I think this is one
sort of sums up kind of where the computing
is in research computing. Hey, team, since last
Tuesday, you guys racked up over 28% of the new
cluster, which combined is over 78 years of
CPU in just three days. And I said, it’s still
only just Friday morning. This is pretty awesome! Happy Friday!>>Then I give them the data points. And so that was kind of interesting. So remember about Mark, he’ll come
back into the picture in a little bit. So scale-out computing is everywhere.>>We’re even helping folks look
at how the NBA functions, and where people are
throwing balls from. I don’t really understand this game too
well, but seemingly, it’s a big deal. There’s hoops and bowls and money.>>And so, our database, we
built a little 500 [INAUDIBLE] parallel processor cluster,
a couple of terabytes of RAM, to be able to build this
for Kirk and his team. And they’re doing computing
in a whole other way. Now this is project we’re
involved with that’s absolutely fascinating, around neural
plasticity connectomics and genomic imprinting– three very heavy
hitting areas of research that we fight with on
a day-to-day basis. The idea that our brains are under
plastic stress when we are young. And much of our adult behavior is
sculpted by experience in infancy. So this is a big dealio.>>And so this is work that’s funded by the
National Institutes of Mental Health. And we are trying to basically,
through a lot of large data and big data analysis, kind
of peer into our human brain through a variety of
different techniques.>>So I wanted to stop and kind of
just pause for a little moment. The challenge with remote
data centers is it’s far away. It can’t possibly work. I need my data close by. I need to do my research in my lab.>>And so I kind of took an example of a
functional magnetic resonance imaging data set from our data
center in Western Mass. and connected it to my
desktop in Cambridge. And I’ll play this little video. Hopefully it will kind of work.>>So this is me going through
checking my GPUs are working. And I’m checking that VNC’s up. And this is a clever VNC. This is a VNC with 3D pieces. And so, as you can see shortly, this
is me spinning this brain around. I’m trying to kind of get it oriented. And then I can move through many
different slices of MRI data. And the only thing that’s
different about this is, it’s coming over the wire
from Western Mass. to my desktop. And its rendering
faster than my desktop, because I don’t have a $4,000
graphics card in my desktop, which we have out Western Mass. Of course, I’m trying to be clever. I’m running GLX gears in the
background, whilst doing all this, to make sure that I can
stress the graphics card, and that it all kind of
works and all the rest of it. But the important thing is,
is this is 100 miles away. And you can see from this that
there’s no obvious latency. Things holding together fairly well.>>And so that, in and of itself,
is an example and some insight into how computing and scale-out
computing is going to happen. We’re all working on
thinner and thinner devices. Our use of tablets is increasing.>>So therefore, my carbon
footprint is basically moving from what used
to do that would’ve been a huge machine
under my desk, to what is now a facility– could be anywhere. It could be anywhere at all. And yet, it’s still able to bring
back high performance graphics to my desktop.>>So, getting near the
end– remember Mark? Well, smart lad is Mark. He decided that he was going to
build a realistic virtual universe. That’s quite a project, when you
think you’ve got to pitch this. I’m going to use a
computer, and I’m going to model the 12 million years after
the Big Bang to represent a day. And then I’m going to do 13.8
billion years of cosmic evolution. All right.>>This actually uses a computer
the was bigger than our computer, and it spilled over onto the national
resources to our friends down in Texas. And to the national facilities,
this was a lot of compute. But we did a lot of
the simulation locally to make sure that the software
worked and the systems worked.>>And it’s days like this when you
realize that you’re supporting science at this level of scale, that
people can now say things like, I’m going to a model a universe. And this is his first model. And this is his team’s first model. There are many other
folks that are going to come behind Mark, who are going to
want to model with high resolution, with more specificity,
with more accuracy.>>And so, in the last couple of minutes,
I just want to show you this video of Mark and Lars’s that to me, again,
as a life scientist , is kind of cute. So this, at the bottom
here, to orient you, this is telling you the
time since the Big Bang. So we’re at about 0.7 billion years. And this is showing the current update. So you’re seeing at the moment,
dark matter and the evolution of the fine structure and early
structures in our known universe.>>And the point with this is that this
is all done inside the computer. This is a set of parameters
and a set of physics and a set of mathematics
and a set of models that are carefully selected, and then
carefully connected to each other to be able to model the interactions.>>So you can see some starts of
some gaseous explosions here. And gas temperature is changing. And you can start to see the structure
of the visible universe change. And the important part with this
is, each little tiny, tiny, tiny dot is a piece of physics and has
a set of mathematics around, informing its friend and its neighbor.>>So from a scaling perspective, these
computers have to all work in concert and talk to each other efficiently. So they can’t be too chatty. They have to store their results. And they have to continue to
inform all of their friends. Indeed, you’ll see now, this model’s
getting more and more complicated. There’s more and more stuff going on. There’s more and more
material flying around.>>And this is what the early
cosmos would’ve looked like. It was a pretty hairy place. There’s explosions all over
the place, powerful collisions. And formation of heavy
metals and elements. And these big clouds smashing into
each other with the extreme force.>>And so now we’re 9.6 billion
years from this initial explosion. You’re starting to see things are
kind of calmed down a little bit, just a little bit, because the
energy is now starting to relax. And so the mathematical
models have got that in place. And you’re starting to see
coalescence of different elements. And starting to see this thing kind
of come together and slowly cool.>>And it’s starting to look a little bit
more like the night sky, a little bit. And it’s [? QSing. ?] We’re now 30.2
billion years and we’re kind of done. And then what they did was
that they took this model, and then looked at the visible universe. And basically then, were
able to take that and overlay it with what you can see. And the fidelity is staggering, as to
how accurate the computer models are.>>Of course, the astrophysicists
and the research groups need even better fidelity
and even higher resolution. But if you think about what
I’ve been talking to you today through this little voyage through both
storage and structure and networking and stacks, the important thing is,
is scale-out computing essential? That was my original hypothesis–
back to our scientific method.>>I hope that at the early
part of this I would predict that I would be able to explain
to you about scale-out computing. And we kind of tested
some of those hypotheses. We went through this conversation. And I’m just going to say scale-out
computing is essential– oh, yes, very much yes.>>So when you’re thinking
about your codes, when you’re doing the CS50 final projects,
when you’re thinking about your legacy to humanity and the resources that we
need to be able to run these computer systems, think very carefully
about the FLOPS per watt, and think about the Chaos Monkey.>>Think about your snowflakes, don’t
do one-offs, reuse libraries, build reusable codes– all of the things
that the tutors have been teaching you in this class. These are fundamental aspects. They’re not just lip service. These are real things.>>And if any of you want to follow me,
I am obsessive with the Twitter thing. I’ve got to somehow give that up. But a lot of the
background information is on our research computing
website at rc.fas.harvard.edu.>>I try and keep a blog up to
date with modern technologies and how we do distributive
computing and so forth. And then our staff are always
available through odybot.org. And odybot is our little helper. He often has little
contests on his website too, where you can try and
spot him around campus. He’s the friendly little
face of research computing.>>And I’ll kind of wrap up there
and thank you all for your time. And I hope you remember that
scale-out computing is a real thing. And there are a lot of people
who’ve got a lot of prior art who will be able to help you. And all of the best of luck with
your future endeavors in making sure that our computing both
scales, is high performing, and helps humanity more
than anything else. So, thank you for your time.

Leave a Reply

Your email address will not be published. Required fields are marked *