Building Software at Google Scale: Bazel

Building Software at Google Scale: Bazel

[MUSIC PLAYING] HELEN ALTSHULER: All right. Hi, everyone. Thanks a lot for joining us. We’re really excited
to see you here. And yes, as Chelsea said,
I was looking forward to getting all 500
or 600 that wanted to get in because part of what
really gets us all excited here at Google and within
the Bazel team is getting a bigger
community involvement. This is an open-source product,
so we would love all of you to know what it is,
try it, use it, learn. And by the way, we have
a table at the end there. After the talks, should
you have questions or would you like to join our
user group that we can do more regular meetups
about specific questions on Bazel, feel free
to stop by that table. So I’m Helen Altshuler, and
I’m fairly new at Google. Just joined five months ago. And prior to Google,
I was the CTO of a fintech startup
called PeerIQ and also spent most of my career
at JP Morgan as a software engineer and afterwards as an
executive director for the risk technology and big
data analytics teams. But today we’ll talk
about our experience here. For me, what’s been
really interesting and unique to notice about
Google during my first few months was that
there’s a huge focus on developer efficiency,
productivity, and happiness. And not the fake focus that
I’ve seen in prior places where it’s more about scorecards
and metrics and dates, but more about tools and
tools built by engineers for engineers to make
them love what they do and more efficient in the
best type of technologies that they’re working at. And that’s really what
makes this company unique. And today, we want
to talk to you and share that experience
that the company has built over the past 10 years
about making its build system available for everyone. So why would you care? Well, first of all, we’ll
tell you a little bit more about why we care. And that’s because Google
has obviously a big scale to deal with. So we’ve got 2 billion lines
of code across 9 million files. We’ve got 25,000 engineers. That was at the end of
at some point last year, and now it’s grown a lot
more than that and something around 45,000 changes per day. And about a third
of those changes are actually by
engineers, and 2/3 are automated changes by our
automated packaging tools. But certainly for comparison, we
looked at some of the published data, and MS Windows overall is
about 50 million lines of code. So we are dealing with
a tremendous scale. And when you have
tremendous scale, you need your systems to
be efficient and fast. So for you, why should you
care about a build system? So you need one when your
team is more than two to three software engineers and
also when you use one or more– well, really, more than one
program and language because, at that scale, it’s
becoming more apparent that you need some structure
and some effectiveness. You can’t just rely on
linking everything by itself. And, of course, Google has
been doing this for 10 years, and it helped us grow,
and it scaled really well. And now, with the open-source
development of Bazel, which was released
last year, you can all take advantage of this. And it is a tool that’s
platform independent. It works on prem or
on any cloud platform and it has a lot
more flexibility. So for those of you– sorry. Actually, before we get to
talk about the build system, I’d like to learn a little
bit more about the audience. So how many of you are
software engineers? OK. A pretty big group here. Product managers? Smaller. Entrepreneurs, startups? OK. So we’ve got a pretty
good group here of people that care about building and
releasing software and making changes and developing products. But for those of you who are not
familiar with the build system. So build system is
basically a software that helps you build your software. It’s a more structural
tool that allows you to translate between machine
code and your source code. It also allows you to create
and understand the dependency so you don’t have
to manually inspect different kinds of
files and figure out what order your different
types of sources need to be working in. So there are multiple
build tools out there. Many of you may have
heard or use Make, Maven– there’s tools available
for almost every language– Gradle, and lots of other ones. But today we’re
going to talk about Bazel and what is Bazel, how
it works, its real internals. And that’s what Ulf is
going to tell us about. [APPLAUSE] ULF ADAMS: Thanks. I’m Ulf. First of all, thanks
for coming here. I’m the tech lead for
Bazel, or one of the tech leads, to be correct. I’ve been with the team
for 7 and 1/2 years. There is basically no
one who has been here for longer than me. And I do a lot of
technical stuff. I’m an engineer by heart. I now also manage
the Munich team, the Munich part of the team. We’re in two sites. We’re both in New
York and Munich. And so we’re doing a lot. All right. That was the wrong direction. Where are we? OK. All right. So what’s special about
Bazel, and why do we even have our own build system? And how is Bazel different? All right, let’s look again
at the challenge that we have. Google, at some
point in our history, we decided to keep
all of the source code in a single repository. And I don’t want to talk
about why we do that, but about what’s
involved with doing that. And so what does that
actually look like? We have 2 million lines–
2 billion lines of code. Now if you think of Google,
you think of Google Search, maybe, or of Gmail. So maybe you can think of some
things that these services do. What does Google Search do? What does Gmail do? Anyone? AUDIENCE: Search an email. ULF ADAMS: Search an email. Excellent. What else do these things do? AUDIENCE: Index. ULF ADAMS: Index. Yes. Profile. What? All right. AUDIENCE: Ads. ULF ADAMS: Ads. Show ads. OK. All right. So you have Google Mail and that
parses email and speaks SMTP and sends email
and handles email. And there’s Google Search,
which [INAUDIBLE] a lot of web pages and has a
huge search index. And then you have login. And it turns out Google Search
needs login as well, right? Google Search gives
you a better experience if you log in and is able to
search all your private data as well. And now coming back
to the big code base, there are libraries
that do this, right? It’s not a single big binary,
a single big thing that we just put in our data center. But it’s actually more like a
collection of micro services, of services and micro
services and then, in turn, a collection of libraries. So we have libraries
for everything. And a lot of these libraries
are shared between services. Not all of them,
but many of them. So there is really a lot of
structure in the code base. And Bazel uses this
structure to run the build. And this structure is explicit
in the Bazel configuration files. Let’s take a look at an example. So this is an– well,
a made-up example. But this is what it would
actually– could actually look like, right? You have a build file. You have a Java
binary rule in it. I do a lot of Java
work, so that’s what I’ve used for an example. But it could be
anything, really. Then you have a couple
of source files. In this case, we actually
look for all the source files in the current directory. And you have dependencies
and you have a main class. And then your Java
file looks like this. So what does Bazel do with that? All right. So it looks at the build file,
it loads it, and it says, this is a package. And this package
contains one rule. And it also contains
one source file. And it contains a generated
file that serves the output file of the Java binary. And there is nothing
else in this package. So Bazel doesn’t
actually go out and look for all the source files
that you have in your tree, but it only looks at the
build files and maybe list a directory or so. And the key idea here is that,
if you have such a big code base, you need to
make sure that you can make sense of the code base
very quickly and efficiently. So you only want to read
the parts of the code base that are actually
needed for your build. If you compare that with
Make, for example, Make reads the entire makefile. If you have a single
makefile for your project, at the end of
reading the makefile, Make knows everything,
which is great. Except if your code base
goes beyond a certain size, you start having– just
reading the makefile can start to take minutes. That’s actually what we had. We were using Make before
we started using Bazel, and it took a long time. And so this is one of the
things that we definitely wanted to do. And so one key attribute
here is that we also know about all the
dependencies of this rule. So the rule depends
on the source files, and it also depends on– the
dependency is declared here. In this case, it’s bar. And then the rule
itself, the Java binary, also depends on tools,
like the Java compiler and Java packaging tools,
that sort of thing. But there is no
other dependency. There is no implicit
dependency on something else. We don’t go out and try to
figure out what you want to do but rely on it, everything
being explicitly specified. So when we look at
this build file, we know exactly
what we need to do. We read this build file, we read
the build file for the tools, for the dependencies, and
then we start building. And so I want to give it
sort of a walkthrough to how Bazel builds stuff. So you tell Bazel,
please build foo. All right. So Bazel starts with that. Now in order to
build foo, it needs to read the build file for foo. OK. Let’s do that. And then it needs to
list the directory. Right? So find the other source
files in this case. OK. So now we have the package. We know what it is. OK. We know that it’s a
dependency on bar, so it reads the bar
build file as well. In this case,
there isn’t a glob. Someone said
explicitly, this depends on, in which
case Bazel doesn’t go out and look for files. All right. So we’ve got the rule now. We know what it is. We’ve analyzed its dependencies. Now we want to build it. So we go back up
to that first node and say, OK, now I
want to build it. So what do we need
to do to build it? I need to do a packaging step. That’s the last
step in the process. And this packaging step depends
on having compiled the files. So we see that we now have this
dependency on the compiler, and then we start compiling
the files in the right order. And then we’re done. All right. So that’s good. Now what happens if you
modify a source file? Let’s take a look at that. Modify a foo main dot Java. And then Bazel
knows because it has tracked all those
dependencies exactly which actions it has to re-execute. It doesn’t have to
look at anything else. And so you can imagine that,
if your software is small, it doesn’t really matter. But as your software
gets bigger, it becomes more
and more important that you only do the
minimum amount of work absolutely necessary for
whatever changes you’ve just made, right? So for those who don’t know
it, most software developers type text in a text
editor or in IDE. And you usually don’t change
everything at once, right? You change one file. You want to fix one bug. You want to add one feature. So you go to the right
file and start editing it, and then you rebuild and
then you rebuild and rebuild and maybe you want to
run some tests on it or you want to run
it on your computer. And so this cycle needs to be
as fast as possible for you, as a software engineer,
to be productive. All right. So let’s take a look at the
more complicated example, adding a file. Let’s say we added a file. Now this foo node
here with a slash at the end, that represents
the directory or the directory listing. And we can figure out that the
directory listing has changed. On Linux systems, we use the
inotify API for that, right? The Linux kernel
has this API, which allows us to ask, tell
me all the files that have changed in this directory. Tell me every time something
in this directory structure changes. And so that fires
when we add a file, and so we know that this
directory listing is out of date. So we redo the
directory listing. And we know exactly which
actions have depended on that. And then we reconstruct
the package. We create the new actions. And then when we
execute it, of course, we take the new
file into account. And we’re done. And again, we’re doing
the minimum amount of work necessary to update
from your change from the previous
state plus your change to the complete new change. And in any case, we’ve
never looked at bar. We’ve never even considered
the existence of bar. All right. So how do we make sure
that that is correct? We need to know all
the dependencies. We need to know exactly what
all the dependencies are. And then when we
run an action, we need to make sure that you don’t
accidentally or intentionally access files that you didn’t
declare as a dependency. And so we use sandboxing on all
the major platforms to do that. And for us, we’ve always
seen that correctness equals performance. If we can ensure that the build
system is sufficiently correct that it gets this right
99.999% of the time, then our developers
never need to do a clean. They never need to
restart from scratch. They always get an
incremental build, which is fast because
we can make sure that we do just the minimum
amount of work necessary. And so if you look at our code
base, 45,000 changes per day is a lot and we build
everything after every change– well, that’s not true. What we actually
do is we only build the stuff that was changed. You change the main parser, we
only rebuild the main server. Everything else we
don’t even touch. And so we can do that with
that number of changes because we do the minimum
amount of changes at each step. All right. I want to talk a little bit
more about time and space complexity. And I hope that everyone will be
able to follow even though this is a bit technical. Let’s say we have
a Java program. I’m sorry to everyone
who doesn’t write Java, but it’s my most used language. All right. So we have a
sequence of library, library A, then library
B depends on library A, library C depends on library
B, and so on up to our binary. And so for each of these,
we have some source files that we want to compile. And so, in order to
compile the source files, you need to know
the dependencies. So when you compile A, you only
compile A. Oh, that’s easy. When you compile B, you need to
make A available for the source files for the compiler
to be able to compile B. When you compile C, you
need to make the libraries A and B available. Now if you look at
this, the first library has one thing that it looks
at, then two, then three, then four, then five. And if you sum that up, in
general 1 plus 2 plus 3 and so on plus n is n times
n plus 1 over 2, which is something
something and squared. And in terms of
complexity, n squared is not that bad, except if
you have a large code base. The larger your code
base, the worse this gets. And it’s not just two times
the size of the code base. That’s two times worse. But it’s actually
four times worse. And so this is a
problem that we’ve been struggling with
for years, and we’ve been working on this–
it sounds weird to say we’ve worked on this
for years because it seems like a simple problem. But you need to apply
this to every programming language you support. And you need to make sure that
you don’t– we have hundreds of people contributing to the
code base and we need to make sure that everyone is aware
of this problem and that they follow the coding patterns
that are needed to solve this. All right. So this is the solution, right? Instead of making a
copy of everything for, you just give a pointer to
whatever it was before here. We don’t really care,
it’s just something. And so that solves
the space problem. But at some point you
actually run the compiler and you need to make
all of this available. You have to iterate eventually. All right. Let’s take another example. You don’t just have one binary,
but you have many binaries. You have many tests, hopefully. Everyone has a
lot of tests, yes? OK. And so– sorry. What can happen here is
that, at each of the tests, you iterate over all
of your dependencies. And then if you have n
over two dependencies and n over two tests, you
have n/2 times n/2, and you get, again,
a quadratic regime, which is, again, problematic. So I wanted to give an example
of how this can kill you, because it can and it does. This is the C compiler. This is GCC. This is the performance of GCC. There is a specific
case where the C compiler runs into a quadratic
regime, and then this happens. The numbers on the left
are seconds, right? So this is the time needed to
compile a single C source file, and as you can see, it goes up
like this if you do it wrong. But if you do it right, it
looks like the green line on the bottom. So in Bazel, we’ve
been working on this, and these are our times. This is not the full build. This is just what
we call analysis. These are the lines
for a CC library and Java library and pi library. The numbers are not
quite up to date, but I checked that they
are still quite close. On the bottom is the
number of dependencies. If you have 1,400 libraries
in a row in your project– I don’t think anyone will
do that, not even us– but if you had,
Bazel would still be able to analyze
that in under a second because we make
sure that we don’t grow quadratically
but only linearly in the number of dependencies. Build systems can’t avoid this. There is no build system
that can get around this. The only thing you can
do is try to push it a little more towards linear
and away from quadratic. Now for us, we don’t
just build stuff. We have a data center
worth of machines. We don’t need a single machine. Let’s just distribute
it across a data center. And that helps us a
lot because it gives us a better factor, right? Let’s say you have 100 machines. You distribute your build
across 100 machines. And your performance
scales quadratically, but you only have 100 libraries. Then if you can do
it right, you can parallelize all the compilers
so they’re all in parallel and get much better
performance, obviously. All right. So I’ve talked about Java. But what about binary
lambda calculus? Anyone using that? I was looking for the
most unusual programming language I could find. No? OK. All right. What about Scala? Anyone using Scala here? All right. So when you have
a build system, we do a lot of language-specific
work, right? At Google, we use the
major four languages like Java and C
and Go and Python. But if you want to
use the build system, you might want to use
a different language. You might want to use it
for Scala, for FORTRAN, or for Haskell. And we support that. We have an extension
language that allows you to extend Bazel for
pretty much any language you can think of. We have literally dozens
of examples of languages that we can support with
this extension mechanism. This extension mechanism
is a little bit unusual. It doesn’t allow you
to freely program, because we want to make
sure that we can still know all of the dependencies
that your software has and that we can do incremental
builds correctly in all cases. So we’ve intentionally
restricted this so that you can only do
things that are safe. We can make sure
that, whatever you do, we can correctly do
incremental builds in all cases and also correctly
parallelize your build. This is a brief example. I’m not going to
go through this. But you can see that
it’s reasonably easy to add support for
something, in this case, that runs shell commands. So in some cases you have a
case where you have a file and you want to do
something with it, and the easiest way you
can think of is shell. And when that happens,
you can use a [INAUDIBLE]. All right. And then this is what your
build file looks like. So it looks similar
to what we had before. And this is an
advantage because it means that, if you’re
a company and you have multiple people
working on your projects and maybe they work with
different languages, the build files all
look very similar. And they are all
similar in ways that make it easy to pick up a
different part of your project. We talk a lot to people who
would like to use Bazel, and they often
tell us, well, they are currently running with a
combination of Make and Maven and shell scripts and Gradle. And so one of the
advantages of Bazel is that it can handle pretty
much all programming languages. I mean, maybe not literally
all, I don’t know. We haven’t tried
all of them yet. But a lot of them. And it makes it
very easy to move between different
parts of the project and between different languages. And we have color
roots, actually, and GOLD roots and Rust roots. And you can go to GitHub and
see the full list of languages that we already
have support for. All right. Talked about that. Talked about that. Workers. Now workers are
pretty cool, and I want to talk about them
because they’re so cool. So in some cases, you want
your builds to be even faster. And now if you look
at your compiler, your compiler has to
do all sorts of things before it can even
compile your code. It loads a standard
library and initializes itself and so on and so forth. And what we support
is we support keeping a process, a hot
process around, a hot compiler process around, so that it
can cache the standard library and other things in memory
and then, especially for incremental builds, this
is particularly interesting because then we reduce the
path on incremental builds by quite a bit. Of course, we have IDE
integration, not all of them yet but we’re working on that. And– all right. So build systems. What’s special about Bazel? Well, we try to
make sure that we do the minimum amount of
work necessary every time, and we make sure
that we know all of the dependencies that go
into it so that we can do that correctly. We sandbox actions so that
you can’t accidentally depend on something
that you didn’t declare. In build systems, we
believe that correctness is driving performance. We need to be very careful
to avoid quadratic behavior. Otherwise, you end
up like C compiler. And we have an
extension language, so you can support pretty much
any language, any programming language you want. All right. Thank you. And over to David. [APPLAUSE] DAVID STANKE: Thank you. So there’s a giant
clock here screaming at me, saying, dude, it’s late. Everyone wants to get back
to drinking and eating. So in true Bazel
fashion, I will not repeat anything
you’ve already heard. I will only tell
you new information. My name is Dave. I’m the product
manager for Bazel. I’ve been at Google
for about three years. And prior to that,
I worked with a lot of different teams, lot of
different tools and languages. And what really struck
me when I came to Google was how productive and happy the
engineers were, and, of course, these are mutually
reinforcing things. And the tools that they
used is a really big part of what makes them so engaged. And one of them is called
Blaze, which Bazel is built on. So it’s really
exciting for me that we are able to take this and
extend it to the community and offer that
productivity to everyone across languages,
across platforms. Bazel’s a little funny as
a new product, in a sense, because it’s been around for
over 10 years from its genesis. In 2008, it became
the default at Google. So it’s really very
mature but, of course, relatively new to the
open-source world. And so we’re looking
to rapidly mature it in the open-source
environment into something that can provide that
same productivity as it has at Google. So our current release
is version 0.4.1. We’re not at 0.2 yet, right? And what we’ve introduced
in that release is the persistent Java compiler. So Ulf was talking
about the ability to maintain a persistent
compiler in memory, and that incremental
builds much faster. Oh, look, now I’m
repeating information. We’ve introduced
sandboxing for OS 10. So sandboxing
means that we won’t rely on assets from the
underlying operating system. And that’s really
important for our story around the repeatable,
reproducible build. So when I’m building something,
my next-door neighbor’s building something,
we want it to behave the same way for
the same code base, and even extending that
to cross-platform builds. Couple things that are recent. One is Windows. We’ve added support for Windows. You can build on Windows. We support Windows 7 Plus! 64-bit variants. Relatively new but
maturing very rapidly. And the IDE support
that Ulf mentioned. What we have is a framework for
IDE support and then specific implementations
for popular IDEs. We’d love to see that
continue to grow. So what’s coming next? In the next few
versions, so here are some of the highlights. We’re going to keep working
to mature Windows support. There’s still work to be done on
OS X. We want to really step up the mobile side of things. Again, our goal
here is to make it so that Bazel can be the
only build system you need, and so therefore we want to
have really solid support across all the platforms. And Android and iOS are
supported by Bazel as of today, but there’s work
to be done there. We’re going to
bundle the GRE in. So Bazel depends on Java. And what that means is
that, to install it, you have to have a running
GRE on your system. We’re going to bundle that
in, make it easier to install, easier to port across systems. We’re adding support
for code coverage. And you run that locally. It spits out this really
nice graphical report of how your coverage is. Some caching, things
like Ulf spoke about. And then the one I think
is the most exciting here is fourth down– I should
have put these in order– the searchable community
repository of Skylark rules. So the idea that we want
for Bazel is it’s not a C build system. It’s not a Java build system. It’s not a FORTRAN build system. It’s all of them. And that’s the
impetus for Skylark is that as the community
of a given language. For example, Rust
is a great example, where that community said, we
want to make work for Bazel. They got together. You can go on GitHub and find
a full set of Bazel rules. And we certainly
were aware of that. We may have helped them. I’m not sure. But this is something
that they as a community were able to do with Bazel. And we want that to continue
to grow and make it so that, whether it’s a new
language you’re creating, trying to support
an older language, Bazel is the place to go. Bazel gives you all that
plumbing, gives you the engine, and you write your
rules to make it work. We had a great example. We were talking
to someone earlier who uses a bunch of Java
and a bunch of FORTRAN. And the beauty of Bazel
is you can use both. Java is supported
out of the box. FORTRAN is something
you can write rules for and you’re good. So then we get to Bazel 1.0. And Bazel 1.0 is not so
much defined by features as it is defined by an
approach to process. We’re going to make GitHub
be the primary core source for where everything happens. We really want to
engage everyone. We need help here. And we’re going to have that
formalized process for how the software is
developed, which leads to being able to have a
long-term support policy so that we can communicate
deprecations. We don’t expect that Bazel’s
going to be changing versions frequently. We want this to be a really
solid platform upon which you can build
those Skylark rules and iterate those at the
speed that you need them. Then from there, we move on to
Bazel 2.0, which really is just the evolution of
that Bazel vision. At that point we want it to be
completely language agnostic, so we’re going to take
those core rules– right now some of the language had
built-in rules– we’re going to take those and
move those out to Skylark, make the core really small and
perform it so that everything happens through the
extension framework. And these things
can then mature at their own appropriate cadences. We want it to be big. We want Bazel to be
widely used and loved. So we hope everyone here will
try it out, experience it, give us feedback, help it mature. And we want it to be
fill in the blank. This is something
that we as Google feel like it’s important that
we work with the community, evolve the state of the art
of software development. So whatever it is
that we as a community need from a development tool,
please, the forums are open. Tell us what you’re looking for. Propose changes. And let’s make this new
approach to building software the incremental, the
parallelizable, something that supports huge code
bases, huge teams, that supports the kind of software
development that we do now. Let’s keep it going with Bazel. So if you haven’t tried it,
what are you waiting for? has
everything you need. You can compile from source
or, if you’re lazy like me, you can get the installers
for all the platforms. Great tutorials
from hello, world all the way up to
complicated stuff. And, of course, you can grab
Skylark rules for any language that you’re working
on from GitHub. Give us a star. We love positive reinforcement. And there’s a public mailing
list at So please join in, listen to the
conversations, and contribute. [MUSIC PLAYING]

7 thoughts to “Building Software at Google Scale: Bazel”

  1. What am I waiting for? The distributed cache (, supposedly a blocker for the 0.5 release, and the major source of Blaze's speedup. Yet it didn't even get a mention in this talk.

  2. It's 2016 and we still writing and supporting our build scripts in text editor even in the best IT company in the world. Why not to create a component gui for that?

  3. this video is helpful in explaining why we need it, but too much about propagandas of Google, too few details of how it works internally

Leave a Reply

Your email address will not be published. Required fields are marked *