Web Performance Testing at YouTube

By | December 9, 2019

Hi. I’m Rick Viscomi, and
I make YouTube fast. In this video,
I’m going to share how we test web
performance and a few tips for doing the same
on your own website. The two test methodologies
that we’re going to look at are real user monitoring, or
RUM, and synthetic testing. Real user monitoring,
just as it sounds, gathers performance
data from real users as they browse the site. For YouTube, this provides
us with the ground truth about how users are
experiencing the site. Synthetic testing, on the other
hand, is more like a laboratory and is commonly used on demand
to analyze our particular use case. Both methodologies
collect performance data, but given these
distinctions, RUM is especially useful
for identifying how fast the page is, and
synthetic for identifying how to make it faster. The simplest way to gather
and analyze RUM data is with one of the existing
analytics products. In fact, Google Analytics
can do this for you. At Google, we use
a custom solution called client side
instrumentation, or CSI. RUM works on three basic levels. First, we instrument
our client side code to provide timing and page
state data to the RUM JavaScript library. Then, the library
packages up the data and sends it off to the
Analytics server for later use. And finally, the
Analytics server provides developers with a
dashboard to view the results. Out of the box, most RUM tools
measure the general page events published by their
navigation timing API. But for us, one of the most
important performance metrics is how quickly a video plays. The metrics you collect
should be customized based on what your
application does and what your users are
actually waiting for. It’s also important to include
any contextual information that might have an effect
on performance. For example,
YouTube users may be exposed to
experimental features, and we would need to
know what effect that might have had on performance. By keeping track
of these variables, we’re better able
to get insights into the performance of
particular demographics. Also, note that the CSI
beacon is sent unobtrusively after the critical path
and video playback. This helps to ensure
that testing does not interfere with the
user experience. To analyze the performance data,
we turn to the CSI dashboard. Here, we’re able to look at the
change in performance over time and compare experiments
against controls. For example, in
this chart, we’re comparing the results of an
experiment against the baseline control. The experiment times
are lower or faster until this last point. We can use the CSI
dashboard to slice data by various demographics
to understand why this is happening, and
when we break the data down by browser, it becomes
clear that browser B is responsible for
the regression. It’s also worth mentioning
that, because we gather data from so many users–
all of whom have different devices on
different networks– we typically look at
the 50th percentile, or the median value, to give
us a representative view of the performance data. OK. That’s it for RUM. Now let’s compare it
to synthetic testing. For that, we use WebPagetest,
which is a free public tool. And instead of gathering
data from real users, we configure synthetic tests
to a particular demographic. The tests run in a
laboratory-like environment to mitigate the effective
network anomalies, and this is achieved
through repetition. So tests that are run many times
are considered to be reliable. And, finally, when
the results are in, we’re able to analyze
our application specific metrics along
with the test network log. WebPagetest gives
you full control over the type of use case
for which you want to test. For example, we may be
interested in the Watch page performance in Chrome
over a DSL connection. And being a free public
tool, WebPagetests are run on ordinary computers
connected to the internet. To test YouTube, however, we
need more privileged access to beta versions of the site. So for this reason, we run
our own private instances of WebPagetest inside of Google. The results are shown on
a waterfall diagram, which is a visualization of
the network activity. In this view, we’re able to
precisely see the activity on the critical path
to video playback. And, finally, WebPagetest
also annotates the waterfall with our application
specific metrics. This gives us the
necessary context to better understand
what exactly is contributing to slow values. So that’s it for our look
at synthetic testing. Remember that RUM is only
as useful as the metrics you provide. CSI is just one of many
tools that does it, but your solution should be just
as unobtrusive and lightweight. Synthetic tests
should be configured to represent common use cases,
like a particular browser, connection, or page demographic. Also, by locating synthetic
test machines inside of your own network, you
can validate performance on a staging server
before launching. To learn more about the
NavigationTiming API, check out the Mozilla Developer
Network article at this URL. Or for more information
on using WebPagetest, check out the link
over here, and also look out for my book
coming later this year. Thanks for watching.

4 thoughts on “Web Performance Testing at YouTube

  1. Calin Lupa Post author

    Nice, great content, thanks for sharing Rick! Looking forward for your book.

  2. Google Developers Post author

    I/O Byte: Web Performance Testing at YouTube
    /by @Rick Viscomi  #iobytes   #youtube   #perfmatters  

    Even though YouTube operates at a scale of over a billion users per month, web performance testing follows the same core principles that you can apply to your own site. In this video, learn about how we measure the page load speed and how we instrument tests to make the experience even faster.

  3. Jesse Holden Post author

    It can't be said enough really, the better your web performance, the more succesfull your web business. I have been using https://wao.io/ for a while now and it's automated web performance optimization is just unreal, definitly recomend checking it out!


Leave a Reply

Your email address will not be published. Required fields are marked *