Google I/O 2021 keynote in 16 minutes

good morning everyone it's great to be back at io today i'm excited to share our latest breakthrough in natural language understanding lambda it's a language model for dialogue applications and it's open domain which means it's designed to converse on any topic and while it's still in research and development we've been using it internally to explore novel interactions for example say you wanted to learn about one of my favorite planets pluto lambda already understands quite a lot about pluto and millions of other topics let's listen to a conversation the team had with pluto a few days ago i'm so curious about you i sense your excitement ask me anything tell me what i would see if i visited you would get to see a massive canyon some frozen icebergs geysers and some craters it sounds beautiful i assure you it is worth the trip however you need to bring your coat because it gets really cold i'll keep that in mind hey i was wondering have you ever had any visitors yes i have had some the most notable was new horizons the spacecraft that visited me let's break down what made it feel so natural first learn concepts as you saw the model talked about the new horizon spacecraft and the coldness of space lambda synthesized these concepts from its training data because none of the responses were predefined lambda answered with sensible responses keeping the dialogue open-ended natural conversations are generative and they never take the same path twice and lambda is able to carry a conversation no matter what we talk about yet it's still early research so it doesn't get everything right sometimes it can give nonsensical responses imagining pluto doing flips or playing fetch with its favorite ball the moon other times it just doesn't keep the conversation go going we believe lambda's natural conversation capabilities have the potential to make information and computing radically more accessible and easier to use we look forward to incorporating better conversational features into products like google assistant search and workspace lambda is a huge step forward in natural conversation but it is still trained only on text when people communicate with each other they do it across images text audio and video so we need to build models that allow people to naturally ask questions across different types of information these are called multimodal models for example when you say show me the part where the lion roars at sunset we will get you to that exact moment in a video advances in ai are helping us reimagine what a map can be but now you can also use it to explore the world around you you'll be able to access live view right from the map and instantly see details about the shops and the restaurants around you including how busy they are recent reviews and photos of those popular dishes in addition there are a host of new features coming to live view later this year we're adding prominent virtual street signs to help you navigate those complex intersections second we'll point you towards key alarm landmarks and places that are important for you like the direction of your hotel third we're bringing it indoors to help you get around some of the hardest to navigate buildings like airports transit stations and malls indoor live you will start rolling out in top train stations and airports in zurich this week and will come to tokyo next month we're bringing you the most detailed street maps we've ever made take this image of columbus circle one of the most complicated intersections in manhattan you can now see where the sidewalks the crosswalks the pedestrian islands are something that might be incredibly helpful if you're taking young children out on a walk or absolutely essential if you're using a wheelchair thanks to our application of advanced ai technology on robust street view and aerial imagery we're on track to launch detailed street maps in 50 new cities by the end of the year so we're making the map more dynamic and more tailored highlighting the most relevant information exactly when you need it if it's 8 a.m on a weekday we'll display the coffee shops and bakeries more prominently in the map while at 5 pm we'll highlight the dinner restaurants that match your tastes you'll start seeing this more tailored map in the coming weeks people have found it really useful especially during this pandemic to see how busy a place is before heading out now we're expanding this capability from specific places like restaurants and shops to neighborhoods with the feature called area business say you're in rome and want to head over to the spanish steps and its nearby shops with area business you'll be able to understand at a glance if it's the right time for you to go based on how busy that part of the city is in real time area busyness will roll out globally in the coming months let's talk about all the ways we're innovating in shopping soon on chrome when you open a new tab you'll be able to see your open carts from the past couple of weeks we'll also find you promotions and discounts for your open carts if you choose to opt in your personal information and what's in your carts are never shared with anyone externally without your permission we capture photos and videos so we can look back and remember there are more than four trillion photos and videos stored in google photos but having so many photos of loved ones screenshots selfies all stored together makes it hard to rediscover the important moments soon we're launching a new way to look back that we're calling little patterns little patterns show the magic in everyday moments by identifying not so obvious moments and resurfacing them to you this feature uses machine learning to translate photos into a series of numbers and then compares how visually or conceptually similar these images are when we find a set of three or more photos with similarities such as shape or color we'll surface them as a pattern when we started testing little patterns we saw some great stories come to life like how one of our engineers traveled the world with their favorite orange backpack or how our product manager christy had a habit of capturing objects of similar shape and color we also want to bring these moments to life with cutting edge effects last year we launched cinematic photos to help you relive your memories in a more vivid way cinematic moments will take these near duplicate images and use neural networks to synthesize the movement between image a and image b we interpolate the photos and fill in the gaps by creating new frames the end result is a vivid moving picture and the cool thing about this effect is it can work on any pair of images whether they were captured on android ios or scanned from a photo album in addition to providing personalized content to look back on we also want to give you more control we heard from you that controls can be helpful for anyone who has been through a tough life event breakup or loss these insights inspired us to give you the control to hide photos of certain people or time periods from our memories feature and soon you'll be able to remove a single photo from a memory rename the memory or remove it entirely instead of form following function what if form followed feeling instead of google blue we imagined material you a new design that includes you as a co-creator letting you transform the look and feel of all your apps by generating personal material palettes that mix color science with a designer's eye a new design that can flex to every screen and fit every device your apps adapt comfortably every place you go beyond light and dark a mode for every mood these selections can travel with your account across every app and every device material u comes first to google pixel this fall including all of your favorite google apps and over the following year we will continue our vision bringing it to the web chrome os wearables smart displays and all of google's products we've overhauled everything from the lock screen to system settings revamping the way we use color shapes light and motion watch what happens when the wallpaper changes like if i use this picture of my kids actually getting along for once i set it as my background and voila the system creates a custom palette based on the colors in my photo the result is a one of a kind design just for you and you'll see it first on google pixel in the fall starting from the lock screen the design is more playful with dynamic lighting pick up your phone and it lights up from the bottom of your screen press the power button to wake up the phone instead and the light ripples out from your touch even the clock is in tune with you when you don't have any notifications it appears larger on the lock screen so you know you're all caught up the notification shade is more intuitive with a crisp at a glance view of your app notifications whatever you're currently listening to or watching and quick settings that give you control over the os with just a swipe and a tap and now you can invoke the google assistant by long pressing the power button and the team also reduced the cpu time of android system server by a whopping 22 percent and with android 12 we're going even further to keep your information safe to give people more transparency and control we've created a new privacy dashboard that shows you what type of data was accessed and when this dashboard reports on all the apps on your phone including all of your google apps and we've made it really easy to revoke an app's permission directly from the dashboard we've also added an indicator to make it clear when an app is using your camera or microphone but let's take that a step further if you don't want any apps to access the microphone or camera even if you've granted them permission in the past we've added two new toggles in quick settings so you can completely disable those sensors for every app android's private compute core enables things like now playing which tells you what song is playing in the background and smart reply which suggests responses to your chats based on your personal reply patterns and there's more to come later this year all of the sensitive audio and language processing happens exclusively on your device and like the rest of android private compute core is open source it's fully inspectable and verifiable by the security community with a single tap you can unlock and sign into your chromebook when your phone is nearby incoming chat notifications from apps on your phone are right there in chrome os and soon if you want to share a picture one click and you can access your phone's most recent photos to keep movie night on track we're building tv remote features directly into your phone you can use voice search or even type with your phone's keyboard we're also really excited to introduce support for digital car key car key will allow you to lock unlock and start your car all from your phone it works with nfc and ultra wideband technology making it super secure and easy to use and if your friend needs to borrow your car you can remotely and securely share your digital key with them car key is launching this fall with select google pixel and samsung galaxy smartphones and we're working with bmw and others across the industry to bring it to their upcoming cars that was a quick look at android 12 which will launch this fall but you can check out many of these features in the android 12 beta today let's go beyond the phone to what we believe is the next evolution of mobile computing the smartwatch first building a unified platform jointly with samsung focused on battery life performance and making it easier for developers to build great apps for the watch second a whole new consumer experience including updates to your favorite google apps and third a world-class health and fitness service created by the newest addition to the google family fitbit as the world's largest os we have a responsibility to build for everyone but for people of color photography has not always seen us as we want to be seen even in some of our own google products to make smartphone photography truly for everyone we've been working with a group of industry experts to build a more accurate and inclusive camera so far we've partnered with a range of different expert image makers who've taken thousands of images to diversify our image data sets helped improve the accuracy of our auto white balance and auto exposure algorithms and given aesthetic feedback to make our images of people of color more beautiful and more accurate although there's still much to do we're working hard to bring all of what you've seen here and more to google pixel this fall we were all grateful to have video conferencing over the last year it helped us stay in touch with family and friends and kept businesses and schools going but there is no substitute for being together in the room with someone so several years ago we kicked off a project to use technology to explore what's possible we call it project star line first using high resolution cameras and custom built depth sensors we capture your shape and appearance from multiple perspectives and then fuse them together to create an extremely detailed real-time 3d model the resulting data is huge many gigabits per second to send this 3d imagery over existing networks we developed novel compression and streaming algorithms that reduce the data by a factor of more than 100 and we have developed a breakthrough light field display that shows you the realistic representation of someone sitting right in front of you in three dimensions as you move your head and body our system adjusts the images to match your perspective you can talk naturally gesture and make eye contact it's as close as we can get to the feeling of sitting across from someone we have spent thousands of hours testing it in our own offices and the results are promising there's also excitement from our lead enterprise partners we plan to expand access to partners in healthcare and media thank you for joining us today please enjoy the rest of google i o and stay tuned for the developer keynote coming up next i hope to see you in person next year until then stay safe and be well

As found on YouTube

Google I/O Keynote (Google I/O ’17)

[MUSIC PLAYING] [VIDEO PLAYBACK] [PAPER CRUMPLING] [MUSIC PLAYING] [SQUEAK] [SQUEAK] [SEAGULL CRYING] [CARS HONKING] [ZAP] [CHALK ON CHALKBOARD] [CAR HONKING] [SCRAPING] [TEARING] [CHEERING AND APPLAUSE] [CHIMES] [FOOTSTEPS] [BIRDS CHIRPING] [TAPPING] – Hm. [BIRDS CHIRPING] [POP] – [GASP] [CHUCKLES] [MUSIC PLAYING] – Hm? [BIRDS CHIRPING] [HEAVY FOOTSTEPS] – Mm. [TING] [THUNDERING IN DISTANCE] [RAINFALL] [THUNDER] [THUNDER] – [STRAINING] [THUNDER] – [GASP] – [MANY STRAINING] – [SIGH] – Hmm. [GLEAMING] – Huh? – Oh? – [GASP] Hmm. [CLACK] – Woohoo! Whoa. [ROCKETING] [TAP] [THUMP] [END PLAYBACK] [APPLAUSE] SUNDAR PICHAI: Good morning. Welcome to Google I/O. [CHEERING] AUDIENCE: I love you, Sundar! [LAUGHTER] SUNDAR PICHAI: I
love you guys, too. [LAUGHTER] Can't believe it's
one year already. It's a beautiful day.

We're being joined
by over 7,000 people, and we are live streaming
this, as always, to over 400 events
in 85 countries. Last year was the 10th year
since Google I/O started, and so we moved it closer
to home at Shoreline, back where it all began. It seems to have gone well. I checked the Wikipedia
entry from last year. There were some
mentions of sunburn, so we have plenty of
sunscreen all around. It's on us. Use it liberally. It's been a very busy year
since last year, no different from my 13 years at Google. That's because
we've been focused ever more on our core mission
of organizing the world's information. And we're doing it for everyone. And we approach it by
applying [? deep ?] computer science and technical
insights to solve problems at scale.

That approach has served
us very, very well. This is what allowed
us to scale up seven of our most
important products and platforms to over a billion
monthly active users each. And it's not just
the scale at which these products
are working, users engage with them very heavily. YouTube, not just has
over a billion users, but every single day, users
watch over 1 billion hours of videos on YouTube. Google Maps. Every single day, users navigate
over 1 billion kilometers with Google Maps. So the scale is
inspiring to see, and there are other products
approaching this scale. We launched Google
Drive five years ago, and today, it is over 800
million monthly active users. And every single week, there
are over 3 billion objects uploaded to Google Drive.

Two years ago at Google I/O,
we launched Photos as a way to organize user's photos
using machine learning. And today, we are over
500 million active users, and every single day, users
upload 1.2 billion photos to Google. So the scale of these
products are amazing, but they are all still
working up their way to what's Android, which
I'm excited as of this week, we crossed over 2 billion
active devices of Android.

[APPLAUSE] As you can see, the robot is
pretty happy, too, behind me, so it's a privilege to
serve users of this scale. And this is all
because of the growth of mobile and smartphones, but
computing is evolving again. We spoke last year about this
important shift in computing from a mobile first to
a AI first approach. Mobile made us reimagine every
product we were working on. We had to take into account that
the user interaction model had fundamentally changed,
with multi-touch, location, identity, payments, and so on. Similarly, in a
AI first world, we are rethinking all our products
and applying machine learning and AI to solve user problems. And we are doing this across
every one of our products. So today, if you
use Google Search, we rank differently
using machine learning.

Or if you're using Google
Maps, Street View automatically recognizes restaurant
signs, street signs, using machine learning. Duo with video calling
uses machine learning for low bandwidth situations. And Smart Reply and Allo last
year had great reception. And so today, we
are excited that we are rolling out Smart Reply to
over 1 billion users of Gmail. It works really well. Here's a sample email. If you get an email like this,
the machine learning systems learn to be
conversational, and it can reply, I'm fine with
Saturday, or whatever. So it's really nice to see.

Just like with every platform
shift, how users interact with computing changes. Mobile brought multi-touch. We evolved beyond
keyboard and mouse. Similarly, we now
voice and vision as two new important
modalities for computing. Humans are interacting
with computing in more natural and immersive ways. Let's start with voice. We've been using
voice as an input across many of our products. That's because computers
are getting much better at understanding speech. We have had significant
breakthroughs, but the pace, even
since last year, has been pretty amazing to see. Our word error rate
continues to improve, even in very noisy environments.

This is why if you speak to
Google on your phone or Google Home, we can pick up
your voice accurately, even in noisy environments. When we were
shipping Google Home, we had originally planned to
include eight microphones so that we could accurately
locate the source of where the user was speaking from. But thanks to deep
learning, we use a technique called neural beamforming. We were able to ship it
with just two microphones and achieve the same quality. Deep learning is what allowed
us about two weeks ago to announce support for
multiple users in Google Home, so that we can recognize up
to six people in your house and personalize the experience
for each and every one. So voice is becoming
an important modality in our products. The same thing is
happening with vision. Similar to speech, we are
seeing great improvements in computer vision. So when we look at
a picture like this, we are able to understand the
attributes behind the picture.

We realize it's your
boy in a birthday party. There was cake and
family involved, and your boy was happy. So we can understand
all that better now. And our computer
vision systems now, for the task of the
image recognition, are even better than humans. So it's astounding
progress and we're using it across our products. So if you used the
Google Pixel, it has the best-in-class camera,
and we do a lot of work with computer vision. You can take a low light picture
like this, which is noisy, and we automatically make
it much clearer for you.

Or coming very soon, if you
take a picture of your daughter at a baseball game, and there
is something obstructing it, we can do the hard work
remove the obstruction– [APPLAUSE] –and– [APPLAUSE] –have the picture of what
matters to you in front of you. We are clearly at an
inflection point with vision, and so today, we are
announcing a new initiative called Google Lens. [APPLAUSE] Google Lens is a set of
vision-based computing capabilities that can understand
what you're looking at and help you take action
based on that information.

We'll ship it first in
Google Assistant and Photos, and it'll come to
other products. So how does it work? So for example, if
you run into something and you want to know
what it is, say, a flower, you can invoke Google
Lens from your Assistant, point your phone at it, and we
can tell you what flower it is. It's great for someone
like me with allergies. [LAUGHTER] Or if you've ever been
at a friend's place and you have
crawled under a desk just to get the username and
password from a Wi-Fi router, you can point your phone at it. [APPLAUSE] And we can automatically
do the hard work for you. Or if you're walking
in a street downtown and you see a set of
restaurants across you, you can point your phone.

Because we know where you are
and we have our Knowledge Graph and we know what
you're looking at, we can give you the
right information in a meaningful way. As you can see, we're
beginning to understand images and videos. All of Google was built because
we started understanding text and web pages. So the fact that computers can
understand images and videos has profound implications
for our core mission. When we started
working on Search, we wanted to do it at scale. This is why we rethought our
computational architecture. We designed our data
centers from the ground up. And we put a lot
of effort in them. Now that we are evolving for
this machine learning and AI world, we are rethinking our
computational architecture again. We are building what we think
of as AI first data centers. This is why last year,
we launched the tensor processing units. They are custom hardware
for machine learning. They were about 15 to 30 times
faster and 30 to 80 times more power efficient than CPUs
and GPUs at that time. We use TPUs across
all our products, every time you do a search,
every time you speak to Google.

In fact, TPUs are what powered
AlphaGo in its historic match against Lee Sedol. I see now machine learning
as two components. Training, that is, how
we build the neural net. Training is very
computationally intensive, and inference is what
we do at real time, so that when you
show it a picture, we'd recognize whether it's
a dog or a cat, and so on.

Last year's TPU were
optimized for inference. Training is computationally
very intensive. To give you a sense, each one of
our machine translation models takes a training of
over three billion words for a week on about 100 GPUs. So we've been working
hard and I'm really excited to announce our next
generation of TPUs, Cloud TPUs, which are optimized for
both training and inference. What you see behind me
is one Cloud TPU board. It has four chips in
it, and each board is capable of 180
trillion floating point operations per second. [WHOOPING] And we've designed it
for our data centers, so you can easily stack them. You can put 64 of these
into one big supercomputer. We call these TPU
pods, and each pod is capable of 11.5 petaflops. It is an important advance
in technical infrastructure for the AI era. The reason we named
it cloud TPU is because we're bringing it
through the Google Cloud Platform. So cloud TPUs are
coming to Google Compute Engine as of today. [APPLAUSE] We want Google Cloud to be
the best cloud for machine learning, and so we want
to provide our customers with a wide range
of hardware, be it CPUs, GPUs, including the
great GPUs Nvidia announced last week, and now Cloud TPUs.

So this lays the foundation
for significant progress. So we are focused
on driving the shift and applying AI to
solving problems. At Google, we are bringing
our AI efforts together under Google.ai. It's a collection
of efforts and teams across the company focused on
bringing the benefits of AI to everyone. Google.ai will focus
on three areas, state-of-the-art research,
tools, and infrastructure– like TensorFlow and Cloud TPUs– and applied AI.

So let me talk a little
bit about these areas. Talking about research, we're
excited about designing better machine learning
models, but today it is really time consuming. It's a painstaking effort of a
few engineers and scientists, mainly machine learning PhDs. We want it to be possible
for hundreds of thousands of developers to use
machine learning. So what better way to do
this than getting neural nets to design better neural nets? We call this approach AutoML. It's learning to learn. So the way it works is we take
a set of candidate neural nets. Think of these as
little baby neural nets. And we actually use a neural net
to iterate through them till we arrive at the best neural net.

We use a reinforcement
learning approach. And it's– the
results are promising. To do this is
computationally hard, but Cloud TPUs put it in
the realm of possibility. We are already approaching state
of the art in standard tasks like, say, for our
image recognition. So whenever I spend
time with the team and think about neural nets
building their own neural nets, it reminds me of one of my
favorite movies, "Inception." And I tell them
we must go deeper.

[LAUGHTER] So we are taking all
these AI advances and applying them to
newer, harder problems across a wide range
of disciplines. One such area is health care. Last year, I spoke about our
work on diabetic retinopathy. It's a preventable
cause of blindness. This year, we
published our paper in the "Journal of the
American Medical Association," and [? verily ?] is working
on bringing products to the medical community. Another such area is pathology. Pathology is a
very complex area. If you take an area like
breast cancer diagnosis, even amongst highly
trained pathologists, agreement on some
forms of breast cancer can be as low as 48%.

That's because
each pathologist is reviewing the equivalent of
1,000 10-megapixel images for every case. This is a large data problem,
but one which machine learning is uniquely equipped to solve. So we built neural nets
to detect cancer spreading to adjacent lymph nodes. It's early days,
but our neural nets show a much higher
degree of accuracy, 89% compared to previous
methods of 73%. There are important caveats we
do have higher false positives, but already giving this in
the hands of pathologists, they can improve diagnosis. In general, I think this is
a great approach for machine learning, providing
tools for people to do what they do better. And we're applying it
across even basic sciences. Take biology. We are training
neural nets to improve the accuracy of DNA sequencing.

[? Deep ?] [? Piriant ?] is a
new tool from Google.ai that identifies genetic variants
more accurately than state-of-the-art methods. Reducing errors is in
important in applications. We can more accurately
identify whether or not a patient has genetic disease
and can help with better diagnosis and treatment. We're applying it to chemistry. We're using machine
learning to predict the properties of molecules. Today, it takes an incredible
amount of computing resources to hunt for new
molecules, and we think we can
[? accelerate ?] timelines by orders of magnitude.

This opens up possibilities
in drug discovery or material sciences. I'm entirely
confident one day, AI will invent new molecules that
behave in predefined ways. Not everything we are
doing is so profound. We are doing even
simple and fun things, like a simple tool which
can help people draw. We call this AutoDraw. Just like today when
you type in Google, we give you suggestions,
we can do the same when you're trying to draw,
even I can draw with this thing.

So it may look
like fun and games, but pushing computers
to do things like this is what helps them
be creative and actually gain knowledge. So we are very excited about
progress even in these areas as well. So we are making
impressive progress in applying machine learning,
and we are applying it across all our products, but
the most important product we are using this is for Google
Search and Google Assistant. We are evolving
Google Search to be more assistive for our users. This is why last
year at Google I/O, we spoke about the Assistant,
and since then, we've launched it on Google
Pixel and Google Home. Scott and team are going
to talk more about it, but before that, let's take a
look at the many amazing ways people have been using
the Google Assistant. [VIDEO PLAYBACK] – OK, Google. [MUSIC PLAYING] – Hey, Google? – Hey, Google. – OK, Google. – Hey, Google. [BLING] – Play some dance music. – Sure. [BLING] – This is "Fresh Air." My guest will be– – Kimmy Schmidt on Netflix. [BLING] – OK, Google. Count to 100.

– Sure. 1, 2, 3– – Play vacuum
harmonica on my TV. [VACUUMING] [HARMONICA PLAYS] – –71, 72– – No! – –73– – Play the "Wonder
Woman" trailer. – Hey, Google. Talk to Domino's. – Talk to Lonely Planet. – Talk to Quora. – Show me my photos
from last weekend. [BLING] [SCREAMING] – Your car is parked at 22B. [BEEP BEEP] – Today in the news– [BLING] – Turn the living
room lights on. – OK, turning on the lights. – I'm back, baby. – Hey, Google. Drop a beat. – Flip a coin. – Call Jill. – Set a timer. – Talk to Headspace. [TING] – And then just
for a moment, I'd like you to let go
of any focus at all. Just let your mind do
whatever it wants to do. – Done. – Hey, Google. Good night. – Turning off all the things. See you tomorrow. [END PLAYBACK] [MUSIC PLAYING] [APPLAUSE] SCOTT HUFFMAN: Hey, everyone. Last year at I/O, we introduced
the Google Assistant, a way for you to have a
conversation with Google to get things done
in your world.

Today, as Sundar
mentioned, we're well on our way,
with the Assistant available on over
100 million devices. But just as Google
Search simplified the web and made it more
useful for everyone, your Google Assistant
simplifies all the technology in your life. You should be able
to just express what you want
throughout your day and the right things
should happen. That's what the Google
Assistant is all about. It's your own individual Google. So that video we
saw really captures the momentum of this project. We've made such big strides
and there's so much more to talk about today. The Assistant is becoming
even more conversational, always available wherever you
need it, and ready to help get even more things done.

First, we fundamentally believe
that the Google Assistant should be, hands
down, the easiest way to accomplish tasks, and
that's through conversation. It comes so naturally to
humans, and now Google is getting really good
at conversation, too. Almost 70% of requests
to the Assistant are expressed in
natural language, not the typical keywords that
people type in a search box. And many requests or follow-ups
that continue the conversation. We're really starting to crack
the hard computer science challenge of conversationality
by combining our strengths in speech recognition, natural
language understanding, and contextual meaning. Now recently, we made
the Assistant even more conversational, so each
member of the family gets relevant
responses just for them by asking with their own voice. And we're continuing to make
interacting with your Assistant more natural. For example, it doesn't always
feel comfortable to speak out loud to your Assistant,
so today, we're adding the ability to type to
your Assistant on the phone.

Now, this is great when
you're in a public place and you don't want
to be overheard. The Assistant's also learning
conversation beyond just words. With another person,
it's really natural to talk about what
you're looking at. Sundar spoke earlier about
how AI and deep learning have led to tremendous
strides in computer vision. Soon, with the smarts
of Google Lens, your Assistant will be able to
have a conversation about what you see. And this is really cool,
and Ibrahim is here to help me show you a couple
of examples of what we'll launch in the coming months. So, last time I
traveled to Osaka, I came across a line of
people waiting to try something that smelled amazing.

Now, I don't speak Japanese,
so I couldn't read the sign out front, but Google Translate
knows over 100 languages, and my Assistant will help
with visual translation. I just tap the Google Lens
icon, point the camera, and my Assistant can instantly
translate the menu to English. And now, I can continue
the conversation. IBRAHIM ULUKAYA: What
does it look like? GOOGLE ASSISTANT: These
pictures should match. SCOTT HUFFMAN: All right. It looks pretty yummy. Now notice, I never had to
type the name of the dish.

My Assistant used visual
context and answered my question conversationally. Let's look at another example. Some of the most tedious
things I do on my phone stem from what I see– a business card I
want to save, details from a receipt I need
to track, and so on. With Google Lens,
my Assistant will be able to help with
those kinds of tasks, too. I love live music,
and sometimes I see info for shows around
town that look like fun.

Now, I can just tap
the Google Lens icon and point the camera
at the venue's marquee. My Assistant instantly
recognizes what I'm looking at. Now, if I wanted to, I could
tap to hear some of this band's songs, and my Assistant offers
other helpful suggestions right in the viewfinder. There's one to buy
tickets from Ticketmaster, and another to add the
show to my calendar. With just a tap, my Assistant
adds the concert details to my schedule. GOOGLE ASSISTANT: Saving event.

Saved Stone Foxes for
May 17th at 9:00 PM. SCOTT HUFFMAN: Awesome. [APPLAUSE] My Assistant will help me
keep track of the event, so I won't miss the
show, and I didn't have to open a bunch of
apps or type anything. Thanks Ibrahim. So that's how the
Assistant is getting better at conversation– by understanding language and
voices, with new input choices, and with the power
of Google Lens. Second, the
Assistant is becoming a more connected experience
that's available everywhere you need help, from your living
room to your morning jog, from your commute to
errands around town, your Assistant should
know how to use all of your connected
devices for your benefit. Now, we're making good progress
in bringing the Assistant to those 2 billion
phones, and other devices powered by Android, like TVs,
wearables, and car systems. And today, I'm
excited to announce that the Google Assistant is
now available on the iPhone. [APPLAUSE] Woo. So no matter what
smartphone you use, you can now get help from
the same smart assistant throughout the day at
home, and on the go. The Assistant brings together
all your favorite Google features on the iPhone.

Just ask to get package
delivery details from Gmail, watch videos from your
favorite YouTube creators, get answers from Google
Search, and much more. You can even turn on the
lights and heat up the house before you get home. Now, Android devices and iPhones
are just part of the story. We think the Assistant should
be available on all kinds of devices where people
might want to ask for help. The new Google Assistant SDK
allows any device manufacturer to easily build the Google
Assistant into whatever they're building. Speakers, toys,
drink-mixing robots, whatever crazy device
all of you think up, now can incorporate
the Google Assistant. And we're working with many
of the world's best consumer brands and their
suppliers, so keep an eye out for the badge that says,
"Google Assistant built-in" when you do your holiday
shopping this year. Now obviously, another aspect
of being useful to people everywhere is support
for many languages. I'm excited to announce
that starting this summer, the Google Assistant
will begin rolling out in French, German,
Brazilian Portuguese, and Japanese on both
Android phones and iPhones.

By the end of the
year, we'll also support Italian,
Spanish and Korean. So that's how the Assistant is
becoming more conversational, and how it will be available
in even more contexts. Finally, the
Assistant needs to be able to get all kinds of
useful things done for people. People sometimes ask if
the Assistant is just a new way to search. Now of course, you
can ask your Assistant to get all sorts of
answers from Google Search, but beyond finding
information, users are also asking
the Assistant to do all sorts of things for them.

Now as you've already
seen, the Assistant can tap into capabilities across
many Google Apps and services, but Google's features are
just part of the story. We also open the Assistant
to third-party developers who are building some
really useful integrations. I'll turn it over to Valerie
to share more about how the developer platform
is getting stronger. [MUSIC PLAYING] [APPLAUSE] VALERIE NYGAARD: Hi. OK, so with the actions
on Google Platform, it's been awesome to
see how developers like you have been engaging
with the Google Assistant. Like honestly, you've built
some really cool integrations. Like, I can ask Food Network
about the recipe that's on TV right now. I can work out with
Fitstar, ask CNBC about the news, or
my husband and I can play name that tune
with SongPop, which he is surprisingly good at.

Until now, these
experiences have been available through the
Assistant on Google Home. But today, we're
also bringing them to Android phones and iPhones. It's over 100 million
devices on Android alone. So now people can get
to Google features and third-party
services from anywhere, and they can even pick up where
they left off across devices. So, not only are
third-party integrations available in more places. They'll be able to do more. Starting today,
actions on Google will be supporting transactions. It's a complete end-to-end
solution for developers, including payments, identity,
notifications, receipts, even account creation. The platform handles
all the complexity. Let me show you
how one will work. GOOGLE ASSISTANT:
Hi, how can I help? VALERIE NYGAARD: I'd like
delivery from Panera. PANERA: Hi, this is Panera. I'll need your delivery address.

Which one can I get from Google? GOOGLE ASSISTANT: We'll
go with 1600 Amphitheater. PANERA: What can I
get you started with? VALERIE NYGAARD: I'll have the
strawberry poppy seed salad with steak instead of chicken. PANERA: Got it. How about one of
these cool drinks? VALERIE NYGAARD: And here, I can
just swipe through my options. See what looks good. Agave lemonade. PANERA: Great. Are you ready to check out? VALERIE NYGAARD: Yep. PANERA: OK, the total is $18.40. Are you ready to
place the order? VALERIE NYGAARD: Yes. I'll just scan my fingerprint to
pay with Google, and that's it. [APPLAUSE] PANERA: Thanks. You're all set. VALERIE NYGAARD:
Yeah, super easy, like I was talking to
someone at the store. So here I was a new
Panera customer. I didn't have to install
anything or create an account. You've also probably
noticed I didn't have to enter my address
or my credit card.

I just saved those
earlier with Google, and Panera used
built-in platform calls to request the information. Now, I was in control over what
I shared every step of the way. So– AUDIENCE: Woo! VALERIE NYGAARD: [CHUCKLES]
The developer platform's also getting much stronger for
home automation integrations. Actions on Google can now
support any smart home developer that wants to
add conversational control. Today, over 70 smart
home companies work with the Google Assistant,
so now in my Google Home or from my phone, I can lock my
front door with August locks, control a range
of LG appliances, or check in on my son's room
by putting the Nest cam on TV. All right, now
that we're talking about making your home smarter,
we also have a lot of news to share today about Google
Home, our own smart speaker with the Google
Assistant built in. Here to tell you more
is Rishi Chandra.

[MUSIC PLAYING] [APPLAUSE] RISHI CHANDRA: Thanks, Valerie. You know, it's really
hard to believe we launched Google Home a
little over six months ago, and we've been really
busy ever since. Since launch, we've added
50 new features, including some my favorites like
support for Google Shopping, where I can use my voice
to order items from Costco right to my front door. Or I can get step-by-step
cooking instructions from over 5 million recipes. Or I can even play my favorite
song just by using the lyrics. Now in April, we launched in
the UK to some great reviews. And starting this
summer, we're going to be launching in
Canada, Australia, France, Germany, and Japan. [APPLAUSE] And with support
for multiple users, we can unlock the full
potential of Google Home to offer a truly
personal experience. So now, you can schedule
a meeting, set a reminder, or get your own daily
briefing with My Day by using your own voice.

And get your commute, your
calendar appointments, and your news sources. Now today, I'd like you
share four new features we'll be rolling out
over the coming months. So first, we're
announcing support for proactive assistance
coming to Google Home. Home is great at providing
personally relevant information for you when you
ask for it, but we think it'd be even more
helpful if it can automatically notify you of those timely
and important messages. And we do this by understanding
the context of your daily life, and proactively looking for
that really helpful information, and providing for you
and a hands-free way. So for example, let's say I'm
relaxing and [? playing game ?] with the kids. Well, I can see that the Google
Home lights just turned on. Hey, Google, what's up? GOOGLE ASSISTANT: Hi, Rishi. Traffic's heavy
right now, so you'll need to leave in 14 minutes
to get to Shoreline Athletic Fields by 3:30 PM. RISHI CHANDRA:
That's pretty nice. The Assistant saw the game
coming up on my calendar, and got my attention
because I had to leave earlier than normal.

So now, my daughter can
make it to that soccer game right on time. Now, we're going
to start simple, with really important messages
like reminders, traffic delays, and flight status changes. And with multiple-user
support, you have the ability to control the
type of proactive notifications you want over time. All right, and second,
another really common activity we do in the home today is
communicate with others. And a phone call is still the
easiest way to reach someone. So today, I'm excited to
announce hands-free calling coming to Google Home. [CHEERING AND APPLAUSE] It's really simple to use. Just ask the Google
Assistant to make a call, and we'll connect you. You can call any landline
or mobile number in the US or Canada completely free. And it's all done
in a hands-free way. For example, let's say I forgot
to call my mom on Mother's Day. Well now, I can
call her while I'm scrambling to get the kids
ready for school in the morning. I just see and say, hey Google. Call mom. GOOGLE ASSISTANT:
Sure, calling mom.

[RINGING] [RINGING] SPEAKER 1: So, you're
finally calling. Mother's Day was three days ago. RISHI CHANDRA: Yeah,
sorry about that. They made me rehearse
for I/O on Mother's Day. Speaking of which, you're
on stage right now. Say hi to everyone. SPEAKER 1: Oh, hi, everyone. AUDIENCE: Hi. RISHI CHANDRA: So, hopefully,
this makes up for not calling, right? SPEAKER 1: No, it doesn't. You still need to visit
and bring flowers.

RISHI CHANDRA: OK, I'm on it. Bye. SPEAKER 1: Bye. RISHI CHANDRA: It's that simple. We're just making a standard
phone call through Google Home. So mom didn't need to learn anything new. She just needs to answer her phone. There's no additional setup,
apps, or even phone required. And since the Assistant
recognized my voice, we called my mom.

If my wife had asked,
we would have called her mom. We can personalize calling
just like everything else. And now, anyone home can
call friends, family, even businesses. Maybe even a local florist to
get some flowers for your mom. Now, by default, we're going to
call out with a private number, but you also have the option
to link your mobile number to the Google Assistant. And we'll use that
number whenever we recognize your voice. So whoever you call [? must ?]
know it's coming from you. Now, we're rolling out
hands-free calling in the US to all existing
Google Home devices over the next few months. It's the ultimate
hands-free speakerphone. No setup required, call anyone,
including personal contacts or businesses, and even dial out
with your personal number when we detect your voice. We can't wait for
you to try it out.

OK, third, let's talk a
little about entertainment. We designed Google Home
to be a great speaker, one that can put in any
room in the house or wirelessly connect to other
Chromecast built-in speaker systems. Well today, we're
announcing that Spotify, in addition to their
subscription service, will be adding their free
music service to Google Home, so it's even easier to play
your Spotify playlists. [APPLAUSE] We'll also be adding support
for SoundCloud and Deezer to the largest global
music services today. [APPLAUSE] And these music
services will join many of the others
already available through the Assistant. And finally, we'll be
adding Bluetooth support to all existing
Google Home devices. So you can play any audio from
your iOS or Android device. AUDIENCE: Yes! [APPLAUSE] But Google Home can do
much more than just audio. Last year, we
launched the ability to use your voice to play
YouTube, Netflix, and Google Photos right on your TV.

And today, we're announcing
additional partners, including HBO NOW. [APPLAUSE] So just say you want to watch,
and we'll play it for you all in a hands-free way. With Google Home, we want to
make it really easy to play your favorite entertainment. OK, finally, I want
to talk a little bit how we see the Assistant
evolving to help you in a more visual way. Voice responses are great,
but sometimes a picture is worth a thousand words. So today, we're announcing
support for visual responses with Google Home. Now to do that,
we need a screen. Well, fortunately,
many of us already have a ton of screens in
our home today, our phones, our tablets, even our TVs.

The Google Assistant
should smartly take advantage of all
these different devices to provide you the best
response on the right device. For example, with Google
Home, I can easily get location information. OK, Google. Where is my next event? GOOGLE ASSISTANT:
Your Pokemon GO hike is at Rancho San
Antonio Reserve. RISHI CHANDRA: It's for my kids. GOOGLE ASSISTANT: It's
at 11:00 AM today. RISHI CHANDRA: It's for my kids. Relax. [LAUGHTER] But if I want to
view the directions, the best place to do
it is on my phone. Well soon, you could
just say, OK, Google. Let's go. GOOGLE ASSISTANT: All right,
I'm sending the best route to your phone. RISHI CHANDRA: And will
automatically your phone– and notify your phone,
whether it's Android or iOS, and take you straight
to Google Maps.

So you can glance at directions,
interact with the map, or just start navigation. It's really simple. Now TVs are another
natural place to get help from the
Google Assistant, and we've a great place to start
with over 50 million Chromecast and Chromecast built-in devices. So today, we're
announcing that we'll be updating Chromecast to show
visual responses on your TV when you ask for help
from Google Home. For example, I can
now say, OK, Google. Show my calendar for Saturday. GOOGLE ASSISTANT:
Showing it on your TV. RISHI CHANDRA: It'll show
up right on TV screen. I'll immediately get
results from the Assistant. [APPLAUSE] And since the Assistant
detected my voice, we're showing my calendar. Others would see their
calendar by using their voice. We can personalize the
experience, even on the TV. They can continue to
follow-up the conversation. Looks like I have a
biking trip to Santa Cruz.

What's the weather in
Santa Cruz this weekend? GOOGLE ASSISTANT: This
weekend in Santa Cruz, it will be clear and
sunny most of the time. RISHI CHANDRA: So
it's really easy. It's all hands-free. Your Assistant can provide
a visual response to a TV to a lot of different
types of questions. We talked about how
easy it is to play what you want to watch
on the TV screen, but what about those times
you don't know what to watch? Well, soon, you could
just ask, hey, Google. What's on YouTube? GOOGLE ASSISTANT: Here you go.

RISHI CHANDRA: And it'll show
me my personalized results right on the TV screen. If I don't like
any of the options, I can continue the
conversation with my voice. Show my Watch Later list. GOOGLE ASSISTANT: All right. RISHI CHANDRA: Play
"Send My Love." GOOGLE ASSISTANT: Playing
"Send My Love" from YouTube. [MUSIC – "SEND MY LOVE"] RISHI CHANDRA:
It's really simple. Again, no remotes
or phone required. In a short conversation, I found
something really interesting to watch using Google Home. I can even do it
with other things. OK, Google. What's on my DVR? GOOGLE ASSISTANT: Here you go. RISHI CHANDRA:
Here we're showing how it works with YouTube
TV, a new live TV streaming service that gives you
live sports and shows from popular TV networks. And YouTube TV
includes a cloud DVR, so I can easily play
my saved episodes. Everything can be done
in a hands-free way all from the
comfort of my couch. And over time, we're going
to bring all those developer actions that Valerie had already
talked about right to the TV screen.

So we'll do even more over
time with Google Home. And that's our update
for Google Home. Proactive assistance will bring
important information to you at the right time, simple
and easy hands-free calling, more entertainment
options, and evolving the Assistant to provide
visual responses in the home. Next up is Anil, who's going
to talk about Google Photos. [APPLAUSE] [MUSIC PLAYING] ANIL SABHARWAL:
Two years ago, we launched Google Photos
with an audacious goal– to be the home for
all of your photos, automatically organized
and brought to life so that you could easily
share and save what matters.

In doing so, we took a
fundamentally different approach. We built a product from the
ground up with AI at its core. And that's enabled
us to do things in ways that only Google can. Like when you're looking for
that one photo you can't find, Google Photos
organizes your library by people, places, and things. Simply type, "Anil
pineapple Hawaii," and instantly find this gem. [LAUGHTER] Or when you come home
from vacation, overwhelmed by the hundreds of
photos you took, Google Photos will
give you an album curated with only the
best shots, removing duplicates and blurry images. This is the secret ingredient
behind Google Photos, and the momentum we've seen
in these two short years is remarkable. As Sundar mentioned, we now
have more than half a billion monthly active users, uploading
more than 1.2 billion photos and videos per day. And today, I'm
excited to show you three new features
we're launching to make it even easier
to send and receive the meaningful
moments in your life.

Now, at first glance, it
might seem like photo sharing is a solved problem. After all, there's no shortage
of apps out there that are great at keeping you
and your friends and family connected, but we
think there's still a big and different problem
that needs to be addressed. Let me show you what I mean. [VIDEO PLAYBACK] – If there's one
thing you know, it's that you're a
great photographer. If there's a second
thing you know, it's that you're kind
of a terrible person. – What? – Yeah, you heard me. The only photo of the
birthday girl in focus? Never sent it. The best picture of
the entire wedding? Kept it to yourself. This masterpiece of
your best friend? We were going to
send it, but then you were like, oh,
remember that sandwich? I love that sandwich.

If only something could say,
hey, Eric looks great in these. You want to send them to him? And you can be like, great idea. Well, it can. Wait, it can? Yup. With Google Photos. [END PLAYBACK] [APPLAUSE] ANIL SABHARWAL:
So today, to make us all a little less
terrible people, we're announcing Suggested
Sharing, because we've all been there, right? Like when you're
taking that group photo and you insist that it be
taken with your camera, because you know if
it's not on your camera, you are never seeing
that photo ever again. [LAUGHTER] Now thanks to the machine
learning in Google Photos, we'll not only remind you so
you don't forget to share, we'll even suggest
the photos and people you should share with. In one tap, you're done. Let's have a look at
Suggested Sharing in action. I'm once again joined onstage
by my friend, and Google Photos product lead, David Leib. [APPLAUSE] All right, so here
are a bunch of photos Dave took while bowling
with the team last weekend.

He was too busy
enjoying the moment, so he never got around
to sharing them. But this time, Google
Photos sent him a reminder via
notification, and also by badging the new Sharing tab. The Sharing tab is
where you're going be able to find all of
your Google Photos sharing activity, and at the top,
your personal suggestions based on your sharing habits and
what's most important to you. Here is the Sharing
Suggestion that Dave got from his day bowling. Google Photos recognized
this was a meaningful moment, it selected the right
shots, and it figured out who he should send it to based
on who was in the photos. In this case, it's Janvi,
Jason, and a few others who were also at the event. Dave can now review
the photos selected, as well as update
the recipients.

Or if he's happy with
it, he can just tap Send. And that's it. Google Photos will even
send an SMS or an email to anyone who
doesn't have the app. And that way, everyone can view
and save the full resolution photos, even if they don't
have Google Photos accounts. And because Google
photo sharing works on any device,
including iOS, let's have a look at what
Janvi sees on her iPhone. She receives a notification,
and tapping on it lets her quickly jump
right into the album. And look at all the photos
that Dave has shared with her. But notice here at
the bottom, she's asked to contribute the photos
she took from the event, with Google Photos automatically
identifying and suggesting the right ones.

Janvi can review the suggestions
and then simply tap Add. Now all of these photos
are finally pulled together in one place, and Dave gets
some photos he's actually in. [LAUGHTER] Which is great, because a
home for all your photos really should include
photos of you. Now, though Suggested Sharing
takes the work out of sharing, sometimes there's a
special person in your life whom you share just
about everything with. Your partner, your best
friend, your sibling. Wouldn't it be great if
Google Photos automatically shared photos with that person? For example, I would love it
if every photo I ever took of my kids was automatically
shared with my wife. And that's why today, we're also
announcing Shared Libraries. [APPLAUSE] Let me show you how it works. So here, we're now looking
at my Google Photos account. >From the menu, I
now have the option to go ahead and
share my library, which I'm going to go ahead
and do with my wife, Jess.

Importantly, I have complete
control over which photos I automatically share. I can share them all,
or I can share a subset, like only photos of
the kids, or only photos from a
certain date forward, like when we first met. In this case, I'm going
to go ahead and share all. [LAUGHTER] [LAUGHS] We did not meet today. [LAUGHTER] And that's all there is to it. I've now gone ahead and shared
my library with my wife, Jess. So, let's switch to her phone
to see what the experience looks like from her end. She receives a notification,
and after accepting, she can now go to see
all the photos that I've shared with her, which she
can access really easily from the menu. If she see something
she likes, she can go ahead and
select those photos and simply save
them to her library.

We'll even notify
her periodically as I take new photos. Now, this is great,
but what if Jess doesn't want to have to keep
coming back to this view and checking if I shared
new photos with her? She just wants every photo
I take of her or the kids to automatically be
saved to her library, just as if she took
the photos herself. With Shared Libraries,
she can do just that, choosing to autosave
photos of specific people. Now, any time I
take photos of her or the kids, without either
of us having to do anything, they'll automatically appear
in the main view of her app. Let me show you. Now, I couldn't justify
pulling the kids out of school today just to have
their photo taken, but I do have the
next best thing.

[APPLAUSE] Let me introduce you to
[? Eva ?] and [? Lilly. ?] All righty here. So I'm going to go ahead,
take a photo with the girls. Smile, kids! [LAUGHTER] Wow, fantastic. And since this is too
good of an opportunity, I'm going to have to
take one with all of you here, too, all right? [CHEERING] Here we go. Woo! Brilliant. All right. OK, so thank you, girls. Much appreciated. Back to school we go. [LAUGHTER] All right. So, using nothing more
than the standard camera app on my phone, I've
gone ahead and taken one photo with my kids and
one photo with all of you here in the audience. Google Photos is going to
back these two photos up. It's going to share
them with Jess, and then it's going to
recognize the photo that has my kids in them
and automatically save just that one to her library,
like you can see right here. [APPLAUSE] Now finally, Jess and I can
stop worrying about whose phone we're using to take the photos. All the photos of our family
are in my Google Photos app, and they automatically
appear in hers too.

And best of all,
these family photos are part of both of
our search results, and they're included in
the great collages, movies, and other fun creations that
Google Photos makes for us. But notice how only the
photos with the kids showed up in Jess's main view. But because I shared my
entire library with her, I can simply go to the
menu, and Jess can now see all of the photos, including
the one with all of you. [APPLAUSE] And that's how easy sharing
can be in Google Photos. Spend less time worrying
about sharing your memories, and more time actually
enjoying them. Suggested Sharing
and Shared Libraries will be rolling out on
Android, iOS, and web in the coming weeks.

Finally, we know
sharing doesn't always happen through apps and screens. There's still something
pretty special about looking at and even gathering around
an actual printed photo. But printing photos and
albums today is hard. You have to hunt across
devices and accounts to find the right
photos, select the best among the duplicates
and blurry images, upload them to a
printing service, and then arrange them
across dozens of pages. It can take hours of sitting
in front of a computer just to do one thing. Thankfully, our machine
learning and Google Photos already does most of
this work for you, and today, we're
bringing it all together with the launch of Photo Books. [APPLAUSE] They're beautiful, high quality
with a clean and modern design, but the best part
is that they're incredibly easy to make,
even on your phone. What used to take hours
now only takes minutes. I recently made a book
for Jess on Mother's Day. And let me show you just
how easy and fast that was. First, thanks to
unlimited storage, all my life's moments are
already here in Google Photos. No need to upload them to
another website or app.

So I'll select a
bunch of photos here. And the good news is I
don't have to figure out which are the right photos
and which are the good ones because this is where
Google Photos really shines. I'm just going to go
ahead and hit plus. Select Photo book. I'm going to pick
a hardcover book. We offer both a softcover
and a hardcover. And notice what happens. Google Photos is going to
pick the best photos for me automatically, automatically
suggesting photo– 40, in this case. [APPLAUSE] How awesome is that? And it's even going to go ahead
and lay them all out for me. All that's left for me to do
is make a couple of tweaks, check out, and in
a few days, I'll end up with one of these
beautiful printed photo books. [APPLAUSE] And soon, we'll make it
even easier to get started, applying machine learning
to create personalized photo books you'll love.

So when you go to Photo
Books from the menu, you'll see pre-made books
tailored just for you. Your trip to the
Grand Canyon, time with your family during
the holidays, or your pet, or even your kids artwork,
all easily customizable. We'll even notify you when
there are new Photo Books suggestions. AUDIENCE: [INAUDIBLE] ANIL SABHARWAL: Photo Books
are available today in the US on photos.google.com,
and they'll be rolling out on Android
and iOS next week, and will be expanding
to more countries soon.

[APPLAUSE] I am really excited about this
launch, and I want all of you to be the first to try it out. And that's why
everyone here at I/O will be receiving a free
hardcover photo book. [APPLAUSE] It's a great example of
machine learning at work. AUDIENCE: [? $10? ?] Take
that photo [INAUDIBLE] ANIL SABHARWAL: So those are
the three big updates related to sharing in Google Photos. Suggested Sharing, Shared
Libraries, and Photo Books. Three new features built
from the ground up with AI at their core. I can't wait for all of you
to try them out real soon. Now before I go, I want to
touch on what Sundar mentioned earlier, which is the way we're
taking photos is changing.

Instead of the occasional
photo with friends and family, we now take 30 identical
photos of a sunset. We're also taking different
types of photos, not just photos to capture
personal memory, but as a way to
get things done– whiteboards we want to remember,
receipts we need to file, books we'd like to read. And that's where Google Lens
and its vision-based computing capabilities comes in. It can understand
what's in an image and help you get things done. Scott showed how Google
Lens and the Assistant can identify what you're looking
at and help you on the fly. But what about after
you've taken the photo? There are lots of photos
you want to keep, and then look back on later to
learn more and take action.

And for that, we're
bringing Google Lens right into Google Photos. Let me show you. So let's say you took
a trip to Chicago. There's some beautiful
architecture there. And during your boat tour
down the Chicago River, you took lots of
photos, but it's hard to remember which
building is which later on. Now, by activating
Lens, you can identify some of the cool
buildings in your photos, like the second
tallest skyscraper in the US, Willis Tower. You can even pull up
directions and get the hours for the viewing deck. And later, while visiting
the Art Institute, you might take photos of a
few paintings you really love. In one tap, you can learn
more about the painting and the artist. And the screenshot that
your friend sent you of that bike rental place? Just activate Lens, and you
can tap the phone number and make the call
right from the photo.

[APPLAUSE] Lens will be rolling out in
Google Photos later this year, and we'll be continually
improving the experience so it recognizes
even more objects and lets you do
even more with them. And those are the updates
for Google Photos. [CHEERING AND APPLAUSE] Now, let's see what's
next from YouTube. [MUSIC PLAYING] SUSAN WOJCICKI: All right. Good morning, everyone. I am thrilled to be
here at my first ever I/O on behalf of YouTube. [APPLAUSE] Thank you. So that opening video
that we all just saw, that's a perfect glimpse into
what makes YouTube so special– the incredible
diversity of content. A billion people
around the globe come to YouTube every
month to watch videos from new and unique voices. And we're hard at
work to make sure that we can reach
the next billion viewers, which you'll hear about
in a later I/O session today.

We want to give
everyone the opportunity to watch the content on YouTube. So, YouTube is different
from traditional media in a number of ways. First of all, YouTube is open. Anyone in the world can upload
a video that everyone can watch. You can be a vlogger
broadcasting from your bedroom, a gamer live streaming
from your console, or a citizen
journalist documenting events live from your
phone on the front lines. And what we've seen
is that openness leads to important
conversations that help shape society,
from advancing LGBTQ rights to highlighting
the plight of refugees, to encouraging body positivity. And we've seen in our
numbers that users really want to engage with this
type of diverse content.

We are proud that last year we
passed a billion hours a day being watched on YouTube,
and our viewership is not slowing down. The second way that
YouTube is different from traditional media is that
it's not a one-way broadcast. It's a two way conversation. Viewers interact directly
with their favorite creators via comments, mobile live
streaming, fan polls, animated GIFs, and VR. And these features enable
viewers to come together, and to build communities
around their favorite content. So one of my favorite stories
of a YouTube community is the e-NABLE network. A few years ago, an
engineering professor named Jon Schull saw a YouTube
video about a carpenter who had lost two of his fingers. The carpenter worked
with a colleague for over a year to build
an affordable 3D-printed prosthesis that would enable
him to go back to work.

They then applied
this technology for a young boy who was
born without any fingers. So inspired by this
video, the professor posted a single
comment on the video asking for volunteers
with 3D printers to help print
affordable prostheses. The network has since grown
into a community of over 6,000 people who have
designed, printed, and distributed these
prosthetics to children in over 50 countries. [APPLAUSE] So today, thousands
of children have regained the ability
to walk, touch, play, and all because
of the one video– one comment– and that
incredible YouTube community that formed to help. And that's just one example of
the many passionate communities that are coming together
on YouTube around video. So, the third feature
of this new medium is that video works
on-demand on any screen. Over 60% of our watchtime now
comes from mobile devices. But actually our
fastest growing screen isn't the one in your pocket. It's the one in
your living room. Our watchtime in our living room
is growing at over 90% a year. So, let's now welcome Sarah Ali,
Head of Living Room Products, to the stage to talk about the
latest features in the living room.

[MUSIC PLAYING] [APPLAUSE] SARAH ALI: Thank you, Susan. So earlier today,
you heard from Rishi about how people
are watching YouTube on the TV via the Assistant. But another way
people are enjoying video is through the
YouTube app, which is available over half a billion
smart TVs, game consoles, and streaming devices. And that number continues
to grow around the world. So, when I think
about why YouTube is so compelling
in the living room, it isn't just about
the size of the screen. It's about giving
you an experience that TV just can't match. First, YouTube offers
you the largest library of on-demand content. Second, our recommendations
build channels and lineups based on your
personal interests, and what you enjoy watching. And third, it's a two-way
interactive experience with features like
voice control.

And today, I'm super
excited to announce that we're taking the
interactive experience a step further by introducing
360 video in the YouTube app on the big screen. And you know that
you can already watch 360 videos on your phone
or in your Daydream headset. But soon, you'll be
able to feel like you're in the middle of the action,
right from your couch, and on the biggest
screen you own. Now, one of my personal
interests outside of work is to travel. And one place I'd
love to visit is Alaska to check out
the Northern Lights. So, let's do a voice search. Aurora Borealis 360. Great. Let's choose that first video. And now, using my TV remote, I'm
able to pan around this video, checking out this awesome
view from every single angle. Traveling is great,
especially when I don't have to get on a flight,
but 360 is now a brand-new way to attend concerts. I didn't make it to Coachella,
but here I can experience it like I was on stage.

And to enhance the
experience even further, we are also introducing
live 360 in the living room. Soon, you'll be able to
witness moments and events as they unfold in a new,
truly immersive way. So whether you have a Sony
Android TV, or an Xbox One console, soon, you'll
be able to explore 360 videos right from
the comfort of your couch and along with your
friends and family. And now, to help
show you another way we're enabling
interactivity, please join me in welcoming Barbara McDonald,
who's the lead of something we call Super Chat. [MUSIC PLAYING] [APPLAUSE] BARBARA MACDONALD:
Good morning I/O, and to everybody
on the live stream.

As Susan mentioned, what
makes YouTube special is the relationships
that creators are able to foster with their fans. And one of the best ways to
connect with your fans is to bring them live, behind
the scenes of your videos, offering up can't-miss content. In the past year, the
number of creators live streaming on
YouTube has grown by 4x. This growth is
awesome, and we want to do even more to deepen the
connection between creators and their fans
during live streams. That's why earlier this year,
we rolled out a new feature called Super Chat. When a creator is
live streaming, fans can purchase Super
Chats which are highlighted, fun, chat messages. Not only do fans
love the recognition, but creators earn
extra money from it. In the past three
months since launch, we've been amazed by
the different ways creators are using Super Chat. Even April, our favorite
pregnant giraffe, who unfortunately could
not be here with us today, has raised tens of
thousands of dollars for her home, the
Animal Adventure Park.

But, OK. [CLAPPING] OK, we can clap for that. [APPLAUSE] [LAUGHS] But enough talking from me. We are going to do a live
stream right here, right now, to show all of you
how Super Chat works. And to help me, I am
very excited to introduce top YouTube creators with
9 million subscribers and over 1 billion
lifetime channel views. On the grass back
there, The Slow Mo Guys! [CHEERING AND APPLAUSE] GAVIN FREE: Hello, everyone. DANIEL GRUCHY: Wow, hey. Happy to be here. How's it going? BARBARA MACDONALD:
It's great to have you. So let's pull up
their live stream. And just look. Chat is flying. Now, I love The
Slow Mo Guys, and I want to make sure that
they see my message, so I'm going to Super Chat them. Pulled up the stream. And right from within live chat,
I am able to enter my message, select my amount, make
the purchase, and send.

Boom. See how much that
message stands out? And it gets to the top. It's cool, right? DANIEL GRUCHY: Yeah,
thanks, Barbara. It's actually lovely
at the minute. Although, I feel like there's
a high chance of showers. GAVIN FREE: Very local
showers, like, specifically to this stage. DANIEL GRUCHY: Very sudden. Yeah. BARBARA MACDONALD:
Ooh, I wonder. I wonder. Well, because we know developers
are incredibly creative, we wanted to see what you can
do to make Super Chat even more interactive. So we've launched an API for it.

And today, we're taking
it to the next level with a new developer
integration that triggers actions in the real world. This means that when a fan
sends a Super Chat to a creator, things can happen in real life,
such as turning the lights on or off in the creator's
studio, flying a drone around,
or pushing buttons on their toys and gadgets. The Slow Mo Guys are going to
create their next slow motion video using Super Chat's API. We have now rigged things up so
that when I send my next Super Chat, it will
automatically trigger the lights and a big horn
in this amphitheater, OK? And that is going to signal our
friends back there on the lawn to unleash a truckload of water
balloons at The Slow Mo Guys. GAVIN FREE: I'm scared. [CHEERING] DANIEL GRUCHY: Yeah. BARBARA MACDONALD: Yeah. [LAUGHS] DANIEL GRUCHY: That's right. For every dollar, we're going
to take another balloon. So, more money
means more balloons. Although, I did hear
a guy over here go, oh, we're going to
really nail these guys.

All right, that's going to
be at least $4 right there. So, yeah. Each dollar donated goes to
the causes Susan mentioned earlier, the e-NABLE network. BARBARA MACDONALD: OK, so, how
much do you think we can send? I can start at $1 and go
anywhere upwards from there. So, it's for charity. How do we think– $100. How's that sound? AUDIENCE: More. BARBARA MACDONALD: OK,
higher, higher. $200? $200? GAVIN FREE: How about
$500 for 500 balloons? BARBARA MACDONALD: $500? I can do that. I can do that. OK. So I'm going to send my
Super Chat and hit Send. $500. Boom. [HORN BLOWS] DANIEL GRUCHY: Oh! Balloons, oh [INAUDIBLE] god! Agh! BARBARA MACDONALD: [LAUGHS] DANIEL GRUCHY: Ugh. Yep. All right. All right. BARBARA MACDONALD: Keep going. Keep going. DANIEL GRUCHY: Oh! BARBARA MACDONALD: It's 500. DANIEL GRUCHY: It's finished. It's finished. GAVIN FREE: It never ends, ah! DANIEL GRUCHY: Ah! [INAUDIBLE] BARBARA MACDONALD:
That was amazing.

Thank you, everybody,
for your help. So this obviously just
scratches the surface of what is possible using
Super Chat's open APIs. And we are super excited
to see what all of you will do with it next. So Susan, how about
you come back out here, and let's check out the
video we've all made. [VIDEO PLAYBACK] [MUSIC PLAYING] [APPLAUSE] BARBARA MACDONALD: [LAUGHS] AUDIENCE: [? Yeah, guys! ?] BARBARA MACDONALD: Wow.

[APPLAUSE] Thank you, Slow Mo Guys. Thank you, Barbara. I'm really happy to
announce that YouTube is going to match The
Slow Mo Guys' Super Chat earnings from today
100x to make sure that we're supplying
prosthetics to children in need around the world. [APPLAUSE] So that 360 living room demo
and the Super Chat demo– those are just two
examples of how we are working to connect
people around the globe together with video.

Now, I hope that what
you've seen today is that the future of media
is a future of openness and diversity. A future filled with
conversations, and community. And a future that works
across all screens. Together with creators,
viewers, and partners, we are building the
platform of that future. Thank you, I/O, and please– [APPLAUSE] Please welcome
Dave Burke, joining us to talk about Android. [CHEERING AND APPLAUSE] [VIDEO PLAYBACK] [MUSIC – JACKIE WILSON, "HIGHER
AND HIGHER"] [BUZZING] [CHEERING] [SATELLITE BEEPS] – Yay! Woohoo! [FIREWORKS LAUNCHING] – Yay! Woohoo! [FIREWORKS LAUNCHING] [END PLAYBACK] [CHEERING AND APPLAUSE] DAVE BURKE: Hi, everybody. It's great to be here
at Google I/O 2017. As you can see, we
found some new ways to hardware accelerate Android.

This time, with jet packs. But seriously, 2 billion
active devices is incredible. And that's just
smartphones and tablets. We're also seeing new momentum
in areas such as TVs, and cars, and watches, and
laptops, and beyond. So let me take a
moment and give you a quick update on how Android
is doing in those areas. Android Wear 2.0 launched
earlier this year with a new update for
Android and iPhone users. And with you partners like
Emporio Armani, Movado, and New Balance, we now enable
24 of the world's top watch brands. Android Auto. We've seen a 10x user
growth since last year It's supported by more than 300 car
models and the Android Auto mobile app. And just this week,
Audi and Volvo announced that their
next generation nav systems will be powered by
Android for a more seamless, connected car experience. Android TV. We partnered with over 100
cable operators and hardware manufacturers around the world. And we're now seeing 1
million device activations every two months.

And there are more than
3,000 Android TV apps in the Play Store. This year, we're releasing a
brand-new launcher interface, and bringing the Google
Assistant to Android TV. Android Things previewed
late last year, and already there are thousands
of developers in over 60 countries using it to
build connected devices with easy access to the
Google Assistant, TensorFlow, and more. The full launch is
coming later this year. Chromebooks comprise almost 60%
of K-12 laptops sold in the US, and the momentum is
growing globally. And now, with the added
ability to run Android apps, you get to target laptops, too. Now, of course,
platforms are only as good as the apps they run. The Google Play ecosystem
is more vibrant than ever. Android users installed a
staggering 82 billion apps and games in the past year. That's 11 apps for every
person on the planet. All right, so let's come
back to smartphones. And the real reason I'm here
is to talk about Android O. Two months ago, we launched our
very first developer preview. So you could kick the tires
on some of the new APIs.

And of course, it's very
much a work in progress, but you can expect the
release later this summer. Today, we want to walk you
through two themes in O that we're excited about. The first is something
called Fluid Experiences. It's pretty incredible what you
can do on a mobile phone today, and how much we rely on them
as computers in our pockets. But there are still
certain things are tough to do
on a small screen, so we're doing a
couple of features in O that we think will
help with this, which I'll cover
in just a moment. The second theme is
something we call Vitals. And the concept here is to
keep vital system behavior in a healthy state so we can
maximize the user's battery, performance, and reliability. So let's jump
straight in and walk through four new
fluid experiences, with live demos,
done wirelessly.

What could possibly go wrong? [LAUGHTER] All right. These days, we do a lot of
[? wants ?] on our phones, whether it's paying
for groceries while reading a text
message you just received, or looking up guitar chords
while listening to a new song. But conventional
multi-window techniques don't translate well to mobile. They're just too fiddly to
set up when you're on the go. We think Picture-in-Picture
is the answer for many cases. So let's take a look.

My kids recently asked me
to build a lemonade stand. So I opened up YouTube, and I
started researching DIY videos. And I found this one. Now, at the same
time, I want to be able to jot down the
materials I need to build for this lemonade stand. So to multitask, all I do
is press the Home button, and boom, I get
Picture-in-Picture.

You can think of it as a kind
of automatic multi-window. I can move it out of the
way, I can launch Keep, I can add some more materials. So I know I need to get
some wood glue, like so. Then when I'm done, I just
simply swipe it away like that. It's brilliant. Picture-in-Picture lets you
do more with your phone. It works great when
video calling with Duo.

For example, maybe I
need to check my calendar while planning a
barbecue with friends. And there are lots of
other great use cases. For example,
Picture-in-Picture for Maps navigation, or watching
Netflix in the background, and a lot more. And we're also excited
to see what you come up with for this feature. We're also making
notification interactions more fluid for users. >From the beginning,
Android has really blazed a trail when it comes
to its advanced notification system. In O, we're extending the
reach of notifications with something we call
Notification Dots. It's a new way
for app developers to indicate that there's
activity in their app, and to drive engagement. So take a look. You'll notice that the Instagram
app icon has a dot in it. And this is it
indicating that there's a notification
associated with the app. So if I pull down the
shade, sure enough, you can see there's
a notification. In this case,
someone's commented on a photo I'm tagged in. What's really cool is I can
long press the app icon, and we now show the
notification in place.

One of the things I really
like about the Notification Dot mechanism is that it works
with zero effort from the app developer. We even extract the color
of the dot from your icon. Oh, and you get your erase
the icon by simply swiping the notification like that. So you're always in control. Another great feature in O that
helps make your experience more fluid is Autofill. Now, if you use
Chrome, you're probably already familiar with Autofill
for quickly filling out a username and
password, or credit card information with a single tap. With O, we've extended
Autofill to apps. Let's say I'm setting up a
new phone for the first time, and I open Twitter.

And I want to log in. Now, because I use twitter.com
all the time on Chrome, the system will automatically
suggest my username. I can simply tap it. I get my password. And then, boom. I'm logged in. It's pretty awesome. [APPLAUSE] Autofill takes the
pain out of setting up a new phone or tablet. Once the user opts
in, Autofill will work for most applications. We also provide
APIs for developers to customize Autofill
for their experience. I want to show you
one more demo of how we're making Android more fluid
by improving copy and paste. The feature is called
Smart Text selection.

So let's take a look. In Android, you typically
long press or double tap a word to select it. For example, I can open Gmail. I can start composing. If I double tap the word "bite,"
it gets selected like so. Now, we know from user
studies that phone numbers are the most copy-and-pasted items. The second most common are
named entities like businesses, and people, and places. In O, we're applying
on-device machine learning– in this case, a [? feed ?]
[? for a ?] neural network– to recognize these more
complicated entities. So watch this. I can double tap anywhere on
the phrase, "Old Coffee House," and all of it is
selected for me. No more fiddling around
with text selection handles. [APPLAUSE] It even works for addresses. So if I double tap on the
address, all of it is selected. And what's more– [APPLAUSE] There is more. What's more is the
machine learning model classifies
this as an address and automatically suggests Maps. So I can get directions
to it with a single click. And of course, it works as
you'd expect for phone numbers. You get the phone
dialer suggested.

And for email addresses,
you get Gmail suggested. All of this neural
networking processing happens on-device in real time,
and without any data leaving the device. It's pretty awesome. Now, on-device
machine learning helps to make your phone smarter. And we want to help
you build experiences like what you just saw. So we're doing two
things to help. First, I'm excited to
announce that we're creating a specialized version
of TensorFlow, Google's open source machine
learning library, which we call TensorFlow Lite. It's a library for apps
designed to be fast and small, yet still enabling
state-of-the-art techniques like [? compnets ?] and LSTMs. Second, we're introducing
a new framework at Android to hardware accelerate
neural computation. TensorFlow Lite will leverage
a new neural network API to tap into silicon-specific
accelerators. And over time, we expect to
see DSPs specifically designed for neural network
inference and training.

We think these new
capabilities will help power our next
generation of on-device speech processing, visual search,
augmented reality, and more. TensorFlow Lite will soon
be part of that open source TensorFlow project, and
the neural network API will be made available later
in an update to O this year. OK, so that's a
quick tour of some of the fluid experiences
in O.

Let's switch gears and talk about Vitals. So to tell you more,
I want to hand it over to Steph, who's been
instrumental in driving this project. Thank you. [MUSIC PLAYING] STEPHANIE SAAD
CUTHBERTSON: Hi, everyone. OK, so all the features
Dave talked about are cool. But we think your phones'
foundations are even more important–
battery life, security, startup time, and stability. After all, if your battery dies
at 4:00 PM, none of the other features that Dave talked
about really matter. So in O, we're investing
in what we call Vitals, keeping your phone secure
and in a healthy state to maximize power
and performance. We've invested in three
foundational building blocks– security enhancements,
OS optimizations, and tools to help
developers build great apps. First, security. Android was built with
security in mind from day one with application sandboxing. As Android has matured, we've
developed vast mobile security services. Now, we use machine learning to
continuously comb apps uploaded to Play, flagging
potentially harmful apps.

Then, we scan over 50
billion apps every day, scanning every installed app
on every connected device. And when we find a
potentially harmful app, we disable it or remove it. And we found most
Android users don't know these services
come built-in with Android devices with Play. So for greater
peace of mind, we're making them more
visible and accessible, and doubling down
on our commitment to security, with the
introduction of Google Play Protect. [APPLAUSE] So here, you can see
Play Protect has recently scanned all your apps. No problems found. That's Google Play Protect.

It's available out of the
box on every Android device with Google Play. Second, OS optimizations. The single biggest visible
change in O is boot time. On Pixel, for example,
you'll find, in most cases, your boot time is
now twice as fast. And we've made all
apps faster by default. We did this through extensive
changes to our runtime. Now, this is really cool stuff,
like concurrent compacting garbage collection
and code locality. But all you really need
to know is that your apps will run faster and smoother. Take Google Sheets–
aggregate performance over a bunch of common actions
is now over two times as fast. And that's all from the OS. There are no changes to the app. But we found apps
could still have a huge impact on performance. Some apps were running
in the background, and they were consuming tons
of system resources, especially draining battery.

So in O, we're
adding Wise Limits to background location
and background execution. These boundaries put
sensible limits on usage. They're protecting battery
life and freeing up memory. Now, our third theme is helping
developers build great apps. And here, I want
to speak directly to all the developers
in the audience. Wouldn't it be cool if Android's
engineering team could show you what causes performance issues? Today, we've launched
Play Console Dashboards that analyze every
app and pinpoint six top issues that
cause battery drain, crashes, and slow UI. For each issue the app
has, we show how many users are affected and provide
guidance on the best way to fix. Now, imagine if developers could
also have a powerful profiler to visualize what's
happening inside the app. In Android Studio, we've also
launched new unified profiling tools for network,
memory, and CPU. So, developers can
now see everything on a unified timeline, and
then dive into each profiler. For example, on CPU, you
can see every thread.

You can look at the call
stack, and the time every call is taking. You can visualize
where the CPU is going. And you can jump to
the exact line of code. OK, so that's Android Vitals. [APPLAUSE] How we're investing
in your phone's foundational security
and performance. Later today, you'll see
Android's developer story from end to end. Our hard work to
help developers build great apps at every stage– writing code, tuning,
launching, and growing.

But there is one more thing. One thing we think would
be an incredible complement to the story. And it is one thing our team
has never done for developers. We have never added a
new programming language to Android. And today, we're making
Kotlin an officially supported language in Android. [APPLAUSE] So, Kotlin– Kotlin is one our
developer community has already asked for. It makes developers so
much more productive. It is fully Android
runtime compatible. It is totally interoperable
with your existing code. It has fabulous IDE support. And it's mature and
production ready from day one. We are also announcing our
plans to partner with JetBrains, creating a foundation
for Kotlin. I am so happy JetBrains CEO,
Max Shafirov, is here today. [APPLAUSE] This new language is
wonderful, but we also thought we should increase
our investment in our existing languages. So we're doing that, too. Please join us at the
developer keynote later today to hear our story
from end to end.

OK, so let's wrap up. There are tons more features
in Android O, which we don't have time to go into today. Everything from
redesign settings, to Project Treble, which
is one of the biggest changes to the
foundations of Android to date, to downloadable fonts
with new emoji, and much more. If you want to try some of
these features for yourself– and you do– I'm happy to announce we're
making the first beta release of O available today. Head over to android.com/beta. [APPLAUSE] But there's more. [LAUGHS] You probably
thought we were done talking about
Android O, but, I'd like you to hear some
more about Android. And from that, please
welcome Sameer. Thank you. [MUSIC PLAYING] [APPLAUSE] SAMEER SAMAT: Thanks, Steph. Hi, everyone. >From the beginning,
Android's mission has been to bring the power
of computing to everyone. And we've seen tremendous
growth over the last few years, from the high end to
entry-level devices, in countries like
Indonesia, Brazil and India. In fact, there are now
more users of Android in India than there
are in the US.

And every minute,
seven Brazilians come online for the first time. Now, all this
progress is amazing. For those of us who
have a smartphone, we intuitively understand
the profound impact that computing is having
on our daily lives. And that's why our team
gets so excited about how we can help bring this
technology to everyone. So we took a step back
to think about what it would take to get
smartphones to more people. There are a few
things that are clear. Devices would need to
be more affordable, with entry-level prices
dropping significantly. This means hardware that uses
less power-packed processors and far less memory
than on premium devices. But the hardware is
only half the equation. The software also
has to be tuned for users' needs around
limited data connectivity and multilingual use. We learned a lot
from our past efforts here with Project
Svelte and KitKat, and the original
Android One program.

But we felt like the time was
right to take our investment to the next level. So today, I'm
excited to give you a sneak peek into a
new experience we're building for entry-level
Android devices. Internally, we
call it Android Go. Android Go focuses
on three things. First, optimizing the
latest release of Android to run smoothly on
entry-level devices, starting with Android
O. Second, a rebuilt set of Google Apps that
use less memory, storage space, and mobile data. And third, a version
of the Play Store that contains the
whole app catalog, but highlights the apps
designed by all of you for the next billion users. And all three of these
things will ship together as a single experience starting
on Android O devices with 1 gigabyte or less of memory.

Let's take a look at
some of the things we're working on for Android Go. First, let's talk about
the operating system. For manufacturers to make more
affordable entry-level devices, the prices of their
components have to come down. Let's take one example. Memory is an
expensive component. So we're making a
number of optimizations to the system UI and the
kernel to allow an Android O device built with
the Go configuration to run smoothly with as
little as 512 megabytes to 1 gigabyte of memory.

Now on-device
performance is critical, but data costs and
intermittent connectivity are also big
challenges for users. One person put it best
to me when she said, mobile data feels like currency. And she wanted more control
over the way she spent it. So on these devices, we're
putting data management front and center in Quick Settings. And we've created an API that
carriers can integrate with, so you can see exactly how much
prepaid data you have left, and even top up right
there on the device. But beyond the OS,
the Google Apps are also getting
smarter about data. For example, on these devices,
the Chrome Data Saver feature will be turned on by
default. Data Saver transcodes content on the
server and simplifies pages when you're on a
slow connection. And, well, now we're
making the savings more visible here in the UI.

In aggregate, this
feature is saving users over 750 terabytes
of data every day. I'm really excited that the
YouTube team has designed a new app called YouTube Go for
their users with limited data connectivity. Feedback on the new YouTube
app has been phenomenal, and we're taking many of the
lessons we've learned here and applying them to
several of our Google Apps. Let me show you some of the
things I love about YouTube Go. First, there's a new
preview experience, so you can get a sneak
peek inside a video before you decide to spend
your data to watch it. And when you're sure this
is the video for you, you can select the
streaming quality you want, and see exactly how much mobile
data that's going to cost you. But my favorite
feature of YouTube Go is the ability to save videos
while you're connected. So you can watch them
later when you might not have access to data. And if you want to share any
of those videos with a friend, you can use the built-in
peer-to-peer sharing feature to connect two of your
devices together directly, and share the files
across without using any of your mobile data at all.

[APPLAUSE] But beyond data
management, the Google Apps will also make it
easier to seamlessly go between multiple
languages, which is a really common use case
for people coming online today. For example, Gboard now
supports over 191 languages, including the recent addition
of 22 Indian languages. And there's even a
transliteration feature, which allows you to
spell words phonetically on a QWERTY keyboard to type
in your native language script. Now, Gboard is super cool,
so I want to show it to you. I grew up in the US, so for any
of my family that's watching, don't get too
excited by the demo. I haven't learned Hindi yet. And I'm sorry, mom, OK? [LAUGHTER] So let's say, I want to send a
quick note to my aunt in India. I can open up Allo,
and using Gboard, I can type how it
sounds phonetically. [HINDI], which means
"how are you" in Hindi. And transliteration
automatically gives me Hindi script. That's pretty cool. Now, let's say I want to ask
her how my I/O speech is going, but I don't know how to
say that in Hindi at all.

I can use the built-in
Google Translate feature to say, "how is this going?" And seamlessly, I
get Hindi script, all built right
into the keyboard. [APPLAUSE] My family is apparently
a tough audience. All right. Well, the Google Apps
are getting Go-ified, what had always
propelled Android forward is the apps from all of you. And no surprise, many of
our developer partners have optimized
their apps already.

So to better connect users
with these experiences, we'll be highlighting
them in the Play Store. One example is right
here on Play's home page. To be eligible for
these new sections, we published a set
of best practices called "Building for Billions,"
which includes recommendations we've seen make a big difference
in the consumer experience. Things such as designing
a useful offline state, reducing your APK size to
less than 10 megabytes, and using GCM or JobScheduler
for better battery and memory performance. And also in "Building
for Billions," you'll find best practices for
optimizing your web experience. We've seen developers
build amazing things with new technologies, such
as progressive web apps. And we hope you can come
to our developer keynote later today to learn
a whole lot more.

OK, that was a quick walkthrough
of some of the things coming in Android Go. Starting with Android
O, all devices with 1 gigabyte of RAM or less
will get the Go configuration. And going forward,
every Android release will have a Go configuration. We'll be unveiling much
more later this year, with the first devices
shipping in 2018. We look forward to
seeing what you'll build, and how we can bring computing
to the next several billion users. Next up– next up, you'll
be hearing from Clay on one of Google's newest platforms
that we're really excited about– VR and AR. Thank you. [APPLAUSE] [MUSIC PLAYING] CLAY BAVOR: Thank you, Sameer. So, Sundar talked about how
technologies like machine learning and
conversational interfaces make computing more intuitive by
enabling our computers to work more like we do. And we see VR and AR
in the same light. They enable us to
experience computing just as we experience
the real world. Virtual reality can
be transporting. And you can experience
not just what it's like to see
someplace, but what it's like to really be there.

And augmented reality uses
your surroundings as context, and puts computing
into the real world. A lot has happened since
Google I/O last year, and I'm excited to share a
bit of what we've been up to. So let's start with VR. Last year, we announced
Daydream, our platform for mobile virtual reality. And then in October, to
kick-start the Daydream ecosystem, we released
Daydream View, a VR headset made by Google. And it's super comfortable. It's really easy to use. And there's tons to do with it. You can play inside
alternate worlds, and games like "Virtual
Virtual Reality." And you can see any
part of our world with apps like Street View.

And you can visit other worlds
with apps like Hello Mars. There's already a great
selection of Daydream phones out there, and we're
working with partners to get Daydream on even more. First, I'm pleased
that LG's next flagship phone, which launches later this
year, will support Daydream. And there's another. I'm excited to announce that
the Samsung Galaxy S8 and S8 Plus will add Daydream support
this summer with a software update. [APPLAUSE] So, Samsung, of
course, they make many of the most popular
phones in the world.

And we're delighted to have
them supporting Daydream. So great momentum in
Daydream's first six months. Let's talk about what's next. So with Daydream,
we showed that you can create high
quality mobile VR experiences with
just a smartphone and a simple headset. And there are a lot of nice
things about smartphone VR. It's easy. There aren't a bunch of cables
and things to fuss with. You can choose from a bunch
of great compatible phones. And of course, it's portable. You can throw your
headset in a bag. We asked, how we take the
best parts of smartphone VR and create a kind of device
with an even better experience? Well, I'm excited to announce
that an entirely new kind of VR device is coming to Daydream–
what we call standalone VR headsets. And we're working with
partners to make them. So what's a standalone headset? Well, the idea is that
you have everything you need for VR built right
into the headset itself. There's no cables, no phone,
and certainly, no big PC. And the whole device is
designed just for VR. And that's cool for
a couple of reasons.

First, it's easy to use. Getting into VR is as easy
as picking the thing up. And it's one step
and two seconds. And second, presence. And by that, I mean really
feeling like you're there. By building every part of the
device specifically for VR, we've been able to optimize
everything– the displays, the optics, the sensors– all to deliver a stronger
sense of being transported. And nothing
heightens the feeling of presence like
precise tracking– how the headset
tracks your movement. And we've dramatically improved
tracking with the technology that we call WorldSense. So WorldSense enables what's
known as positional tracking. And with it, your view
in the virtual world exactly matches your
movement in the real world. And it works by using
a handful of sensors on the device that look
out into your surroundings. And that means it
works anywhere. There's no setup.

There's no cameras to install. And with it, you really
feel like you're there. Now, just as we did with
Daydream-ready smartphones, we're taking a platform approach
with standalone headsets, working with partners to
build some great devices. To start, we worked
with Qualcomm to create a Daydream
standalone headset reference design, a sort of
device blueprint that partners can build from. And we're working closely
with two amazing consumer electronics companies to
build the first headsets. First, HTC, the company
that created the VIVE. [APPLAUSE] We're excited about it, too. [CHEERING AND APPLAUSE] They're a leader
in VR, and we're delighted to be working
with them on a standalone VR headset for Daydream. And second, Lenovo. We've been partners for years,
working together on Tango. And now, we're excited
to work with them on VR.

These devices will start to
come to market later this year. So that's the update on VR. Great momentum with apps,
more Daydream-ready phones on the way, and a new category
of devices that we think people are going to love. So let's turn to
augmented reality. And a lot of us were
introduced to the idea of AR last year with Pokemon GO. And the app gave
us a glimpse of AR, and it showed us
just how cool it can be to have digital
objects show up in our world. Well, we've been working
in this space since 2013 with Tango, a sensing
technology that enables devices to understand
space more like we do. Two years ago in 2015, we
released a developer kit. And last year, we shipped the
first consumer-ready Tango phone. And I'm excited to announce
that the second generation Tango phone, the ASUS ZenFone AR
will go on sale this summer.

Now, looking at the slides,
you may notice a trend. The devices are getting smaller. And you can imagine
far more devices having this capability in the future. It's been awesome to
see what developers have done with the technology. And one thing we've
seen clearly is that AR is most
powerful when it's tightly coupled to the real
world, and the more precisely, the better. That's why we've been
working with the Google Maps team on a service that
can give devices access to very precise location
information indoors. It's kind of like
GPS, but instead of talking to satellites
to figure out where it is, your phone looks for
distinct visual features in the environment, and it
triangulates with those.

So you have GPS. We call this VPS, Google's
Visual Positioning Service. And we think it's going
to be incredibly useful in a whole bunch of places. For example, imagine you're at
Lowe's, the home improvement store that has
basically everything. And if you've been there,
you know it's really big. And we've all had
that moment when you're struggling to find that
one, weird, random screwdriver thing. Well, imagine in the
future, your phone could just take you to
that exact screwdriver and point it out to
you on the shelf. Turns out we can
do this with VPS.

And let me show you how. And this is working today. So here we are walking
down an aisle at Lowe's. And the phone will find
these key visual feature points, which you can
see there in yellow. By comparing the feature points
against previously observed ones, those colorful
dots in the back, the phone can figure out exactly
where it is in space down to within a few centimeters. So GPS can get you to
the door, and then VPS can get you to the exact
item that you're looking for.

Further out– [APPLAUSE] Further out, imagine
what this technology could mean to people with
impaired vision, for example. VPS and an audio-based
interface could transform how they make
their way through the world. And it combines so many things
that Google is good at– mapping, computer vision,
distributed computing. And we think precise
location will be critical for
camera-based interfaces. So VPS will be one of the core
capabilities of Google Lens. We're really excited about
the possibilities here. So the last thing
I wanted to share is something that
we've been working on that brings many
of these capabilities together in a really
important area. And that's education. Two years ago, we
launched Expeditions, which is a tool for teachers
to take their classes on virtual reality field trips.

And 2 million
students have used it. Today, we're excited
to announce that we're adding a new capability
to Expeditions– AR mode, which enables kind
of the ultimate show-and-tell right in the classroom. If we could roll
the video, please. [VIDEO PLAYBACK] – All right, who wants
to see a volcano? 3, 2, 1. – Whoa! – Look at that lava. Look at that smoke
coming out of that. Pretend you're an airplane
and fly over the tornado. – That's the top of it. – What do you see? – It's either a
asteroid, meteorite– – We're learning
about DNA and genes– things that we can't see. And so, the most exciting thing
for me with the AR technology was that I could see
kids get an "aha" moment that I couldn't get by
just telling them about it. – The minute I saw it
pop up on the screen, it made me want to
get up and walk to it. – You actually you get
to turn around and look at things from all angles, so
it gave us a nice perspective.

– See if you can
figure out what that might be based on what you know
about the respiratory system. – I got to see where the
alveoli branched off, and I could look inside them
and see how everything worked, which I never saw before. And it was really, really cool. [END PLAYBACK] CLAY BAVOR: We're just
delighted with the response we're seeing so far. And we'll be rolling this
out later in the year. So, VR and AR, two
different flavors of what you might call immersive
computing– computing that works more like we do. We think that's a big idea. And in time, we see VR
and AR changing how we work and play, live and learn.

And all that I
talked about here, these are just the first steps. But we can see where
all of this goes, and we're incredibly
excited about what's ahead. Thanks so much. Back to Sundar. [APPLAUSE] [VIDEO PLAYBACK] – We wanted to make machine
learning have an open source project so that everyone
outside of Google could use the same system
we're using inside Google. [MUSIC PLAYING] [END PLAYBACK] [APPLAUSE] SUNDAR PICHAI: It's incredible
[? with ?] any open source platform, when you see what
people can do on top of it. We're really excited about the
momentum behind TensorFlow. It's already the most popular
ML repository on GitHub. And we're going to
push it further. We are also announcing the
TensorFlow Research Cloud.

We are giving away
1,000 cloud TPUs, which is 180 petaflops
of computing to academics and researchers for free so that
they can do more stuff with it. I'm always amazed by the stories
I hear from developers when I meet them. I want to highlight
one young developer today, Abu Qader from Chicago. He has used TensorFlow to help
improve health for everyone. Let's take a look. [VIDEO PLAYBACK] [MUSIC PLAYING] [CHATTER] – My name is Abu. I am a high school student. 17 years old. My freshman year, I remember
Googling machine learning. I had no clue what it meant. That's a really cool
thing about the internet, is that someone's already doing
it, so you can just YouTube it, and [CLICK] it's right there. Within a minute, I really saw
what machine learning can do. It kind of like hit
something within me. This need to build
things to help people.

My parents are immigrants
from Afghanistan. It's not easy coming in. The only reason we made it
through some of the times that we did was because people
showed acts of kindness. Seeing that at an early
age was enough for me to understand that
helping people always comes back to you. [INAUDIBLE] – How are you? – And then it kind of hit me– a way where I could actually
generally help people. Mammograms are the cheapest
imaging format there is. It's the most accessible to
people all around the world. But one of the biggest problems
that we see in breast cancer is misdiagnosis. So I decided I
was going to build a system for early detection
of breast cancer tumors, that's successful to everyone,
and that's more accurate. How was I going to do it? Machine learning. The biggest, most extensive
resource that I've used, is this platform
called TensorFlow. And I've spent so
many hours going really deep into these
open source libraries and just figuring
out how it works. Eventually, I wrote
a whole system that can help radiologists
make their decisions. All right. – Ready? – Yeah. I'm by no means a wizard
at machine learning.

I'm completely self-taught. I'm in high school. I YouTubed and just
found my way through it. You don't know about
that kid in Brazil that might have a groundbreaking
idea, or that kid in Somalia. You don't know that
they have these ideas. But if you can open
source your tools, you can give them a
little bit of hope that they can actually conquer
what they're thinking of.

[END PLAYBACK] [CHEERING AND APPLAUSE] Abu started this as
a school project, and he's continued to
build it on his own. We are very, very fortunate
to have Abu and his family here with us today. [CHEERING AND APPLAUSE] Thank you for joining us. Enjoy I/O. We've been talking
about machine learning in terms of how it will power
new experiences and research. But it's also important we think
about how this technology can have an immediate
impact on people's lives by creating opportunities
for economic empowerment. 46% of US employers say
they faced talent shortages and have issues filling open job
positions while job seekers may be looking for openings
right next door.

There is a big disconnect here. Just like we focused
our contributions to teachers and students
through Google for Education, we want to better connect
employers and job seekers through a new initiative,
Google for Jobs. Google for Jobs
is our commitment to use our products to
help people find work. It's a complex,
multifaceted problem, but we've been investing
a lot over the past year, and we have made
significant progress. Last November, we announced
the Cloud Jobs API. Think of it as the first
fully end-to-end, pre-trained, vertical machine learning
model through Google Cloud, which we give to employers– FedEx, Johnson & Johnson,
HealthSouth, CareerBuilder, and we're expanding to
many more employers. So in Johnson &
Johnson's career site, they found that applicants
were 18% more likely to apply to a job suggesting the matching
is working more efficiently. And so far, over 4
and 1/2 million people have interacted with this API. But as we started
working on this, we realized the first
step for many people when they start looking for
a job is searching on Google.

So, it's like other
Search challenges we have worked in the past. So we built a new feature
in Search with a goal that no matter who you
are or what kind of job you are looking for, you can
find the job postings that are right for you. And as part of this
effort, we worked hard to include jobs across
experience and wage levels, including jobs that have
traditionally been much harder to search and classify– think retail jobs,
hospitality jobs, et cetera. To do this, well, we have
worked with many partners– LinkedIn, Monster, Facebook,
CareerBuilder, Glassdoor, and many more. So let's take a look
at how it works. Let's say you come to
Google and you start searching for retail jobs. And you're from Pittsburgh. We understand that. You can scroll down and click
into this immersive experience. And we immediately start showing
the most relevant jobs for you. And you can filter. You can choose Full-time. And as you can see, you
can drill down easily.

I want to look at jobs which are
posted in the past three days. So you can do that. Now, you're looking at retail
jobs in Pittsburgh, posted within the last three days. You can also filter
by job titles. It turns out employees
and employers use many different terminologies. For example, retail could
mean a store clerk, a sales representative, store manager. We use machine
learning to cluster automatically, and so that we
can bring all the relevant jobs for you. As you scroll through it,
you will notice that we even show commute times. It turns out to be an important
criteria for many people. And we'll soon add a
filter for that as well.

And if you find something
that's of interest to you– so maybe the retail
position [? in ?] Ross. And you can click on it, and you
end up going to it right away. And you're one click away. You can scroll to find more
information if you want. And you're one click away from
clicking and applying there. It's a powerful tool. We are addressing jobs of every
skill level and experience level. And we are committed to making
these tools work for everyone. As part of building
it, we literally talked to hundreds of people. So whether you are in a
community college looking for a barista job, a
teacher who is relocating across the country and you
want teaching jobs, or someone who is looking for
work in construction, the product should
do a great job of bringing that
information to you. We are rolling this out in
the US in the coming weeks, and then we are
going to expand it to more countries in the future.

I'm personally enthusiastic
for this initiative because it addresses
an important need and taps our core
capabilities as a company, from searching and
organizing information, to AI and machine learning. It's been a busy morning. We've talked about
this important shift from a mobile first
to a AI first world. And we're driving it forward
across all our products and platforms so that all of you
can build powerful experiences for new users everywhere.

It will take all of
us working together to bring the benefits of
technology to everyone. I believe we are on the verge
of solving some of the most important problems we face. That's our hope. Let's do it together. Thanks for your time today,
and enjoy Google I/O. [APPLAUSE] [MUSIC PLAYING].

As found on YouTube

Build Fast and Smooth Web Apps from Feature Phone to Desktop (Google I/O ’19)

oh right good morning thank you for showing up at 8:30 I am Mariko I'm from web developer ecosystem team within Chrome let's talk about building fast and smooth smooth web apps so it's safe to assume I think that all of us use Internet using computers touch device or your phone every day but what if I tell you that millions more people are accessing the Internet or coming to access the internet using device like this a feature phone so for those of you feel remember a days of feature phone you might remember typing the text on those keys I actually have it here I grew up in Japan where the first popular mobile web network was built and using feature phone to access the website and doing all of the emails and text was like my part of high school and college years so it feels City nostalgic to me but also I went into the industry after that so I remember the days building a website specifically for feature phone in Japanese we call agaric a site oh it's really hard it you needed you to subset of HTML or sometimes completely different markup language yeah so bad memories a little bit but don't worry these phones are new phones that's coming out to the market it's developed right now sold right now all over the world and it's not just me telling you that these phones or popular according to a counterpoint research 303 hundred and seventy million smart feature phones are expected to be sold between this year and 2021 so what is that smart feature phone right this is a new feature phone so smart feature phone means it has a new OS sometimes such as kayo s there is a few other OS to Cairo S is awesome because it's all web-based so you can build an app using web technologies it has a modern web-browser it's a little bit about virgins behind but it supports HTML CSS in JavaScript for those who develop the website for feature phones in Japan it's a great news and then it also comes with app ecosystem which means you can download Google Maps YouTube games on your phone just like you would do with your smartphone so software wise it feels like a smart phone but it comes in the sides of hardware so the screens are usually small I think most of the KY OS devices or QVGA screen size the navigation wise it does have Mouse shows up on the screen but you have to navigate it with your d-pad and if you want to input anything any text on the field you have to use the number key a and the t9 key input to do that so that part is a feature phone so what does it look like to browse the web today using these new feature phone here's a website you might have visited once or twice before Google com no it's great it looks great but we all agree that Google com the top phase it's just a bunch of links in one input field so huh that's not impressive let's see how it's like to build load a web apps right so here's a Skoosh it's a web apps using web assembly you can do image completions it's a PWA we build it last year for c.d.s how does it perform not great the CSS layout is completely off so the thing is when we build a Skoosh our team built it we wanted Skoosh to be the best in class web app so the first load is only 16 kilobyte it lost nearly fast even on this feature phone but we didn't need it quite tested on the screen sides this small we tested it on the desktop tablet and iPhones but we never thought of having somebody accessing the site using these phones news for the one of the depressing layout of the website is at i/o website not great not great it's so frustrating to navigate i/o so using this or oh it's almost unusable but not everything on the web is bad my favorite is Twitter so Twitter mobile I'm evaldo twitter.com when you access that on the smart feature phone you almost get the same exact value you get the same exact same experience you can tweet you can lead to it you can load video you can search for gifs you can like a touch image everything that's provided in the Twitter website available on the smart feature phone that brings me to the project that I'm going to discuss today so I briefly mentioned I am part of web developer ecosystem team in chrome and within that I'm in the small subset of the team that tried to build a real-world web applications that we can share so that we can share our learnings by building something new right so our first project what Skoosh it's a image compression application that completely built within a browser we use what was them to provide a new were file formats like web P for browsers that doesn't support web beat and you can select all of the knobs and like bunch of settings to see how much better the compression can be and you can download it and I'm upload it to your blog so that's the idea and then after we build this last November we got to discussing what should be the next project and we decided let's just build a game we wanted to build a game because whenever we asked web developers what is the way of really good ad everybody says it's a document and we wanted to say completely opposite of document to see if we can handle that so games seem like a good topic and developing a game came with a bunch of problems that we faced as a web developer everyday such as like how do we handle a lot of inputs coming from all of these UI or can we really provide a graphics heavy application on the web and on top of that because we were hearing that the feature phones are getting popular decided that our apt this year it's going to support everything from feature from the desktop and everything in between so I would like to explain what we built first and then we're gonna get into how we build it introducing procs proxy the game of proximity inspired by the legendary game a minesweeper game is situated in the space and your job is to find black hole you can play it works on any kind of devices from desktop to tablet to d-pad even with screen readers one dimmed button column 9 of 16 hidden button column 10 of 16 it is a PWA so you can download it and save it on your desktop or on your phone and play the game wherever you want even when you're offline so that's the game we built and you can access the app at the procs dot app that's the URL so let's discuss how we build procs so even before we go into the project we decided on the baseline like three of us got together Jake Sora and me and talked about like what is this app going to be three points we decided that every device will get a same core experience meaning we are not gonna build a three different apps for desktop tablet and feature phone we decided that it has to be accessible both on the input device so like all of the mouse keyboard touch and d-pad to support it and then we said why not make sure that the screen readers accessible to performance wise our team Nene likes to build a performant web app so we said it has to be really really good performance so we said our performance budget to be initial payload of less than twenty five kilobyte time to interactive less than five seconds on throw 3G network and animation should learn 60 frames per second whenever you can so let's with that understanding three of us got to working so let's let me explain how this app is laid out so the game started with a game logic file the jaicro on the way on the long-haul flight because he wanted to play a minesweeper but there was no internet on the flight and he wrote it he's that kind of engineer game logic just contains the logic there is no UI element to it it just of like how big of the field is where the mine is and with certain points are clicked how should it the deal that kind of thing on top of that we built a new iron state services so you I we use pre-act and State Services is a thin leper on top of game logic to communicate game logic and the UI service we also rotate our own landing pipeline and which we will get into later later so for y and we also have a little bit of utilities to goo it all together so simple issue you can understand all of this could learn in main thread everything in one file well it could be separated but it could learn in main thread however from the get-go we knew we wanted to do graphics heavy design we wanted to animate it and we were like not sure about this so we decided to move a game logic and state service into web worker so web worker those of you who are not familiar is a way to learn your JavaScript off the main thread and separate thread to communicate between worker and main thread you use API called post message and it's like not an enjoyable experience keeping track of the message passing it's it's a lot of work likely team members are my foods sitting there wrote a library called a comlink it's a sin leper or abstraction on top of post message to make using worker a lot more enjoyable in fact we improve the comlink by doing this project calling vfo so we were working on this project using comlink and testing it on the feature phone and realize that it's not really working great and the eyes data comm link has a processing in intensive work happening so we fixed that and comlink v4 was the East so if you're interested in doing our offloading all of the tasks to worker you should definitely check out connect UI wise we use pre-act we chose pre-act because hmm three of us use pre-act on the previous project and we liked it and and also it's still the smallest UI library out there that fits in a performance budget the app wise it is a standard single page application all of the custom elements are blended on one div and then appended into the body but we knew we have an aggressive goal of initial payload less than 25 kilobyte so we decided to do a kind of strategy where we will build time pre-render the first load the first interaction using a little hack using puppeteer so basically we have app and whenever we build our up we learn a puppeteer and puppeteer will download oh by the way puppeteer is a way to control a headrest Chrome from your script so puppeteer will open up the headrest Chrome do the thing store download app react react will just build a HTML or put it into the Dom and then we just grab whatever the output that was in there and put it into index.html and that's what gets uploaded to our static site hosting notify and then that gets served to the user as a first payload this is just showing you how easy it is to start with puppeteer you just started instance browser puppeteer at launch and then create a page go to that link evaluate that link and just put it into the HTML and that's all there to it so the basic architecture whites two key points we use Walker to flee app the main thread as much as possible because we knew going in we want to do gothic heavy thing main thread can only up the graphic stuff main thread only can only do like Walker cannot do the graphic stuff so free that app for graph and then we also plea lender at the build time for speedy loading of the initial bundle so that leads me to talk about graphics perhaps the biggest performance choice we made was to have our own graphic rendering pipeline so uh when you show up Ram was to completely use the Dom we were thinking oh yeah we can just have a table and then put a bunch of buttons in it and you know we can use the CSS animation like transform opacity that think that lands on the GPU to do the animations and that would be great right well it turns out we think that we might hit a Chrome browser bug where when we what when all of this was in the one layer and I want to update a just a single button chrome was the painting entire table which is not a great for performance well way to solve these problem if you see it is to put a buttons or some elements that you want to update in the separate layers using things like world change transform but we have a lot of buttons on the game because each of them ice the game cell it's going to be a layers and that might solve a painting problem but then it creates excessive amount of layers and that hooks up the memory so we are creating another problem so we're like hmm this louder is not good we should go for another loud so we decided to do our all of our graphics in canvas in fact we have two canvas or no screen one is for background animation and one is for each animation that is doing the game cell so these are generated and Leonard every frame 60 frames per second using requestanimationframe so if you're not familiar with the crystal means your frame browser requestanimationframe is a way to schedule your script and every tick the browser the flash is the black graphics so you put some kind of JavaScript in our case the drawing drawing call for the canvas and you put that in callback of the request animation frame and next stick that one at the next stick and if you're doing animation stuff you probably want to recursively call that so that you put that task into every frame and then this is how we update our animation if you were curious about all the stuff that I was talking about painting layers compositors CSS requestanimationframe I wrote a full part blog series about inside look at modern web browser that explains what happens when you down when your code gets the browser and how its executed so you should check that out and so we have to canvas now and there's a few other things that we did for the graphics for performance for example the background animation which we call it nebula animation in fact it's only a 1/5 of screen size so whatever device you have we only create a canvas size 1/5 of your device and just stretch it out to full screen we were lucky because the design the designer came up was already blurry image so clearing a small one and stretching it didn't make much and saved us a lot of memoria do that so the for the grid animation weekly we basically do a sprite animation and we generate these sprites on the client side so we do not send any image data down the wire to users we send the JavaScript to generate the sprite and then JavaScript creates the sprites and once that's done it's saved in index DB this way we can create the most optimal sprites for each of devices so different devices have a different device pixel ratio some of them have won some bottom two sometimes three so this way any device is accessing our site we don't need to create an image they can create an image or two sprites on the client side so that's graphics let's talk about accessibility which is exciting point that I I work so nice I'm really excited so as I explained the game is now into canvas and we can totally build the game just with campus a lot of games do you know whenever use abuse of mouse or clicks on you just get the coordinate of the mouse and then you write your own JavaScript to say hey did it hit the square underneath and then you just lead roll the animation but we decided we are going to keep a Dom that table and buttons remember that was throwing a painting Yeller so we just fixed that by putting in opacity zero there's a reason why we kept the dumb version on top of canvas because if we have an element we can focus on it or we can attach our event listeners so when you are playing the procs what you are seeing on screen is a canvas but but your javascript is interacting with is an invisible buttons and tables and this way we can tap into browsers native accessibility features so here's a screenshot of me playing a game with voiceover it is written down to what the voice whoever said it says hidden button column fifteen or sixteen hidden is a state of the button and that's the only part that we manipulated from our JavaScript by adding a Ilia label everything from button to column sixteen fifteen or sixteen kind of like suggesting users where the locations are that just came out of the box by using the table so when we clear the start a new game we generate a table right clear the table and in theory you should just add a low grid and the screen reader things should be all taken care but somehow maybe because we are displaying the table opacity:0 the browser wasn't quite legislate as a grid so inside when we create all of the lows and columns we also needed to specify that this TR element is a low and this TD element is grid so and that's all this problem but at the beginning we were like why is it not working the documentation said that if we put the grid it should work so that was a fun challenge to do code or foe and inside of each cell we generate a button that user click on it speaking of buttons we use a accessibility technique called tab loving tab index when we clear a button for each cell first top left corner of the cell gets tab index of 0 which means it's tab accessible when keyboard navigation user hits the tab the focus on that button but then everything else we set having the X of minus 1 which means it cannot access it cannot focus by having it this way the keyboard user doesn't have to tab hundred times to get to the end of the game so that they can get to the other menu button down at the bottom of the screen and when the key screen keyboard user access the game they focus on one cell and they switch to using an auto key and a low key will emit like which direction how many times that the focus should move so the cell focus method basically takes like okay current focus is here just gonna make it and on top of all or focus on focus of all and then the new button that will be focused it's going to be focus of all by putting the zero and then just set the focus let's set the focus and this is how we implemented loving tabindex I did not any I did not know any of this until I did this project embarrassingly we got a lot of help from Rob Dodson who is on our team who does a lot of accessibility work and he has a guide on web dev for accessibility to all which explains all of the techniques that I explained right now so you should definitely check that out and thank you very much Rob so this is a half way of the project and we were feeling great our animations were learning future phones game was working fine but then we notice you know I explained that they feature phone to desktop everything in between so we were testing iPad Android go phone which is a low and side of the Android phone and we were noticing that the game is clashing on Android golf on quite a lot we were like how like feature phones or the weakest powered phone enjoy Goa's should be upgrade why is it why is it crashing what turns out how many pixels are on the screen matters a lot so on the feature phones case they may have a weaker chips but they only need to care about a certain number of pixels to drive right on under the girl phone which is a smart phone with touchscreen with much bigger screen they might have a slightly powerful hardware but they have four times or more pixels to drive and that makes them sad and just shuts down some time so we decided that okay at this point we should just check if the hardware can support animation and if not just lender a static version so basically when the game allows we do a little bit of a check saying can this harder we're a support animation and if they can we load a WebGL animations and if not we support a canvas city static graphics studied graphic is something we need to make anyway for accessibility for those who prefers a reduced motion action so we just expose that to a little lower grade hardware phones – so this is just a our our applets – checking this might be a little naive but this is just one check shader box this class is just a class abstract on top of WebGL and we check a high precision vertex shader and if that's possible then we just learned animation and if not we go with a studio animations oh sorry 2d static Len during we we know it is a little naive but we found that the devices that supports high P usually can handle animations of course if user have a preference set for prefers reduce motion we check for the media query and then that becomes a default – let's talk about supporting different with devices because I've been kissing keyboard touchscreen deeper right so game has a two main functions click to open the cell and click to flag the cell when you are playing on the mouse it is just regular click and right-click if you are doing that on the keyboard then we assign a toggle button to the F key for the flag so you click on it and then you can go back and forth with the mode and then navigation wise so you use number key and enter I sorry the arrow key and enter for the phones and tap we went with just tapped and toggle method so you can see at the bottom we have a toggle button and every time a user wants to switch to that mode you just click on that toggle and you might be feeling looking at this play video of a phone saying like I want to pinch zoom in to see if how many cells are left right I want to do means out to see like how our game is doing and this is something we discussed while we were developing this and we actually needed to discuss whether to do it or not because we had this goal of the app has to be performant planning smoothly we had this debate about like yeah but if we support pink zoom then we lose native scroll and that means scroll gets slow and it's that ok so eventually we just decided that we for this app we are going to go with native scroll over pinch zoom so we're not supporting the pinch zoom yet maybe hopefully the thing about pinch zoom is that once you have the pinch zoom action then you lose the native scroll and you need to implement your own scroll physics right and that it's not gonna be comparable to comparable fast to the native scroll and on a web platform we don't have the way to tap into browsers scroll physics we would love to have them and we would love to explore the possibility but for now native scroll only which also gave I also wanted this interaction with a double tap to flag I was like why do I have to toko the mode can I just double tap and if like it and it was immediately shut down because of performance if you implement the tablet app that means we all have to wait for single tap to check it's a double tap is it double tap which means we create a few milliseconds delay for the user the interaction so based on the baseline that we agree that it has to be smooth we said no that what up possibilities or in the future hasn't implemented it long tap holding on the thing we can do that performant Lee this is just a question of a UX and design we don't know how to flatter notify the user that you hold it enough to flag it now you can take it off and you know the black hole is not gonna get revealed so once we figure out the design we might implement it which brings me to this phone that does not have any of it yeah I mean it has a click you can click the button but like there's no right click there's no touch screen there a toggle key for F F doesn't exist it's only number key so what are we – we yeah we added a custom key navigation to the number key so when you click on five the number go the sales focus goes up when you click on hit on zero goes down and then when you hit on eight it is a click action and then when you hit a hash find the the sharp side then that's the mode toggle so another thing we found was that we need to show users where the focus are it's very hard on these small screen to see which button are they about to click they also have a mouse indicator or two but it's sometimes you can't really see like it's this mouse pointing to the one button or the other button so we made sure that we highlight the focus and tell the user this highlighted element will be open once you click on it another thing we added which my favorite is a key shortcut guide so if you access our game on the feature phone then you will see these tiny icon indicating you can click on the hash button to start the game or you can click on asterisk button to open the informations this is piece of UX that I took from a 2000 mobile development in Japan so the Japan had a feature phone web network and when you go to a long document site or something they usually have a table of contents on top and then the in page link inside and those are usually map to number keys so you will have a number emoji right next to the table of contents indicating oh if you want to go to chapter 3 just click on 3 and then you can just move down so I took that UX and then put it into this game another thing that's really important if you're designing a website or with game for feature phone it's to have a way for user to get away from that view so I have a close button there whenever user opens the settings model which is quite a long because it also contains how to play the game and Scrolls down whenever they think oh ok I got it they can just tap up though press on the asterisk key and in just close that model if we didn't do this UX and have the standard design that we have for the smartphone and desktop and all which is a floating X button this happens in the middle of the page user have to scroll up up up up up hit the top and finally the mouse can move to that close button to close it and this is really frustrating so the element itself the floating X button and the close button at the bottom is the same element but depending on the device if its feature form or not I just changed the CSS to put the location differently so that's the feature from designs let's talk about a fine strategy as I talked about this game is a PWA it used serviceworker to cache all of the resources so even if you're fine you can play the game and whenever you do offline there's always question of how do we update the game if there's a new update is there we might have seen these Mauro saying like hey update is available little this app or dismisses and then use the old version this is directly took from the previous project that we did but in this case we did not want to block the users of wanting to play the game or playing the game right now so we hit this logic inside of this page this button to be exact so whenever user comes to the app and hit start and whenever there is a network they make a call to hey is there updated and if there is update then it starts fetching down the new version and once that's done it Lowe's a new version of app skip the opening screen because we already know the how the game settings for it and launch it into game directly so when user sees this page the game is already updated to new versions and this is how we do the offline and versioning lastly I want to talk about resource loading so after all of this WebGL and feature phones and all of that our total packet became a hundred kill by gzipped we feel quite good about this site and out of 100 kilobyte gzipped packet twenty kilobyte is a first payload so we hit the goal of under 25 kilobyte D zipped first payload basically this is just an index.html that gets sense when the first request goes in which means this index.html contains this page I like actually this page so all of the animations and the opening title role that we hunter crafted with CSS animations or lazy loaded so this is the minimal set of features and the buttons that user need to start interacting the first action the first action could be starting the game first action could be opening the information icon clicking the information icon to open the settings or clicking the fullscreen button to go into the full screen mode so we even subset the font so we look at all of the glyphs that's used on this page subset the font and inline it into the index.html so nearly our index.html is the twenty kilobyte of data that kind of look like this yeah all of a lot of inlining but once that gets to the users and users start interacting then little by little chunk so download at least lazy loaded and then game fully interactive for doing this we used all up we really loved and enjoyed using dollop we even wrote our own plugins for things that load up didn't really provide out of the box but we even felt comfortable doing that and kind of mixing and matching it which was not the case on the previous project that we used a different building process and Lola worked really great for our setup using Walker's so as I mentioned our codes are separated into the worker and main thread and commlink is a shared dependency to communicate each other if you do this in web pack then webpack creates a two different chunks of the dependency and you know separated it and that's just duplicating it but load up out of the box just keep it as a one chunk and then share that as a dependency for worker and the main thread so this was out of the box great feeling for a project for module loading because JavaScript modules are not supported in web workers we used AMD and Soma wrote a a tiny pregnant called Lola plugin Lee loader which is a AMD like loader that is Leedy tiny specifically made for all app output so you might want to check that out but that's part of our build process and even doing that tools cannot be the help the fine tuning of shipping down the data we needed to go in and look at our index.html and what gets loaded and then see why is our indexing getting bigger and bigger and bigger so if you wanna check what kind of refactoring we did there's an epic PR up on the github code I need stuff smaller by Jake and things that he did or things he discovered was things like this so our game screen has a element called top bar that has a you know the number of cells that's open the timer counter all of that but those are only for game time right whenever the game is not learning like opening screen or win screen it's only just a title banner but when we are loading the index dot HTML which only need title banner they also load the logic of timers and logic of the open count and everything so we separate that element into top bar the full on version and top bar simple and just load it two differently and those shot some data so this is great if somebody have a time to go in and dig through and every now and then check if we're doing great but we try to keep reminding our self that every P for the quest that we make be conscious about the sides so every poll request we make on our Lippo we run a little script called Travis size report on a Travis CI to just check what changed it is says this file name changed or this file size changed and you know this screenshot isn't you know particularly interesting nothing really changed but sometimes you find an unexpected change like oh why did this file name changed or why did this file were suddenly this big so this was a good reminder so that's the process we took I would like to end with the three learnings that we had I definitely think that having a set baseline for the project was great we started with the story the project with understanding of what's important to us for this project and how we make decisions and that get got us you know showing up to the stand up saying hey I want to implement double top and immediately Jake says no you can do it's not performant and I'm like I wouldn't be offended or not feel like defending because I was just going oh that's right performance was an important thing for this project another thing we think the worker is a crucial for learning a smooth application we need to learn JavaScript off of the main set as much as possible I don't think we could make this game possible on the future phone if we didn't do the worker lastly if you're feeling like I'm not game developer I'm not gonna do the game on the web one thing you can take away from this talk is just study what's on the first instruction for your website or your web applications and just remove everything you don't need that makes your first load data small and an user can get to your service quicker so if you want to check out the app here's the link for the game all of us source code are open sourced on the github so you should check that out future Lee Quest or bug fix yourself very much welcome and if you have any questions or want to play them a game on the big touchscreen all of us will be at the sandbox tent a after this thank you very much [Music]

As found on YouTube