Improving app performance with benchmarking (Google I/O’19)

[MUSIC PLAYING] CHRIS CRAIK: Hey, everybody. We're here to talk to you
about benchmarking today. I'm Chris. DUSTIN LAM: And I'm Dustin. And we're engineers who work
on Android Toolkit stuff. So first, I'd like to
start off with a note. We're not here to talk
about benchmarking devices. I know a lot of people out there
are probably really excited compare phone A to
phone B, but that's not what this talk is about. This talk is about benchmarking
Android app code, specifically, Kotlin and Java-based app code,
and a little bit of NDK code, too.

So what is benchmarking? There's probably a couple
of definitions out there. But for all intents and
purposes of this talk, we're talking about
measuring code performance. Right? We want to know how
much time it takes to run some chunk of code. And we want to
measure this in a way where we can see our
improvements reflected in those measurements. And of course, unlike
profiling, we're not running the full
application in here. We're looking for a tool
for rapid feedback loops. We want to iterate and quickly. So you publish an app. Very exciting, right? We're going to be
rich and famous. And then your users
start writing reviews. It's the first of five
out of five stars. We're doing great,
hearts in the eyes. Was this review helpful? Yes, for my self-esteem. Five out of five again. Awesome. Little hesitation
space, exclamation mark. We're doing great. And then the inevitable
one star review, "laggg". Oh. You're so upset you can't
even spell it right. Well, we've been there too. And we'd like to share
what we've learned.

So let's go on a benchmarking
adventure, very exciting. We'll start off with a test. We'll call it, myFirstBenchmark. And here's an
object that we might be interested in benchmarking. If you haven't seen this before,
this is the Jetpack WorkManager library. And it's basically an
abstraction over async tasks. And here's a synchronous test
API we can use to measure. So what's the first
thing we might do? Well, we could get
the system time, and then get the system time
after we've done the work, subtract that out, output
that as a result, and great, we have our first
benchmark, right? We're done. Well, not quite. We'll find that as we run
this over and over again, we'll get vastly
varying results.

And that's no good
for measurements. But what if we ran it in a loop? And maybe we'll run
this loop five times, and then we'll
report the average. So great, problem solved, right? There's still a few issues here. Well, first, five is kind
of a random magic number. And ideally we'd like
to dynamically compute this number. We want to report
the result when we're ready and confident
that this number won't change too much. We're also measuring
immediately. And the issue here is
that the first couple runs of some piece
of code might not be indicative of what the
real user will experience. And this can be due
to code paging memory or just-in-time
compilization optimizations. And we're also
including outliers. So everything on device
is a shared resource. And this can cause
complications, background
interference that might cause certain runs to
be much, much slower than they should be. And if we include those
results, our measurements will be completely off. And make sure we're not running
an emulator, because while it's very tempting to run
tests on emulator, emulators don't emulate
real world performance.

So the lesson here is that
benchmarking is tricky. It's often inaccurate. We're measuring the wrong
things at the wrong time, right? And it's unstable. There's tons of variance. How many loops should we run? And it's hard to
set up correctly. That's worst of all,
because benchmarking is a tool to help us
focus developer efforts. So it's hard to set up how much
time we're really saving here. So I'd like to introduce the
Jetpack Benchmark library. It's out now, and
it's in alpha one. This is a previously
internal tool I've been working on for years. We've been working hard to get
it available to the public now.

So we hope you
guys will enjoy it. First, I'm going to go over a
little bit about what it looks like and how might how you
might use it in your code, and then we'll talk
about the internals and how we solve a
lot of these issues. So Jetpack Benchmark–
it's a tool for measuring code
performance, of course. Thank you.

We like to prevent common
measuring mistakes. There's a lot of common pitfalls
that we've seen in the past, and we want to pass
those lessons on to you. And best of all, it's
integrated with Android Studio. Who doesn't love Android
Studio integration, right? Let's jump back to
our previous example. Here's our previous benchmark. It's verbose, and
it's full of errors. But this is really
only the code that we care about– the highlighted
code here, about three lines. So let's fix that. First, we apply the benchmark
rule, and then all of this code becomes one simple API
call, measureRepeated. And we can focus on the code
that we really care about. Let's check on
another example here. This is a database benchmark. So first, let's initialize
our database with Room. And if you haven't
seen Room before, it's another Jetpack library. It's just an abstraction
over databases. All that matters
really here is we're initializing some
object that allows us to make database queries.

So first let's
clear all the tables and install some test data,
create our measuring loop, and then we can measure the
code that we care about. In this case, some
kind of complex query, not sure what it is. Doesn't really matter. But there's an issue here. Depending on your
database implementation, your query could
be cached, right? If we know that the query
results won't be changed, shouldn't we just
save that result and use that in
the future instead of having to use complex
query all over again? So what we're
going have to do is we have to take this
clearAndInsertTestData method and do it inside
the loop, right? Every single time,
bust the cache, so we can measure what
we actually care about. But there's another
problem here. We're now measuring more
than we actually care. So what are we going to do? Well, we could do
the previous thing where we take the system
time before and after, maybe save that to some variable,
output the result, then subtract that
from our final result.

And then we'll have
our actual result that we care about, right? Well, this is,
again, very verbose. And fortunately, this
is a common use case that we run into too. So we created an API
for this as well. And that's the
runWithTimingDisabled block. So I know a lot
of Java developers out there are probably
wondering, well, what about us? Well, of course, we
have Java APIs as well. And it's slightly different. So I'd like to go through
this line by line. A lot of code has changed here. So we have to create this
BenchmarkState variable. And this is how the library's
going to commit to your code when it's done.

We create a while loop and
call state.keepRunning. And inside that block,
we can run all the code that we want to measure. If we ever need to do
any setup initialization, we can just call this
pauseTiming method, do our initialization,
and then resume when we're ready to measure again. So that's great. I've got one more
example for you guys, and it's a UI example. A very compelling case for
benchmarking might be UI. And we've designed this library
to integrate on top of existing test infrastructure. And that's because we
want you to be able to use all your favorite tools. For example, the
ActivityTestRule, which is an abstraction
that'll help you with activity lifecycle
and all the setup you need in order
to make a UI test. So we simply mark this
test as an @UIThreadTest to run it on the main thread. And then we get a reference
to a recyclerView. Then, as before, we just
create a measureRepeated loop and measure the code
that we care about.

In this case, we're scrolling
by the height of one item. Let's talk about
studio integration. With Studio 3.5, we're releasing
a new benchmark Module Template to help you get up and
running with your benchmarks. So let's walk through what
adding a benchmark module looks like. So here's a typical
Android project. We've got an app module
and a library module, because we love modulization
and we're great developers. We right click the
project, click New, Module. And we'll get this Create
New Module wizard that you've probably seen before. And if you scroll all
the way to the bottom, you'll see this new
Benchmark Module icon. Click Next, and it gives
a little configuration dialog you can use to
choose your module name, change the package name.

And we've got templates in both
Kotlin and Java-based code. So click Finish, come
back to the project. Now we have our
benchmark module. Let's take a look inside. We've got a benchmark here
that you can just run, up and running. Similar as before,
set BenchmarkRule, measureRepeated loop, and
it works out of the box. You can just run
it, and your results will appear directly
in Studio in case you're developing locally and
you want to iterate quickly.

We've also got JSON
and XML output data. We'll pull off from connected
instrument to tests from device onto your host machine. And we've done this
with the intention of being ingested by
continuous integration tools. So if you want to
look for regressions and look for improvements
you've made over time, this is a great way to do that. As with any module, there's
also a build.gradle file. And there's a couple of things
here I'd like to point out. So first, there's a bench
there's a benchmark plugin that we're going to be shipping
along with the library. There's a custom runner that
extends off the Andrew JUnit runner. And it's also open
to be extended, in case you need to do that.

We've also pre-built it with
some pre-built proguard rules that'll work out of the box with
the Jetpack Benchmark library. And of course, the
library itself– can't forget that,
very important. Let's talk about
the Gradle plugin. So that's this line you saw
before, the apply plugin line. It's going to help you pull
your benchmark reports when you run gradlew,
connect an Android test, or connected check for CI.

And we've also got some gradle
tasks for CPU clock stability in there as well. The test runner, that's
the AndroidBenchmarkRunner. It's kind of an
in-depth topic, so we'll talk more about that later. But suffice to say it's got
a lot of baked in tricks to stabilize your benchmarks. We've also got proguard rules. And that's in this file here. Our template supplies
pre-configured proguard rules. And that's important because
proguard optimizes your code. So you want to make sure
you're running your benchmark in a configuration that
represents real user performance, right? So we want to do this
in a release production environment if possible.

So that's why we bundle
these rules for you. And we realize that
tests generally don't use proguard
or R8, but this is important for benchmarks. And it's probably why we set
it up as a separate module. So we've also included
an Android manifest here that's going to run
with your Android tests. And if you notice here, we've
set debuggable to be false. And while this is normally
enabled by default for tests, it's great because it
allows us to use things like connecting a
debugger and use all of our favorite
debugging tools.

And it's great when you're
testing for correctness. But it's not so
great for benchmarks. We've seen before that runs have
been between 0% to 80% slower. And 80% is not the number
that should be worrying you. It's the hyphen. It's the range, the
variability here. It's hard to account for. Let's take a look at an example. Here are some benchmarks
that we have in AOSP that we use in the Jetpack team. And along the x-axis, you'll
see several different types of benchmarks that we do. And along the y-axis is
normalized duration, how long the benchmark took to run.

Blue is with debuggable false
and red is with debuggable on. And we've normalized
each benchmark, benchmark by benchmark,
not across the board. So you'll see here in the
deserialization example that there's hardly
any difference. If you look really closely,
it's like one pixel off. 1% difference, right? But over here in the
inflateSimple benchmark, there's a huge difference here. And again, the hard part
here is the variability. It's really hard to account for. So we want to make sure that
the optimizations and the code change we're making
are actually going to have an impact on our users. We need to make sure
debuggable is off. So that leaves me at about
benchmark configuration. And here I'd just like to
give some tips that we also bundled with the template. But you should
definitely be setting up your benchmarks this way. So first, as before, we'd like
to turn off debuggability. We also want to make sure
code coverage is false. If you're using
something like JCoCo, this actually modifies
the dex in order to support what it needs to do. And that's great if you're
trying to get code coverage, but when you're running a
benchmark, that's not so great.

Of course, as before, you
like to enable proguard. And we currently support
library modules for alpha one. We're really pushing developers
to modulize their app this year. And I'd like to do
a little shout out. Please check out the How to
Create a Modular Android App Architecture talk. That was on Tuesday,
so you should be able to find it on YouTube. It's by Florina and Yigit. But what if you forget? As a library, we can do a lot,
but we can't do everything. We can't just print
out a device and have you run on that, right? But that would be great. Fortunately, we've got warnings. So we're going to
corrupt your output and make it very visible to you. And hopefully this
is something that you can catch in your continuous
integration tests. So here's an example,
a couple of warnings, debuggable as before.

If you're running
on an emulator, that's also pretty bad. If you're missing the
runner, then you're not using any of our tricks to
help stabilize your benchmarks. And if you're low on battery– now, a surprising thing
about being low in battery is that while you might expect
the device will throttle itself, it'll try
and save power.

However, we've found
that on many devices it still does this even
while it's charging. So this is definitely something
you want to watch out for. So that's a bit about
what the lab looks like and how you would use it. And now Chris is going to
talk about how it works. CHRIS CRAIK: All right, so
that was a lot of information about what it looks
like to use the library, but there's a lot of
implementation behind it to implement all
of these behaviors that Dustin was talking about. So first of all, let's
talk about CPU clock, specifically
frequency and how it's kind of the enemy of stability.

Because when you go up and down
massively, you change results, and you make it very
hard to discover regressions and improvements. So from the perspective
of benchmarking, there are really two
problems that CPU clocks introduce to us. And the first is ramping. Clocks generally start
out low when the device isn't doing anything. And then once work
is scheduled, they will ramp slowly
over time in order to get to a
high-performance mode. On the other side
of this, though, when the device gets
hot because it's been running for a long time,
the clocks will dive quickly, and you'll get
terrible performance.

So both of these are problems. Let's talk about
them one at a time. So first, ramping– clocks will
generally increase under load. And what we've seen is that it
takes about 100 milliseconds. So here we have a clip of
a little systrace here. And the only thing
that's important to note is the clock frequency
at the bottom versus when the work started on the top. At the very beginning
of the trace, you see time equals
0 milliseconds. The frequency is 300
megahertz– incredibly slow. A frequency that most of your
app code is never going to see. However, after about
75 milliseconds, you see that we go all the
way up to 2.56 gigahertz. If we were measuring in
between these two times, that would be bad. However, the solution for this
is actually pretty simple. Benchmark just runs warmup
in order to account for this. So we spin the measurement
loop for 250 milliseconds before we ever start measuring. And then we only start measuring
once timing stabilizes. This also has the
nice side effect of handling the
Android runtime's just-in-time compilation. By the time that
your performance numbers are stabilizing,
JIT has stabilized as well.

You're seeing performance
that's corresponding to what your user would see in a
frequently hot code path. So that was ramping, but diving
is a much bigger problem. When the device gets hot,
the clocks dive quickly. And this is called
thermal throttling. Generally, the CPU will
lower its frequency once it gets to
a very high level because it wants to avoid
overheating and damaging the chip. This can happen
unexpectedly, though, and massively affects
performance while we're running our benchmarks. So take a look at this
sample where I'm just taking a pretty simple benchmark. It's just doing a tiny bit
of matrix math in a loop. And at the beginning, I'm
getting a relatively stable performance. Over the first minute or so,
it looks pretty stable at 2 milliseconds, all is well. But less than a minute in,
performance becomes terrible. Look at that. Less than two minutes in, we
are up to three and a half times the performance that we expect.

However, the device runs at this
low clock for a little while. It cools down. And it's back down. OK. Well, we're good now, right? No. We're still doing the work. So over the result of
this five minute test, we have thermal throttling
taking us up and down and making these results
pretty terrible, right? We can't extract a
whole lot of information from these because we
are dynamically going between 2 and 7 milliseconds. We can't trust
these measurements. So our solution to
throttling, we found, is different per
device because we have different tools
that we can use in different configurations. So the first solution to thermal
throttling is the simplest– lock clocks. What if we could just
set the clock frequency? Well, unfortunately, this is
ideal, but this requires root. This is not a great solution
for the average person. Because although we can set
it to a medium frequency, this will keep it from
thermal throttling because the device handles
medium frequency just fine.

We do provide a gradle
plug-in for you, though, if you do
have a device that is rooted– simply
gradlew lockClocks and your device is locked. But requiring root, though, is
really not a general solution. We don't recommend it. So what else do we have? Well, there is this API added
in Android N, Window.setSustai nedPerformanceMode. And this was originally
designed for VR and for games, but it's incredibly
appealing for benchmarks.

This is because it lowers
the max clocks specifically to solve this problem,
prevent throttling. Perfect, right? And it also works on
GPUs as well as CPUs, so it's even useful
if you're wanting to do rendering benchmarking,
say if you're a game. However, it comes along
with a lot of difficulties. It's designed for VR, not
for headless benchmarks. So first of all, it
requires an activity running with a flag set. It also has two
separate modes if it thinks that you're
single-threaded versus multi-threaded.

And it's also only supported
on some N+ devices. So let's talk through
each of these. Maybe we can solve some of this. So the first is
the activity flag. And in order to
solve this problem, we actually just launch
an activity for you around any headless test. We inject an activity that
launches any time that we might need to use the
setSustainedPerformanceMode so that it's up at all times. We set this flag, also, on any
activities that you launch. So if you're doing a UI test,
like Dustin was showing before with the Recycler view, that
gets this property as well. And it works together with
ActivityTestRule and Activity Scenario, so you don't have
to worry about adapting to this new model.

And in addition, it also calls
it out in the UI of the test while it's running. It pops up there
and says, hey, I'm in setSustainedPerformanceMode. Now, OK, so we've
got a way to solve the problem of needing
the flag, but how do we handle the
two different modes? So first, let's
describe what these are. The setSustainedPerformanceMode
can either operate in a single or
a multi-threaded mode.

Either that means you have your
single threaded application– so we could probably use
one core at max frequency– or you're multi-threaded,
in which case it will set all of the
cores to a lower frequency to prevent throttling. That's fine for
a game, but we're trying to run potentially
different benchmarks with different threading
models, and switching between these modes will
create inconsistency. So what do we do
about this problem? How do we make– we'd really like to just
pick the multi-threaded mode. That's the lower of both.

That sounds good, but
how do we force that? Well, the way that we do this
is in our Android benchmark runner. When sustained performance
mode is in use, we create a new thread. And this thread spins. And this is a really
strange way to do this, but it turns out that this is
actually a pretty efficient way to force us into a
multi-threaded mode. And it gives us that incredibly
sustained performance that we were looking for. We also set a
thread name for it. So if you see this in a trace,
you understand what's going on. And we do the best we can. We set thread priority to
be the lowest possible so that it interferes with
your test minimally.

But here's a look of what
that looks like in systrace. So on the top there, you
can see clock frequency. And over the time of
the benchmark running, when the test starts, the
activity launches– bam, we're suddenly locked
to half clocks. Perfect. We're not going to thermal
throttle in that configuration. Once the test is done,
the activity finishes, and then the clocks are free to
ramp back up slowly over time. So we talked about
how we solve the issue with having an activity
running with a flag set.

We talked about the
two different modes. But we still have
this last issue. This is only on some N+ devices. So you can check whether this is
supported on a specific device by calling PowerManager.isS
ustainedPerforma nceModeSupported. By the way, this is
supported for anything that is VR certified. So you have that. However, in the Firebase
Test Lab, for example, 11 out of the 17 Nougat+ OS
device combos have support for it. So it's not terribly hard
to find something that you can use like this for CI.

But again, this is still
not a general solution. This requires
platform support that isn't available on every phone. So our final solution to
this problem at the very end here is Thread.sleep. The simplest solution–
many devices aren't rooted. Many devices don't have
setSustainedPerformanceMode, so we use Thread.sleep. So what we do here is
we detect a slowdown in between every benchmark
by running a little tiny mini benchmark to see if the device
has started thermal throttling. If it does, we throw away
the current benchmark data, and we sleep to let
the device cool down.

So we saw this
previous slide how performance was oscillating
all over the place and we couldn't get stable
numbers out of this. Well, in this particular
case, our device doesn't have root, so we
can't use lock clocks. It can't use
setSustainedPerformanceMode. Not available, OK. Can't use that. So we have to fall
back on Thread.sleep. So let's see how that actually
performs in this graph. That is a lot better. It is just completely flat. And if you see here, we have
standard deviation for the two different approaches. For the default, we've
got 2.25 milliseconds. And for Thread.sleep,
you've got to look closely, 0.02 milliseconds– massively
more stable, much more consistent performance. But there was a
sacrifice with this.

So because we were
sleeping every time that we detected
thermal throttling, it takes longer to run. This one takes
about eight minutes. The original takes
about four and a half. So in summary, our solution
for thermal strategy does have three
different steps, and we use the best solution
for your device that's available for
your particular phone. So clocks are one
thing, but what about background interference? What about other things
running on your device? So first of all, you have to
ask, am I in the foreground? So if you're not,
something else is. And this is important to
think about in the context of performance. Tests generally run,
if they don't have UI, on top of the launcher,
because there's nothing to launch, nothing to display. However, this means that the OS
thinks the launcher, right now, is the important app. When you're running
in the background, say for instance,
behind the launcher, that means you get
a lot of sources of potential performance
interference.

You might have a live
wallpaper rendering. You might have home
screen widgets updating. You could have
the launcher doing hotword detection or
other miscellaneous work that you don't know about. The status bar is probably
repainting every now and then with a notification coming up. The clock's changing,
Wi-Fi, whatever. And starting at Nougat,
it's possible for devices to have a foreground-exclusive
core, a core that is only available to be used
by the foreground application.

You just can't touch that
if you're in the background. So we want to come
to the foreground. And we actually have a solution
for this already, right? We have our old activity. Remember this guy? So this also solves
this particular problem. And that's why we use this in
all benchmark configurations, regardless of whatever
your clocks are. The benchmark
keeps this activity in the foreground at all times,
unless you have your own, in order to guarantee
that you can use all cores with minimum
rendering interference from whatever is going
on underneath you.

But there's another
important source of background interference. And that is contention. So everything on the
device from CPU to disk is a shared resource. And if your benchmark
is being, let's say, kicked off of a CPU
because of some conflict with another task, if you're
accessing disk and system services at the same time
that someone else is, or if somebody has random
background work happening, for instance, the
system doing something, an app background job,
something along those lines. Those can create contention. So for example, if we
were running our benchmark and warmup just finished,
so we're just right about to start
taking measurements. However, we have this
other process here that's about to start doing some work.

Our first few loops are fine,
but then the other process kicks in and starts
running as well. All of a sudden, we see,
oh, well, actually some of these loops are just giving
me flat out bad numbers. Not very useful numbers that
overlap with this background work. Some of these runs are
still totally fine, but some of them aren't.

So that's why we have this idea
of measure twice, report once. And by twice I, of
course, mean many times. We will measure every– we will measure several loops. And understanding
that most contention is for a short
duration, we can ignore those loops that are most
likely to have hit contention. So what we do, in fact, is
that we report and track the minimum number
observed, not the average, because the average is
susceptible to contention.

This way, the number that we
report by the benchmarking library is immune to
tiny little hiccups that happen every now and
then and might otherwise interfere with your numbers. All right, so let's
talk about how to go about using this library. So the first thing that
we want to recommend is– don't benchmark everything. Start with tracing. Start with profiling tools. Maybe you have measurement
going on on real user devices that tell you some
particular part is slow. Well, that's a good
place to start.

Benchmark what you know is slow
so that you can iterate on it and improve it. We generally
recommend benchmarking synchronous blocks, because
these are the easiest to measure and these are the
easiest to improve over time. So if you're measuring something
that's single-threaded, it is much more likely
to be stable and isolated from other tasks. There's no thread hopping. That means you're taking
the scheduler entirely out of the mix. And that way, for instance,
you might measure UI separately from network separately
from database and rendering.

We also generally recommend
fairly small blocks. These are faster to run. And here we've been probably
less than 50 milliseconds. However, the loop itself
only has around 6 nanoseconds overhead. And this is running on a fairly
old device at half clocks. So you can measure really
small amounts of work with a benchmarking library. Another important
thing to remember is that the benchmarking library
is primarily for hot code. Because we run all of
your code with warmup and we run it in a
loop, that probably means that the code inside
is going to be JIT-ed.

Now, that's great if it's
something like work that's done by your Recycler
view, but if it's only run once in a while
by your app, it's not as likely to get JIT-ed. So be very careful when
you're benchmarking startup. We generally recommend
to only benchmark the code that is inside
of a loop during startup so that is more likely
to be measured correctly. Another thing to
keep in mind is to be aware of caches that might
be anywhere in your code, or even in someone else's code.

So here's a simple
example of a benchmark where we access a file and
check whether it exists. Maybe we observed that this
took a couple of milliseconds during startup. The problem is that the
OS is going to take this and it's going to
know that nothing has changed and just served
you a cached value every time. The benchmark is going
to be very different than the behavior at startup. So be aware of this. You can sometimes take measures
to create things differently, for example, like Dustin
showed at the beginning with the database. Another thing to consider
is potentially avoid overparameterizing. So correctness tests, it's
really easy to, say, sweep over five different
variables, recognize that, OK, over all 4,000 of these
tests, none of them failed. Great. You get a pass. That is a much harder
task for benchmarking, because we're getting something
that isn't a simple pass/fail. More data can be more
difficult to deal with. We recommend to start targeting
benchmarks at real world parameters instead of maybe
going as enthusiastically towards parameterization as
you might during a unit test. One thing I really want
to re-emphasize, though, that Dustin mentioned
before, is, please, do not compare devices.

This library is not designed
to compare the performance of one device versus another. We're really focused
on comparing code, whether that's framework
code, whether that's app code, on the same device,
same operating system version. The library optimizes
for stability. It will have some
factor of difference, that can vary from device
to device, over what you see on a benchmark versus
what you see in reality. And that's because we do not
account for real-world clocks. For example, if the user
touches down on their device, starts scrolling, generally
the clocks will ramp.

If a user uses their
device for a while and it gets hot and thermal
throttles, it performs poorly, the clocks go down. None of that is accounted for
in the benchmarking library. And that's why we don't
recommend comparing devices with our library. So you might be
wondering, OK, how do I go about in
integrating this into my CI? So there are different
tiers of adoption that you can start with to
look at exactly how deeply you want to get into benchmarking.

The first is quick and local. It's very reasonable to do
a trace, write a benchmark, and measure the
performance, make a change, check in that change
without having to monitor the performance over
time and detect regressions. And in general, this is because,
unlike correctness tests, benchmarks usually get better
because you're deleting code. Now, that said, if
you can monitor them over time, potentially manually,
that's a totally reasonable way to go. And that's even better. But we want to recognize
that there's still value in manual
monitoring of benchmarks, even if you don't have
automatic regression detection. Because automatic regression
detection is a complex problem. However, if you do want to go
all the way to that point– and we recommend it– it is something to
recognize that it's not as simple as detecting, when
does my benchmark go down by, say, 1%? Because there's going to be all
sorts of times where it flakes a little bit, it goes
down a little bit, and make a trade for one
benchmark versus another, or you check in a feature
that just absolutely needs to make this one code
path slightly slower.

So if you're prepared to receive
emails for all those things, by all means, just
something to keep in mind. And so let's go through
a few closing notes. We do have, as Dustin
mentioned earlier, an NDK sample that shows how to
use some of our infrastructure together with C++ code. That's available on our
android-performance GitHub repository. And it actually wraps the
existing Google benchmark C++ library to get numbers directly
from infrastructure that you might already be using
for C++ benchmarking. It captures the results together
with Java and Kotlin benchmarks in our final output. And it applies all of our
Android-specific tricks regarding stabilizing clocks. So Dustin mentioned that this is
actually a fairly old library. So why haven't you
heard of it before? Well, this library
has been around since about 2016, used inside
of the Android platform. And we use it all
over the place. We use it for text, for
graphics, for views, for resources, for
SQLite, for optimizing all of these different components. But for a long
time it's been very difficult to use externally
because we kind of needed you to have a rooted
device to run this on.

That's easy for a
platform developer, much harder for
an app developer. But more recently,
we have overhauled it for non-rooted devices. We've switched over to
Kotlin, which gave us some really nice benefits in
terms of function inlining for that measurement
loop and allowed us to minimize overhead. So now libraries such as
Room, Navigation, and Slices all use our benchmarking
library to make improvements. So here's an example
from the platform from a couple of years ago. I think this was in
O when we noticed that, hey, toggling the
visibility of a relatively large tree of views
was kind of expensive.

So this was our CI
over the process of checking in a few changes. So at the beginning,
modifying these views took 350 microseconds. And we checked in
an optimization for how outlines were changed
when views were stored. And so it went
down a little bit. And we actually realized,
OK, most of the work here is actually
in View.invalidate. It's doing a lot of
work that it doesn't need to anymore in a modern
hardware-accelerated pipeline. So we checked in a
complete overhaul of that. And of course, we
quickly reverted it because it broke the world. But we didn't learn
a lot of things that we needed to
change along that path. And after we were able
to check that back in, we were able to get a
total improvement of 100%– of 50% of taking that view
toggling from 0.35 milliseconds per, say, 64 views or
whatever, down to half of that, which is
a huge improvement. And we did this all before
we had automatic regression testing. So in summary here, we've seen
through all of these tricks that we've talked
about, benchmarking is a really complex problem.

It's really hard to
measure accurately. It's completely foiled
by clock instability. And background interference
can be a real source of pain when you're trying
to detect changes. That's why with
Jetpack Benchmarking we've provided a simple API. We bundled in all
the lessons we've learned about getting stable
numbers out of a device. And the alpha is available now.

Thank you so much for coming. If you remember only one
thing, d.android.com/benchmark. Thank you all so much. [MUSIC PLAYING] .

As found on YouTube

Machine learning models + IoT data = a smarter world (Google I/O ’18)

[Music] hi everybody and welcome to this session the last one of the day everybody having a good i/o so far nice excited for the concert tonight yeah so thanks for spending some time with us beforehand I'm Laurence Moroney and Kass and we're here today to talk about machine learning plus IOT equals a smarter world now that's a little bit of a riff on a famous computer science book anybody know what that is algorithms plus data structures equals programs by Niklaus Wirth it's one of the seminal works in comp size so we said hey we want to do a little bit of a riff on that and it's kind of it's also really about what's going on in the world today that there's two major trends going on number one is the rapid rise of IOT and number two is the rapid rise of AI and machine learning and in many ways these are kind of you know that they're intersecting with each other to create some really cool new scenarios so we wanted to talk through a few of these scenarios and give some demonstrations the first one we'll give is like let's take a look at like really cheap IOT sensors that like will do something like in this case we're gonna take a look at those that measure air quality and how they can push data to the cloud and how we can use that for machine learning and we'll also look at how the cloud can be empowering too for IOT ai is products that help you collecting data and analyzing it and learning some intelligence with GD plotting models and then the third one will be as like some of our devices are like practically many computers like raspberry PI's and our handheld devices that what can we do about on device inference with them as well so we'll take a look at some of the examples of that and some pretty cool stuff that's happening there but first of all let's sum numbers according to IHS research they're expected to be 31 billion connected IOT devices by the end of this year that's a lot of devices and those devices combined we found the same research is showing a 27.6% compound annual growth rate of the data that they are pushing to the cloud so 31 billion by the end of this year and every year the amount of data that they're growing is growing pushing it's growing by 27.6% and that's a lot of data now what are we going to do with that data we'll obviously there's the usual things fill sinks filtering sorting querying all that kind of stuff but we can also use that to start training some models and all of that data and all of those models machine learn models can then be used to generate intelligence and to help us to start making intelligent decisions based on the data that's being generated by our devices so the first scenario I wanted to look at was thinking about simple sensors that give us smart solutions I just wanted something with four asses on it easier to remember that way and like an example of this is like this these little devices and if you've ever done any kind of hardware hacking and there's like a device the one on the left here is called an esp8266 and this little chip actually has GPIO pins on it so those import output pins and it also has a Wi-Fi circuit on it and you know when you combine that with something like the one on the right you know that's an air quality sensor and a gas sensor now you can start putting together something very quickly and very cheaply that will push data to the cloud and I'm gonna go off-script for a second and an interesting story Kaz and I had prepared this talk and we were rehearsing it last week with like the VP of our division and one of the feedback he gave us was like hey you don't have any small simple sensors on there you're doing lots of cloud stuff in you doing lots of mobile stuff but you don't have any IOT sensors so you should really build one and put it in there and this was like last Wednesday and the the environment that we live in now is so cool that we were able to thanks to Arduino and thanks to the maker community to be able to build the demo that I'm gonna show shortly you know in just a couple of days and the longest part of that can anybody guess what the longest time in building the demo was no it wasn't actually testing it was waiting for the parts to be shipped from Amazon I'm an Amazon Prime subscriber and it took two days of it it only took three days to build the whole thing so you know that it's it's an amazing time to be alive I think you know the amount of things that you can do the amount of creativity and the kind of things that you can build it's just I find it so cool and I'm really happy to be alive right now so let's take a look at like an example of how you'd use one of these like consider a bill like this one and say in a building like this we can put sensors like this for air quality around the building and when they're around the building then we can start tracking the air quality in that building but you know that's just an instant status we can tell the stuff here on the south side of the building the air quality isn't great you know and then on the north side of the building the air quality is very nice but what would usually happen if you work in an office like this people on the south side are going to go and adjust the thermometer adjust the thermostat the fans will kick in and then people on the north side are gonna be too cold you know it's like what wouldn't be great if we could intelligently adapt the building or what if there was a disaster and it was like maybe a carbon monoxide leak and with our sensors are gonna show that carbon monoxide leak and I don't know fix how well you can see it here but the leak is primarily centered around the exit and if people are being generated being driven towards the exit they might be driven towards danger but when we've got cheap sensors like this we could say hey look instantly we can have an alternative exit but what if we can go even further sorry what if we can go even further and then think about scenarios where machine learning comes involved and we can start predicting things like maybe we can predict the best times of the day to keep the fans on to maximize air quality and reduce our energy footprints it's good for the environment or maybe we could if there was a gas leak or something like that we could predict the path of the gas leak best on based on testing and based on machine learning models so emergency responders would know where it's safe to go and where it's dangerous to go and you know we all of our emergency responders and we want to make sure that they're safe or maybe we could look at what the impact by tradition by modeling what would be of like rearranging our office what if we have more people sitting in the north and less in the south or what happens if we host a big event and you know this this building is actually there one of the offices in Google Seattle and open that top top left hand side there that room called center of the universe is actually where we host a lot of events and it'd be good for us to understand the air quality when we do so so if you've got a lot of devices like this there's a product called cloud IOT core and the idea behind cloud IOT core is it's fully managed and allows you to easily secure connect manage and ingest data from all of your devices and then once you have that then you can start doing interesting things like pulling the data out using bigquery using cloud ml using cloud data lab using cloud studio to be able to then build models and then run inference on these models and have analytics behind that but let me give a little demo of environmental monitoring and what it would look like so if we can switch to the wolf vision so this is the device I was talking about so this is a simple Arduino device I'm not using the Wi-Fi in this case I'm just using a wired one so I have a field on top of the Arduino with a with a wired Internet just so we can make sure that the demo would work here and then this little device is the air quality sensor and so it's like it's right now it's measuring the air quality in the room and so that's measuring for particulates it's measuring for gases carbon dioxide carbon monoxide natural gas all that kind of thing so we kind of want to trigger it by kind of polluting the air a little bit any any ideas how we could do that I'll just breathe on it because we all breathe out carbon dioxide right so I'm gonna breathe on it I've always wanted to do that in a session and now if we switch to my laptop so on my laptop we can now see I have a firebase running here in Isaiah using firebase cloud firestore and on cloud firestore I'm actually detecting and I'm getting the readings from this little device so we can see the readings up here once they come up look here at the reading like 57 that was a moment ago when I breathed on it 54 I've actually gotten it up to 70 I must have eaten something bad the visual background here is about 42 so what's going on here how has this been done well with Arduino it's super simple to write code in C and can we switch back to the slides it'll be a little easier to see the code so if I go back to the slides we can see this is the kind of code that you would have written and C on an Arduino and all I'm doing is I'm reading pin zero that this is plugged into data pin zero on there I'm reading that and then every two seconds I'm reading it again and then what I'm doing with that is I'm posting that to a URL and then that URL is writing it's a firebase for me and that's all you have to do it's really that simple to build a sensor to start monitoring air quality very very cheap very very simple and then when you aggregate the data from all of these sensors together if you've got thousands of them in the cloud then you can start building some really intelligent solutions but there's something to take into account when you're doing IOT and that's think in terms of security so Dana security is very very important obviously these things are just generating data and pushing it up to my database but a smart hacker might be able to figure out how I'm doing that and then get access to my database you got to think in terms of data security and a pattern that I usually like to follow for this and for good reason too is instead of pushing from the devices directly up to your databases have something within your firewall that they are pushing to and let that act as a proxy and then that proxy will proxy out to your databases for you so then you know you'll have a secure connection between that proxy and the databases and then hopefully that secure connection will be enough and the reason for that is also to do it as an internal thing was that the power limitations on these things mean that a lot of these will only support HTTP they won't support HTTPS so that's why in my code a moments ago if you looked closely you would have seen that I was just calling oh I I was just making a client to connect server come at a tea you know in that case because it's just an open HTTP connection so that was to my internal proxy and then the internal proxy was sending it out to the cloud so things to consider you know very cheap very easy for you to build systems like this one coding with something in our card we know and see is super simple there's tons of API since tons of examples there that would do that you can just adapt and that's what I did in this case I'd never touched an Arduino before last Friday and it took a cup and I'm not particularly smart and it took it just a couple of hours to figure out how to hack some of the code that they had provided to be able to build this and write my data up to firebase so that's on a small simple sensor but now let's take a look at what happens when you go cloud scale okay thank you guys so that was a very simple scenario and lift you could take a look at another scenario where I will to have a little bit smarter devices such as kabuna if you take a picture with a camera you may want to detect some object what kind of object you to have in your images and tensorflow provide an API hope to do that that is object detection API so by using the object detection API you could have the labels of each in image or you can have the boundary boxes and you could have this course and it's really simple to use the API example you can just download the module file for the object a prediction API and you can have the dictionary imported cancels with the model file of the model then you can pass your images to the year additionally and that's it so you would get the output dictionary so we'd have the detection results such as the number of the object you have or classes objects or the boundary boxes and this course it's so simple so for example you can have the raspberry pi with the camera attached to a shopping cart so that you can take a picture of the inside of the shopping cart and apply the object detection API to what kind of to detect what kind of it items you are having or how many of them you have in the cart and we can use the cloud IOT core combined with the cloud and machine learning engine to build them a system that is that provides the production level scalability and availability for example if you want to build a smart communal system for the shopping cart you can have a Raspberry Pi and a camera attached to the shopping cart then use the quad IOT core to collect all the data stored data or storing data on the cloud pub/sub which could be here back-end for your server and we could use the google kubernetes engine or gke as an Orchestrator for orchestrating everything happening article outside and finally the GK we would be sending the data to the ML Benji that's where we are realigning the prediction with the object detection API let's take a look at the demonstration so I'd like to used here and the webcam to show house which is a laptop place yeah so this is how it looks right it's a very small crowd miniature card and we have a Raspberry Pi and display and camera so if you put something like this is a fake eggplant I bought it in a Japanese store also known as an aubergine for us Europeans or tomato then known as a tomato how detection API would look like so please switch to the art as you asked him yeah so let's wait a while I need to refresh walking maybe network issue oh yeah so thank you so it's so easy to detect what kind of the object and how many of them you have in the shopping cart so please go back to the slide so that worked so that was a very simple scenario where we just counted the items and crushed by them but you may think that what's the point of using cloud for this kind of detection and actually that is true you can run the object detection API just inside the tensile floor just inside the raspberry pie box and you don't have to use the ground but if you can collect all the data at the cloud side with the thousands of you because then you can extract certain or some collective intelligence from a thousand W path because for example you can train a machine learning model with the order of the shopping items you are adding to it occurred thousands of things you can you can trigger a machine learning model that can predict what will be the next item you will be putting into the shopping cart that will be reflecting the location proximity starting from the vegetables and fruit and meat like the chicken and the spices and pasta based on the already past the history of the shoppers and you can also have a small display with it showing the recipe recommendation with your next item to add that could be working as a navigator for the shoppers so let's call it as a smart shopping navigator and what kind of machine learning model we should design for implementing the machine are the navigator at first we have to represent all kinda many items you have in depart so if you have the tomato the card we may we may have this one hot vector that represents the tomato in your cat in this case you would have a 1.0 as a value in the vector and if you put eggplant in the cart you would have another 1.0 Vario in the vector if you have the chicken then you will have another one that 0 for chicken so this multiple one hot vectors represents the cart I can history and we stencil all you can write code like this so you would have a dictionary for shopping items and you can call the one what function to encode the shopping items as a vector and then you can have the multiple vectors that represents the hard item history and now we have the history and how do you detect the changes in the history in this case you can use the convolution single dimensional composition or one decomposition and Composition in machine learning is usually used to detect the certain patterns in a local group of the big data for example if you have a 2d images you can apply a 2d completion to find some shapes or patterns in the images that is how CNN or convolutional neural network works for image recognition but you can also apply the convolution to do one single dimensional data such as time series data for example you can apply the 1 decomposition on the cut I can history we can detect that what kind of changes that happen inside the card items and then you can flatten the output to get the result and we stencil off you can write the code like this you can call the one combo 1d function to apply the single dimensional composition and then pour the flatten to get the result so now we have the cut I can change history and we may also want to taking other factors such as seasonality whether it's a winter or summer or time of day because shoppers may want to choose different food items based on the season argue whether it's a summer hot day or whether it's a cold day so we put everything into a single MLP multi-layer perceptron which is a classic or neural network with the three layers to predict the next two items to add with stencil all you can write code to concatenate with everything into tensile and you would define your three layers of the MLP and that's it let's take a look at is how this mashup endemic Navigator works so laptop please yeah please laptop if you put the eggplant then it should detect the egg pan I see your shadow so the you are watching the same screen I'm watching here so at the right side you see the what kind of the recipes and I came so recommended so it looks like the system to recommends the pastor eggplant baked and I want you to make a pasta so I would put the tomato then the Navigator would show the other things you have to add would be I already have it but eggplant tomato and I want to make the ayah maybe I have to easier I want to make a pastor the mouse is also working masses walking or somehow it doesn't show what pastor man you may uh yeah I could try putting just chicken so that you can just follow the items on the screen so that you can get you can find the all the items to make the eggplant tomato pasta that how it works okay thank you so please go back to the screen just right so as you have seen on the demonstration by showing the recipe ideas and next item to add it works just like a car navigator for the shoppers so the shoppers can just follow the next item to add to fill the cut means the all the item is required to cook with certain recipe and that was the example how the crowd can be empowering to for the IOT devices not only for collecting data but also you can analyze it and learn some collective intelligence from it it's not done just an IOT anymore it's a Internet of smart things and this demolition demonstration was actually built by a Google Cloud partner called gurus notes and they have open sourced everything's on the get half so if you're interested please go to the github and search with smart shopping navigator to find out what kind of code in your drive and this is also they also provide you a production solution called Magellan blocks where you to have the user interface to build a whole data pipeline for collecting the IOT data and analyzing it and learning a training the model so with that I directly about the stage to Lourdes Thank You Cass pretty cool stuff right so now the third scenario that we wanted to look at was like on device inference and training and like there's a few different types of devices you can do inference and training on so for example you can do it on a mobile phone or a tablet you can do it on a Raspberry Pi or you can do it on an Android things device and there's a really cool video of doing it on a mobile phone or tablet that I've put at this QR code so take a look at this QR code watch this video I'm not gonna show it here but the idea behind this video is that if you went to the tensorflow summit you would have seen it but it's farmers in Tanzania in Africa who don't have cloud connectivity and who rely on a plant called cassava that if the cassava plant gets diseased it can kill an impact an entire crop and it's hard for humans to eyeball it and see it but they built a machine learned system that all you do is they wave their phone over the leaves and they can diagnose diseases early it's really really cool so that kind of on device inference without any cloud connectivity is possible through something called tensor flow light with tensor flow light you can also do inference and training on a Raspberry Pi and I have a car here that I'm gonna show in a moment that's driven off of a Raspberry Pi and if you were in the sandbox area you would have seen cars like this one self-driving around but basically there's a little camera at the front here and what I would do is I would manually drive the car record what I'm doing and then based on the telemetry that I'm sending to the car to drive it around and the corresponding video that's giving us the features and labels that we want that the car will then remember and then use that to self drive if we switch back to slides for a moment and then finally there's the third one which was shown in a moment is a nun Android things device all this is made possible because a tensorflow light and a strongly recommend check out tensorflow light and check out the talks on tensorflow light but the idea behind tensorflow light is that if you want to train a machine learn to model we have a bunch of converters that will then flatten that model and shrink it to make it more mobile-friendly there's an interpreter core then which is used to execute that model so you can do inference on the model so for example things like if you want to do an image classification it'll do that locally on the device without roundtrip to the cloud and then there's all the stuff that's needed like operation kernels and hardware acceleration to take advantage of the mobile hardware that you have it runs on Raspberry Pi it runs on iOS and of course it runs on Android and it works really great on a pixel I have a pixel two up here and if you want to do it see a demo of it come see me afterwards and I can show like real-time inference on the pixel it doesn't really work on the stage that well now here's a video that I shot of me doing real-time inference and you can see like it's so the coffee mug correctly and here you know it this coffee mug kind of looks a little bit like a mixing bowl so it got like at some points it felt like a mixing goal and I know that's a horrible product placement for my show coffee with a Googler but I had to do it and here it was able to recognize a mouse and stuff like that so this is what it would look like running on Android and so for the self-driving car that I've been talking about so you can see on the right is the Raspberry Pi and on the left is something called a pulse wave modulation controller and that's what actually drives the car itself so I have tensorflow light running on the Raspberry Pi and that's what will actually drive the car and this is built on a project called donkey car and if you want to go and take a look at donkey car and details about donkey car they're at this URL so should we take a look at what it would look like to actually train the car so if we can switch to my laptop please and so I've actually started up the Raspberry Pi on the car and there's a little web server running on that so let me go and see if I can access it hopefully it will obey oh now it's refusing to connect oh sorry and so what I'm doing is it runs on Python so I'm just running the Python code sorry has to make you hold it so long so this is actually booting up the web server that's on the car that I would then use the train and drive it so I'll take a moment just to boot that up but what will happen is like so if I want to control the car when I'm training it obviously it's hard for me to do it on it so it has that little web server the web server is connected to a hot spot on my pixel and my laptop's connected to the same one so it's starting up that server it looks like that service done yep and here it is so as calves moves it around you'll see that one and if I were to like try and drive the car I made a bit of a mistake if you've been over to the sandbox you would have seen the cars they're using are really small ones and I didn't read the specs properly and I bought this one and it's an absolute beast it goes really really fast I don't dare put it on the floor and start driving because it will take off and it will land about three rows back so watch out calves and you see it's like this what it looks like if I'm driving and training it you know I can steer and I can control it like this so it's you know what it's doing right now it's recording what it would see on the webcam from the car actually moving and storing all that stuff that were then used to train a tensorflow model for that specific area so when you see the cars over in the sandbox they're on that figure of an eight track so we drove them around that figure of eight we train them in that way and then instead of launching it in training mode as I have it now you just launch it in drive mode and they go and they drive themselves so you can build one of these as a donkey car it's okay we can put it down we can build one of these using the donkey car project it costs this one is a little bit more expensive but they generally cost maybe about two hundred dollars total for everything and you can build your own little self-driving car and it's all open source it's really really cool so thank you can we go back to the slides please the motor on the car is really loud so I'm gonna turn it off because we hear it wizzy okay so that's what it's like to train a donkey car and that's the kind of with on-device inference you can then have a believe it or not a model self-driving car I think that's really cool now the next thing was Android things that I mentioned and did everybody get one of these right so these are just so cool so this is like just a little it's this one isn't actually a Raspberry Pi it's a different developer board but a similar concept and with Android things being able to execute stuff and with tensorflow light running on Android things you can start doing inference and it has anybody done the code lab have you tried it out to do inference on these try it out and build it and you'll be able to do things like this so this afternoon back in my hotel you know on this device I kind of pointed it at a water bottle and I thought it was interesting that it gives you three things that it thinks it is it thought it was a water bottle a pop bottle or toilet tissue I don't know if it's if it's because of the circular parts of it or anything like that and then I tried it on this one and you know it's a coffee cup a coffee mug a cup or a measuring cup and particularly way the handle is placed I thought it would be really interesting to try and fool it with the handle the way it's placed now I thought it might think it was a picture because sometimes when I do mugs like that it kind of classifies that as a picture and then of course this one I tried on my Android my little French Android and I thought he was a piggy bank a teapot I guess he's a little teapot the way he's standing or maybe even a spotlights and maybe it's the shape of it or something like that but that's the stuff that's built in and like when you when you assemble this device that you've got that app is actually already running on here and that app is open-source and one of the really cool things you can do is that the image model that that app is trained for you can actually retrain that for your own images I've been working on a version of it that I'm going to add to the car and maybe we'll be able to talk about it at a future i/o where I'm retraining the car to recognize things like traffic lights so if it sees a red light it will stop if it sees a green light it will go and all the code for you to do that is available with this kit when you get it so give it a try and for those of you watching online just check out Android things you can you can buy these kits they're relatively cheap and there's so much fun to play with so Anna code labs that I mentioned if you're here at i/o you can go over and try it out if you want to try it on your machines at home the URL for it is at this QR code go give it a you know have a play with it it's it's it's a lot of fun so in recap you know we spoke a little bit about Internet of Things in AI and about the trends that are happening that the explosive growth that is going on the growth of actual devices and the amount of data that they're producing and then the things that you can do with that we looked at like with a sensor data on an Arduino we looked at cloud scale AI and we looked at on device inference now there's a whole bunch of things that you can do to go and take a look at okay um come on slide animate there we go so things that you can try out number one writing data from your Internet of Things device to cloud storage there's great details on it at this URL if you want to explore the tensorflow object detection API which is what cads was using here when it was detecting tomatoes and eggplants or aubergines and that kind of stuff you can find detail than that at this link IOT in the cloud ml engine details for that are at this link and this one and tensorflow and tensorflow light and all of the mobile scenarios that I was showing here were based on sensor flow lights you can learn about that at this link or attend the talks if you here and then finally exploring Android things itself I find it super cool and it's really nice because if you're already an Android developer you can start building stuff using your existing skill sets when I was doing the thing for the Arduino here I had to give myself a crash course in C you know but if you like I said if you're already an Android developer oops and you're used to using Java or Kotlin to build your Android apps then you can start building things apps with that too so with all of that we just want to say thank you very much feedback or I was always welcome we'd love to hear it and it's at this URL thank [Music]

As found on YouTube

Intro to the Google Assistant: Build Your First Action (Google I/O’19)

if you're new to developing for the Google assistant you have come to the right talk and if you're an experienced assistant developer don't worry we're gonna tell you what's new our mission for the Google assistant is to be the best way to get things done that is an ambitious mission I think it sort of rhymes with organize the world's information and make it useful and accessible and just like Google searches mission from 20 years ago our mission today for the assistant requires a large and vibrant ecosystem of developers and that's where all of you come in so whether you're joining us here the amphitheater shoreline watching on the livestream or on YouTube later this talk is going to tell you how you can make your experiences shine on the Google assistant if you're a Content owner this talk is going to tell you about schema.org markup and templates and how you can make your content look great on the assistant and if you're an Android developer wait I hear there's a few Android developers here today where are the Android developers where are you yes thank you if you're an Android developer this talk is going to tell you how you can use app actions to voice enable your Android apps and if you're an innovator in this new generation of conversational computing this talk is going to cover interactive canvas and conversational actions how you can use HTML CSS and JavaScript to build rich immersive actions for the assistant and if you're among the few the proud the hardware developers at i/o any hardware developers a couple this talk is going to tell you about the new innovations in the smart home SDK but before we do any of that Naomi is gonna tell us a little bit about why now is the right time for you to invest in the Google Assistant we're gonna start by going back in time I want you to think about when you first use the computer for some of you it was the 80s maybe you played a game or store to receipt or if it was the mid-80s you even used a word processor for others it may have been the 90s games are a little bit better and you navigated via the mouse instead of the command line ten years after computers entered our home cell phones entered many of our hands and by the mid to late 90s communication was more portable than it had ever been before but it was still very early you remember the days when text messaging was a game of back and forth between the inbox and the sent folders yes we've come a long way now another 10 years later in about 2007 the first smartphones entered the market and then mobile computing exploded so you may notice a trend here about every 10 years or so there's a fundamental shift in the way that we are computing from desktop to mobile and now to conversation so what does these chips mean for all of you as developers well it means that you have a lot to think about a lot to build on and a whole lot to build with and this is because each new wave is additive to the one that came before we're still clicking we're still typing we're still tapping and yes now we're also talking we're looking for more assistance in our daily lives and we're turning to the devices around us to get things done now Google's approach to this era of conversational computing is the Google assistant it's a conversation between the user and Google that helps them get things done in addition to the Google assistant there are also assistant enabled devices like Google Home Android phones and more and finally there's actions on Google which is our third party platform this enables developers to build their own experience on the assistant it's an entirely new way to engage with your users as they're using conversation to get their things done and it was announced on this very stage just three years ago now in just a few short years we've seen an incredible evolution in terms of how users are talking to the assistant and this presents opportunities for developers across the globe now think back to those first use cases on conversational platforms they were very very simple and straightforward they were limited to things like turn on the music turn on the lights turn off the music turn off the lights again simple straightforward commands that fulfilled users very low and limited expect a shion's but there have been three incredible shifts in querying that have occurred over the last couple of years first we're seeing that users are having longer conversations in fact query strings on the assistant are about 20% longer than similar queries on search second they're more conversational they're 200 times more conversational than search so queries are going from whether nine four zero four three two something like do I need an umbrella today like you might ask a friend a family member or even a real-life assistant and third queries are action-oriented it's 40 times more action-oriented than search so users aren't just searching for information but they're actually looking to get things done they're finding that restaurant for Friday night and they're booking that dinner reservation and the evolution of the query is due to a couple things happening simultaneously so first we're seeing that technology is improving natural language processing and understanding improvements have actually decreased the word error rate which is a key metric for speech recognition it's now better than what humans can understand simultaneously the number of assistant ready devices has soared so it's turned this new way of computing into an ambient one it's always there when we need it no matter what environment we're in or what device were on it's magical but it poses a really new challenge for all of us which is how do we reach the right user with the right experience in the right moment all at the same time so sync – a pretty typical day we'll talk about some of the touch points where the assistant might be helpful so first you wake up good start now if you're anything like me you really would love to keep your eyes shut for that extra 20 seconds but you also need to kick start your day and find out where you need to be and when well the assistant can help with that now you're waiting for the subway you're in a crowded loud station you have a couple of moments of idle time before that train comes to maybe pre-order your cup of coffee or by your friend that birthday gift you've been meaning to send them you're the assistant on your mobile or your watch can help in those moments as well and finally you're sitting on the couch the end of a long day your laptop or your mobile phone are probably not too far away but neither is your Google home it's there to help you so across these moments and more Google is handling the orchestration that's required to deliver that personalized experience for the user with the context appropriate content so you as the developer don't have to worry about which experience which device which manner you can just leave that to us so what does this mean for the developer well you have more ways than ever to be there for your user you can reach users across the globe in over 19 languages across 80 countries on over 1 billion divisive today with over 1 million actions but more than that it's actually easier than it's ever been and this is something we're all really excited about I know we're all balancing far too many projects for the numbers of hours in a day so today we're going to talk about how you can get started if you have an hour a week or even an entire quarter to build for the assistant we'll talk about how to use existing ecosystems as well as how to build net new for the assistant and we'll focus on four major pillars so first we'll talk about how to use your existing content this leverages what you're already doing in search so web developers we're gonna be looking at you for this one second we'll talk about how to extend your existing Android investments leverage the continued investments you're making in your mobile apps and app developers I heard you before so you're gonna want to pay attention to that section third we'll talk about how to build net new for the assistant so if you're an innovator in the conversational space we will share how to get started and finally hardware developers saw a few hands go up before where if you're looking to control your existing device cloud our smart home section will appeal to all of you within each section we'll talk about what it is and how to get started but before we do Dan is gonna tee up a very sweet example that we will use throughout the rest of our presentation so a single unifying example that shows all the different ways you can use Google assistant now this gets me thinking about two things one I love s'mores I have a sweet tooth and I'm also an engineer here in Silicon Valley home of tech startups of all kinds and the tech hub of the world so how can I combine my love of s'mores and my love of Technology talking it over with my colleagues Brad and Naomi we thought of the idea of using a fictional example company that you as a developer can show all of the different ways that assistant can help you and your company with things like building a global brand through Google assistant increasing your global sales customer growth and acquisition and even things like user re-engagement like the very important metric of daily active users and so the first pillar that we have is how you can leverage your existing content with Google assistant so like many of your companies s'more s'mores has a lot of existing content that's ready for the assistant they have a website they have a podcast and of course they have recipes so that we can all understand how to make that perfect s'more at our next bonfire now also just like you they spend a great deal of time optimizing their site for search so we're gonna talk about how they and of course how you can extend existing efforts and optimizations to the Google assistant now Google's presented ways to optimize your content for search since the 90s we work hard to understand the content of a page but we also take explicit cues from developers who share details about their site via structured data structured data is a standardized format for providing information about a webpage and classifying that page content for example on a recipe page you can disambiguate the ingredients from the cooking time the temperature the calories and so on and because of this markup we can provide users with richer content on the search results page answers to questions and a whole lot more and this brings Google search beyond just 10 blue links and over the last year we've been hard at work to enhance the Google search experience and enable developers to extend their content from search to other Google properties like the assistant so for sites with content in popular areas like news podcasts and recipes we have structured data markup to make your content available in richer ways on search same optimizations that you make for search will also help your content people with discoverable and accessible on the assistant and it's just standard RSS you've seen this before and that's the approach we've always taken we're using industry standards and insuring those optimizations are ready for search and the assistant to and I'm so excited now to announce two brand new programs that we're adding how-to guides and fa Q's so the additional optimizations that you make for search will soon yields a richer experience and an automatic extension to the assistant let's dive into each so how-to guides enable developers to mark up their how-to content and make it discoverable to users on both search and the assistant what displays is then a step-by-step guide to the user on anything from how to change a flat tire to of course how to make that perfect some more so on the left here you can see a nice image based preview of the how-to content in some more some more site it allows the user to engage with your brand further upstream in their journey and it differentiates your results on the search results page and if you don't have images don't worry we have a text-based version of this feature as well now on the right you can see the full guided experience on a home hub device again all powered by the same markup on some more some more site now the best thing about creating a step-by-step guide is that you actually don't have to be technical to do so now I know we're at i/o and i/o is the Developers Conference but if you have one hour to dip your toes in the assistant pool and you don't have a developer who can devote the time to adding the markup don't worry we have ways for you to get your content onto the assistant even as simply as using a spreadsheet so now you can combine your existing YouTube how-to videos and a simple spreadsheet and actions console to get a guided how to experience across many assistent enabled devices so s'mores s'mores calm now has two ways for how they can get their step-by-step guides on how to make that perfect some more onto the assistance if they have a developer with some extra time they can add the markup or they can use a simple spreadsheet to extend their existing YouTube content now we're gonna switch gears a little why to think about how times you turn to search for answers to questions maybe some of you are even doing it right now that's okay maybe you're trying to find out the return policy to your favorite store or if there's a delivery fee for your favorite restaurant blackout dates for travel the list truly goes on and on while our epic use markup enables a rich answer experience on search giving users answers directly from your customer service page so the same optimization will then enable queries on the assistant to be answered by the mark-up you already did and it's so easy to implement so when a user queries something like what's more smores delivery fee on the assistant a Google will soon be able to render the answer from that same markup on your customer service page and here are some developers that have already gotten started with FAQ s and how-to guides and we'd love to have you join us tomorrow at 10:30 in the morning to learn more about how to enhance your search and assistance presence with structured data of course the talk will also be live streamed or you can catch it later on YouTube so as you've seen there are three main ways that s'mores s'mores calm and you can leverage existing content first you can ensure the hygiene of your structured data markup for podcasts news or recipe content you can add the new epic use markup to your customer service site or you can use a new template to bring your content to the Google assistant we're so excited about the ways that we're making it even easier for developers to extend existing content from search to the assistance but we're also making it even easier for companies to engage their existing ecosystems like Android so let's talk more about app actions all right thank you how about that Sun huh you enjoying the Sun in shoreline I can't see anything without these so I'm gonna go with this so we're my android developers again Android developers yes just like many of you the s'mores s'mores company has a popular Android app they want to make it as easy to order s'mores as it is to enjoy them but just like many of you they face the high cost of driving app installs coupled with the reality that users are you using fewer and fewer apps each year the sea of icons found on many users phones might be a contributing factor it's hard for users to remember your icon much less find it what we need is a new way for users to find and launch your android apps one that's focused more on what users are trying to do rather than what icon to click last year at i/o we gave a sneak peek at app actions a simple way for Android developers to connect their apps with the helpfulness of the assistant and with the Google assistant on nearly 1 billion Android phones app actions is a great way for you to reconnect with your users app actions uses Google's natural language understanding technology so it's easy for users to naturally invoke your application and finally app actions doesn't just launch your app it launches deeply into your app so we fast forward users right to the good parts now to help you understand this concept let's walk through an example of course using our s'more s'mores app let's first take a look at how this looks the traditional way so first of course I find the s'more s'mores app in the sea of icons does anybody see it next I select the cracker okay that that makes sense and then I have to choose a marshmallow all right and then I get to pick the chocolate and the course a toast level okay I've got to say how many I want that's important too and then finally I can review my order and confirm now that's a long intent to fulfillment chain it's a long way from I first had the desire for a warm delicious s'mores before I got it successfully ordered and that means there's opportunities for drop-off all along the way now let's take a look at what this looks like once s'more s'mores has enabled app actions first you'll notice I get to naturally invoke the app with just my voice I can say order one milk chocolate from s'more s'mores and then immediately we jump right to the good part of this application confirming that order notice we got all the parameters correct and then we just confirm and we're done we're ordered it's a short path from when I had the intent for the warm delicious more before I got it ordered but of course we didn't build app actions just for stability s'mores we had a few partners that have already started to look at app actions so for example I can say to the assistant order a maple glazed donut from Dunkin Donuts of course I might need to work that off so I can say start a run on Nike Run Club and I might want to settle that bet from last night by saying send $50 to naomi on paypal so let's take a look though at what enables this technology what's going on under the covers here foundationally at Google we connect users that express some intent with third parties that can fulfill it and app actions is the mechanism that you as app developers can use to indicate what your Android app can do each app action each built in and it represents an atomic thing a user could want to do including all possible arguments for that so you just need to implement that built-in intent and I handle the arguments that we pass to you the cool thing about built-in intents is that they model all the different ways users might express an intent for example these are the ways users could say start an exercise notice as an app developer you don't need to handle all of this complexity with this different kinds of grammar we handle all of that for you you just implement the built in intent so speaking of which let's take a look at how it looks for you as a developer to implement that well the first thing you'll do is open up Android studio and add an actions xml file you notice on that second line there it's order menu item that is the name of the built-in intent that we have implemented for our s'more s'mores app and then in that fulfillment line you'll notice a custom schema URL so you could of course use an HTTP URL as well this just tells us where in the application we should fast forward into and then at the and then you'll notice we map the arguments there so the menu item name is the argument name from the built in intent and then notice our URL is expecting item name and then finally at the bottom there we're giving some inventory what are the kinds of things users might say for this application and just for brevity I put the usual now we just need to handle that in our on create function so very simply we parse that item name parameter out of the URL we check to see if it's that identifier and if so we just pre populate the UI with exactly what we want this is a very simple integration that you can get done very quickly on the assistant so the good news is that you can build and test with app actions starting today we're releasing built-in intents in these four categories in finance food ordering ride-sharing and fitness so if your app is in one of those categories you can build and test right now and of course the team is already working on the next set of intense so if you have ideas or thoughts on what intent would be great we love it if you gave us feedback at this URL but there's one more thing app actions and built-in intents it also enables slices slices is a way that you as an Android developer can create a declarative version of part of your application and embed it into Google surfaces like the Google assistant so in this case we're implementing the track order built-in intent and then you can see in line there that's our Android slice showing up right in line in the Google assistant making it quick and easy for users to get that information and then launch into your app if they need more advanced functionality so what did we see here you can enable users to invoke your app with just their voice with app actions there's a simple integration model all you need to do is map the intents in your app to us common set of built-in intents and then the good news is you can build and test starting today with more intense coming soon so we've seen how you can leverage your existing content with Google Assistant we've seen how you can integrate your Android applications with Google assistant now I want to talk about conversation specifically conversation actions that is how you can build new custom experiences for the Google assistant so why our conversation actions so important well for one it's a way that you can natively control the devices capabilities so if the device has a screen showing image if the device supports touch show a suggestion chip that they can tap it's a way that you can increase your brand awareness through things like custom dialogue and agent personas you can grow your user re-engagement through well-crafted conversation design and things like action links and furthermore you can drive habits and interactions with features like routines daily updates and push notifications now what exactly our conversation actions it's the idea that you as a developer have full control over the dialogue the back-and-forth of what's said this is distinctly different than that of app actions or content actions or even smart home actions where you have some type of fixed markup or maybe you're using a built in intent something that how defines already what your the material that you're trying to access Google assistant takes that fixed markup and ingests it applies its own natural language understanding and matches what the user says automatically to the access of that material with conversation actions that's flipped you as a developer define custom intents you define the types of phrases the that a user might say to match that custom intent you even define that information that you want extracted out of what they say for those same intents so with conversation actions we need some type of tool that can help us do this and that is dialogue flow out-of-the-box it provides two key elements the concept of user intent or what the user actually wants and the concept of entity abstraction the way that you glean information out of what they say let's dive in a little bit with a small example so if we take some oars I would like a large s'more with dark chocolate and I want it to go dialogue flow can take this phrase as a whole and match it to the user intent that they want to purchase a snack of some kind now you see here a few words highlighted in the sentence large dialogue flow can understand that they want a large snack s'more the type of snack dark chocolate the topping of that snack and they want it to go rather than for a delivery or for their so when we took it take a look at this at a sequence of dialogue and expand it a little bit more the user might say something like hey gee talk to s'more s'mores s'mores s'mores in this case is the invocation name of your action Google assistant takes that audio transcribes it into text applies its natural language understanding and invokes your conversation action from that point forward it's between you as a developer and dialogue flow that's controlling the responses back to the user and so let's take a look at a live demo here I have dialog flow and a few intents that I've already defined I have a small shop where you can order a snap you can order whatever you last ordered a gift card and so let's take a deeper look into ordering a snack when I look at this I have a few things I have my contexts I have my training phrases that I've already supplied these are the phrases that I think as a developer the user might say that app that matches the intent of them wanting to purchase a snack of some kind if I scroll down I can see the types of parameters and the relating entities of those parameters specifically things like delivery pickup the type of snack the size of the snack toppings etc now if I scroll down further you'll see the responses that I've created as well that reference the entities that I've also defined if I scroll down even further you'll see the fulfillment if I wanted to have a custom fulfillment I can have a standard webhook call for this type of intent now let's look at an example here if I say one large s'more with milk chocolate you'll notice instantly without additional input from me dialog flow has highlighted several key elements within this phrase large it knows that's the size of the snack that I want s'more the type of snack chocolate type of milk so there you go that's that's pretty powerful stuff now let's take a look at it in the phrase of a full conversation if I say I would like to order a large s'more with dark chocolate instantly it gets the information it has the various contexts it matched it to the intent of ordering of snack and we scroll down it also still has the various entities and parameters that it's extracted now the default response here is that I've defined a required parameter of delivery or pickup and so it's asking me will it be for delivery or pickup I respond delivery and there you go it understands you've ordered a large dark chocolate s'more and you want it for delivery there you go so this is powerful stuff for Google assistant now let's go back to the slides here what we have is a way to build conversation now Google assistant supports a huge array of devices and surfaces and we want to be able to scale across them all different devices support different capabilities a smart speaker is voice only a car the voice forward it has a screen but you still want to have voice control to intermodal like your cell phone to maybe your SmartWatch which is screen only we need some type of feature that can fully utilize the screen and in the case of s'more s'mores they want to be able to build a rich game something that's voice first custom full-screen visuals they want to build a turn-based battle system that's multiple surface and supports all these different kinds so today I'm happy to announce the brand new API called interactive canvas this is an API that enables pixel level control of rendering any HTML any CSS and JavaScript let me reiterate that is a full webview running on the device if it supports full screen visuals animations even video playback now it wouldn't be Google i/o without another live demo let's take a look here what I have is a hub and for this demo I'm gonna need some some audience interaction here I'm gonna be playing a trivia game and it's gonna be a really hard question so I need you guys all to scream out as loud as you can the answer to this question hey Google Play HQ University hears HQ University yes that's me the host who's khalifa host malone the trap Trebek and I'm here to welcome you to HQ niversity you've been accepting to this elite program to help smarten up and sharpen up your trivia skills to help you win HQ the rules are very simple my assistant Alfredo is gonna ask you a series of questions that just like HQ start easy and get harder you will respond with the answer you think is correct and the H universe will reward you let's get down to the nitty-gritty let's get this show on the road baby Alfredo pinin with Camaro numero uno ready question 1 if you put on a hoodie what type of clothing are you wearing your choices are cloak sweatshirt or cape so what's the answer sweatshirt yeah baby you did it we did it awesome next so that is interactive canvas a way that you can build fullscreen visuals and custom animations so how does something like this work well first off we have to start with a regular conversation action where the user says something to the device that in turn goes to the actions on Google platform and turn to dialogue flow and finally your custom fulfillment your fulfillment supplies a certain type of response in this case it needs to supply a rich response when we add the web application the canvas it supplies a immersive response which tells the Google home to load the web app your your web application in turn has a specific JavaScript library called the interactive canvas API let's look at some code for this when we look at this this is the custom fulfillment you can see here I have the default welcome intent that's supplying a new immersive response that supplies the URL of the web application and the initial state of that web application what does it look like on the web application side well when we see this there's two main elements that you need to include there's the CSS stylesheet this supplies the specific padding for the header on the devices and things like that and then the actual JavaScript library this library itself manages the state of your web application with the states of the conversation so that you can control both in unison so some key takeaways around conversation is that dialogue flow is the tool for developers to where you can build custom conversations you control the back-and-forth of your dialogue and – we've announced the interactive canvas API the pixel level control over the display for games where you can run HTML any CSS and any JavaScript now I want to switch it up a little bit talk about smart home the ability for you to control any hardware with your voice now traditionally smart home has been all about cloud to cloud communication so when I turn on my Phillips hue light bulbs what's actually happening is that Google assistant takes in the audio transcribes it into text applies natural language understanding and sends a specific response to philips hughes servers philips you in turn controls the light bulb and so now i'm glad to announce a new api the local home SDK this provides new local control over your devices with post assistants latency of under 200 milliseconds supports local discovery protocols like UDP broadcast mdns UPnP for actually controlling devices UDP TCP and HTTPS now with smart home there's device types and device traits it supports all of them out of the box with the exception of two-factor authentication now my favorite part is that it's come as you are meaning you don't need to change the embedded code on your device for the messages what's actually happening is that you develop a JavaScript application that runs on the home device that's pretty awesome so let's take a look at how this works when the user says something like hey G turn on the lights again that audio sent up to Google assistant transcribes it into text applies natural language understanding Google assistant creates a execution intent and sends this intent this structured JSON response down to the Google home where the home is running your JavaScript application your JavaScript application in turn understands the intent of trying to turn it on with the exact device ID things like that and constructs the specific message that your device support in turn turning on the light so I want to show more traits and more types today we're always adding more device types that we support with goop with the smart home api's and today we're releasing even more with things like open closed starts off with zones lock and lock even devices and pipes like your door boiler garage door now again we're adding these all the time and today we're releasing a huge number more so I want to show now how s'more s'mores can use smart home so what I have here is a toaster oven some of you might have already seen this up here and wondering what it's used for so I have a toaster oven here with inside of it some s'mores that I want to eat and I want to toast perfectly I also have a Google a ìwhy vision kit which inside of that is a Raspberry Pi zero and I'm using this to control the power to this toaster oven and so let's take a look hey Google turn on my s'mores toaster okay turning s'mores toaster on awesome so there you have it a smart home device that's being controlled via voice with Google assistant and so let's recap some key takeaways one we announced the local home SDK where you can control real world devices using local Wi-Fi with your voice and second we've announced a huge number of new traits and device types these these are available today that you can use in your own devices and so what to do next all right we have covered a lot today but we are so excited to share with you all of the ways you can get started building your first action so to quickly recap you can use your web content and leverage what you're already doing in search you can use app actions and leverage the Android ecosystem you're already participating in you can build a custom experience for the assistant by building a conversation action or you can extend your hardware and build smart home actions to control the devices around your home but this was just the beginning there are 12 more sessions this week that will dive into topics we only had a chance to introduce in our time together today so these additional talks will be geared toward Android app developers web developers hardware developers or anyone who just wants to learn how to build with conversation or some insights around this new ecosystem so please check them out live watch them on the live stream or of course tune in later on YouTube now for those of you here with us this week we have a sandbox out back office hours and a code lab now I think I heard our toasters ready so it's time for us to go enjoy some mores but visit our developers site talk to us on Twitter and we can't wait to see the rest of i/o thank you so much for joining us today [Applause] [Music] you [Music]

As found on YouTube

How to get one-meter location-accuracy from Android devices (Google I/O ’18)

[Music] we're going to show you how recent changes in hardware and standards make one meter location accuracy possible in some cases as soon as this year I'll give you a short overview now then Roy will introduce Wi-Fi round-trip time technology and standards and show you a live demo then way we'll explain the Wi-Fi api's to now return and talk about new GPS technology and api's at the end they'll be loading up for the next session so we'll take questions right outside that door and will be available at office hours at 1:30 p.m. today so it's a great time for location applications because technology hardware standards and Android api's are all evolving simultaneously to enable accuracy that has not been possible previously in phones so eventually this means high accuracy for everyone but today we want to take you under the hood of location because we want to give you the opportunity to get a head start on the future we also want to highlight the need to protect and respect the user the more people who use location the more people the more careful we and you have to be will highlight where you must get user permissions and we'll close with some guidelines for making great location apps so where are we today with indoor location accuracy if you think you've noticed that your phone seems to be more accurate when you're inside shopping malls and office blocks then it was a few years ago you're not imagining it with each release of the fused location provider we have had steady improvement of the Android algorithms and machine learning for Wi-Fi locations there continues to be improvement and you'll see indoor accuracy of better than 10 meters but round-trip time is the technology that will take us to the 1 meter level meanwhile what about GPS well terms of GPS accuracy in the open sky there has been not much change in the last few years you're out in the open sky your GPS accuracy from your phone is 5 meters and that's been constant but wrong measurements roar GNSS measurements from the phones you can now improve on this and with changes in satellite and receiver hardware the improvements can become dramatic now everyone's familiar with the blue dot but to get the blue dot you need the location provider of course and to get location you need measurements specifically range measurements from Wi-Fi access points or from GPS satellites today we'll show you how one meter measurement accuracy is available in phones the key technologies are Wi-Fi round-trip time GPS dual frequency and carrier phase measurements and we'll show you how to use accurate measurements to create accurate location now if you just want to wait a year or two this will find its way into the worldwide ecosystem and the fused location provider but we want to give you a chance for a one to two year lead by taking accurate measurements and turning them into accurate location we want to work with you to accelerate the future to take it and bring it closer to the present so you might wonder well why do I need better location accuracy anyway well let's just look at two instances where existing apps could use much better location accuracy so for indoor routing or navigation of the kind that you used to in your cars you need much better accuracy than you have outdoors you need one meter accuracy because indoor features like the distance between cubes or aisles only a few meters and even for the most loved outdoor applications such as directions and specially directions in traffic we could use higher accuracy than we have now for example when you came here this morning in a car you probably had the time estimated by the average traffic speed what you really want is the traffic speed in the lane that you're in so that you could ask how fast would it be if I could take the carpool lane they of course many other use cases and I'll mention a few before we finish but the important thing is that we are sure that you will have many more ideas than we have and that's the beauty of the open Android ecosystem so now here's Roy to tell you about Wi-Fi round-trip time thanks Frank and I'm very excited to be here today to tell you about a new positioning technology in Android P we call Wi-Fi round-trip time or RT t you'll hear me say that acronym a lot which is basically measuring the time of flight of RF signals it has the potential to estimate your indoor position to an accuracy of one to two meters now we're going to hit the ground running today before I tell you about the details of our TT and we're going to show you a video of indoor navigation based powered by OTT and I want to emphasize first of all that this is not a product but an internal prototype to explore the power of the technology and how it can also be used to support other context-aware applications this prototype also showcases some of the magic that Google could offer to its employees today so we're gonna roll the video and what you should keep in mind if this is a bit like car GPS except we're working indoors so in a moment you'll see there's an application it's a mapping application and we're searching for a conference room we found that conference room it's plotted the route that's the shortest route and now we're off we're following the route and as I make progress you can see the route is turning gray my position from our TT is the big rage lot and I'm deliberately making an error here so the system is rerouting and it's rerouting again if I get about 20 feet away it starts the rerouting process and I'm following the route and you can see the code of flying by and there and coming in and I've arrived at my destination conference room Tefo so that is the power of our TT and the thing to think about thank you the thing to think about here is that if you didn't have 1 to 2 meter accuracy then when that system rerouted it would jump potentially between aisles that were surrounding me and it would be a terrible user experience so that's why it's so important to have this kind of accuracy so before I get into the details of a Wi-Fi RTT I want to tell you about how we calculate location indoors now we use Wi-Fi RSSI which stands for received strict signal strength indication and basically we can calculate distance as a function of signal strength now the figure that you see on the left-hand side here the access point which is in the center has a heat map of signal strength the green is the strongest and the red is the weakest at the edges and I've placed two phones on this diagram at the transition between the weak and the strong notice that the phone on the right is further away from the access point than the phone on the left so say the signal strength different distance and it's this variability in distance for signal strengths that unfortunately makes it very hard to get accurate our readings from our SSI on a regular basis but there are lots of algorithms and tricks that we can pull to improve this but it can be improve further and that's where Wi-Fi RTT comes into place so Wi-Fi RTT round-trip time it uses time of flight instead of signal strength it measures the time it takes to send a packet a Wi-Fi RF packet from an access point to a phone and back again and because radio signals travel at the same speed as visible light if we multiply the total time by the speed of light and we divide by 2 we get distance round-trip time divided by 2 and we get the range from the phone to the access point so that's the basic principle now if you want to calculate position we have to use a process called multilateration and more on that in a minute but the key thing to think about here is the more ranges you have the more constraints you get in a more accurate position you can achieve and if you can use at least four ranges than we think you can get typically an accuracy of one to two meters in most buildings so why am I telling you about Wi-Fi RTT today why not last year or before well what I want you to take away is that 2018 is the year of Wi-Fi our RTT in android your takeaways are that we are releasing a public API in Android P based on the I Triple E eight to eleven MC protocol and furthermore we're also integrating aspects of this protocol into the fused location provider which is the main location API that people use to put location on a map and anytime there are our TT access points in the vicinity the accuracy of that position will be greater a little bit of history the 82 11 standard was ratified in 2016 the end of it and in early 2017 the Wi-Fi Alliance started doing Interop between silicon vendors to make sure the chips followed the protocol and that's when we started doing a lot of work to validate how it could be integrated into android and by the fall of this year of course we will release the API so that all of you can I have access and build your own applications around that technology so now diving into the principles of how Wi-Fi MTT works so the ranging process starts with the standard Wi-Fi scan the phone discovers the access points which are round and based on certain bits which is set inside the beacons and the probe responses we can figure out which of those access points are RTT capable and the phone chooses one of those two range – and it starts by making a request to the access point and as a result the access point will start a ping pong protocol back the ping is sent to the phone is called an FTM or fine timing measurement packet and the pond which sent back to the access point is an acknowledgment of that packet the time stamps are recorded at each end that each device records them but for the phone to calculate the total round-trip time it needs to have all of those time stamps so the access point sends one more packet third message which contains the missing to the phone then simply calculates the round-trip time by subtracting the time stamps from the AP and as its own turnaround time which all the time stands that it recorded so that leaves the time of flight we multiplied by the speed of light to get distance we divide by two and we get the range that we care about now it turns out if you do this process multiple times you will in fact get more accuracy and so that's what the protocol allows for it allows for a burst in Android we're typically doing a burst of about eight of these of these events and as a consequence the system can calculate statistics so the mean and the variance which allows us to more accurately plot a position on a map and knowing the accuracy also allows us to calculate a path miraculous well so now you have ranges how do you get a position so I just want to give you a feel for how you go about doing this now there's lots of different mathematical approaches and I'm just picking one because it's relatively easy to explain but this is where the power of developers comes in for you to figure out your own own way to do it so if you know a phone is at a certain range from an access point that tells you that it can be anywhere on the circumference of a circle of that radius that radius R and I've written the circle equation for that circumference on the right hand side that center x1y1 now if you want to find a position you've got to constrain it so if you take four ranges to four separate access points as I've shown on the diagram on the right hand side you then can see that if those ranges were accurate those circles would intersect at a single point how do you find that point programmatically well if you write those four equations out on the left hand side you see they're nasty circle equations it may be difficult but in fact it's actually very straightforward you take you pick one of them you subtract it from all the others and you end up with a set of line equations the square terms disappear and those lines are in fact drawn on this diagram as well and then it's very easy to find out where two lines intersect now there's one problem with what I've just told you and the problem is that we're assuming the measurements are perfect in reality no measurements are perfect everything has error and there will be no exact solution to that equation so let me give you a more realistic example so here we have several access points which we've ranged to and I've exaggerated the problem here and you can see some of those circles don't intersect how do you solve that well in fact you do the same thing as you did before you subtract the circles so you get the lines but this time they don't intersect in a point they intersect in a polygon in this case it's a it's a triangle and your phone lies probably somewhere maximum-likelihood in the center of that triangle and then we can apply some college math least squares solution and get a maximum likelihood you can find standard packages which do this on the net you can then also refine this position further by repeating this process particularly as the phone moves and then you can calculate trajectory and use filtering techniques like caramel filters and other things so that's the basic principles okay now like any new technology there are challenges and so we've experienced some of these early on little robots which you can see on the right hand side is used by us to measure the range from the phone which is carrying to an access point and it validates that range against the marks which are on the floor which provide us with ground truth what we find is that sometimes there is a constant range calibration offset may be as much as a half a meter and sometimes also you see multipath effects where the non line of sight path from the access point to the phone is actually received rather than the line of sight path that one can be solved by the vendor using something called antenna diversity but all of these things are algorithms which the vendors are improving and and basically we need to go through a sort of teething process of getting rid of these bugs and Google can help in this process by providing reference platforms and reference applications so the vendors can calibrate their own platforms before you guys even get to use them which will be the ideal situation now I've assumed that you want to start as an early adopter and start using this API but as we move into the relatively near future we expect you to just use the fuse location provider because we're going to be putting the RTG capability into it so at the moment fuse location provider uses GPS when is available cell signal strength Wi-Fi RSSI and also fuses with the onboard sensors no inertial navigation from accelerometer and gyro now we're adding Wi-Fi RTT into that mix and it will increase the accuracy whenever RTT capable access points are available okay so the one other thing to remember is that when you were doing it yourself you had to know the position of the access points in the fused location provider we will know those positions automatically for you we'll crowdsource those positions and so you won't have to worry about that and that would make life a lot more a lot easier for you to write applications okay so now we're going to take it up a notch and we're going to give you a live RTG demo in collaboration some of our colleagues in Gia so what I have over here on the podium is is a phone let's bring it back okay there we go which is running an RTG system in combination with Google mobile maps and what we're doing is we're using a number of access points which are around the room so you I'm only go you saw the blue boxes which are on the slide so these were provided by one of our partners and you can see them around the room towards the back on the side notes a couple in the center over here now the thing to bear in mind is that you would this phone because we're just in a tent would normally receive GPS signals so we've disabled GPS you're only using our cheechee with this phone and and what I'm going to do is I'm going to walk around the aisles and you can see on this car that I've already got a plot of where anger to go so I'm gonna start moving now and it's like going towards the corner of the stage and you should see the blue dot with my little man inside following me and we of course we expect an accuracy of 1 to 2 meters and so I'm walking on the aisle the aisle here is about it's about two meters across there abouts and you can see it's very nicely following within that accuracy I think the the demo environment has been very good to us so far so we're going along and it's a little bit of lag it's going around the back here and we're approaching a turnaround point where I'm going to walk up the aisle and we're rerouting as I come back to make my path a little bit shorter and you can see we're going very nicely it's still well within the one to two meters and if you had GPS shown here as well I mean typically you would be expecting to see five meters but that's accruals outdoors and indoors in a typical building you've only gone it you're only going to have indoor location technology such as this so now I'm approaching the corner of the stage and at this point I'll hand back over to way thank you very much thank you good boy he's gonna tell you about the details of the API hey thanks alright what a great demo so now you must be very good to try round-trip time rendering yourself let me walk you through the are TTIP ad in he to sing how you can add RTG in your own application so as I mentioned RTD measures the round-trip time between two Wi-Fi devices so both your mobile phone and your access points need to support either 2000 MC protocol and as you sell RTD can give you very fine location down to one meter accuracy so your application need to declare access sign location permission and of course post location and Wi-Fi scan need to be enabled on the mobile device okay so how do you know whether your mobile phone support a cutie in P we ID'd a new system feature called feature Wi-Fi RTT so you can simply check whether this is written true i a mobile device our pixel phones running p DV 2 and above where's about RG t so how do you know whether your access points support our duty as normal you will need to do a Wi-Fi scan and get a list of Wi-Fi scan results each right to sue this needs some work once a scan results and check whether this method is 802 11 MC responder return to this will tell you whether the access points support our duty so after you get a list of RTD never deities simple atom to scan request builder to build a scan request RTT is done by Wi-Fi RTG manager which you can simply get by getting the system serviced Wi-Fi RTP rendering service okay now we're ready to start our TV running by sending the RTD request to the RTP manager with the rhaenyra darker back our TV will start surely RTT text no more than hundreds of milliseconds and when the finishes you would get a list of information including the status RTT Mifare the MAC address which IP you have just arranged and also the most importantly the distance between the mobile phone and the access point so here is the list of information you can get from our key tyranny results you can get the distance it also gets distance and dimension which is a sensation from multiple ranges in multiple FTMs and you can also get number of attempted FGM measurements and number of successful moments so the ratio of successful measurements over account environments will give you an idea of how good the Wi-Fi environment is 43 reindeer okay so I mentioned our pencil devices were some hierarchy tea how about access points were beginning to some exit points supporting lmmse protocol in production and we are very excited to let you know Google Wi-Fi will soon support lmmse protocol yeah thank you by the end of this year of the sheriff Wi-Fi we have our TV enabled by default and worldwide were also beginning to sin the deployment of our DDI vs South Korea is actually leading the development of our of our TTIP s and of course this is just the beginning of the long journey we're very eager to sing a larger penetration rate of our TVs in upcoming years with that I'm going to hand over to Frank to talk about one meter GPS thank you okay so let's move to the great outdoors and speak about GPS I'm going to show you some basics of GPS just enough to explain what's new to the satellites and what's new in the phones and how you can exploit these changes to get better location accuracy from GPS when you're outdoors under open sky so GPS works like this it sends a code from the satellite and the code encodes the time at the satellite the net travels to you through space and arrives at you and your GPS receiver will compare that time with the time in its clock the difference between those two tells you how far you are away it's kind of like you have a tape measure where one ends at the satellite and you're holding the school at any moment you can look down and read a number which is the difference in these two times if you move further away you read a bigger number do you move a little bit closer you read a smaller number okay but now the actual GPS tape measure is kind of special first of all it's really long secondly the tick marks occur only every 300 meters because these bits of the code occur at a rate of one microsecond so one microsecond times the speed of light is about 300 meters so this is like a tape measure where instead of having all these inches on here you only have a mark every 300 meters and your GPS receiver essentially interpolates between those marks and there's your five meter accuracy okay but there's more to it than that because how does that code get through space in the first place well it's carried on a carrier wave radio wave which for GPS has a wavelength that is less than twenty centimeters and your GPS receiver can measure where you are on this wave and as long as it keeps tracking it can measure relative motion with great precision and this is because it the receiver will measure the face and as you move that phase will change but now what about getting your absolute location the trouble with the carrier phase ruler if you like is that it's it's kind of like a ruler with very precise markings on it but no numbers at all because one wavelength looks just like the next so your your receiver can tell you the phase of the wave you're on but it doesn't know are you the green dot or the red dot so how do you solve that problem well for that you need to introduce a new concept which is GPS reference stations so these are GPS receivers at fixed sites measuring the same thing at the same time they communicate that data to you with well-known algorithms you can combine this data and over some period of time you can work out where you are relative to the reference station with great precision with this carrier phase precision now you know where the reference station is so now you know where you are with great precision ok so this concept is not new this has been in commercial GPS receivers since the 1980s for surveying hence our a little surveyor there holding the GPS antenna on the stick what is new is the availability of these carrier phase measurements from phones and dual frequency measurements in phones right now all of your smart phones all smart phones everywhere have GPS or GNSS on one frequency band only it's known as l1 but there's a new frequency in town it's called l5 and it's supported by all these GNSS systems gps Galileo bado ku CSS and IRNSS and the availability of a second frequency means that you get much faster convergence to carry a phase accuracy if you're doing this kind of procedure and why well we just went through the ambiguity that you have on a single wave well now look what happens if you introduce a second wave at a different frequency immediately you can disambiguate because you could not have the same phase on that second wave as on both of those wavelengths so you could not be on the red dot if you're at the peak of B of the red wave and so you can disambiguate and get much faster convergence to the very high accuracy that you want all right so what about hardware well in the last few months several companies that produce consumer GPS chips have announced the availability of dual frequency l1 l5 GPS chips both for the automobile market and for the phone market and these chips are now being designed into cars and phones now let's talk about the measurements themselves and the AP is the the phone must support the GNSS measurements API and your app is going to need access fine location permission and location needs to be on so these are the basic requirements so how do you know if a particular phone supports these measurements will add a high level you can just go to a website that we maintain geo / GNSS tools as part of the Android developer site and there's just a table there that lists phones that support the GNSS measurements and also which characteristics they support so it'll tell you which phones support the measurements and which of those support the carrier phase measurements programmatically you do this as follows you need a method on status changed and it will return a ninja integer that tells you the capability of the phone either if the phone just does not support the measurements at all or if they supported but location is off or if they support it and location is on in that last case you're good to go so now let's get into some details of the api's the most relevant methods for what we're talking about here are the following three there's get constellation type which tells you which of the different GNSS constellations a particular satellite belongs to there's get carrier frequency hertz which tells you whether you're on the l1 or the l5 band for a particular signal and then most importantly there's get accumulated Delta range meters which is how far along that wave the receiver has tracked you since it began tracking the signal and then there's something else that I need to explain which is duty cycling so right now when you're navigating with your phone and you see the blue dot moving along for example when maybe when you're navigated here this morning you might think that the GPS is on continuously and it's actually not what's happening in the phone is that GPS will by default be on for a fraction of a second and then off for the remaining fraction of a second and then repeat and this is to save battery so you perceived that the GPS is on all the time because the blue dot will move along continually but actually it's duty cycling internally now for this carrier phase processing you have to continually track the carrier wave because remember the carrier wave is the ruler with no numbers on it so if the GPS was on and your receiver measured your phase and you get the data from the reference station you'd start processing if the GPS then goes off for a fraction of a second well now you've lost where you were it'll start again you'll reacquire you'll be a different phase on the react position you'll start again well you'll never solve the problem back right you need the tape measure to stay out and you need to process and to do that you need to disable duty cycling and you can do that in Android P with a developer option which I'll talk about some more in a minute okay so now add some details of the API what I've shown here on the right is a screen shot of an application that we've put out it's called GNSS logger and this is enables you to log the raw measurements in the phone now the nice thing about this app is it's a reference app the the code is open source and available to you on github so when you build your app please you could make use of our code and when you do build an app that needs role measurement you will need the Android location manager API with the method register GNSS measurements callback and this method requires you to pass it a GNSS measurements event callback shown here you construct this callback and then override the method on status changed and that will give you the integers status that we discussed to tell you if measurements are supported if they are you then override the method on Jia's on GNSS measurements received and this allows you to receive a GNSS measurement event every epoch for example every second and this event gives you the values we've been talking about constellation type carrier frequency and accumulated Delta range now for duty cycling that's a developer option so you access that through the developer page on your phone as you see there on P and this allows you to disable the duty cycling now keep in mind this introduces a trade-off between getting the continuous measurements and battery life there will be an impact on battery life how much well even when GPS is on continually it'll use less than 20% of the power that screen on users so that gives you a feel for the magnitude now this is a developer option precisely because it's a trade-off in in battery life and we're very concerned about maximizing battery life but if you and we together can prove that there's value in this option and people want it then it will be upgraded to a fully supported API in the future so here's a block diagram that shows the basic architecture that we expect if you implement an app for high accuracy than the bottom of the block diagram on the Left you've got the GPS GNSS chip the GNSS measurements come up through the api's we've just described and then your app lives at the top in the application layer you're going to need access to a reference network to get the data that the reference stations are tracking they are publicly available a reference networks I've listed one down the bottom the International GNSS service IGS org and you can get data from them free then you need to process that data in some kind of position library and that does all this carrier phase processing and that too is available as open source code there's another example down there our TK Lib org has an open source package for precise positioning and then you're good to go now I mentioned that dual frequency gives you much faster convergence to the high accuracy but you don't have to wait until the dual frequency phones come out you can start doing this with single frequency phones and here's an example of someone who's already done that this is an app created by the French Space Agency and they they're doing exactly what we show on the block diagram on the left and they're achieving sub-meter accuracy after a few minutes of convergence here's some more external analysis that's been done in a similar way this is from a paper called positioning with Android GNSS this is using one of those chips that I showed you the chip that goes in cell phones that does dual frequency and what's been shown here is the cumulative results over many different starts of the GPS and what you see is that most of the time the accuracy is better than a meter you see that on the vertical axis which is 0 to 1 meters the accuracy gets to better than a meter in less than one minute and then continues to converge as long as the phone continues to track that carrier phase continuously here's a another similar but different paper this is using one of the chips that's meant for cars and so this was tested in a car driving around that track there and what the plot here is showing is the accuracy after the initial convergence while the car was driving so you see with GNSS alone the accuracy is 1 to 2 meters and with this carrier phase processing it's at a couple of decimeters so for you to build this what are you going to need well of course you need the device location to be enabled and your app has to have location permissions so that's going to come from the user you need the basic GNSS measurements that's been available since Android n you also need this continuous carrier phase I've been talking about and that's available in P with the developer option it would be nice to have dual frequency for fast convergence and that's coming soon you need a reference network such as the one I already mentioned there are also commercial reference networks out there and commercially available software to do the same thing but I recommend you start with a free stuff and go from there and then finally there's the app from you so in summary everything we've been showing you here we have indoor and outdoor technology that's been evolving kind of in parallel in each case we have a new technology and Android P gives you something to ax let's talk about indoors again the new technology is Wi-Fi round-trip time and round-trip time enabled access point we give you public API to access these measurements but you need access point infrastructure so this is where some of you can do this this year because if you have a customer who owns or controls a venue they can upgrade their access points sometimes just a firmware upgrade and then you have the infrastructure Android P comes out later this year and you can implement something like what Reuters demoed and have indoor navigation or many other apps for example someone goes in a store where's the milk you can make the world a better place for all of us by saving us from the tyranny of having to ask directions from strangers okay and if you if you're not one of those people who has access to this now in a few years the infrastructure will naturally evolve as access points upgrade to Roger time and this will be available from fused location provider as Roy said now outdoors okay for this carrier phase process they can it's not just outdoors but outdoors with open sky and what do you need dual frequency and continuous carrier phase and we give you the API and the developer option to make use of that you will need reference station access as I mentioned and then applicate well what can you do outdoors with open sky well we already mentioned the traffic example and there's many other examples that readily come to mind where existing GPS accuracy doesn't cut it for example geocaching where people go look for treasures would be nice to have one meter accuracy precision sports monitoring imagine a snowboarder who wants to measure her tracks very precisely after the fact five meters not good enough one meter would be great speaking of sports they're more and more drone apps where you kind of follow me and the drone will fly along and video you well it'd be nice if the videos you and not the person next door to you and so on I'm sure there are hundreds of apps and you're probably thinking of some right now and that's the whole point we want you to do that and you and ask together bend the arc of Technology history closer to the present and I'm really looking forward to next year to see some of you back here and see what you've created and so finally I want to leave you with a couple of pointers when you build location apps please build great location apps you must have user trust please provide the user with transparency and control you're gonna have to ask for location permissions for this explain to them what you're doing how it benefits them when things go wrong make your app recover gracefully if these measurements are unavailable for some moment or something goes wrong you can fall back to fused location provider location so think about that and finally respect the battery life trade-offs that we've discussed so I must remind you to fill out your surveys please at that site and as I mentioned will be available outside the door here for any questions or so from all three of us thank you [Music]

As found on YouTube

Intro to the Google Assistant: Build Your First Action (Google I/O’19)

if you're new to developing for the Google assistant you have come to the right talk and if you're an experienced assistant developer don't worry we're gonna tell you what's new our mission for the Google assistant is to be the best way to get things done that is an ambitious mission I think it sort of rhymes with organize the world's information and make it useful and accessible and just like Google searches mission from 20 years ago our mission today for the assistant requires a large and vibrant ecosystem of developers and that's where all of you come in so whether you're joining us here the amphitheater shoreline watching on the livestream or on YouTube later this talk is going to tell you how you can make your experiences shine on the Google assistant if you're a Content owner this talk is going to tell you about schema.org markup and templates and how you can make your content look great on the assistant and if you're an Android developer wait I hear there's a few Android developers here today where are the Android developers where are you yes thank you if you're an Android developer this talk is going to tell you how you can use app actions to voice enable your Android apps and if you're an innovator in this new generation of conversational computing this talk is going to cover interactive canvas and conversational actions how you can use HTML CSS and JavaScript to build rich immersive actions for the assistant and if you're among the few the proud the hardware developers at i/o any hardware developers a couple this talk is going to tell you about the new innovations in the smart home SDK but before we do any of that Naomi is gonna tell us a little bit about why now is the right time for you to invest in the Google Assistant we're gonna start by going back in time I want you to think about when you first use the computer for some of you it was the 80s maybe you played a game or store to receipt or if it was the mid-80s you even used a word processor for others it may have been the 90s games are a little bit better and you navigated via the mouse instead of the command line ten years after computers entered our home cell phones entered many of our hands and by the mid to late 90s communication was more portable than it had ever been before but it was still very early you remember the days when text messaging was a game of back and forth between the inbox and the sent folders yes we've come a long way now another 10 years later in about 2007 the first smartphones entered the market and then mobile computing exploded so you may notice a trend here about every 10 years or so there's a fundamental shift in the way that we are computing from desktop to mobile and now to conversation so what does these chips mean for all of you as developers well it means that you have a lot to think about a lot to build on and a whole lot to build with and this is because each new wave is additive to the one that came before we're still clicking we're still typing we're still tapping and yes now we're also talking we're looking for more assistance in our daily lives and we're turning to the devices around us to get things done now Google's approach to this era of conversational computing is the Google assistant it's a conversation between the user and Google that helps them get things done in addition to the Google assistant there are also assistant enabled devices like Google Home Android phones and more and finally there's actions on Google which is our third party platform this enables developers to build their own experience on the assistant it's an entirely new way to engage with your users as they're using conversation to get their things done and it was announced on this very stage just three years ago now in just a few short years we've seen an incredible evolution in terms of how users are talking to the assistant and this presents opportunities for developers across the globe now think back to those first use cases on conversational platforms they were very very simple and straightforward they were limited to things like turn on the music turn on the lights turn off the music turn off the lights again simple straightforward commands that fulfilled users very low and limited expect a shion's but there have been three incredible shifts in querying that have occurred over the last couple of years first we're seeing that users are having longer conversations in fact query strings on the assistant are about 20% longer than similar queries on search second they're more conversational they're 200 times more conversational than search so queries are going from whether nine four zero four three two something like do I need an umbrella today like you might ask a friend a family member or even a real-life assistant and third queries are action-oriented it's 40 times more action-oriented than search so users aren't just searching for information but they're actually looking to get things done they're finding that restaurant for Friday night and they're booking that dinner reservation and the evolution of the query is due to a couple things happening simultaneously so first we're seeing that technology is improving natural language processing and understanding improvements have actually decreased the word error rate which is a key metric for speech recognition it's now better than what humans can understand simultaneously the number of assistant ready devices has soared so it's turned this new way of computing into an ambient one it's always there when we need it no matter what environment we're in or what device were on it's magical but it poses a really new challenge for all of us which is how do we reach the right user with the right experience in the right moment all at the same time so sync – a pretty typical day we'll talk about some of the touch points where the assistant might be helpful so first you wake up good start now if you're anything like me you really would love to keep your eyes shut for that extra 20 seconds but you also need to kick start your day and find out where you need to be and when well the assistant can help with that now you're waiting for the subway you're in a crowded loud station you have a couple of moments of idle time before that train comes to maybe pre-order your cup of coffee or by your friend that birthday gift you've been meaning to send them you're the assistant on your mobile or your watch can help in those moments as well and finally you're sitting on the couch the end of a long day your laptop or your mobile phone are probably not too far away but neither is your Google home it's there to help you so across these moments and more Google is handling the orchestration that's required to deliver that personalized experience for the user with the context appropriate content so you as the developer don't have to worry about which experience which device which manner you can just leave that to us so what does this mean for the developer well you have more ways than ever to be there for your user you can reach users across the globe in over 19 languages across 80 countries on over 1 billion divisive today with over 1 million actions but more than that it's actually easier than it's ever been and this is something we're all really excited about I know we're all balancing far too many projects for the numbers of hours in a day so today we're going to talk about how you can get started if you have an hour a week or even an entire quarter to build for the assistant we'll talk about how to use existing ecosystems as well as how to build net new for the assistant and we'll focus on four major pillars so first we'll talk about how to use your existing content this leverages what you're already doing in search so web developers we're gonna be looking at you for this one second we'll talk about how to extend your existing Android investments leverage the continued investments you're making in your mobile apps and app developers I heard you before so you're gonna want to pay attention to that section third we'll talk about how to build net new for the assistant so if you're an innovator in the conversational space we will share how to get started and finally hardware developers saw a few hands go up before where if you're looking to control your existing device cloud our smart home section will appeal to all of you within each section we'll talk about what it is and how to get started but before we do Dan is gonna tee up a very sweet example that we will use throughout the rest of our presentation so a single unifying example that shows all the different ways you can use Google assistant now this gets me thinking about two things one I love s'mores I have a sweet tooth and I'm also an engineer here in Silicon Valley home of tech startups of all kinds and the tech hub of the world so how can I combine my love of s'mores and my love of Technology talking it over with my colleagues Brad and Naomi we thought of the idea of using a fictional example company that you as a developer can show all of the different ways that assistant can help you and your company with things like building a global brand through Google assistant increasing your global sales customer growth and acquisition and even things like user re-engagement like the very important metric of daily active users and so the first pillar that we have is how you can leverage your existing content with Google assistant so like many of your companies s'more s'mores has a lot of existing content that's ready for the assistant they have a website they have a podcast and of course they have recipes so that we can all understand how to make that perfect s'more at our next bonfire now also just like you they spend a great deal of time optimizing their site for search so we're gonna talk about how they and of course how you can extend existing efforts and optimizations to the Google assistant now Google's presented ways to optimize your content for search since the 90s we work hard to understand the content of a page but we also take explicit cues from developers who share details about their site via structured data structured data is a standardized format for providing information about a webpage and classifying that page content for example on a recipe page you can disambiguate the ingredients from the cooking time the temperature the calories and so on and because of this markup we can provide users with richer content on the search results page answers to questions and a whole lot more and this brings Google search beyond just 10 blue links and over the last year we've been hard at work to enhance the Google search experience and enable developers to extend their content from search to other Google properties like the assistant so for sites with content in popular areas like news podcasts and recipes we have structured data markup to make your content available in richer ways on search same optimizations that you make for search will also help your content people with discoverable and accessible on the assistant and it's just standard RSS you've seen this before and that's the approach we've always taken we're using industry standards and insuring those optimizations are ready for search and the assistant to and I'm so excited now to announce two brand new programs that we're adding how-to guides and fa Q's so the additional optimizations that you make for search will soon yields a richer experience and an automatic extension to the assistant let's dive into each so how-to guides enable developers to mark up their how-to content and make it discoverable to users on both search and the assistant what displays is then a step-by-step guide to the user on anything from how to change a flat tire to of course how to make that perfect some more so on the left here you can see a nice image based preview of the how-to content in some more some more site it allows the user to engage with your brand further upstream in their journey and it differentiates your results on the search results page and if you don't have images don't worry we have a text-based version of this feature as well now on the right you can see the full guided experience on a home hub device again all powered by the same markup on some more some more site now the best thing about creating a step-by-step guide is that you actually don't have to be technical to do so now I know we're at i/o and i/o is the Developers Conference but if you have one hour to dip your toes in the assistant pool and you don't have a developer who can devote the time to adding the markup don't worry we have ways for you to get your content onto the assistant even as simply as using a spreadsheet so now you can combine your existing YouTube how-to videos and a simple spreadsheet and actions console to get a guided how to experience across many assistent enabled devices so s'mores s'mores calm now has two ways for how they can get their step-by-step guides on how to make that perfect some more onto the assistance if they have a developer with some extra time they can add the markup or they can use a simple spreadsheet to extend their existing YouTube content now we're gonna switch gears a little why to think about how times you turn to search for answers to questions maybe some of you are even doing it right now that's okay maybe you're trying to find out the return policy to your favorite store or if there's a delivery fee for your favorite restaurant blackout dates for travel the list truly goes on and on while our epic use markup enables a rich answer experience on search giving users answers directly from your customer service page so the same optimization will then enable queries on the assistant to be answered by the mark-up you already did and it's so easy to implement so when a user queries something like what's more smores delivery fee on the assistant a Google will soon be able to render the answer from that same markup on your customer service page and here are some developers that have already gotten started with FAQ s and how-to guides and we'd love to have you join us tomorrow at 10:30 in the morning to learn more about how to enhance your search and assistance presence with structured data of course the talk will also be live streamed or you can catch it later on YouTube so as you've seen there are three main ways that s'mores s'mores calm and you can leverage existing content first you can ensure the hygiene of your structured data markup for podcasts news or recipe content you can add the new epic use markup to your customer service site or you can use a new template to bring your content to the Google assistant we're so excited about the ways that we're making it even easier for developers to extend existing content from search to the assistance but we're also making it even easier for companies to engage their existing ecosystems like Android so let's talk more about app actions all right thank you how about that Sun huh you enjoying the Sun in shoreline I can't see anything without these so I'm gonna go with this so we're my android developers again Android developers yes just like many of you the s'mores s'mores company has a popular Android app they want to make it as easy to order s'mores as it is to enjoy them but just like many of you they face the high cost of driving app installs coupled with the reality that users are you using fewer and fewer apps each year the sea of icons found on many users phones might be a contributing factor it's hard for users to remember your icon much less find it what we need is a new way for users to find and launch your android apps one that's focused more on what users are trying to do rather than what icon to click last year at i/o we gave a sneak peek at app actions a simple way for Android developers to connect their apps with the helpfulness of the assistant and with the Google assistant on nearly 1 billion Android phones app actions is a great way for you to reconnect with your users app actions uses Google's natural language understanding technology so it's easy for users to naturally invoke your application and finally app actions doesn't just launch your app it launches deeply into your app so we fast forward users right to the good parts now to help you understand this concept let's walk through an example of course using our s'more s'mores app let's first take a look at how this looks the traditional way so first of course I find the s'more s'mores app in the sea of icons does anybody see it next I select the cracker okay that that makes sense and then I have to choose a marshmallow all right and then I get to pick the chocolate and the course a toast level okay I've got to say how many I want that's important too and then finally I can review my order and confirm now that's a long intent to fulfillment chain it's a long way from I first had the desire for a warm delicious s'mores before I got it successfully ordered and that means there's opportunities for drop-off all along the way now let's take a look at what this looks like once s'more s'mores has enabled app actions first you'll notice I get to naturally invoke the app with just my voice I can say order one milk chocolate from s'more s'mores and then immediately we jump right to the good part of this application confirming that order notice we got all the parameters correct and then we just confirm and we're done we're ordered it's a short path from when I had the intent for the warm delicious more before I got it ordered but of course we didn't build app actions just for stability s'mores we had a few partners that have already started to look at app actions so for example I can say to the assistant order a maple glazed donut from Dunkin Donuts of course I might need to work that off so I can say start a run on Nike Run Club and I might want to settle that bet from last night by saying send $50 to naomi on paypal so let's take a look though at what enables this technology what's going on under the covers here foundationally at Google we connect users that express some intent with third parties that can fulfill it and app actions is the mechanism that you as app developers can use to indicate what your Android app can do each app action each built in and it represents an atomic thing a user could want to do including all possible arguments for that so you just need to implement that built-in intent and I handle the arguments that we pass to you the cool thing about built-in intents is that they model all the different ways users might express an intent for example these are the ways users could say start an exercise notice as an app developer you don't need to handle all of this complexity with this different kinds of grammar we handle all of that for you you just implement the built in intent so speaking of which let's take a look at how it looks for you as a developer to implement that well the first thing you'll do is open up Android studio and add an actions xml file you notice on that second line there it's order menu item that is the name of the built-in intent that we have implemented for our s'more s'mores app and then in that fulfillment line you'll notice a custom schema URL so you could of course use an HTTP URL as well this just tells us where in the application we should fast forward into and then at the and then you'll notice we map the arguments there so the menu item name is the argument name from the built in intent and then notice our URL is expecting item name and then finally at the bottom there we're giving some inventory what are the kinds of things users might say for this application and just for brevity I put the usual now we just need to handle that in our on create function so very simply we parse that item name parameter out of the URL we check to see if it's that identifier and if so we just pre populate the UI with exactly what we want this is a very simple integration that you can get done very quickly on the assistant so the good news is that you can build and test with app actions starting today we're releasing built-in intents in these four categories in finance food ordering ride-sharing and fitness so if your app is in one of those categories you can build and test right now and of course the team is already working on the next set of intense so if you have ideas or thoughts on what intent would be great we love it if you gave us feedback at this URL but there's one more thing app actions and built-in intents it also enables slices slices is a way that you as an Android developer can create a declarative version of part of your application and embed it into Google surfaces like the Google assistant so in this case we're implementing the track order built-in intent and then you can see in line there that's our Android slice showing up right in line in the Google assistant making it quick and easy for users to get that information and then launch into your app if they need more advanced functionality so what did we see here you can enable users to invoke your app with just their voice with app actions there's a simple integration model all you need to do is map the intents in your app to us common set of built-in intents and then the good news is you can build and test starting today with more intense coming soon so we've seen how you can leverage your existing content with Google Assistant we've seen how you can integrate your Android applications with Google assistant now I want to talk about conversation specifically conversation actions that is how you can build new custom experiences for the Google assistant so why our conversation actions so important well for one it's a way that you can natively control the devices capabilities so if the device has a screen showing image if the device supports touch show a suggestion chip that they can tap it's a way that you can increase your brand awareness through things like custom dialogue and agent personas you can grow your user re-engagement through well-crafted conversation design and things like action links and furthermore you can drive habits and interactions with features like routines daily updates and push notifications now what exactly our conversation actions it's the idea that you as a developer have full control over the dialogue the back-and-forth of what's said this is distinctly different than that of app actions or content actions or even smart home actions where you have some type of fixed markup or maybe you're using a built in intent something that how defines already what your the material that you're trying to access Google assistant takes that fixed markup and ingests it applies its own natural language understanding and matches what the user says automatically to the access of that material with conversation actions that's flipped you as a developer define custom intents you define the types of phrases the that a user might say to match that custom intent you even define that information that you want extracted out of what they say for those same intents so with conversation actions we need some type of tool that can help us do this and that is dialogue flow out-of-the-box it provides two key elements the concept of user intent or what the user actually wants and the concept of entity abstraction the way that you glean information out of what they say let's dive in a little bit with a small example so if we take some oars I would like a large s'more with dark chocolate and I want it to go dialogue flow can take this phrase as a whole and match it to the user intent that they want to purchase a snack of some kind now you see here a few words highlighted in the sentence large dialogue flow can understand that they want a large snack s'more the type of snack dark chocolate the topping of that snack and they want it to go rather than for a delivery or for their so when we took it take a look at this at a sequence of dialogue and expand it a little bit more the user might say something like hey gee talk to s'more s'mores s'mores s'mores in this case is the invocation name of your action Google assistant takes that audio transcribes it into text applies its natural language understanding and invokes your conversation action from that point forward it's between you as a developer and dialogue flow that's controlling the responses back to the user and so let's take a look at a live demo here I have dialog flow and a few intents that I've already defined I have a small shop where you can order a snap you can order whatever you last ordered a gift card and so let's take a deeper look into ordering a snack when I look at this I have a few things I have my contexts I have my training phrases that I've already supplied these are the phrases that I think as a developer the user might say that app that matches the intent of them wanting to purchase a snack of some kind if I scroll down I can see the types of parameters and the relating entities of those parameters specifically things like delivery pickup the type of snack the size of the snack toppings etc now if I scroll down further you'll see the responses that I've created as well that reference the entities that I've also defined if I scroll down even further you'll see the fulfillment if I wanted to have a custom fulfillment I can have a standard webhook call for this type of intent now let's look at an example here if I say one large s'more with milk chocolate you'll notice instantly without additional input from me dialog flow has highlighted several key elements within this phrase large it knows that's the size of the snack that I want s'more the type of snack chocolate type of milk so there you go that's that's pretty powerful stuff now let's take a look at it in the phrase of a full conversation if I say I would like to order a large s'more with dark chocolate instantly it gets the information it has the various contexts it matched it to the intent of ordering of snack and we scroll down it also still has the various entities and parameters that it's extracted now the default response here is that I've defined a required parameter of delivery or pickup and so it's asking me will it be for delivery or pickup I respond delivery and there you go it understands you've ordered a large dark chocolate s'more and you want it for delivery there you go so this is powerful stuff for Google assistant now let's go back to the slides here what we have is a way to build conversation now Google assistant supports a huge array of devices and surfaces and we want to be able to scale across them all different devices support different capabilities a smart speaker is voice only a car the voice forward it has a screen but you still want to have voice control to intermodal like your cell phone to maybe your SmartWatch which is screen only we need some type of feature that can fully utilize the screen and in the case of s'more s'mores they want to be able to build a rich game something that's voice first custom full-screen visuals they want to build a turn-based battle system that's multiple surface and supports all these different kinds so today I'm happy to announce the brand new API called interactive canvas this is an API that enables pixel level control of rendering any HTML any CSS and JavaScript let me reiterate that is a full webview running on the device if it supports full screen visuals animations even video playback now it wouldn't be Google i/o without another live demo let's take a look here what I have is a hub and for this demo I'm gonna need some some audience interaction here I'm gonna be playing a trivia game and it's gonna be a really hard question so I need you guys all to scream out as loud as you can the answer to this question hey Google Play HQ University hears HQ University yes that's me the host who's khalifa host malone the trap Trebek and I'm here to welcome you to HQ niversity you've been accepting to this elite program to help smarten up and sharpen up your trivia skills to help you win HQ the rules are very simple my assistant Alfredo is gonna ask you a series of questions that just like HQ start easy and get harder you will respond with the answer you think is correct and the H universe will reward you let's get down to the nitty-gritty let's get this show on the road baby Alfredo pinin with Camaro numero uno ready question 1 if you put on a hoodie what type of clothing are you wearing your choices are cloak sweatshirt or cape so what's the answer sweatshirt yeah baby you did it we did it awesome next so that is interactive canvas a way that you can build fullscreen visuals and custom animations so how does something like this work well first off we have to start with a regular conversation action where the user says something to the device that in turn goes to the actions on Google platform and turn to dialogue flow and finally your custom fulfillment your fulfillment supplies a certain type of response in this case it needs to supply a rich response when we add the web application the canvas it supplies a immersive response which tells the Google home to load the web app your your web application in turn has a specific JavaScript library called the interactive canvas API let's look at some code for this when we look at this this is the custom fulfillment you can see here I have the default welcome intent that's supplying a new immersive response that supplies the URL of the web application and the initial state of that web application what does it look like on the web application side well when we see this there's two main elements that you need to include there's the CSS stylesheet this supplies the specific padding for the header on the devices and things like that and then the actual JavaScript library this library itself manages the state of your web application with the states of the conversation so that you can control both in unison so some key takeaways around conversation is that dialogue flow is the tool for developers to where you can build custom conversations you control the back-and-forth of your dialogue and – we've announced the interactive canvas API the pixel level control over the display for games where you can run HTML any CSS and any JavaScript now I want to switch it up a little bit talk about smart home the ability for you to control any hardware with your voice now traditionally smart home has been all about cloud to cloud communication so when I turn on my Phillips hue light bulbs what's actually happening is that Google assistant takes in the audio transcribes it into text applies natural language understanding and sends a specific response to philips hughes servers philips you in turn controls the light bulb and so now i'm glad to announce a new api the local home SDK this provides new local control over your devices with post assistants latency of under 200 milliseconds supports local discovery protocols like UDP broadcast mdns UPnP for actually controlling devices UDP TCP and HTTPS now with smart home there's device types and device traits it supports all of them out of the box with the exception of two-factor authentication now my favorite part is that it's come as you are meaning you don't need to change the embedded code on your device for the messages what's actually happening is that you develop a JavaScript application that runs on the home device that's pretty awesome so let's take a look at how this works when the user says something like hey G turn on the lights again that audio sent up to Google assistant transcribes it into text applies natural language understanding Google assistant creates a execution intent and sends this intent this structured JSON response down to the Google home where the home is running your JavaScript application your JavaScript application in turn understands the intent of trying to turn it on with the exact device ID things like that and constructs the specific message that your device support in turn turning on the light so I want to show more traits and more types today we're always adding more device types that we support with goop with the smart home api's and today we're releasing even more with things like open closed starts off with zones lock and lock even devices and pipes like your door boiler garage door now again we're adding these all the time and today we're releasing a huge number more so I want to show now how s'more s'mores can use smart home so what I have here is a toaster oven some of you might have already seen this up here and wondering what it's used for so I have a toaster oven here with inside of it some s'mores that I want to eat and I want to toast perfectly I also have a Google a ìwhy vision kit which inside of that is a Raspberry Pi zero and I'm using this to control the power to this toaster oven and so let's take a look hey Google turn on my s'mores toaster okay turning s'mores toaster on awesome so there you have it a smart home device that's being controlled via voice with Google assistant and so let's recap some key takeaways one we announced the local home SDK where you can control real world devices using local Wi-Fi with your voice and second we've announced a huge number of new traits and device types these these are available today that you can use in your own devices and so what to do next all right we have covered a lot today but we are so excited to share with you all of the ways you can get started building your first action so to quickly recap you can use your web content and leverage what you're already doing in search you can use app actions and leverage the Android ecosystem you're already participating in you can build a custom experience for the assistant by building a conversation action or you can extend your hardware and build smart home actions to control the devices around your home but this was just the beginning there are 12 more sessions this week that will dive into topics we only had a chance to introduce in our time together today so these additional talks will be geared toward Android app developers web developers hardware developers or anyone who just wants to learn how to build with conversation or some insights around this new ecosystem so please check them out live watch them on the live stream or of course tune in later on YouTube now for those of you here with us this week we have a sandbox out back office hours and a code lab now I think I heard our toasters ready so it's time for us to go enjoy some mores but visit our developers site talk to us on Twitter and we can't wait to see the rest of i/o thank you so much for joining us today [Applause] [Music] you [Music]

As found on YouTube