NGC11 - Pardon the Noise & Excuse the Dust

Hi, I'm Rachel Blum.

Also known as

+Rachel Blum

@ groby

Chrome Frontend Engineer

Previously: Blizzard, EA, Pandemic Studios, Midway Games...

Platform Zoo

Home Computers
Unix boxes
PC's
Arcade
Console
MMO

Javascript performance

Demo Source:savagelook.com

Graphics

This page intentionally left blank

WebGL for Fast Graphics

Demo Source: helloracer.com

WebSockets

Low latency, persistent, full duplex
UTF8
Simple to use

var socket = new WebSocket("ws://server.com");

socket.onopen = function(event) { 
socket.send("Hello Server"); }

socket.onmessage = function(event) { 
alert("Server says: " + event.data); }

Binary in Chrome 15 (crbug.com/93652)
Unreliable being discussed (crbug.com/91596)

Plenty of options...

WebStorage
IndexedDB
File API
Application Cache

File System

Large file storage
Intelligently cache your resources
Store large binary blobs
Supported only on Chrome

Application Cache

Caches your entire app locally!
Why?
1. Your assets (should) stay fairly consistent
2. No control over browser caching
3. App is 'zippier' when loading assets
4. Javascript API to App Cache

Cache Manifest File

    <html manifest="example.appcache">... </html>

Serve with mime-type: text/cache-manifest

CACHE MANIFEST
# 2010-11-17-v0.0.1

# Explicitly cached entries
CACHE:
index.html
stylesheet.css
images/logo.png
http://img.example.com/logo2.png
scripts/main.js

# static.html will be served if the user is offline
FALLBACK:
/ /static.html

# Resources that require the user to be online.
NETWORK:
*
# login.php, http://api.twitter.com, etc.

More App Cache debugging

Javascript Console

Page Visibility API

Determine if your app is visible or not:

document.addEventListener('visibilitychange', function(e) {
  console.log('hidden:' + document.hidden,
              'state:' + document.visibilityState)
}, false);

Demo

Available since Chrome 13

Visibility API

Fullscreen content

<video width="300" src="movie.webm" controls></video>
<button onclick="enterFullscreen()">Get Huge!</button>

function enterFullscreen() {
  var elem = document.querySelector('body');
  elem.onwebkitfullscreenchange = function(e) {
    console.log("Entered fullscreen!");
    elem.onwebkitfullscreenchange = onFullscreenExit;
  };
  elem.webkitRequestFullScreen();
}

Demo

Fullscreen API

Control of the entire document and elements:

document.webkitIsFullScreen ( bool )
document.webkitCurrentFullScreenElement ( DOMElement )
document.webkitFullScreenKeyboardInputAllowed ( bool )
document.webkitCancelFullScreen();

Element.webkitRequestFullScreen();
//Element.webkitRequestFullScreenWithKeys();

Enabled in Chrome 16

Spec: https://wiki.mozilla.org/Gecko:FullScreenAPI

Fullscreen API

Control of media elements:

MediaElement.webkitSupportsFullscreen ( bool )
MediaElement.webkitDisplayingFullscreen ( bool )
MediaElement.webkitEnterFullScreen()
MediaElement.webkitExitFullScreen()

New pseudo-classes, and attributes:

:-webkit-full-screen
:-webkit-full-screen-document
:-webkit-full-screen-root-with-target
allowFullScreen attribute added for <iframe>

Know when you're connected

Status:

if (navigator.onLine) {
  console.log('ONLINE!');
} else {
  console.log('Connection flaky');
}

Online/offline events:

window.addEventListener('online', function(e) {
  // Re-sync data with server.
}, false);

window.addEventListener('offline', function(e) {
  // Queue up events for server.
}, false);

Available since Chrome 14

Online Overview

Smarter animations

Common technique for JS animations:

window.setTimeout(function() {
  // move element. Call this again.
}, 1000 / 60); // 60fps.

Preferred technique:

window.requestAnimationFrame = window.webkitRequestAnimationFrame ||
    window.mozRequestAnimationFrame || window.msRequestAnimationFrame;

var reqId_ = null;
(function callback(time) { // time is the Unix time.
  // move element.
  reqId_ = window.requestAnimationFrame(callback, opt_elem /* bounding elem */);
})();

Now possible to synchronize JS-based animations with CSS/SVG animations
Chrome caps sampling at 60fps (no big, your monitor is ~60Hz)
Callback won't be invoked if tab/element is not visible ( display:none )
Tells the browser your intentions are to animate something

In the olden days...

    <bgsound="xfiles.mid" controls="console" loop="5">

    
    <embed src="hamsterdance.wav" autostart="true"
            loop="true" hidden="true">
    <noembed>
      <bgsound src="hamsterdance.wav">
    </noembed>

.... and of course embedded flash

But... <audio>!

Yes, but:

Codec support issues
Latency issues
Sample-accurate scheduling
Glitches
Real-time processing and analysis
Low-level audio manipulation
Effects, and filters, oh my!

Scheduled playback

var ctx = new window.webkitAudioContext();

function playSound(arrayBuffer) { // Obtain arrayBuffer from XHR2.
  ctx.decodeAudioData(arrayBuffer, function(buffer) {
    var src = ctx.createBufferSource();
    src.buffer = buffer;
    src.loop = false;
    src.connect(ctx.destination);
    src.noteOn(0); // Play immediately.
  }, function(e) {
    console.log(e);
  });
}

Shoot:

Streaming Source

Long sounds can be streamed, via an <audio> node.

    var context = new webkitAudioContext();

    function setupStreamingAudio() {
      var audioTag = document.getElementById('audioTagID');
      audioTag.audioSource.connect(context.destination);
    }

Javascript Source Node

Generate wave forms in JS.

  var ctx = new webkitAudioContext();

  src = ctx.createJavaScriptNode(1024, 1, 1);
  src.onaudioprocess = function(e) { computeWave(e); }
  src.connect(ctx.destination);

  var sample_count = 0;
  computeWave = function (e) {
    var data = e.outputBuffer.getChannelData(0);
    for (var i = 0; i < data.length; ++i) {
      data[i] = Math.sin(sample_count++);
    }
  }

Sample-accurate scheduling

  // Play the bass (kick) drum on beats 1, 5
  playSound(kick, time);
  playSound(kick, time + 4 * eighthNoteTime);

  // Play the snare drum on beats 3, 7
  playSound(snare, time + 2 * eighthNoteTime);
  playSound(snare, time + 6 * eighthNoteTime);

  // Play the hi-hat every eighthh note.
  for (var i = 0; i < 8; ++i) {
    playSound(hihat, time + i * eighthNoteTime);
  }

function playSound(buffer, time) {
  var source = context.createBufferSource();
  source.buffer = buffer;
  source.connect(context.destination);
  source.noteOn(time);
}

Gain Control - Cross Fade

  var source = context.createBufferSource();
  var gainNode = context.createGainNode();
  source.buffer = buffer;
  source.connect(gainNode);
  gainNode.connect(context.destination);

Drums Organ

Filter Effects

BiquadAudioNode filter

low-pass
high-pass
band-pass
low shelf
high-shelf
peaking/parametric EQ
notch
allpass

Filter Demo

DelayNode

delay time can be modulated for chorus effects
feedback effects are possible (ping-pong delay, etc.)

DynamicCompressorNode

for overall control and sweetening of the mix
pumps up overall volume without peaks getting too loud
good for both music and games (controls many sounds playing at the same time)
very common and important audio effect

AudioPannerNode

positions a sound in space (left, right, up, down, forward, backward, closer, farther, etc.)
3D environments

ConvolverNode

Provides room ambience and other special effects

small / large room
cathedral
concert hall
cave
tunnel
hallway
forest
extreme filters
backward effects
weird comb filters
etc. etc.

Convolution Demo

Node Graph - Doppler Effect

     // Connect dry mix
     source.connect(panner);
     panner.connect(dryGainNode);
     dryGainNode.connect(masterGainNode);
     
     // Connect wet mix
     panner.connect(convolver);
     convolver.connect(wetGainNode);
     wetGainNode.connect(masterGainNode);
     
     // Connect master gain
     masterGainNode.connect(context.destination);

Doppler Effect Demo

Behind the scenes

http://www.flickr.com/photos/grrphoto/152833988/

2008

early prototype

2009

6/2009 - API design
9/2009 - WebKit & JS bindings
12/2009 - continued design work, first presentation to Apple

2010

1/2010 initial Web Audio integration
5/2010 - Audio incubator, Mozilla Audio Data
9/2010 - redesign & refactor, merge approval
late 2010 - land merge in WebKit

Early 2011

W3C Audio Group
Chrome 10 - OSX Web Audio behind a flag
lots of porting work
Chrome 12 - Windows & Linux behind a flag
Dinahmoe ships Plink & ToneCraft

late 2011

9/2011 - Chrome 14 - Web Audio enabled by default
10/2011 - Tomas Senart creates AudioJedit
10/2011 - Apple enables Web Audio in nightlies

Participation

In rough order of time spent:

Participate in W3C groups
Write your own code
File bugs (crbugs.com)
Talk to Developer Relations!

Mouse Lock

Web Events WG

Draft W3C Spec

crbug.com/72754 (filed 2/11/11)

    element.lockMouse();
    element.unlockMouse();
    element.setMousePosition(x,y);

Pepper API sample: ppapi/examples/mouse_lock

Synthetic Events

document.addEventListener("click", function (e) {
  if (e._isSynthetic)
    return;
  // send a synthetic click
  var ee = document.createEvent("MouseEvents");
  ee._isSynthetic = true;
  x = myCursor.x;
  y = myCursor.y;
  ee.initMouseEvent("click", true, true, null, 1,
                    x + e.screenX - e.clientX,
                    y + e.screenY - e.clientY,
                    x,
                    y);
  var target = document.elementFromPoint(x, y)
  if (target)
    target.dispatchEvent(ee)
});

Why so long?

Among other things:

Time to first draft: June 2011
W3C spec & discussions
Coordinate with WebKit
Security implications
Permissions UI

Joystick API

W3C Draft Spec

crbug.com/79050

Supported in Mozilla nightlies

function runAnimation() {
    window.requestAnimationFrame(runAnimation);

    for (var i = 0; i < navigator.gamepads.length; ++i) {
        var pad = navigator.gamepads[i];
        // todo; simple demo of displaying pad.axes and pad.buttons
    }
}

window.requestAnimationFrame(runAnimation);

What is WebRTC?

Plugin-free high quality real-time voice/video communication in the browser

Browser engine:
- Echo cancellation, noise reduction, automatic gain control
- Network jitter management
Web APIs:
- Audio/video capture and playback
- Device management
- Builds on the PeerConnection API

WebRTC

W3C Draft Spec
webrtc.org
code.webrtc.org - open-source, royalty free BSD license
Supported by Google, Mozilla and Opera.

What's still missing?

Orientation Lock
Hardware Feature Detect

CPU performance
GPU performance
RAM size
Display DPI
...

High Performance Timers
App Bug Reports (crbug.com/83642)

Pardon the noise & excuse the dust

Hi, I'm Rachel Blum.

Platform Zoo

Terra Firma

Javascript performance

Graphics

WebGL for Fast Graphics

WebSockets

Asset Storage

Plenty of options...

File System

Application Cache

Cache Manifest File

Debugging App Cache

More App Cache debugging

That's a lot of work...

Shiny and New

Page Visibility API

Fullscreen content

Fullscreen API

Fullscreen API

Know when you're connected

Smarter animations

Web Audio API

In the olden days...

But... <audio>!

Scheduled playback

Streaming Source

Javascript Source Node

Sample-accurate scheduling

Gain Control - Cross Fade

Filter Effects

DelayNode

DynamicCompressorNode

AudioPannerNode

ConvolverNode

Node Graph - Doppler Effect

Behind the scenes

2008

2009

2010

Early 2011

late 2011

What can I do?

Participation

In progress...

Mouse Lock

Mouse Lock

Synthetic Events

Why so long?

Joystick API

Joystick API

WebRTC

What is WebRTC?

Architecture

WebRTC

Terra Incognita

What's still missing?

And so much more...

The future is built today

... Let's make it good!