Motion Detection with JavaScript

I recently gave a talk at RevolutionConf about writing a motion detecting web app with JavaScript. This is basically that talk, but in blog form. Live demos and all the source code are available at the end of this article.

The Premise

I wanted to see what my pets do when I’m away. But I didn’t want a continuous live stream, since that would be pretty boring (they sleep a lot). Instead, I decided to build something that would watch for motion, take a snapshot of anything that happens, then upload the snapshot to Twitter for remote viewing.

Basic flow of motion detection web app

Just for kicks, I decided to do this as a web app, all in JavaScript.

Accessing the Webcam

The first step is to access the webcam. This is actually really easy with WebRTC, which is natively supported by modern browsers… unless your browser is Safari. Boo. Another wrinkle is that WebRTC has some syntax differences from browser to browser, but the adapter.js shim will fix that for you.

Anyway, to grab a stream from a webcam, start with a <video> element in your HTML.

Then add a bit of JavaScript.

This will attempt to grab a 640px by 480px stream from an attached webcam. Users will be prompted to permit access, but assuming they do, the stream will be displayed in the <video> element on the page. Check out this quick demo to see it in action.

Grabbing Still Frames

We need to capture still frames from the streaming video so that we can do motion detection (more on this later) and potentially upload them as images to Twitter. I settled on an interval of 100ms between captures, which is 10 frames per second.

We start by grabbing the <video> element with the stream on it from the page. Then a <canvas> element is created in memory, though you could also have it on the page for display.

A simple setInterval() allows us to capture a new still frame every 100ms. Each capture is drawn onto the <canvas> element by calling drawImage() and passing in the <video> element. It’s smart enough to just draw whatever is visible on the <video> element at that very moment.

Once something is drawn on a <canvas>, you can save it off as an encoded image string. We can use this string later when uploading to Twitter.

Of course, we don’t want to save and upload every still frame we capture, just the ones with motion. Fortunately, <canvas> gives us the tools to detect motion.

Diffing

So, what exactly is “motion”? A video (such as a stream from a webcam) is just a bunch of still images shown in rapid succession. Movement is perceived when there are changes between these images. So to check if motion has occurred between two frames of a video, we check for differences between the two images, also known as “diffing”.

We’ve already covered how to draw images onto a <canvas> from a <video>. By default, drawing something onto a <canvas> just covers up whatever was already there, but this can be changed to show you the differences instead.

Here’s an example of an image drawn on top of another with the aforementioned difference setting. Dark pixels indicate areas with little to no motion. Brighter pixels show areas where something moved (in this case, mostly my cat’s head).

Diff image

Scoring

We can see that motion happened on the <canvas>, but how do we turn this into data that we can programmatically evaluate or “score” for motion? The answer is getImageData(). This function returns an ImageData object that has a data property. This property is a long array of numbers, with every chunk of 4 numbers representing a single pixel (red, green, blue, and transparency).

Pixel data

Remember, when diffing, the brighter pixels indicate more difference which means more motion. So the higher the combined red, green, and blue values of a pixel, the more motion that occurred in that pixel. By scoring every pixel like this, we can determine if values are significant enough to consider a capture as having motion.

Here’s a quick algorithm to do this.

Post-Processing the Diff

Not only can we read pixel data from ImageData.data, we can also write adjustments back to it. By tweaking the pixel data this way and then redrawing it with putImageData(), you can essentially do post-processing on the diff image.

I like doing this to make the diff image monochrome green (set red and blue values to 0) and then amplifying the brightness (multiply green values by some constant). This makes it really easy to see where motion is. It also makes it easier to see the ambient visual noise, in case you need to tweak some threshold values to discern between noise and actual motion.

Diff image with post-processing

On top of that, I like to downscale the diff image to 10% of its original width and height. Even a modest 640px by 480px image is a lot of data (307,200 pixels!) to churn through, so downscaling helps lighten the load. Sure, the image becomes super pixelated, but it’s still plenty enough for the purposes of motion detection.

Diff image downscaled

Throttling

One important thing I haven’t covered is throttling. Without throttling, any continuous motion will cause rapid-fire captures to be saved and uploaded to Twitter, 10 times a second. That’s no good.

There are a couple ways to handle this. This is what I went with.

Sequence for throttling

The important thing is the “chilling” state, which is just a cooldown timer. I also added a “considering” state, during which captures can occur continuously, but only the highest scoring capture is kept and uploaded. Hopefully, the highest scoring capture is the most entertaining capture, since it has the most motion.

The Back-End

Even though all the motion detection is done client-side, we still need a back-end since we’re using the Twitter API (can’t have our secret Twitter API key revealed on the front-end).

I won’t go into specifics about the back-end, since it’s not directly related to motion detection. I’ll just mention that I wrote it in node.js, so it still satisfies my “all in JavaScript” goal.

The Results

You can check out the finished web app for yourself. I call it Diff Cam Feed. It’ll ask you to sign in with Twitter so it can upload to your feed (that’s all it does, nothing shady, I promise).

Diff Cam Feed screenshot

I set it up on my laptop for a few trial runs around my apartment. Results were decent. Just my pets, doing pet things.

Various captures of pets

It also works fine on a Raspberry Pi running Raspbian, after installing UV4L.

Diff Cam Feed on a Raspberry Pi

Then I tried it on an old Android phone I had lying around (Samsung Galaxy S4). It works great, as long as you run it in Firefox. For some reason, Chrome on Android dies after a few minutes.

Diff Cam Feed on an Android phone

Conclusion

Overall, this turned out to be a fun side project. The motion detection looks really cool and isn’t very difficult to do. I made it all open source, so you can check out the code in this GitHub repo.

It’s not hard to imagine other uses for motion detection in a web app. If this interests you, then check out Diff Cam Engine. It wraps up a lot of motion detection stuff for you, so you can kickstart your own project.

Lastly, I’ve set up a website with several demos illustrating the concepts I talked about. Feel free to play around with them and check out the code.

Thanks for reading!

JavaScript String Replace Magic

Replacing strings in JavaScript is a fairly common task, but there are some nuances that might surprise the uninitiated, not to mention some really powerful features just below the surface. Read on to see.

String.prototype.replace()

The Simple Case

The most obvious way to do string replacement is by passing 2 strings to replace(), the string to find and the string to replace it with.

replace() does not change the value of str, rather it returns a new string with the applied replacements (this example doesn’t do anything with the returned string, but you’d probably want to).

It’s worth mentioning that this approach is case sensitive. Searching for “Salt” in the above example would come up empty.

Replacing All Occurrences

There is one major caveat: this approach only replaces the first occurrence.

This is often not what you want. To replace all occurrences, you’ll want to use the regex flavored version of replace().

The g flag is crucial. That’s what makes it a global search, finding all occurrences. But what if you want to specify a string to replace via a variable, instead of hardcoding “badger”? It’s a little more typing, but not hard to do with a RegExp object.

We’re dealing with regex now, so don’t forget to escape characters that have special meaning or you’ll get weird syntax errors. Just stick a \ in front of them.

Doing More with Regex

Opening the door to regex lets us do more interesting things. For example, replacing multiple words at once. While we’re at it, let’s solve that case-sensitivity issue from earlier.

The | element lets us create an “or” list of terms. And you probably spotted the i flag, which makes the replacements case insensitive.

Here’s something fancier. The following looks for 3 digit numbers and replaces them with the same number, but with dashes between the digits.

\d is a special regex element that matches any single digit, so we use 3 of those to find 3 digit numbers. Each one is wrapped in parenthesis (), which lets us reference them in the replacement string. $1 is replaced with whatever was matched in the first set of parenthesis, $2 with what’s in the second, and so on.

We’ve barely scratched the surface of what can be done with regex. You can do some amazing things, but that’s an entire post (if not an entire book) of its own.

Replacement Functions

Sometimes you may want to do something completely custom when replacing text, something that not even regex can handle. Our next example replaces fractions with their decimal equivalents.

\d+ means “one or more digits in a row”. We do this twice, both times wrapped in parenthesis, so we can catch the numerator and denominator. In the middle is \/, which is simply an escaped slash /. Put it all together and we’ve got a fraction finder.

This time, instead of a string for the second parameter of replace(), we’re providing a function. This function is called every time a match is found and its return value is what replaces the match. The function takes a couple parameters, though we’re only using 3 in this example. The parameters are, in order:

  • match – The matching string that was found.
  • submatch1, submatch2, … – Each set of parenthesis used will add a submatch parameter. It’s the same concept as the $1, $2, etc. stuff from earlier.
  • offset – How far into the string this match is, in number of characters.
  • string – The entire original string that we called replace() on.

The code within the function does the actual fraction/decimal conversion. It takes the first submatch (numerator) and second submatch (denominator), converts them both to integers, divides them to get the decimal value, then returns that as a string.

We’re just dividing numbers here, but you have the freedom to write whatever custom string replacement code you want. Go nuts.

1 comment » Related topics:

Highlight Text Inside a Textarea

Let’s start with the bad news. You can’t actually highlight text in a <textarea>. Any markup you would add to do so would simply display as plain text within the <textarea>. The good news is, with some carefully crafted CSS and JavaScript, you can fake it.

This article will show you how to pull it off, but if you just want the solution, feel free to go straight to the jQuery plugin: highlight-within-textarea.

Highlight within textarea

Why?

Way back in 2010, I made a site called Regex Storm that would highlight regex matches in a <textarea>, like this. It’s not hard to imagine other applications for this sort of “find and highlight” functionality. 5 years is a long time though, so I decided to revisit the concept and fix it up for modern browsers.

The Plan

The basic idea is to carefully position a <div> behind the <textarea>. JavaScript will be used to copy any text entered into the <textarea> to the <div>. A bit more JavaScript will make that both elements scroll as one. With everything perfectly aligned, we can add markup inside the <div> to highlight text, which will show through the <textarea>, completing the illusion.

A demo should make this all clear. Scroll around and edit the text. Then click Toggle Perspective and edit and scroll some more.

See the Pen Highlight Text Inside a Textarea by Will Boyd (@lonekorean) on CodePen.

HTML

It takes a couple of elements to achieve the layout we need.

The container <div> simply serves as an anchor to position other elements within. The highlights <div> is nested within the backdrop <div>, which may seem like 1 more <div> than we need, but doing it this way fixes some subtle scrolling bugs (which we’ll get to later).

The <mark> tag is the best semantic choice for highlighting text inside the highlights <div>. We’ll be inserting these tags with JavaScript.

CSS

Elements must be laid out with pixel perfection. This can be tricky, as many browsers have very subtle differences that normally aren’t worth caring about, but are very noticeable in this case.

For example, Firefox adds a single pixel of margin to the top and bottom of a <textarea>. And iOS adds a border-radius to <textarea>. So we have to fix these styles (and others I haven’t specifically mentioned).

CSS is used to make text inside the highlights <div> wrap and scroll exactly like the text inside the <textarea>.

It’s also important to set the visibility of things so that our stacked elements display properly together. Remember, the <textarea> sits on top of the highlights <div>.

JavaScript

The JavaScript has 3 main responsibilities.

  1. Keep text in the highlights <div> synced with text in the <textarea>.
  2. Highlight text in the highlights <div> with <mark>.
  3. Make the highlights <div> scroll in tandem with the <textarea>.

Let’s start by binding 2 events.

The input event triggers whenever text in the <textarea> is changed. The callback function takes this text, applies highlights to it, then inserts it into the highlights <div>.

And here’s the applyHighlights() function.

applyHighlights() actually does 2 things. On line 3, it fixes a bug where a trailing carriage return causes the highlights <div> to become misaligned. Then on line 4, it does the actual highlighting by inserting <mark> tags. The regex in this example highlights all capitalized words, but this can be customized.

The scroll callback copies the scroll position of the <textarea> to the backdrop <div>, so that they both scroll in tandem.

Wrap Up

That covers the basic mechanics of (fake) highlighting within a <textarea>. Please note that there are some finer details that this article didn’t discuss and the demo didn’t account for. For the more comprehensive, more bullet-proof solution, check out the highlight-within-textarea jQuery plugin on GitHub.

Ember in Atlanta

Every once in a while I’ll make a “what I’m up to” post. I guess it’s time for one now. Along with some bonus rambling about Ember and Less.

Laptop stickers

Back in Atlanta

We moved out of Seattle and back to Atlanta. Seattle is a good city, really enjoyed living downtown, but in my experience the Seattle Freeze is real — especially when compared to Southern Hospitality. Regardless, we both had good jobs waiting for us in Atlanta, so things worked out nicely. I’m now working for Pardot/Salesforce, still doing front-end development. So far, it’s great.

Ember vs. Backbone

The new gig has me working in Ember instead of Backbone. Ember is a little less mature, still evolving, deciding the best way to do things. It’s fun to watch it evolve, but frustrating when so much seems to change out from under you.

Ember is really big on convention over configuration. It does a lot of “magic” behind the scenes instead of burdening you with explicit declaration of behavior, like you’d more commonly see in Backbone. This is great when you know the nuances, but until you do, a lot of it is not obvious. This contributes to a steeper learning curve.

Ember is more opinionated. I kind of like it that way. While there might be several ways to do something in Backbone, Ember would tell you that there is one right way to do that something.

Take this all with a grain of salt. These are just my opinions so far. I’m still learning Ember.

Less vs. Sass

The new job has me leaving Sass for Less. Honestly, it doesn’t matter. They both do variables, arithmetic, nesting, and mixins just fine, which is 99% of what we need. Leaves me wondering why this stuff isn’t baked into CSS (aside from calc() and Firefox’s syntactically awkward var(--variable) implementation).

Later

I’m working on a little something I hope to share soon. It’s part of a bigger plan, but I don’t want to get too far ahead of myself. Until next time.